pytest-randomly Reorders Tests and Surfaces Order-Dependent Failures in CI

pytest-randomly shuffles test order each run (and reseeds random/numpy). A test that passes in file order but fails when shuffled is depending on shared state or another test running first - a real bug the shuffle exposed.

What this error means

CI fails intermittently with a test that passes locally in default order. The header prints Using --randomly-seed=N, and re-running with that exact seed reproduces the failure deterministically - pointing at order coupling, not flakiness.

Using --randomly-seed=305419896
...
tests/test_cache.py::test_hit FAILED   # passes in file order, fails shuffled
AssertionError: assert cache.get('k') == 'v'   # a prior test populated it

Common causes

Shared mutable state between tests

A module-level global, a class attribute, a singleton, or a real cache mutated by one test leaks into another. In file order it happens to work; shuffled, it does not.

Implicit fixture-ordering assumption

A test relies on another test (or a session fixture side effect) having run first. Randomized order removes that guarantee.

How to fix it

Reproduce with the printed seed, then isolate state

Pin the seed to reproduce, then make each test set up and tear down its own state.

pytest -p randomly --randomly-seed=305419896
# fix: reset shared state per test
@pytest.fixture(autouse=True)
def _clear_cache():
    cache.clear()
    yield
    cache.clear()

Make tests independent of order

1. Replace shared globals/singletons with fixtures scoped to the test.
2. Never depend on another test having populated state.
3. Keep the shuffle on in CI so coupling is caught, not hidden.

How to prevent it

• Keep tests independent - no reliance on execution order or leaked state.
• Use function-scoped fixtures that reset shared resources.
• Run pytest-randomly in CI and reproduce failures with the printed seed.

Related guides