pytest-randomly Exposes Order-Dependent Test Failures in CI

pytest-randomly shuffles test order each run. A test that passes in file order but fails when shuffled depends on hidden shared state another test set up - the plugin surfaces a real bug, it does not cause one.

What this error means

A test passes locally but fails intermittently in CI, and the run prints Using --randomly-seed=<N>. Re-running with the same seed reproduces it; a different seed may pass - the classic signature of order-dependent, state-leaking tests.

pytest output

Using --randomly-seed=305418216

tests/test_b.py::test_reads_cache FAILED
# passes when run alone, fails after test_a mutates a shared/global value

Common causes

Shared/global state leaks between tests

A module-level variable, singleton, cache, or env var mutated by one test is read by another. In file order it happened to work; shuffled, the dependency breaks.

Fixture scope too broad / not reset

A session/module-scoped fixture carries state across tests that assume a clean slate, so order changes the outcome.

How to fix it

Reproduce with the printed seed, then isolate

Pin the seed to reproduce deterministically while you find the leak.

Terminal

pytest -p randomly --randomly-seed=305418216
# bisect to the pair: run the two tests together
pytest tests/test_a.py tests/test_b.py -p randomly --randomly-seed=305418216

Fix the shared state, do not just disable shuffling

Reset global/singleton/env state in a fixture teardown or autouse fixture.
Narrow fixture scope (function scope) where tests assume a clean slate.
Avoid module-level mutable state; build fresh objects per test.

How to prevent it

Keep tests independent - no reliance on execution order.
Reset shared/global state between tests via fixtures.
Keep pytest-randomly enabled so order dependence surfaces early.