How to Run Tests in Random Order to Surface Flakes in CI

Randomizing order surfaces hidden test coupling, and printing the seed lets you replay the exact failing order.

Use pytest-randomly (or Jest test shuffling) to run tests in a random but reproducible order. When a run fails, rerun with the printed seed to reproduce and then fix the coupling.

Steps

• Enable random ordering with a seeded plugin.
• Capture the seed printed on each run.
• Replay the failing seed to reproduce, then fix the dependency.

Randomize and replay (pytest)

pip install pytest-randomly
pytest                       # prints: Using --randomly-seed=931204
pytest -p randomly -p no:cacheprovider --randomly-seed=931204

Shuffle Jest tests

// jest.config.js
module.exports = { randomize: true, seed: 1 };
// or: npx jest --shard=1/1 --seed=1

Gotchas

• Random order that always passes proves nothing; you need the failing seed to fix anything.
• Always log the seed, or a random failure becomes impossible to reproduce.

Related guides