Self-Healing CI: Containing Tests That Fail Only Under Parallel Load
A test that passes alone but fails when run alongside many others is reacting to contention, not necessarily a logic error -- and a clean retry often clears it.
The problem
A test fails intermittently only when the suite runs at high parallelism -- it passes in isolation or on re-run. Contention for CPU, I/O, or a shared fixture made it slow or out of order on one crowded run. A human re-runs the suite and it goes green with no code change.
TimeoutError: test exceeded 5000ms (suite running with --maxWorkers=16)
# passes when run alone or with --runInBandWhy it happens
Running many tests at once multiplies pressure on shared resources -- CPU schedulers, disk, sockets, ephemeral ports, and shared fixtures -- so a test that is fine alone can time out or interleave badly under load even though its logic is correct.
The contention is conditional and transient, but blindly retrying everything is risky: a test that fails because of a genuine ordering bug must still surface rather than be papered over.
The manual fix
Manual handling for parallel-load flakiness:
- Re-run the suite and confirm the failure does not reproduce.
- Lower worker count or isolate the contended test so it does not share state under load.
- Add a bounded retry only for tests known to be load-sensitive, and fix the underlying contention.
How this gets automated
Self-healing CI treats load-driven flakiness conservatively: it retries failures whose signature matches transient contention (a timeout or resource error only under high parallelism) while leaving consistent, reproducible failures to surface as real. The aim is to remove load noise without ever masking a genuine concurrency or ordering bug.