Flaky Tests in CI: Causes, Detection, and Containment
A flaky test passes and fails on the same code. Left unmanaged, flakiness erodes trust in CI until a red build means nothing.
Flaky tests are nondeterministic: their result depends on something other than the code under test. They are corrosive because they train teams to ignore failures - the opposite of what CI is for.
Common causes
- Race conditions and timing assumptions (sleep-based waits).
- Test-order dependence and shared mutable state.
- Reliance on real networks, clocks, or external services.
- Resource pressure (a slow, loaded runner missing a timeout).
Detecting flakiness
Track per-test pass/fail history across runs. A test that fails and then passes on the same commit is flaky by definition. Re-running the whole suite repeatedly on a known-good commit surfaces the worst offenders.
Containing it without hiding bugs
Quarantine known-flaky tests so they do not block merges, file tickets to fix the root nondeterminism, and apply bounded retries only to tests known to be flaky. The principle: retry the noise, never the signal - a consistently reproducing failure must always surface.
Key takeaways
- Flaky = nondeterministic; the result depends on something other than the code.
- Detect via per-test history; a fail-then-pass on one commit is the tell.
- Contain with quarantine + targeted retries, and fix the root cause.
- Never blanket-retry everything - that hides real regressions.