Self-Healing CI: Containing Flaky Tests Without Hiding Real Bugs

A flaky test passes and fails on the same commit. The art of automating around it is retrying the genuinely-transient failure without ever hiding a real regression.

The problem

A test suite fails on a test that passes when re-run, with no code change. Timing, ordering, shared state, or an external dependency made it nondeterministic. Teams re-run the job and it goes green.

Why it happens

Flaky tests depend on something nondeterministic: a race, a clock, test-execution order, or a network call. The same commit can pass or fail depending on conditions unrelated to correctness.

Blindly retrying every failure is dangerous - it can mask a real, intermittent bug. The goal is to contain known flakiness, not to paper over regressions.

The manual fix

Manual handling usually looks like:

Re-run the job and confirm the failure does not reproduce.
Quarantine or mark the test as flaky and file a ticket to fix the nondeterminism.
Add a bounded, test-level retry only for tests known to be flaky.

How this gets automated

Self-healing CI treats flakiness conservatively: it identifies failures that match a transient, non-deterministic signature and retries those specifically, while leaving consistent, reproducible failures to surface as real. The point is not to retry everything - it is to remove the noise of known-transient failures so a red build reliably means something is actually wrong.