What Is a Flaky Test? Causes and Fixes Explained
A flaky test is one that returns different results on the same code and the same inputs, passing one run and failing the next for no real reason.
Flaky tests are the most corrosive kind of CI failure because they are not reproducible on demand. The code did not change, the test did not change, yet the result flips. Over time, developers learn to ignore red builds, and a real regression slips through unnoticed.
What "flaky" actually means
A test is flaky when its outcome depends on something other than the behavior it claims to verify. Run it a hundred times and it passes ninety-seven; the three failures are not telling you the code is broken. The signal-to-noise ratio of your test suite drops, and so does confidence in every result it produces.
Common causes
- Timing and async: a test asserts before an operation finishes, so it depends on how fast the machine happens to be.
- Shared state: tests leak data between each other, so order or parallelism changes the result.
- External dependencies: a real network call, clock, or random seed introduces nondeterminism.
- Resource pressure: a slow or memory-starved runner makes an otherwise-fine test time out.
Flaky tests vs transient infrastructure failures
It is worth separating two things that both look like flakiness. A flaky test is a problem in the test itself, fixable only by changing the test or the code. A transient infrastructure failure (a network blip, a registry timeout, an out-of-memory kill on the runner) is environmental and fixable by retrying. They demand different responses.
How teams contain flakiness
The durable fix is to make tests deterministic: remove real I/O, control time and randomness, and isolate state. Until then, teams quarantine known-flaky tests so they do not block merges, and track a flaky-test rate to keep the problem visible rather than ignored.
The Latchkey angle
Latchkey self-healing managed runners cannot fix a genuinely flaky test for you, but they do remove the environmental noise that masquerades as flakiness: when a job fails on a transient or mechanical issue such as a network blip or an out-of-memory kill, the runner detects it and retries automatically, so a one-off blip does not fail your build. That leaves a smaller, clearer set of failures that are actually test bugs.
Key takeaways
- A flaky test passes and fails on identical code and inputs.
- Causes are usually timing, shared state, external dependencies, or resource pressure.
- Flaky tests (test bugs) differ from transient infrastructure failures (retryable).
- Make tests deterministic; quarantine and measure the rest.