# Flaky Tests in CI: Causes, Detection, and Containment

> What makes tests flaky, how to detect flakiness, and how to contain it with quarantines and bounded retries without hiding real bugs.

Source: https://latchkey.dev/learn/ci-cd-concepts/flaky-tests-in-ci  
Updated: 2026-06-25

A flaky test passes and fails on the same code. Left unmanaged, flakiness erodes trust in CI until a red build means nothing.

Flaky tests are nondeterministic: their result depends on something other than the code under test. They are corrosive because they train teams to ignore failures - the opposite of what CI is for.

## Common causes

- Race conditions and timing assumptions (sleep-based waits).
- Test-order dependence and shared mutable state.
- Reliance on real networks, clocks, or external services.
- Resource pressure (a slow, loaded runner missing a timeout).

## Detecting flakiness

Track per-test pass/fail history across runs. A test that fails and then passes on the same commit is flaky by definition. Re-running the whole suite repeatedly on a known-good commit surfaces the worst offenders.

## Containing it without hiding bugs

Quarantine known-flaky tests so they do not block merges, file tickets to fix the root nondeterminism, and apply bounded retries only to tests known to be flaky. The principle: retry the noise, never the signal - a consistently reproducing failure must always surface.

## FAQ

### What is Flaky tests in CI?

Flaky tests are nondeterministic: their result depends on something other than the code under test. They are corrosive because they train teams to ignore failures - the opposite of what CI is for.

### Detecting flakiness?

Track per-test pass/fail history across runs. A test that fails and then passes on the same commit is flaky by definition. Re-running the whole suite repeatedly on a known-good commit surfaces the worst offenders.

### Containing it without hiding bugs?

Quarantine known-flaky tests so they do not block merges, file tickets to fix the root nondeterminism, and apply bounded retries only to tests known to be flaky. The principle: retry the noise, never the signal - a consistently reproducing failure must always surface.

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt