# How to Reduce CI Flakiness Without Just Adding Retries

> Measure your flake rate, classify failures into the four causes that actually occur, and fix them at source. Includes when a retry is correct and when it hides a real defect.

Source: https://latchkey.dev/learn/optimize-ci/how-to-reduce-ci-flakiness  
Updated: 2026-08-20

A blanket retry turns a visible problem into an invisible one. Measure the flake rate first, classify what is actually failing, and retry only the class that is genuinely transient.

Flakiness is expensive in two ways that compound: you pay for the failed run and the re-run, and every red build that turns out to be noise makes the next real failure slightly less believable. The second cost is the one that eventually breaks a team.

The instinct is to add a retry. That is correct for exactly one of the four causes below and actively harmful for the others, because it converts a defect you could have found into background noise you have learned to ignore.

## Measure the rate before you fix anything

You cannot tell whether flakiness is improving without a baseline, and the number is usually worse than people estimate.

```Terminal
# what share of recent runs are retries of a failed run?
gh run list --limit 200 --json databaseId --jq '.[].databaseId' |
while read -r id; do gh api "/repos/{owner}/{repo}/actions/runs/$id" --jq '.run_attempt'; done |
awk '{t++; if ($1>1) r++} END {printf "re-run rate: %.1f%% (%d of %d)\n", 100*r/t, r, t}'

# which jobs fail most often?
gh run list --limit 100 --json databaseId --jq '.[].databaseId' |
while read -r id; do
  gh api "/repos/{owner}/{repo}/actions/runs/$id/jobs" \
    --jq '.jobs[] | select(.conclusion=="failure") | .name'
done | sort | uniq -c | sort -rn | head
```

## The four causes, and what each one needs

| Cause | How it presents | Correct response |
| --- | --- | --- |
| Genuinely transient infrastructure | Registry 5xx, DNS blip, spot reclaim | Retry. This is the one retries are for |
| Shared state between tests | Passes alone, fails in the suite | Isolate the state. A retry hides it |
| Race conditions and timing | Fails ~1 run in 20, no pattern | Fix the synchronisation. A retry hides it |
| Resource exhaustion | Exit 137, disk full, timeouts under load | Raise limits or reduce parallelism |

> Only the first row is safely retryable. Retrying the other three converts a reproducible defect into an intermittent one, which is strictly harder to fix later.

## Prove which one you have

```Terminal
# shared state: does it pass alone?
<runner> path/to/one.test

# order dependence: shuffle the suite
<runner> --shuffle --seed 12345   # then replay with the same seed

# race: run it repeatedly and count
for i in $(seq 1 30); do <runner> path/to/one.test >/dev/null 2>&1 || echo "fail $i"; done

# resource: watch while it runs
/usr/bin/time -v <runner> 2>&1 | grep -E "Maximum resident|Exit status"
```

> Runners have fewer cores than a laptop, so timing-sensitive tests fail under contention that never occurs locally. Reproduce with reduced parallelism before concluding a test is not flaky.

## Quarantine, do not delete

A test that fails intermittently and cannot be fixed today should be quarantined into a non-blocking job rather than deleted or retried in place. Deleting loses the coverage; retrying in place hides the trend.

- Move it to a job that runs but does not gate merges.
- Record why it was quarantined and by when it will be fixed. Quarantine without an expiry becomes deletion with extra steps.
- Track the quarantine list size as a number the team sees. A growing list is a signal; an invisible one is a graveyard.

## Retry correctly when you do retry

```.github/workflows/ci.yml
# retry the transient class only, at the step level, with a bound
- uses: nick-fields/retry@v3
  with:
    max_attempts: 3
    timeout_minutes: 10
    retry_on: error          # not on test assertion failures
    command: npm run test:integration

# and always record that a retry happened, or your real
# failure rate becomes invisible
```

> Never retry a whole workflow blindly. Retry the specific step whose failure mode is transient, keep the attempt count bounded, and make retries visible in your metrics.

## Measure before you optimise

Pipeline optimisation usually targets the step people assume is slow. Get the real per-step timings first, because the answer is frequently dependency install or a cold cache rather than the build itself.

```Terminal
# per-job timings for the last 20 runs
gh run list --limit 20 --json databaseId,conclusion,createdAt,updatedAt \
  --jq '.[] | "\(.conclusion)\t\(.createdAt)\t\(.updatedAt)"'

# per-step timing inside one run
gh run view <run-id> --log | grep -E "^\S+\s+.*Run |##\[group\]" | head -40
```

> Compare a cold-cache run against a warm one. If most of the difference is install time, caching is the win; if it is not, caching will change nothing and the build itself needs the attention.

## FAQ

### What is an acceptable CI flake rate?

Below one percent of runs is generally tolerable; above five percent people stop trusting red builds, which is the expensive failure mode. Measure your re-run rate to get the number rather than estimating it.

### Should I just retry flaky tests automatically?

Only for genuinely transient infrastructure failures such as registry timeouts or a reclaimed instance. Retrying shared-state bugs, races, or resource exhaustion hides a reproducible defect and makes it harder to fix later.

### How do I know if a test is flaky or genuinely broken?

Run it in isolation, then run the suite with a shuffled order, then run it repeatedly. Passing alone but failing in the suite is shared state; failing roughly one run in twenty with no pattern is a race; failing consistently is not flakiness at all.

### Why do tests fail in CI but pass locally?

Runners have fewer cores, no TTY, a different locale and timezone, less memory, and a cold cache. Timing-sensitive tests fail under contention, and snapshot tests containing formatted dates fail on a timezone difference. Pin `TZ` and `LANG` and reproduce with reduced parallelism.

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt