How to Reduce CI Flakiness Without Just Adding Retries
A blanket retry turns a visible problem into an invisible one. Measure the flake rate first, classify what is actually failing, and retry only the class that is genuinely transient.
Flakiness is expensive in two ways that compound: you pay for the failed run and the re-run, and every red build that turns out to be noise makes the next real failure slightly less believable. The second cost is the one that eventually breaks a team.
The instinct is to add a retry. That is correct for exactly one of the four causes below and actively harmful for the others, because it converts a defect you could have found into background noise you have learned to ignore.
Measure the rate before you fix anything
You cannot tell whether flakiness is improving without a baseline, and the number is usually worse than people estimate.
# what share of recent runs are retries of a failed run?
gh run list --limit 200 --json databaseId --jq '.[].databaseId' |
while read -r id; do gh api "/repos/{owner}/{repo}/actions/runs/$id" --jq '.run_attempt'; done |
awk '{t++; if ($1>1) r++} END {printf "re-run rate: %.1f%% (%d of %d)\n", 100*r/t, r, t}'
# which jobs fail most often?
gh run list --limit 100 --json databaseId --jq '.[].databaseId' |
while read -r id; do
gh api "/repos/{owner}/{repo}/actions/runs/$id/jobs" \
--jq '.jobs[] | select(.conclusion=="failure") | .name'
done | sort | uniq -c | sort -rn | headThe four causes, and what each one needs
| Cause | How it presents | Correct response |
|---|---|---|
| Genuinely transient infrastructure | Registry 5xx, DNS blip, spot reclaim | Retry. This is the one retries are for |
| Shared state between tests | Passes alone, fails in the suite | Isolate the state. A retry hides it |
| Race conditions and timing | Fails ~1 run in 20, no pattern | Fix the synchronisation. A retry hides it |
| Resource exhaustion | Exit 137, disk full, timeouts under load | Raise limits or reduce parallelism |
Prove which one you have
# shared state: does it pass alone?
<runner> path/to/one.test
# order dependence: shuffle the suite
<runner> --shuffle --seed 12345 # then replay with the same seed
# race: run it repeatedly and count
for i in $(seq 1 30); do <runner> path/to/one.test >/dev/null 2>&1 || echo "fail $i"; done
# resource: watch while it runs
/usr/bin/time -v <runner> 2>&1 | grep -E "Maximum resident|Exit status"Quarantine, do not delete
A test that fails intermittently and cannot be fixed today should be quarantined into a non-blocking job rather than deleted or retried in place. Deleting loses the coverage; retrying in place hides the trend.
- Move it to a job that runs but does not gate merges.
- Record why it was quarantined and by when it will be fixed. Quarantine without an expiry becomes deletion with extra steps.
- Track the quarantine list size as a number the team sees. A growing list is a signal; an invisible one is a graveyard.
Retry correctly when you do retry
# retry the transient class only, at the step level, with a bound
- uses: nick-fields/retry@v3
with:
max_attempts: 3
timeout_minutes: 10
retry_on: error # not on test assertion failures
command: npm run test:integration
# and always record that a retry happened, or your real
# failure rate becomes invisibleMeasure before you optimise
Pipeline optimisation usually targets the step people assume is slow. Get the real per-step timings first, because the answer is frequently dependency install or a cold cache rather than the build itself.
# per-job timings for the last 20 runs
gh run list --limit 20 --json databaseId,conclusion,createdAt,updatedAt \
--jq '.[] | "\(.conclusion)\t\(.createdAt)\t\(.updatedAt)"'
# per-step timing inside one run
gh run view <run-id> --log | grep -E "^\S+\s+.*Run |##\[group\]" | head -40Frequently asked questions
What is an acceptable CI flake rate?
Should I just retry flaky tests automatically?
How do I know if a test is flaky or genuinely broken?
Why do tests fail in CI but pass locally?
TZ and LANG and reproduce with reduced parallelism.