GitHub Actions flaky tests that pass on retry

By Kaveh Alemi·Latchkey

A GitHub Actions flaky tests retry policy is worth exactly what the classification behind it is worth: tests that pass on the second attempt depend on something other than the code under test, and a blanket retry hides which something. Measure the rate first, classify the failure into one of four causes, and retry only the one class where a retry is honest.

What this error means

The same commit produces two different results. Attempt one fails on a test that asserts something ordinary, attempt two passes, and the diff between them is empty because there is no diff: the run number changed and nothing else. In the run history the two attempts sit under one workflow run, which is the clearest evidence you will get, because a flaky failure is only visible as a pair. The failure itself is usually unremarkable, an assertion off by one element or a timeout waiting for an element that did appear, which is why it reads as noise rather than as a defect with a cause.

Run attempt 1  conclusion: failure   e2e (ubuntu-latest)
Run attempt 2  conclusion: success   e2e (ubuntu-latest)
Head SHA 9f1c2ab, unchanged between attempts

Measure the rate before you fix anything

You cannot tell whether flakiness is improving without a number, and the number is almost always worse than the estimate. The cheapest proxy is the re-run rate: what share of recent runs needed a second attempt. It takes one command and it gives you a baseline that survives arguments.

Then find out which jobs are producing it. A flake rate spread evenly across a suite is a resource or environment problem; a flake rate concentrated in four tests is four tests.

Keep the number somewhere visible. In our experience a rate near one percent reads as a tax, and at several percent people stop reading red builds, at which point a real regression is indistinguishable from noise. Those thresholds are ours rather than a standard; the trend is the part that matters.

# what share of recent runs are re-runs?
gh run list --limit 200 --json databaseId --jq '.[].databaseId' |
while read -r id; do gh api "/repos/{owner}/{repo}/actions/runs/$id" --jq '.run_attempt'; done |
awk '{t++; if ($1>1) r++} END {printf "re-run rate: %.1f%% (%d of %d)\n", 100*r/t, r, t}'

# which jobs fail most often?
gh run list --limit 100 --json databaseId --jq '.[].databaseId' |
while read -r id; do
  gh api "/repos/{owner}/{repo}/actions/runs/$id/jobs" \
    --jq '.jobs[] | select(.conclusion=="failure") | .name'
done | sort | uniq -c | sort -rn | head

Common causes

Shared state between tests

The largest category in most suites. A test leaves a row in the database, a file in a temporary directory, a stubbed global or a mocked timer, and the next test to read it fails. It passes alone and fails in CI, where the whole suite runs in one process. The isolation is at fault, not the environment.

You added a retry and the failure rate went up

This is what the obvious fix does. Retrying a shared-state bug or a race makes the red build go away and leaves the defect in place, and because the retry is not recorded the real failure rate becomes invisible. Worse, the retried test still fails sometimes, so the flake budget is spent on the same tests forever while the suite accumulates more of them.

Order dependence and races

A test that depends on running after another, or a fixed sleep standing in for a condition that usually completes in time. CI is slower and more contended than a laptop, so a margin that held locally does not hold. A shuffled run with a recorded seed turns both into something you can reproduce on demand.

The runner is not your laptop

Fewer cores, less memory, no TTY, a different timezone and locale, a cold cache. A suite that runs four workers on an eight core machine runs four workers on a two core runner, and timing-sensitive tests fail under the contention that follows. Pin TZ, LANG and the worker count.

Genuinely transient infrastructure

A registry timeout, a DNS blip, a reclaimed machine, a rate-limited pull. The one class where the failure has nothing to do with your code and a bounded retry is correct. It is also the smallest class in most repositories, which is why retrying everything is a bad trade.

How to fix it

Classify before you touch the test

1. Run the failing test alone. Passing alone and failing in the suite is shared state.
2. Shuffle with a fixed seed and replay it. Reproducing on one ordering is order dependence.
3. Loop it thirty times. A couple of failures with no pattern is a race; failures only under parallelism are resource contention.

Fix shared state at the boundary, not in the test

Give each test its own database schema or transaction that rolls back, its own temporary directory, and a fresh set of globals. A fixture that resets state is better than a test that tolerates it, because the next test written against the same boundary inherits the isolation for free.

- run: npx vitest run --isolate --pool=forks --poolOptions.forks.maxForks=2
  env:
    TZ: UTC
    LANG: C.UTF-8

Quarantine what you cannot fix today, and put a clock on it

A known-flaky test should stop gating merges without being deleted, because deleting it loses the coverage it provides on the days it works. Move it to a non-blocking job that still runs and still reports, list what is in quarantine somewhere people see, and require a fix-or-delete decision when the cap expires. A quarantine without an expiry is a graveyard.

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - run: npm test -- --testPathIgnorePatterns=quarantine

  quarantined:
    runs-on: ubuntu-latest
    continue-on-error: true      # reports, never blocks
    steps:
      - run: npm test -- --testPathPattern=quarantine

Retry the transient class only, with a bound and a record

Where the failure is genuinely infrastructure, retry the step rather than the assertion, cap the attempts, and make the retry leave a trace. A retry nobody records is a measurement destroyed.

- uses: nick-fields/retry@v4
  with:
    max_attempts: 3
    timeout_minutes: 10
    retry_on: error            # not on test assertion failures
    command: npm run test:integration
- if: always()
  run: echo "attempt=${{ github.run_attempt }}" >> "$GITHUB_STEP_SUMMARY"

Three probes that name the cause

Run the test alone. If it passes by itself and fails in the suite, another test left state behind: a database row, a global, a mocked clock, a temporary file. That is shared state, and a retry fixes it about half the time, which is the worst outcome because it makes the bug intermittent instead of visible.

Shuffle the suite with a fixed seed, then replay the same seed. A failure that reproduces on one ordering and not another is order dependence, and the seed gives you a deterministic reproduction to work from.

Run it thirty times and count. A failure landing a couple of times in thirty with no pattern is a race, usually a fixed sleep standing in for a condition. One that appears only under parallelism is resource contention, the fourth cause.

# shared state: does it pass alone?
<runner> path/to/one.test

# order dependence: shuffle, then replay the same seed
<runner> --shuffle --seed 12345

# race: repeat and count
for i in $(seq 1 30); do <runner> path/to/one.test > /tmp/flake-$i.log 2>&1 || echo "fail $i"; done

# resource: watch what it peaks at
/usr/bin/time -v <runner> 2>&1 | grep -E "Maximum resident|Exit status"

What Latchkey does, and the rule it holds itself to

Latchkey does re-run a failed workflow automatically, and the gate on it is the interesting part. From our own audit of failed CI runs: "A test is only re-run when our own analytics have already flagged that exact workflow as one that passes on retry." Not a class of failure, not a heuristic on the error text, that workflow, from its own history. And when it does re-run, "it runs exactly once".

The sentence that makes it a rule rather than a feature is the next one: "Without that evidence, a failing test stays red." A tool that guesses when it does not know is worse than no tool, because it spends your trust rather than its own.

That is deliberately narrow, and this page is the reason. Three of the four causes below are defects a retry converts into a rate, and no engine can tell from a log which one it is looking at. Evidence that this workflow has passed on retry before is the only honest license to try again, and once is the only honest number of attempts.

How to prevent it

• Publish the re-run rate somewhere the team sees it, and treat a rise as a defect rather than as weather.
• Isolate test state at the fixture boundary so new tests inherit the isolation.
• Pin TZ, LANG and worker counts in CI so the runner is a known environment rather than a surprise.
• Shuffle the suite in CI with a recorded seed, so order dependence is found by the pipeline rather than by a release.
• Cap how long a test may stay quarantined, and enforce the cap.

Frequently asked questions

Related guides

References

• Latchkey: we audited 156,808 failed CI runs, and the rule for re-running one
• GitHub Actions: re-running workflows and jobs, and run_attempt
• GitHub-hosted runners: standard runner specifications
• Workflow syntax: continue-on-error and job-level controls
• GitHub Actions documentation