Flaky test or infrastructure failure? How to tell from one red job

By Kaveh Alemi·Latchkey

Telling a flaky test vs infrastructure failure apart comes down to who stopped the job: exit code 137 or 143, a runner that lost communication with the server, a network or registry error, or a full disk points at the machine, while an assertion message, a test runner timeout, or a failure that depends on order or shared state points at the test. Re-running on a fresh machine is the right call for the first group and hides a bug in the second.

One job is red and someone has a finger on the re-run button. Whether that is the right move depends on a single question: did the machine fail, or did the test? The log almost always says, if you know which lines to read.

This page is the decision for one failure. Finding every flaky test in a repository is find flaky tests in GitHub Actions; fixing a test once you know it is flaky is GitHub Actions flaky tests that pass on retry.

Read the failed step from the bottom

Start with the last lines of the step that failed, not the first error you see while scrolling. The last line says how the process ended, and that is the first branch of the tree: a test runner that finished and reported failures exits with its own code, usually 1, while a process stopped from outside exits with 128 plus the signal number.

Then note which step failed. A failure in checkout, dependency install, image pull or browser download happened before a single test ran, so it cannot be a flaky test, whatever it looks like.

# the failed steps only, last lines first
gh run view <run-id> --log-failed | tail -n 40

# which step failed, by name
gh api "repos/{owner}/{repo}/actions/jobs/<job-id>" \
  --jq '.steps[] | select(.conclusion == "failure") | .name'

Signals that point at the machine

These end a job from outside the code. Each has a page in the failures library that covers causes and fixes; the table is the lookup. Exit codes are 128 plus the signal: 137 is SIGKILL (9), 143 is SIGTERM (15).

A canceled step goes through the same ladder on purpose. GitHub documents that the runner sends SIGINT, waits 7500 ms, sends SIGTERM, waits 2500 ms, then kills the process tree, so a 143 on a run someone canceled, or one that hit timeout-minutes, is the runner doing its job.

Signals that point at the test

The test runner finished and is telling you what it saw. An assertion message with expected and received values (expect(received).toBe(expected) in Jest, Vitest and Playwright, AssertionError in Python and Node) is the test disagreeing with the code. A runner-level timeout such as Playwright's Test timeout of 30000ms exceeded. is the test waiting for something that did not happen in time, which on a healthy machine is usually a race or a fixed wait.

The rest show up only across runs. Passing alone and failing in the suite is shared state. Failing in one order and passing in another is order dependence. Failing only under parallel workers is contention between tests. The three probes that separate them are on GitHub Actions flaky tests that pass on retry.

The middle, where both are true

Some failures carry a machine signal and a test cause. A test step killed with 137 hit the machine's memory ceiling, but if peak memory grows every week, a leak in the code under test is the real defect. A test timeout on a two-core runner may be CPU contention from too many workers rather than a race. An ECONNRESET raised inside a test that calls a real external service is a test design problem: the test depends on a network it does not control.

The tie-breaker is a fresh machine. Re-run the job, which on GitHub-hosted runners means a new virtual machine. The same failure in the same place again is deterministic, so read it as a bug. A failure that moves or vanishes was environmental or flaky, and the signal above tells you which.

# re-run only the failed jobs, on new machines, and wait for the result
gh run rerun <run-id> --failed
gh run watch <run-id> --exit-status && echo "passed on a fresh machine: environmental or flaky"

# still red: compare where each attempt stopped
gh run view <run-id> --attempt 1 --log-failed | tail -n 20
gh run view <run-id> --attempt 2 --log-failed | tail -n 20

The tree as a script

The script reads the failed steps of one run and prints the first machine signal it finds, else the first test signal, else asks you to read the log. The machine check runs first on purpose: a test that fails after the runner ran out of disk is not evidence about the test.

It is a first pass, not a verdict. Extend the patterns with the messages your own stack prints, and keep the output with the run so the decision is recorded next to the evidence.

#!/usr/bin/env bash
# Usage: triage.sh RUN_ID   (inside the repository, or with GH_REPO set)
log=$(gh run view "$1" --log-failed)

machine='exit code 137|exit code 143|lost communication with the server|received a shutdown signal|No space left on device|ECONNRESET|EAI_AGAIN|ETIMEDOUT|429 Too Many Requests'
in_test='AssertionError|expect\(received\)|Test timeout of [0-9]+ms exceeded'

if sig=$(grep -m1 -oE "$machine" <<<"$log"); then
  echo "machine: $sig"
elif sig=$(grep -m1 -oE "$in_test" <<<"$log"); then
  echo "test: $sig"
else
  echo "unclassified: read the failed step"
fi

Key takeaways

• A process stopped from outside exits 128 plus the signal: 137 is SIGKILL, 143 is SIGTERM.
• A failure before the test step (checkout, install, pull, browser download) cannot be a flaky test.
• Assertion messages and runner timeouts point at the test; order, shared state and parallelism confirm it across runs.
• When both are true, re-run on a fresh machine: a failure that repeats in place is a bug.

Frequently asked questions

Related guides

References

• GitHub Docs: workflow cancellation reference (read 2026-11-12)
• GitHub Docs: GitHub-hosted runners
• GitHub CLI manual: gh run view
• Bash reference manual: exit status (128 plus the signal number)
• GitHub Actions documentation