# Find flaky tests in GitHub Actions from data you already have

> Find flaky tests in GitHub Actions from run attempts, the REST API and JUnit XML, with scripts that list every job and test that flipped on a commit.

Source: https://latchkey.dev/learn/flaky-tests/find-flaky-tests-in-github-actions  
Updated: 2026-11-12

To find flaky tests in GitHub Actions, look for one pattern: the same commit producing both a failure and a pass. GitHub already records it per job as run attempts, and your test runner records it per test in JUnit XML, so two short scripts can list every job and every test that flipped.

Everyone on the team knows the suite is flaky. Nobody has a list. The re-run button gets pressed, the build goes green, and the evidence that something disagreed with itself sits in the run history where nobody reads it.

That evidence is already stored. This page shows where: the run attempt number, four REST endpoints, and the JUnit file your test runner can write. It ends with a script for jobs and one for tests, both run against real data before they were printed here.

## Flaky has a definition you can query

A test is flaky when the same code gives two different answers. In GitHub Actions the cleanest form of that evidence is a re-run: GitHub documents that re-runs use the same `GITHUB_SHA` and `GITHUB_REF` as the original event, so a job that failed on attempt 1 and passed on attempt 2 ran the same commit twice and disagreed with itself.

Three identifiers carry it. The run id stays fixed across re-runs, the attempt number goes up by one each time, and the commit SHA also ties together separate runs of the same commit, such as a push run and a pull request run.

[GitHub Actions flaky tests that pass on retry](/learn/failures/flaky-tests-passing-on-retry-in-ci) measures how often this happens across a repository and covers quarantine. This page names the jobs and the tests.

| Identifier | Where you read it | On a re-run |
| --- | --- | --- |
| `github.run_id` | Workflow context; `id` on the REST run object | Unchanged |
| `github.run_attempt` | Workflow context; `run_attempt` on REST run and job objects | Starts at 1, plus one per re-run |
| `GITHUB_SHA` | Environment; `head_sha` on REST run and job objects | Unchanged |

> From the contexts reference and the re-running workflows and jobs page on docs.github.com, read 2026-11-12. A run can be re-run at most 50 times, and only up to 30 days after the initial run.

## The endpoints that hold every attempt

Four REST endpoints are enough. The trap is the jobs list: its `filter` parameter defaults to `latest`, which returns only the jobs from the most recent execution. A run that failed and was re-run green looks clean under the default. Pass `filter=all` to get every attempt, each job carrying its own `run_attempt` and `conclusion`.

After a failed-jobs re-run, the jobs that had already passed reappear under the new attempt number with their original start times. Grouping by job name and sorting by attempt handles that without a special case.

```Terminal
# default filter=latest: only the jobs of the last attempt
gh api "repos/OWNER/REPO/actions/runs/RUN_ID/jobs" \
  --jq '.jobs[] | "\(.run_attempt) \(.conclusion) \(.name)"'

# filter=all: every attempt, so a failure followed by a pass shows up
gh api "repos/OWNER/REPO/actions/runs/RUN_ID/jobs?filter=all&per_page=100" \
  --jq '.jobs[] | "\(.run_attempt) \(.conclusion) \(.name)"'
```

| Endpoint | What it returns |
| --- | --- |
| `GET /repos/{owner}/{repo}/actions/runs?head_sha={sha}` | Every workflow run for one commit, each with `run_attempt` |
| `GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt_number}` | The run as it was on one attempt |
| `GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs?filter=all` | Jobs from every attempt; the default `latest` hides the earlier ones |
| `GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt_number}/jobs` | Jobs for one attempt only |

> Endpoints and the `filter` default from the REST reference for workflow runs and workflow jobs, read 2026-11-12. The reappearing jobs were observed on a public run with a failed-jobs re-run on the same date.

## List the jobs that failed then passed on one commit

The script takes a repository and a SHA, finds every run on that commit that was re-run, pulls all of its jobs, and prints each job whose last attempt passed after an earlier one failed. It needs only `gh` and `jq`.

Run against `microsoft/vscode` at commit `0dcccc4` on 2026-11-12, it printed two jobs: `Linux / Electron-Smoke (shard 3/3)` and `PR Gate`, both `1:failure 2:success`. The second is a gate that waits on the first and failed only because its dependency did. Read the output from the top of the dependency chain, or you will count one flake twice.

Loop it over the SHAs from `gh run list --json headSha` for the last week and you have flaky jobs by name rather than a rate.

```Terminal
#!/usr/bin/env bash
# Usage: flaky-jobs.sh OWNER/REPO [SHA]
set -euo pipefail
repo=$1
sha=${2:-$(git rev-parse HEAD)}

gh api "repos/$repo/actions/runs?head_sha=$sha&per_page=100" --paginate \
  --jq '.workflow_runs[] | select(.run_attempt > 1) | .id' |
while read -r run; do
  gh api "repos/$repo/actions/runs/$run/jobs?filter=all&per_page=100" --paginate \
    --jq '.jobs[] | {run: .run_id, job: .name, attempt: .run_attempt, conclusion}'
done |
jq -rs 'group_by([.run, .job])[]
  | sort_by(.attempt)
  | select(.[-1].conclusion == "success" and any(.[]; .conclusion == "failure"))
  | "\(.[0].run)\t\(.[0].job)\t" + (map("\(.attempt):\(.conclusion)") | join(" "))'
```

## Go below the job: one JUnit report per attempt

A job name tells you where to look, not which test. For that you need the test runner's own record, and nearly every runner can write JUnit XML: pytest with `--junitxml`, Playwright with its `junit` reporter, Maven Surefire by default. Upload it on every outcome with the attempt number in the artifact name.

The name matters. Artifact names must be unique within a run, and the run keeps the uploads from earlier attempts, so `${{ github.run_attempt }}` in the name gives each attempt its own file instead of a collision on the re-run. In a matrix job, add the matrix values too.

Check one thing before trusting the files: retries inside the runner. If the runner retries a failing test and it passes, its JUnit output can record a plain pass. Playwright does exactly that unless you turn on `includeRetries`, as [flaky Playwright tests in CI](/learn/flaky-tests/flaky-playwright-tests-in-ci) shows; Surefire writes `flakyFailure` elements instead.

```.github/workflows/ci.yml
steps:
  - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  # set up your language here
  - run: python -m pytest --junitxml=reports/junit.xml
  - if: ${{ !cancelled() }}
    uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
    with:
      name: junit-${{ github.run_attempt }}-${{ github.job }}
      path: reports/junit.xml
      retention-days: 14
```

> Artifact naming rule from the actions/upload-artifact README at v7.0.2. The run-level artifact list was checked on a public run on 2026-11-12: artifacts uploaded on attempts 1 and 2 were both listed and both downloaded with `gh run download`.

## Compare the attempts test by test

`gh run download` with a pattern puts each matching artifact in its own directory named after the artifact. The Python below walks those directories, reads the attempt number from the name, and prints each test that both passed and failed. It uses only the standard library.

Download several runs of the same SHA into one directory and it still works, because the definition does not care which run produced the disagreement. We checked it against two Playwright JUnit files in which one test failed on attempt 1 and passed on attempt 2: it printed that test and nothing else.

```flaky_tests.py
import re, sys, pathlib, xml.etree.ElementTree as ET
from collections import defaultdict

seen = defaultdict(dict)  # test -> {attempt: "pass" or "fail"}
for path in pathlib.Path(sys.argv[1]).rglob("*.xml"):
    match = re.search(r"junit-(\d+)", str(path))
    if not match:
        continue
    for case in ET.parse(path).getroot().iter("testcase"):
        if case.find("skipped") is not None:
            continue
        failed = case.find("failure") is not None or case.find("error") is not None
        test = f'{case.get("classname")}::{case.get("name")}'
        seen[test][int(match.group(1))] = "fail" if failed else "pass"

for test, runs in sorted(seen.items()):
    if {"pass", "fail"} <= set(runs.values()):
        print(test, " ".join(f"{a}:{r}" for a, r in sorted(runs.items())))

# gh run download "$run_id" --pattern 'junit-*' --dir reports
# python3 flaky_tests.py reports
```

## When to stop scripting and use a service

The scripts answer the question one commit at a time. Answering it across months means storing every report, which is what flaky-test services do. Trunk Flaky Tests and BuildPulse ingest the JUnit XML your runner already writes through an upload step; Datadog's Test Optimization detects flaky tests from the results it collects. The trade is an upload step and a vendor against a script and storage you own.

Either way, a name on the list is not yet a diagnosis. A job killed with exit code 137 lands on it as easily as a racy assertion, and they need opposite fixes. [Flaky test or infrastructure failure](/learn/flaky-tests/flaky-test-or-infrastructure-failure) sorts one from the other.

> Vendor behavior from docs.trunk.io (GitHub Actions quickstart), the buildpulse/buildpulse-action README and docs.datadoghq.com (Flaky Tests Management), read 2026-11-12.

## FAQ

### How do you identify flaky tests?

Look for the same commit producing both a pass and a fail. Re-runs in GitHub Actions use the same SHA, so a failed attempt followed by a passed one is direct evidence, and JUnit reports kept per attempt narrow it from the job to the test.

### How do I see previous attempts of a GitHub Actions run?

In the web UI, open the Latest dropdown to the right of the run name and pick an earlier attempt. From the terminal, `gh run view <run-id> --attempt 1` shows one attempt. From the REST API, list jobs with `filter=all` or call the attempt endpoints with an attempt number.

### What is github.run_attempt?

A number for each attempt of a workflow run. It starts at 1 and increases with each re-run, while `github.run_id` and `github.run_number` stay the same. Put it in artifact names and log lines so each attempt keeps its own evidence.

### How many times should I rerun a test to know if it is flaky?

There is no fixed number, because it depends on how often the test fails. A test that fails one run in fifty has about a 64 percent chance of failing at least once in fifty runs, from 1 minus 0.98 to the power of 50, so a clean streak of fifty is weak evidence. Use the attempt history you already have before spending runs on repetition.

## References

- [GitHub REST API: workflow runs, including run attempts (read 2026-11-12)](https://docs.github.com/en/rest/actions/workflow-runs)
- [GitHub REST API: workflow jobs and the filter parameter (read 2026-11-12)](https://docs.github.com/en/rest/actions/workflow-jobs)
- [GitHub Docs: re-running workflows and jobs](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs)
- [GitHub Docs: contexts reference, github.run_attempt](https://docs.github.com/en/actions/reference/workflows-and-actions/contexts)
- [actions/upload-artifact README](https://github.com/actions/upload-artifact)

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt