How to Avoid the Cost of Blind Retries in CI

Blind retries make a suite look green while hiding real regressions and burning extra CI minutes, so track how often tests need a retry and drive that number down.

A retry that passes on the second attempt hides whichever bug caused the first failure. Measure the retry rate, alert when it climbs, and treat retries as a temporary bridge to a real fix, not a permanent policy.

Steps

• Record how many tests only passed on retry each run.
• Alert or fail when the retry rate exceeds a threshold.
• Prioritize the highest-retry tests for a proper deflake.

Log the retry rate

# playwright: count flaky (passed-on-retry) tests from JSON
npx playwright test --reporter=json > report.json
jq '[.suites[].specs[] | select(.tests[].results | length > 1)] | length' report.json

Gotchas

• A test that passes only on retry is still a bug; the retry just delayed the discovery.
• Retries add CI minutes on every run, so a growing retry rate quietly grows your bill.
• Managed runners can auto-heal transient infrastructure failures, but a flaky test still needs a real fix.

Related guides