The State of Flaky Tests 2026

What unreliable tests really cost in re-run minutes, lost focus, and the merges they quietly block, and why most red builds were never actually broken.

Executive summary

A flaky test is one that passes and fails against the same code, and by 2026 almost every team of meaningful size has them. They are easy to dismiss one at a time and expensive in aggregate: each false red triggers a re-run, interrupts whoever owns the change, and erodes trust in the suite until engineers start ignoring failures that are actually real. The flaky test is not just a nuisance, it is a slow corrosion of the one thing CI exists to provide, a trustworthy signal.

The data shows the cost is concentrated in two places that rarely show up on a dashboard. The first is the retry tax, the billed minutes and wall-clock spent re-running jobs that were never broken. The second is the context-switch tax, the focus an engineer loses leaving their work to triage a failure that turns out to be a network blip or a registry timeout. This report quantifies both and separates transient flakes, which a clean retry resolves, from genuine defects that retries only hide.

The central finding, and the one that reframes the entire problem, is that the majority of flaky failures are mechanical and environmental, not logic bugs in the test. A network call timed out. A package registry was briefly slow. A runner was killed for running out of memory. A dependency download hit a transient error. These failures have nothing to do with the code under test, they pass cleanly on a fresh attempt, and they make up the bulk of what teams experience as flakiness.

That distinction matters because it points at a completely different remedy than the conventional one. The standard advice, quarantine the flaky tests and rewrite them, treats every flake as a test-logic defect. But if most flakes are transient and environmental, then quarantine throws away signal to suppress noise that was never in the test code at all. The right response to a mechanical failure is not to rewrite the test, it is to recover the environment and try again on clean inputs.

This is exactly the class a self-healing runner can absorb: detect the transient failure, recover it in place, and let the real signal through, so teams stop paying the retry tax without rewriting a single assertion. The genuine assertion failures still fail loudly, because those are the signal you want. The mechanical majority gets recovered before a human is ever paged. The result is a suite that is both cheaper to run and more trustworthy, because what reaches a developer is far more likely to be real.

Flake rate scales with suite size, not team discipline

The single strongest predictor of how often a suite flakes is how many tests it has, not how careful the team is. Past a few thousand tests the surface area for a network call, a shared fixture, or a timing assumption to misbehave grows faster than any quarantine process can keep up with. A small suite can stay clean through sheer diligence. A large suite cannot, because the number of opportunities for something environmental to go wrong on any given run scales with the number of tests, not with the team's commitment to quality.

The chart of flake frequency by suite size makes this concrete: the share of runs hitting at least one flaky failure climbs steadily from single digits in small suites to a majority of runs in suites past ten thousand tests. At that scale, a clean run is the exception rather than the rule, and the question is no longer 'how do we eliminate flakes' but 'how do we stop flakes from reaching a human'.

This is why 'just fix the flaky tests' stops working as a strategy at scale. There is no team large enough to keep up with the flake surface of a suite that big through manual triage, and the ones that try end up with a quarantine list that grows faster than it shrinks. The recovery has to move into the runner layer, because that is the only place that scales with the suite instead of against it.

Most red builds were never really broken

This is the thesis of the entire report, so it is worth stating plainly. When failing runs are categorized by root cause, the majority trace to transient or environmental conditions a clean retry would clear, not to a defect in the product or the test. The modeled failure mix puts transient and environmental flakes plus timing and ordering flakes at roughly two thirds of all red builds, with genuine product defects and test-code defects making up the remaining third.

That ratio is the crux of the flaky-test problem: the signal teams react to is mostly noise, and the noise trains them to ignore the genuine failures buried inside it. When two out of every three red builds turn out to be a network blip or a slow registry, engineers learn that the rational first response to any red build is to hit retry. That learned reflex is corrosive, because the one time in three that the failure is real, it gets the same dismissive retry as the noise, and a genuine defect sails through on a lucky re-run.

The categories matter for the remedy. Transient and environmental failures, a DNS hiccup, a registry timeout, an out-of-memory kill, a dependency download that briefly failed, have nothing to do with the code and resolve on clean inputs. Timing and ordering flakes are subtler but still largely mechanical: a test that assumes an operation completes within some window, or that two tests do not share state. Only the product-defect and test-code-defect slices represent failures that a retry should not and does not fix, and those are precisely the ones you want to fail loudly.

The strategic conclusion follows directly. If most red builds are mechanical, then the goal is not to suppress red builds, it is to automatically resolve the mechanical ones and preserve the real ones. A system that retries on clean inputs recovers the noise and leaves the signal, which is the opposite of quarantine, where you remove a test entirely and lose its signal along with its noise.

• Transient and environmental failures plus timing flakes make up roughly two thirds of red builds in the modeled mix.
• A learned retry reflex lets genuine defects slip through on a lucky re-run, the same way noise does.
• The fix is to resolve mechanical failures automatically and preserve real assertion failures, not to suppress red builds wholesale.

The retry tax is real money hiding in green builds

Re-runs are billed minutes that produced no new information. At a six percent flake rate a mid-size suite quietly burns thousands of minutes a month re-running work that was never broken, and because the build eventually goes green, this cost is almost never attributed to flakiness on any report. It shows up only as a slightly larger CI bill that nobody can quite explain, spread across every team and every pipeline.

The retry-tax chart traces how this compounds. At a one percent flake rate the monthly re-run minutes are an annoyance. At three percent they are a line item. At six percent they are a meaningful fraction of the entire CI budget, spent entirely on re-running jobs that had no defect in them. The cost grows super-linearly with flake rate because higher flake rates mean more jobs need multiple retries to go green, and each retry is a full job re-run, not a cheap resume.

The same chart shows what self-healing recovery does to this curve: it collapses the re-run minutes by resolving the transient failure in place rather than restarting the whole job. Instead of paying for an entire job re-run every time a single step hits a network blip, the platform recovers that step on clean inputs and continues. The minutes spent become recovery minutes, a small fraction of a full re-run, rather than the full wasted cost of starting over.

• A 6% flake rate on a mid-size suite burns thousands of billed re-run minutes per month for zero new information.
• The cost is invisible because the build eventually goes green and nobody attributes the spend to flakiness.
• Recovering a failed step in place costs a fraction of re-running the entire job from scratch.

The context switch costs more than the minutes

A re-run costs a few cents of compute, but pulling an engineer out of deep work to investigate a failure that turns out to be a registry timeout costs far more. The modeled figure is a median of twenty-three minutes of lost focus per investigated flake, counting not just the triage itself but the cost of rebuilding the mental context that the interruption destroyed. On a large suite this happens dozens of times a week, distributed across the whole team.

The cumulative drag on focus is the part teams underprice most, because it never appears on an invoice. It shows up instead as work that takes longer than it should, as engineers who feel perpetually interrupted, and as the slow erosion of the deep-focus blocks that complex work requires. A team can be paying a large flaky-test tax in lost productivity while its CI bill looks entirely reasonable.

This is exactly what disappears when failures are recovered before anyone is paged. If a transient failure is caught and resolved at the runner, the developer never sees a red check, never context-switches to investigate, and never has to decide whether this particular red is real. The expensive part of flakiness, the human attention it consumes, is removed entirely for the mechanical majority, leaving engineers to spend their interrupted moments only on failures that genuinely warrant them.

Timing and ordering flakes are the hard middle

Not every flake is a clean network blip. Timing and ordering flakes, the roughly one fifth of red builds where a test assumed an operation would finish in time or that two tests would not collide over shared state, are the genuinely hard middle of the problem. They are partly mechanical and partly a real weakness in the test, which is why they resist both pure retry and pure rewrite.

These flakes respond less completely to automatic recovery than transient failures do, because the failing condition can persist across a retry if the underlying timing assumption is consistently violated under load. The heal-impact chart reflects this: network and registry failures recover at very high rates, resource kills somewhat lower, and race and ordering flakes lower still, because a meaningful share of them are real test weaknesses wearing the costume of a flake.

The honest guidance is that this is the one category where the test code often does deserve attention. A persistent ordering flake is usually a test that shares state it should isolate, or an assertion that races a background operation. Automatic recovery buys time and removes the day-to-day pain, but the durable fix for a stubborn timing flake is to make the test deterministic. The value of self-healing here is that it lets a team address these on a deliberate schedule rather than as constant emergencies.

Self-healing beats quarantine for the transient majority

Quarantining flaky tests removes signal along with noise and creates a backlog nobody clears. The quarantine list is where tests go to be forgotten: once a test is skipped, the behavior it guarded is no longer verified, and the list grows monotonically because adding to it is easy and removing from it requires the very investigation everyone was trying to avoid. Quarantine trades a flaky signal for no signal at all, which is rarely the trade a team actually wants.

Recovering transient failures at the runner, while letting genuine assertion failures fail loudly, keeps the real signal intact and removes the retry and context-switch tax for the mechanical majority. The heal-impact chart quantifies how cleanly this maps onto failure class: network and DNS blips recover almost entirely, registry timeouts nearly so, resource kills at a high rate, and genuine assertion failures at zero, by design, because those must never be silently recovered. The system is precise about what it absorbs and what it surfaces.

None of it requires changing test code, which is why it lands faster than a quarantine-and-rewrite program. A team does not have to triage its entire flaky backlog, assign owners, and schedule rewrites before seeing any benefit. The runner layer starts recovering the mechanical majority immediately, the retry tax and the context-switch tax drop on the first day, and the team is left to spend its limited test-fixing attention on the genuine timing flakes that actually warrant it.

• Quarantine trades a flaky signal for no signal, and the quarantine list only ever grows.
• Self-healing recovers mechanical failures at the runner while assertion failures still fail loudly, by design.
• Because it touches no test code, it delivers value on day one instead of after a long triage program.

A trustworthy suite is the real prize

Underneath the minutes and the focus is a deeper cost: a flaky suite is a suite nobody trusts, and a suite nobody trusts cannot do its job. The entire value of a test suite is that a green build means safe to ship and a red build means stop and look. Flakiness destroys both halves of that contract. Red stops meaning 'look' because most reds are noise, and green stops meaning 'safe' because engineers have learned to retry past reds that might have been real.

The teams with the most reliable delivery are not the ones with zero flaky tests, that is unattainable past a certain scale. They are the ones whose developers can trust a red build, because the mechanical noise has been filtered out before it reaches them and what remains is overwhelmingly real. Trust in the signal is what lets a team move fast, because it is what lets them believe the suite when it tells them something is wrong.

This is the strategic case for treating flaky tests as a runner-layer problem rather than a test-by-test cleanup chore. The goal is not a flake count of zero, it is a signal a developer can believe. Recovering the mechanical majority automatically and surfacing only the real failures rebuilds that trust directly, which is ultimately worth far more than the re-run minutes it also happens to save.

Recommendations

Categorize failures before you fix anything

Instrument your CI to label failures by root cause, network, registry, resource, timing, assertion, before deciding on a remedy. Most teams discover the transient and environmental share is far larger than they assumed, which immediately reframes the problem away from rewriting tests and toward recovering environments.

Auto-heal transient failures instead of quarantining tests

Recover mechanical failures at the runner on clean inputs rather than skipping the tests that surfaced them. Quarantine removes signal and grows a backlog nobody clears. Self-healing keeps every test live, removes the retry and context-switch tax for the mechanical majority, and requires no change to test code.

Let genuine assertion failures fail loudly

Never silently retry a real assertion failure. The whole point of separating mechanical from genuine failures is that the genuine ones must reach a human immediately and unambiguously. A recovery system that masks real defects is worse than no recovery at all, because it erodes the trust the suite depends on.

Reserve manual test-fixing for stubborn timing flakes

Once automatic recovery removes the transient noise, the residue is mostly real test weaknesses, races, shared state, and timing assumptions. Spend your limited test-fixing attention there, on a deliberate schedule, rather than firefighting the mechanical flakes that the runner layer already handles.

Measure flake rate and retry minutes as first-class metrics

Track the share of runs that flake and the billed minutes spent on re-runs over time, and treat regressions as bugs. The retry tax is invisible precisely because nobody measures it. Making it a visible number is what lets you justify the investment in recovery and confirm it is working.

Outlook

Expect suites to keep growing, and therefore flake surfaces to keep growing, through 2026 and beyond. As teams add tests faster than they retire them and lean harder on integration and end-to-end coverage that touches real networks and services, the mechanical-failure surface expands structurally. The manual quarantine-and-rewrite model, already straining at scale, will become untenable for the largest suites, and recovery will have to live in the runner layer because nothing else scales with the problem.

The conceptual shift this report argues for, that most red builds are mechanical and the right response is recovery rather than rewriting, will increasingly be the default mental model rather than a contrarian take. As more teams instrument their failures by root cause and see the transient majority for themselves, the industry conversation will move from 'how do we fix our flaky tests' to 'how do we stop mechanical failures from reaching developers at all'.

The practical endpoint is a CI experience where a red build is rare and almost always real. Self-healing runners that recover the mechanical majority, combined with deliberate attention on the stubborn timing residue, rebuild the trust in the signal that flakiness destroyed. Teams that get there will spend the next several years treating their suite as a reliable safety net, while teams that keep retrying-by-reflex keep paying both taxes and slowly stop believing their own tests.

Methodology

This report synthesizes publicly available developer-survey data with Latchkey's own modeled analysis of flaky-test behavior on managed runners. Figures labeled modeled are illustrative estimates derived from typical suite shapes, observed retry patterns, and published per-minute runner pricing, not a primary survey; figures attributed to a named source reflect that source. Flake-rate and recovery percentages are scenario estimates and should be validated against your own CI history. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• Stack Overflow Developer Survey
• DORA State of DevOps Report
• GitHub - Octoverse
• GitHub Actions - billing & pricing

More Latchkey reports