The State of Pipeline Reliability 2026
When CI itself is the outage: the uptime, recovery time, and runner-availability metrics that decide how often a whole team can actually merge.
Executive summary
Pipeline reliability is the uptime metric nobody puts on a status page. When CI is red on main, or a job sits queued because no runner is free, delivery stops for the whole team just as surely as a production outage stops customers. The difference is that nobody is paging on it, so the cost accumulates quietly as stalled merges and idle engineers rather than a visible incident with a postmortem and an owner.
That invisibility is precisely why pipeline reliability is chronically under-managed. A production outage has a dashboard, an alert, and a name; a blocked pipeline has a developer quietly waiting, a pull request that cannot merge, and a release that slips a day. The minutes add up across a team, but because no single moment looks severe, the aggregate cost rarely gets measured, let alone defended with a budget the way production uptime is.
The data splits the problem into three failure modes. Pipelines break and stay broken until someone recovers them, which is a mean-time-to-recovery problem dominated by detection and triage rather than the fix itself. Runners are not available when work arrives, which is a queue-starvation problem that spikes exactly when the team is busiest. And transient infrastructure failures masquerade as real breakages, which inflates both of the first two. Each is measurable, and each is addressable without rewriting the application.
This report quantifies all three. We look at how effective CI uptime falls as organizations grow, how much of recovery time is humans noticing and diagnosing rather than remediating, where pipeline wait time actually goes, and what share of pipeline-blocking incidents trace to transient causes that should never have reached a human. The throughline is that the largest reliability wins come from the infrastructure layer, not the test layer.
Managed runners change the recovery math directly. Self-healing recovery clears the transient share of breakages before a human is involved, collapsing the part of MTTR that humans are slowest at, and an elastic warm pool removes most of the queue-starvation tax so the pipeline is available when work arrives. The result is a pipeline that recovers itself when it is broken and has capacity when it is not, which is where the largest reliability gains in this report come from.
Share of working hours main pipeline is green and unblocked, by org size. · Source: Latchkey analysis (modeled)
Median minutes from red main to restored green, by recovery approach. · Source: Latchkey analysis (modeled)
Email me the report
The full report is right here on this page, free. Want the link in your inbox to read later or share, plus new Latchkey reports as they drop? Drop your email and we will send it over.
Sent! Check your inbox for the report link.
No spam. Unsubscribe anytime.
Effective uptime falls as orgs grow
Small teams keep main green most of the time almost by default. There are few enough contributors, services, and matrix jobs that the odds of something being red or blocked at any given moment stay low, and when something does break, the person who broke it is usually still online and fixes it quickly. Reliability at small scale is mostly a function of small numbers.
As repositories, contributors, and matrix jobs multiply, those odds climb steadily. Every additional service is another pipeline that can be red; every additional contributor is another source of breaking changes; every additional matrix leg is another cell that can flake. By the time an org is in the hundreds of engineers, a meaningful slice of every working day has a blocked pipeline somewhere on the critical path, even though no single incident looks severe enough to escalate.
The chart below shows effective uptime declining from near-perfect at small scale to noticeably lower past a few hundred engineers. The decline is not caused by worse engineering; it is the arithmetic of more independent things that can each be broken at once. That is why large orgs cannot fix pipeline reliability with discipline alone and have to attack it at the infrastructure layer, where a single improvement applies across every pipeline.
Most recovery time is detection and triage, not the fix
When a pipeline breaks, the clock is dominated by noticing it, figuring out whether it is real, and deciding what to do, not by the actual remediation. The fix for a transient failure is often a single re-run that takes seconds, but the minutes that pile up before that re-run, the time until someone notices the red check, context-switches into it, and concludes it is not a real regression, are where mean-time-to-recovery actually goes.
This is why the recovery approach matters so much more than it looks. The chart below shows median recovery time falling from over an hour with manual triage, to roughly forty minutes with an on-call runbook, to under half an hour with blanket auto-retry, to single digits with a self-healing runner. Each step down the ladder removes more of the human-latency portion of MTTR, which is the slowest and most variable part.
Self-healing recovery wins because it resolves the transient majority before a human is in the loop at all. When a step fails on a known-transient signal, the platform retries it on a fresh environment automatically, so the breakage never reaches a developer, never starts the detection-and-triage clock, and never consumes the attention that manual recovery requires. Elite delivery teams already hold time-to-restore in the Less than one hour band, and collapsing human latency is how that band is held as the org grows.
- Detection and triage, not remediation, dominate the recovery clock for the transient majority of breakages.
- Recovery approach moves median MTTR from over an hour (manual) to single digits (self-heal).
- Self-healing recovery removes the human-latency portion entirely by resolving transient failures pre-triage.
Queue starvation is a hidden reliability tax
A pipeline that is green but cannot get a runner is just as blocked as one that is red. The developer is waiting either way, the merge is stalled either way, and the only difference is that a queued job does not even produce a red check to investigate, so it is even less visible than a breakage. Availability is a reliability dimension that uptime-of-the-tests metrics miss entirely.
A large share of total wait time is spent simply waiting for a runner to be free, and the chart below puts it at nearly a third of the queue-to-finish wall-clock for a busy repository. Worse, that share is not evenly distributed: it spikes exactly when the team is busiest, because a self-hosted or fixed-capacity fleet sized for the average starves during peak, and a fleet sized for peak sits idle and expensive the rest of the time.
An elastic warm pool that scales with demand removes most of this. When capacity expands to meet a burst and contracts when the burst passes, queue starvation stops being a recurring tax and becomes a non-event, and the cold-start share shrinks too because warm capacity is already running when work arrives. The chart shows runner-availability wait and cold-start provisioning as the two slices an elastic pool directly attacks.
- A green pipeline that cannot get a runner is as blocked as a red one, and even less visible.
- Runner-availability wait spikes during peak, exactly when fixed-capacity fleets starve.
- An elastic warm pool turns availability from a recurring tax into a non-event and shrinks cold starts too.
Estimated split of total queue-to-finish wait for a busy repo. · Source: Latchkey analysis (modeled)
A large share of breakages are transient, not real
When pipeline-blocking incidents are categorized by root cause, a substantial fraction trace to transient infrastructure or flaky gates rather than genuine code regressions. The chart below puts transient infrastructure and runner issues as the single largest category, with flaky test gates close behind, and real code regressions accounting for only about a fifth of what blocks a pipeline.
This distribution is the central fact of pipeline reliability, because it means the majority of breakages are incidents that should never have reached a human. A network blip pulling a dependency, a registry timeout, an OOM kill on an undersized runner, a race that only shows up under load: none of these are the application being wrong, and all of them pass on a clean retry. They are mechanical, not fundamental.
These are precisely the incidents self-healing recovery is designed to absorb. By retrying transient failures on a fresh environment before anyone is paged, automated recovery reserves on-call attention for the roughly one-in-five incidents that are real regressions worth a human's time. The chart makes the case visually: shrink the two transient categories and the pipeline's apparent reliability improves dramatically without touching a single test.
- Transient infrastructure and flaky gates together outnumber real code regressions by roughly three to one.
- Every transient category passes on a clean retry, because the application was never the problem.
- Self-healing recovery reserves human on-call attention for the minority of incidents that are genuine.
Estimated share of pipeline-blocking incidents by root cause. · Source: Latchkey analysis (modeled)
Reliability is bought at the runner layer, not the test layer
Teams chasing CI reliability often start by hardening tests, quarantining flaky cases, and adding retries inside the suite. That work has value, but it attacks the smaller half of the problem and the slower-moving one, because it requires touching application and test code case by case while the failure categories keep regenerating as the suite grows.
The larger and faster wins come from the infrastructure underneath: available capacity so work never starves, and automated recovery so transient failures never become incidents in the first place. Both are properties of the runner layer rather than the test layer, which means a single improvement applies uniformly across every pipeline in the org instead of one suite at a time.
Managed runners deliver both with no ops burden, which is why the break-even increasingly favors managed for teams without a dedicated platform group, and often even for those that have one. The relevant comparison is not the per-minute price of a runner but the total cost of an unreliable pipeline: the stalled merges, the idle engineers, and the on-call hours spent recovering breakages that an elastic, self-healing layer would have absorbed automatically.
The cost of an unreliable pipeline is paid in stalled merges
The true cost of pipeline unreliability is not the CI invoice, it is the delivery throughput lost while the pipeline is red or starved. A team with a 47-minute median recovery time and a couple of blocking incidents per repo per month is losing hours of merge-ready work to waiting, and at organizational scale across many repositories that compounds into a real drag on how fast the company can ship.
This cost is structurally invisible because it never appears as a line item. There is no invoice for an engineer who finished a change at eleven and could not merge it until two because main was red and nobody had triaged it yet. The work was done; the value was simply deferred, and deferred value does not show up on any dashboard that finance or engineering leadership routinely watches.
Making the cost visible is the first step to defending against it. Tracking effective pipeline uptime, time-to-restore, and queue-starvation share turns an invisible drag into a measurable metric that can be improved and reported on, the same way production uptime is. The teams that instrument these numbers stop treating pipeline reliability as background noise and start treating it as the delivery-throughput lever it actually is.
Cold starts inflate both recovery and queue time
Cold-start provisioning is a quiet contributor to both failure modes in this report. The chart of where wait time goes shows cold start and provisioning as a meaningful slice on its own, and it also worsens recovery: a self-hosted runner that has to boot, register, and warm its caches before it can even attempt a re-run adds minutes to every recovery on top of the human-latency cost.
The problem is worst exactly when reliability matters most. During a burst, when queue starvation is already biting, new capacity is also coldest, so the jobs that wait longest for a runner are also the ones that pay the largest cold-start penalty once they finally get one. The two effects stack rather than cancel, which is why peak load is where fixed-capacity fleets feel most unreliable.
A warm pool that keeps capacity provisioned and caches hot removes most of the cold-start penalty from both paths. Work that arrives during a burst lands on a runner that is already up rather than one that has to boot, and a re-run after a transient failure executes immediately instead of waiting on provisioning. The same elastic, warm infrastructure that solves availability also shrinks the cold-start slice the chart shows.
Recommendations
Instrument effective uptime, time-to-restore, and queue-starvation share
You cannot defend a reliability budget you cannot see. Track the share of working hours main is green and unblocked, the median minutes from red to restored, and the fraction of wait time spent on runner availability. Making the invisible drag measurable is the precondition for improving it the way production uptime is improved.
Auto-heal transient failures before they reach a human
The largest category of pipeline-blocking incidents is transient and passes on a clean retry. Retrying automatically on a fresh environment collapses the detection-and-triage portion of MTTR, which is the slowest part, and reserves on-call attention for the minority of incidents that are real regressions.
Run an elastic warm pool sized to demand, not to the average
Queue starvation spikes at peak precisely because fixed-capacity fleets are sized wrong for bursts. An elastic warm pool that expands during demand and contracts after removes most of the availability wait and shrinks cold starts, turning the single largest slice of queue time into a non-event.
Buy reliability at the runner layer, not one test suite at a time
Hardening individual suites attacks the smaller, slower-moving half of the problem. Available capacity and automated recovery are properties of the runner layer, so a single improvement applies across every pipeline in the org at once instead of suite by suite. Prioritize the infrastructure fix.
Treat pipeline reliability as a delivery-throughput metric
The real cost of an unreliable pipeline is stalled merges and deferred value, not the CI invoice. Frame uptime, recovery time, and availability as delivery-throughput levers and report them alongside the DORA metrics, so leadership sees the drag that an unreliable pipeline puts on how fast the company can ship.
Outlook
Expect pipeline reliability to graduate from an invisible background cost to a measured, owned metric over the next two years, following the same path production uptime took a decade ago. As more teams instrument effective uptime and time-to-restore, the stalled-merge cost that today hides in deferred value will become a number leadership tracks, and the runner-layer fixes that move it will get the budget they currently lack.
The architectural direction is clear. Elastic warm pools, ephemeral per-job environments, and self-healing recovery are converging into a single expected baseline, because they solve availability, isolation, and recovery at the same layer with one investment. The teams adopting that baseline get reliability as a property of their infrastructure rather than as the output of constant manual vigilance, which is what makes it durable rather than a heroic streak that ends the moment attention lapses.
For most teams the practical takeaway is that pipeline reliability does not require a heroic project, it requires moving the work off humans. The transient majority of breakages should be recovered automatically, the availability tax should be removed by elasticity, and the residual real-regression minority should be what on-call actually spends its attention on. The organizations that internalize that will spend the next two years merging on demand while their peers keep losing hours a day to a pipeline nobody is paging on.
Methodology
This report synthesizes publicly available DevOps research, including the DORA State of DevOps performer bands, with Latchkey's own modeled analysis of runner availability and pipeline recovery on managed infrastructure. Figures labeled "modeled" are illustrative estimates derived from typical pipeline shapes, observed queue behavior, and published runner pricing, not a primary survey; figures attributed to a named source reflect that source. Uptime, MTTR, and queue percentages are scenario estimates intended to show direction and magnitude, and should be validated against your own CI telemetry. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.