CI pipeline reliability: what to measure, and what to fix first
CI pipeline reliability is the share of red builds that mean something, not the share of runs that end green. Once a team reruns first and reads second, the suite has stopped being a signal, and adding tests to it will not bring the signal back.
Pass rate is the number every dashboard leads with and the least useful one available. A suite that passes 95 percent of the time can be excellent or worthless, depending entirely on what the other 5 percent is: regressions caught before they shipped, or noise that everyone has learned to click through on the way to merging.
The question that separates those two is narrower. When the pipeline goes red, how often is the code actually broken? That ratio decides whether anyone acts on a failure, and every other reliability problem is downstream of it. A team that trusts its red builds will fix them quickly; a team that does not will rerun them, and the rerun is where the cost lives.
Reliability is a ratio, not a rate
Define it precisely and the rest of the page follows: a pipeline is reliable when a failure is evidence. Every failure that is not evidence spends the same minutes, produces the same notification and trains the same reflex, which is to run it again. That reflex is the actual failure mode, because it is applied uniformly, including to the failures that were telling the truth.
This is why pass rate misleads. It moves when you quarantine a flaky test and it moves when you delete a real one, and it moves in the same direction for both. A ratio between real and spurious failures does not have that property, which makes it worth the effort of collecting even roughly.
Four numbers worth tracking
None of these needs a platform team to collect. All four can be derived from the workflow run history that GitHub already stores, and the first one can be computed from reruns alone.
- Rerun success rate. Of the failures that were rerun with no code change, what share passed. High is bad, and the number is a direct measure of how much of your CI output is noise.
- Failure cause mix. Test failures against infrastructure failures against environment failures. When infrastructure is on top, the machine is failing your code rather than the other way around.
- Mean time to recovery. How long the default branch stays red. It is the number that tells you whether failures are being prioritized, and it moves faster than any of the others when a team starts caring.
- Time to first signal. How long a developer waits to learn anything at all. Past the length of a coffee break, they have context-switched, and the feedback arrives to somebody who has moved on.
Flaky and broken are different problems
A broken build fails the same way every time on the same commit. A flaky one does not, and the difference is not about severity: it is about what the failure is evidence of. Broken is evidence about the code. Flaky is evidence about the environment, the test, or the interaction between them, and it is still evidence, which is the part teams give away when they add a blanket retry.
The trap is that flaky failures are cheap to make disappear and expensive to leave. One retry wrapper removes the symptom from the pull request and removes the data at the same time, so nobody can say next quarter whether flakiness grew or shrank. Quarantine is the better move for a genuinely flaky test: it keeps the test running, stops it blocking a merge, and leaves a list. A list can be worked down and can be shown to be shrinking. A retry policy produces neither, and it charges you the minutes twice.
Infrastructure flakiness is a different animal again, because the fix usually is not in your repository. A registry that returned 503, a kernel that killed the heaviest process, a disk that filled with Docker layers. Those are worth separating out precisely because they are fixable once rather than test by test.
MTTR for CI, and when the clock really starts
Borrowed from incident response, mean time to recovery for CI means the time from the default branch going red to it going green again. It is the best single number for whether a team treats a broken pipeline as work, and it is easy to measure badly.
The clock starts when the run went red, not when somebody noticed. Teams that start it at acknowledgement report a flattering number and miss the largest component, which is the gap before anyone looks. It also should not stop at "the fix was merged": it stops when a run on the default branch is green, because a fix that did not work is not a recovery.
One caution: MTTR responds to the wrong incentive as readily as the right one. Reverting quickly improves it, which is usually correct. Disabling a test also improves it, which is usually not. Read it beside the failure cause mix, or it will reward the shortcut.
How much of it is the runner?
This is the number everyone wants and very few people have honestly. We ran the study: an audit of 156,808 failed CI runs across eight open source repositories, including PostHog, Grafana, ClickHouse, Home Assistant and Next.js, over January to March 2026. The replay repaired about 1,300 runs automatically and proposed roughly 700 fixes, which is 1.3 percent of the corpus.
What the audit does not do is claim the remaining 98.7 percent were real defects. Its own words: "I do not know the split, and I am not going to invent one." That refusal is the honest position, and it is worth repeating here rather than rounding into a statistic, because the vendor pages that quote a confident percentage for environment failures are almost never quoting a measurement.
What the 1.3 percent does establish is a floor, and the post is careful to say what kind: "a floor produced by the weakest possible version of the system, not as a hit rate". A replay reads logs and cannot stand on the runner, so it cannot free disk and watch what happens, raise a ceiling and re-run the step, or check that a fix held. Your own number is knowable in a way the industry average is not, and the way to get it is the rerun success rate above. Count the failures that passed on an unchanged rerun for a month. That is your noise, measured on your pipeline, and it is the only version of this statistic worth acting on.
Fix in this order
The ordering is not about severity. It is about which fixes are cheapest and which failures are eroding trust fastest, and those turn out to be the same ones.
- Infrastructure failures first. Timeouts, out-of-memory kills, disk exhaustion, registry errors. They fail builds that had nothing wrong with them, they are usually one configuration change each, and they are what teaches a team that red means nothing.
- Then genuinely flaky tests. Quarantine so they stop blocking, track so quarantine does not become a graveyard, and fix on a schedule somebody owns.
- Then feedback time. A correct answer nobody waits for is not much better than a wrong one. Cache, shard, and cut work that does not need to run on every commit.
- Then the long tail. Rare failures are worth fixing last and worth recording first, because the recording is what tells you when one stops being rare.
The question the dashboard cannot answer
Ask whether anyone still reads a failure notification. If a red build produces a rerun rather than a look, the pipeline has stopped being a signal whatever the chart says, and the remedy is to remove noise rather than to add tests. It is a question for the team rather than for the tooling, and it predicts every metric above.
The reason it is worth asking out loud is that the numbers improve for bad reasons too. Quarantine everything and the pass rate rises. Retry everything and the rerun success rate becomes invisible. The only check on that is a person who still expects a red build to mean the code is broken, and notices when it stops being true.
Key takeaways
- Reliability is the share of red builds that are evidence, not the share of runs that are green.
- Rerun success rate is the one number to start with: it measures your noise directly.
- Quarantine a flaky test, do not retry it. Retrying removes the symptom and the data together.
- Start the MTTR clock when the run went red, not when someone noticed.
- Nobody can tell you what share of your failures is environmental. Count your own reruns instead.