CI Pipeline Reliability: What to Measure, and What to Fix First

By Daniel Zoghalchali·Latchkey

A pipeline is reliable when a red build means something. Once people re-run first and read second, the suite has stopped being a signal, and no amount of coverage fixes that.

Reliability is usually measured as pass rate, which is the least useful number available. A suite that passes 95% of the time can be trustworthy or worthless depending entirely on what the other 5% is: real regressions caught, or noise everyone has learned to click through.

The useful question is narrower. When CI goes red, how often is the code actually broken? That ratio is what determines whether anyone acts on a failure, and everything else is downstream of it.

The four metrics worth tracking

Fix in this order

Infrastructure failures first: timeouts, OOM kills, disk exhaustion, registry errors. They are the cheapest to fix and they are eroding trust fastest, because they fail builds that had nothing wrong with them.
Then genuinely flaky tests: quarantine them so they stop blocking, track them so quarantine does not become a graveyard, and fix them on a schedule.
Then feedback time. A correct answer nobody waits for is not much better than a wrong one.
Only then chase the long tail of rare failures.

Why quarantine beats retry

Retrying a flaky test hides it, and hidden flakiness compounds: the count grows silently until a large share of every run is noise. Quarantine keeps the test running but stops it blocking the merge, and it produces a list. A list can be worked through and can be shown to be shrinking. A retry policy produces neither, and it charges you the minutes twice.

The reliability metric that is not a metric

Ask whether anyone still reads a failure notification. If a red build produces a rerun rather than a look, the pipeline has stopped being a signal regardless of what the dashboard says, and the fix is to remove the noise rather than to add more tests. That is a question for the team, not for the dashboard, and it is the one that predicts the others.

Applying this to your pipeline

Measure before changing. Most CI optimisation targets the wrong step because the slow one is assumed rather than timed.
Cache what is expensive to produce and cheap to validate, and key the cache to the exact tool version.
Fail fast: run the cheapest checks that can reject a change first, so an expensive job never starts on code that cannot pass.
Prefer determinism over speed when they conflict. A fast pipeline nobody trusts gets re-run, which is slower than a slow one that is believed.

Frequently asked questions

What makes a CI pipeline reliable?

That a failure means the code is broken. Reliability is not a high pass rate; it is a high correlation between red builds and real defects. Once failures are routinely resolved by a rerun, the pipeline is producing noise and people stop reading it.

What is a good rerun success rate?

Low. A high rerun success rate means most failures pass again on unchanged code, which is a direct measure of how much of your CI output is noise. It is the most diagnostic single number available, and it points at infrastructure rather than tests.

Should I retry flaky tests automatically?

Prefer quarantine. Retrying hides flakiness and charges the minutes twice, and hidden flakiness grows because nothing forces anyone to look at it. Quarantine keeps the test running without blocking merges and leaves a list that can be worked down.

What should I fix first in an unreliable pipeline?

Infrastructure failures. Timeouts, out-of-memory kills, disk exhaustion and registry errors fail builds that had nothing wrong with them, they are usually cheaper to fix than flaky tests, and they are what teaches a team to distrust a red build.