# CI Pipeline Reliability: What to Measure, and What to Fix First

> The metrics that tell you whether CI is trustworthy, why pass rate is the wrong headline number, and the order to fix flakiness, infrastructure failures and slow feedback in.

Source: https://latchkey.dev/learn/ci-explained/ci-pipeline-reliability-explained  
Updated: 2026-08-20

A pipeline is reliable when a red build means something. Once people re-run first and read second, the suite has stopped being a signal, and no amount of coverage fixes that.

Reliability is usually measured as pass rate, which is the least useful number available. A suite that passes 95% of the time can be trustworthy or worthless depending entirely on what the other 5% is: real regressions caught, or noise everyone has learned to click through.

The useful question is narrower. When CI goes red, how often is the code actually broken? That ratio is what determines whether anyone acts on a failure, and everything else is downstream of it.

## The four metrics worth tracking

| Metric | What it tells you | Bad sign |
| --- | --- | --- |
| Rerun success rate | Share of failures that pass on an unchanged rerun | High: most failures are noise, not code |
| Mean time to recovery | How long a workflow stays broken | Rising: failures are not being prioritised |
| Failure cause mix | Split of test, infrastructure and environment failures | Infrastructure dominating |
| Time to first signal | How long until a developer learns something is wrong | Longer than a coffee: people move on and context is lost |

> Rerun success rate is the one to start with. If most failures pass on a rerun, you do not have a testing problem, you have an infrastructure problem wearing a testing problem costume.

## Fix in this order

- Infrastructure failures first: timeouts, OOM kills, disk exhaustion, registry errors. They are the cheapest to fix and they are eroding trust fastest, because they fail builds that had nothing wrong with them.
- Then genuinely flaky tests: quarantine them so they stop blocking, track them so quarantine does not become a graveyard, and fix them on a schedule.
- Then feedback time. A correct answer nobody waits for is not much better than a wrong one.
- Only then chase the long tail of rare failures.

## Why quarantine beats retry

Retrying a flaky test hides it, and hidden flakiness compounds: the count grows silently until a large share of every run is noise. Quarantine keeps the test running but stops it blocking the merge, and it produces a list. A list can be worked through and can be shown to be shrinking. A retry policy produces neither, and it charges you the minutes twice.

## The reliability metric that is not a metric

Ask whether anyone still reads a failure notification. If a red build produces a rerun rather than a look, the pipeline has stopped being a signal regardless of what the dashboard says, and the fix is to remove the noise rather than to add more tests. That is a question for the team, not for the dashboard, and it is the one that predicts the others.

## Applying this to your pipeline

- Measure before changing. Most CI optimisation targets the wrong step because the slow one is assumed rather than timed.
- Cache what is expensive to produce and cheap to validate, and key the cache to the exact tool version.
- Fail fast: run the cheapest checks that can reject a change first, so an expensive job never starts on code that cannot pass.
- Prefer determinism over speed when they conflict. A fast pipeline nobody trusts gets re-run, which is slower than a slow one that is believed.

## FAQ

### What makes a CI pipeline reliable?

That a failure means the code is broken. Reliability is not a high pass rate; it is a high correlation between red builds and real defects. Once failures are routinely resolved by a rerun, the pipeline is producing noise and people stop reading it.

### What is a good rerun success rate?

Low. A high rerun success rate means most failures pass again on unchanged code, which is a direct measure of how much of your CI output is noise. It is the most diagnostic single number available, and it points at infrastructure rather than tests.

### Should I retry flaky tests automatically?

Prefer quarantine. Retrying hides flakiness and charges the minutes twice, and hidden flakiness grows because nothing forces anyone to look at it. Quarantine keeps the test running without blocking merges and leaves a list that can be worked down.

### What should I fix first in an unreliable pipeline?

Infrastructure failures. Timeouts, out-of-memory kills, disk exhaustion and registry errors fail builds that had nothing wrong with them, they are usually cheaper to fix than flaky tests, and they are what teaches a team to distrust a red build.

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt