Making CI Reliable: Handling Transient Failures

Half of "broken CI" is not your code; it is the infrastructure having a bad second.

A red build does not always mean a real bug. Network blips, registry timeouts, package mirror hiccups, and out-of-memory kills produce failures that vanish on a re-run. These transient failures erode trust in CI and waste time. This lesson separates them from real failures and covers how to handle each.

Transient vs real failures

A real failure is deterministic: the same commit fails the same way every time because the code or a test is wrong. A transient (or mechanical) failure is non-deterministic infrastructure noise: a DNS timeout, a registry 503, a network reset, an OOM kill. The tell is that re-running the exact same commit passes.

Common transient failures

• Network blips: DNS resolution failures or connection resets while fetching dependencies.
• Registry timeouts: a container or package registry returning 5xx or timing out under load.
• Out-of-memory kills: a job killed by the OS when a step briefly exceeds available memory.
• Runner provisioning hiccups: a machine that comes up degraded.

Step-level retries

For known-flaky network steps you can add a retry wrapper, but applying this everywhere is tedious and clutters every workflow. Use it surgically for the steps that genuinely need it.

- name: Install deps (with retry)
  uses: nick-fields/retry@v3
  with:
    max_attempts: 3
    timeout_minutes: 10
    command: npm ci

Letting the platform handle it

Per-step retry config does not scale, and it cannot rescue a job whose underlying machine went bad. Managed runners such as Latchkey automatically retry transient and mechanical failures (network blips, registry timeouts, OOM) at the infrastructure level, without per-step retry configuration in your workflow. You write the pipeline once and let the platform absorb the infrastructure noise, so a green build means your code is actually green.

Key takeaways

• Distinguish transient (non-deterministic, passes on re-run) from real (deterministic) failures.
• Step-level retries help for specific flaky steps but do not scale across a whole pipeline.
• Managed runners like Latchkey auto-retry transient/mechanical failures without per-step config.

Related guides