Self-Healing CI: Recovering from apt/yum Mirror Failures

When apt-get update fails to reach a mirror, the package you wanted still exists - the specific mirror it picked was briefly down or out of sync.

The problem

An apt-get update/apt-get install or yum/dnf step fails because a distribution mirror returned a 404, a connection error, or stale metadata. The package list is fine; a human re-runs the job - which often resolves to a different, healthy mirror - and it installs cleanly.

Err:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
  404  Not Found [IP: ...]
E: Failed to fetch ... Unable to connect to mirror

Why it happens

Distribution archives are served by a pool of community and regional mirrors. Any individual mirror can be briefly unreachable, mid-sync, or serving stale index files, which surfaces as a fetch error even though the package is available elsewhere.

Mirror selection is often round-robin or geo-based, so a retry frequently lands on a different, healthy mirror with no change to your build.

The manual fix

The manual fix is to retry - ideally after refreshing metadata or pinning a known-good mirror:

1. Re-run the job so package resolution can pick a healthy mirror.
2. Refresh package metadata (e.g. apt-get update) before retrying the install.
3. Point at a reliable mirror or an internal package cache to reduce dependence on the public pool.

apt-get update && apt-get install -y --no-install-recommends <package>
# or for RHEL-family
dnf clean all && dnf install -y <package>

How this gets automated

A failed mirror fetch has a recognizable transient signature, and the safe response is to refresh metadata and retry - typically onto a different mirror. A self-healing CI pipeline detects the mirror failure, retries the package operation with backoff, and only surfaces the step if the package is genuinely missing rather than the mirror being briefly down.

Related guides