Self-Healing CI: Recovering a Transient apt Mirror 404
A 404 from one apt mirror usually means it is briefly out of sync, not that the package is gone -- refreshing metadata and retrying clears it.
The problem
An apt-get install fails because a mirror returned a 404 for a package file the local index still references. The package exists; the chosen mirror was mid-sync or the local index was stale. A human refreshes metadata and retries -- often landing on a healthy mirror -- and it installs cleanly.
Err:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libfoo amd64 1.2.3
404 Not Found [IP: ...]
E: Failed to fetch ... 404 Not FoundWhy it happens
A mirror publishes its package index and the package files it points to at slightly different moments. If the local index references a version a mirror has not finished mirroring -- or has just rotated out -- the file fetch returns 404 even though the package is available once metadata is refreshed.
It is transient and self-correcting: re-running apt-get update to pull a consistent index, ideally from a healthy mirror, makes the same install succeed with no change to the package you want.
The manual fix
The manual fix is to refresh the index and retry:
- Re-run
apt-get updateto fetch a fresh, consistent index. - Clear stale package lists if the 404 persists, then update again.
- Point at a reliable mirror or internal cache to reduce dependence on the public pool.
apt-get update
apt-get install -y --no-install-recommends <package>How this gets automated
A transient apt 404 has a recognizable mid-sync signature, and the remedy is well-defined: refresh metadata and retry, typically onto a consistent mirror. A self-healing CI pipeline detects the fetch failure, refreshes the package index, retries the install, and only surfaces the step if the package is genuinely missing rather than a mirror being briefly out of sync.