# apt Hash Sum mismatch in CI

> An apt Hash Sum mismatch in GitHub Actions means a mirror served an index its own Release file does not vouch for. Here is the fix that lasts.

Source: https://latchkey.dev/learn/failures/apt-hash-sum-mismatch-in-ci  
Updated: 2026-09-20

An apt Hash Sum mismatch in GitHub Actions means the package index that arrived does not match the checksum the repository published for it in its Release file, so apt threw it away rather than trust it. Almost every one of these is a mirror caught between syncs or a cache serving a stale copy, which is why the same Dockerfile builds twenty minutes later with nothing changed.

## What this error means

The step runs `apt-get update`, prints a `Get:` line or two, then an `Err:` for one index, and ends on two lines starting `E:`: the fetch that failed, and "Some index files failed to download". apt exits 100, which apt-get(8) documents as its code for any error, so the step fails before anything is installed. The part worth reading is the block under the error, where apt prints both hashes: the one the Release file published, and the one it computed over the bytes that arrived. When those two lines are there, the diagnosis is finished, and neither your package list nor your signing keys are involved. The run below serves that disagreement on purpose. An apt repository on loopback publishes a Release file recording the checksum of one index while the server hands out a different index of the same length, so only the hash can fail. Everything apt does with it is apt's own work. The second attempt is served the matching index, which is what a mirror that has finished syncing looks like, and each attempt deletes this repository's cached lists first so both fetch.

```Actions log, apt-get update step
E: Failed to fetch http://127.0.0.1:8899/dists/repro/main/binary-amd64/Packages.gz  Hash Sum mismatch
   Hashes of expected file:
    - Filesize:194 [weak]
    - SHA256:29ce1c9573dd99d27d7502dc31a11d72b29945d4ff15d119d014db5da7503a9f
   Hashes of received file:
    - SHA256:288976285ce9f2b135482209b294cac0f2bc6020a5cb9894b072e7e1b11b7090
    - Filesize:194 [weak]
   Last modification reported: Sun, 20 Sep 2026 10:11:49 +0000
   Release file created at: Sun, 20 Sep 2026 10:11:49 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
```

## Common causes

### The mirror was publishing while you were reading it

The documented race, and the common case. A mirror rebuilding its indexes serves a Release file and a Packages file that are seconds apart in age, and apt catches the seam. It is not correlated with your project, your base image or your package list, which is exactly why it looks random. This one clears on a retry, often because the retry lands on a different mirror behind the same name.

### A cache in front of apt is answering with an older index

A pull-through apt cache, a corporate proxy or a CDN can hold an index past the point where the Release file moved on. The tell is that retrying does not help: every attempt gets the same stale copy from the same cache, so the same two hashes print every time. In our experience a mismatch that survives three attempts is a caching layer rather than a mirror.

### A restored `/var/lib/apt/lists` is pairing an old index with a new Release

Caching apt lists between runs, or reusing a warm Docker layer that carries them, puts a stale index on disk before apt starts. Layer caching does this quietly: a `RUN apt-get update` in one layer and a `RUN apt-get install` in another can be months apart in practice, because only the second one was invalidated.

### The download was cut short and apt kept what arrived

Rarer, and usually visible in the Filesize line rather than only in the SHA256 line. A proxy that truncates, a link that drops mid-transfer or a disk that filled during the fetch will all produce a file that hashes differently because it is a different length.

## How to fix it

### Let apt retry the fetch itself

1. Set `Acquire::Retries`, documented as "Number of retries to perform. If this is non-zero APT will retry failed files the given number of times".
2. Set it for the command rather than in a config file you then have to ship into every image.
3. Keep the number small. Three to five attempts covers a mirror mid-sync and fails fast on a stale cache, which is the answer you want.

```Terminal
sudo apt-get -o Acquire::Retries=5 update
```

### Clear the lists and update again, in the same layer

Dropping the cached indexes makes the next update fetch a consistent set in one pass. In a Dockerfile this has to happen inside the same `RUN` as the update, because a separate layer is exactly the thing that goes stale, and the install has to be in there too.

```Dockerfile
RUN rm -rf /var/lib/apt/lists/* \
 && apt-get -o Acquire::Retries=5 update \
 && apt-get install -y --no-install-recommends ca-certificates \
 && rm -rf /var/lib/apt/lists/*
```

### Take the cache out of the path, or point at a mirror you control

When the mismatch repeats with identical hashes, stop retrying and look at what is between the runner and the archive. Bypass the proxy for apt, invalidate it, or point apt at one mirror that you know finishes its syncs before it publishes.

```Terminal
echo 'Acquire::http::Proxy "DIRECT";' | sudo tee /etc/apt/apt.conf.d/99no-proxy
sudo sed -i "s|http://archive.ubuntu.com|http://mirror.example.com|" /etc/apt/sources.list.d/ubuntu.sources
```

### Stop restoring apt index files between runs

Cache the downloaded `.deb` archives if you want to save bandwidth, and never cache the index directory. The archives are content addressed by name and version; the indexes are a snapshot of a moving repository, and restoring one into a later run is the third cause above, on purpose.

```.github/workflows/ci.yml
- uses: actions/cache@v6
  with:
    path: /var/cache/apt/archives
    key: apt-${{ runner.os }}-${{ hashFiles('**/apt-packages.txt') }}
```

## How to prevent it

- Put `-o Acquire::Retries=5` on every `apt-get update` in CI, including the ones inside Dockerfiles.
- Keep `apt-get update` and `apt-get install` in one `RUN`, and delete the lists at the end of it.
- Pin vendor repositories to a mirror that publishes by-hash paths where you have the choice.
- Cache apt archives, never apt lists.

## What apt compared, and what it did not

apt fetches the Release file first, and that file is a list of checksums for every index the repository publishes. It then fetches an index, hashes the bytes, and compares. A mismatch means those two fetches saw different states of the same repository, and nothing more than that.

| What apt checked | Where the expected value came from | What a mismatch means |
| --- | --- | --- |
| SHA256 of the index | The `SHA256:` block in `Release` | The two fetches saw different content |
| File size | The same block, printed as `Filesize` and marked `[weak]` | Usually a truncated or swapped file |
| The signature on `Release` | `InRelease`, or `Release.gpg` | A key problem, which prints differently |

> If your log says `NO_PUBKEY`, `EXPKEYSIG` or "the following signatures were invalid", you have a signing problem rather than this one, and clearing the lists will not touch it.

## Acquire-By-Hash exists because of this exact race

Ubuntu documents the sequence plainly: apt downloads the Release file, the mirror updates its Packages file, apt downloads the Packages file, and the hash it was promised no longer describes the file it got. The repository was never broken. It was two states of the same repository, read a few seconds apart.

The fix for the race was to stop asking for a file by name. Under acquire-by-hash, apt asks for the index at a path built from its checksum, and every by-hash file has to exist before the new Release file is published, so there is no window. apt turns this on by default: `Acquire::By-Hash` is documented as "True by default, but automatically disabled if the source indicates no support for it". That last clause is why you still see this on repositories that do not publish by-hash paths, which includes plenty of vendor apt repositories that CI jobs add.

## What happened on the recorded run

The run above failed the first attempt and passed the second, with no change to the command. The runner is a Latchkey managed runner, and what stands between those two attempts is its own retry of the failed step rather than anything the workflow asked for. No pattern is named here: naming one claims a specific production detector fired, and that claim belongs to a page whose evidence record carries it.

The mechanism is worth borrowing whether or not you run on a managed runner: this failure clears on a second fetch precisely when its cause is a mirror mid-sync, and it does not clear at all when the cause is a cache in front of that mirror. That is the single most useful split on this page, and the next section is organized around it. See [how self-healing works](/documentation/self-healing) for what the runner does with a failing step in general.

## FAQ

### Why does apt-get update fail with Hash Sum mismatch only in CI?

Because CI fetches indexes far more often than a laptop does, from shared addresses, at the times mirrors publish. Nothing about the runner is special; it simply takes many more shots at the race. The flakiness reports on the GitHub runner images repository describe the same thing, with the same fix: retry the update, and stop caching the index files.

### Does Hash Sum mismatch mean a package was tampered with?

Almost never. A tampered index would also have to be signed by a key apt trusts, and a signing failure prints as an invalid signature or a missing public key instead. What this message says is that two fetches from the same repository disagreed, which Ubuntu documents as an ordinary consequence of a mirror updating while you read it.

### How do I make apt retry a failed index download?

Set `Acquire::Retries` on the command with `-o`, or in a file under `/etc/apt/apt.conf.d/` for an image you build yourself. Three to five is enough: the transient cause clears within a couple of attempts, and a higher number only delays the failure you would rather see quickly.

### Should I cache /var/lib/apt/lists between CI runs?

No. That directory is a snapshot of a repository that keeps moving, and restoring it is how you pair an old index with a new Release file. Cache `/var/cache/apt/archives` instead if you want to save the download, and note that the archives directory is root owned, which is the usual reason a cache step on it fails with a permission error.

## References

- [Ubuntu wiki: AptByHash, the race this message comes from](https://wiki.ubuntu.com/AptByHash)
- [apt.conf(5): Acquire::Retries and Acquire::By-Hash](https://manpages.ubuntu.com/manpages/noble/en/man5/apt.conf.5.html)
- [apt-get(8): the update command and its exit status](https://manpages.ubuntu.com/manpages/noble/en/man8/apt-get.8.html)
- [actions/runner-images#7067: sudo apt-get update is flaky on ubuntu-20.04-xl](https://github.com/actions/runner-images/issues/7067)

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt