Docker buildx imagetools create failed in CI
When docker buildx imagetools create failed in CI, the command has three stages and the wording tells you which one you are in, because buildx wraps the last two with prefixes it writes itself. It never prints a sentence about creating a manifest list, so searching for one finds other people guessing.
What this error means
A job that builds each architecture separately and then combines them fails on the combine step, often after every build job went green. The command reads sources from a registry, assembles an index in memory and pushes it, so nothing local is involved and rerunning the build jobs changes nothing. Errors arrive in one of three shapes depending on the stage. No run is recorded for this page; the shapes below are set out as buildx and the resolver format them.
--- one per-architecture tag was never pushed
ERROR: ghcr.io/acme/api:1.4.2-arm64: not found
--- a source that is an old schema 1 image
ERROR: schema1 manifests are not allowed in manifest lists
--- the assembled index could not be written to the target
ERROR: publish sha256:9f2c1b7d to ghcr.io/acme/api:1.4.2: unexpected status from PUT request to https://ghcr.io/v2/acme/api/manifests/1.4.2: 403 ForbiddenThree stages, and the prefix that tells you which one broke
The command resolves every source you named, combines them into one index, then copies any blobs the target repository is missing and pushes the index under each tag. Buildx labels the last two stages when it reports a failure from them, and leaves the first unlabeled because it has not started its own work yet.
So a bare message with a reference in it is resolution. A message beginning with the word copy is the blob transfer between repositories. A message beginning with the word publish is the final write of the index under a tag. That single distinction saves most of the guessing, because the fixes for the three are unrelated.
| Stage | How a failure from it is prefixed | Where to look |
|---|---|---|
| Resolve the sources | No prefix, just the reference and the reason | The build jobs that were supposed to push those tags |
| Combine into an index | A sentence about the manifests themselves | The format of the sources, and the platforms they declare |
| Copy blobs to the target repository | copy then a digest, a source and a target | Read access to the source repository from this job |
| Publish the index under each tag | publish then a digest and a tag | Write access to the target repository |
Common causes
A per-architecture tag was never pushed
The command reads its sources from the registry, so a build job that failed, was skipped, or pushed under a slightly different tag leaves nothing to resolve. The resolver reports the reference followed by not found, with no wrapper, because buildx had not begun copying anything yet. In our experience a matrix leg that was skipped by a path filter is the usual culprit.
The job can read the target but not write it
Combining needs write access to the repository the tag names, and reading the sources needs only read. A workflow whose token was scoped for the build jobs will resolve everything and then fail at publish. The prefix makes this unambiguous, which is the main reason to read it.
A source is in a format that cannot go in a list
Buildx refuses to put a schema 1 manifest into a manifest list and says so directly. This turns up when one architecture is being taken from an old vendor image rather than built, and that image predates the current formats. The fix is to rebuild or remirror that source, not to change the command.
Sources span repositories with nothing to anchor them
Bare digests get their repository inferred from the other references, and that only works when there is exactly one. Sources in two repositories, or a digest with no tag alongside it, leave buildx without an answer and it says it cannot infer the repository for that argument. Fully qualifying every source removes the ambiguity.
How to fix it
Gate the combine on every source actually resolving
Check the sources in the publish job before combining them. It converts a confusing failure into a named one and costs a couple of seconds, and it catches the skipped matrix leg that is the commonest cause.
- name: Every architecture must be present
run: |
for arch in amd64 arm64; do
docker buildx imagetools inspect "ghcr.io/acme/api:1.4.2-$arch" >/dev/null \
|| { echo "::error::1.4.2-$arch is missing, check the build matrix"; exit 1; }
doneGive the publish job write permission of its own
- Declare the package write permission on the publish job rather than inheriting it.
- Log in in that job, because a login in the build job does not carry across.
- Keep the dry run step ahead of the real one so read failures surface before write failures.
jobs:
combine:
needs: [build]
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}Fully qualify every source when more than one repository is involved
Name the repository on every argument rather than relying on inference. Buildx only infers when all the references agree, and a pipeline that stages in one registry and publishes to another never satisfies that.
docker buildx imagetools create \
--tag ghcr.io/acme/api:1.4.2 \
staging.example.com/acme/api:1.4.2-amd64 \
staging.example.com/acme/api:1.4.2-arm64Retry only when the status says the registry faltered
A publish failure carrying a 5xx status is worth another attempt, and one carrying a 403 is not. Because buildx puts the stage prefix on the front, a retry wrapper can read the line and decide, instead of repeating a permission failure three times.
for i in 1 2 3; do
out=$(docker buildx imagetools create --tag "$TAG" $SOURCES 2>&1) && exit 0
echo "$out"
case "$out" in *" 50"[0-9]" "*) sleep $((i * 15)) ;; *) exit 1 ;; esac
done
exit 1The command has its own argument errors, and they are specific
Several failures never reach a registry at all, because buildx validates what you asked for first. Running with no sources, running without a tag and without a dry run, and naming sources across more than one repository without a tag to anchor them each produce their own sentence. They are worth recognizing because they look like registry problems in a wall of log output and are not.
The multi-repository one catches people building in a staging registry and publishing to a production one. Buildx can infer the repository for a bare digest only when every reference points at the same repository, so mixing them requires you to name the repository explicitly.
- name: Combine, with the sources fully qualified
run: |
docker buildx imagetools create \
--tag ghcr.io/acme/api:1.4.2 \
ghcr.io/acme/api:1.4.2-amd64 \
ghcr.io/acme/api:1.4.2-arm64Dry run first, because it separates reading from writing
The command takes a dry run option that resolves every source and prints the index it would push, without writing anything. That splits the problem cleanly: if the dry run succeeds, every source exists and is combinable, and whatever failed is about writing to the target. If it fails, no amount of token widening will help.
It is cheap enough to keep in the workflow permanently as a step before the real one, and it gives the publish job a failure that names a missing architecture instead of a permissions error twenty seconds later.
docker buildx imagetools create --dry-run \
--tag ghcr.io/acme/api:1.4.2 \
ghcr.io/acme/api:1.4.2-amd64 \
ghcr.io/acme/api:1.4.2-arm64Why no recorded run backs this page
Every failure on this page is a statement about registry state, not about a runner. A missing source tag, a source in an old format, and a target that will not take a write are all conditions we would have to create in a registry first, and the recording would then show our registry refusing our own request. The runner contributes nothing to any of the three.
The part worth trusting is which wrapper buildx attaches at which stage, and that is a property of the code rather than of any one run. It was read in the command source and is quoted above, which is what lets a reader classify their own error without reproducing ours.
How to prevent it
- Run the dry run as its own step, always, before the real combine.
- Make every build matrix leg required by the publish job rather than optional.
- Fully qualify source references so inference is never load bearing.
- Keep the per-architecture tags and the final tag in one repository when you can.