Insecure registry HTTP response to HTTPS client in CI

By Kaveh Alemi·Latchkey

An insecure registry HTTP response to HTTPS client failure that happens during a build rather than during a pull is BuildKit talking, and BuildKit has its own registry configuration that owes nothing to the daemon. Adding the host to the daemon configuration file changes nothing for a builder running in its own container. The clause itself, http: server gave HTTP response to HTTPS client, is written by the Go standard library on one five byte test, and it reads the same whichever program was asking.

What this error means

The build stops in a step that loads metadata for a base image, or in the export that pushes, and names a registry you run yourself on a plain port. A docker pull of the same reference from the same job may succeed, which is the signal that two different programs are involved and only one of them has been configured. When neither is configured the pull fails as well, carrying the same final clause behind a daemon prefix instead of a build step number. The failure is deterministic and repeats on every run. No run is recorded for this page, so both lines below are assembled from the strings each program formats.

--- during a build, from BuildKit, containerd and the Go client
#8 [internal] load metadata for registry:5000/acme/base:1.4.2
#8 ERROR: failed to authorize: failed to fetch anonymous token: Get "https://registry:5000/token": http: server gave HTTP response to HTTPS client
--- the same clause on a plain pull, from the daemon prefix and the Go client
Error response from daemon: Get "https://10.20.30.40:5000/v2/": http: server gave HTTP response to HTTPS client

Two programs, two configurations, one job

In a GitHub Actions job that uses buildx with the container driver, there are two registry clients. The Docker daemon has one, configured through its own daemon configuration file, and it is what a docker pull step uses. BuildKit has another, running inside the builder container, and it is what resolves base images during a build and what pushes the result.

They do not share settings. The builder container does not mount the host daemon configuration and would not read it if it did, because BuildKit takes its registry settings from a file of its own. That file also splits into two options what the daemon combines into one: plain HTTP is a separate switch from accepting a certificate that cannot be verified, and the documentation says not to enable both together.

Common causes

The daemon was configured and the builder was not

The commonest shape, because the daemon setting is the advice people find first and it genuinely fixes docker pull. The build keeps failing, which makes it look as though the setting did not take. It did; it applies to a program that is not the one failing.

The reference uses a service name rather than loopback

A registry service container is insecure by default only at the loopback address, because that range is built into the daemon and is not a general rule. Referencing it by service name, which is what a container builder must do, removes that exemption and requires real configuration.

The registry section does not match the reference

BuildKit keys its registry configuration by host, and the reference in the build carries a port. A section written without the port does not apply, in the same way that a daemon entry without the port does not. In our experience this is what remains after someone has already found the right file.

The wrong one of the two options was enabled

BuildKit separates plain HTTP from accepting an unverifiable certificate, and the documentation says not to turn both on at once. Enabling the certificate option against a registry that serves plain HTTP does not help, because the problem is not the certificate; there is no TLS on the other end at all.

Something in front of the registry answered, or TLS terminates elsewhere

A proxy that terminates TLS leaves the registry answering HTTPS on its public name and plain HTTP on the address behind it, so a reference written against that address gets a plain answer from a registry that does hold a certificate. Anything answering in front of the registry produces the identical line, because the five byte test reads what the bytes spell, not who sent them.

How to fix it

Declare the registry when the builder is created

Pass the BuildKit configuration inline at setup time and key the section by the exact host and port the build references. This is the fix that matches where the failure actually is. Before you do this, read the last fix: if something is answering in front of the registry, or the certificate exists under another name, declaring the host plain HTTP cements a misconfiguration rather than fixing one.

- uses: docker/setup-buildx-action@v3
  with:
    driver: docker-container
    buildkitd-config-inline: |
      [registry."registry.example.internal:5000"]
        http = true

Or build on the docker driver so there is only one configuration

1. Set the buildx driver to docker, which makes the daemon the builder.
2. Configure the insecure host in the daemon configuration file as usual.
3. Accept the trade: the docker driver cannot do multi-platform builds or the full cache backends.

- uses: docker/setup-buildx-action@v3
  with:
    driver: docker

Give the registry a certificate and stop configuring exceptions

On a job-scoped registry a self-signed certificate is a few seconds of work and removes the whole class of problem, because both clients then do the ordinary thing. Mount the authority into the builder and reference the registry over HTTPS from everywhere.

- uses: docker/setup-buildx-action@v3
  with:
    buildkitd-config-inline: |
      [registry."registry.example.internal:5000"]
        ca = ["/etc/buildkit/certs/internal-ca.pem"]

Prove which client is failing before changing anything

One pull and one metadata load answer the question. If the pull succeeds and the build fails on the same reference, the daemon is configured and the builder is not, and you know which file to edit without guessing.

docker pull registry.example.internal:5000/acme/base:1.4.2 && echo "daemon can reach it"
docker buildx imagetools inspect registry.example.internal:5000/acme/base:1.4.2 \
  || echo "the buildx client cannot"

Read who is answering before you declare the host plain HTTP

Turning the HTTP option on cements a misconfiguration when the registry holds a certificate under another name. Ask both spellings and read the first bytes back: whichever completes a TLS handshake is the name the build should reference, and the other is the address behind the proxy.

curl -sS -i http://10.20.30.40:5000/v2/ | head -5
curl -sS -i https://registry.example.com/v2/ | head -5

The five byte comparison that decides this wording

The final clause is written by the Go standard library, not by Docker and not by BuildKit, and Go reaches it on one narrow condition. A TLS client reads a five byte record header before anything else. When those bytes are not a valid header, the Go TLS package returns a record header error carrying them. The HTTP client then compares those five bytes against the text that begins every HTTP status line, and only on an exact match does it substitute the scheme mismatch error you are reading.

So this is a precise claim rather than a vague TLS complaint. It does not mean the certificate is wrong, or expired, or signed by an authority the runner does not trust, all of which produce different errors that name the certificate. It means the other end answered in plain HTTP on a port that was addressed over HTTPS. Everything in front of that clause is wrapping, which is why the same five words arrive behind a build step number in one job and behind a daemon prefix in the next.

- name: Ask the registry both ways
  run: |
    curl -sS -o /dev/null -w "http  %{http_code}\n" http://10.20.30.40:5000/v2/ || true
    curl -sS -o /dev/null -w "https %{http_code}\n" https://10.20.30.40:5000/v2/ || \
      echo "https failed at TLS, which is the point"

Why localhost works and a service alias does not

A registry started as a service container in a workflow is reachable from steps at a loopback address and from other containers at its service name. The daemon treats the loopback ranges as insecure without being told, so a reference written against the loopback address works with no configuration at all.

The service name has no such exemption anywhere. A build that resolves registry:5000 is going through BuildKit, to a name that is not loopback, with no matching configuration, so the TLS attempt goes out and comes back as a plain HTTP response. The same registry, two references, two completely different outcomes.

Why an HTTPS attempt appears even for a host you marked insecure

The daemon always builds an HTTPS endpoint for a registry and appends a plain HTTP one only for a host on its insecure list, which Docker insecure-registries not configured in CI sets out along with the matching rules that decide whether an entry applies at all. Without a matching entry there is no second endpoint, so the single HTTPS attempt is the whole conversation.

That ordering matters when you are reading a log rather than writing a configuration. People watch an HTTPS request go out to a host they have marked insecure and conclude the setting was ignored. It was not. Marking a host insecure adds the plain endpoint; it does not remove the encrypted one. What tells you the setting took is whether a plain HTTP attempt follows the failed HTTPS attempt.

It is also worth knowing what you agreed to. The flag that adds the plain endpoint is the same flag that switches certificate verification off for that host, so one entry does two jobs and there is no way to ask for only the first. A container builder spells the same choice out as two separate options instead, which is the one place its configuration is stricter than the daemon.

Configure the builder when you create it

The buildx setup action takes an inline BuildKit configuration, which is the point at which to declare the registry. Doing it at builder creation is better than editing files afterwards, because the builder container is created once and the configuration is part of that creation rather than something applied to a running process.

Name the host exactly as your build references it. A registry section is keyed by host and port, so a section for registry does not cover registry:5000 any more than the daemon equivalent would.

- uses: docker/setup-buildx-action@v3
  with:
    buildkitd-config-inline: |
      [registry."registry:5000"]
        http = true
- uses: docker/build-push-action@v7
  with:
    push: true
    tags: registry:5000/acme/api:1.4.2

Why no recorded run backs this page

A run of this would record our service container, our builder configuration and our reference spelling, and every one of those is the variable the reader needs to change. The failure is not a property of a runner that a log could demonstrate; it is a mismatch between which program made the request and which configuration was written, and a transcript of our mismatch does not identify theirs.

The claim at the centre of the page is worse suited to a log still: a comparison of five bytes inside the Go standard library is not something a transcript can show. A recorded run would print the sentence, which nobody doubts, and say nothing about the condition that produces it, which is the part readers get wrong when they reach for a certificate fix.

What does transfer is the separation itself: that BuildKit has its own registry configuration, that it splits plain HTTP from certificate trust, and that loopback is exempt in the daemon and nowhere else. Those are quoted from the configuration documentation and the daemon source rather than from a job of ours.

How to prevent it

• Configure the builder in the same step that creates it, never afterwards.
• Use one spelling of the registry host across steps, builds and configuration.
• Prefer a certificate over an exception once more than one client needs the registry.
• Remember that a service container is only implicitly insecure at the loopback address.
• Record which name carries the certificate when something in front of the registry terminates TLS.

Frequently asked questions

Related guides

References

• BuildKit: the daemon configuration file and its registry options
• Docker docs: BuildKit TOML configuration
• containerd: the resolver and authorizer wrappers around a transport failure
• Go: the scheme mismatch error and the five byte comparison
• moby: the endpoint lookup that appends a plain HTTP endpoint
• Docker documentation
• Docker build cache
• GitHub Actions documentation