Docker exec user process caused: no such file or directory in CI

By Kaveh Alemi·Latchkey

Docker exec user process caused: no such file or directory means runc got all the way to the last step of container startup, called exec on your entrypoint, and the kernel answered that the target does not exist. The file you are staring at is almost always present; what is absent is the interpreter named on its shebang line, or the dynamic loader the binary asks the kernel for.

What this error means

A container dies the instant it starts, in a job where the build was green, and the message names a Go source file and a line number that have nothing to do with your code. The line number moves between reports of the same failure, which is the first clue about how the text is made: two public issues quote this failure at standard_init_linux.go:219 and at standard_init_linux.go:211 and neither number means anything you can look up. Nothing you wrote appears in the message at all, which is why the usual next move is to check that the entrypoint file is in the image, find that it is, and get stuck.

docker run reviewdog/action-golangci-lint:v1.8
standard_init_linux.go:219: exec user process caused: no such file or directory

This line exists as a literal in no source file

Search the whole message and you will find pages quoting it and no code containing it, which is a good reason to think somebody made it up. Nobody did. runc builds it at the moment of failure out of four pieces with three different origins, and only one of them is a string a person typed.

In runc v1.0.0, libcontainer/generic_error.go formats every system error as "%s:%d: %s caused: %s", filling the first two verbs from a stack frame captured when the error was created. That is where standard_init_linux.go and the line number come from: they are the file and line of the failing call in that particular runc build, not part of any message. The third verb is the only literal, exec user process, passed from libcontainer/standard_init_linux.go where runc calls exec on your entrypoint. The fourth is the kernel errno rendered by Go, and for this page it is errno 2.

Common causes

A carriage return on the shebang line

The most common cause in CI by a distance, and the one the reporter in the issue above landed on. A script committed from Windows, or checked out with autocrlf on, carries CRLF endings, so the interpreter path picks up a trailing carriage return and stops matching a real file. The image looks correct in every listing, because the carriage return is invisible in all of them.

The shebang names an interpreter the image does not ship

A script written against bash and run in an Alpine image fails here, because Alpine provides sh and ash through BusyBox and no bash at all. In our experience this arrives when a script that worked in a developer shell is copied into a slim base image, where the shell was never the same program.

The binary needs a dynamic loader the image does not have

A compiled executable built against glibc and copied into a musl image asks the kernel for a loader that is not present, and the kernel reports that absence with the same errno as a missing script. This is the cause when the entrypoint is not a script at all and nothing about shells is involved.

The entrypoint path only ever existed in a builder stage

A multi-stage build that copies artifacts but not the wrapper script leaves an ENTRYPOINT naming a path nothing created. This reads like the shebang cases and is not one: here the named file really is absent, so a directory listing inside the container settles it in one command.

How to fix it

Normalize line endings, and keep them normalized

1. Strip carriage returns in the build, so an image is correct whatever the checkout did.
2. Add a .gitattributes rule so shell scripts are checked out with LF on every platform.
3. Verify once with od -c, which shows the carriage return that every editor hides.

# .gitattributes
*.sh text eol=lf
entrypoint.sh text eol=lf

# Dockerfile, belt and braces
RUN sed -i 's/\r$//' /entrypoint.sh && chmod +x /entrypoint.sh

Point the shebang at a shell the base image really has

Write #!/bin/sh unless you need a bash feature, and if you need one, install bash in the image deliberately rather than assuming it. A shebang is a hard dependency on a path, so treat it like any other dependency and declare it where the image is built.

#!/bin/sh
set -eu
exec "$@"

Match the binary to the libc of the image that will run it

Either build a static binary, or build against the same libc as the base image, or move to a base image with the libc your binary already wants. Checking with ldd inside the target image answers it before the pipeline does, and the answer is unambiguous.

# musl target for an Alpine runtime image
CGO_ENABLED=0 go build -o /out/app ./cmd/app
# or keep glibc and pick a glibc base
FROM debian:bookworm-slim

Smoke run the image in the job that builds it

Run the built image once with its real entrypoint before anything publishes or deploys it. This is the step that separates a green build from a working image, and it costs seconds against a build that already took minutes.

- name: Smoke test the entrypoint
  run: |
    docker run --rm --entrypoint sh "$IMAGE" -c 'ls -l /entrypoint.sh'
    docker run --rm "$IMAGE" --version

The file is there. Something it points at is not

Once you accept that the error is about an exec that the kernel refused, the shortlist is short. The kernel resolves the shebang line before it resolves anything you would call the program, so a script whose first line reads #!/bin/bash in an image with no bash produces exactly this, and so does a script whose first line reads #!/bin/sh followed by a carriage return. In the second case the kernel is looking for an interpreter literally named /bin/sh\r, and it is right that no such file exists.

The third member of the shortlist is a compiled binary rather than a script. A dynamically linked executable names its loader in the ELF header, and a glibc binary copied into an Alpine image names a loader that musl does not provide. The kernel reports the missing loader with the same errno as a missing script, so the message cannot tell the two apart and you have to.

# which of the three is it? three commands, in order
docker run --rm --entrypoint sh "$IMAGE" -c 'head -c 32 /entrypoint.sh | od -c | head -2'
docker run --rm --entrypoint sh "$IMAGE" -c 'command -v bash || echo "no bash here"'
docker run --rm --entrypoint sh "$IMAGE" -c 'ldd /usr/local/bin/app || true'

Newer runtimes print a different line for the same failure

If your CI log does not look like the one above, that is expected, and it is worth knowing before you decide the message was invented. runc dropped the whole standard_init_linux.go construction after v1.0.0. From v1.1.0 onward libcontainer/system/linux.go returns a plain Go path error from its Exec function, which renders as the operation, the path and the errno, and the layers above wrap it in unable to start container process and error during container init.

The practical consequence is that a log check written against the old wording quietly stops matching. Match on the errno words at the end, which have survived every rewording, rather than on the Go file name at the front, which has not.

What the runner does about it

No repair, and none is claimed: this slug has no entry in our heal evidence, so nothing on this page may say the runner fixes it. It also has no recorded run, and the reason is specific to this failure rather than to our harness. This error is perfectly deterministic. A script with a carriage return on its shebang line has one on every attempt, so a recorded run would be one line, a retry, and the identical line again. The capture would document that we ran the command, not why the command failed.

What a runner can honestly offer here is the diagnosis being cheap. The three commands above cost a couple of seconds after a build and turn an opaque runtime error into a named missing interpreter. Put them behind a smoke-test step and this page stops being one you need.

How to prevent it

• Enforce LF endings on shell scripts through .gitattributes, not through developer discipline.
• Pin the base image, so an interpreter present today is present next month.
• Build and run the same libc, or build static binaries and stop thinking about it.
• Smoke run every image in the job that produced it, before any step depends on it.

Frequently asked questions

Related guides

References

• runc v1.0.0: the format string that assembles this line, libcontainer/generic_error.go
• runc v1.0.0: the "exec user process" literal, libcontainer/standard_init_linux.go
• reviewdog/action-golangci-lint#70: the line quoted above, from a container action
• Docker docs: Dockerfile reference, ENTRYPOINT and CMD forms
• Docker documentation
• Docker build cache
• GitHub Actions documentation