Exec user process caused: permission denied in Docker CI

By Daniel Zoghalchali·Latchkey

Exec user process caused: permission denied means runc found the entrypoint exactly where you said it would be and the kernel refused to execute it, which in a container almost always means the file has no executable bit. It is the same construction as the missing-file version of this error with one difference that decides everything: the errno at the end is 13 rather than 2.

What this error means

The container exits immediately, before any of your own logging runs, and the message names a Go source file and a line number from runc rather than anything you wrote. Two public reports of this exact failure quote it as standard_init_linux.go:228 and as standard_init_linux.go:211, and they render the tail differently too, one with a colon and one with quotation marks. Both are the same problem. The file itself is present in the image and a listing will confirm that, which is what makes the message feel wrong until you look at the mode column.

standard_init_linux.go:228: exec user process caused: permission denied

Two renderings, one failure, and what that proves

The two wordings in the wild are not two bugs. runc formats system errors from a stack frame plus a cause plus the underlying error, and that format string changed once: through v1.0.0-rc90 the final part was rendered as a quoted string, and from v1.0.0-rc91 it became a colon and a bare string. So exec user process caused "permission denied" and exec user process caused: permission denied differ only in which runc produced them.

The line numbers differ for the same kind of reason. They are captured from the stack at the moment the error is created, so they point into whatever runc build you are running and are not part of any message. This is worth knowing before you search: the whole line is assembled at run time and appears as a literal in no source file, so finding nothing when you search all of it is expected rather than suspicious.

Common causes

The script was committed without the executable bit

The dominant cause. Git tracks the bit and nothing else about the mode, so a file added without it is copied into every image without it. The review that introduced the script showed its contents and not its mode, which is why this survives code review reliably.

The file was generated or extracted during the build

A script produced by an earlier stage, unpacked from a tarball, or written inline in the Dockerfile has the default mode rather than an executable one. Nothing lost the bit here; it was never set, which is a different story with the same ending.

The entrypoint is a binary without the bit

The same failure applies to compiled programs, and it is easier to miss because people associate chmod with scripts. An artifact downloaded in the build and copied into the final stage commonly arrives non-executable, in our experience more often than a checked-in script does.

A mount or a security policy is blocking execution

Rarer, and worth knowing so you do not chase a mode that is already correct. A volume mounted with noexec, or a policy that forbids executing from a path, produces the same errno with the bit plainly set. Check the mode first, because it is one command, and look here only when the mode is right.

How to fix it

Set the mode as the file enters the image

1. Use COPY --chmod=755 so the mode is correct no matter what the build context held.
2. Name the entrypoint by absolute path, so the error would name a path you can check.
3. Rebuild and run the image once with no arguments to confirm it starts.

COPY --chmod=755 entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

Fix the bit in the repository too

Record the executable bit in git so every build from this tree agrees, including ones on machines that do not preserve modes. One command sets it and the change shows up in the diff as a mode change, which is exactly where the next reviewer should see it.

git update-index --chmod=+x entrypoint.sh
git ls-files -s entrypoint.sh   # expect 100755

Chmod in the Dockerfile when COPY --chmod is not available

On a builder without BuildKit, or when the file is produced rather than copied, set the mode in a RUN step instead. Do it in the same stage that ships, because a mode set in a builder stage does not follow an artifact that a later COPY brings across.

RUN chmod 0755 /usr/local/bin/entrypoint.sh \
 && /usr/local/bin/entrypoint.sh --version

Check the mode in the job that builds the image

One listing after the build tells you the mode of the entrypoint before anything tries to run it. This catches the generated-file and downloaded-binary cases that a repository check cannot see, because those files do not exist until the build runs.

docker run --rm --entrypoint sh "$IMAGE" -c 'ls -l /usr/local/bin/entrypoint.sh'

Where the executable bit goes missing

A file copied into an image keeps the mode it had in the build context, and a build context is a git checkout. Git records only whether a file is executable, and it records that faithfully, so a script committed without the bit arrives in every image without it forever. Nobody notices at review time because the diff shows the content and not the mode.

The other route is a file that never had a mode to keep. Scripts extracted from an archive, generated by an earlier step, or written by a heredoc in the Dockerfile all start at the default mode, which is not executable. In both cases the fix is the same and it belongs in the image, not in the workflow.

# what the image actually has
docker run --rm --entrypoint sh "$IMAGE" -c 'ls -l /entrypoint.sh'
# and what git recorded, which is where it came from
git ls-files -s entrypoint.sh

Set the mode where the file enters the image

The cleanest fix is a single flag. BuildKit lets COPY set the mode as the file lands, so the image is correct without a second layer and without depending on what the checkout did. That one change removes the whole class, including the case where somebody clones the repository on Windows and the mode never existed.

Fix it in git as well, so that local runs and other images built from the same tree agree with the one you just fixed. The two together mean the bit is right in the repository and right in the image regardless of how either is produced.

# in the Dockerfile
COPY --chmod=755 entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

# and once, in the repository
git update-index --chmod=+x entrypoint.sh

What the runner does about it

No repair, and none is claimed: the slug is not in our heal evidence, so the page may not say a runner fixes it. There is no recorded run either, and the argument for this one is narrow and specific. This failure is a single bit on a single file inside an image the runner did not build. A capture would show one line, and that line is already quoted verbatim in two public issues at two different line numbers and in two different renderings, which is worth more than a third copy: the pair is what proves the line number and the punctuation are not part of the message. A run of our own would add a fourth variant and no new information.

There is also nothing here for a runner to repair even in principle. Changing the mode of a file inside somebody else's image, between attempts, would be modifying the artifact under test, which is the one thing CI must not do.

How to prevent it

• Always use COPY --chmod for entrypoints and wrapper scripts.
• Keep the executable bit recorded in git, and review mode changes as changes.
• Set the mode in the stage that ships, not in a builder stage.
• Smoke run the image in the job that built it, so a bad mode never reaches a deploy.

Frequently asked questions

Related guides

References

• runc v1.0.0: the format string behind both renderings, libcontainer/generic_error.go
• seglo/kafka-lag-exporter#431: the colon rendering at line 228
• banzaicloud/drone-kaniko#38: the quoted rendering at line 211, same failure
• Docker docs: Dockerfile reference, COPY and the --chmod option
• Docker documentation
• Docker build cache
• GitHub Actions documentation