Skip to content
EnglishPortuguês日本語
Start Free
Latchkey LogoLatchkey home
EnglishPortuguês日本語
Start Free

Fork retry resource temporarily unavailable CI failures

·Latchkey

Fork retry resource temporarily unavailable CI failures mean the kernel refused to create a new process with EAGAIN, and EAGAIN in fork(2) is a limit on the number of processes and threads, not a shortage of memory. Count the threads your build is running and compare that against ulimit -u and the cgroup PID limit before you touch the runner size.

Two errno paths out of fork: EAGAIN retries and says Resource temporarily unavailable, ENOMEM does not
The two failure paths in bash make_child. The retry loop runs only while errno is EAGAIN, which is why an out-of-memory fork prints once and never prints "retry".

What this error means

The step prints the same line several times and then a shorter version of it, after which almost every command fails because nothing can start a child process. The repetition is not noise: bash retries on this specific error and prints once before each retry, with a doubling sleep in between, so a step can sit there for several seconds looking hung before it gives up. The last line drops the retry: because it comes from a different branch of the same function, the one that runs after the loop has stopped trying. The prefix is worth reading too. In a workflow run: step it is not the word bash but the path of the temporary script the runner wrote, because bash names the error after BASH_SOURCE[0] whenever it is not interactive.

Reconstructed from sys_error in bash make_child (jobs.c)
fork: retry: Resource temporarily unavailable
fork: retry: Resource temporarily unavailable
fork: retry: Resource temporarily unavailable
fork: Resource temporarily unavailable

Read the errno before you believe anything else

Two different failures produce a line that starts fork: and they have opposite fixes. Resource temporarily unavailable is the C library's text for EAGAIN, and Cannot allocate memory is its text for ENOMEM. If your log says the first one, this page applies. If it says the second, you have a memory problem and the runner size is the lever.

The distinction is visible in bash's own source rather than a matter of interpretation. In make_child, the retry loop is written as while ((pid = fork ()) < 0 && errno == EAGAIN && forksleep < FORKSLEEP_MAX), so the retry behaviour is conditional on EAGAIN specifically. A fork that fails for any other reason falls straight past the loop to the branch below it, which prints once, kills the current pipeline, and sets the exit status to EX_NOEXEC, defined as 126 in bash's own header.

That gives you a free diagnostic. Repeated fork: retry: lines are proof the error was EAGAIN, because no other errno reaches that line. A single fork: line with no retries above it is something else, and the something else is usually memory.

Terminal
ulimit -u                      # max user processes and threads for this uid
cat /sys/fs/cgroup/pids.max    # the cgroup PID ceiling, often lower than ulimit
ps -eLf | wc -l                # threads currently alive, not processes
free -m                        # rule memory in or out in the same breath

Common causes

Nested parallelism multiplied out

The common case, and it is arithmetic rather than a bug. A test runner given one worker per core starts a build tool that is also given one job per core, and each of those starts a language runtime with its own thread pool. On a 4 vCPU runner that is not twelve threads, it is four times four times some pool size, and it can reach four figures without anything looking wrong in the workflow file. The ceiling is reached by multiplication, which is why the same command runs fine on a laptop with a much higher limit.

A container or job with a low pids limit

Running your build inside a container adds a second ceiling. The cgroup PIDs controller enforces pids.max independently of ulimit -u, and it is frequently the lower of the two, so a build that has plenty of headroom by the shell limit is stopped by the cgroup anyway. This is the version that appears when a job is moved into a container and nothing else changes, and it will not reproduce outside that container.

Something is leaking processes or threads

A step that starts a background service on every iteration of a loop, a test suite that spawns a helper per test and never waits for it, or a tool that leaves zombies behind will climb steadily toward the limit rather than jumping to it. The signature is that the failure happens late in a long step rather than immediately, and that it moves earlier as the suite grows. Counting with ps -eLf | wc -l at the start and end of the step turns this from a theory into a number.

The limit is genuinely too low for the work

Some builds legitimately need thousands of threads, and a runner image or container configured conservatively will refuse them. This is the least common of the four and the only one where raising the limit is the right answer rather than a way of postponing the question, so establish it by measurement before you conclude it.

How to fix it

Measure the ceiling and the count, in that order

  1. Print ulimit -u and cat /sys/fs/cgroup/pids.max in the failing step. The effective ceiling is the lower of the two, and inside a container it is usually the second.
  2. Count threads rather than processes with ps -eLf | wc -l, because RLIMIT_NPROC counts threads and ps -ef does not show them.
  3. Sample the count at the start of the step and again just before the failure. A number that climbs is a leak; a number that jumps is parallelism.
.github/workflows/ci.yml
- name: Show the ceilings and the current count
  run: |
    echo "ulimit -u: $(ulimit -u)"
    echo "pids.max: $(cat /sys/fs/cgroup/pids.max 2>/dev/null || echo not-in-a-cgroup-v2-container)"
    echo "threads:  $(ps -eLf | wc -l)"

Cap the parallelism where it multiplies

Fix the product, not one of its factors. Pin the worker count of the test runner and the job count of the build tool to the vCPU count of the runner rather than leaving both on automatic, and cap any thread pool that defaults to the core count. On a standard 2 vCPU GitHub-hosted runner in a private repository this usually means smaller numbers than a developer laptop suggests, and the build often gets faster as well, because oversubscribed workers spend their time in the scheduler.

.github/workflows/ci.yml
- run: make -j2
- run: npx jest --maxWorkers=2
- run: pytest -n 2
  env:
    RAYON_NUM_THREADS: "2"
    GOMAXPROCS: "2"

Raise the limit only once you know which limit binds

If the work genuinely needs the threads, raise the ceiling that is actually stopping you. Raising ulimit -u does nothing when the cgroup PID limit is the lower of the two, which is the single most common wasted change on this failure. In a container, the PID limit is set where the container is created rather than from inside it, so this is a change to the job definition rather than a line in the script.

.github/workflows/ci.yml
- name: Raise the shell limit for this step only
  run: |
    ulimit -u 8192
    ./run-the-heavy-build.sh

Stop the leak rather than absorbing it

Where the count climbs, find the step that starts something and does not wait for it. Background services started with an ampersand and never stopped, helper processes left running after a test times out, and shells that exit without reaping their children all accumulate. Adding an explicit stop, or running the service under a supervisor that the step can kill on exit, removes the failure instead of moving it further into the suite.

Why this is not an out-of-memory error, even though it looks like one

The manual page for fork(2) lists what EAGAIN means, and memory is not on the list. It names the RLIMIT_NPROC soft resource limit, "which limits the number of processes and threads for a real user ID"; the system-wide /proc/sys/kernel/threads-max; the maximum number of PIDs in /proc/sys/kernel/pid_max; and "the PID limit (pids.max) imposed by the cgroup 'process number' (PIDs) controller". Every one of those is a count.

Memory exhaustion in fork(2) is ENOMEM, a separate entry with separate text. So an explanation that says this error means the runner ran out of memory to back a new process is describing a real failure mode and attaching it to the wrong message, and the fix it leads to, a larger runner, buys memory you were not short of while leaving the process ceiling exactly where it was.

There is a real memory failure on GitHub Actions runners and it prints something else entirely. If your build is being killed rather than refused, exit code 137 in GitHub Actions is the page you want, and the tell is that the process dies mid-work rather than failing to start.

Why there is no recorded run on this page

We reproduce failures on our own runners and publish the log. This one we deliberately do not, because the reproduction is a fork bomb by another name: the only way to get the kernel to refuse a process is to exhaust the PID ceiling, and a runner in that state cannot reliably run the harness that is supposed to be recording it. What you would get is a log proving our instrumentation survived, which is not the same claim as a log proving your build failed.

The numbers that would make such a run useful are not properties of the failure anyway. ulimit -u and pids.max are properties of a particular runner image and a particular container, so a figure captured on our machines would be evidence about our machines and would be read as evidence about yours. The two commands at the top of this page take two seconds and answer the question for the runner you are actually on, which is the only one that matters.

How to prevent it

  • Pin worker and job counts to the runner size rather than leaving them on automatic detection.
  • Print ulimit -u, pids.max and the thread count in any job that has failed this way before.
  • Stop background services explicitly at the end of the step that started them.
  • When you move a job into a container, check the cgroup PID limit before assuming the shell limit applies.
  • Read the errno on every fork: line: Resource temporarily unavailable and Cannot allocate memory need opposite fixes.

Frequently asked questions

Does "fork: retry: Resource temporarily unavailable" mean I ran out of memory?
No. That text is the error string for EAGAIN, and the manual page for fork(2) lists EAGAIN as a limit on the number of processes and threads: RLIMIT_NPROC, the system thread maximum, the PID maximum, or the cgroup PID limit. Running out of memory in a fork produces ENOMEM and the text Cannot allocate memory instead.
Why is the message printed several times?
Because bash retries this specific error. Its make_child function loops while the fork fails with EAGAIN, printing one line and sleeping a doubling interval before each retry, and only when the loop ends does it print the shorter fork: line, kill the pipeline and set the exit status. Repeated lines are therefore proof that the errno was EAGAIN.
Should I use a bigger runner to fix this?
Usually not. A bigger runner buys vCPU and memory, and this failure is a ceiling on the number of processes and threads. It can even make things worse, because most build tools scale their worker count with the core count, so a larger machine starts more threads and reaches the same limit sooner.
Why does my build hit this in a container but not outside one?
The cgroup PIDs controller enforces its own pids.max on top of the shell limit, and inside a container it is frequently the lower of the two. That limit is set when the container is created, so raising ulimit -u from inside the job changes nothing. Print both numbers and act on whichever is smaller.

Related guides

GitHub Actions exit code 137: the step was killed, not crashedExit code 137 in GitHub Actions is SIGKILL, almost always the out-of-memory killer. Job killed no output watchdog errors in CIA job killed no output watchdog message comes from Travis or CircleCI. GitHub Actions runner specs: vCPU, RAM and disk by sizeGitHub Actions runner specs for every size: vCPU, RAM and disk for standard, larger, macOS and ARM runners, The runner has received a shutdown signalWhen the runner has received a shutdown signal in GitHub Actions, the agent was stopped mid-job.

References

A fork retry is a pid limit, so a bigger runner will not help. Latchkey is $0.0025/min at 2 vCPU. Start free → 30-day trial · No credit card