Self-Healing CI: Recovering a Job OOM-Killed Mid-Run
A job that is killed for memory partway through is the runner enforcing a ceiling, not your code failing -- retrying with adequate memory completes it.
The problem
A job is killed mid-run by the out-of-memory killer after a step’s memory use spiked past the runner’s ceiling. The build was progressing fine; it briefly needed more memory than the runner had. A human re-runs on a larger runner, or lowers peak memory, and the job completes unchanged.
[12345.678] Out of memory: Killed process 6789 (node)
##[error] The runner has lost communication with the server.Why it happens
A runner enforces a memory ceiling, and when a build, compiler, bundler, or test step spikes past it the kernel’s OOM killer terminates the heaviest process mid-run, cutting the job off even though it was healthy until the spike.
It is mechanical, not a code defect, when the memory growth is a legitimate peak rather than a leak: the same commit completes once it has adequate memory headroom.
The manual fix
The manual recovery is to add headroom or lower peak memory, then retry:
- Re-run on a runner with more memory.
- Lower peak memory: reduce parallelism, cap language heaps, or split the heavy step.
- Confirm it was the OOM killer from the runner’s memory metrics around the failure.
How this gets automated
An OOM kill has a clear, detectable signature, and the right response is well-defined: retry with adequate resources rather than masking a code bug. A self-healing CI pipeline recognizes the out-of-memory condition, retries the step with the memory it needs, and only escalates if the failure is a genuine leak rather than a mechanical peak, so the developer never sees the red build.