Self-Healing CI: Recovering from OOM-Killed Jobs (Exit 137)

When a build is killed for using too much memory, the failure is about the runner, not your code. That makes it a perfect candidate for automatic recovery.

The problem

A job dies with exit code 137 - the process was SIGKILL’d, almost always by the out-of-memory killer. The build was working; it just briefly needed more memory than the runner had. A human then re-runs the job, often on a bigger machine, and it passes.

Typical symptom

Killed
##[error] Process completed with exit code 137.

Why it happens

CI runners enforce a memory ceiling. When a build, bundler, compiler, or test suite spikes past it, the kernel’s OOM killer terminates the heaviest process with an uncatchable SIGKILL.

Because SIGKILL cannot be trapped, the job has no chance to clean up or report a useful error - you just see Killed and exit 137. The signal says nothing about *why*, which is what makes it look scary even though it is mechanical.

The manual fix

The manual recovery is almost always one of these, applied by hand after the failure:

Re-run the job on a runner with more memory.
Lower peak memory: reduce build/test parallelism, cap language heaps (e.g. Node’s --max-old-space-size), or split the heavy step.
Confirm it was memory (not a watchdog) by checking the runner’s memory metrics around the failure.

How this gets automated

An OOM kill has a clear, detectable signature, and the right response is well-defined: retry with adequate resources rather than masking a code bug. A self-healing CI pipeline recognizes the out-of-memory condition automatically, retries the step with the resources it needs, and only escalates to a human if the failure is genuinely not transient. The developer never sees the red build - the job simply completes.

Frequently asked questions

Isn’t automatically retrying an OOM just hiding a real problem?

Only if the memory growth is a genuine leak in your code. Most CI OOMs are mechanical - a build that legitimately needs a bit more headroom than the default runner provides. Self-healing handles the mechanical case automatically and surfaces the genuine-bug case instead of silently masking it.