Self-Healing CI: Recovering When the Runner Fills Up Mid-Build
By Daniel Zoghalchali·Latchkey
When a build runs fine until the disk fills, the failure is about capacity, not your code -- reclaim space and the same commit completes.
The problem
A build progresses normally and then fails because the runner ran out of disk space partway through -- intermediate artifacts, layers, and caches accumulated until a write failed. A human adds a cleanup step or re-runs on a fresh runner and the build passes unchanged.
Typical symptom
tar: write error: No space left on device
##[error] Process completed with exit code 2.
Why it happens
A build writes a growing pile of intermediate output -- compiled objects, container layers, downloaded dependencies, logs -- and on a runner with limited disk that total can cross the ceiling mid-build even though each step is reasonable on its own.
It is mechanical and self-correcting: nothing is wrong with the code, and the same commit succeeds the moment adequate space is available again.
The manual fix
The manual fix is to reclaim space and rerun:
Prune intermediate build output, unused images, and caches before the heavy step.
Remove large preinstalled toolchains the job does not use.
Re-run on a runner with a larger disk if the build legitimately needs it.
A mid-build disk-exhaustion failure has a clear signature and a well-defined remedy: reclaim space, then retry. A self-healing CI pipeline detects the out-of-space condition, frees space on the runner with safe cleanup or retries with adequate capacity, and can suggest a durable change so the build stops outgrowing its disk.
Frequently asked questions
What causes Self-Healing CI: recovering when the runner fills up Mid-Build?
A build progresses normally and then fails because the runner ran out of disk space partway through -- intermediate artifacts, layers, and caches accumulated until a write failed. A human adds a cleanup step or re-runs on a fresh runner and the build passes unchanged.
How do I fix Self-Healing CI: recovering when the runner fills up Mid-Build manually?
[object Object]
Can Self-Healing CI: recovering when the runner fills up Mid-Build be fixed automatically?
A mid-build disk-exhaustion failure has a clear signature and a well-defined remedy: reclaim space, then retry. A self-healing CI pipeline detects the out-of-space condition, frees space on the runner with safe cleanup or retries with adequate capacity, and can suggest a durable change so the build stops outgrowing its disk.
