Self-Healing CI: Automatic Recovery from "No Space Left on Device"
Running out of disk is one of the most recoverable CI failures there is: the fix is to reclaim space and try again, with no change to your code.
The problem
A build fails with no space left on device. The runner filled up with image layers, caches, and temporary files. Someone adds a cleanup step or re-runs on a fresh runner, and the build passes unchanged.
write /var/lib/.../blob: no space left on device
##[error] Process completed with exit code 1.Why it happens
Build caches, downloaded dependencies, container layers, and temp files accumulate during a job. On a small or reused runner disk, a normal build can cross the limit partway through.
The failure is transient in the sense that nothing is wrong with the code - the same commit succeeds the moment space is available again.
The manual fix
The manual fix is to reclaim space and rerun:
- Prune unused container images, build cache, and volumes.
- Delete large preinstalled toolchains the job does not use.
- Add a
.dockerignore/ trim the build context so less is written. - Re-run the job once space is available.
docker system prune --all --force --volumes
sudo rm -rf /usr/share/dotnet /opt/ghc
df -hHow this gets automated
Because the trigger (a disk-space error) and the remedy (reclaim space, then retry) are both well-defined, this class of failure can be handled with no human involvement. A self-healing pipeline detects the disk-exhaustion condition, frees space on the runner using safe cleanup, retries the step, and - where appropriate - suggests a durable change so it stops recurring.