Self-Healing CI: Recovering from Inode / Disk-Pressure Exhaustion

A disk can be far from full on bytes yet still refuse writes because it ran out of inodes - the fix is to clear small files, not to add gigabytes.

The problem

A build fails with no space left on device even though df -h shows free space, because the filesystem ran out of inodes after a job created huge numbers of tiny files (node_modules trees, caches, temp files). A human prunes those files or re-runs on a fresh runner and the build passes unchanged.

Typical symptom

OSError: [Errno 28] No space left on device
$ df -i
Filesystem      Inodes  IUsed IFree IUse% Mounted on
/dev/root      1310720 1310720    0  100% /

Why it happens

A filesystem has a finite number of inodes, one per file or directory entry, set at format time. Workloads that generate millions of tiny files can exhaust inodes long before they exhaust raw bytes, so the disk reports space but refuses new files.

The failure is mechanical, not a code bug: the same commit succeeds once the tiny-file clutter is cleared and inodes are available again.

The manual fix

The manual fix is to reclaim inodes by deleting small-file clutter, then retry:

Confirm it is inodes, not bytes, with df -i.
Delete dense small-file directories the job no longer needs (stale caches, old dependency trees, temp dirs).
Re-run the job once inodes are free.

Manual cleanup

df -i
rm -rf ~/.cache/* /tmp/* node_modules/.cache
df -i

How this gets automated

Inode exhaustion has a distinct, detectable signature separate from byte-level disk-full, and the remedy is well-defined: reclaim inodes by clearing small-file clutter, then retry. A self-healing CI pipeline detects the disk-pressure condition, frees inodes with safe cleanup, retries the step, and can suggest a durable change so it stops recurring.

Frequently asked questions

Why does the disk say it has free space but writes still fail?

Because byte capacity and inode capacity are separate limits. df -h shows bytes; df -i shows inodes. A workload that creates a flood of tiny files can hit 100% inode usage with gigabytes of bytes still free, and the only fix is to delete files, not to add storage.