Self-Healing CI: Recovering Exhausted /dev/shm Shared Memory
A browser or process that crashes because /dev/shm is full hit a small shared-memory ceiling, not a bug -- more shared memory or a fallback flag fixes it.
The problem
A step crashes because the shared-memory filesystem (/dev/shm) ran out of space -- common with headless browsers and some databases that default to using it. The code is fine; the runner’s /dev/shm was too small for the workload. A human raises the shared-memory size or sets a fallback flag and the step passes unchanged.
[crash] Failed to allocate shared memory: /dev/shm is full
# headless Chrome commonly hits this without --disable-dev-shm-usageWhy it happens
Some tools use the shared-memory tmpfs at /dev/shm for scratch space, and containers often mount it small by default. A workload that needs more shared memory than the small default provides exhausts it and crashes, even though plenty of regular memory is free.
It is a mechanical ceiling on one specific resource, not a code bug: giving the workload adequate /dev/shm (or pointing it at regular temp space) makes the same step succeed.
The manual fix
The manual fix is to provide adequate shared memory or a fallback, then retry:
- Increase the shared-memory size for the container/step.
- Use the tool’s fallback flag where it exists (e.g. headless Chrome’s
--disable-dev-shm-usage). - Re-run the step once shared memory is adequate.
How this gets automated
A /dev/shm exhaustion has a distinct shared-memory signature separate from a general OOM, and the right response is to provide adequate shared memory and retry. A self-healing CI pipeline detects the shared-memory exhaustion condition, retries the step with the resources it needs, and only escalates if the failure recurs with adequate shared memory, which would indicate a genuine leak rather than a small default.