GitLab CI "no space left on device" - Runner Disk Exhausted

By Kaveh Alemi·Latchkey

A write failed because the runner host or container ran out of disk. On long-lived runners this accumulates from cached layers, build dirs, and old artifacts that are never pruned.

What this error means

A step fails with "no space left on device" - often during checkout, dependency install, or docker build. It commonly affects one busy runner while others are fine, and clears after cleanup.

Job log

fatal: cannot create directory at 'node_modules': No space left on device
# or
write /var/lib/docker/tmp/...: no space left on device

Diagnose it: which rule matched, and on which runner?

GitLab evaluates rules: top to bottom and the first match wins, including one that sets when: never. A job that does not run, or runs when you did not expect it to, is nearly always matching an earlier rule than the one you are reading.

.gitlab-ci.yml

# validate the definition against the project
curl -s --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/ci/lint" \
  --data-urlencode "content=$(cat .gitlab-ci.yml)"

# what the job actually sees
script:
  - env | grep -E "^CI_(PIPELINE_SOURCE|COMMIT_REF_NAME|RUNNER)" | sort

Common causes

Build/cache buildup on a long-lived runner

Each job leaves caches and build directories. Without cleanup, a persistent runner’s disk slowly fills until writes fail.

Dangling Docker images and volumes

On a dind or socket runner, old layers, stopped containers, and anonymous volumes consume the bulk of the disk over time.

How to fix it

Reclaim disk on the runner

Runner host

docker system prune -af --volumes
sudo gitlab-runner cache-clear   # if using local cache
df -h

Prevent buildup automatically

Schedule periodic docker system prune and cache cleanup on persistent runners.
Prefer ephemeral runners (autoscaling/Kubernetes) that start with a clean disk.
Set expire_in on artifacts and bound cache size so storage stays in check.

How to prevent it

Use ephemeral runners or schedule disk cleanup on persistent ones.
Prune Docker images/volumes regularly on dind hosts.
Cap cache size and set artifact expire_in.

Frequently asked questions

What causes GitLab CI "no space left on device"?

There are 2 common causes: build/cache buildup on a long-lived runner and dangling docker images and volumes. Each job leaves caches and build directories.

How do I fix GitLab CI "no space left on device"?

There are 2 fixes depending on which cause you have: reclaim disk on the runner and prevent buildup automatically. Work through them in order, since the first is the most common.

What does GitLab CI "no space left on device" actually mean?

A step fails with "no space left on device" - often during checkout, dependency install, or docker build.

How do I stop GitLab CI "no space left on device" happening again?

Use ephemeral runners or schedule disk cleanup on persistent ones. The prevention section lists 3 changes that keep it from recurring.

Can Latchkey fix this automatically?

Yes. Latchkey runs your GitHub Actions on managed runners that detect this failure, apply the fix, and retry the job automatically - self-healing is on by default.

References

GitHub Actions documentation