Docker "mkdir /var/lib/docker: read-only file system" in CI

By Kaveh Alemi·Latchkey

The daemon stores images, layers, and container state under its data-root (default /var/lib/docker). If that path lands on a read-only mount - a read-only root filesystem, a remounted volume, or a misconfigured data-root - the daemon cannot create directories and fails with read-only file system.

What this error means

A pull, build, or run fails with Error response from daemon: mkdir /var/lib/docker/...: read-only file system. The data-root is on a read-only mount.

docker

Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/.../merged: mkdir /var/lib/docker/overlay2: read-only file system

Diagnose it: read the container, not the compose file

A container that exits immediately in CI has almost always logged the reason and then been cleaned up. Capture the logs and the exit code before changing configuration.

Terminal

# why did it stop?
docker ps -a --format '{{.Names}}\t{{.Status}}\t{{.Image}}'
docker logs <container> 2>&1 | tail -50
docker inspect <container> --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}'

Common causes

A read-only root filesystem

On hardened or immutable hosts, /var/lib/docker may sit on a read-only root, blocking writes.

A misconfigured or remounted data-root

A data-root pointed at a read-only volume cannot store image and container data.

How to fix it

Point data-root at a writable path

Set the daemon data-root to a writable mount and restart.

daemon.json

# /etc/docker/daemon.json
{ "data-root": "/mnt/docker-data" }
# then: sudo systemctl restart docker

Remount the data-root read-write

If the existing mount should be writable, remount it rw.

Terminal

mount | grep /var/lib/docker
sudo mount -o remount,rw /var/lib/docker

How to prevent it

Keep the daemon data-root on a writable mount.
Avoid placing /var/lib/docker on a read-only root.
Verify mount options when provisioning Docker hosts.

Frequently asked questions

What causes Docker "mkdir /var/lib/docker: read-only file system" in CI?

There are 2 common causes: a read-only root filesystem and a misconfigured or remounted data-root. On hardened or immutable hosts, /var/lib/docker may sit on a read-only root, blocking writes.

How do I fix Docker "mkdir /var/lib/docker: read-only file system" in CI?

There are 2 fixes depending on which cause you have: point data-root at a writable path and remount the data-root read-write. Work through them in order, since the first is the most common.

What does Docker "mkdir /var/lib/docker: read-only file system" in CI actually mean?

A pull, build, or run fails with Error response from daemon: mkdir /var/lib/docker/...: read-only file system.

How do I stop Docker "mkdir /var/lib/docker: read-only file system" in CI happening again?

Keep the daemon data-root on a writable mount. The prevention section lists 3 changes that keep it from recurring.