could not get lock /var/lib/dpkg/lock, and who is holding it

By Kaveh Alemi·Latchkey

A could not get lock /var/lib/dpkg/lock failure in CI is apt reporting that it asked the kernel for a write lock on a dpkg file and did not get it, and the rest of the line says whether another process was found holding that lock. That distinction is the whole page, because one version is a race you can wait out and the other is a permission or path problem that no amount of waiting will clear.

What this error means

An apt-get update, apt-get install or Ansible apt task stops with one or two lines beginning E:, naming a file under /var/lib/dpkg/. There are three shapes and they are not interchangeable. Could not get lock <file>. It is held by process <pid> (<name>) means the kernel told apt exactly which process owns the lock. Could not get lock <file>. It is held by process <pid> is the same finding with the process name missing, which happens when the holder exited between the two calls apt makes. Could not get lock <file> with an errno appended and no holder at all is a different condition entirely: apt asked who held the lock only when the failure looked like contention, so the absence of that clause means the lock attempt failed for some other reason, usually that you are not root or the path is not writable. Underneath, a second E: line names which of the two dpkg locks apt was taking, and its closing question is the tell: is another process using it? and are you root? are two different branches of the same error, chosen by errno.

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 3412 (unattended-upgr)
N: Be aware that removing the lock file is not a solution and may break your system.
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

Read the second line, not the first

Both lines come from the same failed attempt, and the second one is more specific than the first. GetLock in apt-pkg/contrib/fileutl.cc is what produces the Could not get lock line. It opens the file, asks for a write lock with F_SETLK, and only if that fails with EACCES or EAGAIN does it call F_GETLK to find out who has it. That second call is what fills in It is held by process. Any other errno skips the lookup, so the message comes out without a holder and with an errno string on the end instead.

The caller then adds its own line. debSystem::Lock in apt-pkg/deb/debsystem.cc takes two locks in order, the frontend lock at /var/lib/dpkg/lock-frontend and then the administration directory lock at /var/lib/dpkg/lock, and for each one it picks between two sentences using the same errno test. On EACCES or EAGAIN it asks is another process using it?. On anything else it asks are you root?. So a log that ends in are you root? is not telling you about a busy machine at all, and the fixes that everybody reaches for first will do nothing to it.

apt also volunteers an opinion about the obvious workaround, on a N: notice line right under the error, and it is worth repeating because deleting the lock file is still the top answer on the internet. The notice reads: "Be aware that removing the lock file is not a solution and may break your system."

Common causes

A boot-time updater still holds the lock when your first package step runs

Cloud images start unattended-upgrades and apt-daily timers on boot, and a job that begins installing packages in its first seconds arrives while one of them is mid-transaction. This is the case that names a holder, usually unattended-upgr, and it is genuinely transient: the holder finishes and the lock frees. It is also the case that a two-minute wait almost always survives, which is why the same workflow can pass a hundred times and fail on the hundred and first.

Two package steps in the same job run at the same time

A matrix that backgrounds an install, a setup action that shells out to apt while your own step is already running, or a & in a run block, and now two processes want the same exclusive lock. In our experience this is the version that repeats rather than flickers, because the overlap is structural rather than a matter of timing, and the holder named in the message is another apt-get rather than a system updater.

The step is not running as root

Inside a container that drops to a non-root user, or in a run block where the sudo went missing from the second command of a two-command line, the lock file cannot be opened for writing at all. The errno is not EACCES or EAGAIN in the sense apt tests for, so no holder is looked up and the second line asks are you root?. Every retry you add makes the job slower and none of them make it pass.

The dpkg directory is on a mount that cannot take a write lock

A read-only overlay, an unusual bind mount, or a network filesystem changes which errno comes back. apt has special handling for two of these: EROFS produces a warning about a read only lock file and continues, and ENOLCK produces a warning about an NFS mounted lock file and continues. Anything else falls through to the error path with no holder named, which is why an odd mount and a missing sudo look identical in the log.

How to fix it

Set a lock timeout on every apt-get call

1. Add -o DPkg::Lock::Timeout=180 to apt-get update and to every apt-get install. The option is read by debSystem::Lock, so it covers both the frontend lock and the administration directory lock.
2. Use a number rather than minus one in CI. Minus one waits forever, which turns a stuck updater into a job that burns its whole timeout with no output.
3. Put it on the command line rather than in /etc/apt/apt.conf.d, so the next person reading the workflow can see that the wait is deliberate.

sudo apt-get -o DPkg::Lock::Timeout=180 update
sudo apt-get -o DPkg::Lock::Timeout=180 install -y --no-install-recommends jq

Wait for the boot-time updaters instead of racing them

On an image you control, the cleaner move is to let cloud-init finish before any package work starts. It blocks until the first-boot work is done, which includes the package activity that holds the lock, and it exits non-zero if that work failed, so a broken image fails loudly instead of failing your install a minute later.

- name: Wait for first-boot package work
  run: sudo cloud-init status --wait

- name: Install packages
  run: sudo apt-get -o DPkg::Lock::Timeout=180 install -y --no-install-recommends libpq-dev

Fix the root case rather than retrying it

If the second line asks are you root?, stop adding retries. Run the step as root, or in a container add the package install to the image build where it runs as root, or grant the user write access to /var/lib/dpkg. The reason to treat this separately is that the fastest way to tell the two apart is already in your log, and the fix for one is inert on the other.

- name: Install packages
  run: |
    set -euo pipefail
    id -u
    sudo -n true || { echo "no passwordless sudo in this environment"; exit 1; }
    sudo apt-get -o DPkg::Lock::Timeout=180 install -y --no-install-recommends libpq-dev

Serialize your own package steps

Where the holder named in the message is another apt process of yours, the answer is one package step per job rather than a longer wait. Collect everything a job installs into a single apt-get install with all the package names, which is also faster, since apt resolves and downloads once instead of once per call.

sudo apt-get -o DPkg::Lock::Timeout=180 install -y --no-install-recommends \
  libpq-dev libvips-dev pkg-config

apt waits two minutes in CI. apt-get waits none.

This is the part that makes the failure look random, and it is a one-line difference in apt itself. In apt-private/private-cmndline.cc, apt 2.7.14 and 2.8.1 set Binary::apt::Dpkg::Lock::Timeout to minus one when standard input is a terminal and to 120 when it is not. The key is scoped to the apt binary by name. apt-get is not covered by it, so DPkg::Lock::Timeout stays at its lookup default of zero, and GetLockMaybeWait in debsystem.cc returns the immediate GetLock result when the timeout is zero.

Put those together and the behaviour in a job is this. apt install nginx in a non-interactive shell retries the lock once a second for two minutes before giving up. apt-get install nginx in the same shell fails on the first attempt. Almost every CI script uses apt-get, because that is what the Docker and Ubuntu documentation tells you to use for scripting, so almost every CI script has opted out of the wait without knowing it existed.

The fix is to ask for the wait explicitly rather than to switch binaries, since apt also prints an unstable, human-facing progress format that is worse to parse. Passing the option on the command line puts the choice in the workflow where a reviewer can see it.

- name: Install packages
  run: |
    sudo apt-get -o DPkg::Lock::Timeout=180 update
    sudo apt-get -o DPkg::Lock::Timeout=180 install -y --no-install-recommends libpq-dev

Why this page carries no recorded run

The dpkg lock is a property of one machine at one instant, and the way to make it fail on demand is to hold the lock yourself and then run apt. A job that does that proves only that the kernel implements fcntl locking, which was never in doubt, and it bills a runner minute to demonstrate it. What the page actually needs is the branch that chooses each sentence, and that is readable in apt at a named version without running anything.

So the block at the top is assembled from those two functions rather than pasted from a log, and it is labelled that way. It is also a composite in a second sense: the E: and N: prefixes come from a third function, and the process name in the middle comes from the kernel by way of GetProcessName. Searching for the whole line in apt will find nothing, and that is expected rather than evidence of invention. Nothing on this page claims Latchkey repairs this failure, because content/heal-evidence.mjs has no record for this slug.

How to prevent it

• Put -o DPkg::Lock::Timeout on every apt-get call in the repository, not just the one that failed last week.
• Bake packages into the image where you can. A package that is already installed takes no lock at all.
• Collapse several installs into one call, so a job never contends with itself.
• Alert on the are you root? wording separately from the is another process using it? wording, because only one of them is worth a retry.

Frequently asked questions

Related guides

References

• Debian/apt: GetLock in apt-pkg/contrib/fileutl.cc, where the holder lookup happens
• Debian/apt: debSystem::Lock and LockInner in apt-pkg/deb/debsystem.cc
• Debian/apt 2.8.1: the binary-scoped Dpkg::Lock::Timeout default in apt-private/private-cmndline.cc
• cloud-init: the status subcommand and what --wait blocks on
• GitHub Actions documentation