CI Deadlock - Tests or Build Stuck With No Progress
A deadlock is different from a transient hang: two or more parties are each waiting on the other, so the program can never make progress. A retry runs into the same wall - this is usually a real bug to fix, though a job-level timeout still recovers the runner.
What this error means
A test suite or build freezes with no output and never finishes, hitting the job timeout. Unlike a flaky network hang, it reproduces - the deadlock is in the code or in resource ordering, not in the network. Some runtimes detect it outright (Go’s "all goroutines are asleep - deadlock!").
fatal error: all goroutines are asleep - deadlock!
# or (no detector - just a hang):
PASS test_a
PASS test_b
# (frozen, then job timeout)Common causes
Circular lock / resource acquisition
Thread A holds lock 1 and waits for lock 2 while thread B holds lock 2 and waits for lock 1. Neither can proceed - a classic deadlock that reproduces deterministically.
Waiting on a result that is never produced
A consumer blocks on a channel/queue/future that nothing ever fulfills (e.g. the producer exited early or a barrier was never reached), so it waits forever.
How to fix it
Capture a stack dump at the hang
Get thread/goroutine stacks so you can see what each party is waiting on. This is real debugging, not a retry.
# Java: thread dump
jstack <pid>
# Go: send SIGQUIT to dump all goroutine stacks
kill -QUIT <pid>
# Python: py-spy dump --pid <pid>Bound the wait so it fails fast
- Add per-test and per-step timeouts so a deadlock surfaces quickly instead of burning the full job budget.
- Fix the root cause: consistent lock ordering, or guaranteeing the awaited result is always produced.
- Add timeouts to blocking primitives (channel receives,
Future.get, lock acquisition).
How to prevent it
- Enforce a consistent lock-acquisition order across the codebase.
- Put timeouts on every blocking wait so a deadlock fails fast.
- Run tests with a per-test timeout to catch hangs in CI.