Self-Healing CI: Recovering When the Runner Briefly Loses Communication
A job marked failed because the runner briefly lost contact with the control server is an infrastructure blip, not your build failing -- a rerun completes it.
The problem
A job is failed because the runner momentarily lost its connection to the CI control server -- a brief network partition or heartbeat miss -- even though the work itself was healthy. The code is fine; the runner just dropped contact for a moment. A human re-runs the job and it completes unchanged.
##[error] The runner has lost communication with the server. Verify the machine is running and has a healthy network connection.Why it happens
A runner stays in contact with the control server through periodic heartbeats over the network. A brief partition, a momentary DNS or routing hiccup, or transient congestion can cause a missed heartbeat that marks the job failed even though the runner and the build were healthy.
It is an infrastructure-level connectivity blip, not a build error: nothing your code did caused it, and a rerun on a healthy connection completes the same work.
The manual fix
Manual handling for a lost-communication failure:
- Re-run the job -- the connection is almost always healthy on the retry.
- Make long steps idempotent/resumable so an interruption does not leave corrupt state.
- Investigate only if disconnections are frequent, which points to a real network problem rather than a blip.
How this gets automated
A lost-communication failure has an unmistakable infrastructure signature -- a connectivity drop, not an application error -- and the safe response is to rerun on a healthy runner. A self-healing CI pipeline recognizes the communication loss, reruns the job, and only counts a failure that reproduces with healthy connectivity, so a momentary partition never shows up as a red build.