What Is a Stale Runner? The Drifted Machine That Flakes Builds
A stale runner is one that has drifted from its intended state - an outdated agent version, an unpatched image, or accumulated leftover state - and now causes failures.
Stale runners are a leading cause of "it works on most runners but not that one". Persistent runners drift over time; the longer they live, the further from a clean baseline they get and the more mysterious their failures become.
How runners go stale
- Agent drift: the runner version falls behind and is rejected or deprecated.
- Image drift: the OS and tools age out of date with security patches missing.
- State drift: leftover files, caches, and processes accumulate across jobs.
- Registration drift: ghost runners linger offline after their machine is gone.
Why staleness causes flakiness
A drifted runner behaves differently from a fresh one: a build that passes on clean machines fails on the stale one because of leftover state or an old tool. These are maddening to debug because the code is fine - the machine is not.
Preventing staleness
Ephemeral runners cannot go stale: each is fresh and destroyed after one job. For persistent runners, you need automated agent updates, regular image rebuilds, and per-job cleanup - which is ongoing ops work.
Staleness on managed runners
Latchkey runners are ephemeral and rebuilt from a maintained image, so staleness simply does not occur - every job gets a current, clean machine.
Spotting a stale runner
Telltale signs are a job that fails only on one specific runner, deprecation warnings about the runner version, or disk-full and leftover-process errors that clear after a reboot. When the symptom moves with the machine rather than the code, suspect staleness.
Key takeaways
- A stale runner has drifted in agent, image, or state and now fails.
- It is a leading cause of "fails only on that one runner" flakiness.
- Ephemeral runners cannot go stale; persistent ones need active upkeep.
- Managed ephemeral runners avoid staleness entirely.