What Is a Mechanical Failure in CI? Explained
A mechanical failure is one rooted in the machine or infrastructure running the job, such as an out-of-memory kill or a preempted instance, rather than in your code.
When people say a build "failed", they often mean the tests failed. But a meaningful fraction of failures never reach your code at all: the runner ran out of memory, the disk filled, the instance was reclaimed. These mechanical failures are environmental, and they behave very differently from logic errors.
What counts as mechanical
- Out-of-memory kills (the process is SIGKILLed, often surfacing as exit code 137).
- Disk-full or inode-exhaustion conditions on the runner.
- Spot or preemptible instance reclamation mid-job (often SIGTERM, exit 143).
- Runner provisioning hiccups and brief provider incidents.
Mechanical vs logical failures
A logical failure comes from your code or tests: an assertion is false, a type does not check, a file is missing. A mechanical failure comes from the machine: it ran out of a resource or was taken away. Logical failures reproduce every time; mechanical ones usually do not, because they depend on transient resource conditions.
Why they look confusing
Mechanical failures often surface as cryptic signals rather than helpful errors. An out-of-memory kill leaves only the word "Killed" and exit code 137; a preemption may show a generic cancellation. Without context, these get misfiled as flaky tests or random failures.
The right response: retry
Because mechanical failures are environmental, the correct response is usually to retry, ideally on a fresh runner with adequate resources. Retrying a logical failure is pointless, but retrying a mechanical one frequently succeeds because the transient resource condition has passed.
The Latchkey angle
Mechanical failures are precisely what Latchkey self-healing managed runners are built to catch. The platform detects mechanical and transient failures such as out-of-memory kills, network blips, and registry timeouts and automatically retries, so a one-off blip does not fail your build, while real code failures still fail fast.
Key takeaways
- A mechanical failure is in the machine or infrastructure, not your code.
- Out-of-memory kills, disk-full, and preemptions are typical examples.
- They rarely reproduce, unlike logical (code) failures.
- The right response is usually a retry, often on a fresh runner.