Self-Healing CI: Recovering from a Spot Instance Interruption
When a spot instance is reclaimed, the build did not fail -- the machine was taken away. Running it again on fresh capacity is the right move.
The problem
A job stops abruptly mid-run because the cloud provider reclaimed the underlying spot instance with little warning. The code is fine; the runner disappeared. A human re-runs the job on new capacity and it completes unchanged.
The runner has lost communication with the server.
# the spot instance received an interruption notice and was reclaimedWhy it happens
Spot/interruptible capacity is offered cheaply in exchange for the provider’s right to reclaim it when it needs the capacity elsewhere. When that happens, the instance is interrupted on short notice and any job on it is cut off mid-flight.
The interruption reflects the host going away, not anything your build did, which is why a plain re-run on fresh capacity succeeds with no change.
The manual fix
The manual recovery is to re-run on fresh capacity:
- Re-run the job so it lands on a new instance.
- Make long steps idempotent/resumable so an interruption does not leave corrupt state.
- Run critical jobs on on-demand capacity to avoid interruptions entirely.
How this gets automated
A spot interruption has an unmistakable signature -- the host is reclaimed, not an application error -- and the safe response is to retry on new capacity. A self-healing CI pipeline recognizes the interruption, reruns the job on a fresh instance, and only counts a failure that reproduces without an interruption, so a reclaimed instance never shows up as a red build.