Self-Healing CI: Recovering from Spot-Instance Preemption (Exit 143)
When a spot runner is reclaimed mid-job, the build did not fail - the machine was taken away. The right move is to run it again on fresh capacity.
The problem
A job dies with exit code 143 (SIGTERM) partway through because the underlying spot/preemptible instance was reclaimed by the cloud provider. Nothing is wrong with the code; a human re-runs the job on a new instance and it completes unchanged.
Received SIGTERM, terminating...
##[error] The runner has received a shutdown signal.
Process completed with exit code 143.Why it happens
Spot and preemptible instances trade a lower price for the provider’s right to reclaim them on short notice. When capacity is needed elsewhere, the instance gets a termination signal (SIGTERM) and the job is cut off mid-flight.
Exit 143 is 128 + 15 - the process exiting on signal 15 (SIGTERM). It reflects the host going away, not anything your build did, which is why a plain re-run on fresh capacity succeeds.
The manual fix
The manual recovery is to re-run on fresh capacity:
- Re-run the job so it lands on a new instance.
- Make long steps idempotent/resumable so a reclaim mid-step does not corrupt state.
- For critical jobs, run on on-demand capacity to avoid reclamation entirely.
How this gets automated
A preemption has an unmistakable signature - a termination signal from the host, not an application error - and the safe response is to retry on new capacity. A self-healing CI pipeline recognizes the preemption, retries the job on a fresh instance, and only counts a failure that reproduces without a reclaim - so an interrupted spot job never shows up as a red build.