How to Run CI Runners on Spot Instances and Handle Interruption

Spot and preemptible instances cut runner cost sharply, but the cloud can reclaim them, so runners must be ephemeral and interruption-aware.

Spot (AWS) and preemptible (GCP) instances are far cheaper than on-demand but can be reclaimed with a short notice. Run only ephemeral runners on them so a lost instance means at most one lost job, and watch the interruption endpoint to drain gracefully. Route long or non-retryable jobs to on-demand capacity with labels.

Steps

• Use only ephemeral runners on spot, so reclamation loses at most one job.
• Poll the instance metadata interruption notice and stop accepting new jobs when it fires.
• Let CI retry the interrupted job on a fresh runner; route critical jobs to on-demand via labels.

Watch for a spot interruption

#!/usr/bin/env bash
# AWS: interruption notice appears ~2 minutes before reclaim
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
while true; do
  CODE=$(curl -s -o /dev/null -w '%{http_code}' \
    -H "X-aws-ec2-metadata-token: $TOKEN" \
    http://169.254.169.254/latest/meta-data/spot/instance-action)
  [ "$CODE" = "200" ] && { echo "spot interruption; draining"; ./drain-runner.sh; break; }
  sleep 5
done

Gotchas

• Never run a stateful or non-idempotent deploy step on a spot runner; it may vanish mid-job.
• The interruption notice is short (around two minutes on AWS), so drain quickly and rely on job retry.

Related guides