How to Set Metric Gates on a Canary in GitHub Actions

A canary is only safe if each traffic bump is gated on metrics; a breach must abort and revert automatically rather than continue to full rollout.

Between weight increases, query error rate and latency; proceed only if both are in budget, otherwise abort the rollout so the canary weight returns to zero.

Steps

• Increase canary weight in stages (e.g. 10%, 50%, 100%).
• After each stage, query the metric gates over a bake window.
• Abort and revert on any breach instead of continuing.

Workflow

- name: Gated canary
  env: { PROM: ${{ vars.PROM_URL }} }
  run: |
    set -e
    for w in 10 50 100; do
      ./set-canary-weight.sh "$w"
      sleep 120
      Q='sum(rate(http_requests_total{code=~"5..",track="canary"}[2m]))/sum(rate(http_requests_total{track="canary"}[2m]))'
      ERR=$(curl -sfG "$PROM/api/v1/query" --data-urlencode "query=$Q" | jq -r '.data.result[0].value[1] // "0"')
      echo "weight $w error=$ERR"
      if awk -v e="$ERR" 'BEGIN{exit !(e>0.02)}'; then
        echo "gate failed at $w%; aborting"; ./set-canary-weight.sh 0; exit 1
      fi
    done

Gotchas

• Compare canary metrics against the stable track, not an absolute baseline, to isolate the release effect.
• Setting weight back to 0 on breach is the rollback; verify traffic actually drained afterward.

Related guides