How to Analyze Canary Metrics From Prometheus in CI/CD

A metric query against Prometheus turns "looks fine" into a hard pass or fail for promoting a canary.

The heart of any progressive strategy is the analysis step. Query Prometheus for the canary success rate and latency over a short window, compare against thresholds, and let the result decide whether the pipeline promotes or rolls back.

How it works

You issue PromQL queries scoped to the canary version label, evaluate them against successCondition thresholds, and exit non-zero when they fail. This is the same logic Argo Rollouts and Flagger run internally; doing it explicitly in CI works when you have no rollout controller.

Success-rate and latency queries

queries.promql

# Success rate for the canary over the last 5 minutes:
sum(rate(http_requests_total{app="myapp",track="canary",status!~"5.."}[5m]))
/ sum(rate(http_requests_total{app="myapp",track="canary"}[5m]))

# p99 latency for the canary:
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{app="myapp",track="canary"}[5m])) by (le))

Evaluate the queries and gate promotion

.github/workflows/ci.yml

analyze:
  runs-on: ubuntu-latest
  steps:
    - name: Query and evaluate canary metrics
      run: |
        Q='sum(rate(http_requests_total{app="myapp",track="canary",status!~"5.."}[5m]))/sum(rate(http_requests_total{app="myapp",track="canary"}[5m]))'
        SR=$(curl -fsS --data-urlencode "query=$Q" https://prom/api/v1/query | jq -r '.data.result[0].value[1]')
        echo "canary success rate: $SR"
        awk "BEGIN{exit !($SR > 0.99)}" || { echo "success rate below 99%, roll back"; exit 1; }

Avoid noisy verdicts

Use a window long enough to gather meaningful samples but short enough to catch regressions fast (2 to 5 minutes is common). Require a minimum request count before trusting a rate, or a low-traffic canary can pass on too few samples.