How to Alert on Pipeline Failures With Alertmanager and PagerDuty

A Prometheus rule fires when CI failures cross a threshold, and Alertmanager routes that alert to PagerDuty.

Write an alerting rule over your ci_jobs_total counter, then add a PagerDuty receiver in Alertmanager keyed on a routing/integration key. Group by workflow so one flurry of failures does not page ten times.

Steps

• Define a Prometheus rule that fires when failures exceed a threshold.
• Add a pagerduty_configs receiver in Alertmanager with the integration key.
• Route the alert to that receiver and group by workflow.

Alert rule

groups:
  - name: ci
    rules:
      - alert: CIPipelineFailing
        expr: |
          sum(increase(ci_jobs_total{conclusion="failure"}[15m])) by (workflow) > 3
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "CI failing for {{ $labels.workflow }}"

Alertmanager receiver

receivers:
  - name: pagerduty
    pagerduty_configs:
      - routing_key: ${PAGERDUTY_KEY}
        severity: critical

Related guides