How to Detect Performance Regressions Against a Baseline in CI

A baseline comparison catches gradual slowdowns a fixed budget misses by failing when this run is meaningfully slower than the last good one.

Store a baseline summary as an artifact or in the repo, restore it in CI, then compare the current p95 to it. Fail when the regression exceeds an allowed percentage such as 10 percent.

Steps

• Persist a baseline summary.json from a known-good run.
• Restore it in the PR job (cache or a committed file).
• Compare current vs baseline and fail if slower by more than the allowance.

Comparison step

CUR=$(jq '.metrics.http_req_duration.values["p(95)"]' summary.json)
BASE=$(jq '.metrics.http_req_duration.values["p(95)"]' baseline.json)
awk -v c="$CUR" -v b="$BASE" 'BEGIN{
  limit=b*1.10;
  printf "current %.0f  baseline %.0f  limit %.0f\n", c, b, limit;
  exit (c>limit)?1:0
}'

Gotchas

• Runner variance adds noise; use a generous allowance or run against a fixed environment.
• Refresh the baseline only from the default branch, or a bad merge poisons it.

Related guides