hyperfine --export-json: CI Performance Gates
hyperfine --export-json results.json writes each benchmark result, including mean, stddev, min, and max in seconds, so a script can assert a performance budget.
A benchmark only protects you if CI fails when it regresses. The JSON export plus jq plus a threshold check is the whole pattern.
What it does
The --export-json flag serializes every benchmarked command into a JSON object with a results array; each entry has command, mean, stddev, median, min, max, and times (all in seconds). You parse mean with jq and compare it against a budget.
Common usage
hyperfine --warmup 3 --export-json bench.json './app --run'
# fail the job if mean exceeds 0.5s
mean=$(jq '.results[0].mean' bench.json)
awk -v m="$mean" 'BEGIN{ exit (m > 0.5) }' \
|| { echo "Perf regression: ${mean}s > 0.50s budget"; exit 1; }Options
| Flag / field | What it does |
|---|---|
| --export-json <file> | Write full results as JSON |
| --export-csv <file> | Write results as CSV |
| .results[].mean | Mean time in seconds (JSON field) |
| .results[].stddev | Standard deviation in seconds |
| .results[].min / .max | Fastest / slowest observed run |
| --output <where> | Control whether the benchmarked command output shows |
In CI
Commit a baseline JSON and compare the new mean against baseline.mean * 1.1 rather than a hardcoded second value, so the gate tracks the machine. Because shared runners vary, gate on a generous threshold (10-20 percent) and use --warmup to keep the first run from tripping it.
Common errors in CI
jq: error: Cannot index ... with "results" means hyperfine wrote nothing (the benchmark failed before exporting); check its exit code first. awk comparisons treat the value as a string if the locale uses commas for decimals; force LC_ALL=C. Times are in seconds, not milliseconds, so a budget of 500 will never trip.