benchstat: Compare Go Benchmark Runs Statistically
benchstat summarizes repeated Go benchmark output and, given two files, reports the percentage change and whether it is statistically significant.
A single benchmark run is noisy. benchstat takes many runs, computes a robust central value, and tells you whether a change between baseline and branch is real or just noise.
What it does
benchstat reads the output of go test -bench (ideally with -count>1), computes a median or mean with a variation estimate, and when given two inputs prints the delta and a p-value, marking changes that are not statistically significant as "~".
Common usage
go install golang.org/x/perf/cmd/benchstat@latest
go test -bench=. -count=10 ./mypkg > old.txt # on main
go test -bench=. -count=10 ./mypkg > new.txt # on branch
benchstat old.txt new.txtOptions
| Argument / Flag | What it does |
|---|---|
| old.txt new.txt | Compare two result sets and show the delta |
| single file | Summarize one result set (mean and variation) |
| -col <key> | Group results by a label column (newer benchstat) |
| -filter <expr> | Select a subset of results (newer benchstat) |
In CI
The standard regression gate: benchmark main into old.txt, the PR into new.txt (both with -count=10), run benchstat old.txt new.txt, and fail the job if a key metric grew beyond a threshold. Changes flagged "~" are not significant and should not fail the build.
Common errors in CI
"benchstat: no data" means the input files are empty or not in go test benchmark format; confirm the -bench step actually produced BenchmarkXxx lines. If every delta is "~", the runs are too few or too noisy to be significant; increase -count. A newer benchstat changed its CLI (it now favors -col over positional old/new in some workflows), so pin a version if a script depends on the older interface.