hyperfine: Usage, Options & Common CI Errors
hyperfine measures how long a command takes, with statistics across runs.
hyperfine benchmarks CLI commands with warmups, multiple runs, and statistical output. In CI it is used for perf regression checks, where exporting JSON and tolerating noise matter.
What it does
hyperfine runs a command many times, applies warmup runs, detects statistical outliers, and reports mean, min, max, and standard deviation. It can compare multiple commands and export results as JSON/CSV/Markdown for regression gating.
Common usage
hyperfine 'grep foo bigfile'
hyperfine --warmup 3 --runs 20 './build.sh'
hyperfine 'cmd-a' 'cmd-b' # compare two commands
hyperfine --export-json bench.json './script'
hyperfine -N 'echo hi' # -N: no intermediate shellOptions
| Flag | What it does |
|---|---|
| --warmup <N> | Run N warmup iterations before timing |
| --runs <N> | Force exactly N timed runs |
| --prepare <cmd> | Run before each timed run (e.g. clear cache) |
| --export-json <file> | Write results for later comparison |
| -N / --shell=none | Execute directly without a shell wrapper |
| -i / --ignore-failure | Do not abort when the command exits non-zero |
Common errors in CI
hyperfine aborts with "Command terminated with non-zero exit code: N. Use the '-i'/'--ignore-failure' option ..." if the benchmarked command fails - fix the command or pass -i for commands expected to fail. "Statistical outliers were detected" is a warning that the shared CI runner is noisy; add --warmup and more --runs, or use --shell=none to cut overhead. Comparing means across different runner classes produces false regressions - pin the runner or compare relative deltas.