hyperfine: Benchmark Commands with Warmup and JSON
hyperfine runs a command many times and reports the mean, standard deviation, min, and max of its wall-clock time.
For a repeatable performance number you can gate on, hyperfine beats hand-rolled time loops. It warms up caches, discards outliers, and can export JSON for regression checks.
What it does
hyperfine executes a shell command a configurable number of times, times each run, and prints statistics (mean, stddev, min, max). It supports warmup runs, parameterized sweeps, and comparison of multiple commands in one invocation.
Common usage
hyperfine 'grep -R TODO src/'
hyperfine --warmup 3 --runs 20 'cargo build --release'
# compare two commands and pick a relative speedup
hyperfine 'sort file.txt' 'sort --parallel=4 file.txt'Options
| Flag | What it does |
|---|---|
| --warmup N | Run the command N times before measuring (fills caches) |
| --runs N (-r) | Perform exactly N measured runs |
| --min-runs / --max-runs | Bounds when hyperfine chooses the run count |
| --prepare <cmd> | Run a setup command before each timed run |
| --export-json <file> | Write full results as JSON |
| --export-markdown <file> | Write a Markdown comparison table |
| -N / --shell=none | Skip the intermediate shell for lower overhead |
In CI
Export JSON with --export-json and parse the mean to gate a PR. A common pattern: run hyperfine on main and on the branch, then fail if the branch mean exceeds the baseline by a threshold. Use --warmup so disk and page caches do not skew the first result.
Common errors in CI
"Command terminated with non-zero exit code" means the benchmarked command itself failed; add --ignore-failure only if you intend to time failing runs, otherwise fix the command. Highly variable results on shared runners come from noisy neighbors; raise --warmup and --runs, or pin the job to a less contended runner. "The flag --shell=none cannot be used with a command that contains shell syntax" means your command has pipes or globs; drop -N.