Criterion.rs: Statistical Rust Benchmarks
Criterion.rs runs Rust benchmarks with statistical rigor, estimating each measurement with confidence intervals and comparing against a saved baseline.
Criterion.rs works on stable Rust (unlike the built-in nightly bench) and remembers previous runs, so it can report whether a change is a real improvement or regression.
What it does
Criterion.rs is a benchmarking crate driven by cargo bench. It runs each benchmark many times, fits a statistical model, and reports the estimate with a confidence interval. It stores results so a later run prints the change (improved, regressed, or no change) versus the previous or a named baseline.
Common usage
cargo bench
# save a named baseline (e.g. on main)
cargo bench -- --save-baseline main
# compare the current code against that baseline
cargo bench -- --baseline mainOptions
| Argument | What it does |
|---|---|
| --save-baseline <name> | Store results under a baseline name |
| --baseline <name> | Compare against a previously saved baseline |
| --load-baseline <name> | Compare two saved baselines without rerunning |
| <filter> | Only run benchmarks whose name matches |
| --sample-size <n> | Number of samples per benchmark |
In CI
On the base commit run cargo bench -- --save-baseline main, then on the branch run cargo bench -- --baseline main and inspect the reported change. Runner noise inflates variance, so keep sample sizes reasonable and prefer dedicated over shared runners for stable numbers.
Common errors in CI
"no benchmarks to run" or a benchmark that does not run means the [[bench]] target is missing harness = false in Cargo.toml, so cargo uses the default libtest harness instead of Criterion. Missing baselines ("baseline main not found") mean the --save-baseline step did not run or the target/criterion directory was not persisted between jobs; cache or pass it forward. High variance warnings come from noisy runners.