The 2026 CI/CD Benchmark Report

Build time and cost per pipeline, benchmarked by language, runner size, and provider so every variable is isolated.

Executive summary

A benchmark is only useful if it is comparable, so this report fixes the pipeline shape and varies one thing at a time: first the language, then the runner size, then the runner provider. That discipline is what makes it possible to say not just 'Rust is slow' but how much of the cost is the stack, how much is the runner, and how much is recoverable. Without isolating the variables, a benchmark collapses into a single number that hides every decision that actually moves it.

Two axes dominate the results. Language sets the floor: compiled and generics-heavy stacks carry several times the build time of fast-compiling ones, and because cost is build time times a fixed per-minute rate, that spread shows up directly in cost per run. Runner size then trades money for wall-clock, but the trade is not linear: doubling the cores rarely halves the time once a build is no longer CPU-bound, so past a point extra cores mostly add cost.

The third axis is the provider, and it is where the easy win lives. Running the identical benchmark on a managed Linux runner lands the same build at roughly 69 percent lower cost per run, before counting the premium-OS minutes it displaces. Because nothing in the workflow changes except the runner label, this is the rare benchmark lever that improves cost without any engineering trade-off, which is why it deserves its own axis rather than being folded into the language tables.

The benchmark also makes a point about what headline numbers leave out. A clean run assumes every job passes the first time, but real pipelines pay for flaky re-runs that the published figures exclude. Since most of those failures are transient and pass on retry, the gap between the benchmark and the real bill is largely avoidable, and closing it is a matter of recovering transient failures automatically rather than re-running by hand.

Read together, the three axes give a team a way to locate its own pipeline. The language axis tells you your floor, the size axis tells you whether you are over-provisioned, and the provider axis tells you how much you are leaving on the table by running the same work on a more expensive runner. The full report publishes the per-language and per-size tables and the methodology to reproduce them, so a team can place its own numbers on the same grid.

Language sets a 5x floor on build time before any tuning

The fastest and slowest benchmarked stacks differ by roughly 5x on identical hardware, with Go-style fast compiles at the bottom near five minutes and Rust at the top near twenty-five. That spread is structural: it comes from the compile model, the type system, and the amount of code generation each stack does, none of which a team changes by tuning its pipeline.

Because cost is build time times a fixed per-minute rate, the same spread shows up directly in cost per run. A Go run models at four cents and a Rust run at twenty, a 5x difference that exactly mirrors the time difference, because on a single runner the only thing converting time to money is the per-minute rate. The language you build in is therefore a cost decision as much as a speed one.

The practical implication is that the realistic optimization target is always relative to a stack's own floor, not a global average. Telling a Rust team to hit a Go team's build time is meaningless; the right question is whether the Rust pipeline is near its own achievable floor or sitting well above it because of cold caches and oversized matrices. The benchmark gives each stack its floor so a team can measure the gap that is actually closeable.

Bigger runners buy wall-clock with diminishing returns

Moving a Java build from two to four cores cuts time meaningfully, from seventeen minutes to eleven in the model, but four to eight returns less and eight to sixteen returns almost nothing, dropping only from eight minutes to seven. Each doubling of cores costs proportionally more per minute while the time saved shrinks, so the cost-per-improvement worsens at every step up the curve.

The reason is that a build stops being CPU-bound once its parallel work is exhausted. Past that point the long pole is a serial step, a single large compilation unit or an unshardable test, that no number of extra cores can speed up. The cores are real, but the work cannot use them, so the additional capacity is paid for and left idle.

Right-sizing therefore means finding the knee of this curve rather than buying the biggest runner. The knee is the size where wall-clock stops improving meaningfully, and choosing it captures most of the speed at a fraction of the cost of the largest tier. A managed layer that right-sizes per job lands at that knee automatically instead of defaulting every job to one oversized tier.

• Java build models at 17 min on 2-core, 11 on 4-core, 8 on 8-core, 7 on 16-core: returns shrink at each step.
• Past the point a build is CPU-bound, extra cores add cost without adding speed.
• Right-sizing means picking the knee of the curve, not the largest available runner.

Premium-OS runners dominate the cost table

The same Java benchmark that costs $0.14 on hosted Linux costs $1.36 on hosted macOS, nearly ten times as much. That multiple is not a rounding error in a cross-platform matrix; it is the dominant term, and it tracks the published OS multipliers directly, with macOS at 10x and Windows at 2x the Linux rate.

Any benchmark that mixes operating systems is therefore really a benchmark of OS placement. A pipeline that runs its OS-agnostic work, linting, unit tests, dependency install, on macOS is paying the macOS rate to do work that would pass identically on Linux. The expensive runner ends up doing cheap work, and the bill is dominated by a placement decision rather than by the work itself.

The single biggest cost decision is thus whether a job genuinely needs a premium OS at all. Reserving macOS and Windows for signing, packaging, and platform UI tests, and running everything else on Linux, frequently cuts a cross-platform bill by a third or more with no loss of coverage, because the genuinely platform-specific surface is far smaller than the matrix that grew around it.

• The Java benchmark costs $0.14 on hosted Linux and $1.36 on hosted macOS, nearly 10x.
• OS multipliers from GitHub pricing: Windows 2x, macOS 10x the Linux rate.
• A mixed-OS benchmark is really a benchmark of OS placement, not of the work itself.

The provider axis is the cheapest win on the board

Holding language, pipeline, and runner size fixed and changing only the provider, the managed Linux runner lands the identical build at about 69 percent lower cost per run, four cents against fourteen for the Java benchmark. Nothing in the workflow changes except the runner label, so there is no migration of logic, no rewrite, and no behavioral risk.

This is the rare benchmark lever that improves cost without any engineering trade-off. Every other lever asks for something: sharding tests takes work, right-sizing requires tuning, dropping a premium OS means moving jobs. The provider swap asks only that the same job run on a cheaper runner of the same shape, which is why it is the first place a cost-conscious team should look.

The saving comes from the per-minute rate, with the published Linux hosted rate at 0.008 and the managed blended rate modeling at 0.0025. Because the rate is lower and the job is identical, the cost falls proportionally on every run, and the saving compounds across the thousands of runs a busy repository executes in a week.

Build time by language converges less than the spread suggests

The 5x spread in build time is real, but it is measured on the build step in isolation. Once a real pipeline adds dependency install, environment setup, and a cold runner, the relative gap between languages narrows, because every stack pays similar overhead for the surrounding phases regardless of how fast its build step is.

This matters for how a team reads the benchmark. The build-by-language chart shows the structural floor, but a team's actual wall-clock includes the shared prologue, and a slow prologue can make a fast-compiling language feel as slow as a slow-compiling one. A Go pipeline with a cold cache and a cold runner can lose its compile-speed advantage to setup tax.

The takeaway is to read the language axis as the floor and the surrounding phases as how far above the floor you sit. A team near its language floor has little to gain from caching and warm capacity and should look at sharding or right-sizing; a team well above its floor should fix the prologue first, because that is shared overhead the benchmark's clean build step does not show.

Cost per run scales with the matrix, not the single leg

The benchmark reports one leg, but real pipelines run a matrix of language versions, operating systems, and dependency sets, and the bill is the per-leg cost times the matrix width. A four-cent Go run becomes a much larger number once it is multiplied across a six-way matrix that runs on every push.

That multiplication is where the per-leg decisions compound. A leg that runs OS-agnostic work on a premium OS is expensive once; across a wide matrix run thousands of times a week it is expensive thousands of times. The same is true of an oversized runner or a cold cache: each per-leg inefficiency is paid on every leg of every run.

Reading the benchmark at the matrix level rather than the single-leg level changes the optimization order. The biggest savings come from fixing the decision that repeats across every leg, OS placement, runner size, cache warmth, before tuning any individual stack, because a per-leg fix multiplied by the matrix width is far larger than a one-time improvement to a single leg.

Benchmarks understate cost until you price in re-runs

A clean benchmark run assumes every job passes the first time, but real pipelines pay for flaky re-runs that the headline numbers exclude. A transient failure, a registry timeout, a network blip, or an out-of-memory kill turns a green run red, a developer re-runs it, and the effective cost per change is suddenly double the benchmark for that run.

Most of those failures are transient and pass on retry, which means the re-run produced no new information; it just paid the full benchmark cost again to confirm what the first run would have shown without the hiccup. A small flake rate on a busy repository quietly lifts the effective cost per run well above the published figure, and the gap is invisible because it hides in re-run noise rather than in any single number.

Auto-healing those transient failures keeps effective cost per run close to the benchmark instead of inflating it with avoidable repeat runs. When a known-transient failure is retried on a fresh environment automatically, before a human triggers a full re-run, the pipeline recovers without paying for itself twice, and the real bill stays near the benchmark the report publishes.

Recommendations

Benchmark your own pipeline by isolating one variable at a time

Fix the pipeline shape and vary language, then size, then provider, so you can attribute cost to the right cause. A single blended number hides whether your spend is the stack, the runner, or the provider, and you cannot fix what you cannot attribute.

Right-size to the knee of the curve, not the largest tier

Extra cores stop buying wall-clock once a build is no longer CPU-bound. Find the runner size where time stops improving meaningfully and choose that, capturing most of the speed at a fraction of the cost of the biggest runner.

Treat OS placement as the biggest cost decision in the matrix

A macOS minute costs roughly ten times a Linux one. Run OS-agnostic work on Linux and reserve macOS and Windows for signing, packaging, and platform tests. Because the matrix multiplies the saving across every leg of every run, this is usually the largest single cost reduction available.

Take the provider win because it carries no engineering trade-off

Running the identical job on a managed Linux runner lands it at roughly 69% lower cost per run with no change to the workflow. Unlike sharding or right-sizing, the provider swap asks for no rewrite and introduces no behavioral risk, so it is the first lever to pull.

Auto-heal transient failures so the real bill matches the benchmark

Flaky re-runs inflate effective cost per run above the published benchmark. Retrying mechanical failures on a fresh environment keeps the real cost close to the clean benchmark and stops a small flake rate from quietly doubling the cost of affected changes.

Outlook

Through 2026 and into 2027, expect benchmarking to shift from comparing languages to comparing pipeline operations, because the language floor is fixed but the gap above it is where teams actually differ. As repositories grow and matrices widen, the cost of running the same work on the wrong runner, the wrong OS, or with a cold cache becomes the dominant variable, and the teams that benchmark their operations rather than their stack will pull ahead on both speed and spend.

The provider axis will keep widening as managed runner layers mature. The combination of a lower per-minute rate, warm pools that remove cold starts, persistent caches, and self-healing that keeps the real bill near the benchmark is converging into a baseline that is simply cheaper and faster than self-managed hosted runners for the same work. As that gap grows, the provider swap moves from an easy win to an obvious default.

For most teams the practical takeaway is that the benchmark is a map, not a verdict. Locate your pipeline on the language, size, and provider axes, fix the variable that is costing you the most, and price in the re-runs the clean numbers hide. The organizations that do that will run their CI near the benchmark floor while their peers pay a premium for the gap between the clean number and the real one.

Methodology

This report combines published GitHub-hosted runner pricing with Latchkey analysis of a fixed benchmark pipeline run across languages and runner sizes. Figures labeled "modeled" are illustrative estimates derived from representative per-stack builds and public pricing, not a primary survey. Cost per run is computed as billed minutes times the published per-minute rate for the runner (Linux 0.008, managed 0.0025, a 69% reduction), and OS multiples (Linux 1x, Windows 2x, macOS 10x) are from GitHub pricing. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• GitHub Actions - billing & pricing
• DORA State of DevOps Report
• Stack Overflow Developer Survey
• GitHub - Octoverse

More Latchkey reports