The State of Load Testing in CI 2026

How teams fit performance and load tests into pipelines without blowing the compute budget or lying to themselves about fidelity.

Executive summary

Load testing is the part of CI most teams know they should do and least often actually automate. Functional tests gate every merge, but performance tests tend to live in a separate, manually triggered corner of the toolchain because they are compute-hungry, slow, and easy to misread when the test environment does not resemble production. The result is a quiet gap: teams ship code that passes every functional check and then discover under real traffic that a hot path got twice as slow three releases ago, with no commit to point at and no test that would have caught it.

The teams making progress in 2026 treat performance as a budgeted, scheduled signal rather than a per-push gate. They run focused load legs on representative hardware, fan out load generation across multiple runners to reach realistic concurrency, and compare against a tracked baseline instead of an absolute threshold. None of this is exotic, and that is the point: the hard problems in continuous load testing are cost and fidelity, not tooling. The tools have been good for years; what stops teams is the bill and the credibility of the numbers.

This report quantifies what load tests cost relative to ordinary CI, why undersized or shared runners produce numbers you cannot trust, and how cheap, fast-scaling managed fan-out changes the calculus of running performance work continuously rather than occasionally. The throughline is that most of the reasons teams give for not automating load tests, expense and unreliability, are mechanical and addressable, not fundamental properties of performance testing.

Three figures frame the year. Only a minority of teams run any load test inside CI at all, which means the practice is still a differentiator rather than table stakes. A full load leg consumes several times the compute of a functional leg, so the cost objection is real and has to be designed around rather than ignored. And a managed per-minute rate well below hosted pricing is what turns wide fan-out from a rare, hand-triggered event into something a team can afford to run on a schedule.

For engineering leaders the practical message is that continuous load testing does not require a performance engineering team or a heroic platform project. It requires a budget, a baseline, representative runners, and a fan-out mechanism that is cheap and fast enough to use routinely. The teams that put those four things in place catch regressions while they are still one commit deep, and they do it without the per-push cost that makes naive performance gating collapse.

Load tests are a compute outlier, so they need a budget not a gate

A full load test leg consumes several times the compute of an ordinary functional test, which is exactly why bolting it onto every push backfires. A functional leg validates correctness in a few minutes on a small runner; a load leg has to sustain traffic long enough for queues, garbage collection, connection pools, and caches to reach steady state, and it often needs more than one machine to generate that traffic. The compute multiple is not waste, it is the price of a signal that says anything about behavior under sustained pressure.

Because the cost is structural rather than incidental, the teams that sustain continuous performance testing stop trying to make it cheap enough to gate every commit and instead make it cheap enough to run often. They treat performance as a scheduled, budgeted signal against a tracked baseline, reserving the heavy legs for merges to main or nightly windows rather than blocking every developer push on an expensive run. A regression caught the next morning is still caught one commit deep, which is most of the value of a per-push gate at a fraction of the cost.

The baseline is what makes the budgeted approach work. An absolute threshold ("p95 under 200ms") rots as the application grows and gets disabled the first time it blocks a legitimate change. A baseline comparison ("p95 within 10 percent of last week") tracks the application as it evolves and flags the direction of travel, which is the question that actually matters: not whether this build is fast in the abstract, but whether this change made things worse. The chart below shows why the per-leg cost forces this discipline rather than the per-push reflex teams bring from functional testing.

• A full load leg models at roughly 7x the compute of a functional leg, so per-push gating multiplies the most expensive test by your merge volume.
• Scheduled and merge-to-main windows catch regressions one commit deep at a fraction of per-push cost.
• Baseline comparison survives application growth; absolute thresholds get disabled the first time they block a legitimate change.

Most fidelity loss is a runner-sizing problem

The biggest reason a CI load result fails to predict production is that the runner generating or serving the load is smaller than the real system, so the numbers describe the runner more than the application. A two-core runner saturates its own CPU long before it stresses the service under test, and the resulting curve flattens into a ceiling that is an artifact of the test harness, not a property of the code. Teams read that ceiling as a result and ship against it.

Unrealistic concurrency and scaled-down data compound the sizing problem. A load test that drives ten virtual users against a service built for ten thousand exercises none of the contention, lock, and pool-exhaustion behavior that defines real performance, and a test database with a thousand rows hides the query plans that only go quadratic at a million. Each of these is a fidelity leak, and together they let a pipeline produce green performance numbers that have almost no predictive relationship to production.

Fixing fidelity is mostly a matter of running on representative hardware and driving realistic concurrency, not buying a fancier load tool. The single highest-leverage move is to put the service under test and the load generators on runners sized like production rather than on whatever the default CI tier happens to be. Most of the remaining gap closes by seeding realistic data volumes and driving concurrency high enough to provoke contention, both of which are configuration rather than new infrastructure.

• An undersized runner produces a throughput ceiling that is an artifact of the harness, not a property of the service.
• Low virtual-user counts skip the contention, locking, and pool-exhaustion behavior that defines real performance.
• Scaled-down test data hides query plans that only degrade at production row counts.

Fan-out is the only way to reach honest concurrency

A single runner can rarely generate enough concurrent load to stress a modern service, so it produces optimistic numbers that collapse under real traffic. The load generator becomes the bottleneck before the service does, and the test measures how fast one machine can open connections rather than how the service behaves under a realistic crowd. The fix is not a bigger single runner, which hits its own ceiling, but more runners generating load in parallel.

Distributing load generation across several runners reaches a realistic target far faster and more truthfully. The modeled fan-out curve below shows the effect clearly: a load that takes forty minutes to drive from one runner reaches the same concurrency in a fraction of the time when spread across eight, and crucially it actually reaches the target rather than topping out below it. Fan-out buys both speed and honesty at once, because the same parallelism that shortens the run is what lets the test apply pressure a single machine never could.

This makes the cost and speed of spinning up parallel runners the practical limit on how honest a load test can be. If standing up eight load generators is slow and expensive, teams ration concurrency down to what one or two runners can produce and quietly accept optimistic numbers. If wide fan-out is cheap and fast to start, realistic concurrency becomes the default rather than a special occasion, which is where the runner layer stops being plumbing and starts being the thing that determines whether the test is trustworthy.

Runner OS choice quietly inflates the load-test bill

Because load legs are long and compute-heavy, running them on an expensive runner class multiplies an already large cost. A macOS minute lists at roughly 10 times a Linux minute on hosted runners, so a long load leg that wandered onto a macOS runner pays that multiple for every minute it sustains traffic. The leg that was already the most expensive thing in the pipeline becomes an order of magnitude worse for no benefit.

Performance work almost never needs a non-Linux host. Load generation is HTTP, gRPC, and socket traffic that behaves identically across operating systems, and the service under test is almost always deployed on Linux in production anyway, so a Linux runner is the higher-fidelity choice as well as the cheaper one. There is rarely a defensible reason for a load leg to run anywhere but Linux.

Keeping the heavy generation legs on Linux is one of the simplest ways to keep continuous load testing affordable. It is a one-line decision in most pipeline configs and it removes the single largest avoidable cost multiplier from the most compute-hungry leg in the suite. The cost chart contrasts these per-minute rates so the multiple is concrete rather than abstract.

Cheap, fast fan-out changes what is affordable

When parallel runners are slow to start or costly per minute, teams ration load testing into rare manual events. Every fan-out adds startup latency and per-minute cost, so a wide, honest test becomes a quarterly project rather than a routine signal, and routine is exactly what catches regressions early. The economics of the runner layer decide whether continuous load testing is a practice or an aspiration.

Managed runners that scale out quickly at roughly 69 percent lower cost per minute (Linux at $0.008/min hosted versus managed at $0.0025/min) make wide fan-out routine rather than exceptional. When eight load generators cost a fraction of what they used to and start in seconds rather than minutes, the friction that pushed load testing into a corner of the toolchain disappears, and teams run it on every merge to main without thinking about the bill.

Self-healing behavior closes the last gap. A long load run is exactly the kind of job that a transient connection failure aborts partway through, wasting all the compute spent so far and producing no result. Automatic recovery of those transient failures clears the network blips, registry timeouts, and cold-cache stalls that otherwise abort a long run, so the expensive minutes already spent are not thrown away. For continuous load testing, where every run is long and every abort is costly, that reliability is as much a cost lever as the per-minute rate itself.

• Managed fan-out models near 69% below hosted Linux rates, turning a quarterly project into a per-merge signal.
• Fast scale-out removes the startup latency that pushes wide load tests into rare manual windows.
• Auto-recovery of transient failures stops a network blip from aborting a long, already-paid-for run.

Soak tests find what smoke tests cannot

A short smoke load test confirms the service does not fall over under brief pressure, but the failures that take down production are usually slow: a memory leak that only matters after an hour, a connection pool that exhausts under sustained traffic, a cache that degrades as it fills. These are invisible to a two-minute load test and obvious to a soak test that holds traffic long enough for the slow failure to surface.

The reason soak tests are rare is the same reason load tests are rare, multiplied. They are the longest and most compute-heavy leg of all, which makes them the hardest to justify on expensive, slow-starting runners. The modeled compute multiple for a soak test sits between a smoke test and a full load test precisely because it trades intensity for duration, and duration on a metered runner is money.

Cheap, reliable runners change the math here too. When a soak test can run overnight on inexpensive managed capacity that will not abort on a transient blip, holding traffic for an hour stops being an extravagance. The teams with the best performance track records run soak tests on a schedule for exactly the failure classes that smoke tests structurally cannot reach.

Performance results need a baseline, not a verdict

A load test that emits a pass or fail tells a developer almost nothing actionable. The useful artifact is a trend: how this build compares to the recent baseline, and whether the direction is improvement, noise, or regression. A single number with no history cannot distinguish a real slowdown from the normal variance of a load test, so it either cries wolf and gets ignored or stays silent and misses the regression.

Tracking the baseline over time also turns load testing into a conversation rather than a gate. A ten percent p95 regression that is flagged but not blocking lets a team decide whether the change is worth the cost, which is a judgment call that an automated gate cannot make well. The signal informs the merge decision instead of overriding it, which is what keeps developers from disabling it the first time it is wrong.

This is why the budgeted, scheduled approach and the baseline approach reinforce each other. Running on a schedule produces the history a baseline needs, and comparing against a baseline produces a signal worth running on a schedule. Together they make continuous performance testing something teams actually keep, rather than a gate they add in a burst of discipline and quietly remove a month later.

Recommendations

Run load tests on a schedule against a tracked baseline

Stop trying to gate every push on an expensive load leg. Run focused load tests on merges to main or in nightly windows, and compare each run against a recent baseline rather than an absolute threshold. This catches regressions one commit deep without paying the per-leg cost on every developer push, and the baseline survives the application growth that breaks fixed thresholds.

Size load-test runners like production, not like the default tier

Most fidelity loss is a sizing problem. Put both the service under test and the load generators on runners sized like production rather than on whatever the default CI tier is, seed realistic data volumes, and drive concurrency high enough to provoke contention. This closes the majority of the gap between CI numbers and production behavior without any new tooling.

Fan out load generation to reach honest concurrency

A single runner caps out before it stresses a modern service. Distribute load generation across several runners so the test actually reaches realistic concurrency, and pick a runner layer where wide fan-out is cheap and fast to start so honest concurrency is the default rather than a special occasion.

Keep performance legs on Linux

Load generation behaves identically across operating systems and the service almost always ships on Linux, so there is rarely a reason for a load leg to run on a macOS or Windows runner at several times the cost. Reserve the expensive runner classes for genuinely platform-specific work and keep the heavy, OS-agnostic generation legs on Linux.

Choose a runner layer that recovers transient failures

Long load and soak runs are exactly the jobs a network blip aborts partway through, wasting everything spent so far. A runner layer that retries transient failures on a fresh environment protects the expensive minutes already invested and is what makes overnight soak tests reliable enough to schedule.

Outlook

Expect continuous load testing to move from differentiator to expectation over the next two years, following the same path functional CI took a decade ago. The blocker was never the tools; it was the cost and the credibility of the numbers. As managed fan-out drives the per-run cost down and runner sizing closes the fidelity gap, the excuses for not automating performance testing fall away one by one, and the teams that adopt it early will catch regressions their competitors ship to production.

The architectural direction points at the runner layer doing more of the work. Cheap, fast-scaling, self-healing capacity is what turns wide fan-out and overnight soak tests from rare projects into routine signals, and the same managed infrastructure that makes ordinary CI cheaper makes continuous performance testing affordable for the first time. The cost story and the fidelity story increasingly point at the same place: representative, elastic runners that start fast and do not abort on a blip.

For most teams the takeaway is that the gap between knowing they should load test and actually doing it is now mechanical, not fundamental. A budget, a baseline, production-sized runners, and a cheap fan-out mechanism are enough to make performance a continuous signal rather than a quarterly fire drill. The organizations that internalize that will spend the next two years catching slowdowns early while their peers keep finding them in production.

Methodology

This report combines publicly available developer data and published GitHub Actions pricing with Latchkey's analysis of performance and load-testing pipelines. Figures labeled "modeled" reflect typical load, soak, and fan-out behavior observed across managed runner usage, expressed as illustrative estimates rather than a primary survey. The managed per-minute rate and the roughly 69 percent saving are modeled blended figures derived from published per-minute rates. Fidelity splits and fan-out curves depend heavily on service shape, data volume, and concurrency target and will vary by project. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• Stack Overflow Developer Survey
• GitHub Actions - billing & pricing
• AWS EC2 On-Demand pricing
• DORA State of DevOps Report

More Latchkey reports