The State of ML & GPU CI/CD 2026

GPU runner cost, model-test pipelines, data and cache bottlenecks, and what scarce-GPU queue time does to ML delivery.

Executive summary

ML CI/CD looks like ordinary CI/CD until you price the accelerator. A GPU minute costs roughly ten times a Linux CPU minute, model-test jobs run long, and the data they need is large, which makes the economics and the bottlenecks of an ML pipeline fundamentally different from a web-app pipeline. The same pipeline shape that is cheap and fast on CPU becomes expensive and slow the moment it touches a GPU, and most of the lessons teams learned optimizing application CI do not transfer cleanly.

This report focuses on where GPU time actually goes. A striking share of it is not spent on math at all but on moving data into place: pulling datasets, warming caches, and reloading model weights on a cold runner. Because the accelerator is billed the whole time, including the minutes it sits idle waiting for data, those bottlenecks are far more expensive than the equivalent stalls on a CPU runner. A cold cache that wastes five minutes on a cheap CPU runner is an annoyance; the same five minutes on an A100-class accelerator is a line item.

The scarcity problem compounds the cost problem. GPUs are limited and contended, so jobs queue for the accelerator before they can even start, stretching feedback loops for the teams that can least afford slow iteration. A model engineer who waits minutes for a GPU on every iteration loses far more than the wall-clock time, because the cost of a slow loop is paid in lost focus and abandoned experiments, not just queue minutes. Warm GPU pools, persistent caches, and recovery of transient failures are where the time and money come back.

Three figures frame the year. A T4-class GPU minute models at roughly ten times a Linux CPU minute, and high-end accelerators stretch that multiple much further. Nearly half of billed GPU minutes go to data loading and cache misses rather than to model compute, which means the meter runs hardest on work that is not the work. And during peak hours a scarce accelerator class can keep a job waiting several minutes before it even starts, an invisible tax that never appears as a line on a per-minute bill.

For ML leaders the practical message is that GPU CI cost is dominated by inefficiency, not by compute. The accelerator is expensive, but the largest reclaimable spend is in the data path, the cold starts, and the transient failures, all of which are addressable without touching a single model. The teams that treat the GPU as scarce, expensive capacity to be kept busy on real work, rather than as a CPU runner that happens to cost more, are the ones whose ML delivery stays fast and affordable as their model count grows.

The accelerator changes the whole cost model

A GPU minute is an order of magnitude more expensive than a CPU minute, and high-end accelerators stretch that gap further still. The cost chart below shows the ladder: a Linux CPU runner sits near the floor, a T4-class GPU is roughly ten times higher, and A10 and A100 classes climb steeply from there. The same wall-clock minute that costs almost nothing on a CPU runner becomes a meaningful expense on an accelerator, and the most expensive classes turn ordinary inefficiency into serious money.

This reorders every optimization priority an ML team inherits from application CI. On a CPU pipeline, a few wasted minutes barely register and teams rationally ignore them. On a GPU pipeline, any inefficiency that wastes wall-clock time on the accelerator, from a slow data load to a transient failure to a cold start, is multiplied by ten or more, so the same waste that was invisible on CPU is suddenly the dominant cost. The expensive resource is the one that has to stay busy.

The strategic consequence is that GPU CI is a utilization problem before it is a compute problem. The question is not how to make the model faster, which is hard and slow to change, but how to keep the expensive accelerator doing model work rather than waiting on data, queues, or retries. That framing, the accelerator as scarce capacity to be kept busy, is the lens the rest of this report applies, because it is where the reclaimable spend actually lives.

• A T4-class GPU minute models at roughly 10x a Linux CPU minute; A100-class climbs far higher.
• Inefficiency that is invisible on CPU becomes the dominant cost on a GPU at 10x the rate.
• GPU CI is a utilization problem first: keep the expensive accelerator busy on real model work.

Most GPU time is data movement, not math

A large share of billed GPU minutes is spent pulling datasets, warming caches, and reloading weights rather than running the model. The minute split below is stark: nearly half of every billed GPU minute goes to data loading, artifact reload, and cold-start overhead instead of to the model compute the GPU was rented for. The accelerator sits with its tensor cores idle while a dataset streams over the network, and the meter does not care that no math is happening.

Because the accelerator meter runs the entire time, including those idle waits, the data path is the highest-leverage GPU cost optimization available, often outweighing model-level tuning by a wide margin. A team can spend weeks shaving ten percent off model runtime, or it can put the dataset and weights on a fast local cache and reclaim the much larger fraction the GPU was burning on data movement. The second is cheaper to do and saves more, which inverts the instinct to optimize the model first.

The fix is structural rather than algorithmic. Datasets and model weights should live close to the runner and persist between runs, so the GPU spends its expensive minutes on compute rather than on fetching the same data it fetched yesterday. This is the same caching discipline that helps CPU pipelines, but the payoff is amplified by the GPU rate: every minute of data movement removed is a minute of accelerator time reclaimed at accelerator prices.

• Nearly half of billed GPU minutes go to data loading, weight reload, and cold start, not model compute.
• The meter runs while the accelerator sits idle waiting on the network, so idle waits cost full rate.
• Fixing the data path usually reclaims more spend than model-level tuning, at less effort.

Scarce GPUs turn queueing into a delivery bottleneck

High-demand accelerator classes are contended, so jobs wait for a GPU before they can begin. The pickup-time chart below shows the gradient: an abundant T4 class starts quickly, a moderately contended A10 waits longer, and a scarce A100 class at peak hours can leave a job queued for several minutes before any work happens. That wait is pure latency, paid before the first useful minute, and it lands hardest on the most powerful and most contended hardware.

During peak hours that queue stretches feedback loops for exactly the teams whose iteration speed matters most. A model engineer iterating on an experiment pays the scarcity tax on every loop, and the compounding cost is not the queue minutes themselves but the broken focus and the experiments not run because the loop got too slow to stay in flow. The accelerator scarcity that looks like an infrastructure detail is actually a direct drag on research velocity.

The wait is also invisible on a per-minute bill, which is what makes it insidious. A finance view of GPU spend sees only the minutes the accelerator was billed, not the minutes a job spent waiting to get one, so the scarcity cost never appears as a number even though it directly slows shipping. A warm GPU pool collapses that wait toward zero by keeping accelerators provisioned and ready, turning the queue from minutes into seconds and making the hidden tax visible by removing it.

Persistent caches pay for themselves fastest on GPU

Reloading datasets and model weights from cold on every run multiplies expensive GPU minutes. The cache-impact chart shows the spread: a cold cache on a cold runner is the worst case, warming either the cache or the runner helps, and warming both together cuts the per-run GPU minutes dramatically. Every one of those reclaimed minutes is an accelerator minute, which is what makes the saving so large relative to the same optimization on CPU.

A persistent cache that survives between runs removes the repeated load entirely. Instead of pulling the same multi-gigabyte dataset and reloading the same weights on every invocation, the runner restores them from a local store in a fraction of the time, and the GPU starts doing real work almost immediately. On a CPU pipeline this saves cheap minutes; on a GPU pipeline it saves the most expensive minutes in the whole organization's CI, which is why caching is the cheapest GPU optimization a team can adopt.

The reason caching is underused on GPU is the same reason it is underused everywhere: it is conceptually simple and operationally finicky, and a cache keyed wrong either serves stale data or never hits. But the payoff-to-effort ratio is higher on GPU than anywhere else precisely because of the rate multiplier. A team that measures its GPU cache hit rate the way it measures model accuracy will find the data path is where its accelerator budget was quietly leaking.

Warm GPU pools and auto-heal cut the most expensive minutes

Cold starts and transient failures cost the same wall-clock time on a GPU as on a CPU, but at roughly ten times the rate, which is what makes them the highest-value thing to eliminate in an ML pipeline. A cold start that wastes provisioning minutes and a transient failure that forces a re-run both burn accelerator time at accelerator prices, and on a long model-test job those wasted minutes add up to the largest absolute spend in the pipeline.

A warm GPU pool removes the cold-start wait by keeping accelerators provisioned and ready, so a job starts computing in seconds instead of waiting minutes for a GPU to come online. This attacks both the scarcity queue and the cold-start overhead at once, because a warm pool is a pool of accelerators already past the slow part. The pickup-time and cache-impact charts both show the warm-pool case as the best outcome, and that is not a coincidence: warming the runner removes the most expensive idle minutes in both dimensions.

Automatic recovery of transient failures removes the other large waste. A model-test job that fails on a network blip pulling a dataset, an out-of-memory kill, or a registry timeout has burned expensive GPU minutes before failing, and re-running it from scratch burns them again. Retrying on a fresh environment before a human sees a red check means those failures never cost a second full GPU run, which is exactly where managed GPU runners reclaim the largest absolute spend in an ML pipeline.

• A cold start or transient failure wastes the same minutes as on CPU, but at ~10x the rate.
• A warm GPU pool removes both the scarcity queue and the cold-start overhead at once.
• Auto-recovery stops a transient failure from forcing a second full, expensive GPU run.

Model-test pipelines are shaped nothing like app pipelines

An application CI pipeline is many short, parallel jobs that each validate a slice of the code; a model-test pipeline is a few long, serial jobs that each load a large dataset, run a model, and check the result. That structural difference changes which optimizations matter. Sharding and parallelism, the workhorses of fast application CI, help less when the bottleneck is a single long job loading a single large dataset onto a single scarce accelerator.

The long, data-heavy shape also makes every stall more expensive in absolute terms. A short application job that hiccups loses seconds; a long GPU job that stalls on data movement or a transient failure loses minutes of accelerator time, and there is no parallel sibling job to hide the loss behind. The pipeline's critical path runs straight through the most expensive resource, so anything that slows that path is paid at full GPU rate with no amortization.

This is why the optimization priorities invert for ML. Where an application team reaches first for sharding and concurrency, an ML team reaches first for the data path, the cache, and the warm pool, because those are what keep the single long critical-path job from wasting accelerator minutes. Treating a model-test pipeline like an application pipeline, and optimizing for parallelism it cannot use, is a common and expensive mistake.

GPU cost discipline is utilization, not just rate shopping

The instinct when a GPU bill grows is to shop for a cheaper accelerator rate, but the minute split shows why that is the smaller lever. If nearly half of billed GPU minutes are spent on data movement and cold starts rather than compute, a team that only negotiates a lower rate is still paying for the same wasted minutes, just slightly less per minute. The larger win is removing the wasted minutes entirely so the accelerator does compute instead of waiting.

Utilization is the metric that actually governs GPU CI cost. A GPU kept busy on model work at a moderate rate is cheaper than a GPU at a discounted rate that spends half its time idle on data loads and queues. The data path, the cache, and the warm pool all raise utilization by removing the idle minutes, which is why they outperform rate shopping as a cost strategy even though they are less obvious.

The combined effect is what makes managed GPU runners compelling for ML CI: they bring a lower per-minute rate and the warm-pool, persistent-cache, self-healing utilization improvements together, so the rate reduction and the wasted-minute reduction compound. For an ML team, that combination is what keeps accelerator spend proportional to the model work being done rather than to the inefficiency surrounding it.

Recommendations

Put datasets and weights on a persistent cache close to the runner

Nearly half of GPU minutes go to data movement, so the data path is the highest-leverage optimization on the most expensive resource you run. Cache datasets and model weights so they persist between runs and sit close to the accelerator, and measure the GPU cache hit rate the way you measure model accuracy. This usually reclaims more spend than model-level tuning at far less effort.

Use a warm GPU pool to remove scarcity queueing and cold starts

A scarce accelerator class can keep a job queued for minutes before it starts, an invisible tax on iteration speed that never shows on a per-minute bill. A warm pool of provisioned accelerators collapses that wait toward zero and removes the cold-start overhead at the same time, attacking the two largest non-compute time sinks in one move.

Auto-heal transient failures before they force a second GPU run

A model-test job that fails on a network blip, an OOM, or a registry timeout has already burned expensive accelerator minutes, and re-running it burns them again. Retry transient failures on a fresh environment before a human sees a red check, so the largest absolute waste in the pipeline, a full repeated GPU run, never happens.

Optimize for the long critical path, not for parallelism

Model-test pipelines are a few long serial jobs, not many short parallel ones, so the sharding and concurrency that speed application CI help little here. Focus on the single long critical-path job: keep its data warm, its runner provisioned, and its failures recoverable, because that path runs straight through the most expensive resource you own.

Manage GPU cost by utilization, not by rate alone

A discounted accelerator that idles half its time on data loads costs more than a busy one at a moderate rate. Treat utilization as the governing metric and combine a lower per-minute rate with warm pools and persistent caches, so the rate reduction and the wasted-minute reduction compound rather than competing for attention.

Outlook

Expect GPU scarcity and GPU cost to stay the defining constraints of ML CI/CD through 2026 and beyond, because accelerator demand is growing faster than supply and the rate gap to CPU is not closing. The teams that treat the GPU as scarce capacity to be kept busy will pull further ahead of those that treat it as an expensive CPU runner, because the inefficiency the second group tolerates is multiplied by a rate that keeps the stakes high.

The architectural direction is toward warm, cached, self-healing GPU capacity becoming the expected baseline rather than a luxury. The same managed-runner properties that lower ordinary CI cost, warm pools, persistent caches, and automatic recovery, are worth disproportionately more on accelerators because every reclaimed minute is reclaimed at accelerator prices. The cost story and the iteration-speed story point at the same infrastructure, which is what makes the direction durable.

For most ML teams the practical takeaway is that the accelerator bill is dominated by inefficiency they can remove without touching a model. The data path, the cold starts, and the transient failures are where the reclaimable spend lives, and addressing them keeps GPU cost proportional to real model work as the team's model count grows. The organizations that internalize that will spend the next two years iterating fast on cheap, busy accelerators while their peers pay full rate for idle ones.

Methodology

This report combines published cloud GPU pricing with Latchkey's own modeling of model-test pipeline shapes, cache behavior, and accelerator contention. Figures labeled "modeled" are illustrative estimates derived from public pricing and representative ML pipeline profiles, not a primary survey; figures attributed to a named source reflect that source. The roughly 10x GPU-to-CPU minute multiple and the GPU per-minute rates are approximate and derived from published on-demand instance pricing. The managed utilization improvements and the roughly 70 percent rate saving are modeled blended figures. Minute splits and queue waits depend heavily on dataset size, model shape, and accelerator scarcity and will vary by team. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• AWS EC2 On-Demand pricing
• GitHub Actions - billing & pricing
• Stack Overflow Developer Survey
• CNCF Annual Survey

More Latchkey reports