The State of Data Pipeline CI 2026

Data teams adopted CI discipline faster than the CI underneath caught up: meaningful data validation needs a real warehouse, so the runs are long and a real share of failures are transient infrastructure noise, not defects.

Executive summary

Data teams have adopted software engineering discipline, version control, code review, and CI on every change, faster than almost any other discipline outside core application development. A decade ago a data warehouse was edited by hand and a broken transformation was discovered in a dashboard; today the transformation lives in version control, its changes go through pull requests, and a pipeline validates them before they merge. dbt made models testable, Airflow made pipelines reviewable, and data-quality frameworks made expectations executable.

The practice has matured; the CI underneath it has not kept pace. The tooling that lets data teams write tests is excellent, but the runners those tests execute on are the same ephemeral hosted machines that application teams use, and the data workload does not behave like an application workload. Meaningful data tests require a real warehouse, integration runs move real volumes of data, and the dependency on external systems means a fraction of failures are transient infrastructure errors rather than genuine defects. The discipline arrived faster than the infrastructure to support it.

Data CI is structurally slow and structurally flaky for reasons application CI is not. It is slow because the validation that matters, checking that a transformation produces the right joins, types, and row counts against actual data, requires querying a real warehouse and waiting for it to move data, which no amount of CPU on the runner can speed up. It is flaky because every one of those queries crosses the network to an external system that can time out, rate-limit, or drop a connection, and when it does, the CI run fails through no fault of the code.

This report quantifies where data-pipeline CI time goes, why integration runs dominate, how much of the failure rate is noise rather than signal, and what a slow, flaky data CI run actually costs once re-runs and triage are priced in. The throughline, consistent with every Latchkey report, is that a meaningful share of this pain is mechanical: the slow part is structural and has to be managed, but the flaky part is transient infrastructure error that a self-healing runner can absorb before it ever reaches a human.

The stakes are higher than convenience. When data CI is slow and flaky enough, teams quietly stop trusting it, moving the expensive integration stage to a nightly job or a manual trigger and letting changes merge with only the cheap checks. That defeats the purpose of CI by letting data defects through, and it is a direct consequence of an integration suite that is too slow and too noisy to leave on every pull request. The fix is not fewer data tests; it is faster, more reliable runs that keep the full validation on the critical path.

Integration runs against a real warehouse dominate the clock

Unlike application CI, where most tests run in memory against mocks and stubs, meaningful data tests require a real warehouse to validate joins, types, and volumes against actual data. You cannot mock your way to confidence that a transformation produces the right result, because the thing under test is precisely how real data flows through real SQL, and that requires a real query engine holding real rows. The validation that matters is, by its nature, an integration test.

Those integration runs are the single largest slice of data-pipeline CI wall-clock. The modeled split of where a data CI run spends its time, shown below, puts warehouse integration runs at well over half the clock, with the cheaper model tests, DAG validation, and setup sharing the remainder. The time is not spent on the runner's CPU; it is spent waiting for an external warehouse to scan, join, and aggregate data, which is a fundamentally different cost than a compute-bound application test.

This is why the same tuning that speeds application CI barely moves the data CI needle: the bottleneck is data movement, not CPU. Adding cores to the runner does almost nothing when the runner is idle, waiting on a warehouse query to return. The levers that matter for data CI are different, scoping the integration stage to what actually changed, keeping the warehouse connection reliable, and not re-running the whole expensive suite because of a transient failure, and they target data movement and reliability rather than raw compute.

A fifth of data CI failures are not defects

Because data tests depend on external warehouses and networks, a meaningful share of failures are transient: a warehouse timeout under load, a dropped connection mid-query, a rate-limited request when too many CI runs hit the warehouse at once. Roughly a fifth of failed data CI runs are infrastructure noise rather than genuine data or logic defects, and the code under test was correct the whole time. The failure was the path to the warehouse, not the transformation.

Every one of those non-defect failures costs a full re-run plus an engineer's attention to triage a non-bug. The re-run is expensive because data CI runs are long, so a transient failure on a multi-minute integration suite means paying for the whole suite again. The triage is expensive because a red check demands a human decide whether it is real, and a developer pulled away from their work to investigate a warehouse timeout has lost focus on a problem that did not exist in their code.

The breakdown of what data CI failures actually are, shown below, separates the real data and logic defects from the transient warehouse, network, and fixture-ordering noise. The transient categories are a substantial minority of all failures, and they share the defining property that they pass on a clean retry. That is exactly the signature of a mechanical failure, and exactly the kind of failure a self-healing runner can absorb before it ever reaches a pull request or a human.

• Roughly a fifth of failed data CI runs are transient warehouse, network, or fixture-ordering noise, not real defects.
• Each non-defect failure costs a full re-run of a long integration suite plus an engineer interrupted to triage a non-bug.
• Transient failures pass on a clean retry, which is the signature of a mechanical failure a self-healing runner can absorb.

dbt made data testable, and made CI longer

dbt turned models and their expectations into executable tests, which is a clear win for data quality. Where a transformation used to be trusted on inspection, it now ships with assertions, uniqueness, not-null, referential integrity, accepted values, that run in CI and fail the build when the data violates them. The expectations that lived in a senior analyst's head are now code, and code can be checked automatically on every change.

That win comes with a direct cost in CI time, because those tests run against materialized data. A dbt data test is not a unit test that checks a function in isolation; it is a query that materializes a model and then asserts properties of the resulting rows, which means it pays warehouse cost to run. The more models and the more expectations, the more warehouse queries, and the longer the CI run, so the very thoroughness that makes dbt valuable also lengthens the pipeline.

The teams that stay fast separate cheap parse-and-compile checks from expensive data tests and gate the expensive stage on what actually changed. Parsing the project and compiling the SQL is fast and can run on every change; materializing and testing every model is slow and should run in proportion to the change, testing the models a diff actually touches and their downstream dependents rather than rebuilding and re-testing the entire project on every push. This is the data-CI equivalent of test impact analysis, and it is what keeps the dbt thoroughness from becoming a merge-blocking tax.

Slow integration runs push teams to skip them

When the integration stage runs for many minutes on every change, teams quietly start skipping it. The pressure is entirely rational at the individual level: an engineer with an approved, obviously-correct change does not want to wait ten minutes for a warehouse integration suite, so the team moves it to a nightly job or a manual trigger and lets the fast checks gate the merge. Each such decision feels reasonable, and collectively they hollow out the CI.

Moving the integration stage off the pull-request path defeats the purpose of CI by letting data defects merge unvalidated. A defect that the integration suite would have caught now ships, and it is discovered later, in a nightly run or, worse, in a dashboard or a downstream consumer's report, where it is far more expensive to diagnose and far more damaging to trust in the data. The integration suite was the safety net, and slowness is what made the team take it down.

The modeled wall-clock by validation type, shown below, makes the temptation concrete: the warehouse integration stage dwarfs the lint, unit, and even the dbt data-test stages, so it is the obvious thing to cut when CI feels slow. But the fix is not fewer integration tests; it is faster, more reliable runs so the full suite can stay on the critical path without becoming the reason merges stall. The goal is to make the integration suite cheap enough to keep, not to give teams a reason to abandon it.

Data validation belongs on every change, not in a nightly batch

The whole value of CI is catching a defect at the moment it is introduced, while the author still has the change in their head and the diff is small. A data defect caught in a nightly batch run is caught hours after the author moved on, against a day's worth of merged changes that now have to be bisected to find the culprit. The later the validation, the more expensive the defect, and a nightly batch is about the latest validation can be while still calling itself CI.

Data-quality frameworks made expectations executable precisely so they could run continuously, on every change, as a gate rather than as a periodic audit. An assertion that a key is unique or that a metric stays within bounds is most valuable when it blocks the change that would have violated it, not when it reports the violation the next morning. Pushing those expectations to a nightly job converts a gate into an alarm, and an alarm that fires after the damage is done is a weaker thing than a gate that prevents it.

Keeping data validation on every change requires the integration suite to be fast and reliable enough that no one is tempted to move it off the critical path. That is the same requirement as the previous finding, viewed from the other side: the reason teams move validation to a nightly batch is that the per-change run is too slow and too flaky, so the route to per-change validation runs through making that run cheap and trustworthy. Faster, self-healing runs are what let the expectations stay where they belong, gating every pull request.

The re-run tax is hidden inside the cost of data CI

Because a fifth of data CI failures are transient, a data team is paying a quiet re-run tax that rarely shows up as a line item. Every transient failure triggers a re-run of a long, warehouse-bound suite, and those re-run minutes are real spend on work that produced no new information, the suite would have passed the first time if the warehouse had not hiccuped. The bill includes the cost of failures that were never about the code.

The modeled cost per data CI run, shown below, prices this explicitly: the same run costs noticeably more on a hosted path once re-runs are included than it does without them, and the gap between the hosted-with-re-runs case and a managed self-healing path is the re-run tax made visible. The transient-failure rate does not just slow the team down; it inflates the compute bill by paying repeatedly for runs that transient infrastructure failed.

What makes this tax addressable is that it is mechanical. The transient failures pass on a clean retry, so a runner that retries them automatically on a fresh environment removes both the re-run minutes and the engineer triage they would otherwise trigger. The re-run tax is not a fixed cost of doing data CI; it is the cost of letting transient infrastructure failures reach the pull request, and a self-healing runner is what stops them before they do.

Auto-healing runners turn flaky data CI reliable

Data CI is the clearest case for self-healing runners of any ecosystem Latchkey models, because a large fraction of its failures are transient warehouse and network errors that pass on a clean retry without any code change. The defining pain of data CI, that it is structurally flaky because it depends on external systems, is exactly the pain self-healing is built to absorb: a known-transient failure is retried automatically on a fresh environment before a human ever sees a red check.

A managed runner that auto-heals those failures removes the re-run tax and the false-alarm triage at the same time. The warehouse timeout, the dropped connection, the rate-limited query, each of these is caught and retried by the platform rather than surfaced to the developer, so the failure never reaches the pull request and the engineer is never pulled away to investigate a non-bug. The transient share of failures simply stops being a thing the team experiences, even though the underlying warehouse still hiccups occasionally.

Alongside reliability, the managed path lands the compute around 69% cheaper than hosted rates, and because it removes the re-run tax it cuts the wasted minutes on top of cutting the per-minute price. The combined effect keeps the slow integration suite reliable enough to leave on every pull request, which is the outcome that actually matters: not a faster warehouse, which CI cannot deliver, but a run trustworthy enough that no one is tempted to take the validation off the critical path.

• Data CI is the clearest self-healing case: a large share of its failures are transient warehouse and network errors that pass on a clean retry.
• Auto-healing catches the timeout, dropped connection, or rate-limited query before it reaches the developer, removing both the re-run and the triage.
• The transient share of failures stops being something the team experiences, even though the warehouse still hiccups occasionally.
• The managed path also lands compute around 69% cheaper than hosted rates while removing the re-run tax on top.

Recommendations

Separate cheap checks from the expensive warehouse stage

Run lint, parse, and compile checks on every change as a fast gate, and treat the warehouse integration suite as a separate, expensive stage. This lets the cheap checks give quick feedback while the expensive stage runs in proportion to the change, and it makes the slow part visible and manageable rather than buried inside one long undifferentiated run.

Gate data tests on what actually changed

Materializing and testing every model on every push is the data-CI equivalent of rebuilding the whole repository for a one-line change. Use dbt's dependency graph to test the models a diff touches and their downstream dependents, not the entire project, so the expensive warehouse work scales with the change rather than with the size of the project.

Auto-heal transient warehouse and network failures

Roughly a fifth of data CI failures are transient infrastructure errors that pass on a clean retry. Retry these mechanical failures automatically on a fresh environment so a warehouse timeout or dropped connection is a quiet recovery rather than a full re-run of a long suite plus an engineer pulled in to triage a non-bug. This removes both the re-run tax and the false-alarm triage.

Keep the integration suite on every pull request

The value of data validation is catching a defect when it is introduced, not in a nightly batch hours later against a day of merged changes. Resist the temptation to move the integration suite off the critical path; instead make it fast and reliable enough to keep there, because a gate that prevents a defect is worth far more than an alarm that reports it the next morning.

Move integration-heavy data CI onto managed runners

Data CI's pain, slow warehouse-bound runs and a real transient-failure rate, maps directly onto what a managed runner with self-healing addresses. Moving the integration leg onto managed infrastructure lands the compute cheaper, removes the re-run tax by auto-healing transient failures, and keeps the slow suite reliable enough to leave on every change. Of all ecosystems, data CI sees the clearest reliability return.

Outlook

Expect data CI to keep adopting the rest of software engineering's practices, contract testing between pipelines, data versioning, environment isolation per pull request, which will make the validation richer and, left unaddressed, longer and more dependent on external systems. The discipline will continue to outpace the infrastructure unless data teams treat the runner layer as a first-class concern rather than inheriting the same ephemeral hosted machines that application teams use for a fundamentally different workload.

The transient-failure problem is structural and will not go away, because data validation that means anything has to cross the network to a real warehouse, and real warehouses under real load will always occasionally time out, rate-limit, and drop connections. That makes self-healing not a temporary patch but a durable requirement for data CI: as long as the validation depends on external systems, a share of failures will be infrastructure noise, and a runner that absorbs that noise is what keeps the signal trustworthy.

The durable takeaway is that data teams have already done the hard cultural work of bringing CI discipline to data, and the remaining gap is infrastructure rather than practice. A runner layer that gates the expensive warehouse stage on what changed, auto-heals the transient failures, and lands the compute cheaper is what lets the mature data-testing practice run on every pull request without stalling merges. The teams that close that gap will keep their validation on the critical path while their peers quietly retreat to nightly batches and the unvalidated merges that follow.

Methodology

This report combines public data-engineering ecosystem data and dbt and Airflow documentation with Latchkey's own analysis of CI runner economics. Stage timings and failure-mix splits are modeled on a representative analytics pipeline with warehouse integration tests and will vary with data volume, warehouse choice, and test coverage. Cost figures derive from published GitHub-hosted per-minute rates and Latchkey managed rates. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• Stack Overflow Developer Survey
• JetBrains Developer Ecosystem Survey
• GitHub Actions - billing & pricing
• CNCF Annual Survey

More Latchkey reports