Self-healing CI vs AI code fixes: which failures belong where

By Kaveh Alemi·Latchkey

Self-healing CI vs AI code fixes sounds like a competition between two tools and is really a routing question with one right answer per failure. A runner that repairs infrastructure problems and an agent that repairs code are solving disjoint sets, and almost all of the damage in practice comes from sending a failure to the wrong one.

Failure classes routed to two owners: infrastructure to the runner layer, defects to the coding agent — Which owner each failure class belongs to, and what going to the wrong one costs. Class taxonomy from the open-source CI Doctor catalog, counted 2026-09-21.

Both halves of this are now ordinary. Runners that detect a transient failure and retry with a remediation have existed for a few years; coding agents that read a stack trace and open a pull request have existed for about as long. What has not settled is which one gets handed which failure.

The cost of getting it wrong runs in both directions and neither is symmetric with the other. Ask an agent to fix an infrastructure failure and it edits code that was never wrong. Ask a runner to retry a real defect and you have built a machine for hiding bugs.

Two layers, and why conflating them is expensive

The runner layer sits below your code. It owns the machine, the network path to every registry, the disk, the memory ceiling and the tool versions on the image. Everything that goes wrong there is an environment problem, and the correct response is to change the environment and run the same code again.

The agent layer sits above your code and owns the diff. Everything it can fix is a fact about your repository: a wrong assertion, a type error, a missing migration, a configuration file that does not match the code. The correct response is a change to the repository, and running the same code again is precisely the wrong move.

So the boundary is not about capability, and arguments framed as which is smarter miss it entirely. A model can read a disk-full error perfectly well; the problem is that the only fix is more disk, and it has no disk to give. Routing is the whole game.

Somebody has already written the taxonomy down

You do not have to invent the classification, and it is useful that a written one exists in public. The open-source CI Doctor skill carries a catalog of CI failures with a class column, and the five its instructions actually reason about are the environment ones: network, setup, memory, timeout and disk. A row with no class is a real failure.

The proportions are worth knowing before you build anything on this. Counted on 2026-09-21, the catalog holds 4,555 rows, of which 760 carry a class and 3,795 do not. So on that corpus roughly one failure signature in six is an environment problem and five in six are bugs. The transient ones are over-represented in any given week, because a bug fails once and a flaky registry fails all month, but the long tail is still code.

Class	Who should take it	What good looks like	What the wrong owner does
`network`	The runner layer	Retry with backoff, then authenticate or cache the source	An agent edits retry logic into your application code
`memory`	The runner layer	A bigger ceiling, or less parallelism, then rerun	An agent rewrites a working algorithm to use less memory
`disk`	The runner layer	Prune caches and layers, reclaim space, larger disk	An agent deletes test fixtures to make the number fit
`setup`	The runner layer, then a durable pin	Install or pin the tool in the image or a setup step	An agent adds an install command to an unrelated script
`timeout`	The runner layer first, the agent if it repeats	Retry once; investigate only if it reproduces	Either owner raises the limit and calls it fixed
No class: a real defect	The agent, or a human	A diff, a test that now fails before it passes	A runner retries, and a real bug ships green on attempt two

The failure the runner must never take

A retry is only honest when the code that is about to run again is the same code that should have passed the first time. The moment that stops being true, a retry is a mechanism for hiding a defect, and it hides it in the most expensive possible way: intermittently, on somebody else's branch, weeks later.

This is why a retry policy written as a blanket rule is worse than none. Retrying every red job, or wrapping a step in continue-on-error, converts a deterministic failure into a flaky one and moves the discovery from a pull request to production. What separates a repair from a hidden bug is the classification happening before the retry, not after it. Self-healing CI explained works through the grades of that in detail.

The failure the agent should not take

Hand a coding agent an out-of-memory kill and watch what happens: it reads exit code 137, finds the test that was running, and proposes a change to that test. The change is often plausible and occasionally even makes the symptom go away, which is the worst outcome, because now a real memory ceiling is masked by a weakened test.

The pattern generalizes. An agent asked to fix an environment failure will always find something in the repository to change, because that is the only surface it has. Giving it the failure class along with the log is the cheap fix: an agent told that this is a network class failure will say so and stop, and an agent told nothing will edit something.

The band in the middle: flaky tests

A test that fails one time in thirty is the case where both layers have a claim and both are partly wrong. It is not an environment failure, so no amount of retry-with-remediation fixes it, and the runner that retries it is doing the thing the previous section warns about. It is also not a clean defect, so an agent handed one failing run has no reliable signal to work from.

The honest handling is neither: quarantine, measure, and treat the flake rate as a number somebody owns. Retrying a flaky test is a decision to keep it, so it should be a decision rather than a default, and it belongs beside a count of how often it fires. The measurement side of that is in CI pipeline reliability explained.

What an agent is genuinely better at here

Three things, and none of them is repairing infrastructure. Reading a long log and finding the eight lines that matter, which is a search problem models are unusually good at. Proposing a diff for a defect with a clear signature, which is the mainstream case. And explaining why a failure happened in terms of your repository, which is the part a runner cannot do because it has never read your code.

The productive arrangement is therefore sequential rather than competitive. The runner absorbs the environment classes so they never reach a human or an agent at all. What is left is, by construction, about the code, which is exactly the input an agent is good with. Getting that input into a shape an agent can read cheaply is its own problem, covered in CI failure reports your agent can read.

How to route it in your own pipeline

Four steps, and the first one is the one most teams skip, which is why the rest never gets done.

Classify before you route. Pull the exit code and the distinctive error tokens out of the failing step, and decide which class it is. Without this step every later rule is guesswork.
Send the environment classes to whatever handles your infrastructure, and make sure that path is allowed to retry. If nothing handles them, they are being handled by a human rerunning the job, which is the same fix at a worse price.
Send the unclassed failures to the agent or the author, with the failing step's log rather than the whole run, and with the class attached so it does not have to guess.
Count what each path handled, monthly. If the environment path is growing, that is an infrastructure problem getting worse rather than a feature working harder, and the classes tell you which one.

What this page does not claim

No run backs it, and it is an argument rather than a measurement. The one number in it, the six-to-one split between unclassed and classed failure signatures, is a property of one public catalog counted on one date, and it describes how many distinct failures exist of each kind rather than how often each fires in your logs. Those are different quantities and conflating them would overstate the case considerably.

It also makes no claim about what any particular product repairs. Which failures a given runner actually handles is a per-failure question with per-failure evidence, and the pages in this library that assert a repair carry a recorded run behind it. This page is only about who should own which class, which is true regardless of what you run.

Frequently asked questions

Should CI automatically retry a failed job?

Only after classifying it. A retry is honest when the same code should have passed and the environment stopped it: a registry timeout, a transient reset, a disk that filled. It is dishonest when the failure is a defect, because retrying converts a deterministic bug into a flaky one and moves the discovery downstream. A blanket retry policy is worse than none for exactly that reason.

Which CI failures are not bugs in my code?

The environment classes. The classification table in the open-source CI Doctor skill names five: network, setup, memory, timeout and disk. Counted on 2026-09-21, all classed rows together accounted for 760 of its 4,555 failure signatures, with the other 3,795 carrying no class at all, meaning a real defect. Roughly one signature in six is environmental, though transient failures fire far more often than that ratio suggests.

Why does my coding agent edit the wrong thing after an out-of-memory failure?

Because the repository is the only surface it has. Given exit code 137 it finds the test that was running and proposes a change to it, which sometimes makes the symptom go away and masks a real memory ceiling. The cheap fix is to hand it the failure class along with the log: an agent told that this is a memory class failure will say so instead of editing.

Who should own a flaky test, the runner or the agent?

Neither, as a default. A test failing one time in thirty is not an environment failure, so retry-with-remediation does not fix it, and it is not a clean defect either, so a single failing run gives an agent nothing reliable to work from. Quarantine it, count how often it fires, and give the flake rate an owner. Retrying it is a decision to keep it, which is fine as long as it is a decision.