Self-healing CI, explained: what it repairs and what it must not
Self-healing CI diagnoses a failing step on the runner and repairs it while the job is still running, so a transient failure never reaches a person. The hard part is not the repair: it is the line between a failure that should be fixed and one that must be allowed to fail.
Most CI failures are not bugs. A registry times out, a runner fills its disk, a mirror returns a 503, a shared library is missing from the image. The job fails, someone reads a log, someone clicks re-run, and the second attempt passes without a line of code changing. That loop costs the minutes of the failed run plus a human interruption, and it teaches a team to re-run first and read later, which is how a real failure gets ignored.
Self-healing addresses that loop directly. It is worth being precise about the term, because three different things are sold under it, and only one of them removes the interruption.
Three grades, frequently conflated
Explain is a model summarizing why the job failed, after it failed. Propose is a pull request suggesting a fix for you to review. Repair is the failing step being fixed and retried inside the same run. Only the third removes the interruption; in the first two a person still has to read something and act, which is the cost the loop was imposing in the first place.
A vendor who says "self-healing" and means the first grade is describing a summarizer. The question that separates them is simple: when the fix is applied, is the job still running?
- Explain: a summary after the failure. The job is already red and someone is already reading.
- Propose: a suggested patch. Better, and still a review queue.
- Repair: the step is fixed and retried in the same job, and the pipeline stays green.
What the runner repairs
On a Latchkey runner the decision is made by a pattern library rather than by a model guessing. Counted in that library on 2026-09-20, it holds 112 patterns: 82 live and 30 in shadow mode. Of the live ones, 41 carry a repair plan and 41 are explicit passthroughs, which is to say the library spends as much of itself on deciding what not to touch as on fixing anything.
The 41 live repairs break down by category as 30 network, 4 memory, 4 configuration, 2 missing tools and 1 disk. The shape is not an accident: network failures are where CI wastes most of its minutes, and they are the safest class to retry, because a registry that answered with a 503 has told you the request, not your code, was the problem. The repair actions are correspondingly dull. Thirty of the 41 begin with a retry under exponential backoff. Four set an environment variable, such as a raised heap ceiling. Four retry the step on its own: the kernel OOM kill, the Python MemoryError, and the npm and Cargo lockfile-drift patterns, which the build that ships the library to runners augments with a command rewrite first. Two install a missing system package from a vetted list, and one frees disk space before retrying.
Before the pattern library runs, a much cheaper stage looks at the exit code alone. That table carries 19 entries, of which only 2 are decisive enough to heal on and 4 to fail on, with 13 marked as not decisive: exit code 1 tells you nothing, and a system that pretends otherwise is guessing. A novel failure that neither stage recognizes goes to a bounded AI diagnosis with a time budget of about four minutes.
What it must not repair
A failing test, a compile error, a type error, a lint violation. These are the signal CI exists to produce, and a system that makes them go away is not fixing your pipeline, it is disabling it. The correct behavior for a genuine failure is to fail, with the original logs, exactly as if nothing had intervened.
That claim is easy to make and easy to check, so here is the check. content/repro/self-healing-ci-explained.sh ships a real bug: a port parser that accepts an empty string because Number("") is zero. The second test asserts a throw that never comes. Run on a latchkey-small runner on 2026-09-20, the job failed with the test runner's own output and exit code 1. Self-heal engaged, which is the pair of wrapper lines at the end, and did nothing else. Six of the seven stack frames and the dumped assertion object are elided between the two ends of the excerpt; nothing that remains has been edited.
node v20.20.2, fixture /home/runner/fixture
.X
Failed tests:
✖ rejects an empty string (0.721164ms)
AssertionError [ERR_ASSERTION]: Missing expected exception (RangeError).
at TestContext.<anonymous> (file:///home/runner/fixture/parse.test.mjs:10:10)
[stack frames and the assertion object elided]
[latchkey-bash-wrapper] BEGIN sidecar POST (boot_wait=30s max_time=320s url=http://localhost/diagnose socket=/run/latchkey-self-heal/sock)
[latchkey-bash-wrapper] END sidecar POST ok (attempts=2 http=200)Shadow mode, and "diagnosed but not retried"
Two phrases in this area mean something specific. A pattern in shadow mode is telemetry only: it matches, it is recorded, and at runtime it returns passthrough, so the job fails exactly as it would if the pattern did not exist. Shadow is where a borderline pattern lives while it earns confidence, and each one carries the number of clean matches it needs before promotion. Thirty of the 112 patterns are there today, most of them the broadest network matchers, which are precisely the ones you would not want firing on a guess.
"Diagnosed but not retried" is the outcome in the run above: the sidecar was asked, it answered, and the answer was that nothing should be done. The failure is still recorded, which matters more than it sounds. Those are the failures latchkey watch polls for, and they are the ones worth handing to a coding agent, because by construction they are the failures the deterministic layer has already declined to paper over.
How it differs from retry steps, re-runs, and an agent
A retry step wrapper, or a re-run of failed jobs, repeats work and hopes for a different outcome. It doubles the minutes on every failure including the real ones, it hides flakiness rather than surfacing it, and it delays the feedback a genuine bug should have produced immediately. Blanket retry also has no idea what it is retrying: the same action is applied to a DNS blip and to a broken assertion.
A repair is narrower and better recorded. A specific cause is diagnosed, one action is applied to it, and the step is retried once from a state that has changed. That leaves an audit trail you can count, which is the data you need to fix the underlying problem rather than absorbing it forever.
An agent fixing code is the complement, not a competitor. The runner owns the environment, where the failure is not about your code and there is nothing to review; the agent owns the code, where every change needs a human, a diff and a test. Ask a vendor which of the two they are doing, and be suspicious of an answer that covers both.
Four failures, worked end to end
Each of the categories above has a page that reproduces the failure on a real runner and shows what happened, including the runs where the repair did not work. They are the honest version of this page: the log, the exit code, and what the engine did about it.
- Memory: exit code 137 in GitHub Actions, the kernel OOM kill, and JavaScript heap out of memory, where the repair raises a ceiling.
- Disk: no space left on device, where the engine reclaims space mid-job, and its inode variant, which is in shadow mode.
- Missing tools: error while loading shared libraries, where a system package is installed from a vetted list.
- Network: Docker Hub pull rate limits and npm network errors, the category that carries 30 of the 41 live repairs.
Key takeaways
- Only repair inside the running job removes the interruption; explain and propose still need a person.
- The library is as much about refusing to act as about acting: 41 live repairs against 41 live passthroughs.
- A failing assertion is passthrough by design, and the recorded run above shows it failing normally.
- Shadow mode means matched, recorded and deliberately not acted on.
Frequently asked questions
How do I retry failed steps in GitHub Actions workflows?
Will self-healing hide real bugs in my code?
Should I automatically re-run failed GitHub Actions workflows?
How can I prevent an out of memory error in GitHub Actions?
NODE_OPTIONS or the JVM heap flags explicitly, cap test worker counts, and move to a larger runner when the work genuinely needs the memory. Self-healing raises a ceiling for you when the failure is recognizable, which buys the run but does not change the sizing.