Self-Healing CI Explained: What It Repairs, and What It Must Not
Self-healing CI repairs a failing step on the machine while the job is still running. The hard part is not the repair; it is drawing the line between a failure that should be fixed and one that must be allowed to fail.
Most CI failures are not bugs. A registry times out, a runner runs out of disk, a package mirror returns a 503, a tool is missing from the image. The build fails, someone reads a log, someone clicks re-run, and the second attempt passes without a single line of code changing. That loop costs the minutes of the failed run plus a human interruption, and it teaches the team to re-run first and read later, which is how a real failure gets ignored.
Self-healing addresses that loop directly, and it is worth being precise about what "self-healing" means, because three quite different things are sold under the name.
Three grades, frequently conflated
What is safe to heal
- Network failures: registry timeouts and 5xx responses from package and image registries.
- Resource exhaustion: out-of-memory kills and disk-full errors, where the job is correct and the machine was not big enough.
- Missing tooling: a command that is absent from the image, installed from a vetted allowlist rather than arbitrary input.
- Environment drift: a missing variable, a toolchain version mismatch, a file-handle limit set too low.
What must never be healed
A failing test, a compile error, a type error, a lint violation. These are the signal CI exists to produce, and a system that makes them go away is not fixing your pipeline, it is disabling it. The correct behaviour for a genuine failure is to fail, with the original logs, exactly as if nothing had intervened. Any tool in this category should tell you plainly where that line sits, because a vendor who is vague about it is describing a retry loop.
Self-healing is not blanket retry
Retrying a whole job on failure is the crude version, and it is worse than it looks: it doubles the minutes for every failure including the real ones, it hides flakiness instead of surfacing it, and it delays the feedback a genuine bug should have produced immediately. Repairing a specific diagnosed cause is a different operation. It also leaves an audit trail, so you can see what was wrong and how often, which is the data you need to fix the underlying problem.
Applying this to your pipeline
- Measure before changing. Most CI optimisation targets the wrong step because the slow one is assumed rather than timed.
- Cache what is expensive to produce and cheap to validate, and key the cache to the exact tool version.
- Fail fast: run the cheapest checks that can reject a change first, so an expensive job never starts on code that cannot pass.
- Prefer determinism over speed when they conflict. A fast pipeline nobody trusts gets re-run, which is slower than a slow one that is believed.