Skip to content
Latchkey home
Documentation menu

Self-healing: what it does

How Latchkey runners detect and fix transient CI failures during the run, what they will never touch, and how heal proposals become reviewable pull requests.

of diagnosis
3 stages
exit codes, patterns, then AI
healable categories
5
network, config, missing tools, memory, disk
code changes in-run
0
fixes only touch the ephemeral runner
by default
On
every runner, every plan

In short

  1. Every runner detects and fixes transient failures (network, config, missing tools, memory, disk) during the run, in the steps self-heal reaches; on by default.
  2. Three-stage diagnosis: exit codes, then a pattern library, then bounded AI; fixes only ever touch the ephemeral runner.
  3. Durable fixes arrive as heal PRs you review; a failure that repeats on every try still fails, with original logs intact.
  4. Failures in your own code that self-heal records arrive as a structured bundle over MCP, ready for your coding agent: root cause, exact failing file and captured logs, where Latchkey kept them.
app.latchkey.dev / acme / runs / job

The job page for job test in the acme/api CI pipeline, under the trail Runs / acme/api CI / the run's id / Job. In This job the job is Running: started Oct 14, 11:47, Queue to start 12.8s, Cost under a cent so far. The log's bar names the open step, Install dependencies, beside Search the log, Raw and Follow. The step is Running, and its output shows npm error code ETIMEDOUT: a request to registry.npmjs.org failed with connect ETIMEDOUT. Two [latchkey-bash-wrapper] lines follow as the runner consults its heal service. The step's heal record then appears above its output: What Latchkey did, Latchkey matched this failure to a known pattern and retried this step only, in place. The steps before it did not run again., with Pattern match and the Open the heal link, then One log covers every try of this step. A [latchkey] line then reads self-heal is running this step again, try 1 of 3 (reason: network). The retry prints added 1291 packages, and audited 1292 packages in 15s, then found 0 vulnerabilities, and a last [latchkey] line reads the re-run passed, so the step keeps the re-run's exit code. The page then shows the job Passed and Healed, with Pattern match beside its name and Took 1m 26s. Install dependencies reads Passed, then Healed, in 59s; Run the suite is Passed in 24s; and the log's foot reads Complete; the job itself never ran again.

FIG. 3.8-AA registry timeout fails npm ci; the runner matches it to a known pattern, retries that step in place, and the job goes green without running again. The job page marks the job and the step Healed and says under the step what Latchkey did.

Every Latchkey runner ships with self-healing built in, enabled by default. When a workflow step fails, the runner diagnoses the failure locally and, when it is a known transient class, fixes it and retries the step in place. Your build goes green without a human re-run, and every intervention is recorded for you to inspect.

The same holds for a job Latchkey runs from your workflow file: when one of its steps fails, a heal retries that step only, in place, and never re-runs the steps before it. A retried step keeps the secrets it was handed; self-heal never re-runs it from outside the step. Self-heal does not run at all on a step that a trial run would leave out: a step written advisory: skip, a step that can publish and writes no advisory:, and every step of a job that writes advisory: hold or names an environment and writes no advisory:. Nor does it run on a step that starts once its job holds more than 64 values for the runner's heal service to hide: each distinct value of a secret its steps have been handed so far, each service setting of 28 bytes or more, and the credentials Latchkey hands its steps, up to two plus one for each identity token a step is issued. A step that then fails says in its log that self-heal is off for it, and the job's log still hides every one of those values.

self-heal · diagnosis, fastest first

Three failing steps go through self-heal diagnosis, fastest first. A build killed with exit 137 is decided at stage 1, the exit-code lookup, as out of memory, and is retried once with nothing changed. An npm ci that fails with ETIMEDOUT gets no match at stage 1 and matches the NPM_NETWORK_TIMEOUT pattern at stage 2, and is retried with backoff. An npm test that fails with an AssertionError matches nothing at stages 1 and 2, and the bounded AI at stage 3 finds a real failure in your code, so it stands with its original logs and the failure bundle goes to your agent.

FIG. 3.8-BDiagnosis runs fastest first: an out-of-memory kill is decided by its exit code, a registry timeout by a known pattern, and only what neither recognizes reaches the bounded AI. A failure in your code that repeats on every try still fails.
01
Step fails
A step exits non-zero; the runner captures its output in-line
02
Diagnose
The three-stage cascade identifies the failure class
03
Fix in-runner
Retry with backoff, raise memory, free disk, install a tool
04
Retry in place
The step re-runs on the same machine, same workspace
05
Green
The job continues; the run is marked Healed and the heal is recorded on its job

The three-stage diagnosis cascade#

Diagnosis runs fastest-first, so the common cases cost almost nothing:

01
Exit-code lookup (instant)

Some failures identify themselves. Exit 137, the kernel killing a process that ran out of memory, is decided and fixed right at this stage. Exit 127 (command not found) is flagged here too, but its actual fix comes from stage 2's pattern library and install allowlist.

02
Pattern library (deterministic)

The step output is matched against a curated library of known failure signatures: registry timeouts and 5xx responses across npm, yarn, pnpm, pip, uv, Go modules, cargo, NuGet, Docker and GitHub registries, Composer, Bundler, Maven, apt, and git; DNS and TLS hiccups; rate limits; heap exhaustion; disk-full errors; lockfile drift. A pattern match is deterministic: same failure, same fix, every time.

03
Bounded AI diagnosis (novel failures)

Anything the first two stages do not recognize goes to a sandboxed AI agent with a strict time budget (about four minutes). It reads the captured output, reasons about the root cause, and either applies a safe in-runner fix, proposes a durable fix as a pull request, or concludes the failure is real and lets it stand.

What it catches, by category#

The biggest source of flaky CI: package registries and external services having a bad moment. Timeouts, 5xx responses, DNS resolution failures, TLS handshake errors, and rate limits across every major ecosystem. The fix is a retry with exponential backoff; the walkthrough below shows this exact case end to end.

output
1npm ERR! code ETIMEDOUT
2npm ERR! network request to https://registry.npmjs.org/react failed
3npm ERR! network This is a problem related to network connectivity.

Out-of-memory kills (exit 137) and language-runtime heap exhaustion. An exit 137 is retried once with nothing changed. Heap exhaustion that Node or the JVM reports is retried with a raised limit, through NODE_OPTIONS=--max-old-space-size=6144 or _JAVA_OPTIONS=-Xmx4g -Xms512m.

output
1FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Disk-full failures ("No space left on device", ENOSPC). The fix frees space by pruning Docker layers and npm, pip, yarn, Gradle, and cargo caches plus /tmp, then retries the step.

output
1Error: ENOSPC: no space left on device, write

A step assumes a tool that the job never installed (exit 127, "command not found"). The fix installs the missing system package from a vetted allowlist (the same package set found on GitHub-hosted runner images) and retries; tools outside the allowlist are deliberately left alone.

output
1/usr/bin/bash: line 1: ffmpeg: command not found

Known config mismatches with known-safe rewrites, such as a lockfile out of sync with the manifest, where npm ci fails but npm install is the documented remediation.

output
1npm ERR! `npm ci` can only install packages when your package.json and
2package-lock.json are in sync.

The fix toolbox#

Every fix is applied inside the ephemeral runner only, and the runner is destroyed after the job:

Retry with backofffix
For transient network failures: wait, then re-run the step, escalating the delay.
Set environmentfix
Raise memory limits for the retry: NODE_OPTIONS, _JAVA_OPTIONS, GRADLE_OPTS, and friends.
Free diskfix
Prune Docker layers, package caches, and /tmp when the disk fills mid-job.
Install a packagefix
apt-get install a missing tool, restricted to an allowlist: the same package set found on GitHub-hosted runner images.
Command rewritefix
Swap a known-bad invocation for the documented remediation (npm ci to npm install on lockfile drift).

A heal, end to end#

Here is one real heal, an npm registry timeout, from failing step to green job. Your step output streams unchanged, in real time; during a heal the runner adds lines of its own: two while the on-runner heal service is consulted, then one before and one after each re-run of the step:

job log
1npm ci
2npm ERR! code ETIMEDOUT
3npm ERR! syscall connect
4npm ERR! network request to https://registry.npmjs.org/lodash failed, reason: connect ETIMEDOUT 104.16.92.83:443
5npm ERR! network This is a problem related to network connectivity.
6[latchkey-bash-wrapper] BEGIN sidecar POST (boot_wait=30s max_time=320s url=http://localhost/diagnose socket=/run/latchkey-self-heal/sock)
7[latchkey-bash-wrapper] END sidecar POST ok (attempts=1 http=200)
8[latchkey] self-heal is running this step again, try 1 of 3 (reason: network).
9npm ci
10added 1291 packages, and audited 1292 packages in 42s
11found 0 vulnerabilities
12[latchkey] the re-run passed, so the step keeps the re-run's exit code.
  1. The step fails. npm ci exits non-zero after npm exhausts its own internal retries. The runner captured the step output as it streamed, so the diagnosis starts with the full picture already in hand.
  2. Stage 1, exit code. Exit 1 is not decisive on its own, so the cascade moves on.
  3. Stage 2, pattern match. The ETIMEDOUT lines match a known npm network signature in the pattern library: verdict heal, category network, no AI involved. The two [latchkey-bash-wrapper] lines are the only trace of this consultation in your logs.
  4. Retry with backoff. The plan allows up to 3 attempts with an escalating delay starting at 2 seconds. The runner waits, then re-runs the step on the same machine, in the same workspace: a [latchkey] line names the try, and the second npm ci in the log is that retry.
  5. Green. The retry passes and the job continues as if the timeout never happened. One [latchkey] line after the retry says it passed, and heals that change the machine announce it too (for example [latchkey-bash-wrapper] installed package: ffmpeg), so no intervention is invisible.

The narrative you did not see in the logs is on the dashboard: the run on Latchkey CI, or the job on Latchkey on GH, shows Healed beside its result on the Runs page; on a Latchkey CI job, the healed step's record says what Latchkey did, with its Pattern match stage, and Open the heal opens the Heal Details drawer with the stage-by-stage story and the action taken ("Retried the npm command with exponential backoff after a network timeout").

Heal pull requests#

Some root causes deserve a durable fix in your repository, not just an in-run rescue: a setup step missing from the workflow, a missing engines pin in package.json, a too-low job timeout. When a successful heal traces back to a structural cause like these, self-healing produces a structured proposal and opens a heal PR with a typed, deterministic edit.

01
Root cause found
The diagnosis points at something in the repo
02
Structured proposal
Error summary, root cause, and a typed change
03
PR opened
An ordinary pull request on a latchkey branch
04
You review & merge
Or close it; nothing merges itself
  • Each PR is titled "Latchkey heal: <error summary>"; the body explains the Error, the Root cause, and the Fix, and links back to the workflow run, so review takes minutes.
  • Proposal PRs carry a verification note: Verified at runtime when Latchkey applied the fix on the live runner and the failing job passed on retry, or Proposed fix, not verified by a passing run otherwise. On Latchkey on GH, a Latchkey verification section joins the PR once its own run of the failing workflow passes or fails, saying which.
  • On Latchkey on GH, when retries rescued a workflow's failing runs, the AI Insight finding for that workflow counts them in Pro ("Latchkey auto-retried this workflow N times; M passed on the retry"); auto-retries pause when a workflow's retry pass rate is poor.
  • Heal PRs are never auto-merged, and pull requests from forks are never touched.
  • PR creation uses the GitHub App's permissions; if you never merge a heal PR, nothing in your repository changes.

Declined fixes: stop proposing#

Not every proposal class is welcome on every repository. Self-heal proposal findings on the AI Insight page carry a Stop proposing action (owners and admins): Latchkey then permanently stops opening PRs for that failure class in that workflow file on that repository until you undo it under Settings, Self-healing, Declined fixes. Self-healing still runs on those failures; it just stops proposing workflow changes.

What it will and will never do#

Self-healing does
  • Retry failed steps with backoff, on the same runner
  • Raise memory limits and free disk space for the run
  • Install allowlisted missing system packages
  • Propose durable fixes as reviewable pull requests
Self-healing never
  • Modifies your code or repository during a run
  • Merges anything: repo changes only arrive as PRs you review
  • Hides a failure that repeats: a step that fails on every try fails with its original logs
  • Touches pull requests from forks

Hand off to your coding agent#

Self-healing fixes the environment, never your source. When the real problem is a bug in your own code that fails on every try, the build fails, and when self-heal records the failure, it arrives as a structured bundle served over the Latchkey MCP server, ready for your own coding agent to fix from. The bundle hands your agent the pieces it would otherwise reconstruct by hand, where Latchkey kept them:

  • The root cause, in plain language.
  • The failing step's exit code and the exact source file where the error surfaced.
  • The captured logs of the failing step, including output GitHub hides in its log viewer, with secrets stripped before they leave Latchkey; a very large log is cut.
  • What self-healing found and why it stood down, plus the workflow definition.

In Claude Code, the built-in /mcp__latchkey__fix command does the whole round trip in one step: it grabs the most recent unfixed failure and gets to work. Any MCP-capable agent (Cursor, Codex, and others) can read the same bundles through the server's tools. Setup, API keys, and the ready-made connect command are in Connect your AI agent.

Observability: every heal is on the record#

  • Heal records on each healed Latchkey CI job (from the Runs page) say what Latchkey did, and Open the heal opens the full report with its category, verdict, a plain-language description of the action taken and, for AI-agent heals, the agent's full transcript.
  • Healed is flagged on the Runs page: Healed sits beside the result of a healed Latchkey CI run, or of a healed job on Latchkey on GH, and on a Latchkey CI job Open the heal shows the report. On Latchkey on GH, list_heals and get_heal on the MCP server read the heal's record.
  • The Self-heal outcome report on AI Insight shows how often healing saves a run and which categories dominate, which is itself a signal about your infrastructure.
SignalWhat it tells you
Outcome badgeHealed (the fix succeeded, or a fix was applied whose result is not recorded), Re-ran unchanged (a flaky-test re-run: self-heal ran the failed step again as written, so nothing was fixed), No Action (the system intentionally left the step's result as it was, such as on a failure in your own code), Failed (a fix or a flaky-test re-run was attempted but did not recover the step), or Pending (outcome not yet recorded).
Fix-type pillHow the heal was decided: Auto-fix (a deterministic exit-code rule), Pattern match (a known failure signature with a known fix), or Agent fix (the agent investigated).
Agent transcriptThe step-by-step record of an agent heal: the plan, per-turn hypothesis and reasoning, each action and its result, and durations.

Controls#

  • Workspace-level switch, Self-heal for this workspace, in Settings, Self-healing and in the Self-heal section of the Runners page, which share it (owners and admins), on by default for new workspaces; changes take effect within about a minute.
  • Max retries per step on the Self-heal card of the Runners page or in Settings, Self-healing (owners and admins): how many times self-heal may run a failed step again before the step keeps its failure, from 1 to 6, 3 unless you change it.
  • Flake Shield, only in Settings, Self-healing (owners and admins): what self-heal does when a test step fails on Latchkey CI, while self-heal is on; the setting does not change Latchkey on GH jobs. A workspace can choose Off or Learn only. With Learn only, self-heal can run a failing test step again to learn whether its failing tests are flaky, and the step keeps its failure whatever the re-run shows; flaky-test re-runs in learn-only mode are billed as runner minutes (Runner usage and free minutes). Re-run and Re-run and quarantine are each shown only to a workspace already on that mode, which can keep it or move off it.
  • There is no per-repository toggle; the workspace switch applies to all monitored repositories.
  • Declined fixes, also under Settings, Self-healing, lists the proposal classes you have stopped, with the option to undo each one.

Common questions#

Q.01Does self-healing slow my builds down?

No. It only activates when a step fails; passing steps run with zero added latency. A healed failure costs the diagnosis plus the retry, which is almost always far cheaper than a human noticing a red build and clicking re-run.

Q.02Can it see my secrets?

It can see a secret your step prints. When a step fails, self-heal reads the step's output on the runner. When its AI stage runs, the end of that output and any of the job's files it reads go to our AI model provider, and the heal record keeps up to 1 MiB of each output stream for 90 days. On GitHub Actions the output is captured before GitHub masks secrets. The runner removes values a bash step hid with ::add-mask::, secrets in a known format and, on GitHub Actions, every secret the workflow file maps into the step's env: before anything is sent or kept, and known secret patterns are removed again when a heal record is served. A step whose workflow file uses a secret any other way is not diagnosed. On Latchkey CI, the secrets Latchkey delivered to the job are masked on the runner first. The runner itself is destroyed after the job. The full list is on Security and GitHub permissions.

Q.03Can it retry forever and run up my bill?

No. Retries are bounded per step, the AI stage has a hard time budget, and the job itself has a time limit: a Latchkey CI job can set timeout-minutes up to 180 (30 by default), a CLI job runs for up to 3 hours, and a Latchkey on GH runner for up to 4 hours from its start, or up to 8 when the job is still running at 4. There is also no separate fee for self-healing: extra runtime during a heal bills at the runner's standard per-minute rate.

Q.04What happens when it cannot fix a failure?

The step still fails, with its original logs intact. If self-heal re-ran it first, each re-run's output follows in the same log, marked by its [latchkey] lines. If it ran the step while testing a fix, one [latchkey] line counts those tries, and their output is not in the log. When self-heal diagnosed the failure, the diagnosis is under Open the heal on a Latchkey CI job and, on GitHub Actions, in its failure bundle on the MCP server: unhealable is a first-class verdict, not an error. A step self-heal does not reach fails with its original logs and is not retried; a failed uses: step is still read after the run and can get a diagnosis there.

To see which real-world failures healing covers, browse the self-healing pattern library on Learn - each entry shows the manual fix next to what the runner does automatically. For anything currently failing your builds, start from the full CI/CD error library.