Documentation menu

Getting started

Dashboard & analytics

Managed runners

Caching

Team & notifications

Billing & plans

Help

Self-healing: what it does

How Latchkey runners detect and fix transient CI failures during the run, what they will never touch, and how heal proposals become reviewable pull requests.

The Self-Healing section of the Runners page — The Self-Healing section: heal KPIs, trends, and the Recent Heals feed.

Every Latchkey runner ships with self-healing built in, enabled by default. When a workflow step fails, the runner diagnoses the failure locally and, when it is a known transient class, fixes it and retries the step in place. Your build goes green without a human re-run, and every intervention is recorded for you to inspect.

3 stages

of diagnosis

exit codes, patterns, then AI

failure categories

network, memory, disk, tools, config, code

code changes in-run

fixes only touch the ephemeral runner

by default

every runner, every plan

01Step failsA step exits non-zero; the runner captures its output in-line

02DiagnoseThe three-stage cascade identifies the failure class

03Fix in-runnerRetry with backoff, raise memory, free disk, install a tool

04Retry in placeThe step re-runs on the same machine, same workspace

05GreenThe job continues; the heal is recorded in Recent Heals

The three-stage diagnosis cascade#

Diagnosis runs fastest-first, so the common cases cost almost nothing:

Exit-code lookup (instant)

Some failures identify themselves. Exit 137 is the kernel killing a process that ran out of memory; 127 means the command was not found. A table of known exit codes maps these to their fix immediately.

Pattern library (deterministic)

The step output is matched against a curated library of known failure signatures: registry timeouts and 5xx responses across npm, yarn, pnpm, pip, uv, Go modules, cargo, NuGet, Docker and GitHub registries, Composer, Bundler, Maven, apt, and git; DNS and TLS hiccups; rate limits; heap exhaustion; disk-full errors; lockfile drift. A pattern match is deterministic: same failure, same fix, every time.

Bounded AI diagnosis (novel failures)

Anything the first two stages do not recognize goes to a sandboxed AI agent with a strict time budget (about three minutes). It reads the captured output, reasons about the root cause, and either applies a safe in-runner fix, proposes a durable fix as a pull request, or concludes the failure is real and lets it stand.

What it catches, by category#

The biggest source of flaky CI: package registries and external services having a bad moment. Timeouts, 5xx responses, DNS resolution failures, TLS handshake errors, and rate limits across every major ecosystem.

output

npm ERR! code ETIMEDOUT
npm ERR! network request to https://registry.npmjs.org/react failed
npm ERR! network This is a problem related to network connectivity.

# self-heal: network/registry timeout -> retry with backoff -> passed on retry 2

Out-of-memory kills (exit 137) and language-runtime heap exhaustion. The fix raises the limit for the retry via environment variables such as NODE_OPTIONS=--max-old-space-size=4096 or _JAVA_OPTIONS=-Xmx4g.

output

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

# self-heal: memory/node heap -> set NODE_OPTIONS=--max-old-space-size=4096 -> retried

ENOSPC and disk-full failures. The fix frees space by pruning Docker layers and npm, pip, yarn, Gradle, and cargo caches plus /tmp, then retries the step.

output

Error: ENOSPC: no space left on device, write

# self-heal: disk/full -> pruned docker + package caches (14.2 GB freed) -> retried

A step assumes a tool that the job never installed (exit 127, "command not found"). The fix installs the missing system package from a vetted allowlist (the same package set found on GitHub-hosted runner images) and retries; tools outside the allowlist are deliberately left alone.

output

/usr/bin/bash: line 1: ffmpeg: command not found

# self-heal: tool_missing -> apt-get install ffmpeg (allowlisted) -> retried

Known config mismatches with known-safe rewrites, such as a lockfile out of sync with the manifest, where npm ci fails but npm install is the documented remediation.

output

npm ERR! `npm ci` can only install packages when your package.json and
package-lock.json are in sync.

# self-heal: config/lockfile drift -> command rewrite: npm ci -> npm install -> retried

The fix toolbox#

Every fix is applied inside the ephemeral runner only, and the runner is destroyed after the job:

fixRetry with backoffFor transient network failures: wait, then re-run the step, escalating the delay.

fixSet environmentRaise memory limits for the retry: NODE_OPTIONS, _JAVA_OPTIONS, and friends.

fixFree diskPrune Docker layers, package caches, and /tmp when the disk fills mid-job.

fixInstall a packageapt-get install a missing tool, restricted to an allowlist: the same package set found on GitHub-hosted runner images.

fixCommand rewriteSwap a known-bad invocation for the documented remediation (npm ci to npm install on lockfile drift).

What a heal looks like in your logs#

Your step output streams unchanged, in real time. When a heal happens, the runner's diagnostic lines appear alongside it, so the intervention is never invisible:

terminal

$ npm ci
npm ERR! code ETIMEDOUT
npm ERR! network request to https://registry.npmjs.org/lodash failed
# [latchkey-bash-wrapper] step failed (exit 1), diagnosing...
# [latchkey-bash-wrapper] verdict: network/registry-timeout (stage 2 pattern match)
# [latchkey-bash-wrapper] action: retry with backoff (attempt 1 of 3, waiting 5s)
$ npm ci
added 1291 packages in 42s
# [latchkey-bash-wrapper] heal successful, continuing job

Heal pull requests#

Some root causes deserve a durable fix in your repository, not just an in-run rescue: a setup step missing from the workflow, a missing engines pin in package.json, a too-low job timeout. When a successful heal traces back to a structural cause like these, self-healing produces a structured proposal and opens a heal PR with a typed, deterministic edit.

01Root cause foundThe diagnosis points at something in the repo

02Structured proposalError summary, root cause, and a typed change

03PR openedAn ordinary pull request on a latchkey branch

04You review & mergeOr close it; nothing merges itself

Each PR is titled "Latchkey heal: <error summary>"; the body explains the Error, the Root cause, and the Fix, and links back to the workflow run, so review takes minutes.
Heal PRs are never auto-merged, and pull requests from forks are never touched.
PR creation uses the GitHub App's permissions; if you never merge a heal PR, nothing in your repository changes.

What it will and will never do#

Self-healing does

Retry failed steps with backoff, on the same runner
Raise memory limits and free disk space for the run
Install allowlisted missing system packages
Propose durable fixes as reviewable pull requests

Self-healing never

Modifies your code or repository during a run
Merges anything: repo changes only arrive as PRs you review
Hides real failures: unhealable steps fail with original logs
Touches pull requests from forks

Hand off to your coding agent#

Self-healing fixes the environment, never your source. When the real problem is a bug in your own code, the build fails truthfully, and the failure arrives as a complete, structured bundle served over the Latchkey MCP server, ready for your own coding agent to fix from. The bundle hands your agent what it would otherwise reconstruct by hand:

The root cause, in plain language.
The failing step's exit code and the exact source file where the error surfaced.
The full, untruncated logs of the failing step, including output GitHub hides in its log viewer, with secrets stripped before they leave Latchkey.
What self-healing already investigated and why it stood down, plus the workflow definition.

In Claude Code, the built-in /mcp__latchkey__fix command does the whole round trip in one step: it grabs the most recent unfixed failure and gets to work. Any MCP-capable agent (Cursor, Codex, and others) can read the same bundles through the server's tools. Setup, API keys, and the ready-made connect command are in Connect your AI agent.

Observability: every heal is on the record#

Recent Heals (on the Runners page) lists every intervention with its category, verdict, and a plain-language description of the action taken; AI-diagnosed heals include the agent's reasoning iterations.
Healed runs are flagged in the Pipeline Performance runs table and deep-link to their heal report.
Heal KPIs and trends on the Runners page show how often healing saves a run and which categories dominate, which is itself a signal about your infrastructure.

Signal	What it tells you
Outcome badge	Healed (the fix succeeded), No Action (the system intentionally did not act, such as on a failure in your own code), Failed (a fix was attempted but did not recover the step), or Pending (outcome not yet recorded).
Fix-type pill	How the heal was decided: Auto-fix (a deterministic exit-code rule), Pattern match (a known failure signature with a known fix), or Agent fix (the agent investigated).
Agent transcript	The step-by-step record of an agent heal: the plan, per-turn hypothesis and reasoning, each action and its result, and durations.

Controls#

Workspace-level mode in Settings, Self-Healing (owners and admins); changes take effect within about a minute.
Per-configuration toggle on the Runners page, to exclude a specific runner type.
There is no per-repository toggle today; the workspace mode applies to all monitored repositories.

Common questions#

Does self-healing slow my builds down?

No. It only activates when a step fails; passing steps run with zero added latency. A healed failure costs the diagnosis plus the retry, which is almost always far cheaper than a human noticing a red build and clicking re-run.

Can it see my secrets?

Diagnosis runs locally on your runner against the step output your job already prints. Nothing new is exposed: secrets masked by GitHub Actions stay masked, and the runner is destroyed after the job.

Can it retry forever and run up my bill?

No. Retries are bounded per step, the AI stage has a hard time budget, and the runner itself has a 4-hour lifetime cap. There is also no separate fee for self-healing: extra runtime during a heal bills at the runner's standard per-minute rate.

What happens when it cannot fix a failure?

The step fails exactly as it would on any other runner, with original logs intact, plus a diagnosis you can read in Recent Heals. Unhealable is a first-class verdict, not an error.