Skip to content
Latchkey
Self-Healing CI/CD

The failures that should never reach a human - and how they get automated.

A catalog of transient and mechanical CI/CD failures that can be automatically detected, fixed, and retried. Each page shows the problem, the manual fix, and how self-healing CI platforms like Latchkey automate it away so your pipeline just stays green.

Transient failures

Network, registry, and timeout flake that retries clean.

Cargo registry timeoutA timeout fetching a crate from the cargo registry is a transient transport blip, not a missing crate. See th… Composer hits a GitHub rate-limitA Composer install that hits a GitHub API rate-limit is a temporary throttle, not a missing package. See the… Alpine apk fetch failsAn apk fetch that fails is a transient mirror or network blip, not a missing package. See the manual fix and… Flaky network file downloadA one-off failed file download (curl, wget, fetch) is a transient blip, not a build bug. See the manual fix a… Go module proxy returns 5xxA 5xx from the Go module proxy is a transient backend blip, not a missing module. See the manual fix and how… Maven Central download timeoutA timeout pulling an artifact from Maven Central is a transient transport blip, not a missing dependency. See… NuGet restore network blipA NuGet restore that fails on a network blip is a transient transport problem, not a missing package. See the… pip index SSL errorA transient SSL error talking to a pip index is a momentary handshake or proxy blip, not a broken package. Se… Cloud deploy API rate-limited (429)A 429 from a cloud provider API during deploy is a throttle that resets, not a deploy bug. See the manual fix… Git submodule fetch timeoutA submodule fetch that times out is a transport blip, not a repo problem. See the manual fix and how self-hea… Transient step timeoutA step that times out once but completes on retry hit a transient slowdown, not a hang. See the manual fix an… Yarn install network timeoutA Yarn install that times out fetching a package is a transient transport blip, not a missing dependency. See… Artifact upload times outAn artifact upload that times out is a transient transfer problem, not a build failure. See the manual fix an… Azure Blob Storage throttles a requestA 503 ServerBusy throttle from Azure Blob Storage is a temporary rate limit, not a build failure. See the man… Electron download times outAn Electron download that times out is a transient fetch blip on a large binary, not a broken install. See th… npm registry returns 503A 503 from the npm registry is a transient backend blip, not a missing package. See the manual fix and how se… S3 artifact store returns 503A 503 (SlowDown) from S3 during an artifact transfer is a transient throttle, not a build failure. See the ma… Docker Hub pull-rate limit (429)Docker Hub pull-rate limits (429 Too Many Requests) are transient and shared. See the manual fix and how self… git clone / fetch network failuresGit clone and fetch network errors in CI are transient transport failures. See the manual fix and how self-he… Transient registry / 5xx failuresRegistry timeouts, 429s, and 5xx errors are transient by definition. See why they happen and how self-healing… Intermittent segfault in a native testA native test that segfaults intermittently may be a timing or resource flake, not a real crash. See the manu… Flaky test containmentFlaky tests fail intermittently with no code change. Learn how to identify them and how self-healing CI retri… Test fails only under parallel loadA test that fails only when run in parallel is a resource-contention flake, not always a real bug. See the ma… DNS resolution failuresA "could not resolve host" error in CI is a transient DNS blip, not a config bug. See the manual fix and how… Network timeouts & flakinessNetwork timeouts during dependency downloads and API calls are transient. Learn the manual mitigations and ho… Registry returns 5xx mid-pullA registry that returns a 5xx partway through an image pull hit a transient backend error, not a bad image. S… Registry auth token briefly expiredA registry push or pull that fails on an expired short-lived token is a timing issue, not bad credentials. Se… CI cache service returns 503A 503 from the CI cache service is a transient backend blip, not a build bug. See the manual fix and how self… CDN cold cache 504A 504 from a CDN on a cold cache is a transient origin-fetch blip, not a broken asset. See the manual fix and… Clock-skew TLS validation failureA TLS validation failure caused by a skewed runner clock is a time problem, not a bad certificate. See the ma… conda solve network blipA conda environment solve that fails fetching channel metadata is a transient network blip, not an unsolvable… Webhook delivery delayedA step that fails waiting on a delayed webhook hit a timing problem, not a missed event. See the manual fix a… DNS rebind during deployA deploy step that fails because a hostname briefly resolved to a stale or changing address is a DNS-propagat… Docker layer pull resets mid-transferA Docker image pull that resets mid-layer is a transient transport blip, not a bad image. See the manual fix… Browser binary download fails (Playwright/Cypress)A failed Playwright or Cypress browser download is a transient fetch problem, not a test bug. See the manual… geckodriver download failsA failed geckodriver download is a transient fetch problem, not a Firefox test bug. See the manual fix and ho… Homebrew bottle download failsA failed Homebrew bottle download is a transient fetch problem, not a missing formula. See the manual fix and… Node prebuilt native binary download failsA failed prebuilt native binary download during npm install is a transient fetch blip, not a build-from-sourc… Git clone hangs on a large repoA clone that hangs on a large repository is a stalled transfer, not a repo problem. See the manual fix and ho… Message queue connection dropA step that fails because a message-queue connection dropped is a transient transport blip, not a broker outa… Proxy / egress connection resetA connection reset through an egress proxy is a transient network blip, not a config bug. See the manual fix… GCS resumable upload retractedA GCS resumable upload that the server retracts mid-flight is a transient blip, not a build failure. See the… CDN purge / propagation is slowA post-deploy check that fails because a CDN purge had not propagated hit a timing issue, not a deploy bug. S… Slow cold start times out a stepA step that times out waiting on a slow cold start hit a warmup delay, not a hang. See the manual fix and how… Third-party status blip during deployA deploy that fails on a brief third-party outage is upstream, not your config. See the manual fix and how se… Transient 429 from an API in testsA test that fails on a 429 from an API it calls hit a rate limit, not a logic bug. See the manual fix and how… Transient SSH connection dropAn SSH connection that drops mid-step is a transient network problem, not an auth or host bug. See the manual… ECR auth token expires mid-pushAn ECR push that fails because the auth token expired mid-push is a timing issue, not bad credentials. See th… Clock-skew TLS / cert failuresA "certificate not yet valid" TLS error is often clock skew on the runner, not a bad cert. See the manual fix… Flaky external-service integration testsIntegration tests that hit external services flake on the service, not your code. See the manual fix and how… Spot instance interrupted mid-jobA job cut off because the spot instance was interrupted is infrastructure, not a build bug. See the manual fi… Spot / SIGTERM preemption (exit 143)A job killed by spot-instance preemption (SIGTERM / exit 143) is infrastructure, not a build bug. See the man… TLS handshake timeoutsTLS handshake timeouts in CI are transient connection failures, not certificate bugs. See the manual fix and… npm / pip / cargo registry timeoutsPackage-manager registry timeouts during install are transient, not lockfile bugs. See the manual fix and how… Package mirror briefly unavailableA package mirror that is briefly unreachable is a transient availability blip, not a missing package. See the… TLS cert chain temporarily unavailableA TLS verification failure from a temporarily unavailable chain or OCSP responder is a transient blip, not an… Homebrew analytics call blockedA brew step that stalls or errors on a blocked analytics call is a non-essential network blip, not a broken i… Runner briefly loses communicationA job that fails because the runner briefly lost contact with the control server is a network blip, not a cod…

Resource failures

OOM and disk problems fixed by adjusting the runner.

"No space left on device" recoveryDisk-exhaustion failures are recoverable without a code change. See the manual cleanup and how self-healing C… Corrupted dependency cacheA corrupted or partial dependency cache fails the build until cleared. See the manual fix and how self-healin… "fork: cannot allocate memory" / PID limitsA "cannot allocate memory" on fork is a resource ceiling, not a code bug. See the manual fix and how self-hea… "Too many open files" (ulimit)An EMFILE "too many open files" error is a file-descriptor limit, not a code bug. See the manual fix and how… Cypress binary cache miss download failsA Cypress binary download triggered by a cache miss that then fails is a transient fetch blip, not a test bug… Job OOM-killed mid-runA job killed by the out-of-memory killer mid-run hit a memory ceiling, not a code bug. See the manual fix and… Shared memory (/dev/shm) exhaustedA crash from a full /dev/shm is a shared-memory ceiling, not a code bug. See the manual fix and how self-heal… Inode / disk-pressure exhaustionRunning out of inodes fails writes even with free bytes left. See the manual fix and how self-healing CI recl… OOM-killed jobs (exit 137)Out-of-memory kills (exit 137) are mechanical, not code bugs. See the manual fix and how self-healing CI auto… Metaspace / native-thread OOMJVM metaspace and "unable to create native thread" OOMs are resource ceilings, not heap leaks. See the manual… npm EINTEGRITY on a corrupt cacheAn npm EINTEGRITY error is a corrupted or partial cache entry, not a bad dependency. See the manual fix and h… GPU briefly unavailableA job that fails because a GPU was not available yet hit a device-readiness race, not a code bug. See the man… Runner runs out of disk mid-buildA runner that fills its disk partway through a build hit a capacity ceiling, not a code bug. See the manual c…

Setup & environment

Missing packages and config gaps with durable fixes.

Missing compiler / build toolA missing compiler (gcc, clang, cc) fails native builds until installed. See the manual fix and how self-heal… Missing system packagesA missing system library or tool fails the build until someone installs it. See the manual fix and how self-h… Port briefly in use (EADDRINUSE)An "address already in use" error is usually a brief port race, not a real conflict. See the manual fix and h… ChromeDriver download version mismatchA ChromeDriver that fails to download a matching version is a transient setup race, not a test bug. See the m… Database failover during migrationA migration step that fails because the database failed over to a new primary is a transient infrastructure e… Docker daemon not ready at startA "cannot connect to the Docker daemon" error at job start is a startup race, not a config bug. See the manua… Gradle distribution download failsA failed Gradle wrapper distribution download is a transient fetch problem, not a build-script bug. See the m… Helm repo index download failsA failed Helm repo index download is a transient fetch blip, not a missing chart. See the manual fix and how… Service container healthcheck flapsA service container whose healthcheck flaps before settling is a startup-stabilization issue, not a broken se… Kubernetes rollout briefly not readyA deploy step that fails because a Kubernetes rollout was not ready in time often just needed longer. See the… Load balancer warmup 502A 502 from a load balancer while new targets are warming up is a transient readiness blip, not a broken deplo… Playwright host validation failsA Playwright host-dependency validation that fails fetching system packages is a transient setup blip, not a… Transient apt mirror 404A 404 from an apt mirror is usually a mid-sync or stale-index blip, not a removed package. See the manual fix… apt GPG keyserver timeoutA timeout importing a GPG key from a keyserver is a transient reachability blip, not a bad repo. See the manu… Flaky DNS inside a service containerA name-resolution failure between containers in CI is usually a transient internal-DNS blip. See the manual f… apt "Hash Sum mismatch"An apt "Hash Sum mismatch" is a stale or mid-sync index, not a broken package. See the manual fix and how sel… apt / yum mirror failuresOS package mirror errors during apt or yum are transient mirror problems, not build bugs. See the manual fix… Database service container slow to accept connectionsTests that fail because a database service container is not ready yet hit a startup race, not a config bug. S… Kubernetes API server briefly unreachableA kubectl call that fails because the API server was briefly unreachable is a transient connectivity blip, no…
Explore other topics
GitHub ActionsWorkflow, runner, and YAML errors - diagnosed and fixed. Node.js & npmnpm, yarn, and pnpm failures in CI - solved. DockerBuild, run, compose, and registry errors - explained. Pythonpip, poetry, venv, and pytest failures - fixed. Java & JVMMaven, Gradle, and JVM failures in CI - resolved. GoGo build, module, and test failures - diagnosed.