Skip to content
Latchkey
Runner Infrastructure

OOM, disk, network, and timeout failures - the infra layer.

Diagnose the infrastructure-level failures that break CI regardless of language: out-of-memory kills, disk exhaustion, network flakiness, timeouts, and rate limits. Where Latchkey self-healing has the most impact.

Memory & OOM

OOM kills, exit 137, and heap limits.

runner container "OOMKilled"Fix an ARC runner container killed with OOMKilled in CI - the job exceeded the pod memory limit, so Kubernete… "bash: fork: Cannot allocate memory"Fix "bash: fork: Cannot allocate memory" in CI - the shell could not spawn a child process because the runner… "Cannot allocate memory"Fix "Cannot allocate memory" (ENOMEM) errors in CI - the runner ran out of usable RAM or hit a cgroup limit b… fork: cannot allocate memoryFix "fork: Cannot allocate memory" (ENOMEM) in CI when the runner cannot spawn a new process. How to free mem… "cannot fork: Resource temporarily unavailable"Fix "cannot fork: Resource temporarily unavailable" in CI - the runner hit the per-user process limit (ulimit… "fork: retry: Resource temporarily unavailable"Fix "fork: retry: Resource temporarily unavailable" (EAGAIN) in CI - the runner hit a process/thread limit or… JS heap out of memoryFix "JavaScript heap out of memory" in CI when Node/V8 exceeds its heap limit during builds or tests. How to… "Out of memory: Killed process"Read the kernel "Out of memory: Killed process" / oom-kill line in dmesg to confirm a CI OOM, identify the vi… "unable to create new native thread"Fix JVM "java.lang.OutOfMemoryError: unable to create new native thread" in CI - the runner hit a thread/proc… "Too many open files"Fix "Too many open files" / EMFILE / ENFILE in CI - a process or the runner hit the open file-descriptor limi… "memory cgroup out of memory"Fix "Memory cgroup out of memory: Killed process" in CI - a container or systemd cgroup memory limit was exce… "Killed" (no stack trace)A CI step that just prints "Killed" and dies with no stack trace is almost always an OOM SIGKILL. Learn how t… Swap exhausted / thrashingWhy a CI runner that runs low on memory thrashes swap, slows to a crawl, and then OOM-kills the job. How to d… Exit code 137 (OOM)Why CI steps exit with code 137 and "Killed": the OOM killer sent SIGKILL after the job exceeded runner memor… Container memory limitFix a CI container killed for exceeding its memory limit (cgroup OOM, exit 137) even with free host RAM. How… ICE: Killed (OOM)Fix "internal compiler error: Killed (program cc1plus)" in CI: the compiler was OOM-killed, not a real ICE. H… Go runtime OOMFix "fatal error: runtime: out of memory" in Go CI when the runtime cannot get memory from the OS. How to red… unable to create native threadFix "OutOfMemoryError: unable to create new native thread" in CI when the JVM hits a thread or memory limit.… java OutOfMemoryErrorFix java.lang.OutOfMemoryError in CI when the JVM exceeds its heap or the runner runs out of RAM. How to tune… Jest worker OOMFix "Jest worker ran out of memory" in CI when too many workers or a leak exhausted RAM. How to lower workers… node-gyp killed (SIGKILL)Fix a node-gyp native build killed with signal 9 in CI: the compiler was OOM-killed during install. How to lo… "spawn ENOMEM"Fix Node.js "Error: spawn ENOMEM" in CI - the runner had no memory to fork a child process for child_process.… Python MemoryErrorFix Python MemoryError in CI when a script or test exceeds the runner RAM. How to lower memory usage and tell… Exit code 137 (OOM killed)Why CI steps exit with code 137 and "Killed" - the kernel OOM killer sent SIGKILL after the job exceeded runn… "metric ... exceeds threshold"Fix a TeamCity build failed by a metric failure condition in CI - a build metric such as artifact size or tes… Webpack heap OOMFix a webpack build that crashes with "JavaScript heap out of memory" in CI when large bundles exhaust the V8…

Disk & storage

No space left, cache bloat, and cleanup.

/dev/shm full (browsers)Fix Chrome/Chromium crashes from a full /dev/shm in CI browser tests. The shared-memory mount is too small. H… apt-get out of diskFix "No space left on device" during apt-get in CI when the package cache and unpacked files fill the disk. H… "failed to delete ephemeral runner"Fix ARC "failed to delete ephemeral runner" with a stuck finalizer in CI - the EphemeralRunner will not termi… "INSTALLATION FAILED" (gha-runner-scale-set)Fix a failed Helm install of the ARC gha-runner-scale-set chart in CI - bad values, a name clash, or a missin… "no matches for kind AutoscalingRunnerSet"Fix ARC "no matches for kind AutoscalingRunnerSet" in CI - the CRD is not installed because the controller is… cert-manager webhook (legacy ARC)Fix legacy ARC cert-manager errors in CI - the older actions-runner-controller needs cert-manager for its adm… artifact download "404"Fix Buildkite "artifact download" 404 in CI - the requested artifact does not exist for the build, usually be… "no files matched"Fix Buildkite "buildkite-agent artifact upload: no files matched" in CI - the glob pattern matched nothing, s… "Error uploading artifacts"Fix Buildkite "Error uploading artifacts" in CI - the agent matched files but could not store them, usually a… "reference is not a tree"Fix Buildkite checkout "fatal: reference is not a tree" in CI - the agent working copy does not have the comm… "/tmp" fullFix "No space left on device" from /tmp in CI - the temp filesystem (often small or tmpfs) filled with build… Disk quota exceededFix "Disk quota exceeded" (EDQUOT) in CI when a per-user or per-volume quota caps writes even with free space… "failed to register layer: no space left"Fix "failed to register layer: write ...: no space left on device" during docker pull in CI - image layers fi… "gzip: stdout: No space left on device"Fix "gzip: stdout: No space left on device" in CI - compressing an artifact or stream filled the runner disk.… "inotify watch limit reached"Fix "ENOSPC: System limit for number of file watchers reached" in CI - inotify ran out of watches. Raise fs.i… "No space left on device"Fix "No space left on device" (ENOSPC) in CI - the runner disk filled with caches, layers, and artifacts. How… ENOSPC in CIFix "No space left on device" (ENOSPC) in CI when the runner disk fills with caches, layers, and artifacts. H… Inode exhaustion (ENOSPC)Fix "No space left on device" when df shows free bytes - the runner ran out of inodes, not space. How to conf… Inode exhaustion (ENOSPC)Fix "No space left on device" when df shows free bytes: the runner ran out of inodes, not space. How to confi… "Read-only file system"Fix "Read-only file system" (EROFS) in CI - a write hit a read-only mount, or the kernel remounted the filesy… "tar: write error"Fix "tar: write error: No space left on device" in CI - an archive extract or pack filled the runner disk mid… /tmp directory fullFix ENOSPC from a full /tmp in CI. Temp files or a small tmpfs filled the temp directory. How to clear it, re… Cache directory bloatWhy dependency and build caches bloat over time and fill a CI runner disk - and how to bound, prune, and key… "DiskPressure" (evicted)Fix CI runner "DiskPressure" evictions - a Kubernetes-hosted runner node ran low on disk and the kubelet evic… docker build out of diskFix "write error: no space left on device" during docker build in CI when layers and the build cache fill the… Layer cache filled diskFix a runner disk filled by the Docker layer/build cache in CI when /var/lib/docker grew across jobs. How to… "ENOSPC: ... write"Fix Node.js "ENOSPC: no space left on device, write" in CI - a write/writeFile failed because the runner disk… "[Errno 28] No space left on device"Fix Python "OSError/IOError: [Errno 28] No space left on device" in CI - a write failed because the runner di… npm install out of diskFix ENOSPC during npm install in CI when node_modules and the npm cache exhaust the runner disk. How to recla… tar no space (cache)Fix "tar: write error: No space left on device" when CI restores a cache that outgrew the runner disk. How to… "checkout directory ... is not empty"Fix TeamCity "The checkout directory is not empty" in CI - leftover files in the agent checkout directory blo… "free disk space is below the threshold"Fix TeamCity "Agent has less than N MB of free disk space" in CI - the agent free-space requirement is not me…

Network & flakiness

Timeouts, DNS, and transient 5xx.

"407 Proxy Authentication Required"Fix HTTP 407 "Proxy Authentication Required" in CI - the corporate proxy demands credentials that the request… "Connection reset by peer"Fix "Connection reset by peer" (ECONNRESET at the socket layer) in CI - the remote end or an intermediary sen… "Connection timed out"Fix "Connection timed out" in CI - the TCP handshake got no response because a firewall silently dropped the… "EAI_AGAIN"Fix "EAI_AGAIN" in CI - getaddrinfo returned a temporary DNS failure, meaning the resolver was unreachable or… "ECONNRESET"Fix "ECONNRESET" during a download in CI - an established connection was reset by the peer or an intermediary… "Temporary failure resolving"Fix apt "Temporary failure resolving <mirror>" in CI - apt-get could not resolve the package mirror hostname… apt Failed to fetchFix "apt-get update ... Failed to fetch" in CI when a package mirror is briefly unreachable. How to retry and… "Actions service ... 403"Fix ARC "Actions service responded with 403" in CI - the credential reached the Actions service but is not au… "failed to get registration token"Fix ARC "failed to get registration token" with a 404 Not Found in CI - the GitHub App or PAT the controller… "secret ... not found" (githubConfigSecret)Fix ARC "githubConfigSecret not found" in CI - the scale set references a secret that does not exist in the r… "no runner group found"Fix ARC "no runner group found" in CI - the runnerGroup named in the scale set values does not exist for the… "the GitHub App ... does not have permission"Fix ARC "the GitHub App does not have permission" in CI - the App authenticated but its permission set does n… "401 Bad credentials"Fix ARC "401 Bad credentials" in CI - the App private key, App ID, installation ID, or PAT in githubConfigSec… "failed to create session"Fix ARC RunnerScaleSet listener "failed to create session" in CI - the listener could not open a message sess… runner "ImagePullBackOff"Fix an ARC runner pod in ImagePullBackOff in CI - Kubernetes cannot pull the runner or DinD image because of… runnerScaleSetName must match runs-onFix ARC jobs never matching a runner in CI - the scale set name (the Helm installation name) must equal the w… "Agent lost" mid-jobFix Buildkite "Agent lost" and "The agent disconnected" in CI - the agent stopped sending heartbeats mid-job,… "command exited with status 1"Fix Buildkite "The command exited with status 1" in CI - a step command returned a non-zero exit code, so Bui… environment hook secret errorFix a Buildkite environment hook secret failure in CI - the agent environment hook could not load a secret, s… "Failed to upload pipeline"Fix Buildkite "Failed to upload pipeline" in CI - the buildkite-agent pipeline upload step rejected the pipel… "Permission denied (publickey)"Fix Buildkite checkout "git@...: Permission denied (publickey)" in CI - the agent has no SSH key authorized t… "meta-data get ... key not found"Fix Buildkite "buildkite-agent meta-data get: key not found" in CI - a step read a meta-data key that no earl… "No command to run"Fix Buildkite "No command to run" in CI - a command step has no command defined, so the agent has nothing to… "permission denied ... docker.sock"Fix Buildkite "permission denied while trying to connect to the Docker daemon socket" in CI - the agent user… "pipeline.yml could not be parsed"Fix Buildkite "Failed to parse pipeline.yml" / "Invalid YAML" in CI - the pipeline definition has a syntax er… plugin "failed to fetch"Fix Buildkite "Failed to fetch plugin" in CI - the agent could not clone the plugin repository at the pinned… "could not connect to buildkite"Fix the Buildkite agent "could not connect" error in CI - the agent process cannot reach the Buildkite Agent… "Unable to authenticate: 401"Fix Buildkite "Unable to authenticate: 401" in CI - the agent registration token is missing, wrong, or revoke… docker plugin "failed to pull"Fix a Buildkite docker plugin image pull failure in CI - the agent could not pull the image named in the plug… docker-compose "service not found"Fix the Buildkite docker-compose plugin "service not found" error in CI - the `run` service named in the plug… dynamic pipeline "exit status 1"Fix a Buildkite dynamic pipeline "exit status 1" in CI - the script that generates and uploads steps failed b… "AccessDenied" (secrets bucket)Fix Buildkite S3 secrets "AccessDenied" in CI - the agent cannot read the secrets bucket the S3 Secrets hook… "502 Bad Gateway" / "503"Fix "502 Bad Gateway" and "503 Service Unavailable" from a package mirror or proxy in CI - a transient upstre… "apt: Failed to fetch ... Connection timed out"Fix "Failed to fetch ... Connection timed out" from apt-get update in CI - a transient mirror/network blip. A… "Clock skew detected"Fix "Clock skew detected" and "warning: Clock skew detected. Your build may be incomplete" in CI - the runner… "Connection reset by peer"Fix "Connection reset by peer" (ECONNRESET) in CI - an open connection was dropped mid-transfer by the remote… ECONNRESET in CIFix "Connection reset by peer" (ECONNRESET) in CI when a remote drops an in-flight connection. How to retry a… "Connection timed out"Fix "Connection timed out" (ETIMEDOUT) in CI - a runner could not establish or complete a connection in time.… ETIMEDOUT downloadingFix "Connection timed out" / ETIMEDOUT in CI while downloading deps, a transient network stall. How to retry… "Could not resolve host"Fix "Could not resolve host" in CI - DNS lookups failing on the runner due to transient resolver blips, missi… Could not resolve host (DNS)Fix "Could not resolve host" in CI, a transient DNS failure reaching a registry. How to retry it and why a DN… "curl: (35) SSL connect error"Fix "curl: (35) SSL connect error" in CI - the TCP connection opened but the TLS handshake failed or stalled.… "curl: (6) Could not resolve host"Fix "curl: (6) Could not resolve host" in CI - curl could not turn the hostname into an IP. A transient DNS b… "curl: (7) Failed to connect"Fix "curl: (7) Failed to connect to host port" in CI - the name resolved but the TCP connection was refused o… read tcp resetFix "read tcp ... connection reset by peer" in CI when a Go/Docker client loses an in-flight TCP read. How to… "Temporary failure in name resolution"Fix "Temporary failure in name resolution" (EAI_AGAIN) in CI - a transient DNS failure on the runner, distinc… Temporary name resolutionFix "Temporary failure in name resolution" (EAI_AGAIN) in CI, a transient DNS resolver failure. How to retry… "TLS handshake timeout"Fix "net/http: TLS handshake timeout" and stalled TLS negotiation in CI - a transient failure to complete the… TLS handshake timeoutFix "net/http: TLS handshake timeout" in CI when a secure connection stalls mid-handshake. How to retry and w… 429 rate limitFix HTTP 429 Too Many Requests in CI when a registry or API rate-limits the runner. How to back off and why a… 503/502 from registryFix 503 / 502 from a package registry in CI, a transient upstream error. How to retry and why a 5xx self-heal… "Hash Sum mismatch"Fix apt "Hash Sum mismatch" in CI - a package index downloaded inconsistently, usually from a mid-sync mirror… Proxy connection errorFix CI proxy errors - "Received HTTP code 407 from proxy after CONNECT", 403s, and proxy timeouts - from miss… Transient 5xx (502/503/504)Fix transient 5xx errors (502, 503, 504) from package mirrors and registries in CI - the textbook retry-recov… "(56) OpenSSL SSL_read"Fix curl "(56) OpenSSL SSL_read: ... errno 104" in CI - the TLS connection was torn down while reading, usual… "Could not resolve proxy"Fix curl "Could not resolve proxy: <name>" in CI - the proxy hostname in HTTP_PROXY/HTTPS_PROXY could not be… "Failed to connect ... port 443"Fix curl "Failed to connect to <host> port 443" in CI - the connection to the HTTPS port was refused or dropp… curl (6) resolve hostFix "curl: (6) Could not resolve host" in CI, a transient DNS failure during a download. How to retry and why… "unable to get local issuer certificate"Fix "unable to get local issuer certificate" in CI - the client received the leaf certificate but has no CA b… "TLS handshake timeout"Fix Docker "net/http: TLS handshake timeout" during a pull in CI - the TCP connection opened but the TLS nego… "received unexpected HTTP status: 503"Fix "received unexpected HTTP status: 503 Service Unavailable" in CI - the registry or a proxy in front of it… "Could not resolve host"Fix "Could not resolve host" in CI - the runner or container could not turn a hostname into an IP because DNS… "runner has not been able to update telemetry"Understand "The runner has not been able to update telemetry" in CI - a runner-to-control-plane connectivity… "Client.Timeout exceeded while awaiting headers"Fix Go "context deadline exceeded (Client.Timeout exceeded while awaiting headers)" in CI - the request conne… "lookup ...: no such host"Fix Go "dial tcp: lookup <host>: no such host" in CI - DNS returned NXDOMAIN, meaning the name definitively d… "x509: certificate signed by unknown authority"Fix "x509: certificate signed by unknown authority" in CI - a Go client or the Docker daemon does not trust t… NO_PROXY not honoredFix NO_PROXY being ignored in CI - traffic to an internal host still goes through the corporate proxy because… "self signed certificate in certificate chain"Fix "self signed certificate in certificate chain" in CI - a TLS-inspecting proxy presents a certificate sign… "tunneling socket could not be established"Fix Node "tunneling socket could not be established" in CI - the HTTPS_PROXY CONNECT tunnel to the proxy fail… "ETIMEDOUT"Fix npm "ETIMEDOUT" in CI - the request to the package registry timed out because the connection to the regis… Outbound blocked (proxy)Fix outbound requests blocked by a proxy or firewall in CI (connection refused, proxy 403). How to configure… "certificate verify failed: unable to get local issuer certificate"Fix Python "certificate verify failed: unable to get local issuer certificate" in CI - requests/urllib3 canno… ".teamcity" DSL compilation errorFix TeamCity ".teamcity" DSL compilation errors when settings are stored in VCS - the project fails to update… "Access denied. ... permission"Fix TeamCity "Access denied. You do not have enough permissions" in CI - the user or token running an action… "Build agent ... disconnected"Fix TeamCity "Build agent ... was disconnected" in CI - the agent lost its connection to the server mid-build… "repository not found"Fix TeamCity "Failed to collect changes ... repository not found" in CI - the VCS root URL points at a reposi… "Cannot connect to the Docker daemon"Fix TeamCity "Cannot connect to the Docker daemon" in CI - a Docker build step or the Docker wrapper cannot r… "Failed to collect changes: Authentication failed"Fix TeamCity "Failed to collect changes ... Authentication failed" in CI - the VCS root credentials cannot au… "Failed to load ... DSL" (settings.kts)Fix TeamCity "Failed to load DSL" / settings.kts compilation errors in CI - the versioned Kotlin DSL settings… "Failed to resolve artifact dependency"Fix TeamCity "Failed to resolve artifact dependency" in CI - the build could not download artifacts from a so… "Failed to start build" (VCS root)Fix TeamCity "Failed to start build" caused by a VCS root error in CI - checkout could not begin because the… "No enabled compatible agents"Fix TeamCity "No enabled compatible agents can run this build" in CI - no connected agent in the assigned poo… "Parameter ... is not specified"Fix TeamCity "Parameter ... is not specified" and unresolved %param% references in CI - a build parameter the… "agent ... unauthorized"Fix TeamCity "Agent is not authorized" / "unauthorized" in CI - a connected agent will not pick up builds unt… "Upgrading" agent stuckFix a TeamCity agent stuck "Upgrading" or failing to upgrade after a server update in CI - the agent cannot d… "command not found"Fix TeamCity "command not found" in a build step in CI - the tool the step calls is not installed or not on t… "snapshot dependency ... failed"Fix a TeamCity snapshot dependency failure in CI - a dependency build in the chain failed, so the dependent b… "certificate has expired"Fix "SSL certificate ... has expired" in CI - the server certificate (or an intermediate) is past its notAfte…

Timeouts & hangs

Step timeouts, deadlocks, and cancellation.

AutoscalingRunnerSet: no runners createdFix an ARC AutoscalingRunnerSet that creates no runners in CI - jobs queue but no ephemeral runner pods appea… "Cannot connect to the Docker daemon"Fix ARC "Cannot connect to the Docker daemon at unix:///var/run/docker.sock" in CI - the dind sidecar is not… "ephemeral runner ... failed to start"Fix ARC "ephemeral runner failed to start" in CI - the runner pod was created but its container could not lau… "failed more than ... times"Fix ARC "EphemeralRunner has failed more than 5 times" in CI - the controller gave up recreating a runner tha… "failed to get desired runner count"Fix ARC "failed to get desired runner count" in CI - the listener could not read how many runners the Actions… "permission denied ... docker.sock"Fix ARC "permission denied while trying to connect to the Docker daemon socket" in CI - the runner user canno… listener "CrashLoopBackOff"Fix an ARC listener pod in CrashLoopBackOff in CI - the RunnerScaleSet listener keeps restarting because it c… runner "Init:Error"Fix an ARC runner pod in Init:Error in CI - the init container that sets up Docker-in-Docker or externals fai… runner pod "Pending" / UnschedulableFix an ARC runner pod stuck in Pending in CI - the scheduler cannot place it because of insufficient CPU or m… "Waiting for a runner to pick up this job"Fix an ARC workflow stuck on "Waiting for a runner to pick up this job" in CI - the runs-on label does not ma… "No agents matching the following tags"Fix Buildkite "No agents matching the following tags" in CI - a step targets a queue or tag that no connected… BUILDKITE_PARALLEL_JOB missingFix Buildkite parallelism where BUILDKITE_PARALLEL_JOB is unset in CI - the step lacks a parallelism count, s… "Waiting for agent" never endsFix a Buildkite job stuck on "Waiting for agent" in CI - no agent with capacity and matching tags is availabl… "##[error]The operation was canceled" (timeout)Decode "##[error]The operation was canceled" that follows a timeout in CI - the job ran past its limit and wa… command timed outFix "command timed out after Nm" in CI when a wrapped command exceeds its own timeout. How to tell a hang fro… context deadline exceededFix "context deadline exceeded" in CI when a Go/Docker/kubectl client times out an operation. How to retry, r… "context deadline exceeded"Fix "context deadline exceeded" in CI - a Go (or client) operation ran past its context timeout, usually a sl… Job exceeded max timeFix "The job exceeded the maximum execution time and was terminated" in CI. How to split work, cache, raise t… "exceeded the maximum execution time of N minutes"Fix "The job has exceeded the maximum execution time of N minutes" in CI - the job ran past its configured ti… "wait-for-it: timeout occurred"Fix "wait-for-it.sh: timeout occurred after waiting N seconds" in CI - a readiness wait gave up because a ser… Deadlock (no progress)Diagnose a genuine deadlock in CI - threads or processes waiting on each other forever. Why this is a real bu… Deadlock then killedFix a CI step that produces no output, deadlocks, and is killed on timeout. How to find the deadlock, avoid s… Exit code 143 (SIGTERM)Fix exit code 143 in CI - the process got SIGTERM, often because a spot/preemptible runner was reclaimed mid-… Hung step / stuck processDiagnose a CI step that hangs with no output until the job times out - a process waiting on input, a stuck ne… timeout-minutes exceededFix a CI step cancelled after timeout-minutes elapsed. How to tell a hung step from genuinely slow work, rais… Test suite hangsFix a CI test suite that hangs and is killed on timeout. How to find the stuck test, close open handles, and… Healthcheck timed outFix a CI job that times out waiting for a service container or healthcheck to become ready. How to wait prope… Watchdog kill (no output)Fix CI watchdog kills - a job terminated for producing no output for too long, even though it was still worki… "The action has timed out"Fix "Error: The action has timed out" in CI - an action step ran past its time limit, often waiting on a slow… "exceeded the maximum execution time"Fix "The job running on runner ... has exceeded the maximum execution time" in CI - a job that ran past its t… "lost communication with the server"Fix "The runner has received a shutdown signal" / "lost communication with the server" in CI - a runner that… "The operation was canceled"Decode "The operation was canceled" in CI - timeouts, concurrency cancellation, lost runners, and manual canc… "The runner has received a shutdown signal"Understand "The runner has received a shutdown signal" in CI - the runner host was told to stop mid-job (pree… "No idle compatible agents"Fix TeamCity "There are no idle compatible agents which can run this build" in CI - compatible agents exist b… "Process exited with code 1"Fix TeamCity "Process exited with code 1" in CI - a command-line build step returned a non-zero exit code, so… "Tests failed: N"Fix TeamCity "Tests failed: N (M new)" in CI - the test reporting detected failing tests, which fails the bui… "an error message is logged by build runner"Fix a TeamCity build failed by the "an error message is logged by build runner" failure condition in CI - a m… "Database ... connection" failureFix a TeamCity server database connection failure in CI - the server cannot reach or authenticate to its exte…
Explore other topics