# Self-Healing CI: Recovering When a GPU Is Briefly Unavailable

> A job that fails because a GPU was not available yet hit a device-readiness race, not a code bug. See the manual fix and how self-healing CI retries on a GPU runner.

Source: https://latchkey.dev/learn/self-healing-ci/self-healing-gpu-briefly-unavailable  
Updated: 2026-06-26

A GPU job that reports "no device" was usually scheduled before the GPU was ready or onto capacity that briefly lacked one -- a retry on proper GPU capacity clears it.

## What makes a failure safely retryable

Automatic retry is only correct for failures that are genuinely transient. Retrying a deterministic failure wastes minutes and hides a real defect, so the classification matters more than the retry mechanism.

- Safe to retry: network timeouts, registry 5xx, transient DNS failures, a service container that was not ready, a spot instance reclaimed mid-run.
- Not safe to retry: assertion failures, compile errors, lint violations, anything that fails identically on every attempt.
- Ambiguous, and worth investigating rather than retrying: out-of-memory kills, disk exhaustion, and flaky tests. These repeat under load and a retry only hides the trend.
- Always record that a retry happened. A pipeline that silently retries is a pipeline whose real failure rate you do not know.

## FAQ

### What causes Self-Healing CI: recovering when a GPU is briefly unavailable?

A GPU workload fails with no CUDA-capable device is detected or a device-initialization error. The code and drivers are fine; the GPU was briefly unavailable -- the device was not initialized yet, or the job landed on capacity without an accessible GPU. A human re-runs on a GPU runner and it passes unchanged.

### How do I fix Self-Healing CI: recovering when a GPU is briefly unavailable manually?

[object Object]

### Can Self-Healing CI: recovering when a GPU is briefly unavailable be fixed automatically?

A "no device" failure at job start has a clear device-readiness signature, and the safe response is to wait for the GPU and retry, or retry onto proper GPU capacity. A self-healing CI pipeline detects the device-unavailable condition, retries against a ready GPU, and only escalates if no GPU is genuinely available, which is the real

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt