Self-Healing CI: Recovering When a GPU Is Briefly Unavailable
A GPU job that reports "no device" was usually scheduled before the GPU was ready or onto capacity that briefly lacked one -- a retry on proper GPU capacity clears it.
The problem
A GPU workload fails with no CUDA-capable device is detected or a device-initialization error. The code and drivers are fine; the GPU was briefly unavailable -- the device was not initialized yet, or the job landed on capacity without an accessible GPU. A human re-runs on a GPU runner and it passes unchanged.
RuntimeError: No CUDA-capable device is detected
# or
nvidia-smi: couldn't communicate with the NVIDIA driverWhy it happens
GPU devices must be initialized and exposed to the job before the workload starts. A job that begins before the driver/device is fully ready -- or that is scheduled onto capacity that does not currently have an accessible GPU -- sees no device even though correct GPU capacity is available moments later.
It is a device-readiness or scheduling race, not a code bug: the same workload succeeds once it runs against an initialized GPU, with no change to the job.
The manual fix
Manual handling for a briefly-unavailable GPU:
- Re-run the job so it lands on ready GPU capacity.
- Add a readiness check (e.g.
nvidia-smi) before the workload and wait for the device. - Ensure the job targets a GPU runner class and that drivers are initialized at boot.
for i in $(seq 1 20); do nvidia-smi >/dev/null 2>&1 && break; sleep 2; done
nvidia-smiHow this gets automated
A "no device" failure at job start has a clear device-readiness signature, and the safe response is to wait for the GPU and retry, or retry onto proper GPU capacity. A self-healing CI pipeline detects the device-unavailable condition, retries against a ready GPU, and only escalates if no GPU is genuinely available, which is the real signal of a capacity or configuration problem rather than a startup race.