nvidia-smi --gpu-reset: Reset a Wedged GPU
nvidia-smi --gpu-reset reinitializes a single GPU to clear a wedged or errored state without rebooting the host.
When a job crashes and leaves a GPU with stuck memory or an Xid error, a targeted reset can recover it, but only if nothing else is using the card.
What it does
nvidia-smi --gpu-reset (or -r) performs a secondary bus reset on the specified GPU, clearing its state, memory, and ECC counters. It requires that no processes hold the GPU and typically needs root.
Common usage
# reset a single GPU by index (must be idle)
sudo nvidia-smi --gpu-reset -i 0
# confirm nothing is using it first
nvidia-smi -i 0
# short form
sudo nvidia-smi -r -i 0Options
| Flag | What it does |
|---|---|
| --gpu-reset / -r | Reset the targeted GPU |
| -i <id> | GPU index or UUID to reset (required in practice) |
| -q | Inspect ECC/Xid state before deciding to reset |
In CI
A cleanup step can reset a GPU between jobs on a self-hosted runner so a leaked allocation from a crashed test does not poison the next one. Kill or wait for all GPU processes first, since the reset refuses while the device is in use. Latchkey-style self-healing runners automate exactly this kind of recovery so a wedged GPU does not silently fail the next build.
Common errors in CI
"The requested operation is not available because a process is still using the GPU" or "GPU is currently in use by another process" means a training/inference process still holds it; kill it and retry. "Insufficient Permissions" means you need root. "Reset is not supported" appears on some consumer GPUs and inside VMs/containers where the reset ioctl is blocked.