nvidia-smi: Check the GPU and Driver in CI
nvidia-smi prints the GPUs the NVIDIA driver can see, their driver and CUDA versions, memory use, and running processes.
The first line of any GPU pipeline is nvidia-smi. If it prints a table the driver works; if it errors, nothing downstream will run.
What it does
nvidia-smi (NVIDIA System Management Interface) queries the NVIDIA kernel driver via NVML and prints a table of each GPU: name, driver version, the maximum CUDA version the driver supports, temperature, power, memory used/total, and utilization, plus a list of compute processes.
Common usage
nvidia-smi
# refresh every 1s (like top)
nvidia-smi -l 1
# just the driver and CUDA version line
nvidia-smi --query-gpu=driver_version,name --format=csvOptions
| Flag | What it does |
|---|---|
| -L | List each GPU by index, name, and UUID |
| -q | Full per-GPU detail dump (ECC, clocks, power) |
| -l <sec> | Loop, refreshing every N seconds |
| -i <id> | Restrict to a specific GPU index or UUID |
| --query-gpu=... --format=csv | Scriptable CSV of named fields |
| dmon | Scrolling one-line-per-sample device monitor |
In CI
Run nvidia-smi as the very first step of any GPU job and fail fast if it errors. The "CUDA Version" it shows is the highest CUDA the driver supports, not the toolkit installed; do not confuse it with nvcc --version.
Common errors in CI
"nvidia-smi: command not found" means the NVIDIA driver/utilities are not installed on the runner (or the container was started without the GPU). "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running." means the kernel module is not loaded or mismatched, common after a kernel update without a matching driver rebuild. In a container this usually means it was launched without --gpus all or the NVIDIA Container Toolkit.