nvidia-container-cli: Inspect Container GPU Access
nvidia-container-cli is the low-level utility the NVIDIA Container Toolkit uses to discover GPUs and mount driver libraries and device nodes into a container.
When --gpus all fails, nvidia-container-cli tells you whether the failure is the host driver, the toolkit, or the container config, before you touch Docker at all.
What it does
nvidia-container-cli queries NVML and the driver to report available GPUs (info) and to list the exact driver files and device nodes it would inject into a container (list). The container runtime calls it under the hood on docker run --gpus.
Common usage
# what GPUs and driver does the toolkit see?
nvidia-container-cli info
# which files would be mounted into the container?
nvidia-container-cli list
# check the toolkit version
nvidia-container-cli --versionOptions
| Command | What it does |
|---|---|
| info | Report NVML/driver version and detected GPUs |
| list | List driver libraries and device nodes to mount |
| --version | Print the container CLI version |
| configure | Low-level container setup (used by the runtime) |
In CI
Add nvidia-container-cli info as a health check on GPU runners. If it succeeds but docker run --gpus fails, the problem is the Docker runtime registration (fix with nvidia-ctk), not the driver. If info itself fails, the host driver is broken and no container will get a GPU.
Common errors in CI
"nvidia-container-cli: initialization error: nvml error: driver not loaded" means the host NVIDIA kernel module is not loaded (reinstall or rebuild the driver, often after a kernel update). "nvidia-container-cli: mount error: file creation failed" points at a read-only or misconfigured container rootfs. A version mismatch between the toolkit and the driver also surfaces here first.