dcgmi: Data Center GPU Manager Diagnostics
dcgmi is the CLI for NVIDIA Data Center GPU Manager (DCGM), used to run health diagnostics and pull detailed telemetry on datacenter GPUs.
Beyond nvidia-smi, DCGM runs active diagnostics that catch failing GPUs before a long job wastes time on them. dcgmi diag is the go-to pre-flight check.
What it does
dcgmi talks to the DCGM host engine (nv-hostengine) to enumerate GPUs, stream telemetry, and run tiered diagnostic suites (dcgmi diag -r), which execute real workloads to validate memory, PCIe, and compute health.
Common usage
# list GPUs DCGM manages
dcgmi discovery -l
# quick health diagnostic (level 1)
dcgmi diag -r 1
# deeper, longer diagnostic (level 3)
dcgmi diag -r 3Options
| Command | What it does |
|---|---|
| discovery -l | List the GPUs DCGM sees |
| diag -r 1 | Quick sanity diagnostic |
| diag -r 2 | Medium diagnostic (minutes) |
| diag -r 3 | Long, thorough diagnostic |
| dmon | Stream DCGM field telemetry |
| nv-hostengine | The daemon dcgmi connects to |
In CI
Run dcgmi diag -r 1 as a pre-flight health check on self-hosted GPU runners; if it flags an issue, take the runner out of rotation instead of scheduling training on a failing card. Higher -r levels take longer, so save -r 3 for periodic maintenance jobs.
Common errors in CI
"Error: Unable to connect to host engine. Host engine connection invalid/disconnected." means nv-hostengine is not running; start it (or run dcgmi with an embedded engine). A diag reporting "Fail" on a subtest indicates a genuinely unhealthy GPU. Inside a container, DCGM needs the right GPU capabilities and often a privileged or specially configured runtime to run diagnostics.