nvidia-smi dmon: Stream GPU Metrics
nvidia-smi dmon prints a scrolling, one-line-per-sample stream of GPU metrics that is easy to log or pipe.
Unlike the full table, dmon is built for time series: it emits fixed columns per sample, so you can capture utilization over the life of a job.
What it does
nvidia-smi dmon samples selected metric groups at a fixed interval and prints one row per sample per GPU: SM utilization, memory utilization, power, temperature, clocks, and more, depending on the groups you select with -s.
Common usage
# default metric set, one sample per second
nvidia-smi dmon
# power, utilization, memory; 30 samples then stop
nvidia-smi dmon -s pucm -c 30
# background it and log to a file during a training step
nvidia-smi dmon -s um -d 5 -o DT > gpu.log &Options
| Flag | What it does |
|---|---|
| -s <groups> | Metric groups: p=power/temp, u=utilization, c=clocks, m=memory |
| -c <count> | Sample N times then exit |
| -d <sec> | Delay between samples (default 1s) |
| -i <id> | Limit to specific GPU(s) |
| -o DT | Prefix each line with date and time |
In CI
Start dmon in the background with -o DT before a training step and kill it after, so you get a timestamped utilization log attached to the job artifacts. Use -c to bound the sample count so a leftover process cannot run forever.
Common errors in CI
dmon writing nothing usually means it was backgrounded without redirecting stdout, or the job exited before the first -d interval elapsed. "Unable to determine the device handle for GPU" mid-stream indicates the GPU fell off the bus or the driver crashed; the training process will fail too.