ncu: Kernel Profiling with Nsight Compute
ncu profiles individual CUDA kernels at the hardware-counter level, reporting occupancy, memory throughput, and instruction mix.
Where nsys shows the timeline, Nsight Compute (ncu) zooms into a single kernel and explains why it is slow. It needs counter permissions, a frequent CI snag.
What it does
ncu replays selected kernels while collecting GPU performance counters, then reports metrics like achieved occupancy, memory bandwidth, and warp stalls. Because it can serialize and replay kernels, profiling adds significant overhead.
Common usage
# full detail for every kernel (slow)
ncu -o report python infer.py
# profile only kernels matching a name, skip the rest
ncu -k myKernel -c 10 -o report ./app
# a lighter metric set
ncu --set basic -o report ./appOptions
| Flag | What it does |
|---|---|
| -o <name> | Write a .ncu-rep report |
| -k <regex> | Only profile kernels whose name matches |
| -c <count> | Profile at most N kernel launches |
| -s <count> | Skip the first N launches |
| --set <basic|full> | Which metric set to collect |
| --target-processes all | Profile child processes too |
In CI
Scope with -k and -c so a run profiles a handful of launches instead of every kernel, which otherwise makes jobs run for very long. To collect counters, the driver must permit non-admin performance counters; set that up on the runner or run ncu as root.
Common errors in CI
"ERR_NVGPUCTRPERM: The user does not have permission to access NVIDIA GPU Performance Counters on the target device." is the classic one: run as root or set the driver module parameter NVreg_RestrictProfilingToAdminUsers=0. "No kernels were profiled" means the -k regex matched nothing or -s skipped them all. In a container, counter access also needs the right --gpus capabilities.