nsys profile: Timeline Profiling with Nsight Systems
nsys profile runs your program and records a system-wide timeline of CUDA API calls, kernel launches, memory transfers, and CPU activity into a report file.
Nsight Systems answers "where is the time going" across CPU and GPU. Its CLI, nsys, runs headless, so it fits in CI to catch throughput regressions.
What it does
nsys profile launches your application and traces selected subsystems (CUDA, NVTX, OS runtime, cuDNN, cuBLAS) into a .nsys-rep report. nsys stats then prints summary tables (kernel time, memory copy time) from that report without a GUI.
Common usage
# profile and write report1.nsys-rep
nsys profile -o report1 python train.py
# trace only CUDA and NVTX
nsys profile --trace=cuda,nvtx -o report python train.py
# print summary stats from an existing report
nsys stats report1.nsys-repOptions
| Flag | What it does |
|---|---|
| -o <name> | Output report file name |
| --trace=<list> | Subsystems to trace: cuda,nvtx,osrt,cudnn,cublas |
| --stats=true | Print summary stats after profiling |
| -d <sec> / --duration | Profile for a fixed number of seconds |
| --force-overwrite=true | Overwrite an existing report file |
In CI
Run nsys stats in the job and fail if a key kernel exceeds a time budget, catching performance regressions in a merge. Use --force-overwrite=true so a re-run does not error on an existing report, and archive the .nsys-rep as an artifact for local inspection.
Common errors in CI
"Failed to create export file" or a permission error means the output directory is not writable; write to /tmp. On locked-down kernels, "CPU sampling requires CAP_SYS_ADMIN" or "perf_event_paranoid" warnings appear; either lower kernel.perf_event_paranoid or restrict --trace to cuda so it does not need CPU sampling. A container without --privileged may block OS runtime tracing.