trtexec --onnx: Build and Benchmark TensorRT

trtexec builds a TensorRT engine from an ONNX model and measures its inference latency and throughput, all from the command line.

trtexec is the fastest way to validate that a model converts to TensorRT and to benchmark it. In CI it catches conversion regressions and latency creep.

What it does

trtexec parses an ONNX model, builds an optimized TensorRT engine for the current GPU (optionally in FP16/INT8), and runs timed inference to report latency and throughput. --saveEngine caches the built engine for reuse.

Common usage

# build from ONNX and benchmark
trtexec --onnx=model.onnx
# FP16, save the engine, set workspace
trtexec --onnx=model.onnx --fp16 \
  --saveEngine=model.plan --memPoolSize=workspace:4096
# handle dynamic shapes
trtexec --onnx=model.onnx \
  --minShapes=input:1x3x224x224 \
  --optShapes=input:8x3x224x224 \
  --maxShapes=input:16x3x224x224

Options

In CI

Build the engine once with --saveEngine and reuse it with --loadEngine to keep pipelines fast, since building is slow. An engine is specific to the GPU architecture and TensorRT version it was built on, so rebuild when either changes. Gate merges on the reported latency staying under budget.

Common errors in CI

"[E] [TRT] ModelImporter.cpp ... No importer registered for op: <Op>" means an ONNX operator TensorRT does not support; simplify or replace the op. "Error Code 4: Internal Error (... exceeds memory pool)" means --memPoolSize is too small; raise the workspace. "FP16 is not supported on this platform" means the GPU lacks fast FP16; drop --fp16. "The engine plan file is not compatible with this version of TensorRT" means a --loadEngine built on a different TensorRT/GPU; rebuild it.

Related guides