PyTorch "CUDA error: device-side assert triggered" in CI

A CUDA kernel tripped an internal assertion and aborted the context. The most frequent cause is an index out of bounds, such as a class label outside the valid range for CrossEntropyLoss or an index beyond an embedding table.

What this error means

A step fails with "RuntimeError: CUDA error: device-side assert triggered". Because CUDA is asynchronous, the reported stack often points at a later call than the real kernel.

/opt/conda/.../IndexKernel.cu:91: operator(): block: [0,0,0], thread: [32,0,0]
Assertion `srcIndex < srcSelectDimSize` failed.
RuntimeError: CUDA error: device-side assert triggered

Common causes

A label or index out of valid range

A target class index >= num_classes for a loss, or an index past the end of an embedding/lookup table, trips a bounds assert inside the kernel.

Async error reported at the wrong call site

CUDA queues work asynchronously, so the traceback can blame an unrelated later op, hiding which kernel actually asserted.

How to fix it

Run synchronously to find the real kernel

1. Set CUDA_LAUNCH_BLOCKING=1 so the error surfaces at the offending op.
2. Optionally reproduce on CPU, where the bounds error gives an exact index.
3. Inspect the labels/indices feeding the named op for out-of-range values.

CUDA_LAUNCH_BLOCKING=1 python train.py

Clamp or validate indices and labels

Ensure target classes are in [0, num_classes) and embedding indices are within the table size before the forward pass.

assert targets.max().item() < num_classes
assert targets.min().item() >= 0

How to prevent it

• Validate label ranges against num_classes before training.
• Keep CUDA_LAUNCH_BLOCKING=1 in CI for clearer GPU tracebacks.
• Add a CPU smoke test that catches index errors deterministically.

Related guides