PyTorch "CUDA out of memory. Tried to allocate" in CI

The GPU allocator could not reserve a block large enough for the next tensor. The message reports how much it tried to allocate versus how much was free, which tells you whether to shrink the workload or free cached memory.

What this error means

Training or inference aborts with "torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate ... GiB" and a breakdown of reserved versus allocated memory.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0;
15.78 GiB total capacity; 13.90 GiB already allocated; 1.20 GiB free; 14.10 GiB
reserved in total by PyTorch)

Common causes

Batch size or model too large for the CI GPU

The runner GPU has less memory than the development machine, so a batch size or sequence length that fits locally overflows in CI.

Accumulated tensors or autograd graph not freed

Holding references to per-step tensors, or summing losses without .item(), keeps the autograd graph alive and grows memory until it overflows.

How to fix it

Shrink the per-step memory footprint

1. Lower the batch size (or use gradient accumulation) for the CI run.
2. Detach metrics with .item() or .detach() so the graph is freed.
3. Wrap evaluation in torch.no_grad() to avoid storing activations.

with torch.no_grad():
    out = model(batch)   # no autograd graph retained in eval

Free cached memory between phases

Release the allocator cache between stages so fragmentation does not block a large allocation.

import torch, gc
del intermediate
gc.collect()
torch.cuda.empty_cache()

How to prevent it

• Size CI batch and sequence length to the runner GPU, not the dev box.
• Detach logged metrics so the autograd graph does not accumulate.
• Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation.

Related guides