ML job "Killed" (host OOM) during training in CI

The process died with a bare "Killed" and exit code 137. This is the Linux OOM killer reclaiming host RAM, not a GPU memory error: data loading, a large dataset in memory, or too many worker processes exceeded the runner's RAM.

What this error means

A training or data-prep step ends with only "Killed" (no Python traceback) and exit code 137, often while loading data or spawning DataLoader workers.

python

Loading dataset...
Killed
Error: Process completed with exit code 137.

Common causes

Host RAM exceeded by data or workers

Loading a dataset fully into memory, large in-memory caches, or many DataLoader workers each copying data can overshoot the runner's RAM, triggering the OOM killer.

A memory leak across epochs

Holding references (growing lists, un-freed batches) increases RSS each step until the kernel kills the process.

How to fix it

Reduce host memory pressure

Stream or memory-map the dataset instead of loading it all into RAM.
Lower DataLoader num_workers and prefetch_factor so fewer copies exist.
Free large objects between phases and avoid accumulating Python lists.

train.py

loader = DataLoader(ds, batch_size=32, num_workers=2, pin_memory=False)

Confirm it was the OOM killer

Check the kernel log to verify the OOM killer fired and which process it chose.

Terminal

dmesg | grep -i "killed process"

How to prevent it

Size the runner RAM to the dataset and worker count.
Stream or shard large datasets rather than loading them whole.
Watch RSS across epochs to catch leaks before CI OOMs.