Python "MemoryError" loading a dataset in CI

Python asked the OS for more memory than the runner could provide while loading data into RAM. The allocation failed and the interpreter raised MemoryError - the dataset, or an intermediate copy of it, exceeds the runner's memory.

What this error means

A data-loading step fails with "MemoryError" (or the process is OOM-killed) when reading a large CSV/array fully into memory on a small runner.

python

Traceback (most recent call last):
  File "load.py", line 9, in <module>
    df = pd.read_csv("big.csv")
MemoryError: Unable to allocate 6.40 GiB for an array with shape (800000000,) and data type float64

Common causes

Loading the whole dataset into memory at once

Reading a large file fully (or building a big in-memory copy) exceeds the runner's RAM.

A runner with too little memory for the workload

CI runners are often smaller than developer machines, so data that fits locally OOMs in CI.

How to fix it

Stream or chunk the data

Process the dataset in chunks instead of loading it all, and use memory-efficient dtypes.

load.py

for chunk in pd.read_csv("big.csv", chunksize=100_000):
    process(chunk)

Run on a larger-memory runner

When the workload genuinely needs more RAM, run the job on a higher-memory runner class.

How to prevent it

Stream or chunk large inputs rather than loading them whole.
Use compact dtypes and drop unused columns early.
Size CI runners to the real memory footprint of the job.