Python MemoryError in CI

Python raised MemoryError because an allocation exceeded available RAM. In CI this is common with data libraries (pandas, numpy) loading or copying large structures.

What this error means

A step ends with a traceback in MemoryError, often during a large read, concat, or array allocation. Using a larger runner or streaming the data clears a one-off spike with no logic change.

Traceback (most recent call last):
  File "etl.py", line 88, in <module>
    df = pd.concat(frames)
MemoryError

Common causes

Loading more data than fits in RAM

Reading a large dataset fully into memory, or copying it, can exceed the runner ceiling.

An unbounded accumulation in the code

Building a list/array that grows without limit eventually exhausts memory; this is a code fix, not a runner fix.

How to fix it

Reduce peak memory

Process data in chunks instead of all at once.

for chunk in pd.read_csv("big.csv", chunksize=100_000):
    process(chunk)

Right-size or rework

1. Move data-heavy jobs to a runner with more RAM for a genuine one-off spike.
2. If memory grows unboundedly, fix the accumulation in your code.
3. Lower worker/process parallelism in the step.

How to prevent it

• Stream or chunk large datasets in CI.
• Right-size memory for data pipelines.
• Profile peak memory for data-heavy tests.

Related guides