pandas "MemoryError" on a large DataFrame in CI

pandas (via NumPy) raises MemoryError or "Unable to allocate ... for an array" when an operation needs more contiguous RAM than the runner has. CI runners are memory-constrained, so loading or pivoting a large file is a common trigger.

What this error means

A read or transform fails with "MemoryError" or "numpy.core._exceptions._ArrayMemoryError: Unable to allocate X GiB for an array with shape ...", usually on read_csv, a merge, or a pivot.

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 3.73 GiB
for an array with shape (500000000,) and data type float64

Common causes

Reading the whole file into memory

A full read_csv of a large file allocates the entire frame at once, exceeding a small runner.

A wide operation that explodes memory

A merge with duplicate keys or a pivot can multiply row count and blow past available RAM.

How to fix it

Read in chunks and downcast dtypes

1. Process the file in chunks instead of one allocation.
2. Read only needed columns and downcast numeric dtypes.
3. Aggregate incrementally so the full frame never materializes.

for chunk in pd.read_csv("big.csv", chunksize=100_000, usecols=cols):
    process(chunk)

Use a smaller fixture or a streaming engine in CI

Test on a sampled file, or use a streaming/out-of-core engine when the full dataset is essential.

df = pd.read_csv("big.csv", nrows=50_000)

How to prevent it

• Use sampled fixtures for pandas tests in CI.
• Read only needed columns and downcast dtypes.
• Process large files in chunks rather than one allocation.

Related guides