Spark "java.lang.OutOfMemoryError: Java heap space" in CI

A Spark JVM (the driver or an executor) ran out of heap and threw OutOfMemoryError. On a small CI runner this usually means an action pulled too much data into one JVM, or memory limits were left at defaults.

What this error means

A Spark job fails with "java.lang.OutOfMemoryError: Java heap space", frequently during a collect(), a wide shuffle, or a large toPandas() on a memory-constrained runner.

java.lang.OutOfMemoryError: Java heap space
  at org.apache.spark.util.collection.ExternalAppendOnlyMap...
  ... Job aborted due to stage failure

Common causes

An action pulls too much into one JVM

collect() or toPandas() brings the whole dataset into the driver heap, which a small runner cannot hold.

Default memory settings on a constrained runner

Driver/executor memory left at defaults is too small for the shuffle or cache the job performs in CI.

How to fix it

Avoid collecting the full dataset

1. Replace collect()/toPandas() on large frames with limit(), aggregation, or writing to storage.
2. Sample the data in CI tests instead of materializing everything.
3. Re-run with a bounded result set.

# bound the data instead of collecting all of it
df.limit(1000).collect()

Raise driver/executor memory for the job

Give the JVM more heap when the work genuinely needs it and the runner has the RAM.

spark = (SparkSession.builder
    .config("spark.driver.memory", "4g")
    .config("spark.executor.memory", "4g")
    .getOrCreate())

How to prevent it

• Use sampled or small fixtures for Spark tests in CI.
• Avoid full collect()/toPandas() on large frames.
• Size driver/executor memory to the runner and the workload.

Related guides