How to Run PySpark Tests in CI

PySpark runs on the JVM, so the number-one CI failure is a missing or mismatched Java - not the Python package.

The pyspark wheel bundles Spark, but it needs a compatible JDK at runtime. In CI you install Java, set JAVA_HOME, and run Spark in local mode for tests.

Why it fails in CI

No JDK or JAVA_HOME unset → "JAVA_HOME is not set" / gateway not started.
A Java version Spark does not support yet → JVM startup errors.
The heavy Spark/JVM startup is slow and occasionally flaky on small runners.

Install and run it reliably

Install a Spark-supported JDK, export JAVA_HOME, install pyspark, and run a local Spark session in tests.

Terminal

# install a supported JDK (Spark supports e.g. Java 17 on recent versions)
apt-get update && apt-get install -y openjdk-17-jre-headless
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64

pip install pyspark
python -c "from pyspark.sql import SparkSession; SparkSession.builder.master('local[2]').getOrCreate().stop()"

Cache & speed

Cache ~/.cache/pip and the Ivy/jars directory (~/.ivy2) if you pull Spark packages. JVM startup is heavy; larger managed runners speed it up and transient startup flakes auto-retry.

.github/workflows/ci.yml

- uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      ~/.ivy2
    key: pyspark-${{ runner.os }}-${{ hashFiles('**/requirements*.txt') }}

Common errors

JAVA_HOME is not set → install a JDK and export JAVA_HOME.
Java gateway process exited before sending its port number → wrong/absent Java; pin a supported JDK.
UnsupportedClassVersionError → JDK too new for the Spark version; downgrade Java.

Key takeaways

PySpark needs a Spark-compatible JDK with JAVA_HOME set.
Run Spark in local[*] mode for CI tests.
Pin the JDK to the Spark version to avoid JVM startup errors.