How to Unit Test PySpark Jobs With Local Spark in CI

A local[*] SparkSession runs Spark in-process, so you can test DataFrame logic without any cluster.

Create a session-scoped SparkSession fixture, build small DataFrames, run your transformation, and assert on collected rows.

Steps

• Install pyspark and a JDK in the job.
• Provide a local[*] SparkSession pytest fixture.
• Assert on the output of your transformation.

Test

import pytest
from pyspark.sql import SparkSession
from my_jobs.transform import add_total

@pytest.fixture(scope="session")
def spark():
    return SparkSession.builder.master("local[*]").appName("ci").getOrCreate()

def test_add_total(spark):
    df = spark.createDataFrame([(1, 2), (3, 4)], ["a", "b"])
    out = add_total(df).collect()
    assert out[0]["total"] == 3
    assert out[1]["total"] == 7

Workflow

steps:
  - uses: actions/checkout@v4
  - uses: actions/setup-java@v4
    with:
      distribution: temurin
      java-version: '17'
  - run: pip install pyspark==3.5.1 pytest
  - run: pytest tests -q

Gotchas

• PySpark needs a JDK; install Java before running tests.
• Reuse one session-scoped SparkSession so JVM startup is paid once.

Related guides