PySpark "Python worker failed to connect back" in CI

For Python UDFs and RDD code, Spark launches a Python worker process and the JVM connects to it over a local socket. When the worker cannot start or the JVM cannot reach it, Spark raises "Python worker failed to connect back".

What this error means

A Spark job fails with "org.apache.spark.SparkException: Python worker failed to connect back", sometimes alongside a PySpark/Python version mismatch or a missing PYSPARK_PYTHON.

org.apache.spark.SparkException: Python worker failed to connect back.
  at org.apache.spark.api.python.PythonWorkerFactory...
Caused by: java.net.SocketTimeoutException: Accept timed out

Common causes

PYSPARK_PYTHON points at a missing or wrong interpreter

Spark launches the worker with PYSPARK_PYTHON; if that interpreter is absent or differs from the driver, the worker cannot start or handshake.

Local networking blocks the worker socket

Loopback restrictions or a firewall on the runner stop the JVM from connecting back to the worker process.

How to fix it

Pin the Python interpreter for driver and workers

1. Set PYSPARK_PYTHON (and PYSPARK_DRIVER_PYTHON) to the same interpreter that has PySpark installed.
2. Confirm that interpreter exists on the runner.
3. Re-run so driver and worker use one Python.

export PYSPARK_PYTHON=$(which python)
export PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON

Ensure loopback is reachable

Bind Spark to localhost so the worker socket is reachable on a constrained runner.

spark = (SparkSession.builder
    .config("spark.driver.bindAddress", "127.0.0.1")
    .config("spark.driver.host", "127.0.0.1")
    .getOrCreate())

How to prevent it

• Set PYSPARK_PYTHON to the interpreter that has PySpark.
• Keep driver and worker Python versions identical.
• Bind Spark to loopback on single-node CI runners.

Related guides