PySpark "py4j.protocol.Py4JJavaError" in CI

Py4JJavaError is the Python-side wrapper for any exception thrown inside the Spark JVM. The Python traceback is generic; the real cause is the Java exception and message printed underneath it.

What this error means

A PySpark action (collect, count, write) fails with "py4j.protocol.Py4JJavaError: An error occurred while calling o123.collect" followed by a JVM stack trace and the underlying Java exception.

py4j.protocol.Py4JJavaError: An error occurred while calling o95.collect.
: org.apache.spark.SparkException: Job aborted due to stage failure:
  Task 0 in stage 3.0 failed 1 times, most recent failure:
  ... java.lang.NullPointerException

Common causes

A JVM exception during execution

The Java stack trace under the Py4JJavaError holds the real cause (a NullPointerException, a parse error, a type mismatch); the Python layer only relays it.

A data or schema problem in the job

Bad input, a divide-by-zero in a UDF, or a malformed file makes a task throw on the JVM side and abort the stage.

How to fix it

Read the Java exception, not the Python wrapper

1. Scroll past "Py4JJavaError" to the first : org.apache.spark... and the root Java exception.
2. Fix the real cause (the data, the UDF, or the schema) that the JVM reported.
3. Re-run the failing action to confirm.

Reproduce with full logs

Run the action with Spark logs at INFO and a small input so the JVM exception is easy to read.

spark.sparkContext.setLogLevel("INFO")

How to prevent it

• Validate input schemas before actions that materialize data.
• Guard UDFs against nulls and bad values.
• Keep Spark logs readable so the JVM cause is visible in CI output.

Related guides