Python multiprocessing Hangs or Crashes on fork vs spawn in CI

Python’s multiprocessing start method matters in CI. fork after threads have started can deadlock; spawn re-imports the module and requires picklable targets. A mismatch between the method your code assumes and the platform default causes hangs or errors.

What this error means

A multiprocessing job hangs forever, crashes with RuntimeError: context has already been set, or fails to pickle the worker target - and behaves differently on Linux (fork default historically) vs macOS/newer Python (spawn default).

RuntimeError: context has already been set
# or, under spawn:
AttributeError: Can't pickle local object 'main.<locals>.worker'
# or the job simply hangs after "Starting worker pool"

Common causes

fork after threads/locks are held

Forking a process that already started threads (or holds a lock) can leave the child in an inconsistent state and deadlock. Many libraries (logging, gRPC, CUDA) are unsafe to fork after init.

spawn re-imports and needs picklable targets

Under spawn (macOS default, and the direction CPython is moving), the child re-imports the entry module and unpickles the target. A lambda/closure target or top-level side effects then break.

How to fix it

Set the start method explicitly

Choose a method deliberately instead of relying on the platform default, and guard the entry point.

import multiprocessing as mp

def worker(x):   # top-level, picklable
    return x * 2

if __name__ == "__main__":          # required for spawn
    mp.set_start_method("spawn")    # consistent across platforms
    with mp.Pool(4) as pool:
        print(pool.map(worker, range(8)))

Make targets importable and avoid fork-after-init

1. Define worker functions at module top level so spawn can pickle them.
2. Create the pool before starting threads or initializing fork-unsafe libraries.
3. Wrap the launch in if __name__ == "__main__": so re-import does not recurse.

How to prevent it

• Call set_start_method (or use get_context) explicitly for cross-platform CI.
• Keep worker targets at module scope and picklable.
• Guard process launches behind if __name__ == "__main__":.

Related guides