Python multiprocessing "can't pickle" error in CI

The spawn/forkserver start method serializes the callable and its arguments with pickle to hand them to a worker process. An object that cannot be pickled - a lambda, a local function, a lock, an open file - makes the dispatch fail.

What this error means

A run using multiprocessing or a ProcessPool fails with "TypeError: cannot pickle 'X' object" or "PicklingError: Can't pickle <function <lambda>>". It often only appears on spawn-based platforms.

_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f...>:
attribute lookup <lambda> on __main__ failed

Common causes

The target or argument is unpicklable

Lambdas, locally-defined functions, open sockets/files, locks, and DB connections cannot be pickled, so they cannot cross the process boundary.

Spawn start method requires picklable everything

On spawn (default on macOS and Windows, used by some CI), the child re-imports the module, so anything passed must round-trip through pickle.

How to fix it

Pass picklable, top-level callables

1. Replace lambdas and nested functions with module-level functions.
2. Pass plain data (ids, paths) and reconstruct connections inside the worker.
3. Move unpicklable resources out of the arguments.

def work(item_id):  # top-level, picklable
    conn = connect()  # built inside the worker
    return process(conn, item_id)

Initialize resources per worker

Use a pool initializer to create non-picklable resources in each worker instead of passing them in.

from multiprocessing import Pool
with Pool(initializer=setup_worker) as pool:
    pool.map(work, item_ids)

How to prevent it

• Pass only picklable data to worker processes.
• Define worker targets at module top level, not inline.
• Construct connections, locks, and file handles inside the worker.

Related guides