Python multiprocessing "can't pickle" error in CI
The spawn/forkserver start method serializes the callable and its arguments with pickle to hand them to a worker process. An object that cannot be pickled - a lambda, a local function, a lock, an open file - makes the dispatch fail.
What this error means
A run using multiprocessing or a ProcessPool fails with "TypeError: cannot pickle 'X' object" or "PicklingError: Can't pickle <function <lambda>>". It often only appears on spawn-based platforms.
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f...>:
attribute lookup <lambda> on __main__ failedDiagnose it: is it the build backend or a missing system library?
Python packaging failures in CI split into build-backend configuration problems and missing system headers. The traceback usually points at the backend even when the real cause is an absent -dev package.
python -m pip install --upgrade pip build
python -m build --wheel 2>&1 | tail -40
# a compiler error naming a .h file is a system dependency, not a Python one
# e.g. "Python.h: No such file" -> python3-dev
# "openssl/ssl.h" -> libssl-devCommon causes
The target or argument is unpicklable
Lambdas, locally-defined functions, open sockets/files, locks, and DB connections cannot be pickled, so they cannot cross the process boundary.
Spawn start method requires picklable everything
On spawn (default on macOS and Windows, used by some CI), the child re-imports the module, so anything passed must round-trip through pickle.
How to fix it
Pass picklable, top-level callables
- Replace lambdas and nested functions with module-level functions.
- Pass plain data (ids, paths) and reconstruct connections inside the worker.
- Move unpicklable resources out of the arguments.
def work(item_id): # top-level, picklable
conn = connect() # built inside the worker
return process(conn, item_id)Initialize resources per worker
Use a pool initializer to create non-picklable resources in each worker instead of passing them in.
from multiprocessing import Pool
with Pool(initializer=setup_worker) as pool:
pool.map(work, item_ids)How to prevent it
- Pass only picklable data to worker processes.
- Define worker targets at module top level, not inline.
- Construct connections, locks, and file handles inside the worker.