Python "OSError: [Errno 24] Too many open files" in CI

The process hit its open-file-descriptor limit. Either code is leaking file handles or sockets without closing them, or the runner’s ulimit -n is too low for legitimate concurrency.

What this error means

A long run, a test suite, or a high-concurrency task fails with OSError: [Errno 24] Too many open files. It often appears only after many iterations or under load, and may surface from sockets, temp files, or subprocess pipes.

Python traceback

OSError: [Errno 24] Too many open files: 'data/chunk_4821.bin'
# or from sockets
OSError: [Errno 24] Too many open files
  File ".../http/client.py", line 941, in send

Common causes

Leaked file or socket handles

Files, sockets, or subprocess pipes opened without a with block (or never closed) accumulate until the descriptor table is exhausted.

ulimit -n too low for the workload

The runner’s soft limit on open files (often 1024) is too small for genuinely high concurrency, so even leak-free code can exhaust it.

How to fix it

Close handles with context managers

Use with so files and sockets are closed deterministically, even on error.

Python

with open(path, "rb") as f:
    data = f.read()
# requests: use a session and close it, or 'with requests.Session() as s:'

Raise the descriptor limit

Terminal

ulimit -n 65535
python run.py
# inside Python (soft up to hard limit):
# import resource; resource.setrlimit(resource.RLIMIT_NOFILE, (65535, 65535))

Find the leak

Print len(os.listdir("/proc/self/fd")) periodically to confirm growth.
Audit code paths that open files/sockets without a with or explicit close().
Cap parallelism (pool size, concurrent connections) to a sane bound.

How to prevent it

Always open files/sockets with context managers.
Bound concurrency and connection-pool sizes.
Raise ulimit -n for genuinely high-fan-out workloads.