Self-Healing CI: Recovering "Too Many Open Files" (ulimit)
A "too many open files" failure means the job hit its file-descriptor ceiling - raising the limit or closing handles fixes it, no code change required.
The problem
A step fails with EMFILE: too many open files or Too many open files while opening files, sockets, or watchers. The code is fine; the process exceeded the file-descriptor (ulimit -n) ceiling. A human raises the limit or reduces concurrency and the step passes unchanged.
Error: EMFILE: too many open files, open '...'
# or
accept: too many open filesWhy it happens
Every open file, socket, and file watcher consumes a file descriptor, and the OS caps how many a process may hold. Tools that watch large trees or open many connections in parallel can exceed a low default ulimit -n even though no resource is genuinely scarce.
CI runners often ship with a conservative default descriptor limit, so a step that works locally on a higher limit fails on the runner - a mechanical ceiling, not a bug in your build.
The manual fix
The manual fix is to raise the descriptor limit or reduce open handles, then retry:
- Raise the soft file-descriptor limit (e.g.
ulimit -n) for the step. - Reduce concurrency so fewer files/sockets are open simultaneously.
- Ensure tooling closes handles and uses efficient file watching.
ulimit -n 65535
# then run the stepHow this gets automated
An EMFILE has a clear descriptor-exhaustion signature, and the safe response is to provide headroom and retry. A self-healing CI pipeline detects the open-files limit condition, retries the step with an adequate descriptor limit, and only escalates if the exhaustion recurs - which would indicate a genuine descriptor leak rather than a low default.