How to Find a Memory Leak in CI

A job that OOM-kills only in CI usually has a leak that local runs never push hard enough to expose. Capturing memory over the run pins it down.

CI OOM kills are often a slow leak that accumulates across a long test run. The approach is to make memory visible over time, then bisect to the test or worker that grows without bound.

1. Watch memory across the run

Sample RSS during the job so you can see whether memory climbs steadily (a leak) or spikes once (a single heavy step).

- run: |
    ( while true; do
        date +%s; grep VmRSS /proc/$$/status; sleep 5;
      done ) & node --max-old-space-size=2048 ./run-tests.js

2. Take a heap snapshot at the peak

Capture a heap snapshot to see what is retained. Run a smaller batch so the snapshot is tractable.

- run: node --heapsnapshot-signal=SIGUSR2 ./run-tests.js &
- run: sleep 60 && kill -USR2 $!

3. Bisect to the leaking test

Run test files in isolation and compare end-of-run memory. A file whose memory does not return to baseline after teardown is the leak source. Common culprits: un-torn-down servers, growing module-level caches, and event listeners never removed.

4. Right-size RAM so a leak is loud, not silent

On an undersized hosted runner a leak hides until it OOM-kills randomly. A right-sized Latchkey runner gives the headroom to capture the climb cleanly, and self-heal retries the rare genuine infra OOM on a fresh machine.

Key takeaways

• Sample RSS over the run to distinguish a steady leak from a one-time spike.
• Take a heap snapshot at the peak to see what is retained.
• Bisect test files; the one that never returns to baseline is the leak.

Related guides