Self-Healing CI: Recovering Metaspace / Native-Thread OOM
Not every JVM OOM is a heap problem - metaspace and native-thread OOMs come from other ceilings, and the fix is more headroom, not a heap dump.
The problem
A JVM-based build or test run fails with OutOfMemoryError: Metaspace or unable to create new native thread. The heap may be fine; the job ran out of metaspace or native-memory/thread budget. A human raises the limit or runs on a bigger machine and it passes unchanged.
java.lang.OutOfMemoryError: Metaspace
# or
java.lang.OutOfMemoryError: unable to create new native threadWhy it happens
The JVM has several memory regions beyond the heap. Metaspace holds class metadata and grows as more classes load; native threads each consume their own stack outside the heap. Either can exhaust before the heap does, producing an OOM that a bigger heap would not fix.
CI loads a lot of classes and spawns many worker threads (parallel tests, build daemons), so metaspace and native-thread pressure are common on default limits - a mechanical ceiling, not necessarily a leak.
The manual fix
The manual fix is to give the right region more headroom or lower thread pressure, then retry:
- Raise metaspace and reduce thread fan-out (e.g.
-XX:MaxMetaspaceSize, lower test/build worker counts). - Re-run on a runner with more memory and a higher process/thread allowance.
- Confirm which region ran out from the error before changing heap settings that will not help.
How this gets automated
A metaspace or native-thread OOM has a distinct signature separate from a heap OOM, and the right response is to retry with appropriate headroom for that region. A self-healing CI pipeline recognizes the non-heap OOM condition, retries the step with adequate resources, and only escalates if the failure is a genuine, repeating leak rather than a mechanical ceiling.
Frequently asked questions
I already raised `-Xmx` and still get an OOM - why?
-Xmx only sizes the heap. A Metaspace or "unable to create native thread" OOM comes from a different region, so a bigger heap does nothing - it can even make native-thread OOMs worse by leaving less native memory. The fix is to size the region that actually ran out, or add overall headroom.