dmesg: Find the OOM Killer in CI Logs
dmesg prints the kernel ring buffer, where an out-of-memory kill leaves an "Out of memory: Killed process" line that explains an otherwise silent exit 137.
When a CI job dies with exit code 137 and no application error, the kernel OOM killer is the usual cause. dmesg is where it records the kill.
What it does
dmesg reads the kernel ring buffer: boot messages, driver output, and critically the OOM killer’s decisions. Each kill logs an "Out of memory: Killed process <pid> (<name>)" line plus a memory table. -T prints human-readable timestamps instead of seconds-since-boot.
Common usage
dmesg -T | grep -i "out of memory"
dmesg -T | grep -iE "killed process|oom-kill"
dmesg --level=err,warn -T # only warnings and errors
dmesg -w # follow new kernel messagesOptions
| Flag | What it does |
|---|---|
| -T | Human-readable timestamps |
| -w | Wait for and print new messages (follow) |
| --level=<list> | Filter by level, e.g. err,warn |
| -H | Human output with relative time and paging |
| -c | Clear the ring buffer after printing |
In CI
Exit 137 = 128 + 9, a SIGKILL, and on a memory-starved runner that SIGKILL comes from the OOM killer. After such a failure, dmesg -T | grep -i "killed process" names the process and the moment. Pair it with free -m to confirm the runner was actually out of memory. A Docker-imposed memory limit kills the same way and also shows here.
Common errors in CI
"dmesg: read kernel buffer failed: Operation not permitted" appears when kernel.dmesg_restrict=1 (common on hardened hosts and inside containers); read the host or use journalctl -k with sudo instead. In a container the buffer reflects the host kernel, so an OOM may have been the host’s, not the container’s; cross-check with the cgroup memory events.