The OOM killer: reading its kill logs and preventing the next one

Your process didn't crash. It was executed. When memory runs out, the Linux OOM killer picks a victim, sends it SIGKILL, and writes a detailed confession to the kernel log - which almost nobody reads past the first line. That's a waste, because the OOM killer's log is a complete memory autopsy: who was using what, why the kernel couldn't reclaim, and why it chose the process it chose. Learn to read it and "the app randomly dies sometimes" becomes a ticket with a fix attached.

Step 1: confirm it was actually the OOM killer

A process that vanished isn't necessarily OOM-killed. Check the kernel log:

dmesg -T | grep -iE 'killed process|out of memory|oom'
journalctl -k --since -24h | grep -i oom     # survives reboots if journald is persistent

If there's no OOM line, stop - you're debugging something else (a crash, a signal from a human or a supervisor, or a systemd unit hitting its own limits). If the process died with exit code 137 inside a container, that's SIGKILL - often OOM but not always; the kernel log is the ground truth.

Step 2: read the kill report

A real report, trimmed:

[Tue Sep  8 03:12:44 2026] java invoked oom-killer: gfp_mask=0x140cca, order=0, oom_score_adj=0
[Tue Sep  8 03:12:44 2026] Mem-Info:
[Tue Sep  8 03:12:44 2026] Tasks state (memory values in pages):
[Tue Sep  8 03:12:44 2026] [  pid  ]   uid  tgid total_vm      rss ... oom_score_adj name
[Tue Sep  8 03:12:44 2026] [   812 ]     0   812   295013     8102 ...     -250 systemd-journal
[Tue Sep  8 03:12:44 2026] [  4172 ]  1001  4172  9613410  7811234 ...        0 java
[Tue Sep  8 03:12:44 2026] Out of memory: Killed process 4172 (java) total-vm:38453640kB,
anon-rss:31244936kB, file-rss:1024kB, shmem-rss:0kB, UID:1001 pgtables:61232kB oom_score_adj:0

How to read it, in order of usefulness:

• invoked oom-killer names the process that asked for memory when the well ran dry, not necessarily the hog, just the last straw.
• Killed process names the victim. anon-rss is the number that matters: actual anonymous memory resident in RAM. total-vm is virtual address space and routinely 4–10× larger; don't let anyone panic about it.
• The task table is your whole-system memory census at the moment of death. Sort it mentally by rss. If the victim owned 30 GB of a 32 GB box, the diagnosis is done. If the victim was small, someone bigger with a negative oom_score_adj was protected, and the kernel shot the messenger.
• A memory: usage 2097152kB, limit 2097152kB line (with a cgroup path) means this was a cgroup OOM, not a system OOM: the container hit its own limit while the host had free memory. Different problem, different fix - raise the limit or shrink the workload, and in Kubernetes that's the OOMKilled / exit 137 investigation.

How the victim gets chosen

Each process has an oom_score - roughly, its share of usable memory - visible live:

cat /proc/<pid>/oom_score
cat /proc/<pid>/oom_score_adj    # -1000 (never kill) .. 1000 (kill me first)

The kernel kills the highest score. oom_score_adj is how you put a thumb on the scale: sshd ships with a negative value so you can still log in; you can protect a critical daemon with a systemd drop-in:

# /etc/systemd/system/postgres.service.d/oom.conf
[Service]
OOMScoreAdjust=-800

Use this sparingly. -1000 means the kernel will kill everything else first, including things you'll miss. Protecting the database and letting the OOM killer eat your monitoring agent is a legitimate trade - decide it on purpose, in a config review, not at 3 AM.

Preventing the next one

Bound the blast, don't just add RAM. The systemic fix is cgroup limits so one process can't take the box down:

[Service]
MemoryMax=4G        # hard cap: cgroup OOM-kills the service, host survives
MemoryHigh=3G       # soft cap: throttles/reclaims before the cliff

A service that OOMs inside its own 4 GB sandbox is an incident for one service. A host-level OOM is a lottery where every process bought a ticket.

Understand overcommit before touching it. By default (vm.overcommit_memory=0) Linux hands out address space optimistically and settles the bill at page-fault time - that's why the OOM killer exists. vm.overcommit_memory=2 (strict accounting) makes malloc fail instead of the kernel killing later, which sounds safer and breaks a surprising amount of software that never checks malloc returns. Know the trade before flipping it.

Watch pressure, not just usage. By the time free memory hits zero, you were in trouble for minutes - the box was reclaiming, swapping, and stalling first. Rising major faults and swap activity (vmstat 1 → si/so) show up early, and PSI (/proc/pressure/memory) quantifies the stall. Memory pressure also masquerades as an I/O problem and drives load average through the roof before anything dies.

A little swap is a shock absorber, not a fix. It converts an instant kill into a slow degradation you can alert on. If you run with zero swap, the cliff has no railing.

The forensic habit

The OOM kill report is timestamped evidence. Correlate it: what deployed just before, what cron fired, what request pattern changed. journalctl --since/--until around the kill time reconstructs the sequence - the technique is the same journal forensics workflow you'd use for any incident. The engineers who handle OOM incidents well aren't the ones who add RAM fastest; they're the ones who can point at the line in dmesg and say this process, this much anon-rss, this cgroup - and then verify the fix by watching the pressure metrics, not by waiting a week to see if it dies again.

FAQ

How do I know if the OOM killer killed my process? dmesg -T | grep -i 'killed process' or journalctl -k | grep -i oom. If there's no kernel log line, it wasn't the OOM killer - look for other SIGKILL sources (a supervisor, a cgroup manager, a human).

What's the difference between a cgroup OOM and a system OOM? A cgroup OOM means one container/service hit its own memory limit; the host was fine. A system OOM means the whole box ran out. The kill report shows a cgroup path and limit for the former. Fixes differ: adjust the limit or the workload vs. find the host-level hog.

Does total-vm matter in the kill report? Rarely. It's reserved address space, not RAM. anon-rss is the real footprint. A JVM with total-vm of 38 GB and anon-rss of 2 GB is not your problem.

Can I disable the OOM killer? You can (vm.overcommit_memory=2, or panic-on-oom), but the alternative to killing is often a hung or panicked machine. Bounding services with cgroup limits is the option that keeps the host alive and the failure small.