skip to content
‹ All posts

OOMKilled and exit code 137: finding the real memory hog

OOMKilled Kubernetes pods and exit code 137: how to read Last State, use kubectl top, dodge JVM container traps, and decide raise-the-limit vs fix-the-leak.

#kubernetes#debugging
#kubernetes

An OOMKilled Kubernetes pod is the kernel telling you a story with the ending torn out. The container crossed its memory limit, the cgroup OOM killer sent SIGKILL, and the process got zero chance to log, flush, or say goodbye - which is why the application logs so often end mid-sentence. Exit code 137 is the arithmetic of that death: 128 + 9, where 9 is SIGKILL.

The first move is to confirm the kill actually was an OOM kill, because 137 alone doesn't prove it:

kubectl describe pod <pod>

Look at the container's Last State:

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

Reason: OOMKilled is the confirmation. Exit 137 with a different reason (Error, Completed after eviction, a failed liveness probe whose kill escalated to SIGKILL) is a different investigation - the exit code reference covers the full table. Assume nothing from 137 alone.

Which container, and against what limit

A pod is not the unit of death; a container is. In a pod with sidecars, find the actual victim - describe shows Last State per container, and usage is per container too:

kubectl top pod <pod> --containers

Then get the limit it died against:

kubectl get pod <pod> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits.memory}{"\n"}{end}'

Two early findings worth their weight:

  • No limit set, still OOMKilled? Then the node ran out of memory and the kernel's system-level OOM killer chose your process - a node problem wearing a pod costume. The node's kernel log has the verdict; reading OOM killer logs is its own skill.
  • Limit suspiciously small? 128Mi on a JVM, or a limit inherited from a LimitRange default nobody remembers setting. Check for namespace defaults:
kubectl get limitrange -n <namespace> -o yaml

Watch it die: usage vs limit over time

kubectl top gives you a snapshot; OOM kills are about trajectory. If you have Prometheus, plot container_memory_working_set_bytes against the limit - that's the number the OOM killer judges, not RSS. Without metrics, the cheap version is a watch loop while you reproduce:

watch -n5 'kubectl top pod <pod> --containers'

The shape of the curve is the diagnosis:

  • Slow climb over hours/days, then death - a leak, or an unbounded cache. Raising the limit reschedules the outage; it doesn't cancel it.
  • Flat, then a spike on some trigger - a large request, a batch job, a report endpoint. The limit is sized for the average, not the peak. Find the trigger in the access logs at the kill timestamp.
  • Dies immediately at startup - the working set never fit. Common after a dependency bump or when someone set the limit by copying another service's YAML.

The kill timestamp is in Last State: Terminated / Finished: - correlate it with deploys, cron schedules, and traffic. OOM kills at :00 every hour are not a mystery, they're a cron job.

The JVM (and friends) gotcha

The classic false leak: the app is sized against the host, not the container. An older JVM defaults its heap to a fraction of visible memory - and inside a container it may see the node's 64 GB, size a heap far beyond a 512Mi limit, and get killed while behaving exactly as configured. Modern JVMs (8u191+, 10+) are container-aware, but only if nobody overrode it with a hardcoded -Xmx from the pre-container era.

kubectl exec <pod> -- java -XX:+PrintFlagsFinal -version | grep -i maxheap

Sane setup: -XX:MaxRAMPercentage=75.0 and let the limit drive the heap - the remaining 25% is not generosity, it's metaspace, threads, and off-heap buffers, which is why a JVM with a heap below the limit can still be OOMKilled. The same shape exists elsewhere: Go's GOMEMLIMIT unset, Python workers multiplied by a WORKERS env var tuned for a bigger box, Node's --max-old-space-size copied from a Dockerfile written for different hardware.

Raise the limit or fix the leak?

The honest decision rule:

Raise the limit when the workload legitimately needs the memory - the working set grew with the data, the spike is a real workload you intend to serve, the old limit was a guess. Set it from observed peak working set plus headroom (~20–30%), and fix requests at the same time, because requests-vs-limits misconfiguration is its own pager, the mistakes that page you covers that.

Fix the app when the curve climbs without bound, when the spike is a bug (unbounded query result, missing pagination, cache without eviction), or when you've already raised the limit once for the same symptom. The second limit raise for the same curve is not a fix; it's a payment plan.

Either way, verify like you mean it: after the change, watch the restart counter stay flat and the working set stabilize below the limit under real traffic - not just for the five minutes after deploy. If the pod is also crash-looping for non-memory reasons in between OOM kills, untangle that separately with the CrashLoopBackOff field guide.

FAQ

Is exit code 137 always an OOM kill? No. 137 means the process died by SIGKILL; the OOM killer is just its most famous sender. Node eviction, a kubectl delete that outran the grace period, or a liveness-probe kill that escalated all produce 137. Only Reason: OOMKilled in the container's Last State confirms memory as the cause.

Why was my pod OOMKilled below its memory limit? Usually accounting: the kernel judges the cgroup's working set, which includes page cache your process caused and, depending on runtime, sibling processes in the container. Also check for a lower per-container limit than you think (LimitRange defaults) and, on nodes under pressure, system-level OOM kills that ignore your limit entirely.

Do memory limits cause throttling like CPU limits do? No - that asymmetry trips people up. CPU limits throttle; memory limits kill. There is no "memory slowdown" phase: the container is either under the limit or it's dead. That's why memory limits deserve headroom and CPU limits deserve skepticism.

Should I just remove memory limits? Tempting, but the limit isn't the villain - it's the blast door. Without limits, a leaking pod eats the node and takes innocent neighbors with it via node-level OOM or eviction - the memory-shaped twin of a node whose disk quietly fills up. Keep limits; size them from measured working sets instead of folklore.