skip to content
‹ All posts

By Riya Madaan

Evicted pods: node pressure and who gets killed first

Kubernetes pod evicted? How node-pressure eviction works, what the eviction message means, why QoS class decides who dies first, and how to clean up.

#kubernetes

A Kubernetes pod evicted is the kubelet playing lifeboat captain. A node ran low on memory, disk, or PIDs; the kubelet had to throw someone overboard to keep the node itself alive; and it chose your pod by a ranking you probably didn't know you'd entered. The pod object sticks around as a corpse with Status: Failed, Reason: Evicted so you can read the verdict:

kubectl get pods -A --field-selector=status.phase=Failed
kubectl describe pod <evicted-pod>
Status:      Failed
Reason:      Evicted
Message:     The node was low on resource: memory. Threshold quantity: 100Mi, available: 87Mi.
             Container api was using 1200Mi, request is 256Mi, has larger consumption of memory.

Every word of that message is load-bearing. Which resource. What the threshold was. And, the part people skip - why this pod: it was using far more than it requested. Let's unpack the machinery. (Evicted is one of several statuses that are really node problems; the pod status reference maps the rest.)

Why do pods get evicted?

Because the node ran low on memory, disk, or PIDs, and the kubelet killed pods to keep the node alive. The kubelet watches a set of eviction signals: memory.available, nodefs.available, imagefs.available, nodefs.inodesFree, pid.available. Each has thresholds:

  • Hard thresholds (e.g. memory.available<100Mi): cross it and the kubelet evicts immediately, with no grace period beyond the pod-level minimum.
  • Soft thresholds: cross it for longer than a grace period and eviction starts, honoring a configured termination grace.

You can read a node's current state from its conditions:

kubectl describe node <node> | grep -A8 Conditions

MemoryPressure, DiskPressure, PIDPressure set to True mean the kubelet is in eviction mode right now - and the scheduler also stops placing new pods there (which is one of the answers to pods stuck in Pending).

Disk pressure has an extra wrinkle: before evicting anyone, the kubelet tries garbage collection - deleting dead containers and unused images. If you see ImageGCFailed or constant image churn in node events, the node is fighting for disk before your pods start paying the price.

Which pods get evicted first?

Pods using more than they requested, lowest priority first. For memory pressure, the kubelet ranks pods by, in order:

  1. Whether usage exceeds requests. Pods using more than they requested are first in line, regardless of QoS class. A Burstable pod within its request outranks (survives) one over it.
  2. Pod priority (priorityClassName). Lower priority dies earlier.
  3. How far over the request the usage is.

QoS class is the shorthand everyone quotes, and it maps closely onto that ranking:

QoS class Definition Practical fate under memory pressure
BestEffort No requests, no limits Using anything > requested nothing → first out
Burstable Requests < limits (or partial) Evicted if over request; safe-ish under it
Guaranteed requests == limits on every container Last to go - usage can't exceed request
kubectl get pod <pod> -o jsonpath='{.status.qosClass}'

This is the operational argument for setting honest requests: they're not just scheduler hints, they're your position in the lifeboat queue. The full set of request/limit tradeoffs is in Requests and limits: the misconfigurations that page you.

Eviction vs OOMKilled: don't conflate them

Both involve memory; the fixes differ.

  • OOMKilled (exit code 137): the container exceeded its own memory limit; the kernel killed the process; the kubelet restarts it in place. That's a per-container ceiling - see OOMKilled and exit code 137.
  • Evicted: the node ran short; the kubelet killed the whole pod; a controller-managed pod gets rescheduled elsewhere, a bare pod just dies.

If pods are being evicted while their limits are nowhere near reached, the node is oversubscribed: the sum of workloads' actual usage exceeded the node's allocatable. That's a requests-accuracy problem (requests far below real usage let the scheduler overpack the node), not an application bug.

Also distinct: API-initiated eviction (drains, kubectl drain, descheduler) goes through the Eviction API and respects PodDisruptionBudgets. Node-pressure eviction does not respect PDBs - the kubelet is protecting the node and asks no one's permission.

The diagnostic path when you find evicted pods

# 1. Read the verdict - which resource, which threshold
kubectl describe pod <evicted-pod> | sed -n '/Message/,/^$/p'

# 2. Was it one node or many?
kubectl get pods -A --field-selector=status.phase=Failed \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,MSG:.status.message

# 3. Is the node still under pressure?
kubectl describe node <node> | grep -A8 Conditions
kubectl top node <node>

# 4. For disk pressure: what's eating the disk? (on the node)
df -h /var/lib/kubelet /var/lib/containerd
du -sh /var/log/pods/* | sort -h | tail

One node evicting is a node problem (log flood, image bloat, one greedy pod). Many nodes evicting at once is a capacity or requests-honesty problem - and adding replicas of the greedy service will make it worse. A node filling its disk until the kubelet starts evicting is one of the practice labs, if you want to walk this path on a live cluster.

Cleaning up the corpses

Evicted pods are kept, by design, as evidence - the equivalent of the flight recorder. They don't consume node resources, but hundreds of them clutter every kubectl get pods. Once you've read what they had to say:

kubectl delete pods -A --field-selector=status.phase=Failed

The kube-controller-manager also garbage-collects them when the terminated-pod count crosses a threshold (--terminated-pod-gc-threshold, default 12500 - i.e., effectively never in a small cluster). Delete the evidence after the diagnosis, not before: an engineer who wipes the Failed pods first has destroyed the only record of which resource and which threshold, in a graded session, that's evidence-destruction with a blast-radius cost, mid-incident it's just self-sabotage.

FAQ

Do evicted pods come back automatically? Pods owned by a Deployment/ReplicaSet/StatefulSet are replaced - the controller creates a new pod, scheduled onto a (hopefully) healthier node. Bare pods and completed Jobs' pods are not resurrected. The evicted pod object itself never restarts; it's a record, not a process.

Why was my pod evicted when the node had free memory in kubectl top? kubectl top shows a sample, evictions act on the kubelet's threshold math over memory.available, which includes reclaim behavior and can move fast. Also check whether the eviction was for disk or PIDs - the message names the resource, and it's frequently ephemeral-storage, not memory.

Can I stop a specific pod from ever being evicted? Make it Guaranteed (requests == limits everywhere) and give it a high priorityClassName. Critical system pods use exactly this. Nothing makes a pod truly unevictable on a dying node, nor should it.

What does "Pod ephemeral local storage usage exceeds the total limit" mean? That's a limit eviction, not node pressure: the pod blew through its own ephemeral-storage limit (usually logs or temp files). Raise the limit or stop writing, emptyDir counts against it.