Pod stuck in Pending: the complete diagnostic tree

A pod stuck in Pending means one precise thing in Kubernetes: no node has been chosen for it. Nothing is crashing, nothing is pulling, nothing is running - the scheduler looked at your pod and every node in the cluster, and found no legal match. The good news: the scheduler writes down exactly why it failed, per constraint, per node count. The first move is always to read that verdict:

kubectl describe pod <pod> | grep -A6 Events

You're looking for a FailedScheduling event like:

0/8 nodes are available: 3 Insufficient cpu, 3 node(s) had untolerated taint
{node-role.kubernetes.io/control-plane: }, 2 node(s) didn't match Pod's node affinity/selector.

That line is the whole diagnosis - it partitions every node in the cluster into the reason it was rejected. The rest of this guide is the tree for each reason, fastest check first.

Insufficient cpu / Insufficient memory

The most common. No node has enough unreserved capacity for the pod's requests - note: requests, not actual usage. A cluster can be 20% utilized and still unschedulable because everyone's requests are padded.

kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources.requests}'
kubectl describe nodes | grep -A5 "Allocated resources"

Compare what the pod asks for against what each node has left. Three usual findings:

• The request has a typo. memory: 4000Mi meant to be 400Mi, or the infamous cpu: 4000m that should have been 400m. Fix the manifest.
• The request is honest and the cluster is full. Scale the node pool, or evict something less important - priority and preemption exist for this.
• Requests across the fleet are folklore. Everyone asked for 2 CPUs "to be safe" and the scheduler believes them. That cleanup is its own project, requests and limits mistakes is the map.

If nodes are also under actual pressure and killing tenants to cope, you'll see it as evictions rather than Pending - a different symptom of the same capacity disease.

node(s) had untolerated taint

Nodes can carry taints; pods need matching tolerations to land on them. The event names the exact taint, so this one hands you the answer:

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'

Common cases, in order of "wait, that's it?":

• Control-plane taint. node-role.kubernetes.io/control-plane:NoSchedule on a small cluster where the only spare capacity is the control-plane node. Either tolerate it deliberately or add workers - don't strip the taint in a hurry.
• Dedicated node pools. dedicated=gpu:NoSchedule and your pod lacks the toleration, add the toleration and usually a matching nodeSelector, or you'll tolerate your way onto the wrong hardware.
• Condition taints. node.kubernetes.io/not-ready, unreachable, disk-pressure, memory-pressure. These aren't scheduling bugs - they're node health problems that taint the node automatically. Chase the node, not the pod: Node NotReady: what to check, in order.

didn't match node affinity/selector

The pod demands label X; no node has label X. Check both sides of the contract:

kubectl get pod <pod> -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}'
kubectl get nodes --show-labels

The classics: a label with a typo (disktype=ssd vs disk-type=ssd), a node pool that was replaced and came back without the custom label, topology.kubernetes.io/zone pinning to a zone with no capacity, or requiredDuringScheduling where preferred was meant. Also check pod anti-affinity: requiredDuringScheduling anti-affinity against your own app with topologyKey: kubernetes.io/hostname caps you at one replica per node - replica 4 on a 3-node cluster will be Pending forever, by design.

Volume problems: unbound PVCs and zone mismatches

pod has unbound immediate PersistentVolumeClaims

The pod is waiting on storage, so the real patient is the claim:

kubectl get pvc -n <namespace>
kubectl describe pvc <claim>

If the PVC is Pending too, follow the storage tree - wrong or missing StorageClass, a provisioner that isn't installed, no matching PV - in PVC stuck in Pending. One case looks like a bug but isn't: a StorageClass with volumeBindingMode: WaitForFirstConsumer keeps the PVC Pending until a pod schedules - that's normal sequencing. And on multi-zone clusters, remember a volume already bound in zone A vetoes every node in zone B; the event says node(s) had volume node affinity conflict.

The quota case: the pod never even appears

ResourceQuota rejections don't produce Pending pods - they prevent the pod from being created at all. Symptom: the Deployment says 2/3 ready and there's no third pod to describe. Look one level up:

kubectl describe rs <replicaset> | grep -A4 Events
# "failed quota: compute-quota: must specify limits.memory"
kubectl describe quota -n <namespace>

Either the namespace is over quota, or quota requires requests/limits the pod doesn't set. Fix the spec or negotiate the quota - but know that this branch starts from the ReplicaSet's events, not the pod's.

No events at all

describe shows Pending and an empty event list. Now the scheduler itself is the suspect:

kubectl get pod <pod> -o jsonpath='{.spec.schedulerName}'
kubectl get pods -n kube-system | grep scheduler

Either the pod names a scheduler that doesn't exist (schedulerName: my-scheduler, an artifact of copied YAML), or the scheduler is down/crash-looping - check it like any other broken pod with the CrashLoopBackOff field guide. And if your pod is actually ContainerCreating rather than Pending: that's post-scheduling, a different tree entirely - image pulls, volume mounts, CNI.

FAQ

How long will a pod stay Pending before Kubernetes gives up? Forever. There is no timeout - the scheduler retries every unschedulable pod indefinitely, and the moment capacity or labels change it will place it. A pod Pending for three days is not "stuck" in the system's eyes; it's patiently waiting for a cluster that fits.

Why is my pod Pending when the cluster has plenty of free memory? Because scheduling is done on requests, not usage. If existing pods request far more than they use, the node's allocatable capacity is spoken for on paper even while the hardware idles. Compare Allocated resources in kubectl describe node output against kubectl top node and you'll see the gap.

Can I force a Pending pod onto a specific node? Setting nodeName directly bypasses the scheduler entirely - the kubelet will try to run it even if taints and resources say no, and it will fail there instead. It's occasionally useful for debugging, and almost always the wrong production fix. Fix the constraint the scheduler reported instead.

Does deleting and recreating a Pending pod help? Rarely. Pending isn't backoff - it's an unsatisfiable constraint, and the replacement pod inherits the same spec and the same verdict. The exception is stale state around volumes or a fixed scheduler; after changing the cause, recreating the pod is fine and often fastest.