skip to content
‹ All posts

By Riya Madaan · Updated

Node NotReady: what to check, in order

A Kubernetes node NotReady is a race against pod eviction. The ordered checklist: kubelet, CNI, certs, and node pressure - fastest check first.

#kubernetes

A Kubernetes node NotReady is a page with a timer attached. It doesn't just break one node, after about five minutes the control plane starts evicting that node's pods, and the failure spreads to wherever they land. The good news: nearly every NotReady comes down to one of five causes, and the first command tells you which branch you're in. This post is the triage order for the whole node; if you've already narrowed it to a kubelet that won't start, go straight to kubelet not starting for the journal patterns.

kubectl describe node <node>

Scroll to Conditions. Everything below hangs off what you find there.

Symptom Likely cause First command
Ready = Unknown, stale lease renewTime Kubelet down, machine down, or network partition systemctl status kubelet
Kubelet crash-looping (swap, cgroup driver, bad config) Kubelet can't start journalctl -u kubelet -e --no-pager
Ready = False, cni plugin not initialized CNI pod not running on the node kubectl get pods -n kube-system -o wide --field-selector spec.nodeName=<node>
DiskPressure = True Kubelet or image filesystem full df -h /var/lib/kubelet /var/lib/containerd
x509: certificate has expired in the kubelet journal Kubelet client cert expired openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet-client-current.pem
Node goes NotReady, then disappears from kubectl get nodes Spot reclaim or autoscaler scale-down Check the autoscaling group's activity log

What's the difference between Ready=Unknown and Ready=False?

Unknown means the control plane stopped hearing from the node; False means the kubelet is alive and reporting that something is broken. They point in opposite directions.

Ready = Unknown - the node has stopped reporting entirely. The kubelet is dead, the machine is down, or there's a network partition between the node and the API server. The control plane marks a node Unknown when it hasn't heard a heartbeat within the node-monitor grace period (40 seconds by default).

Ready = False - the kubelet is alive and talking, but reporting that something on the node is broken. The condition's Message field usually names the culprit outright. Read it before touching anything.

You can also check the heartbeat directly:

kubectl get lease -n kube-node-lease <node> -o yaml

A stale renewTime confirms the kubelet stopped renewing its lease - you're in the Unknown branch. Now walk the causes, fastest check first.

1. The kubelet is down

The most common cause by a wide margin. SSH to the node:

systemctl status kubelet
journalctl -u kubelet -e --no-pager

If it's crash-looping, the last few log lines almost always say why: a config file that doesn't parse, swap enabled (failed to run Kubelet: running with swap on is not supported), a cgroup driver mismatch with the container runtime, or a disk so full the kubelet can't write its own state. The kubelet's failure modes are a topic of their own, covered in the kubelet post linked above.

Restarting the kubelet is low-risk: running containers keep running while it's down. But read the logs first - a restart that works destroys your evidence for why it stopped.

2. The CNI never came up

If kubectl describe node shows Ready = False with a message like:

container runtime network not ready: NetworkReady=false
reason:NetworkPluginNotReady message:Network plugin returns error:
cni plugin not initialized

the kubelet is fine; the network plugin isn't. Check whether the CNI DaemonSet pod is actually running on this node:

kubectl get pods -n kube-system -o wide \
  --field-selector spec.nodeName=<node>

If the Calico/Cilium/flannel pod on that node is missing or crash-looping, that's your problem, not the node. On the node itself, ls /etc/cni/net.d/ - an empty directory means the CNI never installed its config, which is common on freshly provisioned nodes where the CNI DaemonSet hasn't scheduled yet.

3. Node pressure conditions

Still in kubectl describe node conditions: MemoryPressure, DiskPressure, or PIDPressure set to True. The node is technically alive but the kubelet has started refusing new pods and evicting existing ones.

DiskPressure is the one you'll meet most. Check both filesystems the kubelet cares about:

df -h /var/lib/kubelet /var/lib/containerd

Image garbage collection usually can't save you fast enough once you're over the eviction threshold; deleting unused images manually (crictl rmi --prune) can. Disk-pressure eviction is one of the faults in the practice labs if you'd rather meet it first on a cluster that isn't yours. Who gets evicted under pressure, and in what order, is its own topic - covered in evicted pods and node pressure.

4. Expired kubelet certificates

If journalctl -u kubelet is full of:

x509: certificate has expired or is not yet valid

the kubelet's client certificate expired and it can no longer authenticate to the API server. Check the expiry directly:

openssl x509 -noout -enddate \
  -in /var/lib/kubelet/pki/kubelet-client-current.pem

Kubelet certs normally rotate themselves, but rotation fails quietly when the node was powered off past expiry or the cluster CA changed. This failure rarely visits one node alone - if one cert expired, check the rest of the cluster before it checks you. Full recovery steps live in Kubernetes certificate expiry.

5. The cloud provider took the node

Spot/preemptible instance reclaimed, autoscaler scale-down, failed cloud health check. The tell: the node goes NotReady and then disappears from kubectl get nodes entirely, or the cloud console shows the instance stopped. Check your autoscaling group's activity log before spending an hour debugging a machine that no longer exists.

What happens to pods on a NotReady node?

They get evicted and rescheduled after about five minutes. When a node goes NotReady or Unknown, the control plane adds node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taints. Almost every pod carries a default toleration for these with tolerationSeconds: 300 - meaning after 5 minutes, pods on the node are marked for deletion and rescheduled elsewhere.

That's the timer. If the cluster has spare capacity, this is self-healing. If it doesn't, the evicted pods pile up unschedulable - see pod stuck in Pending for that follow-on failure. DaemonSet pods stay put; StatefulSet pods on an Unknown node will not be replaced until the node comes back or is deleted, because the control plane can't prove the old pod stopped.

Once the node recovers: verify Ready, then check whether you cordoned it during debugging.

kubectl uncordon <node>

Leaving a healthy node cordoned is the kind of leftover damage that's invisible until the next incident - the fix isn't done until the cleanup is.

FAQ

What does NotReady actually mean? The kubelet posts a status heartbeat to the API server via its node lease. NotReady means either the kubelet reported a failing condition (Ready = False) or the control plane stopped hearing from it at all (Ready = Unknown, after the ~40-second grace period). It's a symptom with five common causes, not a diagnosis.

How long until pods get evicted from a NotReady node? Five minutes by default - the tolerationSeconds: 300 on the automatic not-ready/unreachable tolerations. You can tune it per pod, which is worth doing for workloads where failover is expensive.

Is it safe to restart the kubelet? Yes. Running containers are managed by the container runtime, not the kubelet, and keep running while it restarts. Read journalctl -u kubelet before restarting so you don't lose the reason it died.

Should I just reboot the node? Only after you've read the kubelet logs and checked disk. A reboot fixes the symptom, erases the evidence, and books the same page for next week.