By Riya Madaan · Updated
Node NotReady: what to check, in order
A Kubernetes node NotReady is a race against pod eviction. The ordered checklist: kubelet, CNI, certs, and node pressure - fastest check first.
A Kubernetes node NotReady is a page with a timer attached. It doesn't just break one node, after about five minutes the control plane starts evicting that node's pods, and the failure spreads to wherever they land. The good news: nearly every NotReady comes down to one of five causes, and the first command tells you which branch you're in. This post is the triage order for the whole node; if you've already narrowed it to a kubelet that won't start, go straight to kubelet not starting for the journal patterns.
kubectl describe node <node>
Scroll to Conditions. Everything below hangs off what you find there.
| Symptom | Likely cause | First command |
|---|---|---|
Ready = Unknown, stale lease renewTime |
Kubelet down, machine down, or network partition | systemctl status kubelet |
| Kubelet crash-looping (swap, cgroup driver, bad config) | Kubelet can't start | journalctl -u kubelet -e --no-pager |
Ready = False, cni plugin not initialized |
CNI pod not running on the node | kubectl get pods -n kube-system -o wide --field-selector spec.nodeName=<node> |
DiskPressure = True |
Kubelet or image filesystem full | df -h /var/lib/kubelet /var/lib/containerd |
x509: certificate has expired in the kubelet journal |
Kubelet client cert expired | openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet-client-current.pem |
Node goes NotReady, then disappears from kubectl get nodes |
Spot reclaim or autoscaler scale-down | Check the autoscaling group's activity log |
What's the difference between Ready=Unknown and Ready=False?
Unknown means the control plane stopped hearing from the node; False means the kubelet is
alive and reporting that something is broken. They point in opposite directions.
Ready = Unknown - the node has stopped reporting entirely. The kubelet is dead, the
machine is down, or there's a network partition between the node and the API server. The
control plane marks a node Unknown when it hasn't heard a heartbeat within the
node-monitor grace period (40 seconds by default).
Ready = False - the kubelet is alive and talking, but reporting that something on the
node is broken. The condition's Message field usually names the culprit outright. Read it
before touching anything.
You can also check the heartbeat directly:
kubectl get lease -n kube-node-lease <node> -o yaml
A stale renewTime confirms the kubelet stopped renewing its lease - you're in the
Unknown branch. Now walk the causes, fastest check first.
1. The kubelet is down
The most common cause by a wide margin. SSH to the node:
systemctl status kubelet
journalctl -u kubelet -e --no-pager
If it's crash-looping, the last few log lines almost always say why: a config file that
doesn't parse, swap enabled (failed to run Kubelet: running with swap on is not supported),
a cgroup driver mismatch with the container runtime, or a disk so full the kubelet can't
write its own state. The kubelet's failure modes are a topic of their own, covered in the
kubelet post linked above.
Restarting the kubelet is low-risk: running containers keep running while it's down. But read the logs first - a restart that works destroys your evidence for why it stopped.
2. The CNI never came up
If kubectl describe node shows Ready = False with a message like:
container runtime network not ready: NetworkReady=false
reason:NetworkPluginNotReady message:Network plugin returns error:
cni plugin not initialized
the kubelet is fine; the network plugin isn't. Check whether the CNI DaemonSet pod is actually running on this node:
kubectl get pods -n kube-system -o wide \
--field-selector spec.nodeName=<node>
If the Calico/Cilium/flannel pod on that node is missing or crash-looping, that's your
problem, not the node. On the node itself, ls /etc/cni/net.d/ - an empty directory means
the CNI never installed its config, which is common on freshly provisioned nodes where the
CNI DaemonSet hasn't scheduled yet.
3. Node pressure conditions
Still in kubectl describe node conditions: MemoryPressure, DiskPressure, or
PIDPressure set to True. The node is technically alive but the kubelet has started
refusing new pods and evicting existing ones.
DiskPressure is the one you'll meet most. Check both filesystems the kubelet cares about:
df -h /var/lib/kubelet /var/lib/containerd
Image garbage collection usually can't save you fast enough once you're over the eviction
threshold; deleting unused images manually (crictl rmi --prune) can. Disk-pressure eviction
is one of the faults in the practice labs if you'd rather meet it first on a
cluster that isn't yours. Who gets evicted
under pressure, and in what order, is its own topic - covered in
evicted pods and node pressure.
4. Expired kubelet certificates
If journalctl -u kubelet is full of:
x509: certificate has expired or is not yet valid
the kubelet's client certificate expired and it can no longer authenticate to the API server. Check the expiry directly:
openssl x509 -noout -enddate \
-in /var/lib/kubelet/pki/kubelet-client-current.pem
Kubelet certs normally rotate themselves, but rotation fails quietly when the node was powered off past expiry or the cluster CA changed. This failure rarely visits one node alone - if one cert expired, check the rest of the cluster before it checks you. Full recovery steps live in Kubernetes certificate expiry.
5. The cloud provider took the node
Spot/preemptible instance reclaimed, autoscaler scale-down, failed cloud health check. The
tell: the node goes NotReady and then disappears from kubectl get nodes entirely, or
the cloud console shows the instance stopped. Check your autoscaling group's activity log
before spending an hour debugging a machine that no longer exists.
What happens to pods on a NotReady node?
They get evicted and rescheduled after about five minutes. When a node goes NotReady or Unknown, the control plane adds node.kubernetes.io/not-ready
or node.kubernetes.io/unreachable taints. Almost every pod carries a default toleration for
these with tolerationSeconds: 300 - meaning after 5 minutes, pods on the node are marked
for deletion and rescheduled elsewhere.
That's the timer. If the cluster has spare capacity, this is self-healing. If it doesn't,
the evicted pods pile up unschedulable - see
pod stuck in Pending for that follow-on failure.
DaemonSet pods stay put; StatefulSet pods on an Unknown node will not be replaced until
the node comes back or is deleted, because the control plane can't prove the old pod stopped.
Once the node recovers: verify Ready, then check whether you cordoned it during debugging.
kubectl uncordon <node>
Leaving a healthy node cordoned is the kind of leftover damage that's invisible until the next incident - the fix isn't done until the cleanup is.
FAQ
What does NotReady actually mean?
The kubelet posts a status heartbeat to the API server via its node lease. NotReady means
either the kubelet reported a failing condition (Ready = False) or the control plane
stopped hearing from it at all (Ready = Unknown, after the ~40-second grace period). It's
a symptom with five common causes, not a diagnosis.
How long until pods get evicted from a NotReady node?
Five minutes by default - the tolerationSeconds: 300 on the automatic not-ready/unreachable
tolerations. You can tune it per pod, which is worth doing for workloads where failover is
expensive.
Is it safe to restart the kubelet?
Yes. Running containers are managed by the container runtime, not the kubelet, and keep
running while it restarts. Read journalctl -u kubelet before restarting so you don't lose
the reason it died.
Should I just reboot the node? Only after you've read the kubelet logs and checked disk. A reboot fixes the symptom, erases the evidence, and books the same page for next week.