skip to content
‹ All posts

By Riya Madaan · Updated

Rolling update stuck: maxUnavailable, PDBs, hung rollouts

Kubernetes rollout stuck? How to read rollout status, decode progressDeadlineExceeded, spot PDB deadlocks, and roll back without making it worse.

#kubernetes

A Kubernetes rollout stuck at "2 out of 5 new replicas have been updated" is the platform equivalent of an elevator between floors. Nothing is down, exactly - old pods are still serving - but the deploy isn't finishing, CI is red, and every minute you wait, the odds increase that someone "helps" by deleting things.

The stuck rollout is almost never the Deployment controller misbehaving. It's the controller refusing to proceed because some condition it's waiting on will never come true. Your job is to find which condition. Here's the ordered path.

Symptom Likely cause First command
Progressing False ProgressDeadlineExceeded No new pod became ready within progressDeadlineSeconds kubectl describe deployment <name>
New pods in ImagePullBackOff New tag missing or registry auth broken kubectl describe pod <new-pod>
New pods in CrashLoopBackOff New build crashes on boot kubectl logs <new-pod> --previous
New pods Pending No headroom for surge pods kubectl describe pod <new-pod>
New pods Running but 0/1 READY Readiness probe never passes kubectl describe pod <new-pod>
Node drain hangs mid-rollout PDB allows zero disruptions kubectl get pdb -A

1. Ask the rollout what it's waiting for

kubectl rollout status deployment/<name>
kubectl describe deployment <name>
kubectl get rs -l app=<name>

rollout status gives you the live counters. describe gives you Conditions - the one to read is Progressing. If it says ProgressDeadlineExceeded, the controller has formally given up: no progress for progressDeadlineSeconds (default 600). That doesn't roll anything back automatically - it just sets the condition and stops the clock. The rollout stays wedged until you act.

The ReplicaSet listing tells you the shape of the problem: an old RS holding replicas and a new RS that can't grow. Now find out why the new pods aren't becoming ready.

2. Why aren't the new pods becoming ready?

Usually one of four things: a bad image, a crash on boot, no room to schedule, or a readiness probe the new build never passes. This is the most common answer by far.

kubectl get pods -l app=<name>
kubectl describe pod <new-pod>

Match the symptom to its guide:

  • ImagePullBackOff - the new tag doesn't exist or auth broke. See Fixing ImagePullBackOff. A rollout wedged on a bad image is one of the practice labs, if you want to unwedge one for real.
  • CrashLoopBackOff - the new code crashes on boot. kubectl logs --previous; full tree in Debugging CrashLoopBackOff.
  • Pending - the cluster can't fit the surge pods. Remember rolling updates need headroom: with default maxSurge: 25%, a 20-replica deployment briefly wants 25 pods.
  • Running but 0/1 READY - failing readiness probe. This is the silent one: no crash, no event drama, just a probe that the new build never passes. describe pod shows Readiness probe failed. A readiness gate from a service mesh (Pod readiness gates in describe output) can hold pods un-ready the same way.

A rollout stuck on readiness is the controller doing exactly its job: it will not shift traffic to pods that say they aren't ready. Fix the probe or fix the app - don't loosen maxUnavailable to force it through.

3. maxUnavailable and maxSurge arithmetic

strategy:
  rollingUpdate:
    maxUnavailable: 25%
    maxSurge: 25%

Two configurations produce mysterious-looking stalls:

  • maxUnavailable: 0 with maxSurge: 0 is an illegal combination - the API rejects it, but maxUnavailable: 0 with a small surge on a full cluster means the rollout can only proceed by scheduling extra pods, and if they stay Pending, nothing ever advances.
  • Percentages round. maxUnavailable: 25% on 2 replicas rounds down to 0; the controller must surge, and needs room to do it.

4. Can a PodDisruptionBudget block a rollout?

Not directly. PodDisruptionBudgets don't block rolling updates on their own - deleting a pod because its ReplicaSet scaled down is not an "eviction" in the PDB sense. But PDBs absolutely deadlock the adjacent flows that stuck rollouts get tangled with: node drains during the rollout, cluster-autoscaler consolidation, and any controller that evicts rather than deletes.

The classic: maxUnavailable: 0 (or minAvailable: 100%) in a PDB, plus a node drain, plus a rollout whose new pods aren't ready yet. The drain waits on the PDB, the PDB waits on ready pods, the ready pods wait on the rollout, and the rollout is waiting on capacity the drain was supposed to free.

kubectl get pdb -A
kubectl describe pdb <name>   # look at ALLOWED DISRUPTIONS: 0

ALLOWED DISRUPTIONS: 0 with the rollout mid-flight is your deadlock signature. A PDB that permits zero disruptions is not protection - it's a standing outage of every maintenance operation, and it will fail the blast-radius question in any serious postmortem.

5. Rollback: the honest option

If the new version is the problem and the fix isn't a two-minute one:

kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name>              # to previous revision
kubectl rollout undo deployment/<name> --to-revision=4
kubectl rollout status deployment/<name>            # verify it completes

Two caveats worth knowing before you need them. rollout undo creates a new revision that copies the old template - history moves forward, which is what you want for auditability. And kubectl rollout restart does not fix a stuck rollout - it stacks another rollout on top of the wedged one. Restart is for "re-pull the image / re-read the config" cases, not for "the template is broken."

If GitOps owns the cluster, undo through Git, or Argo/Flux will cheerfully re-apply the broken version thirty seconds after you fixed it.

6. Verify like you mean it

A rollout isn't done when rollout status returns. It's done when:

kubectl get deployment <name>   # READY, UP-TO-DATE, AVAILABLE all agree
kubectl get rs -l app=<name>    # old RS scaled to 0
kubectl get endpoints <svc>     # endpoints repopulated

plus one real request through the front door. Declaring victory off the controller's counters alone - without checking the service actually serves - is one of the most common failure patterns we see in graded lab sessions, and it's exactly the wasted-confidence moment covered in common troubleshooting interview mistakes.

FAQ

Why does my rollout say ProgressDeadlineExceeded? No progress (no new pod became ready) for progressDeadlineSeconds - 600s by default. It's a condition, not an action: Kubernetes doesn't auto-rollback. Find why new pods aren't ready, fix or rollout undo.

Does kubectl rollout undo lose my history? No. Undo creates a new revision with the previous template. rollout history keeps up to revisionHistoryLimit old ReplicaSets (default 10) for exactly this purpose.

Can I pause a rollout instead of rolling back? Yes: kubectl rollout pause deployment/<name> freezes it (old and new pods keep serving), resume continues. Useful when you want to investigate the new pods without the controller churning underneath you.

Why is my StatefulSet update stuck when Deployments roll fine? StatefulSets update in reverse ordinal order and stop at the first pod that never becomes ready - one wedged pod halts the entire sequence. Same diagnosis (describe the stuck pod), but expect stricter ordering and check partition in the update strategy.