skip to content
‹ All posts

By Riya Madaan

PVC stuck in Pending and volume mount failures

A PVC stuck in Pending in Kubernetes: StorageClass typos, missing provisioners, WaitForFirstConsumer, access modes, and the mount failures that follow.

#kubernetes

A PVC pending in Kubernetes means one thing: no PersistentVolume has bound to the claim. The reasons range from "you typo'd the StorageClass" to "your cloud provider is refusing you a disk", and the claim itself will tell you which - if you ask it. First move, always:

kubectl describe pvc <name>

The Events: section at the bottom is the entire diagnosis, nine times out of ten. Here are the messages you'll see there, ordered from most common to least, and what each one means.

Symptom Likely cause First command
no persistent volumes available ... no storage class is set No default StorageClass kubectl get storageclass
storageclass ... not found Typo in storageClassName kubectl get pvc <name> -o jsonpath='{.spec.storageClassName}'
waiting for first consumer WaitForFirstConsumer, no pod scheduled yet kubectl get storageclass <name> -o jsonpath='{.volumeBindingMode}'
waiting for a volume to be created Provisioner missing or failing kubectl get pods -n kube-system | grep -i -e csi -e provisioner
ProvisioningFailed on ReadWriteMany Block storage only supports RWO kubectl describe pvc <name>
Pod Pending with volume node affinity conflict Volume in a different zone from the nodes kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}'
Multi-Attach error for volume Disk still attached to an old node kubectl describe pod <pod>

"no persistent volumes available for this claim and no storage class is set"

The claim has no storageClassName, the cluster has no default StorageClass, and there's no pre-provisioned PV waiting around to match. Nothing is even trying to create a volume.

kubectl get storageclass

Look for (default) next to a name. If nothing is marked default, either set one:

kubectl patch storageclass <name> \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

or set storageClassName explicitly in the claim. On managed clusters (EKS, GKE, AKS) a default usually exists; on kubeadm clusters it usually doesn't, and this is the number-one cause of a first PVC sitting in Pending forever. If you'd like to work through a stuck claim on a cluster that's actually broken, there's a PVC stuck Pending scenario in the practice labs.

"storageclass.storage.k8s.io "fastssd" not found"

The claim names a StorageClass that doesn't exist. Compare the claim against reality:

kubectl get pvc <name> -o jsonpath='{.spec.storageClassName}'
kubectl get storageclass

StorageClass names are case-sensitive and this field is immutable on the claim - you can't edit it in place. Delete and recreate the PVC with the right name. (If the pod using it is also wedged, see pod stuck Terminating and finalizers before you start force-deleting things.)

Why is my PVC Pending with "waiting for first consumer"?

Because nothing will be provisioned until a pod that uses the claim schedules, and this one is not a bug. The StorageClass has volumeBindingMode: WaitForFirstConsumer, which delays provisioning until a pod actually schedules - so the volume gets created in the same zone as the node the pod lands on. A PVC in Pending with this event and no pod referencing it is behaving exactly as designed.

If a pod does reference it and both are stuck, the problem is the pod's scheduling, not the storage. Go debug the pod: Pod stuck in Pending: the complete diagnostic tree.

"waiting for a volume to be created, either by external provisioner..."

The claim found its StorageClass, asked the provisioner for a volume, and nothing answered. Check that the provisioner actually runs:

kubectl get pods -n kube-system | grep -i -e csi -e provisioner
kubectl get storageclass <name> -o jsonpath='{.provisioner}'

Common causes: the CSI driver was never installed (again: bare kubeadm clusters), its pods are crash-looping, or its cloud credentials expired. The provisioner's own logs say why:

kubectl logs -n kube-system -l app=ebs-csi-controller -c csi-provisioner --tail=50

Cloud-side quota errors ("VolumeLimitExceeded", out of IOPS quota, unavailable disk type in that zone) show up here, not in the PVC events.

Why does a ReadWriteMany claim fail with ProvisioningFailed?

Because the provisioner behind it is block storage, and block storage only does ReadWriteOnce. Requesting ReadWriteMany from a block-storage provisioner is the classic. EBS, GCE PD, and Azure Disk are single-node block devices - they do RWO, full stop. The provisioner rejects the claim and the event says so.

If you genuinely need many pods writing one filesystem, you need file storage (EFS, Filestore, Azure Files, NFS, CephFS). If you don't - most workloads don't - change the claim to ReadWriteOnce and give each replica its own volume via a StatefulSet's volumeClaimTemplates.

Zone and topology mismatch

The volume exists in us-east-1a; every schedulable node is in us-east-1b. The PV carries a node-affinity constraint and the scheduler can't satisfy it, so the pod pends with volume node affinity conflict. Check where the volume actually lives:

kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}' | jq

WaitForFirstConsumer exists precisely to prevent this. If you're on Immediate binding in a multi-zone cluster, that's the fix - for the next volume. For this one, either bring up a node in the right zone or recreate the volume.

The mount-side failures: FailedMount and FailedAttachVolume

A bound PVC can still fail at pod start. These show in kubectl describe pod, not the PVC:

  • FailedAttachVolume / Multi-Attach error for volume - the disk is still attached to another node. Classic after a node dies ungracefully: the old node never detached, the pod rescheduled, and an RWO volume can't attach twice. If the old node is genuinely gone, the attach controller usually recovers in ~6 minutes; a NotReady node that's actually half-alive can hold the attachment indefinitely, and deleting the stuck pod on it (or the Node object, if it's truly dead) releases it.
  • FailedMount with a timeout - attach succeeded, mount didn't. Check the CSI node plugin on that specific node (kubectl get pods -n kube-system -o wide | grep csi), then kubelet logs on the node for the real filesystem error (bad fstype, filesystem corruption, secret for the mount missing).

FAQ

Why is my PVC Pending when the StorageClass exists and the provisioner is healthy? Most often WaitForFirstConsumer: nothing provisions until a pod that uses the claim schedules. Check kubectl get storageclass <name> -o jsonpath='{.volumeBindingMode}'. If it's WaitForFirstConsumer and no pod references the PVC, Pending is correct behavior.

Can I change the StorageClass or size of an existing PVC? storageClassName and access modes are immutable - recreate the claim. Size can only grow, and only if the StorageClass has allowVolumeExpansion: true; edit spec.resources.requests.storage upward and the CSI driver resizes, sometimes requiring a pod restart for the filesystem step.

Deleting the PVC - do I lose the data? Depends on the StorageClass reclaimPolicy. Delete (the usual default) destroys the underlying disk when the claim goes. Retain keeps the PV as Released, and you can manually rebind or salvage it. Check before deleting, not after: kubectl get pv <name> -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'.

My pod says "Multi-Attach error" but the other node is dead. How long do I wait? The attach/detach controller force-detaches roughly six minutes after the node is marked unhealthy and its pods are terminated. If the stale pod is stuck Terminating on the dead node, force-delete the pod first - the detach won't start while Kubernetes thinks the pod still runs there.