CreateContainerConfigError: the ConfigMap/Secret mistakes behind it

CreateContainerConfigError is one of the most honest statuses Kubernetes has. It means: the image pulled fine, the node is fine, the scheduler did its job - but when the kubelet tried to assemble the container's configuration, something it needed didn't exist. In practice that something is a ConfigMap or Secret, referenced by a name or key that's wrong, missing, or in the other namespace.

The status alone doesn't tell you which. The event does:

kubectl describe pod <pod> | sed -n '/Events:/,$p'

Warning  Failed  kubelet  Error: configmap "app-config" not found

That one line is the whole diagnosis. Here's the taxonomy of what it can say, and what each variant means.

Decoding the CreateContainerConfigError event messages

configmap "X" not found / secret "X" not found

The pod references an object that doesn't exist in this namespace. Verify:

kubectl get configmap,secret -n <namespace>
kubectl get pod <pod> -o yaml | grep -A3 -E 'configMapRef|secretRef|configMapKeyRef|secretKeyRef'

The three usual causes, in order of frequency:

1. Namespace mismatch. The ConfigMap exists - in default, while the pod runs in prod. ConfigMaps and Secrets are namespaced; pods can only reference objects in their own namespace. kubectl get configmap app-config -A settles it in one command.
2. Deploy ordering. The Deployment applied before the ConfigMap did (a Helm hook mishap, a kustomize overlay that dropped a resource, CI applying manifests in the wrong order). The pod will recover on its own once the object appears - the kubelet retries.
3. A rename that missed a reference. Someone renamed app-config to app-config-v2 and updated four of five references.

couldn't find key X in ConfigMap Y / couldn't find key X in Secret Y

The object exists but the specific key doesn't. This is the subtler failure because kubectl get configmap looks fine at a glance. Compare actual keys against expected:

kubectl get configmap app-config -o jsonpath='{.data}' | jq 'keys'
kubectl get secret app-secrets -o jsonpath='{.data}' | jq 'keys'

Watch for case sensitivity (DATABASE_URL vs database_url) and for Secrets created with --from-file, where the key is the filename, extension included: --from-file=tls.crt creates key tls.crt, not tls.

secret "X" not found - but only on some nodes

If it's an imagePullSecret that's missing you get ImagePullBackOff instead, so this class is genuinely about env/volume references. But a pod that fails on one node and runs on another usually means you're looking at a stale pod from an old ReplicaSet with old references. Check kubectl get rs and confirm which template the failing pod came from.

The optional flag: making references non-fatal on purpose

Both env and volume references accept optional: true:

envFrom:
  - configMapRef:
      name: feature-flags
      optional: true

With optional: true, a missing ConfigMap yields an empty env instead of CreateContainerConfigError. This is the right tool for genuinely optional config - and a foot-gun when someone adds it to silence an error. The pod starts, the app runs without the config it needed, and you've traded a loud, cheap failure for a quiet, expensive one. If you add optional: true mid-incident to get pods running, write it down and revert it - the config is still missing, you've just stopped being told.

ConfigMap volumes vs env vars: different failure modes

The same missing object fails differently depending on how it's consumed:

So if you're staring at ContainerCreating with a MountVolume.SetUp failed ... configmap "app-config" not found event, it's the same disease with a different presentation - volume materialization happens earlier in the container lifecycle than env resolution. And note that a mount that succeeds with wrong contents doesn't error at all: the app starts, misreads its config, and exits - which lands you in CrashLoopBackOff territory instead.

Projected volumes fail the same way as ConfigMap volumes, with one addition: a projected serviceAccountToken source can fail if the ServiceAccount doesn't exist - the event names it.

The five-minute diagnostic path

# 1. What exactly is missing?
kubectl describe pod <pod> | sed -n '/Events:/,$p'

# 2. Does the object exist in THIS namespace?
kubectl get configmap,secret -n <namespace>

# 3. If it exists: do the keys match?
kubectl get configmap <name> -o jsonpath='{.data}' | jq 'keys'

# 4. What does the pod actually reference?
kubectl get pod <pod> -o yaml | grep -B2 -A4 -E 'configMap|secret'

# 5. After fixing: confirm recovery (kubelet retries automatically)
kubectl get pods -w

No restart needed - once the object or key exists, the kubelet's next sync starts the container. If the pod doesn't recover within a minute of the fix, you fixed the wrong namespace or the wrong key. Re-read the event; it's still telling the truth.

This exact failure - a renamed Secret key behind a CreateContainerConfigError - is one of our practice scenarios, and it's a reliable separator: the fast transcripts read the event first; the slow ones restart the Deployment three times before running describe.

Related statuses that get misdiagnosed as config errors

• CreateContainerError (no "Config") is different: the config resolved fine but the runtime couldn't create the container - bad entrypoint, runtime failure, or a container name collision on the node.
• Pending with unbound PVCs is a storage problem; see PVC stuck in Pending.
• Exit code 1 immediately after fixing the config means the app got the config and rejected it - see Kubernetes exit codes explained for reading what the app is telling you.

FAQ

Does CreateContainerConfigError retry on its own? Yes. The kubelet retries container creation on every sync loop. Create the missing object (or fix the key) and the pod starts without any restart, rollout, or delete.

Why does my pod work in staging but hit CreateContainerConfigError in prod? Because ConfigMaps and Secrets are per-namespace and per-cluster. The reference in the manifest is identical; the object it points to has to exist in each environment separately. Diff them: kubectl get configmap app-config -o yaml --context staging vs prod.

Is CreateContainerConfigError ever caused by RBAC? Not for the pod itself - the kubelet reads the ConfigMap/Secret, not your user. But the thing that was supposed to create the object (an operator, external-secrets, a CI job) may have been blocked by RBAC, which is why the object is missing. Check that controller's logs.

How do I prevent this class of failure? Deploy config and workload atomically (one Helm release, one kustomization), gate rollouts on kubectl rollout status, and avoid optional: true unless the config is truly optional.