Debugging CrashLoopBackOff in Kubernetes: a field guide

CrashLoopBackOff is not an error. It's Kubernetes telling you: this container keeps dying, and I've stopped trying so hard. The actual error is one layer down, and the fastest engineers all use the same first move:

kubectl logs <pod> --previous

--previous matters. The current container may have zero log lines because it died before printing any; the previous instance is where the crash actually happened. If you only remember one flag from this post, remember that one.

What CrashLoopBackOff actually means

It's a status, not a diagnosis, and it describes the kubelet's behaviour, not your app's. The loop runs like this:

1. Your container exits. Any exit, clean or not, counts.
2. The pod's restartPolicy (Always by default) tells the kubelet to restart it.
3. It exits again. The kubelet starts backing off: it waits 10s before the next attempt, then 20s, 40s, 80s, and so on, capped at 5 minutes.
4. During that wait, kubectl get pods shows CrashLoopBackOff.

So the word you're reading describes step 4, the waiting. The failure happened in step 1, and nothing in the string CrashLoopBackOff tells you what it was. That's why the status alone is never actionable, and why every path below starts by going one layer down.

Two consequences worth knowing. The back-off timer resets once a container has run successfully for about 10 minutes, which is why a pod that crashes every 11 minutes never shows the status and quietly restarts forever instead. And a container that exits 0 still crash-loops under restartPolicy: Always, a batch job pasted into a Deployment does exactly this, and the logs look perfectly healthy.

Triage in three commands

kubectl logs <pod> --previous            # what the app said before it died
kubectl describe pod <pod>               # exit code, Last State, events
kubectl get events --sort-by=.lastTimestamp -n <ns>   # what the cluster did about it

describe is where the exit code lives, under Last State: Terminated. Read it before you change anything, it usually names the cause outright:

Anything above 128 is "killed by signal n", where n is the code minus 128. That single piece of arithmetic resolves more incidents than any dashboard.

Here are the eight causes that account for nearly every CrashLoopBackOff in the wild, with the fastest diagnostic path for each.

1. The application is exiting on purpose

A missing env var, an unreachable database, a config file that doesn't parse. The app starts, fails its own preconditions, and exits non-zero.

FATAL: could not connect to db: hostname "pg-primary" not found

Path: kubectl logs --previous shows the app's own error. Fix the config, not the pod. Check kubectl describe pod → Environment: and mounted ConfigMaps/Secrets against what the app expects.

2. OOMKilled

The container exceeds its memory limit and the kernel kills it. Logs often end mid-sentence.

Path: kubectl describe pod → Last State: Terminated, Reason: OOMKilled, Exit Code: 137. Raise the limit or fix the leak, which one, and how to tell them apart, is its own field guide. Exit code 137 = SIGKILL; if you see it, check OOM before anything else.

3. Failing liveness probe

The app is fine but slow to start; the liveness probe kills it before it's ready; repeat forever.

Path: kubectl describe pod → events say Liveness probe failed. The fix is almost never "disable the probe" - it's a startupProbe or a longer initialDelaySeconds. (Disabling the probe and never re-enabling it is exactly the kind of thing a blast-radius score catches.)

4. Bad command or entrypoint

command:/args: in the manifest override the image's entrypoint - with a typo, a wrong path, or a shell-ism that doesn't work without a shell.

Path: exit code 127 (command not found) or 126 (not executable) in kubectl describe pod. Compare the manifest against docker inspect of the image.

5. Missing or broken volume mount

A ConfigMap that doesn't exist, a Secret key that was renamed, a PVC stuck pending - the container can't start because a mount can't materialize.

Path: this usually shows as CreateContainerConfigError first, but a mount that succeeds with wrong contents becomes cause #1. kubectl describe pod events, then kubectl get configmap,secret and check the actual keys.

6. Crash in an init container

The main container is innocent; an init container is the one crash-looping.

Path: kubectl get pod shows Init:CrashLoopBackOff. Then kubectl logs <pod> -c <init-container> --previous.

7. Port already in use

Two containers in one pod binding the same port, or a container binding twice on restart.

Path: bind: address already in use in the logs. Check the pod spec for sidecars sharing the network namespace.

8. The image itself is wrong

Wrong tag pushed, latest moved under you, architecture mismatch (arm64 image, amd64 node, exec format error).

Path: logs show exec format error or the app version doesn't match what you expect. kubectl describe pod → Image: and compare digests, not tags. If the image never got pulled at all you'd be looking at ImagePullBackOff instead, a different status with a completely different cause list.

How to fix it without making it worse

Every cause above has a fast wrong answer, and they share a shape: suppress the symptom so the status goes green.

• Deleting the pod. It comes back, crashes again, and you've destroyed the --previous logs that would have told you why.
• Removing the liveness probe. The status clears because nothing is checking any more.
• Doubling the memory limit without reading the working set. Buys an hour if it's a leak, and hides the leak from whoever is on call next.
• restartPolicy: Never "to stop the noise". Now it fails silently.

None of these are fixes; they're ways of making the cluster stop mentioning the problem. Change one thing, then check the same evidence you started from, logs --previous, describe, the exit code, and confirm the state actually moved.

The meta-skill

Notice what the fast path always is: read what the system is telling you before changing anything. logs --previous, describe pod, exit codes. The engineers who fix these in five minutes aren't the ones who memorized this list - they're the ones with a diagnostic habit: hypothesis, evidence, then change.

That habit is measurable. It's what we grade, and it's what a Kubernetes troubleshooting interview should be testing in the first place.

FAQ

How do I fix CrashLoopBackOff? You don't fix the status, you fix the exit. Run kubectl logs <pod> --previous for the app's own last words, then kubectl describe pod <pod> for the exit code under Last State. The code names the category, 137 is memory, 127 is a bad command, 1 is the app rejecting its own config, and the category decides the fix.

Why are the logs empty for a CrashLoopBackOff pod? Because you're reading the wrong container. Without --previous, kubectl logs targets the current attempt, which may not have started yet during a back-off wait. If --previous is also empty, the process died before writing anything: check the exit code, and suspect the entrypoint, a missing mount, or an architecture mismatch rather than the application.

How long does Kubernetes wait between restarts? The kubelet backs off exponentially, roughly 10s, 20s, 40s, and so on, capped at 5 minutes between attempts. The timer resets once the container stays up for about 10 minutes, which is why an app that crashes on a long cycle restarts quietly instead of showing the status.

What is the difference between CrashLoopBackOff and ImagePullBackOff? CrashLoopBackOff means the container ran and died. ImagePullBackOff means it never ran, because the registry refused to hand over the image. Same back-off machinery, opposite halves of the lifecycle, the full map of which status points at which subsystem is in the pod status taxonomy.