blog / debugging
Debugging methodology and field guides
How to narrow a failure fast: the USE method, reading logs as evidence, and the habits that separate a ten-minute fix from a two-hour one.
25 posts
Posts
Kubernetes NetworkPolicy debugging: when pods can't talk
Kubernetes NetworkPolicy not working - or working too well? A test method for silent packet drops: default-deny surprises, selector typos, DNS egress.
Zombie processes: what defunct really means
Zombie processes explained: what <defunct> means, why kill -9 does nothing, how to find the negligent parent, and when a zombie process on Linux matters.
PVC stuck in Pending and volume mount failures
A PVC stuck in Pending in Kubernetes: StorageClass typos, missing provisioners, WaitForFirstConsumer, access modes, and the mount failures that follow.
Requests and limits: the misconfigurations that page you
Kubernetes resource limits and requests, minus the folklore: CPU throttling, limit=request tradeoffs, missing requests, and the LimitRange safety net.
Linux DNS not resolving: resolv.conf and systemd-resolved
Linux DNS not resolving? Trace the real resolution path - nsswitch, /etc/hosts, resolv.conf, systemd-resolved's stub - and why dig works but curl fails.
How to run a live debugging interview that isn't torture
A practical guide to running a live debugging interview: environment prep, hint policy, think-aloud prompts, timeboxing, and a scoring sheet that works.
Ingress 502, 503, 504: which layer is lying to you
Kubernetes ingress 502, 503, and 504 errors decoded: which layer failed, which logs to read, and the fastest path from load balancer to pod to find it.
Evicted pods: node pressure and who gets killed first
Kubernetes pod evicted? How node-pressure eviction works, what the eviction message means, why QoS class decides who dies first, and how to clean up.
Address already in use: who owns this port?
bind: address already in use - find the process holding the port with ss and lsof, tell TIME_WAIT from a live listener, and fix it without rebooting.
Debugging Kubernetes DNS: from nslookup to CoreDNS
Kubernetes DNS troubleshooting from symptom to CoreDNS: reproduce with a debug pod, decode the ndots trap, read CoreDNS logs, check upstream resolvers.
Think-aloud debugging: narrating through an interview
Technical interview communication for debuggers: why narration scores, exact phrases to use, and how to avoid both dead silence and nervous rambling.
Rolling update stuck: maxUnavailable, PDBs, hung rollouts
Kubernetes rollout stuck? How to read rollout status, decode progressDeadlineExceeded, spot PDB deadlocks, and roll back without making it worse.
systemd service won't start: a working checklist
A systemd service failed to start - the checklist that finds why: systemctl status, journalctl -xeu, exit code 203, ExecStart paths, deps, and limits.
Node NotReady: what to check, in order
A Kubernetes node NotReady is a race against pod eviction. The ordered checklist: kubelet, CNI, certs, and node pressure - fastest check first.
Debugging under pressure: staying methodical on the clock
Debugging under pressure is a trainable skill: hypothesis discipline, timeboxing, note-taking, and panic recovery for interviews and 3 AM incidents alike.
HPA not scaling: metrics-server and the usual suspects
HPA not scaling? The ordered checklist: metrics-server health, missing resource requests, misread targets, behavior windows, and what the conditions mean.
Reading Linux OOM killer logs and preventing the next kill
The Linux OOM killer just ended your process. How to read the dmesg kill report, what oom_score means, and how cgroup limits stop the next kill.
Pod stuck in Pending: the complete diagnostic tree
Pod stuck in Pending? Kubernetes scheduler events decoded - insufficient resources, taints, affinity, PVC binding, quota, and the silent no-events case.
CreateContainerConfigError: fixing ConfigMap and Secret refs
CreateContainerConfigError decoded: every ConfigMap and Secret mistake that causes it, the exact describe events, and the fastest fix for each.
High load average: what it measures and how to debug it
High load average on Linux doesn't always mean busy CPUs. What load really counts, how to split CPU from I/O wait, and a vmstat/pidstat/iostat path.
OOMKilled and exit code 137: finding the real memory hog
OOMKilled Kubernetes pods and exit code 137: how to read Last State, use kubectl top, dodge JVM container traps, and decide raise-the-limit vs fix-the-leak.
Kubernetes pod status reference: every state and its cause
Every Kubernetes pod status, from Pending to Terminating: what each one means, which subsystem gave up, and the first command to run.
Disk full: find it, fix it, and the du/df mismatch
df says the disk is full but du can't find the files? A field guide to disk full errors on Linux: lsof +L1, deleted-but-open files, inodes, safe cleanup.
Fixing ImagePullBackOff: every cause, fastest check first
Every cause of kubernetes ImagePullBackOff, ordered fastest check first: image typos, missing tags, registry auth, Docker Hub rate limits, and network failures.
Debugging CrashLoopBackOff in Kubernetes: a field guide
What CrashLoopBackOff really means, how to read the exit code, and the 8 causes it comes from - each with the fastest diagnostic path first.