skip to content

blog / debugging

Debugging methodology and field guides

How to narrow a failure fast: the USE method, reading logs as evidence, and the habits that separate a ten-minute fix from a two-hour one.

25 posts

Posts

Kubernetes NetworkPolicy debugging: when pods can't talk

Kubernetes NetworkPolicy not working - or working too well? A test method for silent packet drops: default-deny surprises, selector typos, DNS egress.

#kubernetes

Zombie processes: what defunct really means

Zombie processes explained: what <defunct> means, why kill -9 does nothing, how to find the negligent parent, and when a zombie process on Linux matters.

#linux

PVC stuck in Pending and volume mount failures

A PVC stuck in Pending in Kubernetes: StorageClass typos, missing provisioners, WaitForFirstConsumer, access modes, and the mount failures that follow.

#kubernetes

Requests and limits: the misconfigurations that page you

Kubernetes resource limits and requests, minus the folklore: CPU throttling, limit=request tradeoffs, missing requests, and the LimitRange safety net.

#kubernetes

Linux DNS not resolving: resolv.conf and systemd-resolved

Linux DNS not resolving? Trace the real resolution path - nsswitch, /etc/hosts, resolv.conf, systemd-resolved's stub - and why dig works but curl fails.

#linux

How to run a live debugging interview that isn't torture

A practical guide to running a live debugging interview: environment prep, hint policy, think-aloud prompts, timeboxing, and a scoring sheet that works.

#hiring

Ingress 502, 503, 504: which layer is lying to you

Kubernetes ingress 502, 503, and 504 errors decoded: which layer failed, which logs to read, and the fastest path from load balancer to pod to find it.

#kubernetes

Evicted pods: node pressure and who gets killed first

Kubernetes pod evicted? How node-pressure eviction works, what the eviction message means, why QoS class decides who dies first, and how to clean up.

#kubernetes

Address already in use: who owns this port?

bind: address already in use - find the process holding the port with ss and lsof, tell TIME_WAIT from a live listener, and fix it without rebooting.

#linux

Debugging Kubernetes DNS: from nslookup to CoreDNS

Kubernetes DNS troubleshooting from symptom to CoreDNS: reproduce with a debug pod, decode the ndots trap, read CoreDNS logs, check upstream resolvers.

#kubernetes

Think-aloud debugging: narrating through an interview

Technical interview communication for debuggers: why narration scores, exact phrases to use, and how to avoid both dead silence and nervous rambling.

#interview-prep

Rolling update stuck: maxUnavailable, PDBs, hung rollouts

Kubernetes rollout stuck? How to read rollout status, decode progressDeadlineExceeded, spot PDB deadlocks, and roll back without making it worse.

#kubernetes

systemd service won't start: a working checklist

A systemd service failed to start - the checklist that finds why: systemctl status, journalctl -xeu, exit code 203, ExecStart paths, deps, and limits.

#linux

Node NotReady: what to check, in order

A Kubernetes node NotReady is a race against pod eviction. The ordered checklist: kubelet, CNI, certs, and node pressure - fastest check first.

#kubernetes

Debugging under pressure: staying methodical on the clock

Debugging under pressure is a trainable skill: hypothesis discipline, timeboxing, note-taking, and panic recovery for interviews and 3 AM incidents alike.

#interview-prep

HPA not scaling: metrics-server and the usual suspects

HPA not scaling? The ordered checklist: metrics-server health, missing resource requests, misread targets, behavior windows, and what the conditions mean.

#kubernetes

Reading Linux OOM killer logs and preventing the next kill

The Linux OOM killer just ended your process. How to read the dmesg kill report, what oom_score means, and how cgroup limits stop the next kill.

#linux

Pod stuck in Pending: the complete diagnostic tree

Pod stuck in Pending? Kubernetes scheduler events decoded - insufficient resources, taints, affinity, PVC binding, quota, and the silent no-events case.

#kubernetes

CreateContainerConfigError: fixing ConfigMap and Secret refs

CreateContainerConfigError decoded: every ConfigMap and Secret mistake that causes it, the exact describe events, and the fastest fix for each.

#kubernetes

High load average: what it measures and how to debug it

High load average on Linux doesn't always mean busy CPUs. What load really counts, how to split CPU from I/O wait, and a vmstat/pidstat/iostat path.

#linux

OOMKilled and exit code 137: finding the real memory hog

OOMKilled Kubernetes pods and exit code 137: how to read Last State, use kubectl top, dodge JVM container traps, and decide raise-the-limit vs fix-the-leak.

#kubernetes

Kubernetes pod status reference: every state and its cause

Every Kubernetes pod status, from Pending to Terminating: what each one means, which subsystem gave up, and the first command to run.

#kubernetes

Disk full: find it, fix it, and the du/df mismatch

df says the disk is full but du can't find the files? A field guide to disk full errors on Linux: lsof +L1, deleted-but-open files, inodes, safe cleanup.

#linux

Fixing ImagePullBackOff: every cause, fastest check first

Every cause of kubernetes ImagePullBackOff, ordered fastest check first: image typos, missing tags, registry auth, Docker Hub rate limits, and network failures.

#kubernetes

Debugging CrashLoopBackOff in Kubernetes: a field guide

What CrashLoopBackOff really means, how to read the exit code, and the 8 causes it comes from - each with the fastest diagnostic path first.

#kubernetes