skip to content
‹ All posts

By Riya Madaan

Zombie processes: what defunct really means

Zombie processes explained: what <defunct> means, why kill -9 does nothing, how to find the negligent parent, and when a zombie process on Linux matters.

#linux

A zombie process on Linux is not a stuck program, isn't using CPU, isn't holding memory worth mentioning, and cannot be killed - because it's already dead. <defunct> in ps output means the process has exited and the kernel is keeping only its obituary: a process table entry with the exit status, waiting for the parent to collect it via wait(). The zombie is not the bug. The parent that never called wait() is the bug. Everything about debugging zombies follows from that one sentence.

Symptom Likely cause First command
<defunct> entries in ps Exited children not yet reaped ps -eo state,pid,ppid,cmd | awk '$1=="Z"'
Hundreds of zombies sharing one PPID Parent missing a SIGCHLD handler or wait() loop ps -fp <ppid>
Zombies pile up while the parent looks idle Parent blocked on something else (pipe, full disk, NFS) cat /proc/<ppid>/status | grep -E 'State|Threads'
fork: retry: Resource temporarily unavailable PID table exhausted ls /proc | grep -c '^[0-9]'
Zombies accumulating inside a container App running as PID 1 doesn't reap orphans ps -o pid,ppid,stat,cmd (inside the container)

Confirm and count

ps -eo state,pid,ppid,cmd | awk '$1=="Z"'
# Z  40412  3117  [worker] <defunct>
# Z  40477  3117  [worker] <defunct>

ps -eo state= | grep -c Z      # just the count
top                            # header line: "Tasks: ... 214 zombie"

Two things to note immediately: the count and the PPID column. A handful of transient zombies that appear and vanish within seconds is normal life - every child briefly passes through the zombie state between exiting and being reaped. Hundreds of persistent zombies sharing one PPID is a bug in exactly one place.

Find the negligent parent

ps -fp 3117
# UID   PID  PPID  CMD
# app  3117     1  /usr/local/bin/job-runner

The parent is your actual suspect. It spawned children and never collected their exit statuses - a missing SIGCHLD handler, a wait() loop that died, an event loop that got wedged. Check whether the parent is itself healthy:

cat /proc/3117/status | grep -E 'State|Threads'
strace -p 3117 -e trace=wait4,waitid -f    # is it even trying to reap? (briefly)

If the parent is blocked on something unrelated - a dead pipe, a full disk, an NFS hang - the zombies are a symptom of that incident, and fixing it un-wedges the reaping.

How do I kill a zombie process?

You cannot kill a zombie. kill -9 sends SIGKILL to a process that no longer runs; there is nothing left to receive it. Your options:

  1. Nudge the parent: kill -CHLD 3117 politely asks it to reap. Works only if the parent has a functioning handler that got out of sync - rare, but free to try.
  2. Kill (or restart) the parent. When the parent dies, its zombies are re-parented to PID 1 (or the nearest subreaper), which reaps them promptly. This is the reliable fix:
systemctl restart job-runner    # or kill 3117, if something supervises it
ps -eo state= | grep -c Z       # verify: count drops to ~0

Then file the real fix - the parent's code needs a SIGCHLD handler or a waitpid(-1, …, WNOHANG) loop. Zombies always come back until the code does.

Are zombie processes harmful?

Only in numbers. Zombies consume no memory pages and no CPU - but each one holds a PID. The failure mode is PID (task) exhaustion:

cat /proc/sys/kernel/pid_max        # often 32768, or ~4M on modern systemd hosts
cat /proc/sys/kernel/threads-max
ls /proc | grep -c '^[0-9]'         # current PID usage

When the table fills, fork() fails with EAGAIN across the whole box - bash: fork: retry: Resource temporarily unavailable - and suddenly you can't even run the commands to debug it (pro tip: echo /proc/[0-9]* uses shell builtins, no fork needed). Zombie leaks share this blast radius with fork bombs and file-descriptor leaks; the sibling failure is too many open files, and both reward the same prevention: per-service caps (TasksMax= in systemd, pids cgroup controller) so one leaking service can't starve the host.

The container angle: PID 1 has responsibilities

In a container, your app is usually PID 1 - and PID 1 inherits every orphan in the PID namespace. Ordinary processes get their orphans handled by init; a containerized app that never expected to be init silently accumulates zombies from any process its children spawn and abandon (shell-outs, sidecars-of-convenience, ffmpeg invocations, browser workers).

The fixes are standard and cheap:

  • Run a minimal init as PID 1: docker run --init (injects tini), or bake tini/dumb-init into the image entrypoint. It does exactly two jobs: forward signals and reap zombies.
  • Kubernetes: shareProcessNamespace: true in the pod spec makes the pause container PID 1, which reaps for everyone in the pod.
  • Or make your app a subreaper (prctl(PR_SET_CHILD_SUBREAPER, 1)) and actually reap, the option for people who enjoy doing init's job by hand.

A container hitting its pod's pids limit fails in ways that look mysterious from the outside

  • exec probes failing, kubectl exec refusing - and lands you back in the CrashLoopBackOff diagnostic tree with an unusual root cause. Worth having seen once before it happens in production. (It's also a staple of hands-on interview scenarios precisely because kill -9-ing a zombie is the tell of someone who hasn't.) There's no zombie lab yet, but the practice labs put you on a broken Kubernetes cluster where the same rule holds: find the real culprit before you kill anything.

The 60-second version

ps -eo state,pid,ppid,cmd | awk '$1=="Z"' | awk '{print $3}' | sort | uniq -c | sort -rn

One line: zombie count grouped by parent PID. One dominant PPID → inspect that parent, restart it, verify the count drains, fix its reaping code. Distributed PPIDs with small counts → normal churn, close the ticket.

FAQ

Are zombie processes bad? A few short-lived ones: no, that's the normal exit path. Hundreds that persist: yes, but as a symptom - a parent isn't reaping - and as a slow PID leak that can eventually exhaust the process table and break fork() for the whole machine.

How do I kill a zombie process? You can't; it's already dead. Kill or restart its parent (found via the PPID column in ps -eo state,pid,ppid,cmd). The orphaned zombies get re-parented to init and reaped immediately.

Do zombies use memory or CPU? Effectively none - no address space, no runnable code. What each one holds is a slot in the process table (a PID). The resource they exhaust is kernel.pid_max, not RAM.

Why do containers accumulate zombies? Because the app runs as PID 1 in its namespace and inherits orphans it never reaps. Use docker run --init, tini/dumb-init as entrypoint, or shareProcessNamespace: true in Kubernetes so a real init handles reaping.