High load average: what it actually measures and how to debug it

A high load average on Linux is the most misread number in operations. The alert says load is 64, the box has 16 cores, someone declares "CPU is pegged" - and then top shows the CPUs 80% idle. Both facts are true, because Linux load average doesn't measure CPU. It measures demand: the number of tasks that are either running, waiting to run, or stuck in uninterruptible sleep - usually waiting on disk I/O or NFS.

That last clause is the whole game. On most Unixes load = CPU queue. On Linux, tasks in D state (uninterruptible, typically blocked on I/O) count too. So "load 64 on 16 cores" has two completely different diagnoses - a CPU stampede or an I/O stall - and the fix for one does nothing for the other.

Step 0: normalize by CPU count

uptime
# 14:02:11 up 41 days,  3:12,  1 user,  load average: 64.02, 31.17, 12.06
nproc
# 16

Load is an absolute count, not a percentage. Load 12 on 16 cores is a calm Tuesday; load 12 on 2 cores is an incident. Also read the three numbers as a trend: 1-, 5-, and 15-minute exponentially damped averages. 64, 31, 12 means it's getting worse right now; 12, 31, 64 means the storm is passing and you're mostly doing forensics.

Step 1: CPU or I/O? One command decides

vmstat 1 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
# 41  2      0 812340  94520 6123400   0    0     3    12 4200 9800 88 10  1  1  0

Read two columns and two more:

• r - tasks runnable (running + waiting for CPU). r persistently above core count → CPU saturation.
• b - tasks blocked in uninterruptible sleep. High b → I/O (or lock) stall.
• us/sy high with wa low → real CPU work (or a syscall storm if sy dominates).
• wa high with b high and idle CPUs → the disk is the bottleneck, not the processor.

This single split determines your entire next hour. Don't skip it.

Path A: high load average from CPU

Find who:

top -o %CPU          # quick look; press 1 to see per-core
pidstat 1 5          # per-process CPU over time, better for bursty offenders

Then decide why: a runaway process (one PID at 1600%), too much legitimate work (many processes each modest - capacity problem), or a fork storm:

vmstat -s | grep forks      # snapshot; run twice, diff it
pidstat -w 1                # context switches; voluntary vs involuntary

High involuntary context switches across many processes = more runnable tasks than CPUs, i.e. genuine saturation. If sy (system time) is the bulk, profile the syscalls before blaming the app's math: strace -c -p <pid> for a quick histogram (briefly - it slows the target) or perf top if it's installed.

Path B: high load average from D-state tasks

Idle CPUs, high load, b column nonzero. Find the blocked tasks:

ps -eo state,pid,ppid,wchan:32,cmd | awk '$1=="D"'

wchan tells you what they're waiting on (e.g. io_schedule, rpc_wait_bit_killable, that one means NFS). Then confirm the device is actually saturated:

iostat -xz 1
# Device  r/s   w/s   rkB/s  wkB/s  await  aqu-sz  %util
# nvme0n1 12.0 890.4  48.0 113974.2  38.20   34.1  99.6

%util near 100 with await far above the device's normal latency = the disk is the queue. Now find the process generating the I/O:

pidstat -d 1 5       # per-process kB read/written
iotop -oPa           # if installed: cumulative I/O per process

Typical endings for this story: a backup or find walking the disk, a database checkpoint or compaction, a log-flood filling the page cache with dirty pages, swap thrashing (check si/ so in vmstat - nonzero means memory pressure, and you may be one bad allocation away from the OOM killer), or an NFS server on the other end having its own incident. NFS deserves special mention: an unreachable hard-mounted NFS server can pin dozens of processes in D state and send load to absurd numbers while the machine does nothing.

Step 3: widen the lens before you fix

A load number alone never tells you the fix. Before acting, spend two minutes running the whole-system sweep - the USE method gives you the checklist (utilization, saturation, errors for each resource) so you don't tunnel on the first red number you saw. And if this box is a Kubernetes node, remember that CPU pressure shows up one layer up as throttling and evictions before it shows up as node load - the requests and limits failure modes are the same incident wearing a different costume.

Fixes map cleanly once you know the branch:

• CPU, one offender → fix/kill/renice the offender (renice, cgroup CPU quota).
• CPU, everything busy → capacity: more cores, fewer workers, or admission control.
• I/O, one offender → throttle it (ionice, cgroup io.max) or move the work off-peak.
• I/O, device dying → check dmesg for resets and SMART for media errors; that's not a tuning problem.

Verify

Load average is a lagging indicator - the 1-minute number takes minutes to fall after the cause is gone. Verify on the leading metrics instead: vmstat's r and b back near zero, iostat %util back to baseline, then watch load drain.

FAQ

What is a "good" load average? As a rough rule, sustained load below the core count means tasks aren't queueing. Short spikes above it are normal. Sustained load at 2–3× cores means work is waiting - for CPU or disk, which is what the vmstat split tells you.

Why is my load high but CPU idle? Tasks in uninterruptible (D) sleep count toward Linux load. That's almost always disk I/O, NFS, or (rarer) a kernel lock. vmstat's b column and ps filtered for state D confirm it; iostat -xz 1 finds the saturated device.

Do the three load numbers mean 1, 5, and 15 minutes exactly? They're exponentially damped moving averages over those windows, not simple means - good for trend direction ("is it climbing or draining?"), bad for precise arithmetic.

Can I kill a process stuck in D state? Not usually - it won't receive the signal until the I/O it's waiting on completes or fails. Fix the underlying I/O (restore the NFS server, unstick the device); the processes then either proceed or die on their pending signals.