skip to content
‹ All posts

By Riya Madaan

Linux interview questions: scenarios from real 3 AM pages

Linux interview questions drawn from real 3 AM pages: disk full, runaway load, OOM kills. What strong answers sound like and how to grade the weak ones.

#hiring

The standard Linux interview questions have been stable for twenty years: "What does chmod 755 mean?" "Explain inodes." "What's the difference between a hard link and a soft link?" They were mediocre predictors in 2006 and they're worse now, because every answer is one tab away and the candidates you actually want stopped memorizing this stuff the day man pages became searchable.

Nobody gets paged because they forgot what an inode is. They get paged because the disk is full, the load average is 40, or the OOM killer just ate the database. So build your Linux interview questions out of those pages. Here's a scenario bank drawn from the incidents that actually happen, with rubric hints for grading the answers.

Scenario Linux interview questions from real pages

1. "Disk is at 100%. You delete a 40 GB log file. df still says 100%. What happened?"

The single best Linux interview question in existence, because it's a real page, it has a precise answer, and it can't be bluffed.

Strong answer: a process still holds the deleted file open, so the kernel can't free the blocks until the file descriptor closes. Finds the culprit with lsof +L1 (or lsof | grep deleted), then chooses between restarting the process, or - the senior move - truncating via the proc filesystem (: > /proc/<pid>/fd/<n>) to free space without killing anything. Extra credit for mentioning the other df/du mismatch causes: mounts shadowing full directories, and inode exhaustion (df -i) masquerading as "disk full" with free bytes.

Weak answer: "Delete more files" or "reboot the server." Rebooting would fix it, which is exactly why this question grades judgment, not just knowledge: the reboot trades a mystery for an outage. Our disk full field guide covers the full tree; a strong candidate reconstructs its spine unprompted.

2. "Load average is 60. CPU is 85% idle. Users are screaming. Explain, then debug."

Strong answer: knows that Linux load average counts runnable and uninterruptible (D-state) tasks - so high load with idle CPU means processes stuck waiting, almost always on I/O (or NFS, its favorite pathological case). Reaches for vmstat 1 (blocked column, wait time), iostat -x (device saturation), and ps to find D-state processes. Debugs the storage, not the CPU.

Weak answer: "Load of 60 means you need more cores." That's the answer of someone who has read about load average but never been paged by it. The mechanics are in our high load average guide - the question tests whether the candidate carries the model, not the definition.

3. "MySQL died at 3:12 AM. Nothing in the app logs. Where do you look, and what will it say?"

Strong answer: goes straight to the kernel: dmesg -T or journalctl -k around the timestamp, expecting OOM killer forensics - the "Out of memory: Killed process" line, the score table, which cgroup hit its limit. Then reasons about why: did the database genuinely leak, did something else balloon and the killer chose the biggest victim, are overcommit settings and cgroup limits sane. Distinguishes "MySQL was the problem" from "MySQL was the largest bystander."

Weak answer: "Check the MySQL error log" (already implied empty by the prompt, looping on exhausted evidence is itself a datum) or "just restart it and add a monitor." Restart-and-hope is how the 3:12 AM page becomes a 4:40 AM page. Reading kill logs properly is a skill worth its own page - we wrote it up in the OOM killer guide.

4. "A service won't start after a config change. systemctl start returns immediately with no error, but the service is dead."

Strong answer: systemctl status first, then journalctl -xeu <unit> for the actual failure. Knows the usual suspects: ExecStart path wrong, permissions, a missing dependency, an environment file that didn't load, Type= mismatch (a forking service with Type=simple "succeeds" and dies). Checks whether the process started and exited versus never started.

Weak answer: edits the config back without ever reading the journal. Undoing changes blindly is rollback theater - sometimes right, but worthless without knowing what failed.

5. "You need to free port 8080. ss -ltnp shows nothing listening, but binds still fail."

A smaller question, but it finds people with real scars: TIME_WAIT states, a process in a different network namespace (containers!), or the check itself run without privileges so the process column came back empty. Strong candidates ask how the bind fails and reach for ss -atnp, fuser, or a look inside the container netns. Weak candidates suggest rebooting again.

Grading: rubric hints that keep it fair

Across all five scenarios, score the same dimensions - not "did they name lsof +L1":

  • Evidence order. Kernel logs, journals, and process state before mutating anything. First-move-is-a-read is the strongest single marker of seniority.
  • Model correctness. Do they carry an accurate mental model (what load average counts, why deleted files hold space), or a folk model that survives until it's 3 AM?
  • Blast radius. Reboots, service restarts, and kill -9 all "work." Do they reach for the smallest intervention first, and can they articulate the tradeoff when they do choose the big hammer?
  • Verification. After the fix: does df actually drop, does load actually recover, does the service stay up? "Should be fine now" is a 2, not a 4.

Write anchors for each level before the interviews start, ask every candidate the same scenarios, and note evidence for each score. Gut feel doesn't survive a hiring debate; timestamps do.

From talking about pages to taking one

The honest limit of this whole question bank: it grades the story of debugging. The candidate who describes lsof +L1 fluently and the candidate who can actually drive it under time pressure overlap less than you'd hope - and the fluent-describer failure mode is getting worse now that any candidate can have an LLM whispering perfect answers in a remote screen.

The structural upgrade is to stop asking and start watching: a real Linux box with a real planted fault - the disk genuinely full, the service genuinely dead - a 45-minute time box, and a recording. The scenario bank above converts directly: every question on this page is a plantable fault. When you make that jump, grade it like an incident review, process over outcome - we wrote up exactly how in how to grade an incident, and our own rubric weights it the same way.

FAQ

What are the best Linux interview questions for sysadmin and SRE roles?

Scenarios built from real pages: the deleted-but-open file holding disk space, high load with idle CPU, OOM killer forensics in dmesg, a systemd service that dies silently. They're better than trivia because strong and weak answers sound different - one contains a method, the other contains a reboot.

How do I grade open-ended Linux scenario questions consistently?

Fix the scenarios, fix the dimensions (evidence order, model correctness, blast radius, verification), and write anchors for each score level before interviewing. Then note the specific moment that justified each score. Consistency comes from the sheet, not the interviewer's memory.

Are commands like awk and sed worth testing in interviews?

Lightly. A candidate who can pull a field out of a log with something - awk, cut, grep, even a Python one-liner - has a real everyday skill. Testing exotic sed flags tests whether they've memorized your favorite trick. Fluency matters; specific incantations don't.

Should Linux interviews be hands-on instead of verbal?

Verbal scenarios are a strong screen and cost nothing. For the decision round, hands-on wins: every scenario above can be planted as a real fault on a real box, and watching a candidate actually find the deleted file descriptor tells you what no description can. Record it so two graders can score the same session.


Faultybox runs your candidates through a real broken cluster and shows you exactly how they fixed it - command log included. Start free →