By Riya Madaan
How to practice Linux troubleshooting deliberately
Practice Linux troubleshooting the deliberate way: break-it-yourself drills, a skill ladder from beginner to on-call ready, and feedback loops that stick.
Ask an engineer how they got good at Linux troubleshooting and you'll usually get some version of "years of production incidents." True, and useless as advice - you can't schedule incidents. But you can manufacture them. If you want to practice Linux troubleshooting in a way that actually compounds, you need the same three ingredients as any deliberate practice: problems slightly beyond your level, immediate feedback, and repetition with variation.
Reading man pages gives you none of those. A broken VM gives you all three.
What do I need to practice Linux troubleshooting?
A machine you can wreck without ceremony. Setting one up takes ten minutes, once:
# Any of these works - pick the one you'll actually use
multipass launch -n lab # Ubuntu VM, trivially rebuildable
# or: a $5/month cloud VM you snapshot and restore
# or: docker run -it --name lab ubuntu:24.04 bash (fine for many drills)
The requirement is psychological, not technical: rebuilding must be so cheap that you feel zero hesitation before trying something destructive. Hesitation is the enemy of practice volume.
Break-it-yourself drills to practice Linux troubleshooting
The core method: plant a fault, walk away for an hour (or better, have a friend plant it), then debug it under a 20-minute timer as if paged. Each drill below names the sabotage and the skill it builds.
# 1. Disk full - the classic
fallocate -l 8G /var/log/bigfile # find it with df, du, ncdu
# 2. Disk "full" but du disagrees - deleted-but-open file
fallocate -l 8G /tmp/ghost && tail -f /tmp/ghost & rm /tmp/ghost
# learn: lsof +L1, why df and du diverge
# 3. Inode exhaustion - df says space is fine
mkdir /tmp/inodes && cd /tmp/inodes && touch f{1..500000}
# learn: df -i, the "No space left" lie
# 4. Runaway load
stress-ng --cpu 8 --io 4 --timeout 600 &
# learn: load vs CPU%, vmstat, pidstat - see our load average guide
# 5. Port conflict
python3 -m http.server 8080 & # then try starting "your" service on 8080
# learn: ss -ltnp, fuser, who owns a port
# 6. Broken systemd service
sed -i 's|ExecStart=/usr|ExecStart=/usrr|' /etc/systemd/system/myapp.service
systemctl daemon-reload && systemctl restart myapp
# learn: systemctl status → journalctl -xeu → reading unit files
# 7. DNS sabotage
echo "nameserver 203.0.113.1" > /etc/resolv.conf
# learn: resolvectl, the resolution path, stub resolvers
# 8. Permission trap
chmod 600 /var/www/html/index.html # as the wrong owner
# learn: EACCES in logs, ls -l habits, effective UIDs
Rotate through these, then re-plant them with variations (different filesystem, different service, a cron job that re-breaks it). The variation matters: you're training the diagnostic loop, not memorizing eight answers.
Details for the deeper versions of several of these live in our field guides, disk full and the du/df mismatch and high load average are good companions to drills 1-4.
Why doesn't more practice make me faster?
Because practice without review is just repetition. After every drill, three questions, written down:
- What was my path? List the commands in order. Honestly.
- What was the shortest path? Now that you know the answer, what three commands would have found it?
- What's the delta? That gap - the wandering - is the thing to shrink next time.
This ten-minute review is where the skill actually forms. Engineers who skip it plateau; they've debugged the same disk-full fault twelve times with the same wandering path.
If you want the review done to you: SadServers is free and excellent - pre-broken servers with a defined pass condition, faults you didn't plant, so the diagnosis is real. KillerCoda has free interactive scenarios too. Faultybox is the graded tier of the same idea, though its practice labs are Kubernetes clusters, not Linux boxes: a real broken cluster, and afterward a report with your full timestamped command log and a review of your process - diagnosis quality, efficiency, what you risked along the way - which is the feedback loop most self-taught debuggers never get.
The skill ladder
Self-assess honestly; practice at the rung you fail, not the rung that feels good:
- Rung 1 - Fluency. You can use
ps,top,df,du,ss,journalctl,dmesg,lsofwithout looking up flags. - Rung 2 - Single-fault diagnosis. Given one planted fault from a known category, you find it in under 20 minutes.
- Rung 3 - Unknown-fault diagnosis. Faults you didn't plant, category unknown (SadServers level). You run a method - observe, hypothesize, test - instead of a checklist.
- Rung 4 - Pressure and narration. Same as rung 3, but timed and narrated out loud, the way an interview or a real page works. The method holds even when the clock is running.
- Rung 5 - Judgment. You choose safe paths: you notice when a fix is destructive, you verify before declaring victory, you clean up. This is the rung interviews and blast-radius scoring actually measure.
A realistic schedule: three 30-minute drills a week gets a working engineer from rung 1 to rung 4 in about two months.
FAQ
Can I practice Linux troubleshooting without a homelab? Yes. A free-tier cloud VM, Multipass on your laptop, or even a container covers almost every drill above. SadServers and KillerCoda run entirely in the browser. The barrier is habit, not hardware.
How is this different from just studying for the LFCS or RHCSA? Cert prep teaches you to perform known tasks; these drills teach you to diagnose unknown faults. Both are useful, but interviews and pagers test the second one. The cert task list is actually a decent source of things to sabotage, though.
How long until this shows up in interviews? Usually within a month of consistent drills, because the visible behaviors - going to evidence first, stating hypotheses, verifying - appear early. Depth keeps compounding for years, but method is learnable fast.
Should I practice on my own workstation? Plant faults only in disposable environments. Half the value of a lab is the fearlessness, and you can't be fearless on the machine your life is on. (Also: drill 3 on your real disk will ruin your afternoon.)