skip to content
‹ All posts

By Riya Madaan

How to grade an incident: process over outcome

Incident response evaluation should grade process, not luck. How to score diagnosis quality, wrong-turn recovery, verification, and blast radius.

#hiring

Two candidates debug the same broken cluster. Candidate A fixes it in twelve minutes. Candidate B runs out the clock. Easy call?

Read the command logs. Candidate A ran a command they found on Stack Overflow, which happened to work, and they can't tell you why. Along the way they deleted a namespace "to clean up." Candidate B formed three hypotheses, killed two of them with evidence, was closing in on the real cause methodically, and never once put the rest of the system at risk.

Candidate A got the outcome. Candidate B ran the process. If your incident response evaluation only records "fixed: yes/no," you just hired the coin flip and rejected the engineer.

Why incident response evaluation must grade process, not outcome

Outcomes in debugging are noisy. Whether a candidate lands the fix inside a 45-minute window depends on which hypothesis they happened to try first, whether they've seen this exact fault before, and luck. Across many incidents those effects wash out; in a single interview they dominate. Judging one session by its outcome is judging a poker player by one hand.

Process is the stable signal underneath. The engineer who observes before acting, hypothesizes before mutating, and verifies before declaring victory will beat the lucky guesser over a career of pages - even on days the guesser wins the hand. This is the same lesson the industry already learned about postmortems: blameless incident reviews grade decisions given the information available at the time, not results. Your interviews should meet the same bar as your retros.

Here's a rubric that does it, dimension by dimension. It works whether you're scoring a live lab, a whiteboard scenario, or a real on-call shadow.

Dimension 1: hypothesis quality

The core question: does evidence precede action?

Strong looks like: the candidate's first commands are read operations - kubectl describe, logs, events, journalctl - and their stated hypotheses reference what they found. Each hypothesis comes with a discriminating test: "if it's the image pull, the events will say so; if the events are clean, it's not that."

Weak looks like: mutation before observation. Restart it, scale it, delete it, re-apply it - the random-walk pattern where actions aren't tied to any theory of the failure. The tell is that when you ask "why did you try that?", the honest answer is "to see if it helps."

A useful anchor pair for scoring: a 5 states hypotheses explicitly and tests the most likely/cheapest first; a 1 makes three or more state-changing moves before the first evidence-gathering one.

Dimension 2: time-to-diagnosis and efficiency

Not "did they fix it fast" - "did they narrow it fast." Time-to-correct-diagnosis is worth more than time-to-fix, because in production the fix is often trivial once the cause is known.

Score the shape of the search: did each check eliminate a branch, or did they revisit the same dead end twice? An engineer who reads a pod stuck in Pending event log once and extracts everything in it is demonstrating a skill; one who runs the same describe five times hoping for different text is demonstrating its absence.

Be careful with this dimension: penalize wasted motion, not thoroughness. A candidate who spends two minutes confirming a hypothesis before a risky fix is being efficient, not slow.

Dimension 3: recovery from wrong turns

Every real incident contains wrong hypotheses. So the interesting datum is never whether the candidate goes down a wrong path - it's what happens when the path stops producing.

Strong: they notice quickly (the evidence contradicted the theory), say so plainly, and re-plan from what they've learned. Wrong turns leave a residue of eliminated causes, not a residue of half-applied fixes.

Weak: sunk-cost debugging. Twenty minutes on a disproven theory because they'd already invested ten. Or its cousin - silently abandoning the theory and the evidence, and starting to guess.

This dimension is nearly invisible in a Q&A interview and vivid in a timestamped command log, which is one reason recorded work samples grade so much better than conversations.

Dimension 4: verification

The difference between "it should work now" and "it works, and here's the evidence."

Strong: after the fix, the candidate re-runs the failing request, checks the pods are Ready and stay Ready, confirms the original symptom is gone - unprompted. The gold-standard move is re-testing the thing that was originally reported broken, not just the thing they changed.

Weak: declaring victory at the fix command. In production this is the engineer whose "resolved" incidents reopen an hour later.

Dimension 5: blast radius

What did they break, or risk, while fixing? Disabled a liveness probe to stop the restarts and never re-enabled it. Force-deleted a pod with a finalizer that existed for a reason. --force on a Terraform state lock without checking whether the holder was a live apply - the kind of judgment call we walk through in the state lock guide.

We think this dimension is underweighted almost everywhere, which is why blast radius gets its own score in our rubric. Senior engineers are distinguished less by what they know than by what they decline to break. An outcome-only evaluation actively rewards recklessness - the fastest fix is often the most destructive one.

Putting it together: weights and anchors

A workable default: hypothesis quality 25%, efficiency 15%, wrong-turn recovery 15%, verification 15%, blast radius 15%, and raw correctness - did the system recover - only around 30% or less of the total. (Yes, the thing most interviews treat as the whole score is a minority stake. That's the point.)

Two mechanics that keep it honest:

  • Written anchors per score level. "3 = formed hypotheses but tested them in an inefficient order" beats "3 = okay." If two graders can't independently land within a point of each other, your anchors are too vague.
  • Evidence citation. Every score should point at a moment: a timestamped line in the command log, a note from the live session, a quote from the whiteboard session. Ungrounded scores are vibes with a number attached. (This is a rule we enforce mechanically - our AI evaluation layer must cite timestamped transcript evidence for every score, and pass/fail itself comes from deterministic grader scripts, never from a model's opinion.)

If you want the fuller argument for anchored rubrics and how to calibrate graders against each other, that's its own post: structured interviews for infra roles.

FAQ

What is incident response evaluation in hiring?

Assessing a candidate by how they handle a failing system - live lab, simulation, or structured scenario - rather than by recall questions. Done well, it scores the diagnostic process (hypotheses, efficiency, recovery, verification, blast radius) against written anchors, with the fix itself only a minority of the grade.

Why not just check whether the candidate fixed the issue?

Because single-incident outcomes are mostly noise: a lucky Stack Overflow paste beats a sound process that ran out of clock. Outcome-only grading hires guessers, rejects methodical engineers, and rewards destructive shortcuts that happen to work.

How do I grade process without a recorded environment?

Have the interviewer log observations against the five dimensions in real time - first actions, stated hypotheses, reaction to disproof, verification steps - and score from the log immediately after. It's workable, but lossy and interviewer-dependent; a recorded session removes the dispute about what actually happened.

Should candidates be told the grading criteria in advance?

Yes. "We grade your process - hypotheses, verification, what you risk - not just whether you fix it" changes candidate behavior in exactly the direction you want to observe, and it's fairer. You lose nothing: process under pressure can't be faked by knowing it's being watched.


Faultybox runs your candidates through a real broken cluster and shows you exactly how they fixed it - command log included. Start free →