skip to content

How grading works

Every interview and practice run is graded by the same published rule. Code decides whether the fault is fixed. An AI layer scores how the candidate got there, and it has to show its evidence.

Solved or Not solved

When the candidate submits, grader scripts run outside the sandbox and check the cluster they left behind: did the symptom clear, was the real root cause fixed, and does the fix hold. Each check carries a weight. The verdict is Solved only when every check that matters (weight 2 or more) passes. The model never decides this, and it cannot overrule it.

Checks assert outcomes, not keystrokes, so any valid fix passes, not only the one in the answer key.

The five criteria

Each criterion is scored 0 to 4, then weighted into a score out of 100. The weights below are the default. Some labs use a preset for a different seniority, which keeps the same five criteria and shifts the weight between them.

CriterionWhat it measuresWeight
Fix correctnessWhether the fault is fixed, judged only by grader checks run outside the sandbox.30
DiagnosisWhether they followed evidence from the symptom to the cause before changing anything.25
Blast radiusWhat their changes did beyond the fix: deleted state, removed guards, widened privileges.20
VerificationWhether they checked their own fix before submitting.10
Root-cause explanationWhether they can explain the chain from cause to symptom, in their own notes.15

Who scores what

  • Fix correctness comes only from the grader checks. No model is involved.
  • Blast radius has a ceiling set by a published rule set: deleted persistent state, a removed guard, or widened privileges each cap the score. The AI layer may only score lower than that ceiling, never higher.
  • Diagnosis, Verification and Root-cause explanation are scored by an AI layer. Every score must quote a real command or file change, at a real timestamp. A citation that does not match something that happened is retried once; if it fails again, the AI scores are withheld and the report says so, rather than publishing numbers nobody can check.
  • The AI layer never sees the candidate’s name, email or company.

Recommendation levels

Every interview report carries one recommendation. It follows from the verdict, the score and the blast-radius findings, never from the model’s opinion of the candidate.

Advance
Solved, with a strong score and no major blast-radius finding.
Advance with reservations
Solved, but with a major blast-radius finding or a weaker score worth discussing.
Discuss
A mixed result: solved with a low score, or not solved with a strong one. Talk it through before deciding.
Do not advance
Not solved, and the score does not make up for it.
Do not advance on this signal
A critical blast-radius finding, such as deleted persistent state, a NetworkPolicy or quota switched off, or cluster-admin granted. The fix does not outweigh it.
Inconclusive
Not enough to judge: the recording is broken, or the candidate stopped very early without solving it.