By Riya Madaan
Deterministic grading, or: never let an LLM decide pass/fail
LLM evaluation reliability is a real problem, so we designed around it: deterministic scripts decide pass/fail, AI only explains with cited evidence.
We build an AI-heavy product, and we will not let an LLM decide whether a candidate passes.
That sentence confuses people, so this post is the long version. It's about LLM evaluation reliability - the awkward fact that language models are inconsistent judges - and about a trust architecture that takes the fact seriously instead of hoping it away. The rule is simple: scripts decide, AI explains. Everything else follows from it.
The LLM evaluation reliability problem
The industry's default move right now is "LLM-as-judge": show a model the candidate's work, get back a score. It's cheap, it scales, and it produces numbers with a straight face. It has three failure modes that should terrify anyone using those numbers to reject people.
It's non-deterministic. Run the same transcript through the same judge twice and you can get different scores. A grade that changes on re-roll isn't a measurement; it's a weather report.
It's persuadable. LLM judges reward confident narration, tidy formatting, and fluent explanation - which correlate with communication skill, not debugging skill. A candidate who talks like a postmortem reads better than one who quietly fixes the thing. Worse, transcripts contain candidate-written text, and a judge that reads free text can be steered by it.
It's unauditable. When a rejected candidate asks "why?", "the model felt the diagnosis was weak" is not an answer you can defend - not to the candidate, not to your own engineers, and not to anyone reviewing your hiring process later. A score with no traceable basis is a liability wearing a number.
None of this means LLMs are useless for evaluation. It means they're miscast. They're being asked to be the judge when they should be the court reporter.
Deterministic grading: the script decides
Whether a system recovered is not a matter of opinion. Either the deployment is serving traffic or it isn't. Either the data survived or it didn't. Either the certificate validates or your cluster is still on fire.
So the pass/fail verdict at Faultybox comes from grader scripts - deterministic checks that run from outside the candidate's sandbox against ground truth. Did the pods come back? Does the service answer? Does the fix survive the pod being recreated? Run them a thousand times, get the same answer a thousand times. No model in the loop, no temperature, no vibes. The candidate can't talk their way past a script, and neither can a model the candidate is using - which is one reason AI-allowed interviews don't scare us.
This is the same trust structure as your CI pipeline. You wouldn't merge to main because an LLM read your diff and felt good about it. Tests decide; reviews explain. Interviews deserve the same rigor, because the stakes are a person's job.
Where the AI actually belongs
Correctness is only about 30% of our default rubric, because the outcome was never the interesting part. The interesting part is judgment: diagnosis quality, blast radius, verification, root-cause explanation. Judgment lives in the transcript, and reading transcripts at scale is exactly what LLMs are good at.
So the AI layer scores judgment - but under two constraints that change everything:
Anchored rubrics. The model doesn't invent standards. It scores against written anchors - concrete descriptions of what a 2 and a 4 look like for each dimension, versioned and pinned per interview. The rubric is the law; the model applies it.
Mandatory citations. Every score must cite timestamped evidence from the command log. Not "the candidate diagnosed efficiently" but "formed the correct hypothesis at 04:12 after reading the probe config, confirmed via endpoint check at 06:40." A claim without a timestamp doesn't ship.
The citation requirement is the load-bearing part. It converts the AI's output from verdict to argument - and arguments can be checked. A hiring manager who doubts a score goes to that timestamp in the command log and reads what actually happened. If the evidence doesn't support the claim, the failure is visible, not buried in a number. That's why evidence beats scores: the score is an index into evidence, never a substitute for it.
Why the architecture, not the model, is the trust story
Models will improve. Judges will get more consistent. Doesn't matter - the architecture is still right, for the same reason you still write tests when your engineers get better. Decisions that affect people's livelihoods should rest on checks that are reproducible, auditable, and immune to persuasion, with interpretation layered on top and pinned to evidence. That division of labor is stable no matter how good the interpreter gets.
The alternative - a single opaque score from a model, no citations, no command log - asks everyone to trust the number. Candidates won't. Hiring managers quietly don't. And when the number is wrong, nobody can even tell.
Scripts decide. AI explains, with receipts. Anything else is astrology with an API.
Faultybox runs your candidates through a real broken cluster and shows you exactly how they fixed it - command log included. Start free →