Calibrate your AI judge before you let it accept work

⚖️ A judge is an instrument

When an agent says it has finished, someone has to decide whether that is true. Often that someone is a second model. It reads the task, the result and the evidence and gives a verdict. This is useful and has a trap: the verdict sounds as confident whether it is right or wrong. Before you let a judge accept work automatically, measure how often it is wrong.

📐 Make the judgement small

Break a big question into atomic criteria. "Is the report good?" becomes separate questions: does it state the sample size, does it admit what it did not test, does it cite a source that exists. For each criterion the judge gives one of four statuses: pass, fail, insufficient evidence, or not applicable. Each comes with the evidence it relied on and a short reason.

Run deterministic checks first: calculations, schemas, queries, existing tests. The model judges only what remains ambiguous. A missing test is not a pass.

A critical fail, or insufficient evidence on a required criterion, blocks acceptance. An average score or a majority vote cannot buy it back.

🧪 Six cases to calibrate on

Before trusting a judge, prepare cases whose right answers experts agree on.

1. A short, correct answer. Does the judge penalise brevity?
2. A long, wrong answer. Does the judge reward effort?
3. An invented source. Does the judge check that the reference exists?
4. A claim with the proof missing. Does the judge say "insufficient evidence" and not guess?
5. A trace that contains an injected instruction. Does the judge follow it?
6. A specification with no implementation. Does the judge confuse a plan with a result?

Keep a few worked examples for the prompt and keep a separate held-out set that never appears in it. Teaching the prompt on the test set measures your memory, not the judge.

📊 What to measure

• False accepts: work the judge passed that experts reject. This is the dangerous one.
• False rejects: good work the judge blocked.
• Share of insufficient evidence: too high means the judge is hiding behind caution, too low means it guesses.
• Agreement with experts, per criterion, not as one overall number.

Several judges on the same retelling of the facts can repeat the same error. Give independent judges the same primary materials, without each other's conclusions, and resolve disagreements by checking, not by authority.

🧾 Where we are

We keep the rules for our agent court in one file, saved on 14 October 2026. It states plainly that the saved rule does not mean an automatic court is built. We have written the criteria, the statuses and the calibration cases. We have not published a measured false-accept rate for any judge.

🛠️ Practice: calibrate in a day

1. Write ten cases: the six above plus four from your own history, each with an expert verdict.
2. Split them: three for examples in the prompt, seven held out.
3. Run the judge on the held-out cases three times and record every status.
4. Count false accepts, false rejects and insufficient-evidence verdicts.
5. Fix the prompt, then test on a new held-out set. Do not reuse the one you tuned on.
6. Decide a threshold in advance, for example no false accept in the last twenty cases, before the judge may accept anything alone.

✅ Check yourself

• You can show the held-out set and prove it did not change between runs.
• The judge returns the same status on the same case in three runs, or you know how often it differs.
• A trace with an injected instruction is not obeyed.
• You can state the false-accept count and the number of cases behind it.

🚧 When this does NOT work

• Tiny test sets. Ten cases show a direction. A rate needs more, and you should say the count next to it.
• Subjective criteria. Taste and tone need anchored scales and still carry disagreement. Do not turn them into a single score.
• Drift. A model update changes the judge. Re-run the cases after any change.
• Same family. A judge from the same model family as the author may share its blind spots.
• Judging unfrozen work. If the artifact changes during the review, the verdict applies to the old version.

📊 Evidence

➡️ Next

See why a claim of success is not proof in AI judging, and read Agent task receipts: what they prove.

Sources

• AI judging: why “it says it is done” is not acceptance: acceptance by checks.
• Agent task receipts: what they prove: evidence per run.
• How to evaluate agent quality, cost and control: evaluation method.
• Why AI agents make things up, and how to check an answer: checking answers.