AI judging: why “it says it is done” is not acceptance

A task ends with text. Acceptance is a different event. Deterministic checks decide whether work is done — tests, builds, schema validation, a reproducible command — and a model's own summary is only a claim about the work.

This page describes the rule we apply, and how to run a judging pass without turning it into an expensive ritual.

🔹 The evidence ladder

Every statement in a completion report sits on one of three levels. They are not interchangeable:

A report that says "implemented" is not evidence that it works. A report that says "tested" without a command, a return code and an output is not evidence either.

🔹 What a judge is allowed to do

A useful judging pass is read-only:

• it may read the repository, the logs and the diff;
• it may run the acceptance command if that command is safe and cheap;
• it must not edit the work it is judging;
• it must not approve its own changes.

If the judge is also the author, you have not added a check — you have added a second opinion from the same source.

🔹 A judging brief that works

Goal: <one sentence>
Acceptance: <command> must exit 0 and print <marker>; <file> must contain <value>.
Evidence required: command, return code, output excerpt, UTC timestamp.
Forbidden: editing files; running paid services; claiming a check you did not run.
Verdict: PASS | FAIL | NOT_RUN, per acceptance criterion.

Three details make this brief useful:

1. Per-criterion verdicts. PASS, FAIL, NOT_RUN — with NOT_RUN never silently becoming PASS.
2. A return code, not "it worked". A pipeline whose last command is tail reports tail's exit code, not the test's.
3. A timestamp. Without one, "verified" is undated and cannot be trusted later.

📊 What it costs

A judging pass is a task, so it has a receipt like any other. The economics are simple:

• Judging a draft costs a fraction of producing it.
• Judging after a bad result has already been published costs rework plus trust.

Budget the judge into the plan. If a lane's output will be accepted on the strength of a check, that check is part of the task's price.

🚧 Failure modes worth watching

• Rubber-stamping. A judge that returns PASS on everything is worse than no judge: it launders bad work.
• Verdict matching the question. If the judge repeats the wording of the brief ("all criteria met"), ask for the command output, not the sentence.
• Unfalsifiable claims. "Improved performance" without a before/after number is not a result.
• A whitelist of known failures. A test that is red is fixed or removed, not added to a list of "known red" — otherwise the list hides the next defect.

🚧 When this does NOT work

• There is no machine-checkable acceptance. Then the honest verdict is NOT_RUN, and a human must decide.
• The check is expensive. Do not run a full heavy suite on every draft; run the narrow check, and schedule the heavy one.
• You need a legal sign-off. A model's verdict is not an audit opinion and cannot stand in for one.

➡️ Next

• Splitting work across agents — where judging fits in a parallel run.
• Sources and verification — how this wiki marks its own claims.
• Costs, budgets and limits — pricing the check.