Reviewing AI-written code: why we test it in a clean checkout

🎲 The risk with code you did not type

Vibe coding means accepting generated code because it seems to work. It is fast, and for a throwaway script it is fine. The trouble starts when the same habit reaches shared code. An agent works in your working tree, which holds files you changed and never committed, local settings and cached results. The tests it runs see all of that. A teammate who checks out the commit sees none of it.

🧾 One incident, in numbers

On 14 October 2026 an agent adopted a 1,123-line analytics module from a live server snapshot into git. In the working tree the module passed 56 of 56 tests. In a clean checkout of the same commit it failed 10 of 56, because it depended on uncommitted edits to a neighbouring file. The commit was reverted the same day. A second slip followed: the revert command has no quiet flag, the first attempt failed, and an amend then rewrote someone else's commit.

Nothing in the diff looked wrong. The failure was in what the diff did not contain.

🔍 The review routine

Reading the diff is necessary, and it is not enough. Our routine has five gates.

1. Scope. Does the diff touch only the files the task names? Unexpected paths, deleted files and changed modes are the first things to look for.
2. Reading. Can the reviewer explain what each new function does and what it returns on bad input? If not, the code is not accepted yet.
3. Clean checkout. Create a fresh worktree from the target branch, apply only the commit, and run the tests there. A pass in the working tree does not count.
4. Exit codes. Run the test command without a pipe. pytest | tail -1 && git commit commits on red, because the exit code belongs to tail.
5. Rollback ready. Know the command that undoes the change before you merge it, and check it on a throwaway branch.

🛠️ Practice: set up the clean-checkout gate

1. Pick one repository that agents write to.
2. Write a short script that creates a temporary worktree from the target branch, applies the candidate commit, runs the test suite and removes the worktree.
3. Make the script exit non-zero on any test failure and on an unexpected file in the status output.
4. Run it on the last five agent commits. Note how many fail.
5. Make the script a required step before merge. A pass in the author's tree is information, not approval.
6. Add the rule to your agent instructions: "show the clean-checkout result, not the local one".

✅ Check yourself

• Take a commit that depends on an uncommitted file. The script fails and names the missing import or file.
• Introduce a failing test and run the gate through a pipe. The gate still fails, because you read the exit code of the test command itself.
• Someone new can run the gate from the README in under five minutes.
• You can say which of the last ten agent commits touched a path the task did not name.

🚧 When this does NOT work

• Tests that do not test the behaviour. A clean checkout with weak tests gives a clean pass for broken code. Agents also write the tests they pass.
• Dependencies on the environment. A test that needs a database, a key or a host cannot pass in a fresh worktree. Mark it clearly and run it where those exist, and do not fix it by faking the data.
• Large diffs. Past a few hundred lines a human review turns into skimming. Ask for smaller changes.
• Generated code you will never touch again. A one-off script may not deserve the gate. Decide that on purpose.
• Review by the same model. A model that wrote the code often agrees with it. Use a deterministic check first, a different reviewer second.

📊 Evidence

➡️ Next

Write acceptance down before the work starts in Spec-driven development with existing task packets, and see why a claim of success is not proof in AI judging.

Sources

• AI judging: why “it says it is done” is not acceptance: acceptance by checks.
• Spec-driven development with existing task packets: handing work over with criteria.
• Agent task receipts: what they prove: what a receipt shows about a run.
• Recovery after a failed, stuck or uncertain run: what to do when a run goes wrong.