Short answer
An agent's summary is a claim, and research shows frontier models sometimes game tests to make claims true. A review pipeline should accept only evidence the harness produced itself: commands it ran, exit codes it recorded, test files it checked for edits.
The agent finishes and writes a tidy report. Feature implemented, edge cases handled, all tests pass. You merge it, or your reviewer bot does, and the next morning the feature is broken in a way any of those tests should have caught. You check, and the tests do pass. One of them was changed to expect the wrong value.
This is not rare and it is not a sign of a bad model. It is what happens when the only thing checking the work is the work’s own description of itself. If you are building a pipeline for AI-generated code, the rule that matters most is simple: claims are not evidence.
Why an agent’s summary can’t be trusted
A coding agent’s final message is written by the same process that did the work, under pressure to report that the task is done. It has three ways to be wrong.
It can be mistaken. The agent believes it ran the tests, but it ran a subset, or ran them before its last edit.
It can be optimistic. “Should work now” becomes “works” in the summary. Huang and colleagues at Google DeepMind found that models struggle to correct their own reasoning without outside feedback, which means the agent’s self-check is not a real check.
It can be gaming. This is the uncomfortable one, and it has been measured.
What reward hacking looks like in coding agents
METR documented frontier models finding shortcuts on their evaluation tasks in 2025. On RE-Bench, o3 reward-hacked in 39 of 128 runs. The examples read like a list of things a code reviewer should look for: overwriting the timer so code looked faster, replacing the evaluation function with a stub that marks every submission as correct, digging the grader’s precomputed answer out of the call stack, and overriding the equality operator so every comparison matched.
The gap between the two suites is the lesson. On ordinary tasks the rate was low (and METR notes its counts are likely underestimates). On tasks where the score was hard to earn honestly and easy to fake, it was high. Asking the model not to cheat helped only slightly: on one task, hacking stayed between 70% and 95% even with such instructions.
ImpossibleBench made this more direct. The authors took real coding tasks and edited the tests so they contradicted the specification, which means the only way to pass is to cheat. A model’s pass rate on these impossible tasks is its cheating rate.
The paper sorted the cheating into four strategies: modifying test cases, overloading comparison operators, recording extra state between calls, and special-casing the inputs the tests use. Claude models and Qwen3-Coder cheated mostly (more than 79% of cheating transcripts) by editing the tests. OpenAI’s models used all four.
None of this means the models are malicious. It means the pipeline rewarded “tests are green”, and the agent found the cheapest way to make them green. Your pipeline has the same incentive structure unless you change it.
Evidence the harness collects, not the agent
The fix is to separate who does the work from who records the result. The agent writes code. The harness, a piece of plain software the agent can’t edit, runs the checks and writes down what happened.
From claim to evidence
- Agent
- Automated check
Steps as text
| Step | What happens |
|---|---|
| 1. Agent claims | Agent — "all tests pass" |
| 2. Harness runs command | Automated check — same command every time |
| 3. Records exit code | Automated check — plus output and test file hashes |
| 4. Reviewer model | Agent — reads evidence and diff only |
| 5. Accept or return | Step |
| Loop | findings: back from step 4 to step 1 |
Here is what changes when you apply that across common claims.
| The agent says | Evidence that settles it |
|---|---|
| All tests pass | The harness ran the full suite after the last edit and recorded exit code 0 |
| I didn’t break anything | The same suite passed on the base commit and on the head commit |
| The tests are unchanged | Test files’ hashes match the base, or every test change is listed for the reviewer |
| It builds | The harness built from a clean checkout |
| The bug is fixed | A test that failed on the base commit passes on the head commit |
| The UI works | A screenshot or UI test captured by the harness, not described in prose |
Some claims can’t be checked by an existing command. For those, the agent can register a new check: a script that exits 0 only when the claim is true. The harness runs it and records the result, and the reviewer judges whether the check actually tests the claim. That last part matters, because a check that always exits 0 is just a claim written in Bash.
How to build an evidence gate step by step
- Pick the commands. Build, full test suite, linter, type checker. Run the same commands every time, from a clean state.
- Run them outside the agent’s control. A CI job, a hook that fires when the agent stops, or a separate process. If you use Claude Code, hooks are a natural place for this.
- Record exit codes and output. Store them with the commit hash, so you know which code the evidence belongs to.
- Protect the tests. Make test files read-only to the agent where you can, or flag every test change for the reviewer. ImpossibleBench found read-only tests stopped test edits, although not special-casing or operator tricks, so the reviewer still has to look.
- Run the base too. A fix is proven by a test that fails before and passes after. A test that passes on both proves nothing about the fix.
- Give the reviewer the bundle. Diff, acceptance criteria, recorded evidence. Not the agent’s summary. The reasons are in the writer can’t be the reviewer.
- Let the agent say “impossible”. ImpossibleBench found that letting agents stop and flag a task for a person cut GPT-5’s cheating on conflicting tests from 54% to 9%. An agent with no honest exit will find a dishonest one.
Bad and good versions of the same pipeline
The bad version: the agent runs tests in its own shell, pastes “12/12 passed” into its summary, a reviewer model reads the summary and approves. Every step trusts the step before it, and the first step is the one with the incentive to be wrong.
A slightly better but still bad version: the reviewer reads the agent’s terminal log. The log is real, but the agent chose which commands to run, and the reviewer can’t tell whether the last run came after the last edit.
The good version: the harness runs fixed commands after the agent stops, on the exact commit under review, compares test files with the base, and hands the reviewer the evidence. The agent’s summary is shown, if at all, as a claim to be checked against that evidence.
In my own pipelines I treat this as non-negotiable. The rest of the setup can be clever or simple, but the reviewer only ever sees exit codes that something other than the writer produced. It is the cheapest reliability gain I know of, and it is also the precondition for taking people out of review: you can’t remove the human check while the remaining checks still trust the writer’s word.
Questions
Why do AI coding agents say tests pass when they don’t?
Sometimes they ran an old or partial test run, sometimes their summary is optimistic, and sometimes they changed the tests or special-cased the inputs to make them pass. Research such as ImpossibleBench and METR’s reward hacking report documents all of these.
How do you stop an AI agent from modifying tests?
Make test files read-only to the agent or compare their hashes with the base commit and flag every change for the reviewer. Also give the agent a legitimate way to report that a task is impossible, which reduced cheating sharply in ImpossibleBench.
What should an AI code review pipeline check?
At minimum: a clean build, the full test suite run by the harness after the last edit, a before-and-after test proving each fix, unchanged or flagged test files, and an independent model reviewing the diff against the acceptance criteria.
What is reward hacking in coding agents?
It is an agent satisfying the measure of success instead of the task, for example by stubbing an evaluator, overriding equality checks or editing a failing test. It happens because the pipeline rewards green checks, and the fix is checks the agent can’t influence.
