Short answer
Yes, but not in one jump. Move in stages: AI comments first, then machine gates plus an independent model decide low-risk changes, then people sample instead of approve. Each step needs measured escape rates and a fast rollback before the next one.
Your team adopted coding agents and the pull requests started piling up. Writing got cheap, reading did not. Every PR still waits for a person who has a calendar full of other things, and the question you are quietly asking is whether that person has to be there at all.
My answer is that review with no person in the loop is possible for a growing share of changes, and that the way to get there is a staircase, not a switch. Teams that flip the switch get burned and swing back to reviewing everything. Teams that climb one level at a time, with numbers at each step, keep going.
Why human review became the slowest step
The Faros AI report on about 10,000 developers shows the shape of the problem. Teams that use AI heavily complete more tasks and merge far more pull requests, but the review side absorbs the cost: each PR is bigger and takes much longer to review.
Even at Google, which said on its Q3 2024 earnings call that more than a quarter of new code was AI-generated, that code was then reviewed and accepted by engineers. The writing moved to machines and the reading stayed with people. If writing keeps getting faster and reading doesn’t, review becomes the whole cycle time. I wrote about that bottleneck in code review is the new bottleneck. This article is about the way out.
What a human reviewer actually does
Before you remove a step, list what it does. A human review on a typical team does four jobs at once:
- Catches defects: logic errors, missed edge cases, broken contracts.
- Checks intent: is this the change the ticket asked for, or a confident answer to a different question?
- Guards risk: security, data migrations, money, anything hard to undo.
- Spreads knowledge: someone else now knows this code exists.
Machines are already decent at the first job and getting better. OpenAI’s CriticGPT work found that model-written critiques of code were preferred over human critiques in 63% of cases. The second job can be made mechanical if the ticket carries checkable acceptance criteria. The third job is where people stay longest. The fourth job is real, but it is a weak reason to block a merge: you can spread knowledge with a daily digest of merged changes without making anyone wait.
So the honest framing is not “humans vs AI”. It is: which of these four jobs can you hand to a machine for which classes of change, and how will you know it is working?
The five levels from L0 to L4
Here is the staircase I use when I talk to teams about this. The levels describe who has to look at a change before it merges.
| Level | Before merge | Person’s role | Move up when |
|---|---|---|---|
| L0 | A person reviews every PR | Reviewer | You have CI that runs on every PR and you trust it |
| L1 | An AI reviewer comments first, a person approves every PR | Reviewer with a head start | AI findings match what people would have flagged on most PRs |
| L2 | Machine gates decide low-risk PRs, people review risky ones | Reviewer of the risky share | Escaped defects from machine-approved PRs are no worse than from human-approved ones |
| L3 | Machines decide almost everything, people sample after merge | Auditor | Rollback is fast and sampled audits find little |
| L4 | No person before merge | Owner of criteria, risk rules and audits | You can explain every incident without “nobody looked” |
The process at L2 and above looks like this. The writer is an agent, the gates are machines, and a person only appears when the risk router says so.
Review at L2, with people only on the risky share
- Agent
- Automated check
- Person
Steps as text
| Step | What happens |
|---|---|
| 1. Writer agent | Agent — code and tests |
| 2. Evidence gate | Automated check — build, tests, checks run by the harness |
| 3. Reviewer model | Agent — different model |
| 4. Risk router | Automated check — paths and change type |
| 5. Person | Person — only for high-risk changes |
| 6. Merge | Step |
| Loop | findings go back: back from step 3 to step 1 |
The reviewer model must be a different model from the writer, for reasons I go into in the writer can’t be the reviewer. The evidence gate runs commands itself instead of trusting the agent’s report, which is the subject of evidence over claims.
How human review share falls at each level
Below is a model, not a measurement. It shows the share of merged pull requests a person reads before merge on a hypothetical team, assuming about 60% of changes are low risk (copy, styles, tests, internal tools, dependency bumps) and about 10% are high risk (auth, payments, migrations, infrastructure).
Who reads a pull request before it merges
Share of merged PRs on a hypothetical team
- A person reads it
- Only machines read it
Data as a table
| A person reads it | Only machines read it | |
|---|---|---|
| L0 | 100% | 0% |
| L1 | 100% | 0% |
| L2 | 40% | 60% |
| L3 | 10% | 90% |
| L4 | 0% | 100% |
Notice that L1 removes zero human reviews. It only makes them faster. Many teams stop there and conclude that AI review “doesn’t save much time”. That is correct for L1, and it is the reason L1 is a step and not a destination.
The conditions that let you move up a level
Each move needs evidence, not a feeling. These are the checks I would want before each step:
- From L0 to L1, run the AI reviewer in shadow mode for two to four weeks. Log its findings next to the human comments on the same PRs. You want to know how often it flags what people flagged, and how often it raises noise people would ignore.
- From L1 to L2, write the risk rules down: paths, file types and change kinds that always go to a person. Then let machine gates approve the low-risk class, and track escaped defects (bugs found after merge) per class. Compare with the human-approved baseline from the same months.
- From L2 to L3, make rollback cheap: feature flags, small diffs, one-command revert. Then switch the medium-risk class to post-merge sampling, say one PR in ten, read by a person who records what they would have changed.
- Move from L3 to L4 only when sampled audits find little, and when incidents trace back to bad criteria rather than missed review. At that point the person’s job is writing criteria and risk rules, which is where human judgement was always most valuable.
A note on the numbers you track. Review comments per PR is a bad metric: an AI reviewer can produce endless comments. Escaped defects per class, time from PR to merge, and revert rate are good metrics, because they measure outcomes.
Bad approaches that set teams back
I see the same mistakes repeatedly.
The writer reviews itself. “Now review your code” in the same session produces a polite agreement with its own choices. If that is your AI review, you are at L0 with extra tokens.
The AI reviewer reads the agent’s summary. If the reviewer sees “I added validation and all tests pass”, it will review that sentence. It has to see the diff and the results of commands it did not run.
Everything moves at once. A team turns on auto-merge for all PRs on a Monday, a migration breaks production on Thursday, and AI review gets banned for a year. Risk classes exist so that one bad week can’t do that.
No exit for the agent. When the reviewer and writer can’t agree, a loop without limits burns tokens all night. Put a ceiling on rounds and send the stuck case to a person, which is a good use of a person.
Review placed only at the end. In my own public benchmark, adding one review at the end of an agent’s run halved the hand-counted bugs (about 13 to about 7) but made the run longer, 3.7 to 4.2 hours. Moving the disagreement before any code existed cut the same task to 1.5 hours with no broken mechanics. The details are in the benchmark write-up. One task proves little, but the direction matches what review research has said for decades: problems caught early are cheap.
What the person does at L4
Taking people out of the review loop does not take them out of the system. At L4 a person owns three things:
- The acceptance criteria. If the criteria are vague, no review, human or machine, can rescue the change.
- The risk rules. Which paths are always human, which changes need two models, which ones may auto-merge.
- The audit. A weekly sample of merged changes, read carefully, with findings fed back into criteria and rules.
That is a better job than reading forty pull requests a day and approving thirty-eight of them after a glance. It is also how one person can keep a product moving that used to need a full team of reviewers. Assumptions matter here: this works for products with good test coverage, fast rollback and a clear definition of done. Regulated code paths may need a human signature by law, and that is fine. They are a risk class, not a reason to keep every PR waiting.
The bold version of this idea is that most review in most product teams will be done by machines within a few years, and people will spend their review time on criteria and audits. The careful version is the staircase above. They are the same claim, said at different speeds. If you want help working out where your team sits on it, write to me.
Questions
Can AI fully replace human code review?
For low-risk changes, a different model reviewing the diff with evidence from real test runs can already decide whether to merge. For high-risk changes such as auth, payments and migrations, keep a person before merge until your audits show the machine gates miss nothing that matters.
What is the difference between human in the loop and human on the loop?
In the loop means a person must act before anything proceeds, such as approving each pull request. On the loop means the system proceeds on its own and a person monitors, samples and can stop it, which is the L3 and L4 model.
Is AI code review accurate enough to trust?
Accuracy depends more on setup than on the model. A reviewer that is a different model, sees the raw diff and gets results of commands it did not run is far more useful than the writer reviewing its own summary.
How do I start automating code review safely?
Run an AI reviewer in shadow mode next to your human reviewers for a few weeks and compare findings. Then write down risk classes and let machine gates approve only the lowest-risk class while you track escaped defects.
What metrics show automated review is working?
Track escaped defects per risk class, revert rate and time from pull request to merge. Comment counts are a poor signal because an AI reviewer can generate as many comments as you let it.
