Short answer
An LLM judge agrees with people about as often as people agree with each other on open-ended answers, but it flips with answer order, favours longer output and can be misled by the answer it grades. For code, let it judge against executed evidence and a reference, never on its own reading alone.
You want to grade agent output automatically. Maybe you are comparing two coding agents on your own tasks, maybe you want a model to decide whether a pull request meets its criteria. Asking another LLM “is this good?” is the obvious move, and the results look plausible. The question is how much of that plausibility is real.
Short version: an LLM judge is a good tool with known failure modes, and the failure modes are predictable enough to design around. Used naively on code, it will tell you confident things that are false.
Where LLM-as-a-judge comes from
The term comes from Zheng and colleagues, who built MT-bench and Chatbot Arena in 2023 and asked whether GPT-4 could stand in for human raters of chatbot answers. Their headline result is the one everyone quotes: when ties are excluded, GPT-4’s verdicts agreed with human experts 85% of the time, slightly more than human experts agreed with each other.
With ties and inconsistent votes counted, the numbers drop to 66% for GPT-4 versus humans and 63% between humans, against a random baseline of 33%. Either way, the conclusion holds: on open-ended chat answers, a strong model judge is about as consistent with people as people are with each other.
Two things to notice before you apply that to code. Those were conversation answers, judged on helpfulness and quality, where humans disagree a lot. And “agrees with people as often as people agree” is a low bar when people themselves disagree a fifth of the time.
The biases every judge has
The same paper measured the ways judges go wrong, and later work added more.
Position bias is the largest. Zheng’s team gave judges two similar answers, then swapped the order. A consistent judge gives the same verdict both times.
Wang and colleagues showed how far that goes: with ChatGPT as the judge, they made Vicuna-13B beat ChatGPT on 66 of 80 queries just by changing the order in which answers appeared.
Verbosity bias is the second. In a “repetitive list” test, where an answer was padded with a rephrased copy of its own list, Claude-v1 and GPT-3.5 preferred the padded version 91.3% of the time. GPT-4 resisted much better, failing 8.7% of the time. Newer models are better at this, but the pull towards longer output has not gone away, and agent output is often long.
Self-preference is the third. Panickssery and colleagues found that models recognise their own text above chance, and that this recognition is linked to scoring their own text higher. If your judge and your writer are the same model, you have built a grader that likes the student’s handwriting.
A fourth problem matters more as models converge. Kim and colleagues found that when two models both get a question wrong, they agree on the wrong answer 60% of the time on one leaderboard dataset, and that the effect is stronger among larger, more accurate models. A judge from a similar model family may share the writer’s mistakes, so it will not see them.
Why judging code is harder than judging chat
The most important result for code is easy to miss in the paper. Zheng’s team found that GPT-4, asked to grade answers to basic math questions it could solve itself, was misled by the answers in front of it and marked wrong answers as correct. On ten math questions, scored in both answer orders, the default judge failed 14 times out of 20.
Code is closer to math than to chat. It is right or wrong, and reading it is not the same as running it. A judge shown a plausible function with a confident docstring will often accept it, for the same reason GPT-4 accepted the wrong math answer: the candidate’s reasoning pulls the judge along. The fix in the paper was a reference answer the judge produced independently before seeing the candidate. For code, the equivalent reference is better still: the program’s actual behaviour, from tests the harness ran.
Which judging tasks a model is reliable for
Here is how I would map judging tasks for code. It is my reading of the research above plus practice, not a measured ranking.
Where an LLM judge is reliable for code work
Data as a table
| Item | Can the verdict be checked by running something | How subjective the question is |
|---|---|---|
| Naming and readability | No | Subjective |
| Matches the ticket | Yes | Subjective |
| Logic is correct | No | Objective |
| Tests pass | Yes | Objective |
| Security review | No | Objective |
“Logic is correct”, judged by reading alone, sits in the worst quadrant: objective, so a wrong verdict is simply wrong, and unverified, so nothing catches it. “Tests pass” needs no judge at all. The productive middle is a judge that reads executed evidence and decides whether the change does what the ticket asked.
There is good news here too. OpenAI’s CriticGPT work trained models to critique code and found their critiques were preferred over human critiques in 63% of cases. Models are useful at finding problems. They are less reliable at certifying that no problems exist. Use them as critics more than as certifiers.
How to set up a code judge you can trust
These are the practices I apply, each tied to a failure mode above.
- Give the judge evidence. Build output, test results and exit codes that the harness collected, plus the diff. Never the writer’s summary.
- Use a different model from the writer, from a different provider where you can. This reduces self-preference and some correlated mistakes.
- Swap and repeat for comparisons. When comparing two outputs, judge in both orders and count a win only if it holds both times. Zheng’s team used exactly this conservative rule.
- Ask binary questions against a rubric. “Does the change handle an empty list? Yes or no, with the line” beats “rate this from 1 to 10”.
- Let it generate a reference first. For logic questions, have the judge state the expected behaviour before seeing the candidate.
- Calibrate on labelled examples. Take 30 to 50 past changes that people already judged, run your judge on them, and measure agreement before you let it decide anything.
- Watch for length. If longer outputs win systematically, your judge is grading effort, not correctness.
The bad approaches are the mirror image: the same model judging itself, a single 1–10 score, one ordering, no evidence, and no calibration set. A judge built that way will agree with your agent most of the time, which is exactly what makes it dangerous.
For the broader design of a reviewer that differs from the writer, see the writer can’t be the reviewer. For why the evidence must come from the harness, see evidence over claims. And if you use a judge to compare agents on your own work, run your own evaluation with a calibration set first.
When to use an LLM judge, and when not to
Use one when the question is about fit and intent, when evidence is available, and when you have measured its agreement with people on your own examples. Use one as a critic that lists problems, then verify the problems.
Don’t use one as the final word on whether code works, as a grader of its own writer, or as a single-number score on outputs of different lengths. In those places it does not just make errors. It makes confident errors that look like a passed check, which is worse than having no check.
Questions
What is LLM-as-a-judge?
It is using a language model to evaluate outputs, usually other models’ answers, instead of human raters. The term comes from a 2023 paper that found GPT-4’s verdicts on chat answers agreed with human experts about as often as experts agreed with each other.
How accurate is LLM-as-a-judge?
On open-ended chat answers with ties excluded, GPT-4 agreed with human experts 85% of the time, against 81% between experts. On objective questions it can be misled by the answer it grades, so accuracy on code depends on giving it executed evidence.
What are the biases of LLM judges?
The documented ones are position bias (favouring the first or second answer), verbosity bias (favouring longer answers) and self-preference (favouring its own outputs). Judges also share errors with the models they grade, especially among large models.
How do you reduce position bias in an LLM judge?
Judge each pair twice with the order swapped and count a win only when both verdicts agree; otherwise call it a tie. Few-shot examples also helped GPT-4 in the original study, raising consistency from 65% to 77.5%.
Is LLM-as-a-judge good enough to review code?
It is good at finding likely problems and checking a change against its ticket when it has test results and the diff. It is not reliable at confirming that code is correct by reading it, so pair it with execution.