Short answer
Public benchmarks like SWE-bench Verified are saturated and partly contaminated, and they measure Python bug fixes with tests already written. To choose an AI coding agent, replay 20 to 30 of your own closed tickets, hide the real fix, run each setup several times on the same budget, and measure hidden-test pass rate, cost and the human minutes needed to merge.
Every model launch comes with a chart where the new model beats the old ones on a coding benchmark. A team picking an agent reads three of those charts, sees three winners, and ends up choosing by whichever demo looked best in a meeting.
The benchmarks aren’t lying, exactly. They answer a narrow question very well: how often can this agent fix a known kind of bug in a known set of open-source projects? Your question is different. You want to know which agent, with which setup, gets your tickets merged in your codebase at a cost you can live with. Nobody has published that number. You have to measure it, and it is less work than most teams assume.
What SWE-bench actually measures
SWE-bench, the benchmark most launch charts cite, was built from real GitHub issues in popular open-source Python repositories. Each task gives the agent an issue and a repository, and the agent passes if hidden tests from the real fix go green. SWE-bench Verified is a 500-task subset that human annotators checked for clear descriptions, correct tests and solvable problems.
That design is clever and also narrow. Every task is Python. Every task already has a test that defines “fixed”. Nobody asks whether the patch is readable, whether it matches the codebase’s style, or what it cost to produce.
Scores on it rose fast. On the official leaderboard, where results are run and listed by the SWE-bench team, the best submission went from single digits in late 2023 to 79.2% by December 2025.
Best score on the SWE-bench Verified leaderboard
Share of 500 tasks resolved, best submission so far
Data as a table
| Best submission | |
|---|---|
| Late 2023 | 4.4% |
| Mid 2024 | 38.4% |
| End 2024 | 62.2% |
| Mid 2025 | 75.2% |
| End 2025 | 79.2% |
| Early 2026 | 79.2% |
Vendors’ own announcements, collected by aggregator sites, report much higher numbers for newer models by mid-2026, in the 90s. Those are self-reported with each vendor’s own harness, which is part of the problem: the same model scores differently depending on the scaffold around it.
Why public benchmarks won’t pick your agent
Three things break the link between a leaderboard and your team.
The benchmark is saturated and contaminated. In February 2026 OpenAI said it would stop reporting SWE-bench Verified. According to Decrypt’s account, an audit of 138 tasks that its model consistently failed found 59.4% of them broken, for example tests that required a function name the issue never mentioned. It also found that several frontier models could reproduce known fixes from a task ID and a short hint, which suggests the solutions were in their training data. OpenAI recommends SWE-bench Pro instead, where models scoring above 70% on Verified reportedly land around 23%.
The task shape is not your task shape. The same leaderboard data makes this concrete. One agent, Sonar Foundation Agent on Claude Opus 4.5, scored 79.2% on the hand-checked Verified subset and 52.6% on the full, unfiltered SWE-bench test set.
A clean issue with a test attached is the easy case. Your backlog has vague tickets, UI work, migrations, flaky integration tests and code in languages the benchmark never touches.
Feeling is not measurement either. In METR’s randomised trial, experienced developers using AI tools on their own repositories were 19% slower, while believing they had been about 20% faster. If your team picks an agent by how the demo felt, it is measuring the wrong thing with a broken instrument.
There is one public measure I do find useful for planning: METR’s time horizon, the length of task an agent completes half the time, which doubled about every seven months from 2019 to 2024. It tells you roughly how big a ticket you can hand over today. It still doesn’t tell you which agent fits your code.
How to build an evaluation from your own tickets
The good news is that you already own the best benchmark for your team: your closed tickets and the commits that fixed them.
An agent evaluation built from your own history
- Person
- Agent
- Automated check
Steps as text
| Step | What happens |
|---|---|
| 1. Pick tickets | Person — 20 to 30 closed ones |
| 2. Rewind the repo | Step — parent of the real fix |
| 3. Run each setup | Agent — same budget, 3 times |
| 4. Hidden checks | Automated check — tests from the real fix |
| 5. Human grading | Person — would you merge it |
| 6. Scorecard | Step — pass rate, cost, minutes |
| Loop | rerun on each new model: back from step 6 to step 3 |
- Pick 20 to 30 tickets that were closed in the last year. Mix them on purpose: a few bug fixes, a few small features, one refactor, one with a vague description, one in the part of the code everyone avoids.
- For each one, check out the commit before the real fix, and keep the fix’s tests aside. The agent must never see them.
- Write the prompt the way your team writes tickets, not the way you wish it did. If you plan to give the agent acceptance criteria in production, write them here too; acceptance criteria an agent can’t talk its way around covers how.
- Run every setup you’re considering on every ticket, at least three times. Agents vary a lot from run to run, and one run per task tells you about luck. Give every setup the same time and token ceiling.
- Score with the hidden tests first, then have an engineer grade a sample: would they merge it as is, after small edits, or not at all? Time how long it takes to get each patch to mergeable.
- Keep the harness. When a new model or tool version ships, rerun the same set in an afternoon.
A setup means more than a model. Test the combinations you would really use: Claude Code alone, Codex alone, one writing and the other reviewing, different reasoning efforts. In my experience the gap between setups with the same model is often bigger than the gap between models, which is why I compare them in Claude Code vs Codex. My own public benchmark is a single-task version of the same idea, with the prompt and procedure published so anyone can rerun it.
What to measure on each run
| Metric | How to get it | Why it matters |
|---|---|---|
| Hidden-test pass rate | Run the real fix’s tests on the agent’s patch | The closest thing to “it works” you can automate |
| Mergeable without edits | Engineer grades a sample blind to the setup | Passing tests and readable code are different things |
| Human minutes to merge | Time the edits needed | This is where the real cost sits |
| Cost per resolved ticket | Tokens or plan usage divided by tickets solved | A cheap agent that fails often is not cheap |
| Wall-clock time | Start to final answer | Matters for overnight and queued work |
| Run-to-run spread | Pass rate across the repeat runs | A setup that is great once and poor twice is a risk |
| False “done” claims | Agent said finished, hidden checks failed | Tells you how much you can trust its reports |
The last row is the one teams skip and later regret. An agent that fails honestly costs you a retry. An agent that reports success on broken work costs you a bug in production. Measure how often each setup claims more than it delivered, and favour setups that prove their work with commands, as I describe in evidence over claims.
Bad and good ways to evaluate AI coding agents
Bad: choosing by leaderboard rank. The leaderboard measures a task shape you probably don’t have, on data the models may have seen.
Bad: one impressive demo. A demo is one run of one task chosen by someone who wanted it to work.
Bad: letting the agent see the tests, or grading by its summary. Both measure how well it writes reports.
Bad: using a model as the only judge. A model grading another model is useful as a first filter, with known biases, and I go into where it fails in LLM-as-a-judge for code. Keep people on a sample.
Good: your own tickets, hidden checks, repeated runs, equal budgets, and a scorecard you rerun on every release. Twenty tickets measured honestly beat any number on a launch chart.
I expect this to become normal practice within a couple of years: teams will keep a private evaluation set the way they keep a test suite, and switch models when the numbers say so rather than when a launch post does. The teams that start now will also be the first to trust agents with work nobody reviews by hand, because they’ll have the data to justify it.
Questions
What is the best benchmark for AI coding agents?
No public benchmark predicts performance on your codebase well. SWE-bench Verified is widely cited but saturated and partly contaminated, and OpenAI now recommends SWE-bench Pro, but the most useful benchmark is a set of your own closed tickets with hidden tests.
How do you evaluate an AI coding agent?
Replay real tickets from your history with the real fix hidden, run each agent setup several times under the same budget, and score hidden-test pass rate, cost per resolved ticket and the human minutes needed to make each patch mergeable.
Is SWE-bench still a good benchmark?
It is still useful for comparing agents on Python bug fixes, but scores near the top no longer separate models well. OpenAI stopped reporting SWE-bench Verified in February 2026, citing broken tests and training-data contamination.
How many tasks do I need to evaluate a coding agent?
Twenty to thirty varied tasks, each run at least three times, is enough to see large differences between setups. Small differences need more tasks, so treat a gap of a few points as a tie.