Short answer
Do AI coding tools make developers faster? It depends on what you measure. A Copilot trial on a small greenfield task found 55.8% faster completion; METR's trial with experienced developers on their own large repos found 19% slower. Team telemetry shows more output but slower review, and no clear company-level gain.
Search for whether AI makes programmers faster and you get two headlines that can’t both be true. One says developers finish 55.8% faster with Copilot. The other says experienced developers were 19% slower with AI and didn’t notice. Vendors quote the first, sceptics quote the second, and a manager deciding on a budget is left with noise.
Both results are real. They measured different tasks, different people and different things. Once you read what each study actually timed, the contradiction mostly goes away, and what is left is more useful than either headline.
The four kinds of evidence
AI productivity evidence comes in four kinds, and they answer different questions.
Controlled trials time people on tasks, with and without AI. They are the most rigorous, but the task shapes the result.
Telemetry studies count what happens in real repositories: tasks closed, pull requests merged, review time, bugs. They show the whole pipeline but can only show correlation.
Surveys ask developers what they experience. They capture things telemetry misses, such as friction and trust, but people are poor judges of their own speed.
Single-team measurements, including mine, tell you about one setup on one task. Useful for direction, never for a general claim.
Copilot, 55.8% faster: a small task from scratch
The 2023 Copilot trial (arXiv 2302.06590) gave 95 developers one task: implement an HTTP server in JavaScript as fast as possible. The group with GitHub Copilot finished 55.8% faster than the control group.
That is a clean result about a specific situation. The task was small, self-contained and greenfield. There was no existing codebase to understand, no team conventions, no reviewer and no production. This is the situation where autocomplete and code generation shine, because almost all of the work is typing out a well-known pattern.
What it does not tell you is how long a feature takes to reach users in a ten-year-old codebase with four people involved.
METR, 19% slower: experts in their own large repos
METR’s randomised trial from July 2025 sits at the other end. Sixteen experienced open-source developers worked on 246 real issues in repositories they had contributed to for years. The repositories averaged more than 22,000 stars and over a million lines of code, and issues took about two hours on average. Each issue was randomly assigned to allow or disallow AI, mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet.
With AI allowed, tasks took 19% longer.
The perception gap is the part worth remembering. Before the study, developers forecast that AI would cut their completion time by 24%. Afterwards, they still believed it had cut it by 20%.
This says something uncomfortable about self-reported gains. If experts who just spent weeks being measured could misjudge the direction of the effect, a survey where people estimate their own speed-up is weak evidence on its own.
The study also has limits METR states openly: a small number of developers, people who knew their code extremely well (the case where an assistant helps least), and tools from early 2025.
METR’s follow-up: probably faster, and harder to measure
METR ran a second, larger study with late-2025 tools, published in February 2026: 57 developers, 143 repositories and over 800 tasks. For the ten returning developers the estimated change in task time was minus 18%, meaning faster, with a confidence interval from minus 38% to plus 9%. For the 47 new developers it was minus 4%, with an interval from minus 15% to plus 9%. Both intervals include zero.
METR itself calls this data an unreliable signal, and the reasons are interesting. Many developers declined to take part because they didn’t want to work without AI. Between 30% and 50% said they skipped tasks they didn’t want to do without it. Some ran several agents at once, which made time per task hard to log. METR thinks these effects push the estimate down, so the true speed-up could be larger, but it says its data is only very weak evidence for how large.
The honest summary: individual task speed with current tools is probably positive for many developers, the size is uncertain, and it depends heavily on the task and on how well the person already knows the code.
Team telemetry: more output, slower review
Faros AI’s 2025 report measures something the trials don’t: what happens to the pipeline. Across more than 10,000 developers on 1,255 teams, high-AI teams completed 21% more tasks and merged 98% more pull requests. They also saw review time per pull request rise 91%, pull request size grow 154% and bugs per developer rise 9%. Faros found no significant correlation between AI adoption and improvement at the company level.
DORA’s 2025 report, based on nearly 5,000 technology professionals, found close to 90% use AI at work and more than 80% believe it made them more productive. It also found AI now correlates with higher throughput while delivery instability keeps rising.
Atlassian’s 2025 developer experience survey of 3,500 developers and managers adds the time budget. 68% said AI saves them more than 10 hours a week. 50% said they lose more than 10 hours a week to organisational friction. Developers reported spending only 16% of their time coding.
Put those together and you get the real picture. AI speeds up the slice of work it touches. That slice is small, and the stages around it (review, waiting, coordination) absorb or even cancel the gain. I explain the maths of that in Amdahl’s law for software teams and the review side in code review is the new bottleneck.
What each study measured, side by side
| Study | Who | What was timed or counted | What it can’t tell you |
|---|---|---|---|
| Copilot trial, 2023 | 95 developers | time to build one small HTTP server | effect on real codebases and teams |
| METR, 2025 | 16 experienced OSS developers | time per real issue in their own repos | effect for newer tools or less familiar code |
| METR update, 2026 | 57 developers | time per task, late-2025 tools | a reliable size of the effect |
| Faros AI, 2025 | 10,000+ developers, 1,255 teams | tasks, PRs, review time, bugs | cause and effect |
| DORA, 2025 | about 5,000 professionals | survey answers, delivery outcomes | precise time savings |
| Atlassian, 2025 | 3,500 developers and managers | self-reported hours saved and lost | measured time |
How to read the next AI productivity claim
Bad ways to use these studies: picking the one that supports what you already think, quoting a self-reported percentage as if it were a measurement, or treating a greenfield toy task as evidence about your monolith.
Better questions to ask of any claim:
- What exactly was timed: a task, a pull request, or delivery to users?
- Whose code was it, and how well did they know it?
- Was there a control group, or is it a before-and-after or a survey?
- Did anyone count what happened downstream: review time, rework, incidents?
- Which tools, and from when? A study from early 2025 describes different tools from the ones you can buy today.
And the best answer is your own data. Pick ten real tickets, run them through the setup you are considering, and measure lead time to merge and defects found later, not just how fast the first draft appeared. I describe how to do that in run your own benchmark.
Where the gains will actually come from
My reading of all this is optimistic, with a condition. The tools are good and getting better fast, and the newer METR data points the right way even if it can’t pin down a number. The ceiling on what a team gets is not the model. It is the process around it. Teams that keep human review, approvals and hand-offs exactly as they were will see the Faros pattern: busier developers, longer queues, flat delivery. Teams that redesign those stages for cheap code will see what the trial headlines promise, at the level of the product instead of a single task.
Questions
Does AI make developers more productive?
On small, well-defined tasks, yes: a controlled Copilot trial found 55.8% faster completion. On real work in large, familiar codebases the evidence is mixed, and team-level studies show more output but no clear company-level gain so far.
Why did the METR study find developers were slower with AI?
The developers were experts working in their own very large repositories, where they already knew what to do and an assistant adds checking and prompting overhead. METR’s later study with newer tools pointed towards a speed-up but called its own data unreliable.
Are self-reported AI productivity gains reliable?
Not on their own. In the METR trial developers believed AI made them 20% faster when it actually made them 19% slower, so surveys are better at capturing friction and sentiment than speed.
How should my team measure AI coding productivity?
Measure lead time from ticket to merge and defects found after merge on your own tickets, with and without the tooling. Counting lines or pull requests rewards volume, which AI inflates for free.