The historical result
METR’s July 2025 report studied 16 experienced open-source developers across 246 real issues. Tasks allowed AI or disallowed it at random. In that setting, assisted tasks took 19% longer. The authors explicitly limit the result to the studied developers, repositories and early-2025 tools.
The update that changes how the result should be used
In February 2026, METR described a later experiment as an unreliable signal of the current productivity effect. Developers and tasks were increasingly selected out of participation, and parallel-agent work complicated time measurement. Its raw follow-up estimates suggested speedup, but the researchers caution against treating them as a precise current effect.
Why repository familiarity changes the question
A contributor who already knows the architecture may spend more time correcting an assistant’s plausible assumptions than writing a narrow patch directly. A newcomer may gain value from explanations and navigation before producing any code. These are different tasks. A useful team pilot should record familiarity, task category and the acceptance contract rather than merging every activity into one average. The goal is to locate where assistance helps your work, including where it adds friction.
Measure waiting and parallel work honestly
An agent can run while a person does another task. Counting the whole agent runtime as human effort can overstate cost; omitting review and supervision can understate it. Record both elapsed delivery time and active human time. If concurrent work is allowed, specify how shared review and context-switching time are allocated. This makes a comparison more informative than a single stopwatch number and prevents parallelism from appearing as impossible labor savings.
Avoid selecting only the easy wins
Agree on the task inventory before seeing which mode feels attractive. Include tasks that fail, are abandoned or need substantial revision. Preserve the original requirement and record why scope changed. If participants choose their own assisted tasks, describe the resulting selection bias in the report. It may be a practical deployment choice, but it answers whether people use AI effectively on chosen tasks, not whether AI improves every task in the backlog.
A useful purchasing conclusion
Do not quote a dated slowdown as proof that current AI never helps, or a vendor speedup as proof that review can be removed. Buy access for a bounded pilot, define useful completed work and inspect costs after the full workflow. Keep the evidence dated as tools evolve. A team that discovers one reliable, valuable use case has learned more than one that forces every request through AI to improve a usage statistic.
Primary sources
Figures and product details above are dated and attributed; they are not current guarantees.
Tools & capabilities referenced
Operators studying this story often evaluate adjacent AI products. Explore Roseram guides where available:
PRACTICAL NEXT STEPS
Take the evidence into your own project.
Calculate the whole project cost
Compare subscriptions, usage, launch work and renewals.
Test an existing-project workflow
Check import, source loading, preview and publication.
Verify the output you buy
Require evidence tied to a real customer task.
Wiki brief
This is a research case study, not a startup success story or a product fraud allegation. It helps teams distinguish measured task performance from confident impressions about AI.
Operator takeaways
- Read the original finding together with the later update.
- Separate active human time from elapsed delivery time.
- Record failures and selection effects before estimating savings.
Related case studies
Plan → build → review with routed models and transparent usage.