How to measure AI coding productivity: cost per accepted task

Define useful completed work before starting the pilot

Select a recurring task whose output can be judged independently: a bug fix, a data transformation or an accessible form. Write the acceptance criteria before assigning a tool. Name the relevant repository revision, owner and verification command. Keep scope comparable across observations. A ten-line correction and a database migration should not become interchangeable tasks simply because each creates one pull request.

Create an inventory before participants choose the tasks they prefer to do with AI. Include difficult and unsuccessful work. If people select only tasks they believe suit the assistant, describe that selection in the report. It can be a sensible operating choice, but it answers a narrower question than whether every backlog task improves. Keep changed requirements and abandoned attempts in the ledger.

Technical references: METR: early-2025 developer productivity experiment · METR: February 2026 experiment design update

Record a baseline that includes review and rework

Use the existing release requirements for the baseline and the assisted work. Record when the task begins, when an acceptable change is ready and how much active human time it consumes. Include reading source, clarifying requirements, reviewing the diff and correcting mistakes. Do not omit a senior reviewer’s time merely because the author finished quickly.

Keep elapsed time and human effort separate. A model may work while a developer attends to another task. Counting every minute of model runtime as labor exaggerates effort; counting only typing omits supervision and review. Record overlap explicitly, using an agreed allocation for shared review or context switching. An honest small ledger is more useful than a dashboard with an unexplained productivity score.

Compare tasks without changing the quality contract

Where feasible, assign comparable tasks before observing which method looks attractive. Do not relax the acceptance checks for generated code to make assisted work appear faster. Capture reviewer comments and failed checks with the revision they refer to. If a task changes scope, record the change and explain whether it remains comparable. Never include production credentials or sensitive customer material simply to make the pilot realistic.

A trial may be observational rather than randomized. That limits causal conclusions because task complexity, experience and interruptions can differ. Record those factors and state what the comparison supports. Avoid claiming that a subscription caused every improvement after its purchase. A team can still identify promising task categories without pretending it conducted a controlled laboratory experiment.

Technical references: GitHub: enterprise research with Accenture · METR: February 2026 experiment design update

Calculate cost per accepted task with explicit assumptions

Add the expenses attributable to the trial: licenses, measured model usage, runtime charges and agreed labor costs. State the allocation period for subscriptions used across several projects. Divide by tasks that meet the acceptance contract, while retaining the cost of failed attempts in the numerator. Do not count a rejected change as a successful deliverable because the model produced an answer.

For illustration only, a pilot costing $600 across ten accepted tasks has a $60 cost per accepted task. This is arithmetic, not a vendor quote or a prediction. Compare it with a baseline using the same expense categories. If labor is estimated rather than timed, label it. Report both cash expense and time where a single blended number would hide how the team actually pays for the work.

total_cost = licenses_allocated + usage_cost + runtime_cost + human_review_cost
cost_per_accepted_task = total_cost / accepted_tasks
# Include failed-attempt costs; never divide by zero.
# State currency, period and labor-rate assumptions.

Inspect quality after acceptance, not just at generation

Track regressions, reverted changes and defects discovered after release. Define a reasonable observation period and keep severity visible. A cosmetic correction and a data-access failure should not be reduced to an interchangeable defect count. Inspect unauthorized data access and error handling independently of the happy-path tests generated by the assistant.

Do not ask the same model to invent both the requirement and the sole evidence that its output meets it. Use the business contract, existing tests and an independent review to identify meaningful failure cases. A change can pass a generated unit test while failing the deployed user journey. Keep local verification, staging behavior and production outcomes in separate fields.

Technical references: GitHub: pull request reviews

Give the agent a measurable task, then make a bounded decision

Use the brief below for a task that belongs to the pilot. Keep the agent’s goal focused on the outcome, not on maximizing code volume. Ask for the files changed, verification evidence and remaining limitations. The human pilot owner should collect time and cost independently rather than accepting a model’s estimate of its own efficiency.

Review the pilot by task category and identify where assistance helps, where it adds review burden and where evidence is too limited. Expand access for the demonstrated use cases, adjust the workflow for weak ones and repeat after material tool changes. Date the conclusion. Neither a historical slowdown nor a vendor success story establishes the behavior of every current tool on every future task.

Complete the bounded task in the supplied repository revision. Preserve existing public behavior except for the specified change. Read the relevant source and instructions first. Apply a reviewable change and run the acceptance checks. Report changed files, exact checks and failures, remaining limitations, and a safe revert path. Stop if source, permissions or requirements are missing. Do not replace real evidence with a successful-looking mock.

Technical references: METR: early-2025 developer productivity experiment · METR: February 2026 experiment design update

Specific answers

Common questions