Accenture GitHub Copilot study: what enterprise evidence actually measures

The published research design

GitHub describes a randomized controlled trial at Accenture, followed by broader adoption analysis and surveys. It reports an 84% increase in successful builds and substantial developer satisfaction. The publisher sells the evaluated tool; its study is useful evidence with that commercial context.

Keep the denominator visible

Successful build counts, build pass rates and delivery speed answer different questions. A team doing more work may produce more successful builds without changing its pass rate. A small code change and a major migration also should not be treated as equivalent units. Before using a headline in a purchasing proposal, identify the unit, comparison group and observation period. If those details are unavailable, keep the claim attributed and avoid deriving an invented financial return from it.

Adoption is an intermediate outcome

Installation shows access. Accepting a suggestion shows interaction. Merging a reviewed change shows a further delivery step. None automatically establishes that customers received a useful improvement. A pilot should keep those stages separate so the team can diagnose where value is lost. If people install the tool but abandon suggestions, examine task fit and context. If suggestions are accepted but create rework, inspect the review and testing process before increasing usage targets.

Build a modest comparison with your existing workflow

Pick a representative collection of maintenance tasks and specify acceptance criteria before work starts. Track completed tasks, total elapsed time, reviewer effort and escaped defects. Keep the same release requirements for assisted and unassisted work. Random assignment can help reduce selection bias when feasible, but do not interrupt critical delivery just to manufacture an experiment. When the comparison is observational, explain that task complexity and developer experience may affect the result.

What the budget should include

Include license expense, onboarding, security review, failed attempts and the time needed to repair generated changes. Use a cost per accepted task rather than dividing a subscription by the number of suggestions. A low-cost tool can still be expensive if it consumes senior review time; a higher-cost tool may be worthwhile on tasks with clear, repeatable acceptance checks. The correct purchasing threshold depends on your workload and the value of the delivered outcome.

Use the findings without copying the conclusion

The study concerns a particular enterprise setting and a dated tool configuration. It does not establish the result for every language, repository or present-day model. Treat the case as a model for asking better questions: which tasks improve, which measures change, and what work remains with people? Keep qualitative feedback alongside delivery evidence, because satisfaction and useful completed work can move in different directions.

Tools & capabilities referenced

Operators studying this story often evaluate adjacent AI products. Explore Roseram guides where available:

Operator takeaways

• Distinguish build counts from pass rates.
• Compare cost per accepted task, including review.
• Keep adoption and customer outcomes in separate measures.

Related case studies