Rakuten Claude Code case study: measure the release, not the demo

What Rakuten reported

Anthropic’s account reports feature time to market falling from 24 to five working days. It also describes a seven-hour coding session on a vLLM task with occasional guidance, checked against a reference method at 99.9% numerical accuracy. These are attributed customer claims, not our measurements.

Why numerical accuracy is a narrow result

A comparison with a reference calculation answers a specific question about output agreement. It does not show that an entire application is 99.9% correct, secure or maintainable. An operator should write down the reference, tolerance, input distribution and failure conditions before deciding what a numerical result means. That distinction matters when a compelling benchmark appears beside a much broader business claim.

Design an agentic coding pilot you can inspect

Choose a bounded change with an existing baseline: for example, replacing a slow calculation while preserving its public interface. Give the agent the relevant implementation, a representative input fixture and the exact command that produces the comparison. Keep credentials and production writes outside the task. Ask for an explanation of changed assumptions and collect the diff, test output and revision together. If the source cannot be loaded, stop rather than allowing a plausible replacement to stand in for the actual repository.

Measure the whole release path

Record time spent clarifying requirements, generating edits, reviewing changes, correcting failures and preparing the release. A task can finish generation quickly and still create a long review queue. Include unsuccessful attempts in the pilot instead of reporting only the most impressive run. Compare work of similar scope and complexity, then describe the uncertainty around the result. A small team can use a simple task ledger; it does not need an enterprise dashboard to avoid misleading itself.

Where a small team should start

Start with a reversible task whose outcome is easy to observe: a tested data transformation, a documented interface or a narrow component change. Put an owner beside each acceptance check. Have someone inspect behavior that the agent’s own tests may have missed, such as invalid input or a changed dependency. Release to a controlled environment before inviting customers. This turns a vendor story into a useful evaluation plan without assuming the same speedup will occur in your codebase.

Evidence limits and the buyer decision

The linked account is published by the tool vendor and does not supply a randomized counterfactual for the feature-delivery claim. Use it to identify a workflow worth trying. Choose a subscription or wider rollout only after your own pilot captures completed tasks, review cost and regression outcomes. Faster releases are valuable when they preserve the customer contract and leave the team able to maintain what was shipped.

Tools & capabilities referenced

Operators studying this story often evaluate adjacent AI products. Explore Roseram guides where available:

Operator takeaways

• Define the reference and tolerance before evaluating generated calculations.
• Measure review and release time alongside generation.
• Keep reversible scope and repository evidence attached to each task.

Related case studies