Developers can be highly satisfied with an AI coding tool without reporting much time saved. New research suggests that evaluating AI coding productivity requires a much broader view than satisfaction or time savings alone.
Key Takeaways
- Satisfaction and time savings measure different things. In one large GitHub Copilot study, the relationship between them was weak.
- Researchers identified six factors that shape how developers experience AI coding productivity, including task completion, cognitive load, technical expertise, and ownership of work.
- These factors aren’t an ROI formula. They help identify the productivity, quality, and long-term effects that should inform an ROI calculation.
Satisfaction and Time Savings Tell Different Stories
Researchers from Carnegie Mellon University and BNY Mellon surveyed 2,989 BNY Mellon developers with access to GitHub Copilot about satisfaction and perceived weekly time savings, then conducted 11 follow-up interviews.
86% said they were satisfied or very satisfied, yet roughly 60% reported saving less than one hour per week. Among 2,754 developers who answered both questions, satisfaction and perceived time savings had only a weak positive correlation (r=0.34). Nearly 400 developers who saved 30 minutes or less still reported being very satisfied.
The study covered one company and one AI coding tool, so the numbers aren’t a universal benchmark. But the finding is useful: developer satisfaction and measurable productivity gains aren’t the same thing.
The 6 Factors Developers Said Matter
The follow-up interviews surfaced six factors that developers associated with AI coding productivity across development, deployment, and longer-term career effects.
1. Self-Sufficiency
Does the tool reduce the need to leave the task to ask a colleague, search documentation, or look for an answer elsewhere?
Less context switching can make development easier even when the time saved is difficult to isolate.
2. Frustration and Cognitive Load
Does AI reduce the mental effort required to complete the task, or does reviewing, correcting, and re-prompting its output create new work?
A tool can feel helpful because it removes friction even when total completion time barely changes.
3. Rate and Quality of Task Completion
Does AI help developers complete useful work faster without sacrificing quality?
The researchers specifically caution against treating output volume as progress. The important question is whether real engineering work gets completed, not simply whether more code gets generated.
4. Ease of Peer Review
What happens after AI-assisted code is reviewed?
Look at whether review becomes easier or harder, whether reviewers find more problems, and whether AI-assisted changes require additional checking or rework.
5. Technical Expertise
Does long-term reliance on AI help developers build expertise or make it harder to develop skills such as debugging and understanding system architecture?
The study specifically surfaced concerns about junior developers relying on coding assistants before developing those instincts themselves.
6. Ownership of Work
Do developers still understand AI-assisted code well enough to maintain, troubleshoot, and take responsibility for it later?
Code that ships successfully can still create a longer-term problem if nobody understands it well enough to fix it when something breaks.
What Existing Developer Productivity Frameworks Miss
Not all six factors are new to developer productivity measurement.
The researchers found that three factors overlap with existing frameworks: cognitive load and peer review with the SPACE developer productivity framework, and task completion with DORA metrics. The other three factors aren’t directly covered by either framework: self-sufficiency, technical expertise, and ownership of work.
That matters as AI writes more of the code itself. Traditional delivery metrics can show whether work moved through the system. They are less equipped to show whether developers are becoming more capable, more dependent on AI, or less familiar with code they now maintain.
What This Means for AI Coding ROI
The six factors aren’t six entries in an ROI formula. They’re a reminder that the inputs behind an AI coding ROI number need to capture more than tool usage, satisfaction, or estimated hours saved.
Some signals can already be measured with engineering data. Developer Productivity Benchmarks can connect AI-assisted work with delivery and quality measures. Agent Effectiveness adds visibility into how coding agents work before changes ship.
Verification is one example. In Larridin’s analysis of 1.4 million agent sessions, sessions that closed without a verification step produced roughly 2.7 times more follow-up fixes than those that included verification. That doesn’t directly measure peer-review ease, but it shows why productivity measurement needs to follow AI-assisted work past initial output.
Other factors are harder to capture through telemetry alone. Self-sufficiency, cognitive load, technical expertise, and ownership may still require periodic surveys or structured conversations with engineering teams.
That gives an AI coding measurement program two complementary views:
- Operational data: task completion, delivery, quality, verification, rework, and cost
- Developer experience: self-sufficiency, cognitive load, expertise development, and ownership
Then calculate ROI from evidence that reflects what actually changed.
Frequently Asked Questions
Why isn't developer satisfaction enough to measure AI coding productivity?
Satisfaction can reflect reduced frustration, easier access to information, or a better development experience without corresponding directly to time savings or completed work. The BNY Mellon study found only a weak relationship between satisfaction and perceived time savings.
Can all six factors be measured quantitatively?
Not equally well. Task completion, delivery, review, and quality can often be measured using engineering data. Factors such as cognitive load, technical expertise, and ownership may require surveys, interviews, or other qualitative checks.
How should these factors be used to measure AI coding ROI?
Use them to improve the evidence behind the ROI calculation. Combine delivery and quality data with actual AI costs and periodic developer feedback rather than relying on a single proxy such as adoption, satisfaction, or estimated time saved.
Measure the Inputs Behind AI Coding ROI
Larridin connects AI-assisted engineering activity with delivery, quality, agent behavior, and AI spend so teams can see what is actually driving the return from coding tools.