At review time, 94% of technology leaders rated AI-generated code higher quality than human-written code. Yet 82% reported a major production failure tied to AI code in the past six months. The contradiction isn’t a coincidence. It’s a measurement problem.
AI-generated code can look clean, compile, and pass an initial review yet still not be ready for production. Reviewers may not see missing context, unusual inputs, dependency behavior, or security weaknesses until the code runs in the real environment.
New Relic’s survey makes that gap visible. While 94% of respondents rated AI-generated code higher quality at review, 78% reported more incidents after it shipped. The finding doesn’t prove that every incident came from poor AI output. It does show that review-time confidence isn’t enough evidence of production quality.
Leaders need to know which AI-assisted changes required remediation, how much rework followed, and whether the pattern improves as AI code share grows.
New Relic found that 62% of respondents said their teams often trusted AI-generated code enough to ship it without line-by-line manual verification. That level of scrutiny isn’t scalable. What leaders need is a consistent verification discipline that matches controls to the risk of the change.
Routine boilerplate code may only need automated tests and standard review. Authentication, data access, and core business rules should trigger deeper review, stronger tests, and security scanning.
Veracode tested more than 150 large language models across 80 coding tasks in Java, JavaScript, C#, and Python. Across all models and tasks, 45% of generations contained a known security flaw when no security guidance was provided. Syntax correctness exceeded 95%, while security performance stayed near 55%.
Cross-site scripting had a 15% security pass rate, and log injection had a 13% pass rate. Those weaknesses require secure design, context-aware review, and automated application security testing beyond basic syntax and functional checks.
New Relic found that 74% of respondents said at least 25% of AI-generated code needed significant post-deployment rework because of poor context, incomplete data, or flawed system assumptions. That pulls experienced engineers away from planned work.
That’s why leaders should track code durability and code turnover, not just whether a change deployed successfully. A change can avoid an immediate incident and still be deleted, reverted, or substantially rewritten within 30 or 90 days.
Define the failure metric before trying to improve it. Measure the percentage of AI-assisted changes that cause an incident, rollback, hotfix, or other production remediation, then compare that rate with human-only changes.
Break the results down by team, repository, tool, change type, and risk level. Use tool telemetry, commit metadata, and engineering records to identify AI-assisted work. Document unattributed changes rather than forcing false precision.
Set minimum controls by change type instead of letting each engineer decide how much verification is enough. Requirements may include independent tests, security scans, reviewer context, and additional approval for high-risk code.
Track whether those controls happen. A policy that isn’t visible in workflow data won’t reduce the failure rate.
New Relic found that 96% of respondents considered observability very or extremely important when working with AI-generated code. Production telemetry can show which deployment preceded an error, latency spike, rollback, or incident. Connecting that record to available AI source signals creates the feedback loop review metrics lack.
Categorize failures by root cause to identify recurring problems such as missing context, insecure patterns, insufficient tests, or oversized changes.
AI code share is a composition metric, not a performance goal. Read it alongside change failure rate, rework, code turnover, review depth, lead time, and recovery time. Larridin’s AI Dev Productivity platform brings AI code share, code durability, delivery, and quality signals into the same engineering view.
Rising AI code share with stable or improving quality can indicate healthy adoption. Rising code share alongside more failures, rework, or turnover is a pattern to investigate, not proof that AI caused the problem. Our DORA metrics article explains why delivery metrics need to move together.
Generic tool training won’t solve a context, testing, or security problem. Use failure and rework data to identify which capabilities teams need: scoping work for AI, supplying codebase context, designing independent tests, reviewing security-sensitive logic, or knowing when not to delegate a task.
AI proficiency is the measurable ability to use AI effectively to produce business outcomes. Larridin’s AI Fluency capability helps identify gaps by team and role, so enablement can target the areas most likely to improve quality.
Review, compilation, and syntax checks only evaluate part of the quality picture. AI-generated code may lack local system context, mishandle unusual inputs, interact poorly with dependencies, or introduce security weaknesses. Production behavior, rework, and durability reveal problems that weren’t visible at the initial gate.
In Veracode’s controlled testing, 45% of generation tasks produced code with a known security flaw when models received no security-specific guidance. It isn’t a claim that 45% of every organization’s AI-generated code is vulnerable. Internal testing and production data are needed to measure your actual rate.
Compare production failures per AI-assisted change with the rate for comparable human-only changes. Segment by team, repository, change type, and risk level, then watch the trend as AI code share changes. Aggregate incident counts can show that reliability changed, but they can’t identify AI’s contribution without source attribution.
There’s no universal target. Higher AI code share is useful only when delivery, quality, and cost outcomes hold or improve. A team with high AI code share and low rework may be using the tools well. The same code share paired with rising failures or turnover calls for investigation.
Larridin’s AI Dev Productivity platform measures AI code share alongside durability, delivery, and quality signals, showing whether AI-assisted work creates durable value or shifts cost into review and rework.
Book a discovery call to see how AI-assisted code is performing in your engineering organization.