Coding-agent benchmarks show what models can do on controlled tasks. Production traces show how those agents behave inside your codebase with your workflows, reviewers, and deadlines.
Scale AI's SWE Atlas evaluates coding agents across 284 held-out engineering tasks covering Codebase QnA, Test Writing, and Refactoring. The tasks were written by professional engineers and use real open-source repositories.
Two results help illustrate the limits of current coding agents. Top models score in the 40s on a percentage scale, with none crossing 50% across the full suite. And when the same task is attempted three times, models are two to three times more likely to succeed at least once than to succeed all three times.
That consistency gap matters. An agent that can solve a task sometimes may still require substantial oversight in production.
SWE Atlas compares model capability under controlled conditions. A production engineering environment adds variables the benchmark isn't intended to capture: your codebase, task mix, review practices, integrations, and what happens to the code after it ships.
A production view starts with real agent sessions rather than held-out evaluation tasks. Four categories are especially useful:
Category | What to Measure in Production | SWE Atlas Parallel |
|---|---|---|
Investigation depth | How much surrounding code and context the agent examines before making a change | Codebase QnA |
Completion | Whether the change is fully finished or leaves stale code, missed call sites, or cleanup for a person | Refactoring |
Verification | Whether the agent checks its work and how much human intervention happens before the change ships | Test Writing |
Consistency | How much outcomes vary across similar tasks and sessions | Repeated-task reliability |
These categories take the same broad concerns behind benchmark evaluation and apply them to work happening inside a live engineering environment.
Verification is one example of why production data matters. Larridin found that agent sessions that close without a verification step produce 2.7 times more follow-up fixes than sessions that include verification. A held-out benchmark can't show that downstream effect because there is no production codebase to revisit after the task ends.
Instead of asking engineers to run separate evaluation tasks, production measurement captures agent sessions as part of normal work and connects them to what happens afterward.
Larridin's Agent Effectiveness supports Claude Code, Codex, Cursor, Cline, and custom agents. It connects sessions to the pull requests they produce, scores agent behavior across multiple dimensions, and compares results with an organization's own history and industry benchmarks.
Larridin has analyzed 1.4 million agent sessions to support those benchmarks.
That produces a different kind of measurement from SWE Atlas. The benchmark asks whether a model can complete a standardized task. Production measurement asks how an agent actually worked inside your environment and what happened to the result.
Neither view replaces the other.
Benchmark data can help answer:
Production data can help answer:
A public benchmark also shouldn't be confused with the benchmarking available inside a production product. Larridin can compare an organization's agent performance with its own history and industry p50 and top-quartile benchmarks, but it does not currently publish a public cross-tool leaderboard ranking Claude Code, Cursor, Codex, and other agents against one another.
Yes. They provide a useful, standardized view of model capability. They just don't tell you how the same agent will behave inside your specific codebase and engineering process.
A single score can hide very different problems. An agent may produce correct code but fail to verify it, leave cleanup behind, or perform inconsistently across similar tasks. Measuring those behaviors separately makes it easier to see what needs attention.
Measure real sessions inside your own engineering environment and connect them to what actually ships. Look at investigation, completion, verification, consistency, human intervention, and downstream outcomes rather than relying only on a model's benchmark score.
Larridin's Agent Effectiveness measures coding-agent behavior across real engineering sessions and connects those sessions to the work they produce.