You approved Copilot. You rolled out Cursor. Engineers are using both. And when someone asks whether the investment is making the team ship faster, the honest answer is: you are not sure.
License counts do not answer that question. Vendor dashboards do not answer that question. You need to connect AI tool usage to the engineering outcomes that actually matter.
Key Takeaway
Engineering AI investment is real—88% of business executives are increasing AI budgets in the next 12 months—but most engineering leaders cannot connect that spend to deployment velocity, code quality, or cycle time improvement. Larridin surfaces usage depth by engineer, team, and tool—and links it to the outcome signals that prove whether AI assistance is actually accelerating delivery.
Quick Navigation
- The Engineering AI Measurement Gap
- What Engineering Leaders Need to Measure
- What Larridin Gives You
- Frequently Asked Questions
- Connect Coding-Agent Sessions to Shipped Work
- Build or Buy the Measurement Pipeline
Key Terms
- Token ROI: The ratio of value delivered to token consumption across AI tools used in engineering workflows. Measures whether AI assistance is producing output worth its cost—not just whether it is being used.
- AI Proficiency (Engineering): The degree to which engineers have integrated AI assistance into their actual coding, review, and architecture workflows—distinct from simply having a Copilot license activated.
- Deployment Velocity: The speed at which code moves from development to production. One of the primary engineering outcome metrics that AI assistance should improve—and the one most useful for connecting AI investment to business value.
- Shadow AI (Engineering Context): AI tools engineers use that fall outside sanctioned tooling—personal ChatGPT subscriptions, alternative coding assistants, experimental API usage. Common in engineering teams and invisible to most governance frameworks.
- Utilization x Proficiency x Value: Larridin's core measurement framework. Who uses AI (utilization), how well they use it (proficiency), and what business impact it creates (value). All three are required to evaluate engineering AI investment meaningfully.
The Engineering AI Measurement Gap
Engineering teams have adopted AI tools faster than almost any other function. OpenAI's research found enterprise AI usage jumped 8x in a year—and engineering was the early concentration of that growth. The tools are in use. The licenses are paid for. The question that most engineering leaders cannot answer is whether any of it is showing up in outcomes.
The measurement gap has a specific shape in engineering contexts:
Adoption is easy to see. Proficiency is not. Whether an engineer has Copilot enabled tells you nothing about whether they are using it in ways that accelerate their work. Larridin's research shows the top 6% of AI users save more than double the hours of the average user. That distribution exists in engineering teams too. Identifying which engineers are at which level of proficiency is the first step to improving the average.
Vendor dashboards measure their own tool, not your outcomes. GitHub tells you how many Copilot suggestions were accepted. It does not tell you whether acceptance rates correlate with deployment velocity, PR cycle time, or defect rates. You need an independent layer that connects usage signals to the engineering metrics you actually track.
Shadow AI is especially common in engineering. Engineers are more likely than most employees to find, evaluate, and start using AI tools on their own—often through personal subscriptions, API keys, or experimental tooling. Three out of four CISOs have found unsanctioned AI tools in their environments—and engineering teams are a primary source. That usage is invisible to both procurement and governance frameworks.
What Engineering Leaders Need to Measure
Evaluating engineering AI investment requires three connected signals:
-
Usage depth by tool, team, and engineer Not whether AI tools are activated—whether they are genuinely embedded in daily workflows. Session frequency, feature engagement, which tools are being used for which task types. This is the input signal.
-
Proficiency distribution Who on the team is using AI at an advanced level, and what does that look like? Who is using it superficially? Where are the gaps that targeted support could close? Larridin's proficiency signals surface this by individual, team, and role—so enablement is targeted, not broadcast.
-
Outcome correlation Do the engineers using AI tools most deeply and proficiently ship faster, produce fewer defects, and close PRs more quickly? Connecting usage signals to GitHub and Jira outcome data answers the question your CFO is asking. Larridin's engineering productivity module is built specifically to establish this correlation.
For Coding Agents, Connect the Session to the Work That Ships
Usage depth and proficiency are useful inputs. Agent-assisted engineering adds another requirement: reliable linkage between an agent session and the commits, pull requests, and downstream outcomes it contributes to. The build-versus-buy guide to agent engineering measurement explains why that linkage is harder than collecting activity.
An agent tracks a session ID, source control tracks commits, and a pull request carries review and outcome records. Those identifiers do not automatically connect. A system can count sessions, commits, and reviews without showing which agent-assisted change was verified, shipped, or later reworked.
For engineering leaders, the test is whether measurement can follow useful work through the delivery lifecycle. Connect the session to its contribution, review and verification, deployment, and subsequent durability or correction where the available data supports it. Keep uncertain or missing links explicit rather than attributing every change near an AI session to that session.
Assess delivery, quality, and cost together over comparable periods. A shorter coding session is not an acceleration if additional review or remediation offsets it. Correlation between deeper AI use and delivery performance is evidence to investigate, not proof of causation; account for differences in task complexity, repositories, staffing, and workflow design.
Build or Buy the Measurement Pipeline Without Losing the Outcome Focus
You can collect agent and engineering data internally. Claude Code supports OpenTelemetry for usage, cost, and tool-activity data, while GitHub and GitLab expose repository, commit, pull request, and review data through APIs. Data access is the starting point. The ongoing work is making those sources reliable and useful together.
What an Internal Pipeline Has to Own
- Collection and storage: Agent telemetry, repository and review records, infrastructure, and appropriate access controls.
- Identity and work attribution: Consistent mappings between people, tools, sessions, commits, and pull requests.
- A common data model: Normalized fields, definitions, time periods, and costs across coding assistants and agents.
- Outcome reporting: Delivery, verification, quality, rework, and cost measures that answer the engineering decision.
- Maintenance: Integration changes, missing data, schema updates, data-quality troubleshooting, and support for newly adopted tools.
Several tools mean several schemas and levels of detail. Cross-tool comparison requires normalization, and each new or changed tool can create more maintenance. A pipeline that works on launch day is not a permanent measurement capability unless someone owns its reliability.
When Building In-House Is a Practical Choice
A focused internal pipeline can be a good fit when most engineers use the same tool, measurement needs are specific, and platform engineering has capacity to build and maintain the system. Owning the schema and underlying data model may justify that effort. Use existing source-control and delivery reporting where it already meets the need rather than rebuilding it unnecessarily.
When Buying Is a Better Fit
A platform becomes more attractive when engineers use multiple coding assistants and agents, new tools arrive frequently, session-to-work linkage is essential, or nobody has clear ownership of the internal pipeline. Buying shifts much of the integration and normalization work to the provider for supported tools; it does not eliminate internal administration or the need to validate coverage.
The linked guide describes Larridin’s Agent Effectiveness as supporting Claude Code, Codex, Cursor, Cline, and custom agents, with links between agent sessions and the pull requests they produce. Confirm supported versions, integrations, linkage completeness, and downstream outcome coverage in your environment before relying on that evidence.
Compare the Ongoing Cost and the Evidence You Receive
For an internal build, include engineering and data work, storage and observability, reporting, schema maintenance, new-tool support, and troubleshooting. For a purchased platform, include subscriptions, implementation, integrations, internal administration, governance, supported-tool limitations, and reduced control over the underlying pipeline.
Validate either approach against a known sample of agent sessions and shipped changes. Test identity matching, missing records, review and rework signals, and cost reconciliation. The choice is not simply build cost versus subscription price. It is whether the measurement capability remains reliable enough to explain if AI is accelerating delivery, at what quality, and at what full cost.
What Larridin Gives Engineering Leaders
Larridin surfaces the usage and proficiency signals that make engineering AI investment measurable.
Full AI tool visibility across your engineering org Larridin captures usage across every AI tool your engineers are using—Copilot, Cursor, ChatGPT, Claude, and the tools they found themselves that procurement does not know about. No vendor integration required. Independent telemetry that shows the real picture.
Proficiency depth, not just activation counts Larridin surfaces usage depth by engineer, team, and tool—session frequency, workflow integration, tool diversity. This distinguishes engineers who are genuinely augmented by AI from those who have a license they rarely open.
Your internal engineering AI champions, identified Larridin identifies the engineers already using AI at the highest level of proficiency—the internal benchmarks for what great looks like on your team. These are the people whose practices are worth understanding, documenting, and scaling deliberately across the rest of the org.
The foundation for outcome correlation Larridin's usage data pairs with GitHub and Jira outcome signals—deployment velocity, PR cycle time, code quality indicators — to establish whether AI tool investment is showing up in the metrics that matter. This is the core of Larridin's engineering productivity use case.
Spend accountability for your AI tooling budget Which AI tools are your engineers using deeply, and which are effectively shelf-ware? Larridin surfaces utilization by tool so license optimization conversations are based on actual data. See how this connects to the broader AI spend management framework.
Frequently Asked Questions
Does Larridin integrate with GitHub or Jira to correlate AI usage with engineering outcomes?
Larridin's engineering productivity module is built to establish this correlation—connecting Larridin's AI usage telemetry with GitHub and Jira outcome data to measure deployment velocity, PR cycle time, and code quality in relation to AI tool adoption and proficiency. This is the use case that directly answers "is Copilot making us ship faster."
Does Larridin monitor code content or what engineers are prompting AI tools with?
No. Larridin's zero-knowledge architecture means it captures usage patterns—which tools, how often, session depth—without reading or recording prompt content, code, or any other work product. Privacy architecture details here.
How does Larridin handle the AI tools engineers are using that IT does not know about?
Larridin captures usage across sanctioned and unsanctioned tools via browser extension and desktop agent telemetry — no vendor integration required. This is specifically what surfaces the AI tools engineers have found and started using on their own, which are invisible to vendor dashboards and procurement records.
How quickly does Larridin deploy for an engineering organization?
Typically one day for initial deployment. Browser extension and desktop agent installation requires minimal IT involvement. Baseline visibility across your engineering team's AI tool usage within the first week.
Can we build our own agent engineering measurement system?
Yes. Agent telemetry and repository, pull request, review, and delivery data can support a focused internal pipeline. Define the required outcomes first, then assign ownership for collection, session-to-work linkage, normalization, and ongoing maintenance.
What is the hardest part of measuring agent-assisted engineering?
A major challenge is connecting agent sessions to the changes they contribute to and the outcomes that follow. Activity counts alone cannot establish whether a specific agent-assisted change shipped, remained useful, or created additional review and rework.
Is building measurement in-house cheaper than buying?
Not necessarily. Compare development, infrastructure, maintenance, new-tool support, and engineering opportunity cost with subscriptions, implementation, administration, and coverage limitations. A narrow, well-owned internal pipeline and a multi-tool platform solve different operating problems.
Ready to measure whether your AI investment is actually accelerating delivery?
We've built a dedicated resource center for engineering leaders navigating AI measurement — see the Developer Productivity Hub.