See our AI Impact podcast episodes on YouTube.
Chamberlain Group doubled average pull requests per engineer between January and July. Over the same year, its AI spend grew from under $50 to about $1,000 per engineer. And this was a good thing!
Dan Kirsche, Chief Technology Officer at Chamberlain Group, is an expert, yet approachable voice on the current emergency of capability and costs, and how it can be made to be a good thing. In the latest episode of AI Impact, he walks host Ameya Kanitkar through the metrics he kept and dropped, why a hard cap on AI spend backfired, and how he's working toward a cost-per-feature metric that puts a dollar figure on AI productivity.
Key points
Dan taught himself BASIC at age eight and has made a career of engineering transformations. At Chamberlain, the company behind the Chamberlain and LiftMaster garage door openers and the myQ app, he leads 600 to 700 hardware and software engineers as the company becomes a software-led organization. Several key points emerged.
Make the tools available, then make AI part of the job
Chamberlain started with GitHub Copilot and got adoption in pockets. As Cursor and Claude Code pulled ahead, Dan made them freely available, along with Claude for Enterprise for hardware teams and managers, without taking any tools away. "The first thing is making your tools available," he says; many companies gate access instead.
Even so, adoption stalled at 40% to 50%, while managers insisted everyone loved the tools. So Dan made AI a constant topic, in town halls, in a monthly meeting for directors and above, and in his one-on-ones. He also required every team to reserve time for AI projects in its quarterly capacity plan, and reviews those plans.
Keep the metrics that work, and drop the ones that don't
Dan is wary of metrics that drive bad behavior, and lines of code is the classic example. He started with two simple ones: average pull requests per engineer, and days with commits. Pull requests proved useful, and doubled from January to July.
Days with commits didn't move, because engineers commit daily but squash those commits before opening a pull request, so he dropped it. Every metric is reviewed monthly and shared with the whole team. Ameya notes that, in Larridin's experience, companies typically see velocity gains of 40% to 2x, which puts Chamberlain near the top of the range.
$1,000 per engineer, soft caps, and a cost-per-feature metric
When the CFO or board asks for business value rather than activity, Dan admits it's "probably the hardest question in engineering." His answer for now: stay within budget. Chamberlain covered the jump to about $1,000 per engineer partly by cutting contractor spend, as teams delivered more themselves.
With spend growing about 40% a month, he's watching costs more closely and welcomes cheaper models. When some engineers hit $2,000 a month, the company tried a hard cap, and Cursor usage dropped about 30% the next month. It reversed course and now uses soft caps that auto-approve requests for more, because, as Ameya puts it, the heaviest spenders are often the most productive.
Next, Dan is building a cost-per-feature metric that compares work done by humans augmented with AI, by AI agents alone, and without AI.
AI-first hiring, and AI in the product
For senior roles, Dan won't hire anyone who can't describe real experience using AI. He's still hiring interns, who tend to be AI-native, because the best engineers he's worked with came from intern programs.
To ease job fears, he tells his team that Chamberlain doubled productivity in six months and could 10x it in the next year, but that the goal is to grow the business, not to eliminate roles. And unlike most companies, which focus on AI for their own teams, Chamberlain is building AI features for customers, with business buy-in from day one.
Running AI on video from millions of devices makes costs explode, so the team takes a cost-first approach, ahead of new features planned for CES.
Conclusion
See the full episode on the AI Impact Podcast playlist on the Larridin YouTube channel. To get a demo of Larridin, contact us.
Quotes
- "Now we're transforming into a software-led organization." (Dan Kirsche)
- "From January to July, we've seen a doubling in the average number of pull requests per engineer." (Dan Kirsche)
- "If you didn't have experience leveraging AI that you could talk about, you wouldn't make it." (Dan Kirsche)
- "In the last six months, we doubled productivity, and in the next year, I think we're going to 10x our productivity." (Dan Kirsche)
- "This is the first year where we have millions of dollars of runaway spend that nobody budgeted for." (Ameya Kanitkar)
- "By cutting their spend, you actually kneecap the most productive people on your team." (Ameya Kanitkar)
- "Pairing a senior engineer with taste and judgment with two or three very hungry, AI-native junior engineers works really well in practice." (Ameya Kanitkar)
Q&As
Ameya had questions; Dan had answers.
Q: As you focus on making the engineering organization more AI-native, how are you driving that transition? What are your key challenges?
A: The first thing is making your tools available. I think a lot of people miss that, or they gate it. ... So we just said, here's Cursor. Anybody who wants it, start using it. ... We kept measuring. ... It was 40%, maybe 50%. ... How do we drive adoption? The next thing I did was just talk about it a lot, and also provide education. ... And I set it from the top: you have to reserve a certain amount of time for AI. I need to see AI projects.
Q: The CFO or the board says, "Show me the business value, not just the activity." How do you tie productivity growth back to the business?
A: You're asking probably the hardest question in engineering. ... I try to measure outcomes. ... Where we're going with it, with the CFO question, is this: we have a budget, and we're sticking to our budget. Maybe we shift more to token spend and less in other areas. ... We're starting to put together a new metric, which is cost per feature. ... Can you measure an average cost per feature for humans augmented with AI? Can you measure it with pure AI agents?
Q: How did you budget for AI spend? Where did the budget come from, and how much is spend per engineer?
A: We probably started with spend of zero to maybe $50 per engineer at the beginning of the year. ... Today ... we're at somewhere around $1,000 per engineer. ... There are certain areas of the budget that we were able to reduce. One of those is contractor spend. ... We were seeing a 40% increase in spend per month, and another 40% increase is pretty significant. So I can tell you we're far more focused on spend.
Q: Tactically, do you have soft caps, hard caps, or "eat whatever you want"?
A: They're soft caps. ... If you need to use more, you can just request it, and it'll auto-approve. ... In the spirit of experimenting, we did put a hard cap in. ... That was the totally wrong thing to do, because the following month, the usage of Cursor dropped by about 30%. So we pulled back really quickly.
Q: Have you changed how you hire, and who you look for?
A: We shifted a lot of our focus to where this talent is, and the answer is the West Coast. ... The second was AI: AI-first development was really important. ... If you didn't have experience leveraging AI that you could talk about, you wouldn't make it. ... I don't believe we'll stop or slow down hiring early-career engineers. ... At every company I've been at, the best engineers have always been those who came from the interns.
Q: Did I miss anything? Is there something important I haven't asked about?
A: Most organizations are thinking about AI in terms of their teams. ... We're extremely focused on building features for our users, for our customers, that leverage AI. ... Our second challenge was, how do you do GenAI at scale? ... All of a sudden, you're doing all of these inferences with frontier models, and the costs explode. So we had to get rigor around being cost-first.
Topics in this episode
Key topics in this episode include:
- AI impact: Dan told his team that Chamberlain doubled engineering productivity in the last six months, and that he expects a 10x gain in the next year as better models and autonomous agents arrive.
- AI tool usage: Dan made Copilot, Cursor, Claude for Enterprise and Claude Code freely available, then pushed adoption from about 40-50% higher with town halls, education and AI time reserved in every team's quarterly plan.
- AI costs: Chamberlain's AI spend grew from under $50 to about $1,000 per engineer this year; Dan offset it partly by reducing contractor spend, and uses soft caps after a hard cap backfired.
- Token costs: Dan says Chamberlain is shifting budget toward token spend, and that his heaviest users, the "tip of the spear," are the ones who use a lot of tokens to find new uses.
- AI ROI: To answer the CFO's question, Dan is building a cost-per-feature metric that compares features built by humans augmented with AI, by AI agents alone, and without AI.
- Human productivity with AI: Chamberlain doubled the average number of pull requests per engineer from January to July, and dropped a "days with commits" metric that didn't track real productivity.
- AI agent productivity: Dan wants to measure the cost per feature delivered by pure AI agents, and is building AI features, such as video intelligence, that run at the scale of millions of devices.
- Enterprise AI: Dan describes rolling out AI across a 600- to 700-person hardware and software engineering organization spread across the US, Taipei, India and Mexico.
- AI and the Board: Ameya describes the common pushback when a CTO tells the board the team is shipping 2x faster: "Show me the business value of it. Don't just tell me the activity."
Full Transcript
See this latest episode of AI Impact.
Chapter 1: Introduction (0:00)
Dan: In the last six months, we doubled productivity. Now we're transforming into a software-led organization. The degree to which AI is changing the way you do work, and the speed at which it's moving: it's a challenge to catch up. The first thing is making your tools available. Here's Cursor. Anybody who wants it, start using it. How do we drive adoption? So the next thing I did was just talk about it a lot, and also provided education. We're extremely focused on building features for our users, for our customers, that leverage AI.
Ameya: Hello, everyone. Welcome to the AI Impact podcast. This is the podcast where we discuss how AI is changing things, and what the impact is. Today we have a very special guest, Dan Kirsche. He's the CTO of Chamberlain Group. It's a company that has stood the test of time, around for over 100 years, and it's reinventing itself in the world of AI. I'm very, very excited to talk to you, Dan. Welcome to the show.
Dan: Well, thank you. I'm excited to be here.
Ameya: Fantastic. Maybe a good place to begin is to set the stage for us. Tell us a little bit about your background, how you got here, and how you're now the CTO of the company. You have this long career; how did you get here?
Dan: Yeah, sure. I'd say I probably have a somewhat interesting background on how I got here. I like to start at the beginning, when I was a kid. As a kid, I was always tinkering. I loved pushing buttons and taking things apart. I remember when I was about eight years old, my dad came home with an IBM PC-compatible 286, and I just loved it. I couldn't stop playing with it. I taught myself BASIC. That's when I fell in love with software development. I learned BASIC on it day and night. Overall, as a kid, I was always a tinkerer. And luckily, when I grew up and got into the professional world, I was able to continue doing that. I stayed within software engineering and continued to be a builder. And what I realized as I got into leadership is that I was really good at engineering transformation. I focus a lot on business value and business needs, and align that with engineering technologies and capabilities. So as I've gone through a few companies, I've been brought in to do that transformation: getting engineering better aligned with the business, getting us to be more innovative, and moving quickly. For probably the last six or seven years, I was in fintech. That was a really interesting experience for me, because I got to figure out how to do an engineering transformation in something that's so regulated and requires a lot of security. How do you continue to be innovative? How do you continue to move fast, but still build with the right security protocols in place? So, great learnings.
Chapter 2: Chamberlain's software-led transformation (3:27)
Dan: Today I'm at Chamberlain. Chamberlain Group has been extremely exciting and fun. This is the first time I've been at a hardware-software company, and I joined because, again, of transformation. Most of us probably recognize Chamberlain as the garage door opener. We're in about 36 million homes. We have the Chamberlain brand, we have the LiftMaster brand, and we also have the myQ app, which a lot of people use on a daily basis. It sends notifications when you're leaving home and when you're coming home, and you can even actuate it: you can open the garage and close the garage. And I came in because we were in the middle of our transformation. We started as an industrial manufacturer. We create really good garage door openers. We also have an equally sized commercial business: think warehouses, where you open the dock doors, the dock levelers. So we're a great manufacturer. We've had a ton of success, and now we're transforming into a software-led organization, and that's super exciting for me. When I joined, I had a lot of conversations with our CEO: are we really going to be able to move at the speed I want to move at? Can we really be innovative? And it was a resounding yes, and it's been very true. It's such an exciting experience. I keep talking about the garage door openers, because that's what everybody knows us for, but we have a complete ecosystem. We really focus on the perimeter of the home, access intelligence: knowing the perimeter of the home and making the right decisions to open and close. That includes a whole suite of video cameras, door locks and doorbells. And all of that is information. I haven't gotten to the AI parts, but the punch line is that with all of this information, that's where AI really shines. So I spend a lot of my time thinking about how we leverage all of these data points to make a better product and have a better customer experience. And as everyone is experiencing, AI is moving so fast. What you thought wasn't possible yesterday is possible today. So we're just trying to keep up, trying to stay innovative, trying to build quickly, and kind of scrambling, like everyone else at companies today, just trying to keep up.
Ameya: Fantastic. It's very funny that you mentioned you started tinkering with computers at age eight. I was talking to somebody at Google, and they did a study of who did best at Google, a very talented group of people. They looked at universities, what education people had, which other companies they'd worked at, all that stuff. But the one thing they found with the strongest correlation with the highest output was how early you were exposed to computers, and how early you started tinkering with them. And this obviously resonates. You have such a storied career, and you started at age eight. So I guess that study works.
Dan: Yeah, I guess so. I didn't realize it when I was a kid; I was having fun. I don't know if you remember, but back in the day there was no internet. There were BBSs, bulletin board systems. I don't know if you had a chance to use them, but you could download games, and you could download pictures, and man, I had a ball getting stuff that I probably shouldn't have gotten. I remember the whole progression of modems.
Ameya: Then you start opening up your computer. There's the weird sound of the modem and all this.
Dan: Yeah. Yeah. It was a fun time.
Chapter 3: Structuring hardware and software teams (7:47)
Ameya: You touched on something very critical at the end, which is that you're now in the AI era and you want to transform your business. Let's talk a little bit about the company itself. You mentioned the consumer applications, the commercial applications, all the data and information. But how is the company set up? You mentioned hardware and software. How is the org set up? Is the industrial side separate? Is the software engineering group different from hardware? How is everything set up to run this? The reason I'm asking is that if you go and talk to a SaaS company, people generally know how product teams are set up. I'm very interested in understanding how these traditional industrial manufacturers, which are now software platforms to some extent, actually a very large extent, are set up internally.
Dan: Yeah. I'll talk about the engineering organization. Actually, before I get into engineering: at the top level, from a business entity perspective, you have our different business units. We have our residential business. We have partnerships; we have partnerships with auto companies, for example. Tesla is a great example: you can open and close your garage from your car, because we're fully integrated with Teslas. So we have partnerships. Then we have enterprise, which is software that manages logistics from when a truck comes to a warehouse: getting into the yard, then to a dock, and then back out. We have a lot of software that manages the logistics there. Then we have core, which is our core commercial business: the openers, the parking lot gates. And then we have community: myQ Community. Because we have locks and openers and gates, it's great to integrate the software and offer a solution to multi-unit buildings. So those are our separate, distinct businesses. Then, if you go into engineering, under me we have hardware and we have software, which I think makes a lot of sense. It's a lot of fun for me, but it makes sense because you have to build both together. It's a really deep integration, and I think that's really where you get the value. So we have mechanical engineering, electrical engineering, firmware. And then think about all of the software: front end, mobile, web, back end, cloud engineering. And of course, new today is our AI platform. All of that reports up to the CTO.
Ameya: Can you give us a sense of how big this is, hardware plus software? How many engineers?
Dan: In total, we're about 600 or 700 folks. And as I was saying about the transformation we're going through, if you look back to 2018, which I'd say is really when we were trying to transition to more of a software organization, 30% of our team were software engineers. Fast-forward to today, and we've flipped that: 30% of our organization is on the hardware side and 70% is on the software side.
Ameya: All right. Fantastic. So it's a fairly large org. The reason I asked is that my next question is about transformation, and with large changes, it's just different. It's not just a technology change; it's a bunch of other changes.
Chapter 4: Driving AI tool adoption (12:17)
Ameya: Talk a little bit about how, as you focus on making the engineering organization more AI-forward and more AI-native, you're driving that transition. What are some of your key challenges? What are some of your key tenets for driving that transformation?
Dan: It's a pretty large organization, but it gets even more complicated, because we have multiple offices internationally. We have an office in Taipei, an office in India and an office in Mexico. So it's not just trying to transform one group in the same time zone. It's the whole world at once, and I think that adds a lot more complexity to how you approach this. Let me start with this: it's a really hard problem. The degree to which AI is changing the way you do work, and the speed at which it's moving: it's a challenge to adopt, and then it's a challenge to catch up. And it doesn't stop. The speed is always there, and you just have to keep moving forward. So how do you do that? For us, the journey probably started around the beginning of last year, when we had Copilot, back in the early days.
Ameya: It feels so long ago.
Dan: Yeah, it feels so long ago. We had GitHub, so it was an easy thing to just extend to Copilot, and we got adoption in pockets. But then the tools really started moving. You got Cursor, and then Claude Code, and things were really picking up, and you were seeing the difference in capabilities. That was something I saw early on: we really need to get this adoption going. So the first thing is making your tools available. I think a lot of people miss that, or they gate it, and it becomes hard to get the tool. So we just said, here's Cursor. Anybody who wants it, start using it. Here's Claude: we got Claude for Enterprise, which more of our hardware team and our managers use. We also got Claude Code for our software engineers. And you could choose. We didn't take anything away: you had Copilot, you had Cursor, and Claude. It can be a scary thing, because everyone is so used to software development and engineering in a specific way, with agile methodologies and so on: you write code. So it can be kind of scary. I took it slow at first. I eased everybody into these tools and got everybody comfortable with them. We did that for a few months, and we kept measuring. Measuring is super important. For adoption, it was a very simple metric. I think the more straightforward the metric, the better. It was purely about adoption, and it just wasn't going as fast as I would have liked. I've used these tools myself. I don't write production code anymore, but I still write code, either at home or for internal tools, and I saw the immense value. In my mind it was, why aren't people just running to this thing, because clearly it's the efficiency and productivity. And I'm looking at the adoption, and it was 40%, maybe 50%. I'm asking engineering managers, "What are you seeing?" "Oh man, everyone loves it. It's great." Well, why aren't I seeing it in the metrics? So that's when my staff and I really started to iterate on this and figure it out. How do we drive adoption? The next thing I did was just talk about it a lot, and also provide education. In town halls, it's always a topic: what we're doing at Chamberlain, what we're doing with AI. I have a monthly director-plus meeting, director and above, and somebody presents about AI: somebody using it in a different way, giving people ideas. And in my one-on-ones, I also hold my team accountable for driving that change within their organizations. So it's always front and center with everybody. It's always a topic, and everyone knows it. When I'm walking around the office, people tell me, "Oh, you know what? I started using Cursor, and it's been great." So that was another piece of it. Another piece is getting everybody to think more innovatively. I spend a lot of time driving the culture here. I think culture is so important: building a culture of innovation. Annually, I revisit our engineering vision. We have pillars, and one of those pillars is innovation, and I reinforce that at the beginning of the year and in every town hall. Get people comfortable with taking risks. It's okay to fail. Engineers forget about that. I think deep down, most engineers love being innovative and taking risks. But the truth is, we get bogged down by all the business demands. There are so many feature requests, so many things we need to get done, and there are dates. We have to be able to commit to a date, and if you're committing to a date, you can't take risks; you have to do what you know. So how do you do that? I try to foster that culture, and we can go into that, too.
Ameya: Can I ask you a question here? Did you allow them to... I mean, the thing about these tools is that there's definitely a learning curve. As you mentioned, it's like taking a risk. You and I know it usually pays off, but I can understand why there may be some hesitation in some pockets. Did you allow them to fail? Outside of the push from the top, and innovation being part of a rubric for calibration, was there a stick, or was there another carrot?
Dan: There were two big things that started where I began seeing a change. One is that we do capacity planning on a quarterly basis. Every engineering manager works with their product counterparts and figures out what they're going to do this quarter. You know the capacity of your team, and you get your requests from product. And I set it from the top: you have to reserve a certain amount of time for AI. I need to see AI projects. And we do reviews of these capacity plans, so it's not just "go build and figure it out."
Chapter 5: Measuring AI developer productivity (20:29)
Ameya: Did you change the expectations? For example, let's say you're doing X amount of work in a quarter, and now the expectation is that you do X plus, I don't know, 20%. Was there a change in expectations there?
Dan: Yes. That's another really good point. Like everybody else, since the early days of software engineering, we've been trying to figure out how to measure software engineers' productivity, which is still an enigma today. I think we're a little better off. But you have to be careful what you're measuring, because you don't want to drive bad or unexpected behaviors. The traditional one is lines of code. How awful is lines of code? So we don't want to do that.
Ameya: It always falls in the wrong place.
Dan: Exactly. So we had to be careful. There were two metrics we started tracking and setting expectations on. One was pull requests. I think you're seeing that today; it's becoming somewhat of a proxy for productivity. So we measured the average number of pull requests per engineer. We took a snapshot at the beginning of the year, and every month we do KPI reviews. Every month we'd review this at the leadership level and see how all the different departments within software engineering were performing. That brought clarity to expectations, and it was shared down to everybody. Everyone had access to see pull requests, and they knew that was a priority. That was one. The second one was days with commits. The thought was, if you're using AI, you're going to be more productive, and most likely the number of days with commits should go up. So those were our two metrics. And like any metric, you have to watch it, learn and change. What we realized is that pull requests are actually really helpful. That turned out to be a really good one. And like I said, it's kind of become standard practice. Whenever Anthropic talks about how great Claude Code is, it's always about the increased number of pull requests. So that's been great. From January to July, we've seen a doubling in the average number of pull requests per engineer. So that's been really cool. We ended up dropping days with commits.
Ameya: The days with commits.
Dan: Yeah. We dropped that one. It wasn't moving, and that didn't make sense: why are our PRs going up, but not days with commits? What we realized is that engineers will most likely commit every day, but then they squash their commits before they create a PR, so you lose it. So, okay, that one doesn't make sense, and we tossed it out.
Chapter 6: Tying productivity to business value (23:38)
Ameya: I love how you're iterating on these metrics. Let me ask you a question. We measure developer productivity as part of our platform, so we look at a lot of this across companies, and what you shared absolutely resonates: all the velocity metrics are moving in the right direction. Some companies are at 2x, some are up 40%, some a little higher. So you're absolutely in the same ballpark as the industry; in fact, I'd say fairly high up. One pushback we get from the business, which is very common, and I'm very curious whether you've had this conversation internally... A caricatured way to describe it is that the CTO goes to the board and says, "Hey, we're shipping 2x faster," and all that, and the CFO or the board pushes back and says, "Okay, show me the business value of it. Don't just tell me the activity." It's a little bit of a caricature, but I'm curious: how do you tie all this 50% or 100% growth in productivity, at least at the PR level, back to the business? How do you navigate that?
Dan: Yeah. You're asking probably the hardest question in engineering. That's the hardest job right there. And the truth is, I don't have a great answer to that. I try to measure outcomes. I think it's so important to measure outcomes as well as possible, but there are so many challenges with it. Is the feature actually the problem? Sometimes a really simple feature has significant business value, and sometimes a super complex one has very little business value. So that's such a hard problem to solve. Where we're going with it, with the CFO question, is this: we have a budget, and we're sticking to our budget. Maybe we shift more to token spend and less in other areas, but we're balancing the budget, so net spend isn't going up. We track it a lot. And we're starting to put together a new metric, which is cost per feature. That's where we're going. We're absolutely trying to get to some sort of dollar measurement. Can you measure an average cost per feature for humans augmented with AI? Can you measure it with pure AI agents? Can you measure it without? That's really what we're trying to get to right now, because that's what we can control. I don't have the great answer for tying AI productivity to business value.
Ameya: I guess a lot of the time... and by the way, this is common across the board. People can feel it; they know it. But a lot of times the way it manifests is that expectations go up, in terms of how much we can do with the same size teams, things like that.
Chapter 7: Funding $1,000 per engineer (27:11)
Ameya: Actually, let's talk about budgets, because the AI budget specifically is a very interesting topic. Most customers we've spoken to didn't plan for this. They might have planned a $30 or $40 a month budget per engineer at the beginning of the year, and now the average spend is $300 or $400 per engineer per month, and they didn't budget for it. This is the first year where we have millions of dollars of runaway spend that nobody budgeted for in December or January, but now the spend is there. I'm sure that in 2027, companies are going to be a lot more disciplined about this. But I'm curious: how did you budget for this? Where did the budget come from? And do you have any sense of how much the spend is per engineer per month?
Dan: To give you a sense of magnitude, we probably started with spend of, like you said, zero to maybe $50 per engineer at the beginning of the year, because I think there was just a pocket of engineers using it. Today, and I actually just had a town hall with my team, we're at somewhere around $1,000 per engineer.
Ameya: Yeah. That's big, right?
Dan: Right. Yeah. It is big. We've had to be creative. There are certain areas of the budget that we were able to reduce. One of those is contractor spend, which makes a ton of sense: we've been able to reduce contractors because our teams are able to deliver more. There were other aspects of this, too. While that seems like a lot to change within one year, we were on top of it. We were watching it, so we were able to adjust our spend as it was progressing. Now, though, we're at the point where we were seeing a 40% increase in spend per month, and another 40% increase is pretty significant. So I can tell you we're far more focused on spend, and I'm very happy with all the recent announcements about models getting cheaper. I really hope that balances out going forward.
Chapter 8: Why hard caps backfired (30:00)
Ameya: There's a paradox: things get cheaper, and you actually increase demand. But we'll see. There are also so many open-source models, which aren't so cheap. Tactically speaking, how do you do it? Do you have soft caps, hard caps, or "eat whatever you want"? How is it actually executed?
Dan: They're soft caps. Basically, because we want to keep track of it, we do have caps. If you need to use more, you can just request it, and it'll auto-approve. And that's a really good question, because, in the spirit of experimenting, we did put a hard cap in. This was earlier this year, when we started seeing spend go up significantly and we were really nervous. It was, "Oh my gosh, there are some people spending $2,000 a month," when most people were spending a hundred. So we put a cap in place. And that was the totally wrong thing to do, because the following month, the usage of Cursor dropped by about 30%. So we pulled back really quickly. Sent out an email: "Sorry, guys."
Ameya: You know what's funny, and I'm curious whether you see this: there are some people who spend a lot more tokens and aren't productive, but typically there are a lot of people who spend more and are also very productive. So by cutting their spend, you actually kneecap the most productive people on your team. And what I'm hearing more and more is that that's where soft caps start to work, because they essentially ask you to justify things to get to the next level, but you're not hard-blocked.
Dan: Yeah, that's exactly right. It's clear that usage is going to keep going up, and we do want the engineers who are using this the most to be the tip of the spear, figuring out the next thing we could use this on that we never thought of. And usually that costs a lot of tokens. So, like you said, with the soft cap, we're trying to balance it, but not cut people off.
Chapter 9: Hiring AI-first engineers (32:16)
Ameya: Yeah, that's very smart. Let's talk a little bit about team dynamics, and the hiring side. Have you changed, or do you plan to change, how you hire and who you look for? How has it changed between pre-AI and post-AI, or I guess between last year and this year? What has changed, and what hasn't?
Dan: There are a couple of things going on. We changed recruiting and hiring significantly this year. Part of it is that as we go through this transformation, scale is challenging. We have 10 million connected households. We have 15 million connected users on the platform. We send out close to 100 million notifications a day. So it's pretty significant scale, and we're growing rapidly. Given that degree of scale, we're in an increasingly smaller pool of talent that has seen this scale. So we shifted a lot of our focus, and this doesn't necessarily have to do with AI, to where this talent is, and the answer is the West Coast. We really focused on hiring top talent from the best organizations, and a lot of it was on the West Coast. That was one thing we did. The second was AI: AI-first development was really important. I try to interview everybody at manager and above, and everybody at staff engineer and above, and if you didn't have experience leveraging AI that you could talk about, you wouldn't make it. There are obviously more and more people using AI with an AI-first mindset, but around midyear, or earlier in the year, it was challenging to find people. Some would say, "Oh yeah, I'm going to learn it. I haven't done it yet." Well, you're not innovative enough. So that changed a lot; it became a topic. What didn't change, of course, is pretty straightforward: culture. We spend a lot of time on behavioral questions, making sure you're a fit for Chamberlain. That stayed the same.
Ameya: Very interesting. The other dimension I'd love to explore: did your profile change? For example, I was talking to some very senior leaders at Bloomberg, and I asked them the same question: do you have more junior engineers or more senior engineers? How has that profile changed? The answer can be no change, which is also fine, but I'm curious.
Dan: To be honest, I don't think AI has changed it. We were focused on more senior-level talent because of the scaling challenges and complexity; what we're building is fairly complex, so we were looking for more senior people. But I don't believe we'll stop or slow down hiring early-career engineers. We continue to have interns, and we continue to make offers to our interns. We haven't really changed our practices, and the truth is, it's because they're usually AI-native. They have lots of energy. They generally crave learning. They'll learn quickly and move fast. And at every company I've been at, the best engineers have always been those who came from the interns. So that's a pipeline I don't want to change.
Ameya: No, very true. Someone joked to me that "we hire very AI-native junior talent out of college, and they're very AI-native because they've spent the last four years cheating through all their exams." But yes, that's true. What we've found is that pairing a senior engineer with taste and judgment with two or three very hungry, AI-native junior engineers works really well in practice. They almost become their GPT-7 or 8, and then you have a senior person... you know what I'm saying? They're almost their own agents.
Dan: Yeah, we've seen the exact same thing.
Ameya: Did you face any challenges on the cultural side? It's one thing when you're running a startup or a Silicon Valley company, where you're always looking for something new. It's another thing to run a very innovative company that's also [unclear] years old. How has that been? I'd say that's a fairly unique challenge, in your seat.
Dan: It is. Like I said, I've joined other organizations and done engineering transformations before. This one is different because it's just so much faster, but the principles are kind of the same. There are always folks who are worried about change. They're not used to change; they hesitate. And it's so important to lay out the vision. Where are we going? What are we ultimately trying to get to? So I talk a lot about that. In particular, with AI, a big component was that people were scared for their jobs. You hear this all the time. In a recent town hall, I was talking to my team, and I made the statement that in the last six months, we doubled productivity, and in the next year, I think we're going to 10x our productivity, with all the new tooling, the better models and the autonomous agents that are coming out. We're going to be massively more productive. And naturally, everyone's going to say, "Well, that means you need a tenth of the people." And the truth is, that's not the way it goes. What we're trying to do is really hard. The business is growing. We're looking to 10x our digital subscription business within a span of probably five years. That's not easy; it doesn't just happen. We have to build toward it, and we're going to build increasingly complex use cases. So laying it out: we're not looking to eliminate roles. We're looking to be an even better engineering organization. I think that eased a lot of folks. The other piece is the metrics, which is really important. Make them very clear, so everyone can see them. Metrics have to be really transparent. And then repeat: repeat the message, repeat the metrics. Some people opt out, and that happens, and that's totally acceptable. It's usually a small portion who opt out. It's just not for them. And you bring in folks who are excited about all of this. Ultimately, I think we've done a really great job transforming into this kind of AI-first organization.
Chapter 10: Building AI product features (40:24)
Ameya: Fantastic. Dan, we spoke about hiring, and the type of hiring you're doing. We spoke about how you're driving change with AI, the whole AI transformation, and the challenges of running hardware and software, having the right incentives, the right metrics, budget, costs: the tactical things that work for you at scale. Did I miss anything? Is there something important I haven't asked about?
Dan: Yeah. I think something somewhat unique to us is how we're using AI. Most organizations are thinking about AI in terms of their teams: how do you make your teams more productive, more capable? I think that's at least 90% of the use cases out there. I don't know if that's fact, but it's what it seems like. There's so much of that. We're extremely focused on building features for our users, for our customers, that leverage AI. To me, that's so interesting. It's such a different sort of challenge. So I think that's important to touch on: how do you use AI not just to make yourself more productive, but as a feature? Because I see this all the time: the engineering team will say, "Oh my gosh, GenAI, this is this great thing. Let's go do a pilot." And then you do a pilot, and it doesn't go anywhere. Nothing happens with it, because it's just technology. There's no business behind it. I can tell you that at Chamberlain, our CEO is super passionate about AI, and about AI as features in our product.
Ameya: In the product.
Dan: Exactly. And I give him credit: he sets very ambitious goals for us, and we figure out how to achieve them. At the beginning of this year, we set very ambitious goals for new capabilities we were going to build and launch in time for next year's CES. So here we are, a few months before CES, and we've been slowly figuring out how to do this. We started with: what's the product? What's the value to the customer? We did a lot of research, qualitative research and behavioral research, and we got to a feature set, a product we thought would be great for our customers and had a lot of business value. So we got business buy-in, which was really cool. From day one, we were all unified, and that enabled us to prioritize. That's always the big question: can you actually prioritize these features? That was our first challenge. Our second challenge was, how do you do GenAI at scale? I talked about the millions of devices and millions of customers we have. All of a sudden, you're doing all of these inferences with frontier models, and the costs explode. So we had to get rigor around being cost-first: finding the right models, and the right ways to optimize. We've been very structured in this. We've focused on the cost of these features and the customer outcome.
Ameya: That's right. It's not just about adding features. It needs to be scalable and cost-effective. You have a marginal cost on all these things.
Dan: Yeah. There's this really large variable cost. For every user, every camera, you're now running video through a large language model. How do you manage that? We had to tackle that. And I do believe that at CES, we'll have some pretty exciting new features. We've really focused on intelligent access, and now we're moving into intelligent security. So I really do believe we've got some really interesting things.
Ameya: Well, I look forward to all the new announcements at CES. It sounds like you're already done a few months ahead. This was an absolutely fantastic conversation. Thank you so much for coming on the show.
Dan: Absolutely. Thank you for having me.
This transcript was prepared by AI and may contain mistakes.