# Measuring AI ROI in Software Engineering

**Podcast:** Engineering Enablement by DX
**Published:** 2026-04-03

## Transcript

At DX, we've also developed true throughput, which is weighted PR throughput, which actually does incorporate lines of code as one of the mechanisms to weight PR throughput.
But again, I think all these metrics are imperfect.
So you're dealing with degrees of imperfection.
I do think lines of code is significantly less perfect than PR throughput as a high level signal.
Quick note before we dive into the episode.
I'm excited to share the AI measurement framework, which offers research-based metrics for tracking AI adoption, measuring impact.
and making smarter investment decisions.
We've developed the AI measurement framework through ongoing research as well as practical experience helping hundreds of companies like Booking.com and Block.
To learn more about the AI measurement framework and how to use it, head over to getdx.com slash AI-framework.
Again, that's getdx.com slash AI-framework.
Welcome.
Thanks everyone for joining.
The discussion today is going to be hosted by Jesse Adamitz, who is a...
Senior Director of Platform Engineering from Twilio.
He's also a longtime DX customer, super thoughtful thought leader.
He's been on our Engineering and Able-in podcast before.
You might recognize him.
And he's going to be kind of guiding this conversation with Abhi AMA style around questions that we're hearing from you all and just the industry in general around sort of the evolution of AI impact.
From there, I'll pass it over to you, Jesse, to kick off this conversation.
Yeah, I feel like I'm here to get as many questions answered, even just for myself.
I feel like things have been moving very quick the last even, well, obviously a year plus, but the last couple of months, there seems to have been this click, maybe around background agents or this conversation about ROI, things like that.
So suddenly it feels like pieces fall into place very quickly.
And then we're all asking the same questions.
So like, maybe, maybe let's start there.
I'll be like, again, I think it's gone from a lot of leaders like us are saying, cool, how do we?
how do we adopt this?
But adoption seems to have moved very quickly.
And now we're all saying, how do we finish the rollout or enable going, like enable everybody doing more of the same things faster.
And then of course, you've got people seeing the adoption and saying, how do we measure it?
So what are you seeing from leaders about maybe how they're applying AI across the SDLC?
You know, I think we've moved on from CodeGen specifically, or like autocomplete.
And so now it's like, where else are we seeing it in the SDLC?
First of all, I want to just acknowledge like every leader I talk to, the impression I leave with is like, this is a crazy, it's an unbelievable time to be a developer productivity or platform leader.
Never before has there been this amount of change, this level of attention and focus on developer productivity.
So it's both a...
stressful and very demanding time to be a leader in this space, but I think also very exciting and, you know, huge abundance of opportunities and opportunity to steer our organizations.
When I go talk to leaders, I would say, first of all, this question of like, what is the future SDLC?
So what is beyond just code tools?
I think that's actually the question everyone is trying to figure out.
And I've started seeing when I go meet with customers, they share with me, you know, their pitch decks, their vision docs, their 12, 36 month hypothesis for how they're going to incorporate AI across the SDLC.
But it's very much in the visioning state, right?
I would say everyone is trying to figure out what does this look like 12 months from now?
No one has it nailed.
A few of the themes that I pick up on, I would say are.
One, looking at the wider SDLC, and I use the term like wider, so spanning more of the SDLC process.
So areas like review, areas like planning and prototyping.
Those are themes that come up when I go talk to leaders in terms of where they're seeing new bottlenecks emerge and new opportunities to greater impact productivity.
Another theme I hear about is just developer experience.
So, you know, especially when I go talk to DX customers, they're looking at the data on where are the bottlenecks for developers, where are the friction points for developers?
How do we leverage AI to make a dent in those areas?
Whether that's code-based maintainability, even deep work, documentation, of course, which also is critical for successful AI reliability.
So developer experience is another theme.
And then lastly, you already touched on this, Jesse, in your intro, but background agents, which go by many names.
I've heard everything from async engineering, agent-driven engineering, background agents, autonomous, like full agentic engineering.
So there's a lot of different labels for this, but generally the idea of like, how can we unleash agents outside of the human?
nudging approach of working with AI, but rather more in the background, more proactive, more autonomous.
How can we basically give more work to agents then without being bottlenecked by the human workflow?
So those are kind of the themes I'm hearing about.
The review one is a big one for me or for us that I've seen a lot lately that You know, and to your point, we saw it come up in our DX data, for example, that like it turns out the generating the code wasn't actually really ever the problem.
Turns out we probably had code review problems and things like that a long time ago, but now it's being amplified.
Right.
And so that actually came up in our most recent snapshot of just like code review is now the bottleneck.
And it's interesting to see it.
This is one of those like hindsight 2020, I guess.
But for those who read like the Phoenix project and things like that way back when and these analogies to assembly lines and just moving the bottleneck from one station to the other.
It's like, it's all rushing back where I'm like, oh, look, it's that's happening.
So yeah, it's super interesting.
If we if we maybe go one step back, right, we talked about, again, this shift from adoption, but I imagine there's still lots of folks like it's it's easy to feel left behind in the conversation, even week to week, honestly, I'm sure there's some people thinking like, how do you ensure the company is ready?
for AI, is there such thing as ready?
Like, is there a checklist where before you go enable mass enablement, you do this or, or teams just kind of letting it happen and then catching up?
What are you seeing there?
I think this is a really relevant question of this audience, because I think this is where the bulk of the work and opportunity lies for platform leaders.
developer productivity leaders.
It's true.
A lot of leaders I go talk to will lament or comment that their environment just does not seem well suited or ready for AI.
What they typically mean by this is it's actually a lot of the things that we associate with developer experience.
So they'll talk about, hey, like we don't have like standardized develop.
development environments that are actually like, you know, have the necessary tools for agents to produce good code.
We don't have good feedback loops, you know, things like CI, tests, security guardrails in place to allow agents to produce code that feels safe, right?
We don't have documentation about our software systems, our different repositories, how these systems interrelate.
So when you go have an agent try to work on a feature, it goes and spins up Kafka, even though you're already have rabbit MQ in place.
Right.
So, um, these type, this I've heard a lot of leaders call this AI readiness.
And I think, uh, it often comes up that the irony is like, these are actually the same things that we've been talking about for a long time in terms of developer experience.
Those same bottlenecks are bottlenecks for AI to.
be reliable and effective.
And so this is an area I see a lot of organizations reinvesting in now, sort of reframing as agent effectiveness or agent experience and sort of reinvesting.
But it's a journey, right?
Like no different than it's been a journey for a lot of us to improve these areas for developers, to tackle these areas in this new world.
It still takes time.
And so it's really a transformation that everyone needs to undergo in order to get their environment in a state where AI can work reliably.
I imagine tangential to that is when you're getting ready to enable or you're getting ready to, you know, again, you're catching up from feeling left behind, like tool assessment feels like a thing that I think a lot of this audience has probably either gone through or going through.
And I'm curious, is the important thing, which tool you use, like Cloud Code versus Cursor versus, you know, anything else?
Like, is the differentiator like how people are using the AI tools?
or the tools they're using or kind of what have you seen there?
I mean, it would be incorrect to say that there aren't material differences in the tools.
It's also true that the tools are evolving so quickly that You know, there's no advice I could give on this call right now that would hold true for probably longer than 10 days.
I would say that generally speaking, I don't think success hinges on the tool you choose, right?
The models continue to improve.
So all of us to some extent are, and I've heard leaders talk about, look, you know, what can we do other than sit around waiting for models to improve?
I think that's kind of the question for all of us.
Like what can we do to...
increase the leverage from these tools.
I think that ties back to the things we talked about, right?
Making sure context and the environment that AI works in is as rich and effective as possible.
There's also other things, right, we can do in terms of enabling teams and developers to learn how to use these tools effectively, tools we can build as platform leaders to deploy these tools and incorporate them into the SDLC seamlessly.
So a lot of things we can do, but yes, I don't think tool choice is the most important decision right now, because I think all of us on this call, like the landscape is, you know, it's not fair to call it a three horse race, but like, you know, we understand kind of where the tools are at and at this point, the rate at which models are improving.
So that's stabilized a little bit to a certain extent.
I think it's time to look at, you know, what do we do beyond those tools?
Yeah, you said something there that resonated about like the model.
the rapid improvement of the models I kind of saw recently.
And I think we're feeling it too, that where enablement for us a few months ago might've looked like, Hey, how do we roll out a Twilio wide Cloud MD type file to kind of go and embed our standards and give everybody a leg up.
But we're seeing like now the better pattern is every time the model increments like delete.
CloudMD and start over because the instructions are stale.
Tangential to that, I think in the enablement part and talking about the speed of it, I think what comes up a lot is investing in education and knowledge sharing, right?
Like what I see is so far there tends to be these folks that are really far on one end of the spectrum.
Like their monitors are not big enough to run enough terminals that are like adversarially coding against each other.
But then you've got the other folks that are like, I prompted six months ago and like, it wasn't great.
And the middle ground is very sparse.
So then you start talking about like, okay, well, how can these people teach those people?
But also at the same time, like the second we sit down to be like, okay, what would a curriculum look like?
The ball moved.
Do others feel that pain from like the conversations you're having?
Yeah, I mean, 100%.
I mean, I think similar to how difficult it is to really ground ourselves in reality at an industry level, right?
Like what is actually a realistic expectation around what we should be able to achieve today and the next 12 months.
That's a very difficult question for leaders to answer right now.
I think it's similarly actually pretty difficult to answer that question, even just as a developer.
within an organization, because probably in your organization, Jesse, there's probably developers sharing clips of like them doing crazy things with, with, with, crazy things.
And, you know, then you start wanting to ask questions like, okay, well, like, does that rep that is that workflow?
Is that portable to like all developers within your organization?
Is it actually reliable?
Was it cost effective?
Right.
That's actually the elephant in the room.
So I really think we're still at a point of fostering experimentation, acknowledging that we don't know exactly like what right the right way, the official way, what will look like.
I think it's more important right now to encourage that experimentation.
Ask good questions.
Ask hard questions about cost.
I think that that's increasingly important.
And we'll talk more about that today.
I don't think there's a playbook that can be written right now, to your point.
I think it would be futile to attempt to do so.
Maybe along those lines, something I think I uncovered talking to other leaders and reading last year was one really tactical approach, perhaps around the enablement is to make it someone's job.
Right.
Again, not necessarily a new, a new thought, but specifically around AI enablement, like, Hey, if there was a team responsible for thinking about the tools, thinking about the workflows, sharing those more broadly, et cetera.
So I'd share like, that's an experiment we're trying.
We recently set up a productivity team, but it's interesting because it immediately comes up of course of like, well, how do you measure the success of the enablement team?
Any, any thoughts there?
Yeah.
And I don't think this.
question and answer really has changed with AI.
I think we've always had a little bit of a challenge in terms of, well, how should we think about the quote unquote productivity of an enabling team?
Because, you know, sort of pure product velocity and end velocity doesn't really apply.
I think our advice has always been to think of it more in terms of outcomes.
And really the outcome that an enabling team is looking to deliver is productivity increase.
to the teams they serve right so i think that the outcome to be focused on in terms of success is you know to what degree have you lifted up the the metrics or end velocity for the teams you're enabling rather than you know what is our product or edge velocity as a platform team right so we're starting to scratch on on more of the like measuring, right?
Conversations I've definitely been having and I know others are having, you know, it's an interesting shift.
I think what platform or developer enablement teams used to work on was the cost of it was more just like, well, we employ these folks and that's the cost of like them making people more effective and having leverage.
But now with AI, there's actual ROI conversations, right?
Because it's like dollar in, how many dollars out?
What's your thoughts currently?
Like, how do we...
Do we measure ROI?
How do we measure ROI?
How do you maybe combat the conversations if that's the tactic around correlation versus causation?
You know, hey, we still shouldn't be measuring ROI or if we do, it has to be this way.
Like what's the current thinking from your perspective?
Well, I'll say a few things.
You know, first of all, the correlation versus causation conundrum, nine out of 10 organizations.
I go meet with, I feel is making a little bit of a mistake around the correlation versus causation.
Generally speaking, analyses that focus on the question of do people who use AI more have higher code throughput, like that analysis is generally, I would say a little bit flawed because typically the people who use AI more are the people who coded more in the first place, right?
So that's a problem.
I think in most organizational settings, a longitudinal analysis is more telling.
And you may have seen on our newsletter, we're about to publish a meta longitudinal analysis across a bunch of companies.
And the findings are really illuminating.
Longitudinal analyses are also sometimes challenging just due to data availability.
Like you need longitudinal data that's clean.
You also need to still account for compounds.
So like one, I think.
pretty serious compound variable right now is actually just the heightened pressure at a lot of organizations to increase throughput, right?
Like leaders are demanding higher throughput and higher levels of AI and code activity.
And so I think there is just increases in throughput due to, you know, good arts law there.
So longitudinal analysis is my recommendation, you know, when possible.
And that doesn't mean that You know, just comparing cohorts cross-sectionally, like I mentioned, is useless, right?
You know, we operate in businesses.
We have to answer certain questions the best we can, even with imperfect data.
And so it's okay to do that.
But I just want to call out statistically, you know, I don't think that's the thing to be drawing long-term conclusions from.
In terms of ROI of AI in general, I think the more time I've spent with customers in our research looking at the data, I think it's helpful to think of it in two.
bucket.
One is amplification, which Jesse, I know you're a big fan of use that term already.
Right.
And that is really about like, how much more productive are humans, right?
Thanks to AI.
And for this, we look at things like throughput.
So you know, our engineer is able to deliver more you by using AI.
Time savings, like how much time do developers feel like they're saving in their work?
Specific workflows, things to AI or new tools that incorporate AI.
Developer experience scores, right?
So to what extent are AI initiatives or AI tools just improving the broader developer experience, which in turn, we have a conversion of developer experience to time savings with the developer experience index.
So that's amplification.
I think that's one bucket.
The other bucket, which is...
in some cases more theoretical than actual depending on where you are in your journey but you know i would call it augmentation so it's really like to what extent and this goes back to like background agents a little bit we're talking about but so like augmentation is to what extent are you actually extending your organizational engineering capacity by incorporating agents as like headcount into your workforce right so like how much work are agents driving and delivering?
How much are we?
And of course, with all this, the divisor needs to be cost.
So like, how much work are agents delivering?
And how much is an interesting question, right?
Like we can look at that in terms of like throughput.
One thing, and we had this as part of our AI measurement firm published last year, a unit we like to think about as like human equivalent hours.
So like, how much work are agents delivering?
How much would that have taken a human?
And then you divide that by how much Did it cost?
Right.
How much should we spend?
And the result is actually like an agent hourly rate, which I think is a really interesting metric because.
Yeah.
Yeah.
That's going to hopefully be pretty low.
So it tells you, hey, like this is a place to invest, like by invest, like agents are a really efficient workforce for us.
Right.
It's a high ROI place of that.
So that's how we think about augmentation.
So again, I think amplification, augmentation, I think this is a good way to think about.
like different bits of data we're collecting.
I think it's also a good way for leaders like you and others on this call to tell the story, right?
When you go to executives and you're talking about what are we doing with AI?
Well, to a certain extent, we're amplifying our humans.
And to a certain extent, we're trying to augment our workforce with agents.
And so I think it's a nice mental model.
There's a couple of things there.
The background agent thing I mentioned earlier, it feels like that's kind of just clicked in the last like.
two months, not even click that like, oh, all of a sudden everybody's doing it, but everybody all of a sudden understands it at least a little bit more of like, oh, what it could be.
I think how we've been thinking about it, and maybe it's along the lines of that, like, you know, staff augmentation of like, you know, everybody's got KTLO.
Everybody's got these heaps of like.
there's just these third-party dependencies that need to be upgraded and this and that and you know there's solutions for that kind of stuff like dependbot or renovate or whatever but there's an interesting thing that we stumbled upon just recently where it's like well even if like a version of a thing gets bumped what if the signature changes like the the renovated dependable like they're not solving that so if you then tell an agent hey did you notice that package got bumped like does anything else have to happen that that's that's work that obviously we would point an engineer out historically but what if instead you just woke up and it was done and so to me you know you mentioned the amplification i i get to internally i don't have to sell anything so i don't use the word productivity but i've always used the experience part and to me that's just it's just wildly good experience right like you wake up and it's like oh all these things just got done and and it just makes people happier hopefully right so so that's super interesting you mentioned the the kind of cost you know, cost of an agent kind of thing too.
Something that I was thinking about recently and from talking to others too, you know, maybe a difference between like enterprise and startup, I've been looking at like the cost of people burning tokens.
But depending on your context, it's what I've found is it's either a lot or a little, but the same token burn, right?
So in public, you know, FANG companies, maybe like compensation is very high and you look at the token.
cost of one of those engineers.
Token cost as a percent of comp is like very, very low.
And you're kind of like, well, why would we hire?
And I don't actually mean to take it from the headcount perspective.
Them burning those tokens is free in comparison, like compared to how much more they can do.
But I would imagine that's a challenge at a startup where...
comp is, you know, dollar comp is a little bit lower and somebody might be burning half their salary in tokens.
So I don't know if you have thoughts there, but that's kind of like recent observation.
The presumption right now, the working assumption is that, you know, every token dollar spent is a good dollar spent.
Right.
I wouldn't disagree with that, I think.
But I do think that we're going to have to refine that a little bit.
Right.
I mean, there's a lot of cases of token spend that is probably difficult to rationalize when you really zoom in.
And, you know, I saw a comment earlier about just better analytics on like how tokens are being spent, how engineers are prompting these tools.
Like, and of course, what, what's the, what's the outcome are we getting?
You know, we just published some data again, the full papers coming on.
Look, we're seeing like 10 to 15% overall throughput gains longitudinally year over year.
Let's suppose two thirds of that is attributable to AI.
So let's call it, you know, 8 to 10%.
What does that mean in terms of spend, right?
Like, so what's the ROI we want?
Does that mean 8 to 10% of an engineer's salary worth of token spend?
I don't think anyone's figured out the formula, but I think when you actually zoom into the numbers, this working assumption of, oh, just like.
burn tokens at all costs, I think, you know, starts to become a little murkier.
So I don't have the answer.
I would just say that I think more, you know, I think we're all starting to think more about this.
And then, of course, the costs are so variable and still changing.
The pricing models are changing.
It's hard to pin down right now.
So I think, you know, I would see this as a next 12 month initiative for a lot of leaders, not something where they're like.
sold today right now and there's just a good call in the chat too so i'll acknowledge that like it is definitely model dependent and things like that you choose the wrong model to do a lot of the token spend and it's not palatable versus a cheaper model things like that we saw you know cloud code introduce reviews and it's like okay it's 25 bucks a review you know you need some guardrails around that yeah that that i would not bucket that in the free category i'm curious you know i think I like to think the industry has matured enough that we understand like lines of code is not a good measure of an engineer.
Is it a bad measure of an agent?
Like, like is, is measuring agents and, and what they would do without a person any different or do we have thoughts there yet?
I think, and I haven't seen an argument against this still, obviously disputing lines of code is what folks like me in this space have, you know, spent a lot of time over the years.
You know, I would say.
you know, lines of code is a noisy metric with a simple reason.
A low effort change can be many lines of code.
High effort change can be few lines of code.
And so it's just noisy, right?
And I think it becomes even noisier with AI generated code, which has the tendency to be inflated in terms of line count, right?
That's not, I don't think that's a controversial statement.
And so for those reasons, post AI and pre AI, we've always preferred.
metrics like PR throughput, which we feel again, still imperfect, but it provides a little bit more of a normalized signal of like change throughput.
So how many atomic changes are we pushing through the system?
And, you know, I think that's a less noisy metric.
We at DX, we've also developed a true throughput, right, which is like weighted PR throughput, which actually does incorporate lines of code as one of.
the mechanisms to sort of weight PR throughput.
But again, I think all these metrics are imperfect.
So you're dealing with like degrees of imperfection.
I do think lines of code is significantly less perfect than PR throughput as a high level signal.
There's other research and just kind of empirical, like a lot of big tech companies have spent a lot of time trying to figure this question out to have generally landed at change throughput as their preferred signal for this type of thing.
Yeah, it makes me think like Again, back to kind of like DXi as maybe the really thing, the really clear, clear, unclear, because it's calculated, right?
But like the thing to actually optimize for what you were just saying made me think back again to experience where I've got somebody on my team, for example, that it's funny how like the art of possible is just limitless now where he specifically has said to me before, he's like, I want to do 20 changes a day.
And where that used to be like, oh, that's a great goal.
Like, I understand you're being hyperbolic.
He's not anymore, right?
Like he's actually approaching something like 20 changes a day and it's tangible.
And he's one of those, like he's far on the end of the spectrum.
Monitor's not big enough for enough agents, but it's quite interesting to see the realization here that like, yeah, anything is actually possible now.
So it's wild.
I'm curious, maybe as the next question there, like one thing we're seeing the impact this has on engineers role, like the role of the individual contributor.
I think one way I've certainly seen, I think the industry has started talking about is.
The role of the software engineer will go more to like orchestrator or I've seen folks even just equated to leadership where typically as software engineers get higher in their career, they take on a lot more leadership and they are doing that orchestration even outside of management.
But now it's like, you know, will it be that we're expecting level one and level two engineers to like lead teams, quote unquote?
What are you thinking there on the orchestration of work rather than just doing work?
This is one of those examples where it's a little bit difficult to get down to reality.
Certainly, there's a lot of examples out there within our organizations and on Twitter of developers being orchestrators, right?
To what extent is that actually how engineers should work?
day to day in most organizations, I think that's a little murkier.
To what extent are engineers well positioned to be effective at that way of working is another question.
And to what extent do engineers actually enjoy working in that way is the third question.
I think a couple thoughts on that.
One, I think that a lot of developers prefer to being in the code versus spending a lot of energy drafting requirements and specs and reviewing other people's code.
So take that for what it's worth.
Like my guess would be most engineers would prefer to be in the code than just reviewing code and drafting requirements for agents.
To what extent are developers well positioned to be effective at this?
Well, I would posit that like on most teams, it's the definition.
It's actually the defining and deciding of what.
to do that is more the bottleneck than the coding and doing it part and it's also i would say arguably the the harder and rarer skill to hone right to to have that good judgment so suddenly saying hey like if we just took all our engineers today and just gave them three ics with infinite coding capacity that report to them like what would be the outcome like would our organizations be more productive in terms of like outcomes, I would probably guess no.
And this goes back to like coding was never the bottleneck, right?
And just more engineers, it's like the mythical man, like just adding engineers is actually make things go faster.
So, so I don't have the clear answer to this, but I would say it's not as simple as, oh yeah, like every developer is just going to be an orchestrator and that's going to work well and be a creative to our organizations.
I think, I think we have to see, and I think not all engineers are going to want to, or be well suited for being orchestrators.
What does that mean for the role?
What does that mean in terms of upskilling?
And maybe this is just one of the really difficult human bottlenecks to actualizing the ROI we envision out of AI.
It actually boils down to human judgment, and that's the bottleneck.
Yeah, super interesting.
How are we seeing AI influence work from home or RTO policies?
Well, right now, AI is putting a lot of pressure on all companies, and therefore leaders are, I would say, generally...
cutting work from home.
Like, you know, it's more about the kind of meta power balance of the workforce versus like corporations.
And I think AI, the dynamic on the market right now and the employment market is such that I think, you know, there's a shift toward leaders cutting our work from home policies and RTO.
So that's what's happening now in terms of how's that impacting productivity?
I don't know.
There's too much other stuff happening to find that signal.
You mentioned kind of the engineers may not want to spend their time like developing requirements kind of thing.
That's an interesting one.
And I think it was also another maybe prior observation of the AI adoption that was like, oh, maybe I'll imply a little bit of two cents here.
It's like.
Theoretically, we've always said like, oh, you should write a well-formed spec so you understand what you're building and stuff like that.
And maybe that goes in the same bucket of like, you should write good docs.
And a lot of times we cut corners there as an industry.
But it is true that if you give a well-formed spec to the AI, it does significantly better.
Do you have any pulse on how folks are...
doing there?
Like, are they starting to write better documentation?
Are they doing spec first?
Or is it maybe still, you know, if the alternative is still just vibe coding?
What are folks doing?
I saw a funny, like comics circulating today, like, like the like the best spec is actually just code, right?
Which was kind of funny.
Spec driven, but like, buzzwords aside, I think it's true, like, The AI is only as effective as the instructions given to it.
So that's table stakes, right?
And by instructions, it's not just there's different types of instructions, like literally like do X, Y, Z, right?
That's one way.
You know, understanding of the broader requirements, understanding of the broader system environment it's working in and specific tools and choices to make.
I think this is a hard problem.
Because I think it's like just human tendency.
We're lazy.
And like doing this type of thing is not super gratifying because you're not producing.
You're just setting up the foundation for producing.
I wouldn't say I've seen any, again, specific playbook for how to do this super well at an enterprise scale.
But certainly I'm seeing a lot of platform leaders pay attention to the fact that Things like documentation, having good kind of base level foundational context for AI is really table stakes.
And then, of course, at the individual engineer level, how to properly steer these tools is a skill and a challenge as well.
I'll share even internally, we've seen an explosion of GitHub seat requests, more seats than we have engineers.
And we were kind of scratching our head of like.
what's going on.
But it's like everyone's a developer.
And of course, if they're writing code, we don't want them to like store it in Google Drive.
And so we should give them a license.
But yeah, so like that this expectation of non-engineers or is there an expectation more broadly?
Is it certain companies?
Like what's do we think there's a pattern there yet?
Well, what is a pattern is what you just described, which is I think a lot of organizations are trying to make a cultural shift where.
you know, all employees, particularly in R&D, so your designers and product managers, business analysts are all expected to, there's a blurring of the edges of roles, I think, is sort of like what we all see as possible.
And so there's a push in that direction.
You know, again, in terms of like, how do we manage that?
It sounds like you're navigating that right now, Jesse.
So yeah, I guess they need GitHub seats.
They need...
uh you know cloud code licenses like what does this mean in terms of how teams managing production software need to operate again comes back to like guardrails and you know does this introduce more review bottlenecks and i think those are kind of like situational questions that yeah i'd share from the platform perspective i mean it it poses new questions for sure right like where You know, we give out AWS accounts for different reasons, or we have certain guardrails around what can go into production, things like that.
Or we even assume like, hey, every code base is going to make its way to production.
So we have branch protections and things like that.
It's this new shape, right?
Where it's like, okay, well, yeah, that team in marketing needs an AWS account, but it shouldn't look like that one, right?
It should not have PCI and all this other stuff.
So it's interesting because now platform is potentially serving an even broader audience and not similar personas other than they write code.
So that is a bunch of new conversation where it's like, oh, we know how to create GitHub orgs.
No, this one's different.
And what does a new paved path look like?
We talked a little bit about the SDLC earlier.
One question that somebody asked is like, Do we think maybe sprint ceremonies in like their traditional sense, like do we think these are being challenged even?
Like is the two week sprint like dead, broken, dying?
I think there's a lot of questioning of traditional process and workflow in general going on.
I mean, from like my point of view, in terms of what I'm seeing at companies and also what we're seeing at our company, at both DX and Atlassian, I don't think planning is dead.
Planning and prioritization.
Right.
Like back to what we talked about, like you can have all the energy capacity in the world, but if you don't direct it at the right things, you waste money and time.
Right.
With one caveat, that prototyping can be done a lot faster.
So certainly you can explore and spike on things faster.
But when it comes down to like, you know, what is our focus?
I think you still need.
rituals and processes for determining where you're going to spend your focus, your mental capacity, your tokens and your people.
I won't comment on like two week sprint cycles.
Again, I think that's more of a human.
I don't think AI really affects the cadence in which we want to meet and plan.
The question about whether we should be using like story points was certainly interesting.
I still don't think that goes away.
Again, it's not reality to think of all coding tasks as being reduced to a cost of zero, right?
Like humans are in the loop.
Humans are steering and overseeing and reviewing the code.
So there's still significant cost and time that goes into developing things.
And so I think it still makes sense to when we're prioritizing, you know, kind of like understand the relative time and cost of different things is part of prioritization.
Whether you do that in story points or t-shirt sizes or some other unit.
I think is, you know, an open question as it always has been.
I've also seen discussions of like, well, do we incorporate like token cost estimate into estimation?
I don't, I don't, I haven't 100% like understood like what the value of that would be.
That's kind of like an implementation concern.
I don't think you necessarily know upfront how many tokens you're going to burn when you go work on something, right?
Because it's an iterative process.
So that's my thoughts on that.
Okay, we have somehow come to the very top of our time there.
I want to make sure I take a quick second to say thank you to Jesse for hosting the conversation today and Abhi for taking the time to answer these questions.
And of course, thank you all so much for joining us today.