# Roblox AI Native Engineering Strategy

**Podcast:** Engineering Enablement by DX
**Published:** 2026-10-30

## Transcript

We already have a different name inside of Roblox.
We call ourselves builders.
Everyone is a builder at Roblox.
And what we've seen is the lines blur between engineers, designers, product managers, to the point where these tools have been like an equalizer.
And we have product managers that are like fixing bugs and shipping new features into production by themselves.
Now we're seeing that as almost like part of the job is that you can lean in and actually build as much as a software engineer.
Welcome to the Engineering Enablement Podcast.
I'm your host this week, Brian Houck.
Before we jump into the episode, I want to tell you about the CAFES framework, a new research paper co-authored by researchers from DX, Capital One, GitHub, University of Victoria, and Google.
It explores why context quality has become the central bottleneck in agentic workflows and introduces five properties to help teams make their AI agents more effective.
When teams provide better context, they build more effective agents and achieve far better engineering outcomes.
To learn more and read the paper, head over to getdx.com forward slash cafes.
Hello, welcome to the Engineering Enablement Podcast.
I'm your host, Brian Houck.
Very excited for this week's guest.
We are joined by Andrew Swerdlow, who is VP of Engineering at Roblox.
where he leads AI acceleration and core platforms.
Andrew's working on some really interesting problems on what it takes to make software engineering genuinely AI native.
Before Roblox, Andrew led trust and safety engineering and AI fairness at Instagram.
He spent nearly 16 years at Google working across developer tooling, privacy, and products like YouTube and Android.
Andrew is also the author of Tech Leadership, the Blueprint for Evolving from Individual Contributor to Tech Leader.
And on a more personal note, Andrew and I are of the same sort of branch of the developer productivity tree he studied under the amazing Dr.
Peggy Story.
And so it's always fun to have someone that sort of came from the same lineage of developer productivity as I did.
Andrew, welcome to the show.
Thank you so much.
I am so happy to be here.
I'm so grateful that you gave me the invite and happy to always talk with you.
So, yeah, I'm looking forward to today's conversation.
Great.
Let's jump in.
You've said that we've solved typing, but we haven't solved trust.
What does it actually take to close that gap and get from AI that can generate code to AI that you're comfortable letting act autonomously?
Yeah, I think that that is really the hot topic right now, because what we've noticed is that it's pretty easy to generate code.
And, you know, that's what we're seeing.
A lot of PRs are being generated.
But the overall lead time into production hasn't really moved all that much.
It still takes about the same amount of time to ship into production.
And so we start looking at, well, we have all this code and where's it getting bottlenecked at?
And the first stop is actually it's code review.
And hopefully we're going to talk a little bit about what we've been doing to solve code review.
testing and debugging production and all these other areas where I don't think AI has been as effective yet.
And so what we really need is we need AI to be infused across the entire software development lifecycle to actually get the full promise of it.
But to do that, to actually give AI the keys to be able to do a deployment, you have to trust it that it's not going to create a salve.
And so...
that's where the gap is right now is being able to know that an ai agent is is not going to cause a regression in production or create a save or ultimately you know ship some sort of slop or mistakes into into your into your product and so One of the ways that we've been thinking about this at Roblox is how do we put in place those guardrails?
What are the sets of controls and system design that would be needed to prevent bad things from happening?
And as it turns out, those sets of guardrails, those sets of trust systems are actually the same ones that we've been saying for almost a decade are important for us to invest in.
So it's things like great unit test coverage, it's integration testing, it's canaries into production, it's the ability to detect and rollback issues.
And so actually the companies who I see moving the fastest are ones that had been investing in continuous deployment and good testing infrastructure all along, because now they have those guardrails that are in place that help them establish trust.
So that's like a piece of it.
So, I mean, I've said on this podcast before that.
AI acts as an amplifier, right?
Like it shines a spotlight on a lot of the properties of your engineering system that were always true.
As many organizations think about trust, a big question is where do we actually need humans in the loop?
And I know like you've talked about humans in the loop can often act as a tax.
And so when is human oversight actually good safety engineering?
crutch because we haven't built the right systems in order to operate without having a human there.
Yeah.
And I think to be clear, humans need to be accountable for the products and code that we build.
And so I'm a strong believer in that.
I do think where we've decided right now to put humans in the loop are not necessarily the right spots.
And so really been thinking about, you know, how do we sort of move from, you know, things like humans reviewing every single line of code that AI generates to humans teaching systems about what kind of code could be good in production and what kind of code is bad in production.
And so that way you're actually teaching a system versus reviewing all the output of AI.
But to do that, we found that there are some systemic issues with the current technology suite that is available to us.
So if you look at most of the frontier models or open-weight models, these things are trained off of the internet data, publicly available data, or like smaller sets of data.
We're talking petabytes of data.
But companies store exabytes of data.
That's like orders of magnitude different in terms of the amount of data.
And so it's no wonder that a frontier model or an external model doesn't understand your organization like internal employees do because it doesn't have access to the same level of knowledge base and experience as businesses.
And so one of the things that we thought about earlier at Roblox is how do we take Roblox?
best engineers and capture their knowledge and infuse it into agentic interactions.
And that's been a journey that we've been on for the last year and a half now and some really exciting results with that.
So there's actually a few really interesting places we can go.
with that.
And I think one of the things I'd like to dig into a little bit more is this notion that, you know, frontier model can know an enormous amount about software engineering, but doesn't know anything about like your actual organization.
I think certainly trying to encode some of those like best practices and the guidance from like your senior engineers is important but what else does an agent need to understand about your engineering system that isn't already in the model itself Yeah.
So, you know, if you're working at a large company with a large code base, you might have picked up some patterns that might be different than the industry norm.
So, for example, maybe you're using like an older version of Python internally.
Right.
And so maybe you're still on, you know, two dot X or whatever.
And meanwhile, the entire industry is like multiple versions further.
you know a good model would probably tell you hey there's like you know some wrong pattern here you need to fix that but actually there might be a conscious decision internally inside of a company to say we don't want to migrate yet that's actually a large set of work and like we're not ready for that and so we actually want to use an old outdated you know language version or library and so you know how how would how would a model know that you have to kind of instruct it on like what behavior is expected and so You know, the way that we think about this is that all this knowledge of like how you should build inside of your company is already there, right?
We have this like very rich historical information of what it took to build the existing product that we have in production now.
And so being able to mine that data and be able to extract all of the interesting insights from that evolution of your product and turn it into some type of judgment that can be used by agents, either for code review or even generating new code, that was our goal.
And so we actually built.
it's a system and it sort of has a couple different components to it.
The first is this idea of extracting, you know, out of the 20 years of historical information of Roblox, can we actually pull meaningful insights out of the code base?
So looking at all of the different commit histories, looking at code review comments, looking at sev data, Slack information, all of those sorts of things, and turn that into interesting signals that could be used to guide agentic interactions.
And so we did a whole bunch of research around this and extracted a ton of data and, you know, did some analysis.
And what we found is that the highest fidelity signal is actually code review comments because in code review comments, this is like your engineers telling people how they should write code inside of your organization.
But it's a noisy, it's a noisy set of data, right?
So people are kind of knitting on things or making sort of relevant comments or using different language patterns.
And so a tough part of the challenge is how do you like extract a huge amount of data, normalize it, cluster it and make it make it useful for agents.
And so we we came up with a method to do that.
And one of the things that we found is that there is sort of like a temporal element to it that actually most of those comments from 20 years of history are not relevant.
are relevant.
Sort of like decays over time.
Exactly.
Decays over time because patterns change, you know, the software changes.
And so we found the sweet spot for this data money activity is looking at just like the past 18 months to two years worth of worth of data.
And so we we have this process where we extract all the comments.
We use an LLM to sort of like normalize those comments and put them into sort of a rule format.
And then we discard a bunch of things that don't look like a rule that, you know, hard for us to parse.
And then we actually We do a clustering of them and we essentially have a simple process of just inserting those comments into a vector database and looking at sort of like clusters of similar comments.
And once we have those, we extract those into something that's like a rule.
So we have another LLM process that actually takes that and puts it into a rule format.
And then we created this nice UI that takes this rule, which we call an exemplar.
and puts in front of different repo owners.
And so you can actually go to a nice UI and see like a set of like 50 rules for your repo that were automatically extracted through this data analysis.
And the cool thing about it is that these have citations.
So you can see this particular rule was cited 50 times.
And if you kind of click on it, you can see like, who are the people that were citing this rule?
And you can kind of see, you know, some reputation level signals around those people.
Oh, that's like the senior engineer.
That's like the person who's most knowledgeable.
This must.
be an authoritative rule.
And so we say that a repo owner can go through that list and then easily click to add each of those rules to the repo.
Essentially, it gets stored as a YAML file in that repo and then becomes a rule that gets followed for things like code review or even new code generation.
And so that's like the exemplar piece of pulling out what we consider historical information.
So it's like, I feel that...
good code reviewers, like, you know, people who leave high quality comments, they have always been worth their weight in gold.
And I think that we like historically have dramatically undervalued engineers who are phenomenal at leaving code review comments.
And I'm curious, like more than just.
frequency?
Do you have some sort of weighting for senior devs versus junior devs?
And even among senior devs, those who have a strong reputation, is there some sort of weighting in how you treat these exemplars?
So right now, it's actually more about frequency.
It seems to be the strongest signal.
We did actually build an expertise classifier that looked at a bunch of different interesting signals like tenure, level, and then also experience within a repo itself because you might have maybe a relatively new engineer just most familiar with that code base and so we actually did build a classifier to help us with ranking of these exemplars but we found that that was less effective than actually just like the repeated citation and so we kind of combined the two a little bit so that you can see a ranked ordering list of exemplars for your repo.
And then you can actually look and spot check who the people are that are citing them and sort of implicitly know whether or not that's an authoritative voice.
But we don't do any formal ranking based off of that right now.
Maybe something to experiment again with it in the future.
So as you mine out all of this wealth of engineering knowledge from the code review comments in order to make your own AI code reviewers more effective, Does that run the risk of you are going to like dry up that future source of sort of like engineering knowledge for the future as you no longer have to like have your engineers like leave their own comments?
Yeah.
So, you know, let's let's let's talk through that.
Right.
Like what would that lead to?
So suppose that we built a system so good that, you know, our code review agents used to write.
all code review feedback for everyone and that people are happy with it they accept it they don't reject it i'm not sure that would be a bad thing i feel like that's actually sort of the end state but we're not there right now we're at a point where You know, I think like about 60% or actually this week's data is almost 70% of all PR feedback from our code review agent is actually accepted by humans.
That still leaves on the table the 30% that wasn't accepted for some reason, as well as, you know, things that maybe have, that might've been missing.
And so I think there's still like a fairly big gap that needs to be closed for code review to be perfect.
But if it was perfect, if it was like only being used and that was like the main source of truth and all comments were generated by AI, I think that means that we.
achieved a high level of abstraction where we can start to trust that system fully.
Got it.
So it's like we aren't going to need that source anymore.
So you had said 70% of AI-generated code review comments are accepted.
How does that compare to human-generated code review comments?
Oh, that's a good question.
So it's what we've observed and sort of like corresponds with the literature is approximately 50 to 55% of code review comments are accepted by by humans that were written by humans.
And so we've been able to do this sort of like before and after as well as we have some some level of holdout groups, not every repo has opted into code review agents at this point.
And so we have some some strong conviction that our code review agent actually produces more acceptable comments than even humans at this point.
Oh, that's really interesting.
I would also suspect.
But it was a journey, I would say.
It was a journey.
Yeah, it didn't quite get there overnight.
Our first attempt was just kind of a naive, like, let's run an LLM on a PR and get it to provide some feedback for us.
And with the noisy feedback that it was providing, only about 20% of the feedback was actually accepted by humans.
And so it's sort of been this long journey over the last year and a half to get to 70%.
So we got to this.
part of the conversation, because we were talking about like, what is all of the organization specific knowledge that you have to be able to feed into the models that they weren't sort of natively trained on?
And you have this wealth of engineering knowledge that's, you know, comes from the code review comments.
But that is all about sort of engineering artifacts that have already been created, like from the decisions we have made, what can we comment about them?
And I'm, you know, it's so much of, you know, why we are making the decisions to begin with, why we are building these things like are often.
never written down.
And I'm curious, like, was that a problem that you faced in Roblox where it's, you know, you have all this information that just lived in engineers' heads, lived in hallway conversations, or have you always had processes to sort of make sure that those things get written down?
I think like most companies, documentation is very hard at Roblox and every company I've ever worked at.
And so what I've learned is the code, the systems around the code, those are the sources of truth.
Those are the most accurate.
you know, sources of references that we can look at.
And so yes, like, like, trying to extract this information directly out of docs, like was not the highest fidelity signal.
And, you know, like slack was actually pretty good.
We were able to like, you know, see some interesting, you know, opportunities in slack, but You know, for us, code review comments was the most solid and attributable and turned into the most concrete forms of knowledge encapsulation.
There are, like some other experiments that we're doing, looking at things like incident reports and, you know, these sorts of things.
We've also provided sort of a way for people to author their own rules that don't need to necessarily be implicitly extracted.
We have a playground where you can author rules and you can test it against like sets of PRs and see whether or not it actually finds the right issues.
So we do have some.
rules that are that are hand-authored a lot of those come out of like sev so we say we have like an incident we want to make sure that we have a exemplar that would prevent that stuff from happening again we go to the authoring tool we can kind of write up in natural language with some like one-shot examples about like what we want to be able to detect and prevent and then we can like test it on like the pr that costs a sev or even you know build some prs that you know try to try to tickle that that that particular rule And so that's another way for us to, you know, sort of like, you know, merge both explicit and implicit exemplar rules.
And I think that that sort of has bridged the gap for us.
I think we're always looking for more signals, but we also want to do it in a way that doesn't become noisy and that is high value.
We don't want things that are nits.
We want things that are really important and would potentially prevent defects from escaping into production.
The interesting thing about this is that.
After we started this process, we had a whole new surface area where we could start to add things like post-mortem follow-up action items too.
Because in the past, whenever you had an incident, usually you'd be like, hey, we need to add some tests.
We need to add an alert.
We need to do these things.
And maybe you update a runbook or engineers just know not to do a certain thing and you create a doc for it.
But those things decay over time.
People leave the company.
Tribal knowledge leaves.
Now there's a way to programmatically encapsulate.
those learnings into something that gets injected into every code review.
And that's something that's an entirely new capability for us.
So I am a huge proponent of code reviews, not just as a way of finding defects.
In fact, I think they are probably not a particularly great way of finding defects.
I think they are a phenomenal way of transferring knowledge of.
you know, raising transparency around like what our peers are working on.
It's a way of creating sort of shared understanding.
And as we automate this away, we sort of lose some of that.
Now, some of that we might not need anymore, as you mentioned, but we still need some, you know, like I still feel like there's something lost if we don't understand what it is our peers are doing, what it is that like.
is being built by the engineering system.
I'm curious, like how you think about that and, and, and what you've seen so far.
Yeah.
I think, um, my, my intuition here is that the things that.
that matter the most right now is is still intent it's the intent of like what this change is supposed to be doing it's the architecture designs about like how you're achieving those goals and it's about like maybe the spec is about as detailed as maybe we'll end up going in the future because i think the the underlying syntax of like you know how a function is labeled and you know what sort of you know naming convention we're using all that kind of stuff i feel like will be below the agentic abstraction layer in very short period of time we're already seeing it now where we've seen this shift from everyone wanting to operate in an IDE, looking at every line of code to being, you know, functioning mostly in a terminal at this point.
And it's not to say that people aren't still looking at the code, but it's less and less.
And so if you project that out into the future, six months, one year, two years, I think most people will be living at a higher level of abstraction for these systems.
And those three things that I just mentioned feel like they matter a lot to me.
It's the intent, it's the architecture, and it's the spec of the system that you're building.
So I love that.
I agree that as we move away from code as sort of like the first order artifact and we really move to intent as like the atomic unit of work, then it's like, how do we track that?
And that's the thing that we're trying to understand.
But I believe I've heard you say that design docs are dead as like we move to a world where it's just like so easy to rapidly create prototypes and like help me resolve that in my head.
Yeah, first of all, I want to give credit.
That was Malta from Vercel, the CTO of Vercel.
He was giving a talk at Roblox.
I think you were there for that one.
But he used to be like one of the main like design doc folks from Google back in the Google days and, you know, evangelizing the value of design docs.
And his thoughts on this were that you don't need a design doc when you can make a prototype faster.
And I agree with that.
I think that, you know, if your goal is to get feedback on like the contour.
of a system and, you know, iterate on some of the details.
that a prototype feels like a good way to do that.
And so I think the time that you would spend doing those sorts of reviews should actually just be going into prototype evaluation and iteration.
So I think, yes, design docs might be dead, long live specs.
And specs, I suppose you could say, is a form of a design doc, but it's more, I think, tied to the direct implementation of the thing that you're trying to build and kind of going through the details of the exact implementation and architecture.
So as we move more towards then spec-driven development, spec being the sort of engineering artifact that expresses our intent, what makes a specification good enough that you can just sort of hand it to an agent and it can work autonomously off of it?
What are the attributes of a good spec?
Yeah.
So I feel like this is still an area of active discussion.
feel like we really consolidated on like a canonical like spec format for for all of roblox it's sort of picked up and used in in different areas at different various degrees but the things that i like about spec driven development is that you know it's it's it's a lot cheaper and it's a lot easier to like iterate and align on like a higher level like design and and specification than it is to actually like you know go and and um spend the tokens on on on building something um and so you know when we're kind of moving into budget conscious, you know, kind of faster iterations, doing like the cheap thing first, which is iterating on some words on a paper is a lot easier to change than even the fast prototype that you can build with AI.
And so, you know, what is a good spec encapsulate?
It allows you to to see what the overall system is trying to achieve, what are the approaches for building the subcomponents, separation of concerns, all of those sorts of things.
And what we found more recently is that even code reviews themselves are actually just being replaced by spec review.
And so you can actually just send out the spec for people and have them provide feedback on that.
And that actually gets incorporated.
It's almost like the code review process itself.
I mean, I don't feel like we have it.
figured out yet, right?
It's just like, I think everyone is trying to scramble on like, what is like the right level of detail, the right level of consistency.
Like I think one of the things we've been wildly inconsistent about as an industry is specs and how we define specs.
And like, one of the things I'm thinking about is as you talk through this, Andrew, is like, historically, the way that we might evaluate a great engineer is, do they provide helpful code review comments?
Are they great at mentoring the engineers around them?
And do they write high-quality code?
Do our principal devs, are they great coders?
Do you envision that we're going to start shifting how we evaluate what it means to be a great engineer to sort of the quality and the consistency of the specifications we write?
Like, how does it change what makes a great software engineer?
So in my experience, what I'm seeing so far is that.
Very experienced seasoned engineers.
They are still able to leverage these systems to a higher degree of impact than more junior folks.
The pattern that I see is like a junior person who hasn't kind of built the scar tissue of building large scale systems or having to maintain those systems.
Sometimes they're a little too trusting of AI.
And, you know, AI in its sort of, you know, sycophantic, you know, way will kind of, you know, build a system that you ask and maybe not push back enough.
Therefore, you'll end up with something that might be suboptimal.
It might work initially, but, you know, it might not be something that will scale or be maintainable or, you know, those sorts of things.
Whereas.
An engineer who's built a bunch of these systems, who knows like the operating problems, they're going to be a little more skeptical of maybe like the first response from an agent and they're going to push back on it and they're going to be, you know, asking like questions about like, you know, boundary conditions and, you know, these sorts of things where, you know, maybe the junior person wouldn't have.
And so I do think it's like this like level of skepticism, this level of pushback, this level of iteration to get to really the ideal outcome.
is something that we're going to want to see in more junior engineers.
They have to develop that.
And I think it's not going to come naturally to them until they actually see that, hey, if I just like, you know, go with a basic spec and take whatever the AI is giving me as the output, if that's going to have to be rebuilt five times.
So sort of like, you know, the half-life of the code in production is pretty short because we didn't do a good job at like coming up with a good spec or a good prompt to design it.
And so, yeah, I think there's going to be a lot of new dimensions for how we think about evaluating people.
Certainly.
You know, most companies were not looking at code output as like the single metric anyway.
And so all the other things that we were evaluating on teamwork and, you know, cross-functional behaviors and leadership and all that kind of stuff isn't going away.
I think it's just more across the execution dimension where, you know, output is easy, longevity and quality and product fit is, you know, is stuff that matters more.
You have to have those scars.
Exactly.
Scar tissue.
As you say, like.
Yes, I think I, on one hand, it's, these are things we always should have valued more, I think.
And it's just now I, it'll be a bit of a forcing function, which I think is a good thing.
I also, as you are talking through this wonder, like, will we change, you know, how will we change the way that we interview for junior engineers?
And like, well, in the future, we'll.
And, you know, junior dev interviews look a lot more like senior dev interviews where it's a lot more sort of system design and sort of system architecture versus, you know, I think we'll move away from, you know, coding whiteboard problems.
Yeah, I've seen a few different approaches that companies are taking.
Some of them actually want to see how well you can use AI, right?
And so instead of doing a coding interview, it's a coding interview with AI.
Can you actually build a system zero to one?
What's your process?
Is it efficient?
I think efficiency is a key dimension that we're going to...
dig into and prize a lot more than we did in the past.
Because if you think about it this way, you could have, you know, two engineers, one engineer is sort of prompting their way to sort of vibe coding to get to a solution and maybe burning a lot of tokens, right?
And actually, the cost is a lot higher for that engineer, opposed to another engineer that's maybe kind of working in plan mode, iterating on a spec, and then ultimately coming up with very nice and elegant solution.
And then, you know, single sort of pass to get it implemented.
leveraging a lot less tokens to do that.
And so my intuition is that, you know, we need ways to evaluate that, to look at, you know, how people are building better, looking at their tool setup, their process.
And that becomes the thing that we evaluate, not just whether or not they were able to converge to a solution, because I think the convergence to a solution will become a commodity.
But how you get there is a thing that we need to evaluate.
So we've touched on a lot of different places where software engineering may be changing, changing the way we do workflows around things like code review or even sort of incident management.
And a term I hear thrown around a lot is AI native development.
And so when you think about what makes an engineering organization AI native.
Beyond just like, oh, we're using a lot of AI tools.
Like, what does that even mean?
Like, what are those attributes to truly be AI native?
Yeah, I feel like this is a term that will probably have a shelf life, by the way, because I don't think it'll be long before every company can claim to be AI native.
You're just an engineering organization.
Yeah, you're just an engineering organization, right?
But I do think there's probably a spectrum of where people are thinking about in terms of AI native.
And the V0 here is that If you use AI deeply for all of your engineering work, that's probably like, you know, the base table stakes there.
And I think most companies are getting to that point.
And in fact, you know, even at Roblox, we've been trying to measure like how AI native we are.
And one of the dimensions that we've been looking at, which many other companies do, is how much of our code is actually written by AI.
I think that was probably the first question that we all started asking ourselves about a year and a half, two years ago.
So for Roblox, most of our code is written by AI.
And we can say that concretely inside.
scientifically because we've built a pretty sophisticated fingerprinting system that looks at every single generated line of code across any different tool, whether it's cursor, copilot, codex, cloud code, and we store that fingerprint.
And then every day we look in our GitHub repo and we try to do that fingerprint matching about what actually made it to a merge line of code in our production code base.
And so we're able to kind of see if all the code is generated and see what goes into production and see what's not.
generated by ai and so that's actually been the interesting metric is how much is actually still brain coded or hand authored real quick question on that andrew because like even like a given line of code might not be all ai versus all human and so like how do you think about that like how much of the code has to be ai for it to be an ai authored line It has to be all.
So we actually take a hash of the line of code and we do a direct hash mapping.
And so, you know, for us, if someone does go in and like, you know, tweaks like, you know, a case, we would consider that hand authored.
And so we're kind of giving like, you know, humans the benefit of the doubt there a little bit.
But if someone had to like go and like specifically look at that piece of line of code, it's talking about attention and it's talking about like their like understanding.
And then the interesting question is like, well, that metric no longer has value for us.
Right.
And so like, like, but actually this.
system that we built has a great amount of value for us because now what we can see is we can see all the lines of code which sort of harness model version were used to produce that code and then you can start to map that to things like well how good is that code what is the half-life of that code in production does it get rolled back because there was a like a bug or a sev so like which models produce like bugs and sevs and those sorts of things And then, you know, we even, you know, try to store things like session information, prompt information associated to what did it take to generate that line of code?
How much did it cost?
How many turns of an engineer did it take to actually produce that production line?
So then we can start to ask some like very interesting things about, you know, what like the cost of a line of production code or.
Costs is in like actual, like not just human time and effort, but also.
Token costs, I assume.
Full token costs, not just like to do the last generation, but maybe even, you know, all of the successive turns in a single session, or maybe even doing session mapping for similar sessions used to like prototype before actually implementing that line.
So there's some very interesting metrics that we're starting to just like kind of scratch the surface with, but also, you know, capturing that prompt stream has been super useful for us because it helps us understand overall efficiency and waste.
And, you know, some very like fascinating metrics to come out of that around things like.
60% of all of our token uses is actually not to generate code, but it's to understand code or to ask questions and that sort of thing.
And so that's a huge amount of spend for essentially knowledge management and understanding.
And only 40% is actually used to...
to generate production code or not even production code, just to generate code and only some subset of it actually like survives to make it into main.
And then we look at things like waste, right?
So we saw a huge amount of waste around context bloats and tool failure silently, cash misses, all that kind of stuff can be looked at once you have great observability into your agentic coding tools.
And so we've actually modified some version of OTEL tracing to be able to leverage our existing observability.
ability stack, you know, based off of like Grafana and open telemetry and using that to instrument the agents as much as possible to be able to do that analysis and find opportunities for efficiency improvements, not just sort of like at the system level, but also using it to coach engineers on how they might, you know, do a better job with their agentic software development.
Like, hey, you have this like 30 turn session with like a totally full context window that's going through compaction all the time.
Did you know that that actually might not produce the best?
results for you and is awfully expensive.
And so, you know, hey, like this can be great information to teach people how to use these tools even better.
Like one other thing that we're experimenting with that we've noticed is like compaction, which is sort of like this process that, you know, Harness will do, do try to like reduce your context window size to like try to make it more efficient.
It's sort of a little opaque and non-deterministic.
And so we actually have been trying to build something around like checkpointing sessions so that if you like create like actual like like like forced checkpoints of longer running sessions and then prompt people to actually start a new session with like the previous checkpoint.
We're seeing some interesting interesting value in that.
One, because you can actually like pick up a session that maybe drops, you know, maybe unexpectedly quit or, you know, works out in some way or you can actually be more efficient with your tokens by, you know.
starting from a previous checkpoint in a new context window.
So I think there's lots of interesting technologies that can be built to help engineers.
And the more automatic you make them for people, I think that will be the win.
That's really interesting because not only is it more efficient, but there's sort of this agentic failure mode where...
agents get distracted.
And so if you have, you know, some hyper relevant context buried in a much larger context window, agents will get distracted.
I'm curious as you, uh, sort of like start new sessions over if that allows the agent to refocus.
So not only is it more efficient, I wonder if you get better outcomes actually out of it as well.
Yeah, that's a great point.
So for CodeReview, we have like a very specific example of that.
So one of the things that we do for CodeReview is we say for every exemplar for a particular PR, each exemplar has to run in its own agent.
its own agent so that way it has its own context window and what we found is we found better better results and more determinism by you know kind of lowering the amount of things that you're asking an agent to do at any specific time And so, you know, the more that you can kind of like separate concerns, the better because LLMs by their nature can be non-deterministic.
And the more that you put in the context window, the more, the less likely it's going to be that you'll actually achieve all the outcomes of the things that you're trying to put in there.
So one of our first attempts of doing our code review agent was actually just taking all of the exemplars, shoving them into a single rule file and having one agent just go and run them on.
What we found is that that decayed very quickly.
It was sort of proportional to the size of the context window and the number of things that you put in there.
And so separating things into their own.
agentic runs certainly in our experience is producing much better outcomes.
So if you have this like incredible map of, you know, tasks that that's trying to be completed to what is the, you know, the model, the harness that you're using and can trace it through, through some outcomes, then I'm curious, like.
how much that feeds into things like model selection and whether like how, how fine tuned that is, or you still leave that to the discretion of individual developers.
So model routing is like, you know, like a huge topic right now.
It's all the hotness.
um we've had like an intuition like many other companies that you know certain models perform better on certain tasks and you know are more efficient and so you know we have experimented a lot with you know building some of our own model routing technologies but what i would say is that most of the frontier companies are actually trying to provide some of these services as well so you know there's you know, a very famous, you know, IDE that was recently acquired by SpaceX that, you know, has model writing capabilities that are pretty good.
And, you know, they are tunable so that you can balance things like cost or intelligence and, you know, those sorts of things.
My expectation is that, you know, all of the Frontier Labs will provide their own routers.
And so I wouldn't spend too much time on, like, building a brand new router from scratch itself.
I would, you know, kind of, you know, leverage some existing thing out there.
This is one of the mistakes.
that we made early in our sort of agentic transformation of the company was we were trying to build things that we shouldn't have probably built because the industry is moving fast, faster than we can move.
And so trying to focus on the things that.
you know, we have the most leverage on that are unique to us has been where we want to put our energy.
And so that's why the whole idea of extracting, you know, all of the judgment and information from our own knowledge sources and inject that into agentic interactions was an important effort for us because that's something that most companies are not going to do at the scale that we wanted it to.
But building a router for a harness that already has thousands of engineers trying to improve the product is probably not a winning bet to invest your time on.
Although.
My cynical side does need to point out that leaving that to the frontier labs or organizations that are associated with frontier labs, like XAI is sort of in that family over there, then they have different economic incentives and also are constrained by a set number of families of models.
And I'm curious, who will end up filling that gap, I wonder?
Yeah.
So when I talk to other folks doing similar jobs to me, other companies, like one of the pieces of philosophy I try to try to leave them with is that, you know, this is still a very dynamic, fast moving domain right now.
And I would say almost on like a weekly or biweekly basis there.
emerging new technologies that really change the landscape and so one of the things that you should do as an engineering leader building a gentic organization is is try to create agility and flexibility and so you know at roblox we are we are we you know very multi-harness multi-model trying not to close the door we we want to use whatever is the best tool most economically efficient And so we allow engineers to use a broad set of frontier level tools as well as open weight, open source models.
And so we believe that, you know, trying to create vendor neutral abstractions or not go through one way doors for specific product features kind of allows us to like very easily kind of migrate between whether it's Codex or Cloud Code or Cursor or Gemini.
You know, these things should be up to the discretion of engineering.
They should vote with their feet on which tool might be the best.
And so if.
someone's providing a model router capability and it's not effective and it's not doing a good job, then engineers are going to vote with their feet.
And, you know, we've seen that.
We've seen that already before where someone will launch a feature, they don't like it.
And then, you know, within days, people have migrated already to a new model and harness almost seamlessly.
There is some work to do, but I feel like that's the job of the dev infra teams is to like make that seamless, make sure that, you know, skills are transferable and MCPs and, you know, no dependencies on special metadata files that can't easily just be rewritten or symlinked, that sort of thing.
As you describe this new future, sort of everyone is an AI native engineering organization, and we are really focused on driving specs through the system and our understanding of sort of line level code might decrease over time.
A lot of the metrics that we have historically used to track developer productivity, certainly, I don't know if they become meaningless, but their meaning changes dramatically.
And so like...
How are you thinking about measuring if you're truly getting to that AI data engineering organization?
Yeah, I think, you know, one of the key things is still goes back to like one of the other topics we're talking about is autonomy, that, you know, truly agentic native companies that are leveraging autonomy really, really well are going to see the most benefits.
But, you know, autonomy has to be earned through trust and through safety.
And so, you know, really thinking about like those early guardrails that we talked about at the beginning around, you know, test generate, you know, test coverage, integration testing, canaries, those are the things that are going to get you to autonomy.
But how do you actually know whether or not you're running autonomously, I think is a question.
And so what we've.
put together is a rudimentary metric around this.
And essentially it's long-term session that, you know, that converges.
And so we have like a metric that we look at that looks at like the P50, the, you know, P75, P99, P99.9, and looks for, you know, how long are sessions getting?
Because that means that, you know, an agent was able to do something that took more than, you know, 30 seconds.
Because if it's only 30 seconds, a person is going to have to be on their keyboard, interacting every 30 seconds to give a new prompt.
And what we want to see is more long-running tasks that actually...
achieve tough projects.
And so we haven't quite.
How do you tie that together?
Because I'm like, I could make a task run for a real long time that doesn't do a whole lot.
Yes, yes, yes, yes.
So we actually, we do, we use AI to evaluate the sessions because as I said before, we have this like very rich session information and we can see all the behaviors of the agents, whether or not like code actually ended up merged into production.
We've built this sort of like, you know, end to end pipeline of data.
And then, you know, when we start analyzing long, long, long running sessions, autonomous sessions, we actually ask LLMs, you know, did this survive into production?
Does this look?
like a good use of time.
How complex was this task?
What kind of task was it?
We developed like a taxonomy that we had PRs with.
We use an LLM to determine whether or not a PR is a new feature, a bug fix, like all those sorts of things, mostly because we wanted another metric besides autonomy, which was actually feature velocity.
And so...
This is like something that I think most of us have wanted for a really long time.
And I have had several failed attempts in my career to build a feature velocity metric by like annotating code, you know, that sort of thing.
But we've kind of reached a point now where LLMs do a very good job at analyzing a PR and being able to determine whether or not something is a feature, a bug request, or bug fix, like that sort of thing.
And so every PR that we have right now is tagged with one of those dimensions.
And we can actually look at, you know, a normalized feature velocity rate per engineer.
And we can see.
whether or not the introduction of new tools actually increases feature velocity.
Because in my opinion, feature velocity is the closest proxy to innovation that a company can have.
Because if you're shipping new and interesting features, you're most likely an innovative company.
And if you're innovating, you're probably going to find some great new ways to delight customers.
And so that would be an interesting metric that I like to use.
Besides that, I do think there are still some other really important metrics that you and me both share as a thing that we like in terms of how are engineers feeling, right?
Do they feel?
they're productive do they feel like their output is more and you know are they happy with the tools and all those sorts of things and so you know We can give everyone access to all these tools, but if they feel like they're still being slowed down by something, code review, overload, or whatever it is, issues with debugging production, then we haven't really solved the productivity problem with these tools.
And so still looking at the qualitative feedback coming in from our engineers, a set of quantitative metrics that are shifting away from just the raw PR metrics into something newer, maybe more like a feature velocity metric.
Those are the things that I think will matter most in terms of measurement in the future.
I love that because it's, you know, I'm a huge proponent of measuring across lots of dimensions and having checks and balances.
And I think if you are analyzing the sessions themselves, I can tell you a lot about efficiency.
Obviously, developer sentiment can tell you about, you know, what is the sort of the human cost of doing it.
And then feature velocity, like I am a firm believer as we move to.
sort of intent-driven development, then sort of our innovation, our innovation velocity, our sort of idea to customers is the way I talk about it, is the most important speed metric.
And that is effectively feature velocity.
Like how quickly can we deliver new capabilities to our, to our customers?
And so I, I, I love that.
I'm curious as you like, as you were looking at evaluating the sessions to see sort of like how long running are they?
that in and of itself can get pretty token intensive.
And so like, do you sample, are you evaluating everything?
Like in practice, how do you recommend organizations look at that?
Yeah, you know, efficiency is still like really top of mind for us.
So we've successfully driven the adoption of all these tools.
Everyone has like all the tools.
They're using them all the time.
And now we're kind of looking at the ROI and the budget.
And so that's always top of mind for us.
And we have an entire AI efficiency team that is looking at, you know, how do we sort of optimize all of the different types of workflows?
So, for example, our CodeReview agent, it actually does an ensemble based approach for CodeReview where it uses like multiple frontier models.
evaluate the same exemplars and it's sort of like alignment engine react loop.
And that could be very expensive when you're talking about many thousands of PRs per day, you know, kind of fanning that out for like every exemplar, you know, like could be, you know, 50, 100 exemplars per PR and then multiple malls.
And so we focus a lot on trying to make these systems optimal.
And so one of the engineers, a great engineer on the team is the tech lead.
His name is Kushan, go Kushan.
He came up with a really nice caching strategy for all of this, where, you know, we noticed that when you select a set of exemplars.
you're actually operating in a very specific area of the diff.
So maybe it's like exemplars around testing, for example.
So actually the type of context that you need to evaluate those exemplars for that PR.
is going to be all the tests.
And so what we do is we pre warm the cache with a bunch of this sort of metadata around the PR around like, you know, those tests, and then we leverage like that cache between agents to be able to run those exemplar evaluations across like the same context data.
And it reduces the overall cost to run each of those exemplars by 90% because most of the data, most of the cost is actually the context data.
And so like, that's just like an example.
But for the session, analysis.
So for session analysis, actually, there's a lot of like deterministic, just like, like normal code that's being used to capture that data and storing it in an efficient way for us.
And so we actually use, like I said, open telemetry as like sort of like our tracing data, plus like some other pieces that we ended up building ourselves and probably might open source in the near future and storing it in like a really like efficient database.
So we actually use like StarRocks database that, you know, is quite efficient.
And then, you know, we do run like deep analysis on most of those prompts, but we try to do it in a more cost effective way because it's actually, it's not real time data.
This is sort of like more like async data that can be run at nighttime on, you know, cheaper models and more efficiently, even open weights, you know, can be used to like extract insights from these things.
This is not the stuff that you need the high.
level of reasons that reasoning like extra high you know top frontier model you can actually offload that to more efficient systems so it's about choosing the right thing for the right system yep the the right tool for the right task I do think the hardest part of AI right now is actually system design and infrastructure.
And so the hardest problems to solve for us right now are, you know, how do we build like trust into our systems?
How do we make these things cost effective?
And how do we extract the ROI that we all want?
So sort of my final question for you today, as you describe...
the future of what engineering organizations might be looking like and sort of operating at these higher levels of abstraction.
You know, we've talked a lot about how the role of a software engineer may change.
I'm curious, like in the future, you know we even call each other software engineers or it's like is there another term for it do you think yeah yeah you know we we already have a like a different name inside of roblox we call ourselves builders um and everyone is a builder at roblox and and what we've seen is like the lines blur between engineers designers product managers to the point where you know these tools have been like an equalizer.
And we have product managers that are like fixing bugs and, you know, shipping new features into production like by themselves.
We had less of that before.
There was always a few like really, you know, hands-on PMs.
But now we're seeing that as almost like part of the job is that, you know, you can lean in and actually build as much as a software engineer.
But I do think software engineering as like a discipline and a domain and a skill set is still here to stay.
You know, I do believe that.
Some of our most senior engineers that have experience building high scale systems, they know where the pitfalls are.
They know how to come up with the right spec and the right system architecture so that we build the right thing once and not have to rebuild it over and over again.
And so I don't see that changing.
I just see the details of the work shifting.
But I think this is a transformation that we've seen in our industry multiple times.
My parents were in university and studying.
They were using punch cards and writing assembly and all these sorts of things.
And then higher abstractions and languages came in.
And for a while, I think people were probably still looking underneath the hood and checking the assembly and maybe correcting things.
And then eventually the abstractions became strong enough that you don't need to go and poke under the hood.
And people just trusted those language abstractions.
And if there was a problem, you would solve the problem using the abstraction as well.
And I think we're just in that transition period with AI right now where, you know, we're still trying to poke under the hood and make sure everything is optimal and correct.
But as we build trust.
Yeah, we're still building trust.
Exactly.
Exactly.
As you think about then automating code review, like one of the concerns is, you know, you have the model vouching for itself.
And so I'm curious then, like, how do you know how much you can trust the AI?
to say, oh yeah, this AI generated code is good.
How do you build, like how much trust do you have there?
Yeah, it's interesting.
It's like AI writes the code, AI reviews the code, AI deploys the code.
So like, you know, where is actually the human judgment in the loop there?
And so sort of talked a little bit before where I'm a strong believer that, you know, people need to be accountable for their code.
And so I think that that has just shifted.
So right now, the way we think about it is that all exemplars have to be explicitly reviewed and approved by a human.
And that essentially creates this boundary condition to say this is how like our code review agents should behave.
And so that way it can improve classes of changes and not just a specific code review change.
And we believe that that's like the right spot for humans to be in the loop.
And we found that this has improved accuracy and acceptability of the feedback to the point where we've surpassed even even like the human level of of feedback acceptance.
And so much so that we've actually decided that we want to allow some PRs to flow into production autonomously just with AI code review only.
And so the approach to that has been we have a programmatic check that looks at did the code review agent find anything that would be an issue?
No.
OK, passes that check.
You know, is this a repo that people have opted in for autonomous deployment check?
Is it instead of these other things that are that are verified explicitly by code?
If it passes all those checks, well, I'll do one more final pass where we'll have a LLM review the overall PR and try to do a risk based assessment of it to be able to say, hey, this like even though it's a small PR, this looks like something risky.
And so, you know, yes or no to whether or not we should do deployment for that.
And.
What this means is this means that if it passes those two checks, we can actually do a full auto merge into production.
It reminds me a lot of Meta's system.
They call it diff risk score, which is like, can you sort of assess the risk of allowing this change through?
And if it, you know, the ramifications of letting it through are low or it's easy to call back, then you can do it fully autonomously.
It sounds like you've built something very similar.
Yeah.
And right now we're taking a very conservative approach.
So, you know, the agent looks at things like, hey, is this updating documentation?
Is it updating tests?
Is it like, you know, doing these sorts of things where we have pretty high confidence that the blast radius would be quite low if like, you know, something didn't go right, if there was an issue there.
And, you know, our job right now is to like tune this system to the point where we kind of adjust the risk spectrum to a point where we're able to incorporate more automated deployments, but also not cost saves.
So that's actually the work that's ahead of us.
How has that been received by developers themselves?
So actually, you know, an interesting part of this journey has been like the people side.
So the technology, you know, has been interesting and fun to build and lots of experimentation.
But there's also culture change that has to happen as well.
You know, there's we have entire engineering practices and culture that are just like predicated on this idea that every, you know, production PR needs to go through like a human code review.
Right.
And so like convincing people that these systems are safe enough that, you know, they do as good or a better job than humans.
has been a journey, not just for engineers, but also for our compliance team and for leadership.
They all want to see the data as well.
And so we spent a great deal of time doing evals, doing measurements, even injecting faults into the system so that we can see whether or not we catch them, just to build confidence and trust.
And so that's actually been an important part of the transformation as well.
Awesome.
I did have this one line that I was trying to hope to say.
So I think like one of the things that, you know, I really want to tell folks is like, you know, stop counting tokens.
That metric doesn't matter.
Start counting what code gets accepted, what got rolled back and what shipped with no human intervention.
I think those are the things that are going to matter most in the future for agentic native development companies.
Totally agree.
I think counting tokens is a good way of measuring cost and not a ton else.
Exactly, exactly, exactly.
Andrew, this was phenomenal.
Thank you so much for joining us.
Thank you.
Thank you.
Yeah, it's been a really pleasure and you're really excited to have these conversations.
It's always, always very thrilling to talk to you, Brian.
Thank you.
Have a great day.
Thank you.