# Harness Engineering: Scaling Autonomous AI Code Production

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-06-09

## Transcript

We wanted Codex to be able to connect to our Electron app.
Chrome DevTools protocol, MCP, was how we started out.
Eventually, I kind of took a peek at the code, and it was entirely different.
Still, Codex was connecting to the Electron app over Chrome DevTools protocol, but it was actually a local TypeScript daemon that was being spun up that was providing a little CLI interface instead of MCP because we found that there were only two or three tool calls that we actually needed.
It was more context efficient.
It was faster.
This had happened underneath me without me knowing.
My workflow was not interrupted.
That same dependency edge and entity relationship existed.
But I almost didn't have to know how it was implemented.
Did you find that cool or worrying?
I found it shocking.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
Hey everyone, hope you're enjoying the episode so far.
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
But for whatever reason, many of you have yet to subscribe to the channel.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
All right, back to the episode.
Hello and welcome to another episode of the AI Native Dev and we're here live at AI Native DevCon 2026 in London and joining me is Ryan Lopopolo who is a member of technical staff at OpenAI.
So, Ryan, how are you?
Doing great, excited to be here.
The conf is popping, it's really fun.
I go on stage later today and I'm just excited to get after it.
Absolutely, looking forward.
And tell us a little bit about the session that you're giving later.
Yeah, so I'm going to be talking about all things harness engineering, which is sort of, I guess, kind of like some of the discourse that I helped define earlier in the year here, where talk about what it means to kind of set up agents to do highly complex, autonomous, software engineering-like work in your code bases with high confidence and almost headless nature, if you can get to it.
A lot of fancy context management techniques in order to get these agents to learn from their own mistakes.
produce high quality code and go for a very long time without human intervention.
Absolutely.
And we're going to talk about Harness Engineering.
We'll talk about some of the methodologies that you've used internally at OpenAI and talk through maybe some of the bits that have been enjoyable, some of the bits that have been scary, and it's going to be great fun.
So Ryan, why don't we kick off with Harness Engineering.
For the layman, what is Harness Engineering?
Yeah, so we have these wonderful technology and these coding agents.
these fantastic models that are very smart and able to produce a ton of code.
And harness engineering is this idea that to produce code that is acceptable, to produce code that we trust, that we can build confidence and business systems around, there are a bunch of little decisions that go into the process of producing PRs, producing systems that we have to make along the way.
And in order to get the agent to know and understand those, we have to write it down.
We have to make sure that, like, The content of those choices, all those non-functional requirements that go into making good code, get surfaced to the agent at the right time.
So it's a combination of getting context into the repository in a place where these coding agents can see it, surfacing it to the agent at the right time, and making sure that all the code that we produce follows the golden thread of what it means for us to produce high-quality software.
Reviewer agents, tests, lints.
big refactoring loops, all sort of ways to get the agent to close the loop and make sure that it proves to me and the rest of the team that the code it has produced is acceptable and can be merged.
So we've had prompt engineering, context engineering, and harness engineering.
Now, what would you say are the main differences or overlap between context engineering and harness engineering?
I think really the...
Two only levers you have when interacting with agents is which context you provide and which tools you provide.
And harness engineering is kind of a combination of both with the goal of making sure the context that we give the agent is correct so that it can work its reasoning magic to produce good code.
And one thing that I think is unique about harness engineering is we're actually using the tool calls themselves to prompt inject the agent to manage its context window.
One thing that's pretty neat here is the way that we structure tests and lints and well-formed tests in the code base is actually very different than how I would structure those messages for a human, right?
I would have an exhaustive list of failures that I give the human and have them wade through the logs in order to figure out what's wrong.
Whereas with agents, due to the nature of how they invoke tools, we want those messages to be somewhat compressed, but also semantically meaningful, right?
Rather than giving sort of a mechanical ESLint failure, for example, I can give a prose error message that says, you screwed it up in this way, in this file, you know, please open it and follow the runbook in file XYZ to go fix it.
Because we've seen you make this mistake before and we know that we can trust you to do the right thing.
Yeah.
And that level of integration, that deeper integration really unlocks that next step, right?
Now, interesting, I'm looking at some of your background here.
You built Stripe's efficiency engineering team, led developer productivity at Brex for 350 engineers.
There's a bunch of infrastructure and DX background there.
Would you say that helped you or hindered you from the point of view of Harness Engineering?
It definitely helped.
It kind of got me used to this idea of working through others.
I'm trying to improve productivity for an engineering organization of 350 engineers.
I can't be hands-on in the weeds of every PR that's being produced.
And I can't work directly on the underlying systems.
I kind of have to steer from the background, make sure that the right things are happening by default, the right systems are in place in order to have high trust in the organization to do good work.
And that's exactly what it means in order to be high-functioning with these agents.
in order to move beyond sort of coding agents as pair programming assistants to go massively parallel where i've got the 15 team looks windows open i need to be able to take my hands off the wheel to a certain extent so this way of operating the sort of systems thinking mentality kind of comes pretty naturally to me because that's kind of the expertise i've had to build up over the course of my career yeah Yeah, interesting.
We'll actually touch a little bit about, I guess, the part of that, which is, well, how much trust do you need then to have those 15 open?
But we'll cover that in just a second.
I'd love to, you know, when you have those 15 open, there's obviously a huge amount of agent written code and various different roles in the workflow and the pipeline.
Now, a really interesting constraint that you imposed on yourself and the team was no human written code.
That's right.
And I guess, first of all, How fearful were you at the time when you said, no one's to write any code here in this project?
How confident were you that that was the right course?
It was definitely a bit of a radical idea at the time.
Kind of when I started out building in this way was even before we had proper reasoning models, GPT-5 came out in August of 2025, and I started working in this hands-off way in June.
So this is, you know.
2003 era Codex Mini had just came out, the very first versions of Codex CLI.
And to be honest, back then the models were way less capable.
It was very painful in order to operate hands-off like this.
And sort of by necessity, working with the model, working with the coding agent, I kind of had to stick myself in as a very chunky tool for the model to delegate to.
If it got stuck or if it needed to do something.
The only thing it could do at the beginning was ask Ryan to do something for it.
Eventually, I get tired of being asked the same thing all the time.
Initially, back then, it was installing dependencies with cargo.
This is just not something that Codex was able to do reliably back then.
So I would type the commands in my terminal and give the agent some dependencies to work with.
And sort of automating that was the first step.
And it kind of got me in this mode of...
paying very close attention to where I'm spending my time, figuring out ways to build up a tool call that removes the need for me to spend that time.
So then I can kind of figure out the other ways that I can help the agent get more and more chunky tools to give it more capability.
And to start with zero like that, you kind of viscerally experience the failure of the agent.
I think to kind of have a good sense of The tool calls as they are making, the mistakes the agent is making sort of in the process of writing the code is really, really important in order to build confidence in it.
Because to see what mistakes it's making, I'm also able to see the parts that it does really good at and can start to retool the way I go about my week to lean into the stuff Codex is really good at.
It's really good at following instructions.
It's really good at writing tests and calling them.
So collapsing a lot of the engineering process into these very well-paved roads for the agent helps me build confidence in it.
It's funny how you say it.
It's really good at following instructions and really good at writing tests.
And I'm thinking, wow, the complete opposite of most developers.
Maybe that's wrong to say.
So that's really interesting.
And I guess when you think about your team, like this is a decision that you made, but when you think about your team, how comfortable?
How on board was everyone in joining this?
I got very lucky that as we grew the team, we grew it all with new hires to the company.
So they kind of get dropped into this insane environment and being like, oh, I guess this is just the way we do things now.
Let me just go whole hog at it.
It usually took about two weeks for folks to ramp up to this way of operating.
Some growing pains.
very early on in growing the team um probably a good amount of dog food as well you were able to go back to the team i mean the the product we were building too also helped a bunch we were building sort of a data analyst agent uh a data analyst agent using codecs inside in order to write sql and call databricks and all these sort of things um means a lot of the techniques that we were using to write the code were techniques we needed to deploy into this agent to have it be effective a very cool virtuous cycle there It took about two weeks for folks to onboard.
Initially, very painful in terms of figuring out how to coach folks to not produce slop.
A big part of maintaining high velocity with these tools is just not permitting slop to enter the code base.
But once we sort of had figured it out, by the time we got our fifth, sixth, seventh hire onto the team, within those two weeks, we were actually seeing like...
velocity and PR throughput go up 5, 10, 15%, which is, again, not the common path when you onboard quickly to a team, like the pre-singularity type, right?
And the reason for that is because as we accrued more and more context, more and more capability into the code base, sort of everyone invoking Codex as the sole entry point to the code base was getting the best parts of everybody by default.
Rather than having to spend one, two, three months absorbing the best practices of the team, they're already there.
Which means new hires to the team were able to very quickly supply their best judgment and context, which means now everybody is being more effective very, very quickly.
You're effectively onboarding the new hire through the agent already.
That's right.
Brilliant.
So there's a really nice phrase I'm sure you've heard.
I've said it a million times as well.
You can go fast.
You can only go fast when you have breaks.
That's right.
Now, in this scenario, going fast is using codecs and AI agents to generate that code for us at a faster velocity than we would be able to manually write.
The breaks, of course, are those tests, those reviews that validate the code, that show and highlight where we need to actually spend time and fix and adjust before we go live.
Another interesting thing is the zero human review.
Now, you're putting AI to be the brakes now, which is a little bit more worrying.
In terms of, I guess, an alarming proposal, how comfortable were the engineers when you said, right, we're now going to do zero human review on this code.
We want agents to be able to do that review.
Yeah, so sort of like the idea of post-merge code review, if at all sort of thing is, you know, kind of like not a new thing, right?
When I started my career back in the early 2010s, like this was a normal part of what it meant to write software is kind of have a high trust environment with your team, bias to action, bias to throughput, and, you know, have some representative sample of the code that you are looking at.
Otherwise, have lots of synchronous communication on it as a team to make sure that everyone has the same mental model of the system.
And it's very cool to kind of like have this come back into vogue with agents because if the entry point to the code base is always the agent, it's...
really not required for every engineer on the team to have a full and complete mental model of the system.
So long as that they are able to trust that everyone else on the team is enabling Codex to have that level of expertise.
Which is not to say that like there is no review at all in the system, right?
To kind of provide a little bit more nuance here, we do have some cases where we require two-party traditional review pre-merge.
And most of the time, that is centered around things like highly complex plans, sort of phased milestones that would take place over the course of an actual week, these sorts of things.
And the reason for that is these plans, these milestones, these exec plans function as the prompt you're going to give the agents.
And if you are relying on a text document to drive implementation, being crisp around what words are in there and whether or not they fully under or misspecify the task is quite important.
Because if you misspecify the task up front, you are going to get garbage.
So that's kind of where we focus the...
the bulk of the human review and so the review here like it's it's different to how we would normally think of a code right yeah it's like first of all a code review you know we will argue about details and namings and things like you know almost more style guide style comments um or rather that's what sometimes happens.
Unfortunately, yes.
Whereas now what we're looking at is actually a much more, I guess there's probably a couple of reasons.
One, because it's actually more interesting to actually, you know, review at that higher level anyway, because they're more meaningful input from our point of view, where we need to kind of like much more think about what's the greatest value I can provide here.
And it is that more meaningful decision making at the top.
But also, I guess we're probably more...
abstracted away from the code.
So there is an amount that we just have to trust where actually it's almost a waste of our time to go too deep into the implementation because we need that to be abstracted away and we need mechanisms of where testing just occurs to make sure that that succeeds.
That's right.
This is sort of, again, that sort of like group tech lead sort of mentality, right?
Like when I led developer productivity at Brex, right?
I don't have my hands in the weeds of every bit of Terraform every team is writing.
But I do care at a high level about guiding principles.
I want the infrastructure to be reproducible.
I want it to provision quickly.
I want us to have modules in place that solve common developer workflows.
But how you structure your underlying modules, whether you're using for each in an idiomatic way, I don't really care too much about that.
As long as the code is like...
locally coherent and actually solves the business problems that I'm hoping to do.
And ideally, that's a decision that maybe even change over time, depending on how something should be structured.
So it's not something we should always be so connected to at all times.
Anyway, I think we mentioned kind of like trust a little bit.
I can't remember what the magic ratio of trust is, where it doesn't matter how many times something works, it's how frequently something doesn't work is the key, which breaks the trust.
But I guess at what stage were you and the team, like, truly thinking, oh, actually, do you know what?
I completely trust this.
I guess from my point of view, and I'll talk about myself here when I say laziness, but there is a time when I'm using a genetic development where I'm just kind of like assuming and just being lazy about not reviewing things in enough depth.
But in terms of like true trust as in, yeah, I'm going to hand this off to the agent and believe that this is going to be high quality that I can actually push to production.
At what moment would you say in the...
the process, would you say you were truly trusting of what was being delivered?
I think there were two sort of phases here.
One was, do I trust the agent to be able to produce code well?
And the magic moment for that was sort of working on what was essentially a very early version of what the Codex app is today, talking with a teammate around, man, I wish I had voice input here.
And then being able to just give a prompt and half an hour later see that in the real world in the app was fantastic.
You kind of got this feeling of confidence that it really is down to my capacity for scheduling prompts to go into this magic machine to see my ideas come to life.
So that was sort of like the first phase of getting into this mindset of our role as engineers should be to figure out ways to feed the machine, to schedule work in order to kind of...
have our vision come to fruition.
The next sort of phase was, is this code trustworthy?
Is it high quality?
And this sort of came to us as a team sort of out of necessity.
When we onboarded our third engineer to the team, we significantly increased our PR throughput.
But at the same time, we're unable to keep pace with reviews and making sure that slop didn't enter the code base.
We kind of started to kind of pump the brakes a bit, like you mentioned, and take one day a week Fridays to do garbage collection where a lot of long synchronous standups in this way of working, right, in order to socialize that mental model.
So we would kind of compile a list over the course of the week of.
things that we didn't quite like, right?
We want to bias toward throughput, bias toward merging, but still we want to make sure we maintain our quality bore.
So every Friday was trying to eliminate that slop, but also finding ways to systematically eliminate it going forward, right?
We never wanted to give the same feedback twice in the same way that you want, you know, a new engineer you're onboarding to the team to actually internalize what it means to do a good job.
So...
That kind of stacked into sort of these programmatic guardrails and then a very long outer loop of sort of these automated CI jobs that are looking for slop, identifying slop, kind of compiling a list of golden principles for what it means for us to write good code, finding divergences from it in the code base, proposing PRs.
And at that point, we were able to see, oh.
The agents are able to write code the way we expect.
We just kind of need to put the right loops and systems in place for it to kind of close that feedback cycle for itself.
And presumably that was given to it by context or skills or something like that.
This is before skills were a thing.
There was a time before skills?
Yeah, I know.
I know.
Only what?
Eight months ago at this point.
We kind of did a very cheap hack.
kind of had just a GitHub issue where engineers and agents would leave comments around sort of underlying principles of good software development, right?
How to properly make a React snapshot test, what it means to write reliable...
network code, how to think about structuring a lint.
And you end up with this big issue with 100, 150 comments on it.
And that was sort of the seed for what it meant for the agent to go sort of spider through the code base, stack rank a set of violations, propose PRs.
And then one interesting thing there is we had like an on-call whose job it was to supervise all the agent produced.
stuff all the headless stuff because we we didn't quite trust it yet we needed eyes on it in order to build that confidence uh human with thumbs up thumbs down merge or not leave review feedback which means the next time that anti-slopification loop spins up it looks at all the prs it had produced it ingests that human feedback compares it with the session logs from the last time it ran says which mistakes did i make which priorities did i miss Did I produce misaligned code and why?
Let me figure out how not to do that going forward, produce some markdown, save it as an artifact in the GitHub run.
So the next time I run, I have this extra context that I have learned from the actual human feedback in the system.
And is that the correct way?
Is that the way you would recommend others?
set up their agents to almost like iterate through?
So it's almost like build the code and then do these almost like audits or reviews in post and then raise PRs where you're almost, you know, you're outside of the guardrails?
Or would you today look at adding that into the agent at code creation time?
It's a mix of both.
It's a mix of both.
Yeah.
We do think sort of these asynchronous loops are a big part of what it means to be effective.
Early versions of Codex Security via Aardvark also worked in the same way, right?
Where I'm getting a security review of my code after it has ended up in main.
It is maybe finding a way in which it is vulnerable, producing a patch.
We put up a PR, we review it, and it merges.
And I think this sort of flow is really, really nice because it allows mistakes to actually make it to main, which means they can be learned from.
Anytime we are able to come up with a correction, one, we want to fix it.
but also repeating patterns sort of thing, which we identify with agents and loops and distillation and all of that.
That's the stuff we want to pull forward earlier into the pipeline.
Yeah.
And when you do push with mistakes or things that you want to correct, are you less fearful of pushing those mistakes to production these days because you know how fast those can be reviewed and fixed by an agent?
It's definitely a spectrum, right?
I will say that for this project, we didn't have continuous deployment, right?
We're building a native app.
We are manually cutting a release branch and doing smoke testing.
That smoke testing is initially all by humans, where we look at where we're spending the time and try and get agents to do some of it, right?
But it is important, I think, to have some level of human supervision over the release process in this world.
But...
if I were to kind of like put my group tech lead hat on again, right?
Some mistakes are able to be made.
Like the consequences of them are low.
And much like I want engineers who are becoming more senior to learn from their own mistakes, I also kind of want to see mistakes that the agents make so that we as their supervisors can figure out how to get them to learn as well.
Because it feels like in this kind of scenario, debt and fix is cheap.
Because it's about, well, we'll run an audit.
So long as you have those guardrails and rules in place to say these are some of our best practices, I can put something in which isn't ideal.
And no, it's not going to take six months for a human to come and review this code.
It can be done in a more regular time frame.
Yeah.
Like with this sort of like new world where the production of code is so cheap, I actually think that you...
can, at least working with agents, flip sort of like the DevOps shift left thing onto its head a bit, where the cheapest thing I can do is literally produce a change to my prompt, whether that is like one of those golden principles that I'm yeeting into that GitHub issue or, you know, a prompt that I socialize with my teammates, right?
That's the cheapest thing I can do in order to fix the code that is produced.
And, you know, if we find that like the mistakes are a bit more...
systematic, I can kind of think about pulling that further left.
I can add some docs or I can operationalize a review agent that is going to judge every PR according to those docs.
Or I can do more shifting left by making deterministic tests that suss out this behavior and on and on earlier in the process.
But the cheapest amount of effort I can spend to shake out that misbehavior is kind of where I want to start.
Because...
these things are magic reasoners that most of the time they do pretty well with very little effort on my part.
It's super interesting.
And I'd almost love to jump ahead a couple of questions because I kind of want to ask the question, the way you talk about that whereby you make minor changes over here to a prompt and then implementations and things can just change.
And it kind of makes me feel that...
When we talk about the thing that persists, it's less about the code and it's more about our prompts and our context and our functional requirements.
And it brings me to something called Symfony, which is a ghost library because it's spec.
It's all about specs and context versus implementation or code.
And I guess when we think about that, the code just becomes a throwaway.
The implementation is just, it's there today, it might change tomorrow.
But the thing that persists, the thing that we care about and we should put most time in, like you say, lifting up those reviews more to the functional requirements and things like that, is the specification and the prompts.
Is this a future?
Is this like when we think about the relationship between the specs and the code, how much is that changing today?
Yeah, so it's kind of funny where, You think traditionally in spec-driven development, right, you start with a spec and then you kind of refine it as the code is produced sort of thing to make it more concrete, to suss out those hidden decisions and ambiguity in the spec.
And for things like Symfony, for all these lints that we're producing, for things like...
documentation on critical user journeys or product features or click flows and screens, I've actually found it is much easier to produce the code first, to have that code sort of like present a straw man for what the world could look like, engage with that as a team to refine it, and then distill the spec out of that.
Because sort of produced artifacts, whether they are code, a spreadsheet.
a Google Doc, right?
These things are very information dense in terms of the decisions that were made to go into producing something that we have accepted as good.
So for Symfony, the thing that we shipped is very much a spec.
But the thing that we started with was kind of a vibed implementation of Symfony in TypeScript in our monorepo, where when we thought it was good and it was solving the problems that we wanted it to, we worked on producing a spec out of that to share with the world.
And it was this very, very cool process where we had this three-phase pipeline.
We started with Symfony as it was, and we gave it to Codex to say, produce a spec markdown file that is able to reproduce this system.
We end up with a spec.
Give that to a new agent that says, without access to the original, it says, you have a spec, please implement this system.
It goes for a while with goal in order to produce this thing.
When it says it's done, we give that system, the spec, and the original implementation to a third agent who's going to act as the judge to say, we are attempting to replicate the original version of Symfony.
This was the spec that was given to an agent to produce it, and this is what they came up with.
Where are their misalignments in the derived artifact versus the original system?
And propose changes to the spec.
in order to make the next attempt better.
That's like a very token intensive process, but you know, working at the token factory.
I suspect you get a discount.
But what you end up with is this really refined spec that is reliably able to produce the system as it is written.
while still providing space and ambiguity for consumers of this ghost library to adapt it to their actual business context, their actual repository and tool chain and issue tracker and all these other sorts of things, which I think is pretty cool because it means that we end up specifying in very tight, high detail the parts that actually matter.
business logic critical parts while still leaving the flexibility for folks to evolve and change it as they see fit.
Yeah.
And what I love about that is the process that you just described.
And I think we need to learn from everything that we've done in software engineering for many, many years going into agentic development.
But the process that you just described is a very pure prototype.
It's us rapidly developing our application.
So we make the learnings we need to be able to say, oh, this worked, that didn't.
Actually, I'd prefer it to be like this, or let's change the UI, which ultimately breaks a number of flows.
So we make those learnings.
But the difference is, because development is cheap now, we are able to say, I can just throw this away.
I'm not tempted to use that, but I'm going to try and distill.
everything that I have learned and the functional requirements, et cetera, into that spec.
And I love that iterable kind of like that iterations through to make, to really, you know, I've seen implementation to spec gaps and this is really good to be able to really fill that in to make sure that the spec captures everything that we need.
Yeah, it's not like this process is new, right?
Like if I think about other areas in my career where I've seen this, right?
Like partnering with data science, they want to cobble together a model.
in a Jupyter notebook to prove out the idea first.
And once they get something that they are confident in, then it sort of gets handed off to engineering in order to productionize it, to actually deploy it into the MapReduce pipeline to operate at scale sort of thing.
If we're thinking about, you know, doing rapid prototype of a design like in Figma and then handing it off in order to be productionized, also the same sort of context, right?
But now, because like you said, production of code is cheap, we can actually do that in our production systems.
One neat way that we made that idea real in this app we are building is in Dev, it's an Electron app, we had a window, a screen that the agent could launch, which would basically give it the native rendering canvas that we're shipping, as well as our full design system and component library, which means it's actually able to prototype new screens directly in the surface that we're deploying, produce screenshots of them, give them to our designer.
And now it's a super, super tight loop on does our vision match the reality?
Are we able to produce this experience in the surface that we're going to ship, which is just wild and super, super cool.
Yeah.
I'd love to talk about capabilities and I'll take a slight deviation here.
I'd love to talk about Codex and the industry perception of Codex.
On socials, in community discussions and all kinds of things, particularly over the last month, I'm hearing people rave about Codex and moving from other well-known agents, Claude and things like that, over to Codex.
And they are...
they're stating all of a sudden I'm more productive on codex or I'm actually, it's actually more accurate in terms of doing these various things.
Internally, I guess you're probably maybe seeing this and like, you know, has there been a moment or has there been a feature or something whereby, you know, codex leveled up because of a number of things or is it just, you know, the gradual hard work of a group of...
of teams?
Yeah, so the product velocity on Codex is fantastically quick.
We just recently celebrated a milestone of having 5 million weekly active users.
Wow, congratulations.
Tebow just did a weekly reset, so go fast, go burn those legs down.
I will say that there is a very tight and virtuous cycle between the development of Codex itself and our research team, which means there is this fantastic flywheel on every point revision of the GPT-5 series where the capability just improves in leaps and bounds.
I would say big lifts from 5.2 to 5.3, background tool calling, parallel tool calling means that Codex was able to do more work.
all the time in order to make more progress more quickly on more complex changes.
The unification of the normal GPT-5 model and the codex model in 5.4 means that you are not only getting a super-powered code-producing agent, but one that is also incredibly generally intelligent, a good writer, which means I'm able to start using codex for other parts of the software development loop other than producing code.
The slide deck that I'm going to be presenting later today is 100% written by Codex using the app connectors in Google Sheets, which is not something I would have imagined to be possible last year.
Some super-powered stuff that came out in 5.5 with computer use, the browser embedded in the app.
These things make it so much easier in order to close the loop, where a ton of the jank that we had to put in place in the 5.1, 5.2 era just doesn't need to exist because we are giving more, more chunky, powerful tools because there is this virtuous loop between.
the harness and the app that we are deploying and sort of our research environment and what it means to get the model to be effective at using these things just a constant march of capability improving and one thing that i love about codex is it'll just do the full job it won't give me much nonsense about it i can treat it like i would another member of the team yeah right I'm not shoulder surfing all seven of my teammates in order to bop them on the head if they do a bad thing.
I give them a task.
I check in maybe occasionally at stand-up and I trust that I will get a produced PR or several that does the job.
And I have the same level of trust in Codex and its ability to be autonomous and complete.
And you mentioned Codex and then you mentioned models and model versions.
What would you say moves the needle most?
Is it updates to the agent or updates to the model?
It's a mix of both, but I have to say, the new model releases I think are the most exciting bit to me.
The most sort of powerful levers we have in terms of increasing capability are continuing to train the model.
And this is a thing that goes faster and faster over time.
We're super excited every time a new model comes out.
Getting to test them internally is also fun to be able to kind of see the leaps and bounds of new progress we were able to make.
Each revision of the 5 series from 5.2 onward, I think 5.2, I think is singularity era here with these coding agents, has seen increases in PR throughput time and time again.
Whereas at the beginning of sort of the 5.2 era, we were looking at maybe three and a half PRs per engineer per week on my team.
Now with 5.5, we're looking at...
70, which is just fantastic.
That's more than linear scaling sort of thing.
It's truly amazing.
And is that people changing or is that capabilities doing a better job?
Or a mix of both?
A mix of both, I would say.
As the model capability improves in leaps and bounds like this, there is certainly less effort that needs to go into the enablement step.
The harness engineering gets easier.
The same techniques.
we would use, but the need to build chunky tools in order to attach the agent to the outside world is just things that we need to do less and less.
To give you an example, in order to get Codex to drive our Electron app before computer use was a thing, we needed to boot up a graphical Ubuntu in a Docker container with a virtual display driver and X server, have engineers install X-CORTS on their Mac, in order to proxy over to that headless host and wire up FFMpeg to this gross contraption in order to record videos of what the app was doing, how the agent had changed it.
And this is like entirely obsoleted and like a single toggle in the app to turn on computer use, like so, so much easier.
And I expect this to only get better and better over time.
Yeah, it kind of reminds me of thinking back to, you know, when I've been in the industry 25 years now, but when I think back to, wow, people actually used to write like...
bytecode or machine code or binary and it's like this is the same thing right it's like wow people used to actually do that and connect yeah i don't need to write tcl in order to test my use uis anymore that is definitely an improvement so let's jump back into into harness engineering then um and let's specifically look at a good use case the frontier project um tell us a little bit or walk me through what the harness actually looks like for that um including like I guess what gets updated, what's in it, but also ownership.
Who owns that?
Who changes that?
Yeah, so a little bit of context.
OpenAI Frontier is sort of our enterprise platform for agents, and it covers everything from how to build agents with the API, agents SDK, the codex harness, to how you would sort of observe and govern agents running around your enterprise.
It's very, very cool to kind of see all of the harness aligning on Codex for all the things that we do, which means there's a single highly leveraged interface between the product we are building and research, which means sort of all the post-training improvements that we do on the model accrue leverage to every product that we build.
And right now there's a very, very clear interface in order to inject new capability into Codex via plugins.
These things are skills and scripts.
Same thing as always.
I'm talking about context and tools.
Very, very cool that we have ended up sort of in this like IOC TL sort of extensibility where really the biggest levers we as agent builders, as product builders can give the model give codex is to give it chunkier and chunkier tools that connect to the actual business problems that we're trying to solve and to do.
context shaping and management for it.
Another exciting part of Frontier that me and my team have been working on is sort of like the organizational context management pieces to be able to figure out what work is actually done in your enterprise or what the data ontology is of your data warehouse and how that is actually used in order to answer metrics questions and these things.
Have been doing some really, really interesting experimentation on what it means to potentially give each agent in your enterprise sort of a sidecar that is continuously managing context for it.
This idea that all agents are coding agents, so we should give them things that are native to that, little Git repositories that they can grapple and search over.
And all of that is to say the techniques that we're using to write the code.
are things that are going to translate really naturally to building agents in the enterprise.
And all of that comes from using that base codex harness in order to build these agents.
Yeah, yeah.
And I guess when we think about, you know, a minute ago, we were just talking about bytecode and, you know, machine code and things like that.
The code that gets created, or let's go back then.
The code that we used to write years and years ago when we did assembly and so forth, it's really machine readable.
Less than human readable.
Then we kind of moved into the next generation of programming languages.
And actually, we were writing more for us as humans.
Sometimes we actually took the decision to write more maintainable, human maintainable code versus most performant code because it was less likely that we would add bugs and things like that when we make changes.
Today, and I'll ask two questions, today and in...
maybe a couple of years' time.
I hate asking questions for a couple of years' time when we can barely think about this week.
But today in a couple of years' time, is it more important that code is legible to agents or humans?
I would say if I had to pick only one thing that was legible to me, it would be the reference documentation of the system, the interfaces, and the rendering of all of that into mermaid entity relationship diagrams.
system diagrams and sequence diagrams.
And that's today.
That's today.
If I could only pick one, right?
Still, of course, like I am peering down into the machine code, right?
And I am looking at the diffs as the agent is producing them, but I am able to do that less and less as I gain trust in it.
And I am able to do that.
I'm able to ignore the code as it is produced for more and more complex tasks.
Not all of them, but for increasingly many of them.
The things where I find that me and the team provide the most outsized value are on defining the interfaces, defining what the components of the system are, how each component should be structured, and how the code relates to each other.
Things like wiring up all the system dependencies together and using Scala cake pattern style dependency wiring I find to be very useful.
to pay attention to because it actually means that I am able to draw the lines on that system diagram in my own head and in the reference documentation.
Now, sort of the concrete implementation of those things, to be honest, I might not even know what language they are in.
Very funny story.
To think about those system diagram lines, looking at the local development process, we wanted Codex to be able to connect to our Electron app.
Chrome DevTools protocol, MCP was how we started out.
Eventually, I kind of took a peek at the code and it was entirely different.
Still, Codex was connecting to the Electron app over Chrome DevTools protocol, but it was actually a local TypeScript daemon that was being spun up that was providing a little CLI interface instead of MCP because we found that there were only two or three tool calls that we actually needed.
It was more context efficient.
It was faster.
And this had happened underneath me without...
me knowing my workflow was not interrupted.
That same dependency edge and entity relationship existed, but I almost didn't have to know how it was implemented.
Did you find that cool or worrying?
I found it shocking.
I'd find it cool.
I found it shocking, but also I found it great, right?
It basically means I lived that high trust relationship in a very, very real way.
Someone on my team imagined a way the world could be better, went after it.
and did it in such a way that disrupted literally no one on the team.
And because it's the agents that are writing code, they have no opinions around the tools or their structure, as long as we give them things that work and permit them to do their job.
Yeah, super interesting.
And I think it's definitely a...
It feels to me like humans are going to be the slowest, the slowest here in this, in adapting to this kind of level of trust.
Yeah, it's a consistent challenge.
My attention, my ability to focus across all these threads, it's still something we are contending with.
So when do you feel like, you know, in your usage with these projects, what are the circumstances that you see today where you actually want to go in and look at the code?
Sort of if we were to kind of put a sort of like two axis graph together here of sort of low and high ambiguity and low and high complexity in this sort of high, high bucket here, I think that's largely two shapes of projects or tickets or epics, whatever you want to call them.
Things where I'm literally starting from zero to do a net new thing.
where maybe the shape of the interface is where it lives, what the experience is, I don't know yet.
Or sort of the most difficult refactors that I want to do where I either need to break interfaces or redefine them, maybe go backwards and delete code in order to achieve this thing.
But I don't know what the end shape that I want is.
These are the places where I pay the most attention.
This is honestly where I want to be, right?
Because this is what it means to kind of like...
peer six months into the future to unblock my team.
It's also, you know, code is cheap to produce here, right?
I find myself operating in this space differently than I did a year ago where I'm willing to vibe out a 50,000 line of diff PR and throw it away only to learn the areas where the agent is likely to fail, where it is going to struggle putting those interfaces into place that I like.
And usually I'll actually push up those PRs.
trash them immediately and then crank out 15 PRs out of that to kind of like do the prep work or do the staging in order to prepare myself for the actual cutover that I want to do.
Yeah, yeah.
Let's jump to a quote that you made a while back, which is, you called it borderline negligent not to use a billion tokens a day.
It's a strong claim.
Who were you aiming to shake the ladder of?
This idea that the amount of intelligence we are able to extract from the models is in a way linear with token consumption.
I think at the time it was quite controversial, but we see it being increasingly true.
This is ultimately why test time compute exists.
In order to get the models to be smarter and have more and more rich side effects out into the world, we want to move workflows to high token consumption use cases.
I think in order to get to a billion tokens a day, you must be thinking in a grander world than pair programming with the model.
You must find ways to be in parallel.
You must find ways to put these asynchronous loops into place.
You must find ways to build agents that are able to side effect onto organizations and teams rather than just yourself.
And in order to do that, we have kind of had to invent a bunch of patterns on what it means to do that.
have automations that are seeded from the repository context that everyone in the team can use rather than sort of single player mode sort of things.
And it's definitely not the case that everyone today can say, I'm going to consume a billion tokens a day and poof, magic, it happens.
There is work that has to be done in order to make that safe, in order to make the output aligned, to make sure those tokens are burned in ways that are productive, that make it to main and improve the...
and experience that I am shipping to users, these sorts of things.
And I think at this point in the industry with this technology, we don't really have a durable, scalable, locked-in sense of what those patterns are.
So, you know, to kind of poke aggressively like that and get to a billion tokens a day is really to kind of challenge people to figure out what it is that's going to work.
Sort of.
feels like we have just invented the concept of CICD, right?
And like people are kind of scrambling to figure out what that means.
And only 15 years later did this become a thing that's like fairly standardized, that is table stakes.
And that's, I think, what needs to happen right now with agents and producing software.
We need to figure out those patterns.
So let's continue this flow then, harness engineering.
If we follow that to its logical conclusion, engineers and well harness engineering is is part of every project every team uh developers on that team are using a billion tokens uh every day open ai very happy with that um what does an engineering team look like in that scenario i have found it very valuable to have a diverse team in terms of perspective experience and expertise um i know myself i probably bias more toward backend and infrastructure sort of work.
You probably see in my sort of profile here, which means that when it was just me on the team, I was producing quite terrible React code.
We had some components that were 6,000 lines long, a ton of really bad re-renderings, these use effect hooks that were like...
four overlapping closures, like very hard to reason about.
And to bring folks onto the team that actually were able to say, Ryan, this is garbage, really helped.
And to be able to kind of like have these full stack teams, I find really, really useful.
The other sort of thing is, I think we no longer need to focus so much as software engineers on developing code production as a skill.
Really what I want to see engineers sort of nurture in their own careers is this sort of systems thinking mentality, what it means to set up teams for success, how far can I peer into the future to solve problems to increase the throughput at which we are producing code and shipping to customers.
Probably for me, my time has been split 30-30-30 between sort of the hardest refactors and zero-to-one product ideation.
talking to customers and prioritizing scheduling and staffing work.
Whereas before, I was probably spending closer to 50 or 70% of my time producing code.
So I am able to step back a bit and focus on these other highly cross-functional, high-priority work streams in order to unblock teams of agents from going and doing the code production.
And the most important piece there.
talking to users and understanding the requirements and actually mapping those back to the functional requirements of the app versus thinking too deeply about how an app should be built in the implementation and not thinking enough about Okay, what does the actual user want rather than, you know, me building this backend to be absolutely perfect in its design and so forth.
Yeah, and one of the things that's part of that is deciding what not to build.
Yes.
Very, very common trap.
I see people falling into these days with agents.
And particularly because, yeah, coding's cheap.
We can build everything, but we need to kind of like stop and ask ourselves, should we build this?
Because I guess it's still us as humans that are consuming in the majority, in the most part.
Well, I suppose there will be these middleware things where, I guess, other agents and other apps can consume.
But when we think about humans and users actually using this, we need to develop at the pace that they're comfortable consuming as well.
So super interesting, super interesting balance.
Ryan, this has been...
Absolutely fascinating, really enlightening to chat with you.
So a massive thank you for spending the time to chat with us today and really looking forward to your session later today, which is going to be live on the main stage and also recorded for our listeners to watch back in their own time as well.
So Ryan.
Massive thank you.
Appreciate it.
Amazing.
Thank you for having me.
It's such a fun conversation.
And I will catch you later at the conference and on the live stream.
Amazing.
Thanks very much for tuning in and hope to catch you on the next episode.
Bye for now.
Bye, folks.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast.
It's a community.
And we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more and I hope to see you there.