# Agentic Coding: Why Code Review Still Matters

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-09-22

## Transcript

We wanted to be living in the future.
And so we said, cool, let's try this.
We'll still have pull requests, but we'll just say, cool, if you read the plan, the code's not as important.
Yeah, we did this experiment.
We ran this for four or five months before we realized, oh, this code base is actually unusable now.
It became slower and slower.
Change over here caused a regression over here.
And then we decided, no, this is bad.
And then we literally rebuilt the whole product from scratch.
And the first two weeks of that, my co-founder, Kyle, was literally in VS Code, typing every, not even cursor, typing every character by hand to plumb out the data.
architecture for the new version.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pagiani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
Yesterday, you were writing code.
Today, you're prompting agents.
Tomorrow, you'll be running a software factory.
Argentic coding is moving fast enough to give anyone whiplash.
This June, AI DevCon London sold out.
Hundreds of developers, engineering leads and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify and GitHub cut through the hype and get to what actually works.
This November, we're bringing it back to New York.
Bigger, bolder and full of people actually building the Argentic future.
Three days of real talks, Real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.
Our early bird pricing ends on September 30th, so make sure you lock in your place before prices increase.
As a podcast listener, you can get 15% off your ticket using promo code POD15.
We have so much planned, and we can't wait to have you join us in New York.
Hello, everyone.
Welcome back to the AI Native Dev.
Thanks for tuning in.
Today we have a really fun conversation about software factories and how to do them or not do them and how did we think and we didn't think and all that and to sort of dig into the timeline and sort of really kind of informed opinions about this.
We have Dex Horthy, who is the CEO and co-founder of Human Layer.
Hi, Dex.
Thanks for tuning in here.
Stoked to be here.
Dex, you've done a lot of super interesting things beyond human layer on it, but I think there were these sort of 12-factor agents that you wrote before trying to kind of codify and explain, you know, how do we actually build agents on it.
Today, I think you're sort of starting to refer to like 12-factor.
factories on it.
Oh, you got that already?
Well, we just sort of know that you've done it.
I actually don't know.
Maybe we'll dig into that a little bit on it.
And I think also what's sort of super interesting is that you've been sharing kind of strong opinions loosely held a little bit or maybe like changing over time.
Maybe they were strongly held but pried out of your hands over time on it, which I think makes for sort of good insights and conversations to have.
Thanks for tuning in.
And maybe I'll kind of dig into it straight away, which is you've sort of changed your mind substantially and declaratively around kind of the way to do agentic development of it, you know, really around specifically kind of lights off development, right, versus not.
So what happened there?
Like, what did you believe before?
And what's the turnabout?
Yeah, I mean, the journey kind of starts around, you know, around when Opus 4 came out.
And when they launched Opus 4, they also launched the Cloud Agent SDK, which means you could use...
It had a very small interface at that point.
Right now, the Cloud CLI is basically like the JVM of agentic coding.
It's got like 200 flags on it.
Yeah, it's a freaking battleship.
But it was the first time you could use Cloud, like small agentic loops as part of a broader, more deterministic piece of software.
So we were building like...
terminal multiplexers for cloud code of like, Hey, what would it look like to build a two E where you can manage a bunch of sessions in parallel or like manage, see, get an inbox of like, what are all the approvals waiting on you and things like that.
And we spent a lot of time trying to get, how do we get these agents to work for longer?
Um, it's very interesting.
The, the biggest change between summer of 2025 and now is actually like the planning meta game.
I don't think a lot of people realize this.
A lot of people just write plans because they put plan mode in the thing or they read something online and they found some skills.
Hey, we're going to make a plan.
And I think depending on where you are in your journey and how the models have changed, in retrospect, I realized that writing a plan file in the summer, in July of 2025, the number one most high leverage thing that that got you was it was a really reliable way to get a model to work for longer.
Yeah.
Okay, we're going to do everything on this plan.
And to kind of follow instructions, I think, a little bit at the time, right?
It was, well, it didn't always follow instructions, but you had, you also had things like a spec kit, for instance, that was trying to like really micromanage the agent and sort of like say, okay, not only is this the plan, but at any time, just come back to me at the end of this step and I will spit out the precise set of additional plans that you're going to do.
But I think it got them, yeah, it got them to sort of kept things a little bit on track.
or at least more than it was if you didn't plan.
And it was a way to also make the work a little resumable, right?
It was the first version of, first, the compaction of intent.
Like, hey, I'm going to take this whole thing that I want to build, which could be specified as every line of code I want to write.
Or you could put it into this file as a couple hundred lines of markdown.
It's like, here's what we're going to do.
And then you could keep that up to date with what was done, what changed.
And so you could resume from a clean context window later, which, again, also in July of 2025, the models were not that good at long context.
this term called the dumb zone or whatever it was, which at that time was around 100,000 tokens.
If you went over 100,000 tokens, you got way worse results.
Obviously, that number is higher now.
Yeah.
Although there is probably still a dumb zone.
There's still a spot in which I still like.
I reuse that term on it, and I feel like it has an element of too much focus is no focus, right?
When you have too much information in it, then finding the area of attention is still something where if you're intentional about it, by...
yourself making the decisions around what's in the context window and what's not, then you can, you're more likely to achieve the goal that you're trying to achieve.
Yeah.
So anyways, getting back to like our journey is like, we had just come away from, um, Sean Grove had given this really interesting talk.
I recommend watching it.
He was a researcher at open AI at the time.
And he was talking about basically like the spec is the new code.
Um, and this idea that like the way we think about assembly now, is the way we're going to think about code in the future.
And the spec will be the thing that is verifiable and executable and compiles into code.
And that, like, checking in the code, he made this really good analogy of, like, hey, you write a bunch of Java code, and then you compile the jar, right?
And the jar goes into production or whatever, whatever your compiled asset is.
And then you commit the code to the repo.
This is in the future, like, you'll basically...
just commit the specs.
But right now, all of the interesting stuff from how people were using coding agents in June of 2025 was basically you send a bunch of prompts and then you get your result out and then you commit that and then you push that and that becomes part of production.
But like that, all of the interesting, useful, more high leverage stuff is actually all the prompts you sent in that conversation that should be recorded somehow.
And that like having a long chat with an agent and then throwing away the chat and shipping the code is going to be the new version of like, what if we wrote, you know, a bunch of Java code and then compiled the jar and checked in the compiled jar and threw away the code that we wrote.
Right.
Yeah.
I think, I mean, I very much kind of remember that it needs to be sort of held.
Frankly, like this conversation is not just sort of past tense, right?
Like I think today there's a lot.
to be claimed, and I know this is very much kind of in your world, which is like those are the very artifacts that you're supposed to be reviewing and discussing and preserving on at the time.
But I think, so you were like touting, okay, we want to do like context engineering and kind of capture all of these sort of intents on it.
But I guess what was the advocacy then, like at the time when you were guiding people to say, hey, be amazing with sort of agentic development, like what is it?
that you should do.
And I'm interested because I know that you kind of went from almost, it feels like you went from a context engineering kind of advocate to almost like an antagonist to the group on it, that it's like a little bit more like you really need to be more defined and more reviewing.
And I've heard you say now that you should review the code for now.
So I guess what's the, what has changed?
Yeah, I mean, this thing happens with models as new models come out and get smarter.
You never know when the singularity is and the only way to stay ahead is try to live in the future.
And so I said, okay, this is a thing people are saying might work.
And people in the lab, I think there was a thing that was happening last year a lot where it was just like, oh, the labs have way more alpha than any of us.
So if they say this is a thing you should do, then you should go do it.
And so the OpenAI said, oh, this is working for us.
then we wanted to try it.
We wanted to be living in the future.
And so we said, cool, let's try this.
Let's try, like, we'll still have pull requests, but like, we'll just say like, cool, if you read the plan, like the code's not as important.
Ton of issues with that.
First of all, the plans contained all of the code that was going to be written.
And so you would read this plan of every single, it was literally like inline diffs.
You read the plan.
And then you would go write the code.
And then if you didn't read the code, you would basically, if you wanted to also review the code because there was drift or you were larger.
We were a three-person startup.
So we had this kind of option.
We were pre-product market fit.
It was like, okay, go, go, go, go, go.
But we kind of recommended this to some larger teams, even teams with like 50 developers.
Then they would say, yeah, we wrote the plans.
And then the person went and did the implementation.
And then the implementation would drift.
And so they would have to review.
review the code too.
They weren't ready to let go of reviewing the code, which is good for them.
But they were actually doing more work.
They were reviewing basically almost the same shape of code twice.
Yeah, depending on the detail level of the spec.
But either way, combined with some sort of sense of discomfort, because now you're reviewing someone else's code, and so you're kind of a familiarity and control over it.
It's oftentimes harder to apply your opinions when editing than it is when you kind of write it in the first place.
Right.
Once the code is written, you're kind of committed to a path and it's like much more work to change it to the point where sometimes it just makes sense to throughout the implementation, like drop a new prompt in or redo your plan and then go again.
But yeah, we did this experiment.
We ran this for like four or five months before we realized like, oh, this code base is actually unusable now.
Like it became slower and slower.
There was like...
every single change caused you know change over here caused a regression over here i think this is also one of the things we're like what if we let the model own the system architecture too like human in the loop reviewing it was like hey how would you solve this well well i make a golang demon and then i would use a unix socket then i would connect these things and then when we run the mcp we ended up with this like system of six different services where you have a desktop app that talks to a socket over Rust.
And then that's a Golang daemon on the back end, which then launches and manages Cloud Code Sessions, which then launch MCP servers to handle the approval system.
And then those MCP servers call back to the front end and put things in a SQLite database via Unix socket.
And that shows in the UI.
And the user approves something that goes back to the MCP, back to Cloud, back to the Golang daemon, and back to the UI.
It was just this incredibly complex system that...
the models developed if you let them decide how it should be architected.
How much is that, though?
How much is it that the models don't know how to do it versus you just sort of saying, like, hey, figure out an architecture?
Like, how much of that can be prompted?
And I know we're sort of still in a little bit of, like, a past tense.
So maybe some of this, like, we need to talk about, like, a current perspective on it.
But I think there's a difference between saying, hey, models can't architect and models can't architect yet.
Or they can't architect if you don't give them enough.
Yeah.
And I think fast-forwarding a little bit.
So we ran this experiment, and then we decided, no, this is bad.
And then we literally rebuilt the whole product from scratch.
And the first two weeks of that, my co-founder, Kyle, was literally in VS Code, typing every, not even cursor, typing every character by hand to plumb out the data architecture for the new version.
Yeah, exactly.
But I'm really glad those patterns have stayed really, really solid.
And especially if you're starting a new project, the decisions you make early on just constantly cascade into the future.
And we've changed some basic things, but the core ideas behind it are still really strong.
And we built on electric SQL and durable streams and have this kind of unidirectional data flow where clients don't have to...
There's no signaling logic from the back end and there's no signal receiving logic on the front end.
That's all just handled by the sync engine.
which was a much better way to do it.
I think any piece of software built today should use a real-time sync database where clients can just connect and get updates live.
But anyway, so we basically were like, okay, we need to read the code.
We need to understand the architecture.
And then also, there's so many slop patterns that get in and that these models will, left unattended, will create a system.
I don't know, there was this thing, I think it was Donald Knuth or one of these OG...
C language guys had this sort of principle that was like, in order to debug code, you need to be much smarter than you were when you wrote it.
And so if you write code that is the smartest code, the most clever thing that you can come up with, you are by definition not smart enough to debug it.
And so the thing you're building always has to be more simple than...
than your actual peak intelligence, which I don't know.
I think just in the news last week was like Steve Yege, who spent the last year running around every podcast on the internet and saying like, hey, and I love Steve.
Steve's a brilliant guy, a huge fan.
But I am going to call him out for going running around the internet and telling everybody, hey, if you're still trying to read the code, you're not going to make it.
And like, just let the models cook kind of thing.
And then last week kind of said, yeah, I used Fable to build this thing.
And Fable made a system so complicated that it itself could not debug it.
And you really have to keep an eye on these things and understand the system you're building.
So sort of the counter argument to this, I kind of understand and I relate.
And we sort of still have a Tesla as we sort of build.
And we are sort of a team of sort of 20 odd people on the engineering side.
So we do need some sort of supervision, by the way, like still compliance systems require kind of a name to every sort of code review and all that.
So we have a lot of like...
AI-assisted code review, and some changes are allowed to sort of go autonomously, and there's some supervision of complexity, et cetera, et cetera.
So I wouldn't sort of say, hey, entirely lights off.
But I think the counter argument to reviewing the code is that you're stuck, that when you review the code and you apply...
a change, right, or you sort of give more specific instructions or open VS Code and actually kind of handcraft some sort of portion of it, then what you're not doing is you are not building a mechanism or a system or an automation that enables the creation of code that is to your preferences, right?
That is, you know, when you talk about those specs, like I always think of specs as this.
kind of this slider between determinism and adaptability.
You know, it's like how many degrees of freedom do you want to give the LLM, the agent, to build with, like these are the decisions that you have made, and within those you should be allowed to play, or like you're enabling the agent to play, and you need to define the sort of the outcomes that you want to see out of it, and you should be comfortable with sort of not caring as long as the outcomes.
are achieved.
And so if you're kind of building towards that, you would like to achieve that result, you have to codify your intent.
And the way that you would do that is not by reviewing the code, but rather by continuously interacting with the system.
There's like a balance to it.
Like this idea of like plan before you build, like the only reason to do planning or alignment or whatever it is, whether it's you in the model or you in the model and your team is like, because you have this factory and this is pre AI, right?
But there's all these pieces and I've talked about this before, so I'm not going to go in depth on this, but you have people making stuff and then someone builds it and you have a review thing.
And then you have a rollout and you have monitoring and you have all this stuff.
But basically this idea here is like what people learn.
30, 40, 50 years ago is that this build step is like can take hours or days and the review step can also take hours or days.
And so we as humans before AI, we're like, okay, we're going to talk about the architecture.
We're going to do this sprint planning thing.
We're going to align with everybody on the team to make sure that this is at least directionally correct in the hopes that we can decrease the chance of rework and reduce the review time.
Because we talked about it ahead of time and we've all done this.
Everyone's read a perfect PR where they're like, yeah, we talked about this for an hour and now this thing is a thousand or two thousand lines of code.
But it's following our patterns and everything is correct.
So it's easy to review.
When people say we are drowning in PRs, we have too many PRs.
My pushback is like, no, you don't have too many PRs.
You have too many bad PRs because you're letting people turn off their brains when they're coding.
And so like the other side of this, though, is like you don't want to over plan.
Right.
So like I can YOLO a two sentence prompt and there's a 50% chance I'll have to rework it.
When I say rework, I mean like I'm going to go back and forth with the model a couple more times.
Please don't YOLO a two sentence prompt and not read the code and send it to someone else to review.
I'm assuming you're going to actually abolish it.
Or you could like handwrite a detailed spec.
You could sit there at a meeting and spend a bunch of time and spend five hours writing a spec.
And maybe now there's a 10% chance.
You may still have to change some stuff.
but it's less.
Or you could do the extreme of write all the code by hand.
There's a 0% chance you'd have to rework it because of something the agent got wrong because you wrote it.
You may still have to rework it.
We've had code review for 30, 40 years, but you're not going to have to change it because of something the agent, because you left the agent some leeway to make decisions on some of the smaller details.
And I think about this in terms of expected pain.
Like, what is the chance you'll have to change something later?
And how painful is that?
Like, I never argue with a model about, like, how, what color a button is going to be or something like that.
Because I know that if it's not the color I like, I'll just send one prompt and it will fix it.
But if, like, the database schema is wrong, then we're probably going to have to throw it out and start over.
And so I want to get aligned on that first.
And so the idea is like you have to build this intuition of like based on the size of the thing and the complexity of the code base and how good the code is today and how many bad patterns are in there that the model might follow.
Like where do you want to land on this curve?
And that's what I think about like today.
Planning is all about leverage.
How do you find the sweet spot for the task?
You're going to get wrong sometimes.
That's why I said like get a lot of reps.
But this is kind of how I think.
I think it's similar to what you said but just kind of visualize in terms of like what are plans for?
is to kind of at every stage of your, in human layer we have a bunch of different planning workflows.
We have like just do a design doc and then go ship it.
We have like design and then the order we're going to do it.
We have like a PRD flow that's like three or four stages.
But depending on the size of the feature, you want to have one or more stages where you are just like slicing off 50% of the potential bad outcomes.
You're narrowing the outcome space in a way that is like a good use of your time.
I think I have a whole bunch of thoughts on it.
I think direction was sort of aligned on it.
You have to control a little bit, or you have to decide what you want to control for and what you don't want to control for on it.
I have a set of questions here.
One is, how much of that planning is around kind of guiding the decisions versus expressing your intent?
Because I think when you're...
When you're planning, you say, well, I want the architecture to be like that.
And implicitly, it's, well, because I want to make sure that it, you know, scales cheaply.
I want to make sure that it is, you know, resilient to like additional kind of whatever data fields on it, you know, whatever it is.
But you don't say that.
You sort of say, I want the architecture to be like that.
And so you're sort of limiting to your own creativity of how you would implement.
But you actually never capture.
your kind of actual intent of why is it?
It's one of these, you know, ask why 17 times to get to it.
So I guess when you're going through that planning, how much are you trying to extract and do you separate what do you want to achieve versus how the implementation should look?
I mean, I think there's a reason why Matt Pocock's skills and specifically like the grill me skill has.
uh, the most, it's in like the top 25 stars on any GitHub repo of all time.
And it's like the top three stars on GitHub for repos that like mostly just contain skills is because like people are bad at specifying their intent.
And, uh, and I think the, the, a good AI assisting.
assisted planning process involves like having the model get enough context to understand what's going on and enabling a user.
The fastest way to get a good description of your intent is to just like off the cuff, say kind of what you want.
I'll turn on voice mode and I'll ramble for 60 seconds.
And the things I know I want, like, oh, okay, we're going to, this has to scale to end users or this cannot use this pass that has to use this, like let's use Redis and not the Postgres database.
Cause I know the speak, like anything, you know, you should specify, but the, the onus of like, hey, I need to sit there and just like think for six hours and decide exactly what I want.
It's much easier and much faster to be able to take a very rough, like large, unspecified thing, underspecified thing, and use AI to relentlessly interview the user.
As to, hey, look, here's what you asked for.
Here's some alternatives.
Are you sure you don't want these?
Or, hey, you asked for this, but there's a ton of open questions there.
Here's what I think they are.
And walking down that decision tree.
So I think we're in agreement over there that it's hard to extract things out of it.
But when you think about reviewing the code and the likes, I guess what I struggle with a bit is this, okay, if you extracted the intent, correctly, and you have defined your kind of verifications correctly, right?
You know what you're looking for, which are both big ifs.
They're both hard to do on it.
Do you still need it?
Like the models are getting pretty good.
You know, let's put aside, you know, that is hard to do.
Let's put costs aside for a sec, right?
Like let's assume you can kind of iterate and you can define because you've defined what good looks like on it.
Like, do you think, like, is it, what's the limitation over here?
Is it?
Is it actual like models are just not good enough yet in your view?
Or is it, I don't know if you have like a guess as to like when would they be?
Or is it more, or is it something else, you know, that it's like a, that is sort of, that causes you not to sort of say like what you really need to do is like invest in the capturing of the intent and the verifiers.
And instead you're saying you should review the code.
There's like this there was this post from the Haskell community like six months ago That was like a sufficiently detailed spec is code Yeah, but those are like they are implementation specs like these are they're talking about cases in which I am dictating an algorithm to you, right?
Yeah And so, yeah, if you give, I mean, the reason why I think specs are so powerful and people maybe get a little bit like one-shotted by a good spec with verification is like if you give the model a good way to get feedback on what it's doing and to like a hill climb against some objective that it itself can measure, whether it's qualitative or quantitative, it will move mountains for you.
And it's, again, just like plans were a great way to get a model to move.
work unintended for longer, a good spec and good, we call it like back pressure, right?
Is how do you give the model the ability to get feedback on what it did without a human having to do that part?
You can get a lot more done and you can do more in parallel because that thing's going to go work for two hours and go kick off another one.
Right.
I think that I'm entirely aligned with.
So I think the difficulty, as I interpret what you've sort of said, and sort of maybe a bit of like to my sort of self-view over here as well, which is like in a perfect world, you'd be able to express intent very well and you'd be able to express your verifiers, right?
Of like, how is it that you would verify that it's right very well?
And when you do, you don't need to review the code.
Like if you're able to do that, then you will achieve.
uh something in which you can allow autonomy right like what is the level in which you'll allow like what do you need to see to be able to allow autonomy yeah um i mean i think it's a great way to kind of like prototype and understand what's possible like i'm working right now i'm just a little experiment and i don't actually think this is like a good There are, I'm working on a code search harness, basically.
And I know like Tejas at MorphLM built a custom model just optimized for code search called WarpGrep.
But what I have been working on is like a code search harness for JEV.
So, like, can you get Jev to select the next correct action?
Like, basically using Jev's classifiers as tool calls of, like, cool, next I want to read this file or the final answer of, like, hey, yeah, now we have enough context to answer the question or whatever it is.
And the problem is Jev only has a 32K or 64K, like, input context window.
I forgot.
I think 64K has some asterisks on it.
But that means that you cannot put an entire code base in there.
or you cannot say, here's all the information we have so far.
And so you have to get really creative about how do you split things up and how do you merge them back together.
And so I'm kind of taking like an evolutionary algorithm sort of approach of like, hey, cool, go do a bunch of subagents, have them try different things, try them hill climb different dimensions of the Pareto frontier for cost and speed and accuracy and 45 different code search evals right now and more coming.
I'm not reading any of that code.
And if it actually works and it seems good, then I will probably take the root of it and be like, cool, explain this algorithm to me, turn that into a spec, and then reimplement that with actual clean architecture because there's just slop everywhere right now.
The code is just doing all this crazy regex matching and splitting.
So I think not reading the code in order to prove what's possible and to prototype is very feasible.
Everyone says, I don't know, there was this thing that people said in like the Sonnet 3.5, Opus 4 kind of days, which was like, yeah, sure, the code is slop, but I'm not going to read it because like by the time I need to fix it, that'll be GPT-7's problem.
And the challenge is like, folks, we're at GPT-6 and every really good builder I know has mentioned that like, you know, Astra got much better at computer use.
It got much better at video game slop.
It got much better at...
blender, but it didn't actually get that much better at writing code that is maintainable.
And I think this is because to train a model at something, you need RL environments for it.
And to have a good RL environment, you need a Oracle, you need a thing that can verify whether the product meets the needs of the user and maintainability, you don't find out something is unmaintainable until two months later.
There's no fast Oracle for software maintainability.
I was riffing about this with Adios Manny, who was at Google for a while, and that was kind of his take.
It's like, yeah, maintainability has no fast Oracle, and it doesn't seem like...
There was this benchmark.
Have you heard of Slopcode Bench?
I'll give you the very high level, which is like you give a model a code base.
You give it a challenge.
You say, hey, go write this feature.
And then you have the code base.
And then you say, cool, now go write this feature.
And it has to inherit its code base at every step.
And it's like, now go write this feature.
And basically what the benchmark shows is, one, it's very under-saturated.
I think GPT 5.5 in May got a 14.8%.
And they're getting ready to publish the results for Astra, I think.
I was helping them.
run it with some inference and stuff.
But it's not my benchmark, by the way.
This is our lab out of University of Wisconsin.
But we've been providing runs and doing stuff like this.
And Astra got in the first run we did was 16.3%.
So the models are getting better in this dimension of can they maintain a code base over time?
But not as fast as everyone thinks.
Like I said, we said GPT-7 would solve all of this.
We're GPT-6.
We are getting, we are RLing and hill climbing in other domains, but like the frontier of like model maintaining code base over time does not appear to be getting better that quickly.
Hey everyone, hope you're enjoying the episode so far.
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
Whether that's talking about the latest tools, the most efficient workflows or defining best practices.
But for whatever reason, many of you have yet to subscribe to the channel.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
All right, back to the episode.
I love this conversation.
I think about this all the time.
I feel like the counter argument to an extent is sort of this notion of, sort of of maintenance agents right and i i think in general like through the tesla journey right we started talking about sort of spec centric software development spec driven software development and then we moved into this sort of context and skills uh and to me the the sort of the premise of that evolution was that uh is maybe the acceptance that when you think about an amazing developer on your team They're not amazing just because they know what you are building, but also because they understand how you are building.
And really, to me, context is sort of the breadth of it, right?
I think like you look at skills, for instance, they will have, I think, three primary categories.
They will have specs, which are more about like how does your product or another product work on it so they can build it.
They will have...
workflows around how do you want to sort of perform some action on it.
And then they will have opinions.
They will have policies, best practices, and like choices where it's not necessarily correct or incorrect.
It's just your choice.
We got in the room and we agreed this is our security policy, right?
This is our sort of balancing act.
Or we say...
I don't know, we had a couple of these at Sprout Social my first gig out of college.
And they just had a list of things.
It was like, do not wrap APIs in other APIs.
Exactly.
Minimize code's traversal depth.
And my boss at the time, this guy, Alan.
crazy smart dude, he had this thing of like three, he called it three copies press hard.
It's like at the top of your endpoint, you always have like a, okay, log a thing and then send a metric and then do another thing.
And every engineer wants to like collapse those into a...
function, like an abstraction, like send metrics.
And his take is just like, no, just duplicate the three lines of code everywhere.
Because as soon as you put an abstraction over it, it ends up like leaking a bunch of stuff and it becomes very complex and it gets a bunch of responsibilities.
Just call the three different methods in line.
And this is what you do when you go to the car shop or the car mechanic, right?
They say three copies, press hard.
You write down, you sign the thing and then...
One goes to management and one goes to one goes to processing and one one you keep.
And they've just figured this out that it's like, no, we're not going to take one thing and then like make copies of it or whatever.
It's just like, yeah, it seems a little weird and archaic and like, OK, you could be smarter about it.
Anyways, yeah, these are the kinds of opinions that like are very hard.
This taste that engineers have over five, 10, 20 years.
Yeah, we haven't figured out how to put that in a model yet.
We just haven't figured out how to give models that that taste.
So I guess I guess what I'm asking is.
Okay, so you can't put that in a model.
And I don't know if they ever will, because like models almost by definition, they are generic intelligence.
The one you use and the one I use are the same.
And so even if they are smart, they're the same opinion.
You know, they will run, you know, design is probably the easiest examples of that.
You know, like you might like dark mode or you might hate dark mode, right?
Like, you know, you might sort of choose that.
And so...
But you can't put it into the model, but you can put it into context.
And so you can define, and if you've captured, and that to me is another planning aspect, right?
Like you've planned how you want to build, and now you can run a whole bunch of maintenance agents that would run on that code base, that would review code that was created by an implementer agent who took in more like feature requirements, right?
Like sort of requests on it and will ensure, and you codify your taste, your preferences.
I guess, why isn't that?
the way to achieve a non or low slop code and kind of invest in that path versus reviewing the code.
Yeah, I mean, I could see a world where, I mean, the way this usually starts is people just add.
end up adding one after another after another of bullet points into their clot MD or their agents MD of always do this and never do this.
And then you end up with a bunch of instructions and most of them don't get followed because it's too much context.
So we do this thing internally at HumanLayer where we have deterministic detectors for, some of them deterministic, some of them are non-deterministic, but very simple of like, we don't like this pattern.
The desired end state looks like this.
There's a bunch of code that looks like that.
Whether it's like React Doctor deterministically pulling warnings out or Oxlint pulling warnings out.
It's like every night we have four or five different, people call them agents.
It's the same harness with a different user message basically.
Five agents if you must.
Five sort of skill-powered workflows or sort of prompt power workflows.
Exactly.
And they run on crons and GitHub actions.
And every morning we wake up to five pull requests that make our code base a little bit better along some dimension.
Yeah.
I think the challenge is, is like...
You have to know exactly what you want.
You have to codify all of your opinions somewhere.
You probably can't be in your agent's MD.
Maybe you can put them in a skill or you can break them up into a set of skills.
But the more opinions you have, the more clever you have to be.
This is context engineering.
This is always the thing.
The more clever you have to be about splitting them up and paralyzing and orchestrating multiple different things to go do this.
So I get the point around sort of the needs to review the codes today because of the limitations.
And as I mentioned in TESOL, we still sort of review the code.
But to an extent, the sort of the bet that we're making in our own internal software factory is that really it's all about extracting correctness out of observations.
And so what we do is we take today maybe some amount of sort of risk or burden that really forces the process of not writing the code and sort of reviewing the code.
interact with the systems.
There's exceptions to that.
There's asterisks next to it.
But by default, it has to be kind of a feedback that you provide about what is wrong, you know, as part of pull requests, as part of systems.
And sort of these sort of continuously improving skill-powered workflows, so like these feedback loops that really help feed back into the system.
And I think there's a good case to be made that today we are maybe making more effort in sort of investing and honing those skills on the sort of the bet, which is always TBD, that it will set us up better.
And we're sort of seeing some compounding speed out of that.
So I don't know how you think about that.
And how does that align or conflict with like reviewing the code?
Yeah, I mean, we are very bullish on this compounding thing.
I think probably like...
the anti-pattern I see in a lot of orgs and most of them are probably larger than yours or mine is is someone gets excited and they say I'm going to build the software factory and they kind of throw everything out and they say cool let me design this whole thing from scratch it's going to have this compounding thing and we will never have to read the code and whatever it is and like I think that's uh like any piece of software the big design up front is like is always going to lose to the like iterate fast and get 1% better every single day or even 1% better every single week.
So I think like the idea of having a process, the thing is, is like you cannot afford to put slop code in your code base because every time you let bad code in, you are making the entire system worse.
People talk about software.
Yeah, people talk about software factories, but there is a big difference between a traditional factory and a software factory.
In a factory, if a car comes out of the end of the line and it's bad, that's cool.
Whoever was going to buy that car or whatever is going to deal with it.
But if you ship bad code in your software factory, it degrades every future piece of work that goes through that factory because there's more bad patterns for the model to see.
You do want to continually be making progress towards you have to want to have some ramp where you can make the factory better every single day or every single week.
But it's like the number one priority is like stop the stop the bleeding or stop the, you know, close the windows on the house before you turn on the air conditioning.
You have to.
Yeah, exactly.
But no, this idea of like, cool, you know.
We have humans in the loop whose job it is to make sure that the code is good and that nothing goes into prod or into the code base that is going to make the quality worse and going to make future quality worse because the agent will see that and think it's okay.
But a byproduct, the kind of exhaust of that process is you get a ton of real examples that you can use to feed improving the factory and make it a little bit better.
Anytime a human has to put a comment on an agent-written PR and say, hey, this is wrong, we don't do it this way, agent goes and fixes it, it's fine.
You need to be, like, siphoning that off.
Every time someone sends, even in a prompt, you should be reviewing the session traces, maybe not reading them verbatim, but analyzing the session traces of everyone on your team and saying, where are people getting frustrated?
Where are they telling the model they did something wrong?
And then figuring out how to improve the entire system, whether it's the base skills, whether it's the code review skill, whatever it is.
So that, you know, if you're not getting better than you're getting worse.
And so, yes, I 100% agree on this idea of compounding engineering or whatever you want to call it.
You do always want to be working on the thing that builds the thing.
And if you do a good job of this, then you're sliding scale of like you're spending less and less time actually reviewing individual code and more and more time doing things that are going to improve.
You're basically shifting from like caring about your position to caring about your velocity.
Yeah.
From kind of building the car to building the factory, right?
producing that.
And how, so that's a really good point, though.
I think one takeaway from what you're saying is, like, yes, you should improve the sort of the context eventually that guides the factory and the production, but you should also remember that one of the primary, if not the primary context that eventually guides the factory is the actual existing code base.
And so if that...
code base is full of slop, then the reality is that that will be a pattern because the agents somewhat legitimately default to mimicking the patterns that they see within the code.
And so if you let that go sideways too much, and it's sort of a counter view to saying that tech debt is deflationary and that you can accumulate debt today.
I mean, that would maybe be true if literally every pull request that you're creating, you don't take in any code.
You just sort of produce it from scratch.
But that is not realistic because you don't actually have that.
So really good point.
Maybe so going a little bit more future looking a little bit and thinking about what are the artifacts that might stick around?
So like how long do you think?
code will still be the source of truth.
We're talking about rebuilding, replacing, editing increasingly, but we still end up kind of capturing the actual decisions in the code.
Do you foresee that two years from now that is still the case?
Or do you think new artifacts emerge that are the primary and the code is more like the bytecode that you've produced?
Right.
Yeah, I don't want to predict the timeline.
I think a lot of people's predictions have been, Either way too conservative or way too liberal.
We're with Fortune 500 companies, publicly traded companies, large private companies.
The reason why they love working with us is because we're like, cool.
Do not mortgage your code base for the future.
Yes, there may be a 1% chance that you miss out on some crazy, crazy upside.
But we are going to let you move as fast as possible without giving up control or giving up taste.
I was at a dinner in San Francisco probably six weeks ago.
And we were talking about software factories and loops engineering or whatever we were calling it back then.
And this question was like, cool, raise your hand if AI is writing 100% of your code.
Like, everybody, hand up.
Raise your hand if you're still reading the code.
And it's like 50% of the people.
The other 50% have been like, no, I made 70 linter rules custom by hand, and I caught every single anti-pattern.
Whatever your approach is that gives you the confidence to ship it.
But then the question came, it's like, how long until we don't review the code?
And I was like, eh, maybe one, two years.
It's possible that that could be the case.
And then there was a really interesting question.
It was like, do you ever think there will be a time when you don't review anything at all?
And one of the founders there, who is a veteran of a lot of this stuff, kind of said one of the most interesting things that I think about almost every day now, which is like, there will probably always be alpha in reviewing something.
We don't know what it is.
I can't say what it's going to be.
It's probably going to change over the next two years.
But if you don't review anything, then you're just letting the AI cook.
And then you're going to get the same product, the same email, the same company that everybody else gets.
And so there will always be something if you want to rise above the median.
I think the generic token farm.
of just like, hey, look, yeah, we spent tokens and we create value is like very undifferentiated.
So I think there will always be a place for smart, creative, thoughtful, caring humans to be reviewing something.
For now, I think it's still the code, but eventually it might be something else.
I don't know when or what it's going to look like.
But there will always be something to review and there will always be a thing that you can add to the picture.
Yeah, I think we're sort of seeing a lot of that with like Claude presentations.
You know, for a while, like...
You know, everybody was amazed by how great it was that Claude, like at the breakthrough, like it was creating these good-looking presentations.
And now every single presentation looks like a Claude presentation.
And you just sort of yearn for something that looks a little bit different.
But how – so I agree with that.
I think you have to differentiate.
You have to capture those.
I do muse, though, about a better way to capture preferences.
I think a lot about sort of outsourcing, right?
You have a lot of like great companies that have sort of outsourced their, whether it's their sort of software stacks or even more so like a lot of manufacturing companies that they don't actually manufacture everything.
They design the thing that will be manufactured.
Someone else is producing it.
And so they really kind of define the outcomes, but they work with these entities.
I guess, does that...
analogy kind of resonate with you, which is like create artifacts that document references, opinions, taste, and for the most part, outsource implementation to the agents?
Yeah, so two things.
Yeah, I think absolutely.
People say, oh, the new moat is taste.
I think that's really blurry, but I do think it's safe to say the moat you can have is understanding the problem better than anyone else, like knowing what features to ask the factory for, understanding your customers better than anyone else is incredibly valuable still.
And that's, again, that's where you can context engineer the agent in a way that none of your competitors can if you understand the problem better.
I'm ambivalent about taste.
Like part of me feels that taste is just a preference that you haven't bothered writing down.
You know, it's just like I'll know it when I see it type thing.
is something that just, okay, like get the agent to see it often enough and it will capture it.
It still is a differentiator.
Like it is still the case that if your written down taste is better than my written down taste for some purpose.
And so for those purposes, your product or your output will be better than mine.
And so there's still value in it.
I still think taste should be codified and I think still it is, still think that it is the differentiator.
But it is, it's not this elusive human versus AI difference.
It's just a choice piece.
Well, what about you, man?
Like, what do you think the future of...
reviewing things or building or factories, like what, what, what spicy takes do you have or, or mild takes?
I don't know.
Like I think there's, there's, there's the market for spicy takes is way too frothy right now.
And I actually love a more middle of the road of like, look, yes, I think this is directionally correct, but actually like it's not as crazy as everyone says it is because everyone is incentivized to, you know, engagement farm on the internet and you can actually, you get paid money if you get people excited about something regardless of, uh, regardless of, whether it's true or not, but what are your spicy or otherwise takes on where this is going?
I've got a fairly kind of skill or context-centric view of the world.
I think the fact that, I think software engineering is a creative profession.
creating software.
And the fact that it has engineering in the name is a mistake of history.
Like it just happens to be that you had to be an engineer.
You have to be an engineer to be able to create those.
And really what we want is to be able to capture taste and preferences and balances.
And I think those will be the kind of units of code, if you will.
that you are building.
This is my definition of what secure is, of what good design is, of what even a specific subscreen, like you can still get granular or not in terms of its preference.
And even some elements of how controlling you are, whether you're Apple and you want to design everything down to the last bit and gram, or whether you're fine with some...
more sort of generic OEM component in there.
And so those are your choices.
And those units will still be developed in the sense that you still need to know that which ones you are using and which ones depends on what.
And you still need evals and tests to be able to know that when you modify them, they do not regress.
And you still need to observe them and sort of see how they work and deduplicate them and align them when you have multiple team members creating them.
And so I guess my belief is that software development would revolve around skills.
and that the role of code will diminish over time, that it has to be sort of duplicate.
I don't know when it goes away.
It's more about, we've referred to at the beginning of TESOL, we've had the notion of a spec and a shadow spec, sort of the spec being the thing that you create and the sort of shadow spec being the capturing of the LLM's decisions.
What is the difference between a shadow spec and code?
I don't know.
It's like degrees of freedom for it.
And today you already do that, right?
You already use Java and you're delegating garbage collection.
And it happens in different ways on different systems.
And you don't care because you trust that it is good enough.
And so I think it's sort of going into that world.
And the conversation we have right now is really about what is the journey towards it, right?
And what should you build today so you're not left stranded for tomorrow, but that still actually gets you somewhere today?
that balance has been tough.
Yeah.
And so I won't say software engineer, maybe I'll just say builder, but like as a builder, as a person who wants to create things of value, I think right now your two biggest bottlenecks are how do I let the model pull as efficiently and easily and like mentally pleasant for me, the human, how do I enable the model to brain drain me as quickly as possible on all the things that...
are my tastes and preferences and things.
And then I think we're in alignment there.
And then if you're still reading the code, there's like a corollary of what is the fastest, most efficient, most visually effective way for a model to tell me what it has done.
Like, yeah, maybe I might not read every line of code.
But between, if I'm reading every line of code now and someday I'm reading a spec, like between there, there will be.
you know, laddering up of like, cool, don't tell me the code, but give me things.
Give me something that is, you know, 50% faster or 80% faster than reading every line of code that gives me almost as much confidence or almost as much understanding.
How can you, what are the like patterns where we can leverage?
How do humans think and understand things to both?
get that into the model's context as quickly as possible, and then let the model context engineer the human of, like, what's the fastest way that I can explain to this person what has been done so they can be confident that it works.
Yeah, no, that's well said.
So maybe one last question, a bit along those veins.
I think the advice you give today is different than the advice you gave 12 months ago, right, sort of seeing the evolution of the surrounding.
What's your bet on what aspects of your advice will change 12 months from now?
Yeah, that's a great question.
There are some things that we got right almost 18 months ago that I have not changed, right?
Like the context engineering thing I think will always be true as long as we're using transformer-based LLMs.
That's just like the math of how they work, and you will get better quality if you use less context, so you'll always be able to engineer things to get better results.
What will change in the next 12 months?
I think I spent a lot of time thinking about infrastructure.
I think the entire software factory, this isn't going to be like AI advice, but I think the entire software factory stack will decompose into good interfaces and open components, kind of like what we saw happen in the Kubernetes world in the last decade of basically the more open and portable you can make these ideas, the more likely they are to get broad adoption and you need a certain amount of momentum.
to actually become the default solution, just like what happened to VMware, just like what happened with AWS, what happened with Kubernetes.
And so I think a lot of these like walled garden, like you can build a software factory entirely on Cloud.
You can use Cloud Code as the harness.
You can use Cloud Automations as your orchestrator.
You can use Cloud like managed code sandboxes, managed agent, whatever they call it, as your runtime environment.
And you can kind of do this whole verticalized stack.
And I just, I think that The advice I will keep giving people today is like, I think dev tools and dev tool stacks are probably going to stay or become even more open, open source, open protocols, whatever it is.
And so I would, I would encourage people to prepare for that.
And you can build anything now.
Don't.
don't totally know about on the code, especially if you're building critical systems.
Amen to that in terms of the sort of the open composable systems.
I had early on on the podcast, we had Matt from Netlify, the founder of Netlify.
And he was kind of making this passionate case, which I relate to, which is that the web had elements of it that were sort of closed, that eventually the sort of the open web won.
And I think we're all better off for it.
And I think similarly in the kind of the world of agents.
The closed ecosystems are very tempting because they naturally align and all the different pieces can come together on it.
But this belief that eventually open and composable leads to greater innovation and greater flexibility for organizations.
So I think this composability will win over.
The ordering, that's a whole different ballgame.
Yeah, which chips are going to fall in what order or dominoes, I suppose.
Yeah, how much of a lead do these sort of frontier the factories, the Devins, the OpenAI, the Anthropics of the world, how much of a lead do they actually have?
And how it's like, for them, it's like they're going to war every single day because I think that Open is catching up faster than anyone is ready to notice yet.
Dex, thanks a lot for the great conversation here.
Super, super fun.
Deep insights.
Also, thanks in general for sharing the whole journey here and the learnings and the practices along the way on it.
I think we learned as a community, and you're definitely sort of a chief amongst the educators over there, and that is much appreciated.
Oh, and thank you, man.
I am a huge fan of you and the team, and you guys were super early to this game, and you have, I mean, Patrick Dubois is like...
been a fanboy of his since he invented DevOps days in like 2009 or whatever.
So you guys clearly have a really good squad and like, I'm excited to see what you guys are cooking.
Thanks.
And thanks everyone for tuning in and I hope you join us for the next one.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast, it's a community, and we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more, and I hope to see you there.