# Intent-Driven Development and AI Factory Economics

**Podcast:** Engineering with AI
**Published:** 2026-09-20

## Transcript

doesn't mean the assertions are clear like what you mentioned mutation testing is there so then the next level of tightening needs to happen to say hey if i run mutations how much percentage of mutants survive uh that is the next level you need to your test development skill you take like mad book or superpowers or anything or for that matter your own skill.
So what I usually do is I use like a cloud dynamic workflow or some sort of a RALF loop, wherein I put in a rubric of these measurements.
I put in the base problem statement, which is the substrate.
And then I put in the skill and I put it through like a few iterations to see where they are missing.
And then I have a repository of rules, which I can use and pick one at a time, keep adding it, keep removing them.
And then I come out with a skill file.
which I then use for that project.
So long story short, tuning the skill is necessary for your code base, your team, and your models that you're using gives you a little bit more consistent results than just generally saying follow TDD or generally taking a TDD skill off the market.
Hari Krishnan is the creator at intentdriven.dev.
He's also been a guest on this podcast before, and his episode does really well.
He also has some really big news, so we wanted to have him on today to talk about that.
Hari, thanks for being on the show today.
Really appreciate your time.
Thanks, Kyle.
And I'm really glad to be back here.
And yeah, I'm glad to see that your audience are able to resonate with what we spoke about last time.
Yeah, glad to be back.
Yeah, for sure.
Well, and I hear you've got some really big news right now, Hari.
Like you've been working on, like, do you have a book deal?
Like, come on, man, tell me the story.
Just launched the MEAP, the Manning Early Access Program of my book called Spectrum Development, Engineering with Intent.
It went live on the 31st of August.
So that's the big news.
Again, thank you for your wishes.
I've been working on it for about three to six months.
I was writing the book even before I...
closed on the publisher, but thanks to the publisher, now it's out there.
Looking forward to feedback, and it's early access, so obviously anyone who's reading it can give us the feedback and figure out how we shape the book and take it forward.
And early access program, I'm familiar with it.
I can almost see the icon in my head.
But I forget, does this mean all the chapters are there, and you're sort of getting a pre-read before the paper book, or is this only some selected chapters?
That's a great question.
Thanks.
I almost missed mentioning it.
Because it's an MEAP, it's only the first three chapters that are live right now.
It's running at a 50% site-wide sale, which means people could get on it and read the chapters now.
And of course, you get it at 50%.
And eventually, when the prior book goes into print, you will get your copy.
Okay, you get the whole thing.
Nice, nice, nice.
So much has changed, and you weren't on all that long ago.
What stands out to you is something that is one of the bigger changes.
One of the things I noticed, I think your radar itself is brilliant.
It was not there last time and now it's on.
So thank you for doing that.
I think the space needs it.
I refer to it quite regularly just to see where people are at.
And of course, like other thought leaders who've been on your show, I get a sense of what their headspace is like in terms of these concepts that are going on.
So going by that, I think a topic we discussed last time is intent articulation.
And pretty much like my website goes, it's called intent-driven development.
Early on, I used to think I'm indexing or over-indexing on this term called intent because it naturally felt that that's what it means.
Whether you're doing specification or you're doing loops.
You're articulating intent to the agent so that then the agent can build something rather than trying to interpret what your intent is.
You make that intent explicit.
That was the core idea of it.
Yeah.
And I think it's good to see that resonate and a lot more information coming out there, which is in support of that.
Like, for example, the recent playbook that Anthropic released called the AI Native SDLC playbook or something like that.
And one of the primary things is it's called this thing called intent.md, which you capture your intent.
Thereafter, that drives the rest of the software development lifecycle itself.
Well, of course, the details are important and each of us needs to read it and see if it makes sense to us and whatnot.
But what I take away is intent articulation is of utmost importance.
And paying attention to that is something that all of us need to do.
How we go about doing it is all essentially building the harnesses about.
Yeah.
The harness is about getting intent out.
And that's another concept which I've been toying with, which I've been calling the intent harness.
So I'm glad to see this is taking shape right now.
Yeah.
Yeah.
You had your finger on the pulse.
I do remember when that came out, like, and I think it's like an intent MD file, like you say, and it's not like this was deep inside, you know, the article.
It was like in the first few paragraphs, right?
Because I only skimmed it and I saw it in there.
And as soon as I saw this file name, I'm like, I'm thinking, okay, Harry would love this.
This is.
Definitely up your alley.
So for the audience, because of course I remember, actually, I'm not sure I do.
How do we think about intent versus like specification, right?
Like, you know, I think of the story format as a, I would like, so that, is that towards intent or am I missing some sort of nuance when I think about it that way?
What is meant by intent here?
And you described it as.
the loop, you know, other kinds of things, that that's what the intent is helping you really nail down.
But I'm not sure I got the semantics right in my head yet.
Got it.
I think a simple one-line explainer in my head is usually when we are prompting or wipe coding our way to getting work done with agents, we are going from a vague idea we have directly to implementation, right?
And then whatever's missing...
Based on the implementation, we course correct and then we iterate on top of it to arrive at what we want to do.
But the issue here is the interpretation gap between idea to implementation.
The agent has to interpret whatever has not been said by the person who wants the implementation to be built.
So in my head, intent is the ability to reduce that gap between your idea and the implementation by reducing interpretation that agent.
is trying to do.
So the thought process behind intent-driven development is that as agents get more capable and models become more powerful, they obviously are working towards a factory-like setup, right?
Long-running, complex features could be built.
And we cannot be the bottleneck to keep clarifying what's going on.
And we cannot have this ping-pong relationship going on between us and the agent.
Agent needs to independently deliver, but also stay...
Align with what we want.
So which means we need a process which will elicit intent from us, make that clear.
And where it becomes clear is an artifact such as specifications.
So the authoring of specification, for example, is a conversation between the human and the agent, thereby bringing out those details, reducing implicit understanding so that...
then the agent can go on to build the complex stuff.
So I'll stop there if that's making sense.
Okay, no, that helps, that helps.
So one thing I'm getting from this is, don't worry, you don't have to like sit down and know your full intent perfectly ahead of time.
It is still an iterative process and the intent.md is where we work on that together with the agent that we're working with.
Okay, so that's one thing I'm getting from this.
Another thing I'm getting from this is that it's sort of maybe a bigger picture than the user story, right?
As a...
you know, as a person who at google.com, I would like to enter, uh, the name of the product that I'm looking for so that I can find out if my favorite shop carries it.
That's a great user story, but it doesn't do much in terms of the intent of like, well, our intent is to be the best e-commerce search in the world.
It seems like there's more of the directional and not just the specification, or it's a nice combination of the two, but I just made that up on the spot.
So am I getting closer now or am I still missing it?
You are getting closer in the sense that intent by its nature is implicit, right?
I know what I want, but I don't know whether, like, let's say Kyle understands what I want, right?
And there's that conversation between us.
I think it's a throwback to behavior-driven development, if you wish, right?
The whole idea of BDD, if I'm not wrong, is about pushing that conversation between people rather than, I think, okay, so taking one step back, right?
I think the wipe coding or prompting approach to...
talking to an agent is very instructional to say like, you know, I want you to do this versus if you get into a conversation, then if the agent asks you, is this what you meant?
And then you clarify.
That could be pretty much just the intent elicitation itself.
You don't technically need user stories, but user stories give a format which makes it a lot more easier.
But is that the only format?
Maybe not.
You have a lot more going on there.
Sure.
Right.
Yeah.
Okay.
Okay.
So I'm forgetting the important part.
The user story is a reason to have a conversation between a software developer and an analyst.
And so I'm forgetting the important part.
So the intent in D is a representation of that elicitation of the intent.
Wherein we'll arrive on, are we talking about the same thing?
Okay.
The best of what I understood from the Anthropic playbook yet, I don't think I would say I have fully grok what they have.
you know, mentioned there.
But they clearly have an intent.md and they also have a spec.md.
So I'm still trying to discern what's the difference between how they are interpreting it.
So far, what I'm trying to say is intent as I understand it and intent as like eventually we'll all converge, I guess.
But the idea is to make things explicit.
So one other article that's been coming to mind while we've been talking.
is earlier this week, our former colleague Martin published something from Giles Edwards Alexander, another form of ThoughtWorker.
I think he's still with ThoughtWorks.
Talking about this concept of the accidental blackboard, he's talking about is like having a single document, which becomes a shared repository for many agents, including people, right, that might want to use it as a reference.
And the idea of like many scientists or engineers in one room around a single blackboard, and we're all sort of using it to share ideas.
And yeah, that's what's coming to mind when you talk about this.
The intent MD file becomes that blackboard that other agents can read from, and the agent and I that are discussing the intent can use describe and so on.
Yes, and I think I could hazard a guess as to how that makes sense.
And this point is that agents, just like people, if you're anthropomorphizing them, need to communicate.
And essentially, the way I've been using tools like Linear or Asana or Jira, is not more like it's essentially the coordination between them, right?
So we need to communicate.
We need to say, hey, Hari is done with this.
So Kyle needs to pick up and run with the next task.
So we need that kind of concept to play off of.
And maybe I myself want to put something out of there on the blackboard, which I might otherwise forget.
I come back to it and pick it up.
So anything that humans have done.
as a way to think through.
I think that applies very much to agent.
So far, my hypothesis, at least, and it's been holding.
Yeah, yeah, yeah, yeah.
I tried to use Notion that way for a little while, and it worked somewhat pretty well.
I was trying to use Google Docs before, and that didn't work very well at all.
We ended up creating new documents all the time.
At least with Notion, we could sort of update document pieces a little bit at a time.
I haven't tried linear, though.
That's a neat idea there.
Cool.
Another thing that I'm reflecting on is that often the best conversations that I've had on the show, I will find out later someone was writing a book.
And I'm wondering if just that process of going and drafting is, you know, people talk about how writing is thinking, right?
And so some of the most thoughtful conversations I've had with people, the one like you and I had before, I later find there was a book coming.
And it's, yeah, there's a bit of a premonition there for me now.
Interesting.
Yeah, glad you mentioned that.
I think the whole idea of writing the book I started six months ago was more about like, you know, just thinking through the process myself, like what are we really talking about?
And essentially in this area of intent driven development, there's very fragmented knowledge out there.
And I myself wanted something which I could refer back to to say, hey, this is what I mean.
And this is what I want to index back on for every concept so that I'm consistent about how.
I prompt my agents and whatnot.
So that's how it started.
So I guess you're absolutely right.
Thinking through happens with the writing.
It's a very, how do you say, recursive process.
Yeah.
And I think it's almost in defense of the book as well, you know, or at least of long form.
I think there's something, you know, I could probably rip out a page about most things.
But if you ask me to write 200 pages about something, I'm really going to have to push myself, right?
And I'm going to have to think deeply and I'm going to check my research and my do research as well.
So, yeah, I think there's something to that volume or that size that is also a forcing function.
Cool.
So other things have changed since we last spoke.
A lot of guests and myself are thinking about like, hey, we were living in the era of token maxing there for a while where we were all rewarded by how many tokens did we spend?
I don't remember actually getting a bonus based on that, but we remember those times.
That seems to have changed.
How is this impacting you in your practice?
How is this impacting teams you have a chance to observe?
Like, what are you seeing?
I think it's quite a reckoning for all of us.
For the longest of time, we had a little bit of a free ride, I would say.
I at least would not really think about tokens as much.
But I think the first conversation now that happens with most of my clients, at least, is that what is the per-seat budget that we're looking for?
How does your harness?
Everything comes down to that, right?
How does your harness and your SDLC, the native SDLC?
you know, justify what is the amount of token.
And even to that matter, the amount of unit work done per certain unit token is how we have to start looking at it because tokens are not made equal, so to speak, right?
Because some models tend to be token hungry, but still they're very cheap.
So we have to sort of normalize those things.
So the economics of it is fairly involved, actually.
And I wouldn't say I have a full handle on it.
But what I'm trying to do at this point is try and expand energy carefully, so to speak.
So if there is specification process, for example, I think in my case, I do a lot of specification, I do a lot of grant work, execution, and then verification.
So the execution part is what can afford to be on lighter models or cheaper models.
The specification and exploration has to be more on the frontier models.
So division of labor is how I've been able to get some amount of rational in terms of keeping the costs under control.
But honestly, do I have a control on how much I will be using?
I don't yet know that because it's quite difficult because month on month, the economics of it has been changing and how much work gets done keeps varying.
So I think it will need another quarter, at least for me to arrive at saying, hey, this harness for this kind of work, you will be able to predictably say.
you know, a per seat cost of $100 to $200, maybe $300 at most, you could comfortably do it, right?
And the one thing that I'm guarding for is worst case scenario, should you run out of tokens and whatnot, is you should not be so dependent on the model that I cannot do anything else, right?
Like if I have to switch models and whatnot.
So which is why I am heavily investing in the strategic context in terms of my specifications and other ways of indexing that code so that...
Even if I don't have a full mental model, I can operate on it.
Even if I switch vendor, harness, model, I'm not impacted by it.
I'm trying to guard for that.
That's the current going on.
Yeah, yeah.
It's interesting.
And I'm working with a team that over the last little while has been working their way through not only the economics, but also how they think about sort of information security and PII and their own.
competitive, you know, the nature of their data, you know, and those kinds of things.
Anyway, so they sort of started with Anthropic Cloud Team and they're paying a certain amount per developer.
So it's not the enterprise account.
As I understand it, once you hop to enterprise, like you're paying by the token.
But if you're at the team level, you're not yet doing that.
And so it's closer to like a ProMax plan, which...
These amounts sound expensive.
You know, $200 a month is not uncommon.
I think that's what I'm paying, for instance, personally.
But when you start to look at how many tokens you actually use and what they normally cost on the API, you start to realize, oh my God, I've been getting away with murder.
That's cheap compared to what the per token pricing would be.
But as you say, a token isn't a token.
And I've also seen those same folks sort of say like, oh.
I'm here on open router and I can see that, you know, prices for this, you know, tokens over here are better and tokens over there are cheap.
But to your point, that's not necessarily the same thing.
The models use them differently and do different amounts of work with them.
And they may be faster, they may be slower, they might do better work with those more tokens or not.
So, yeah, it's a very confusing time.
Another experiment that I'm aware of, well, it's one that I'm doing, is trying to rent, you know, very large machines from EC2 at the moment that have, you know, sort of very large GPUs in them.
You know, H100s, call it four of those.
And just running your own, you know, go download Mimo from Xiaomi.
Go download, you know, whichever one of the latest quins or something.
And just see how far you can push the economics that way.
Rack rate EC2 instances are probably not the cheapest way to get a machine with four H100s, but it is something with which you can do some experimentation.
The part that I haven't gotten to yet, and I'll tell you how it goes once I do, is how many people can you throw at a machine like that, right?
Like if all it can do is answer one person very well, then this is not the right economics.
But if it can answer 100 people, then maybe, right?
But to me, that's the part which I have not yet discovered.
But yeah, I think that we all have to do this kind of economic work.
This is such interesting times.
As you say, the economics are a big part of it.
And eventually you might get down to the point where you have local models right on your laptop.
And there are experiments on those.
I think one got popular called Colibri.
The Colibri project was essentially to take a GLM 5.2 instance and load it up on demand, like not load the entire thing into memory, but sort of stream the model into memory as needed.
Of course, it is much slower, but if you can plan your workloads efficiently, then you could very well hand it off overnight.
Because it's going to be slow, but it's not bad, right?
You're getting GLM 5.2.
and get that work done.
So I was experimenting with that a month ago or so, but it might be leading to something is what I'm thinking.
Maybe you will have models which run locally or maybe on a small tower which is sitting right on your desktop, maybe not even on a cloud, but might be slower.
But that's where, again, it's important for us to get our harness.
Right.
So that we don't really have to wait for the agent to finish.
Let it take its time, but it's free.
So that's OK.
Yeah.
I made the mistake of searching eBay or Kijiji or one of these the other day for like used Mac studios.
And I went down this rabbit hole.
There's all these Mac studios for a little while.
They were selling with 512 gigabytes of RAM.
Right.
Which is that that's usually the size of a small SSD.
And that's unified memory.
And those things are selling for like fifty thousand dollars.
Apple doesn't sell that version anymore because memory has gotten expensive, but I'm not sure that explains that kind of pricing.
Yeah, so you can tell we're all sort of looking into this.
I doubt anyone is doing that thinking like, oh, I really want the fastest, you know, Logic Pro performance I can get.
That's likely someone who's trying to do local models.
Yeah.
There is also this concept that of...
A friend of mine was talking about called Rob Winkle.
He's talking about RLMs, which is recursive language models, which you could take like smaller models.
We could recursively keep asking the questions.
And, you know, that reduces your need for very advanced model.
You could just take the same question.
you know, reminisce on top of it and get the solution eventually.
That's another area I'm just trying to learn right now.
So RLM is another area to look into.
Last time we spoke, we touched upon a topic, which is essentially a spectrum development and you could define the spec.
But then once you define the spec, like what happens thereafter?
Like, how do you know if the...
application is built according to that spec or not.
So I've been experimenting a little just to see how far you could push that and just have like a quick thing that I could demonstrate or show you like how it looks like.
Let me quickly share my screen and talk about it.
Brilliant.
So, okay.
I think just to cut to the chase and show you a example of what it looks like, I'll quickly just expand this.
reduce this out so that you can see my full screen.
All right.
So my specification in OpenSpec format, again, OpenSpec by default doesn't really capture it in Gherkin format.
Close to it, but not really Gherkin, right?
What I have done here is a customized schema of OpenSpec, which I called intent-driven schema.
And that schema, you know, changes the specification format to contain Gherkin within the markdown.
And this Gherkin is what then gets translated into something that we are all familiar with, which is step definitions with Cucumber.js or whichever tool you choose.
Again, I'm using JavaScript here, so it's written the same.
use cases here in JavaScript.
So that makes it equivalent.
So what happens is in this approach, I think in your radar, I also noticed you have ATDD, right?
Which is acceptance test driven development.
So this becomes the acceptance test for the, you know, the agent to build the application against.
And so thereby, like, you know, it doesn't deviate from what's in the spec.
So even though like, you know, if the spec itself is like it's plain English, so to speak, I can still.
But this becomes more executed according to an acceptance test.
So that is keeping the application alive.
But that's only one part of the equation, right?
Which is essentially, if you look at what's happening, I think what we spoke about last time was this guy, which is spec first.
or spec anchored, I think these are the two areas that we looked at.
So spec-driven development by itself, by default is spec first because you're using specification to think through the process and that itself is a good value, right?
But worst case scenario, even if you are not really following strictly, you could be anchored, trying to keep spec and the implementation in close coordination.
But what we really want is more around you know, spec as source, which is, I only start with the spec for all implementations, right?
That's where BDD comes in.
So where I showed you how I take the spec, convert it into an acceptance test, and then that drives the implementation.
So thereby, you know, your code gets built.
But the problem I faced with this is obviously...
The outer shell is constraining the code sufficiently well enough in terms of behavior.
But what gets built internally is largely spaghetti.
It doesn't really make a lot of nice sense.
Of course, you can introduce DDD concepts and whatnot.
But what I found most useful is putting DDD into that equation really makes sense because BDD acts as the macro.
level check on the unit of work, keeping the overall behavior aligned with what you want to build.
And TDD kind of makes sure that, you know, and I am a mockist school of TDD, so to speak.
So when you're mocking and you're building out units, so sort of somewhere between these two guidelines, so to speak, or how do I say, envelope of light within which the aircraft has to navigate.
These units essentially are built fairly reasonably well.
Of course, there's more to this, but at least at a top level, the thought that evolved in my head is more like, how do these concepts fit in?
Like, you know, SDD, of course, gets you the thinking process to first spec first to spec anchored.
BDD helps you reach a little bit of spec as source.
And of course, if you want cleaner code and nice design.
That's where the testing, the test-driven development, niceties, that helps.
So I'm still building on this concept.
It's still very early stage, but you have to sort of live with this ugly little diagram that I have to kind of convey the concept.
I totally get it.
I've often given the advice to teams, if you have some code, which is a bit of a ball of spaghetti, that going in and applying unit tests to it, posthumously, this is now, this isn't, you know, TDD in advance, this is adding unit test after the fact, will force the spaghetti to detangle because each time you're adding a unit test, you're going to have to sort of like clean up this area in order to make it testable instead of an object that news up another object that news up another object and, you know, inception all the way down, you're going to end up putting in factories and you're going to end up putting in parameters and, you know, inversion of control and so on.
And so that's going to, it's a way to do it.
And test driven, on the other side sort of forces that that happens from the beginning so you don't have to stop and go back and do it.
So yeah, to me, I like that.
The BDD kind of keeping the behavior and making sure that the intent is captured and executed correctly but doesn't give you good structure and then TDD does.
Yeah, I like that.
And it really sort of shows you where spectrum and development fits.
It sort of fits in, like you say, this sort of outer envelope, which is driving these existing processes that we already know and love.
Ooh, that's interesting.
Nice.
So taking off from what you mentioned, right, that makes sense.
in a pre-existing code which does not have specifications or tests which is the large you know majority of what we have to deal with on a day-to-day basis I actually found you know using characterization tests as you would say like from Michael Feather's kind of legacy code refactoring that concept works like a charm because then you can isolate what area you're working on characterize it and isolate it and in the process of isolation you also nicely modularize it and you know Quality keeps incrementally changing.
Again, none of what I mentioned has to be day one.
You could sort of have your medicine and eat your veggies and eventually get better.
Yeah, yeah, yeah.
I like it.
I like it.
Okay, cool.
The other thing that's coming up for me, and this has kind of come up a couple of times now as we've been talking, is that...
I recently in my practice with Claude, and I don't know if this is them improving Claude code or if this is one of the plugins like superpowers that I pulled in.
I think it's probably a little bit of both there.
But Claude has started asking me like, OK, we've got the spec now and you've approved the spec.
And I have a plan based on that spec.
Are we ready to proceed?
And then they'll say, do you want to go sub-agent driven?
Or since this is small.
we can sort of do it in this one context session.
And my understanding is that part of what we're doing there is that making sure that each implementing agent gets to start with a fresh context.
It doesn't have to go through the entire conversation that we use to get to that intent, just the intent itself and not the entire conversation that led to the spec, just the spec itself and so on.
And so thereby it can have a cleaner context window.
But I can still keep talking to this one that remembers the conversation, which is great.
And I've started to think, like, for one thing, with your open weights conversation, talking about using frontier models for that sort of container conversation, I could imagine using a very good open weights model that's more of a sonnet level thing, I think, and less of a fable or an opus for the actual execution, for the actual development, especially with these guardrails in place, I think.
But the other thing that this is all bringing to mind is that, like, this is starting to very much to feel like, yeah, we're capturing the intent in this outer container.
And then, you know, when each one of these little subagents does pick up, like, yeah, superpowers is feeding in TDD.
And so that is helping me build a larger test suite and helping me build confidence.
So, yeah, it's, but it seems like this new concept of intent on top of it can really speed up the process of understanding, like, where are we right now?
Are we?
are we still building a SPAC hub or are we shifted to building a plan or are we implementing and really sort of better organizing all that.
So that's exciting.
I think I mentioned this concept last time, which is I let agents fight it out sort of to figure out what's the right thing to do.
So I call that skill something called the adversarial authoring skill.
So the adversarial authoring, in my case, I'm using two.
which is the author.
In this case, it's 5.5.
I was just experimenting.
And review is GPT 5.6 teras.
So this is reviewing what the other fellow wrote.
And if you look at my artifacts which get generated, you will notice that...
Let me just pull that up.
So this is the proposal.
This is one of the artifacts that OpenSpec generates for each feature that I'm...
or change that I'm proposing, right?
So the author proposed poll voting.
This is a voting application.
So I'm trying to figure out what's going on.
Now, what happens is these two agents, like one agent writes it, the other one reviews it.
And then they have like a document as this is the proposal.
And this is the council notes.
So the council notes are essentially the two agents argued as to what were the challenges raised by the reviewer.
And then the resolutions that were accepted between them.
And so basically what comes to me for review is already something that two agents have kind of like fought it out a little bit so that I can do that.
And this is also a way for me to kind of...
evaluate newer models, right?
For example, if I'm onboarding a newer model, I usually keep my reviewer constant.
I onboard a newer model.
So then I see if my review notes is going bigger or smaller, what's going on there.
So kind of gives me a vague sense.
I can't say this is scientific and all that, but it gives me a pulse on how these agents are behaving and where they are disagreeing or agreeing.
So that's one thing.
And I also have agents for developer and QA.
So they, again, sort of keep the guy, the QA is more responsible for the XPS test.
The developer is more on a test.
So they kind of fight it out with each other.
And yeah, this sort of helps.
Sort of a sonic level model.
That's brilliant.
Yeah.
So the sub-agent is what makes sense, right?
And I think I was testing out QN 3.8 max and kind of made sense to verify it.
So similar to the adversarial authoring, we also have like the senior developer and senior QA.
So these are like, you know, opposing roles and the QA is responsible for acceptance.
The developer writes the tests, the unit tests in the code.
And in this case, I was testing out QN 3.8 max and against the QA and I could assign a different model to it.
So just like what you mentioned, which is we could use sub-agents to do the work.
And also...
structure the harness such that you have a little bit more control in terms of how much context window we use, right?
It's really, really critical.
So instead of trying to use up the context window to its extent, sub-agents are a great way.
And I think since you're using superpowers, it does a pretty decent job of like handing off and bringing back the important aspects and not have to overload your...
context into a dumb zone, so to speak.
Yeah, for sure.
For sure.
A lot, obviously, of the challenges.
Like, how do you keep this so that we're in the smart mode?
Yes.
So a couple of things are coming to mind with this.
One, your technique of looking at the outputs is definitely...
So one of the things, when I started out on my project to rent large EC2 machines...
I tried to cheap out and go with this set of chips called the L40S.
This is out of love ways.
And I thought to myself, oh, these are just like two generations back.
These should be fine.
But I misunderstood.
They were two generations back, but they were also one of the chipsets that NVIDIA was aiming at like lighter weight work.
I think the memory bandwidth is less, I think is what's going on.
Like they still have plenty of memory, but the bandwidth is less and maybe there's less.
Processing power, something.
I'm sure I'm getting the details wrong.
But point being, the L40Ss were not really meant for what I was doing with them.
And it worked.
You know, I was able to get open code to talk to it.
And I was able to, you know, get it to go through.
I had a sample code base that was an open source project I'd worked on a year and a half ago.
And I fed that in there and said, like, go analyze this code and tell me what's wrong with it.
And it, I mean, it did, but.
It wasn't good.
You know, the results.
I did the same code base and the exact same prompt and Opus, and it came back in about four minutes.
And it had like really specific advice, you know, not just like, hey, you might have a SQL injection here, but it's in this line and it's also in these files.
And here's how you might go about fixing it.
But then Quint 3.6, which is a very well thought of model, you know, but running on this hardware that was not well thought of.
And it took seven-ish minutes.
So, okay, seven versus four.
It's not terrible.
But the output was not nearly as smart.
It was like, yes, there's SQL injection and you need to fix it in that file.
And that's it.
There was no, like, pointer to which line of code.
There was no explanation of how it got it right or wrong.
And, you know, and there was, so it's just that the quality wasn't as good.
Which, this mirrors something Birgitta talked about in one of her articles recently.
the exact same model, but on different hardware will produce better or worse results.
So I think we're used to thinking about this as like, oh, how many billion parameters?
Well, that one's going to be good.
And sure, maybe, but just like we talked about how tokens are not really fungible, I'm not sure those parameters are fungible.
But the other thing is just running that on different hardware, you're going to get different results.
Like bigger hardware for some models, it's not just going to be faster, it's going to be smarter.
and what it says back, which is really kind of amazing.
Again, it's the economics of all of this in a way.
It becomes one of the interesting questions.
But that reminds me of one more thing she said.
When she was on the show, she cautioned me against TD for agents.
She said she had done some research.
She had done, if I remember right, it was about three runs of maybe four different scenarios.
So, you know, our...
Our size here is small.
But when she looked through the results of these like 12 different sessions in this three by four grid, she found that when she asked, I think it was it was an anthropic model, which one I don't remember.
But when she asked it to use TDD in advance, it actually produced results that were on average worse.
Right.
Versus when she did not, the results were on average better.
Now, I asked her, are you using, like, is this part of, like, the superpowers version, or is this just, like, some other prompt that she wrote?
And it wasn't.
So she's talked about how she wants to go back and sort of, like, rerun this against, like, more battle-proven prompts, I guess, or approaches, at least, to how you might have an agent to TDD.
But, yeah, her early caution was we might be anthropomorphizing them a little bit too much when we ask them to TDD.
It may not actually be better.
Like you, I'm finding that it at least gives me confidence to see the test suite growing.
And especially when I have it go run like a mutation check against those tests to tell me like how good were they?
And it goes and finds mutants and kills them and it proves them.
It gives me confidence, you know.
But I haven't done what she's done.
I haven't actually gone in and like run the test to find out.
So, yeah, it's an interesting sort of...
Is it a debate?
Is it a finding?
I don't know.
But it's an interesting little nugget that I came across that I kind of can't let go of.
Got it.
I think my experience with test-driven development has been all over the place, right?
The same models have done a very good job when I just asked them to do TDD, the same model, on a different day, and probably based on how angry it is or how hard the processor is running, it gives me a different result.
But what I've noticed is...
TDD, it's tried to game through that, right?
It's trying to finish off as soon as possible and not really realizing that TDD is important to get the result right.
So that's where I kind of use like constant ways of...
asking it for evidence, right?
So for example, the simplest of hacks it does is it does a vertical slice.
It writes all the tests first and then it writes the code later.
Obviously, the code is going to be not so great because the design is not being driven by the test.
When you go, sorry, it does a horizontal slice, I meant.
It does all the tests at once.
Instead, we want vertical slices of smaller units.
But how do you get vertical slices for it to be done without it cheating, right?
So then I had to make it like, you know, you have to give me commits for each vertical slice.
And unless you give me commits, another agent will reject that particular PR itself.
So that forced it into vertical slices.
Then when we got into writing tests, of course, it's not like, ideally, if it wrote the test and only wrote enough code.
to get that test passing, this should be 100% branch coverage.
There is no reason for it to have no branch coverage, right?
But it still does.
So obviously, I need to have like a branch coverage to verify what's going on and keep it at that.
Now, even if branch coverage is right, it doesn't mean the assertions are clear, like what you mentioned, mutation testing is there.
So then the next level of tightening needs to happen to say, hey, if I run mutations, how much percentage of mutants survive?
That is the next level.
So what I did is I took any test-driven development skill you take, like MadBook or Superpowers or anything, or for that matter, your own skill.
There's a certain amount of tuning you need to do for your models.
and your code base for you to get a little bit of consistent results over time.
So what I usually do is I use like a cloud dynamic workflow or some sort of a RALF loop, wherein I put in a rubric of this measurements.
I put in the base problem statement, which is a substrate.
And then I put in the skill and I put it through like a few iterations to see where they are missing.
And then I have a repository of rules, which I can use and pick one at a time, keep adding it, keep removing them.
And then I...
come out with a skill file, which I then use for that project.
So long story short, tuning the skill is necessary for your code base, your team, and your models that you're using gives you a little bit more consistent results than just generally saying follow TDD or generally taking a TDD skill off the market and seeing what's going on.
That's my experience so far.
Yeah, yeah, yeah.
Oh, gosh.
This whole notion of skills and how it continues to be useful.
When I first saw them, I'm like, I mean, cool, I guess, but that's not going to work.
We're just going to type English into a file.
But honestly, it's kept going.
It's kept proving value.
And to your point, the tuning of them is an important thing.
I think some of that can be just sort of about the code base itself, about the other expected practices.
To your point, I just sort of let...
superpowers push in whatever it's doing.
But sometimes that means it'll start suggesting, well, you haven't done a PR.
Well, I don't do PRs.
I'm a solo coder on, and I, I always merge domain.
So I don't do, I merge domain frequently and, and I don't, so stop telling me to do PRs.
But so, yeah, I mean, to your point, these things all need adaptation, but I think you're saying something which is deeper and more nuanced than that.
Not just like, oh, this isn't my process.
but that literally tuning the skill can make it more effective on your code base.
And though the same skill might have went perfectly well on somebody else's.
So that's interesting.
Pretty much the same for us humans also, right?
I could be a very good engineer as far as it comes to extreme programming and test-tune development, but it takes me a while for me to tune myself into the code base when I start a new project.
And that tuning or rather that onboarding experience is what I think may be missing.
for agents and skills.
So that's what I'm experimenting again.
I don't want to say like I have a handle on it, but this has helped is what I would say.
I love it.
Hari, you're staying a step ahead of at least me and possibly some of our other audience members as well.
So yeah, I love how you're doing that.
Nice work.
So Hari, another thing that's coming to mind for me, In one of my recent interviews, this is with Ian Johnson.
He is a software developer in Austin, Texas, and he's currently working on a book called Harness Engineering, which I'm sure it is one of many with that title right now.
But I really liked him.
He seems sharp.
And I think he's the first person to come on the show and at least explain explicitly to me that they, as part of their process, as part of their team in a startup, they have built...
something that would resemble a factory, right?
And you and I talked about dark factories and we talked about like, are we ready?
Is this coming later?
And these kinds of questions.
And I think that's still something of a question, but.
Let me explain what he's done, as I understand it, and bounce it off of you.
As I say, other guests may have done it, and they didn't call it out so explicitly, but he called it out explicitly as we are doing a factory.
And what they do, they have a software developer assign the work to themselves.
So they're not assigning it to Claude or to Agent Alpha or something.
I'll assign it to Kyle.
From there...
I'll then somehow, and I don't remember exactly how, but do some sort of process.
I'll say, agent, go work on this.
Do I mark it with a tag or something?
And the next cron shop finds it.
Who knows?
But an agent is going to pick that up and it's going to do the work.
It's going to pull in the story from JIRA.
It's going to pull in the code that it needs.
It's going to make the change that it needs and write a PR.
Or maybe it'll give me some sort of preliminary step before PR.
where I get to do some review before we actually get to like, okay, let's put it into main bio PR.
But the agent basically does the work.
Now, I'm still assigned though.
And I think what I like about the way he's describing this is it's like, I get to look at this as a software developer and be like, library upgrade for a library that, you know, I feel pretty confident is going to be an easy one.
You know, throw that to the agent, you know, bug fix for like something that's probably going to be a fairly simple change, you know, throw that into the agent, you know, a new user interface change, but it's one that we've really well thought through.
And this agent, this area of the user interface is really clean.
Throw that to the agent.
New feature.
Maybe I'll do this one interactively.
And we sort of get to leverage our own decision process around like what to throw to the agent and what to work on interactively, just like you would today.
Like sit down with your favorite agent of choice, whether that's, you know, Quinn 3.8 or MIMO or GLT or whichever one, you know, and work interactively on it.
And I think I liked that.
I think I liked that it's still assigned to a human.
But I can imagine greater parallelism this way, right?
I can imagine greater work getting done, but there's still sort of a, who is the responsible developer to it?
So this may be me naively describing something that like everyone's already doing, or it may be me describing, you know, something that's unique or at least specific to this team.
But that's how I see it.
I hadn't heard a description like that of other teams which have decided on how a factory should be.
This doesn't particularly seem like a dark factory to me, but maybe kind of like it doesn't have the overnight quality, but I guess it is.
But I wanted your take.
Like, have you seen others doing things like this?
Do you think of this as what does this fit your definition of factory?
Like, what do you think of what you've heard in the description so far?
So I think the way I look at it is the factory is able to this indenting what you want out of the factory.
And then like, you know, you place an order and something gets manufactured, comes out of the other side, something like that, right?
That's, and everything internally is automated, so to speak.
And of course, like this QC, and if you want to be at the end of the line, like, you know, as the last person stamping your seal of approval before it goes out, that's okay.
But like, you know, everything in between, obviously, will be in various stages of automation.
And of course, like when a new factory is being commissioned, we're not going to have like everything run off by themselves.
We're probably going to have humans and robots work side by side and then slowly wean off on the humans and eventually trust the process enough so that kind of runs itself.
And like, I don't think it's very different from like our...
continuous delivery, so to speak.
If you want to think about it that way, almost every commit goes to production, right?
And the kind of rigor that requires is a lot more than what people would give credit for.
Same thing goes for the factories also.
So the small microcosm of that is how I would say my website works.
Like I don't actively look at the code that happens at all.
I keep pushing tasks of what needs to happen.
Let's say my book launched, right?
So I need to put a...
section on the website saying the book is live and go buy it and whatnot.
So I just push a task into my linear and leave it there.
Sometime whenever the agent comes on, it'll go pick it up.
It'll do the job.
It'll send me a PR with the design and I'll approve it and send it off.
Then at some time around, it'll build it and it'll host a temporary version of it and say, is this okay?
I'll say, okay.
And then it does it, right?
And I think it also does a periodic check of the SEO.
Like I think, wow, we figured out a few issues around.
So it does that.
It keeps coming back to me.
It's running.
And essentially my primary interface to writing code, at least for my pet projects, has become my mobile, right?
It just keeps pinging me and I keep responding to it.
And that's pretty much all.
And the coordination is happening through a task board or something like that.
I don't know whether I can call it like my micro factory, so to speak, but that's how I'm thinking about it.
And in due course, I hope what will happen is as teams, like let's say we are a team and we are ideating and we are creating, pushing it out into the board.
And there are agents which go keep building it and like, you know, come back with.
reasonable amount of accuracy so that you don't have, you're not burdened by review and you can push it out with a fair bit of confidence.
So that's the part.
The last part is the most important thing.
How confident are you to push it out without looking at it, right?
It's again very parallel to how CD is, right?
In continuous delivery, do you have the, how do you say, do you believe your system enough to say every comment goes to production without ever looking at it?
That's the similar question I would ask in terms of where is your confidence you're building in this factory situation.
Yeah, yeah, yeah.
Oh, I like that.
I like that.
Something I didn't get to ask Ian, but I get to ask you now because you mentioned it.
I like how you talked about like, oh, give me a preview so that I can see it.
And that reminded me of a project that I'm doing where it's essentially a static website with, you know, Cloudflare pages, workers.
I forget which one.
And, you know, that automatically built you a preview.
So it's very easy to go and look at the preview.
But then I thought, well, this other project I'm doing, I don't know how easy it would be to see this preview.
Like I had a second pre-production environment there for a little while.
I could have said, like, go shape it to that, you know, pre-prod.
And I could have looked at it there maybe.
But I don't have built out the tech to do an ephemeral, you know, instance of my application.
And unfortunately, you know, there are parts of it that have grown just complicated enough that that wouldn't be a simple automation to do.
It would be possible, but I guess it's a couple of days, maybe a week of sitting down with Claude to get to the point where I could have that ephemeral, you know, preview, which maybe I ought to do.
But again, it's just me, so I don't.
But how do you think about getting this, like, let's say this were an enterprise environment.
And not just Kyle's little static website or Kyle's somewhat clunky larger application.
What if this is enterprise?
And you've got multiple teams, multiple domains.
You've got this team over here, and they're working on this back-of-house thing, which is critical to billing and critical to analytics and critical to revenue recognition.
They're publicly traded.
And then you've got this other part over here, which is more focused on e-commerce and sales.
what are we even previewing, right?
Like, and we sort of change, when we change the architecture of the e-commerce area, for instance, and stop depending on this third-party service and introduce this new first-party version of the same thing, I can only imagine that keeping that, like, give me a preview running could be pretty complicated.
Now, I don't think you're saying Kyle will have, the factory will be aware of all of this at once.
Or maybe you are.
Maybe eventually.
But at least on a team-by-team, sort of subcomponent-by-subcomponent level, how are we going to keep it all straight, I guess?
Because each team is going to have the same challenge of how do I make my thing deployable enough, but I have my dependencies.
Are they deployable enough to have this safe preview environment?
That's really pushing a fair amount of rigor, engineering discipline on teams in order to be able to maintain that.
but it could be powerful.
How do you react to where I'm going?
Am I overstating it, understating it, or do you think I'm right on?
I think it's a very relevant question.
That's the hardest question also to answer, right?
Some of the teams that I'm working with, we're trying to push the idea of like, the way teams are organized now, right?
Like, you know, each team owns certain things.
Maybe it's a result of Conway's law or the way you put it.
And, you know, they're organized in a certain way because they ship certain things, right?
But when you're looking at a factory, I honestly don't know if it should continue to be organized that way because the modularity of what gets shipped versus...
Like who's sitting in, okay, let's take this analogy, right?
You're talking about a factory.
If you have the shop floor, you also have the executive rooms, right?
Now, obviously, if agents are running the shop floor, the humans are not there.
Humans have to be in the executive room.
The humans on the executive room, they should not ideally be in that same bounded boxes, right?
They are operating at a certain level of abstraction, which could be both good and bad.
And I don't think we have fully figured that out yet.
Now, what's going to happen is in terms of reviews, at least what I can tell you, what does a preview look like for an enterprise situation is what are your domain-driven boundaries?
Like, you know, if there's a feature going in, you're going to do an analysis of that.
You probably will get a C4 diagram of what's the change like.
Are you approving of the C4 diagram?
And then if you approve of the C4 diagram, then the task list and whatnot.
that gets generated out of it.
How closely does that resemble that C4?
How can you keep those things aligned?
Then that comes as a preview and then it goes in.
And nothing takes away from the fact that you still have your regression suite, which is deterministic, which you cannot, you know, hand off to saying agent will verify this manually, right?
Like it's as bad as saying, you know, I'll just depend on manual testing to get a job done.
That's not going to happen.
So you still have...
all your elements of software engineering and which is why I strongly believe that the more we get into this factory business, there's a lot more work to do in terms of software engineering.
We're just exposing every ugly part of our, you know, STLC that we have ignored and have been lazy about.
Now you cannot be, right?
So it's a painful process to say, I want a factory and that means you automatically need top of the line software engineering.
And that means also reorganizing your team structure as more around the entire product rather than saying, I am billing, I am shipping, I'm fulfillment.
Maybe you're an expert in that area.
You probably will still encode those skills and the knowledge, the wikis for that.
But are you going to be isolated in that team and responsible for shipping that unit?
Maybe, yes, you still probably need like what you said, Baudian.
Maybe that seal of approval, that HITL is still required because someone has to be taking responsibility for that part of the harness.
But the bigger question here is, again, sorry if I'm rambling, bigger question here is when you're building the factory, the factory itself becomes a deliverable, right?
And there has to be a...
a set of engineers who work on that.
And that's the harder problem.
And I think that itself would be a couple of hours of discussion as to what it is.
Yeah, you're right.
I'm realizing as you speak that, right, when we think about manufacturing is not a product.
Batteries are not a product.
It is not an easy thing to go and, well, I mean, you can buy a building shell for a factory, but that does not give you an up and running assembly line that's functional, right?
Or any other kind of manufacturing arrangement that is functional.
It's just a shell.
Yeah.
And the creation of a factory itself is a lot of engineering to do it.
So, right.
So conversely, this idea that, oh, don't worry, the robots are going to do all the work.
is going to require a whole lot of software engineering to get right in the first place.
And maybe agents will help us with that.
Certainly they will.
But yeah, almost certainly they will.
But it's still going to require some brains to get it done.
Ooh, this is interesting.
One final thought, and then I need to be respectful of your time.
But I'm thinking back about a team that was suddenly going to get helped out by...
a team of contract engineers, I think in Eastern Europe in this case.
And it was like, look at what we're doing for you.
We're giving you all this help.
You know, you, you said you didn't have enough people.
We got you this team of four people suddenly.
And, but then as part of the, the business arrangement, somebody in the engineering leadership on the customer side, who I was working for at the time, decided like, yeah, we better make sure that we check all the PRs, this new team in Eastern Europe.
We've got to check all their code.
It's got to be an FTE.
But what that team found out pretty quickly is that suddenly all they were doing was code review because the four people in Poland or whatever it was were creating way more code than they could possibly keep up with on the PR side and write their own.
And suddenly they didn't like their jobs very much.
They started to feel like, you know, somebody else is getting dry all this code.
I'm just doing PRs all day.
And so that's obviously an analogy to sort of where we are now.
You know, it's a different flavored analogy.
I hope we can find ways to like our jobs better than that team did.
But I think we will.
I don't think it's a perfect analogy.
I don't know.
Yeah.
Yeah, I think it's always going to be that joy of coding versus joy of building.
I don't know if that's even the right way to put it.
But coding, as I saw, while I enjoyed the craft of putting things together by hand, it also was limiting in terms of how much you can get done.
especially for smaller teams and individuals who are building stuff out.
The moat was always that I don't have enough money to build stuff, but I have so many of these ideas and I would like to build it.
And now, if you honestly take back...
step back a little and look at it.
The joy of coding is also joy of building, right?
It's solving your problem.
So I don't know.
But yes, certainly it's not fun to be reviewing code.
And that's the difference in my analogy.
The team in Poland had gotten to take on both the building and the coding.
And the team, I think it was Ohio at that point, were only doing review.
And that's why they didn't like it.
But to your point, that's not what's happening here.
We are working with the agents, but we're not just PRing and they're doing all the building.
Like, it is a harness, right?
Like, we still have the reins.
So we are still getting that to, I personally feel like I'm experiencing that process of building, even if I'm not experiencing the coding anymore.
Yeah, cool.
That gives me hope.
Awesome.
Definitely gives us all hope, I suppose.
Yeah, yeah, yeah.
So, Hari, tell us one more time before we go where they can find the book.
And congratulations once again.
I'm really excited to go and read it.
But yeah, tell me in the audience where we can go find it one more time.
So the book is called Spectrum Development and Engineering with Intent.
And of course, intent is what we've been speaking about.
It's live on MAP Manning.
And right now, three chapters are available to you.
And soon enough, we'll be putting out more chapters.
You get to read it as I write it.
And of course, if you have any comments to make, do jump in on the live book forum and you get to shape the book as it evolves.
And yeah, right now, you have a 50% discount going on because it's introductory.
So that's where the book is at.
Awesome, sir.
Hari, thank you so much for talking with us today.
Appreciate having on.
As always, it's a joy talking to you, Kyle.
Again, glad to be back here.
Thank you for the opportunity.