# Constraint Engineering: Building Reliable AI Workflows

**Podcast:** Engineering with AI
**Published:** 2026-10-19

## Transcript

Ian is a staff software engineer at Parental, which provides paid parental leave insurance to employers.
And over a decade in software development, he's found a niche in educating and describing and getting the industry talking about what it is that we're doing.
I admire the effort he's put in at Takoda.dev in doing just that.
And today, he's our guest on the Engineering with AI podcast.
Ian, thank you so much for coming on the show today.
Really appreciate having you on.
Hi, great to be here, Dial.
Let's hear a little bit more about you.
What would you add to that?
I know you've got a bunch of projects that you're writing about and talking about right now, and I wouldn't even know where to start.
Yes, I'm a staff engineer at Parenteau.
And since I've been there at the beginning of the year, I have been doing a lot of experimentation with figuring out how to deliver software better in an end-to-end product-driven way to actually get it delivered.
Those experiments led me to...
find patterns in areas that eventually culminated in blog posts and then a book.
In addition to that, I've also been starting to create different projects, open source projects, and different teaching resources just to kind of distribute out the knowledge that I've gained for about the last nine months.
Oh, so this is pretty recent then?
Yeah, so...
The main thing that got me kicked off here was that I joined Parento at the beginning of the year.
And going into that, I knew I wanted to use quad code.
The first thing I needed to do was to figure out how to do that in a safe way.
The first three weeks or so was actually just a lot of testing and putting...
characterization tests and other layers of tools around the project, like linting and static analysis, things like that.
And then once I had those guardrails in place, then I felt comfortable to be able to start using Claude more.
And that's sort of what enabled me to start to experiment with this stuff more because I was able to easily back it out or cancel it if the test went red.
Yeah, yeah.
Things like that.
One of the things I picked up about your background, and maybe I'm getting, I hope I'm getting the details right, but it looks like you're in Austin, but you're working with a team out of New York.
Is it all remote or are you like the remote and they're there or like, how does that part work?
Yeah.
Yeah.
So we are all remote.
It's a remote first company.
There are quite a few folks in New York, but there's a, we have some folks who are in California.
I'm in Austin.
We have somebody in Florida as well.
So we're distributed across the United States fully remote.
Cool.
Sometimes I wonder if in that situation that like long form writing works better than like short form Slack messages, right?
Because you can hope that people will read it asynchronously versus if you're in person with them, you can just kind of tell by the look on their face that they're not sold yet and keep going, you know?
But for all remote teams, I've often thought that like long form writing is a good way to go.
So yeah, but okay, cool.
Nice.
Thank you.
So, but recently it's, it's not just the blog posts and the descriptions.
I mean, you've, you've, you've got some projects going and one of them's a book, if I understand it right.
Like, tell us about that.
Yeah.
So I recently wrote a book named Tarnas Engineering, building reliable workflows around non-deterministic agents.
And that is.
The high-level patterns that I was able to understand from harness engineering through those experiments, a lot of the information that came from there either came from experiments at work or experiments with open-source software.
And then it's at that point that I started to notice some recurring patterns, and so I started to record them and discovered that there's, I would say that there's like three layers around the harness.
Each one of them has similar concerns, but they have different shapes in the way that they address it.
Okay.
How do you mean?
Yeah.
So let's see.
One example is the way that a rule works.
Actually, a hook is probably a better example.
Let's use a hook.
So if you were to write a hook into your project, that's something that happens at the configuration layer for the tool that you're using, say, Claude.
And a rule will sometimes not hold that well.
A hook is going to be more deterministic because it's actually going to get called.
But it's still at the mercy of the tool, of the loop that you're using.
It's still defined within Claude itself.
And so if you wanted to gain more control over that, then the next step is to bring that up to the harness.
So a hook that uses something, or sorry, a hook that is there for testing is completely under your control.
It's defined in the repository.
And if you wanted to have a more structured approach to that, you could actually...
promote that into a custom harness where the TD-Dub loop is actually in the harness workflow itself, then that allows you to remove that bit from your charter.
So you no longer need to have a hook or a rule because the workflow itself is enforcing it.
So as you go up in scale away from the model, then you need to have more and more layers of determinism around it to kind of restrict what comes out of the non-deterministic model.
Yeah, I got it.
So one of the fun games these days is figuring out what people mean when they use words.
Because I think we all use the word harness when we're talking about this stuff.
But I wonder if we're using it all the same way.
You also mentioned the word charter.
which I think I saw in one of your projects.
And that may be in common use, and I just don't know it yet or not.
Let's start with charter.
That seems like the more specific one.
Yeah.
Yeah.
So charter is not in common use.
It's the term that I use.
By charter, I mean the agent-specific context.
That is the configuration for your agent, your cloud file and your dot cloud folder or your agents file and whatever other, you know, configuration folder that your harness pays attention to.
And the reason why I call it a charter is because it's special context.
There's a lot of things that enter your agent as context, but the things about the charter is that they're special.
First of all, they map to agent primitives.
And then also, if you have something, say, in your cloud file, that's in there every single turn.
So a lot of people call all of this context, and that's true, because it all enters the model as context.
The reason why I treat this context as special is because it directly...
changes the behavior of your coding agents.
And in that way, it's like configuration.
The reason why I call it a charter is because it is like an agreement with your coding agent.
So it's a charter between the team and your harness that says, here are our standards, here's what's acceptable, here's what we want.
Okay.
So like having a team charter in terms of like how we work and like a ways of working, some people will talk about that.
Right.
Okay.
Got it.
No, I think I get that.
And earlier when you were talking about harness versus the hook that you sort of started out with, you talked about how it seemed like you might have described the hook as not part of the harness.
But I'm not sure I got that right.
You might have been talking about it separately, but not as though they were separate.
Just maybe different pieces of the same thing.
So, yeah.
How do you think about that?
Yeah, that one is an interesting one.
And I think that it actually has the same form of several things like tools.
So I would say that the hook is kind of in both places.
The definition of the hook is in your charter because it's in your repository.
But the actual execution of that hook happens in Claude.
So that's the implementation part of it.
The charter is what you define.
So in a lot of ways, what is defined in the charter, that is left in the charter.
And the line between it and the harness is execution.
So that means that your hooks or your tools, that sort of setup is kind of like configuration or...
an npm, like a package.json file, where you're saying, here's all of my dependencies that I'm declaring, and then the tool implements them.
But you still have control over the what.
Got it.
That almost sounds like you're bringing in the specification or requirements or like the Jira story, for lack of a better word.
Is that part of our charter?
Or is that like, that's the workflow that the charter processes?
Yeah.
Yeah, I would say That is not part of the charter.
That's something that the charter constrains.
So a work item like a JIRA card, that would enter either your workflow, if you're working interactively with an agent or a factory or whatever.
And then the charter is what's going to constrain that.
So as that work is going through and the agent is implementing things, then if the tests fail, then that's a charter violation.
And the response of the agent is get it in line with what the expectations are.
Got it.
Okay.
I think I'm following you.
Okay, cool.
Having hearing you talk for a minute, it sounds like we can guess what tools you reach for.
But you've got, you know, a bunch of different work that you do as an individual practitioner, as a person who does work, right?
You've got different jobs that you do from video production to writing to most likely software development.
Maybe answer the question that way.
Like, do you use different tools and all those different things?
Do you, or, you know, I mean, you've kind of mentioned Cloud already.
Like, yeah, let's talk about what tools you use when you're sitting down to do some work.
Yeah.
So, Cloud is definitely a tool that I use.
That's probably the tool that I use the most often.
And that's what I use predominantly to work.
And then, in addition to that, I've also used Codex and Kimmy.
I've had good results with those.
I've been pretty happy with the speed of Kimmy.
And in terms of harnesses, along with Cloud Code, I've also used Pi.
And I have not tried OpenCode yet, but I really want to.
I have tried Codex.
I did mention that.
And also recently, I have been playing with this product called 3i.
Got it.
Okay.
I've heard mentions of this is Pi, the agent SDK, not Raspberry Pi.
And so listeners may be familiar with that sort of a kit that allows you to make your own.
I guess I'm probably getting it wrong.
But triple I, I'm not sure I've heard referenced anywhere.
Can you talk about that one?
Sure.
Yeah.
So three I is basically the idea of.
The idea of loop engineering, graph engineering, and all of these specialized words that we keep inventing and that really are reinventions of things that have already been defined in software engineering.
We're just applying them to agents.
So the insight that I see from that product is it kind of doesn't reinvent everything.
by design, a distributed system.
So it works like a queue.
It has a dead letter queue.
It has retries.
And so your agents all of a sudden are not restricted by some kind of an orchestra that must be an agent that could be inferential.
Instead, it is deterministic code.
It is like every other job queue out there.
It just happens to run agents as jobs.
A worker pool kind of thing almost.
Yeah.
Gastown was sort of oriented like that a little bit when I was playing with it.
They used bead as a way to pass information back and forth between the different agents.
Okay, interesting.
Cool.
And you called it 3i, not triple I.
Okay.
Yeah, I think that that's what it's called.
I'm honestly not sure, but I think that that's what it's called.
Is it like primetime ready, like use it in production, or is it really fun?
Not quite yet.
It is something I've been experimenting with.
It's still pre-1.0.
I think that they're getting close, but there's definitely workflows I can run with it that support my work that maybe I wouldn't use as a production service, but they support production work.
And I would expect that once 1.0 comes along, I think it's going to be very good.
So play with it now if you're interested, but maybe wait till 1.0 for the harder work.
Okay, got it.
So you mentioned Kimmy.
Probably, did you use a specific, I want to say harness, but yeah, let's use the word harness.
Did you use a specific harness with Kimmy?
Or just like, I don't know if they have their own.
Like DeepSeek, I guess, does now.
Or did you use it with, it's probably possible to use it with Claude.
I think that's possible.
Like, how did you go about that?
Yeah, so there is actually a Kimi CLI that I've used, and that's worked pretty well.
It has been pretty fast.
It's pretty minimal.
And the other place where I've used it a lot has been with Pi.
And usually whenever I have API keys, that's kind of like my go-to harness that I'll change things with and see what they're capable of just because Pi is so extensible.
Yeah, yeah.
Cool.
It's funny, where I work right now in my day job, we're not allowed to use things like Open Router, Open Code just yet in order to access back-end models so that we have this bedrock that we can use and we have that cloud code that we can use.
So I'm not used to that world.
I'm aware that it exists.
Since I pay for my own cloud code, I just end up using it.
Anything else would cost me more money kind of thing.
I mean, it could be making it cheaper, I suppose, but I just stay with my subscription.
But yeah, it's such an interesting time with all this explosion of open weights models.
Yeah.
Yeah.
Kimmy for sure is getting a lot of airtime.
Yeah.
There seems to be a lot of interest in MIMO and GLM-52 and a couple of different ones right now.
There's probably one I'm missing.
Of course, Quinn 3.8 came out recently.
Do you do a lot of like playing with different models or is that part of your process?
Not too much.
I only recently started to branch out into Kimi.
Most of what I had done beforehand has been OpenAI or Anthropic.
I just recently started to branch out into Kimi.
I also intend to take a look at GLM and some other things, but I'm only just starting that search right now.
Yeah, yeah.
MinMax, I think was another one, if I remember right.
Anyhow, I have not been able to keep up with them all.
I have a crazy thing I want to try where I rent some enormous $60 an hour machine from AWS and try to run, well, not Kimmy, like multiple trillion parameters, but at least one of the larger ones, just direct and like try to get a sense of like how many people can you put on, right, eight H100s if you happen to have them.
Okay, so we've talked a little bit about the way you work with these tools.
What about like...
coding versus, you know, video production versus writing, like writing with AI is such a, an interesting topic right now.
It is.
And I will, I'm, I'm talking too much, but I'll, I'll, I'll tell one more story, which is that I wrote a book, not about software development, about another subject I'm very passionate about.
And I just used CloudCode to do it.
And I, I didn't, it was kind of in a way, like the only reason I was able to write the book is because I knew.
Claude would be shameless enough to just go decide to write this book if I told it to.
And one of the things that I learned is that I still had to drive it a lot.
Like it was three months of me driving that process.
I know there's a lot of people talking about like, ah, it's all slop, as though it's like two prompts and we just try to make money on whatever comes out the other side.
I doubt anybody's doing that in any serious way.
But just before I even bring up the concept of writing with AI, I want to, you know, if there's going to be a lightning rod, let it be me.
I will admit to having done it.
But yeah, what tools do you use when you're doing video production and writing and all that?
Like, where do you find it useful?
Sure.
Yeah.
So I have not yet implemented any AI tools of video production yet.
That I'm still doing myself manually.
I'm sure that there are tools out there that can help.
I just haven't gotten there yet.
So that's just where I am right now with the AI stuff.
I was hoping you'd have a hint that could save me all the editing time.
This is starting to become too much.
All right, fine.
No, not yet.
But the writing stuff, I definitely do use it for writing.
And that's something that I don't hide.
A lot of folks will have shame about it because there is a lot of slop out there and there is a lot of folks that will call it out.
And they kind of have a point.
So what I've done is my approach is now a pipeline.
So instead of relying on the AI to do.
the writing.
Instead, now I have defined a writing pipeline where there are certain phases we go through and I'm much more involved.
So we start off with the prompt of what I want to write about.
And first it'll come up with just a general structure of, you know, here's how we're going to, the audience, and here's how we're going to like approach this.
And then it'll ask me some general questions.
I'll say, this is what I want to talk about.
These are the specifics.
It'll get some feedback from me.
I'll answer that feedback.
Then it will write a rough draft.
And then as a human, I then step in and I read that rough draft.
I kind of give it a once over.
It is intentionally rough and sometimes I will steer it.
Sometimes I'll accept it.
It depends on how good of a job that it does.
Sometimes it will go off on something that I don't want it to.
And we'll say, I'll cut that part and refocus over here.
That's usually what I have to do at that stage.
The next stage is a slop removal, which removes all of the slop that's in there.
And then it also, and like band words, things like that.
And then it also has some of my own specific word or specific rules for the things that AI does that I don't like, the AI tells.
One example that is.
uh, a pre-summarization sentence.
Like, here's the thing.
Yeah.
Does that a lot.
Um, so I have a couple of custom rules like that.
Uh, after that, I take another look at it.
I give it some feedback and then it does a voice pass.
I've told Claude what voice I want to use and how, you know, the words I want to use, the language level and my audience and things like that.
It'll go ahead and cover that.
And then, again, I just kind of do a general feedback session.
Once it's ready, once it's accepted, then my job is just simply to post it.
There is one more thing I do, though.
which is when I'm ready to post it, I will go through and I'll do a cursory rereading of it.
I will change things on the spot.
I will remove punctuation and make sure that it reads well.
So by the time that it gets to being posted, it's been proofread six times likely.
Yeah, yeah, yeah, yeah, right.
Interesting.
Oh, gosh, I don't want to go too far down this rabbit hole, but maybe just a little deeper.
Okay, so to your point, I think it's almost unfair if someone sees, like I mentioned, a blast radius or load-bearing point or something, and they're like, oh, my God, this is terrible.
So on some level, you're kind of trying to get rid of some of those obvious tells.
It almost feels unfair, right?
Because it does not mean that the person who is like authoring this, like just had the AI do it, right?
Like the AI didn't just do anything at all.
You know, I mean, it's, yeah.
So I like where you're going.
This almost sounds, you described it as a pipeline.
Should I think of this as like a CI pipeline?
I mean, it very well could be a CI pipeline.
It's just the only thing is that it involves me as a person.
If I can automate that away, then I could put it in CI potentially.
Got it.
Got it.
Got it.
Cool.
For video stuff, I'll like run FFMPig by hand and sometimes I'll like tell Claude, this is the three things I need you to do.
But I don't know.
It doesn't seem to quite have the same level of like usefulness that it does in coding for these things.
Yeah.
Yeah.
So the thing that I found with writing that I've appreciated with that approach is that I felt more in control and I've also felt more connected to my readers.
Whenever I was just pumping out AI swap.
which I wasn't really ever doing, but there was less supervision beforehand.
And so with that less supervision, there was more slop.
And my audience didn't enjoy that.
And I didn't like that either.
And I don't want to make people read slop.
And so a lot of that really made me rethink the way that I was writing and decide to become more of a collaborative player, not to make AI right for me, but for AI to help me to write better.
Right.
And OK, so instead of like a manager journalist kind of relationship, it's OK.
Interesting.
Nice.
OK, so let's let's get to teams that you serve.
Right.
And obviously in your day job, like there's a team that you're a member of, you know, in a position of some seniority technical leadership.
We all are every every single one of us in my mind.
So, yeah, I'm interested to hear about the teams that you serve.
So let's kind of start like at the left.
Like when it comes to product management and how we break down work, how we take on work, how we write about it, you know, some people might say, oh, we used to do, you know, product requirements and business requirements documents.
And others would say like, no, no, we did epics and user stories.
I have a sense that no matter what you did before, it might be a little bit different now.
So what's that been like for you?
What was it like before and how has AI changed it?
Yeah.
So much of what we have done has actually been to accelerate the pipeline that already existed beforehand.
So previously we were using Jira, but we weren't using it that much whenever I came on board.
the repository and the project were not set up for many collaborators.
So that was part of getting it set up for that collaboration.
Folks who bring work into teams that you're on, are they using AI to write up what they want?
Part of our process with the product work is to use AI in it.
Now, we usually don't have AI author it to begin with.
Usually we will...
author it ourselves.
It'll come from a human or it'll come from some kind of list or standardization from a human.
And then at that point, we have a refinement step where essentially we create a Jira card, we put everything that's relevant in there, and then AI will take a look at that and make sure, are there falsifiable acceptance criteria?
Is this a small change?
Does this have all the required information I expect to have on it?
Like, is it a bug?
things like that.
And that basically preps it to be able to be picked up.
So that way, no matter where the idea comes from, eventually gets to the point where it's refined well enough that an agent can actually do work on it.
Okay.
Okay.
And is that sort of a goal in state that you're looking for?
Like that's a gate?
Like once an agent could work on it?
Yes.
Yep.
That's the first gate that we use.
Okay.
Got it.
So who, you said a human, which is awesome.
I love that.
With some teams, it might be like, oh, there's a product manager that, you know, sends an email or maybe that product manager, you know, has done something in Figma or maybe they've like authored something in Jira.
In other teams, it might be a tech lead that does that.
Like, how does it work where you are in terms of who writes, you know, that?
Not necessarily the JIRA card, but the thing that comes in that you're writing the card about.
Let's start there.
Yeah, so usually the JIRA cards are written by a product person.
And they are not usually tied to some kind of request or bug.
Sometimes they are.
When it is external work, that'll usually come in through our external channels, through our support channel, and then get handed off to product.
And then products will get that carded up.
And at that point, we'll start to work on it.
And then in addition to that, sometimes we'll have ad hoc work or more often than ad hoc, honestly, work is we'll have work that generates additional.
smaller cards.
And so along with the product person who is defining these things and getting them in there, often the developers are also creating cards, either as some kind of individual change that we need something else, or as we need to break down this card to many other cards.
So we do follow the Agile We follow the spirit of the Agile methodology in that we break things down, we make things small, we make sure that we're able to push things out in small independent releases to be able to experiment quickly.
And we have not yet challenged that.
I know a lot of people are challenging Agile right now.
Right now, we are just doing what we know works fast.
Yeah, yeah, cool.
I like how you say faster.
Is that the goal or is that the observable outcome so far?
That's the observable outcome so far.
Actually, the goal is really more to deliver value, which I feel like we've done so, but it has had the side effect of increasing our speed.
Cool, cool, cool.
Okay, so humans, maybe this is something that might come in from an actual user, which comes in from support, gets shaped that way, and then it comes in as a card, and maybe we shape it from there.
Okay, it comes in a couple of different ways, which totally makes sense.
A lot of product companies will...
We'll understand that.
Yeah.
Okay, cool.
And then developers are sort of picking it up, breaking it down into smaller things.
Okay, nice.
Nice.
Thank you.
I like how you talked about then an agent can work on it.
So is that what happens next?
Like what happens next?
Yeah.
Yeah.
So once a car is well-defined, then that's whenever an agent is able to work on it.
So that's whenever we dispatch an agent.
We either execute it from within our factory or we...
run a session interactively on our own computer, depending on what kind of work it is.
Most of the time whenever we have interactive sessions, that's because we need to check UI or UX or some kind of thing that we really want to ensure that is interactive in a human way, get the feel for it.
But a more automatic change, something on the back end that has unit tests, that can go straight through the factory.
Okay, nice.
So you guys are actually sort of doing, should I think of this as like, you're calling it a factory, but where my brain goes is to like Hermes or OpenClaw, one of these sort of like bots, but that's not the word you're using.
Should I, but help me understand a little bit better.
Like what do you mean when you say factory, I guess?
Yeah, so I would say that A factory is an example of the third layer around your model.
So around your model, you have a charter, and then you have a harness, and then you have the factory, or any orchestrator, really.
But in terms of the factory, we have one that's written in Python, and it is essentially a queue.
It is a daemon that runs a...
a tick every so often, pulls down information, it then pulls those jobs in, dispatches them, and also it has a dashboard UI, a web UI, that we can then interact with it, get logs from it, get information from it, and then also give it information so we can give it additional direction.
Sometimes, as an example, If an agent gets to a point where something is too ambiguous and it can't make a decision, then one of our rules is you don't make a decision on things ambiguous.
Instead, you stop and you raise an interrupt.
And then that interrupt becomes something we see on the dashboard UI.
And we handle it right there.
Kind of like a GitHub PR, except inside the factory.
A PR or an issue or something.
Yeah, yeah.
But okay.
But outside of that.
Okay, cool.
Yeah.
Nice.
Ooh, I like that.
okay, I think you're the first guest who's talked about this.
I don't want to say the first to have done it, but actually having like a factory set up where you're going.
And now this whole charter thing is starting to make more sense to me maybe.
Is it right to think that like if I'm sitting down at my workstation on something interactive, as you say, which I think I know exactly what you mean by that, like regular, I'm using Cloud Code or I'm using Pi and talking to Kimmy, it doesn't matter.
Is that state like, Is it the same charter, I guess, as it would be that if a factory agent had picked it up?
Yes.
Cool.
It's the same charter, and the factory is able to run any harness that you make it, that you build it to run.
So whatever harness is running, then that harness is going to apply its appropriate charter, whatever its configuration happens to be.
Unless you wire that up yourself and you make that a single source of truth so that way any agent can see it.
Walk me through that part.
An agent could choose the charter that is defined by which.
So if that agent, like what kinds of agents would you have in a factory?
And why might one get one piece of a charter and another get another?
So most stock harnesses will have their own definition of whatever their charter is.
It's perhaps a .clawed file or.
something for cursor or codex, et cetera.
And so that's where that starts.
And that charter is going to be used for that model, or sorry, for that harness.
And each one is going to have a different reference point.
Each one is looking at different files.
We do have some standardization.
There's an agents.md standard out there.
There's tool standards and other things that are happening.
That's all really great.
But it's not standardized yet.
So until it is standardized, then each agent kind of has its own configuration surface.
And some even have different capabilities, like Pi doesn't have subagents by default, as an example.
When you say configuration service, do you just mean a JSON endpoint that's got the config in it?
Surface.
Oh, got it, got it.
Thank you.
I'm with you now.
Okay, so maybe in the factory world...
It's less dynamic, less variable.
And maybe when it's interactive, it's more variable.
Is that right?
Got it.
Yeah.
Yeah.
So whenever it's interactive, it's more like your pair programming.
Right.
Because if you run into a guardrail, you get interactive feedback.
Right.
Whereas in a factory, it's just a process running.
It's still happening, but nobody sees it.
Well, and I might use Pi and Kimmy today, and you might be using OpenCode and, you know, an Anthropoc model tomorrow, and somebody else is using CloudCode with Opus.
And so that's why there's good variability.
And MyCloud uses CloudMD and your Pi uses, I don't know what, haven't tried Pi yet, but of course there's variability to your point.
But in our factory, I assume that like it's usually the same kind of approach that happens, whether it's a small feature or a bug, you know, that it's the same model, the same harness every time.
Is that right?
Right.
Yeah.
And it really depends on what.
it is you're using to build it.
If you're using the SDK or if you're using something like LineChain or LineGraph, if you are using the SDK, many of those will assume that it's their tool set.
So the Cloud SDK will pull in the Cloud.empty file automatically.
And so if you want to have a charter that is independent, that's agent agnostic, essentially you need to create your own charter.
and then connect it to those agents in one way or another.
You could simulate them, or you could transcribe them as part of the beginning of the program.
There's lots of things you can do, but a sim leak is probably the cheapest way to do that.
Got it.
Okay, how does this impact architecture and planning?
I wonder if some of architecture is sort of, you know, written into a charter.
So that whether it's the interactive at my desk or yours, or if it's a factory bot doing its thing that, you know, hey, we usually use Postgres and not MySQL, for instance.
Or, you know, anytime we use Python with FastAPI, we like to make sure that blah, blah, blah, blah.
Whatever the architectures are.
I assume some of that is like, you know, sort of installed somewhere.
But other parts of it will be a decision that we're making together.
How has...
all of this changed or impacted that?
Yeah.
Yeah, that's a great question.
I ran across a lot of this whenever I was working with it interactively, and the nature of it has changed a little bit since then.
But in terms of the interactivity part of it, I found myself in particular having to tell Claude over and over, use ports and adapters.
Especially because the framework that I use does not, it fights that a little bit.
And so I, I had to very specifically tell it over and over and over again.
Ports and adapters.
I eventually got tired of typing that.
It became a rule.
It worked a little bit.
Didn't work very well.
It became an all caps rule.
And it didn't work much better.
You heard it here, folks.
All caps.
So that didn't work very well.
Also.
And so what we had to do was to raise the level that we were looking at it from.
So that became a hook check.
It eventually went into our harness.
But since it was a hook check, it was able to run every time.
And we were able to kind of use that to improve over time, to improve.
kind of end over end.
So what would a hook check look like for ports and adapters?
Yes, yeah.
So that's a really good question.
Okay, so a hook for ports and adapters depends on the code base that you're using.
I happen to use Laravel in the background.
I do not like Laravel facades.
Many folks do.
That's fine.
But for ports and adapters, I don't like them because Laravel facades, you have to mock them.
Like in testing, you can call a fake method, but it's not a fake.
It's actually a mock.
I don't like mocks in testing.
And so to get around that, I decided I want to use ports and adapters.
So my hook looks for a sigil that looks like a Laravel facade.
Anything that is a...
that is a title case with a double colon, that's a potential tripwire saying, this is a facade and I don't want to use that.
Exit one.
Okay.
Okay.
So on some level, this is like a JavaScript, the good parts theory of just like, don't do, you know, if it matches.
Oh, man.
PHP 3 comes back into my brain.
Yeah.
And it's better now.
I know.
I've heard.
But yeah, I can definitely imagine all sorts of rules of, like, if you see globals, like, do not do, for instance.
Yes.
Okay.
So on some level, it's sort of a, it's, you could get as fancy as, like, static code analysis or even dynamic code analysis tools.
Yeah.
Yeah, I definitely use static.
analysis for this.
Not dynamic yet, but I've definitely used static analysis for this.
And actually, the thing that you reminded me of with this question is actually what I was able to discover because of that.
Whenever I ran into so many problems with ports and adapters, and I eventually had to settle on a hook, I discovered that as I implemented more and more of them, it was more and more adherent.
Number one, because of the deterministic hooks.
But number two was because of the imitation surface.
It was because of the code that it was looking at.
So whenever I told it, use ports and adapters, and all it saw was facades, it happily copied facades.
But it was only whenever I hit that mark where it was mostly reading ports and adapters where it kind of got with the program to say, oh, this is the pattern I'm going to now replicate.
It had enough examples.
Yeah.
Got it.
Okay, I like that.
I like that.
Cool.
Cool.
How does testing work in your world?
How is testing different than what it was before?
How does testing work?
I don't know.
I'm tempted to start with the factory.
Help me think about that.
Yeah.
So in terms of the factory, I mean, it does have its own tests, but we have, for our card work, we have a TDD loop.
So we do write tests first.
So starting at that level, The model writes the tests, not us, and it writes the test first, like TDD, but not strip TDD.
It doesn't go through that loop.
That's a waste for tokens.
But it will create a list of potential test cases and then present them to us.
And that's our feedback.
So we look at the descriptions of the tests, not the tests themselves.
And we say, are these viable?
Are you missing something?
So it's kind of like a really quick checklist, but also it is a chance for you to catch things it's not seeing, like edge cases.
Or sometimes it can be too complete and you can say, oh, this is not actually a case, but actually it might be a good idea to test that anyway for coverage.
Right, right.
And then after that, they then implement it, it goes from red to green.
proving step where post review will actually have the agent prove it.
It'll spin it up and it'll run through the whole feature end to end and it'll document that.
And so we have this prove step and that happens before we release to our staging environment.
So that way it's more trust as we go along.
And once it's in the staging environment, then we're ready to do UAT.
Oh, nice.
Okay.
I like that.
I've been doing a thing that I would call desk check, which desk check is sort of a before times metaphor of, you know, the product manager will come over to my desk as a software developer, and maybe the QA person will too.
And I'll quickly run them through it.
And at some point, I told Claude that I was going to desk check and it would let me.
But then at some point, I would say, oh, we need to do a desk check, and it would do it itself.
I'm like, oh.
And it almost sounds like this proving that you're talking about.
It's a similar kind of concept.
It's like, go, okay, you said you made the thing.
You say the test is passed.
Great.
Go use it.
Right?
Yeah.
Okay, cool.
Yeah, and the way that we tackle that usually is kind of in the way of, let's say, a testing pyramid, right?
So we have unit tests that cover things.
We have integration tests.
All of those are green.
At the end, we have this end-to-end test.
It tests the happy path so that we know, okay, the feature as it was defined and implemented, that's in place.
And then whenever we hand it over to QA, they can then focus on more exploratory testing.
What happens with this workflow or with this input we didn't think about?
And they can kind of test the boundaries of it because they can rely on the fact that the happy path is stable and tested.
Nice.
I like it.
So my brain sort of leaps up to two facts at the same time when you talk about a manual QA going exploratory.
And one is like, Kyle, this is insurance.
So there's all sorts of like corner cases where like if it goes wrong, like there's money involved and there's probably regulatory.
There's probably like all sorts of good reason to have like somewhat expert manual exploratory.
And another part of my brain leaps to like, oh, my God, that's regression to the old manual stuff.
But I assume it's a good and valuable thing that you guys are doing.
It sounds like it would be more expensive than the proving step, but still probably catches things.
But I guess like, yeah, talk.
But yeah, help me out with that part.
Yeah, it definitely is more expensive than that step.
And that's part of the reason why that step exists is to be able to lower the cost of that.
And we could automate all of that.
That's definitely true.
The reason why we don't automate that is because we are currently building a new product.
And so a lot of that is exploratory in nature.
And so we meaningfully or we specifically separate that so that way we can.
try to explore and see what the UI UX should be and kind of get some questions answered and generate some fast feedback.
That's kind of the story of our factory is every single time there's a possible feedback loop, then we are going to choose the cheapest, fastest feedback loop that there is.
That's nice.
Okay.
Cool.
Cool.
I like it.
Okay.
We've got another.
OK, so anything different about the way that you're seeing infrastructure work, deployment, CICD or literally like, you know, like how we design and decide on what, you know, what sort of infra are we going to deploy this thing to?
What changes about all of that and with AI versus before times?
Yeah, I think.
I think that there's definitely going to be a lot of things happening in the infrastructure as code space.
I mean, that has already happened by and large, but if you're not on infrastructure as code, I think that, you know, you should be heading that way.
And AI is definitely going to help out there.
Lots of possible things that are capable there.
I have not done too much on that side of it.
I've done more on the other side of it, which is CI, CD integration and that sort of thing.
And there's definitely effective things.
I would say that CI, and I should be careful whenever I say CD, continuous delivery is vital for deploying well with agents in a reliable way.
Getting that fast feedback is so important.
And I would point folks to modern software engineering by Dave Farley.
And in it, he talks about just engineering discipline in general, how it applies to software, what we've discovered over time with working with software.
And one of the things that he talks about is feedback loops and having fast feedback loops.
And so a lot of our processes are really tuned around having that feedback.
So that way we can then move as fast as possible, remove as many bottlenecks as possible.
So we have a feedback loop when our tests are read.
We have a feedback loop when our E2E tests fail.
We have a feedback loop when our CI fails.
We have a feedback loop whenever our staging tests fail.
So that way, anytime there's a problem, we'll get that feedback as fast as possible.
And that really helps us to be able to ship quickly.
We still use PRs, but we use PRs just to deploy features.
We don't have long-lived PR branches.
Our PRs will live for a matter of hours, not days or weeks.
It is pretty close to trunk-based.
Sounds like it.
The only real difference is that we're not committing straight to main.
But the reason why we have that is because, again, we're doing a lot of exploratory work.
So we want to be able to keep those changes isolated while we answer questions before we subree to push them into main.
We could do that because we do have pre-commit hooks and things like that.
But that's just not the decision that we have at this point.
And then in terms of the other parts of this.
Feature flags comes to mind as well.
Yeah.
Go ahead, Doug.
Yeah.
So I think the other part of this is that's really important is all the other things that come along with that continuous delivery.
So like CI, having tests in place, having a good test pyramid, having feature flags in place.
All of those things are really important because then you'll be able to.
really take advantage of the speed of your agents because you'll be able to deploy and you'll be able to deploy in a way that is reliable.
You can make it reversible.
You can roll it back.
You can turn the feature flag off and it's not the end of the world.
And that's really what's needed since the agent is so non-deterministic.
Got it.
Got it.
Good answer.
Okay.
What are you seeing change in debugging, observability, and not just observability itself, but responding to alerts, monitoring, those kinds of things?
What do you see changing there?
Yeah, so I think that I have not seen much change in terms of observability for paging.
And that is something that I think is actually probably a good thing.
it might be in our future to be able to hook up agents to pages and to be able to automatically pass things.
That sounds nice on one hand, but on the other hand, I feel like it's a little bit of a disaster waiting to happen because I feel like a human needs to be in the loop there.
Not only to understand what's happening, because the model might not, but also to be able to...
make the kind of decisions that are necessary for human judgment, things that might have to have knowledge about the company or knowledge about HIPAA or some other kind of policy.
So I don't see that really happening very soon.
And so until that happens, I'm going to assume that people are going to be responding to pages.
And so it's for that reason that the quality of our code still matters.
the testing of our code and the understandability of our code.
Like we still need to understand it as humans because we're the ones answering the page.
And in the same vein, I treat that as every step along the process.
I don't ever assign the agent anything.
I assign the whole thing to myself because I'm the one who owns it.
What matters with the change getting out is who the owner is, not necessarily the author.
So if you go into JIRA, assigned to is not set to factory.
It's set to Kyle or Ian.
And one of us is shepherding it through that process.
Correct.
And we still learn the change.
Okay.
I hear you.
And I like where you're going in terms of like, you build it, you run it, you own it.
Like, we better answer the pages.
We better understand it.
I'm with all of them.
Also, I will say that anytime one of my alerts does go off, the first thing I do is tell plot.
I open a new session, so I get clean context.
And I'll be like, I just got this alert, paste, and then I'll go try to fix it on my own.
But then I'll check back with it every couple of minutes.
And it's often beating me to the punch in terms of figuring out what happened.
And I make it ask me before it can just fix it.
It is still guided.
It is assigned to me.
It is not assigned to plot.
But it is usually able to figure out what went wrong faster than I am.
So maybe that just means I'm getting old.
I don't know.
Do you guys use it in that way in terms of like assisting you with the response to the page?
Yeah, usually it is.
It's with assisting a response to the page.
So we don't have it just run dark instead.
We'll supervise it.
But it's not that you're not using the tools.
It's that it's not going through like dark.
Okay, it's just not a factory approach.
Got it.
Right.
Got it.
Yeah.
So in those cases, we kind of treat that as an emergency that needs to be handled by a human.
And so a human needs to be able to understand what's going on and, yes, fix it as fast as possible using the agent, but also understand why.
Yeah, yeah.
And what happened.
Yeah, yeah, yeah.
I'm with you.
Love it.
Okay.
So did your organization or any organization that you've worked with before, did they do much in terms of ensemble?
So pair programming, mob coding, you know, And do you find that you do it more or less now?
Well, let's first start with, did you do it before?
I did do a lot of pair programming before.
Not as much mod programming.
It just wasn't a dynamic in the teams that I was on.
But I do really enjoy pair programming.
I think that it is very highly underrated.
you have a strong pair programming culture, that's kind of the gateway to trunk-based delivery, I think.
Agree, agree.
And I think it also has the ability to remove the need or the contrived need of code review.
Because everybody's talking about code review and about how there's all this information coming in.
We control that by controlling the size of our PRs.
That makes it a little bit more manageable, but it is still a lot.
Whereas whenever you are pair programming, you can get that sign-off immediately because that's really what you need is to have a second pair of eyes.
And so if you're using trunk-based delivery, then...
The industry commits to main.
As long as you have a pre-commit hook, that's all you need.
Yeah.
So I definitely think that that's kind of the way that we're headed.
I almost feel like I've asked a survey question.
What do I mean?
So having worked at eight years for a consultancy that deeply believed in pair programming, I came to love it and I came to see the benefits.
But I also had enough time trying to explain to everybody else that they should do it to realize that most people are not going to, right?
And it is not very common, in fact.
And maybe, you know, when you work for the company that gets hired to teach people to do it, you only see people that don't.
So maybe that colors my experience.
But, you know, my best guess in startups after ThoughtWorks and in my time there is that most people aren't.
You know, and maybe they're doing a couple of hours a day.
They're certainly not doing like six hours a day of programming, pair programming.
And so it sounds like you pushed personally for some amount of it.
But I think some people who did do it all of the time are now struggling with, are we both going to watch three clods together?
Or are we going to watch one clod and I'm going to be on two other clods I'm not telling you about?
Or like, what are we doing in our pair?
And so...
Yeah, and I think it's a valid question.
Some people say, like, no, no, no, it still matters.
We've got to be able to teach each other the tricks of how to use Cloud.
And other people say, like, no, it's a waste of time.
Like, it just, I mean, I love it, but no, we're not those days anymore.
So I wonder if you're having any of those kinds of thoughts about how programming feels post-AI.
Yeah, I definitely think that there are aspects of it that...
It's not as needed.
In some respects, the machine can kind of be your pair, but also it's an asymmetry because you're always the one navigating and it's always the one driving.
And part of the value of pair programming is switching those roles.
I do think that like in general, pair programming is so valuable and we should still be doing it.
I think that it can really enable an organization to get around things like code review, if that's a blocker.
And it'll also disseminate a lot of knowledge.
The team knowing more is always going to be a good thing, even if that includes the way that they work on their harness.
Because if somebody notices something, then that's something that can be promoted to the charter.
And now all of a sudden, you don't have to think about working that way.
It just automatically does.
And I think that pair programming right now is vitally important for junior developers.
I think the junior developers are in a pretty tough spot at the moment.
And one of the things that I think can really go a long way for getting a junior from that level to senior in a way that is comprehension-based is pair programming with the senior, a junior, and Claude.
Because then...
the junior can learn some of those tells that seniors pay attention to.
And I think that that's a really great exercise in order to get them to be able to move up this ladder to seniority.
If we don't have some kind of concerted effort toward this, we are going to have...
less people enter this industry and it's going to become a problem not immediately but in five or six years it will be a large problem so i think that supporting juniors is a very important thing right now I'm with you.
I think there will be perhaps, perhaps it's cynical, but I think some people will hear you say, we'll have fewer people in this industry.
And they'll say, yes, that's right.
We, I don't know that we had too many before, but we had so many before.
But I think, I think there's another risk on top of that, that they might be messing around.
Well, yeah, but.
Like if it's so many fewer, the pendulum has swung too far the wrong way.
So no, I'm with you.
I think that the whole, like, what are we doing with juniors is not yet a thing that we've got our hands on around yet.
Okay.
Well, okay.
But for, and so maybe this color is the next one.
How do you hire staff, train, mentor?
How do you bring new people on board?
Have you had to do that since you've been on a team that was using AI a lot?
We have not had to do that.
We have had to onboard a person onto the team, but not a new hire.
So the onboarding process, we do have that.
And we also have like kind of a framework for our onboarding or hiring in general.
But in terms of the onboarding that we did for the team, the way that we tackle that is over the series of three days.
Because our charter is very complete and has lots of information in it, it is basically a tutor.
And so day one for the new person is just look at the code, ask cloud questions.
How does this thing work as a whole?
Like, what's the important bits?
How do they fit together at a high level?
Day two, we give them a module.
that was going to be the place where they're going to work first.
And day two is going to be understand that module, dig into that module and ask Claude, how does this module work?
What are the important components?
What are the dependencies?
What do I need to know about?
Day three, we have a PR, a real code change for that area of the code.
And we pair during that PR.
Got it.
And during that time, we talk about norms and standards and how to...
interrogate the model and the charter and so forth.
And then after that, they end their third day with a merged PR.
Got it.
Got it.
And they've gone through sort of Sim to Stern and they've used the tools and they've seen how it works.
Nice.
I like that.
I am pretty sure you're one of the first guests who's talked about the whole factory approach.
So let me ask, like, who started it?
Like, whose idea was it at your organization?
Yeah, so I started it.
And the reason why I did was I had heard about software factories, about dark factories.
At the time, I was doing a lot of stuff with harness engineering.
And it was something that I heard about, wanted to learn more about.
And it also aligned with what I was trying to do with AI, because I was trying to really deliver ROI.
I didn't really care about PRs merged.
I cared about results.
And so for me, like...
Actually, one of the metrics that I had introduced to me that I use now is lines of merged code or cost divided by lines of merged code to production.
Just to wrap that around it.
So with the factory, I started off by vibing it just to get a shape of the problem.
And then I tossed that away.
That's very common for me.
I never carry over vibed projects.
And then I started it from an engineering standpoint and decided I'm going to use Python and UV for this.
And after working with it for a while, I initially went down the Lang chain Lang graph path.
And I liked a lot of that.
We also had some restrictions.
At the time, we did not have an API account, so we only had our Cloud SDK.
We only had our subscriptions.
So instead of, that kind of was a blocker.
You need to have an API.
This stuff is all above and beyond what you're paying for your subscription on some level.
Exactly.
Yeah, so rather than go through that, instead of being just swapped over to use the Cloud SDK.
And then with the Cloud SDK, then we were able to, essentially build a factory in a way that was a distributed queue.
So we set it up basically like a queue.
It has a dead letter queue.
It has retry loops, things like that.
Each job is its own phase along the cycle.
And each job will dispatch an agent to do its work.
There is no agent in the factory.
There's no inferential or there's no inference in the factory.
That's all deterministic, good old distributed systems, standard cues.
Got it.
And so the nice part about that is that you know that everything is going to be working in a deterministic way.
You've encapsulated the non-determinism.
The other thing that's nice is that because you don't have an orchestrator agent, you actually get out of some of the limitations that Fred Brooks talks about with parallel work and actors and things like that.
All of a sudden, parallelized work can be done in a faster way as long as there's something persisted to disk.
And that's the trick.
So whenever you persist something to disk, then that's what keeps the factory going.
There is an input and an output criteria for every step.
The input criteria must be met for that step to trigger, and the output criteria must be met for it to complete.
So a card can never start execution before it's been properly defined.
A card can never enter review before it's been executed with a zero exit code.
I like it.
I like it.
Okay, so a qualitative...
Would you do it again?
And if so, would you do it the same way?
I would say yes, I would definitely do it again.
I would tackle it a little bit of a different way.
And I've actually been doing a little bit of experimentation with this recently.
And that's using 3i.
That's the tool I've been using recently.
And it's a harness.
It's a distributed system.
There's a lot of ways you can look at it.
And so the way that I've...
So I have basically been porting our factory over to 3i instead.
And that's been interesting because now instead of maintaining a D-min and a Q and thinking about ticks and how often I have to check and like all of this concern I had to deal with, all of a sudden that's handled for me because 3i does the heavy lifting on the distributed systems part of it.
And so I just kind of like...
spin it up, add a component, and then write some code.
It's very Docker-esque.
Awesome.
Cool.
You've been very generous with your time, and I can see that we're getting to a certain mark on the clock here.
So we got to talk about the book, so that is awesome.
Maybe do you have anything to say about where can people find it?
When does it come out?
Those kinds of things.
Yeah, sure.
So it's available right now and currently it's on LeanPub.
So if you go to LeanPub and search for Harness Engineering, it's one of many books that pops up, lots of books about Harness Engineering right now.
So, yeah, it's called Harness Engineering.
By Ian Johnson, I assume.
Yes.
And I guess one thing that I can kind of leave everyone with is...
I guess, around some of the terminology that you had mentioned.
So you had talked about, you know, harness.
That's one that everybody uses a lot.
I introduced the word charter.
There's a lot of terminology floating around.
And in particular with harness, there's a lot of semantic diffusion happening.
And that's actually something that I had previously gotten wrong while I was working through this.
So I think that in terms of...
the many names that are coming around.
They are kind of all pointing out different patterns and they are kind of talking about specific things.
But I think that it's important to kind of keep in mind like the holistic piece of it.
No matter what you're working on around the model, what you're really doing is you're constraining it.
Whether you are talking about context or whether you're doing the harness or whether you're doing it for factory, you're constraining it.
So the way I like to think of it is that In general, what we're doing is constraint engineering.
And each of these different things like loop engineering or graph engineering, these are all methodologies and sub-disciplines of constraint engineering.
I like it.
I wonder if you already just answered this, but what was something that you had to change your mind about or that you were wrong about or unintuitive about at first?
Yeah, yeah.
So I'll...
Use that example.
Whenever I first learned about harness engineering, I got into it from reading an article from Birgitta Butler.
Yeah.
And as many people did, I'm sure.
Whenever I read it, I thought I understood it.
And I did not.
I took her meaning to mean the extensions of the harness.
That the harness was repo resident.
That the things that I could adjust were things like tools and MCP servers and skills and so forth.
And then later on, as I was working through this, really trying to discover what other people were talking about when they meant harnesses, I kind of got to this point where I realized, oh, the harness is actually its own piece of software.
It has its own tools and its own layer like that.
But that's what made me see the symmetry, because the same patterns that happened in the charter were happening in the Harners.
It's just that they graduated from being inferential to being deterministic.
Oh, interesting.
I like that.
Okay, cool.
Nicely done.
What's something you're most proud of?
I'd say probably the thing I'm most proud of is my family and...
The way that we've been traveling, we haven't traveled for a little bit.
We're back in the States, but we traveled for a year over in Europe and we had a really great time.
And it was just a really great experience.
Sort of a working remote situation?
Yeah, we were working remote and we were just kind of bouncing around Western Europe.
We actually hit three different continents until we ended up back in the United States.
Wow.
Okay, so where?
Tell me a few, or at least one that stood out and that you really loved.
Yeah, so most of what we did was in Western Europe.
France, Germany, Switzerland, the Netherlands, Belgium, England.
Wow.
We went to Morocco for a little bit.
Okay.
And yeah, I think that was essentially what we did in Europe.
Plus Africa.
Yeah.
Yeah.
A bit of North Africa in there too.
Yeah.
So even for a year, which is a long time, that is a lot of places.
So nicely done.
Cool.
Very cool.
And one more, what's something that's giving you joy right now?
I think right now, the thing that's given me joy is probably my sons.
They are at the age where they're just starting to talk.
So it's.
a special time because they're both, it's happened for both of them at the same time.
And so that's probably, you know, one of the, one of the nicest things I've got in my life.
Yeah.
That's awesome.
That's awesome.
When, when guests are on and we can hear their dogs barking, I asked them to send me pictures, but I will protect your, your, your, your family's privacy and I won't ask for that, but that's, that's, that's, that's really cool.
Nice answer.
Thanks, Ian.
Yeah.
Thank you.
Thank you so much for being on the podcast today.
I really appreciate it.
Yeah, thanks for having me.
Great to be here.