# Strategic AI Integration in Software Delivery

**Podcast:** Engineering with AI
**Published:** 2026-07-27

## Transcript

had a machine with 48 gigabytes of memory and one with 64 gigabytes of memory.
One was an M3, one was an M5.
I had like an automated setup to run a task multiple times.
And I had one task that got better quality of output on the M5 machine with more RAM, even though everything else was the same.
Like I would have expected faster responses and all of that.
But why was the quality different?
Like it just solved the task a lot more often.
on the bigger machine than on the smaller.
So that's something that I'm still like very confused about.
So maybe it's because it's a mixture of experts architecture and the bigger machine was loading more experts or something like that.
But there's all of these little idiosyncrasies and things that happen and you're like, what, why?
And so that's really annoying, you know, when you actually have to get your work done.
And you don't really know what to expect, right?
Birgitta Bauchler is a distinguished engineer at ThoughtWorks, where she's been the resident subject matter expert for using AI to improve software delivery for the last three of her nearly 14-year tenure at ThoughtWorks.
Multiple guests of our podcast have quoted her articles recently on harness engineering, maintainability sensors, and most recently, spectra and development.
I've been loving her interviews and recorded conversations on the ThoughtWorks Technology podcast.
One of my favorite Birgitta quotes is, we didn't have enough words for this, and so I went and made some.
And I think it's been incredibly, incredibly useful, especially on the spectra of inside.
We can all sort of intuit that there's so many gradations that this could go, but starting to categorize some of the different points along that dimension.
have been really, really useful for me.
Bjergica, thank you so much for coming on the podcast today.
Yeah, thanks for having me, Kyle.
Let's start with you and your experience.
Yeah, so I've been in technology for over 20 years at this point and basically the whole time in consulting, but not as a PowerPoint consultant, as like a hands-on developer.
So I started as a Java developer a long time ago.
Since then, you know, gone through.
all kinds of different tech stacks, right?
Which interestingly is becoming less relevant, right?
Like you're being asked less, like what is your home stack, right?
So I've always been quite a generalist and, you know, as a consultant, like every two years or so gone to different environments.
So I would consider myself kind of like a domain expert in effective software.
delivery, which is also like a core topic and competency of ThoughtWorks, where I've been for almost 14 years now, right?
And so exactly three years ago, we were recording this in July, right?
2023, ThoughtWorks decided to introduce a full-time global role for a practitioner to just look into this topic.
So this was about half a year, eight months after ChatGPT.
And I remember at the time, we already felt like we were behind, actually not doing enough.
And now in hindsight, when I tell people that I got this role three years ago, everybody's like, oh, wow, that's a long time ago.
So yeah, it's interesting to look back on that.
Yeah.
And I had the experience, like I was the global community of practice lead for our developers.
at ThoughtWorks for a while, which was also all about creating a shared understanding of practices and how we develop software.
So I think of using AI for software delivery as applying AI to the domain of effective software delivery, right?
So it felt kind of like a really good intersection of my skills at the time.
So in the role, I've been experimenting a lot myself, hands-on, sometimes myself, sometimes with some people on a small team.
I've continuously talked to a lot of teams at ThoughtWorks, to clients, both executive teams and directly to the engineers at clients.
I've spoken at conferences.
writing, all of those things.
Yeah, it's been a really interesting role in a time like this.
Yeah, yeah.
To have this extra time and but then also experience how with everything that's going on, there's never enough time to stay on top of everything, right?
Yeah.
Yeah.
No, that's amazing.
What an opportunity to be inside of a global consultancy that really loves thinking about software delivery.
Six months after GPT, I can imagine feeling very behind already because there it was.
It just sort of suddenly appeared.
Right.
But at the same time, like we all know now, like how much has come after it.
So, yeah, that's interesting.
I mean, I think there were quite a few people who already saw it coming.
I was not one of those people.
Because, you know, I'm like very much like more like a business enterprise application developer type person.
We had a few things on the ThoughtWorks technology radar that were kind of like precursors to this, right?
Like BERT and stuff like that.
And so certainly there were other people in the company that saw this coming a lot earlier.
And I mean, Copilot existed before ChatGPT created all of that attention, right?
And then there was...
What was it called?
They were one of the pioneers, started with a T.
Anyway, it'll come to me later, but it shows how some of the pioneers of this actually have also been falling behind over time, right?
Yeah, that sort of first mover disadvantage, like whatever that gets called.
Sometimes our pioneers don't cross the chasm.
Let's start talking about how you use these tools.
You mentioned in a recent post that you vibe-coded a tool.
What are the tools you reach for when you have a choice right now?
Yeah.
To set some more context on how I'm using it, I'm hands-on, yes, but I mean, there's of course the caveat that I haven't actually been on a real team for three years.
I've basically been in the second row, right?
Talking to teams.
And so I've always felt this challenge that especially when you see on LinkedIn and the public discourse in general, like lots of like more experienced technologists, some of them may be more post-technical than others, just coming back to coding, right?
Writing about all of their experiences of using these tools.
And it's always a little bit in a vacuum, right?
Because there's like all of these things that happen when multiple people work on software together, when the software that you build is not a prototype or a POC, but it actually matters.
It has like millions of users, right?
So I've always been like very...
conscious of that and always trying to put that lens on everything I'm doing.
Like, oh, if this was real software, would I push this now?
Or, you know, because it's been my job to kind of explore what this actually means for teams, right?
So just to put that into context, right?
Like I've always done my best, but of course, I'm also like mostly working on code bases that don't matter that much because I have not been on an actual team.
So that's more like second hand, right?
And then the other thing that's maybe a bit...
more unusual about my setup is it is part of my job to compare a lot of different tools.
So I jump around a lot, right?
Which then is also not like a typical experience sometimes, right?
So I have like four or five, six different coding agents or harnesses installed on my machine.
And it's always like a mess when I try different skills and then they're not reachable in one of them and in the other, you know, so it's like a bit of a mess, right?
But if I had to say like, When I'm not in that experimentation mode or something, what I reach for is I'm usually in the anthropic world a lot.
So when I'm trying to explore things or learn things or just discuss things like some thinking, stuff like that, I use Claude, the web version.
And then, yeah, so the three coding harnesses that I jump around with between the most is Claude Code, OpenCode and Pi.
recently as well.
So Pi is this like very lightweight, super small, malleable, as they say, coding agent.
And that's been really nice for using smaller models on my machine recently because it has like a much smaller footprint.
And so, yeah, and then I use it for research as well, of course, right?
Research, I have like as many people have these days, an obsidian vault where I have all my notes and all my content and try to like cross-reference stuff.
And it's almost like a...
lift off your shoulders almost with this knowledge management stuff where in the past I would like think about oh what is the structure I have to set it up in the beginning because it's so hard to change it later and now with something like AI you know you can be like so much more loose about how you how you order your thoughts and your content because AI makes it so much easier to rearrange things right so that has been great for content, for ideas, for writing.
I use it to polish my language sometimes, you know.
People say my English is pretty good, but I'm still a non-native speaker, right?
As a German, I tend to write very long sentences.
So that's been useful.
Those are like the core things I use, I would say.
So much catnip there.
Like I want to dive into small models.
I want to dive into Pi.
Recently on the podcast, we did an article at TechRadar.
I had interviewed now 14 people and went back and essentially put together assess, hold, trial, adopt style ratings based on what they'd said.
And of course, using LLM tools in order to do it, because it would have been a lot to listen to 14 hours worth of audio and take good notes.
People on LinkedIn posts were on promoting this engineering with AI radar.
People are talking about Pi.
People really seem to love it.
Also small models, but we'll get to that.
We'll get to that.
Also on the anthropic side.
So I realize I have not gotten to talk to a developer yet about Fable versus Opus.
Have you gotten to play with it much?
What's your thought?
I have actually not used it a single time yet.
You're in for a treat.
I had other stuff to do, you know, and I was like, it's fine.
I get it.
I get it.
I get it.
Somebody was saying that like, oh, it's proactive.
They have a benchmark.
This is climate tech business that uses these things to do engineering for grids.
And so they have sort of an eval that runs the same prompts through every new model.
And it was you're an electrical engineer and you're faced with such and such good problem.
What do you do?
And he said Fable was much more proactive about not just the way that it solved the problem presented, but, you know, sort of chasing corner cases, the right corner cases.
And that, yes, that that aligns with my with my viewpoint so far, too.
Yeah, it's interesting.
Like, I've also like asked myself, like, why is this not catnip for me?
Like, you get the fable and finally it was available again with catnip for so many people who are deep in the space.
Right.
And it probably should be for me.
But somehow it's for me, like, I'm sure it's like.
one step up again at being like a really good software developer.
I'm also sure it still does some of the same stupid mistakes every now and then.
It might just be like less frequent or something, right?
And so for me, it's always like, yeah, I'm sure the models are going to get better, but there will always be this kind of like floor, glass ceiling, I don't know what to call it.
Like, it will never be 100%, right?
So it's still like...
like I'm still much more interested in like what we build around it and then also like we might not always want to use these like you know in Germany say shooting with cannons at sparrows I think you have a similar thing in English right I like it so um so yeah I mean I believe it it's probably amazing but you know Yeah, yeah.
There's like all of these other things that we'll still have to do in spite of models getting better.
Yeah, for sure.
Well, and like you say, I mean, your current workflow, your job is to try them all out.
So like one more is like there's another job on your list.
I can relate to some of what you said about coding right now.
I'm solo in most of the software development that I do.
In my day job at Navalny, I serve teams, but I don't get to.
put any code in production, right?
But I do have like a side startup where I am playing with these kinds of things.
And so when Fable did switch on, yes, I was using it for that business.
Yeah, okay, fine.
Yet another model.
I'm sure it'll be great.
And I'll leave you with that thought.
Okay, cool.
Do you find that you've got preferences in the kinds of tools that you use?
Like you talked about how Pi is more pliable maybe or smaller.
And, you know, of course.
Yeah, you've tried so many at this point.
Like, do you, and you reference Copilot and that sort of original autocomplete style, you know, versus now the agentic.
Like, what preferences do you have when you're trying different tools out?
What do you like?
Yeah, so, and with the, I mean, I should say, I also was like a huge, always been a huge fan of Cursor.
I've not been using it recently because for like several reasons, we don't have the licenses anymore at the moment in ThoughtWorks, but.
also been great.
I'm also not somebody who's like gung-ho on like, oh, I'll only use the terminal now.
I still have VS Code open all the time.
And I like, I'm also like a very visual person.
So I like just like diffs, like I want to have all the markdowns I have written.
I want to have them rendered nicely.
I have AI generate a lot of mermaid diagrams into my markdown files and I want to see them.
Right.
And, you know.
I mean, in some products, like I'm also using warp on and off, which is this, you know, terminal that can also like open files and, you know, can render mark down there and all of that.
Right.
But I just need this like more visual environment.
I don't know.
So I switched back and forth between the terminal and VS Code a lot in general.
Nice.
And then in terms of like, when do I use which?
So Pi in particular, I've.
been using a lot, one, to play around with the malleability with building extensions and stuff like that.
So I had a use case for that where I wanted to try.
So how do I say this without like going into a whole tangent?
I basically had something.
I wanted the agent to regularly check the status of something.
And I wanted to see if it does that more reliable, if it's accessible as a custom extension, as a custom tool.
than when it's just written in a skill or in the agent MD or something.
And that's what I tried with OpenCode and with Py.
Py is also really easy to use in a more programmatic way.
So when I was trying out those small local models, I was also at some point creating a setup just to quickly have a task run multiple times with a model and see how reliable it was.
And then it's just really nice to have a harness that you can use easily in a programmatic way.
So that was good with Py as well.
And OpenCode in general also because like...
like many open source harnesses, it's a lot easier to switch between different models in those.
Whereas in Cloud Code, it's just like more locked into, by default, more locked into the anthropic models.
And it's just also become very heavyweight, right?
In terms of all the tools that are baked in.
And I mean, lots of functionality is good in one sense, but in the other sense, also it's sometimes a bit much.
And I just sometimes try to go back to basics and feel like I have more control and there's less going on, you know?
Yeah, yeah.
No, I get that.
Okay, cool.
Cool.
Well, you've touched on a few things that could answer this question.
But next up is guardrails.
What are your favorite guardrails for making the tools effective?
And this is a strange question to ask you because you've written like multiple articles that multiple guests have quoted about this exact subject in terms of sensors and everything else.
But yeah, what are the guardrails that come to mind for you?
Yeah, I mean, first of all, I don't like the word guardrails much.
I don't know.
I mean, the guardrail is kind of like the thing, what's the thing next to the highway that your car doesn't get off of it or something, right?
But like in agents or with the LLMs, it's always something like sometimes you hit them, sometimes they suddenly open up and you go right off.
So it always feels like guardrails is not quite the right word because like a lot of things that people call guardrails.
then it sounds as if it's like reliable, right?
But it's not like a lot of the things that we call guard rates are not reliable, right?
So it's always about like, I was saying recently, like when people ask, but how do you make sure that it doesn't do this, it does that?
And it's like in 80% of the cases in the space, what we actually have to ask is, how do we increase the probability that?
Yeah.
Right.
And there are some things that we really want to make sure because they're really important.
They're really like security related, whatever it is, or it has to do with our domain, something that we really want to make sure.
And then we have to see, like, how do we really make sure?
Yeah.
But in all the other cases, it is about like, how do we increase the probability that?
And I think it's actually important that we use that language because otherwise people will get a false sense of security.
Right.
So we have to like fully embrace this, like thinking and probabilities and what is good enough.
Right.
Not just like making sure, but like what is good enough for us.
We have to have that conversation, that definition in every team, in every organization.
Right.
And then, but yeah, in terms of like my, yeah, God, what's then the word, right?
So, I mean, the term that people have started using is harness engineering, right?
And I've also jumped on that bandwagon and wrote an article about it.
I actually don't like it that much as a term.
Ultimately, I think it's just context engineering for coding agents, right?
And the actual thing, the harness, so like cloud code.
Well, the core harness is like cloud code, pie, open code, and so on, right?
And then what we're doing around that is just we're expanding the harness.
And part of that is like trying to like rein in the agent in some way, right?
And so, yeah, maybe this like harness reining, right?
So you have the reins and with the reins you steer.
And, you know, you can also let them lose.
So maybe that's the metaphor.
I don't know.
At some point it breaks down.
Right.
But so and as part of that, yeah, I wrote about that and just kind of like the mental model that I have in my head for it, because it seems like right now a lot of people focus a lot on just writing all of these Markdown files.
Right.
Please do this.
Always do that.
Never do that.
Put it in all caps.
Ten exclamation points.
Yeah.
So and all of those things.
And that's fine.
Right.
But that's basically everything we do feed.
forward, right?
So we anticipate what we want the agent to do and what we've seen it not do right.
So we try to anticipate that and steer it away from it.
And so we feed forward.
But then what we also need to do is give it much more feedback, right?
Because it's like, there will always be a non-zero probability that in that first run, it doesn't generate exactly the best code.
So then we want to give it feedback.
Ideally, automatically first before we even look at it ourselves have like all these different computational tools cpu-based tools that we have just give the first feedback right linter is the is like a really nice simple example right or we can have like a bunch of custom scripts that look for certain patterns in the code that we don't want or stuff like that right yeah and so that's part of that god rail space i think right tests in a way are also guardrails, although that's also kind of like a, you know, like a watermelon, kind of like a fake type of thing a little bit, because the tests are often also generated by AI, right?
So just because the test suite is green doesn't mean that the tests are actually testing, right?
So, you know, that's one of the hardest areas, I think, the functional correctness, like how do I really know that it does what I want it to be?
without like falling back into manual testing world from 10 years ago or so, right?
Yeah, yeah.
I like how you followed harness to reins, to steering, to, and it also sort of infers that there's this other intelligence that you're steering, right?
Like, I mean...
Horses are not quite as smart as people, but still, you know, there's a teamwork to it.
So, yeah, I like that.
Yeah, or I always like early on, I was writing about the agent as like, you know, Kent Beck calls it the genie, right?
Early on, I had this like stubborn donkey, stubborn and eager donkey.
Yes.
It's really, really well read, but yeah.
Yes.
And I mean, and to go back to the steering, like something.
That I found also really interesting when I wrote up about the feedforward feedback is the analogy to cybernetics and control theory and stuff like that, which I actually didn't know that much about as part of this.
Like I researched more about it and got more into it because my colleague Keith Morris.
pointed out what is apparently an old adage that ultimately will always reinvent cybernetics or something like that.
It's like something somebody said at some point.
And I was like, oh, I wonder what that means.
And then got me in the rabbit hole.
And so, and that's basically the idea that we're regulating a system, right?
And that also really resonated with me.
The system that is being created by the coding agent is something that we want to regulate, right?
So we want to think about.
What does the fitness look like?
Like fitness functions and stuff like that, right?
We want to measure what it's doing and then this is maybe too much and then we like pull it back again.
So that was also an analogy that really resonated with me.
Yeah, that's something I've been thinking about.
Fitness functions definitely to me seem like they could.
be something that we're going to be doing a lot more of now because we are not as involved with the coding, right?
Like at the time, I think it was Neil and Rebecca first talking about this function the first time I heard about it.
And I thought to myself like, oh, this is fine.
But, you know, like my teams aren't really making these mistakes.
Like maybe very large teams would.
But that's not something you can take for granted.
That's not a given at all.
Yeah.
And also like thinking about them, one of the ideas is that it forces you to be.
explicit about it, right?
You have to actually think about what does actually fitness mean for our system?
Not every system needs to be highly scalable or like have a latency of like, I don't know, like a super low latency, right?
And that is something I think that lots of teams have not really done very well in the past or like not been because it's kind of, you have a feeling for it as a team because you're also highly aligned, right?
And you have the context.
But now with the agents, actually, like if you want to build If you want to build these sensors and guides and skills and all of those things, you actually have to know what good looks like, what good enough looks like in your system.
And you have to be explicit about it.
It's almost a prerequisite to build a good expanded harness.
Yeah.
You also mentioned writing small tools.
didn't quite call them harnesses, but in order to automate sending prompts, and you talked about your work with sort of local models or small models.
Do you want to, what were you doing there?
That sounds interesting.
So, I mean, obviously with the recent explosion in token costs, it's been like everybody has been talking about like, How do you manage that?
What are alternatives to the big vendors that are maybe more expensive?
And one of those alternatives that people would really love if it works, right?
Everybody just wants it to work is running a model on your developer machine, right?
It's not the only one.
And I think it's definitely not the most realistic.
One, because the models take a lot of memory and you have lots of other stuff usually running on your machine as well while you're doing all of this that also take memory, right?
Another alternative is, of course, like self-hosting open weights models or like turning to other inference providers that maybe with the sovereignty concerns could also be like in your own jurisdiction and so on and so on, right?
Running small models locally, or as people colloquially say, local models, you know, is one of those things that lots of people are interested in now.
And I hadn't done it in a while.
There's always been a lot of noise on LinkedIn and so on.
It works now.
And I'm always wondering, what is it?
Right.
Like what did this person do with it?
And sometimes when you actually ask, it was like a super small little thing, but it wasn't actually a genti coding.
Right.
So I was like, oh, no, I want to see how far has this come for a genti coding.
Also, there's lots of benchmarks, of course, of models.
Right.
I actually didn't want to repeat any benchmarks and automatically compare things.
I wanted to see what it feels like.
Right.
Yeah.
And how plug and play it is for maybe also somebody who doesn't want to spend a week understanding the ins and outs of all the parameters and the set.
settings and the configuration and try out five different run times and stuff like that.
Right.
So, yeah, I, uh, I used one of the like more, uh, user friendly run times, which is, um, LM studio.
I think, uh, there's another one that people mentioned a lot.
That's O L M X O M L X.
Okay.
I always mix up the letters.
I don't know, but like Adam studio is one of the ones that is very user friendly, I would say.
So I use that and I tried a few of the models that.
had gotten like a lot of noise at that, at that moment.
And in particular, QEN 3.6, was it 30 billion or 35 billion parameters, I think.
And that also in my testing was the one that now when I have like smaller, very like specific directed tasks where I feel like a small model could handle that, I usually reach for this model at the moment.
It's my, it's my default.
And yeah, I also try to write up a bit like, I created like a slightly over the top diagram that looks a bit like a conspiracy theory board or something.
Just like listing all the different factors that affect this, right?
Because just to show people like what a messy space is this is right now, right?
Like all of these little screws when you like might actually affect the outcomes.
And also to highlight that this is not just about speed of response, right?
Right.
So the speed is actually...
pretty decent and like amazing when you look back, right?
Okay.
Of what it was, right?
So this like, you can actually run this, you get pretty decent response times, right?
Interestingly, like a lot of reactions that I get to the write-up that I did or also like when I was sharing about it in ThoughtWorks, it's like, oh, here's how you make it even faster.
Use this other runtime for that, right?
I was like, yeah, I can imagine it and it will probably get even faster.
But I'm more interested in like, what is the output, right?
Like, this is actually good code.
Like, how far away is it from the big models, which obviously is very far away, right?
But then also I wanted to get a feeling for which types of tasks would I use this for, right?
Like, what is the size it can handle, the complexity, you know, context window, like, you know, which...
code research size does it kind of like break down, right?
I think that's a lot more interesting, but also a question that's a lot harder to answer than how fast is it, right?
Because that's something we can easily measure, right?
Sure.
So yeah, it was interesting.
Like I said, again, it's like very far away from the big models.
It's still very weird sometimes.
Like I had a machine with 48 gigabytes of memory and one with 64 gigabytes of memory.
One was an M3, one was an M5.
And I had one task that I then ultimately I had like an automated setup to run a task multiple times.
And I had one task that got better quality of output on the M5 machine with more RAM, even though everything else was the same.
Like I would have expected faster responses and all of that.
But why was the quality different?
Like it just solved the task a lot more often.
on the bigger machine than on the smaller.
So that's something that I'm still like very confused about.
So maybe it's because it's a mixture of experts architecture and the bigger machine was loading more experts or something like that.
But there's all of these little idiosyncrasies and things that happen and you're like, what, why?
And so that's really annoying, you know, when you actually have to get your work done and you don't really know what to expect, right?
And that's what I mean by this, like, is it plug and play experience so you can like, kind of rely on like getting similar experiences as somebody on the internet who also tried it, right?
Yeah, to your point, like one of the smaller things that I've done with it is a side project where an author and I are going back through a corpus of articles that she's written over the last couple of years.
There's a thousand different articles and sort of having an LLM go and analyze them.
And to your point, like speed matters, yes.
But I can ask one model, which is, you know, highly quantized and compressed down to fit into, you know, like a Raspberry Pi to analyze this article and it will hallucinate and it'll miss three things.
We'll be looking for like, what companies did we mention in this article?
And it will find one of the seven.
And it will think that we're saying that there's a relationship between it and another one.
And we didn't.
And it's like, I don't care how fast it is.
that's useless.
You know, that's not what we needed it to do at all.
Like didn't meet the brief.
So, but then on the other end, like, yeah, I happen to have a 64 gigabyte, you know, Mac mini.
I loaded up one of the larger models and, and because I had like something else going at the same time, I type high and push enter.
And it was literally like three minutes before it responded back.
It's like, okay, yes, but somewhere in the middle would be nice.
And that's what I mean, right?
Like.
QEN 3.6, this 30, 35 billion, like I said, has a decent speed, both on the M3 and the M5.
It really pushes the RAM, the 48 gigabytes.
So the 64 is a lot better, but you get decent response time, right?
And then I was like, okay, this is runnable.
Now let's look at if it's actually viable in terms of actually giving me some decent result for a certain category of coding tasks.
Can I ask, I assume that That it would be good for, okay, I'm also Anthropic Stack, right?
I'm often talking to Fable or I'm talking to Opus, and it is going to go make a plan for me.
And then when it gets to that plan, thanks to it's probably Superpowers or some other plugin, it's going to come and say like, okay, here's the plan.
Do you want it sub-agent driven in the way that we approach it?
And I haven't double-checked, but I imagine it's starting Sonnet, you know, models in order to actually do the tasks.
which makes sense.
People have talked about how the larger models are almost too wordy to write code.
Like they can get rabbit hole in the wrong direction and Sonnet's much more focused.
So, okay, good.
I imagine when you're saying this, that you're saying that Quen 3.6 is a good replacement for the Sonnet end.
Like once the task has been defined.
Yeah, so that's what I hear a lot of people talk about.
And I've also tried this, right, like kind of plan with like a big, powerful model, which I usually used to do or do anyway, right?
I have it break it down into like multiple phases and then, you know, multiple tasks and stuff like that, and then delegate those tasks or like smaller pieces to a smaller model.
And I think that makes sense.
It's probably, it still depends on what those subtasks are.
Right.
So I wrote in the post as well.
It was a little bit like detox, you know, like from the abundance and kind of like it felt a little bit like when I first tried the co-pilot, it was called multi-file edit at the time.
It wasn't called agent yet.
And in that mode, because at the time it could only change one file at a time.
Right.
And now it could change multiple files, but you had to tell it first which files.
were in the workspace, so to say.
So you had to already have reasoned about your task and thought about, yeah, I think it will have to change these four, five, six files or something, right?
So it's a little bit like that.
You have to actually know a little bit like how big will this change be?
How sprawling will it be, right?
And maybe if that's part of the planning, like the code research and the bigger model has figured out which files you have to change and so on.
I mean, then ultimately, I guess the question is at which point is it, is it so far broken down that then the big model could just write the code as well, right?
Because often in the plan, they already write all the code, right?
Pseudo code or something.
But yeah, I've also tried this.
I've tried starting to build like a game where I had the plan made with Claude and then with Sonnet or Opus.
I don't remember.
And then just like had to delegate that to Quen.
And at first it worked really well.
And then as the game was starting to get more complex, it's like...
It just became too annoying.
So again, it was maybe where the pieces where I know this is quite complex and I'm still going to use Sonnet and like where the things that are really straightforward.
And, you know, so I use it for like writing a script or when I know it's exactly these two files and I kind of roughly know that's going to be the change or just like straightforward things where the prerequisite is almost that I have some awareness already what needs to be done, you know?
Yep.
A hundred percent.
Yeah.
Okay, cool.
Let's shift modes a little bit into not just your process, but the teams that you serve, the teams that you've had an opportunity to observe.
And the way you've described your role, you know, I think of it as sort of a clearinghouse for what's going on inside of ThoughtWorks for AI software delivery, which I'm quite jealous of.
I imagine you got to hear lots of good stories.
But yeah, let's switch to thinking about the teams that you've gotten a chance to observe.
Part of me wants to start with like some broad sweeping question, like, you know, what are the best teams look like?
Yeah.
So let's start there.
What are the best teams look like?
It's really hard to say for me right now because I hear so many things in an anecdotal way, you know?
A lot of our clients or like big enterprises are also chasing for the one way that you need to set up a team and then it works right now, right?
And I think LLMs or the identity of AI is such a...
generic general purpose technology kind of so you can use it for so many things on the team that I sometimes feel a little bit like it's missing the point a little bit when you try to find that one workflow because the whole point is that it's so flexible right yeah yeah and so but then of course on the other hand of course there are things that kind of like make that are generally applicable that you know are more generic right of course it makes sense if a team kind of like shares let's say, prompts with each other, right?
If that's in the form of AgentMD or skills or some other way, right?
Or it makes sense to think about like which parts of the knowledge in the enterprise do we want to make available to an agent so that it has more context when it does its planning and all of that, right?
So there's like these general things.
when you ask for like, I don't know what you said, like the best teams or like the most effective teams with this, right?
It also depends on like how far it goes, right?
There are some organizations now, clearly that are very advanced with this that have like a level of maturity where they can have like unsupervised cloud agents kind of like work on things like, I don't know, documentation edits, CI fixes, like small things, right?
Like there's a lot of really good use cases for like, migrations, right?
Like version upgrades, library upgrades, stuff like that, where you combine a large language model with a code mod, right?
And they kind of like work together to do it.
So that's like one way of like, oh, it's really effective, but that doesn't necessarily make sense for every organization when they actually don't have the infrastructure maturity or the security stance or the, you know, so in those contexts, an effective team is more about like how do, you know, like.
how they use it the best in like a supervised way so that there's enough, like, again, the guardrails and stuff like that.
Right.
So it's, yeah, because it's like, there's so many things, it's like kind of hard to say.
It depends a bit on the context, how far you can go.
Well, I agree.
And you also gave us a nice range there.
Dimension trade-off slider from one to the next.
So thank you.
Okay.
Let's sort of focus in.
I like to think of this as we're going to start on the left and work our way to the right.
What are you seeing change in the way that people approach product management, story breakdown, those kinds of things?
What are you seeing change there?
Yeah, I mean, instead of the product management, and I see because I can, the listeners can't see it, but like we, you have a list of all the things, right?
And I see that this design is not in there, right?
Like user interface design.
So you see that as part of product management, I guess, right?
And so we, We have a team in ThoughtWorks that are like, I think maybe it's two teams that are maintaining this open source medical record system called BAMNI.
You might remember from your days of yore.
Yeah.
And so they did sometime last year, they started just like trying almost anything in AI because they also like their code bases are public.
Their confluence is public.
Their JIRA is public, right?
It's open source, right?
So there were less data security concerns and they had to do a revamp.
of like part of their UI.
And at the same time, they needed to support a certain standard from the domain a little better.
And they also wanted to switch from Angular 1, I think, still to React, right?
So three things at the same time.
So it was like a larger project.
And as part of that, there was an experienced designer who tried a lot of different things in the user interface design space, right?
So also like there was an existing design.
So she tried things like a review of the existing design, right?
Like what are things that work well?
don't work well.
She had access to a lot of user research from like people who had in the past who had actually gone to the clinics that run this like in rural India and so on, right?
And so she had access to that, was included images of like the waiting line and how they were organized and stuff like that.
So she could use that to analyze all of that research and, you know, get insights without having to go again herself like yet another time.
Or she created kind of synthetic user personas that she could chat with, right?
Like of the doctors that work in the hospital.
I think it's very controversial technique among experienced designers, right?
Because it's kind of like part of that profession that, you know, talking to real users and actually like, you know, listening between the lines and actually seeing what they're doing is like really important, right?
Yeah.
But...
Right.
It can actually, if you just like want to cover the basics first, right, it can actually be quite useful to have like an LLM with like training data, especially when it's common domains, right, to like explore kind of like the first level of it.
And so the team works with a domain expert with a doctor who knows the whole domain and everything really well.
And he said afterwards that.
Yes, not everything was correct and all of that, but he felt like when the team came to him to get input from him, it was already like 50, 60 percent there.
So they actually needed less of his time because the work that they could do without him had higher quality.
Oh, that's amazing.
So that's kind of like the more interesting stuff.
There's a talk on YouTube, actually, of our XConf.
ThoughtWorks has this conference on XConf and for XConf India, two of the team members.
Describe what they did.
They're also the technical migration to the React components and so on.
Nice.
I'll link to it in the show notes.
That's amazing.
Cool.
Cool.
Architecture and planning.
Are you seeing it change much there?
Yeah, I mean, architecture is an interesting one.
That's one thing that most people are saying that architecture and the more high-level design, that's where we'll sit as developers, as engineers in the future, right?
That that will still be part of our main job, right?
To stay on top of that.
I do think that it's really useful there to help, right?
If you're actually asking it the right questions, right?
Because there's always trade-offs in all of this, right?
And if you don't ask for those things, if you don't look for those, you just take like whatever the model gives you as a suggestion, right?
So I do think there is this kind of like practice of when you have that experience of what you ask for, when you look for those trade-offs or like what is an even simpler solution or, you know, but I mean, I definitely think it's a really good use case if you have those tools in your toolbox of how you structure it and how you ask those questions.
It can really help order your thoughts and get like a quicker assessment of stuff, right?
One of our guests, Ahmed Misbah, he said, yeah, he's terrible at architecture.
And at first I thought, what?
I've had some really good conversations with Claude about architecture.
But I think he means some of what you're saying there, that like if you ask it the right questions, you can get really good answers.
But his point of view was like, yeah, if you just ask it, it'll tell you to do everything in microservices.
And not that that's always bad, but that that's sometimes not the right answer.
It's always a bit much as well.
Like when I have it right design documents or stuff like that, I usually go through and just delete a lot of stuff because it's just going to confuse future humans and future agents.
Yeah.
But yeah, I do like it as a.
as a helper to create some of this stuff and just make the thinking faster.
I do really hope that with like skills and maybe like honing some of the architecture.
tech tools that we have in our toolbox, that we can make this a great learning and then practice tool for people who are coming into the profession now as well.
I think that's going to be key because a lot of the things that we're doing right now are being done by experienced developers, experienced engineers, and we're building tools and skills and prompts and approaches for ourselves, right?
But we have that pre-existing.
practice, that muscle memory, all of that, right?
And that's not going to be sustainable for the future.
So that doesn't mean that it won't work, that it will be all bad, but we need different types of tools and approaches for people who don't have that.
And maybe they will develop that if we don't help them do it, right?
But something has to come.
But if we insist on only making the right tools for us and not for them, we'll have failed.
And if they fail to make tools at all, we'll have failed.
Yeah, there is a cliff coming.
Will there be good parachutes or stairs down when we actually get to the lip of it?
I don't know.
But yeah, there's obviously an experience cliff coming there.
Yeah, interesting.
Okay.
Obviously, using agentic coding tools in order to develop code is a change in software development, right?
Like we used to open up editors and write it ourselves.
Does anything else come to mind when I ask, like, what's changing about the way software is developed?
with all of this and the teams that you've had a chance to observe and the teams that you serve?
What's changed about software development?
Yeah, I mean, using the coding agents is the biggest one, right?
And I mean, you were asking about story breakdown before, right?
I didn't really like talk about that.
But there's also still this like unresolved question about like where...
does it happen or where does, let's say, requirements breakdown happen in general, right?
Like I sometimes feel these specific words like epic story and all of that.
It's sometimes like muddying the waters a bit.
Ultimately, we always have a large thing that we break down into smaller things and then we break those down into smaller things, smaller things until we say this is small enough for what I want to do, right?
So like the breakdowns, all the spectrum development workflows and developers using them, right?
But where is the...
Are we pairing with the people who have more of the product or the business context?
Or, you know, are we assuming that developers will be doing all of that?
Or is it really just what we used to do in the past as well, where we would get a story and then we would break it down together into tasks, of course, right?
But it seems like people want to do more than that now, right?
But like, where do we have enough product context?
Like either engineers could get that in the future or we pair with other people who...
specialization that is, right?
I think it's not really made explicit yet.
I guess developers think we're going to replace the product people and the product people think they can now also code, right?
Everybody thinks they can do what everybody else is doing, but maybe there's still some stuff that's really useful as a specialized skill, right?
So yeah, it's kind of like maybe back to your question about what does it change about software development?
It's maybe like just still really fuzzy, like how far...
it expands, right?
The responsibility of the coder, let's say, right?
Like I just recently saw a talk by somebody, this was in the context of developing like a technical kind of developer platform.
And they had started doing something where they're trying to make developers responsible as well for the adoption of the features that they're building.
So done is only when you've also like participated in making users actually adopt this.
And so this was one thing they tried to mitigate this thing that when you just can just have high throughput and the features just pile up, but nobody uses them.
Right.
But then when you include developers into making them responsible for also getting people to use it, you know, so it's almost like a new definition of done and like a different version of like stop, stop starting, start finishing.
Yeah, yeah, yeah.
So this was a technical product.
So it probably by far doesn't work in every context, but I found that interesting, right?
So I think maybe if you ask me so generically about what changes about software development, it's this whole like thing that's happening right now where everybody's testing the boundaries of who's doing what, right?
Oh, I love that.
I love that.
To your point, if you're making like an API, it's somewhat, is it easier, I guess, to prove that there's engagement or like maybe, but I think that applies outside of technical products as well, for sure.
Yeah.
Like if- Yeah, it's different skills.
It's different stakeholders.
It's different.
You're right.
I mean- Yeah.
Can you just do that with AI as well?
AI helps you do it, right?
Yeah, we'll find the boundaries of that, right?
I'm starting to think that we used to sort of create these bespoke, you know, handcrafted teams that, you know, had so much of Angular and so much of, you know, Python and so much of like API design, because that's exactly what this project needed.
And the next one needed React and Java and something else.
And now it's going to be more about like, OK, this is the person that's really good about having a sixth sense for, you know, where the product needs to go next.
But this one is really good at finishing it.
And that might be a software engineer and a researcher, you know, or something or people that used to be called those things.
But now they're doing all of the things, you know.
What about testing?
I mean, there's obviously a lot in here.
Yeah.
Yeah.
Testing is just, first of all, kind of a continuation of the.
discussion we just had also about what do developers do.
I mean, I think when all of this started, a lot of people were saying, oh, yeah, developers will have to be testers or something.
Then for some reason, I didn't hear that as much anymore.
And now recently, I've been thinking a lot, again, not about testing, but just like how much more of QA is like QA mindset will have to have in the future, right?
Not just the design, but QA.
And then I always give the example of the best QA I ever worked with.
who was a career changer, actually, and was probably the person who knew the system better than anybody else on the team and knew exactly where to poke and to pull when we were doing certain things.
If we were changing a database, it was different from when we were changing something in the UI.
So kind of just knowing the second order effects, the consequences of like...
what it means when you change certain parts of the system and the risk assessment of that, right?
So this like kind of QA mindset, which is also not a lot of engineers' comfort zone.
But I think if it's again like, you know, let's say the agents are the people on the team and I'm the QA and I don't know exactly what they coded, right?
But I can kind of like tell for this particular feature, what do I have to do, right?
So that's one thing about testing.
And then the other thing that, I mean, just in general, right?
Like there's so much still we have to figure out about how do we know that what is being built is functionally correct in a world where a lot of people seem to want to go, which is we don't look at the code anymore, right?
Yeah, yeah.
So, but then how do I know it's functionally correct?
And I mean, the most interesting thing in that space I've seen so far is this idea of, it seems to be going in a kind of like an acceptance test, like, direction where there's like some level of testing that we are really still very much in control of and very aware of, right?
Where maybe, so this is, this is one called Yvette Oerdek, who has talked about this and written about this a lot.
And I recently talked to her and she gives like multiple examples of how in her code bases for each code base, she thinks about a new kind of like custom test runner.
Almost like what is the way that I would want to see input output what's happening if a new bug comes in?
How can I give the agent the next scenario that, you know, it needs to fix or something?
And she had examples for that between like a 3D game she was building and like for lots of different areas to just like demonstrate how you it would really be like a custom runner almost for every archetype of those things.
But we can now build.
these custom test runners for ourselves very, very easily, right?
Like one of my colleagues, Matteo Vacari, has been practicing that on a few teams now where I think his cases were mostly, at least the ones that I remember that he told me about, HTTP APIs, where you have request response kind of input output.
And he just tries to make sure that there's a very readable version of these input-output pairs somewhere that he can really look at and easily reason about.
And that's kind of like his source of confidence that it's functionally correct.
I don't know then about unit tests and stuff like that.
Maybe we're really going to start just trusting the agent with unit tests at some point and just looking at mutation scores and coverage and stuff like that.
And then in combination with manual testing, maybe we'll do more manual testing.
I don't know.
Yeah, so I think that's a really interesting and crucial area that I think we should be spending more of our airtime on than we are right now.
I like how you talked about how at first we...
you know, kept hearing that we were all going to have to be testers.
I remember that phase.
I also remember a lot of people saying like, ha, TDD with agents doesn't make any sense.
And I think the people who were saying that at the time, me included, meant that like sitting down and trying to ping pong where I write the test and Claude writes the code is a waste of time.
Like it does not make any sense.
But recently, I think it's the Superpowers plugin, which I've already referred to, just pulled in this habit of like, we're going to make the sub agents all do TDD.
So it's now the agent itself writing the test and making it pass.
Yeah.
And that appears to be working.
And I'm building test-based code bases that give me more confidence.
And I don't mean like, oh, it's TDD, so I believe in it.
Like, no, like it actually gives me more confidence, the code base.
But go ahead.
It's interesting that you say that, but it might be a tangent because we have more of your list to go through.
But it's actually my pet project right now because I've always been suspicious of like telling an agent to do TDD just in its agent loop, right?
Yeah.
Tell it, no, I want you to write the test first and then the red test.
And then, you know, and I've always been like, yeah, but why?
Like, why would I, if I don't look at it, if like, why does it matter?
Right.
And I haven't found any eval setup, nothing so far.
Please, like if anybody listens to this and knows one, please let me know that compares it.
Right.
And I mean, to be honest, I don't want to do like a huge setup and compare it, but I've done like some like, like some.
probes, let's say, right?
They have like three different tasks, run it like with TDD prompt, without TDD prompt, check that it actually did TDD, compare.
I mean, first of all, obviously it takes a lot more tokens, right?
So they better be worth it, right?
It is slower.
It's also really hard to get it to do it in the first place and kind of unreliable, right?
Because it doesn't want to do it, right?
And to be honest, with my few samples that I've done so far, there are some indications that it might be worse.
the results.
Interesting.
Because so I had Opus evaluate this and then also had Opus look at how the agent did it and make some, come up with some hypotheses why Opus thinks it might be worse in the cases that I used it with.
And it had like some interesting hypotheses like, you know, in the case where it's not asked to write the test first or do TDD, it actually does like the whole design at first.
And that seems to create richer design than when you...
force it to go step by step by step because with the tokens and the prediction, it just doesn't work the same way as a human would go step by step, I guess.
Okay.
So I don't know.
I mean, would love somebody who has lots of tokens and time at their hand to do like a bigger setup.
But yeah, mine is an anecdata for sure.
Yeah, exactly.
Like I ask people, so what do you expect from it?
And have you seen that work?
I think the one thing that felt something like evidence, I started to notice that it was passing dependencies in as parameters.
And like all of a sudden my method signatures were getting wider.
And I'm like, what the, what is it doing?
And it's because it needed it to be testable.
It needed to be able to pass in an instance of the thing.
And I think that is maybe one why that is like salvageable for TDD, that salvages TDD, that defends its honor in this new, not that it needs it.
But if there's going to be a reason, I don't think it's going to be so that we can read the code better.
I think it's going to be to have created testable code in the first place.
Yeah, I think TDD is a...
is one of those practices that is actually very much for humans, right?
Like I also read again about like Kent Beck's arguments around like why TDD and a lot of them are human stepping back, thinking about it, thinking about the design, all of those.
So it might actually be like, actually to be honest, one of the few cases I've seen so far where something that is good for humans might not be good or might not be as good for agents.
For various reasons.
Oh, that's interesting.
I'll definitely be watching your space for results there.
Okay.
Now, This question sometimes is a stumbler.
I'm just going to say integration.
Now, the first meaning for integration is like, oh, I've got one method and I've got another method.
And I wrote this one and you wrote that one and they need to talk to each other.
And that's integration.
Yes.
But there's also the sort of integration where, OK, some other team that you've never met has been writing something on an entirely different cadence for three years and they've forgotten where half the bugs are in their software.
And suddenly you need to integrate with that thing.
Right.
So I think I mean that.
more, you know, where and that sort of brings up brownfield, greenfield, and that's a part of it.
But maybe also you're dealing with a boundary.
Maybe you're dealing with some sort of API boundary and you're not just going to go through the code base.
Maybe you don't get the code, you know, so it's maybe more like integrating with some big unwieldy API that wasn't really designed for it.
But all of that, think of the hard parts of integration, not the fairly simplistic parts that we've mostly got figured out.
What have you seen change in integration?
A while ago, like I put together just like some, let's say, scenarios about how AI might impact team sizes, team shapes, team, like how teams are designed in an organization.
You know, like when you think about like value stream aligned teams that own certain domains or value streams and like, you know, they always have integrations with other teams and dependencies and you want to reduce the dependencies, but you also want to keep the ownership at a reasonable cognitive load level and all of that.
Right.
And so I.
Can you imagine that dependencies get cheaper between teams, right?
Because it's easier for them to talk to each other.
And there's new ways to document for each other how you use interfaces and stuff like that, right?
Like by providing skills to other teams and stuff like that.
A little bit like what you said, right?
So I haven't heard any concrete stories, but like, yeah, it's like another one of those like little impact points where it's not just about like everybody getting faster, faster, more throughput.
But how could it also actually positively affect some of the organizational challenges we've had?
Yeah, I like that.
Good answer.
Okay.
Deployment?
The actual act of taking the code and putting it somewhere?
Yeah.
Yeah, I've always wondered about that.
Like, I mean, the whole point of like deployment and deployment automation is repeatability, right?
So...
Yes, you could have something where under certain circumstances, AI triggers a deployment or something, but, you know, big whoop, I don't know.
It's not really a big participation in the actual deployment, which I would still very much want to be repeatable CPU-based scripts, right?
Because otherwise, what if something happens like...
how will I even find out what happened?
So it's almost like not having AI be too involved in the deployment and having like really good automation is almost like a driver to be able to use more AI, right?
Because you, like all of that, like continuous integration, continuous deployment, continuous delivery stuff is an engine.
to enable you, like the organizations who already are really good at that have really an advantage to use AI now.
Because they're also able to roll back faster, to find errors faster, to reduce the mean time to restore all of that rain.
Yeah, 100%.
Okay.
What about infrastructure?
So this is thinking about, like, okay, you deployed it somewhere.
No, this is describing it somewhere, I suppose.
Yeah, yeah, yeah.
It's interesting just for myself.
So it's never been like my focus area, kind of like my T-shaped expertise, right?
And recently I had to make a change to the Terraform code that hadn't been touched in a while of this like small application, internal application that I've...
I'm maintaining on the side, but it has some users, not so important, right?
But I could feel myself kind of like hesitating.
I trusted AI less, right?
Because I know how long the feedback loop is.
It's a lot longer than just like trying some front-end code, right?
And lo and behold, I accidentally deleted the DNS configuration, which is then a thing that has an even faster, even longer feedback loop to recreate it.
Right.
But AI could also help me hold my hand through the process and recreate it.
Right.
Yeah.
So, yeah, but that's just like not really like how does AI help with infrastructure, but that was just my experience then doing something that is different from like fast feedback loop kind of local development.
Yeah, so I haven't looked into this space as much, but I know that Keith Morris, who also works at ThoughtWorks, author of the Infrastructure as Code book, he's been looking into this a lot recently.
And I just know that he has been using this tool Swamp a lot and kind of like thinking through what that means.
Unfortunately, I can tease you with it now, but then I can't tell you that much about it, right?
But it actually comes from like some of the people that created it are like ex-chef.
Maybe as well.
I'm not sure.
They had this thing in between called Systems Initiative, where they also tried to come up with like a totally new way of like building infrastructure.
They've pivoted away from that and are like fully focusing on this like building infrastructure code, so to say, or like building infrastructure with the help of AI.
So, yeah, let's give you that teaser.
I don't want to like speak out of turn about any of it.
Yeah.
No, I'll have to see if I can get the key phone next.
Debugging?
When it works, it's magical.
When it doesn't, so what?
Then you just do it yourself next, right?
So it's like so good when you really just, you have the information right there and you just point AI at it and it finds it and you don't have to go through any of it.
It's just so nice, right?
I've also had a case where I needed to debug something and I just had it like.
And I just saw the agent spiral, spiral.
Then I gave it a bit more information.
It spiraled again, right?
And then at some point I was just like, okay, I just have to give it the information, right?
So I set some breakpoints, looked at the actual values, gave it that, right?
So it needs to have the information, right?
I know that, is that in IntelliJ or somewhere?
Somebody told me recently they had a coding agent who was setting the breakpoints as well and then stepping through.
Oh, wow.
I've been waiting for that for like two years.
Like, when can I finally have it do that?
Right.
I think it was an IntelliJ, but I'm not 100% sure.
So that would be cool.
I mean, why would it not be able to do that?
Right.
You just need a tool that integrates like that.
The point is, as long as you can.
give the agent access to all of that information, right?
So when you take it to a high level, like debugging incidents and stuff like that, right?
Like how prepared is your infrastructure?
Does it have good observability data?
Does it have, you know, so then it starts with all of the things you need to do to have good observability, right?
Because that's another one of those where having good data is good for the agent and for the human, right?
Yeah.
Wow.
I love that.
I sometimes think of it in terms of like, oh, if you can just add more unit tests, you don't need the debugger.
And yes, I also sometimes think of it in terms of like if Claude is in it, it seems to get into a mood where it's like, I'm just going to guess.
And it's like sometimes it guesses, right?
If it's if its context window is healthy and all of that, its guesses are pretty good.
But other times, like, again, like if some skill has told it, like, I'm going to encounter, yeah, I think it's a superpowers plugin that I'm thinking of.
Again, I guess I'm a walking ad for superpowers today.
Yeah, but I also wonder if the trend is really towards more acceptance tests again.
Yeah, yeah, yeah, yeah.
I think that's true.
You can always get the agent to like, oh, no, before you try to debug, you know, write a unit test, yeah.
So that's still valid.
But you always have sometimes a situation where you just don't get around it.
For sure.
Yeah.
And you're going to step through.
Yeah.
A hundred percent.
Okay, good.
What about refactoring?
Yeah, I would say it's gotten a lot better.
In the beginning, I was always like very suspicious of what it was doing.
There was also, we had a podcast episode about that on the ThoughtWorks Technology podcast, but probably like two years ago or so with Adam Tornhill from CodeScene, where they had done some experimentation with, at the time, GPT-4 probably was kind of like state-of-the-art or something, and some other big models, and asked them to do refactorings, like refactoring on the small level, you know, like extract method, change argument, stuff like that, right?
I mean, there's different meanings of refactoring that people have, right?
But at the time, they found that actually only in 35% of the cases, it did it in a way that didn't change the behavior as well, accidentally.
I think it has gotten better, but like I've said once or twice, right?
There's always this non-zero probability, right?
Yeah.
So, but yeah, what is like good enough for like whatever you're doing, right?
Like I've definitely felt like with the model like Sonnet or Sonnet and up, I've gotten a lot more comfortable.
having it make like bigger refactorings.
I mentioned code mods before, right?
Those, of course, like, let's not forget that we can give the agent tools that can actually reliably change the code, right?
Right, right, right.
So, and that's also become a lot more common, right?
A lot of the harnesses now have integration with language servers built in that can help you do that.
And so, yeah, that's definitely gotten better.
Okay, you're at least the second guest to mention code mods.
I think Hari did too.
And he...
It was the one referring to your spectrum and development post.
So interesting.
Pair programming, mob coding, these ensemble.
So what I hear from teams is that there are still teams definitely in ThoughtWorks doing pair programming with agents.
I just recently talked to two kind of like early career developers in ThoughtWorks.
Interestingly, both of them were career changers.
So they'd had like the beginning of a previous career and were really good at learning and like what that means and all of that.
So it was like maybe a bit like not quite the average case.
But on their teams, they're also still pairing and also really appreciating that.
And then I think especially in these times where things change under our feet all the time.
I mean, remember, it was just innocuous things in the past, like learning a shortcut in the IDE when you were pairing with somebody.
Oh, my God, that's changing my life.
Right.
Imagine all the things you can learn from each other right now with like everything changing under our feet every day and everybody using them slightly different.
Right.
I think it's super important, especially right now.
And then two brains see more bad code than one brain.
And you can really like develop.
stuff and harnesses for your team together.
Then the big question, of course, is like, oh, what do you do while you see all the, you know, the agent do its work?
So what people say is they just like, they talk about what they're going to do next, what they expect to happen at the end of this, what are we going to test, right?
So it's like just kind of still like constant conversation and planning and thinking through.
Amazing.
And mock coding, I hear a lot of teams like do it as well, especially with...
agentic coding, but I haven't found many details about that.
I found myself encouraging teams to do it more.
A, to make sure that we're sort of all using the tools in a similar way, and B, also because of alignment.
We used to pair with one another on making changes and then switch pairs, which meant that we all sort of knew most of the code base pretty well.
And I feel like we're somewhat disadvantaged about knowing the code base now.
Yeah, but I'm with you.
It's, again, an anecdote.
I'm not sure I've got much to go on there, but it feels right.
What are you noticing about team ratios and role ratios, right?
Like, we kind of hinted at this earlier.
We used to sort of handcraft these teams based on what skills we needed and so on.
But we also used to say, like, oh, so many dev pairs to so many QA people and so on.
What are you seeing there?
Yeah, I haven't really seen much like to say, oh, there's like an interesting pattern there or something.
Like I said before, it depends so much on the organization and their setup of how much they can actually do.
But I'm a little surprised that I'm not seeing more experimentation with this, actually.
I don't know.
It's just that I'm not hearing it.
You know, I'm not going through every team at ThoughtWorks every day asking.
But I would say it's definitely like fewer devs.
That's, you know, like when you have like a good setup, maybe it's Greenfield, maybe it's some like people with some good experience in the space where you're working.
You can get so much further with like just one pair, right?
Yeah.
How do you, how's it changed how you hire staff, train, mentor?
Yeah.
I mean, hire, as far as I know, we offer the use of AI coding assistance in our interviews now.
So it's like an option.
I think.
It's been a while that I talked to the recruiting team last, but they first did kind of like an experiment where they just offered it to candidates and it was kind of like voluntary.
It was like initially actually surprisingly for a long time, people didn't take us up on the offer that much.
But it's been a while.
It's been like almost a year, I think, since I last talked to them.
So I assume that it's different now.
But I've also heard some of those stories where people felt like the candidate was kind of trying to cheat and had some kind of like thing where they could read stuff off the screen.
And so it's, yeah, it's an interesting world where some companies are starting to invite people in person again in the offices, right?
Yeah, yeah.
Yeah, well, I saw that before, AI.
Yeah.
Yeah, for sure.
Yeah, and also for the training.
So we have this ThoughtWorks University kind of like, yeah, boot camp, but not a boot camp grad entry program.
And that also uses AI now in the beginning.
I don't know that many details about that.
That was also like a process.
It didn't like immediately go that way.
And I don't know how much they're using it now.
And the mentoring is interesting.
I mean, I was in a...
panel discussion a while ago, which was with like more experienced people like architects in the room.
And one person was like, oh, how does it work with the mentoring?
I used to love it when I was being mentored and I could mentor other people.
Right.
And I mean, I was like, if it's a room of like experienced people, right.
they should have some influence in their organization to actually say, hey, we still need to do this, right?
And like I was telling you before, like the teams I talked to recently were still pairing, right?
And there was still like this environment of where there was like a tech lead or like a more senior person on the team who was actually interested in helping everybody build good code, right?
Creating better developers, even if they use AI, right?
I also like, it's so hard for me as a more experienced person to imagine.
how you learn these days, right?
So I think we just need to talk to people more and not just say, throw our hands up and say, oh, I guess, you know, at the moment, we don't know how to deal with juniors.
So let the next generation figure that out.
We just won't have any.
Yeah.
And I mean, it's not like we romanticize how we learned as well, right?
Let's just talk to the new people coming in and ask them how they're learning and how we can help them, right?
Okay.
Okay.
Yeah.
So, Birgit, I've used so much of your time today.
I'm sorry for that.
Thank you for being so generous with us.
We got into some tangents.
I'm sorry.
It was also my fault.
You and I apparently have a love of tangents.
So what's something that's giving you joy right now?
Oh, I mean, I really enjoy pattern matching and communication and like figuring out problems and complexities and trying to find easier ways to talk about them.
So that has been quite...
yeah, enjoyable for me, like using all of these things and trying to bring some structure into the madness and then, you know, getting good feedback that it like helps people when I make up some words or something that has been very rewarding.
Amazing.
Birgit, thank you so much for coming on the Engineering with AI podcast today.
Thanks for having me, Kai.