# ThoughtWorks Radar 35: AI Agent Strategy

**Podcast:** Thoughtworks Technology Podcast
**Published:** 2026-10-29

## Transcript

Hello and welcome to the ThoughtWorks Technology Podcast.
I'm your host for this episode, Lily Ryan.
And today we are going to be talking about the Technology Radar Volume 35.
Joining me to discuss this are Doppler members Chris Ford and Effie Eldon.
Chris is a mendicant ThoughtWorker, originally from Australia and currently residing in Barcelona.
Folks in the functional programming community might know him for Music as Code.
He's now ThoughtWorks Chief Architect.
Effie joined ThoughtWorks as a consultant, specializing in, quote unquote, DevOps and duct tape.
before ending up in the global information security team.
She lives in Melbourne with her greyhound noodle, as well as a lot of tech junk that she swears she will do something with one day.
Maybe today's that day, I don't know.
Welcome Effie, welcome Chris.
How are you both?
Yeah, really good, Lily.
Nice to be here today.
Yeah, nice to be here today.
So we're talking about the tech radar.
Volume 35 is going to be out pretty much, I think, when people are hearing this.
And as the both of you were in the room when this was being discussed and debated, I wanted to get into some of the detail about what's in this edition in terms of the themes and what themes are also in relation to the TechRadar, because that's a whole thing in itself.
To get us started, though, Chris, would you mind taking us a bit through how the TechRadar is put together and what that looks like?
Sure.
So you could see the tech radar as a kind of direct democracy of technology within ThoughtWorks.
We have more than 10,000 people globally who are working on different projects with different clients, seeing different things.
So the way we run the tech radar is that we put out a call for all those people to nominate what we call blips or items for the tech radar that they've been observing and they're of interest.
And then what we do is we take all that information and different members of the editorial committee, which we call Doppler, responsible for gathering and curating and getting details and following up to get information for the blips.
And then the Doppler all meet in one physical location over a week and hammer out what we think the hundred or so most relevant blips are.
So really, the TechRadar serves as a kind of sensing mechanism.
for ThoughtWorks originally, and now we share it with everyone else.
Nephi, what's the difference between a blip and a seam?
So the blips are very small, discrete items.
They're submitted by the ThoughtWorkers who are on the ground working for our clients across the world, and they represent a specific tool, technique, platform, language, or framework.
On the other hand, the themes are a bit more emerging.
Over the course of that week, when we are looking at all these proposed blips, Often we notice trends, we notice groupings of things that belong together or specific motivations behind a lot of the new tooling that we're seeing.
And so part of what we do during that week together is note down when a potential theme appears to be emerging.
And then at the end, we look through and look at what's the most prominent theme, what we think is interesting to talk about, what best encapsulates that vibe that we felt over that week.
And that vibe is, I mean, I want to talk a bit about the vibe.
as well, because one of the things that we get the benefit of when we have folks from Doppler here is about what it was like in the room and how it went discussing that stuff over several days, because I understand that the opinions can get pretty heated.
I don't think we're as argumentative as all that, but there's certainly a lot to discuss because certainly with tech the way it is now, it's very common for me to hear some other wise and experienced person at Doppler nominate a tool to solve a problem that I don't know I have yet.
And so there's certainly a lot of rapid adjustment of mental models that I get out of being part of Doppler in response to collaborating with all my colleagues.
Very much a mind opening experience because, you know, that room represents in aggregate the sensing mechanism of all the thought works.
So it's a pretty powerful fire hose.
Yeah, I mean, I think.
As Chris said, I wouldn't say there's much in terms of arguments, but certainly some debate sometimes about what we think we can actually encapsulate into a blip and what feels a bit too complicated to blip, as we put it.
Obviously, it can be easy to think something is simple and can be put into a paragraph until you try and write it, which is why sometimes things initially get up and later don't quite make the cut.
But I think certainly with the themes...
No one in that room would really disagree with what we ended up with for the themes here because they were so prominent and so obvious, I think, at the end of the week exactly, where those clusterings were, where those vibes were taking us.
It's probably not a surprise to anyone listening either that a lot of the themes, all of the themes that we have here are related in some way to LLMs and working with LLMs and how that works and, in fact, other types of models and whatever.
you know, whatever we can be doing with them in practice.
It's been interesting to watch this evolve over time.
And I am really interested to see how you folks have perceived that as well.
But the first thing that we've got is putting more agents to work and talking about evolving over time.
This theme follows on from the volume 34's theme of putting coding agents on a leash.
In this case, it's putting more agents to work.
What's more agents in this case?
Well, I think really what we're seeing here is the impact of an industry trying something out and then encountering new problems all at the same time.
So one of the most obvious and kind of notable cross-cutting concerns of the Blips was that there were a lot of them that were about having some kind of surface where a developer can be running more than one coding agent at the same time.
I guess originally you might have as a CLI or an IDE where you've typed in some instruction and the agent's going away doing stuff in response to your harness and what you told it.
But once you've been doing that for a while, I think everyone who's got that experience finds that the wait time in between instructions and results and also the potential of getting more stuff done draws you into running more sessions at once.
So I think there's a lot of people who started out by just having multiple command line tabs who are looking for user interface that makes it a little bit easier to know when a particular agent needs their attention to get to the next stage or to make it a little bit easier to manage the different workspaces or Git subtrees.
And that's what we're seeing.
So we got a lot of blips that were, you know, around Claude Squad, T3 Code, Orca, and actually many others that I can't even name now.
And they were all solving this one problem that we think a lot of people are hitting at exactly the same time.
I think it's also about the evolution of the tooling and in particular the models, how much they're capable of, how much they're capable of without kind of a human directly providing oversight and feedback turn by turn, you know?
a few years ago it was I'm in my IDE, I have an agent as an assistant on the side and I'm prompting it to do a small bit of code and I look at that or maybe I'm typing and I'm using tab autocomplete to fill out these chunks but as the agents as the models get more capable it's now like I can just throw a card to it and come back and the card's done and so with less need to have a human intimately involved as the code is being written and tested and you know that that feedback loop is progressing we We see humans go, well, you know, I've got Claude on the job.
What am I doing now?
And the answer tends to be, well, what if I've got another Claude on another job?
Whether that's part of the same repo, you know, splitting off into multiple tasks, using things like Git work trees so you can have each agent in an isolated space.
Or whether that's, oh, there's three different projects I'm working on.
I've got a Claude agent on each of those.
And now all I'm having to do is wait for one of them to pop up and need something from me.
How should people be approaching this?
Because it's...
It seems to me that there are a lot of different ways that this could go.
There are a lot of things that people could try.
And maybe the question of should is, I mean, it is definitely opinionated.
But is there a better way than other ways to do this?
My experience with this has been that if I try and do that, my own attention gets pretty split across things.
I will often start something off and then totally forget that I had that thing going or that it was waiting for me.
Or I'll be on some completely different...
part of my work in some other place and come back and realize, oh, actually, that's finished a task I don't even need anymore.
So that's quite fair enough to say, be careful of mistaking activity for progress.
We know that using coding agents can really suck people in.
It's a really fascinating way to create software now.
And I don't think there's anything wrong necessarily with doing multiple things at once.
But I would say that Effie's point about agents being capable of working on tasks for longer is probably the part of it you should lean into.
So if you're able to improve your harness or use a model or create a feedback loop so that you can kick an agent off and need to intervene less often, that's probably superior to being able to spin one or two more plates at the same time and probably also better for your cognitive load and wellbeing.
I think it also comes down to following good engineering practices around properly scoping and shaping and sizing work, having a well-defined task that you can hand to an agent that's going to be able to do through to completion without lots of unknowns that it's going to have to guess or come back to you with, without it being open-ended enough that you might find the agent still going hours later.
By breaking down work properly, scoping it properly, having clear acceptance criteria, proper automated feedback loops so the agent can actually know when it's done and when it has met.
the requirements, that helps you be able to be a bit more hands-off.
But I think the other part of it is that kind of ability to know what needs your attention, where things are at, that observability and visibility into what's going on with your agents.
And I think that's where we're starting to see this new generation of tooling, which is kind of like meta-harnesses, meta-orchestrators operating one level above things like open code, clawed code, codecs that provide interfaces where, for example, an agent that needs your attention is going to pop up to the top as a waiting agent, as opposed to an agent that's working, which will be lower down.
Interfaces that at a glance can tell you, here's all your agents and their current status, which ones are waiting on approval for a command, which ones are finished, which ones are working.
That gives you a bit more visibility compared to a challenge I know I faced early on.
still face to this day, which is, you know, I've got 20 different terminal windows open, I'm going to mouse through to check, is that agent done, is that agent done?
And an agent's sitting there for two hours waiting for me to approve a copy command so it can continue.
One thing that worries me a lot when I'm working in this kind of sense, and I assume that in particular, FE, it also worries you too, given that we're in a similar line of work, is how to know when something has gone wrong.
There's certainly like when an agent is asking for your attention because it has hit a point where it, you know, has to for whatever reason, because you've put a hard rule in place or because there's some kind of point at which it has stopped.
But that doesn't always mean that that's the right point for it to happen or that something hasn't broken earlier on.
How does one set something up so that you can run?
a situation like this confidently with multiple things long running in parallel, even get up and walk away from it maybe, whatever you need to do and feel okay about it, not breaking stuff?
I think it's a good question.
I think there's a few different approaches that are somewhat rapidly evolving at the moment.
I mean, there's, this time around on the radar, we've blipped agent sandboxing in Adopt because it's just something that...
you need to be doing, whether that's coding agents running locally on your laptop or business agents running in the cloud.
The capabilities of models to do great work, but also great harm if they go off on the wrong track, if they get manipulated or prompt injected, or even just fundamentally misunderstand something is also growing.
And so we need actual technical constraints in place so that a model that does go off on the wrong path can't do too much harm.
And I think sandboxing is a big component of that.
I also think that what we're seeing from the frontier labs around built-in safeguards like Claude Code's auto mode and Codex's Approve For Me is another piece of the puzzle there because it means you've at least got a little bit of safety.
It's not something you should rely on primarily, but at least means you can be a little more confident walking away for a few minutes and you won't come back to the entire root directory of your hard drive being deleted, which I saw someone on Twitter complaining about recently.
Chris, Effie mentioned earlier tooling that would help a lot with observability, monitoring, and so on.
But when it's operating at this kind of capacity with lots of agents spinning up their own sub-agents, running for a really long time, and going in parallel, How does observability become meaningful in that situation?
How do you actually get real oversight of that stuff working and not just become kind of like human in a loop seer to some kind of meat proxy really for whatever else is going on and just push button, let's go?
Well, one of the principles I would suggest people follow is that if they know stuff that the agent doesn't, then that will probably explain a lot of the times why the agent does things that they think shouldn't happen.
So for example, one basic thing might be that if your agent can't tell whether the build is red or green, then it's not reasonable to expect that the agent, when it's going away and coding on a big epic or something like that, that it could actually respect build discipline.
So you mentioned observability.
That would be another example of where a good developer...
should be able to keep an eye on production to know the impact of their changes.
And hopefully, if you're developing in a way that we at ThoughtWorks would recommend, your code is going to production very often.
So that production performance of the application should be meaningful feedback in what you do next.
So one of the techniques that we would suggest is that where developers get information to improve their work, also expose that to agents.
So both for the status of the build and...
production, if an agent can be keeping an eye on that while it is iterating and while it is running the tests or whatever other harnesses you have, then it's more likely that it will be kept under control because it will be getting feedback that tell it if it's out of control.
I also think part of keeping an effective human in the loop is not allowing kind of human in the loop fatigue.
As a security professional, for a long time when I was running my agents, I had it strict.
Every time it tries to run a command, I have to approve that.
But obviously when they're running hundreds of commands every minute, that's just not feasible.
When I'm spending all my time reviewing long commands, long lines of bash and Python, I quickly ended up getting into this headspace of skim it for a second, hit enter, not really understanding what that code is or what it does.
I think that's again when using things like sandboxing, like least privilege, like ensuring that the worst case outcome is not that bad, suddenly you don't need to have a human in the loop there because you've actually got hard controls rather than relying on human judgment constantly to prevent something going wrong.
So I think human in the loop where a human is needed, not necessarily in all the possible places you could insert one.
How do you know when a human is needed?
I think one of the trends that we're seeing is that the way that we assure quality is becoming more diverse and actually as a developer what i'm quite excited about it's allowing us to try techniques that were perhaps a bit more considered avant-garde or impractical to implement before so uh i know on this radar we have for example tla plus which is a venerable model checking tool on the radar we also have things like mutation testing so i think that in answer to your question about when a human is needed i would say the answer is whenever there isn't an effective materialization of a check that the agent can use to check its own homework.
So I would expect that most of the time the human activity is higher leverage if reinvested in improving the application space harness and less in inspecting individual pieces of code that go to production.
That latter one is, as Effie hinted, quite...
exhausting and after you've inspected one diff, that doesn't put you in any better position the next diff that comes along.
Whereas if you have implemented a new kind of testing or some other kind of review process, then it's a virtuous cycle.
The other important element here is just about accountability.
An agent fundamentally can't be held responsible, can't be held accountable for things.
And so there are cases where you need a human in the loop because you need a human to be accountable.
for the consequences of a decision, for the consequences of a change.
And exactly how often you need that and when you need that is going to depend a lot on your context, for example, with the regulatory burdens you're under, the sensitivity of systems and data involved.
But fundamentally, we should be having a human hitting the button whenever there's a need for accountability because we can make agents very smart.
We cannot make them accountable.
To build on that, Effie, though, I think you already mentioned this, but human in the loop theater.
is a problem that we're seeing.
So when you do have that need for human accountability, that human has to have the opportunity to absorb enough context to meaningfully make a decision.
If the human is just there to press yes, then it's not genuinely human feedback.
It's just a opportunity to shift blame onto that poor person who approved the unsafe change.
Something that you said earlier about making sure that you have well-defined tasks and you're splitting things up well, I think also comes back into play here because you can, and there are certainly different libraries of skills and other kinds of tooling that you can use that will help you to explode your ideas out, work out what you want, figure out what the shape of certain things should be, getting down into tiny details and set some direction.
And there are also agents that will happily go off and write their own down to an infinite level of granularity as well.
But if you're allowing, I think, too much control to go to that in terms of how much decision making gets taken away from you in that sense of what you're trying to define, I think that too is also an important place for there to be a person who's actually setting some direction and making sure that that direction is still matching.
Yeah, well, I think there's some developers who...
don't really like to engage in analysis or value.
And so it can be really tempting if that's the case for you, that you say, well, here's a bunch of documents.
I don't really want to worry about what the backlog looks like or what the story breakdown looks like.
I'll just chuck it into the agent.
The agent will digest it and then off we go.
I would definitely encourage people to not give in to that temptation because to your point, Lily, I think a human being understanding what...
a meaningful slice of value looks like is still really important.
And if you can organize your work that way, you're going to be having much more meaningful review points that you as a human can give feedback on and that maybe you can demonstrate to your colleagues or to your users.
So that discipline of working out how to turn a bunch of stuff into a set of valuable chunks is not obsolete at all.
In fact, maybe it's more important than it's been in a while.
I think also something we've cautioned against in this radar is surrendering too much of your own cognitive decision-making to agents, particularly around heavyweight frameworks for methodologies for software development with agents.
I think it's very important that you make the workflows that your agent is doing work for you rather than the other way around.
You shouldn't be adapting how you do development to match.
you know, an opinionated framework that is not built for your context, for your team, for the way you're working.
It's much better to pick up things that are a bit more lightweight and actually evolve a workflow that makes sense for you and your team and the space you're working in.
And that way you actually have, you know, full knowledge of exactly what those steps are, what those feedback loops are, what those decision points are, rather than importing thousands of lines of markdown that tell your agent to do things that you probably haven't even read.
I think that's consistent with what our advice has been for a long time in frameworks in general.
You know, I think sometimes it can be tempting to use a really heavyweight web framework without feeling like you have to understand how the pieces work.
But our advice is that in the long term, understanding the constituent parts, which is probably easier if you pick a simpler option, is likely to make it easier for you to maintain the quality of your output over time.
Speaking of heavyweights, I wanted to move us on to the next theme that we've come out of this edition with, which is Beyond Frontier Models, I think we're calling it.
And this is the theme that came out of a whole lot of blips that, as I understand it, were pulled together because folks were seeing a lot of people falling into the trap of thinking, well, I must use Fable for everything because it is the most powerful and the most important and the most cutting edge, and therefore it will be the best tool for every job.
was something that you folks put together because that is a real issue in quite a few different ways.
Chris, could you kick us off in talking about where that goes wrong?
Sure.
So I think the blip to start with is one that we've called blanket frontier model usage.
So to your point, that's just, oh, okay, if powerful is good, more powerful is better.
So I should be using Fable for everything or Astra for everything.
This can be nice because in a way it puts less burden on the programmer to work out what their needs actually are because if they just hit it with a big enough hammer, it'll probably work.
That's both not always the case because there can be advantages to other kinds of models that are not simply how powerful they are, but also it's not very prudent in a time when organizations are worrying about how much their inference costs them because, you know.
The big expensive toy might be the one that is the most fun to play with.
But if everyone does every task, however simple or whatever its nature, with the most expensive model, you're unlikely to control your costs over time.
And switching between different models and different contexts is probably one of the big levers you have to meaningfully keep costs under control while still getting the job done.
I think also part of the reason we've had this explosion of LLMs and AI-based tooling is because the frontier labs and the big labs are starting to build very powerful general purpose models that can do anything without the need for context-specific training, task-specific training.
And, you know, that's really powerful because it means you don't have to do any work, you can just pick something up and use it.
But that doesn't mean that there aren't benefits, particularly for...
larger companies and larger users to look at models that are a bit more fine-tuned, that are a bit more domain or task specific.
And often we find that they can get similar or even better results for a fraction of the cost.
You know, things like GLM, recently releasing GLM Legal, which is a version of the GLM open source model that's been fine-tuned for legal tasks and now outperforms all the frontier models on legal related.
tasks because of that layer of fine tuning.
So I think we've all been very happy to see how far general purpose models can take us, but certainly using a general purpose model for everything all the time is not always going to get the best results and certainly not going to get you the cheapest results.
And to build on that, I think GLM in particular was one of the model families that we discussed as getting creditable.
coding performance without having to go to the big name Frontier Labs.
So there's certainly some distance between open weight models like GLM or, you know, Kimi K3, et cetera, and what you get from the Frontier Labs.
But it's not such a big difference.
And certainly I think that, you know, in my personal coding experience recently using GLM 5.4 to get things done, I didn't.
feel like I was operating in a second class way.
It was actually something that was really, you know, not quite at parity, but certainly in the same ballpark as the Frontier models.
So you shouldn't assume that you've got to either pay up big or stick with something second class.
There are now options and probably the larger and the more canny organizations are looking to balance their inference mix between what they get from Frontier models or the smaller models from Frontier models or locally hosting things themselves or even using the Neo clouds that are springing up to offer open-weight inference at a more reasonable price than you might get from the big labs.
How are things going with local coding models?
And I'm talking about things that run on your own host these days.
What kind of tasks make sense for that kind of thing?
Because that's definitely getting a look in as well.
Yeah, we've got that on the radar.
I would say that we shouldn't...
Local...
models are more popular and more feasible, but they're still kind of large and not necessarily practical unless you have some beefy hardware in all time.
So I think we should encourage people to separate local models, which are interesting and you should have a look at, and maybe for simpler tasks that can be broken down with less kind of context or less processing of a large code base, that's something you could consider.
But to separate that class from open weight in general, it's not simply running my laptop or running on the servers of the big labs.
Open weight is often, I would say the dominant modality of open weight is either self-hosted in your own data center or through NeoClouds.
I've definitely seen this misconception where people think that because I guess open source often means running stuff on your own machine, that open weight needs to mean that as well.
And it's just one modality.
But Effie, I know you've been experimenting with a mixture of all of these different kinds of things with your own work.
So what kind of tasks have you seen split up in a way that can make the best use of the tools that are at hand?
I mean, I certainly think we're starting to see cases where local models are capable of doing tasks where the data or information involved is highly sensitive and you just don't want to take the risk of sending that off your laptop at all.
I've started seeing some...
AI based workflow orchestrators where you know, you know, you've got an LLM call here and LLM call here, then sends the result here, where you can say this particular step should run on my laptop with this model because it deals with looking at a large amount of sensitive data and the results will get sent off.
But as a result, you're keeping hold of that large data store and stopping that ever, ever leaving your laptop.
And I think that's one thing we're going to see more and more is local AI is good enough to handle all of your sensitive private data.
While we're talking about different models and proliferation, we need to talk about decision models like JEV.
I think that's the elephant in the room here.
Chris, your face did stuff when I said that.
What are your takes?
Well, yes, a very good point.
And I think the launch of JEV just before our Doppler meeting was certainly something that was causing a buzz.
in the room because it does maybe shift the calculus of what you should use for what.
So if folks don't know, JEV is a so-called system one model from the idea of thinking fast and slow.
So it's able to make decisions at much less expense and with much less time than if you attempted to use a full LLM to make a quick decision.
So we are seeing people even...
composing decision models with other models where the decision model can make maybe a decision on what grade of model is appropriate to do the real work, for example.
Also, a lot of applications in business contexts of LLMs are really...
So misusing the generality of LLMs, a lot of the time, actually what you need is a decision between a few predictable discrete options so that you can take the next step in your application.
And if you're using an expensive LLM, you're not really making the best use of the technology because there's a lot of power there that isn't really being brought to bear.
And if anything, the extra power of a full LLM might allow it to hallucinate or coming up with answers outside the scope of what you're asking it.
So certainly in those kind of...
user-facing applications, decision models like JEV are getting folks very excited.
I think where we're seeing cases either where kind of general purpose, often frontier models have been used for very simple tasks like sorting your email into different folders, which, you know, it makes so much sense to use something a bit more specialised, a bit...
a lot more cheaper to use like JEV and other decision models which there's been an explosion of including ones that run quite well locally.
It's really, it just makes, it's just so much more practical and because of how much cheaper and faster these models are we're also seeing in cases where people to love to shove an LLM but couldn't do so.
Now there's a whole new world opening up to them, things like you know having a decision model review every line of a 400,000 log line of having a decision model review every line of a 400,000 line log file to highlight ones that are suspicious or that show a certain pattern or that have some other criteria which you can't necessarily build a regex for but it's simple enough that any human eyeball in it could say, yep, that one, that one, that one.
And so we're starting to see, yeah, these new possibilities open up thanks to JEV and the fact that we have these general purpose classifier models now for that kind of more obvious classification.
I should mention that we decided to blip decision models rather than JEV because with the speed of the space moving, there are already other competitors that are built on the same design.
I guess JEV is the one that sparked the initial excitement and it's got maybe one of the better names being named after.
Jevons paradox, which really encapsulates the kind of thing that Effie was talking about of being able to use models in new contexts, but it's already not the only option.
So we didn't want to hone in just on one tech when it's really a space that's opened up.
It really has.
I saw a blog post last week where someone had compared a whole variety of different decision models and gave them all the trolley problem, which was definitely...
You know, an interesting way to benchmark that kind of thing, I suppose.
I would not recommend using the trolley problem as a thing you get a machine to make a decision about, but in case you needed to know where they sat, that is something you can look up.
And this really points to maybe a challenge to the way we build the radar these days, which is that developments come out, evolve, terms shift.
in a time span that's shorter than the six months that we normally do the radar.
So we're maybe used to working in a tech industry that moves fast, but that a sampling rate of once every six months gives us a good look.
Certainly, I think we found even on this radar encountering blips where the thing the blip would be about wasn't out the time of the last radar, and we're already wondering whether it's too old to bother with.
So it's an exciting time to be doing something like a tech radar.
Another interesting thing about that, I suppose, is semantic diffusion.
Because you start to talk about something and you have to stop before you can even get into the conversation because you need to define the term because it has shifted so much even within the space of three or four months.
And the next theme is AI factory, which is one such term.
There's a lot of stuff that goes underneath the heading of AI factory.
Effie, I'll throw this to you first off.
What is an AI factory and according to whom?
I mean, that's a great question.
As you said, there's a lot of semantic diffusion, but I think it's important to distinguish an AI factory from something like a software factory, because in one case, the output is AI, and in the other case, the output is software that's built using AI.
And I think that's a key distinction to make, because there's a lot of overlap between those things and a lot of people talking about both of these spaces.
But fundamentally, one is about how you build and run things with AI, and the other is how you're about...
build things using AI and get software from AI.
Chris, have you got your own definition of AI factory or what have you seen it shift from and to?
Well, I guess of all the terms that is the most overloaded, AI is the most overloaded and generic and has been so for many decades.
But the two kind of semantic clusters, I would say that...
AI factory tends to refer to.
One is this idea of an inference mix, just to call back to our earlier comments about open-weight models between being able to run open-weight models in your own data center, being able to mix in neoclouds, or being able to pick between one or more frontier providers.
So that idea of almost like a token grid is one of the things that people mean when they say AI factory.
And I guess the factory metaphor is...
attractive because people imagine that means, you know, predictable and optimized.
I dare say probably most of the people who are using the term haven't actually visited a real factory, but I also assume that must be how they work.
Otherwise, modern civilization wouldn't continue.
The other kind of main path of the term AI factory that I see is effectively running.
LLMs and AI stuff on Kubernetes, if I could be really frank about it.
So in that second sense, it's less about the kind of the grid of where you're getting your inference from and more about the way that your platform engineering is having to evolve to support the massive demands that both AI coding and customer facing AI apps are going.
So, you know, we've seen a lot of...
blips in this area, but we've got KServ and Cato, for example, on model serving and everything in that family because it's a whole new dominant use case that needs a bunch of tools to be invented very quickly to make it convenient for the people in your platform engineering organization to keep up.
And there's also a lot to tie in here with the token economics issue that everybody's been dealing with as well.
It's not just about how you get to, I mean, the theme that we were just talking about, using the right tool for the right job, but also How to do it more effectively, how to keep track of that cost over a variety of different providers and all of that kind of thing.
That gets bundled in here too, right?
Yeah.
So for example, Lite LLM, maybe despite some visibility of security issues that it might have had over time, I guess that's the hazard of being a popular new product.
It provides routing and abstraction across different models.
So that gives you a way when you do want to external to your organization, pull in inference or connect, you know, you need these kind of abstraction layers to give you that control or also things like, you know, visibility or support the token version of FinOps, et cetera.
If you don't have those layers of abstraction in place and you're just connecting applications directly to frontier providers or any other API, then it's really hard for your organization to manage what is happening.
And when that...
resource of inference is rapidly becoming a significant item on your spend.
It's really something you need to have.
But it's not just about expenditure.
It's also just really about control and management.
I'm just trying to think of anything else I wanted to ask about the AI factory, but honestly, that's kind of it.
Well, maybe I can say what I guess we might have a little bit less to say about the AI factory, but that's because the whole point is it's trying to make a space that is exciting boring.
That's why we're using the metaphor of factory.
It's not supposed to be exciting.
Really, the exciting thing is supposed to be, I don't know, the kinds of coding that we can enable that we couldn't before or the kinds of applications that we can make customer facing.
You might say that the whole AI factory theme is about supplying the kind of infrastructure you need to make inference boring as possible.
So it's good work.
And I almost hope that...
We don't need to have this as a theme in upcoming radars because we see a kind of standardization and stabilization in this part of the space so that maybe we can get on with doing the more creative stuff.
And by more creative stuff, I want to take us to our last theme, which is harness engineering at scale.
And that's the one that I doubt we're going to see any particular standstill on anytime soon.
That's my view anyway.
And another thing where we've had a little bit of semantic diffusion as well since we have landed on this term, what is a harness these days?
Effie, do you have a take?
Oh, I mean, I wouldn't want to try and give a comprehensive definition.
But one thing that's always seemed interesting to me is that a lot of people...
at least used to use the term agent when I was thinking of a harness because they would for example refer to clawed code as an agent and to me the thing that runs inside clawed code is an agent but the tool that's running the agent is the harness and the harness is comprised of everything around the agent that orchestrates the agent and allows it to do what it needs to do.
It's the actual coding, the actual tool that calls the inference, it's the skills and plugins and MCP servers and capabilities that are made at the disposal of the agent.
It is, yeah, it's effectively all the interfaces and knowledge that the agent has at its disposal to enable it to do work, I guess, would be my elevator pitch.
I think I'm yet to read an article about harness engineering that I feel has consistently used the same definition throughout.
I think it's quite common for there to be contradiction even literally within one opinion set of one person.
Where I think maybe my take on this would be that there's a meaningful difference between who is maintaining and creating the harness that should be part of our analysis.
So it is not the same thing when there is code that I rely on to get my job done written by someone who works for OpenAI or Anthropic compared to code that I and my team work on or markdown or whatever else, or markdown code context that my organization contributes to together.
I think you could talk about them all collectively as being the harness because there was one particularly well-known definition of harness as everything that's not the model.
But I think if you can't distinguish between stuff you're relying on the provider of the coding agent doing, stuff you have the control.
to do yourself and stuff that you might have to coordinate with your colleagues to come to agreement on.
And that's really ignoring the main material distinction that will affect programmers day to day.
The one thing I've seen most people agree on is that, and this is going to sound trite, but it's true, that harnesses are important.
Earlier in the same conversation, we were talking about how the harness is the thing that really needs to hold a lot of the guardrails and the context and keep things on the straight and narrow when you're delegating to long running agent sessions.
And all of these things have to come in somewhere, yet the harness is also going to shift context and maintain a ship and all kinds of things depending on which one you're using for which project, in which context, at which point in time.
There's also a whole lot of things being built on top of and around harnesses, regardless of what you think they are.
And I know, Effie, you proposed, I think, agent plugins as a blip for...
this as well.
And we're talking agent plugins here, but this is also a harness thing.
Is this a harness agent diffusion thing?
How did that go?
And what's the plugin ecosystem like?
Yeah.
I mean, so agent plugins is a new open source standard that a few of the major Frontier Labs and similar have adopted.
And it's effectively a standard for, version one at least, is effectively a standard for sharing.
defining and sharing MCP servers and agent skills.
So, you know, a while ago we had the agent skills standard come out and that said you've got a markdown file, it might have some other markdown files as references, it might have some scripts that can be run and an agent should read it by looking in this directory, but we never had a way to encapsulate those and be able to share those around.
Agent plugins doesn't solve all of that.
You still need something like a marketplace, which is another thing you can read up about on the radar, but Agent Plugin is at least a step forward to say, you know, here's the 10 skills that my team have developed for everyone in our part of the world, or here is the eight skills that are relevant to this CLI that I'm downloading from this open source project, or here is the, you know, definition of the three MCP servers you'll need to get your agent up and running with all of the ecosystem that we're using here at the company.
And yeah, define that in a standard way so that...
I can install it just as easily in open code as I can in codecs because before this there has been effectively no standard in each.
Each of the different coding agent harnesses have effectively defined their own definition of what a plugin looks like, how you install them, what they should define, etc.
I think in the write-up for this one, you've drawn a parallel with platform engineering from a couple of years ago that as soon as something comes onto the scene and people are trying to figure out how to build it into their workflows, how to build with it and how to get the best out of it, they're trying lots of different kinds of things.
We get multiple versions of the same sorts of ideas.
Some stick, some don't.
Is it the same thing here?
Is it...
worse because people are vibe coding their own now and there's quite a lot more of it going on?
Is it better because we're getting better clusters or more variety?
What do you think?
So I think when we talk about platform engineering, sometimes people think that means server infrastructure because that's conventionally what's happened in a lot of organizations created a platform organization by renaming their infrastructure organization.
But I think here what we're seeing through enterprise harness shared assets across the organization is something that behaves like a platform in that it is something that is created once and then benefits a whole bunch of other people, but isn't runtime infrastructure.
So in that sense, in the sense of maybe our listeners might be familiar with the book Team Topologies, in that sense of serving a bunch of people by providing something that they can engage with as a service rather than have to go and...
log a ticket or have a conversation in the heat of the moment.
That's where we see the kind of harness engineering, a scale resembling a kind of platform.
I have to say, I don't think that I see many clients doing this effectively or well yet.
What we're seeing right now is the pain point where you have lots of different teams who are all solving almost exactly the same problem.
Maybe even they're solving...
problems around testing or backlog decomposition, but also problems about local knowledge.
How do we want to architect in this organization?
What are the requirements to sort out a path to production in your particular company?
And so the first time, it's fine.
A team solves it, puts it into, mark down a script or something else.
The second time, well, they can probably cut and paste it and it's okay.
But what you then see is that over time you have...
problems both in maintenance because the context you've created and embedded in a project represents a point in time but also you it's really just like it's copy and paste sharing it's not a scalable solution so i don't think that though uh if he's quite right to point out that we have emerging technical standards to make it possible this is really a social and organizational problem and i don't think I can claim to have seen many people who do this well yet.
However, I am convinced from the beginning chaos that we're starting to see as people kind of try to share different bits of context, that it is something that will become very important for organisations to solve.
And I think similar to platform engineering and internal developer platforms, you know, in organisations who don't have that maturity as...
Chris said, we have a lot of reinventing the wheel.
Every team is having to build their own CI pipeline to do the same, you know, eight checks on their code before they get to prod.
And then compliance comes along and rolls out a new tool and you do it with that.
Security says, oh, we're shifting from this thing to that thing.
So suddenly you're having to overhaul your pipeline.
And over time, developers spend more time building that infrastructure.
That is a common concern.
rather than actually building the software, which is the unique, valuable thing.
And that creates a lot of waste.
And also just in a lot of cases, it's not what we want to do.
We're software engineers.
We want to be building software and not spending all our time tinkering, trying to get our agent to work, to integrate correctly with the new code scanning platform that security just mandated.
We'll open a can of worms here, I think, with this question, but I want to ask it anyway, which is that, you know, all architecture and all...
you know, infrastructure and platforms are opinionated and they push something of the intention of the designer onto the users.
My feeling is that this is extra true of things like harness engineering or whatever we want to call it, because it's not just pushing that structural intention.
It's pushing semantic intention on top of things.
And I feel like that is something that we are yet to reckon with as this pattern rolls out.
What do you folks think about that?
So I think that people, are not quite realizing that there is both the technical artifact and the understanding.
And they think that by manipulating the technical artifact or pushing that on people that it automatically transmits the understanding.
So certainly I've seen a lot of people on day one, they use a coding agent, get excited.
On day two, they set up their own kind of system of work and divide things.
And then on day three, they...
publish it or try to make other people do it.
And I think the urge for that, which is very powerful in this, you know, I've now got this new understanding, I want to share it with other people, is quite laudable.
But there's a bit of a problem where if the artifact outpaces the understanding, then you're going to end up with friction.
So I think Effie mentioned earlier the problem with taking wholesale skill libraries that might be open source and chucking them in your project or maybe in this case since we're talking about harness engineering at scale spreading them around your organization but just having copied or made available that collection of workflows and skills doesn't automatically make everyone who's received that technical artifact fluent in the concepts behind it so it is important to share but i think it's also important to share ideas and help people gain expertise so that they can get the most out of these things.
And I think it would be very tempting if you were someone at the center of a technocratic organization to want to focus all your time on pushing out markdown files or scripts or whatever as a way to maybe centrally influence and improve the development of your organization.
But I would definitely say Don't neglect the fact that for those things to really be effective, they have to be wielded by people who get why they're doing what they are and can also apply their own context and manage exceptions.
Particularly because we're talking about...
non-deterministic actors here who are prone to reward hacking.
If you're mandating certain checks, certain steps, you need the human to not only understand that those exist, but when they should be applied and what the purpose of them is.
Otherwise they're going to blindly accept the AI output and the AI might easily, you know, it's the case of, oh, the unit test failed.
So the agent deletes the unit test.
So there's no failing tests.
That's the kind of thing where you know, and a human who fully understands the context and the reason we have the tests is going to say, no, that's not good, go back, you know, restore the test, fix the failing code.
But a human who's just blindly running some of these artifacts produced by another team without understanding the workflow, the intent and what's aiming to be solved there might just readily accept that change from the AI, which is again coming down to that kind of cognitive surrender and human in the loop theater.
I really like that all of the four themes in this episode have been about AI and LLMs, and we've spent most of our time, I think, talking about people, which is where I think this should be at.
Which themes do you folks think will still be strong here in a year, apart from presumably humans as the meta theme?
Effie, I'll start with you.
I mean, I think, as Chris said, AI Factory is one that hopefully we get a degree of...
solved or standardized or at least it becomes as boring as it should be.
So I wouldn't expect that to stick around unless something majorly changes.
I do think that there's a lot more to be explored around putting agents to work and multiple agent orchestration because, you know, we talked about some of the local innovations in that space, but there's still the big question of cloud agents versus local agents and still a lot of rapidly developing technology to enable you to do...
run multiple agents in the cloud, for example, or run agents in a context that they're visible to multiple members of your team.
And I do think that harness engineering is broad enough that there's always going to be something to talk about there.
One of the big clusters of proposed blips that we dealt with the other week, which we haven't even really talked about here, is...
a lot of different frameworks and tools designed to solve the issue of memory and context and knowledge, things that build local knowledge graphs that provide tools for an LLM to access an indexed catalogue of knowledge or an indexed abstract syntax tree of a code base.
And that problem of memory and context is still far from solved, and I think it's going to continue to be.
a lot of innovations in that space.
So harness engineering is not going away anytime soon would be my take.
Yeah, I'd really agree.
I mean, harness engineering at scale is about large groups of people cooperating to get stuff done.
And that has been a challenge of society since society became a thing.
I don't expect that to become redundant anytime soon.
Very possibly we might stop talking about it as one thing because it is such a...
potentially large area starting to share context artifacts across projects and provide guidance etc that maybe we'll have a little bit of a more in focus view of it so we'll be able to pick apart the different parts next uh next um next time uh maybe i'll also mention the one that uh i don't think we've discussed yet which is around frontier models i think that might also be one where it will continue to be massively relevant because we should continue to see innovation and models we'll see more possibilities of different ways to run models or different levels of models.
I'd hope though that we don't have to say beyond frontier models in the future because really the idea that let me just pick the shiniest toy of the set and use that, that's really quite a sign of immaturity in our profession that we...
have been doing that and i'm sure i've been guilty of you know using a outsized model to sort my emails or the example that effi gave earlier so i don't mean to project it to other people but i hope that our fluency with choosing different tools among this set raises so we don't have to say beyond frontier models but i really hope that we're talking about the next decision models type innovation next radar because i think there's a lot more to to come I also just hope that we do continue to have such a plethora of choices when it comes to models because we're starting to see that those open-weight models are getting good enough.
Some people are getting quite concerned.
We're having Anthropic jumping up and down about the cyber capabilities of some of the latest open-weight models and suggesting that they should be curtailed.
We've already had the US start to apply export restrictions to frontier models when it comes to things like usage by China.
And we're now hearing...
as of time of recording, the first memories that might go the other way and that China might start cracking down on what gets released globally by the Chinese labs that have been putting out a lot of the leading open weight models in the near future because of that kind of one-sided capability gap at the moment.
If you've been listening to all of this and you're wondering where you can get more of this, where to keep up, I've got good news to you.
First thing is that Chris Ford is writing a book.
Another book, Agentic Engineering at Scale.
Yeah, that's the title.
Which will be with O'Reilly out early 2027.
And you can find that at O'Reilly.com.
Anywhere you get books, I believe, that will be a thing.
But also, volume 35 of the Tech Radar will be out.
At the time you're listening, I'm fairly sure.
ThoughtWorks.com forward slash radar.
Publication will be in early November.
You can hear about all of these themes.
You can roll down into the smaller details.
You can see all of the stuff that people have been talking about and judge it for yourself.
Effie, Chris, thank you so much for joining me.
It was really great to have you here.
Yeah, it's been good.
Yeah, cheers.
It was a great chat.
Thanks very much, Lily.
And join us next time.
on the ThoughtWorks Technology podcast.
We'll be here to have more conversations about this and more.
You can find us anywhere you get your podcasts.
Like, subscribe, all that jazz.
We'll catch you next time.