# Building Self-Improving AI Software Factories

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-10-13

## Transcript

We notice that agents tend to become crazy with comments.
The more you use the factory, the more time it passes, the longer the comments become.
You need to shift your mindset to thinking of a software factory as something that your code, context and data is part of.
It is not a separate thing.
It's not a separate entity.
It's not some sort of like magic black box that does stuff for you that you can get angry at when it doesn't work.
No, your code, your context, your data is also part of that factory.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts Guy Pajani and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
Yesterday, you were writing code.
Today, you're prompting agents.
Tomorrow, you'll be running a software factory.
Agentic Coding is moving fast enough to give anyone whiplash.
This June, AI DevCon London sold out.
Hundreds of developers, engineering leads and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify and GitHub cut through the hype and get to what actually works.
This November, we're bringing it back to New York.
Bigger, bolder and full of people actually building the agentic future.
Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams or setting the strategy.
As a podcast listener, you can get 15% off your ticket using promo code POD15.
We have so much planned and we can't wait to have you join us in New York.
Hello and welcome to another episode of the AI Native Dev.
And today we're going to be talking about all things agents, harnesses and specifically loops.
And in particular, feedback loops.
We're going to talk about what they are, how you can create them and how you can actually benefit from feedback loops in your organizations.
Joining me today is Maria Goronova, who is actually a repeat guest.
So welcome back, Maria.
How are you?
Thank you, Simon.
Good to be back.
And last time we spoke, you talked to us about abstraction adherence, which is how agents use the right libraries, which is super interesting.
But things have moved on.
Oh, my gosh, so much more.
So much.
And it was in this very office that I heard you speak about all things loops, feedback loops and software factories.
opening questions that you asked the entire group was, what is a software factory?
And I think it means different things to different people.
I'd love to hear from you.
What does it mean to you?
And maybe even, you know, how did it differ from person to person?
The takes were very different from each other.
The responses for sure, everybody had...
a different concept, everybody had a different angle at which they were looking at it.
Some were thinking only about automation, others were thinking about particular designs, about particular architecture.
To me, I think the best way to describe what a software factory is, kind of like go through history really of how agents have changed in the past two or so years.
or less than that.
I was going to say two years is a long time in the world of agents.
No, it's not two years, is it?
I don't know.
It feels like it's been decades, right?
But actually, it's been such a short time.
And I think, you know, software factory is something we have been building towards in those many decades.
So to me, a software factory is something where we have repeatable workflows, we have shared execution environment, and we have feedback loops.
And feedback loops are ways for the factory to maintain itself, to improve itself, to make sure it's operational and it doesn't degrade into quality.
Because once we go into this world of...
Everything is done by agent.
We are supervising the tasks that go into the agent sometimes, but we don't really supervise the code.
We don't really go into the details of writing code or even reviewing code at times.
Things deteriorate very quickly.
It's interesting you didn't mention ticket to PR.
So if someone raises an issue, a PR is created.
But that's kind of like a use case of the factory rather than what the factory is, I suppose.
Yeah, I think that is one single workflow, one single automated workflow in a factory, which happens to be, yeah, I want to write a linear issue or I want to write a Jira ticket or something on the cert and I want this to be then implemented and the PR to be opened and even merged automatically.
But there is so much more to this, especially, again, when it comes to maintenance, when it comes to...
improvements that are not necessarily, you don't need necessarily a human for them.
Instead you just churn through a lot of data.
The agent churns through a lot of data and then it can suggest improvements and implement improvements that make the factory better and keep it operational.
As part of your session, you went through a brief history of agents and the various levels that they went through over the last, what, 30, 40 years maybe?
So you started with V1, just an agent, no context, or sorry, rather, no control over their performance, unless, of course, you own the agent's code itself.
What were the big bottlenecks before context with just an agent?
I think you just said it in the question basically.
So if we only have an agent that we don't own, right?
So maybe we are using cloud code that is closed source or some other closed source agent.
We can't really do anything to improve the behavior of that agent if we don't have context around it, right?
If we're just prompting it and that is all that happens, There is not much control we can exert over that agent.
So an agent without context in a factory, for example, from an autonomy point of view, from a trust point of view, it's very unlikely to do what we want it to do, so very unlikely to actually be of any use.
I think so, yes, because everybody has a different code base.
Everybody has different requirements.
Everybody has different tastes.
So you can't really have one agent that...
simply based on a prompt without anything else around it, can behave in a way that it's compliant, high in quality, you know, things like that.
And then you moved on to V2, which is agent plus context.
What did context really unlock for us then?
And I guess what's still missing from that?
So context by context here.
I mean, things like at the beginning, it was, you know, the Claude MD.
Claude MD, that it was agents MD.
Then we had skills.
Also hooks are also a type of context.
All of these sort of documents that lie around in your repo, especially the ones that are specifically for use by agents.
Yeah.
And...
I think what that unlocked is that now we have control over our experience inside of a particular repo.
You can share it with your team, right?
Like if you have CloudMD, you can have your personal one or you can have one that is in the repo that is shared with the team and it kind of like defines how this team wants an agent to operate in that code base.
And of course, you know, with skills that becomes more sophisticated because now you have different workflows that can be for code and so on.
I think what is missing is that here still it's all very manual and there is no good way of evaluating this, right?
Like how do I answer the question of is my CloudMD good other than vibes?
There is no really established mechanism for doing this and also if I wanted to improve the CloudMD I would just go and manually kind of like vibe something in it or At best, ask my agent to vibe something in it based on what?
More vibes.
It's just there is no really principle.
So I guess we have evals, perhaps, which is maybe more of an offline test, more of a static test, perhaps.
But then perhaps this is where harnesses kick in and loops.
So you mentioned a harness as the agent, the context and a feedback loop.
Tell us what that actually now unlocks.
So feedback loop.
The way I think about it is really it's an agentic workflow, like everything else in this concept.
But it has three distinct stages.
Stage number one is about analyzing some data stream.
So you provided it with some data, let's say, I don't know, the comments that people have given to PR.
to PRs.
So I take all of this data and I give it to the agent and I ask the agent to analyze it.
So the second step here is finding, based on this analysis, finding problems, potential problems or potential inefficiencies, just things that can be better.
In this particular example, if we're looking at PR reviews, maybe we see that the same PR comment is given over and over again to someone's PRs or a factory's PRs.
Like a code review or something like that?
Yeah, like a code review.
It could be anything, I suppose.
Yeah, it could be that there is always, for example, There is a particular way developers in this repo want a class to be written or documented.
And every time the agent doesn't do that, and every time the developers need to comment on it.
Something like this, just a made-up example.
So identifying this problem in the data, and then the third stage will be suggesting a fix.
And this fix could be a code fix, it could be a refactor, it could be a bug fix, or it could be context.
improvements.
So in this case that I mentioned, maybe the repo can benefit from a doc that actually describes how we want the class to be defined in that repo.
And I guess from the data source point of view, let's start with the data source.
What makes a good data source?
Does it matter if it's a human data source or a data source which is actually coming from a set of logs which are generated from another agent or another app or something like that?
It can be anything.
It doesn't have to be human generated.
It could be the agent session logs where we can see agents.
So one thing that we have seen in our internal dark factory, for example, is that agents are able to analyze their own session logs and see places of friction where they...
lose many turns to figure out how to use a tool, for example.
So that would be a potential place for improvement where the agent can now have some context that helps it not have to lose those turns.
But I don't know if I can answer what makes a good data source.
It has to be a bit...
It sounds like anything that displays, concisely displays friction or really has a kind of like a root cause.
of something that can be improved.
Is that fair?
I don't know if you can give it a definition like that because yeah, like it's not obvious.
Sometimes it's not obvious how data source will, with the agent session logs, I didn't know that will happen.
That was kind of, it was cool that it happened and in hindsight, we can explain it, but it's not that.
I thought, oh, let's introduce agent session logs because it's going to identify exactly this kind of problems.
Yeah, yeah.
So the data sources, we should just try multiple data sources.
Yes, I think so.
I think it's a matter of experimenting and seeing what comes out of it.
And yeah.
And what judges the analysis?
So you talk about the analysis.
It's essentially trying to identify issues that repeatedly occur.
What does that?
Is there a specific agent that we use for that?
I know Jev has just interestingly come out.
That's possibly an option, I suppose.
So it depends.
In this particular case, I was talking about feedback loops that kind of receive data over many, many changes, like many PRs, for example, many agent sessions, and so on.
And for that, you just...
The feedback loop is an agentic prompt and the agent works through that by looking at the data and doing certain steps and figuring out things.
But we also have other kind of feedback loops that are more tightly scoped to an individual change, which we call the inner feedback loops.
Those ones...
For example, we can have something, speaking of Jeff, something like verifiers, TESOL verifiers allow you to define a bunch of criteria that you want a change or your code to meet and then use an LLM as a judge approach to verify if that criteria is met.
And you can use Jeff for that, for example, or you can use another model.
And why would I use that?
over what a code review already provides?
I think it's...
I wouldn't say you should use it over what code review already provides.
Or as well as, rather.
I would say as well as, yes.
Like, it is...
It is an additional dimension to it because code review is a little bit more open-ended, a little bit heavier, I guess, even if you do it with an agent.
If you require human time, then it's obviously much more expensive than running an LLM as a judge.
But even when you run an agent, it takes a little bit of time.
It's open-ended.
Verifiers are very tightly scoped.
You've defined specific rules that you want.
your code to follow that are not something that is easy to define with tests or with linting.
It is a form of linting.
I would actually classify it as a form.
It's deterministic in terms of the triggering of the verifier?
Yes, we do trigger the verifier on every change.
So when a PR is up during CI, we would trigger the verifiers.
But the output is, I wouldn't call it deterministic because it's...
It's still done by HLMS, yeah.
Okay, got it.
So you talk about the inner loop.
There's also kind of, I think there's middle loops and outer loops as well, right?
Talk us through what the other loops are as well.
Yes.
So just to clarify, so the inner loop we define loosely as something that operates on a single change to the code base.
The middle loop is more like the examples I was giving before where we have many, many changes to the code base and we get some data from them.
We get the agent logs, the PR comments, but like from all of these changes so that the agent can in a sense generalize by looking at many, many examples across many sessions.
So that is how we, again, loosely defined middle loop.
Outer loop...
is more about production signals.
So if you want, for example, self-heal.
If the service that is running the Dark Factory fails at any point, we want it to be able to detect that it failed, see the error, and then self-heal by fixing that.
So that would be like an outer loop.
Another one would be if we have a certain type of Customer signals, for example, I don't know, daily active users, right?
If we see that going down, we want something to investigate what's going on and find the issue.
So that would be another outer loop.
And what are the triggers for the middle and outer loops?
The inner loops is obviously, the trigger is the thing that happens, maybe it's a pull request that gets raised, so the loop runs on that.
Is it just scheduled for the middle and outer loops?
It depends what exactly you need and what your data is and so on.
For most middle loops that we have, the trigger is a schedule, yes.
So we run them every day or every week, sometimes every month for the very heavy ones.
And they will just continuously kind of monitor the code base and make sure that...
Everything is fine.
The outer loops, they're more likely to be triggered on an event.
As I said, if the service is failing or if some metric is going down that we want to go up, something like that.
Cool.
I'd love to also, I mean, the picture looks wonderful in terms of these loops.
It's self-fixing.
But I presume it doesn't always work perfectly.
Where are the kind of areas that if someone were to create a loop for the first time, they might expect it to go wrong?
Perhaps it's the identification or maybe it's the data source, actually.
Then maybe it's the analysis or the judging on the data to be able to perform that.
that change, maybe it's the change itself.
What are the kind of areas that you would say are the hottest, the hotspots for problems?
So if you wanted to create your first feedback loop, I would say the most important thing is to be very clear with your intent.
You need to be very clear what the data source is, what you expect or what you want the agent to look for in that data source.
And what do you want it to improve based on that data source or based on the findings from the data source?
If you're vague, I think that is probably the place that will be a hotbed for disaster because if you're vague, if you just say, oh yeah, look at my past PRs and improve my repo, this is not...
This is not clear intent.
It turns it into a God loop of trying to fix everything and it actually needs to fix a very specific thing.
But also, the fixes are not going to be deep.
There will be a lack of depth and it will just make up things that maybe are true but not important, that sort of thing.
You want to create depth by really clearly explaining where to look for problems.
For example, simple middle...
feedback loop we have is one that we call comment diet, because we noticed that agents tend to become crazy with comments.
They will just write a lot, like lots and lots of comments and the comments become longer and longer, because the next agent sees long comments and then he thinks it needs to read.
It is, honestly, it's mad.
We've had some crazy files lying around.
So the comment that died feedback loop would go and just the data source is very simple.
It's just the code base.
There is no additional data source.
So just look at the repo and run some scripts to kind of count how many lines, comments versus code there is.
Identify hotbed files that have...
too long of comments, the comments are too long, and then fix that.
Just reduce the lines of comments in a way that doesn't degrade performance, right?
So this is more concrete.
Now I've reduced it to something very, very specific, and it's not going to touch your code, it's not going to touch your architecture, and, you know, big changes like this.
You know that it does just that one thing, and it's focusing all its attention to do that one thing.
And then you can define a separate feedback loop to, I don't know, look at the architecture or the design or something else that you want.
Yeah.
So in terms of feedback loops there, would it be, is it fair to say there's almost like a couple of feedback loops there?
One which is maybe an inner loop that when changes are made, comments aren't too verbose.
And one, or rather, that's maybe a skill.
point on the inner loop.
And on the middle loop, maybe that's more of a scheduled thing doing an audit, essentially, of your code base to check that that's okay and that's happening.
Is that the right way of looking at it?
I think for some things you can divide it this way and have both inner and middle.
I don't know if outer will make sense here.
Probably not.
But yes, you can have inner and middle.
It depends on whether you expect this to be a drift.
or not this kind of behavior.
Because so in this particular case, we noticed that the comment bloat is adrift.
The more you use the factory, the more time it passes, the longer the comments become.
And yes, we can have some gates on the inner loop level.
that check if the comments are too verbose and stuff like that.
But my suspicion is that even if we have those gates, it will still happen over time.
And just to find what drift is, is that a difference between what we expect to happen based on the context and the reality in the code base?
Good question.
I would define drift as drift from some ideal state that we...
hope for.
And the drift happens in human-maintained code bases too, like even before agents, even before all of this.
When many, many developers are contributing to a code base, over time, small errors compound.
And then at one point you have to do a refactor or you discover a big bug or something like that.
There is a very good example for this is actually the rule of three.
You know the rule of three?
Tell me.
Well, if you have to write code once, obviously you write it once.
If you have to write it twice, you write it twice.
But if you have to write it a third time, now is the time to introduce an abstraction.
Right.
So it's kind of like this, like humans do it, agents will do the same.
Like you're going to write the code once, you're going to write it twice.
And then on the third time, you need to introduce an abstraction, but that's not the job of the agent writing the third time code.
That's a job of something else, something beyond that.
That even recognizes it's been done three times, whereas the agent won't necessarily.
Exactly.
So it is kind of like, it's something beyond.
So that's why the middle loop looks at.
many, many sessions because it needs the bigger picture.
It needs to generalize and correct those drifts, correct any spaghetti code, as I like saying, and kind of fold back the repo in a nice state, in a better state that is healthy, that subsequent agents will have a good time working in.
Hey everyone, hope you're enjoying the episode so far.
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
But for whatever reason, many of you have yet to subscribe to the channel.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
All right, back to the episode.
Let's get practical now.
And for people who want to get started, one of the things that you kind of like mentioned is getting the correct mindset.
What's the correct mindset for creating loops and thinking about loops?
You need to shift your mindset.
to thinking of a software factory as something that your code, context, and data is part of.
It is not a separate thing.
It's not a separate entity.
It's not some sort of like magic black box that does stuff for you that you can get angry at when it doesn't work.
No, your code, your context, your data is also part of that factory.
Just to give an example.
Imagine you have project A and project B that semantically do exactly the same thing, right?
When you run things through them, it's exactly the same behavior.
But project A is written well, like the code is well structured, it has good abstractions.
There is lots of context available.
There is, you know, agents in the skills in it.
There is, I don't know, hooks that...
configure the agent.
There is like all sorts of mechanisms that control this agentic behavior.
And project B has none of that.
The code looks like spaghetti.
It's the same code semantically, but it looks like spaghetti.
It is obvious that an agent will perform better in project A, right?
Very much like a human.
A human will also be more confused by the spaghetti code than by the well-structured code.
So...
This is what I mean by this.
You have control over your code, context, data.
You can use all of this as part of the factory to make the factory better because the factory is also your context code and data.
Okay, cool.
And I guess another thing is there are a couple of different types of loops that we talk about internally here at Tesla.
We have a regular loop and then we have a feedback loop.
What's the difference between the two?
Well, I guess for feedback loops, we already talked about them.
It's that idea of a loop that analyzes some data source, finds some problems and fixes them, or rather requests fixes for them.
What we call a general loop in TESL is more of a, any sort of workflow, any agentic workflow can be a loop.
We have, for example, loops that define the issue to PR, what that looks like.
So how an agent should look at an issue, read the issue, plan the change, implement the change and open a PR.
We have loops that look at PR review and can respond when there is a comment on a PR, can then trigger and...
you know, either implement the comment or maybe push back if they don't agree with the comment, that sort of thing.
And feedback loops are type of loops.
It almost feels like the definition of a general loop isn't actually a loop, but I think you mentioned it was a workflow.
It doesn't feel like it loops back on itself, whereas a feedback loop truly does.
Feedback, it loops, you know, takes information, passes it back, and actually does a loop.
Yeah, it is a little meta, I think, because when we say...
Naming is hard.
I think when we say just loop, even in those workflows that are kind of linear in a way, I think we're referring to the loop of the agent inside it, right?
Because this agent is in a loop.
This agent obviously loops through many steps and tools and decisions and so on.
So that's what we've been by loop.
And the feedback loop is like, I don't know, a loopy loop, I guess.
The loopiest of all the loops.
Yeah, I like it.
Oh, no, there is more loopy loops.
There is the meta feedback loops that now would analyze data that kind of tells you if a feedback loop is doing well and then make changes to that feedback loop in order to improve that feedback loop.
So now we are...
Managing the loops, essentially, right?
Yes, yes.
Now we're looping instead of a loop, instead of a loop.
Yeah, and I suppose that could go up any number of depths, I suppose.
Yeah, yeah, it's cool, no?
It's recursive.
That's really interesting, actually, because you don't, you know, a feedback loop, you know, you want it to be correct, you want it to be making good changes, but how do you know it is actually making the right changes?
So it's, everything needs that feedback.
Super interesting.
Yeah, and that's why, so measurement is extremely important.
That's something, I think, we haven't touched on in this discussion, but measuring the impact of changes is actually really very important.
It is because otherwise, if you're just, you know, you're just making changes, but if you can't observe that that's having a good effect, then you can't really trust that anything is happening.
Yeah.
And I was actually going to ask the next question, which was around human interaction.
So when do human, like, obviously with loops and really the goal of this is to really increase that autonomy, increase that trust.
When do humans get involved?
And I guess you can kind of like follow on to your point there in terms of like that observing what good should be and things like that.
When does a human actually enter that, being part of that judging process?
Everywhere, Simon, and all the time.
I think, I think, Many people might find it a little bit daunting that we're moving towards so much automation.
But actually, I think it's just an abstraction level.
We're still doing so much work to analyze what's happening, request the right changes, monitor if the loops are performing well or not, coming up with new metrics that we think will...
you know, reflect better if something is working or not.
All of these things, it never ends.
I don't think the factory is something that can just be built and done.
Yeah.
Like with any software, it's, especially when you make your code base, your context and your data part of it, it constantly evolves and you're part of this evolution.
And that is where the work is rather than code implementation.
Yeah.
And in terms of...
Let's talk a little bit about roles for humans in the software development world going forward with software factories.
I guess software factories, if we have 10 repos, should we have 10 software factories or are there shared components where you might have a shared orchestrator and multiple more specific components that are repo specific?
How much of this...
How much of this rule require humans to manage versus actually building the product?
Is there a role where a human will do both of those as part of their job, do you think?
Or do you think there's more of an infra role where people manage the software factory and more of an app development role where people manage what needs to be built and how it gets built?
A really good question, Simon.
I really like this.
These are all good questions.
Yeah, why am I surprised?
At Tesla internally, we actually, especially when it comes to metrics and improving on those metrics, we actually talk about several different personas.
One of the personas we call an operator, dark factory operator.
So I think this is what you just described as like an infrared role.
This is somebody who really cares about the...
main loops of the dark factory and make sure the dark factory is operational and it can service everyone and so on.
And then we have the steward role.
And the steward role, we have many stewards at TESL.
They're stewards of different areas or different projects, different repos, if you wish.
And it's kind of like the new code owner, if that makes sense.
The code owner is funny, though, because they didn't write the code.
That's the idea, right?
That everything goes through the factory.
So that's why stewards.
So they know about this area.
They are kind of responsible for this code base.
And they make sure that the dark factory operates well in their code base.
So there is, I think...
both roles need to think about the factory and think about this new way of development, but in a different, through different lenses, if that makes sense.
Whether this is one dark factory or many, I think, I think both.
I think it's both one dark factory and also each project, each repo is its own thing, but it can go through the common orchestration.
But then it can define its own, it can and it should define its own feedback loops and its own special purpose loops and so on so that it works better.
Yeah, I think that's really key as well that, you know, you can't just expect one repo to copy paste all the context and everything in terms of how to operate.
and expect it to work just as well in the next one.
Everything needs to iterate and learn from the repo it's in, which is super important.
Yeah, absolutely.
Maria, what advice would you give people who haven't kicked off their first loop but really wants to actually build, they like what they hear and they want to build a loop?
What advice would you give them?
Come to AI Defcon.
Brilliant.
Yeah, no, I think that might be very helpful.
I would say what I was referring to before in terms of how do you get going with one of those feedback loops, be very clear with your intent.
Think about what's the data source.
Think about what...
what signals, especially signals that when repeated make sense, can be found in that data source and then think about what can you improve based on this.
And perhaps focus on those pain points that, you know, what is it that your agents are tripping up on most and almost like start building the loops around those because those are most likely the areas it can loop and improve and feedback the most.
Absolutely, because then it can improve its own behavior.
It becomes self-improvement.
Yeah.
Amazing.
Well, Maria, you've been on the podcast before.
You've looped right around and been on the podcast once again.
Thank you very, very much.
Really amazing insights and appreciate you joining us today.
Thank you, Simon.
Really appreciate it.
Thank you very much, everyone, for listening and see you soon on another episode.
Bye for now.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast, it's a community.
And we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more.
And I hope to see you there.