# Autonomous Code Factories Reshape Engineering Productivity

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-07-28

## Transcript

Two weekends ago, we had quite a busy Friday, and the Dark Factory ended up shipping 150 PRs for two of us over the course of the weekend.
And that was just while we were out hanging about enjoying London in the heat wave.
Last time I looked, I think about only 5% of the Dark Factory codebase had ever had a human look at the PR and approve it.
I don't think people actually go look at code anymore because they're all just using agents.
And so that means, by implication, 95% of our codebase has never had a human look at it.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pagiani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
We just wrapped up two amazing days at AI DevCon in London.
But the great thing is that we get to do it all over again in New York City this November.
You're absolutely right.
We're going to be back in the city that never sleeps on November 3rd and 4th for more amazing sessions, really engaging hands-on workshops and much more.
Yep, all that great networking, partying, eating and drinking that you've come to expect from AI DevCon.
We think we have one of the best hallway tracks in the business and it's the perfect complement to our incredible speakers and presenters.
We'll both be in person and virtual with live streamed access to all main stage keynotes and talks.
Sign up right now for our Super Blind Bird ticket for just $100, only available for a limited time.
We're really excited to be headed back to the Big Apple.
We hope to see you all there.
Hello and welcome to another episode of the AI Native Dev.
Did you know that last week alone, TESOL raised 516 pull requests and the whole company was at a week-long off-site.
The week before, we seemed to raise only 608, so only...
90 or so more.
How do we do this?
It's all about the usage of software factories and the Dart factory.
Joining me is Rob Willoughby, who is the AI engineering lead at TESOL.
And Rob's going to tell us all about how TESOL uses Dart factories and software factories internally to create such amazing numbers.
Rob, welcome.
How are you?
Thank you.
Thank you.
Doing well.
And first time on the AI Native Dev Podcast.
Yes, it is.
First time on a podcast.
Right.
We have to do the induction, unfortunately.
So, no, there's no induction.
I was thinking of making something up then, but yeah, didn't tell you that.
So, Rob, first of all, tell us a little bit about what you do at TESL.
Yeah, so I run the AI engineering team here, which is part of the team that kind of broadly is looking at agents enablements kind of across TESL.
And so what our mandate is, is really to build and then drive adoption of a dark software factory internally that we use for kind of two main reasons.
One is to improve our own engineering velocity.
We want to ship faster, ship more, ship higher quality stuff, be able to use agents better.
But then also, additionally, we use that as a testing ground for our own theses and hypotheses about how to effectively run a factory.
We want to be able to take the learnings from there and put those into our products that we can then go help other companies bring along on the adoption curve as they start to build these factories internally for themselves.
So these numbers, they seem crazy.
Every time I talk to people and say even a few hundred pull requests for not a huge team, not a small team, but a reasonable sized team, people are surprised.
They're like, how do you deal with that?
How do we deal with things like code reviews and the human interaction and things like that?
We'll cover a lot of this.
And actually during this podcast, we're going to talk about how Tesla adopted a dark factory.
What were the sticking points?
What worked?
What didn't work?
How we overcame them.
But for now, let's actually take a look at what a good interaction looks like through the Dark Factory.
Let's now go through a full end-to-end test of two pull requests, one which requires human interaction and the other which will go end-to-end without human interaction, starting from a linear ticket all the way through to a completed pull request.
So this ticket is one that I prepped beforehand, which has to deal with some findings from an audit we did where we can accidentally allow data from GitHub repos, inject commands into the TESOL agent commands that we are showing on the website.
That just has to do with string escaping and all that kind of stuff.
Relatively simple fix, but security important.
This is going to be touching what shows on our website, so we'll require a human PR review in order to land.
All I've got to do is I've got this ticket prepped, saying it's in apps front end, so that's our front end, obviously.
And then I'm going to flip it into to-do.
Because it's been delegated to the Dark Factory, that means that the Dark Factory is going to pick it up and go do this implementation for me.
So that's kicking off the Dark Factory.
Just putting that into a to-do, the Dark Factory recognizes there's a hand over there, and it kicks off.
a new container where it can kind of like do all that exactly and so now i'm hands off at this point so the second ticket is one that is targeting our dark factory itself harness slash geeky mora that's just our own internal name for it This is another one that is around expanding out our metrics.
We aren't actually capturing which inference geo where we're running these queries against, which means that we're not able to properly analyze the effects of caching because our request could go across any different geos, which then affects cache rates and also cost overall.
So because this is within our own dark factory, we have a policy of auto-merging everything because if we can't auto-merge it, that means that we don't have the verification in place to have confidence in the process of writing code or building code.
So you can see that this auto merge label is down here.
This means that as soon as the Dark Factory is done creating it, it will go through PR cycle process using bots, of course.
No human intervention here.
And then it will get auto merged into the repo once all of CI passes and once our review agents approve it.
So I'm going to flip that into to do and we'll check back in on this one at the end of the podcast.
Amazing.
And the fact that that is...
set to auto-emerge.
If that wasn't there, presumably that then would require a human to go in, maybe review the PR and actually push it themselves.
Yeah, exactly.
So auto-merge is an optional label that just gets the bot to literally select auto-merge on the GitHub PR so that it will go in once CI is green.
Awesome.
Amazing.
So now we've raised two tickets, two linear tickets.
One which requires a human review, so the Dart Factory will do all the work and then it will say...
this is done, here's the pull request, approve it, and it will be merged.
The second one, because it's a PR that goes directly to the Dart Factory, it's a PR for the Dart Factory, that will auto-merge, so as soon as it's done, it will squash that down and merge that into the code base.
Yep, correct.
Amazing.
We'll come back to that at the end of the podcast.
So why don't we talk about how Tesla uses the Dart Factory numbers-wise, first of all, and then we'll maybe look back as to what we were doing before the Dart Factory existed.
So when we think numbers-wise, what percentage of Tesla's pull requests today...
use the Dark Factory?
Yeah, so it varies a little bit week over week.
But for the last week, we are between 65% and 70%.
Mind you, this is including pull requests that are building the Dark Factory using the Dark Factory.
So there is a lot of those, as you can expect.
We are heavy users of it internally on our team.
The last time that I looked for the production part of our code base, so the stuff that is serving the registry, our web UI, our CLI, all of that kind of stuff, we are at about 40% of production PRs were raised.
using the Dark Factory.
And all of those 40% require a human review to some extent?
Correct.
Yes.
Our policy is that we only allow auto-merge on stuff that is in our research code base, stuff that is in our Dark Factory code base, and stuff that our GTM team is building using the Dark Factory to help build out their own internal flows too.
And how about weekends?
Does the Dart Factory work while we sleep?
Oh, yes.
We'll happily work weekends and overnight, and as long as there is work in the queue.
It feels like a little bit of the shift in my own thinking about this is less about what is the coding work that I think I can do over the course of the week, but what is the coding work that I can be able to effectively feed the queue over a long enough time.
Because we respect linear blocking relationships and able to handle kind of subtasks and all that kind of stuff, I can actually sequence quite a fair amount of work before I go away for the weekend and come back to, not a fundamentally different architecture, but a relatively different architecture over the course of the weekend.
I think two weekends ago, we had quite a busy Friday, and the Dark Factory ended up shipping 150 PRs for two of us over the course of the weekend.
And that was just while we were out hanging about, enjoying London in the heat wave and some of the sun here.
And was there any interaction that you had over that weekend during those?
Nope, so those were all auto-merged in.
Amazing.
TESOL didn't always use a Dart Factory of course.
What was the engineering styles and culture pre you building the Dart Factory?
So the way I'd framed it was single player agentic engineering in terms of you had people have their local agent or maybe a cloud agent like Cloud Code on the web or cursor agent in the cloud or anything along those lines.
But it was still ultimately being driven by an engineer in the interactive human or the interactive driver seat.
And so you'd figure out with your team what the work you were on to do.
You'd then work with an agent to scope that work, do the implementation, test and verify locally, push up.
and merge the PR.
So very similar to kind of the previous flow prior to agents, prior to LLMs, just on an accelerated timescale because the implementation was now getting cheap.
But it was still that very much so I'm a human driving a ticket through to completion with many touch points along the way.
Yeah, so it wasn't so much a workflow change.
It was more, like you say, it's an acceleration of that code generation, really.
And I guess what were the tripping points at that stage?
What were the things that would slow us down?
Because I guess, you know, speeding us up with the same workflow is going to cause problems, it's going to cause a different bottleneck in, like, in another place.
What were the problems, though?
Yeah, so I think verification and testing was a big problem.
We...
So we run a monorepo where we have a bunch of build dependencies.
And so in order to effectively test everything, you've got to build it locally.
And then you need to walk kind of through a whole bunch of go hit a bunch of endpoints or go click a bunch of buttons in the UI or something along those lines.
And so that meant that any change that you were doing, yes, now your implementation is free or functionally free, but you still have to pay that verification burden in terms of, hey, I've got to go actually check that this did the thing that I wanted it to do rather than just looking at the code.
bottleneck is the code review.
Suddenly you're shifting this, what an engineer is spending the majority of their day doing from writing code to reviewing code.
And there are very few people out there who would have said, I think, in this prior world that they love code review as the most important part of the job.
But it is something that is needed and it's something where we share best practices, share culture through, and now people were just doing it day in and day out.
Because I can churn out 20 PRs in a day using a local agent.
But if we have a policy where every PR needs a human review on it, well, then my team needs to go review 20 PRs.
And yes, then they'll also be churning out a bunch of PRs and then I have to review their PRs.
And we just end up in the cycle of we're actually spending more time reviewing and then PRs get stale and then they have merge conflicts.
And it's this, this, um, this other bottleneck that I think we'd never had to engage with before because we just weren't making enough code before.
So it's very interesting when we compare that previous approach where Tesla was essentially using those coding agencies as an accelerator to writing code.
Come the Friday, we're essentially trying to get as much done as we possibly can looking backwards on that week.
Whereas...
Today with the Dart Factory, what we're actually doing is it's a bad Friday if we don't prep the weekend.
We're actually looking forward to think, can I plan and stack and queue enough for the Dart Factory to have enough work going forward?
It's a very different mindset in terms of you're essentially managing the agent work versus, you know, essentially feeding on a minute to minute basis the agent with tasks to do.
Yeah, definitely.
And it's a little bit of that shift in my mind, like you're shifting up the abstraction stack a bit.
You're no longer thinking about like, what is this ticket that I'm going to try to get done?
You're thinking about what is this feature that I'm going to, I want to do?
What is the sequence of steps that...
someone needs to do it, just like I divvy up to a junior engineer or an intern or something along those lines.
And then how do I make sure that the stuff that I care about, either like the correctness or the UI verification or the clicking the buttons or hitting the endpoints or whatever it is that kind of is important to me, how do I make sure that's encoded in a way where I trust that it's going to get done and the agent is going to do it?
And so I think it's both for me kind of like that stacking up and that forward-looking aspect of it, but the other big shift has been kind of We now have these agents, these models, where we can literally deterministically load context into their context window to enforce the fact that they have read something.
It's very hard for me to get a human to go read a doc on best practices or principles or something along those lines.
So I have this additional leverage from being able to say, you know that I want to build the code base in this way, so you're going to go build it in that way.
and I therefore have a higher level of trust in the code that's going to be outputted.
I can then also start layering in other bits of verification around coding styles or around testing or around enforcement or stuff along those lines.
And this is all about, to that feature in the qubit, the only way that I would trust to be able to build a queue that will then sequence and run autonomously is if I have enough trust that the output of that queue isn't, at a bare minimum, going to break existing functionality, but ideally is going to be something that maybe not.
proud of writing myself, but I would be okay if I had written it myself.
Yeah.
Let's jump through what a dark factory is component by component.
And I'd love for you to walk us through how we started on this dark factory journey, what we implemented first, and really understanding that path so that others can kind of follow and understand what they should do first.
So when we first...
tried to build this Dart Factory, what were the initial steps that we took?
So I think they're just kind of set the lay of the land a bit.
There's a couple important concepts here.
One is the orchestrator, which is what's actually picking up tickets and then passing it off to coding agents, doing work and doing the whole babysitting thing in terms of making sure the PR goes through and all that kind of stuff.
But then importantly, and I'll touch on the second, is the actual context and environment within our repo that we give to the coding agents, which I personally think is where the higher amount of leverage comes from.
But we built the first thing first, the orchestrator first, just because that is the tractable thing.
That's kind of what we think of when we think of a dark software factory, is the mechanism to go from linear tickets to actually produce code.
I think just one of our big learnings previewing this has been that is...
dead simple and not that hard.
It's everything else that surrounds that where you then, because if you don't kind of invest in those other layers, you then end up back in that previous workflow case we were talking about where you just end up overwhelmed with review or you end up overwhelmed with code or you end up doing things that aren't actually driving the whole system forward.
So the orchestrator isn't here an agent orchestrator.
It's essentially almost like a workflow orchestrator.
It's connecting, it's an integrator between agents and other tools or other things like...
Linear is a good example.
Exactly.
And so our journey started with basically a Python queue that connected to Linear and connected to GitHub and ran an agent locally in a Docker sandbox.
We literally had one of our engines not allowed to close our laptop for about a week because it was all running locally on our machine.
And we had much more adoption on that than we had expected internally there.
And so kind of the design decisions that we made around that were trying to keep things as simple as possible and relying on other systems as our source of truth where we could because we didn't want to have to reinvent the wheel there.
So we've made an opinionated choice.
Linear is our source of truth for work to be done.
Everything must flow through linear if it's going to result in a durable artifact that is a PR in our code base or an ocean doc or any of the other artifacts that are produced by the dark factory.
That just allows us the audit trail, the debugging, the visibility on that.
And it means that we can then rely on all the work that linear has done in terms of blocking relationships, subtasks, tagging for the reporting and the mechanisms that we need there.
And we don't need to build it ourselves.
So we have.
basically a poll.
We're working to shift this over to webhooks.
It just hasn't been atop the priority list because we haven't broken linear yet with our polling other than our tests, which we'll mention later.
So we have something that pulls linear for new work delegated to our factory and state to do.
Picks that up.
puts it onto the queue.
There's a bunch of prioritization logic, depending on kind of which project it's coming from, what the priority is, whether it's being blocked, stuff along those lines.
That then gets farmed out to a coding agent.
As I mentioned, originally that was just running in Docker on someone's machine.
We just really want an isolated environment because we don't want these agents to kind of be interacting with each other because each one is meant to be kind of a one-shot implementation.
Mind you, they can...
technically understand what other agents are going on.
And we do see this in the logs because we'll often reference other tickets in the description of a ticket.
And so an agent will go check in on how that ticket is going or what the PR was associated with that.
But that's just kind of emergent behavior in terms of building up the context needed to be able to do effective implementation.
These agents currently for us run in a Daytona sandbox.
We're using Daytona just because of the isolation and security guarantees there.
This runs in our own internal Daytona environment.
And we provision a relatively beefy box for that, technically a VM, or technically a container actually, in order to allow it to spin up the entire monorepo testing stack.
So basically everything that a local engineer can do on their machine, modulo creds that the local engineer has access to that we don't give to the agent, we want the agent to be able to do.
This allows the agent to run our full test suite, including integration tests, spin up the monorepo, click around on a UI, click around on basically anything that we would be doing for validation and testing.
all in their own virtual environment.
That helps us build confidence there.
One really interesting benefit of that is that we've now been able to have the agent post videos and screenshots back onto the PR so that we as humans can go manually verify the work without needing to actually SSH on the box and go click the buttons ourselves.
We can see what the flow looks like from that video, from that evidence, which helps then build trust and build confidence in that it implemented the right thing.
That is kind of the crux of the orchestration flow.
The other bit of the orchestrator is just basically babysitting a PR.
So once the coding agent is done, passes a couple of pre-commit hooks, pushes up a PR.
We're relatively confident that PR at least passes lint.
builds, passes tests.
Like it's a good agentic first shot implementation.
On the PR surface, so this is now we're in GitHub.
We run two different coding agents.
One is CodeRabbit.
One is our own internal code review agent.
Our internal code review agents kind of is anchored on skills.
So we have written three at the moment.
Other ones coming soon.
Skills that are targeting different bits of kind of the way that we, what our code review philosophy is.
So one around security, one around legibility, which is really, is this code easy to maintain in the long term?
And one around kind of catch all functional correctness.
We then, so we run our own code review tool.
This is what's exposed via TESOL change review.
And we are using that as signals on the PR for the agent to go fix.
So CodeRabbit and our...
internals who will post comments back on the PR and we have failures in CI when some tests don't pass or we do some what's called mutation testing, property-based testing, a whole bunch of that kind of stuff that those don't pass.
Those all get posted back as signals on the PR.
The orchestrator is watching the PR and so we'll reawake the agent when those comments are passed back and the agent then goes.
fixes them.
It will take in the comments, it will reason over them, it then responds in line in those comments where appropriate, or as a top level comment, resolving where appropriate as needed, and also potentially escalating back to human for review.
So it has the ability to say, hey, I need context, can pause everything and go.
post that question back on the linear ticket, which then notifies the human.
And so that is kind of the flow in a nutshell, going from linear ticket to orchestrator, to coding agent on box, to PR, to a bunch of cycles on the PR review between automated bots and CI checks back to the agents until it gets to a state that is good.
That's normally when we flip to in review and we'll get the human in order to engage with that because it's kind of like in hopefully the best state that it could possibly be.
And so human, the interaction mode there is twofold.
One, we do have the escape hatch of they can pull down the branch locally, make changes themselves and then push it back up.
That's always an option if kind of it's going to be quicker or they need to do it that way.
What we're trying to push for, though, is doing the interaction mode on that PR surface via comments because that then allows us.
to kind of track that and also integrate that back into kind of what our review philosophy is.
That makes it surfaceable, actionable, so that we can kind of continue to leverage agents or continue to leverage those insights into better review flows so that we don't need to get a human to identify that stuff again in the future.
And that's just for those pull requests that do require a human interaction for a review.
Exactly.
Awesome.
So the orchestrator in this sense actually does sound like kind of what a human would do today in that kind of flow.
Pretty much.
And let's talk a little bit about each step there, because, you know, if we were to, you know, ask an agent to do various things, whether it's coding, whether it's design, whether it's a bit of architecture work or whatever at the various stages, it's not always going to get it right.
How do we manage to ensure that either the agent or the orchestrator is validating?
that what is being built, this is pre-tests, what is being built is being built to our guidance or our standards?
So twofold.
One is we intentionally shift task ownership and task decomposition onto the human at the moment.
So we're not saying give the dark factory your Q1 plans and it will go.
do all of that for you.
We're saying give the DirkFactory a well-scoped linear ticket like you would to a junior or mid-level engineer, and it will go implement that for you.
And so there is an expectation that there has been a fair amount of design work, design thinking, potentially whiteboarding, potentially discussion beforehand, which you are then able to encode in the linear ticket, which then kind of guides the agentic implementation after that.
And so think of it as your job no longer is take ticket and go implement it.
Your job is have the whiteboard So there's a little bit of that shift of prior work up front.
In terms of the actual does it follow what you do, there's one thing is implementation is a...
free or functionally free.
You have to pay token costs.
So kick off six, kick off seven.
Like you've got a couple of ideas, just go kick them off and don't send them an auto merge and go look at the code.
Like I kind of almost view the fact that there is a potential like wide state space of potential solutions as either you don't care about it and so you should accept whatever satisfies it or B, you're unsure and so you should use the agents to then help explore that state space and you can ground your analysis in a real code artifact like how it looks and how it feels rather than just in the theoretical abstract of I think this is going to feel nicer if the interface is designed this way or that way.
And so that kind of approach I think is quite powerful because it's allowing you to kind of do more experimentation.
because you then don't need to run the experiments.
Darfactory does it for you, and you're just kind of looking at the outputs of that.
On the taste and the architecture specifically, our solution to that has been verifiers, which are basically kind of like natural language single statements that can resolve to yes or no to a Boolean about a specific piece of code.
And so they're the mechanism that we're giving engineers to kind of instill.
principles about the code into their own specific parts of the code base that will then only trigger when those parts of the code base are modified.
And that's a deterministic trigger or that's...
It's a deterministic trigger, but it is sent to an LLM as a judge in order for the Boolean evaluation.
So think along the lines of something like these files should only ever import from other library files.
Yes, you can technically write a very convoluted regex in order to do that as a deterministic lint rule.
Or you can encode that principle of like, hey, we want library to be the single source of truth.
And so things should only ever be importing from the library and should never be importing from other modules within the code base in order to kind of maintain dependency, clean dependencies.
And so that kind of taste is really hard to write deterministic rules for.
And we've historically kind of caught it in code review.
agentic code review is expensive and slow, can we shift kind of these discrete chunks into a human legible form that we can then put next to the code so that the agents can go bang their head against the wall in terms of making them pass as part of the CI flow.
So I think that's a little bit of the shift in framing as well.
And it does feel like those, I think you've alluded to that, the verifiers are very, very fast to run.
Yes.
In terms of the task that the agent is trying to do as part of that verifier, it sounds like the scope is much, much smaller as well.
So, for example, if that one check was part of a code review, it feels like that's going to be missed by an LLM because, not all the time, but that's going to be missed by an LLM some of the time because the code review covers a huge number of things.
Whereas if it's a small scoped piece that an LLM just needs to check.
on this file or on this set of files, this one thing is going to have a higher chance of actually validating whether that's done or not.
Correct, yeah.
And so a bit of our philosophy is agenda code review should be the catch-all for things that we miss.
But once we have identified something that we think is important, we should be shifting them to verifiers if we can.
Because as you mentioned, they're a lot cheaper to run.
It's an LLM call for the diff.
And so we can assess that.
And we also run one LLM call per verifier.
So that LLM is really just reasoning about the single verification, not about all of the conflicting priorities that it might have in its head.
So we're trying to kind of sketch out a boundary around The other principle that we have is anything that is a verifier that we can make deterministic, fully deterministic, well then we should make that deterministic.
I think historically a lot of people haven't written really complex lint rules because they're annoying and finicky and they slow human engineers down in their heart to write.
Well, right now we have no human engineers.
Agents are fine with annoying and finicky.
They can very happy to go bang their heads against the wall in order to make something work.
And they're not quick, but like an agent can write it.
doesn't matter to me how convoluted it is.
And so that then allows us to not even pay the LLM cost, then we're just playing the CPU cost of kind of deterministic verification in that regard.
And so it's like a sequencing almost for engineers to kind of build their the things that they had in their head around quality or around what good code looked like or how good code smelled we can kind of put that into the code base in that sequence of the very very quick deterministic verifiers linting rules the uh llm as a judge verifiers that are just over the diff which are natural language and so probably too fuzzy to express, but easy enough for a good reasoning model to understand.
And then the catch-all agentic review.
One of the really interesting things, I think, about that sequence is that we can run automatic promotion.
So every night we look at kind of the PR comments that are being posted by agentic review and human review, and we assess whether any of those should be verifiers.
And so we can kind of continually shift left in that regard to get faster, to get quicker, to get cheaper, but also importantly as well for me, I think, to get more human legible.
look at a verifier, it's much easier for me to reason about that than to read a long skill.md and figure out, is this thing going to be correctly weighted for something that I think is really important against all of the other compounding trade-offs that it might have.
Super interesting.
Let's go into what worked and what didn't work.
Obviously, this unlocked, it sounds like, a ton.
What were the biggest learnings you had early on with this?
Coding agents are really good, and if you give them a wide enough scope, they will do some great things.
I think there was definitely a bit of a kind of bias internally going from a philosophy of I've got to be there babysitting the coding agent to shoot, I can't even look at the logs of this because it's running on a sandbox in the cloud.
better make sure that my tickets are super well scoped and super small and super tiny, kind of like intern level tickets.
And now we've slowly been, and this is cultural, not technical, slowly been relaxing those constraints.
And people have been like, ah, cool.
What is actually the biggest thing that I can give to the dark factory and see it produce something that is relatively valid as a result?
And so the tickets are going to- Was that a trust issue?
It was purely a trust issue.
I think trust- Comfortability seeing seeing the results.
We're asking people to trust that something that is kind of hidden, it's called a dark factory because we're not looking at the logs as it's kind of operating.
We are.
postdoc, but not kind of in the actual operations, they need to be able to trust that that's going to do something that they would be okay with.
Because ultimately, like we still have an accountability mechanism.
The person who filed the ticket is still accountable for the code that got produced.
They still have to, if it's in the production part of the code base, they still have to review the PR.
Like we're not saying how you can abjugate all responsibility because the dark factory did it.
And so we're now asking people to assign their own accountability to something that has been autonomously produced.
And so there definitely is going to be a gradual kind of trust adoption curve.
And we will continue to go through that.
And I think it's going to be never ending because there's always going to be kind of an increased scope or increased verification you can do to build that trust.
I like to think autonomy is earned, not enabled.
You can't just turn something on and say, we can now go at full speed.
It's about building that cultural trust internally in your organization to get to the next stage.
What problems did you have there?
So just on that trust bed, I think one of the things that we've been trying to do on the team to build that trust is by running as far ahead as we can and running into the walls early.
Mucking up, basically.
So we definitely had a few of those.
Let's go deep there.
What are the walls we ran into?
So I think the most interesting, there's two kind of classes of ones, one around context and one around verification.
In terms of context, I pull data for the number of Slack threads sent on our team channel versus the other team channels kind of every week.
Our team, despite being slightly smaller than one of the other teams at the company, is sending about between 10 and 15x as many Slack messages per week on our team channel.
We have had to over-communicate everything that we're doing because we are doing so much more quicker.
And also with AutoMerge On, everything is changing in ways that we don't really understand.
Last time I looked, I think about only 5% of the Dark Factory code base had ever had a human look at the PR and approve it.
And I don't think people actually go look at code anymore because they're all just using agents.
And so that means by implication, 95% of our code base has never had a human look at it.
And so we don't know what's going on there.
So we need to build the signals and the context on top of that to give us trust that this is working the way that we think it's going to be working.
But then also we know what everyone else is going to be doing.
And that 95%, that is of the Dark Factory?
Yes, of the Dark Factory specifically.
Yes, everyone has looked at the production code base.
Again, we have human review required there.
So that has been looked at, but it's for the Dark Factory itself.
And so we needed to figure out mechanisms in order to make sure that everyone knew what they were doing, make sure that everyone was comfortable with their own work getting thrown away, and make sure that we understood how the codebase was changing over time.
And so we definitely still haven't figured this out.
I still will wake up on some days and be like, I don't think that's a good idea.
Why did that land?
But the intention, what we're trying to do there is overshare the context so that we're aware, but then also fix forward.
It's not roll back a PR because it wasn't the thing we wanted to do.
It is, okay, cool, what we missed on the verification side that allowed this to land either in a verifier if it's a human concept or on tests if it's something bad so that we can continually fix forward and build those guardrails, earn that trust on the autonomy side that you mentioned.
In terms of some of the more spectacular failure modes, as part of this, we...
changed our core queue a lot in the early days because we weren't verifying it strongly enough.
And by what I mean by that is we would have a fix.
Actually, concretely, we had an issue where PR comments were being double counted.
And so they would end up on the queue twice until you would get an agent in two different sandboxes responding to the queue.
You could have race conditions where it might produce two different code commits as part of that.
Sometimes the second sandbox would start up after the first one I would be committed.
So to recognize, hey, the work's being done.
But we just.
it wasn't very effective.
And the queue here, just for context of what the queue is, the queue is the list of tasks that the orchestrator needs to pull.
Exactly.
To then pass to an agent or do something with.
Exactly.
Exactly.
And so we would fix that.
It'd be good for a day.
Then it would regress.
Then we'd fix it again.
Then it would regress.
Then we'd fix it again.
There was a point where I think we had like 60 PRs over two or three weeks or sorry, two or three days that we're all trying to fix this.
And so this was just causing constant churn and frustration because we'd say we fixed it and then it would regress and it was just, it was a pain.
And so the way that we wanted to try to fix that because we were still going to continue to let agents do whatever they wanted was really lean into the verification side of that.
And so what we ended up doing was building a quint formal model of our queue behavior, specifically the interpotency of being able to kind of add stuff to the queue so that we could.
verify that on every single PR that we hadn't broken that formal model.
That took, I wouldn't have known how to do that in pre-agent world, just flat out.
I knew what a formal verification was, wouldn't have known how to actually build it.
Took me about a day with Fable and then using the dark factory to go implement this.
And we've had absolutely zero recurrences of that after the fact.
And so that for us was.
very painful in the moment, particularly as we were scaling up.
But then B, a really important insight of like, no, it's not just the kind of verification that we used before of like test and shared human context and like trust that a human has enough context about everything else going on to know that this is a sensitive part of the code base.
No, we actually have to lean in even firmer than we had before because agents don't have that contextual information and we don't want to give it to them because that burns to tokens, which is therefore more expensive and slower.
So that was kind of one really, interesting, annoying, frustrating bit of time.
The other one was an experiment we ran during the offsite, which I was really excited for and which really did not work the way I wanted it to, unfortunately.
And that was explicitly about stress testing this verification layer.
My personal belief is that if we get our verification layer strong enough, and if we are oriented around specifically the behaviors that we care about as an organization, not the actual implementation of the code, I don't care about if you how when I click a button on the UI, the results appear.
I care about when I click a button on the UI, the results appear.
And so we should be able to, just from that verification layer, be able to rebuild the entire system from scratch in a new language.
So we tried to do that.
We currently are a Python implementation.
Elixir is...
ish a good fit for this in terms of having kind of the actor model.
And so we took our verification layer only.
So none of our unit tests, nothing other than kind of our end to end tests or integration tests, this formal model that we were talking about and some of the properties that we that we verify kind of on the surface boundary rather than on internal dynamics.
And we gave that to a coding agent to then go plan out building that in Elixir from scratch.
And then we went into that.
That worked.
It built something.
But we hit kind of two major roadblocks that kind of prevented us from being able to call this a success.
One thing I will note, our queue was perfect.
Queue works totally fine.
So the formal model held up well there.
But what it ended up identifying was even after kind of all the investments on verification, thought we had made, we had some pretty big blind spots.
We weren't actually verifying formally anything to do with the labels on linear, which are a very important routing mechanism.
That was all just built into unit tests, and our end-to-end tests were just testing the happy path there.
And so when we ported over the end-to-end tests, oh, shoot, suddenly the way that we were stacking PRs broke because that wasn't tested in an end-to-end test.
That was just in a unit test.
So that's...
core functionality that wasn't verified externally and so was broken in the elixir implementation we weren't able to kind of push out to production um even though we run in a sandbox production environment and so then i had to mainly babysit an agent to go do that and so that's an area where we want to be investing in in terms of getting the dark factory to be able to make non-destructive changes in production environments or at least the sandbox environment first and then eventually production environments to take the human out of the loop there And then finally, there was just some other bits of the functionality.
I mentioned the stacking of PRs in terms of the feature branch.
I mentioned labels.
I mentioned, or rather, there's also some of our GitHub functionality, specifically in terms of not just comments, but on CI batching, where we try to take in all of the signals on CI failures and send them as a single, as a one-off, rather than sending them as a sequence that isn't captured by an N10 test.
So there was some pretty core functionality and improvements to the system that weren't being captured by this verification layer.
And so...
So despite really wanting to turn it on, well, I did turn it on and then it immediately broke and then I fixed it and then I turned it on again and then it broke some people's PRs and I had to be like, okay, cool, this didn't work.
And so I think the real learning from that is...
This, I think, is doable.
We got pretty far with it.
I think it is an even more ambitious framing, though, than what Anthropic did with the Rust rewrite, where they had kind of the original ZIG source code as an input to that.
But I think it actually allows you to write idiomatic code better because our Elixir implementation, I'm not an Elixir expert, so this is entirely LLM as a judge, agent-vibed.
Our Elixir implementation evidently was idiomatic Elixir.
I don't really know how to assess that, but evidently so, rather than being Python tool supported.
But we need to make sure that we cover everything.
And I don't know yet exactly how we're going to continually lever up on that other than by continuing to do experiments like this to be able to identify where the gaps are, because that's the only way that we can build trust and really let agents rip when we're confident that that boundary, which we do require human review on, is super solid to the extent we can rewrite it.
So one of the things I'm getting here from this discussion, from this conversation, is that in building this Dart Factory, this isn't something that we can necessarily just turn on and expect everyone to use.
A lot of our learnings have come through usage.
And I'd love to kind of switch briefly to adoption and onboarding.
So would we be able to now, with the Dart Factory that we've got now, if we had a new...
team that had never used the dart factory do you feel like they'll be able to just like on one on you know day zero just switch in directly to a to a scenario where they're using a dart factory or do you feel like they would need to grow into it like layer by layer piece by piece So we had a new starter who on day two was adding verifiers to the Dark Factory, which, as I was just saying, is one of the most important parts of the Dark Factory.
So yes, I think you can get new people onboarded to it, but it does require a specific mindset.
I think it needs to be folks who are recognizing the fact that agents are not just doing the same thing but faster, but are fundamentally new workflow primitive in how we do software engineering.
shift in my mind of like, what is the role of the engineer?
I personally don't think engineers are going to go away.
I think there's always going to be need for technical expertise in terms of understanding what the systems are.
But just as your mechanic doesn't build your actual car, but can go debug what happened with your car when it breaks, that I think is going to be more so the role of the engineer in terms of helping build and create the environment for a factory to run in and then debugging why it broke and then improving that.
So more around the outside and that.
I think for kind of a new starter or a new person, it's definitely a mindset shift, but it's still engineering, just engineering at the systems level, which is what you were meant to do as a senior staff, principal level engineer anyways.
We're just now asking junior candidates to start doing that right away.
Hey everyone, hope you're enjoying the episode so far.
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
But for whatever reason, many of you have yet to subscribe to the channel.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
All right, back to the episode.
So an individual...
can join an existing team that has all of this set up for their project.
What about an organization that is trying to, that is, you know, perhaps where Tesla was a little while back and not using a Dart factory or a software factory and perhaps using the Claude code and codecs and things like that to assist them to accelerate that building of software.
Is that something they can snap to directly?
we're going to turn on dark factory and start building this way?
Or is that something that should happen gradually in layers?
Yeah.
So I've been talking a lot about what we're doing with building the dark factory.
But as I mentioned, like we intentionally tried to front run a lot of these issues here.
What we've been doing with the rest of the production code base is that kind of incremental adoption, because we're not going to come out and say, hey, everyone at Tesla, you must use the dark factory.
We've got into that kind of 65% or 65% of PR is going through the dark factory.
by convincing people that this is a fun, new, cool way of working that speeds them up.
And that is very much so by helping them lever up the parts of the code base they own through all these tools and mechanisms that we've identified, specifically around the verification, to be able to allow them to build that trust and to build that faith in automation.
So concretely, what that looks like is just start with...
better context in your repo.
Like that already improves your local developer flow.
Like people, companies, organizations, I think really need to be much more intentional about the context in the repo because that is ultimately.
what goes into the agent, which ultimately, in my mind, and part of my kind of belief of where the value is that we can provide, more important than the actual raw underlying model.
The context that you give will change model results more than swapping a model will.
And so if you feel you're at a good place in that regard, then start layering on the verification bit, because that then also helps you.
trust what agents being driven by local engineers are producing because they are verified two ways that kind of go beyond what you might have done historically so tests are dead cheats right Write tests for everything.
Write end-to-end tests for everything.
Write integration tests for everything.
Do behavioral tests for everything.
Make sure that that kind of layer of the thing that you actually care about, which isn't just what the code does, it's the behavior, it's the product surface, it's the things that is expressed, make sure that is super solid.
Because that mechanism will both help build an adoption kind of from your internal edge team because, hey, your agent now is producing better work without you needing to take kind of the reins and guide it through that, as well as fully producing higher quality outputs over time.
time.
I think then you start layering on into the more non-deterministic aspects.
So adopting verifiers, adopting these kind of discrete units, which are still very human reviewable, very human understandable, but are able to encode kind of fuzzier concepts than you would be able to in a deterministic linting rule.
Again, you can then have something deterministic run that, feed that signal back into a local agent or into a dark factory and use that to kind of encode preferences in the code base and then into agentic review and then kind of levering up in that way.
And so the These pieces that you can adopt, particularly around the verification side, these both do better when you're kind of, or rather allow your engineers to do better when running local flows and give you the building blocks and then move closer and closer to the fully automated factory.
The last point on my mind just around the cultural adoption bit, I think.
The most important thing we've been able to give to the engineers here at TESOL is the feeling that they still own the taste and the quality in their code base.
And they're not just giving that up to an agent, but that we're trying to help them figure out a way to scale out their taste to all of the agents that will be running on their code base.
Previously, let's say you were a staff level engineer who was responsible for a specific part of the code base.
The way that you would enforce kind of quality there is.
doing all the PR reviews or doing presentations or writing docs on what coding standards look like or writing a style guideline or stuff along those lines.
Now you encode that within a set of verifiers, your agents, all the agents, not just your agents, all of the agents of everyone in the company is going to be running that, those verifiers for those bits of the code base.
And suddenly your impact is now scaled across everyone.
And so.
that then forces those agents to engage with it and also allows you the ability to kind of take your learnings over however long you've been in the industry and give them to the agents in a way that makes them better.
And so that, I think, is where the truly scalable kind of shift in mindset is coming from.
It's not just kind of like, what can I do to make my local agent better?
But then can I commit that into the repo in a form that makes all of the agents better and also then gives me the base to build up towards a dark factory future?
Now, a lot of people are using Tesla today.
are using that as the registry to store their context, to evaluate and improve their context, distribute their context across organizations in private and public workspaces.
But the TESL agent, of course, while it can be a great interface to a lot of those functionalities and capabilities, the TESL agent can help here as well in doing a lot of what you just said, right?
Yeah.
So we think of the Tesla agents kind of as like a natural language interface for being able to build the building blocks of her dark factory.
And so.
It has knowledge of all of the bits within the TESL CLI that we've kind of touched upon here.
So when I was talking about our own internal code review, that's TESL change review.
When I was talking about verifiers, that's TESL change verify.
We also have TESL change risk, which is our own internal risk implementation to assess whether a PR needs human review or not.
And so all of these kind of like core primitives are baked into the TESL agent and the TESL agent knows how to set them up in your repo.
As I was mentioning earlier, we don't think the orchestration layer is differentiated.
Everyone probably should build their own orchestrator because it's simple and allows you to then integrate against a whole bunch of your own internal systems.
Like, do you use JIRA?
Do you have your own internal custom ticketing system?
Where are your sources of knowledge?
Like, that is important to work, but the differentiated work, in my mind, comes from the loops that you're able to build on top of that, and also these...
primitives that you're able to compose.
And so Tesla agent will then walk you through how can I set this up in your repo using your context, using your way of doing it.
So for kind of open source repos, we'd recommend using GitHub Actions.
They're easy, they're cheap, they're everywhere.
And so we can then, Tesla agent can then work with you to be like, cool, I want to do a review of all of the PRs that have landed in this repo every single week and derive verifiers from those.
Tessel Agent can do that with you.
And I'm being key there to say with you, not for you, because...
That verification bit, just to reiterate, is something that I think, whereas the human leverage comes from, that's still very much so part of the core engineering job because that's where you're encoding your taste, your preferences, all of that kind of stuff.
You're still in control, which is the key piece.
But the actual go write the GitHub action, do it, yeah, I don't want to do that.
Tesla agent will do that for you.
And if I wanted to get started, I've got a fresh project that I'm building.
I'm an agentic coder.
I'm currently building using various agents.
I've got various skills and context in that repo.
as well.
I'll start up Tesla Agent.
Where do I start?
What's the first command?
What's the first prompt I should write?
So I guess two commands come immediately to mind.
One is do an audit of your code base.
Just ask it to help with that in terms of being able to understand what context do you have?
Where are agents thrashing?
We have the ability in there to also go look at some of your local logs.
If you are using Cloud Code, for example, it knows where to go find your logs for that session.
This stays all local to your machine.
to be clear, but it will help you be able to identify like, hey, this is something that the agents have done that is constantly thrashing and constantly kind of causing issues.
Do you want to write a skill for that?
And then do you want to kind of be publishing and versioning that skill?
And so that kind of gives you an initial set of insight into like where current friction points are or where you're already great and you don't really need to do much.
And then the second one is on those loops.
It's like, what is the first loop?
the agent to suggest you do a loop.
My personal bias would be probably something looking at the PR surface or something looking at like docs, updates or maintenance.
You know, those kind of tasks that you know you should do, but you never want to do.
And so you get still docs.
Agents and loops are great at that.
And so working with the agents who figure out kind of what the highest, either highest value loop is for you to still then be human loop or highest value that you're comfortable running autonomously because it's a really.
interesting and cool feeling to be able to wake up and see, hey, we call the maintenance agents internally.
Hey, my maintenance agent ran overnight, identified three issues, filed PRs, they're all on auto merge and they even landed.
And so that just is great.
Just seeing work done.
Exactly.
And it's not work that I had to think about.
You didn't even have to initiate it.
It's like, I found the signal and I want this signal to go up or go down or do something.
And then maintenance agent goes and does that.
Amazing.
So that's available in Tesla Agent right now.
So if you have the Tesla CLI installed, just type Tesla Agent on the CLI on the terminal and you'll drop into that environment and you can ask those prompts straight away.
I think it's time.
Shall we go back and have a look at some of those linear tickets that we raised?
Let's see if the Dart Factory's done something with them.
Definitely.
This one, as a reminder, is about adding the metrics for where we did the inference geo within the dark factory.
And so we can see that, hey, there's been a PR associated with it, and hey, that PR has been merged.
So I'm just going to click over to the PR here so we can see what happened.
So we've got the summary from Kikimora.
As I mentioned, that's our internal name for our dark factory.
We have CodeRabbit who runs on review.
This GitHub action here is our own individual review harness.
So these are the comments that have been responded to by Kikimora and resolved, given by our own action.
CodeRabbit also then provided some comments as well.
There are some CI failures.
Thank you, Kikimora, for fixing that.
You can then see that CodeRabbit approved it and Kikimora merged it.
So this was...
No human has looked at this up until the point where I'm looking at it now.
This joins that 95% of code that has been merged with no human in our Dart Factory implementation.
Exactly.
And I'm not going to go look at it because I trusted the test pass and therefore it at least isn't broken.
And if it is broken, we will find that out and then we will fix forward.
Fix forward, absolutely.
Awesome.
So this was the first linear ticket that didn't require the human interaction.
Perhaps we jump to the linear ticket that did require that human interaction, which would go into production.
And so this is the ticket where we were looking at fixing the quote issues in terms of what was landing on our...
web UI.
So again, something that was identified in an audit, but probably should be fixed and something that is fully user visible because this is the code or this is what users see on our website.
We're going to see that there has been a PR associated with it and it is also merged.
So clicking over that, we see that 46 minutes ago.
So 10, 15 minutes after we kicked the ticket off.
That's when the initial PR was opened by Kikimaru, our internal name for the Dark Factory.
We see CodeRabbit go kick off as well as our own internal code tool.
A bunch of issues found.
Kikimaru then responds to it.
CodeRabbit approves it.
Then Sahil, who's one of the engineers on our team, as I mentioned, these require human review because they are production touching.
He then approved it and merged it.
So the only human touch points on this PR to get something from linear ticket.
it to production is that required human review because it is still production code and someone clicking the merge button.
And so that, so the fact that Sahil did that was because this was, this was flagged or this was marked as something which requires human review because this is a critical system, maybe a production system or it might not be a production, but it might just be some critical piece of code that we require a human to look at before.
Do you feel like there'll be a stage where we'll fully trust this?
Yes.
There will.
I think so.
One of the things that we are looking to invest more in in the near future is how we do risk classification on PRs.
I would love us to get to a point where we are able to merge straight to production for low risk PRs according to that classification.
And the way that we think about...
that classification, like a lot of else at TESOL, is what is the human visible mechanism for being able to encode that so we can then reason over it rather than an opaque classifier model that's doing a binary classification.
What is the skill we can give to the agent that encodes the things that we think are important so that we can then tune that to that specific risk level?
And so something like this.
Probably would be auto-merged at some point in the future because it is a user visible, but it is just a documentation change.
Doing a full database migration, I still think that one actually might be because that's easy to verify mechanicalistically.
In terms of switching over what infrastructure our pipelines run on, that one might require some human in the loop.
But I think we're working towards that goal.
So during this podcast, we created two linear tickets that have now had...
PRs created and have been merged, one into production, one into the Dart Factory codebase.
That's two of 600 to 700 pull requests that will be created this week at TESOL.
I think we said we're at about 60-70% of all pull requests at TESOL being created through the Dart Factory.
What will it take to get that to 100%?
Building trust, shifting left, so actually stepping back quickly.
We think of it in shifting left and shifting right.
So from the PR, shifting left is helping engineers...
create PRs.
Shifting right is working on the verification of those PRs so that we trust what the output is.
And we think we have historically focused more on the shifting right to build that initial level of trust of like, hey, the code the agent has done is roughly correct.
There is always more that we can do and there's always more that we will do there to kind of improve the quality of the output.
But we think we're in a good enough spot there.
Where we really want to be doubling down and investing in is on how do we make this the easiest way to do work at TESOL.
Not just the fastest or the highest throughput or anything along those lines, but where can we bring the dark factory into making that even easier than possible?
We have our own internal Slack bot that listens ambiently in channels and can file linear tickets for you and all that kind of stuff.
We want to move it into actually being able to propose tickets on linear or being able to do decomposition.
We want to move it into the planning process to be able to help you go from PRD and be there as a coworker to kind of be able to riff ideas off of or something along the way.
those lines.
So making it so that the default is the Dart Factory, not because we're enforcing it, but because that's just what people reach to because it's integrated with the tools that they're using every day.
Amazing.
Rob, this has been truly, truly fascinating and it's an amazing accomplishment to see how much of an impact this has had on the way we build software at TESOL today.
And for our audience, our listeners, if you wanted to try this out and actually build your own software factory, head to tESOL.io for more information about Dart Factories and about the TESOL agent.
Load up the Tesla agent just by typing Tesla agent at the command line once you've got the Tesla CLI installed and just ask it those questions.
Take a look at my environment.
What can I do to improve this?
And how can I add these verifiers, the plug-in reviews, the pieces into the CI and the code reviews?
This is absolutely amazing.
It's definitely the next level of software development.
Rob, it's been an absolute pleasure.
Thank you so much.
Thank you.
It's been fun.
I hope everyone learned something today.
And thanks for listening.
Tune in to the next episode.
Bye for now.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast, it's a community.
And we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more.
And I hope to see you there.