# Continuous AI Turns Repositories Into Software Factories

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-08-18

## Transcript

I mean, chief or not, it's the nature of the factory.
If it produces rubbish, you throw it away, right?
I do understand, of course, that human review is a bottleneck.
When you're creating a factory, you know, a lot of the work is on the quality gates.
You know, there's just no point creating a pull request for the human to review when that thing is low quality.
My job as a factory creator is to deliver high quality pull requests where the reviewer is equipped.
And I love that word equipped.
My job is to equip the reviewer with all the information they need.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
Yesterday, you were writing code.
Today, you're prompting agents.
Tomorrow, you'll be running a software factory.
Argentic coding is moving fast enough to give anyone whiplash.
This June, AI DevCon London sold out.
Hundreds of developers, engineering leads and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify and GitHub cut through the hype and get to what actually works.
This November, we're bringing it back to New York.
Bigger, bolder and full of people actually building the Argentic future.
Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams or setting the strategy.
Our early bird pricing ends on September 30th.
So make sure you lock in your place before prices increase.
As a podcast listener, you can get 15% off your ticket using promo code POD15.
We have so much planned and we can't wait to have you join us in New York.
Hello, everyone.
Welcome back to the AI Native Dev.
Today, we're going to talk about continuous AI.
We're going to talk about repos and what is the container for factories or whatever we call them on that front.
And to dig into all of that, we have with us Don Syme, who is a long-term kind of industry contributor when it comes to programming languages and how we develop.
Big open source contributor, worked a lot on F-Sharp, on various other projects you would have known.
And today, a principal researcher.
at GitHub working on many things, AI development, but specifically also kind of continuous AI and agentic workflows, which we'll cover more.
So Don, thanks for coming on to the show.
I should also point out you've given an amazing talk at our latest AI Native DevCon, and you're a big part here of the London community of development as a whole, but also specifically now as we transform into AI.
So love having you as part of the community.
Guy, it's a pleasure to be here.
And it's been a pleasure to get to know you over the last couple of years as you've created TESL.
And what a wonderful community here in London and also a worldwide AI Native Dev community.
I've used your resources a lot.
I've learned a lot from what you've done.
And, you know, really kudos to kind of create that.
to take that community ethos that was so strong in the developer world and continue it through the really big shifts that have been happening in the developer world.
And yeah, I love what you're doing.
Cool.
I appreciate that.
And it's all about gathering brilliant people and giving them a stage to share their learnings.
So precisely what we're doing here.
So we're going to talk a lot about how do you sort of systemize, if you will, software development in this sort of new era?
And you use the term, you kind of individually and also at GitHub use the term continuous AI when you think about that type of development.
So maybe for starters, just define for us a little bit, what is continuous AI?
Yeah, so the thinking is that There's been a bit of a blind spot in the AI industry, a really big focus on the kind of the use of AI basically coming from the chat modality, the individual productivity modality.
Of course, we see that with chat GPT just exploding in its initial, and then all the kind of productivity enhancements that come around the chat modality.
But we think there's kind of a missing point of view.
which is about collaboration and which is about automation and which is about situated automation.
And we were thinking about this a lot last year at GitHub.
We did some kind of thinking around about it.
We kind of looked, well, you know, we already do a lot of situated automation in CI, CD.
We have continuous integration.
We have continuous deployment.
And then there are this whole range of activities that people want to do in this continuous modality.
The continuous modality is just so central to how we work in the software industry that change is happening in this kind of context, usually the repository, but there can be other kind of context, the organization or multi-repo.
And continuous means you're kind of continually folding in that kind of sense of change and eventing and difference and sort of incrementality that's kind of happening and always reestablishing this crucial invariance like the software actually works in continuous integration or the software is correctly deployed in continuous deployment.
And a lot of these things are very, they're central to how software works.
collaboration and deployment works.
But equally, we want something new, which is also these subjective activities, like continuous documentation.
Now, writing documentation is a kind of...
is a subjective activity.
It's not the sort of thing where those CICD kind of concerns are about like determinism.
We want everything to be exactly right and double check absolutely everything.
Repetition and sort of insistence that it always happens, like a lack of...
Yeah, absolutely.
...less creativity and more consistency.
Yeah, that's right.
But now we have this whole new bucket of things like we want to continuously improve the performance of our software.
We want to...
continuously fix bugs.
We want to continually triage bugs or have every issue labeled correctly within some kind of schema.
We want to continuously look for duplicates in our bugs.
And of course you can kind of do these things each as a one-off.
But in the end you're going to have to put them on some kind of operational permanent kind of basis in the context of change.
And it's so easy for people to kind of...
blank on that and kind of say, oh, look, I can find the duplicate bugs in my repo.
But you've got to make that continuous, right?
You've got to get that on an operational, permanently running kind of basis for that to really be a sustained activity.
So we looked at this kind of collection of things and we said, well, what really characterizes this is they're different to CI, they're different to CD, but they're also similar.
Okay.
And it's that, and that said, well, let's...
partner that CI, CD, and continuous AI.
And the key things is continuous, but also kind of subjective and implemented with kind of AI.
And that's where we land with that term.
When we put that out there, and it kind of seemed to resonate with people, it kind of captured something that other people weren't talking about.
And that's all.
Super interesting.
So let me kind of echo back a little bit and the things that resonate with me.
So it starts from the sort of the single player to multiplayer move on it.
And then I guess naturally over there weaves into automation, right?
Because to an extent, like we talk about continuous, but I think what happens with AI is that it has this jagged edge and sometimes it works amazing and sometimes not.
And we have this kind of hard to determine points in time in which we want.
a human to intervene and apply their judgment, which is clearly a blocker for scaling.
And in that sense, it's similar to continuous integration because before continuous integration was non-continuous integration like you know someone manually i remember running it and sort of running the sequences there was a lot of homebrew uh continuous uh automation as well sort of gauntlet systems or something like your your pr would run the gauntlet of your organization and that kind of all came together into things like github actions and other ci city kind of systems And we see plenty of homebrew for people kind of automating parts of AI kind of workflows as well.
And it's just like, yeah, you know, we should consolidate that into systematic ways of thinking about it and systematic ways of actually kind of doing it.
practice performing so so so it has these maybe like two different routes uh one one going from single player to multiplayer and the collaboration element to it the other is the automation uh and so do this repeatedly maybe starts with sort of homebrew automation and uh and doing it and then consolidating those into the sort of infrastructure that is cicd usage style but extended and expanded into the type of actions that ai can now do that's exactly right are naturally less deterministic in in nature and so that really leans into not just the automation but also the definition of how would you track this activity what is a failed build you know versus a successful one How do you continue that?
So all of those kind of go into that term of continuous AI.
Yeah, yeah, absolutely.
And there's another key aspect here, which is about bounding the context.
I think we're going to kind of talk, bounding the context of automation and bounding the possibilities of what happens in the automation.
People, conversations about some kind of automated AI often go off the rails unless you're actually situated inside.
actually some bounded kind of context.
We're going to do this in the repository with these particular outcomes.
Like it can create a single pull request.
It can create a single issue.
Whatever the job of the automation is on each run, it gets these particular kinds of inputs and doesn't get anything else.
It has a firewall around it and so on.
So to have these kind of bounded, easy to understand, simple kind of simplified view of what...
What automation are we actually doing here?
And because people are already really familiar with CICD automation, a lot of people use GitHub Actions or some other kind of CID systems.
All sorts of people can successfully kind of use those.
It's really a natural starting point for saying, hey, let's automate.
Let's do this together because we're all doing it already.
And now let's kind of automate.
some AI-driven subjective kind of elements of this as well.
And so, you know, situating it there does make a lot of sense.
It's also counterintuitive to a lot of people, I've got to say.
A lot of people listening on this might say, CICD?
I don't want AI anywhere near that, right?
That's like, that's my determinism, and there's a reason for that, which is it.
one of my friends kind of says it's the place where the grown-ups are right you know it's the place that really establishes that your system actually works as it is and a lot of people have plenty of problems with their ci city systems it's not like it's it's it's so important uh that's a major focus for team activity is just kind of get this invariance established there get get the green ticks and get everything kind of checked there.
They use it for everything because it's so useful.
And the thought of adding kind of more subjective activities into that, like labeling or issue triage or bug fixing, that can be disturbing for people.
And it's one reason that we don't say we're putting AI into your CI.
No, no, no, no, no, no.
We don't do that at all.
It's CI, CD.
They stay as they are.
And continuous AI is a separate bucket of things.
You can do none of it if you like.
You can do a lot of it if you like.
You can dial that as you like.
And that's really, really important that we're not mucking with the core ethos of what CIACD does and how crucial that is in the industry.
We want people to double down on that because that's where we really establish software equality in the industry.
I think so I relate to that.
I think like on the context of context, you know, of like defining.
context and defining it.
I do think that there's a lot of analogies to the world of CICD in that environments are actually a core element of CICD as well, right?
They had us move up from works on my machine to something that is defined.
Continuous deployment is about the environment of the production.
And so defining the context is...
just as probably I'm saying it in quotes because it might actually get fairly complex, just a new type of environment variable for it.
But it's a good...
Tee up, though, to a question I had, which is, you know, you used the term continuous AI.
I feel like there's a lot of analogies in the continuous AI narrative to the world of sort of factories and factory lines.
In my own kind of AI Native DevCon talk, I talked about sort of the software stack.
And in one of those paths, I was a little bit stuck with saying, OK, you have models on top of which you have tools, on top of which you have skills.
on top of which you have harnesses, they all kind of compose down.
And then the harness, it was hard to know what's above a harness.
And I wrote their factory line.
The term loop became popular like the next week or something.
So maybe that's the term on it.
Now you have this continuous AI.
But I guess do you feel those are just different words or is like a system harness or a factory line or even a loop?
A loop has a little bit of that optimization piece to it.
The same as continuous AI, I guess, how much, How much is just a different lens to arrive at the same destination versus you're defining something different?
I think some of those conversations are orthogonal because the shift that I'm most interested in is this one from the kind of individual productivity point of view to something which is more continuous and kind of team and kind of integrated into some particular kind of context.
And so...
A lot of the things that are kind of getting developed, like the notion of kind of loop engineering, you can apply that all in a kind of continuous AI setting.
The ledger of record might be different because we typically use the repository and the pull request is the actual kind of ledger of record of what happens and maybe some augmenting kind of things as part of that.
Whereas maybe if you're kind of loop engineering on your local dev machine or something like that, you might be using a different kind of ledger and so on.
But the concepts can actually flow very freely between the two.
And I really like that.
It's like, you know, there's a lot of conceptual kind of competition in that kind of like what comes next kind of space in the individual productivity space.
And I kind of like saying, hey, that's not what we're doing.
We're over here.
And yeah, you do all of that and bring it over, like bring over all the good ideas and rethink them, how they kind of.
work in like what the loops kind of mean.
It might be in lots of steps of a loop like you get to see you set the continuous AI set up like we have goal chases for example continuous goal chases which once a day try and make progress towards some particular defined goal and each day you get a pull request towards that goal and you kind of integrate those but maybe it won't work on the next pull request until you've done the previous one so it's kind of it's not really a is that a loop or is it not a loop it's kind of a loop because it's kind of you know, repeating those pull requests towards the goal.
But it's a cooperative loop that's actually quite slow in its kind of action.
That might be what an enterprise exactly wants to kind of...
to kind of chip away at the kind of all the big backlog of accessibility bugs or something like that that a product has.
Those kind of loops are actually really satisfying for teams where they have like this cadence of like daily improvement towards some kind of goal that is measurable.
You can see you're making progress and managers can kind of understand what's kind of going on.
Whereas, you know, if it's in the kind of loop engineering world where the individual developer is like, yeah, I fixed a thousand kind of accessibility bugs and here's a thousand.
and pull requests for the team to review.
You all have to spend or drop everything you're doing and kind of review those.
That's kind of a little bit disruptive to how teams work.
So we kind of get to shape things a bit differently when you slow things down a bit and kind of move in a different kind of pace.
I feel like I...
To relate to the distinction between the single player and the multiplayer, I think that is very important.
There are still sort of two things here.
One is the pipeline, so the continuous AI element of the pipeline.
This is an aspiration to get to a point in which you can get a recurring or like a repeatable motion of getting a type of input.
to come into the pipeline and process it and get a certain caliber output out of it in a repeatable fashion.
From maybe even two other aspects, what I would say is a loop, which is more look at how that was done.
again and again for it and improve it.
So a self-improving pipeline.
Because you can apply a loop to CI-CD.
I mean, in fact, we do it, you know, and we do it for ourselves.
We do it with the Tesla agent.
For others, which is fix flaky tests.
So flaky tests are an example of something that isn't necessarily a pipeline in its own right.
Maybe it is, but it is about identifying failures in the CI-CD and fixing them.
But I think that's the loop.
And then I think debatable a little bit on the term where there is proactive initiation of something that is like an event-driven, a bug was reported.
That's probably not its own term.
It's probably like either, we're talking about two things, the pipeline and the loop is actually the initiator.
So do you agree with that sort of separation?
We have the pipeline and maybe the loop and the initiation of it.
And how does that relate to the GitHub agentic workflows, like the sort of the core capability that is from GitHub?
I mean, what is the scope of that product?
Is it both of them?
Is it more?
Is it others?
Maybe tell us a little bit about the product side.
Yeah, yeah, because we've talked kind of exceptionally about kind of continuous AI.
And I'm very excited to say the GitHub agentic workflows, which is our implementation of our kind of continuous AI system for GitHub Actions.
is now in public preview.
And, you know, it's going to be part of the, it is part of the GitHub product lineup.
And so, you know, get out there and kind of use that.
So what is GitHub agentic workflows?
We think of it as sort of intelligent automation for GitHub at the high level.
And that...
Concretely, that means you're running the coding agents you know and love, the ones you're very familiar with, your Claude codes or your Copilot CLIs or your Gemini CLIs and the like.
And you're running those with strong guardrails in GitHub Actions.
So that's on an automated kind of basis.
And all of those elements are kind of crucial to us.
First of all, it is intelligent automation continuous AI, like we've kind of been talking about.
You get a choice of coding agent.
We're not tying you into kind of co-pilot, although absolutely that's kind of the default one.
It's a wonderful, wonderful coding agent.
And you get the choices of models and you bring your own key and lots of kind of things about that.
It's all kind of open source as well.
You can kind of look at it all, how it works.
And with strong guardrails is absolutely key.
And we're going to be talking more about security and guardrails later on, I think.
But, you know, the better the guardrails are, the faster you can run with automation.
And it's like train tracks, you know, the stronger the train tracks, the faster you can run.
This is so crucial.
And so many people are kind of missing this in that, in the automated modality, in the continuous AI modality, you've got to be able to sleep at night.
You've got to be able to have those things running when you're not there and responding to things and trusting those kinds of things and making sure that...
making sure all the kind of integrity and information flows are kind of right and reviewable to check you're not kind of leaking information.
I had Ori from Sayara on the podcast here, and he pointed out, if you allow the agent to do something and you run it enough times, it's almost inevitable that it will eventually do that thing.
That's right.
You've got to set the guardrails.
You've got to set the constraints really strongly.
And information flow integrity.
And you do...
need to think about what you're doing.
You shouldn't make your private repositories kind of accessible to agents running in public repositories.
That's, you know, that's just asking for trouble.
You've got to know your basics of agentic security and agentic kind of information flow that people bring in issues.
If you're going to triage those issues, you've got to think twice about triaging issues from third parties.
And we do that mostly through...
forcing you to opt in when you do certain actions, like the default is to kind of ignore issues coming from untrusted contributors, for example.
You've got to opt in if you write an agentic workflow that's going to process that kind of information.
So that's what GitHub agentic workflows are, intelligent automation for GitHub, running the coding agency you know and love with strong guardrails.
in GitHub Actions.
It's a starting point.
We're going to see a lot of different implementations of continuous AI.
We're going to see, we have a thing called GitHub Automations as well, which are more for individual automations for individual productivity.
There's going to be a lot of different kind of automated kind of systems.
There's a whole spectrum of kind of things.
You know, this is how we're cracking open the space to say, hey, you know, that DevOps mentality of CICD.
bring it along to the party.
It sounds like GitHub Agenda workflows are, in my analogy before, maybe more the pipeline than the loop.
So they are the means of interacting it.
They are an event-driven system.
An issue comes along or something, it might sort of trigger that.
But they are not.
It doesn't implicitly...
drive you to create a loop or not.
It's sort of your choice.
Now you can create another time-driven schedule that will now do this on a daily basis.
And so you can do that.
with a programmable sort of system on it, but it is about an agentic pipeline.
Yeah, and you kind of think of it as one step of the loop as well, like taking the issue and creating the pull request.
Now taking the goal and doing one step of that, taking the backlog and doing a small part of that and implementing one, five, ten kind of fixes in one, five, ten PRs.
It doesn't have to be one issue to one pull request.
A single run might create multiple pull requests.
It might process multiple issues.
It might do other kinds of work.
It might work off.
a backlog as a source of input.
It might also be a planning step.
It might sort of decompose that kind of, it might be a research step.
That's actually very common where it says, hey, here every day we're going to do some research on how we can improve performance in this repository.
We're not going to do the work.
We're going to do a lot of investigation of that.
We're going to write a report on like, what are your top five opportunities for improving performance?
And then the...
human the maintainer or the contributors can say let's break that down and actually take that to the next step and actually go and implement those each of those kind of They're actually just doing numbers one and three.
So there's kind of like it allows you to create a lot of different interesting kind of back and forth assistive kind of flows.
Fairly, they're not super fast kind of flows.
It's like CICD, you know, because you really are doing depth research.
That may involve building the product, right?
That may involve taking screenshots and doing research on the product.
And that's where these things can be very different too.
So other kinds of automation you see floating around the industry, because we do have the expectation that a lot of the time these steps can do real software engineering.
They can build the product, they can run, they can take, they can be working off production telemetry data.
For example, they can be kind of getting the latest like 10 failures off your site and say, let's prioritize those and write your research report on how's your site reliability going, you know.
surfacing all this kind of subjective information into useful reports and actionable kind of work to take things forward.
So that makes a lot of sense to me.
And agents are amazingly powerful, and you can run them.
And cloud agents have also become, you know.
much more sort of useful as we sort of run those.
So this is sort of situated cloud agents.
So that's kind of where I wanted.
So I wanted to maybe shift and talk a little bit about sort of the repo as the unit.
So as I think about GitHub Agenda Workflows, which, granted, is a sort of a fairly new product on it, so I'm sure it'll expand many things.
But one of the things that I feel, if I was to explain why is it...
how is it different than any other cloud agent?
You know, it feels like a lot of the core of it is just that sort of repo centricity.
And, you know, one of the sentences kind of in the web pages there talks about turning every repo into a software factory.
And so I guess it feels to me like GitHub agentic workflows are effectively cloud agents that are kind of repo native.
So they're...
They get defined in the repo.
They execute with an expectation that all of their activity and all of that, which other agents can potentially also do, but it's just like a core element.
Is that right?
Is there something else that's sort of fundamentally different?
That's right, and that's absolutely a key.
core design decision about where we started with the CICD kind of analogies, which are also highly repo-centric kind of flows and automations.
And there's a reason why in that there's just so much become simpler.
when you answer that question of what is the bound of operation, where is this running, where is the definition, let's pair that kind of definition.
There are pros and cons to that.
And in the DevOps community, of course, that went back and forth for kind of many years.
Do the definitions of the automation sit in the repo or not?
Yeah, there's a whole world of GitHubs.
And on the terminology, like repo as a factory, I actually kind of prefer to kind of think of repo as a site of production.
a site where things happen, a site where value is kind of produced.
That might turn out to be a factory, but we produce value in all sorts of other kind of ways.
You know, a film studio is not a factory, but it's also a place where value is hopefully kind of produced as well.
So they're all, and the factory analogy is kind of most useful when there's kind of repeating tasks that are the same, that are on the pipelines kind of coming through.
But I think we should think quite broadly about since, AI has such incredible breadth of use.
We should look for other analogies and other kind of inspiration that use a factory sometimes.
That's great.
But also use, you know, people do use words like studios or productions or workbenches and all sorts of other kind of things that kind of make most sense.
So there's another crucial dynamic that...
in the context of the repo and the context of CICD.
And I want to kind of bring this out, which is that when an organization adopts CICD and adopts GitHub or some other similar platform, what they're really doing is saying to the developers in that organization, here's a place called the repo.
where we, as the organization, give you the permission to automate within that bound.
You, the developers, know best what automation you need for your repositories.
You know best what CICD you need.
Therefore, we're going to give you the permission, and you can create these things for workflows which claim some of those capabilities.
They claim the ability to create issues.
Like a security boundary and a decision authority boundary.
Yeah, but we give it to you.
Go for it.
So do what you need to do, okay?
Okay, maybe we have some organizational guidelines, maybe they're reusable actions and so on.
We kind of tame that a little bit to make sure we're kind of keeping a platform centricity in our organization.
But this is an incredibly powerful decision because suddenly developers have access to a limited form of cloud resource.
They have access to compute, they have access to network, and they have access to limited kinds of storage for kind of caching and whatever else goes on.
And that allows developers to be incredibly productive.
And this is why CI-CD is so successful, OK?
Because there is a bound of context where developers are empowered to claim resources.
They do not have to go and ask the...
The cloud team in the company for a new machine, they do not have to go ask for a budget.
The budget is ultimately measured, but this is absolutely crucial organizational dynamic for why CICD is so powerful and so successful.
And we believe that continuous AI or that AI automation in context will go the same way.
There will be a cost center where this stuff does cost money, of course, like CICD.
There will be a cost center where that kind of flows up to.
There can be accounting controls.
There's visibility of where this goes.
There's an authorization context that within that context, we let you use AI.
According to, we still have guidelines and policies and so on, but that allows incredibly creative aspects.
And I think this really matters to me in terms of the kind of the jobs discussion in the industry because there's evidence that...
If you empower people close to the coalface, in a sense, to use AI, not the actual coalface, metaphorical coalface.
Yeah, we're a little bit more posh here.
Because AI doesn't have real coalfaces, I get that.
But then they find amazing productive uses for that, for these technologies.
And so I love GitHub Workflow because it is an empowering technology.
that allows the maintainers of the repos of the people close up close to the software to work out how best to use AI and you have incredible conversations about this with people as you start to unwind you know talking to someone working on children software for children and they say look the thing I struggle with is is is that my the the language used is is getting by by you know is getting written for 15 year olds but I need it I need it suitable for for 10 year olds and of course the ai can make those kind of judgments right there i can read all your documentation strings and all your texts and kind of simplify the language down and just automate that and do that continuously day in day out it's just like incredibly productive.
I think so.
I agree, but I want to challenge it on sort of the two sides of it.
So I think what we're describing here is we're saying a repo is a highly convenient unit.
You know, it is a space, you know, a production site, you know, in which you can say I consolidate a lot of my data on the repo.
And because of that, I can define my automation over here.
My code itself sits over there or my kind of other assets.
potentially context can sort of sit over there and such.
And I think there's a lot of merit to it.
I think also the rise of GitHub Actions, which, you know, functionally have a lot in common with other sort of CI systems, which, you know, are successful in their own right.
But the sort of the rise of GitHub Actions is probably because of that sort of convenience of sort of being built into the repo.
So a lot of value there.
However, there are sort of two other counterfacts to it.
One is...
the actions that happen cross repos so we talk about ci cd as though it's one thing but actually cd oftentimes is not just sort of one repo and so sometimes those things become a little bit tricky and they become another repo so suddenly you have those uh those paths yeah and maybe even more that one is maybe more of a side comment uh and we can get back to it after when we talk about factories as a whole The other one, though, is that in the world of AI, there's been kind of this resurgence of the monorepo.
It never went away, but I think it's fairly clear now that a lot of the agents, a lot of the models have been trained to work in the context of a single repo.
But oftentimes they need information that comes from these other repos.
And so the easiest solution to that is to consolidate everything into a single repo.
But that starts breaking this.
kind of clarity of it's in the same spot because the monorepos can be beastly i mean they can be enormous and they're no longer one thing so using a repo now as a unit of access a unit of security a unit of uh empowerment of who's allowed to do what starts really kind of uh fraying at the edges especially you know what we're seeing like we're doing it here at tesla which is Field teams have access to sort of repos and do that because, again, it's so convenient in terms of agent access.
Pull them in, but you're kind of losing a lot of that identity of a repo as a project.
Yeah, I think you've hit the nail on the head.
There are absolutely real tensions here.
There's kind of a sweet spot somewhere in the middle where, like for some kinds of projects, where just everything works really, really well in an individual repo.
And we kind of recognize that when we walk up to a repo that's just the right size and it's perfectly set up for deployment.
And everything is working really, really smoothly.
And then there are these other tensions that are pulling things in various directions.
I think at GitHub, we have work to do on allowing, making sure.
that the repo is not the only notion of context that actually kind of works effectively, that you could have a collection of repos as a kind of collaborative context, a friend repos or whatever kind of features you might kind of add to that.
And I would, you know, that's one of the things that get...
discussed at GitHub, for sure.
And then there's the question of the organization as a kind of context as a whole.
But of course, organizations can be absolutely enormous, thousands and thousands of repos.
And it's a huge question to kind of scale there.
And even then, you have enterprises which have multiple organizations and so on, which adds a new kind of question.
And there are auth kind of questions.
in all of this kind of work.
And you're right, the coalescing effect into the monorepo, that was a trend that was before AI.
We see it in some projects I'm working on.
I'm saying, hey, look, we should really consolidate this.
Yeah, it's going to work a lot better if we bring it together.
Like there's just no point having these separate.
So yeah, the tensions you say are absolutely real and you always see dynamics where people pull things apart and then they put them back together and they pull things apart.
And I don't have a lot to kind of, I think there's absolutely work we can do at GitHub to tackle, to give more effective solutions to stop you being sort of forced to bring everything into one repo, at least give you more kind of choice and control about where that's, how that works.
And there would also be the question of like, or within a kind of mono repo.
I'm not really an expert on that kind of thing.
Yeah, but I get that there's a question.
Within those controls.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
All right, back to the episode.
So I guess maybe concretely, when we talk about agentic workflows, what type of automations would you say are?
sort of suitable to run within the context of a repo versus which ones are outside.
And when you gave, for instance, the example of a repo pulling in production data example and running it, clearly the production data is not naturally in the repo, you know, for I think many good reasons.
And that system is elsewhere.
So that's one example.
Should that really be a repo-level automation versus something that's outside?
How would you suggest people start thinking about what automation should be done in repo versus which automations?
I guess maybe you technically can do them in the repo, but there's no...
You even shouldn't.
I mean, we have on the...
GitHub Adjunct Workflow site, some really interesting design patterns in this kind of space.
And it's one of the things I think the team are doing really well, is kind of exploring the design patterns.
So there are absolutely cases where we kind of run a whole zoo of agents in a single repo.
And we've written that up as Peli's Agent Factory, Peli's Agent Zoo.
And we've documented 20, 30, 40 agents there that are just really, really interesting kind of repeated things that you want kind of happening in repositories, some really very unexpected ones.
A lot of them are about code improvement.
A lot of them are about performance.
There are ones about styling, for example, things that are maybe a bit really subjective and a really bit unexpected, like GitHubogenic Workflows includes.
a GitHub CLI extension, G-H-A-W, and it has terminal output and we want that well styled.
And it kind of, so the workflow runs in the repository in this case and kind of runs the output and checks it, runs the tool and checks its outputs and say, hey, you could kind of make this look better, organize it better.
And that's actually really, really nice.
And a lot of them have got like 70, 80% kind of PR merge rates.
You can start to kind of track the causality of the repeating kind of action and get statistics on it.
Can I pause on that a sec on it?
So I love those, and there are many, and I highly recommend people go check out.
There are very, very useful automations on it.
But that's a good example, like the styling one, which is the definition of that style.
I mean, it could very well be something that you want to apply.
to multiple repos, is it not?
Oh, yeah, absolutely.
What is the right kind of pattern to think about for, like, where would you then store, is that just a reusable action or reusable?
Reusable skills.
I mean, absolutely.
So you import skill libraries.
So, of course, that's, again, where a lot of the things that are kind of emerging in the individual productivity kind of world, like whole libraries of skills, I know Tesla's kind of done work on these, and, yeah, bring them into your authentic workflows, and you can do that directly.
reference the markdown files, or check the skills into the repo is another thing if you want to go down that.
So it's an interesting boundary there of, so the execution might be defined in it and the skill that you sort of pull in, but if you want to create, for instance, shared access to system or such, you might do that through skills.
The skills might sort of span repos.
Yeah.
And then there are...
I guess MCPs kind of serve as a similar type of shared access.
If you want to access, this sort of gets managed in different ways.
I guess all of those, the tools, the context, and all those separate from the S&P.
They're all a part of the continuous AI narrative.
They're just not, because the continuous AI is broader than the agentic workflows.
Yeah, that's right.
GitHubogenic workflows can bring in MCPs, of course, and a lot of those kind of things flow across.
And you were asking when to run things in repo, when to run things out.
I mean, GitHub is a very open platform.
You can obviously kind of run automations outside thing, outside as well.
tools integrate with GitHub, and that's absolutely great.
I mean, I think our point of view is to give, you know, there are, for instance, data integrity retention kind of policy kind of reasons why you might not like to copy the repo outside.
the GitHub platform.
You've already dealt with this, you've already approved it within your org.
Yeah, there may be legal, whatever reasons, privacy reasons, whatever you can and can't kind of take all that issue and pull request metadata and kind of take it out and then put it back in.
So it's obviously a kind of...
we want a credible uh powerful default kind of option to have in github for these kind of workloads uh we do of course recognize you know lots of kind of depth specialty security agents or penetration testing or all sorts of other kind of things that people do that will happen logically happen outside yeah absolutely we do see some people kind of doing a mix where they their actual focus uh An Israeli startup called HUD is one.
Mai was talking at AI Native DevCon, where their focus is on producing production information, and they kind of need a way to kind of surface reports about what they're finding about bugs and problems in performance results through production probes.
And for them, GitHub Identical Workflows is just a really easy way to do that kind of reporting, kind of regular reporting into the repository kind of context.
And so in that kind of sense, the main intelligence, the main...
The real kind of systems value add is actually over in the production probing that kind of gets done and maybe some AI processing that happens there.
And GitHub agentic workflows is kind of just a secondary role where it's kind of about using some intelligence to kind of surface information.
I think we'll see a lot of that kind of thing, using it as a kind of auxiliary utility in other kind of flows to actually get that information regularly delivered into the room.
It sounds like there's sort of three tiers over there.
You've got information that specifically relates to the code or other content that your repo holds.
So those are the sweet spot.
Like, fine, you automate it.
You bring the compute to the data, if you will, where it is.
You have another sort of pair of things in which a repo might be a convenient way for you to run recurrence on it.
And so the repo is just sort of...
And you see that pattern often, which is you create another repo, like the whole world of GitOps and things like that are worlds in which the repo is basically just, it's purely an automation kind of holder on it.
Yeah, that's kind of side repo pattern where you have an automation running in the side repo and the target repo is actually some kind of monorepo.
Something else.
And in that case, you get the benefit of the kind of enterprise, whatever, legals and security and such approvals if you're already a GitU user.
It's kind of convenient.
Then you might have a third bucket which are cases in which you need other sort of optimizations and specializations whatever bursting to a lot of compute or things like that in which you might want sort of slightly different systems.
You might be applying some proprietary AI knowledge to your repository and that's kind of gonna live off.
to the side.
The side repo one is actually really useful for controlling the authorization about changing the automation as well, like who can write the automation, who can run that, who's the factory runner in a sense.
And so to have that side repo targeting a kind of control, a mono repo is a pretty effective pattern.
So should people, I want to talk a little bit about doing it right and maybe get into philosophy, but maybe one more tactical question.
When people do the automation, there are sort of these two, again, kind of push, almost like monorepo, sort of small repo style debate, which is, should you have one workflow that does multiple things in it?
because there's still a bunch of things you need to configure per workflow and all that.
There's a bunch of convenience to it.
Or should you have, I guess that's how I interpret the agent zoo to be, which is, you know, I've got a hundred different agentic workflows that are sort of running on it.
I guess, what's your sort of bias over there?
What's your guidance?
My particular bias is more towards one workflow, which kind of can do multiple kind of things.
One point of cost control.
Like if you have a...
I'm sorry, I should say my collaborator's bias, Peli DeHalio, is more to have lots of an agent zoo.
That's why we call it Peli's agent zoo.
And part of that's just exploring space.
Like, you know, we don't.
Peli's an amazing guy.
We don't want to kind of constrain what he does.
And, you know, we just like, just go for it.
Just create as many workflows.
And let's see what comes out.
Let's learn, learn, learn.
That's part of what we've been doing here.
But when it comes to actually putting automation, say, into, I'm a repository maintainer for open source technologies, and I don't...
You know, you can create an agent zoo, but that's also a maintenance burden, right?
You know, you've got to maintain that zoo.
You've got to attend to that.
And my attention, like, span is limited, right?
I don't want this to be sucking all my time.
So I prefer to have kind of one point of attention, one point of kind of...
I also maintain multiple repositories, so I'm going to have to look at that across multiple repositories.
Actually, I've made myself a little IDE to kind of help with that, to kind of attend to multiple automations running across, not across multiple, but in multiple automations in different repositories.
So one example is a workflow I've put together called RepoAssist, which is kind of a sort of broad spectrum kind of repository assistant.
It labels issues for you.
It does first response to issues.
So it sort of researches issues and responds to them.
It can choose to implement a PR to kind of fix those issues.
It won't merge the PR.
So my regular daily working with RepoAssist is to kind of look, well, what PRs did it create last night?
Let's take a look at it.
But why do you prefer that, for instance?
Like, for instance, you named there, like, labeling, researching, and opening a PR.
You could have just as well had three different workflows.
Yeah, so this is in sort of a supervisor orchestrator pattern.
And you can have workflows which, you know, are effectively super.
choose to delegate up to more powerful models is an interesting kind of pattern or weaker models for kind of things that's important for cost control.
So why do I prefer it?
I think just conceptual simplicity.
It makes my life simpler and better.
Think about the one entity that you are.
Just to have that.
Cost control really does matter.
in this kind of thing.
Overall complexity, I just find having five workflows considerably more complex than kind of having one.
How does cost manifest here?
Oh, so these workflows are very often done in a scheduled kind of way.
So you may not, with issue triage, you have a choice.
You triage each issue as it kind of comes in and do depth research.
Or you can do it in batch.
Or you can say each day we're going to see what new issues have come in and we're going to kind of catch up in a batch and actually maybe do kind of a batch of them in a single run, which can actually kind of give interesting kind of results as well as it kind of collects information across.
There are things in between that you could imagine having workflows which kind of have a service policy, which says, you know, we're going to triage all issues within half an hour and do depth research on them within half an hour.
That's kind of our policy.
So then that sort of dictates when, how things kind of match.
Yep.
So now if you're...
Repo Assist is a scheduled workflow, so you choose the cadence that it kind of is triggered at, and you hardwire that into the workflow, and you have to go and modify the workflow to change that as things currently stand.
And that's important because you dial up.
You can dial the automation up.
You can dial it back down.
If activity is very high in the repo, you kind of tend to dial it up, and if nothing is happening, you...
well probably you dial it down the software is being maintained and there aren't many issues that remain to be fixed each of those cases i can roughly estimate how much it's going to cost like we have an hour you can see in the histories the average cost per kind of run.
It might be in the cents, it might be in a dollar or something, depending on the models you're using.
And so, you know, you've got a bound, a pretty good bound there.
And there I should add, there are actually very strong cost control features in GitHub Argentic workflows.
I think it's absolutely essential.
It's a thing that's missing from the harness discussion.
I think harnesses should have cost controls and kind of budgeting.
That's an absolutely crucial kind of thing.
And again, it's the kind of thing that becomes natural in the automation world, but it's a little bit...
Unnaturally, individual productivity.
Yeah, it's also hard to know when you do the action again and again, then you can kind of introduce an assessment, an eval, whether you can run it, whether we can model.
Metric, you can absolutely, we've added features that it will do exactly the same work at exactly the same time with multiple models.
It'll only choose the, it will.
choose say the first one as its result but you can use turn that on for a day to get like a sort of examination testing is what I call it where you kind of exact you're doing the models a room full of models is doing exactly the same work at exactly the same time, setting them an exam.
And that really matters when you're using a shared ledger, because if you write anything to that ledger that hints at the answer to that, like a closed issue or something like that, those models will go find it.
It's like, oh, look, I saw another model kind of did it using this closed pull request or something like that.
Now you've got to test those models at exactly the same time.
So we've got a feature for exactly that, which you can kind of turn on for a day and then you kind of get results about like, hey, which models kind of work best.
Work better on there.
In the exams, yeah.
So maybe let me, Viras, we started talking about costs, which I would sort of qualify a little bit as like doing it right.
You're sort of running it.
And maybe I want to touch on two other topics.
One is maybe more around autonomy and sort of the risk of regressions as you rely more and more on these sort of continuous AIs and improvement.
And then the second is safety.
Safety and autonomy.
are related a little bit just like what is it that it's allowed to do so i guess i'll ask the sort of the broader question is as i as i lean in this is sort of like beyond product right but as as one leans into continuous ai and makes changes to the docs to uh you know production deployments fixes How do you know, especially when you think about loops and you think about systems that improve themselves over time, right, gets better and better on it, how do you sort of see or what do you see as a best practice?
for avoiding regressions here?
Do you expect every one of these agentic workflows to generally default to want to have an eval?
If not, when should they have evals to know that they're not changing?
When do you even define what correct and how do you even define beyond vibes what correct behavior is?
Just to clarify, we're talking correctness for the software or correctness for the agentic kind of automation?
So let me kind of reference.
So we're talking about automation.
We talked a lot about how do you sort of systemize the automation of an AI workflow.
But what we didn't talk about that much is around the fact that those workflows are non-deterministic.
And so you run them enough times and they might make mistakes, they might kind of cause problems.
And so I guess what I'm asking around regressions is when we think about CIs, we have some indications.
We know that, you know, tests will sort of run the test eventually could become kind of useless but if they fail we see it we sort of start optimizing them and again it's also like humans have created them i think in the world of ai you give much more sort of vague instructions there's a lot many more degrees of freedom and especially if you have a self-optimizing workflow that modifies itself and and improves how do you how do you know or like what should you have in place to keep degrading to be to be clear in this world the human is always in the loop okay at the at the point of the pull request in particular or the issue so an issue gets created or pull request gets created and no pull request gets merged without the human actually approving that and that will and it's also the case that there are limitations on what pull requests can sort of self-modify the automation, okay?
That GitHub Actions has some built-in kind of limitations about actions creating pull requests, which self-modify the actions.
And those limitations are in there for good reasons, for if you don't want sort of...
malicious actors or somehow kind of mucking with your kind of CICD kind of pipelines.
And some of that thinking does carry over.
So you can have self-improvement kind of loops.
You can have like they can write new skills or they can update new skills and so on, which are kind of separate to that kind of system.
But I do want to kind of say that there are some important kind of controls about how much of that thinking we kind of bring into this world.
And some of that is inherited from GitHub Actions.
But if you just want to.
dive a little into like no pull request gets merged automatically.
But that's not a sort of, that's an untenable sort of destination, right?
Like as you sort of produce and you make more changes, can humans really review?
Like is there a path for scaling this that doesn't require pull requests to be auto-merged?
So that is in some ways a separate discussion.
I mean, in the sense that for GitHub genetic workflows, we have a point of view on that, which is that the human is in the loop.
And the human is in the loop definitely at the point of the modification of the ledger.
Not the full ledger.
Some of the ledger can be modified.
The issues can be written, but the actual code, and it's always been that distinction, the code is kind of the golden truth of what's going on.
And we...
I do understand, of course, that human review is a bottleneck.
So, of course, you can have agentic workflows and other kinds of AI, agentic automation, and all sorts of other algorithmic automation, which do review, review, review.
And when you're...
When you're creating a factory, a lot of the work is on the quality gates.
There's just no point creating a pull request for the human to review when that thing is low quality.
You've just got to create as many high quality gates as possible and automate those to reduce the amount of...
tension you're sucking from the humans involved.
Ultimately, I think you do need, you know, on a, I mean, some products you could almost like auto merge kind of things.
You can, of course, use kind of stacked pull requests or you could just take one long running pull request.
You can push to a pull request branch in an automated way to kind of create a larger and larger and larger feature.
So that's, that is okay.
And that is how some loops kind of work.
And of course we have workflows which, you know, get the CI green.
for example.
So look at all the pull requests, find the ones that are read, work out what's going wrong, make the extra push.
No, not propose, actually make the extra push onto the branch because you're only pushing to a branch.
So we do allow automated pushes to pull requests.
Branches are for, and you can usually constrain that down to be pull requests that have been created by the automation itself.
So you're really kind of carving down to a subset.
So where the industry and different projects will land, I mean, there's no one answer to that, but I do get that there is more and more pressure towards sort of kind of automated, kind of full construction of software and automated absolutely everything.
And we don't go in that direction.
We're being responsible.
You know, there are guidelines about how an absolute human in the loop is absolutely a core guideline of what we're doing here.
I mean, I think I absolutely understand the sort of the driver for it and it's safer.
But I'm also very conscious of the fact that I guess when I think about a factory, I think about a higher level of control plane.
and really everything that is in the current control plane that includes sort of the modifications to the code, we should aspire to make it agentic.
And so maybe it is the next batch of questions around saying, okay, you have kind of continuous actions that are sort of being done in a collaborative fashion on this sort of shared ledger that is the repo, and you propose those, and so you've...
created the automation of these activities at some point.
And I think that point for many organizations is now.
They're at the point in which they already have more changes they can accept.
At Sneak oftentimes, actually very much today.
It's like, hey, the problem is not finding issues.
I have a whole world of issues that I already know of vulnerabilities.
The problem is fixing it, hence the focus from inception on fixing them.
Similarly over here is like if you just got it to the finish line, but you're not able to get it over the finish line and merged, then you just sort of create a review fatigue.
And to be blunt, people are...
not necessarily well equipped anyway at that point in time they haven't written no but that's and this they're not really able to make exactly review right so some of my work is applying uh github agentic workflows to factories inside github okay uh so we have for example a factory for ci performance improvement Okay, now GitHub's a big product, and CI takes quite a while to run.
And we want to improve that, the CI times.
Okay, we want to make incremental, steady sort of improvement by finding the long-running tests and kind of working out maybe the matrix is a bit big or kind of set up, it can become a fixture or whatever.
And my job as a factory creator...
is to deliver high quality pull requests where the reviewer is equipped.
And I love that word equipped.
My job is to equip the reviewer with all the information they need.
to understand why this is being done, what are the risks involved, what are the trade-offs involved, is this a minimal thing, as many quality gate checks on this thing as possible.
Of course, in this context, you get the CICD quality gates kind of as a number one thing.
Now, in this particular example, we're trying to improve CI Perf.
And actually, in this setting, you've got the evidence right there in front of you.
You've got a run that was done with zero changes or the latest run on master or whatever, on main.
And then you've got the run that's done here.
And you see, look, the test took 120 seconds there and it took 15 seconds there.
We can bank that kind of improvement.
We've got the evidence.
You give the evidence to the reviewer.
You say, I found it, and maybe you didn't find the evidence.
You say, this thing should be closed, or auto-close this, get rid of the trash off the production line because this was a dodgy one.
Throw it away.
There's a lot of throwing stuff away in this kind of world, much more than before.
You get trash.
The work has been cheap.
It's easier to throw it away because it was just a guy that sort of ran along.
Cheap in relative proportion.
Yeah, or it's just in the nature.
I mean, cheap or not, it's the nature of the factory if it produces rubbish.
throw it away, right?
You know, the can has got a crush on it.
You know, throw it away.
On our next offsite for our team, I want to go visit some real factories or some real, maybe film studios or something like that.
How do people actually make stuff?
How do they organize this?
How do they tell what's good and what's not?
How do they, you know, how many, you know, let's think about this differently.
I think I have a whole world more of questions on it, but I'm afraid we're sort of like a little bit, a little bit, a little bit out of time over here on it.
So I love, first of all, Don, thanks for sort of coming on.
I love the, I really, really relate to the sort of the term continuous AI.
And I think, you know, there's, there's.
There's kind of a world to define around it.
There's still very much around definitions.
Nice and well.
I love that.
It's always been my joy of working in the tech industry.
So super fun.
Very much like a powerful kind of new repo-centered kind of cloud agent sort of platform in the agentic workflows.
And it sounds like a lot over there that we didn't even sort of scratch the surface over here.
And then some sort of forward-looking questions on the agentic workflows.
to get you to the PR, what happens beyond, how does it relate to the loops, and lots and lots to still explore.
Exciting times.
And yeah, we call it the agentic repository automation revolution is, I think, the title I used for the talk.
And I do think it's going to be at the center of a lot of what practical teams are doing, especially in enterprise, but also open source context.
And it's a great time to be a GitHub and it's going to work.
Thanks for having me on.
Thanks for joining us.
Thanks, everybody, for listening in.
I do very much encourage you to go watch Don's talk, which is on our YouTube channel, the iNativeDev channels.
We'll put it in the show notes.
It's a really excellent talk.
It actually has a lot more content than what we discussed over here.
It's very practical advice.
And I hope you tune in for the next one.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast, it's a community.
And we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more.
And I hope to see you there.