# Agentic AI Organizational Maturity and Continuous Learning Moats

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-07-14

## Transcript

When you mention the word Dark Factory, it's like blasphemy.
It won't work.
I think technically it will work, but you're not organized around it.
The continuous learning is where the knowledge compounds and that seems to be becoming the moat of a company.
Like how fast can you learn?
How fast can you adapt a new idea?
How fast can you do it?
Like even if that means we're writing the whole code base, it doesn't matter.
But like how fast can you get that?
And the people who contribute to that metric are the ones that are really helping you on the journey and not the token billionaires.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pagiani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
We just wrapped up two amazing days at AI DevCon in London.
But the great thing is that we get to do it all over again in New York City this November.
You're absolutely right.
We're going to be back in the city that never sleeps on November 3rd and 4th for more amazing sessions, really engaging hands-on workshops and much more.
Yep, all that great networking, partying, eating and drinking that you've come to expect from AI DevCon.
We think we have one of the best hallway tracks in the business and it's the perfect complement to our incredible speakers and presenters.
We'll both be in person and virtual with live streamed access to all main stage keynotes and talks.
Sign up right now for our Super Blind Bird ticket for just $100, only available for a limited time.
We're really excited to be headed back to the Big Apple.
We hope to see you all there.
Hello and welcome to another episode of the AI Native Dev.
We're still here in San Francisco enjoying AI Engineer.
And we've managed to grab Patrick Dubois.
A third or maybe fourth time for the podcast?
Whoa, okay, yeah, regular.
I'm not sure which, I'm regular, yeah, yeah.
So actually I've talked to a number of people who have said that their favourite episodes are with Patrick Dubois on the podcast, you know that?
Do you need something from us?
No, no, I'm just mentioning it.
Thank you for the compliment.
Observation.
Well, it wasn't my compliment.
Of course.
I prefer the guy ones personally, but no, no, that's fine.
So Patrick, you are obviously...
someone with a wealth of experience in a number of places, of course.
Should I mention DevOps?
You're the stepbrother, I think, of DevOps, right?
And AI has really gripped you more than anything else that I've seen since the DevOps days.
And DevOps days I use as a pun there in terms of the conference, the set of conferences that you built.
What is it about AI that really grips you so much?
So I think I've always taken when there's like a new technology emerging.
And so that happened actually early on with the internet, with cloud, kind of with kind of the collaboration, agile, and then leading to DevOps and then DevSecOps and mobile.
So anytime there's something changing, and I would say I like kind of this emerging chaotic phase of a new technology.
but not just for the technology, but also kind of the impact on the organization and the way we work.
And that's kind of the typical narrative within DevOps is the socio-technical kind of ecosystem.
And I kind of had a hunch that this AI thing was like automation, kind of blowing doors of kind of the ways that we were working.
And that's kind of been, you know, captivating around it.
And one of the things that you've done time and time again with these kind of major shifts in the industry is They're kind of like deep thinking and understanding of what needs to change and how it needs to change.
Obviously, there are a lot of these changes for cloud, for DevOps and these types of motions that we've experienced.
With agentic development, with AI generally, Exactly the same has happened.
And you've done a bunch of research and work to try and help people understand not just the technical challenges and changes they need to make, but also the process changes, the organizational changes.
And in fact, that's what you're speaking about at AI Engineer.
Tell us a little bit about your talk that you're giving tomorrow.
So every new technology will have the impact in the org.
And I feel right now...
A little bit of the technology, I'm not saying it's maturing because that's a broad overstatement, but we kind of get a sense of direction now within AI coding, how it's going from context to harness to loop to what is mentioned, loop craft, loops of loops and so on.
So the technicalities are improving and what you're now seeing in the org is the shift to what does that mean for a developer?
on day to day and that's kind of one way and that's often talked about but what does it mean for a team and a team leader to kind of deal with that change like maybe like one of example is well if your team gets more performant are you running dry on requirements and can people who depend on you for the marketing and the sales if you have 100 features a day how do they package this to the end user?
So there's an impact on the team.
Without massively confusing them every week.
Correct, yeah.
So you have that on a team.
I can see that already a little bit happening maybe more in a shared component, like a platform team who's kind of taking there.
If the loop of loops evaluates to a harness and that becomes more of a factory, is that reusable across teams instead of each team doing their own reusable components?
So their life is changing.
What do they need to provide for kind of the new developers and the new teams doing the work?
And then the VP of engineering, kind of how do they kind of structure the org around that?
So for me, level zero, technical agent enablement.
This is the whole conference, right?
Team enablement, platform enablement, and kind of making sure your organization is enablement.
I think those are like not enough talked on.
But it's also because the one is changing so often that we have older narratives in all the other layers.
And that's kind of where I want to bring the stories.
And it's really interesting that when you talk about, you kind of like mentioned the word maturity there.
And it's like, where are we on this maturity curve?
Because, you know, for me looking at it, there is no maturity.
Because every single time we try and get used to something, there's something else that changes.
Whether it's prompts that we need to start looking at, then it's context, then it's harnesses, then it's, you know, loop engineering.
And it's like every single time we think we know what is the area that we need to spend time on and do research around, there's always something around the corner.
So when we look at maturity, are we just in a continuous exploration and continuous change?
Or do you feel like there are things which we can longer term say, hey, these are things that we need to really focus more on to actually get more mature practices and processes that we expect to keep a long time?
Or as teams, Do we need to actually build that muscle, which is more about continuous change?
And how do we continue to adapt?
What would you say the goals are for teams in that respect?
So I think there is a settling in.
Yes, prompts and specifications will be a keeper.
Yes, harnesses and context will make sense.
The loops will be.
So I don't think this is changing all the time.
It is adding on top of kind of the...
the stack that it's building.
And I think that's quite common in these emerging times where you're thinking like, oh, this was the first slice and that was revolutionary and then they build on top of that.
And that is kind of like building more and more.
I think, I don't know what we, you know, technically in the code generation, if you have teams of agents building on the whole thing, then kind of that is already, you know, one way of kind of solving the coding thing.
Where we might now see things is but how do we prove that the agent did the right thing?
So something that we've been so focused on the generation will move the bottleneck over there, or how can we assess the risk of things that are coming in?
So as the craft matures, it's normal that you feel a little bit lost because every time there's a little bit of a different attention level, but I think it's the industry maturing.
as what is actually the kind of the way the technology kind of works.
Now, that's a different thing from an organizational maturity level, which might not be in the known of the latest, which could be like still working in the old kind of version of completion and think it doesn't work for me and making bad decisions on this.
And that's kind of a challenge in that transient period of like while the stack is maturing.
Now, will it keep evolving?
The rate has been astounding.
that just kind of change over change.
But you can also see some, it's not, it wasn't unpredictable that we'll reach in kind of the, you know, an agent loop, agents of agents.
That was kind of predictable.
But you need the industry to start building that layer out so it becomes accessible for most of the people instead of having to do everything themselves.
But that's the race that we're in, is keeping up.
And for some, that is keeping up the competitive advantage, like almost being in the edge.
And for others, it's more like, yeah, you know, we want to work the new way, but we're not that fast.
We don't have the risk.
We have a lot more things around there.
Where would you say kind of like average organizations are in terms of maturity?
Are we toddlers?
Are we early stages of puberty, maybe?
Where would you say we are?
I think on average, but not that.
high in a big org because the friction of scaling things in an org are different than doing it on a solo or maybe in a team of five.
There are stakeholders, there's like processes in place that will kind of cause that.
So I think it's normal that an enterprise is kind of slower.
Now the typical pattern there is they might have one team leading the charge and then that becomes like multiple teams and then they kind of work on the friction and overcome it.
When maybe, you know, with the comparison of DevOps, when they said, well, continuous delivery, that doesn't work for us.
It was basically, they were saying, we're not ready yet.
It wasn't that the tech was impossible for them to use, but the org wasn't ready.
And I think when you mentioned the word dark factory, it's like blasphemy, right?
Like similarly, like it won't work.
I think technically it will work, but you're not organized around this.
Yeah, super interesting.
Now, I'd love to talk about a new initiative that you've built up over the recent months called the Patents, Agentic Patents.
Folks can take a look at this, tesl.io forward slash patents.
Now, this was a little research project that you built.
And I'd love to actually maybe begin by talking about how you actually built up all the information that's here.
First of all, tell us what is the patent site?
So as the change of the industry is faster and faster, what you typically have is a fragmentation of kind of news.
And there's a firehose of news and things that are changing.
And there's the old information, there's the new information.
My job is to listen in to the emerging things that are happening and kind of also understand how it's like stabilizing as a thing.
So the way that I would normally do that is almost like I would scroll the social media all the time, listening to like conference talks, finding like new ideas and kind of solidifying that.
Now, then I would maybe create a talk of that.
But what I'm now trying to do is to build this almost in an auto-generated website.
and similar to code, it has the same problem.
Like if I would just have it generate, it will be rubbish.
If I had like my own context to it, like my curation, my taste, my harness would be saying, well, not vendors or kind of like things or like it has to have four voices.
So I can put this up.
So I kind of built almost like the research chain.
It's not the deep research per se, but kind of curated and kind of make this.
a tool that I hope is useful for people to say like, hey, I need to talk about defending the ROI to my CFO.
What's the latest thinking about that?
Because there's been a few iterations, like one is like, you know, spent through the roof and then it changes.
So kind of keeping that up to date.
And one of the other reasons was that if you do this through surveys and you ask like, what is changing in the industry?
I feel that is too slow.
because first of all imagine you would do a survey now in an organization about loop engineering everybody's like uh what while there is already the evolved like evolution within the industry that this is where the thinking is going so i think it is i find it more valuable of picking up social signals on the socials to kind of see where it's heading and kind of start extrapolating that and yeah, consolidating that for people, everybody to access it.
Yeah, and you added it.
I think the implementation that you used to actually pull that all together, you used the Karpathy wiki and asked a whole bunch of questions and pulled data from that, which is super interesting.
And actually, we can take a look at the Patterns site here.
And from all of that data, you then displayed it in these kind of categories of agentic coding.
Walk us through, I guess, the sections here that you have, five sections in total.
And tell us a little bit about each.
Yeah.
One thing to note is that this index is hand-created.
And I found that if you let it...
through a bunch of socials, the naming varies and everybody has a different slant on it.
So I think I kind of settled into these different groups where agentic development, everything maybe more what the solo developer is doing, so from vibe coding, spec coding, all the way up to a data factory.
It's almost a progression as we talked about, like when you're kind of building this out, the first thing is, oh, you start with some prompting, oh, you got a better spec.
And I think this is also how the industry has built up the narrative over time on how to do things more technically.
But then the second category is the platform, which I feel the more we're heading into reusable components, whether that's context or harness or the centralized pipeline of the dark factory, I have a belief that that will be something provided by a platform team or the developer experience team as a central piece.
They have already pieces, maybe the MCP proxy and kind of those parts, but slowly they now have a new set of things like, you know, centralized evals for everybody, for all the teams, a registry.
So they're building up kind of those components around that.
And this feels like quite an immature section in terms of adoption from larger organizations, but this is very often.
the piece that unlocks large organizations and helps with scaling, particularly when we think about, you know, those big questions that people, you know, need to ask about, or how are we protecting against X?
Well, we need these guardrails.
We need these policies in place.
And this is all about that kind of like distribution across a large organization.
So this is something that we probably need to, as an industry, get better at.
We'll see more and more solutions.
Whereas that first category...
It feels like, you know, there are people who have already more...
Yeah, exactly.
And people who are successful here in the agentic development, that drives the need, the adoption and the usage across an organization, which then mandates, well, we need this agentic platform to be able to support everyone who's doing things in this agentic development.
Correct.
Yeah.
So it's also the difference between, some explain it by saying there's a solo work.
a solo mode, there's a shared mode and there's a multiplayer mode.
So you unlock a different level if you go from solo to shared and then you ignore the thing.
So there's a compounding effect if everybody starts using the same components, there's a refinement.
So if one team kind of optimizes it, everybody has an optimization.
If you add context for those pieces.
Now, I think you're right in a way that it maybe is a little bit less mature.
But on the other hand, it's not.
AI product, AI inside the product is probably two years ahead from AI coding on components.
They already have the eval system, they already have an observability system.
So it's not foreign to platform teams per se, but the use case for using it for coding is probably unique in there.
And then it has a different way for proxy, how you do evals.
It's not the model, it is the coding agent.
And so there is some newness building up.
So it's almost like we need to potentially reuse existing muscles that we already have, know how to do things, but just apply them slightly differently to the newer ways of working.
And then quality and security, again, that's something that is more painful as an organization as you scale out, if it's not done well, and if it's not done in a way that is, whereby people know what they need to do.
It's similar, I guess, with writing code, right?
Quality and security.
It's painful if you don't know the guardrails and the ways of working.
Talk us through this section.
So I was torn between putting quality and security in a separate section where on the one hand, it does belong in agentic development, in the harness, in the specification as non-functional and so on.
But I found that it...
kind of leads a little bit of its own life.
And I would say that while agentic development has about cranking out the code as a focus and making that code correct, it wasn't about verifying that it was kind of good code.
Now, Harness and Loop kind of changed that a little bit.
But there's still QA testing, there's the code security.
So all the kind of non-functionals probably belong in its category.
And I think that will lead up to, if we remove kind of the bottleneck from AI coding, we solve the specification problem, which is the input.
Now we need to solve the output verification thing.
So I believe this will go on its own path, where maybe some of the latest trends are that after your harness has done things, it's the job of the agent to convince you that it did the right thing.
So not just giving you the PR and looking at the code, but it needs to start helping you do that verification.
Now, within evals, you have an evolution as well.
So imagine you write your code with your prompt.
You have it evaluated with an LLM as a judge.
That gives you a lot of input and output.
Is it correct?
Or you have to look at that.
What you see the more mature people do is they build a verification and almost like a custom tool to deal with all the evals.
Like a fast way of saying yes, no, or I agree that it proves to you.
And I expect something to happen in that quality section as well, where there was a lot.
of tension between like, will we still need an IDE?
Yes, no.
I believe the IDE for writing the code has moved into the CLI, but it's now popping up again as the review interface, which is...
As the IDE is.
Yeah.
Interesting.
And I think that's a move that's going to be made where...
that verification and kind of the proving and it's almost like situational awareness for the user saying yes, no.
Yeah, very interesting.
And why would you say that is, that it would go more to an IDE interface?
Is that just because you would expect someone who is reviewing it to want to...
look through the code and understand?
It might not even be the code, but the fact that they can verify certain features of the API of the website more easily, that there are screenshots, that there's like, you know, recordings of what the agent clicked on.
And so there's a lot more visually than you would say, like, here's a checklist in my terminal.
I personally still use kind of like the CLI inside of the...
IDE just, you know, sometimes to review, sometimes to that.
But I think that is the, well, the current evolution is there's an orchestrator thing within the IDE.
Then there's the evolution about like jumping in when the agents are failing.
I think intent from Augment is kind of doing that, that you can step in when it fails.
And then it's like about having the review making easier.
Again, it's all extrapolations.
It's not that.
We have a crystal ball where the industry is going.
But that's why I think I granted quality and security its own section that I think there will be an evolution there.
Yeah, very much.
Earlier, you talked a little bit about how us humans, the meatbags, are harder to change in terms of adapting to a new process or adapting to use new technology.
The last two are very, very interesting because...
Changing roles is key.
And I think this is probably the hardest thing to predict as well in terms of, you know, how we adjust and how fast we adjust will very much depend on what the short-term view and shape of the org should look like.
And that will be different to something longer term.
And then scaling the org again.
How do you drive adoption through that, you know, those new roles and folks?
So let's talk a little bit about, I guess, the changing roles and changing shape of the development organization.
How do you see that going?
Yeah.
So there's obviously more of the pessimistic view of AI can't do this or we should not be using AI.
Okay.
We shouldn't use AI.
That's more of a conviction than kind of something else.
But it's not there yet.
I would typically say it is a maturity in a way that you can add more and more rules and contexts and harnesses to get it in a certain way.
So one thing I would say is that a skeptical person, get them actually to write the context and have them build a harness.
Where before it felt like, it doesn't work.
What can we do?
Don't use it.
But now I think we're up to a level where we could say, well, you can make it better if you put the effort in.
So it's also one of the reasons where people are saying like, well, we're not using it because it's not good enough because they haven't been working the problem.
Now the other problem in the industry is that they would say, well, we like to do the coding.
We signed up for the coding.
We didn't sign up for a glorified prompter as specification.
And for a while, this stagnated into, yes, we'll all become the perfect spec writer and the code is being generated and so on.
For me, harness engineering and loop engineering, maybe click that that requires a technical component to be built.
So maybe the people who didn't sign up only for kind of the context, they will be working on the harness and the loops and kind of working on that.
So that kind of maybe caters for both audiences who were okay to jump maybe more in requirements and not.
Now, if you scale this out into a org, the question is, how many people will work on the harness?
Maybe that's a subset of all the people that we currently have because it's a shared component.
Now, if a component gets shared, it becomes more complex.
So there's kind of some things and it was similar to DevOps.
Well, you know, DevOps will automate themselves out of a job, but on the other hand...
they could do like things at the scale we didn't imagine before and then we pull the people back so that that is still torn but i think there's now a better outlet for people who still want to do technical pieces and kind of make sure that the agents do a good job and kind of there there's still a part in the review that i mentioned that they can probably start building that review and kind of helping in that piece as well so that that hasn't been solidified into organizations as well so but that's kind of where The roles could go.
As a team lead, I think the trend is that you, instead of encouraging the solo people to improve their own setup, is to push them to a shared thing.
Like, can we reuse your skills, my skills?
Can we put, like, improve that, like, as a team?
So that driving is something I think the team lead does, but it was similar to...
Can we build a shared library instead of everybody inventing their own library?
And that kind of is a collaboration feature, which I would encourage as a team lead to go to.
And then kind of, you know, will they become more product engineers?
That's debatable.
But they have options.
They could do more product engineering.
I think at Lovable, they would say, 80% is working with customers, 200% is improving the technicalities, but I guess it's not for everybody in there.
And one of the things here that I kind of like, and maybe we'll jump into a few of these, hiring in the AI era.
This is super interesting because what's really kind of challenging, I guess, is what made a good developer three years ago.
no longer makes a good developer today.
It's useful to have very key insights into what good architecture, what good reliability needs, mean we need to add into the application or into our design.
But in order to actually be a productive engineering team, to use AI...
to the extent that we can today and look to continuously change to continue using it to the full extent.
It feels like we need a very different type of person, one that is very open to throwing themselves into AI and challenging themselves to continually be better, trying all the new cool stuff and adopting where they think it can benefit.
What would you say a good, if you was to hire an engineer today, What are the key things that you're looking for in an engineer today that you wouldn't have said these would be key things five years ago?
The way that I would phrase it is somebody more as a system thinker.
Is that architect?
Is that kind of like, you know, they care about different things, not just about the code.
So that often qualifies probably more senior people, but it doesn't mean that it has to be senior.
You can be as a junior that you're interested in as well.
Probably not the person that will write the elegant code.
That is kind of, you need to have an eye for taste, but it's not the main thing.
You'll be improving the system there.
And then kind of the collaborating skills and the improvement skills that you're not there for the solo craft.
Like if somebody says, like, I like language X.
Okay, that's great.
Are you only coming if we do this language?
Yeah.
Okay, no, you need to switch languages in that way, similar to a full stack that we defined before, front-end, back-end, it doesn't matter.
You can navigate a bit of the different spaces.
Now, the problem is, like, we used to be learning this maybe by having scars, like, step-by-step in our career.
So that's why it was very similar, again, with the DevOps, where it were, like, people in their 30s.
40s that kind of got enough from the scars learned enough that they were adaptable from there that we see here as well picking it up now if you have two rigid ways of working that like that is probably not where you get hired uh but yeah being open-minded discoverable learning i think the the willingness to learn so i would typically probe it's okay how do we keep how do you keep up what communities are you interested in?
And if they only say their own language, their own stack, that's very limiting there as well.
Very interesting.
And to go back to the patterns, I think it's that type of person that actually when we look at scaling the org and driving adoption, it's typically you don't, when you look at driving adoption, the org, as you mentioned, They mature at different rates and you will have teams.
Typically that style of individual that you just mentioned, you'll have a team which has probably full of those types or has enough of that type to have a big enough momentum that that will be the team that is on the bleeding edge, that is always trying new things out that will learn, oh, these are the best ways of doing things that they can then educate the other teams.
They will get the ROI that other teams will then strive for.
Let's talk about scaling the org.
There are a ton of enterprises that will have, or even kind of like medium and large companies, that will want to be able to scale their AI adoption faster than they are doing today.
What, in your experience, do you see as some of the biggest things that are holding people back?
in the kind of area of driving adoption that you feel they can make, I don't want to say easy wins, because this isn't an easy thing to adjust, right?
But that people need to focus more on in order to unlock greater adoption.
Yeah.
It's the question that I get asked all the time.
How can I speed up my organization?
And I guess there's two parts of it.
Like some parts you can't speed up.
You have to have gone through a little bit of the prompt engineering to know, oh, I need more specifications.
I need more kind of context.
I need more hardness.
And so there is a learning phase to kind of understanding parts of those components that people go through.
So all of a sudden saying, you're going to the Dart factory, like people are like scrambling.
They can't do that.
So that's not per se.
You can speed it up by giving people information, learnings.
Maybe as a team lead, you can kind of say, hey, let's start doing this first step in there.
And I think that's where they can make a little bit of the pace from the team where they say, OK, now everybody's got a fair understanding what prompting is.
Let's move and put all the context and the skills in the repo.
That's like a forcing function they can do that all of a sudden, oh.
if we do that we need to have tests and i don't trust your context and and so so they can pull a little bit of that pace how the team jumps to the next thing so it's almost like an automations that unlock yeah like you know but they control well i think everybody's ready just jump if they don't give the signal jump everybody will keep on doing the same way it's like oh that's limiting oh we're using completion we're not using rules so i think there's a um a value in making that cadence of learning, like either the team or the whole org, like how do you jump in from solo to more shared for like from simple practices to more advanced.
And when you talk about these changes, is it more important for an organization to focus on the teams that are the most advanced?
Or do you feel like there's a greater win with the majority?
Because typically the way I look at it, you'll have an organization where you'll have high inertia teams, teams that don't want to change.
This is how I've done it.
Get off my lawn.
You've got the very fast moving teams, the teams that just really want to experiment and they want to push technology as far as they can go.
They're happy to change.
And then you've got the majority, like the 80% of teams that are willing to change, but when they see value and almost like they are led.
by examples, good examples of teams that are leading a certain area.
So where would you say, like an organization can only spend its effort effectively in certain areas, where would you say it's most effective for an organization to focus, to get the best benefit longer term to improving their overall adoption?
I think regardless whether this is now AI or like Agile or any change, As hard as it sounds, do not spend time on the negatives in the beginning.
Find a success story.
If you can't make that work, the rest will not work.
So it's the super high...
Yeah, because they will show what it could be.
They will also show the benefits and attract other people trying to get the same benefits.
And maybe they'll look up to these people like, hey, what did you do?
And so on.
If you spend time on the negatives, you're not going to get that same thing.
And that becomes later when you have a more adoption already.
Then you're like, okay, why are you still resisting?
We've offered you all the information.
We've seen skill, like parts of it in the org.
What is holding you back?
And that could be a variety of reasons in there.
It probably makes sense is not to over rotate on one team going all the way and nobody else.
So I would say there's like a leveling up.
They spearhead the thing and then you bring others maybe on kind of like parts of the journey already.
And then they kind of like drive kind of towards what the one team is doing.
So it's a fair assessment to say you want to make the fast teams go as fast as they possibly can.
allow then and then the enablement is more for the teams that are following to say let's share the let's share and educate them based on the best practices that these fast teams are going and how can we enable you to take practices that work for your teams you know, learning from these almost like leading teams.
But it's a typical transformation play.
Yeah, that's the same.
I have a hackathon, I share stuff, I have a lunch and learn, I share the success stories, I celebrate like whatever team is doing the best thing.
So that's kind of very common in kind of driving adoption.
This is DevOps, this is cloud, this is developer security.
It's all the same styles.
Correct.
Now, we're given all the success stories.
And that's the reason why I talk about the organizational enablement.
A lot of what happened in the organizations is that, well, we'll give everybody the tools.
And some will pick it up and they go all the way and kind of build their own future and they become very proactive.
Initially, everybody was measuring adoption by saying, well, is everybody using a tool?
Okay.
So that's one way of saying like, okay, we have some kind of...
scaling within the org.
Then people said, well, we're going to look at the people who go crazy on the tokens, that's usage, right?
It's a substitute proxy kind of measurement where you say, well, you know, the people are using more tokens, they're at least enthusiastic about it, but they might not be using this quite efficiently.
If I were to put one measurement right now is how much these people are contributing to the shared components.
and they're fixing the system instead of kind of using more or having a tool.
Because that gives the more buildup of like people improving the shared context, because one context improvement impacts other teams.
And that is typically another skill from like, I can be personally very effective to this is the skill that like helps the whole org.
And that's a different measurement.
And the measurement there is, How many touches does the human still need to do for an agent to do the right thing?
And so kind of measuring that becomes more effective than measuring, you know, kind of adoption just by having license counts.
And if you put that into practice, you'll see like sharing across actually helps that metric to, because if you fix it for one.
Your numbers go up in all the teams.
But I guess that's driving towards that more almost like autonomous factory that we're building.
But I think that is the new metric.
The people who contribute to that metric are the ones that are really helping you on the journey.
Not the token billionaires.
Yeah, no, I totally agree.
So a lot of the conversations that we've had here, it kind of reminds me of...
some of the work that we're doing on Tesla Agent, which is, you know, really, really maps nicely to some of the ways that you're talking about the growth and the scaling.
First of all, if we go back to when you were talking about creating context, sharing that context and making that context the best it can be, I love the way that you kind of mentioned for people who don't want to move to Agentic because they feel like, you know, it's not good enough, it's not where it is.
I love the fact that you say, make them do a lot of work on the skills because that's how you actually get the most out of it.
Now, Tesla Agent, of course, is an agent that allows you to really make those changes well, not just allowing the agent to make those changes with your guidance, but also based on facts from logs and pull requests that have been historically done in those projects.
So that feels like a really strong...
way to kind of like map Tesla agent to some of those things.
But also the way that Tesla agent doesn't try and provide you with a software factory out the box.
What it does is it provides you with a path so that over time you'll get closer and closer and closer to a software factory.
So a big piece of this growing step by step and you know, optimizing as much as we can do in our context and our skills and things like that, as valuable as that is, we need some level of automation to make sure that this is being done continuously.
So how can we, you know, what's a good process of being able to build that automation of the optimization into our daily work?
Maybe I'll first say kind of the parallel with DevOps, right?
A lot of it initially was like, oh, can we deploy faster?
Can we kind of make sure it's happening?
But then the whole monitoring became observability.
And ultimately, that was the feedback channel of the automation.
Like, what do we need to improve, right?
So, and I think part of what you see with the agents, whether that's your skills optimizer that you can put in the loop that it makes you better, whether that's your harness tool that detects.
Things you're doing over and over again that could benefit maybe from better context or becoming a script or a deterministic thing.
So kind of those feedback loops will help you scale this out because you can only do so much and like you contributing the system.
But if it's almost like a positive feedback loop that improves this over time and gives you hints like what to improve.
I think that's really powerful.
Now, the danger of a loop is that it could be a positive and negative feedback loop.
So, you know, once a rule is accepted, it could go wrong, but that's why you need testing, the regression testing, also for harnesses, for anything you put in there, that it's not just I add it and I pray whether it works.
But it's crucial because I often say if you have continuous delivery, and everything's automated, and now with AI, another part is being automated.
The continuous learning is where the knowledge compounds, and that seems to be becoming the moat of a company.
How fast can you learn?
How fast can you adapt a new idea?
How fast can you do it?
Even if that means we're writing the whole code base, it doesn't matter, but how fast can you get that?
And you can't do this without a feedback loop.
helping you to improve that.
And this is a great target, essentially, to make sure we can become as effective as we can be.
But it all comes at a cost as well, right?
And there's a section here, obviously, on scaling the org, which is cost management.
Now, last year, people were just kind of like very much free to, let's try and experiment as much as we can.
Let's try and get as much usage as we can to understand what works, what doesn't.
This year...
Budgets are a little bit more constrained and people are thinking more about the return on the investment and how you manage the cost of everything that you're doing.
Tell us a little bit about how important, I guess, the optimizations are and this automation to make sure we're reeling in a little bit of that cost and we are getting a return rather than just, you know, those billionaire token users that we hear of.
I've been in the position as a VP where I had to say like, oh, look at my budget.
The first year that AI came out, there was no budget for AI.
So where am I looking for it?
So I know it's a struggle, and especially with a lot of the vendors raising their pricing models or changing their pricing models that we didn't cater for.
Now, you can have the instant reaction say, well, everybody stop doing.
Let's remove, reduce the budget and make sure that we keep it in limits.
The problem then is you won't be going on that journey of learning.
Because if you're limiting people, they kind of like work well, I'll still do it manually because I don't have enough tokens and kind of go from there.
Now, others people have kind of start assigning maybe more use cases.
So instead of it saying the agent got better now, is it actually useful for that?
So I think the way you have to look at ROI and cost management is, you know, I need to have a budget for people learning.
and i need to have a budget um that's useful but the constraint of a budget should actually drive you in optimizing that loop because a lot of the people are blindly spending tokens they're using the biggest model out of things they are doing the same thing over and over where they could have been a context change or a harness change that would save them a lot of tokens so if your organization is not your mature you feel like you're over you know, using things.
And the parallel in the cloud was, like, when everybody went to the cloud, every system was a VM.
And everybody was like, this is proving more costly.
And then we learned techniques to put it, like, on the same instance and kind of, like, work from there.
So I think we're in that phase, but it's very risky now for maybe a VP to say, let's close down the full budget.
and kind of stop the usage because then your organization doesn't have that.
I think the right reaction is where are we open spending, but you need to have like observability for your fin ops, for your coding agents and so on.
So you need to work on the telemetry and then say, well, there's no habits.
Don't use this for that.
And kind of have a team work on the optimization.
instead of just closing the budget off.
And let's say, you know, as a VP of engineering, would you say, actually, of all my children, all my teens, I do have...
preferences over where I want to spend money.
And actually the teams that are the highest performing, the ones that are experimenting, I almost want to give them free reign.
Whereas the others, I do want to make sure that they're not just being ridiculous with their spend because we're not actually learning things as much from them.
We're just trying to, it is about ROI.
It's about can we do things better in a cost-effective way?
Whereas these other teams are actually, even if they're spending money and not doing that, we're learning, okay, that's not a path to go, but it's money well spent.
Yeah.
And I think that's the struggle you have with the CFO.
It's like, where's the ROI?
Where's the ROI?
Like they're willing to spend if there's a return.
And I think it's not per se that I would say the best agentic team will get the budget, but I'll put the best agentic team on the most important business project to get like a return back.
And I think that's kind of the challenge is they're not always the teams because The biggest use case might be the most risky use case as well.
So there's a lot of nuances to go there.
Yeah, that makes a ton of sense.
Patrick, this session's gone very fast.
I love these chats learning from you.
And for our listeners, tesl.io forward slash patterns.
And you can read all the wonderful...
you know, research and data as well as some of your takes and your categorization there.
This is not a static site, though.
This is changing over time.
And there's a lot more that I think you're planning on adding to it over time.
So, you know, absolutely take a look at that site.
Very, very useful set of resources for you.
Yeah.
And if you find anything missing or you have an interesting story, I'm also very interested to listen and learn from you.
Thanks very, very much.
And I hope and I'm sure everyone at home enjoyed that session.
Extremely insightful.
TESL.io forward slash patterns to learn more.
Thanks for tuning in and see you next time.
Bye for now.
Thank you.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast.
It's a community.
And we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more and I hope to see you there.