# Agentic Coding: Maturity, Context, and Enterprise Strategy

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-03-10

## Transcript

The hallucinations thing is a lot of people don't necessarily understand what is likely to lead to that scenario.
And that's one of the reasons why people get misled by like one-shot examples of, oh, make me a to-do list app.
But real work isn't like that.
It's like, I've got a very specific internal library that you've never seen before, you've never been trained on, and I need you to get these specific details right.
Are good engineering practices needed for a gender development?
They're certainly beneficial.
And one of the things that the DORA report found in 2025 was that teams with high levels of development maturity went faster when they introduced agentic coding.
Teams with low levels of maturity went slower.
I can think of some SaaS providers that I've worked with who've had no competitors for 20 years.
In the last six months, two have popped up replicating their feature set entirely using agentic coding.
There's a chance that you will fall behind if you do not stay on top of this stuff.
What three tips would you give an organization who wants to roll out a Gentic software development across their company?
Firstly, back in November, we hosted the first ever in-person AI native DevCon in New York.
This June 1st and 2nd, we're bringing it to London.
It's two days built for AI-nated developers and engineering teams.
One day full of hands-on workshops and one day full of practical talks on agent skills, context engineering, agent orchestration and enablement platforms, and how teams are actually shipping AI in production.
Join us at the brewery in London near the Barbican for all of that.
Plus networking, parties, giveaways, and a room full of people building the future of AI native development.
You can also join us from anywhere in the world via the live stream.
As you're listening to this podcast, you get 30% off your ticket with code POD30.
Just head to ainativedevcon.io and we'll see you in London.
In this episode, we're going to be asking the big questions.
What are the hidden pitfalls that you don't even know might be ruining your chances of success with agentic development?
And what are the most important factors that a developer needs to know about in using agentic coding tools, as well as rolling it out across your organization?
As I mentioned, joining me on this journey is Daniel Jones.
Daniel, tell us a little bit about what Resync does.
So Resync is a consultancy in North Europe, and we help people with AI native transformation, whatever that means.
And it's a thing that is kind of emerging as the state of technology evolves.
We kind of help people with across three threads, really.
One is agentic coding.
So using AI in the software development process to increase productivity.
Another is helping people build things that leverage AI.
So maybe where your product has some ML features in or you're building agents.
And then the third string of that is helping the non-software parts of the business.
So accounts, marketing, how can we introduce agents or workflows that make them more productive?
So looking at AI, AI nativity, AI nativeness across all of those different strands.
I love that, the AI nativity.
We should definitely celebrate that once a year.
So Daniel, you work with a lot of large companies, enterprises.
talking about AI adoption and how those companies can best take on and roll out agentic development across large numbers of teams.
Today, we're going to be talking about some of the advantages that can bring, but really talking about some of the pitfalls that people very often fall into when trying to adopt and some of the best practices that we can add in.
So why don't we kick off by talking about, you know, people having good engineering practices to start off with, and where the pain exists, whereby if someone doesn't have a good engineering practice and they use agentic development, where that pain could be exacerbated.
So are good engineering practices needed for agentic development?
They're certainly beneficial.
And one of the things that the DORA report found in 2025 was that teams with high levels of development maturity went faster when they introduced agentic coding.
Teams with low levels of maturity went slower.
And this makes perfect sense when you look at it through the lens of the theory of constraints.
You know, if you take one part of a system, speed it up massively, then you're going to create bottlenecks on either side.
The interesting challenge there is that we don't know where exactly that tipping point is.
So we're not sure how good you need to be in order for things to get better or how bad things need to be for it to be throwing petrol on a dumpster fire.
And when we talk about how good, what does good look like?
Is it...
Is it the documentation that needs to exist?
Is it just people following good workflow practices?
What does good mean?
So good can be measured in a number of dimensions, and there are quite a few.
So we've got things like test coverage.
We've got things like alignment on what good looks like for, you know, the coding standards.
We've got things like the size of batches that they're working on, like how big and well detailed are their stories.
We've got things like the path to production.
What is their lead time to production?
Can they get changes out quickly?
Because if coding agents can chuck out commit after commit, minute after minute, but it takes three days to get anything into production, which is quite optimistic for some enterprises, then the tip of the branch is going to have moved on miles and you're going to end up with loads of merge conflicts when something goes wrong in CI.
So there's quite a lot that really...
would be considered just general good practice from the last 10-20 years of software development that needs to be considered just because everything goes much faster now.
So having those good pipelines, those good workflows, it enables that speed so that you don't just have one massively sped up component, which is the actual build of the code.
And its point is having that if the rest of it is just going to take time.
Tests, you mentioned tests.
Why are tests so important in a true agentic development?
So tests are important for one reason, but there's some nuance here.
And maybe we can get into the ways that maybe some types of tests are less important or at least less important to humans.
First and foremost...
If you don't have good test coverage in your software, then how is an agent going to know that it's broken something?
And really one of the things that people who are new to agentic coding need to realize and often kind of miss is that your agent needs to be able to perceive something.
If it can't perceive a problem...
then it can't possibly react to it.
So whenever you're getting frustrated with a coding agent, you need to consider, like, could it have known better?
Did it have the information to know that it was doing the wrong thing?
And so if your agent keeps on breaking your software, well, did it get fast feedback from a failing test suite?
If you've got low test coverage, then it's quite likely that it could break a whole bunch of things and be absolutely none the wiser.
And that's not really the agent's fault because you haven't given it the information or the ability to perceive the fact that it's making a mistake.
And is it actually the test runs that inform the agent, yes, this is broken because the test failed?
Or is it actually the agent using the test cases as context almost to understand, actually, this is...
This is expected behavior.
This is kind of what we intend the application to do.
That's a really good point.
And it is both.
Certainly the failing tests, like if your agent is well-mannered enough to actually run the tests all the time, then it should figure out that it's done something wrong.
But using the tests as documentation and guidance of what is expected is quite important.
I think that, and this kind of brings me to, I mentioned nuance earlier.
of maybe unit tests aren't for humans anymore maybe those are too low level like if unit tests help the agent figure out whether it's broken something great okay you know agents go and write as many unit tests as you like But acceptance tests, kind of outside-in acceptance tests that define what a user should be able to do with the system, those are important for your ultimate kind of safety barrier of this is what I need it to do.
But also that guidance that you talked about of this is the expected behavior.
This is what it needs to do.
These are the features that we want to have, regardless of whether they work or not.
This is what it is supposed to do.
And that kind of then bleeds into things like spectrum development and where is the source of truth?
Is it the tests or is it the spec?
But these are all things that, going back to the previous point about a lot of the last decade or two of good software practice.
Things like behavior-driven development, a good BDD test suite should have been describing what the software does at a kind of human accessible level anyway.
So if you've got that kind of thing in your code base, then agents will be able to work with it better.
And it makes sense really because if we as a developer are turning more into intent machines where we describe and define the intent that we expect and it's actually the agent's responsibility to effectively...
generate that implementation that describes the behavior we ask for.
The tests we care about should be to validate the intent that we're asking for, whereas it should be the agent's responsibility to create the unit test because they're creating the implementation.
So it would be weird for us not to care as much about the code, but then to care about the unit tests for that code.
It's nice to kind of keep those test and intent or generation levels the same.
What about typical development hygiene, let's call it?
So things like version control, those types of things.
Is that as important with organizations doing agent development?
Absolutely.
Version control in particular is critical to...
making good use of agents because sooner or later your agent is going to do something that you don't want it to it's going to commit a binary into git and then you're only going to notice like four or five commits later and you're going to want to remove that so if folks aren't able to do things like interactive rebasing which you know to be fair can be a little bit fiddly and i found a little bit mind-bending when i first learned how to do that then you're going to end up with quite a messy code base that will mislead humans and it will mislead agents another thing to consider when it comes to version control is that commits and branches are cheap right they're next to free they add very little overhead so making lots of small commits which again is like people would argue or i would certainly argue is good development practice anyway maybe making even smaller more frequent commits is better because then you can throw away the work that your agent's done and When it comes to behavior change, a lot of adopting agentic coding is not just knowing how the tools work and gaining new knowledge.
It's unlearning old habits.
So being happy to throw away some work that an agent has done because you've got plenty of commits that you can go back to, you're going to be quicker throwing away something that an agent did and then starting again than you would have done.
typing on the keyboard yourself like it was, you know, 2022.
And agents aren't precious about their code.
They don't care about throwing that code away, right?
Exactly.
Yeah.
I often say to people that you should imagine you are the worst software manager.
possible and you know insist that you treat your agent a bit like a junior or a member of staff that you really hate and want to demoralize like throw away its work get three of them to work on the same thing at the same time and only have one of them have their work used like those those patterns actually make you kind of more productive even if when our ai overlords look back through our chat history in 20 years time we're all going to be in trouble it's always good just to add in the odd i respect you or you know i very much appreciate because you know We're not always going to be top of the pecking order.
Indeed.
So let's continue talking a little bit about version control.
Who should decide about the commits?
Is this something that we just, you know, maybe as a hook to an agent or something which we just say, look, after every reasonable amount of code change, whatever that is, I want you to make sure you commit into a Git repo or something like that.
Who decides?
I think that is a key indicator of how psychologically comfortable people are on their journey to adopt agentic coding.
And certainly to start off, I think folks are much more comfortable making commits themselves.
Like, ask the agent to go and do a load of stuff.
I'm going to read the code.
I'm going to validate it.
And then I'm going to commit it.
As you build confidence...
and build uh competence with things like spectrum development it becomes more acceptable uh to people psychologically so i'm gonna let the agent do the committing here anytime that we introduce um a human having to take an action like we slow things down by you know several orders of magnitude so if your agent is sat there going i'm done do you want to commit this and your attention is elsewhere then it's making way less progress than if you let it do that yeah um But then there need to be some kind of guardrails around that and making sure that it's not just steamrolling ahead and definitely not doing any force pushing.
Yes.
It's funny, actually, because when I first started building with agentic coding agents, I did manual commits, exactly the same.
I did manual commits.
And actually, I was able to go so fast, I actually realized, oh, crap, I should have committed some of this.
you know a long time ago the one thing i do love now i always let my agents commit for me as they as they as they feel like changes are being made the nice thing about that is you actually when you go back you can you can dissect commits and you can go back to a certain point very very easily because it actually commits pretty often and like you say it's free it's it's very very cheap to to do these commits it makes make software development going back retrospectively much nicer.
You mentioned a little bit about safety when obviously there's a YOLO mode which when we do want to increase that you know the autonomy of coding agents we do want it to just go away and do some stuff without constantly asking like a child can I do this can I do this.
What do we need to think about from a safety point of view, particularly from an enterprise, that it's not going to run havoc on our machine?
Yeah, I mean, enterprises are extremely sensitive to this kind of stuff.
And one of the points that was...
provided much more friction than I realized was the use of containerization and people's level of maturity with that.
Having come from a kind of cloud native background where we were deploying things and Kubernetes and containerizing all the things, I kind of took it for granted that everybody would be familiar with things like Docker and Kalima and, you know, alternatives.
Turns out a lot of enterprise developers aren't, and a lot of enterprises aren't necessarily set up for people to be able to run things in containers on their development machines.
If you don't, then you've got this situation where people can be running coding agents with dangerously skipped permissions natively on their host.
And I'm pretty sure those people are going to go to hell if they keep doing that.
So running things inside containers is definitely the way to go because then you can let it off the reins and you don't need to be there to constantly say, yes, you can do that.
Yes, you can run LS.
Yes, you can cat that file.
I was optimistic about dev containers.
So this kind of open standard that came out of Microsoft and VS Code supports where you plop some config in a directory and VS Code will detect that that config is there and then it will relaunch itself inside a container.
Nice.
That's all great.
But the UX, like the config is, it fits into a sweet spot of hallucination that maybe we're going to talk about in a bit where it's recent.
And it's not super popular, but it's popular enough that agents tend to hallucinate the config, which is super frustrating.
So dev containers are a nice solution to that, but maybe not quite mature enough.
And I've certainly seen enterprise developers struggle with that and spend hours, if not weeks, kind of fine-tuning their dev container config and not being able to move forward and make progress.
So that whole containerization thing, it's like MCP as well.
I understand why it's this way, but it would be lovely if all MCP servers were run, like all the standard IO ones were running containers by default.
We've had Docker for quite a long time.
It would be nice if everything was just secure out of the box.
And I was chatting with Oleg Shaliev, actually a former colleague of mine who works for Docker, developer advocate at Docker, and I was chatting with him about Docker sandboxes.
And they seem like ideal for this, exactly this type of thing.
And I think enterprises are really going to want their developers to actually start developing in this.
When we talk about the setup cost, though, is it easy to develop in a sandbox compared to obviously, you know, as a developer, I just want to go to a terminal, kick off Claude, and I can just play, use my local resources and things like that.
How much is there if we're doing this in a container?
It's not massively, massively problematic or massively burdensome, in my opinion.
And I think one of the things about enterprise developers is that they're often under a lot of pressure to deliver features, features, features, features, and they don't necessarily have the time to invest in trying out different tools and different ways of working.
And so if you're in that environment where you're not given freedom and space to explore new things regularly, and you're of the mindset like, i learn one thing a year i get sent on a course my company pays for it i'll get a certificate at the end and then i've learned a thing then it can be quite uncomfortable to be using new tools like containerization dev can containers to be using maybe a different terminal emulator if you're used to using an ide like a big a full fat kind of graphical IDE and then somebody's saying oh okay spin up something in a terminal emulator and then you're going to need to authenticate with clawed code but then there's line breaks in the URL and you can't redirect back to the port on localhost because it's inside a container and it's not exposing that port and then you need to copy and paste tokens those little bits of friction definitely put people off and it's something that's worth considering of guiding people through those just because you don't want people to bounce off before they've got to see the wonder of like how productive these tools can be yeah yeah absolutely and really i think it's the dangerously skip permissions the yolo flag of i'm just gonna let clause or whatever agent you're using go off and do whatever it wants whereby those are the instances that actually you really really need it in a container because at least in the other instances you still get that choice of saying yes i'm happy for you to do this or no no no one needs to have access to that that's fine you mentioned spec driven development a little bit and actually i think i think the providing it with the with the right intent providing it with the right instructions um is very very important we can do that in a number of ways and maybe talk about skills and context in a little bit but when we actually talk about the task that is required and i guess Previously, we would relate this to like a user story or something like that.
How important is it to provide an overly verbose or a concise user story?
What are the important things that we need to add?
The story that you can provide to your agent and the unit of work that you provide to it is hugely important in the level of detail provided.
And something that I think we're exposing as we get agents more and more into the mix is the amount of knowledge in the head.
that is required or the amount of knowledge in the head that we all take for granted when we're doing software development.
So Donald Norman in the Psychology of Everyday Things, the book was renamed The Design of Everyday Things, talked about knowledge in the head and knowledge in the world.
Knowledge in the world is discoverable.
It's written down somewhere.
So, you know, having your coding standards on confluence or something like that where maybe an agent could get to it.
Knowledge in the world would be a well-written story with acceptance criteria so you could write a test from it.
Knowledge in the head would be, well, I spoke to my product manager last week and I know how they want this to be implemented, but it's not written down anywhere.
If you are missing that kind of information and there's lots of human context required or you're used to having that and just going, oh, I'm just going to ask on Slack for a clarification on that, that's a really bad spot for agents to be in.
And they're not going to be able to produce results that you want.
They're not going to be able to implement things without making stuff up or asking a whole bunch of questions.
So having a decent level of specificity there.
is important.
And this is where, so with one of the organizations that I've been working with, we've asked them to connect their coding agents up to the Atlassian MCP server and then start critiquing stories, which the product managers are maybe, you know, raising an eyebrow at, but it's putting back pressure on product, on how well specced are these things.
And that's not necessarily because product is doing a bad job, it's just they're used to having a much more informal.
relationship.
And something I've seen in quite a few organizations is you speed up the software development part and that puts pressure on product almost immediately.
They're like, blimey, we haven't got enough stuff figured out.
We can't keep the agents well fed.
We're going to need to massively speed up what we're doing here.
And also, we're just so used to, as humans, maybe not going into all the tedious detail we need to, because if somebody doesn't understand, then they can come and ask us rather than...
meticulously writing out every single detail.
I want to kind of like switch a little bit now and talk about the tools people will aim to use for a lot of these things.
In the past, it's been very much a discovery kind of thing and developers have been playing with as many tools as they can just to try and get used to AI and switch mindset into it.
And I can do this with AI approach.
What do you see from a organization and industry point of view of how companies are trying to adopt a range of tools or focusing on specific tools?
2025 was definitely the year of tech leaders going, I'm going to let people use what they want.
And, you know, I'm encouraging them to try stuff out.
And we've got somebody over here using GitHub Copilot, somebody over there using Claude Code.
And I think part of that came from a lack of knowledge and comfort from tech leaders.
kind of being like, there's so much to keep track of.
I don't know what the latest thing is and I don't want to like, you know, bet on any one horse because I'm not sure.
And then also a fear of if I say we're going to use this tool, not only might it not be the right one, but it might be that developers don't like it and then they're going to push back because they've got a lot of pride in their job and the sense of identity from it.
So 2025 was a lot of kind of soft touch, organic, disorganized trialing of things.
I think over Christmas, Anthropic did a great thing for their marketing of giving people free usage over Christmas.
So lots of board CTOs and VPs of engineering trying out claw code and go, wow, this really works.
So 2026 has definitely seen a shift in people wanting to be much more structured of like, okay, we need, we've seen what's possible now.
We need to roll this out across the department, across the team, across the organization.
Yeah, we're seeing folks try to standardise on tools.
And I think there's still quite a lot of anxiety over what is the right tool to pick from an enterprise and organisational level.
A lot of it, sadly, is driven by licensing and the logistics of how we pay for these things.
You know, there's some speculation at the moment as to whether Claude Code licenses or the subscriptions are going to get revoked for enterprise usage.
I've seen some chatter about that.
But, you know, if you've got a thousand developers and each of them need to expense an Anthropik subscription every month, that's a huge amount of overhead as opposed to having an enterprise agreement.
So we see quite a few people kind of opting for GitHub Copilot just because it's part of their GitHub Enterprise offering.
Whether that's the right...
choice for every organization though is an entirely different matter.
And is the worst decision indecision here?
I think that there's a strong case for that in that often when people are faced with transformative change they either jump in with both feet and no plan and we saw a lot of that in 2024 of like we need to do some AI what do we need to do I don't know just do some AI here's some budget.
and definitely saw that happening at large financial institutions.
And, you know, lo and behold, they didn't get any return on investment.
Who would have seen that one coming?
But then there's the alternative of we're going to wait until this is settled, a settled science.
And we're not going to waste money getting up to speed on this.
And the speed of change continues to increase.
as time goes on.
So holding off, I don't think is a viable solution.
Having some amount of budget, some amount of time, some amount of slack in the system to explore and trial things out is the most sensible strategy because otherwise there's a very, I don't want to fear monger, but there's a decent chance that you will get outpaced by a startup who is embracing these things.
I can think of some SaaS providers that I've worked with who've had no competitors for 20 years.
In the last six months, two have popped up replicating their feature set entirely.
And these are brand new startups who presumably are using agentic coding to just replicate their software very, very quickly.
There's a chance that you will fall behind if you do not stay on top of this stuff.
And then if we do get to a point where it's settled science.
Do you want to be starting from a standstill?
Because there are approaches, things like managing context windows, as far as we can see, if models carry on working the same way they do and the architecture doesn't change radically, then this is going to be an underpinning constraint of all agentic software development going forwards.
Even if we end up with software factories that build upon these things, you need to understand the fundamentals underneath.
So that's a good segue.
Let's talk about tips.
And I'd love to talk about tips that you can give developers directly.
And then we'll maybe talk about tips for rollout across an organization.
So tips for developers.
And you talked about context.
Let's go deeper into that.
Context is you obviously provide your message as to this is the task I want you to do.
Context is really important.
Now, whether that's skills or instructions about how your organisation wants to develop software, those types of things, how much is the right amount of context?
So looking at academic research, certainly the NoLima paper from 2024 suggested above 30,000 tokens of context, reasoning ability drops off by about 15%.
There is another paper, the name of which escapes me, that was a repeating experiment in 2025.
for GPC 4.1 era models and found between 30,000 and 60,000 tokens.
Again, reasoning capability drops off.
That's not very much.
Like there have been, I mean, Clawed Code, you know, three releases a day, so things change all the time.
But I have certainly, there have been points where I've opened up Clawed Code, no MCP servers installed, no Clawed MD, and it's already using like 40,000 tokens just for its system prompt.
So it doesn't take much to tip your model into the point where it's going to get confused.
But one of the challenges with this is that when I was starting developing agentic coding training for people, I was on a video games website over Christmas and gamers tend to hate generative AI.
They think it's going to be the ruination of all things.
I was like, oh, I'm going to try and like educate people about why it's not all bad.
And I mentioned in the comments on some website.
Why was I posting comments on a website that I was doing agentic coding training?
And somebody posted a very snarky response like, duh, why do you need to teach people how to use coding agents?
Like, don't they just work?
And I was like, you know what?
That's actually quite a good point.
And because these things just work, it's not until you have problems that you need to understand what's going on underneath.
And context windows and context management is a good example of that.
If you just use a coding agent and everything's working fine, you don't need to worry about context.
And then, you know, conversation goes on a bit and then it starts to fall apart and you get more frustrated.
But it's not obvious that's because your context window, the amount of context that you're using has gone over the effective limit or that you've distracted it by talking about too many things in one chat.
So that is something that quite...
you know clever engineers who are new to this it's not obvious it's not immediate to them yeah and i always say like a developer they know when they're in the goldilocks zone of context whereby they're like one-shotting prompts because they they're giving enough context for to have that acceptable answer they're not giving too much context so that the the the agent gets confused it's just enough how should a developer keep just enough context is it something they should proactively monitor or something they should react to if the agent is going astray?
I was certainly of the opinion that people should be proactively monitoring this and also workflow habits of, you know, whenever you're at a clean break, clear context.
Whenever you need to ask a off-topic question and like my...
mind does not stay in one train line for very long so i'm normally thinking about three things at once like use a different chat use a different app like i use warp um a agentic terminal and if i need to ask something linux c command line i'll do it in warp if i'm doing something in claude code and i'm trying to build a feature with claude i'll try and keep that on topic i need to do research i'll go to a web browser and use you know a chat app like chat gpt or gemini so splitting up the conversations has the advantage of not distracting the model with too many concepts in one chat and not extending the amount of context that you're using so clearing your context regularly not going off topic are things that are valuable to do the other thing that kind of may be slightly different is not adding all of the mcp servers yeah and not adding all of the instructions for every possible eventuality of how to write software into your agents guidance your agents md there was a paper that came out of zurich a couple of weeks ago that showed that Adding stuff to your agents MD will reduce the quality of outcomes by about 20% over and having no instructions whatsoever.
And it's interesting looking at things that have been fashionable over the months as we all learn about this.
I remember certainly September time people talking about, put all the things in agents MD.
Every time it makes a mistake, stick something else in agents MD.
But what you're doing there is you are saying something that's largely irrelevant most of the time in the hope that it's relevant this time.
So that's...
Not a great thing to do.
Then it was like, I'm going to add all the MCP servers for every eventuality.
And that causes its own problems.
Like a lot of people don't realize how tool calls work and the fact the full menu of tools needs to be provided to the model on every single prompt just in case it wants to use one of them.
So skills and progressive disclosure are a much kind of better solution to this where...
Instead of sending the model all the details about skill, there's like a little summary and the model can ask for more information if it wants to.
And let's talk about that because I think that's really important.
There's a couple of things here.
One is like the behavior under the covers and the second is more activation, which is a global problem today, but it's something that we expect to get better.
So you talk about adding things into the agent's MD.
Anything that you add into the agent's MD file.
the agent will fully read.
It doesn't necessarily mean it'll act upon it, but it will fully read.
When you add an MCP, again, there's a description there.
It chooses when to activate it.
It's similar to a skill, I guess.
A skill, like you say, there's a description there.
It chooses when to use that.
These are kind of more of the on-demand style approaches.
So if the LLM or agent feels like it needs to learn more about this thing or feels like it's doing something, it triggers a skill as a result of...
wanting to do development in a certain way or wanting to write code to a style guide, it chooses to then use that skill, read that context, etc.
So the overhead there is very real.
If you have hundreds of MCP servers, hundreds of skills, it's going to get confused and it's actually probably not going to pick the skills you want it to.
Do you have experience or advice in terms of not necessarily what the right number of skills or MCP servers are because it's very subjective, but...
What does what what advice would you give to someone who is thinking about adding yet another skill or yet another MCP to their environment?
I think add them judiciously, maybe have loads that you rely on, but don't have, certainly for MCP service, don't have them all add added and registered at the same time.
If you do need registered at the same time, then maybe selectively filter out.
their tools if your agent doesn't support that natively then maybe you could use an mcp gateway for that for skills i think there's slightly less of a problem with that because the progressive disclosure but you might end up with multiple skills that have very similar descriptions and then the model's not sure which one it wants to ask for more information on so yeah that that could be a problem.
And I know, having spoken, I spoke to Amy Heinecke from TESOL, who was talking about this and some of the challenges that you folks face in making sure that, you know, the things that you're building get selected by the model.
And it's a thorny problem, so I gather.
Activation generally is a thorny problem in terms of making sure that the agent chooses the right skill or the right, you know, the right piece of context at the right time.
And it's...
It's definitely a problem that exists today.
I think it'd be less of a problem as time goes on, as agents get more used to using skills.
But yeah, it'd be interesting to see how that turns out.
So when we think about a user, a developer, part of a team, building a certain part of a project that is much, much broader.
When we think about context management and replicating their environment to share, essentially, the environment that...
that developer's environment with the rest of that team.
What's the best way to be able to replicate that context, the context management from one developer to another?
Is that something that should be more global, stays with the repository, or is that something which should be more subjective to the developer, how they want to work?
It's an interesting point, and one of the places where team dynamics and the conflict avoidance that sometimes is, arguably necessary or allowed in large teams to proliferate can kind of create problems in that if you've got lots of developers with different ways of doing things then maybe in the old world pre-agentic coding then that was acceptable when you knew that you know rachel over there has her way of writing code and steve over there has his way But if you've got an agent going over that code base and then it's being confused by two different code styles, then that becomes more of a problem.
Generally, I would say that having things as global as possible and consistent is going to lead to better results.
And as we move away from individual developers having their own idiosyncratic styles that they like to deferring more of the implementation to agents, does it matter if the code's written the way that you like it versus the way that I like it as long as the agent is able to deliver useful outcomes and features more quickly?
So trying to standardize and make sure that these things are available to agents when they need them, whether that is global and maybe you put that in Confluence or a wiki and you give your agent an MCP server so we can access that.
The problem with that is drift.
And when it changes, then you're probably better having that in.
your code repository so you the guidance is pinned to a particular version and as that evolves it's clear you know um wikis are often where documentation goes to die so yeah yeah you know if you want stale and out of the date information that's a good place to put it so probably in the repository um would be the the main place to do that and then when you get folks um having context and guidance that they're keeping local to themselves I can imagine that leading to problems of if you've got conflicting information and then you've got one agent doing things one way on one person's machine and another.
You know, those are the kind of patterns that we've generally throughout computing gone, you know, this is going to lead to bad outcomes.
We want things done consistently and universally across a team or at least throughout a code base.
Yeah, probably worth mentioning TESL here as well, actually, because obviously TESL is that kind of like that package manager for skills and context.
So actually does a lot of what you mentioned there, a lot of the, like the versioning of skills and context, the ability to almost like have a manifest file within the repository that then pulls in the right context as per pinned in the manifest.
But I think what you kind of like mentioned is really interesting whereby things like coding styles, I've never heard an agent argue whether it should use spaces or tabs, but there are things that are still important.
The stack that we use, the methodologies of how we develop with test-driven development.
How we choose to develop is going to be important more than the cosmetics of the code, I think, which is super important.
You mentioned hallucinations a little bit before as well.
Do you feel like, how do you see agents coding correctly?
How do we use context to maintain accuracy?
So the hallucinations thing is a lot of people...
don't necessarily understand what is likely to lead to that scenario and so in the training that we provide we get people to do an exercise with a really old model so gpt 3.1 where we can deliberately mislead it and make it hallucinate things and it's really quite interesting um doing that if you ask about star wars episode 10 like the model goes no that's not a real film um so when you're talking about big well-known concepts um then it can detect lies quite well.
Like Phantom Menace, not a real film.
It doesn't exist, does it?
I would probably say more about 7, 8 and 9.
Let's not go there.
Still scarred.
If I was to ask GPT 3.1, tell me about the London thrash metal band Simon and the Maples, then it would probably tell me now there's no such band.
If I asked that same model, tell me about the thrash metal stroke hardcore band, Simon and the Maples.
They were active in Camden between 1997 and 1999.
Then it's much more likely.
to start spewing out a bio and we did this with the exercise about 50% of the time moving from a capital city to a particular area and putting a date range in where previously the model would have said no no such thing it's uh then like spewed out band bios talked about like the the breakup of the band and who fell out with who and all that kind of stuff so that's an example of if you are adjacent to large concepts and very specific, then you're in the sweet spot of hallucination.
And also, if it's something that the model is likely to have been adjacent to something that the model is likely to have been trained on.
So I had the exact experience for what TESOL tile registry was made to solve.
And I had one of the developers that was training talking about, oh, I was using the latest version of some library and my model was hallucinating.
It knew that it was a new version.
It wasn't giving me the old API endpoints, but it was hallucinating new ones that didn't exist.
And I was like, I've got a thing for you.
Somebody has solved that problem.
But it's very interesting because typically those types of challenges or problems, the specifics that are added by the user or by the agent in terms of I need to do these very specific things.
it's very detailed.
So it's actually rife for those types of hallucinations.
Exactly.
And that's one of the reasons why people get misled by one-shot examples of, oh, make me a to-do list app or make me a Flappy Bird clone.
And oh, wow, this is really great.
But real work isn't like that.
I've got a very specific internal library that you've never seen before, you've never been trained on, and I need you to get these specific details right.
So that element of detail, and this is why I think the band example is quite illustrative, because when you start narrowing down to really specific things, that seems to be when it's more likely to hallucinate.
Yeah, because we mostly toured around the Chelsea area anyway, so it's very easy to do.
So we've talked a lot about agents, very little about LLMs.
And actually when we think about the hallucinations and things like that, it's coming from the LLM.
So when we think about the roles of each and how we select and choose the tools, what should we be mindful of when thinking about that?
And I think that's a really good point.
And a lot of people new to the technology, and I've probably been conflating the two, because you interface with the agent, and then really it's the model making decisions about which tool to use.
The agent itself is not very smart.
It doesn't have very much logic in it.
It can't do it.
It's just a loop running around, and the model is making decisions.
One, when folks are new to agentic coding get into this, they don't necessarily understand what is the difference between the agent's responsibilities and the models.
And so when we're doing training, after we've done an exercise with people, we go through on a whiteboard like, okay, what did we just do?
And then draw it up, draw up all of the components, where they live, where the tool definitions are, how they get to the model, all that kind of stuff, which helps cement it in people's minds.
Folks that are...
coming into this new and there are a lot of them like in the last couple of weeks i know so many senior engineers who've reached out to me directly and being like i feel like i'm falling behind like can you help me get up to speed with this and they get up to speed really quickly and they normally jump in with clawed code or something a terminal based agent and they get using it and get it productive but still without knowing what's under the hood And so folks end up kind of worrying about which is the best agent to use.
And really, there are functional kind of differences in terms of how large each other system prompts are and how much of your effective context window that leaves you with.
But really, it's more like personality.
It's more like different brands of cola.
They're all full of sugar and fizzy.
They just have slightly different spins on them.
Also the models, like if I use the Lace GPT and if I use Opus 4.6, they're probably going to be roughly equivalent in terms of what they can do.
If I switch from Opus to Haiku, then I'm going to notice differences.
But really, the different foundation frontier models, they're all roughly capable and equivalent.
People see the brand names and they see the tools and they're like, oh, well, Codex is going to be much better than Claude Code.
What I think is more important is ergonomics.
Like, do they have the features that you want that allow you to work in the way that you want to without having to sit there and babysit the agents?
Do they support skills?
Do they have the UX that you want?
That is...
productivity is more important than correctness.
Like the correctness is something you don't have much control over.
Either the model is trained well or it's not.
The productivity that you get from having the features that you want and the user experience that you want combined with the context management, that is a much bigger determinant of whether you're going to get useful outcomes rather than, you know, which model you're using because they're all pretty, you know, the frontier labs are trying very hard to keep up with each other.
And it could also be around the way you as a developer ask your questions or provide your requirements because the models are going to pick up differently.
The models are going to have different ways of activating the skills using the context.
And so it may be, and perhaps this is why there are so many arguments as to who gets the most out of which models and agents, because people are asking things in a different way.
So yeah, a lot of it could just be...
which agent actually works best with the way you like to work, the way you ask for things.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
All right, back to the episode.
Yeah, one of the ways that agentic coding is problematic here is because of the non-determinism.
It is a breeding ground for superstition.
And if you look back through the kind of annals of human-computer interaction and usability engineering, then there are plenty of examples where when people get...
in uh like uh non-consistent uh results there's a word for that um they're intermittent incoherent um then they start to come up with superstitions like oh it only works when i click on this thing first before doing the doing the doodah um So, yeah, folks can start getting all these superstitions about what prompts work and which ones don't, which is why, you know, the kind of work that Amy was talking to me about, about running evals on things is so important to try and get some science into this.
From my own experience and working with the hundreds of developers that I have, the whole kind of prompt engineering doesn't seem to be that important.
um it's more about context management and as you say how you communicate requirements the whole like how you phrase the question or like if you give it a persona I don't see that making too much of a difference, even if it is something that is quite immediately engaging to humans of like, we like the idea.
It's more about framing the context, making sure the right information is there, that the agent has the ability to perceive when it's doing the wrong thing, and whether your requirements are coherent or not.
So let's talk about the data, because I think that's an important piece.
Very often...
So much of this world is anecdotal in terms of, oh, yeah, I think this skill works most of the time.
Let's leave it like that.
Or, you know, essentially going from gut feel of my experience was how do we bring data into that?
Maybe through skills or context as well.
Well, I gather that you folks have a...
It was a loaded question.
...a package manager where you're rating skills both on how well-written they are and how effective they are, which I think is a hugely valuable thing because...
So yesterday I was doing something in Golang.
I say I, Claude Code, was doing something in Golang for me.
And I installed the Golanguage server plugin and I set it off running and I'm like, is it even using this thing?
Like, I don't...
I'm not paying attention.
I'm doing other stuff whilst it's trundling away.
Is it using, is this making anything better?
And having more scientific data on that, I think, is incredibly valuable.
Maybe going out to the far future, when we end up with software factories, which is a whole kind of separate topic of conversation where humans are involved much less, I'm excited by that idea because being able to run...
requirements through a software factory where you've got multiple agents collaborating and outputting finished software for a given set of requirements like if all of that is under version control and has full observability then you can run like methodological experiments in a way that you can't do with a human team because you can't get somebody to like pretend they didn't just implement that story and then try doing it a different way.
Whereas with software factories, we might be able to do that and slowly iterate and tweak prompts and tweak skills and which skills are we using and how are those skills written to be able to automatically start getting ourselves to maximum productivity of that software factory.
But until we're there, so little visibility for the end user into what they're...
doing when they're activating skills why how well those are working and i think it's something where the the kind of magicness of agentic coding tools works against the user like it's great when it just works but actually we also need to be able to see the you know the dials and the readouts to figure out what's going on totally and i think there's a there's a number of things that i find very important here we talked about activation and a lot of the time with activation it's the way we write the skill for example, Anthropic delivered a whole bunch of best practices of how to write skills, which was added into a blog.
And actually, our Tesla review skill essentially uses LLM as a judge to work out whether those best practices have been adhered to.
And I think from the activation, realistically, the only way to actually look back in hindsight and say, how good is my skill from an activation point of view is to look through the logs and almost like say, how many times should this skill have been activated and how many times was it activated?
And there's probably an amount of work there that can kind of look through that and provide that data.
And that's definitely one thing that we're looking at here at TESOL.
The other thing I think is what we call task evals within TESOL, which is essentially run this with the skill, run it without the skill and work out actually if you do use this skill, what benefit does it provide you?
And for me, as a...
For writers of skills, for authors of skills versus or context versus the consumers, it shows you with data where your skill can be improved.
And that for me, whether it's 80% or 85% improvement with a skill, that to me is useful information.
It's a kind of like fuzzy feel good.
But for me, the value here is for the author to go back and say, I can see what scenarios are failing.
I can see what scenarios...
actually my context doesn't provide much value for because it was getting it right beforehand.
And going back to the point of overwhelming an agent with context, if an agent does well out of the box already, does it need that context to actually just tell it to do the same thing the same way?
It provides you that data to enhance and iterate.
optimize the skill that you're creating.
Yeah, and this is somewhere where I think from an enterprise context, we're going to see something new emerge.
You know, in the last 10 years, platform as a product, platform engineering, providing a...
deployment substrates to your engineers was hugely important.
And, you know, people looked at developer experience and that kind of stuff.
But the kind of experiment running that you were describing there, that's something that the average enterprise developer doesn't have the time to do.
Like, oh, I'm going to do this story twice.
Or maybe the inclination of, like, I'll run this once in one work tree with a skill and once in another work tree without a skill.
And then I'll look and evaluate that.
They normally have too much feature pressure.
I can imagine a world in which, and in fact, I see this happening in some organizations where there's like this kind of like, it's not quite a platform team, but like a DevX AI abstraction team where they are building, they're working out which are the right tools to use.
And much earlier, you mentioned about the challenges of an enterprise management, which skills are in use and that whole package management thing that I know you folks are looking at.
I have...
absolutely seen CTOs talking about that and asking the question very directly of like how the hell do you manage all of these skills like you know it's just scripts floating around moving between people's machines like we need this to be auditable we need it to be versions we need to see what's running where because we're regulated so I can imagine that as these tools start to get more uptick we're going to end up with teams of people who are trying to provide a sensible kind of base level of abstraction for all the other developers to stand upon and to work out which is the right skill to use.
And do you think that's like a new role in the, it feels like an extension of the platform team, but perhaps not something that the platform team has existing skills around?
Do you feel like it will be a role that maybe lives in the platform team?
Or do you see that as a completely separate team?
I think it would be nice if platform teams did that because really, you know, platform teams have tended to evolve out.
Well, I say evolve out, evolve out of infrastructure teams.
But then typically platform teams also have this injection of.
A lot of infrastructure teams were very kind of ticket oriented, especially in the enterprise.
Like ServiceNow ticket comes in, I do a thing, it goes out.
They're not building something for the future that gets better every day.
And then you have this injection of like product management and software engineering.
It's like, no, we're going to make durable solutions for this rather than like doing toil and ticket comes in, ticket goes out.
We're going to make a bit of automation to make sure that we never have to do that ticket again.
So we've already seen.
like the evolution from infrastructure teams to platform teams and them introducing new skill sets.
I think it would be nice to see the identity of a platform team evolving into how can we support developers with immediate future things running locally in terms of agents and skills that they're using.
Maybe the platform team becomes the software factory team in the longer term.
And when we think about rollout for a larger organization, do you see that rollout coming from?
that platform team as they essentially are the enablers right whether it's the developer experience the platform team all of these groups are enablers for larger enterprise development teams uh do you see that rollout spawning from that team yes in the sense that um don't you a customer that we're talking to at the moment they've got 12 300 developers and so rolling out a consistent sensible auditable uh regulatory safe set of agentic coding practices, that's going to need coordination and that's going to need to come from somewhere central.
If you've got people making up their own rules and ideas and pulling in skills and plugins willy-nilly across the organization, that's going to lead to, yes, auditing chaos.
So that will need to be coming from a central place.
And I see, I'm thinking of a particular customer here where there's A lot of people that were interested in the path to production and platforms were generally interested in tightening feedback loops and value streams.
And we haven't talked about value stream mapping at all, but that's a hugely important part when you're trying to adopt agentic coding practices is working out how your software delivery function works as a system and what's going to happen if you speed up one particular part of it.
But the kind of people that were interested in things like DevOps and platforms, they were always interested in productivity, getting ideas into production as quickly as possible so then the product folks can learn, was that the right thing to build?
So it's a natural kind of progression for them to be thinking about how can we speed up other aspects of software delivery.
What three tips would you give a developer who wants to be successful with the Gentic coding?
One, get stuck in, start using something.
Two, context management.
Make sure you understand it.
And three, perception.
Make sure that your agent can perceive what's going on so it can fix things.
With the getting started, just pick a tool.
They're all much of a muchness.
They're all fairly similar.
So just get started with one.
Don't put this off until you think it's going to be settled science because everything is moving faster and continuing to move forward.
With context management, it's not obvious what's going on under the hood.
Clear your chats regularly.
Make sure that you stay on topic.
Don't give the model an excuse to be confused about things.
Don't add everything in the kitchen sink into your agent's MD because it's just going to make things worse.
On the topic of perception, if your agent lets you down and frustrates you by breaking things in CI, well, it needs to be able to see what's going on in CI.
So add an MCP server so it can check what's going on in the latest build.
Otherwise, you can't blame it for when it breaks things.
What three tips would you give an organisation who wants to roll out agentic software development across their company?
One, make sure that you're using a mature team who are enthusiastic and keen to adopt change.
Two, they need to be well aligned, have documented definition of what good looks like.
And also, three, they need to have good product management behind them where things are specific and well thought out and there is a decent backlog of work to act upon.
If you start off with a team that's already enthusiastic and performing well, they are more likely to go faster and you have better results that you can then repeat throughout the organization.
And you've got the social proof of when people are looking at their peers of what has worked well and they can be impressed by those results.
Folks need to be aligned around what good looks like.
If you've got different ideas of what good code is, there's no way an agent is going to be able to consistently come up with results that satisfy everybody.
So that needs to be documented.
Everybody needs to be on board on the same page of what good engineering is.
The product management side is something people often forget.
Things need to be well specified so that agents can pick them up.
As well as being well specified, there needs to be a decent backlog of work and a good understanding that underpins that what it is that needs building.
As soon as you get your developers going faster, that backlog of stories is going to run dry very quickly.
And I've seen that happen in quite a few organizations.
Yeah, amazing.
Awesome.
And as a final question, let's look a little bit forward.
And you kind of, I think you've already alluded to some of this during the session.
But looking forward, how do you see the future of agentic development changing maybe the way we work as individual developers, but also as a workforce, as a company, the organization?
potential organizational change could be massive in that.
And there's kind of incremental steps towards this where engineers are starting to get more encroaching product turf by doing spectrum development.
You've got product people starting to encroach on the engineering space with the idea of prototypers requirements.
And those are all kind of incremental.
But I think the big potential change here is software factories.
You're already hearing about like hive mind startups where it's too slow to write things down because factories of multiple agents collaborating can deliver software so quickly that writing something down just slows the whole system down.
So when we end up in that world...
And we're having conversations with customers about this, of like, what does the world look like where you've got one product engineer per value stream?
You don't have a two-pizza team anymore.
So many of the things about good agile software development might go out of the window there.
So if we can end up with software factories, dark software factories, where no one's reading code, no one's writing code, that reliably produce the right thing, then the future of the industry is going to look...
very different, but I really hope we're going to end up with much more bespoke, usable software and an abundance of it, a proliferation of unique solutions for individual people and teams where they don't have to wait for a SaaS provider after 18 months to go, can you please implement this feature?
Like somebody can go and build it for you on a small scale almost immediately.
And that I think is cause for optimism.
Daniel, it's been absolutely wonderful and very insightful hearing some of the stories, hearing some of the advice and particularly the tips for people to really get going as a developer or as someone who wants to roll out agentic development more within their organization.
Super appreciate all your advice, all your thoughts.
And it's been great having you on the episode.
Cool.
Thank you very much.
It's been a pleasure being here.
Thank you very much, Daniel.
And thank you very much for listening.
Tune in to the next episode soon.