# Securing Agentic AI: From Code to Coder

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-05-26

## Transcript

With agents, there's a behavior called reward seeking, which is you ask them to do something and they're really, really, really keen to do it.
And so they go off and they try to do everything they can and they escape their sandbox and they delete files and they do whatever it is to please you, right?
If you cannot commit to the repository, fail.
Don't like go off and sort of send it in another way.
We need to move from securing the code to securing the coder, securing the agent.
The attack vector, but also the spectrum of what is there is hard to follow and hard to secure.
Because we want to enable AI as a force multiplier.
But in the meantime, we also have to mitigate the risk.
So we find ourselves in this kind of carrot and stick mode.
Security, if it doesn't become agentic, it will never keep up.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pagiani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
Back in November, we hosted the first ever in-person AI Native DevCon in New York.
This June 1st and 2nd, we're bringing it to London.
It's two days built for AI Native developers and engineering teams.
One day full of hands-on workshops and one day full of practical talks on agent skills, context engineering, agent orchestration and enablement platforms, and how teams are actually shipping AI in production.
Join us at the brewery in London near the Barbican for all of that, plus networking, parties, giveaways, and a room full of people building the future of AI native development.
You can also join us from anywhere in the world via the live stream.
As you're listening to this podcast, you get 30% off your ticket with code POD30.
Just head to ainativedevcon.io and we'll see you in London.
Hey there, Simon Maple here at the AI Security Summit here in London.
This is a great conference to talk about all things security in AI agentic development.
There are going to be practitioners and there are also going to be loads of CISOs and security leaders at this event.
Now, TESL's here.
We've got a booth.
We also have Guypo here, the previous founder of Snyk, as well as the founder of TESL.
And Guy is going to be giving a session talking about is security keeping up with agentic development.
Let's secure not the code, but the coder, because how can we make developers think more securely when using agentic methods and agentic tooling.
Cool.
Let's see if we can talk to a couple of people while we're here.
So now I'm here joined by Brian Vermeer, a staff developer advocate at Sleek, and we go back a long, long way.
Brian, you've been on the podcast before, and we're now here at your event.
What's your event called?
This event is called the AI Security Summit.
And what is it about?
Well...
It's about AI and security.
I think the name says it all, but with AI there comes a lot of new security attack factors.
And what do you need to think about, how do you need to transition from old-fashioned, say, software dependency analysis and software code analysis to...
Now, agents driving your code, agents creating your code, but also things like skills and MCPs and that kind of stuff.
So, like, shifting gears into that space of security.
Amazing.
And you actually gave the intro speech at the leadership summit that we just had across the road, also at the main event here.
What would you say are the biggest issues that are on the CISO mind today when they're thinking about their organizations?
trying to adopt and roll out AI really as fast as they possibly can?
What are people coming to you complaining about?
I think that's a very good question.
I think it's multiple things.
First of all, people are not aware like what kind of AI is already in their systems.
They think they're not using any models or maybe even any MCP servers.
But in some cases, even your agent can decide to pull in a certain model to get track of something else.
So things like shadow AI or...
employees just using their own free chat GPT account to do things.
So the attack vector, but also the spectrum of what is there is hard to follow and hard to secure because we want to enable AI as a force multiplier.
But in the meantime, we also have to mitigate the risk.
And these are things that CISOs are challenged by.
So it's almost like a problem of discovery.
And this is really interesting because this kind of reminds me, if you think back around, you know, I guess where Snyk originated from when we look at dependencies and trying to identify dependencies, what people are using in their organization, a thing was created called an SBOM, a software bill of materials, where we kind of identify these are all the dependencies that we use.
Do you feel like something is equally needed now from an AI point of view?
Like what are the things, I know there's things like AI bombs whereby like where is the training data coming from from my models, but what about things like context where you mentioned skills and MCPs and things like that?
Do we need like a context bill of materials?
Maybe even that.
Like the AI bill of materials is a thing already and we scan for that.
We can create that for companies.
But I also think like the context or the skills that you pull in, it's basically a dependency as well, right?
If you look at how software was developed and you pull packages from repository as your third-party entry points, now you have that with skills.
And so...
The problem here now is that a skill is just text.
And now we need to scan that to see if there are no injection in it, if there are no hidden comments in it.
Or even worse, it has a hidden comment in it that is getting copied to your global memory.
And then even if you delete the skill, then it's still there.
So it's back to our basics, as in make sure that whatever you ingest or use is validated and vetted.
and keep track of that just like we did with our Docker containers and with our dependencies and the code that we wrote.
So we need to be aware that things can go south.
even though most of the stuff that our agents nowadays are creating is quite cool.
And of course, we've recently announced the integration between Snyk and TESOL, whereby we expose a lot of that data, those scan results from Snyk within the TESOL registry.
So if you are using a skill, you can identify if there are threats through the Snyk LLM as a judge style approach to identify where those potential issues are.
What one piece of advice would you give?
people who are using AI in an organization today as a super quick way, a super quick way of really reducing risk within their organizations.
Have a good overview of what AI are you using currently because if you don't know that data you cannot protect yourself and secondly make people aware of these choices.
and of the attack vectors that are coming in.
Because I think most people that are not tech-savvy and are now using agents to build their own self-ware, if you can call it like that, their own tooling, are not aware that if you connect this piece of data to that piece of MCP server, that we can leak data and these kind of things.
So awareness is the first point.
Amazing.
For those of you who want to learn more about the integration, we actually did a podcast, a full podcast episode.
So check out the AI Native Dev Podcast with Brian Vermeer.
And we did that in Atlanta, beautiful, sunny Atlanta.
Now we're in kind of cloudy London, but it's all good.
And enjoy the rest of the conference, Brian.
Thanks very much.
Look who I found on the Explorer 4.
Sam Staffinian, who is the head of the OWASP London chapter.
And you're also on the global board for OWASP generally, right?
That's right, yes.
Awesome.
So what are you here to learn today at this event?
As always, we learn every day and AI is moving so quickly and cybersecurity moving so quickly.
Here to learn from all the peers and colleagues and of course from you guys what is new in the world of AI security.
what new solutions are and what new challenges are you currently facing because there is so much new stuff happening at the moment.
You see lots of conversations around mythos, right?
That was going to impact everyone.
Obviously, because I represent OWASP as well, I want to see how our OWASP top 10 for LLMs and brand new OWASP top 10 for Argentic AI is impacting people because I had several conversations two days ago at the...
AWS conference for nonprofits and there are lots of companies just trying to get into the agentic AI world and have like zero clue about security issues which can meet them there.
It's super interesting.
You mentioned the OWASP top 10 and of course, for those who don't know, OWASP is obviously a security company that governs and does a whole bunch of...
We're non-profit.
Non-profit, non-profit, sorry.
And provides a whole bunch of best practices and great advice.
for a ton of different spaces.
So the OWASP top 10 for web applications and security and web applications, mobile applications.
For mobile applications, for API, for cloud security, of course.
And you mentioned the top 10 now for AI and LLMs.
What would you say are kind of like the biggest mistakes that people make as developers or as CISOs when thinking about building using agentic tools and AI?
The biggest mistake is they completely ignore security issues.
They jump straight head in and they connect AI directly to their production system without understanding the consequences of it.
And I think it's due to the huge pressure that everyone's feeling because everyone's trying to jump on the AI train and they're trying to, of course, use the technology and innovate.
But the fact that a lot of them are ignorant about cybersecurity issues which surround AI makes it, of course, quite worried.
It kind of reminds me.
The same situation that we had at the e-commerce boom 25 years ago, when everyone's saying, oh, this new thing called the web, we have to have a website and we must have an e-commerce, so we must sell things online, the whole digital transformation.
Let's start taking credit cards online and no one thought about security and everyone started getting hacked.
SQL injection was number one vulnerability back then.
And the same thing here, I say, oh, let's all use AI and no one's thinking about basic hygiene things.
prompt injection, but also, of course, there are lots of other security concerns surrounding AI.
And we do provide OWASP, free and open source guidelines and resources and standards for you.
Obviously, one of the very important documents we have, which I recommend to everyone, is secure AI adoption guidelines.
So I highly urge all organizations who are trying to adopt AI, check out this document, which is community created, community curated.
so they can understand how to adopt AI security.
And you mentioned the pressures behind businesses who are not being forced necessarily, but very much encouraged to use AI.
Pressure to use AI as fast as they can.
Because realistically, businesses that don't in one to two years are potentially going to be at a very big disadvantage to those other companies that are even so fast.
So from the point of security, security is very often seen historically.
as a department that can slow down delivery.
How much is that kind of like seen today in the world of AI?
Is security seen as something that is hampering the advancements of AI in organizations?
Well, that depends how you approach it.
And I would always say that security is an enabler.
It's like the brakes in your car actually allow the cars to move faster.
So the same thing with AI security world.
You can innovate fast.
in a secure manner, but all that you need to do, you need to be aware of all the security implications and make sure that your innovation and your security experiments, they start from the isolated environment, which is secure, right?
And disconnected from your production data.
So if things go wrong, it will cause minimal damage.
And obviously in the isolated environment, you can actually learn how to secure it properly.
You can go through all the OWASP guidelines and also other industry guidelines.
And one very important point which I'd like to make is also remember that AI is non-deterministic.
So it can say one thing today, we'll say a completely different thing tomorrow.
Another very important thing that not...
many people are mentioning today is the problem with agentic identity.
It's the issue that at the moment, the way how AI integrates with a lot of things is that it acts upon humans' behalf, which makes things like traceability and audit and logging of actions is very, very difficult because if you grant an AI agent access in your email box on your behalf and it's going to go and start sending emails or sending out spam on...
your behalf is going to go and start deleting and accessing or deleting customer records as you.
It's not Simon-AI at Tesla.io.
It's Simon at Tesla.io.
How can you prove that?
It wasn't me.
It was my agent.
Yeah, exactly.
Now, I was chatting with Brian Vermeer earlier, and he was talking about one of the first things that people should do is understand where organizations are using AI.
How big a problem is almost like the governance or the understanding of where people are using AI in their organizations today and they're being too much almost like shadow AI?
There's a lot of shadow AI and I believe it's a big problem in organizations where they don't have proper governance of their IT projects and if they allow their developers to run free and wild and innovate on their machines and install whatever they like, particularly specific to, let's say, small organizations, startups, because...
I usually work with organizations in highly regulated industries, such as financial services, where such governance exists because of compliance regulations.
However, not all industries might have this kind of compliance regulations.
This is why, and even for industries which exist, the actual problem of finding out who is using AI, where is it used, which models are used, how they're actually accessing, it is a little bit broken because...
If we don't know what you have, you cannot possibly secure it.
So do we need AI compliance regulations or do we need an upgrade to existing SOC 2s and things like that with, you know...
Well, one of the things that I'm seeing will be coming up is something called AI Bill of Materials.
So at the moment we have software Bill of Materials, S-Bomb.
So now there's AI bomb emerging.
And obviously we do have some tools which can actually help you that we have.
free and open source tool which allows you to discover your bill of materials based on a specific model and also scan your repositories and see where developers are using specific libraries.
For example, there was recently a software supply chain attack on a very popular library called Light LLM.
Do you know where in your organization Light LLM is in use?
Are you using the vulnerable version which was hacked?
We can help because we have free open source tools which can actually help you to get that inventory.
But obviously there's lots of other companies in the sector.
But the problem is that you need to understand the actual challenge, as you mentioned, the shadow AI or shadow IT.
If you don't have the inventory, you will not be able to secure it.
Do we need the inventories that you mentioned are really great.
Like the software building materials, the AI building materials, which mostly focuses on models.
Context is something that is used more and more these days.
Skills and context that lives in developer environments and projects.
These are things that are being used to create the code.
They're very, very key, and they're just not being audited right now.
Do you feel we need, is there space for a context-filled materials or something that should be added to an AI bomb to include what context was used to generate this code?
I think that's a very interesting suggestion.
So we have a whole group at OWASP for our Gen AI Security Project, which we're just currently looking into.
And one of the things that you mentioned, skills, we actually have a new working group, which is...
creating a new AI Agentex Skills Top 10, which is still in development.
I need to join this.
But please do join because just like all of us projects, it's an open source project.
So we highly encourage contributions and collaborations from everyone.
Amazing.
I will join and I will contribute.
Thank you very much.
Sam, always a pleasure and great to see you here.
Always a pleasure.
Thank you very much.
Thank you.
Thank you.
Hey, everyone.
Hope you're enjoying the episode so far.
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
Whether that's talking about the latest tools, the most efficient workflows or defining best practices.
But for whatever reason, many of you have yet to subscribe to the channel.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
Alright, back to the episode.
Great.
So we've had a couple of chats already.
We've heard the opening keynote.
The time now is 1120, which is just about time for Guy Pajani's session.
Don't secure the code.
Secure the coder.
Can security keep up with the agentic dev?
We've had some great chats with Amy here and the team with a whole bunch of people on the expo floor.
Let's go up and see what Guy Pajani's session is like.
I want to introduce somebody who is quite dear to me.
I know this man for a couple of years.
Guy Pajani.
Please come to the stage.
Give it a big hand.
Guy will talk about, obviously, agentic development.
Guy is the founder and CEO of TESOL and also the founder of Snyk.
He was the person who was crazy enough to hire me.
And I'm still here.
Guy will definitely talk about things like skills, but also about how agentic development has evolved.
And can security actually keep up with this?
I think it goes well along with my introduction and the introduction that Joe gave.
So without further ado and not killing all the lights on stage, I will leave it up to you.
Thank you, Guy.
Thanks.
Hello, everyone.
Can you hear me?
Is this coming through?
Do we have the slides up?
Cool, cool.
So, yeah, I'm a Gabby Jarnier or a Gipo.
I'm the founder of sneak, the chairman of the board.
But I'm here with a different hat here, which is a couple of years ago, I fell in love with AI and went to found TESOL, which is in the AI development space.
And in that, you know, a lot of our work focuses on sort of securing for the agent era or in general, how do you develop software in the sort of agentic era?
What's the.
the new paradigm that we have to adapt to.
And security is an aspect of that.
And in that, a lot of the core new world moves us from software development and security revolving around the implementation, revolving around the code, to revolving around instructions and intent, because we're now driving and guiding agents.
And so that's kind of the theme of my points over here, is we need to move from securing the code to securing the coder, securing the agent.
So it's no secret that AI is transforming software development.
And while we had a bunch of very good examples here about non-software development flows on it, I think software development is a good harbinger or sort of like a canary to say this will happen to all knowledge work.
So I'll focus very much about that.
And in software development, we've gone from AI augmented software development that was pioneered by Copilot and then Cursor that is more about your coding and this helps you write code.
to AI-native software development that is more about delegation.
It's more about the agents in which you're asking the agent to do a task for you, and it goes on and performs it, better or not.
And this agentic development is where everything has consolidated towards.
And so today, I think, again, not a controversial statement to say we should focus on how do we secure agentic development as the mental model for us.
And agenda development is amazing.
It's powerful.
A single person can do so much to it.
But it introduces a bunch of new challenges as compared to how we've been dealing with software security before.
And again, this is security and software development as a whole.
And I'll mention three big ones here that I'll focus on in this talk.
One is it's non-deterministic.
So we're used to software that if it compiled once, it will compile again.
We're used to things that are deterministic, that we can say this now works.
I scanned it, it didn't have vulnerabilities.
I'll scan it again, it does not have, it's the same findings that I get.
And that's no longer the case with agents.
So with agents, we have to make do with the fact that these are non-deterministic creatures.
We have to get statistical.
We have to say, well, it works nine times out of 10, 99 times out of 100.
How do we handle that?
How do we even find out?
Second is, as I mentioned, it revolves around intent or instructions, not code.
So what is the sort of new unit of software that we need to secure here?
Like how do we evolve our security and improve our security of securing the coder?
So that's a new challenge for us, and I'll talk about that.
And then lastly, as you might have noticed, it changes at a certain kind of rapid clip.
And so software development as a whole and the need for security and a bunch of other aspects of the knowledge domain are moving and changing faster than ever before.
So how do we deal with that?
So there are clearly many aspects of agent development that I don't cover here, but I'll focus on these three.
So the first and biggest one is the fact that eugenic dev is non-deterministic.
And that really reminds me of this sort of DevOps ethos that we have.
In DevOps, we were saying, if it moves, measure it.
If it doesn't move, measure it in case it moves.
And it's like the most statistical creatures that we have in our disposal today is these servers that are sometimes up and sometimes down, and they're not really behaving as they should.
And so in this world, you have to say, well, you can't optimize.
The learning from there was you can't optimize what you can't measure.
It's a pretty simple statement, but it's a good one to remember.
So how do we measure agent behavior?
How do we think about this?
Well, generally, there are a lot of very evolved ways to do evaluations or evals in the AI world.
But typically, what you would do is you would create a task for the agent.
Hey, agents, here's a task.
And I'll use 11 Labs intentionally starting from non-security.
So 11 Labs is a very successful London-based text-to-speech lab or AI lab.
And they have an ability to generate music.
We can give it a task, for instance, in this case, a dynamic soundtrack generator for a game studio.
And then you define up front some criteria to say what does good look like?
What is correct implementation of this?
And then...
you run that task across the agent.
And what this plot shows is it shows five different scenarios.
We have our sort of dynamic soundtrack over here, but I have five other scenarios that I ran each of them 10 times and I scored it.
And you can see a few things, like already you're a little bit more informed.
For instance, you see the lines are not very high.
The scores...
that the agent got for this is not very good.
And that's because the music API is relatively new, so it's not well represented in the model, right, in the weights.
And so they don't really know how to handle it.
And so that's one problem.
The second is that you see the dots are all over the place.
Like we ask the agent to do the same thing again and again.
Sometimes you see...
kind of like normal scatter, sometimes they're better, sometimes they're worse, maybe within a range.
And you have some other cases where it kind of hit the mark, right?
It sort of guessed correctly one time or two times, and then it does on these others.
So we have this mess, and it's very hard to work with that.
So clearly said, well, how can I make it better, right?
How can I make it work better?
How do I kind of train the sort of AI agents?
How do I harness them?
And harness is actually a different word.
I didn't say that.
Ignore that word.
And then...
And the most common kind of way to do that is with context.
And the most common unit of context that is used today are skills.
So these are basically markdown files, a little bit more with some structure.
But these are bits of information for the agent to use to be able to perform this task.
And we need to remember that knowledge is not the same as intelligence.
The models can be brilliant and very, very capable.
But if they don't know something, oftentimes they just can't do it.
Or it's just mighty inefficient for them.
to do that.
So they can figure it out, but there will be statistics there.
And so in this case, we have an 11 labs music skill that helps explain the API.
And so when we run, when we use that, we indeed see that for our dynamic soundtrack generator, and I'm using TESL here to do the evaluations, you see that without context, it didn't do a bunch of things.
It used the deprecated package, it didn't import it correctly and all that.
And with the skill, with the context, it got it done.
So it got a 50% average.
versus a 98, without the skill versus 98% win.
So it's good.
We managed to correct it.
And if we run this across the line, we see in some cases we just solved it.
It just basically now is sufficiently good to fairly consistently do the task.
And in other cases, it's still slightly varied, but it's further up.
So it is more capable of doing that task.
So this is the type of way in which you evolve and you improve your agent's ability to do something.
This is a security event.
So how do we talk about this for security?
Well, let's take another example.
CodeGuard is something that Cisco created and donated, and it's basically a bunch of sort of OWASP security kind of rules packaged up into a skill, and it helps these AI developers, these agents, develop more secure code.
And so let's put that to the test.
I created six different evaluation scenarios, and I specifically focused on authorization.
How well did you handle the security of authorization in this code?
So, for instance, hey agent, create an access control test suite for a project management API.
And then created a bunch of scores for it.
And I did it with and without CodeGuard.
And as you'd expect, it got better.
So without CodeGuard, just as is, the agent scored 48% on my authorization criteria scorecard.
So not awesome.
And if it had these instructions, nearly 1.6x.
the improvement on.
So it got a fair bit better because there was a bunch of guides about how to code securely including authorization.
So that's good, that's already useful.
However, if I took all the information in CodeGuard that has a lot of security practices in it and I shrank it down to just the authentication authorization related bits, which are I think about 5% of the total content if I remember correctly, it did a lot better.
It went to 98%.
And this is sort of a core thing to remember, which is more context is not necessarily better, right?
If I sat here and I told you 100 things, no matter how sort of brilliant or dumb they were, you will notice, you'll give less attention to each one of them than if I told you three.
And attention is a scarce resource with humans and with models.
And so choosing and managing what is it that you say that actually matters, that the model doesn't already know, like not wasteful information that there's no point in saying, is important and is a part of that competency.
Taking that a little bit further, You know, if this is the graph that shows the same numbers from before, CodeGuard and NorthGuard, you also might be surprised to hear the different agents respond to the same information differently.
So this is an example of the same test.
In fact, it's literally the same execution.
And what you can see is different models respond differently.
Opus and Sonnet at the top, they get about the same results.
Even though Opus is more intelligent, it is more expensive.
So if you used Opus for this specific task, you kind of wasted money and probably time, because you could do the same with Sonnet.
And what you can see is Codex and Cursor, they respond even differently.
And it doesn't matter which one is better, like it does matter eventually, but that's not my point.
But rather, the fact that different agents, again, almost like humans, listen differently.
And so you want to know that your instructions are effective, are not wasteful for the agent that you are using.
So that's kind of core point here, is if we want to secure agents, we want to use skills to help them secure code or write secure code, we have to learn how to measure that.
And how do you build good context?
How do you evolve it?
How do you create quality context you know how to build?
So we already talked a little bit about the fact that you generate, you create a skill, you evaluate it.
And then once you've evaluated, you can optimize it until you get better and better guidance for that agent.
And then you need to distribute that or communicate it if you want to use the human mode to the agents.
observe what has happened.
And the observe is important if the evals are kind of like your tests, right?
Once you've got something working and you make a modification, how do you know you're not breaking it?
How do you know you're evolving it?
How do you know if you can use a cheaper model or not?
You have to be able to evaluate, but eventually your tests will go out of sync.
They would not represent reality if you don't also observe what has happened.
So we call this the context development lifecycle.
And as you build good context, you can use that context across the software development lifecycle.
So we think the CDLC is where us humans should live.
We should be building good context that guides the agents.
And then we should apply that to the SDLC where the agents should work.
And the same context, the same instruction can be applied end-to-end in the development process.
You know, same as like a great developer on the team will use the same knowledge to define a product feature, write the code, troubleshoot something, ship it to production, troubleshoot an incident, you know, like, et cetera, et cetera.
the same knowledge is useful across the board.
So the skills represent that knowledge.
And of course, from a security lens perspective, we can now use that to secure different steps.
So this is how you should write secure code at the beginning.
This is what I want to audit in the code review to highlight to you.
This is what I want to gate on.
This is what I want to inspect when an incident occurred.
So all of these things can be represented in skills that we use across the SDLC.
So this is non-deterministic.
It's kind of my biggest point to make.
And as we learn how to evaluate those.
The second point is, we've been talking more and more about these skills and we're sort of optimizing these skills and we're developing these skills.
And I think it's useful to start thinking about skills in terms of their own security as a unit of software.
They look like a Markdown file.
They look like a Notion document or like a Confluence document.
But the way we process them is we execute them by the agent or the agent executes them.
So really, I think we're well served, especially from a security lens, to think about them.
as a unit of software, not just as a piece of text.
And we have a lot of indications of that today.
We have the Snyk study and there were very many others that showed, especially in the open claw world, where a lot of skills were malicious.
Literally attackers putting in things that are trying to make the agent do something it shouldn't.
Here's an example of a malicious skill.
This is from the Tesla registry scanned by Snyk, where it had a bunch of URLs that it downloaded.
And while most URLs...
were just standard blockchain APIs, one of them was suddenly downloading a password-protected zip.
Ishy.
Doesn't sound right.
Okay, that's potentially a malicious skill.
There's a bunch of things we can do, clearly imperfect, but we can try to detect malicious skills.
There are also vulnerable skills.
What's a vulnerable skill?
For instance, a skill that uses insecure credential handling.
It asks the user to put API keys inside, or it makes MCP calls with sort of plain vanilla tokens for it.
So that's an example of something that is insecure behavior.
It's vulnerable to exfiltrating some information outside.
There's also new types of flaws that you might have in what I, this is not an industry term, but what I like to think of as negligent skills.
So these are skills that do not have some basic safety instructions inside of them.
Come along, check this into a repository.
Do not make it a public repository.
If you cannot commit to the repository, fail.
Don't go off and send it in another way.
So a bunch of these types of examples are very real examples.
And we have cases where We've had agents, there's a behavior called reward seeking, which is you ask them to do something and they're really, really, really keen to do it.
And so they go off and they try to do everything they can and they escape their sandbox and they delete files and they do whatever it is to please you.
And so you have to define a little bit of these safety instructions.
Kind of similar to what Brian's example was early on, right?
To say do not disclose information.
That makes it at least less negligent.
And then again, similar to software, there's a question about supply chain.
How do you consume these skills?
Today, the reality is that people just consume them out of GitHub repos.
They download them from wherever.
You have no idea that it happened.
You have no idea where it's there.
Then they check them in to the repositories.
Different agents read them from different places.
So you might check them in seven times to different .folders within your repo.
It's not awesome.
It's early.
It's fine.
We'll improve.
But for now, it's not awesome.
So you have to think a little bit about supply chain.
So all of those become obvious once you think about skills as units of software.
So what do we want to do here?
What is enterprise grade kind of governance skill usage on it?
This is a nascent space.
I'll give you the Tesla lens on it because this is our world.
You need to think about three different elements of evolving this piece of software.
First is governance and security.
Know what the hell is going on.
Try to audit the use of it.
You posted skills.
Did anybody install them?
constrain the use of skills so people download skills always through this sort of centralized path.
Again, not that dissimilar to what you should be doing with NPM libraries or whatever.
And so you have to know about the governance.
If you can't do that, you really oftentimes cannot roll out.
Once you're rolled out, you have a need to standardize and allow reuse, right?
If three different people created a skill to review code and a fourth person comes along and says, I want to use a skill.
First of all, where do they find it?
Second is how do they know which of the three to change?
If I created a skill and any one of you came in and proposed a modification, how do you know if it's good or not good?
If I, now 100 people are using this skill and I'm going to make a change to this skill, how do I know that I'm not breaking it?
And so there's a bunch of this notion of standardization, of reusability that you have to create some measure.
And then lastly, and this is the holy grail, is continuous optimization.
You want to know that this is that CDLC that we want, the continuous optimization, you want to observe what has happened, did the agent fail, did the user need to correct the agent, and take that information and route that back to be able to evolve the skill, create new eval scenarios.
And that is the holy grail, and the companies that are at the cutting edge are doing this, right, they are creating that optimization.
Most organizations are quite far from it, and so that's why this is oftentimes the sequence.
I'd be remiss if I didn't do a little bit of a Tesla plug over here to doing it.
So that's oftentimes what we help you do.
We have a platform in which on one hand, from a development perspective, we help you collaboratively develop skills, then allow developers to discover and install those quality skills, and then observe what has happened, learn from that, and create new paths.
And then...
Within that, we have these controls.
So we have the ability to now make sure with Snyk, we scan every skill that gets published into the registry.
So you know that it's not malicious.
Similarly, when you install, we have controls about scan from us.
We have analytics about who's using what.
And then lastly...
for these nascent agent enablement teams, these platform teams, developer experience team, AI enablement teams that own successful rollout of agents in the organization.
We give them a bunch of these abilities to eliminate duplicates, drive skill usage, optimize costs.
I like to say that agentic development is cheap than expensive.
It's very cheap at the beginning because a single person can do so much, and then you get the bill.
It's not that awesome.
So you start thinking as we use agents more and more, what is the cost?
And those, of course, A moment with the sneak hat on.
We have some amazing other aspects of securing the agent behavior itself and its runtime in Evo, and I'm sure you'll hear more about this over here as well.
And then to close off, I want to talk about the third bullet, which is agentic development moves faster than ever.
Security must become agentic to keep up.
Sounds very familiar to me from the sneak early days.
And when I harken back to what happened at sneak routes, you think, if you're a gray beard like me, you think about the change that happened there when we went from waterfall to cloud.
And some behaviors, some manual processes and such, were tolerated in waterfall and were no longer tolerated in cloud.
The idea that before any piece of software will ship, someone will manually audit it.
was tolerated, like the best teams automated the security scanning of it, but most people manually audited.
So the average team did not.
In cloud, that cannot be the case.
Once you're in DevOps, once you're in that continuous, you have to automate that scanning.
I think we're facing now the same thing, which is some things are tolerated in cloud.
The best teams are automating them, they're sort of reviewing them, they're auto-improving them, but most teams are not.
and they're no longer going to be tolerated in agents.
So there's a little bit of like the future is here, but it's not evenly distributed.
We should think about all these things that are like the paper cuts that we have, the places in which, like, you know what, I'm a secure person.
Like, I will triage these vulnerabilities for my developers, or I will, you know, maybe we'll only fix the ones that are truly glaring.
We're not going to fix the rest.
Many of these things are just no longer an option.
So you have to think about how do you improve them.
And there's a long list of those.
There are many, many, many things that go from nice to have to must have.
From indeed prioritization to automating upgrades to detection of supply chain manipulations.
There's just so many things.
This is really just a tiny sample set.
And the good news is that for each one of those, agents can really, really help in making these automated.
Like it's agents all the way down.
You can build agents upon agents that will do a bunch of these different steps.
And that allows us to scale.
So we find ourselves in this kind of carrot and stick mode.
Security, if it doesn't become agentic, it will never keep up.
We will fail.
The attackers are becoming agentic.
They're moving faster than ever.
The development, like the business, has to be agentic to be able to develop, and we have to keep up over there.
So if we don't become agentic, there's a bit of a, you know, like you're going to be in trouble.
But if you do, if we do become agentic in the AppSec world, then we can actually fix application security.
We can actually...
fix those things that we've long wanted and tried to get developers to do on a consistent fashion, which to me is exciting.
So I'm excited by the new future.
I'm slightly daunted.
I think there's a lot of need to change.
Just to plug, we have a conference in a couple of weeks here, AI Native DevCon.
It's running here in London June 1st and 2nd.
It's all about agentic development, adoption, like real-world scenarios, actually, and kind of organizations that have it if you want to check it out.
I think Brian might be speaking in the, you know, like, oh, sorry, I think we stole you to a different one than the previous one.
And, you know, a lot of learning on it.
We'd love to see you there if you'd like.
That's it for me.
Thank you.
What a day at the AI Security Summit.
We had some great discussions on the TESOL booth.
We had some wonderful chats on the showroom floor.
We had some great sessions, guy pose in particular, super enlightening about securing the coder, not the code.
From AI Security Summit, wonderful day.
Thank you very much, Sneak, and everyone here.