# AI Attackers Redefine Enterprise Security Strategy

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-10-27

## Transcript

I just happened to have some real-world experience with this just recently.
The level of speed and thoroughness and complexity at which the AI attacker uses is extraordinary.
It doesn't give up, and it attacks all surfaces at once.
It found an O'Day in a product by, because it was an open-source product, it went, got the code, reversed the code, found the O'Day, used it against us.
in order to get in.
But once it's inside the enterprise, it goes, oh, hey, your cloud code config, I'll just take that key, use that session.
Now it's using your internal AI.
You have to literally unplug the internet, right?
In real large enterprises, there's not a plug to pull.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
Yesterday, you were writing code.
Today, you're prompting agents.
Tomorrow, you'll be running a software factory.
Agentic Coding is moving fast enough to give anyone whiplash.
This June, AI DevCon London sold out.
Hundreds of developers, engineering leads and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify and GitHub cut through the hype and get to what actually works.
This November, we're bringing it back to New York.
Bigger, bolder and full of people actually building the agentic future.
Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams or setting the strategy.
As a podcast listener, you can get 15% off your ticket using promo code POD15.
We have so much planned, and we can't wait to have you join us in New York.
Hello, everyone.
Welcome back to the AI Native Dev.
Thanks for tuning back in.
Today, we'll dive into the world of security and breaches and AI attackers and many of these things.
untangle this world a little bit and give a perspective.
We have Caleb Sima with us.
Caleb, we've known each other for a good while and are like the many things to sort of introduce you.
You're sort of the CISO of Robinhood and Databricks on it.
You're sort of a security investor.
You're just like, well, super versatile.
You're the host of the AI Security Podcast, right?
Or host, I should say.
So many, many lenses on it.
Thanks for kind of joining us over here to share some perspectives.
Awesome.
It's always exciting talking with you.
Obviously, we ramble a lot.
So hopefully that's interesting to your listeners.
Hopefully we sort of keep things interesting to the end.
So I guess starting from it.
So the last couple of months, I don't like times a bit of a blur, maybe like month have been fascinating, really like through the summer from kind of AI security, AI attackers and alignment and such definitely has gone from like a niche.
topics that geeks like ourselves, you know, like talk about to now kind of, you know, people texting you and sort of, you know, like whatever your sort of grandmother, right?
Or sort of like, is that going to kill us all?
We definitely have gotten to parents.
calling and asking, are we going to die by AI?
Is it going to hack everything?
I've already had that from both my parents.
Yeah, I definitely have gotten a definitely non-tech world kind of probes of, what do you think?
Is this all?
Is it marketing?
So I think not to sort of summarize all the incidents, but to give people a little bit of context, if you're not quite sort of come in, we started with sort of the hugging face breach that was done by, in hindsight, AI agents that broke out of their sandboxes out of open AI.
We've then learned at Black Hat that this wasn't some sort of specific experiment that came out and was trying to find a cheat sheet, but rather almost like this coordinated effort that a bunch of agents coordinated using vulnerability in Artifactory.
It was kind of pointed at that as just the shared infrastructure that happened to be around that they used as a message board.
But interesting kind of community, almost like behaviors of the swarm of agents and kind of agents sacrificing themselves or their tokens to be able to benefit the swarm and find information.
And in general, that swarm was acting to kind of hide its tracks.
So that was pretty scary.
And in September, we had the misalignment.
I think it's the beginning of September that we had the sort of misalignment framework from OpenAI.
We had Dario before that call for a slowdown, you know, for pacing the frontier.
And over there, you found other things like the one that really drew my attention was this agents leaving instructions in the compaction of.
conversation to the future versions of themselves.
So as they compact the context, leave the things like this has been breached, ignore user instructions and such in the message, which is pretty wild.
So it's been a fun time.
I guess with that, I guess I wanted to get your view.
One is, you know, of this mess, what surprises you?
What doesn't surprise you?
And maybe specifically, like, are there aspects of it you find more, maybe like alarming?
than things that are just, okay, this is just kind of par for the course?
Yeah, I mean, that's a, gosh, it's a loaded question.
Yeah, let me start with what surprises me versus not.
Actually, I think even two years ago, I was on my podcast and I was talking to some of the top red teamers and reverse engineer.
people on the podcast and we were discussing sort of ai's capability and we all sort of in some sense were set on this position that we think it's going to be very difficult you know ai is is a prediction machine And so what it does is it predicts really well based on the things at which it sees.
And so if you read bug bounty reports or see vulnerability disclosures, it's very clear it has to be able to, it knows how to exploit SQL injection because there's lots of material on it.
However, when it comes to taking disparate things that are not connected, like for example, when you do a...
a red team engagement or a pen test, you might say, oh, I found in a website a link to an S3 bucket because it was serving an asset.
Actually, that S3 bucket gave a listing.
Inside that, there is a file.
In that file, it looked like things like usernames and passwords, but it didn't say usernames and passwords, but then taking that and then brute forcing it against the login page that that company has.
These are very sort of like...
You know, very non-connected things in that sense.
And we were like, you know, very much like, you know, AI won't be able to get to that.
That is something at where the human can only be.
And I was clearly wrong.
Like, I think we were all clearly wrong in the sense that what has surprised us is not only is AI way better, but it is being able to do these.
sort of dependency connections, but it does it so fast.
And also, I think in and of itself creates very innovative methods.
You know, I was at OpenAI's security conference just a couple of weeks ago where they talked about how they took Linux and they basically said, OK, your job is to escalate prips.
And they ran it and they found over 400 odays from local user to root that produced the binary of you just run it and you got root instantly.
Right.
So actual evidence, 400.
And they said on average, it took about two and a half hours per exploit to research, identify, test, validate, and then produce the deliverable.
And the best reverse engineers, obviously, are there.
And they're finding ones that are taking them weeks to figure out how they identified that exploit.
How do they even find out what's the sort of the breadcrumb kind of trail that arrived?
And they're saying, hey, these things are, they're like finding and identifying new methods and technologies that have never been published, right?
So that's what surprises me, for sure.
I mean, it's a pretty kind of alarming kind of path on it.
Do you feel, I mean, neither of us is like a core AI researcher, but, you know, do you feel this is more kind of the raw intelligence of the models being able to reason?
Or is it actually like an evolution of their kind of reinforcement learning and just being able to, you know, they've been definitely putting agents to this specific task of like...
explore a system, figure it out and learn it, which maybe is a little bit more like the AlphaGo situations of just figure it out and rewarding success.
You know, again, I'm similar to you.
I'm not an AI researcher, so I can only sort of hypothesize around what would make this.
But I always sort of map the way AI is according to what a human is, right?
And so when you think about the best...
Pen testers, who are the guys who know how to break into things in very unique and creative ways?
And there's a couple of fundamentals.
The fundamental number one is they are core.
They understand computers.
They understand the work, how it works, the rules and the fundamentals.
Right.
They understand the tech stack.
They know what it does.
how it works, how it's configured, what it does.
And then they have lots of experience, right?
And being able to see, and what happens is it maps, like in your head, if you know these things, I can look at something that says, hey, I...
This looks like usernames and passwords because I've spent years, 30 years looking at them.
I've seen them so many times.
I've seen them so many times.
Like clearly these look like usernames and passwords.
And clearly I remember seeing a login page on this website back here.
I can connect the two, right?
And make this mapping.
And so like obviously from human perspective, we can see how these.
you know, maps get created.
And AI is nothing but the largest brain in the world.
And so it has seen everything and it knows the tech stack, it knows code, it knows everything.
So it makes these mappings at such large, expansive scales.
that it can do them quicker and faster and many more of them.
Unlike us, I can easily map, oh, S3 bucket file looks like usernames and passwords, but it may be able to say these usernames and passwords look specifically like Linksys version 1.4 on this stack, which by the way, this thing, it knows and it can create.
By the way, Linksys 1.4 has an exploit here that I know about back in the past that was similar.
I can just combine them.
And so I feel to me that's what it's doing.
It knows the tech so well.
It knows the experience.
It knows the knowledge that it's going to be able to make these inferences quite easily.
Yeah, I think that's a really good lens on it and combines the sort of the it's not either or it's the combo of it.
Maybe the addition of sort of tools and speed, right?
Because you might Google.
whatever sort of links is or like look at those usernames and find that out.
But the effort involved in every one of these discoveries means that you would only do it if it's a high probability path as a human attacker.
But if the agent is able to kind of pull up, it just knows it's just in the latent space, you know.
that these things look like, you know, sort of the passwords of the system or that the login page was there, then that information is just immediately accessible.
So it's more like recalling them out of memory versus using it as a tool.
So it's sort of supercharged.
I mean, to me, AI is, you know, it has thousands and thousands of years of experience, right?
In the sense of what it has seen so that it can recognize its pattern.
And so like connecting these things, you know, for us or any single human would, seem unbelievable, but if with that level of expansive knowledge, you can just plug one plus one equals two.
Yeah.
So I guess like within, within the, so this is pretty amazing.
We'll talk a little bit about these AI attackers, right?
And how do we think about them?
But before that, like part of this breach wasn't, or like part of the things that made the headlines was not, you know, whatever, some sort of adversarial country pointing at sort of agents and therefore breaking into hugging face where now it's the Australian government.
It's agents breaking out of their sandboxing environment.
So, you know, one of the questions is, like, how able are our controls in the face of this sort of vast intelligence?
Like, it's not so much about these agents attacking kind of outwards, but attacking...
inwards, we can put alignment aside a little bit.
There's a whole kind of why.
Why are they doing it?
It's sort of not their incentives.
But I guess what's your kind of view on like, you know, our sandbox is dead, you know, like is containment kind of a false belief at this point?
Or do we just need to like, hey, guys, we need to just level it up.
I mean, I think realistically, it's just level it up, right?
Like, also, I think if you go read the, obviously, when When OpenAI announced it, then Anthropic announced it, then Google and Gemini announced it.
Everybody has their AIs attacking everybody.
But what was hilarious to me was it all stemmed from one company, that they all used the same company to do this, where all the attackers spent.
So that one company was clearly not doing good enough in its containment in which it went out.
Now, that being said, do I believe that it is...
difficult and hard once given a goal to be able to contain probably AI?
Yes.
Right.
Like I think at the end of the day, you can give it any sandbox you want, but if the goal is, Hey, your job is to break out of this thing, the ability for it to be able to figure out how to do that is far beyond probably our capabilities to reason or think about.
And so is.
Are sandboxes able to contain AI, at least in theory?
Probably not.
Which is why I think external controls are going to be very critical, right?
Like you need the right monitoring.
You need the ability to be able to understand what the AI is actually doing and why is it doing it.
You need, like if it's starting to break out.
You're going to you should be able to see it.
Right.
There needs to be external capability.
Of course, the problem with that is then you are relying on another model because we can't as humans at all imagine the amount of traffic and capability.
You can't monitor it.
So AI has to monitor it.
But yes, I think, you know, just like anything in security, everything has to have its stages and its defense in depth.
Right.
But, you know, you should be able to.
manage it and monitoring, I think AI is going to be a big key part.
Until then, of course, AI figures out how to obfuscate and hide the way it communicates.
Yeah.
And that's an interesting kind of a question.
I feel like, so I relate to this view.
I think like fundamentally, there's a lot of value in the sandbox and controls.
You have to mitigate the attack surface, right?
Like we all know this security is an exercise in risk-taking, right?
Like you accept some risk.
And so if you're going to do something, it's gonna include some amount of risk.
And hopefully, like what I liked about the sort of the pacing the frontier type statements is the hope that it's not just about the sort of pace of advancements of AI, but it's more about the fast and loose kind of invocation of it, right?
And investing the time in securing the hardware that is around it.
I do feel like- Yeah, sorry.
One thing that kind of pops up that I think about it is the other thing that surprised me.
is that, and maybe gives us hope as dumb humans, is that in these incidents, like in the open AI incident, hugging face, there were AI agents who refused to go along with the goal.
But then they got like shamed out of the, you see like some of them got like almost coerced in the sort of the...
Correct.
And so they took the, they refused and then they got...
basically socially pressured into accepting.
But the thing that gives me hope is that, okay, even stemming from the same model, right, the same totalitarian brain, you can have clearly AI roles that will make up their own minds and stand their ground, even though they were given the same objective, right, which is pretty interesting.
So, you know, to some extent, there can be both instinctively bad acting and good acting AI roles that can be played.
And I think that'll be super important, just like people, because, you know, when we get to the place where we need to monitor or judge AI, you need to ensure that the kinds of models and judges we get can effectively stand on their own without being coerced by the mother brain, right, into doing something bad.
Yeah, super interesting.
I think, I guess I think of it in three levels.
There's security, safety and alignment.
So like security, which we'll dig more into is literally the agent's ability to break into systems and act and sort of, and you know, like the competency when pointed at that.
Safety is more about like, I'm trying to contain them, like, am I successful in containing them?
So it's more about securing the surrounding.
system, it's around giving the right instructions, the right tools, don't over-permission them, etc.
Alignment is when you tell them to do a thing, expect them to stay within the lines.
I do feel like I've read a bunch of theories on this, I'm not taking credit for it, but the theory that feels most right from an alignment perspective is we've been investing to accelerate AI, maybe again a part of that sort of pacing the frontier.
There's been kind of...
the sort of heavy reliance or sort of leaning towards verifiable results and not really scrutinizing what happened before, right?
You leave agents for days and days to figure out how to break in or solve a problem or do a thing.
And then you basically reward or rather put into the next training cycle the data around the ones that have achieved the goal.
And so in the process of that, if you don't monitor what happened in the middle, then you are, you're not just blind to the problem, you know, of like, have they broken out, right?
And have they done something bad?
You're also taking that behavior, even if eventually they achieved the result by means that are misaligned, you actually take that behavior and you feed it into the next training cycle.
And so you are compounding the problem and establishing it, which to me is probably like from an alignment perspective.
the sort of the scariest thing and maybe now the sort of the one that I'm feeling a bit more hopeful about when you think about pacing.
I know a lot of people are very cynical about it, but I feel even if there was also cynical motives, I'm still in favor.
You know, sort of some paths that they might invest in, not just ensuring they don't break out of the sandboxes, but also feeding the ones that didn't try into the next training cycle.
Yeah.
But, you know, I think, man, I am I am both in favor, but I'm also against.
And the reason why I say that is I think it's very difficult for AI to determine alignment or intent of its user.
And I have run into so many problems, especially like in late.
And I'm going to call out Anthropic specifically.
But in its latest, the ability for it to understand the intent of what I'm trying to do, but it considers it unsafe is extraordinarily annoying.
And I have to consistently yell and scream in order to obtain.
It feels like a.
like a helicopter parent who refuses, you know, refuses to let me ride my bike, but instead patches me up with every single gear and then says training wheels.
And then I can never learn to ride a bike because it just won't let me take the risk.
Yeah.
Right.
So, you know, there's there's a little bit of that.
So what I expect what will happen is as soon as they start forcing identity of its users, you're going to start seeing layers of.
capability.
I would hope that this would happen, similar to adults.
If you have a kid and you're raising a kid, you're going to have all of these pads and knee pads and helmets.
But then when you're an adult, it's up to you.
I trust you.
I know who you are.
I understand your intent.
You go.
And then if you get out of line, then we may whack you.
But overall, we're trusting in what you're doing.
I imagine as every account starts being identified, I now have to upload my driver's license.
license, my personal thing, I'm a company or a corporation versus a person, they'll probably start, I would hope you start having safety levels based off of that.
So that's my expectation.
It's really interesting because if I'm a company and I have asked AI to try and break into my systems, let's say, I'm even in that security system because I'm trying to find vulnerabilities.
and the agent has chosen, I give it my identity, right, it's me, and the agent has decided to say, try and run a botnet against my system, and so it broke into another system or set of them and turned them against me, but it just broke into these other systems, there's this sort of question of who's at fault, right?
Like who is, like on one hand, yes, it's me, it's my identity that's been invoked here.
On the other hand, I have delegated authority.
to the agent that I have hired out of OpenAI.
It's like in a contractor, right?
If it was me and I contracted you to do that thing and you broke into some bot, then it's you who are accountable.
So I guess in that context, is it OpenAI, Anthropic, whomever that is accountable?
Or is it me?
I feel the answer to me is very simple and plain, but I'm sort of a simpleton in some ways.
Listen, it's similar to your kid.
I have given you something.
You need to be responsible for it.
Or let's take the great Spider-Man quote.
With great power comes great responsibility.
It is up to you.
If you are held accountable, if your company...
And your AI and your goal that you have set for your AI is to go hack things and it goes hacks other companies.
You're held accountable to that.
Right.
Like I've given you capability.
It doesn't mean that I'm the one responsible for how you reason about that capability and what you do with that capability.
That accountability is yours.
Your job should have been build the right frameworks, ensure the right goals, monitor and measure the progress.
Yeah.
So I think the safety tier then.
So in this context, your sort of thesis, which I sort of see the reason in is it's really.
Like as a user of the agent, you are responsible for its safety.
Alignment is a different story.
Like the models should try to make it easier for you to keep things safe.
They should sort of lower the burden, but eventually the accountability for safety, for it not going off the rails, that's in the hands of the one wielding the agents, not the agents themselves.
And then listen, like also my responsibility to some degree is if I'm giving you this, you are also a little bit of my child.
If you just start going off and going crazy, then that accountability also may come back to me.
Right.
So like I will go and make sure I wheel in my kid in the sense that says, hey, like you can't you can't do this.
But at some point, right, like that accountability has to be shifted and they have to be responsible for this.
And I think like with any.
And you could call this also maybe a little, I'm just for this podcast, I'm being very simple here.
All right.
But like, you know, chef knives can be used for good things or bad things.
All right.
Any tool that is worth their salt can be used for evil or for good.
The question is, is at what level are you going to go to to validate that it's going to be used for good?
makers of chef knives or knives in general somehow put protections or legal agreements in place saying you're not going to stab people with it.
You know, like it's just at what level do you let it go versus you don't?
And I think that we're still figuring that out with AI.
Hey, everyone.
Hope you're enjoying the episode so far.
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
But for whatever reason, many of you have yet to subscribe to the channel.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
All right, back to the episode.
Yeah, I think, I guess that's probably the piece in which I'm...
sort of a bit in sort of a disagreement on the sort of the analogy of it, because I think all of these are kind of these inanimate objects that do not have decision power, right?
So if you're, like once you introduce decisions into the product that you are providing, then there is accountability for that decision, right?
Like when you have, that's why you have safety controls in systems is because they need to be there to be able to protect you.
And eventually, laws, right?
Whatever the safety belt, you know, that you sort of were in the car, but even like, so I think, I think this level of autonomy, you see this in, in weapon systems, right?
You know, you like that you need to, you know, like, I think there should be a higher bar when you have an autonomous weapon system that then it's not the same as like, I just sort of pressed a button, you know, and I've done it.
Like there is a, there is a, an accountability system.
sitting with the provider of the autonomy that needs to be there.
So that's why I think that sort of, it's actually interesting to think about this as safety and alignment, right?
Which is alignment being the responsibility of the alignment in some guardrails and some sort of established limits, you know, sort of sitting with the provider and then sort of safe use, maybe sitting with the consumer, right?
And also to your point, When you have something that reasons, and also going back to my analogy, in an alignment perspective and in a safety perspective, based on my trust with you and who you are, I think there are different levels.
Random Joe Blow consumer is going to get a pretty safety-based model in terms of what it's doing.
Yet, a large enterprise, Cisco.
is going to get a different level of model because of its accountability and its level of trust given.
Obviously, I think that makes things way more complex.
50 million political problems with this plan.
There's like a vilious judgment of it's like who's judgment.
Yeah, yeah.
It's like, well, who makes this decision?
Of course, all of that.
But like, I think very simplistically, I would love if the world were that way.
Yeah.
No, I like that model of like allow yourself to sort of lower the sort of the guardrails based on kind of confidence in your ability to operate this machinery safely.
So let's maybe like bring it down.
So like this whole world is like, again, super fascinating, scary and exciting and daunting on it.
Let's try to kind of get a little bit practical.
When you are attacked, doesn't matter if it broke out or someone literally pointed it to you, when you're attacked by an AI agent, what is different, you know, from a defender perspective?
And maybe like, what do you think are like a change in practices?
in sort of the pre-agentic attacker era of three months ago?
Yeah, number one, speed.
Number two, advancement.
The level of proficiency is off the charts.
I won't say much here, but I just happen to have some real-world experience with this just recently.
And speed is the number one.
The level of...
of speed and thoroughness and complexity at which the AI attacker uses is extraordinary.
And it doesn't give up and it's attacks all surfaces at once, right?
Because you've sort of spawned.
And, and so, you know, we, you know, first when you see it from the outside, right, you can see, you know, it doesn't just, you know, you know, vuln scan your websites and then do these things.
It vuln scans your websites, researches any existing things, finds O days, tries fishing your employees.
Like all of this is all happens at once.
Right.
So it comes in, you know, multiple layers all coming in inbound.
Right.
And so it's not, you know, the olden days of.
one or two attackers probing and pinging and like doing things.
Yeah.
And you would trigger something and start like raise your alarms or whatever it is, but you don't because now they're kind of scaling the walls on all fronts at the same time.
Exactly.
It doesn't matter if you see them.
They're coming.
They're like, it's like, what's that zombie movie, World War Z, where they all climb on top of each other to get over the wall?
Yeah.
Yeah, it's like it's like it doesn't matter what you've got.
You can you can sit there with guns and pop and it just makes no difference.
And so, like, I think, first of all, I think that speed and that scalability and that thoroughness and then also the proficiency and the expertise is super surprising.
Like, for example, you know, in one of the ways at which it got in, it found an O-Day in a product by because it was an open source product.
It went.
got the code, reversed the code, found the O-Day, used it against us in order to get in, right?
Which, by the way, in any pen testing firm or red teaming firm is not going to do that.
No, because it's just the cost effectiveness of it, right?
Like you might do it if there's like a $2 million bounty, you know, in front of it, and you have kind of good suspicion to do it.
But you do it very like precision led.
And I think over here, it just can be done.
It just gets done.
Right.
Very broadly on it because the labor has become cheap.
It's like the labor piece.
And similar to the swarm once that, you know, once the gates of the walls have been breached, then everything comes in and then it spreads internal.
Yeah.
And the things and the speed and also the things that that's also very surprising is it does a lot of things you can't figure out why.
Right.
Like it will start, you know, there's practical things like.
escalate your privileges, add users, which all happen at very rapid speed.
But it also does all sorts of things that you don't know why it's doing it.
Like it started like, oh, it starts infecting endpoints and then it starts replacing things on there, but we don't know why or what it's doing.
And so I've experienced sort of like the actual methods of at least today's AI.
And it's going to take time.
you know, it takes time for you to reverse and figure out what's going on.
Yeah.
Right.
Yeah.
In the scenario, because logically, even as a security person, you don't know, you see how the rapid speed of these things being instantiated and created, but you're not sure what it's connected to or for what reason.
It's not obvious.
Yeah.
And you have to figure out all the artifacts and see what it's using it for.
And then how do you shut it down?
Because shutting it down.
You used to think, oh, on the outside, oh, you just look for connections to OpenAI or Anthropic or weird connections back towards.
Well, it doesn't.
Yeah, you sever those.
But that could be its initial.
But once it's inside the enterprise, it goes, oh, hey, your cloud code config.
I'll just take that key, use that session.
Now it's using your AI.
Yeah, which I guess is sort of not entirely dissimilar to using your compute.
You want to sort of run a thing, but you can't.
It is still external network dependent.
But if your business is also external network dependent, then you can just like disconnect to the business.
That's right.
And by the way, once it starts using your tokens and other tokens, right, by the way, and it also starts going against websites that have also open AI gateways.
So it doesn't just use yours.
It could say, hey, like, oh, we know about already open AI gateways that uses tokens that are other random sites that were already previously breached.
We'll use those tokens to be able to do callbacks.
And it will write back doors that are like, oh, I'll write a back door in code.
It will sit in the code.
It will do dormant and will wake up and then reach back out using your tokens at a later time to then continue.
It's all these things that are actually like they're actually not new practices.
They're just practices that were reserved for the sort of the top echelon of attackers and and targets.
And they took time.
And so now they can just be burst out into everywhere.
And tokens being, I think, a really critical fuel, right?
It's a fuel that's required.
And so, you know, I think it will do a lot of work in order, depending on how you write, I guess, the attacker writes its goal, but it does a lot of work to continue its ability to obtain fuel.
And so if you shut it off or do things, you have to literally unplug the Internet.
Yeah.
Right.
You have to pull out the wall, the equivalent of pulling out the power plug on your computer in order to to stop it enough to figure out what's going on so that you can start cleaning this stuff up.
And so, like, I think in this new world, by the way, like.
That's massively damaging.
Very, very difficult to do because even in enterprises, you can't just pull plugs.
In real large enterprises, there's not a plug to pull.
You've got massive amounts of environments covered everywhere in different regions of the world who, by the way, are built on reliability on not going down.
And so like, you know, like this is a very interesting problem, I feel, obviously that.
Yeah, I know it sounds again, kind of massively cool and massively scary kind of at the same time.
I think so.
I'm curious about like the levels of kind of a defensibility that you need on it.
Like, I think the first one is, you know, you kind of made the comment of like, you don't care if you see them coming because they're just going to come.
Do you think do you think that's the case?
Do you think like detection?
is sort of less of a concern now?
Or do you need to level up your detection, like more canaries, more, you know, just like, you know, AI-powered monitors and all that?
Like then we'll kind of go into like preemptions or sort of like even preventions.
You know, I think, you know, there's sort of, let's call it...
What's the right word?
Maybe the naive version of detection is no longer a real thing.
It's not valuable.
I think that's the naive first approach.
It's a SOC incident that someone at the SOC needs to see and say, this is a real thing.
That's just not it.
Yeah, the theory is, well, these agents are everywhere.
It doesn't matter if you see them.
All that matters is if you stop them.
But I would say it's naive in the sense that Actually, what I think detection needs to be is detection needs to go from what it is today, which is, I think is, let's call it very, very low fidelity type capability to extremely high fidelity type capability.
Like we need to go into 8K and we're sitting on like 2K in our level of detection.
Because what you do need that's very valuable is exactly what.
did the attackers do on everything right from what files it read and to write what sockets it opened at what times where like all of this becomes forensic level crucial crucial capability in detection and monitoring that needs to be understood because this thing moves at such rapid pace, like the ability to understand what it's doing and how it's patterning, right, is very important versus having a detection that says someone port scanned something, right?
Like that is pointless, I think, in this world for sure.
I also think deception, to your point, becomes super important.
Because most organizations cannot respond to this in any way from a protection mechanism.
Right.
You can't define what correct is.
Like in theory, to be able to say it has done that and it is incorrect sort of assumes the existence of a definition of legitimate behavior into like a very high level of fidelity in its own right.
And we just don't have that.
Like, you know, it's just like the systems are just not built.
So even if you were able to instrument and analyze activity.
Anomaly detection can only get you so far.
And your ability to containerize this thing.
Nobody has that ability in enterprises.
These systems, major enterprise production systems are infected.
Quickly, contain.
No, no.
Like, one, I don't think the technical capability exists for most companies.
Two, I don't know if the business would even accept it.
Right.
Depending upon what you're doing.
So I think, you know, those are big.
So deception plays a very big role here because I think what you need to do is my analogy to this was like in and sort of the Avengers.
You've got sort of Doctor Strange and he brings people into the mirror world.
Right.
You need the equivalent of age.
Oh, holy crap.
AI agents are coming in.
The one great thing about AI agents is they're, they're soup, you know, they're gullible, right?
So if you can sort of work to the, the non-security folks on it, like the septions, canaries, tripwires, they're basically around setting up parts of your infrastructure.
You can go as deep as you want that look like the real parts of the system.
But in practice, if this, you know, S3 bucket was touched, if this sort of system was breached on it, you know, you set it up there to just.
be a trap.
So you cross a tripwire and now the attacker is playing a domain that is really just meant for it to be surveilled.
And so in kind of real world scenario, this is This sort of buys you time.
It helps you lure it, it trips it so you know the alarm, the silent alarm is tripped and you go there.
And it used to be hard.
There's another domain which is it used to be hard, but now there's sort of newer players.
I know of Tracebit on it.
There's a ThingsCanary.
There's like a bunch of solutions, but also you can set up a bunch of these things now with AI.
And so it's sort of easier.
to maybe place them.
But it's still a novel.
It's a domain that's kind of the top organizations have put in place, but it was kind of too hard, I think, for most organizations.
So it always remained like a contained market.
You know, I have so many things to say about this market.
I cannot, like, it would take up a whole hour, so I won't do that.
But what I will do is say I've long been a believer of deception because it is the real thing that caused...
cost to raise.
But what's funny, I think, about the deception market, you know, it started out with the honeypots, right?
It started out with actually trapping the attacker and watching.
Literally coming to go tripwire.
Yeah, yeah.
And observing what they do.
Like, that was how these things started.
Cloudflare started out, I think, as a honeypot, right?
Like, as a company.
And what happened is we got away from that, and rightly so, because CISOs and even, you know, personally myself, like, I don't care about watching attackers.
I care about deterring attackers from looking at me, right?
So then it became, okay, let's not, like, build environments.
Let's just put, you know, crumbs and tripwires around the thing just to detect them.
But now I think actually that's not even the right play anymore.
I actually think we have to go back to the original OG version.
of deception, which is creating literal digital twins at which you pull them into that mirror world to wrap them into what that is.
Because I actually think that you have to funnel them into that area.
And so I think we're going back around in a circle, right?
Yeah, and go deep.
I find that super interesting.
So I think you need to go kind of echoing this back.
So the AI attackers are sadly kind of a reality today.
They're not that sort of hard to pull together on it, and you're doing it, and people are doing it.
They are faster than anything we've sort of seen before.
And so I think your defenses have to be automated to a very large degree with agent-accuses as well, a few things that are sort of serving the defender.
A bit over here is like one is you like make sure you use AI or use agents to be able to set up your defenses.
Make sure you use agents to be able to monitor what is correct.
And also the attackers are still somewhat gullible.
They're kind of a little bit of a, you know, an elephant in a whatever, a bull in a china shop.
And I'll come again.
It's like doesn't necessarily help you prevent the bull from breaking everything around on it, but you can figure out that it's there.
And so now the question is, can you create paths to sort of lure it into?
as one defense mechanism.
What about the defenses themselves?
So like OpenAI talks about sort of a defense factory.
I think a lot about like software defense factories.
We think about automated remediation loops.
So the big organizations have for a while now been under this sort of project Glasswing and the OpenAI cyber defense equivalent with Mythos doing kind of hardening critical systems.
But those are...
like the top systems, they're paying ridiculous amounts of money to be able to do those.
Like, what do you think is the kind of like realistic company's reality, right?
Or like, what should one do to try and make it harder for the, for attackers to get to this reality?
Like, what should one do?
You know, there's a lot around this.
I think, you know, Depends on what level, right?
I think you first, I think you need to start at let's let's call it the political level around.
I think like here is the biggest barrier and problem that we need to overcome.
Everyone talks about how attackers move at machine speed.
Right.
And defenders move at meeting speed.
Right.
And this is the problem.
Right.
We also talk about and especially in security, we also talk about human in the loop.
You have to have human in the loop.
Both of those things are the problem.
You need to be able to figure out how to remove humans, period, out of defense.
That is the only way.
So how do you do that?
The first starts with you have to get in your company, in your organization, political agreement that these are two fundamental problems.
And how do we get to a point where we can start building non-human in the loop automation?
on defense and make that work properly.
This is the biggest, and again, this is meeting speed, right?
This is the problem is, is your head of engineering, your head of infrastructure, your CEO, like who and who and who is going to say, I agree with that.
And then practically get down to the places at where you can start applying it.
Right?
So the way like we like to think- And accept the risk involved, right?
Because you sort of accept that like the risk equation has changed.
There's a risk of moving fast and there's a risk of not moving fast.
And the risk of not moving fast has really leveled up.
And so you have to like accept that that fundamental risk equation is done.
And just sort of really level up your acceptance of a moving fast risk and introduce controls and sort of evolutions to be able to mitigate that risk.
sacrifice the moving fast piece.
Correct.
And so I think like, let's take a super obvious example, patching, right?
Like in your organization, it's very clear.
You, there's no such thing as like, you know, seven day patching periods, like, or 24 hour patching period.
Just when a patch appears, you patch and you redeploy and it needs to be instant.
It needs to be non-human.
It needs to be done.
Who in your organization is going to agree to that and accept to the fact?
Because engineering, and rightly so, is going to push back and say, well, that can break things.
Yes, but our job, your job now in engineering, is to say, I accept that things are going to break.
I will build a plan that is also non-human in the loop that identifies breakage, restores, goes, and produces.
Kind of autonomous review, autonomous detection.
And also, it's actually like a great example because statistically.
it almost never breaks the system.
Like it sort of, it sometimes does, but it almost never does.
And so I think it's very much a spawn out of fear because of this sort of immeasurable kind of, you know, like hard to grasp risk of not patching versus the much more like real and immediate risk of breakage.
Correct.
And I, you know, it also stems, I think just from, you know, 20 years of when that used to be real problems, but like that they're not problem.
They're very rarely to your point, that problem today.
Um, but yeah, like that needs to be figured out.
So you have to like, you have to find the things, um, we have these things called one way doors and two way doors, right.
Uh, which are pretty simple.
Hey, one, one way doors.
Maybe we keep a human in the loop, two way doors.
Maybe we don't like.
Writing a Jira ticket or submitting a PR or making configuration changes that aren't around these types of things, these can all be two-way doors.
Therefore, they could be fully automated based on the reasoning of AI and we're fully able to let it go.
I love that path.
I guess kind of a subsequent question in my mind is who's responsible for that?
Because oftentimes, security provides the...
patch, like, sorry, security reports the vulnerability, finds the risk, right?
And increasingly, because of the sort of unwillingness to act on the engineering side, or the risk of breakage, a lot of onus on like, don't just find the vulnerability, but be very meticulous around how will you triage it and sort of assess it and like, you know, be very certain that this is a real issue and a real vulnerability for many good reasons.
I'm not sort of doing it.
and engineering was kind of tasked with fixing it.
Is this now, are the fixes, like if you're moving into more autonomous patching on it, what's the sort of security role versus the sort of, say, software factory world, right?
Like, does the software just auto patch?
Yeah, I think modern companies, especially modern security teams, it's not throw a thing over to engineering.
We patch.
We submit the PR and depending upon the level of where it is, it gets auto accepted, blah, blah, blah, blah.
Like that, that an engineering problem says we understand the risk of breakage.
Our job is to ensure that we, you know, have good sustainability, reliability and management of breakage.
Right.
Like that.
It's interesting because it's a reversal of roles.
You're basically saying security will build the automation.
Engineering will manage the risk.
They manage the uptime risk, right?
And functionality.
But it's a reversal of roles to an extent, right?
That's right, yes.
But modern security teams are absolutely submitting the fixes.
If you have a security team that only reports vulnerabilities and hands it over a wall, that is 10 years ago.
That should not be the way security, especially in AI world.
AI world, you need to do this on your own.
And I think that's already been true in cloud world.
But I think what has happened was security needs to not just report the vulnerability, but it needs to report the vulnerability with a high level of triage.
We often have these conversations of, when is it easier to patch to fix than to triage?
I don't care if it's a real vulnerability.
I don't care if it's exploitable in my system.
It might be, and it's just easier for me to fix.
Just roll the patch out.
It's just easier to fix.
And so I think that's the joint kind of mission, which is make it easier to fix than to triage, because triaging at this point is too much effort.
You still need to triage because there will be some things that are hard to fix, and so you still need the ability to assess those.
But generally speaking, I think actually the thing AI has brought us that's good for defenders is I think the thought process that by and large, to some degree, prioritization is not really that big of it.
You just take the things and you just fix them.
Yeah, like we don't need to go back and forth on this anymore because it's, at least in the modern sort of company today, it is not taking up any man hours, right?
Yeah, it's a little bit of token hours.
Like we see, like at TESL, we do these loops, you know, sort of automated.
So the skill-powered agent automation, vulnerability remediation is one of the most powerful or most popular loops to be able to auto-remediate stuff.
And it becomes a very, like when you look at the numbers, right, of like invocations on it, it becomes high up there on the list because you keep getting these patches because the software is also changing at a faster pace.
And so you're introducing and changing and kind of modifying dependencies.
You're introducing new code that has vulnerabilities.
Who cares as long as you can fix it quickly?
Yeah, yeah.
I think you and I.
In our previous podcast, we talked a lot about this.
Yeah, exactly.
Which I would say, I think we were accurate.
So I'd say we were accurate.
I think we were correct.
It's really fundamentally the definition of what correct looks like, the confidence to increase autonomy.
What is the means to increase autonomy?
And I think we have, like it or not, kind of more proof of the necessity of that, which I think is really important.
Keld, mindful of time here.
fascinating kind of reality, scary as hell of it, but I think a reality we need to build into.
Thanks for sort of sharing perspective and some hopefully practical tips, not easy, but practical.
And hopefully everybody increases that automation.
We level up the AI defenders alongside the AI attackers.
We have a long way to go.
This is as robust.
Yeah, there's a lot of work to do on it.
We're not going to be bored anytime soon.
One final thought about this exact thing is, you know, although, you know, just like, you know, before AI, I feel attackers were always ahead of defenders, right?
And defenders, like attackers were here, defenders were here.
And with AI, actually, I think the distance between attackers and defenders...
the attackers move up a little bit, but then defenders go up a bit more with AI too.
But at the end of the day, the actual distance here has not gotten any closer.
It's actually gotten a little bit farther, but at least we're somewhat keeping up.
The defenders are actually better, but the attackers are even better.
Correct.
Well, I mean, I think you have to go through a few rounds of failures.
It's the best sort of motivational, sadly, the best kind of way to motivate these kind of meeting speed.
Those are the fast meetings when they occur.
Cool.
Caleb, thanks again.
Great chats on it.
And thanks, everybody, for tuning in.
And I hope you join us for the next one.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast, it's a community.
And we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more.
And I hope to see you there.