# AI Security Arms Race and Open Source Shift

**Podcast:** Dev Interrupted
**Published:** 2026-04-10

## Transcript

Andrew, what opinion have Claude Mythos?
Andrew have many opinion Claude Mythos.
Claude Mythos, big model.
Yeah, all right.
We'll get into why we're talking like a caveman.
But yeah, Andrew, Claude Mythos, it seems like everyone's talking about it.
Is this what you're hearing out there right now?
Oh, yes, indeed.
And welcome to the Friday Deploy, y'all.
It is true that people are talking about Mythos out in the wilds here at HumanX.
I've definitely heard it on people's lips here on the expo floor.
And definitely the security-minded companies have been top of mind for them.
I think Mythos is a really fascinating kind of sea change event in model capabilities, especially in a realm where there's already a huge amount of disparate...
abilities between attackers and defenders in the cybersecurity space.
Anything you're going to put out there that defenders can leverage, an attacker can leverage 10 times better and faster and more aggressively.
The really unbalanced world.
So Mythos entering it is, you know, I think really trepidatious for some people.
What have you been hearing about Mythos?
Yeah, I mean, the cybersecurity, we'll get into that here in a moment.
You know, I've just kind of come to expect that every, I don't know, every month now, maybe every two months, the timelines seem to be condensing where some new major incremental improvement comes out.
So, you know, the current Frontier models are pretty amazing.
So I only kind of expect the next ones to sort of take it a level up.
So, but yeah, so let's get into it.
Yeah, as you mentioned, this is the Friday Deploy.
I'm your host, Ben Lloyd Pearson.
And I'm your host, Andrew Ziegler.
So this week we are covering this AI cybersecurity arms race that we were just discussing.
We'll also go over how to read AI scorecards and benchmarks.
We'll talk about some open source frontier breakthroughs.
And then we're going to get into why Andrew and I started this episode talking like caveman, because it's actually a really cool story.
Andrew, excited.
Yeah, exactly.
But let's kick it off with Project Glasswing.
So this is a new thing that has been announced from Anthropic.
They're partnering with a bunch of major tech companies to use advanced AI models for finding software vulnerabilities before attackers do.
So we were getting into this a little bit, but Claude Mythos has been previewed, or Anthropic released a preview of it.
And along with it, it seems to be becoming a lot of just like concerns and warnings about how it could be used maliciously.
One of those ways is that it's finding a lot of security vulnerabilities in very commonly used libraries.
Like, for example, it found very old bugs within FFmpeg that have been undetected, you know, despite being scanned thousands of times over the years.
And it really does, you know, highlight that there is this urgent arm race between AI-powered defenders and attackers, you know.
So in this, Anthropik is partnering with like the Linux Foundation and a whole bunch of other like big logos.
I don't remember the whole list, but there was a bunch of companies on that list.
And they're committing 100 million in usage tokens to help organizations scan their systems.
So it's really great to see them being proactive.
And, you know, I think the mythos has like some doom and gloom around like how security is about to become a nightmare.
But at the same time, I think it's also really good to focus on.
The good things that AI can do, like longstanding security vulnerabilities in FFMPEG should be fixed, whether or not we have AI.
So we can also use AI to do those things.
So yeah, Andrew, what are you thinking?
Because I know you've been following some of this stuff pretty closely.
The mythos rollout, or rather development, and then them partnering with organizations in this Project Glasswing Initiative, I think is a really great play to see from Anthropic, who sees itself as a partner.
with the software ecosystem, the organizations that they reached out to maintain the software and operating systems that power the entire world.
And by just partnering with that small, very concentrated, large group of organizations, you get like a really widespread over all of the tools we use every day.
So Anthropoc was definitely really smart in partnering with those people first because they acknowledge this inequality between attackers and defenders in the cybersecurity space.
And I definitely think that this is, it speaks to the ethos, I think, of Anthropic as well.
I don't know if necessarily OpenAI would have made that same decision.
I think they probably would have more welcomed the disruption as opposed to trying to gently roll it out.
However, of course, Anthropic too, I think, also has ulterior motives here.
We're living at a time where serving Opus to their customers is extremely difficult for them at scale right now.
There's been a lot of talks about folks having different experiences with Claude Code and with Opus in particular, getting good usage out of it, getting the limits and the mileage out of it that they used to.
And there have been strains around delivering that compute.
So when you're talking about something like Mythos, which is just an order magnitude larger model, you're talking about something that's more expensive.
At the same time that Anthropic has to partner with these organizations to improve their security, they probably also got to beef up their data centers that actually deliver mythos.
So I do think that there's like a dual play going on here.
But Project Glasswing, I think is a great initiative to see from the foundation model provider.
Yeah, I mean, it feels like an arms race.
You know, we have people that want to do really big, good things with AI, and they have to race to stay ahead of these malicious actors that are getting their hands on the same tools.
But, you know, like I said, these issues would persist even if we didn't have AI.
So they would be out there.
They have been out there in the wild, allegedly.
So fixing these vulnerabilities is going to be super important.
I think it's actually a great thing for society at large to have something.
this powerful that can solve security challenges.
But yeah, I'm going to be watching this project pretty closely because, you know, there's so many big companies about it.
I really like the partnership approach, as you mentioned.
And, you know, just generally having AI agents, you know, and all these organizations coordinated on using AI to solve really big challenges is, you know, that's a significant development, I think, just for the industry at large.
I agree.
All right, Andrew, let's talk about...
Benchmarking 101 AI Tool School.
So what's this article?
Yes, this is an article on Substack that I really love.
It's from a blog called In the Weeds, where they do exactly what the title says.
They actually get really in the weeds on some of these more technical and nuanced topics, particularly around AI.
They have a lot of materials around learning different things.
And one of the most recent ones is...
benchmarking 101, understanding how to read a model card, understanding the relevance of a benchmark and how to understand that model's score on it and what it means.
There's a lot of fluency involved in understanding the competency of a model and scorecards have only gotten more complex.
And if you don't already have experience and background in understanding them, they can be really hard when a new model comes out and you see the benchmark results to actually think for yourself and, you know.
Should I try this instead of what I'm currently using?
What might I expect?
What should I be looking for?
And at the same time, when people use models that improve, like you move from Opus 4.5 to 4.6, you can feel it get better or you can feel it perform better, but maybe you lack the verbiage to explain or to really pinpoint what is different.
And what this article aims to do is to make you more capable and fluent in understanding those things as they come out so that you can...
better evaluate those for your team as they happen.
A really great shout out about this too is that they make it so easy to learn.
Like you can just go clone a GitHub repo where they put all of the resources and then you go ask that repo your follow-up questions and dig more into it.
And I think this is like the future of learning.
This is already how I package up most things for other people to consume outside of this podcast is like put it in a repo or put it somewhere where they can just ask questions of it.
So really smart teaching approach from them.
And I highly recommend you go check it out.
Like, and the biggest takeaway is, is that if a bunch of models are scoring 90% or higher on a benchmark, then the benchmark doesn't matter anymore.
And the older a benchmark is, the less it matters too, because the more likely it is to be in someone's training data.
And the best benchmarks are ones that models universally struggle to solve, but humans universally don't struggle to solve.
And a great example of this is the ARC AGI 3 benchmark.
which the best foundation model out there can't even get 1% on, but any human can score 100% on.
These kinds of benchmarks are really important for actually delineating the capabilities of the model.
So strongly recommend you check this one out.
Yeah, first of all, call out the GitHub repo approach to their content.
You know, I love seeing all these new content strategies emerging in the AI era, like new ways of packaging up information for both humans and agents to consume.
It's a really cool approach that we might learn some stuff from.
But yeah, this is a dense article and in a good way.
You know, it's very information dense.
And it is probably the best breakdown I've seen on how benchmarks themselves work.
Because, you know, whenever I've seen like these ratings on like it scores a certain model scores a certain percentage on a benchmark, I kind of my eyes just kind of glaze over.
I've never really understood.
I'm like, cool.
I guess that sounds awesome that they can solve tests, you know.
But it does a really good job at breaking down the success rates for various tests.
And if you have a benchmark that, for example, has low average scores, those are the best ones for differentiating models.
So if everyone's getting an A all the time, you really can't trust that test anymore because it's been, for a variety of reasons, it's no longer a good way to judge model performance.
You know, and the part of the challenge that exists in this space is that like models are improving at this like accelerating rate and solving tests practically as quickly as they can be built, it seems like.
So, you know, we're talking about mythos and how all these new and we're actually going to have another story here in a minute.
How there's just so much innovation happening on the models themselves that the tests are really struggling to keep up.
And then not to mention you have Goodhart's Law.
Like that still applies in the AI world, right?
Like if you tell your model, it's really important that you pass this test, it's going to figure out how to pass the test.
But yeah, and you mentioned ARC AGI 3.
This really does seem to be like a new way of benchmarking.
Like it's a new benchmark of benchmarks in a sense, like because it has dramatically shifted the performance, you know.
And I'm like, they're saying that the best frontier models are scoring under 1% success rate, whereas a human is getting a hundred percent, like pretty easily.
Um, and it's, and it's like, just based on like these random puzzle games, it's actually really cool.
Like I played around with it a little bit and was like, wow, this is actually really neat that like, like for a human, I just intuitively figure it out, you know, without a whole lot of thought.
Um, but you know, I, I actually am wondering if like, we're going to start to see this like pattern of behavior emerge.
Or a pattern emerge where we see patterns of behavior that humans instinctively follow, but it's very difficult to replicate that with an LLM.
Like solving a game that has no instructions on how to do it, just you have to intuitively figure it out by playing the game, is so far a way to differentiate a human from an AI actor.
But I also think there's probably, so like design, for example, I think is, you know.
particularly for like game design.
I think that's like one of those skills that humans have that is really difficult for an LLM to replicate.
And there might be other things like humor, beauty, like being able to judge if something is funny or looks interesting or attractive.
You know, those may be other tastes that like humans are uniquely positioned to understand.
But I do want to make one final point on this.
So I feel like ARK AGI 3 is doing exactly what I've always wanted from this era, and that is bringing back the web arcade.
Now imagining like, wouldn't it be cool if we had like online communities that use puzzle games as the gatekeeping mechanism to participate in them?
Oh, like you can't hang out with us unless your world score is like this good or whatever.
You know, you have to solve, you have to solve like these four puzzles with a hundred percent accuracy before you're allowed to like comment in our community, you know, like stuff like that.
I love the idea of a puzzle gated community as a puzzle person myself.
So I welcome that.
Yeah, yeah, exactly.
All right, so let's talk about some of these other innovations that are happening in the frontier model space.
So a whole bunch of updates coming out in a short time, Angie.
What do we have going on here?
So in the last week, it's kind of like a blink and you miss it scenario.
Four open models have hit the scene that have totally transformed the ability of an average user to own the capabilities of a frontier model.
There have been four major AI models released in the last week.
One of them being Gemma 4, already a really...
well-established model library of on-the-edge and small language model for devices and stuff.
You also have models like Bonsai, which is a much thinner parameterized model that gets great performance on small machines.
You have Trinity, which is a performance at around an Opus 4.5 level and less than 95% of the compute and run cost.
You have Holo 3, which is a state-of-the-art desktop automation agent.
And all of these are released under an Apache 2.0 open source license.
And this is a dramatic shift in how these models have been created and put into the environment.
Gemma and Google, they've never put these models out under this kind of license.
And Apache 2.0 is one of the most permissive open source licenses.
Allows you to fork it, modify it.
and then build a business and sell a product on top of it without owing anything back to the original provider.
All four of these are offered under that same kind of license.
And so what you're going to start seeing are companies that have up until now been pretty beholden and dependent on using OpenAI or Anthropic to get the large-scale compute they need, likely doing what we actually saw Shopify did last week.
We covered a story where they used Quen to fine-tune a local model.
And in the process, unlocked a multi-agent architecture that was profoundly more cheap and profoundly more effective for them because it was trained on their data.
And it was a model that they owned because it was a Quinn model.
This is kind of like a similar kind of trend where this is opening the door for other software leaders like those at Shopify to look at their compute costs and look at the problems that compute is solving.
And then look at these models that are now out there and say, How can we fine tune and serve these ourselves in order to reduce the cost?
And obviously there's boundaries and things around this, like getting the machines and the GPUs that you need.
But when you compare that to a monthly bill from something like OpenAI or Anthropic, it's likely still an order of magnitude cheaper.
I think it totally shifts like the economics around models and the accessibility for them.
And I think you're going to start seeing a lot of specialized fine-tuned models in large and small companies.
What do you think, Ben?
Yeah, well, I don't want to say I told you so too early, but it really is, does feel like my theory of the commoditization of model capabilities being like the, one of the prevailing trends of the near-term future.
You know, things are going to get cheaper and easier to run at a higher level.
of quality.
But yeah, I like that you brought up Shopify because the reason we covered this or one of the reasons we covered that story is that when we're speaking to friends of the show out there who are engineers who are building agentically now and have adopted this agent orchestrator mindset, so often this is just the mind of an engineer.
They immediately wonder, well, can I run this locally?
Can I own the infrastructure?
Can I break it apart and actually...
own the component parts of what I'm building here in this orchestrator.
And a lot of that ends up in like trying to look into stuff like Quen to see if they can run these tasks locally.
But, you know, at a high level, I just think we're in an incredibly interesting space or place right now with frontier model development.
You know, we have stories like Anthropic with Claude Mythos coming out where the highest tier of capabilities are continuing to increase.
Um, but then we also have all these stories here about AI is getting more and more efficient and easier to run on cheaper hardware at a lower cost.
Sometimes locally, you know, um, I think it was Gemma.
They were saying like one of the models you can run on a Raspberry Pi now.
So like everyone go dust those off and make it your, your AI agent.
Already dusted off Ben.
We already, we already got a rack of them, but yeah, you're right.
Yeah.
Uh, so.
You know, it only makes sense to me to do more and more of this work on your local machine or on infrastructure that you own.
You know, particularly as we're seeing more agentic systems emerging like OpenClaw.
And one of the stories that was in this article about the Halo 3, you know, which has, you know, they have that computer use agents that you just point to your desktop and it can navigate around and do things for you, use your web browser.
I actually want to call out just some like big issues with this before.
You know, just sort of separately, you know, first of all, is that like, you know, our desktop experience was built for humans.
It wasn't built for agents, you know, and I feel like instead of trying to force fit our agents onto an experience that makes sense for us, we should be building the experience that makes more sense for our agents.
You know, so to that end, like the user space of a typical operating system just isn't set up with things like a basic permission schema for your AI agent.
And, you know, and that specifically is a recurring theme we're seeing with AI, like the software experiences we've built have the assumption that there will be a human at the center of it operating, operating it all.
But yeah, I mean, to that end, you know, there's just, there's so much happening.
And even if I have some issues with the specifics of where this technology is today, like I still see the promise of it.
And I'm looking forward to like seeing additional iterations, but.
Like right now, it just feels like we're kind of in a peak AI moment because of it's, you know, it's the slope on a slope thing.
Like things are getting better.
It always feels like we're in a peak AI moment right now.
Our benchmark for what a peak hype cycle is needs to be readjusted.
Re-evaluated for sure.
You hit on a really big point about how the user space of a machine is just typically not created for an agent.
I do think this is like what things like OpenClaw and Nemo and things like from NVIDIA that are up.
halfway in kind of more than agentic runtime or are starting to solve, like how do we create that machine that is just intended for the agent to use?
But in the meantime, we have all of these experiences that we have to use every day that are tuned for us.
And there will continue to be experiences that are made exclusively for humans.
And so the need for an agent that's able to operate in that same space will probably never go away.
But like you said, probably becomes less important as agents start to get their own agent space, maybe operating system.
And as we have this fracturing of all of these models and tooling and capabilities, I want to just go back to the last article we covered real quick.
Because one thing that that article really pointed out was how to analyze models for specific capabilities.
So if you need something that can write code, something like an open AI model is typically better at that.
But if you need...
really deep academic understanding Gemini versus Claude is more of an enterprise task solver.
So as we're doing things at a smaller scale and using different local models and different ways of orchestrating this, that becomes more important than ever because you don't want to just be using a single model for all of your problems.
Every model or every task should be form-fitted to the model that you're giving it to.
All right.
Well, we got to close out by explaining why we were speaking like caveman and why this may actually become our default language going forward.
Because honestly, I really like this idea.
But this is a new cloud plugin called caveman.
It reduces AI output tokens by 65% or more in many situations by simplifying all of the output into simple caveman speak while maintaining full technical accuracy.
So you can choose from different levels from like a light mode that is more like a professional but concise version of this to like a maximum compression that just like abbreviates things and takes out any word that doesn't add meaning.
So when they were benchmarking this, you know, they were seeing improvements ranging anywhere from like 22% to like all the way up to like 87% across different coding tasks.
And, you know, it just significantly increases faster response.
times and, you know, lowers your overall token cost, particularly if you're generating artifacts that get fed back into your AI.
So Andrew, what think?
We need Caveman?
Andrew, many thoughts.
I think that the Caveman plugin is fascinating.
I like the idea that people want it to speak less.
Obviously, it's like no one likes the feeling when the AI is kind of just like not.
solving what you need, but then also being way too polite or flu, like flowy with its language.
Just adding words that don't help you.
Just adding words.
The problem with this is that you end up with a polluted context.
If you have an agent that or a model that's just talking and it's not saying the right thing over time, that compounds and compounds and compounds and just creates confusion.
Personally, I have some skepticism that like.
You know, maybe the caveman experience does the same thing, but in the other direction.
Like it's not really able to look back over its own thinking and its responses and really piece it together.
But frankly, it's like it probably still is because a lot of the functions of human language are just there to help make things flow for us or to add little tiny bits of nuance that just aren't really, like you said, necessary for these environments in coding.
I do think that...
This is an interesting shift to see because like there's also been this open GitHub issue on Cloud Code about Cloud being less capable at complex engineering tasks.
And this has been a viral thread.
I mean, Boris is in there himself and there's people giving all sorts of bug reports as well.
And talking about like the degradation, the experience and Opus' ability to speak and reason.
and share its thinking.
Anybody who's been a Cloud Code user has seen over the last few months, you're used to be able to peer into the thinking of the model between all of the steps, but slowly Cloud Code has kind of obfuscated that or rolled it away.
And now most of the thinking steps happen in places where you don't see.
You see the final outputs and you see the tool calls.
And this fundamentally kind of changes the dynamic, I think, of what you think the model is doing because you don't get as clear of a glimpse into its mind.
So in that world where maybe now there's a bit of opacity in understanding the agent's thinking, when you combine that with a sycophant agent or one that's just like spewing stuff that ends up not being correct, then you get a really miscalibration.
Users aren't able to understand like, oh, this is where it derailed in its thinking and this is why it's saying that.
It just gets what it said.
So the idea of...
Trying to like simplify the outputs and try to like make that simpler, I think is like one of the many experiments that users are doing right now around Cloud Code to try to maybe get performance out of it that we had before or to figure out why these small nuances do keep popping up and changing.
I do think that it's like a pretty funny idea.
to think that I'd be sitting there with the agent talking to me like a caveman.
I think I'd be pretty tempted to talk back to it like a caveman.
If you're using this plugin, I really want to know.
I want to know how it helps you because it has like a fun novelty to it.
But I don't know if it's something that I would use for my daily driver.
I kind of like it telling me a little bit more.
Yeah, well, I know you use speech to text a lot, which I do as well.
But I also do occasionally find myself typing into my AI.
And I definitely go straight to caveman mode.
Like one of my favorites, one word sentences right now is the sentence fix.
Tell it everything that's wrong and then just say fix.
No, please is too many tokens.
You're right.
You're right.
Not even the three letter, please.
Yeah, you know, that's all right.
All right.
We can do that.
Yeah, that'll be the light caveman.
We'll sprinkle, we'll sprinkle those.
I mean, cavemen don't necessarily have manners, so I don't think it's going to be a sentence.
Yeah.
But I kind of feel like I legitimately do need this in my life.
And in fact, I would go as far as to say it needs to be a toggle in Claude without having to be a custom plugin.
That's incredible.
The number of times that I have given it a percentage of like, make this 70% shorter.
I am overwhelmed by information and it still doesn't achieve that.
And I have to do like multiple rounds of just like, no, cut more, please.
See, there we go.
Yeah.
So if you use Caveman, let us know what you think.
How are you getting the compression out of it?
I'm getting more miles out of your token.
And do you talk Caveman too now?
Or does it change how you think?
Yeah.
I mean, we were just talking the other day about how like one of our new agents is like super useful, but it's incredibly noisy.
Like it just generates so much information.
And, you know, I kind of liken it to being like, it's almost like a refinery.
bringing in raw ore and processing it into like usable artifacts, you know?
But it's, it's like, you gotta have like hearing protection because it's just like, things are just roaring and like tons of data is coming in and things are changing constantly.
But yeah, I mean, it's like, we do need a simplified layer.
Like it, it is really nice, especially when you're working with some of the more verbose models to have a thing that just flattens the, the, the context into something that is easy for both humans and AI.
to rapidly consume.
So yeah, it's a really cool thing.
I think everyone, I think our listeners should go check it out.
So, you know, even the token cost aside, the human cost, I think is worth it.
Yeah, the cognitive cost.
And I think if I think I'm hearing you correctly, Ben, I think what you want me to do after this call is to go make the agent that I built talk like a caveman.
Now I'm intrigued.
Well, actually, what I want is to have Claude connected to that refinery and have that Claude speaking to me like a caveman, you know?
Caveman orchestrator.
Yeah, exactly.
Yeah, cool.
Well, beyond speaking like caveman, what are your agents up to right now, Andrew?
Well, right now, while I'm on the ground at HumanX, I'm doing my new favorite thing to do at conferences, which is roam around, take pictures and talk with folks at their booths and then drop interesting links, get repos and resources into an agent, get a daily report, daily digest on here's the things you saw.
Here's how they might be relevant to things that you're building.
I love doing that.
It allows me to quickly Scan around and leverage anything that might be uniquely useful for me on the expo floor.
Definitely, I'd call for attack if you're not doing that.
Put an agent in your pocket and walk around.
It's really powerful.
As well, I'm competing in the Intrinsic AI hackathon right now remotely on my phone.
I've been fine-tuning a model to insert wires into a microcontroller as part of a hackathon competition for Intrinsics.
I'm fine-tuning my first model, and it's operating a machine robot in a simulation.
And then in another tab, I'm working on the Gemma for Good hackathon because it's part of Gemma 4 coming out.
Gemma's having an amazing hackathon, a global hackathon to solve a bunch of amazing use cases around AI and making it more accessible with their new model that you can fine tune and make accessible on the edge and on devices.
And so I also have been brainstorming that with an agent while I roam around as well.
So I've been in planning mode.
How about you, Ben?
Incredible.
I feel like I can't follow any of that up, let alone all three of those things combined.
We'll see how it goes.
It's bippin' and boppin' between the terminals and the people.
But I have been really embracing our refinery that we built that I've just described.
And I'm really trying to embrace that.
the, as Yege would call it, the wasteland.
It's like the solved work that's around me.
It's like, now I have an agent that has like solved a workspace for me.
So like, what do I do on top of that?
What do I plug into that?
Like really starting to think about the higher order challenges that I can solve because I have agents that just solve a challenge for me that used to consume significant amounts of time.
So yeah, take advantage of those time savings, cash them.
Yeah, exactly.
All right, cool.
Well, thanks everyone for joining us this week.
That's the Friday deploy.
Make sure you subscribe.
Give us a thumbs up on whatever platform you're listening to.
Rate the podcast.
Yeah, thanks for joining us.
See you next time.
AI is everywhere in software engineering, but most teams still can't prove its impact.
That's where the Apex framework comes in.
Apex is a new operating model for engineering productivity designed to measure AI where it actually matters at the pull request level.
It connects AI activity to delivery outcomes, not just tool usage.
Apex is built on four pillars with AI leverage, predictability, efficiency, and developer experience.
Apex helps you increase throughput without sacrificing delivery confidence or burning out your team.
Because speed without predictability creates chaos and faster coding often shifts bottlenecks downstream.
If you want to operationalize AI the right way, Linear B and Apex gives you the system and the cadence to do it.
download the guide and start measuring what matters