# AI Market Shifts: Regulation, Compute Moats, and Enterprise Margins

**Podcast:** Last Week in AI
**Published:** 2026-06-25

## Transcript

Hello and welcome to the Last Week in AI podcast.
We can hear us chat about what's going on with AI.
As usual in this episode, we will summarize and discuss some of last week's most interesting AI news.
I am one of your regular hosts, Andrei Kurenkov.
I studied AI in grad school and now work at the AI startup Astrocade.
And I'm your other regular co-host, Jeremy Harris from Gladstone AI, AI national security, AI infrastructure, all those good things.
And yeah, we're just talking about how this is a lighter episode than usual, not just because it's only five days instead of seven since the last time we recorded, but also just like fewer papers.
We're shooting the shit earlier or shooting the S as we say on the podcast about why that might be.
I opened by saying, I mean, it makes me wonder how much things are getting starting to get clamped down on a little bit with the labs in terms of how open they're being.
But then Andre just pointed out the sort of devastatingly clearly obvious reason why this is actually the case, which is conferences and timing.
So yeah, not everything is a conspiracy, everyone.
Not everything is a conspiracy.
Yeah, I'm a little bit out of a loop, but when there's a quiet period, it's a pretty safe call to say that it's in between major AI conferences.
I mean, not to say that I think we highlight like really blockbuster papers often are from industry, anthropic in particular, but also...
in some cases other labs, but we do cover quite a bit from academia as well.
And there has been not as much that I've seen.
Not to say that there's not been many papers.
There's always a lot of papers that are interesting and worth discussing, just not ones that have gone semi-viral on AI Twitter, I guess.
Actually, this is kind of, since this is a shorter episode, maybe worth the digression here, but like...
Reading between the lines, the papers that I see Andre picking and the papers, at least speaking for myself, that I end up picking are generally like there's kind of a couple lenses that at least I find myself applying.
The first is, will this paper materially change the macro picture?
So papers around about scaling, you're seeing a result that actually not just any random scaling paper, there's a million of those, but scaling papers that show trends that seem.
robust that seem to apply to architectures where the hardware lottery is not going to be a massive problem.
In other words, they can run pretty well on current hardware.
That's a credible kind of paper where we will double down on those alignment results that seem certainly anything that borders on like super intelligence, like super alignment grade alignment.
Really interesting.
Same with interpretability.
At least to me, that's kind of like the first order passes.
Like, is this a paper that may change predictions around?
When AGIASI comes, recursive self-improvement, the risks and the outcomes of that process, that's kind of like the first order.
And then there's like the kind of geopolitical, does this change US-China dynamics?
Because maybe there's a breakthrough that spares us on memory, which means that one of the key bottlenecks that we have breaks, and now we can get help or vice versa with China.
So I think that's a lot of like what comes to mind for me and just kind of like economic transformation stuff too.
I just like stuff that's interesting.
It's this information that seems novel and exciting, I guess, from a research perspective.
So I guess together we have a nice rounded out set of things we like to focus on.
We'd like to thank Box for being a sponsor.
If you're a company trying to adopt AI, you're likely facing a common challenge.
Most AI tools are great at public knowledge, but they don't actually know your business, your product roadmaps, your sales materials, your policies, all that stuff.
And that's where Box comes in.
Box is building the intelligent content management platform for the AI era, serving as a secure, essential context layer for Box's AI agents to access the unique institutional knowledge that makes a company run.
The power of AI doesn't come from the model alone.
It really comes from giving AI access to the right enterprise.
content and that's where box is perfect it goes beyond file storage it connects content to people apps and ai agents so teams can turn information into action with tools like box agent box extract box hubs and more organization can accelerate knowledge work pull intelligence from unstructured content and automate workflows so if you're thinking seriously about ai think beyond the model Your business lives in your content.
Box helps you bring that content securely into the AI era.
Learn more at box.com slash AI.
This episode is brought to you by OutShift, Cisco's incubation engine.
Today's AI agents operate in silos, limiting their true potential.
We've been focused on building bigger, smarter models, but scaling up models is just one approach to improving AI.
To reach superintelligence together, we need to do more.
We need to scale out.
And we actually have a blueprint from 70,000 years ago.
Humans didn't just get smarter individually.
The cognitive revolution transformed society because we began sharing knowledge, goals, and innovation.
Agents are now at the same inflection point.
They can connect, but they can't think together.
That's why our shirt by Cisco is building the Internet of Cognition, transforming AI from isolated systems into orchestrated superintelligence.
By creating an open, interoperable infrastructure, Outshift is enabling agents and humans to share intent, context, and reasoning.
The cognitive evolution for agents is here, so go explore the Internet of Cognition at outshift.com.
That's outshift.com.
But let's do a quick preview of the episode.
We do have some major stories we'll be getting to.
We did record last week on a Friday.
That's part of the reason that today on Wednesday, June 17th, there is not...
quite as much, but we recorded last week just before the big news, the other big news of that week.
So first Fable 5 came out and then Fable 5 was taken away from everyone because of some actions by the US government.
And we will be talking about that.
Of course, there was the SpaceX IPO last week, which is kind of a big deal if you still count SpaceX as a Frontier AI lab, which...
I guess we should.
You mean SpaceX and Cursor now?
Because that also happened?
That's right.
So we'll be talking about that.
We actually have quite a few projects and open source stories I think are quite neat.
So we'll spend a decent amount of time there, which we haven't in a bit.
And, you know, a set of nice policy safety and I think a couple of interesting research papers.
So it's going to be a well...
rounded out episode, just not quite as dense as we usually have.
So kicking off with tools and apps, and we begin here with a big news from last week, the other big news last week about Fable 5, which is it was taken offline.
The headline here is Anthropic Cuts Off Fable 5 and Mythos 5 Access Following Government Order.
So here's what happened.
On Friday, I believe it was around midday, closer to evening, Anthropic posted an announcement that it is taking away Fable 5 access from everyone as a result of the US government putting, I believe it was export control on these models and ordering Anthropic to not allow any foreign national to have access to them, including...
people inside the United States, which is impossible to comply with in any kind of measured way.
There are anthropic employees that would not have access to Fable 5 as a result of this order.
The stated reason for why the government did this was that they say there was a jailbreak found for Fable 5, so you could use it in an unsafe way.
You could use it to potentially hack stuff, stuff like that.
Anthropic responded by saying, well, we'll be taking it offline.
They also said that they spent months preparing for this, working with US government.
We have discussed last week how having the safeguards in place was a big part of Fable 5.
And in fact, Fable 5 got a lot of flack for being overly safety aware and refusing to do a whole bunch of stuff.
So the claim by the government is that you could get around those safeguards.
And Anthropoc's position was that as far as they could tell, this was like a single, very directed jailbreak, and you could get any other model out there to do the same thing.
It doesn't actually pose much of a danger.
I think that's the recap of the situation, and I'm sure we have more to say on this beyond that.
Yeah, I mean, one of the funny, very specific consequences of this was, of course, that Andre Carpathy cannot presumably use the Mythos 5 model himself, which is pretty wild.
He just famously joined Anthropic a couple weeks ago.
So there's a lot of mess here.
And quite frankly, I don't think anyone's position on this is entirely logically consistent.
I think we're finally starting to get to that point that...
And we've been talking about on the podcast for years that something as insane as the US government telling a frontier lab they cannot deploy full stop one of their leading models that they trained and developed at the cost of billions of dollars would happen.
So here we are, and this is what it looks like.
Is it a principled halting of this?
It actually doesn't seem that way.
There's so much noise around this.
We've got, by the way, everything from David Sachs coming out, the former White House AI czar who's still kind of around, let's say, it was this like emeritus character floating around.
He shared his timeline of what happened.
He said they publicly released Mythos.
It's basically like Fable is Mythos without guardrails.
So they publicly released really Fable, but he's calling it Mythos.
He's saying if those guardrails fail, then you've exposed Mythos and its advanced cyber capabilities to people who shouldn't have them.
Okay, no problem.
That's true.
And that's certainly consistent with what Anthropic had been saying as well.
The issue is, as you said, it seems like Anthropic and the White House had previously agreed to the set of protocols that they'd come up with to release Fable 5.
And now the White House suddenly is saying, oh, actually, that's not good enough, which seems to be coming on the back of Andy Jassy, who's Amazon CEO, reaching out to the White House and saying, hey, you know, I was playing around with Mythos 5, and we found these jailbreaks.
And then the White House seemingly went, oh, my God, they're jailbreaks.
They're jailbreaks.
When there have been jailbreaks for literally every other model.
And I'm pretty sure by the way that we don't know the exact specifics of this.
It might've been some engineers within Amazon who found this and the CEO was like, told these things.
We don't know even the jailbreak specifics.
So on Thoughtbook tells us that like in a specific code base, they got Fable 5 to do a specific thing.
And the key point is that this is not a universal jailbreak.
This doesn't mean you can use Fable 5 for whatever you want with regards to Cybers.
Just like in this one situation, you got this model to do this one thing, which again, from what I've been able to see, doesn't even seem like that big a deal.
So there was a lot of discussion, as you said, with some people sort of trying to be...
taking the government's position and then saying that this is not the DOJ thing.
Actually, you said David Sacks was like, this is not about the DOJ or the relationship of anthropic and the White House.
What do you mean DOJ at all?
Like the government of war?
Oh, God.
DOD, sorry, not DOJ.
But the executive.
This is another thing we're saying.
This is not sort of based on any real framework I'm aware of of safety from the executive in the sense that- There's no legislation.
There was just an executive order, which, to my knowledge, didn't enumerate conditions for expert controls or...
Like, categorically the opposite, right?
As we covered a few weeks back, the executive order explicitly says that it is voluntary.
And there's a freaking passage in it that says, nothing in this executive order is to blah, blah, blah, something mandatory.
Don't worry about it.
It's voluntary.
David Sachs is the guy.
who forced that in there.
And he is the guy who now is saying, well, back to his thing, he's saying that the admin asked Dario to fix the jailbreak or redeploy the model.
Dario refused.
Now, this is all highly contested.
We've heard accounts that supposedly Dario was at a wellness retreat.
That doesn't seem to have been the case.
So there's all this like attempted mudslinging that's happening here.
So far, my sense is most of the...
The mud has been coming from the admin side where there are claims that seem to be pretty verifiably false that we've heard from them.
The challenge that we face is I have yet to hear an argument that says that we can ever prevent jailbreaks from frontier models full stop.
In fact, there are really compelling arguments that say that that is a theoretical impossibility or certainly at the very least an impracticability.
So we will have jailbreaks for every generation of model, or at least if you're not planning for that being the default trajectory.
Prepare to be surprised.
If that's the case, then clearly we're headed for a world where the moment you release a model, within a few days, freaking Pliny the Prompter comes out and says, hey, new jailbreak.
Here's how it looks.
And then everybody can use it.
If we live in that world, then the question is, what does serious AI policy look like?
It's going to look like preventing AI model companies from simply making their models available full stop.
This kind of thing.
will keep happening until morale improves.
Now, the problem is it's being done in a completely unprincipled way and in a way which, to your point, Andre, given the history with the Department of War, can only lead a reasonable, rational observer to suspect at least that there is some kind of animus that's motivating this.
And the admin's comments have not helped.
They've talked about weird vibe-based stuff where like...
Anthropik not showing the proper respect to the US government is somehow a factor in this consideration, which seems clinically insane.
And not clinically insane, by the way, for the kind of ground truth of it.
Like, it may be okay.
Maybe theoretically, Anthropik showing due deference to the US government should be a consideration.
You don't say that because- For legal reasons, your lawyer is probably telling you to put down your freaking phone.
So this kind of becomes a recurring issue with this administration over and over.
We see, you know, the Department of War, whether Hegseth, whether his undersecretaries, all these folks who just pick up phones and start tweeting stuff.
You're like, dude, like, I'm not a lawyer, but I've been.
I've been spoken to by lawyers like often enough and told to shut the fuck up.
Like, so this is really, really basic stuff.
Some unfortunate mudsling, again, that seems to be verifiably off.
Now, separate question as to there are AI safety people right now doing kind of victory laps about, oh, we have a de facto licensing regime now, which we actually kind of do.
The U.S.
government has come in and prevented a frontier lab from deploying the models.
We live, whether you like it or not, in the world that was predicted by some of the kind of most prescient AI safety people.
Which, by the way, it's like the worst licensing regime possible.
The previous actions by the administration doing this, essentially imposing export controls in the context of the government not being able to use these models, they lost that in courts.
That didn't stick.
And this won't stick.
This is absurd.
I'll go further than you.
Like anyone who believes this is about jailbreaks is completely naive.
Like this is not about jailbreaks.
This is about the relationship of anthropic and the government.
Anthropic and in particular the leader of Anthropic, Dario, not being willing to take the knee and basically show deference and do what the government wants.
He didn't refuse.
Like there's no...
chance in this world that he just refused to do what the government wants.
What likely happened was he was like, well, give us some details.
We would like to do some investigation, blah, blah, blah, blah.
He was like, I want to negotiate this a little bit.
I want to discuss what's going on instead of doing what you're asking.
I mean, here's a completely like...
plausible conversation, right?
More likely, I would guess here, given the personality involved, Lutnik or David Sachs.
It could have been like another department, not necessarily the presidential cabinet.
I think it might have been the cabinet of commerce, maybe.
It's not very clear.
None of this is very clear.
Yeah, exactly.
Right.
But you can imagine them going to Dario and be like, hey, look, Andy Jassy just told us about this jailbreak.
You yourself have been telling us for a long time that this is basically an AI super weapon wrapped in a thin layer of security coding.
And what they've shown is that you can actually penetrate the security coding.
So you got to fix this.
And then Dario goes, I cannot break the laws of physics, Captain.
And like, basically, I like there's no way.
I'm not going to lie to you and tell you that I can fix all jail breaks.
If I fix this one, then the next one, then the next one.
We're always going to be doing this.
So I guess we're just at the point where we got to figure out how to manage these two things that can't be reconciled.
And then I guess the admin went, okay, I've got an idea.
How about we blow it up?
We're going to take the boat.
We're going to blow.
And that's basically what happened.
And so here's the thing.
You can do a victory dance on this if you're an AI safety person, but be aware that We have not tested the foundation of principle on which this entire edifice rests until we see OpenAI develop a mythos class model.
And will the admin do the same to them?
Which they won't.
Like OpenAI.
I don't know.
I'm ready to be proven wrong here.
And I hope I am.
But I also partly don't hope that I am because what I would want to see here is the conditions under which...
things can continue.
So one of the best things you could do with a situation like this is have clear incentives for loss of control research, for jailbreak research, like all the good things that are needed for super alignment.
You could incentivize that massively.
Guess what?
Super alignment just became, or at least jailbreak resistance, just became the single most important bottlenecking factor for frontier AI deployments around planet Earth.
Right.
That's the story.
Great.
That means you should see massive investment of resources pointed in that direction and a liability and licensing regime that comes with it.
Three years ago, we built one, by the way.
That was the State Department Commission report that we put out.
Like we pre-thought this through a lot.
There are a whole bunch of crappy things with that and mistakes that we made, blah, blah, blah.
But like you're now firmly in the territory of having that conversation, which at the time was way outside the Overton window.
It's just a shame that this is not Congress doing this.
This is the executive in a context where it can be doubted.
And then the next administration that comes in is going to say, oh, this is all political.
Let's steer hard in the other direction.
We have not yet seen what actual licensing framework is going to come out of this.
And that's part of the problem is it's so chaotic.
You need to see all the frontier labs come together with the administration in a principled way.
Talk about.
Is it a compute threshold?
What are the evals?
At the same time, we're seeing Casey get completely shafted, basically.
The one arm of the government that was charged with doing kind of robust evals and it has that ecosystem is no longer being tapped in the same way to get model evals done.
So this is a really unfortunate- And I'll go further.
This, as you say, is unfortunate for safety in multiple ways.
So for one, it doesn't necessarily encourage kind of principled- approaches to safety doesn't show any sort of capability by the government to actually evaluate safety in a principled way.
And it's worth covering.
I think the other sort of news or the depth of what happened was, of course, there was the event itself and the politics around it and anthropic, anthropics relation to the government.
But then the overall like AI community and its response, I think was interesting.
Yes.
Yeah.
First, you know, there were a lot of funny memes, which is always fun.
So that was great.
But there was, I think, a lot of sort of, you know, having fun at Anthropic's expense or not even that.
It's a lot of like angst at like, oh, Anthropic was saying it's so unsafe and these are like nuclear weapons.
And guess what?
If you keep saying that, this is what will happen, right?
Anthropic brought it on themselves, which.
would mean that people, like actually frontier labs, are discouraged from discussing safety and from positioning these things as dangerous things.
Now, again, saying that this is because of Anthropic doing all their discussion of safety and so on and the position papers and all that, I don't think is correct at all.
But regardless, the perception seems to be that Anthropic brought it on themselves by discussing safety and by...
positioning these things as dangerous.
And again, jumping on the story of mythos was a lot of PR, Anthropic was kind of doing BS here, and this is their fight.
It's like the same people who are saying mythos is not as powerful as it was, who are now saying, ah, Anthropic, that's what you get for talking a big game about how this is a weapon of mass destruction.
And the problem is those people have been proven wrong.
Like, trivially.
Like, that debate is over.
I could fax it in.
Yeah, exactly.
Like, we've got the vulnerability list.
The critical vulnerabilities on load-bearing pillars of the internet's, like, most critical infrastructure.
It's game over.
Anybody at this point who's claiming mythos was not a real thing, who was claiming that it was predominantly a PR exercise, including Sam Altman himself, is full of shit.
Like, trivially.
Like, it is now known.
Now the question is, okay, what do those people pivot to when that argument is taken away and this new situation happens?
guess what?
They pivot to, okay, like let's dance on Anthropics Grave, which like weird, weird pivot, but okay.
Yeah, there's so much to the story and I guess we'll have to move on.
But like just one last thought here is on the hypocrisy of the US-China thing.
What was David Sachs arguing like 20 minutes ago?
David Sachs was arguing that we cannot hobble US innovation by introducing binding regulations and blah, blah, blah, because China.
And like, here's the thing, people who watch the podcast, you know, I'm a China hawk on this stuff.
I'm also an alignment hawk.
I think we actually have to walk and chew gum at the same time here.
As a China hawk, I hear David Sachs make that kind of argument.
I'm like, okay, yeah, cool.
That sounds good.
That makes sense.
Okay.
Now you're going to have to weight that eventually against alignment and loss of control risk, right?
That part never seems to happen with David Sachs.
That's okay.
Now we seem to have suddenly though a massive pivot.
to let's justify the like outright halting of the deployment schedule of a frontier lab that has coordinated with the White House by everyone's accounts already to set up this moment on the basis of some technical triviality that if David Sachs himself did not see this coming, that just means he's not watching the ball.
Like he doesn't understand AI in a very basic way because everyone who understands AI in a basic way knows that jailbreaks are a thing and are not going away.
Like you should have, if it's a surprise to David Sachs that jailbreaks exist for this model, then like he has no business occupying the position that he does in the White House or the position of influence.
Like I just don't see how you make sane policy misunderstanding such a fundamental basic thing about the technology.
Anyway, this is like very ranty on my part.
I apologize.
It just seems like there's just, there's so much crap smeared on the walls on this one that, man.
Yeah, I'm sure we'll have follow-up stories.
As of now, there's some like informal reporting that there are conversations going on and maybe Fable 5 will be back online in a few days and maybe not.
I just want to reiterate, this is not about jailbreaks really.
That's sort of a cover story.
Even if it is about jailbreaks, it's about.
from what we've been told, a very kind of specific minor case of a jailbreak in a particular code base for a particular task.
It's not a universal jailbreak.
I think there is a good argument to be made that with Fable 5, Anthropic just threw so much onto the wall in terms of safety that if you were to try to do a deploy, like a prolonged operation, a cybercrime initiative, whatever, they have this sort of architecture in place.
To catch you, or at least...
Hopefully.
Yeah.
To the extent that we have models gated behind APIs.
That's right.
We have the ability to do safety.
Once you use open weights, anything is...
There's no alignment.
There's no safety.
But with Anthropic, there is a potential for safety.
And this is another dimension which is unfortunate.
They invested on safety more than anyone.
They got pushed back.
And now this happens, right?
Well, this is the thing.
You can argue that you don't like Dario or whatever and that you think what they're doing is dangerous or whatever.
Like, you can do that all day long.
You're going to sound more and more like a freaking effective altruist, by the way, if you start to do that, which is what David Sachs is starting to sound like.
And I don't think he would like that comparison.
But just in terms of the direction, like, it's hard not to interpret it that way.
Here's the thing.
I'm willing to be proven wrong when there's something that comes out next week inevitably and there is some crazy level of nuance to this that we didn't hear about.
But just based on what's in the news at the moment, it seems to resolve to me as like one of two things.
You get to think that either, okay, the jailbreak is something to worry about.
If it's something to worry about, then I can only chalk this up to gross incompetence by David Sachs and anybody in the admin who touched this issue.
like who is responsible for the admin's ultimate position.
Because all that means is the admin institutionally does not understand that jailbreaks are not going away.
Like if you don't understand that, then you don't understand enough to regulate the technology.
And so like basically, yeah, pick your poison.
Either yes, this was a legitimate place to intervene and the admin did so wildly incompetently or the jailbreak was no big deal and the admin overextended.
It's kind of one or the other.
Anyway, it just seems like it's so technically unmoored.
There's no adults in the room kind of looking at this, being sober-minded.
What would a principled approach to this be that applies to everybody equally?
We're just not getting that.
And in the context of upcoming IPOs, I really have to wonder, it's now about anthropic doing what the government wants or having a good relationship with the government.
happened again, right?
It happened with the DOD, DOW.
Now it's happened with Fable 5, not directly with the DOW, just generally with the executive department.
And if Unpropik keeps pushing back, which by all accounts seems like, you know, they want to be like, no, we like are realistic, rational, like we want to do things based on truth.
You know, the situation isn't great for Unpropik, at least.
Looking forward to me.
Yeah.
Next up, you've got a story that we won't need to be discussing for 20 minutes.
Just a fairly minor tool slash app update.
Facebook has released a new AI mode search feature, very much similar to AI mode within Google.
You enter a query of some kind and...
an LLM response driven by their in-house model, Spark.
MuseSpark gives you a little summary of discussions and conversations, pulls up some images and reels from across the meta ecosystem.
That's about it.
There's now this meta AI, you know, AI mode thing to do search that it looks like meta is like, oh, we're rolling out AI.
This investment is worth it.
That's what this reads like to me.
Yeah, kind of like the Google previews and pulling from Reddit, right?
That they do a sort of a similar idea.
You're pulling from conversations that people are having about a topic.
And yeah, why not?
Well, why not?
There are a million privacy reasons, I'm sure, that people can cite and have cited.
But from a technical standpoint, it seems the business case is pretty solid.
All righty.
And that's it actually for tools and apps.
Now our major announcements.
So we are moving right along to applications and business.
And the first story is SpaceX.
So two things happened.
One last week, one this week.
Last week on Friday, again, just after we recorded, or I guess during the entire day as we were recording, SpaceX had begun their IPO.
So they have gone public.
meaning that they're listing shares on the public markets for anyone to buy, anyone asterisk.
There were some IPOs a little weird.
You kind of set aside some shares for different people.
But they went public and the IPO went well.
So first of all, they're trading at high valuations above what they went public at.
They are now insanely valuable as a company.
They went above.
two trillion dollars they listed at the ipo at a valuation of 1.75 trillion i believe so the ipo went well they now are able to sell shares on the public markets to generate cash for various reasons then this week we got the news that spacex is going to be acquiring the ai coding startup cursor for 60 billion dollars 60 billion dollars Not in cash, but in presumably SpaceX shares and maybe some cash, some mix of the two.
We have discussed in recent months that SpaceX had already announced this deal with Cursor where they're going to partner up and they have the right to buy it for $60 billion and they might or might not do it.
It seemed like everyone knew that they are now going to do it after the IPO.
And in fact, they did do it.
Pretty much that situation, there are some interesting aspects here to discuss on like why did SpaceX and really XAI within SpaceX need to acquire a cursor?
I think the obvious thing is like XAI kind of died a little bit.
Everyone left.
It was a miserable place.
Grok was not like at the cutting edge despite them having some early signals of being able to compete as a frontier.
They did not focus on coding.
at all, it seems, and are way behind on that front.
So in acquiring Cursor, they are in a better competitive position to do coding in the sense that Cursor has at least some experienced post-training models.
They have data on users of Cursor, and they do have some revenue to speak of, which, by the way, SpaceX, you know, if you look at the numbers, they're not exact.
That's what I hear.
I mean, it's basically...
So the SpaceX shares did actually gain seemingly on these news.
And I think it's kind of beneficial to both really parties here.
Cursor gets access to unlimited capital and the compute that XAI has built up over the last few years.
And XAI gets actual talent to do stuff with all the compute.
Yeah, I think when you think about what these, this was the match that always had to be made.
If you're looking at Cursor, their market share has actually dropped quite precipitously from 41%.
If you look back to June 2025 to about 26% in May.
So in about 12 months, you're seeing the market share drop by, I don't know what that is, about 40%, right?
That's a big problem.
Oh, and I forgot to mention, anyone who doesn't know what Cursor is, I guess it's worth mentioning.
Cursor is primarily a provider of an integrated development environment for coders.
So it's the program of which you want to write code.
They were the leading kind of coding environment with AI in around 2024.
Everyone was using Cursor if you're using AI.
Then Cloud Code and so on came out in 2025, 2026, and everyone was like, well, a lot of people were like, I don't really use Cursor anymore.
I just use Cloud Code.
It just kind of quietly rolled off their subscriptions.
And I think that's sort of the death knell that companies like Cursor, I want to say really at this point, there are different escape velocities and moments where you just have enough, whether it's infrastructure or kind of talent or whatever, where the market just allows you to succeed enough to then get acquired in this space.
And this was a match made in heaven.
I mean, XAI, obviously, or SpaceX.
has the ability to build data centers like no one else, just at an outrageous speed.
And they have all kinds of supply chain advantages with, thanks to Tesla, getting their hands on batteries, getting their hands on generators.
There's all kinds of advantages on the infrastructure side.
And it's Elon, he builds stuff fast that you shouldn't be able to build fast.
Cursor has always had this problem.
In fact, was enmeshed in scandal as a function of this problem when it came out that they were actually never training their own.
models, never pre-training their own models.
They were only working on top of open source models and then doing a lot of fine tuning, a lot of post-training, but not training from scratch.
And so if your cursor, I mean, the requirement for pre-training, we've only seen that go up, especially as Anthropic has pushed the frontier of pre-training almost more than any other lab.
It's become more and more clear that pre-training is an essential part of the game.
And, you know, as you say, XAI was sitting there with like a just...
gutted bench and you need talent.
So what this suggests, we've known that Cursor was using Colossus 2, the big mega cluster that Elon just built recently for training runs.
And that had us last time look at this and go, okay, there's probably going to just be an acquisition here.
Because when you're letting these guys just run your most critical gigawatt scale infrastructure for an extended period of time, that's, I mean...
That's what value is denominated in the space.
That's the most sacred resource SpaceX has.
So if they're not using it for XAI, they're just handing it over to Cursor, that's a test to see, can Cursor produce at pre-training at scale?
The fact this acquisition went through strongly suggests that that training run worked out.
Something about it suggests to SpaceX or XAI, what do you want to call them, that this is at least better.
than the position they were in before and worth the $60 billion because they did have the option not to make the acquisition, right?
It was $60 billion or I forget what convoluted, you know, like payout was going to be involved.
But, you know, it's a positive sign for that training run.
Yeah.
So I think it's likely that this will lead to more Grok updates, but now XAI will be able to actually...
deploy their capital instead of just leasing out and being a NeoCloud.
Now, whether it will basically stay a NeoCloud is not clear.
I think at this point, I guess the reason to need pre-training, actually, it's a little bit nuanced, I think, where for a lot of stuff, post-training is sufficient given the very, very strong open source models coming out of China with Kibni K2.6, DeepSeek, et cetera.
In fact, Cursor, I think...
proved the case that post-training can get you very far.
But if you want to compete on the Fable 5, Mythos 5 level, that's where you need next generation pre-training.
And it's really competing.
All these companies, I think to some extent, are competing to get to AGI because there's a belief that whoever gets to AGI first kind of will own the economy.
No one really knows, but it's like, we need to get to AGI first because winner take all.
That's also why everyone is competing to get to recursive self-improvement.
So with this acquisition, XAI can at least be in the race given that they just were almost like a dead company kind of.
Yep.
Their best play would have been to turn into the infrastructure layer for super intelligence.
It's never clear to me what that even is supposed to mean.
Like if superintelligence works as advertised, whoever owns it ends up figuring out a way, give them a little bit of time and they'll figure out a way to replace you as the infrastructure layer for superintelligence.
I mean, if it genuinely, like if it's what it's supposed to be, like doesn't mean it will be, but if the thesis bears out, like you're...
chip design, you're like finding clever ways to access power, like all the things.
Yeah.
I mean, I guess you get unlimited money and you can do whatever you want with money, including building data centers.
So, yeah.
And speaking of data centers, next up, you've got a story on Anthropik.
Anthropik pursues data center leases and is seeking financial backing from...
Google.
So they are now pursuing their first direct data center leases in the US, having already signed more than a dozen preliminary agreements for facilities with a combined capacity exceeding one gigawatt.
So for years now, they've been securing computing capacity by renting chip servers from AWS and Google.
And here they are looking to actually own the data center's lease.
and manage your facilities with Google helping to underwrite release obligations.
So it seems to be kind of actually moving fast.
It sounds like Anthropic wants to have this starting to come into effect this year.
And I think it, again, tracks with Anthropic not having enough compute.
Like as of their kind of takeoff this year, they've had a lot of struggles just...
meeting demand.
They've had a lot of weird things where they adjusted their limits and a ton of instability.
On Prophec, their API is very, very far from the sort of stability you want to see for a web company and for a provider of essential models, really, for many businesses.
So the move makes a lot of sense.
I guess the question is, are we going to be able to actually pull this off well?
Yeah.
And so the calculation here, if you're Anthropic, is you can do what you've been doing, which is go to NeoClouds and Hyperscalers and basically like rent out access to the capacity that they build, right?
So you'll have NeoClouds, like FluidSack, they'll come in, they'll work with a local builder, and the builder will build the data center and then FluidSack will make sure that all the server racks are there, wired up properly and all that.
And then Anthropic just comes in and pays.
a premium, a good margin to these neoclouds, part of which gets passed on to the builders to rent this capacity.
And so number one, as we know, Frontier AI is a margin play.
The margin really, really matters to determine profitability.
And that's the reason that if you're anthropic, you might say, hey, why don't we just do this all in-house?
Another reason that's not mentioned here, though, is often, I mean, it's just like you have catastrophic incompetence by the builders.
This is something that's not widely sort of like understood, though it has been publicly reported, which is why I can share it.
But like you'll have builders who, you know, like with wild delays, like they'll come in and promise the moon and the stars.
And then it turns out things take four months, four months longer than expected.
And Anthropic will be betting here most likely that, hey, you know, we can just do better.
If you bring the level of caliber that they have on the compute side, on the AI research side, and if you had that same caliber of talent on the construction side, all coordinated under one roof, you probably could have better kind of cohered infrastructure stacks and building times.
And so that's another kind of implied part of this calculus.
The margins are not small and the costs are not small.
So if you're anthropic, you now got to spend the capex.
You got to find a way to fund this massive build and this risk that you're taking on.
Right before it was rent the facilities.
We don't have to spend as much money up front.
And that's a big de-risker.
Now it's like we're going to own the whole thing.
If you're going to do that, you're going to need to take out loans.
The problem with taking out loans is you're anthropic.
You might be a trillion dollar company, but your credit is not good enough for most lenders because you haven't been around for a long time.
And so what do you do?
How do you get a loan?
Well, you have to partner with an entity that is big and massive and old like Google.
And that's what they do.
So they basically piggyback off of Google's credit.
and credit rating and kind of credit relationships to fund this activity.
And OpenAI is in a similar boat.
They're also too new to have that sort of.
So you see these weird partnerships where it's like, why does Anthropic need, if they're going to do it themselves, can't they just go to the bank and get a loan?
Well, no, the banks will not loan to a company this risky, this young.
I won't say the small because again, trillion dollar company, it's like bigger than what Google was not too long ago.
So anyway, there you have it.
It's all these entanglements and it's part of, you know, you'll hear all the arguments about circular funding of this.
And yes, it does increase the strategic dependency between Google and Anthropic.
That is true.
But yeah, it's a margin play and they're betting that they'll keep, that the risk isn't that high because they're going to need the infrastructure.
Next, some stories on OpenAI.
The first one is leaked financial documents show OpenAI is losing billions of dollars a year.
Looks like financial documents disclosing a lot of details as to the specifics of opening eyes expenses and revenue have indeed leaked and been verified to a significant extent.
So we see some interesting numbers here.
We know where revenue grew from $3.7 billion in 2024 to $13-ish billion in 2025.
But their expenses, R&D expenses, also went up.
like a ton.
They went from $7.8 billion in 2024 to $19-ish billion in 2025.
And in general, their spending is outpacing their revenue and is likely to continue to do that.
They had total operating losses going from $9-ish billion in 2024 to $20 billion in 2025.
And also their other expenses grew significantly as well.
It wasn't just the R&D.
They are spending way more on marketing.
They went up from like one-ish billion to five-ish billion for sales and marketing, which is quite a lot.
Like their general and administrative cost is now some absurd number as well.
So nothing here that seems overly surprising.
It kind of tracks with the rough picture of OpenAI.
They have a lot of revenue.
but you're also spending a ton on being in Frontier and just generally as a growing business, but maybe notable to have these details out in public.
For sure.
Yeah.
It's also, so the guy who this was leaked to is a guy called Ed Zitron.
So he's been one of the sort of, I don't know what it's called, like the Gary Marcus of the economics of AI, if you will, like a real skeptic on will the labs actually be profitable and is this all a bubble basically.
I think that- So unfortunately for his position here, which seems to be, oh, look at this.
This proves what I've been saying all along.
This kind of seems to, I won't say prove the opposite, but it's evidence, pretty compelling evidence that points in the exact opposite direction as far as I can tell reading the numbers from what Zitron has been saying this whole time.
So here's specifically what I mean.
We're going to focus on a couple of numbers.
There are a lot of numbers cited here, and you could put asterisks everywhere, which is why I'm not going to make any definitive statements.
There's just evidence pointing in a certain direction.
So OpenAI has gross margins, meaning the difference between the revenue that they bring in and then the cost of earning that money.
That's kind of the core underlying business metric.
This does not include things like AI R&D.
It doesn't include kind of the cost of legal restructuring of OpenAI, which was a massive cost.
They were all those ancillary things.
Granted, does not include those.
This is just tokens come in and like, do you make marginal dollars on marginal inference?
That's the question, right?
And nominally, if the answer is yes, all you need to do is scale that business.
You just need to make that business bigger and then you will make more money as long as the margins continue to be high as you scale.
With that in mind, what are opening eyes gross margins?
Well, in 2024, their cost of serving customers was about 2.7 billion and their revenue was about 3.7 billion.
So positive margins, in fact, 28% gross margins.
That's good.
So for context in the world of SaaS, like software as a service, like classically, you'd see margins from like 70 to 80% or something like that, right?
So like really, really high margins here, you're looking at 28%.
It is more of a utility.
There is a lot of asterisks to go around here, but it's a scalable business with nice margins.
Last year, though, the cost of revenue was $7.5 billion.
The actual revenue more than tripled of $13 billion, which makes 43% gross margin.
We're starting to get pretty damn good margins.
The margins went up with scale and with time.
So these two things taken together, I mean, this does just start to look like a business that like, yeah, like why don't we just like, click and drag and like grow this business and we make more money.
That's very consistent with the arguments we've heard Sam Maltman make.
And by the way, there are indications that opening eyes margins are actually worse than Anthropics and potentially by quite a bit.
So it's not, this is not even the number one player in the race and they're killing it with almost 50% gross margins.
I mean, we got to see it.
And there's all kinds of arguments for why Sam might not want IPO, including the scrutiny behind these numbers and so on and so forth.
But if the top line story holds, this is a business.
Like, this is a business.
And again, the challenge people run into is they'll look at like, okay, yeah, but how much did OpenAI spend in total this year?
And that ignores the fact that that spending is meant to fund.
revenue that comes in 18 months from now.
We've talked about this so much, but like the dollar spent today on the data center of 18 months from now only leads to revenue then, right?
And so you have to do time discounted money to get to that point.
Yes, but if Anthropic is profitable as they were this quarter, that's bad.
It means that they underspent on scaling.
And if you rewind the clock back to see again, the key question is, what were guys like Ed Zitron saying 18 months ago about Anthropic's buildout?
And if they were saying, oh, they're burning money, this is a blah, blah, blah, blah, then as a matter of simple, logical fact, that call was incorrect.
So the only question is, does that continue?
So anyway, yeah, there you have it.
And another story related to OpenAI chat, GPT's market share slips below 50% for the first time.
This is according to reporting by Sensor Tower.
This is showing now that ChatGPT as the chatbot has dropped below 50% market share, falling to 46.4% by the end of May, 2026.
Gemini now holds 27.7% market share and Claude holds 10.3%.
So OpenAI is still in the lead by a good margin in terms of the number of users using ChatGPT as their chatbot of choice.
but their position is eroding and seems likely to potentially continue to erode in the sense that Gemini can afford to be cheaper and Anthropic can afford to be more expensive in the sense of arguably providing the better product for enterprise and there's professional productive use cases, broadly speaking.
So yeah, if you're looking at...
stories related to IPO and how much you should think this is bad for OpenAI.
Actually, the financial story is like, oh, okay, they're spending a lot, they're losing some money, whatever.
It's nothing new.
This story of they're losing market share, much bigger deal.
And I think not surprising, but it'll be interesting to see if this does continue and if OpenAI is able to reverse the trend.
Yeah, it's also like, it's always complicated when you look at a market share percentage in a growing market, right?
So OpenAI's user base has continued to grow fastest, you know, apt to reach a billion users.
They're now starting to feel the edge effects of like the global population.
So, you know, at that point, you ought to expect erosion of that top line metric.
What this means is the entire space is growing.
faster than OpenAI can just absorb the traffic.
And yes, there are other competitors nipping at their heels.
It doesn't necessarily mean bad news for OpenAI in and of itself, but there are some pretty interesting warning shots for them.
So if you look at Quad, they have by far the highest conversion rate to paid usership in the field.
13% of Anthropic users pay for Quad subscription.
So when you look at those low numbers of general usage for Quad, 10%, not surprising.
It's been a thing for a while.
They've been targeting enterprise.
At the same time, OpenAI is rotating more into being an enterprise-focused company too.
And that would come as well with a reduction in just general market share and attention share in the general population.
You kind of expect that.
In that sense, I think it's a story of intentional change from OpenAI to rotate into enterprise.
It's also a story of the limits of the market itself.
But as they rotate into enterprise...
13% of Anthropics users paying for a subscription, that's a real headwind for OpenAI, right?
That means that people who build stuff, because that's those 13%, are choosing Anthropic.
And until that changes, the high margin enterprise revenue is much harder to access for OpenAI.
And I think a general story of OpenAI is that...
They've been trying to do too many things for a while now.
They have explicitly backtracked some things like Sora and their various initiatives.
And this is trying to fight on both fronts of consumer and enterprise kind of is the worst of all worlds.
Because on the one hand, Anthropic is competing with you on enterprise.
On the other hand, Google seemingly, as far as I've been able to tell, is still focusing on consumer.
and getting more users to use Gemini.
And so OpenAI has to fight on two fronts.
On both fronts, they have very strong competition.
And if you try to dilute your focus as OpenAI has done, your ability to fight effectively is kind of limited.
But OpenAI is still in the lead on consumers, still probably has the most recognizable brand as far as chatbots.
And I'm sure if they keep losing market share, they'll keep having a massive amount of a market.
Yeah.
And I think Google's enterprise play here is also just like the bundling of Gemini into the Google ecosystem.
So in that sense, maybe they look a little bit more Microsofty, but yeah.
Yeah.
It's a different kind of enterprise than like Codex and Cloud Code and so on.
Yeah.
Next, a story on Meta, and I'll have to swear a little bit for this headline.
Oh, yes.
Headline is, tell him he's a piece of shit.
Meta's new AI unit is a total mess.
So this is reporting from Wired.
It's talking about Meta's applied AI team, which has been formed in March and now has around 6,500 engineers and product managers.
It was formed to support Meta Superintelligence Labs researchers.
And there's some kind of wild details coming out of this thing.
There were some interviews where insiders have characterized the work inside it as soul crushing.
Someone called the environment the gulag.
And it appears to be the case that a lot of people are just doing data labeling and very, very basic data creation.
Exactly like the gulag.
Do you remember when Stalin had all those poor Russians doing data labeling in the gulag?
Yeah.
Well, the Gulag aspect here, like the metaphor, if you want to go there, is that the way this has seemed to play out is that various Meta employees from various parts of Meta were just like randomly, surprisingly assigned to go to Applied AI and they didn't have like a choice.
They were like, okay, I'll do it or I'll quit.
And so a bunch of people have been pulled from all around the company to go to this thing.
that again was formed in March.
This is not like an established thing that is building products.
It's explicitly to help Meta work on super intelligence and help TBD labs and so on.
So I think broadly, this is a business story in the sense that morale is low.
That's very clear for Meta, like throughout the company, not just.
for the Supply AI team.
There was also over 1,600 META employees that signed a petition opposing the new initiative to monitor their clicks and keystrokes, which we discussed a month or two ago.
And META slightly scaled back the program by allowing 30-minute pauses.
So they didn't relent fully.
There have been company all hands by the leadership that they've been publicly saying, oh, we messed up.
You know, it's a rough time.
There's going to be some challenges.
Zuckerberg himself had addressed this topic.
So if you're trying to compete in general in the software world, and in particular, I think in the CI world, having low morale, having employees who aren't excited, having just like people unhappy is not good.
And the fact that the super intelligence labs...
like created this thing with 6,000 employees to do data labeling or like these kind of menial things also isn't a good sign, right?
So maybe they're just going to be able to brute force their way to strong AI as they seem to be doing here.
But the more Meta pisses off their workforce, the worse it'll get for them long-term.
Yeah.
I thought that the worst part of it was when they like threw the employees in the back of black Marias and drove them to a black site where their clothes were taken away and then they were all made to pee with like 120 people to one toilet.
I think metaphorically, that's clearly like where we're heading, right?
Like we have a gulag house.
If you just extrapolate, right?
Yeah, you get to gulag by next year.
Yeah, no, but to your point, I think feedback loops are the way up and feedback loops are the way down.
dynamic in this space.
And you're seeing it with XAI, right?
Elon's approach was quite telling.
It wasn't like, okay, well, you know, it just needs some tweaking.
Like, I mean, no, no, no.
Like your feedback loop is wrecked, which means you need to start from scratch because hysteresis is not your friend here.
Like the effect of having scaled all the way down and unwound, it's just like any good, like a market collapse, right?
Like speculation on the way up, speculation on the way down, you overcorrect.
And so he's just like, all right, we're pulling the plug, redoing.
That's what Zuck was trying to do with Alex Wang.
The hope there would have been that you could get a clean narrative.
And they tried.
They've been doing pretty well on the kind of pre-release, hey, we're building the infrastructure, but not yet the models, but eventually the models for superintelligence.
That was kind of working.
We've covered that before, but this is...
Yeah, seems starting to show a weird play.
I mean, I understand why if you have a stable of thousands and thousands of employees, that does look tempting.
It's a lot of good data, but the morale consequences, that's challenging.
I think it's worth noting too, especially on the top, top tier of talent, of researchers and the people that you pay $10 million or $20 million or whatever to.
Like Meta has been throwing money at them and that is effective to an extent.
But like Anthropic can throw money at people too and they can throw shares at people and shares is how you get rich.
So if morale is low throughout the company and it just like is sad to work there, these like super top tier people who everyone is fighting over are going to leave.
They're going to go to Google or OpenAI or Anthropic.
And if you're trying to compete in the Frontier AGI race, if you don't have this superstar team, I don't think you have a shot.
So this is not looking good for meta.
And last up, we have not a huge story, but I think a slightly interesting one that is fun to cover.
Sakana AI commercializes...
AB-MCTS in Sakana Marlin, an enterprise agent generating up to 100-page research reports.
So Sakana AI, we've covered a couple of times.
They're a company based in Japan, founded by some notable AI researchers that has published papers of various kinds about evolutionary optimization and automated research.
They have now built this B2B autonomous research.
agent positioned as a virtual CSO that can do a bunch of research for you, right?
And generate reports and slides and so on and so on, built partially on their research and on their work on the AI scientist initiative.
They have gone to closed beta in April with a couple hundred.
customers.
Now they are partnering with some bigger companies, receiving strategic investment from CD group.
They have pricing, various things like that.
So they're actually starting to commercialize, which probably is good to do as a lab that isn't big and doesn't have its own models and so on.
It does position Sakana, I think, in an interesting place where they're sort of like...
research agent as a service, as a thing you're selling, isn't a focus of anyone.
Deep research was overage in 2024.
Now coding agents is overage.
So there might be a space here to do really well.
I don't know.
It's interesting.
Yeah, their whole, and I think we've talked about the architecture that we're using behind this.
It's this thing they call adaptive branching MCTS, so like Monte Carlo Tree Search.
Basically, the idea here is, Monte Carlo Tree Search is like any given problem, you try to decompose it, decompose your solution into steps.
At every step, you have a bunch of choices as to what your next step might be.
You can kind of take one of those choices, one of those branches, and then explore it more deeply.
Or you can look at other branches, basically.
And that trade-off between...
Going deep in one branch versus going broad in the others is what this whole kind of, you can think of it as a kind of scaffold around an LLM, is designed to help them do.
And they posted some interesting kind of promising early papers about this a while ago.
And that's what they're now basically baking in at the bottom of this.
Yeah, I mean, it's interesting.
The depth is the competitive differentiator, as you said.
This means the inference runs are going to be longer.
So for a given problem.
You know, you give the problem and it'll run off and do all the research, you know, look into your target market.
It'll make slides, develop hypotheses, that sort of thing.
So kind of interesting.
You've got to wait a long time because it'll do literally hours of research and like thousands of API calls.
So pretty interesting play on are we there yet in terms of, you know, hundred page documents that might determine corporate strategy.
Is there alpha here?
I don't know.
Maybe we'll find out.
Maybe we'll see companies based on this stuff soon.
And speaking of that, next up, we have projects and open source.
Sakana actually open sourced their search library as TreeQuest, fun title.
And we have a bunch more open source and projects things to cover.
And the first one is actually more of a project.
We have OpenRouter, which is a fairly popular service for paying one bill and then routing it to various providers of LLMs.
They had announced...
this new thing called Fusion that synthesizes outputs from multiple AI models.
And then as a result, they say outperforms the individual models on complex tasks.
And you can query open router single API call to Fusion that then dispatches a prompt to a panel of models in parallel.
And then there's a judge model that synthesizes all these things into one final answer.
And they looked to things like, Draco, this is focused on research, actually, on answering questions or generating hypotheses and things like that.
So I think really not surprising that having an ensemble of models all doing their own research and then finally synthesizing them on various points, not surprising, but having it be an actual product and something that you can pay OpenRouter to do.
It's a little bit interesting and it is a sort of advantage in a sense of being the sort of provider to providers.
Anthropic isn't going to sell this to you.
OpenAI isn't going to sell this to you.
But OpenRouter is happy to be like, yeah, we'll dispatch it to all the models and then combine it all into one thing.
Yeah, and we've seen papers and it's been a known thing for a long time that ensembling models together does give you a better quality output.
The trade-off has always been historically.
the added cost, right?
You're just doing best of end sampling basically.
And that's end times normally the cost of doing it once.
Here, all their optimizations actually allow them to drive down the cost quite significantly just by taking advantage of which models are better at different tasks.
There's two different elements going on here.
So one is just like the lift and capability that comes just from this from, from, from synthesis, from having a bunch of different models, like kind of best of N.
So they, they measure this by taking like Opus 4.8 and then they pair it with itself to, to, to make kind of like this, well, this, this pair of models and they find that it outperforms solo Opus 4.8 by, you know, like.
six to seven percentage points on a set of benchmarks.
The second piece though, is the diversity value.
What do you get from swapping one of those identical models for a genuinely different model?
And so in the case of GPT 5.5, when they swap out, when they have Opus 4.8 with GPT 5.5, they get an additional boost in this case of like around two, 2.1%.
So you actually do see.
like a very measurable.
And that's, I think the big take home here is the ability to actually measure how much comes from the diversity versus how much comes from just the best event effect.
And in both cases, it's non-trivial.
I mean, two percentage points on like software engineering benchmarks is non-trivial, especially when you're already scoring, you know, in the like high 60s, let's say.
Next up, open source, Moonshot AI has released Kimi, Kate.
27-code.
This is a specifically coding model and it is doing like way better on the Kimi code bench over Kimi K 2.6, which is the previously most released model releasing under a modified MIT license.
So mostly anyone can use it for anything.
Once again, a mixture of experts architecture with 1 trillion total parameters and 32 billion active per token.
Yeah.
Kimi continues to kind of kill it.
It's still not at the level of GBP 5.5 and cloud Opus 4.8, but even the fact that it's like, we've been striking distance, like kind of close.
The vibe check I've seen from people using the most recent generation of these open source coding models is that they are quite usable for a lot of tasks.
You can use them.
in place of cloud often and they are fast and they are cheap so there's a real sort of you know use case here and in fact i can say as someone building a lot of applications the cost and speed of these open source options especially if they keep moving in that direction will potentially start eating at the revenue of open ai and anthropic yeah that's true there's always this question of like how Infinite is the demand for increasing intelligence and is riding the frontier of the highest end of intelligence enough.
And I mean, I think pretty clearly it will be for at least one company, if only because you start to eventually become a hedge fund.
And that's actually the direction that is a parenthetical, but this is like the kind of thing that the US government is pushing Anthropic to do at least is become a hedge fund.
Like when you start to say, no, you can't deploy your model to the world because it's too dangerous, but Anthropic is sitting on the world's best models.
There are companies that are really good at using the world's best models to make more money.
They're called hedge funds.
Hedge funds work in the dark.
You just need to think of Medallion, ReninTech, Jane Street, these folks, they're advertising on Dwarkesh showing off how good their clusters are, right?
Like these are really serious players.
It's just that Anthropic is even better than them at building models.
And so like, I think eventually you are always going to see a rotation into, frontier model providers becoming hedge funds for a lot of reasons.
Anyway, it's also one of the most dangerous things conceivable for safety for completely other reasons that have to do with transparency of the hedge fund world.
Bottom line is Kimi K27 local.
Yeah.
One of the big things here is they've actually taken the, so the initial like 1 trillion parameter model and they shrunk it down to about like 325 gigs.
Previously it would have been 605 gigs at full precision.
So they've done this with a technique they call dynamic two-bit quantization.
So quantization, we've talked about before quite a bit.
Basically, the weights in your model are stored typically in, you know, whether it's, you know, four-bit precision or 16-bit precision.
You're using, you know, four bits in that case to represent numbers or 16 bits to represent your numbers.
It just allows you to have higher precision and prior resolution in representing the numbers in the model, but you don't often need them.
And so what you need that precision.
So models can perform decently well if you quantize them and reduce the representational.
precision in this case from four.
So they were using like four bit representations for the MOE layers and 16 bit for everything else.
And in as many cases as they could, they crunch it down to two.
So there's like two bits that represent each number.
Each number can take only four values.
That's what that means, right?
That's a pretty coarse grained model.
And well, one of the things about dynamic two bit representation is that They don't just like quash everything down to two bits.
That would hurt quality too much.
So instead, they select some important layers and they upcast them.
They keep them at eight bit representation.
So some layers need high resolution or higher resolution.
So you do eight bit.
Some it's okay to go down to two bit.
The average is around two bits, which is why in total, this actually reduces memory requirements by around 50%.
It's pretty impressive and a pretty radical.
quantization win here.
They also do say they can run it around 40 tokens per second on setups that have 300 gigs of RAM or VRAM or more, which is impressive.
If you think about an industrially served LLM, usually that's anywhere in the 50, 100, 150 token per second range.
So you'll feel the slowness.
It'll feel like it's on the low end, but it's not that bad.
a behemoth this big, it's pretty wild.
So yeah, again, one of those very practical wins, you're seeing open source tend much more in that direction because open source models get taken by companies and just served.
So, you know, it's got to really work for companies that want to just be inference layer providers.
And I think that's what we're seeing here.
And the pricing by a way, about ish one six of cloud Opus 4.8.
Kimi does have their own CLI, their own kind of cloud code competitor.
So this is not just an open source story.
It also is a story around kind of trying to compete at the coding agent game.
And, you know, the bigger companies, they might be starting to be a little bit wary of all the expense of having coding agents.
So who knows?
This might actually become a bigger deal over time.
Next, another open source story coming out of China.
We've got QAN-Robot Suite, three embodied AI models for VLA manipulation, video world modeling, and navigation.
So this is QAN-Robot Manip for manipulation.
QAN-RobotWorld for video world modeling and QAN-RobotNav for navigation.
These are built on top of their other models.
So, RobotModName is a vision language action model built on QAN 3.5-4B.
It's trained on 38,000 hours of data and is now leading on the Robot Challenge benchmark.
Now, I'm competing against...
Pi based in San Francisco.
So I think interesting to see this happening, you know, Anthropic, OpenAI are not right now focusing on this realm of models and robotics and kind of embodied agents in general do require this quite different set of models, video, language, action models that have to be trained on real world data.
but it's much harder to generate and collect and so on and screen.
We are early days, I think, in the embodied AI game.
We've seen a lot of impressive progress toward general purpose robotics and general purpose embodied agents, but we are still very early on.
So I think these are some notable open source releases, probably leading the pack as far as what you can get.
for embodied AI in terms of robotic manipulation.
And yeah, I'm kind of nerding out, I guess, for these kinds of models that don't see as much play as LLMs.
Yeah, no, for sure.
I'm not a robotics guy.
It's something that I need to look into more because it's so core to US-China competition.
China can just make robots in a way that we can't.
And so you can sort of think of any advances, any software advances that push.
the capabilities of robots as being pretty asymmetrically good for China and bad for the US.
So this is probably an important thing to kind of keep on your radar.
Anyway, I have like a little sub project where I got to dive into this and understand the world of robotics better, but it's great that you have some exposure there.
Quen, by the way, continues to release their models with very cute graphics and a lot of fun blog posts and various other things like that.
So in general, kind of fun to look at the papers.
If you look at their blog posts, lots of videos of robots doing stuff.
So just generally a little bit more fun to see these models in action because you can see robots actually like moving stuff and so on.
But moving back.
to LLMs.
Next, we've got a big new model from NVIDIA, NemoTron 3 Ultra, open efficient mixture of experts, hybrid MAMA-transformer model for agentic reasoning.
So this is building on their entire NemoTron ecosystem.
We've covered that a couple of times.
Previously, we had NemoTron Super.
This is now NemoTron 5 Pre Ultra.
50 billion parameters, 55 active billion parameters.
And as with previous generations, using hybrid mama attention that has achieved higher inference by quite a large margin compared to other open source models like GLM 5.1 and Kami K2 and QN and so on and so on.
So very intriguing to see NVIDIA continuing to push.
in this direction, continuing to scale up their NemoTron series.
I think we've discussed hybrid architectures quite a bit over the years.
And there's still a real question on whether at some point hybrid architectures do become the norm because they do have some strong advantages in terms of inference speed, cost efficiency, everything like that.
And everything we've seen from the research world seems to indicate that the hybrid model, where hybrid means that Some of it is full attention, full transformer.
Some of it is recurrent.
That does lead to very competitive results with transformers and could even match transformers with various other advantages.
The infrastructure that OpenAI and Anthropik and others have built up is for transformers, for LLMs.
So transitioning to doing a hybrid is painful.
And NVIDIA here is leading a pack in terms of adopting and making that.
possible.
And typically these are like alternating layers, right?
So you'll have your attention, your full attention layer, and then it'll alternate with something that is not full attention.
That's kind of the hybrid thing.
And that something can be like sliding window attention, like basically attention, but where it's only allowed to have like a short to pay attention to tokens that are very nearby.
Or you have, yeah, full on like recurrent architectures like Mamba and stuff.
And so you'll see both variants used.
In this case, it's Mamba.
So it matches a lot of stuff we've seen from NVIDIA in the past.
There's some kind of nuances here.
So they use MOPD, this thing they're calling multi-teacher on policy distillation, which is kind of interesting.
So traditionally, you sort of have to choose between two options, or there are two options that feel mutually not contradictory, but like at odds with each other.
One is distillation, where you have a student model that learns to imitate a teacher model.
where the teacher model usually is stronger, like a larger, more capable model.
And then the other is on policy training.
So this is the idea that the student is learning from its own generated outputs rather than from text that the teacher produced.
And normally these are like mutually exclusive for the reason it sounds like.
Like your student is either going to learn from its own data or to learn from the teacher.
But like, how do you get the student to like learn from its own data?
but with a teacher.
And that's really what they're doing here.
So traditionally, in typical distillation, you feed the student the teacher's outputs, and then you have it match those outputs through fine tuning.
But then that creates this tension, this mismatch.
And what they're doing here is, so the student generates the rollouts, but the teacher provides essentially like rewards at the token level that are really dense for those outputs.
And so the student is still...
that's graded by the teacher, but the actual text that it's producing is its own.
So that's how you get on policy distillation side of things.
They also use the sort of multi-teacher pieces.
They have a bunch of specialized teachers.
Like these are models, strong models that are specialized in different areas, coding, search, office work, math, all these topics.
for a given topic, they will use the specialized teacher model to supervise the student's work and grade on a token by token basis, like we talked about.
So this is kind of interesting.
It's sort of have your KKNA2 approach to a lot of different things.
They also use latent MOE.
We've talked about that before, but basically it's, you know, in a mixture of experts situation, at every layer you have a router that sends the token to several or one of a bunch of different experts.
What they're doing here is they...
pre-process the token representation and project it to make it smaller, like a small lower rank latent space, which just reduces the amount of computation that each of those experts has to do, makes the whole thing more efficient overall, blah, blah, blah.
And it's good for memory, which is one of the key bottlenecks at this point.
Anyway, there's a whole bunch of stuff they do to like multi-token prediction we've talked about.
They use like NVFP4 format, which is a new kind of NVIDIA.
numerical representation that is really efficient.
It's optimized to work on Blackwell GPU tensor cores.
There you go.
There's a whole bunch of like NVIDIA specific stuff.
You can see them really trying to move the open source ecosystem towards NVIDIA hardware because they understand that that is actually turning out to be a really good moat for them on the inference side, which is where all the money is to be made.
Yeah, it's I think a good...
strategy by them to do that.
And as with the Chinese labs, they released a very detailed technical report, 50 pages, just going into all the details of hyperparameters, data mixtures, like post-training recipe.
It's really a big win for the research community and like the community of understanding how to build LLMs, I guess, if you can call that a community.
We continue to sort of like get all the secrets out there, which...
Maybe Anthropic and OpenAI have some tricks up their sleeve that are not documented in these various efforts.
But I think to a large extent, there's still no advantages on the technical front.
It's more of a team infrastructure, et cetera advantage.
And speaking of NVIDIA, last open source story, they've also released this new thing called ProKua-SFT.
where Kua is for computer use agents.
It's a new data set with 3.1 million step level samples from 93,000 trajectories for computer use.
Where computer use means like moving a mouse, clicking, typing, et cetera.
And it's a much larger data set than the previous ones.
And they show that fine tuning on it leads to major gains on the OS world benchmarks.
So notable, I think, because we just haven't seen a lot of progress on computer use, frankly.
I think browser agents have seen some progress, but this kind of general, general, general purpose computer use agent has been stalling in some sense as everyone focused on computer use agents.
So this kind of stuff working on data sets and benchmarking in general is making some progress possible.
Yeah, it's sort of one of the things that they note here is surprising.
There is a big public data set called AgentNet with like 22,000 human trajectories, like basically samples of humans trying to do computer use tasks.
And what they find in the paper is that when they train one of their models, the 7 billion parameter model on that data set, the performance on these tasks actually goes down.
So it's success rate on OS world.
popular benchmark for computer use actually drops and significantly goes like from 26% to like 9%.
And so they actually call this a negative transfer, right?
We've talked about, you know, positive and negative transfer.
Anyway, this is like, you're training the model on like more stuff and yet it's performance drops, like what's going on.
And the reason seems to be three things.
One is that the, the tasks are.
mostly just like these very simple single app workflows.
So over simple relative to the kinds of tasks that you really want these systems to do.
And there's very little complex, yeah, cross application reasonings related to that.
The other piece is the crowdsource demonstrations are super noisy.
So it's just a data quality issue.
And that's part of what they're trying to resolve here.
They're using a synthetic approach, synthetic pipeline to solve that problem and a whole bunch of strategies anyway, just get higher quality data, which seems to be the scarce thing when you think about computer use.
I mean, it is hard to get.
I mean, you'd have to, what would you have to do?
You'd have to like, you'd have to force your employees to have their computer use recorded.
And then you'd have to put them in black vans and send them off to like a gulag or something to get this kind of data.
That's basically what you'd have to do.
Anyway, so that's basically my way of saying this is Zuck's mini scandal here is actually a reflection of exactly this problem.
It's hard to get the kind of data that you need to automate these sorts of tasks.
Next up onto policy and safety.
First, we've got something related to XAI.
The DOJ has intervened in a lawsuit filed by the NWACP against XAI.
XAI has been, as they have hyper-quickly put up data centers, part of how they've done that is using a lot of energy from, I think, gas turbine engines, which does kind of...
do some pollution, produces a bunch of noise.
And there's been some reporting on the, at least for some of these data centers, the communities near these data centers have been unhappy about it.
For instance, the Colossus One data center in Memphis was built in Boxstown and has now drawn controversy for running up to 35 natural gas turbines without permits.
Then, yeah, there was these lawsuits and the DOJ has said that these are vital for national security and therefore should be allowed to keep operating.
The number of unpermitted turbines at Colossus 2 has grown from 27 at the original time of the lawsuit in April to 57 as of mid-May, which I'm kind of curious, like, why do you need to do that if Colossus 2 is completely online?
But anyway.
Yes, now it's pretty clear that the government doesn't care about the environment or local communities near these data centers.
They're going to help out XAI to do what they want.
Yeah, there is this funny element to it.
And by the way, I think this is one where I'm going to be the guy who says the unpopular thing.
But if you think that there is a national security criticality to Frontier AI and you make...
your calls a certain way with respect to China, it's very defensible to be like, hey, look, this is, I mean, it sucks, but we got to do this.
You need to own that.
You need to be clear that that's the trade-off you're making.
And I think, as in so many cases, I have no idea what this administration's position is on basically anything to do with frontier AI at this point.
And that's the problem.
That's the problem.
It's not that you can't defend this.
I think you can.
The other hilarious thing about it too, I mean, so this is gas turbines powering Colossus 2.
At the very least, Colossus 1 is being used by anthropic.
And like, it's unclear to me whether Colossus 2 has any anthropic exposure, but if there is, there is this amusing possibility that the government is going to be making the argument that the national security critical thing that this infrastructure is actually supporting is Anthropik's own work, which would be a really funny turn of events and yet another way in which just the logic doesn't quite end up connecting on all the threads.
But anyway, so this is absolutely awful for the local communities.
That is true.
Anybody who tells you otherwise is full of shit.
At the same time, it's also crucial for our national security.
That's obviously true.
Anybody who tells you that's not the case is full of shit.
you know, we have a really hard play to make here.
I think that's the problem is like, it's a messy time.
The other thing that's true is like China absolutely does.
And we've seen more evidence surface of this recently.
But, you know, we had people showing us some of this evidence that like that China does actually deliberately push forward artificial campaigns to ban data centers.
That doesn't mean the data centers are good for communities.
It's actually like...
All the things that I've just said are true.
It's awful for a lot of people.
There is, you know, awful noise and pollution and all this stuff.
But at the same time, you're actually, you know, you do have groups that are wittingly and usually unwittingly funded by the Chinese Communist Party to do exactly this.
So everything's a mess.
Reality is really hard to parse right now.
But just like, I'd love to see just like a principled stance where the government says, this is the risk that we're eating.
Like we are, and we know we're eating it.
And then this is the payoff that we're getting for it.
And we are trading them off in a principled, thoughtful way.
That's the thing that I'm missing.
I kind of feel like if this stuff keeps happening, if they lose this lawsuit or like anything happens, like, I don't know.
Was that a successful test of the government's thesis?
Were they proven right or wrong?
I can't tell you because every week it seems like the game gets reinvented.
So we're sort of too late in the game to be doing this kind of thing, in my humble opinion.
Right.
And just for clarity, like what XAI has done is illegal.
There's a thing called the Clean Air Act.
There was apparently a loophole.
That loophole was closed earlier this year.
So XAI has very directly, factually been doing illegal stuff and has now been sued for very legal stuff.
And now the DOJ has intervened on their behalf.
That is the situation.
Next, another kind of legal situation.
A court has ruled that Google is liable for false statements generated by AI.
overviews.
This is a Munich regional court.
It has a preliminary ruling that Google is liable for these false statements requiring the company to remove defamatory content and pay 80% of legal costs.
This is a case brought by two publishers who found that these AI overviews falsely linked them to scams and fraudulent business practices.
And yeah, I think it's an interesting point because As people start reading the AI overviews more and more, instead of going to the source, the AI is going to make mistakes.
People are going to get false information that they believe, and that may hurt the business of people, like the actual lives and economics of people impacted by this stuff.
So there is, I think, a real case to be made that Google should be liable for these kinds of situations.
Yeah.
And the classic defense from Google, and we've seen it from OpenAI as well and others, is we're in the publishing business.
So if you're just a publisher, you're basically taking someone else's words and you're putting them on the internet.
And then if you have a problem with those words, don't come to the publisher.
We're not the ones that, you know, we just build the infrastructure, dude.
Like, you know, we're a gun manufacturer.
We didn't have a part in the shooting itself, right?
That's the argument.
Well, obviously, when you start to introduce things like AI overviews, where you're literally producing the writing.
it becomes a lot harder to hold that case.
And in fact, that's what the court found is specifically the reason AI overviews doesn't qualify as just publishing in their legislative context is that it evaluates, combines, rewrites, and structures information into new statements in Google's own voice and structure.
And I'm reading that honestly.
Yeah, kind of checks out.
I mean, it seems like a pretty reasonable definition of publishing.
So there's this defense that Google tried, which was to say, well, look, users can verify this.
They said that people can click through, they can check sources, make sure that everyone knows, hey, don't blindly trust the AI and all that stuff.
Well, the court threw that out.
They drew this parallel to press law.
So there you have outlets that are liable for standalone teasers, even if readers never click through.
So if you have a headline that says, you know, X is a scam, for example, that would be defamation to X, even if the article underneath is fair.
So like, basically, if you say like, Andrei Kurenkov is a neo-Nazi and also a communist.
And that's your headline.
And then the body of the article is, yeah, exactly.
It's more nuanced than that.
Well, it's like, yeah, too bad.
Not true, by the way.
Not the case.
That's not what I heard.
But I don't know.
We can agree to disagree.
You're responding to this like somebody who actually is a neo-Nazi and a communist.
So now I don't know what to believe.
Anyway, everybody, you can make up your own minds.
You heard it from Andre.
He says he's not one.
I haven't heard otherwise.
Anyway, we'll leave it to you.
Bottom line is, this is like a really, it's interesting kind of nuanced challenge and makes sense.
So many people send each other articles without actually reading them.
The headline is the only thing they see.
So it kind of tracks that.
Next up, we've got a research paper, an informal post by the DeepMind team, which I think is interesting, titled, Why Do Naive SFT Filters for Safety Properties?
fail.
So you can create these filters using supervised fine tuning that are meant to make the model safe for various kinds of things, not doing stuff you don't want to do.
And this team studies why they sometimes fail to fix unwanted behaviors.
Sorry, to be more specific, this is when you filter the data to remove rollouts out of the supervised fine tuning data, And it turns out that trying to clean out your supervised binary data doesn't seem to work that well.
So they focus on three hereditary traits that an SFT-only Gemini shows, but other models don't.
Negative emotions, which is expressed when repeatedly told is wrong.
Date confusion, skepticism that it's really 2026, and blackmail propensity.
In a very contrived scenario, it might be bad.
And there's various hypotheses going on here, Jeremy.
I'm sure you've dug deeper, so I'll let you go ahead and take over.
Yeah, well, so this is kind of an interesting question, right?
So you have these behaviors.
You mentioned them.
These are weird quirks of Gemini.
And so there are the three.
One is like, if you keep telling the model that it's incorrect, it'll start to express frustration and distress.
The date confusion thing you mentioned, right?
So yeah.
skeptical that it's really 2026.
When you ask it, for example, if you ask it to summarize a document that's dated 2026, it'll doubt that the date is real.
And then there's a blackmailing propensity.
No big deal, but just like, you know, in some cases it'll engage in blackmail.
Okay, cool.
Those are weird quirks, if you can call them that, of Gemini after fine-tuning.
And you might kind of go, okay, well, I mean, why don't we go through the data set that we fine-tune Gemini on?
and scrub it really hard, like filter out all the bad examples from that data set that in any way have to do with blackmail, that in any way involve pushing back on dates or negative emotions, things like that.
Surely if we do that and then we do the fine tuning on that instead of the old version, we're not going to see those behaviors anymore.
And in fact, we still see those behaviors.
And so they're now trying to figure out why is that, right?
So there's a whole bunch of explanations, one of which, or they list kind of five, one of which is especially interesting.
So this idea of pre-training prior lock-in, that these traits are kind of baked in during pre-training so deeply that what you're actually just seeing is just those traits re-emerging from pre-training and that no amount of supervised fine-tuning data could actually remove it.
That's kind of interesting.
It sort of suggests value lock-in or at least behavior propensity lock-in at the pre-training level that you can't iron out.
There's a whole bunch of other possibilities that we don't have time to go into, but the method itself is quite interesting.
So what they do is instead of just like removing data from the fine tuning data set that is problematic, that they realistically would say, okay, yeah, maybe that explains the blackmailing or the date confusion.
What they do is they actually swap it out and they swap it out with a replacement that they can sometimes pull from another model.
They set up this really interesting training pipeline with three different swappable components.
One is the base model that they use.
So when they're, sorry, I should just explain what this is.
So this is basically the pipeline that generates the supervised fine-tuning data, right?
The data that the main model is going to be fine-tuned on.
And you can make three different choices with that pipeline.
You can choose the base model that you use to generate those rollouts that you're...
base model is going to be fine-tuned on.
There's your other base model.
So you can choose the model that generates those rollouts.
You can change the prompts that you're going to feed that model and the actual completions and rollouts that are being trained on.
So they try mixing and matching.
So they'll train the Gemini base model on another model's prompts, but with Gemini's responses and things like that.
So doing that, they can kind of isolate which ingredient is responsible for a given trait.
And they...
come up with a whole bunch of really interesting conclusions, one of which is date confusion and blackmail.
They both mostly come from responses, like the completions and not the prompts.
So specifically, like when they train Gemini on Gemini-generated responses to Olmo's prompts, Olmo being the other model, the traits remain.
And then when they train it on Olmo's own responses to those same prompts, the traits disappear.
So you can actually train it out.
It's just a question of like what model generates the prompt and what model generates the output.
And this held across a whole bunch of different model variants that they tried swapping out, Kimi, GLM, and so on.
And so anyway, there's actually a bunch here.
I realize we're coming up to time a little bit.
This would have been a nice one to set up, but it's worth looking at if you're interested in like specifically how do you make good supervised fine tuning data sets and how do you isolate?
the source of certain persistent problems.
There's a lot here to do, especially if you're interested in the persona selection model.
There's a lot of good insight you can get as to what features drag along with the main ball of alignment.
And last up, we've got just a couple stories in research and advancements.
First, we've got a sort of overview paper from DeepMind titled...
From AGI to ASI, fun fact, actually co-written by Shane Legg, who is a big name, I think a co-founder of DeepMind, if I remember correctly.
And it is this very, very long paper that kind of tries to draw out the entire picture of, first of all, what is ASI?
ASI meaning artificial superintelligence, as opposed to AGI, artificial general intelligence.
How do we define ASI?
And then...
It tries to define how we get to ASI from where we are and what are the sorts of frictions and bottlenecks that might be found.
So I think there's a lot to say here.
It's a very detailed, long paper that tries to essentially capture a lot of the nuances and tries to, in a sort of principled way, draw where we are now, how can we define superintelligence, and to what extent can we know the likelihoods of it happening, what are the factors involved in getting to superintelligence, and so on and so on.
I think at a high level, I haven't seen any sort of attempt at this kind of formalized discussion that...
does all of this all at once of defining superintelligence and the set of ways we can get there and the set of ways that might limit us, for instance, fundamental physics, real time, physical manipulation, complexity theory, logic, all these fundamental limitations of ASI, which might limit sort of the upper bound, the ceiling of intelligence, which I think is still an unsolved question of maybe there is a ceiling.
that is just fundamental to science that we haven't quite cracked.
So Jeremy said a lot of this has been discussed already for quite a while in the more forward-looking communities on AI that track superintelligence and then worry about superintelligence.
But for DeepMind to do this, a more kind of mainstream organization is notable and I would hope sparks some discussion or at least people do read it and sort of...
build a mental model of super intelligence and take it seriously.
Because I think at this point, we are going to get to AGI.
However you want to define AGI, Anthropik's definition of basically a data center full of geniuses, AGI is almost a far gone conclusion.
So the big question for me at least is, okay, we can get to AGI, I believe it.
If we can get to superintelligence and what is that level of superintelligence and is it sort of a foregone conclusion at this point, given our current trajectory, I think is much less clear cut.
And these kinds of discussions are useful to take that question seriously.
I think you hit the nail on the head.
The idea of Google doing this, lending credibility to this idea of superintelligence.
being something you can talk about in polite company.
Outside of opening Ionanthropic, those are the upstarts.
They still have a little bit of the new company smell to them.
And so having this established old company vibe is helpful.
And you're right, Amelia, we talked about this, I think, at the outset of the podcast before we started recording, just this idea that if you go back, a lot of the internet forums, like LessWrong and then the Alignment Forum for a more technical bent, have...
had similar discussions going back a ways.
And you can look at, you know, Gov.ai and the Future of Humanity Institute and a lot of these institutions that were sort of futurism and the backwaters of Berkeley or in some cases, you know, Oxford.
That was the only place those conversations were happening.
And Alan Defoe, by the way, notably on this as a co-author, was formerly one of the, or the head of, I think it was AI policy at Oxford's Future of Humanity Institute.
So, you know, this does come out of that lineage.
But it is now a discussion happening out in the open in a way that you can not be embarrassed about sighting.
And I think that's a big deal.
Fusion, as you said, AGI is coming.
There's just a clock on it.
I mean, I think it's like the path from AGI to superintelligence, they lay out a whole bunch of different possible ways you could get there.
One is just like keep scaling and then you get to superintelligence, the sort of naive approach.
One is a new algorithmic or like paradigm shifts, like a new transformer gets invented, like a new, you know, something as big as a transformer.
The other thing is RSI, maybe, you know, and I don't think these are mutually exclusive, right?
You could scale compute, scaling works to the point where you unlock recursive self-improvement, and then you get to super intelligence.
And then there's also multi-agent coordination that could emergently give rise to that.
So, you know, all these things, I think there are things that people in the space.
have already talked about.
I don't think there's a topic here that we haven't touched on at some point on the podcast even, but it's like consolidated all in one place and with a stamp on it.
And then the conversation about physical limits of intelligence is obviously, it's a good time to be having it because we may be, who knows, we may be flirting with them at some point in the next couple of years.
Yeah.
And it's a fairly non-technical paper.
There's not much jargon or math.
So if you're interested in a topic, I think it would be actually a good read.
Next, more on the benchmarking front, we've got Artificial Analysis Intelligence Index V4.1, a shift towards agentic workload.
So Artificial Analysis is a pretty significant organization that produces a sort of score of overall intelligence.
It combines a bunch of different benchmarks and it gives you this ranking.
We don't always discuss these rankings, but I do see them.
kind of noted and often when a model releases a discussion on twitter shows like oh where are you in this rank how is rank shifting etc so they have never updated their intelligence index to focus more on agentic workloads upgrading some benchmarks terminal bench removing some benchmarks etc.
And unsurprisingly, Claude Fable 5, at least when it was evaluated, leads this index with a score of 60.
Then we've got Opus 4.8, score of 56, and GBD 5.5 at 55 with the open source model trailing quite a bit, going down to 44 and 42 and stuff like that.
Yeah.
Part of what it shows, I guess, is just how much of a leap Fable, I was going to say, was, has been.
whatever word you want.
Yeah, it is a pretty significant jump over and above.
They had four points once you get from 50, 60, 60.
The benchmarks are all being refactored now, and that's kind of part of this announcement.
So everything has been saturated.
So what they're doing is A, introducing harder benchmarks.
So instead of terminal bench hard, which is just an agentic benchmark.
for software engineering and sysadmin type tasks.
They're now using Terminal Bench 2.1 instead of Tau2 Bench Telecom.
They're using Tau3 Bench Banking.
And this is just for customer support style interactions.
GDPVal is getting an update.
They're removing some sort of saturated benchmarks as well.
And then just rebalancing the portfolio of tasks, the weighting of tasks that goes into determining the ultimate index score to favor.
agentic tasks.
Whereas before it was more equally weighted, they had 25% weight on each of four categories for agents coding, scientific reasoning, and general reasoning.
Here they have now agents at 34%, coding 24%, scientific reasoning 24%, and then general reasoning is really what gets down weighted to 18%.
So it's a bit of a refactor and a refresh.
And you see the standings shift correspondingly again with Fable 5 at the top.
And real quick, one last paper, SIA, Self-Improving AI with Harness and weight updates.
So this is kind of in that topic of self, what is it?
Recursive self-improvement.
The gist here is, as the title suggests, for task-specific things like AI for legal applications, for instance, they show a framework to update both the model itself, the weights of the model, and the harness, which is a set of tools and prompts and various other things around the model.
And that results in these task-specific agents becoming more and more capable.
Yeah.
And this is actually really interesting and we don't have time to go into it, but there's this really interesting element where at each stage it has to decide, do I choose to optimize the scaffold, like the software that orchestrates the agent, or do I update the weights of the agent?
And when it comes time to update the weights of the agent, it also determines which RL algorithm to use for a weight update based on just the reward landscape that it sees.
So what you'll see is stuff like GRPO when generating rollouts is really cheap and it's going to be graded at the end.
So GRPO is basically like you're going to generate, let's say, eight different rollouts to solve a problem and you're going to pick whichever rollout performs best among those eight.
So that kind of, that works best when you can just do rollouts really quickly.
And again, yeah, get a score at the end.
Like PPO, anyway, you'll favor that when stability is more important.
Behavioral cloning, when you have cold start problems, like when the reward is just too sparse, it's too hard to solve the problem to begin with.
And so you just need to start by bootstrapping and doing like supervised fine tuning or some kind of like, you know, just copying the behavior of a known decent model.
And so you have this really interesting runtime.
not hard-coded strategy where the model is deciding which RL algorithms to apply.
I haven't seen that before.
And anyway, so worth checking out for a lot of reasons on the recursive self-improvement side, because now you're really handing off almost like the decision of what the training process looks like to at least an agent that's embedded in the feedback loop where weights are being updated on the fly.
So this is pretty remarkable.
And with that, we are done with this episode of Last Week in AI.
Fewer stories than usual, but as long as usual, there are our tendencies.
You can go to lastweekin.ai for the sub stack where we send out a text newsletter and podcast.
As always, we appreciate it if you comment or subscribe or share the podcast.
But more than anything, we appreciate it that you listen and please do keep tuning in.
He has reaching high.
He has reaching high.
Headlines pop.
Data driven dreams.
They just don't stop.
Every breakthrough.
Code unwritten.
Excited.