# Frontier AI Price Wars and Infrastructure Shifts

**Podcast:** Last Week in AI
**Published:** 2026-07-15

## Transcript

Hello, and welcome to the Last Week in AI podcast, where you can hear a chat about what's going on with AI.
As usual in this episode, we will summarize and discuss some of last week's most interesting AI news.
I'm one of your regular co-hosts, Andrei Kurenkov.
I studied AI in grad school and now work at the startup, Fast Decade.
I'm your other co-host, Jeremy Harris.
I'm wearing a hat because I got back from taking my daughter to the park.
What?
And yeah, actually, I'll be away, by the way, for the next episode.
because I'm on a trip to the States doing a bunch of AI security stuff, but excited to do this one.
We punted the recording from Wednesday this week.
We're now recording on Saturday.
And we were just talking about how a bunch of stuff came out in the interim, which is really great.
Challenging a little bit because the way my schedule works out, I'm not able to like spend the same amount of time between Wednesdays and Saturdays.
So I'll be a little lighter on the details for some of these things, but I've still tried to prepare a bunch of material for it.
Be warned, Andre is going to be carrying some of the weight for some of the more recent releases.
But yeah, hell of a lot happened.
Yeah, it's one of these things where it's like a thing in the industry where we get waves of releases because I guess everyone wants to be one-upping each other.
And everyone kind of knows when each other's stuff is about ready to go.
So everyone just sort of like preps their releases and announcements and then they just do it around the same time.
And that's exactly what happened in this episode.
talking about GP5.6, we'll be talking about GROC 4.5, Muse Spark 1.1, a few other models on top of that.
And beyond that, the big story I think will be in policy and safety with some new vehicle research from Anthropic and a new kind of major think piece from AI 2040.
So it should be a good mix of content in this episode.
This episode is brought to you by OutShift, Cisco's incubation engine.
Today's AI agents operate in silos, limiting their true potential.
We've been focused on building bigger, smarter models, but scaling up is just one approach.
To reach superintelligence together, we need to do more.
We need to scale out.
And we actually have a blueprint from 70,000 years ago.
Humans didn't just get smarter individually.
The cognitive revolution transformed society because we began sharing knowledge, goals, and innovation.
Agents are now at the same inflection point.
They can connect, but they can't think together.
That's why OutShift by Cisco is building the Internet of Cognition, transforming AI from isolated systems into orchestrated superintelligence.
We're creating an open, interoperable infrastructure.
OutShift is enabling agents and humans to share intent.
context, and reasoning.
The cognitive evolution for agents is here.
Explore the Internet of Cognition at outshift.com.
That's outshift.com.
We'd like to thank Box for sponsoring us.
Box is building the intelligent content management platform for the AI era, serving as a secure, essential context layer for Box's AI agents to access the unique institutional knowledge that makes a company run.
And that's a key idea.
The power of AI doesn't come from the model alone.
It comes from giving AI access to the right enterprise content.
Box's recent State of AI in the Enterprise report found that 96% of organizations say agents need access to company-specific content, but only 36% have connected agents to trusted content across many use cases.
And that trust bit is very important.
Box is built with security, compliance, governance, and threat protection in mind, so employees and agents only access the information they're authorized to use.
If you're thinking seriously about adopting AI in your company, think beyond the model.
Your business lives in your content, and Box can help you bring that content securely into the AI era.
Learn more at box.com slash LWAI.
Summer routines live or die by how easy they are.
And honestly, if something takes too much effort, I'm out.
That's why Grooms is my go-to.
It's my one daily pack of gummies covering my greens, vitamins, and minerals.
Plus, it has six grams of prebiotic fiber, which is more than two cups of broccoli.
No mixing powders, no giant pills, no hassle.
I just rip open the pack, and I'm done.
They taste good, and they make it easy to stay on top of my health, even when life gets busy.
Save up to 52% with the code PODCAST at Grooms.co.
That's code PODCAST at G-R-U-N-S dot C-O.
Let's kick it off in tools and apps.
First up, we've got OpenAI rolling out GPT 5.6 after government greenlight and announcing chat GPT work.
So we have now publicly rolled out GPT 5.6, including Sol, Luna, all the variants of a model.
I think we already discussed previously how a model was.
Sol was a little bit confusing in terms of its benchmarks, but generally seen as like fable-ish.
sort of frontier competitive with Anthropic.
And I suppose the bigger news aside from these models being public is this ChatGPT work thing where they are updating the ChatGPT desktop app, which used to be sort of just mirroring the website where you have a chatbot interface into just codecs, basically, into an agentic coder.
And then they kind of hid the actual chat functionality very kind of...
subtly in the UI.
So they're trying to onboard whoever is using this desktop app into using agents like hardcore, right?
Which is an interesting strategy by them.
And also rebranding to ChargedGPT work from Codex, which seems pretty smart given like everyone knows ChargedGPT.
I think not many know about Codex.
Yeah.
And I think in and amongst all this too, The whiplash on the model release permissions, and I will call them permissions from the U.S.
government because that is what they amount to, is pretty wild.
So the U.S.
government just greenlit the release of GBD 5.6, of course.
The request that they had made was for a review period for the U.S.
government to be able to look at GBD 5.6, assess its capabilities, and clear it for release.
The strange thing that we're getting is there was a report out in Axios that said that The government's position on the GPT 5.6 SOL release was that no permission for model releases is required or granted.
with respect to 5.6, and the decisions about the scope of these releases, quotes, rest entirely with the companies.
Now, in theory, of course, this is maybe the case.
There is no law on the books.
There is no regulation on the books.
Officially, they could have just done it, and then who knows what would have happened, you know?
Here's an excerpt, and I'm pulling this off from, I can't remember whose tweet I'm getting this particular excerpt from, but it was really, I thought, good sort of.
context for this whole discussion.
So if you look at the letter that Lutnik sent to Anthropic when he was approving fable going back on the market, kind of weird thing to even say in this context, isn't it?
How is Lutnik approving fable if there is no permission required or granted?
Like what?
Anyway, here's the wording that Lutnik used at the time.
He said, Anthropic has committed to work with the U.S.
government on protocols and standards and releases for the, quotes, covered models.
He says, I reserve the right to reevaluate and adjust the scope of license requirements on the covered models should circumstances change.
What?
And this is the Secretary of Commerce, by the way, not like some sort of oversight thing for AI at all.
It's very ad hoc, essentially, all this stuff.
Sam himself said of the 5.6 solar release, says, bad news.
This is on Twitter at the request of the US government.
It's launching today in limited preview instead of the open access launch we're planning on.
We're working with the government to get to general availability as fast as we can.
This does not sound like a company CEO who has a free hand to do what he wants.
It sounds exactly like someone who understands that in reality what's happening is.
The government, yes, will allow you to do it, but then they'll smack you back by having the BIS export control the crap out of your company and effectively grind your operations to a halt with some of your most important customers on a whim.
That's the reality here.
So there's sort of this funny thing.
I have no objection to licensing regimes.
Again, we put one together like three years ago when we predicted this moment would come.
It's been obvious we would get to a moment like this.
And frankly, a lot of people in the Trump administration in particular have spent the last many years saying that's ridiculous.
It would be irresponsible.
This is craziness.
This is dumerism, all this stuff.
And yet here they are doing exactly that, ostensibly, but just not calling it that, which I think is just an intellectually dishonest play.
I'm being quite forceful on this because I think we need to force a conversation.
at the legislative level, to have actual regulation, to do this with principle.
I'm not saying any answer here is right or wrong, but it's like, it is just obvious to all concerned that the current approach is wildly untenable and just false.
We're getting like lies that are lies in practice, if not in theory, they are lies in practice.
When you look at like, how is this stuff actually being regulated?
There is a licensing regime today in the US for Frontier AI.
Full stop.
You can dress it up.
You can say that it's just like, oh no, it's like, but like ultimately it's also inconsistent.
Opening, I was just asked to, I guess, just show due deference to the feds.
Whereas Anthropic was export controls.
Different mechanisms are being used here in what seems to be a fairly ad hoc way.
And they're being very weakly and poorly justified.
So anyway, there is also all kinds of undisclosed model loopholes, by the way.
So if you have a frontier model that's built by one of these labs and it's just not announced, Some of those models, the internal deployments, if you're concerned about loss of control, at least, or weaponization, the internal deployments are as dangerous as the external deployments for a huge fraction of the risk surface.
China can steal the model.
You can lose control of the model.
A rogue insider threat in your company can weaponize the model.
Like, none of this changes.
And yet, there is literally no effort being expended, seemingly, no requirement to disclose internal deployments.
So not only are you seeing a totally unprincipled uncoordinated, seemingly semi-thoughtless approach being applied here that's rattling markets, causing all this confusion, but it's also failing to grasp that the exact risk they seem to be concerned about, which is AI superweapons, still exists fully.
for a wide range of the kind of threat vectors that you'd be concerned about.
So anyway, I just think this is like a really classic example, by the way, is say super intelligence, right?
A lot of people think that they have something close to frontier potentially.
If that's the case, I mean, totally undisclosed.
They don't even plan on launching a product.
They may have these things running around on their servers, China all in there, Russia all in there, whatever.
Yeah, who the hell knows, right?
So this is just, I mean, it's going to get fixed over time, but there's a pretense here that there is in any way a kind of like, thoughtful approach being applied that I think is just laughable at this point.
This is a chaotic mess and it needs to be fixed.
We need to be able to look back, like literally just, I'm just asking for two weeks of look back where the government looks half coherent in a two-week window.
Please, just give me two weeks where I can go back to look at what did the Congress Secretary say?
And okay, yeah, that's aligned with what we heard the White House say.
You know what I mean?
We're just not getting that.
This is just crazy talk at this point.
Wow, Jeremy.
It sounds like you're getting to understand US politics.
What I will say is it used to be more- coherent.
Like when the White House put out the action plan, you know, it's this roadmap.
They more or less stuck to it for like, I don't know, six months or whatever.
And then the anthropic thing happened.
And I feel like everybody just like got brain cancer or something.
And now like I have no idea what the hell anyone's position is on anything.
And they keep saying stuff that's like, dude, like, do you actually have a preferred lab?
Because it really sounds like you have a preferred lab.
And that's not good.
It's bad for your standing in the courts when you inevitably have to litigate this stuff.
It's bad for the market.
It's bad for the coherence of your security and safety argument.
This is really bad.
So as usual, we had to touch on the politics of this.
And one more thing to say on that front is an interesting, like as a comparison to what happened to Anthropic, I saw after this release that UK AZS, which we've talked about.
I think quite a few times, there was a post saying that when they got access to evaluate GP5 Sol, they were able to get universal jailbreaks within hours of testing, which is what was not shown to be the case with Anthropic.
Supposedly, the problem was like this very directed single jailbreak that kind of wouldn't...
give you much versus universal jailbreak is basically we can make this model do whatever you want.
Either way, it's very clear there's fewer safeguards in place than what Anthropic had.
Anthropic had a very restrictive set of things, even with the initial launch that would roll you back if they detected that you were doing cyber stuff or whatever.
To my knowledge, there are still cyber things in place by OpenAI, but I think...
If you compare them, it's very different.
So good for us to get GPT 5.6.
So we got the vibe check, I guess, of a community.
And it's a good model.
People like it.
It might be Fable level.
It might be better than what Anthropic has.
So it's very nice to see the outcomes of competition and OpenAI sort of regaining its focus and getting back to providing a challenge to Anthropic, which we have been sort of focusing on other stuff for a while.
On that note, right, if it is Fable class, one of the things here that is also pretty bullshit is that Anthropic was held back on their release and therefore their monetization of Fable for weeks.
Meanwhile, like I want to see the side by side of how long OpenAI was forced to delay 5.6.
If the gap is less, basically the US government just handed them weeks and therefore hundreds of millions of dollars.
That's what those weeks bring in in profit.
And so.
This is just another way in which this unprincipled approach of seemingly, I think to many people, picking favorites almost shamelessly.
When you look at the language that's used by just some of these, like it's crazy.
Anyway, so.
Yeah, it's crazy if you don't live in the United States that this could happen.
But why we keep talking about it is that it appears likely that this is just going to be the situation going forward.
So this is not a one-off case of it happened with these two models.
It looks to be the case that the administration is at the point of saying, okay, there's going to be these powerful new AI models and we have a say in what these companies do.
And we can do whatever we want because there's no rules in place or no policies and so on and so on.
So we can expect to keep talking about this, unfortunately.
as new models from Anthropic and from OpenAI get announced.
Yeah.
Just to also like, you know, people listening, you know, I have been a defender of this administration's approach to AI for quite, like basically for the entirety of their first year at this.
You can go back, I mean, I think to a fault in some cases, but, you know, I was trying to find the logic where it was, you know, in Silicon Valley, there's an unfortunate bias in the other direction.
And I think that needs to be corrected for so you can sort of see the signal and the noise.
I think the challenge is this was all kind of cozy as long as an ideologically aligned entity was in the lead.
I think that mythos just like broke everyone's brains.
I think there was also, you know, David Sachs has no business having an opinion when it comes to AI.
Like Marc Andreessen, these are people who fundamentally like do not actually understand the technical side of the safety argument and the security pieces.
And I mean, you can go back and see like individual things that show that.
you know, you can hope that there is enough.
And I think overall, saner minds will prevail to be clear.
Like eventually it becomes just like, these are bioweapons.
These are cyber weapons.
Like no one can really deny that this is a thing.
And I think at that point, you're going to see Congress step in.
You're going to like AI 2040 and all that stuff is, is making, is calling the very easy prediction in that respect.
And we've been talking about that for, for years on the podcast alone.
And so.
Sorry, I mean, on the podcast alone, as opposed to other venues.
So anyhow, I just think we're going to get to sanity.
It's just like people have to be moved different distances to get there, let's say.
It's a bit of a mess, this whole thing.
But if you don't want to hear about the politics, I guess this is good news because GPT 5.6 is out.
They have Sol, Luna, and their small model.
And it is a cheaper alternative to Fable.
While, you know, it's some...
first person accounts being potentially better.
And one of the weird parts with these frontier level models is they do seem to be diverging a little bit in their flavors of intelligence.
So these models always had a sort of character to them and slight quirks and differences.
But in terms of their capabilities, they felt similar to me.
And it now feels like There's this high level differentiation between GPT 5.6 and Fable where when doing very advanced problems of like math and so on, they have different sort of high level pitfalls and ways of reasoning and looking at the problem.
It's quite an interesting phenomenon.
Next up, another new model.
This is Grok 4.5 from Space XAI.
They are calling this an Opus class model.
And if you look at the benchmarks, it's generally competitive Opus 4.8, Terminal Bench, Deep SWE.
This is their coding model alternative for both GPT 5.6 and what Anthropic is providing.
And the big news, I guess, besides that they have a new model, people are saying this is a pre-trained model.
So this is like...
cursor jumping in and training a brand new grok that is much better at coding it's very cheap relative to what the alternatives are there is the pricing is two dollars per million input tokens and six dollars per million output tokens that's i would have to check but maybe two and a half times cheaper than opus if i remember correctly and i don't know how much cheaper it is than fable so it's a relatively very cheap model that appears to be pretty capable.
And from what I've seen of people's conversations, people seem to be saying that it is pretty capable in practice, not just on the benchmarks.
So I think we'll talk about some other model releases too.
The trend is definitely moving toward a price war where token prices are going to be pressured down, especially now where Fable, you can use an opus class model for your everyday needs.
With Frontier models, you probably won't be using as your default.
And that means that this price war might get nasty.
Yeah, it's actually quite interesting to think about where exactly you're going to see the profits accumulate for that reason in the space.
There's this argument that I think is implied.
Maybe it's made explicit.
I just haven't heard it quite that way.
People say there's infinite demand for intelligence, yes.
Is there going to be infinite demand for the frontier of intelligence?
Or does the frontier of intelligence end up getting consumed by the labs themselves in pushing recursive self-improvement?
Like what's the shape of that pyramid basically?
I think that's a question we don't yet know the answer to.
And it's, I think, more non-trivial than I think a lot of people make it out to be.
Myself included.
Myself from like three years ago, certainly included.
So we'll see.
But the price wars are definitely going to be an issue increasingly.
I do think the frontier will continue to command a premium.
And I do think that it's going to be hard for players like Grok to actually end up leapfrogging Anthropic.
We're seeing a lot of stuff like some of the benchmarks that were shared.
originally showing things that, you know, may have worked their way into the training corpus that were like, you know, these cursor oriented benchmarks and things like that.
But one interesting note about this too.
So We don't have, unlike previous Grok announcements, what we don't have is a model card that looks at all the usual risks.
And so 4.5 references a bunch of benchmark comparisons, but we don't have this structured principled assessment of how things are going to work on the safety side.
We know there are some cybersecurity safeguards that were added.
They're kind of hand-waved.
And I mean, ultimately, this kind of hand-waving happens when you're concerned about US government.
stepping in and saying, hey, like you can't actually release this man, of which there is plenty of precedent now.
So anyway, it's also meant for like these long horizon tasks, right?
With nominally full autonomy.
So the stakes do go up, whether it can actually execute, you know, on the kind of meter evals type plots that everybody cares about on cyber and on AI and autonomy remains to be seen.
But kind of interesting, it does seem, it does seem to have at least some pretty good, roughly Pareto.
capability.
So it may not be the most capable, but when you look at capability and cost, there's a point on the curve where you could quite plausibly put Grok 4.5.
To be fair, in the announcement post, we do have a single sentence saying, we've also added new safeguards reflecting the model's cybersecurity capabilities.
So I think they've addressed it pretty well with that.
One sentence.
That's a fair point.
I retract everything.
And to correct my previous statement, it's one-fourth the price of Opus and one-eighth the price of Fable.
So much, much cheaper than the models from Anthropic.
Also cheaper, I think, than GPT-5.6.
So very interesting point in time where we haven't seen getting into this price for territory, I think, until now.
In fact, with the next model, we'll have even more to say.
So moving right along, we've got Meta, and they have released MuseSpark 1.1, which is essentially just their update of the model that is good at coding, or at least somewhat good at coding.
Enough good at coding to make a claim, but you can use this as an agent and potentially part of your workflow.
They are also jumping into a price wars.
I have to double check the prices, but essentially it's very, very aggressive pricing relative to Anthropik and OpenAI.
It makes sense because nobody would even try this as an option unless there's a strong incentives in place.
And I've not seen many vibe checks of people trying it, but the benchmark numbers are looking fairly good.
And I think it...
Would not be surprising if in fact it is sort of starting to compete at least with open source models like GLM and DeepSea.
Just given, you know, they collect data, they have the infra, they should have gotten to this point given the amount of resources they just shoveled into this.
Yeah, it's pretty interesting when you look at like the broad fingerprint of capabilities that's advertised here.
So the first thing, by the way, from a...
You can tell what part of the model announcements I rush to immediately when I see these.
Cyber capabilities, because that's actually going to determine how much money you can make, it turns out, in this current domain.
And by the way, don't think the bio isn't coming soon in exactly the same way for exactly the same for...
even actually worse reasons because you can't patch human anatomy.
Anyway, so they do these dangerous capability evals.
They say they can't rule out the high risk threshold for both chemical and biological and cybersecurity.
And they say that when they bring in their mitigations, they can get the risk down to moderate or lower, which is, you know, kind of acceptable to release range by their assessment.
When you look at cyber, though, the capability jump is huge.
So Cybench, which is the use.
one of the classic cyber benchmarks, the score goes from 65 to 93%.
That's insane.
Like, and remember, right?
Going once you, once you're pushing 85% or even 80% on a benchmark, depending on what it is.
it gets a lot harder to score that incremental point.
So getting from 65 all the way to 93, this is a really, really big jump.
And this is relative to the previous MuseSpark 1.
So this is true basically for all benchmarks, at least with respect to coding.
All of them have very, very significant jumps.
I guess cyber being a part of that.
So it kind of tracks.
And to your point, they did release...
a hundred page safety evaluation report.
So they do seem to have their safety framework actually being put into practice.
It's weird to see Meta doing more safety work now than Grok than SpaceX, but that's where we are.
Yeah.
I mean, Alex Wang, I guess, is scale-pilled and that's why he's putting it out.
But yeah.
And then, so there's also CyberGym, which is more of a real world vulnerability discovery benchmark.
And that's interesting because, so 59% is a score that MUSE 1.1.
gets on that benchmark.
That is well below Opus 4.8.
Opus 4.8 is like 79, right?
So like huge, huge difference, like 20 points.
GBD 5.5 is like 82.
So in a lot of these cases, you see kind of very uneven footprint here, certainly relative to the other frontier models.
It's impressive, like you said, and I think it is, you know, you can think of it as in some domains, probably GLM 5.2-ish kind of adjacent, which is impressive.
Crappy today is impressive two months ago.
Like, so, you know.
It's worth patting them on the back for this.
But yeah, roughly GPT 5.5 comparable on bio.
So anyway, all of which is to say, I think Meta is going to enter the chat on the regulated AI model stuff if this trend continues, probably sometime in the next like three months, I would guess.
So, you know, this is going to be an industry-wide thing.
It's no longer going to be just anthropic and open AI.
The president of the United States is going to have to stare at the markets and say, not today.
And at some point, that's going to have to happen.
And there's going to be a big market correction.
I say that as somebody who is, in various ways, invested as an angel investor, as an investor in hedge funds, as an investor in various things.
I have stakes on this, but I have obviously more on our survival as a species.
So I hope the president is able to say no to the markets when the time comes.
And it's going to be a really tough...
I mean, then there's the national security picture.
And that is a huge challenge.
You can't say no to the national security picture.
You need to say how.
And that's a conversation for another day.
Yeah, and worth noting too that I think it's fair to assume that there are relationships in place and some people have more leeway.
And I would imagine it's true that Meta as one of the big tech companies with tech in general having made a big turn in Silicon Valley, it used to be the case that Tech was progressive.
Tech was sort of aligned with the left a little bit.
That has changed.
Tech is aggressively aligned with whoever is in power and are very happy to work with Trump and basically cozy up to him and gain favor.
So I would expect Meta in a way to have a bit of an advantage with respect to getting permission to not have oversight and so on and so on.
Going back to the pricing, so they are pricing 1.25 per million inputs, 4.25 per million output.
So cheaper than Grok 4.5 even.
What is this?
1.6 price of Opus 4.8.
Getting towards the territory of open source models, I think potentially even at the same level, which used to be super cheap.
So again, part of this price for trend.
Yeah.
And one last...
thought too, is there's always this question of what's the personality going to be?
Like once you have a new entrant in the frontier kind of space or potential one question obviously comes up like, okay, Grok is, is going to.
you know, do anything for you.
Claude is, well, Claudey.
And then ChatGPT is ChatGPT.
It's going to write you some lists.
The trade-off here seems to be, or the personality trade-off seems to be towards over-caution.
At least that's one part of the personality here.
So the false refusal rates on benign requests are higher than peers considerably in some cases, especially on things like cyber, higher than GPT 5.5 and Opus.
So it definitely, it's tuned conservative, not politically, but like it's tuned to refuse to answer questions when there's any doubt.
I find that kind of interesting.
This is like a complete pivot from like the Yan Lacoon era of Facebook meta to now Alex Wang.
This starts to look a lot more like a, I don't know, like a cousin of Anthropic.
That's a real stretch, but there's a flavor of it here anyway.
Speaking of meta, we had some other news as well.
They introduced Muse Image and Muse Video, their new synthetic media generation models.
A bunch of screenshots and videos that showcase that these models are quite good and potentially competitive with the Frontier.
Also benchmarks showing that they're quite strong.
And Muse Video was just in preview.
Muse Image was actually rolled out and made usable across their platforms, resulting in backlash.
Because one of the things you could do with Muse Video, Muse Image, was use it directly from, I believe, Instagram.
And if you simply tagged someone who has a public account, you could make AI generate images of them.
made incredibly easy to make edits of any account on Instagram.
And the backlash was strong enough that they backtracked the release two days or three days after it came out.
So yeah, kind of a fumble on Meta's part here where they appear to be making good progress.
Muse image and Muse video, as far as I can tell, are quite good.
synthetic regeneration models.
And this is a place where there's Google and OpenAI competing, but Anthropic doesn't focus on this stuff and Not a Banana is a little bit old.
ChatGPT is not updated.
So they could get a real advantage here given the amount of data they have of Instagram.
I would imagine that just the training volume is ridiculous, but the product side release here was a bit of a mess.
And next, going back to models, there's a story.
Chinese AI models gain ground with US companies as cost search.
So this is a bit of a trend check where it's showing that the share of tokens that are coming from these open source Chinese models, DeepSeq and Z.ai has gone up significantly.
It has now gone up above 30% weekly since February, 2025 on OpenRouter.
OpenRouter being one service where you can kind of query and direct your queries to different LLMs.
And there's some data there.
We don't know how many requests are hitting the official APIs.
So again, another trend of what we're seeing where GLM 5.2 is incredibly cheap.
It's still, I don't know if it's cheaper than MetaSpark 1.1, but it's...
you know, same territory of much, much, much cheaper than Anthropic and OpenAI while being very capable.
GLM 5.2 had a very positive vibe check from community that you can basically use this as your cloud code driver for the most part.
And it is sort of like one tenth for ish, one tenth ish for price.
So we have some data here from OpenRouter showing that it is starting to get a factor.
OpenRouter, of course, is...
a bit of a niche.
You don't know how big the impact is, but it's happening.
Yeah.
Keep in mind also that China is going to be clamping down on open source at some point in the next year or two.
So many of these trends are, I think, properly viewed as transients, where it just becomes untenable for cyber weapon models or models that can even be used just to amplify the freedom of the individual.
If you're thinking about China.
in ways that the CCP is not going to like.
You are going to see clampdowns.
They'll be justified in various ways.
They're not always going to be justified.
And it's like, oh, this is, you know, we're clamping down to clamp down, but the effect is going to be that.
But in the transient, this is actually really important.
Like if you're trying to think about an offensive cyber campaign as part of the Chinese state, and you want to like get all up in people's systems in the West, we have agents, like AI agents that you can make sleeper agents.
And, you know, our ability to control those.
those agents is only an increase over time.
And so while you have Chinese agents from DeepSeek or GLM 5.2 running on your infrastructure, that is an insider threat.
And that is a way that people need to start thinking about it.
And I mean, I know that they are in various quarters, but increasingly, even at the corporate level, I think people are going to start thinking about it that way.
Sounds like science fiction, but that's just where all the scaling lines are pointing.
So I think it's an important flag that just in the same way that people are suddenly feeling a shock from over-reliance on the frontier labs and like, oh man, the US government could just like shut down my access.
Like, this is no good.
Yes, they can.
They will continue to do this.
Like, until morale improves, the government will keep clamping down on models.
But the same is going to be true and perhaps ought to be true from the open source side increasingly over time.
You know, if you don't want your IP, your commercial secrets to be like sent overseas, increasingly, you're going to want to like look at a model that was made, you know, with...
the hardware that you trust or software you trust.
Yeah, a couple more things I'll say on this front.
I think it's a nuanced picture and I think it's interesting with these open source models because they are open source.
So you can take the actual weights that are public of the model and just look at their performance.
And this is why having Safety Institute and Cyber Protection Institutes is actually very beneficial, I would imagine.
if there is a sleepwear agent component, well, to be quite the big failure of the safety institutes to fail to detect this potential in these models.
And secondly, because they're open source, I mean, you can align it all you want after with just some post training, right?
So the picture is a bit muddled is all I'm saying is because they're open source, the models don't have to be used as is.
Once they hit the West and there are providers on the Western front of just inference that you would probably go through rather than going straight to Z.ai and so on, or trying to host a model yourself, you would go to Fireworks or Grok.
And they typically put these models on their platform with some caution.
So it's a bit of a nuanced picture and we can expect for the near term for these models to keep coming out and keep providing pricing pressure.
Absolutely on the pricing pressure.
And I very much agree with your structural assessment, right?
Having the weights definitely reduces the risk considerably.
One of the challenges is we start to flirt with, we basically need a solution to the alignment problem at a certain point because we still don't know how to even detect sleeper agents, let alone excise them from these models.
And there's like, maybe that'll change.
But there are a lot of people in the alignment community who actually see the sleeper agent kind of intervention as being equivalent to a big part of solving the larger alignment problem, which a lot of people don't think is theoretically possible, but like whatever.
So we fork out and you're right.
Like there's a world where we certainly benefit either way, having access to the weights, how much we benefit is a really big, challenging, thorny, open question.
And I think ultimately it's a leading question that nation states will be asking themselves as they inevitably start to weaponize these models.
Like it's, you know, this is going to be part of the future of subthreshold warfare.
Yeah.
We have seen, by the way, I mean, this is not just speculation.
We do know that these models, if you ask them about things that relate to state-sanctioned discourse, if you ask about Chinese history, the models are going to give you the Chinese history from the government's perspective in every case.
So these models are very much already being influenced by policy and it's not...
Yeah, I totally agree that there is a real potential for them to be, you know, the sleeper agent thing sounds ridiculous, but it's possible, right?
On to applications and business.
And first, we got meta again.
The news is they're planning a cloud business to sell AI computing power.
This is according to Bloomberg.
There are...
doing this cloud infrastructure business internally called Metacompute that would sell outside customers access to AI computing power and Meta's own AI models, competing directly with AWS, Microsoft Azure, and Google Cloud.
Very interesting, kind of mirrors what happened with XAI or space XAI now, where they were a model company and more so beyond being a model company.
They spent absurd gajillions of dollars building up data centers that were kind of leading cutting edge.
And they built as much computing capacity as was possible, basically as quickly as possible.
And were they able to use all their compute?
Who knows?
It's actually, you know, to ramp up your team and your training processes and the infrastructure and so on and so on and so on.
Having four data centers suddenly doesn't mean that you can utilize all four data centers.
effectively.
And I think there is a case to be made that Meta, as an actual business that needs to make money and doesn't get kajillions of dollars from VCs, is starting to look at their balance sheet and saying, we are spending kajillions of dollars on these data centers, and what are we getting in return?
Nothing.
And that would mean that there's a real case for them to seriously move in this direction.
I like that the word kajillion has now become a unit of measure on the last week in AI podcast.
We've given up now.
The economy is unmoored.
Yeah.
There's a lot going on here.
One piece is, it is absolutely true that meta, so every company has the SpaceX, I'll call it problem or incentive, where if you're trying to compete at the frontier, you're going to try to buy hundreds of billions of dollars worth of data centers a year and a half from now.
You're going to build them.
And then you wait.
And if it turns out that you fucked up and you don't have a genuine frontier model because you're SpaceX, because you're meta, because you're whatever, now you have a bunch of excess capacity and like you don't have an economically valuable use case for it.
What is an AI model?
It's a funnel that turns compute into intelligence.
That's what it is.
And so there's two kinds of value.
There's the funnel and then there's the compute.
And then you put those together.
Some companies have one, some companies have the other.
So meta here.
does not have the intelligence.
They don't have the frontier models, but they have the infrastructure.
So yeah, that pushes them towards becoming a NeoCloud.
But the challenge they have with SpaceX is that SpaceX is differentiated by its institutional and cultural capacity to build infrastructure really, really fast.
Meta has historically not been scale-pilled.
It has historically not been the company that believes in superintelligence through scaling.
It has historically not been the company that makes bets that are of the same shape as the anthropic open AI bets.
And so I think like right now, a lot of the story is framed around excess capacity that Meta happens to have right now.
It's unclear whether they're actually going to like pivot in a long-term capacity.
And the same question, by the way, has been raised about SpaceX.
Colossus 1, they rent that out.
Colossus 2, they give it yes to Cursor, but then they acquire Cursor.
And so like, yeah, it's kind of in-house.
Maybe they are trying to- Well, data centers in space, right?
That's- That's an interesting.
Yeah, exactly.
So the question is like whose model runs on an infrastructure?
And what's becoming pretty clear, and that wasn't at least, certainly wasn't obvious to me like three years ago, was that we may end up living in a world where you have- competition among the frontier labs at any given time to figure out who's got the best model.
And then once it's clear that, let's say, Claude is the best model, now everyone reorients their inference compute to serve as Claude.
Because that's just like the most financially, like that's the highest ROI thing.
From the market standpoint, it's just rational.
Now, everyone who wants to be a frontier lab continues to compete along the training compute axis because they still, they want to get that best frontier model for the next generation.
Two weeks later, we want everybody to be competing to run our model.
But you could imagine at least the incentives, one part of them that is legible to me, at least right now, points in this direction where you actually might have more fluidity.
Everybody's competing to build compute.
It's unclear who wins, but whoever wins, they'll be happy enough to like have their model run on other people's infrastructure and rent that.
And everybody else will be happy to amortize the cost, the ungodly eye-watering cost of this infrastructure.
Like people aren't going to pay us to use our shitty model.
We might as well like get money from the leading lab.
so that this whole thing wasn't a giant waste.
And so the question really is, where does that margin go from the actual model training part of the stack?
And that's where, you know, for a while, whatever company happens to be in the lead at that time gets to just earn a lot more margin.
And so I think that's potentially part of this.
You could ask yourself, like, what does meta have over a typical NeoCloud in this space?
And like.
The answer isn't great.
They don't have like the TPU.
They don't have like these massive, like internally developed, really high performing ASICs.
What they have is financial capital.
They're less risky than a core weave because they don't have like crazy debt to equity levels.
They have an actual cashflow business in advertising that can massively support this.
And so if they wanted to, literally just because they're a floating giant ball of cash with a decent technical bench to them, they actually could move into this market.
It's just a question of like, how scale-pill they are and where they see themselves stacking on the kind of model part of the stack.
Yeah, I think I'll say Meta, to me, looks like they have a little bit more of an advantage in the sense that they're a mature company that has data centers for their websites, right?
So they are in the data center business, broadly speaking.
They have, however many, they have built many of them.
So they have the structures in place.
to do that and the talent in place.
They also have worked on custom hardware for inference, which we've discussed.
I think it's moving along to a decent place.
The real challenge is they are not in the cloud business and have not been in the cloud business.
So they're competing with Google Cloud, AWS, which have been clouds for a very long time and have served GPU capacity for a very long time.
One other thing I'll say, is to your point of we've seen this play out earlier this year.
Anthropic had explosive growth with cloud code, explosive.
I forget it's like 10x, 30x, ridiculous in the span of months.
And they ran out of compute.
It was like a ridiculous situation where suddenly you could barely use cloud code if you were on the $20 plan or whatever, because you constantly hit limits and the models got worse.
It was very, very obvious.
They just did not have the compute to serve the demand.
And they had to scramble to make a deal with XAI and so on.
And if the projections about GDP and AI adoption and AI capabilities hold, where we have another 30x jump in demand in like a year, which is plausible if you think that cloud, co-work and so on will continue to be as good for professionals as they have been.
so far, the picture of being a provider of a dental center starts looking very appealing because Anthropic will have to pay you whatever you're asking to continue to serve their customers.
Yeah, you're right.
And it's sort of this game of chicken, right?
That they're playing.
It's like, it's a horrifying proposition for both because while Anthropic is absolutely desperate to open their mouth wider to be able to suck up all the sweet users, that sounds horrible.
Anyway.
You know, you know what I mean?
Yeah.
Let's pretend I didn't say that.
But like, just as Anthropik's desperate for that, on the back end, you're the, let's say you're SpaceX, like you're sitting on these data centers that you built.
You're sitting on a massive capital outlay that you just put out and it's not making you any money.
And it's depreciating rapidly.
Exactly.
Exactly.
So you're actually like in a game of chickens.
Like nobody wants that to go on for very long.
If you don't make money from it in a year.
Like you were screwed.
You messed up.
GPUs are outdated.
Like you wasted your money, right?
If you have a gigawatt cluster, you get a hundred billion dollar, like a gigawatt cluster, roughly ballpark is a hundred billion dollars or so.
If you have that just like hanging out, like its lifespan is like, I don't know what, call it like 10 years or something like order magnitude.
So like, yeah, yeah.
Why don't you go and like light $10 billion on fire?
You're obviously not going to do that.
And so.
you're going to make whatever deal.
Anthropic wants the benefit, but you're dealing with the loss aversion at that point.
And so anyway, it's going to be an interesting economic time.
That's why I think the economic force is pushing for this kind of, call it like compute infrastructure fluidity, where it chases like whoever happens to have the frontier model in that moment.
The argument I think is pretty strong.
I'm very curious what that adds up to for the neoclads and stuff, but it seems like it's there.
And I think this also adds, interestingly, to the price war picture where to do a price war, you need to be able to drive down your costs, ideally, to be able to make this profitable.
And one of the interesting things with Entropic is it appears that they have become profitable.
They've managed to make this business model work, but with rather expensive models, the most expensive models on the market.
When you have more supply of data centers, and this is just speculation on my part, but you would imagine that if you have more supply of compute in general, hitting the market.
And I think all these companies, OpenAI, Beta, all of them have made very, very deep commitments in CAPEX to continue building data centers.
So I would imagine the compute is coming online and will continue to come online in the next year or so.
It's a staggered process.
So as you get more of these data centers finished up and ready to serve demand, you have more supply of compute.
And that means that the tokens can be produced probably cheaper.
And that means you can charge less and great.
Now it can compete in a price war.
And we consumers win basically, but let's see how it goes.
It also, it depends on so many factors, right?
How does dollar denominated demand track with?
the actual intelligence per token?
And then also, can you grow the data center base fast enough for that increased demand?
What Claude suggests is actually, in some cases, you suddenly hit a point where the answer is no.
And so this is really, it's hard because it's a function of what new companies you can create as a function of breaking through those new levels of intelligence per token.
And so a lot of this is just almost literally impossible to predict.
This is what markets are made to discover.
But yeah, it could work out in either direction and lots of room for speculation.
Beauty of markets, discovering what prices should be.
Truly a miracle of, I don't know, modern business.
Next story related to data centers as well.
Like U.S.
energy regulator sets ultimatum for data centers.
So this is FERC, the U.S.
energy regulator issued tailored shock show cause orders.
under some section of the law to each of the six regional grid operators under its jurisdiction, directing them to justify or reform the rules that govern how the data centers, manufacturing facilities, and other large energy users connect to the electric grid.
And this is a very notable point.
Jeremy, I'm sure you know more about this.
At some point, the data centers are...
exceeding what the energy grid can provide.
In the case of XAI, they have much gas turbines that are spewing pollution and so on that are operating to produce electricity.
So yeah, what is the picture here, Jeremy?
Yeah, I think you hit the nail on the head.
A lot of these data centers increasingly are moving to behind the meter energy generation, which means they're not on the grid.
And this is happening really as much as anything, partly it's happening for political reasons because It's untenable to have people's energy prices go through the roof and then you get political pushback and blah, blah, blah.
Like no data centers in my backyard, Bernie type stuff.
But then also there are other advantages to it.
And so now, yeah, regulators, energy regulators, like PJM is an energy, is a grid operator that recently had to basically order generators to run at maximum output and bring idle power plants online because there's a heat wave.
And they couldn't support the heat wave because of all the new load on the grid.
So this is having an impact where you're flirting with brownouts right now.
And if you plot out the rate of energy consumption from data center demand, it's like right now it's in the single digit percentages of total US energy production.
That is only going to increase.
And so you're going to put stress on the grid and eventually you're going to find brownouts become a thing.
And there are all kinds of ancillary issues where the regulators are now getting their feathers all ruffled.
One example, and we're doing a pretty deep dive on this right now.
It'll be public in the next couple months, probably.
But one thing to keep in mind is like, so a lot of the infrastructure, the energy infrastructure that these data centers use is like designed to trip on the same signals.
And so if you have, you can imagine like basically a correlated risk where one kind of signal on the grid basically shuts down a whole bunch of data centers at the same time.
So you've got this on the one hand.
like super high correlated risk because this has been a totally unregulated market in a lot of ways or in this way.
And at the same time, you've got the risk of brownouts and things like that.
The regulators, like Department of Energy is going to have to step in.
They're going to start to insist on big changes.
And so again, this is one of those things where the past trends are real, but also like we're going to learn a lot about the impact of government regulation in the next couple of months.
By the way, does anyone remember climate change?
that the climate change is a thing and pollution and carbon.
Now, I guess it's just me.
Yeah.
Not helping all of this stuff.
Yeah.
I mean, you know, and I think it's the thing that you could make the argument that like for almost arbitrary climate related problems, having better AI that can help us solve these problems.
is a thing, whether it's anthropogenic or not also, because like, you know, either way, if they, the heat, you know, it's increasing and shit, like.
Carbon is science.
You produce more carbon.
It's gonna make things worse.
And, you know.
But just like you could find ways, like regardless, like you could find ways around it that are more creative with it, like blah, blah, blah.
Like, yes, yes, yes.
But it is funny how the discourse is like, it's moving along.
The point also is, I guess, if you may have a rush to add more capacity and build data centers and so on, you're not going to get clean energy, renewable energy to drive these data centers.
And that's already been shown to be the case.
Google, Meta, et cetera, have made deep investments in...
powering their data centers with renewable energy.
And that is sort of like that wayside.
It's not a big priority.
Well, nuclear.
Nuclear is coming online too.
I'm a huge fan of that.
I don't know how the hell we got it in our heads that nuclear is bad, especially given the- Politics.
Yeah.
Yeah, yeah, exactly.
I mean, it's- Yeah, it's starting to actually, I don't know how far along that is.
We've covered many times sort of mini nuclear and sort of promising looking technology, but is it getting to a point where it could be powering data centers?
Yes, there are a bunch of data centers that are like have plans for nuclear buildouts.
SMRs are probably going to be the biggest kind of sort of shift.
Expect the first SMRs, these are small modular reactors.
You can expect the first to come online sometime.
The most optimistic I've heard people like, who are actually like in the data center building space, say, is like 2031, maybe 2030.
So it's like, you know, if you believe in Leopold and whatever, AI 2027, like we're way into super intelligence territory by the time that matters.
But who knows?
Yeah.
And, you know, how much does super intelligence help you with stuff, building nuclear safely?
Anyways.
Onto projects in open source, and we do have a couple of interesting new models that are open source, starting with Nemotron-Labs-Diffusion, a tri-mode language model unifying aderegressive diffusion and self-speculation decoding.
This is from NVIDIA in their Nemotron model of family of models.
And the short version per the title here is...
So you can do decoding, you can do language models in different ways, right?
The standard way is autoregressive, meaning that you output one token at a time effectively.
So you provide the input, you give the output, and you keep sort of doing this loop.
Diffusion, as we've discussed many times, is something typically done with images where you sort of like produce the entire thing at once over multiple steps and sort of make it.
better and better.
And you could do it with text, interestingly.
So instead of going left to right word by word, you kind of produce the entire paragraph at once and denoise it until it's correct.
And then there is self-speculation decoding, which is kind of like standard auto-aggressive, but you have within the model some shortcuts more or less that let you be quicker.
But diffusion also can potentially be quicker.
So in this...
try mode approach, they trained a single model that can do any of these and also has been trained on each of these objectives.
And it allows you to switch between causal and bidirectional attention, meaning that you can use these two training objectives.
And they're releasing here base, instruct, and vision language variants at 3 billion, 8 billion, and 14 billion parameter models.
The models are very quick.
So 6x more tokens per forward pass than, for instance, QAN 3, 8 billion while being comparable.
And I don't know how we got to this point where NVIDIA has produced the most interesting research on neural net architectures and potential alternatives to transformers.
I guess we've seen from academia, Mamba, and so on, but as far as large-scale AI model demonstrations of something that could be better than transformers, I mean, this is pretty exciting stuff.
Yeah, I mean, coming from, I guess, a company that knows the hardware so deeply.
If your hypothesis is that progress is determined by hardware, this may feel like a natural place, but we just hadn't seen it to your point.
The last time, when we started covering the big NVIDIA open source releases, I think you had to go back years to Nemetron, Turing, and LG, the kind of big Microsoft NVIDIA collaboration from, I want to say 2021 or something, was the latest, the biggest case that you could point to.
And now suddenly here they are.
This is a really interesting paradigm.
I mean, as you say, it's this combination of this, you think of it as like autoregressive modeling left to right and the diffusion piece.
And they do a couple of things to kind of make it all work together conceptually.
So the first is they do still have to use just standard traditional autoregressive pre-training to get the model.
up to speed, right?
To set its priors.
And then they turn on this dual track auto complete and diffusion thing.
But you see that a lot.
Like there's often a need to just use autoggression to get these things off the ground.
Once they turn on though, this dual track thing, they do it in blocks.
And so instead of having your entire output window.
The way diffusion works is you kind of start with random noise for all the tokens in the output field, and then you kind of gradually populate them and get more confident as you do more steps of diffusion, and you zero in on what the text needs to look like.
And of course, you get the benefit of being able to look at the whole body of output text at the same time.
Whereas with autoregressive modeling, the challenge is once you choose a token, now you're stuck with that token for the next one.
And so if you go off in the wrong direction, you can't go in like...
20 tokens later be like, ah, that's no good.
And so you'll see the models kind of doubling back a lot as a result of that, because that's the only way they can get themselves back on track.
And so what they do here is they're going to do what they call blockwise diffusion.
So you split the sequence into blocks and the model denoises one block at a time.
It kind of treats the earlier blocks as clean context.
And so it's bidirectional.
It looks, in other words, at the whole chunk of text within a given block.
But it does lock in past blocks as it goes.
So it's causal, it's sometimes referred to as causal.
It's kind of like a mix of the two, basically.
And this is something we've seen already as a potential pattern.
But this is, to my knowledge, the biggest demonstration of this idea and in general, this hybrid approach.
Yeah.
And they even have this interesting...
So the backbone of the Transformers, the residual stream, it's kind of like the...
the vector that gradually gets modified at each layer as more layers add sort of more, I'll call it context, but more information to the vector and let it evolve.
And what they have is what they call dual stream attention.
So they have this like this clean stream that's just causal.
It's just used for the autoregressive part of the modeling.
And then they have a noisy stream.
which they use for the diffusion loss.
And they're processed together in the same forward pass.
And so both of those objectives, the diffusion objective and the autoregressive objective, they're computed at the same time.
And so that's how you kind of get these to work together.
They find that there's this like number that they use to vary the balance of how much autoregressive to how much diffusion to put in the objectives, which is a very common thing.
But the interesting thing is that they find that the losses peak at the same.
They kind of like rise and fall together, both the diffusion and the autogressive losses, which is not intuitive and very interesting.
It suggests that they're complementary rather than fighting for model capacity in some sense.
Kind of interesting across the board and conceptually, I think one of the deeper attempts that I've seen to marry these two concepts together.
And one more open source release, this one a bit less to say, but still interesting.
Tencent has released high free and opened 295 billion.
mixture of experts model with only 21 billion active parameters.
So the short version is, this is not as good as GLM 5.2, but pretty decent, pretty decent intelligence.
And it's smaller.
And it's just another model that's like pretty solid.
It's completely open source.
It's pretty cheap.
You can use it potentially even in a home compute cluster.
So the models for now are continuing to be released, but they may or may not keep doing that.
On to policy and safety.
And we begin with some interoperability research effectively from Anthropic and potentially some kind of safety philosophy, whatever you want to call it.
And this is, I think, the second most exciting news of the week in addition to the model releases.
So we'll probably spend a small amount of time getting into the details.
So the name of the research for Anthropic is verbalizable representations from a global workspace in language models.
And I'll try to provide the correct TLDR and see if I can manage it.
So they begin Anthropic with kind of laying the groundwork conceptually of what workspace is.
And this is a model that has come out of modeling consciousness in humans where...
One aspect of it that you can pinpoint to is consciousness effectively is attention, mirroring attention, kind of directing it.
And you have this sort of workspace where you slot in some stuff to be consciously aware of.
Most of the processing of information, sensor data, et cetera, is unconscious.
You're not aware of it, but it's happening.
And then you kind of bring in bits to be looking at and thinking about.
Like the active thinking you're doing, the active verbalization of things and things you're keeping in mind to draw on kind of at the tip of your consciousness is this workspace notion where a bunch of stuff is happening that you're not conscious of.
And then there's this thing that you can kind of use consciously.
And that's effectively what Anthropik is at least comparing this method to.
So the gist of the method is this kind of idea of via some cool math that relates to Jacobians and whatever else you want to get into.
The gist is you can find the tokens that the model seems to be kind of potentially ready to say that are on its mind, so to speak.
So this is very different from the traditional interoperability technique we've talked about a lot where you can look at.
vectors within a model that you can try to map to concepts.
The difference here is instead of looking at vectors that sort of roughly map onto concepts, which is sort of this like unconscious, diffuse, messy kind of soup of stuff going on inside your model, this is telling you for a given token or given token sequence, this is sort of what explicitly in terms of tokens, in terms of words, is going on inside the model.
And there's a whole lot to say about the technical aspects of this, the interoperability aspects.
We will not be able to get into all of it, but I'm sure Jeremy, you have a bunch more to say.
Yeah.
Everybody be prepared to roll your eyes for the annoying Jeremy Sechelberg.
The classic things that you see.
So imagine every, we talked about the residual stream being the backbone of the transformer, right?
This like vector that keeps getting updated for a given token, every layer, until finally you get to the last layer.
And at the last layer, that vector gets decoded into a probability assignment for every possible next token.
So you have this vector, this list of numbers that in some sense encodes the meaning of the vector you're about to predict.
But then you need to multiply that vector by a matrix, a decoding matrix that turns that into concrete probabilities for the word the, the word apple, and so on.
And so historically, what people would do to try to answer the question you just raised, right?
What is the model thinking at some middle layer?
which is kind of where you want to, if you want to monitor the model for suspicious, sketchy planning, if it's going to kidnap your children and murder you in a dark forest, you want to look and be able to tell, is it thinking those thoughts somewhere intermediate in its intermediate layer?
Basically, what is it not telling you, right?
Inside of it, aside from the output, what is going on?
Exactly, exactly.
And so one historical naive approach people would take is they would take the decoding matrix at the end.
They would literally just like, use it to decode immediately whatever the residual stream was or the residual vector was at any given layer.
So let's look at layer number 27.
Let's just whack it with the same decoding matrix that we would use on the very last layer.
The problem here is that that decoding matrix was never optimized to work with that particular layer.
It'll give you probability assignments over next tokens, but it's not obvious what the hell those are actually supposed to mean.
So somehow what you actually want is the ability to say, okay, If I took this residual vector at layer number 27 and I modified that residual vector a little bit, what would the impact be downstream on the final predicted token?
Let it propagate all the way through those other layers.
And like, if I just modify, you know, the residual stream at layer 27, how then does that change the output all the way up to layer 78 or however many layers there are when it actually gets decoded?
And instead of whacking it with a matrix that was only optimized to work with like layer 78 or whatever, instead, try to come up with some layer that approximates all of the shit that happened between you and like that final layer.
That's what this is.
So that's what the Jacobian here is doing.
It's an approximation and a pretty good one, a pretty principled one of all the crap, all the transformations that happen between, you know, call it layer 27 and the final layer and the output.
And so it actually does give you in some sense, the best estimate that we have as to what the model was thinking, how it was, let's say, how it was trying to nudge the final out.
Yeah, I think the verbalization aspect of this, because it is nuanced, sort of.
You can look at the embeddings, you can look at the activations at layer, you know, whatever, and sort of say, what is these activations telling us?
And that's the traditional way.
So the way this is different from the traditional interoperability techniques is this verbalization aspect where you look specifically at the token outputs, the words the model would lead to as opposed to whatever soup of stuff going on inside of it.
And at least the argument that Anthropik is making is that this is akin to this workspace model and is actually interpretable as something akin to what the model is conscious of.
Whereas true is a whole other topic with different perspectives, but it is a compelling and kind of intriguing argument.
Yeah, exactly.
The connection here to the whole consciousness thing, because it may not be obvious from our description itself, like how does this actually tie into that?
There's a couple of pieces of evidence.
So one, if you ask the model what it's thinking, okay, so if you look at the space of words that the model is thinking about at layer 27, that you can figure out using this technique.
If you swap one sort of one vector for another, so, you know, if it's thinking about soccer, switch it for rugby, that will actually result in changes downstream that match exactly that.
So you can do these pretty surgical interventions.
So this means that there is like some like causally connected thing here.
They also do stuff like tell the model to not think about certain concepts.
And they find that actually these intermediate layers are forced to think about the concept, which is sort of funny.
It's like humans.
If I say don't think of a pink elephant, like.
First thing you're going to do is think of being elephant.
But the key thing is, so the argument for the consciousness thing is that there is a kind of reasoning that we cannot access, but there's also a kind of reasoning that where basically like all thoughts get promoted to some shared reasoning space that the full human brain can access.
And that kind of shared workspace is what consciousness is about.
And this is essentially, it allows them to sort of do a...
a dissection of a language model and show that something similar is happening there where you actually have layer, like the earlier layers, just like the model can't access the thoughts that are going on there.
It's just too early.
The processing of the residual stream isn't yet to a point where it's thinking coherent thoughts.
The thoughts are too raw.
But once you get to those middle layers, you actually do get to that workspace.
And they're able to show through a bunch of experiments that this kind of maps onto intuitively how you might think about that in the context of.
the workspace associated with consciousness.
So quite interesting, quite deep when it comes to that.
I wish we had more time for this one, but hopefully this gives a general sense.
Yeah, honestly, we could spend like two hours deep diving deep into this and maybe we need to do a probability episode.
We keep saying we'll do one.
One day we'll do a deep dive episode again, but it's hard to find the time.
On the consciousness piece, so I think it is...
Important to note nuance here where there's kind of a few things you might be referring to when you say consciousness, right?
There is sort of the functional aspect of consciousness, which is like, there are thoughts, there's like a process going on.
This is how consciousness and thinking and reasoning is working.
And global workspace theory is part of that.
It's sort of trying to map out, okay, here's how thinking happens, right?
A completely different aspect of it is like the experience of consciousness with kind of phenomenal aspect of like, I feel that I'm conscious and I'm aware of it and so on and so on.
And these two things are distinct.
We don't know why, you know, we know our brain is doing stuff.
We know that thinking is happening in some sense, reasoning, data processing, why that data processing seems to result in our kind of conscious, experience of the world is the deepest, hardest problem of philosophy, the hard problem of consciousness.
Anthropic is not making a statement as to that case.
So I think there's a lot of backlash against anthropic whenever they do this kind of stuff, which I'm getting very tired of, of like anthropicators being like, anthropic always mentions safety, always goes on about consciousness.
But when you say consciousness and global workspace theory and so on, What we are claiming here is that functionally it looks similar and you can make a case that there's these kind of like thinking-esque, workspace-esque mechanisms within the model, which is plausible, right?
Because for various reasons, at least you can say we have this model of how thinking happens in humans.
There might be a similar model you can apply to AI.
And this is an intriguing aspect.
Thing you can argue about, but yeah, very interesting kind of new tool in the interoperability toolbox.
And I think another pretty powerful tool potentially if it turns out to be reliable.
Next, Beijing is looking at curbing overseas access to China's top AI models, according to sources.
So Chinese authorities have held meetings over the past month with top tech.
firms, including Alibaba, Bydance, and Z.ai, about potentially restricting overseas access to China's most advanced AI models, including those not yet released.
This is according to three anonymous sources.
These are led by China's Ministry of Commerce.
All secret, but wouldn't be surprising, right, if this were to come to pass.
China has nothing to gain.
basically from releasing open source models to the rest of the world.
So we are getting some indications of it at least being floated and the pieces being put in place.
Yeah.
I mean, we talked about this idea of sleeper agents, and I think that will start to...
I mean, if you start to see the Chinese, the CCP, actually endorsing the continued release of very powerful open source models, that should increase your baseline belief.
in the probability that there are sleeper agents built in there or that there is some kind of game afoot, it doesn't mean that it's guaranteed.
It's just like, just think about the incentives and that's where you land.
They do want to have more market share, obviously, on the open source side.
It is helpful in a whole bunch of ways.
But here, the Chinese Communist Party is clearly.
making that calculation themselves and going, well, you know, we're not so sure that we like this.
So I will say, you know, for people who model the Chinese Communist Party as being this benevolent, this sort of like benefactor of the open source space and how wonderful it is that all the open source models are Chinese.
Yeah, I guess hold your breath for the next two years because you're going to end up seeing a lot of similar moves just as everybody realizes the strategic significance of these models.
And again, it tells you something if they keep.
publishing this.
That tells you only one thing.
I mean, could tell you other things, but it may tell you something quite interesting about what they're putting in those models.
So, you know, we don't know how the restrictions would work.
There was a May round table of a bunch of Chinese legal experts who produced some kind of summary of another proceeding.
And in that case, they were talking about a tiered system with like basic open source models that would be subject to simple filings.
You'd have more advanced technology.
that would face security reviews.
And then the most sensitive frontier models would just straight up be barred from public release or restricted to domestic use.
So everyone seems to kind of be converging on the same obvious truth, which is you can't have weapons of mass destruction in the form of like really, really powerful AI cyber and bio models just in the hands of random people.
And again, it was foretold five years ago by a lot of people.
And a lot of people said it was ridiculous at the time, but like, if you just kept plotting the scaling curves, we were always going to get here.
So this is in some ways the least surprising, surprising news of the week.
Right.
And I think even aside from government interference, DeepSeek is looking to IPO.
Tencent and then Bydance are massive.
So we might just see the same thing that happened in the US happened in China, right?
Where at some point your models are good enough where it doesn't make sense to open source anything.
You're competing with other Chinese companies.
And if you have the best model, you keep it to yourself.
And so either way, I think this open source gravy train is probably going to end as it has in the US.
And next up, the ex-OpenAI employee behind AI 2027 recommends a rosier path.
So speaking of China, this is about AI 2040 Plan A, which per the title is from some of the same team who did AI 2027.
So quick recap, AI 2027 is kind of a narrative that tried to sketch out why you should be afraid of misalignment and potential X-risk and generally how things would go that could lead to a catastrophic outcome or potentially a good outcome.
And it made a bunch of predictions in a sort of narrative framework, some of which have been very accurate about, for instance, the rise of coding agents, governments starting to play an active role in mid-2026, things like that.
And AI 2027...
It does predict a takeoff scenario where at some point we get a course of self-improvement, something, whatever you want to say, and then the models get crazy, crazy good.
Everything is turned upside down and AI can kill us all, right?
So this is the follow-up to that, that kind of tries to lay a narrative of a path where AI maybe can not kill us all, ideally.
And the gist of it- That's the pitch, folks.
That's the pitch.
And the very high level plan is like, okay, so we want AI not to kill us all.
We also expect AI to become super intelligent, probably.
So what we need to do is make sure we have alignment figured out before AI becomes too hard to control and can kill us all.
And the way they say we need to do that is slow down development.
You need the models to not self-improve and explode in intelligence super quickly.
We need to be like, don't make us smarter until we figure out alignment and then they can be smart and solve all our problems.
And most of this AI 2040 thing is devoted to, well, okay, let's say we want to do that.
How do you do that?
And in particular, how do you do that in both US and China?
Because You can do something in the US, but then if China keeps racing forward, then US won't do that because you don't want to stay behind.
And so a lot of it is spent kind of arguing or laying out a story, a narrative of potential cooperation and mechanisms to slow down AI development, which have things like monitoring, access to compute, like pretty strict, direct government oversight of AI model development.
A bunch of responses to this, AI 2040, similar to AI 2027.
Again, could go on for an hour and a half about it.
Higher level, it is interesting to read about, to see one sketch of how people are looking at the future.
And if you are a believer in the fast takeoff scenario, then it's probably nice to read about a potential optimistic outcome.
Probably an unrealistic way to see things in many ways.
But hey, I mean, it's good to be optimistic and say maybe AI will not kill us all.
Yeah, absolutely.
And so full disclosure here, my co-founder and brother Ed and I both had a look at AI 2040 before it came out and gave some feedback on it and stuff.
We have our quibbles with it, as everyone does.
I think the top lines generally make sense.
I think this is a really thoughtful team that's worked on this.
I mean, Thomas Larson and Daniel Cocatayo.
in particular, who I've just known them longer than the other folks who co-authored it, but very, very thoughtful people.
They've been right.
I mean, if you look at AI 2027, more than anybody else at the level of detail that they've offered, they've knocked it out of the park.
There's weird levels of correspondence that we see between what's happening right now and what they predicted.
And AI 2027 is a while ago.
I forget when, but it was like early 2025-ish, maybe.
Exactly.
A while ago.
Yeah, it was something like that, where to the point where it's like, you know, people were laughing at it in ways that now are just like, oh, I guess the US government is just like doing this.
Okay, yeah, like wake up.
That's like, wake up and smell the scaling curves.
That's where they've been pointing.
The funny thing with scaling curves is that something that compounds like 10Xs every year is something that like where, yeah, the 10X will actually happen.
And like, it doesn't do you much good to believe it.
When the numbers are small, it's when the numbers are big that you're actually testing how good is your kind of empirical extrapolation.
So anyway, so it's aged very gracefully.
What they do is they lay out a couple scenarios.
We'll go into detail, but like they've got this plan A scenario that they've presented in a lot of depth here where they're talking about, yeah, the U.S.
and China get together and they're both really scared about loss of control and other things.
So they actually set up this like international way of coordinating around this stuff.
They've got a plan B that's like China won't go willingly.
And so we need to make them slow down.
Plans A and B together are like exactly what we've been focused on over the last like several months as we've been talking to like literally the diplomats who led U.S.-China engagement on WMD risks to see like what is China like at the negotiating table, what's realistic, but also some of the hardware like compute assurance people who.
They've obviously spoken to a lot of the same people.
They're proposing the same solution of network taps and recomputations.
The trouble is when we take that to the intelligence community and ask them about it, it kind of looks like a bit of a non-starter.
So there's a bunch of things here where we're going to be coming out with our report pretty soon.
Our differentiator is our ability to actually talk to the intelligence community, to talk to the diplomats, talk to the builders of the data centers, and to take these kinds of scenarios, just like we did with situational awareness.
and be like, okay, but how does this make contact with the actual reality of institutions and sort of special operations and that sort of thing?
So we're going to be doing the same thing with this by accident.
So I'm not going to kind of like give my full take on this, but I think it's actually really, really thoughtful.
It does an excellent job.
And any quibbles that I might have with it are like...
within the range of like things could go one way or the other and I don't think that they're meaningful.
So they got a bunch of, you know, plan C and plan D.
Things get increasingly shitty as like different governments don't take it seriously enough early enough and then we end up in really bad positions.
It's worth at least skimming, especially if you are compelled by what AI 2027 said and how well its predictions have aged.
I would expect this to age similarly gracefully.
So check it out.
I highly recommend it.
More than anything we've covered in the last sort of like three weeks or so, I would recommend taking a look.
Yeah, exactly.
I think if you look at the discourse as with AI 2027, the mainstream, there's a very strong tendency of kind of the mainstream of AI commentators, whatever you want to call them, AI community roughly, to be at the best dismissive of these kinds of discourses around X-risk and around catastrophic outcomes, around super intelligence.
a lot of skepticism in some cases kind of constructive pushback.
In some cases, just like making fun of it and being like, this is silly.
And in many ways, it's been shown that kind of the dismissal of these kinds of safety concerns is wrong.
Like these are real, yeah, real things to consider.
I am someone who is more on the like extra skeptic side.
I think both.
AI 2027 and AI 2040, very heavily lean on a takeoff scenario of exponential self-improvement.
And that's the key question.
There's no question we get to AGI, but does AGI get us to SAI and does that get us to models that are impossible to control?
I think nothing in our current scaling curves or science really tells us much about fast takeoff, is my personal stance.
But you can make arguments either way.
Anyway.
And that'd be a good episode for us to do at some point.
One of the things that I appreciate about the podcast is it's like there is, you do get like one take.
I think we're both like, I may be more actually outright like ex-risk pilled, but you certainly are more kind of a moderating influence in that respect.
I think that's important because it's the reality has been rough around the edges.
Like one of the key things that people like me have been wrong about is if I had been correct, you know, five years ago, we'd all be dead by now.
I think that's very fair to say.
There are other aspects of the story that I was telling myself that sounded like science fiction at the time and were exactly right, like the mythos thing as an example, but it's never as clean as you want it to be in either direction.
And this is just, we have to proceed accordingly.
And it's not obvious what that means.
Yeah.
The other thing that Silicon Valley is very bad at is converting the technology progress into societal impact.
And it's very easy to overestimate.
The style of the back.
So, you know, if you were, I would have to go back and look at the AI 2027, but part of these projections is like insane changes in GDP and everyone being automated and every job being automated.
And often it's a bit slower than you would imagine.
I think they're pretty much on point, weirdly on the GDP and workforce side.
The big numbers at this stage, as I recall from AI 2027, are more like a CapEx spend.
That's the stuff that sounded crazy.
And now it's just like, yeah, you know, of course we're spending like trillions of dollars on CapEx.
Like, why wouldn't you?
Yeah, that's fair.
That's fair.
But like GDP beyond just like building more AI, I guess.
Yeah.
I can't remember if they made this specific prediction, but it's in the same ballpark.
It's like basically AI is the dominant driver of US GDP right now.
In the same way, exponentials always look cute when they're young and then they become monsters very quickly.
GDP growth, right?
Yeah, GDP growth.
Yeah, yeah.
Sorry, sorry.
Yes, GDP.
Good point.
But that's, I think in line, I'd have to go back and check, but like Daniel posted recently, I think he said something like by his estimate, we're about 75%, running at about 75% of the speed of AI 2027.
And that matches my rough hand wavy sense of it.
But there are quibbles at the margin everywhere.
Yeah.
And we're going to have to close it out to very real.
Lighting paste episode.
So much fun stuff to discuss.
Thank you so much for listening to this week's episode.
As usual, caveat for me, I'm sorry for the releases being a bit choppy and the timing.
It should be getting back on track as my startup gets a bit less crazy.
We appreciate you listening, sharing, rating the podcast, commenting.
We'll respond to comments probably next week.
And more than anything, please keep tuning in.
is reaching high he has reaching high data driven dreams they just don't stop every breakthrough every code unwritten they should change excitement with futures unfolding see what it brings