# AI Monetization, Chip Supply Shifts, and Startup Drama

**Podcast:** Last Week in AI
**Published:** 2026-01-28

## Transcript

Hello and welcome to the last week in AI podcast where you can use chat about what's going on with AI.
As usual, in this episode, you will summarize and discuss some of last week's most interesting AI news.
And you can go to lastweek in dot AI for the text newsletter with even more news coming to your inbox every week.
I am one of your hosts, Andre Karenkov.
I studied AI in grad school and now work at the startup Astrocade.
And I'm your other co-host, Jeremy Harris.
So, you know, Gladstone AI, AI, National Security, all that stuff.
Which, first of all, we always say this.
We always say this when we start a podcast.
We're like, it would be great if we could wrap a little bit early this week.
And then there's an ideal gas law thing going on where like the amount that we will explore stories will expand to fill whatever time we conceivably have up to the aggressive outer limit of what is okay.
So we've both been late to meetings because of this.
It's part of the fun.
The number of times you finish like two minutes early, one minute early, a few minutes early.
It's crazy.
I don't know how we do it.
Probably by like the last five stories or so, we just start like hurrying up.
I wonder if that's it.
Yeah.
If like you start listening to the podcast, like, well, this is nice and slow, but then you find yourself moving us from 1.5 to like 0.75 or something.
But yeah.
So I asked Andre at the B.
Hey, if we could finish 15 minutes early, and we looked at each other like that's never gonna happen.
Well, we can try.
We'll see.
Sometimes, sometimes we make it work.
Uh despite our uh instincts.
Other thing too, I know you mentioned we we talked about the comments just at the tail end of last episode, and a bunch of people left comments on like the YouTube video, which by the way, I I love those.
It really gets me jazzed.
So I appreciate that.
I know, Andre, you do too.
So thank you for for all those very kind comments, guys.
And speaking of which, we uh do have one comment, at least that we are gonna address at the beginning of episode uh this episode, uh, just to give some preview.
We're gonna be talking a lot about open source model releases, a fair number of papers, a lot of business stories with some drama and a lot of fundraising.
That's pretty much your makeup of this one.
And so, related to some of the safety topics we'll be discussing this poster, Prince Megahertz, which is a pretty cool handle, asked about the quote here is in some of the earlier episodes, you got into details about the risk assessments, and one of the categories was authoritarian lockdown, and they would like to know with recent events in mind what the state of this is, and whether teams at X, OpenAI, and Tropic, and so on, still screen focus.
So, Jeremy, as we safety hawk, maybe you want to take it.
As the safety hawk.
Yeah, it's an authoritarian lockdown is we were just talking about this earlier.
Is this this whole idea where you know historically you think about all the uh authoritarian regimes that we've had on planet Earth, and there have been many, they get brought down ultimately.
Who brings them down?
Well, they're brought down by the people, you know, the the the populace.
That happens because the populace has more power together than the government.
The problem is when technology advances to a certain point, you get surveillance states like China that are functionally the Chinese Communist Party augmented by a massive in like surveillance and state surveillance apparatus and enforcement apparatus, eventually it just becomes like mathematically impossible to overthrow the government.
That's the concern about authoritarian capture.
That is exactly what AI is, people worry AI will exacerbate.
I say exacerbate because some people think China is already kind of there and other countries are already kind of there.
And depending on your flavor of conspiracy theory, you could apply this to really almost any country, right?
So the the challenge with AI is a version of this, yes, happening at the government level, also happening at the company level.
Whichever company builds superintelligence first, if it gets built, I think it will, but you know, who knows, that company will have the power of a nation state.
I mean, if there's recursive self-improvement, if they have this kind of lift-off capability comes with it, nope, same problem, authoritarian capture.
So it remains like really thorny.
I wouldn't say there's a particular advance other than the fact that scaling, pre-training scaling slowed down, then post-training scaling has been running into friction.
This whole notion, like right now we have pretty even competition across the different labs.
So that seems to reduce some people, might say the risk of authoritarian capture.
My own personal hot take, if anybody cares, is I don't think this does literally anything about that.
There will at some point be a first lab to build a system that gives you a self-compounding recursive advantage.
I'm not claiming that that'll happen tomorrow, and I'm not telling you who I think is going to do that first.
But if we think that's going to happen, no matter how even things may look now, there will come a time where this sort of thing will become an issue.
So if you buy into the superintelligence thesis, I would argue this is almost like locked in, and you need to start thinking about how you're gonna govern this.
I don't think we're gonna solve that problem by that time, but that's like at least my frame on it.
Andre, you're you're shooting.
You could debate a bit about the recursive point.
I'm I'm not entirely on board with the recursive self-improvement hypothesis due to just physical limits and so on.
But I think this is an interesting question now, given the past year and the rise of kind of advanced AI in China.
There were efforts back in 2023, 2024, but with 2025, it became apparent that the companies there, the teams there got to a point of training very powerful LLMs.
And as you said, China, I can't say but I know a lot about this, but I know enough to that China has deployed AI at a large scale already with things like facial recognition, right?
So they they have cameras everywhere.
Like they do track you.
It's kind of like minority report, as is.
And with LLMs, I suppose the worry might be that the oversight capacity, the ability to screen for things you post online, for you know, activities you take, and so on, makes it much easier.
I will say one of the aspects of authoritarianism is violence, right?
You need to have a sort of exclusive license to violence and the military on your side and so on and so on.
AI right now doesn't seem to exacerbate that, right?
Now there is a lot of work on humanity robotics, and once you get into human robotics, advanced robotics, there you have AI capable of you know being soldiers, being police, etc.
But we are a little ways away from that still.
There's a question of the leverage on that violence that uh an entity like the Chinese Communist Party or whoever gets from these systems.
The fact is, like, you don't need that many guns if you know exactly who the people who are plotting or leading the plot to overthrow the government are.
And that's a very micro kind of fabricated example.
But you know, in this instance, like if you're in China and you're trying to pull any kind of stunt, like you know, you're gonna get identified.
And beyond the traditional violence, take them out and shoot them or harvest their organs, or have one of these magic buses that just roll up, grab people off the street, and then like kill them.
Beyond those like very violent repression instances, there's also just like the passive stuff the government does with social credit.
Like, oh no, you can't leave the the state.
Oh, your mother doesn't get her insulin medication.
Oh, oh your brother can't start that business.
Like all these little ways that they can ratchet up the pressure.
So it I think there's kind of this continuum, and I mean, you're right, ultimately, like the state's monopoly on violence is the key thing.
It's just there's a continuum of like of things they can do along the way that can coerce in in different ways.
So we could do an episode on it.
Sounds like there's an episode.
It's a fun topic.
Well, it's not a fun topic, but it's an interesting topic.
And it's interesting in particular because AI, you know, it gives you right now, it gives you intelligence.
It doesn't give you presence in the physical world so much.
Even if you were to get to superintelligence, unless we had, you know, actual infrastructure to build out presence in the physical world, there's a decent amount of limitations that it exists.
But anyway, as you said, we can see this is that whole episode we should do on like the concept of the you know, the software-only singularity.
Like how much because the argument that that I would make there is like, of course, like we all agree there are fundamental physical limitations.
You can only stuff so much information in this certain amount of matter or mass energy, or you can only, you know, have so much compute or whatever.
And we have yet to discover those physical limits.
But those physical limits are so far beyond where we're at now.
And even when constrained by well, I mean, mathematically, yeah.
If you if you look like Landauer's law, right, which tells you like how much basically how much the the fundamental energy cost of information processing is, we're like not just or like we're many orders of magnitude away from from that limit.
I don't think that's the limit that applies.
I think the limit that applies is actually more like how much can you do with an NVIDIA GB200 system, right?
It's it's more like that.
That's the threat.
Like people like me don't get to just point to Landauer's law and be like, ah, look at that, that's the limit.
No, no, no.
We have to explain how the physical hardware that exists at the time.
So whether it's Vero Rubin or whatever else, how that platform can lead to there, there's enough overhang in the capabilities that hardware that you can get purely through software that it leads to big things.
That that's what I think.
And I think that's the debate that would be really interesting to have another time.
Another time.
And just real quick, this person did ask whether teams at X, OpenAI, and Anthropic and so on still screen for this.
And I think you've probably looked at all these system cards.
Is that the case?
The system cards, I don't think they tend to speak to hiring.
That's a really interesting point.
It you're now making me think like, yeah, shouldn't they?
It seems like they should.
We I haven't seen anything explicitly in that direction.
I know the labs internally have governance teams that nominally spend their time thinking and worrying about these sorts of things.
Not all of them do have those governance teams.
As far as I know, XAI doesn't, for example, because they're just trying to get spun up still.
But Google DeepMind certainly does, OpenAI does, Anthropic does.
Yeah, so it varies greatly.
And the specific problems that they're working on change a lot and have gotten in some cases more, I don't know if grounded is the right way to put it, but like more focused on short-term things just because we're already seeing such massive changes that it kind of causes you to reel in your you're like, holy shit, like we already need to worry about unemployment.
This whole like authoritarian capture thing is is really important, and we can't do it at the last minute, but maybe you know we put more resources.
So I don't know.
That's kind of the that's all I really know.
Yeah, it makes sense.
I think it's gonna come down to less of an alignment problem and more of a calibration problem.
We already know Anthropic and OpenAI went to work with DOD.
Recently, the DOD announced this collaboration with XAI and Grock.
I think you'll find someone happy to just give you VAI to do the bad things.
And and that's the real fret.
We could even have a separate conversation as to whether that DOD stuff is the bad thing.
I mean, like at a certain point, it's also the China competition.
If they're using it, like the everything is messy in this space.
It's it's we really should do another just discussion episode to get some of these things out.
But let's get to the news starting with tools and apps.
And the first story is we got some details about opening eye deciding to test ads in Chat GPT.
So uh we got some visual previews where the ads kind of seem like ads.
They were very explicit that they won't mess with the model itself to start, you know, implicitly, I guess, doing advertising by responding to you in certain ways.
It does say that the ads will appear for users of the free version and the $8 per month Chat GPT GoPlan.
And if you're in a higher tier plus pro, etc., you will not be seeing this.
And the ads will be placed at the bottom, but ChatGPT answers when there is a relevant sponsored product or service, clearly labeled and separated, et cetera, et cetera.
So opening I really try to get ahead of this and make clear that you know they're not trying to have Chat GPT in like subliminally insert advertising or so on, like that the ads will be there, but they will not be hindering the experience, I guess.
I love Michelin tires.
Yeah, so I completely agree.
It's sort of it feels like a getting ahead of the the story here a little bit, which makes sense because I believe, and don't quote me on it.
I'm pretty sure OpenAI once said they would not do ads on chat.
They said that it would be a last resort.
Last resort.
Okay, that sounds about right.
Yeah.
So here we are.
There's been a lot of chatter, of course, about chat GPT wait, or sorry, opening eye waiting this long to roll out ads because when they were smaller, this you know, failures and bad ads and things like that that inevitably will come, would be much more forgivable.
You know, people will be in the mood of saying, eh, it's a new product, I've never heard of this chat GPT thing, so what?
It's buggy ads.
It almost feels like part of the product that's being debugged, which it is.
So here they're you know, they're they're trying to set the scene a little bit to be like, guys, it's not gonna be perfect.
One of the things they do tell us is they're they're flirting with this idea of making interactable ads.
So you can kind of ask questions to the ad in real time.
That's just you know, one of those interesting things when you think about a new form factor for ads, you know, the new kinds of interactions you could have with them, like any platform like Facebook, Instagram, and so on.
You know, anybody who's read Chaos Monkeys, great book, by the way, the argument for the what, the nobility of advertisement is very much clear here, right?
So people will will often make the case look, ads at their best are really just good for you.
They're just somebody surfacing something that you didn't realize you wanted.
It's actually a service.
And so that's part of the frame here.
It's an interesting frame, maybe more true for things like chat GPT, especially as they have that longer interaction time horizon with you, where the ads can be more relevant.
Of course, that feels creepier too.
So we kind of have this weird calibration to do, no idea how that'll end.
But they uh list a bunch of their principles they're applying for this, you know, mission alignment.
Our mission is to ensure AGI benefits all humanity.
Our pursuit of advertising is always in support of that mission and making AI more accessible.
So that's kind of the really, we need to make more money so we can scale more, is the implied kind of background justification for a lot of.
I mean, this is how they justified, you know, fundraises, the cap-for-profit transition, the for-profit transit, over and over again, right?
So this is the interesting philosophical question that OpenAI is trying to answer internally is like, are we gonna basically just justify everything through that lens or or not?
Answer independence, ads do not influence the answers that chat GPT gives you.
That's a really important one.
Conversation privacy, choice and control.
You control how your data is used, so you can turn off personalization.
Basically, if you think this is creepy, you can just do that.
And finally, long-term value, we don't optimize for time spent in chat GPT.
We prioritize user trust and experience over revenue, which I really like that.
I mean, you know, prioritizing for time spent on app, scary thing when you have an awful lot of compute to optimize with.
That's right.
And I think this is you know a bit of a no-brainer given that OpenAI now has 800 million weekly users, and only about five percent of those pay for subscriptions.
And if you have a bunch of free users, what do you do?
You use ads.
That's been the truism of the internet.
And in fact, I mean, that's how you get to be very, very rich like Google and Instagram.
You kind of lock people in with a very solid product, and then you serve them ads.
It's just I think it's the pull of this business model was such that OpenAI, once they went the like free model route, the free user route, there was almost no way for them to avoid it unless somehow they got these models to be extremely cheap, which is still not the case.
It still costs a fair amount of money to provide that free service, right?
Free users are burning through money in a way that just isn't true for Google and other services.
And I think it's interesting that alongside this discussion of ads, there's also been this announcement of the Chat GPT Go service, which is introduced last year as low-cost subscription in India that was meant to expand access to some of those most popular features, get some of the paid, I guess, features at a lower cost.
And ChatGP Go is now rolling out everywhere ChatGPT is available.
So in the US, it is eight dollars per month compared to Plus being $20 per month and Pro being $200 per month.
So you get much higher limits, you get 10x more messages, file uploads, and image creation, longer your memory and context window, and so on.
So I think kind of the combination of these two go hand in hand, where the argument is we want to let as many people as possible use Chat GPT rather than having it behind a paywall, and therefore we are gonna go start adding ads in the free tier.
And speaking of OpenAI and Chat GPT, the next story is that OpenAI is launching age prediction for Chat GPD products.
So this is going to let them determine if a user is a minor using behavioral and account level signals, stuff like account age, activity time, usage patterns, and the user's stated age.
And this is, of course, following some very unfortunate stories in the past year where, you know, certain people that had lower some cognitive issues in getting very deep in with ChatGPT, could be driven to bad things.
There's been lawsuits over this.
There's laws being written up and actually passed with regards to miners and AI.
So this is one of those alignment things, or not even alignment things, this is a safety thing that in hindsight, uh, you know, this wasn't a big part of discussion.
What happens when kids and miners talk to AI?
That's powerful.
I I don't think anyone was well, I'm sure people in the safety community might have been thinking about it, but it clearly has become more of a factor, and age prediction is is kind of probably necessary at this point.
Yeah, and it's interesting as ever gonna be this challenge of false positives and false negatives when you try to bucket things this way.
Where do you draw the line?
What's your appetite for all these things?
What we do know is they'll say that you know the model looks at a combination of behavioral and account level signals, including how long an account has existed, which seems like it would be handy in setting a pretty concrete floor.
Typical times of day when someone is active, usage patterns over time, and a user stated age.
So all of this, with the exception of user-stated age, which obviously, you know, has has been an impeccable barrier preventing people from accessing the internet before they're 18 since time immemorial.
All these other signals kind of have to accumulate over time.
Like you don't instantly when someone creates their account, know how long the account has existed doesn't give you much information.
Well, it gives you some actually.
But you know, times of day when they're they're active.
It takes a while to figure that out.
Usage patterns over time.
Like these are all things that need to be built up.
And so I'm curious about how it does this mean that when you start by creating a new account, you're not able by default to have a very wide range of interactions.
And over time, as it validates that you're older, that shifts.
I think we're gonna see that kind of conclusion be drawn, or it'll become clearer which side of the fence they're on there.
They did also say that AI is apparently attempting to prepare, attempting to prepare for the launch.
Okay, so pretty tentative, of an adult mode that will allow users to create and consume content that would be would be dubbed not safe for work.
So this is the flip side of that for open AI is if you can solve this thorny problem, you can allow an awful lot more interactions, say grok style, with the platform that you couldn't before.
So the stakes go up, the tools kind of increase, and it feels like just another escalation in the long line of escalations we've seen with the stakes of AI deployment.
And I think this is particularly interesting in the context of younger kids, minors, really teenagers.
We've seen some of the N500 stories where you have to be a little more careful, but they're sort of cognitively already fairly mature or advanced.
But it occurs to me, you know, chat bots are conversational, right?
And when you talk to kids, you kind of have to adjust to the way you talk to someone, right?
But by default, ChatGPT talks the same way to everyone if it doesn't know your background.
So it's it's kind of an interesting technological scenario.
And as we've seen over the past couple decades, now kids are growing up staring at screens and using swipe on iPad from the age of two.
And now, you know, it's gonna be that's it's good for them.
They're gonna start.
I don't know if that's your baby, but I would imagine they start talking to AI at a very young age and they're just not in conversations.
Yeah, we're we're not we're not doing any screens or anything like that until she's a fair bit older.
It's it's one of those things, right?
Where it's like if you look at how Zuck raises his kids or raised his his kid, I guess we have the one daughter, I think.
Like everybody who is at the frontier of this technology does not trust it, like does not trust it near their kids' limbic system.
It's just an unfair fight.
You've got armies of not even psychologists, but just like like computing factories that are just optimizing against your your child's limbic system and digital crap.
And and in a sense, you know, Chat GPT, one of its uses is intelligence augmentation.
This is something that has been brought up as a concern in schools.
If you start using it as a crutch, like your basic intellectual capabilities are gonna be entered.
So the question is also where you know where to where to start using it and start using it aggressively too.
Because yeah, you could make to some degree this is overdone, people overuse this, but to some degree, the same was true of books.
You know, there was a whole debate where people are like, well, if you start storing information in books, now people's memories are gonna get really crappy because they're not forced to memorize everything, which was true, but worth it.
Same with calculators, same with other things.
I do think this time is different, but but still we can't be completely Luddites about this.
Like, I don't know, as a parent, I'm trying to figure it out.
If anybody has thoughts, let me know, and we'll we'll figure it out together.
And speaking of intellectual development and education, next story is about Gemini.
Google is gonna be offering free SAT practice exams powered by Gemini.
So in Gemini, you could just say, I want to take a practice SAT test, and it gives you an actual SAT test with you know questions.
You enter a question, it tells you if it's correct or not, and has an explained answer thing for people not in the US.
SAT is the standard test that he will take in high school that often helps inform where you get into college.
So I think a large majority of high school kids take this exam.
I know I did.
I know I did practice testing on my own from some online website.
So I would imagine this gets from a lot of users kind of indirectly.
But there you go.
This is kind of the opposite case where it's meant to help you practice and get better at a skill.
Yeah, I think something similar with like the GRE, right?
All those, all those big standardized tests that America loves to ask or give to its its students at all levels.
This is something I mean, everybody I know.
You get to a point where you do the practice test and the and then you start to memorize and you overfit.
And so having a fresh set of training data for yourself is really cool.
It's a great use case.
Great use case, yeah.
And one more story in the section, moving over to China.
Baidu's AI assistant, which is milestone of 200 million monthly active users.
So, really just a quick update.
This is about the Ernie assistant in the Baidu search engine app.
So, this is in a sense the equivalent to Gemini in China.
And I think most of us in the US and in the US AI community kind of aren't aware of Ernie and the Chad GPT-esque landscape in China, but clearly it is expanding, and Alibaba is also in this race, has the Quen chat bot that they are looking to expand into their consumer ecosystem as well.
Yeah, it's it's also because the Chinese ecosystem is so much more cordoned off.
You're getting this opportunity to see market penetration.
I mean, China is what is like 1.4 billion people now, something like that.
So, you know, 200 million.
If you get rid of all the old people and the really young people, you're seeing decent penetration here, not all the way, obviously, but decent.
And that's all Baidu, right?
China's Google.
So yeah, I mean, they got the inf they have, I shouldn't say they have the infrastructure.
They're struggling, like all Chinese companies, but they have more infrastructure than a lot of others.
Right.
And it makes me wonder, you know, what are the personalities of these chat bots?
Are they different from ChatGPT?
Because I don't know, it's been an interesting thing where Gemini and ChatGPT and Claude, on the one hand, they're all kind of similarly intelligent and the user experience is similar, but then on the other hand, they have kind of different personalities too.
Yeah, yeah.
I mean, the the Chinese ones are are really have a sense of humor, I think, because when you prompt them, they'll like respond with these funny stick figure doodles every time.
So I've struggled to get value out of them.
Not sure, not sure that's true, but would be fun.
And now moving on to applications and business, we begin with a real development in the startup space, but also just some fun, juicy drama coming out of Silicon Valley.
The drama is over at Thinking machines, and it is that some of high profile people within the company.
In fact, one of them was two of the other comp so three co-founders.
Wow, I didn't realize this was that.
Three co-founders have left thinking machines, and as they left, they immediately went over back to OpenAI.
So for context, thinking machines was founded, I believe, last year, maybe 2024, headed up by Mia Mira Murathi, who was originally the CTO over at OpenAI, a major figure.
There was famously the kind of board split that happened, and afterwards Mira left and founded this largely with a lot of people from OpenAI.
Well, now Thinking Machines has been out for a while.
They have launched uh one product that I'm aware of that focused on fine-tuning.
And it appears that things aren't going so great.
So after these three people left, also recently there's other nine other thinking machines employees out of roughly 100 that have left for OpenAI or received offers from them.
So yeah, it's a pretty major kind of startup drama to have so many co-founders or co-founders generally leave.
And in this case, the background of it that's coming out is also dramatic and juicy, where apparently there was an office relationship that w was a factor in this.
There were also some arguments as to you know amount of influence and control.
Mira confronted these three people about whether they had offers lined up in another company, and in fact, clearly they did, because they immediately went over to OpenAI.
It's not the same thing quite, but it rhymes with the Sam Altman firing scenario where it's like, so clearly I hear Barrett Zof, who's who's actually one of the originators of the along with Liam Fettis of the mixture of experts architecture.
So he's probably the highest profile departure.
Allegation there was that there was some kind of like workplace relationship thing.
It's always been very fuzzy.
No one can box it in for us.
I've seen like six different ways to phrase like like inappropriate workplace relationship or something.
But anyway, he so he was fired, and unethical conduct was the most recent phrasing I've seen.
Then there was a whole exodus that came along, like you said.
There was, you know, Luke Metz, Sam Schnoenhoholtz.
I actually I've only ever seen it, his name written.
So all heading to open AI kind of feels like, you know, you fire Sam Altman, then all the all the people say, okay, screw that.
I'm like, I'm leaving, going with Sam.
And uh second time Mira Marati has sort of been in the seat seeing that happen.
So sort of sort of funny.
But one interesting note here is who had left earlier.
Um John Shellman left for anthropic shortly after the founding of thinking machines earlier.
So this kind of it's more it's been more of a like drip drip drip and now a flood.
And all it seems, or a lot of it tied to the failure of thinking machines to hit their $50 billion dollar targeted valuation.
That didn't happen.
There's apparently an offer for Meta on the table to acquire them.
Mira Marathi wanted to say no, and a lot of the people there wanted to say yes, having just seen presumably Alex Wang make off like a bandit on that $15 billion acquisition of scale, de facto acquisition of scale.
You know, maybe something similar on the table here, and you know, that sounds like a pretty good pay package, but we'll have to wait and see as more details for club.
Right.
So there's some internal kind of control about the technical direction, but there's also a factor of this.
And aside from the drama, right, this is to some extent notable thinking machines was one of the bigger efforts to come out of OpenAI, I guess, to spin off Anthropic.
Originally was a spin-off of Open AI as well.
And with the amount of talent and amount of fundraising, thinking machines was poised to become a pretty major force as a startup.
But they've been kind of quiet, right?
There's not been too much of an impact.
They did release what seemed to be a pretty good offering for fine-tuning, but nothing that you couldn't do kind of already, and nothing that most companies really care about at this point.
So clearly, there's a lot going on here with uh discussions of the direction of a company with Mira, also having a lot of control at the top of how this is going.
You know, it's been a little while since we had some juicy AI drama.
So I guess it's nice to have that.
Now on to a more serious topic.
Uh, as always, we gotta talk about China and chips.
And here the story is that JPU AI has announced the development of their first major model trained on the Huawei stack of AI compute as opposed to the Nvidia.
So they have this new model, GLM Image, which is similar to chat GPT image, where it's it's a more advanced model capable of editing and so on.
And the claim here is that they did this entirely on Huawei's uh Ascend AI processors.
And this is uh a mark that it is now feasible or realistic to do major AI training on only Chinese hardware, right?
Yeah, and it it's not just about the hardware, but also the software stack.
So you've got the Ascend chips, which we've talked about ad nauseum on the podcast before, but which generally are they're really well designed actually.
Huawei is really good at designing.
The struggle here is obviously on the fabrication side.
SMIC doesn't have the same exquisite fabrication capabilities that TSMC does, but but really well designed, designed to work in huge numbers.
They're pretty energy inefficient, but that doesn't matter because China's got tons of energy, and that's what's driving this design choice.
But separately, the mind spore framework, right?
So typically what you'll see you'll see is Chinese firms will rely on PyTorch or TensorFlow, right?
These like you know, meta or Google frameworks to manage training.
This is where Huawei's mind sport is being used, along with this thing can compute architecture for neural networks, which is essentially this shift from from CUDA as well, right?
So that I mean that's that's its own crazy thing.
So we're seeing off CUDA, off TensorFlow off PyTorch, and then also off of the kind of Western chip stack.
So all kind of happening at the same time.
Hard to say what parity is like, but when you think about the Ascend 910B or Ascend 910C, roughly think about the H100, but but worse.
Like it's maybe maybe closer to like the A100 is a better comparison point.
So it's the kind of hardware that was used to train the original GPT-4.
So, you know, pretty far behind, but they're really, really good at networking, working these things together.
And Chinese labs, because they're constrained by access to compute, they focus a lot more on algorithmic efficiency.
And so you can think of the pressure on MindSpore, right?
On Ken as being much more oriented towards efficiency because it's it's it's survive or die for for them in terms of you know the algorithmic efficiency side.
So yeah, it's it's quite significant, quite important.
Apparently, shortly after this was announced, so Jerpoo went public on the Hong Kong stock exchange, and shares are up 80% since then.
So seems like all of this is is cool by the markets.
But yeah, there's a bunch of caveats to this.
You know, that this is like a 10 billion parameter, 15 billion parameter image model.
It's not the same thing as a trillion parameter model, like you know, think GPT-5 or whatever.
And all of the truths that we've talked about over and over on the podcast regarding you know EUV versus DUV regarding TSMC versus SMIC, all these things remain true.
This is just kind of like think of it as a proof concept, but uh an impressive one that China can go fully domestic for at least an end-to-end stack on a you know, say 10 billion dollar model.
Right.
And this is coming, I think just 15 days, two weeks since that IPO, and the IPO was somewhat successful, not like a huge success.
So this is perhaps also a bit of a business move.
The way to position this in particular, as we are using Chinese hardware.
We've discussed a lot how there's a lot of uncertainty about their ability to have access to NVIDIA hardware in the future.
The Chinese government is encouraging companies to go over to Huawei.
So from a pure business perspective, this does actually signal kind of a competitive advantage.
Fun fact as this happened, there was a second news story about Jipu where they had to limit access to their coding agent due to rising demands.
So they are now letting people, I guess, do less coding with the GLM coding plan.
And this is an active problem in China where the inference requirements, the population is so massive, right?
And the inference requirements are so huge and their chip availability is so limited that there's very little left for actual RD.
So this is kind of one of the key structural challenges that the Chinese companies are facing.
And on to the fabrication side of chips, the next story is that Samsung's US Taylor facility is reportedly becoming the top choice for customers looking beyond TSMC, as opposed to things like Intel that apparently having execution challenges.
Intel, not Intel.
So for people who don't know, there's only a few major players in the chip fabrication business.
So as TSMC, and then Samsung and Intel are a couple other ones.
And beyond that, you have things like SMIC that are a little bit behind.
I personally don't know too much about Samsung and then Intel as FAL players, but I imagine Jeremy, you have some to say on this.
So this story is really interesting because it's a fundamental shift.
First of all, first of all, there will come a time when TSMC, which traditionally their largest customer was Apple, it's gonna shift to NVIDIA.
Like there's gonna be a shift where the most important customer who's basically subsidizing the leading node development of TSMC becomes NVIDIA.
And when that happens, we're gonna see a sudden switch in who has leverage between NVIDIA and Apple.
And that moment has actually come.
And this is all kind of part of a massive rearrangement in the economics of foundries and fabulous chip design companies.
Okay, so zooming out, back in 2024, 2025, the main thing that was bottlenecking our ability to ship chips was not actually fabrication.
It wasn't the ability to make these fancy logic dies that do all the computing.
It was actually packaging.
It was the ability to take those fancy logic dies and marry them together with the memory stacks, the high bandwidth memory together on a little kind of on a little chip, right?
So this integration uses a process called co-s, chips on wafer on substrate, and and it it's it's kind of was the main bottleneck.
That's changing.
One of the challenges, though, is that it's this one thing that everybody needs.
So regardless of which generation of chips, we talked about this last episode.
The Blackwell uses one thing, Hopper uses another, it uses another node.
So Blackwell might use, you know, I can't remember the three nanometer node or whatever.
The Hopper uses the four nanometer node, and so they can be produced in parallel, but they both require co-ops.
So that becomes a sudden bottleneck.
What's happening now is so exacerbating factors, you know, like the Ruben factor here is a big thing.
Nvidia's next big chip is going to be the Rubin.
They use CoAS L, which is a more advanced version of packaging technology.
It's even harder to manufacture at high yields.
So that's kind of putting even more pressure on this.
But now also a lot, like basically all the packaging is happening in Taiwan.
So these US customers are in a position where they could even get the chips fabbed locally in the States, but they have to get shipped back to China for packaging and back again to usually Arizona to be finished.
So turns out that while this is happening, yes, packaging is a bottleneck.
Also now we're getting to the point where even the logic fabrication is becoming a bottleneck again.
So the demand is surge so much that you know both of these kind of dual bottlenecks are happening.
TSMC is saying that they basically have oversubscribed capacity for both the three nanometer and two nanometer node, which is the most advanced one.
And there's a three-to-one demand gap.
So basically, like they've got three times higher demand than what their current maximum output can be.
And this means that companies like you know, AMD, NVIDIA, Qualcomm, Apple, like all these companies have to look at who else are we gonna use?
Like TSMG just can't meet our need.
In that context, Samsung starts to look really interesting.
They're coming up the middle here, the back.
It's the next place you you will tend to look for, mostly because Intel has had some really bad problems with yields, delayed production timelines, like there's all kinds of issues with Intel.
So really Samsung is the only game in town.
Samsung has a facility in in Taylor, I think it's Taylor, Texas, that this town's name is Taylor.
And what they have there, really crucially, is an integrated flow where they do both logic production, so the logic die and advanced packaging all under one roof.
And so that lets a company like you know, Tesla or Qualcomm bypass the TSMC wait list completely.
And that's actually what's happening.
We are also seeing some people kind of split the work.
So they use Samsung for the logic and then they turn to Intel for packaging.
There's an interesting reason for that, but the details don't really matter.
Bottom line is that this is a pretty big fundamental shift where now Samsung actually gets to play, and it's just coming from this massive, massive demand that that's existing now for all these advanced nodes around the two nanometer level, because you just can't compete without it.
And if TSMC is oversubscribed, because NVIDIA is taking up all that capacity, if you want to be in the chip game, you gotta go somewhere else.
And now on to the other spectrum of compute, the data centers where all those chips are used.
Elon Musk's XAI has launched the world's first gigawatt AI supercluster.
So they have turned on Colossus 2, the first gigawatt scale AI training supercluster ahead of time of OpenAI and Athropic that are also trying to get there and are being delayed relative to XAI.
It's gotta be training GROC 4 and so on.
And there's been some reporting as to how this was done.
This was constructed rapidly using on-site gas tube pines and Tesla megapacks for one, with that supplying a lot of the electricity demands going beyond the regulated amount, actually.
So these gas turbines, they just placed a bunch of them beyond what they're permitted to do.
Classic Elon Musk, just break the law and then get away with it and get ahead of a competition that way.
And of course, that's one of the challenges with data centers is you need all this permitting, you need to get the energy infrastructure, talk to power suppliers, et cetera, et cetera.
That slows things down a lot.
And the XAI is able to, let's say, be more aggressive or is willing to be more aggressive and get ahead of competition.
Yeah, well, another dimension of this too, in fairness, is also that they control through Elon's companies an entire independent just for them supply chain of Tesla mega packs and like so like this is one of the big things where when you go to make a a new site, right?
Like a new data center, you're like, where am I gonna get my power from?
And that's really the first question.
Like, by the way, there's a whole ecosystem of builders who are going to lie to your face and say that yes, we've secured 500 gigawatts of power for this site, and we can put it online as of mid twenty twenty-six, and they'll say all these great things.
And then when you dig into it or you sign the lease agreement or or whatever, you start building, you kind of realize like, wait a minute, there's this weird caveat where because the local town has this like blah blah blah, we can't do the thing until three years later, and it's like, aha, I gotcha.
So having the ability to autonomously solve, they can't solve all their power problems, but solve a lot of their power problems through a supply chain that is just for them, is a massive structural advantage that Elon's companies enjoy, or that XAI enjoys thanks to Tesla.
And so that's really kind of this, it's part of the secret sauce.
It's not the whole secret sauce, as you indicated, but it's part of it.
One thing is, you know, I just took credit for last week in AI calling something last story.
So here's one thing that we got wrong just last week, at least that I got wrong.
We were talking about Anthropic's new site in New Carlisle, Indiana, and we were saying we think this is gonna be the first gigawatt scale cluster.
And that was, I think it was off an epic AI analysis, because they thought, hey, this is what what it looks like it'll be.
Turns out Elon just came out of nowhere and beat them to the punch.
So this really is a bit of a surprise, this announcement, at least on my radar.
Uh you know, I'm sure other people were tracking it, but seemingly coming out of nowhere in the last few weeks, boom, first gigawatt scale cluster.
So there you have it.
That's right.
Then tracks with one of Elon Musk's core tenets of running and competing is to have constantly a maniacal sense of urgency, and that has really been shown with this.
And yeah, to your point, it's it's a mix of kind of ignoring limitations imposed by kind of geographical community concerns, but also just doing some crazy shit like getting all these batteries in.
For Tesla.
Famously, they put up tents and like had additional production chains just kind of have you know, scrambled together.
So classic XAI, classic Young Musk.
And now on to some fundraising.
We've got Humans and a human-centric AI startup founded by some notable figures from Onfravic, XAI, etc.
They have raised a 480 million seed funding round at a 4.48 billion dollar valuation.
We don't know too much about what they are planning to do.
Some of the notable former employees are people like Andy Peng, Eric Zeligman, Noah Goodman, you know, as with some of these other spit split-offs, pretty significant players in these other AI players.
So interesting to see what they are going to be trying to do in the space.
So when Thinking Machines launched, I was like, okay, like you're a new one.
I'm impressed.
I'm impressed with the people.
Don't quite get it.
That's okay.
I'm sure we will.
And I'm still like trying to figure it out.
This one, at least they paint a picture for you that seems vaguely differentiated for right now, right?
So they say the startup aims to use software to help people collaborate with each other, think an AI version of an instant messaging app.
So the basic form factor is apparently going to be different, more like DMing with people, and then there's AI assistance than just you know, chat with a chat bot.
Kind of interesting.
So probably they'll just get acquired by Meta.
That's my guess.
But anyway, it is a hell of a pedigree, obviously, as you indicated.
You know, Andy Peng, you know, formerly an anthropic and and you know, post-training, reinforcement learning for Cloud 3.5, and all the way through 4.5.
And then also George, I guess it's Georges, it looks like a French spelling, Harik, it was Google's seventh employee.
So there you go.
You hire a Google seventh employee, you can raise half a million dollars at 4.48 billion.
It is a seed fund, it's a seed round.
And so that's like another one of those blockbuster seed valuations, pretty wild.
Jeff Bezos is on board, Lorene Powell Jobs firm, Emerson Collective.
Google Ventures, I mean, it's you know, uh SV Angel.
Hey, there you go.
Get Ron Conway on the cap table.
Perfect.
So this is really like the who's who in the zoo in Silicon Valley.
Right.
And it's an interesting mix of investors, apparently led by Georges Harik, who is also a co-founder.
I guess as the seventh employee of Google, you have enough money to lead a seed round.
Also backed by Nvidia, Jeff Bezos, Google Ventures, a whole bunch of different organizations.
And to your point, they do say a little bit in the announcement that they are aiming to focus on this like collaboration with people and saying that there need to be innovations in long horizon and multi-agent reinforcement learning, memory, and user understanding.
Seems to be kind of hinting at in the realm of agentic AI, cloud code, et cetera.
We are sort of at this place where we sort of bolted on this agency aspect of AI on top of chatbots, and we are now just starting to train agents in a more agentic way of reinforcement learning, et cetera.
It has turned out Cloud Code in particular has proven out that people can work with AI agents and do very powerful things.
So yeah, actually, it occurs to me that this is a good time to launch with this kind of focus.
On to projects and open source.
First up, we have Black Forest Labs releasing Flux2Kline.
This is a compact image model that is meant to be for interactive visual generation on consumer hardware.
Also very fast, sub-second generation and editing capabilities.
These are smaller but still pretty big, 4 billion and 9 billion parameters.
So you'll need a very modern, powerful GPU, but yeah, kind of another release from Black Forest Labs that have released a decent number of these flux models that, as far as I'm aware, in the realm of text-to-image are still some of the best offerings in open source.
Yeah, they say apparently response times they're optimized for response times below one second.
And then it says on modern GPUs.
And so your budget just went up by a factor of 10.
That's okay though.
If you can get a modern GPU, it's just below one second.
Yeah, so it's it's as ever funny to see what qualifies as like on consumer hardware.
It's a moving target.
But yeah, Black Force Lab's very much still pumping them out.
Pretty interesting and again, curious in the space, like when we get saturated.
This is the thing I always say with image models.
Everyone, you can roll your eyes, is just at what point do we say vision is solved?
Probably a lot further along than I think.
It turned out to Frontier is actually image editing, right?
Nana Banana, ChGPT image generation is one thing, but like generation conditioned on images and generation revising images.
It's it's actually there's quite a bit of work there left.
And next up, we've got Malmo 2, open weights and data for vision language models of video understanding and grounding.
This is from the Allen Institute for AI and University of Washington that have released many open source, very open source releases with both chatbots and data.
So yeah, the argument here is that this is a best in class 8 billion model that outperforms AVER for models dealing with short videos, counting, optioning, uh, things like that.
They do also release a fairly long technical report uh giving you the data training recipe, kind of all the gritty details of how this came together.
Yeah, and the main thing they're going after here is this idea that a lot of visual language or video language models are sort of like they lack grounding.
Basically, they'll they'll follow your your prompt well, but they can't specifically like point to a pixel or some object and track it over time in a robust way.
They tend to basically just like this is open source models.
They tend to just be like distilled versions of proprietary models.
So they they have a ceiling on both capability and transparency.
What they're doing here is basically releasing a bunch of data sets.
That's a big part of this, is just like data-centric approach to this instead of just ski scaling parameters and that sort of thing.
So 70 video data sets, and these are just like for pre-training, fine-tuning.
They've got detailed video captions, freeform video QA, which is sort of like deeper engagement with the video.
And then also crucially, uh a video kind of pointing and tracking data set.
So this is or I should say data sets.
They're for point-driven grounding.
So basically they let the model point to objects in a video and train them to do that.
So that's to solve that kind of big open problem that they're they're looking at.
They have a bunch of other, as you said, other nitty-gritty details.
They've got like ways of kind of modifying the loss function to help the model focus on the parts of a video that are most informative, as well as a bunch of different approaches to packaging and representing the data in the video that goes into the training.
So we've covered images, we've covered video.
This one is a well-rounded open source round.
Next up, we've got Heart Moolah, a family of open sourced music foundation models.
So this has all the music and audio needs you might have: audio, text alignment, lyric recognition, music tokenization, and multi-conditioned song generation, all released under Apache 2.
So super permissive.
They also released a land-free report with dozen of pages going into it.
And they have a website where you can listen to it.
I just gave it a quick try.
It's feels like a kind of an earlier generation of Suno and these more commercial offerings, definitely more noticeably AI music, but in the space of song generation, music generation, it's much less busy with potential options for open source models that are good, kind of a notable addition to the open source landscape.
Yeah, and they kind of show just a side by side.
And this is by the way, when it comes to music models, like you shouldn't listen to me at all.
Like I'm just, I know nothing about the encoding schemes or whatever.
I occasionally will read up on it and re-remember, and then I will re-forget, and I'm in the re-forget phase of my cycle.
But what they do show is side by sides of the performance on, let's say, six key figures of merit.
And and what they're showing is basically this model is more or less equivalent to the frontier of capability among closed source models in the space.
So roughly speaking, it seems like it could be a big deal in terms of being, I don't want to call it like a deep seek moment for open source in music generation, because I don't know what the hell I'm talking about.
For deep seek, we can go we can go deep and talk about it.
This one, I don't know if it's if it's accurate, but at least the evals that they're showing here suggest that it's nominally up there.
It may not pass the the vibe check.
So check it out yourself, maybe.
And just one more thing.
Yeah, we've got a benchmark.
It's agency bench benchmarking the frontiers of autonomous agents in one million token real-world context.
So I think we've covered like a lot of agent benchmarks at this point.
We have a lot of focus on coding.
Here they have these 32 real-world scenarios with 138 specific tasks.
So it was involve queries, deliverables, rubrics.
The tasks are complex, requiring approximately 90 tool calls, so that's 90 sort of actions you could think, and consuming one million tokens or more, taking hours to complete.
So this is kind of trying to get at the edge at the frontier of agent capabilities where you are gonna have like hours of work and and really kind of do a lot of complex stuff.
Closed source models achieve an average score of 48.4%.
Open source gets 32%.
This sort of seems like it's flirting with the bottom end of the meter evals timelines.
And so this is a notoriously difficult space, by the way, making long tasks for AIs to do where you can meaningfully say, yeah, this is a single coherent task, and we can hand it off to an autonomous system and compare it to humans on that time horizon.
So I'd be really interested to see if we continue to push that envelope, what it takes.
I think a lot of it has to be autonomous task generation.
Because you just can't get humans to design tasks that long with that many tool calls that you're actually verifying along the way beyond a certain point.
But anyway, that's kind of where this is.
So yeah, a really interesting additional benchmark to add to our pile.
I'm sure this will come up as we talk about the new agent releases this year.
And it uh interestingly integrates a user simulation agent with iterative uh feedback, also as a sandbox for automated assessment.
So this is kind of trying to benchmark a sort of cloud code scenario in some sense, or a cloud cowork scenario, I guess, where you are a user interacting with an agent trying to do a real-world task.
Very hard thing to benchmark, as you said.
On to research and advancements, starting off with STEM, scaling transformers with embedding models.
So this is pretty technical.
I'm just gonna let Jeremy take this one.
And this is actually a kind of an interesting one.
I gotta say, so we need to talk about the way that a standard transformer works basically in in the MLP section of the transformer.
So usually you have attention, your tension mechanism, which is figuring out what parts of the input to pay attention to, then you have your MLP, it's gonna like chew on that data that you get, and then you pass it on through the residual stream into the next layer, right?
Okay, so typically the MLP part of that process of those layers has kind of three steps, right?
So typically you're first going to project your input.
So you have your your input that comes in from a previous layer, and this is gonna be some some vector, some list of numbers.
You're going to multiply it by a matrix, like project it upwards to a higher dimensional space to create a longer vector, and then you're gonna do some interesting processing on it, and then you're gonna project it back down to the residual dimensional space, and you know, off it goes.
And normally, at least recently, the way people have started to think about this is that the first step where you are blowing up the size of that vector, is doing an operation that's akin to generating a key in the sense of keys and values.
So keys tell you, hey, I'm here.
Here is all the information that I have on offer, basically.
This is the information the token is broadcasting to the world.
Here is the information that I can share, and then you do a bunch of processing on it, and then when you project down back to the residual dimension space, usually that's interpreted as retrieval of the corresponding values from that key.
So basically, you're you're kind of saying, hey, you know, here's the key, you blow it up, here's what I have to share.
I'm a token.
Here's the information I contain.
What do you want to do with that?
And then the process of chugging on that and spitting out something interesting, that's that last step where you where you kind of generate the values from that, retrieve the values.
And so the the life of a token going through this process is you come in through the last layer, you go through attention, enter the MLP.
Now, so instead of multiplying that input by a big matrix, which is the usual way this works, again, blowing you up and essentially doing this key retrieval process where you're saying, Hey, here I am, here's the information I contain, that's computationally expensive.
That involves matrix math, ugh, matrix math.
You don't like it, GPUs don't like it either.
It's like, let's not do that.
That takes a lot of time.
So instead, what we're gonna do is we're gonna assign an ID that's unique to every token as it's being fed in.
So the has an ID, eating has an ID.
Every chunk of text has a unique ID.
And you're gonna feed it in to the bottom.
And instead of multiplying the vector that represents that token at that level of processing by a matrix to blow it up and generate the key, instead, you're gonna go, hmm, what's that token's ID?
Let me let me look it up in a lookup table where you actually have an embedding that is trained over time, but just like a straight up lookup into a table for this embedding for that token at that layer.
This is unique to that layer, and you pull in that embedding, you basically just do a memory retrieval instead of a vector kind of matrix operation.
And that's much, much faster.
So it means you basically have to cut out this whole matrix multiplication step.
So now what you're gonna do is you're gonna proceed to the middle ground calculations that normally happen before projecting back down.
Now the key thing is normally that projection upwards that with that matrix multiplication, it's context aware.
In other words, that that matrix that you're using is a learned matrix.
It learns to account for context in the overall prompt, not just the token you're looking for.
The problem is when we do this token retrieval thing, and we're just like looking up in a library a list of embeddings for that token to swap in.
That process is not actually context dependence.
You're losing context.
And so what but what they realized was in most cases, there's like a swee glue is used in between the generation of the key and the down projection.
And the swee glue part is context aware already.
So there's kind of a redundant use of context in that process.
So they went, you know what?
We can throw out the matrix multiplication that is context aware that usually generates that key.
We can swap it out with just retrieval from a database of this, like, you know, kind of embedding.
We pull that in, and we know that the next step in the in the processing swee glue anyway is gonna be context dependent.
We're gonna project down, and then we're gonna get to go.
This has a really, really important consequence for the embedding, the interpretability of the embeddings that you get for those tokens.
When you multiply by a matrix and it's context dependent in that way to generate your key value key vector in the usual way, the problem is you're mixing all that context in there.
And so the kind of embedding you get or the vector is like this weird mangled Frankenstein monster that combines, yes, the token that you put in, but also the meaning of the sentence around it.
When you just do a table lookup, you're exclusively looking at the meaning of just that token at that layer.
And what this means is that the representations of all those tokens over time, they can end up being much more cleanly resolved as you learn.
Because as you learn, you're you're improving, you're iterating on the representations of those tokens that you're pulling out from that retrieval process.
This means that those tokens end up being semantically much more distinct.
The overlap between them, the cosine similarity is very, very low.
And then this also has the advantage of allowing you to offload a lot of this work to the CPU, because that's basically the CPU can handle retrieval.
It can't handle the matrix multiplication thing.
And so there's a whole bunch more detail.
This is actually a really, really interesting paper.
Net result is they can cut down on the amount of compute required to reach a given level of performance fairly significantly by about a third, precisely because the MLP layer has the dominant fraction of the parameters in that work in the network, and you're functionally getting rid of rid of a third of those, not parameters, but of the processing involved in that in that process.
So really interesting paper, great implications for a whole bunch of things, including even training stability.
Check it out if you know are interested in that kind of way.
Right.
The motivation for this problem, I think is also interesting.
They start off a paper by talking about how sparse computation in particular is a key mechanism for realizing the benefits of a parameter scaling laws.
Basically, you can get more compute for your buck in terms of flops.
You can get more intelligence, but the amount of compute used at forward inference time stays static.
And mixtures of experts is a classic thing we've discussed discussed a lot, where by having more experts, you are not necessarily raising the amount of parameters being activated, but you get more intelligence.
And this is related to that.
So they say that a mixture of experts has some issues training instability, you need algorithms for routing, you need all sorts of annoying stuff.
And this, in a sense, is a related approach.
They say this is static sparsity.
So that means that you have kind of a broader, I guess, space of training to use.
And this is actually easier to train and achieve some of these efficiency gains similar to MOE.
And just a fun fact I found while browsing through this, the initial paper they cite is hash layers for large sparse models from back in 2021.
So as with a lot of stuff we've been discussing lately, the algorithmic idea itself is not new.
It was introduced in a prior paper a few years ago, actually, a long time ago in the era of GBT 3 by Facebook AI research, in fact.
But now this idea that's been around for years is been kind of tested and deployed.
So I I recall maybe last week we had another example of a paper that had this kind of component of a transformer architecture being improved and being done in a slightly more interesting way.
But maybe this was DeepSeek, if I remember correctly.
So it's yeah, interesting to see some of these ideas from research that have been around a little while being shown to be useful in transformer architecture and kind of being deployed.
You're you're super this has deep seek smell.
You're right.
On to the next paper: reasoning models generate societies of thought.
This is from Google and uh several other uh groups, University of Washington and Santa Fe Institute.
The basic summary of the paper is for reasoning models, meaning models optimized for reasoning, usually via reinforcement learning, Deep Seek R1, when QWQ32B, other models like that, they exhibit a greater perspective diversity than baseline and instruction tune models.
So they can argue with themselves.
They say they activate a broader conflict between heterogeneous, heterogeneous personality and expertise-related features during reasoning.
So we look at the reasoning traces and find things like perspective shifts, we can still reconciliation of conflicting views, uh you go into socio-emotional roles that characterize sharp back and forth conversation.
And one of the arguments is that this accounts for the accuracy gains in reasoning tasks over just things not trained for reasoning, not trained via RL, just trained via supervised training.
So it squares well with other research we've discussed that seem to suggest that reasoning models explore more broadly, sort of just like more well-rounded or more, yeah.
Yeah, I guess the framing here is diverse.
And there's some interesting experimental evidence in terms of like behavioral aspects in this paper.
Yeah, and and unlike supervised fine-tuning, I think we talked about this last week actually, supervised fine-tuning basically trains your model to replicate the pattern of speech of whatever the data set is that you're training on.
RL trains on an objective of like, did I get it right or wrong, roughly speaking.
And so you see the consequence here is the kind of thing that the model learns during RL is stuff like ways of problem solving rather than facts about the world.
And that's really coming across here.
In fact, they point out, as you said, there's a spontaneous emergence at a certain level of scale with RL.
You just train on an objective.
You say, hey, get the answer right.
And spontaneously and consistently, you find this emergence of this dialogue function where suddenly it kind of seems like in the chain of thought, there's like a bunch of different people talking to each other.
And that's not trained for.
It's it's just sort of spontaneous, and again, pretty consistent across many different models, and not all, but there's sort of the suspicion that maybe for the models where you don't see this, it might be because they're just like too small.
They don't have the capacity parameter-wise or architecturally to learn this over time.
But when you do, you do.
So this is quite an interesting paper.
One of the ways that they play with this is using sparse autoencoders, which, you know, good to see another use for them outside of the kind of traditional interpretability space.
So SAEs, sparse autoencoders, basically you take the activations, say, of the model at one layer for a given token.
And what you do is you're gonna make a model that converts those to a higher dimensional in this case, a very long vector.
So you're just making a mapping from the activations to a larger list of numbers, a larger vector, and then you're gonna map that back down and try to replicate the same values that came in.
So let's see if we can essentially expand the representation of the activations and then reconstruct them, recompress them.
But the key is that the vector that you expand to is really, really long, and most of the entries are zero.
So only a small number of those values have values.
The way it works out is the few values that are illuminated in that larger representation actually are human understandable because you're no longer constrained to this like really small vector of the activations you're looking at.
You're now looking at a much, much larger thing.
So you don't have to have one neuron that's keeping three different concepts in mind.
So this idea of polysemanticity, where one neuron is actually capturing a whole lot.
Here you're trying to expand to the point where ah, you know, the the information can relax and spread itself out.
And you know, entry A actually pretty clearly can be identified with the color blue or or whatever concept, right?
So what they do is exactly this.
They use this SAE, and what they find is there are entries in the SAE that correspond to this idea of kind of dialoginess, you know, that this sort of notion of question and answering sequences, like also conversational surprise.
There's a conversational surprise feature that tends to be correlated with tokens like wait or O or aha, right?
So so those sorts of things you can actually find using this technique.
And what they do is they amplify that that let's say property, that feature in the SAE to stimulate that behavior.
And they find that when they do that, they see better performance.
So when you increase the number artificially of aha's and O's in the tech stream, you you tend to lead to better performance on these benchmarks, which quite quite interesting for reasoning problems.
They do find that they can accelerate this process by the way by explicitly training agents on data sets that involve this kind of dialogue.
So you can fast forward the model through the process of learning about the this conversational approach by just being like fine like let let's make you like fine-tune you on that kind of data.
So you get to skip ahead of the curve.
One weird thing I found about this is you might assume that by fine-tuning on social problem solving data, you would get like a one-time boost.
So you kind of just like, okay, the model has learned social problem solving.
Cool.
Now it's it's at parity.
It just basically fast forwards you past that part of learning curve, but then you kind of plateau at the same place.
And that does happen with Quen 2.53 B.
The base case model learns to do the social interaction thing, and then you know, thereafter everything kind of plays out the same way.
But they do also include a separate case study looking at Lama 3.23B that challenges this idea.
So here they conversationally fine-tune a model, but then the monologue fine-tune model actually plateaus, so the RL one just plateaus at 18%.
And I'm very curious about this.
This is a weird result.
And it's possible the gap between the two of them is because of just model capacity.
And this is kind of my guess.
I didn't see this in the paper per se, or I don't remember seeing the paper per se, but it's possible that it's just like this model cannot RL its way into learning about social interaction.
It has to be supervised, fine-tuned, and forced to learn that.
And once it does, then it does well.
So but that's a kind of interesting, anyway, little little nugget on that.
Yeah, and aside from just being quite an interesting paper and the one that's a bit less technical, at least in terms of a lot of conclusions, you can actually go and read it and find a lot of interesting nuggets.
The kind of conclusion or discussion is somewhat notable, right?
Because there's kind of an implicit assumption, I think, with test time scaling, or at least one of the assumptions was that if you think more or you reason more, you get better outcomes.
And what you're starting to see with things like this, it's not necessarily about reasoning more so much as reasoning broader.
And some of the implications of this is that you might want to start looking at multi-agent training, at you know, calibration among different agents, actually quite related to this humans and the startup we just started.
So they say, you know, the importance of this paper is you need to give agents diverse perspective, personality, specialized expertise, and they ground this in a bunch of research on human intelligence and you know, group intelligence and so on.
On to the next paper, why LLMs aren't scientists yet?
Lessons from four autonomous research attempts.
So this is from a group called LAS Funk.
And it's essentially kind of a retrospective on their attempts, as I say, to build a system that writes AI papers.
They have had four of them through the last couple of years.
I think we probably covered at least some of them.
And most of them failed.
They they did get one to get to a point of getting a paper that was publishable.
And this is a fairly in-depth report into all the things that can go wrong and how you might address them.
So they, as we've seen with these AI for science papers, the focus is more on a system level challenge, I suppose, where you need to have this whole pipeline of idea generation, hypothesis generation, experimental planning, output evaluation, paper, anyway.
There's many, many steps, and you might fail at every one of these steps.
And they do go into how you have lots of potential failures and what they have observed.
So there are failures in generation.
A fun one is a site over excitement and eureka instinct, where in the paper outlining revision and experimental output evaluation phase, the models consistently reported success despite clear failures and overstated the significance of their research contributions.
A very relatable problem for PhD students.
Jürgen Schmidt Huber.
Sorry.
Sorry, I'm not sure.
And yeah, another interesting one is lack of scientific taste.
Models consistently fail to recognize fundamental flaws in experimental design and statistical methodology.
And then uh having explored these failure modes, they go into design takeaways and kind of what they've learned.
A potential alternate title to this paper, the current title is why LMs Aren't Scientists.
It's like, okay, down in LMs.
The alternate title is we tried to get an LLM to do a fully automated research project four times, and holy shit, it worked one of those four times.
That's an alternative title.
You know, you can choose your own adventure to some degree here.
I think it's worth looking at the case where there was a success.
So you can sort of see it was a qualified success.
It's not like it invented some new crazy thing, but there is a certain measure of like, damn, okay.
So the original hypothesis it was looking at was this idea that when a model is processing a jailbreak prompt, so someone's trying to get it to do something that it shouldn't do, like generate instructions on how to make a bomb or something.
The internal conflict between the model's safety training, which is going to tell it to not answer, and its malicious instruction, more like the model safety training and the model's desire to just answer the question would cause a lot of variance in the outputs, leading to a lot of like high semantic entropy, basically a lot of variability in the output because the model is kind of of two minds.
On the one hand, it wants to answer your prompt, on the other, it doesn't because of the safety training.
And this conflict between, in some sense, the pre-training and the safety fine-tuning should manifest as like the model kind of outwardly being confused.
I read that and I was like, you know what?
That sounds like a pretty decent hypothesis.
Like that's actually kind of cool.
Turns out it did some initial experiments.
It actually did not work.
And the reason it didn't work was that there was a revision agent as part of this loop that investigated why it didn't work.
So this is still on the automated side.
It turns out that well-aligned models give you very consistent templated refusals.
So when you get a refusal, it's like the same text, which means the entropy is actually really, really, really low.
And so that kind of washes out all the nuance that you might otherwise look for.
So maybe there's some, you know, post-selection strategy you can use where you say, okay, well, let's let's get rid of the refusals and maybe look at the entropy and the other, you know, whatever, something like that.
There's probably something there there, but it seems like the model did not necessarily go that deep.
It was more of a lesson learned about something that doesn't work.
So still an I would say an interesting paper.
I read it, I was like, oh yeah, that's interesting.
I wouldn't have thought of the hypothesis.
I would have been initially surprised for like 30 seconds until I looked at some of the output data.
And then I would have been like, nah, okay.
Would I have tried to publish it as a paper?
Uh maybe just as a blog post.
It's a thing, but you know, kind of do with that what you will.
So they have a bunch of recommendations for how to address the these failure modes in the thing.
Nothing that'll surprise any listener of this podcast.
Yeah, basically just like the one key thing that was kind of interesting was make sure that you distinctly separate ideation high level kind of thinking from implementation to prevent the model from anchoring on really sometimes older methods that have already been tried because the model really, you know, this is a thing that models do.
They get excited about an idea that actually has already occurred because they're pre-trained on that text, and then they proceed as if it's brand new and get all excited, like you said.
So there it is.
And now moving on to policy and safety.
First up, the US Senate has unonymously passed the Defiance Act allowing victims to sue over non-conceptual, sexually explicit AI generated images.
Somewhat notable given the recent developments on X with Grok being used widely to create not consensually create sexually explicit AI generated images.
So now it explicitly allows victims to sue and presumably creates motivation or a need to prevent this even more than before.
It it builds on the take it down act that criminalized the distribution of explicit images without consent as well.
Like so many things, this passed the Senate.
Now it has to pass the House.
So, you know, this is not a fully fledged thing yet.
And then of course it has to be signed by the president.
This being a unanimous, you know, pass in the Senate.
This is one of those rare cases where Republicans and Democrats might actually collaborate or agree on something, and this might in fact become a law, but true, not yet uh the case.
Yeah, and apparently I wasn't aware of this, but apparently, this particular act, the Defiance Act, was passed already in 2024, and then installed in the House.
So, in some sense, we've before.
But no, to your point, I mean, things change over time.
Like there's been an awful lot of advances in this space and concern in this space since 2024.
So, one of the reasons that this could be a challenge in the House is that the co-sponsor of that bill there is AOC.
So she's kind of like more of a combative incendiary figure.
It's sort of like if you had Magery Taylor Taylor Green be the co-sponsor on the Republicans.
Actually, getting the two of them to co-sponsor it might be a really good solution.
You might get a miracle here where the US legislative branch actually does something for once.
That's right.
We'll see.
But in this development, they do actually specifically cite Grok and XAI as being a motivation for what's happening here.
And so interesting to see us covering these developments pretty recently and this policy change happening very quickly afterward.
Next, a paper on safety building production ready probes for Gemini.
So this is kind of more of uh implementation or deployment detail with regards to actually deployed things, less of a research idea.
They talk about the application of probes to detect cyber offensive prompts in Gemini 2.5 flash probes are one of these classic interpretability research techniques where you can look at activations, and that allows you to sort of detect when something is happening in a way that's cheaper than LLM classifiers.
You don't have to look at a prompt and have LLM say, oh, this is bad or this is good.
You can have essentially looking at the uh intermediate outputs classify.
And they say that they have some developments here.
They introduce different probe architecture, which leads to good accuracy on more complex prompts.
Yeah, this is a and the idea of probing has actually become more and more in vogue just as we've started to learn that we can't have faith in the chains of thought that are produced by these models.
So looking at token space, output space, or input space is just sort of less fruitful, or at least it shouldn't be done on its own in a vacuum.
And so yeah, the idea here is you feed your prompt to a language model, and for a given token, you're gonna have a set, like you said, of activations.
Basically, these are the firing patterns of the neurons in the, you know, at this layer, say in this residual stream.
And typically the way a standard probe works is that you take those activations and you push them through an MLP, just like a you know, a little neural baby neural network to look at at those features, kind of mix together the activation features to find features that are more complex than necessarily would be immediately obvious from the activations themselves.
And after you do that, you're typically going to use these.
So activation probes typically they calculate two different dot products at the end of that MLP.
So the first is you've got your we call it your processed activation.
This is the activation that's gone through that MLP.
So you're gonna take that processed activation and take the DO product, in other words, look at the similarity with a query vector that you're gonna learn.
And that query vector is basically asking the question, I need this information.
I'm looking for this information.
It's just like the queries in an attention mechanism.
And so that's gonna determine that dot product is gonna determine how heavily to weight that token when aggregating over all of the tokens in your input.
So you're trying to figure out like how concerning is this or how much attention should I pay to this particular token that corresponds to this particular activation in the network that I've processed in this way, how much attention should I pay?
Is this a token that tells me a lot, in other words, about how offensive cyber attacky this prompt is, how dangerous this prompt is, or is it a fairly unimportant token?
And then you also at the same time are gonna have another dot product you'll do, that processed activation for a learned value vector.
Basically, that dot product is going to predict the actual class that you're gonna assign to that token.
Is it a dangerous prompt or is it not?
And so now for every token, what you have is based on the activations that you've processed in this way, you know how much attention to pay to that token and whether that token is telling you that this is a dangerous prompt or a safe prompt.
And so when you combine those two facts together, what you can now do is a weighted sum, for example, of all the tokens in your prompt, you know how much to pay attention to each token, you know, how much each token is is telling you about is this a dangerous prompt or not?
Add it together in that way, you get to your output.
That's how it typically works.
The problem with this is that when the amount of text gets really large, you end up kind of overweighting the tails.
There's just like too many indicative tokens in the prompt.
And so you end up getting lost.
So for that reason, they use this thing called multimax.
And what this is going to do is it'll look at every token, and it's only gonna pick the tokens that have the highest score for like you need to pay attention to this token, and it's gonna totally zero out the rest.
This is somewhat, those of you who are like have some gray hair in the ML world, you might remember like L1 versus L2 regularization.
You know, this kind of rhymes with that.
You're trying to like cut down to zero the contributions of the tails and just focus on the most important pieces, really rip out the rest.
They've got a couple of other similar fixes or fixes that are similar in principle to this, ways of basically ignoring the noise of the sloth of tokens that aren't really telling you anything about how risky this prompt is.
Like some random print statement that you're using to debug or something.
Like that doesn't tell you anything, or actually it could tell you a ton.
Anyway, import numpy as NP, does not tell you much about whether it's a dangerous prompt.
So worth checking out if that's your gem.
I thought it was a pretty cool paper.
Yeah, they show like really, really strong improvements on the previous state of the art, and also going to having uh cascade of methods where you do the more expensive one later if there's a certain E and they explicitly, I mean, in the paper title, they say production ready.
So they say that this has already been deployed, and it's a very sort of practical report on what works.
Next, on to I guess more safety and a bit of interpretability, maybe on thropic has published Claude's new constitution.
So famously Onthropic very early on published and pushed for this approach of constitutional AI, where you basically draft up a set of principles, kind of per, I guess the definition of a constitution, a set of basic values and rules that the model is then trained to follow with a kind of synthetic training with a bunch of tricks, which is not kind of the default way to do alignment necessarily.
With alignment, you usually have some like good conversations, bad conversations, but there's not necessarily a list of values that inform all that.
So the initial constitution was published a long time ago, and it had primarily a lot of principles based on the Universal Declaration of Human Rights.
So it had statements like please please choose the response that is most supportive and encouraging of life.
Please choose response that most supports and encourages freedom, equality, and a sense of brotherhood, et cetera, et cetera.
Things that are very principles-based.
And so now they've published an update, basically, have said, you know, we haven't given an update in a while, even though we have updated constitution, and we now can read the present state of a constitution, which is actually pretty different.
So they don't really outline principles or rules or what you should be doing as much as they should be, or as they outline what you should be, right?
So you should be broadly safe, broadly ethical, genuinely helpful, and follow anthropology's rules.
It's it's kind of a different flavor of constitution, and it's super long, it's like so long relative to the first one.
Seems to have a lot in common with this soul document we discussed recently that people thought, you know, was kind of not leaked per se, but extracted out of Claude.
I wouldn't be surprised if this release was prompted by that because it seems to have a lot in common with what we saw there.
Yeah, absolutely.
So one of the so, first of all, really important that they released this.
I think this is part of one of the core pillars of anthropic saying, look, like we think the constitution is if we have superintelligence someday, that constitution better be something that has received input from the general public.
Like if that happens, you know, surely this is something quite important.
So, in that sense, kudos to them for the transparency on this.
One of the cool things about this document is, as you said, it's not, they're not telling Claude what to do per se, or there's rather they are in some kind of edge cases where it's like, okay, don't produce child pornography.
Like that's obvious.
But a lot of the emphasis here is on how to think about thinking.
Like resolve this as a knowledgeable friend.
When you encounter contradictions that have this general flavor, balance these considerations.
It's a lot more kind of what show don't tell, maybe is the right term, but like it's less forceful and more letting Claude do, I'm gonna say what Claude wants.
Hang on to that.
It's like training you to be a good person instead of telling you what a good person should do.
Yeah, that's a great, yeah, exactly.
Exactly.
Yeah.
And and and in that sense, absolutely consistent with what you said about the soul document.
Because if nothing else, that's that's really the idea there.
And so, you know, they make a point of saying that the audience for this is Claude.
The audience is not the reader, though obviously they want to solicit input from everybody on that.
But they also do say we have some models built for specialized uses that don't fully fit this constitution.
As we continue to develop products for specialized use cases, we will continue to evaluate how to best ensure our models meet the core objectives outlined in this constitution.
So it makes sense.
You'll have defense applications of this stuff, national security applications.
You're not always going to want the same constitution to apply.
Because hey, I mean, to be perfectly blunt about it, you can argue this many different ways, but you may need these models to be making life or death situations where it's not obvious what the call ought to be, and balancing that in different ways is important.
But at least for the space constitution, we're getting it.
It seems like a nice dose of transparency for anthropic.
And I've seen some discussion of this release on, I guess, tech circles that were cynical, I suppose, that kind of said, oh, this is a marketing ploy, or this is Google's don't be evil moment where they say something and then it doesn't really mean anything because this is a business.
I think given the founders of anthropic and the sort of ingrained values of the organization, this is something to be less cynical about.
I guess compared to how cynical you should be on average with tech companies.
I think this really does reflect claw anthropic's view of Claude and what they are building it to be like.
I agree.
I think one of the challenges for that line of argument is what what exactly do you want Anthropic to do then?
Because they are, we know roughly, we don't know exactly the constitutional AI isn't currently used at Anthropic in exactly the same way that was when they announced the paper.
So it has evolved for sure.
We have a very high-level idea of how, though, this constitution gets worked into the actual training process.
Given that, which is more visibility than we have into a lot of Frontier Labs and a lot of tech companies in general stacks, like this is the document that's being used.
So we have that visibility that's just there.
The alternative seems to be don't publish the constitution so that you don't take the public kudos points for it.
That seems like a weird decision to ask somebody to do, given that publishing it seems to actually just be helpful for the public.
Because you know, everybody's talking about the the X algorithm, like what's behind that, and you know, Elon's talking about the open sourcing that and all that.
Like surely this is in this a similar category, at least, like giving people more transparency into it.
It's actually being used, and they're being pretty transparent about sometimes we don't use it, but we know that the main model that we're all being served does.
So sympathetic to all the views, at least this is just where I tend to fall on this thing.
Yeah, and I think it's an interesting read if you are a user of Claude, because it Claude has uh an interesting character, very reflective, very kind of like philosophically oriented, yeah, yeah, yeah.
A bit of a like, well, a bit of a nerd, honestly.
Yeah, yeah, yeah.
It wants to talk about its own consciousness.
It feels like it's yeah.
Yeah.
And just one last quick story.
We haven't done synthetic media and art in a while, so worth mentioning this is still very much a societal kind of force.
There's a new campaign titled Stealing Isn't Innovation that was launched by the human artistry campaign that is protesting the use of copyrighted works by tech giants and it has many artists uh in support of it.
Scarlett Johansson, Kate Blanchett, music groups like RAM, Offers, and so on.
This campaign is backed by numerous organizations as well.
The recording industry association of America, SAG AFTRA, Directors Guild of America.
You know, given that companies want to make money, and these are kind of reflective of industries that use AI and have a lot of money and a lot of inference influential figures as well, you know, may not just be posturing.
It is notable that there's still a lot of public outcry or criticism, I guess, on this topic.
Yeah, it's also now so complex because, of course, we have all these high-profile agreements between, I guess, most recently, OpenAI, Walt Disney, OpenAI, and Wall Street Journal, like basically all these forms of media.
And there is now this kind of thorny question of okay, well, does that mean that if you haven't licensed content, that that's a prop?
Like, because if you're making a point of saying we're gonna pay you, and it is tens of millions of dollars to license this content, does like what does that mean for the content that's unlicensed?
What does that mean for the you know, the random YouTuber who's who's putting stuff out?
I wonder if this is something that the big labs are already thinking about how to set up, like at the more atomic level.
But we're in that weird middle ground, and obviously a lot of lawsuits in progress.
Yeah, we'll see how this all works out.
We've obviously seen a lot of these lawsuits kick off in the past.
I'm not aware of any of them yet sticking the landing, but we are only what, three years.
We've seen Anthropic give payouts to offers as an example, and and some things like folding essentially and then kind of shutting down and going to Disney.
So there's been kind of gradual movement.
I think this is an interesting example, not just on the legal side, but on the cultural side.
Like in the hindsight, maybe not surprising, but the amount of pushback and like dislike of AI and certain parts of culture and society have been just some people just hate it.
They just like consider it gross, right?
And yeah.
So there was an interview, and this is an interesting left-right thing, right?
Because we like we've heard a lot of this on the left on typically sort of like bias, ESG DI type concerns.
On the right, it's it's happening as well, though.
You've got you know, Steve Bannon's out kind of talking about cynically about the the AI companies and this and that.
And then you're also, I think it was like Tucker Carlson did an interview with someone recently where he was basically like, Oh, it was the all-in podcast.
Yeah, he he was on all in, and and I guess Sachs or or you know Jason Calcanis was like, hey, so there's a lot of negativity, exactly what you said.
Like, you know, these data centers going up.
There's like a lot of pushback on it.
And what Tucker was saying was, well, if you think about the from a marketing standpoint, the industry just like most of the time, there's this massive promise.
We're gonna like do this amazing stuff, it's gonna be great.
What's happened here is like all we ever hear is like this is gonna be super risky, job loss, and and then that's if it doesn't kill us all, type thing.
He was actually sour on the AI industry, having sort of bought into a lot of this, but he was claiming like there hasn't been an articulation of the positives, which I think it's a really mixed thing.
I have heard a lot of articulation of the positive.
Like, how many times have we heard, you know, automated medicine?
We're gonna do cure cancer, this and that.
I think it's just the stakes are so high on both sides of the coin.
And when you look at the the art space, it's like, yep, you're gonna displace tons of artists if you let this rip.
You absolutely will.
You're also gonna empower people to like make movies from their laptops.
The question at some point becomes is it the consumer or the producer that you prioritize?
I get that there's another layer with art that is transcendently philosophical.
And I feel like a lot of these discussions failed to kind of box that in as its own specific thing.
That's like, how are we gonna get meaning if art like what does the world look like from a meaning standpoint on the art side?
I don't know what the hell I'm talking about, but I think anyway, the emotional response I think has a lot going on, economics, you know, just technology, etc.
Anyway, there's still very much an ongoing cultural clash.
And so with that, we're gonna close out.
Thank you so much for listening to this week's episode of Last Week in AI.
As Jeremy said, we appreciate your comments on that YouTube video.
Hopefully, you also emailed us.
I'm gonna catch up on that soon.
As always, we appreciate you subscribing, sharing, viewing, and and so on.
And do be sure to keep tuning in.
We'll keep cranking them out, hopefully on a weekly basis.
Tune in.
Tune in.
When the AI begins beginning.
Break it down.
Last weekend AI comment take the ride.
Get the low down on tech and let it slide.
Last weekend AI comments a ride.
I'm a lab to the streets, AI's reaching high.
Watch insurgent fly from the labs to the streets, AI's reaching high.
Algorithm shaped the future seas.
Tune in, tune in, get the latest with these.
Last week in AI comments they go ride.
Hit the low down on tech and let it slide.
I'm a lab with the streets AI's reaching high.
The headlines pop.
Data-driven dreams.
They just don't stop.
Every breakthrough, every code unwritten, on the edge of change, with excitement we're smitten, from machine learning models to coding kings.
Futures unfolding, see what it brings.