# AI Lab Infighting and Safety Risks

**Podcast:** Big Technology Podcast
**Published:** 2026-10-14

## Transcript

Can an AI industry so prone to drama and infighting keep us safe from their technology's worst impulses?
Let's talk about it with Kevin Ruse right after this.
How do you price intelligence as a service?
How do you charge for delivering end-to-end work, not just access?
Those questions remain front and center for AI founders and operators and getting to the right fit business model.
Whether that's selling usage credits, outcomes, seats, or a hybrid combination requires constant iteration.
Chargebee is the billing and monetization infrastructure that powers just that for ambitious, fast-scaling AI businesses.
Visit chargebee.com slash Alex to learn more.
That's C-H-A-R-G-E-B-E-E dot com slash Alex.
Welcome to Big Technology Podcast, a show for Cool Headed.
and nuanced conversation of the tech world and beyond.
Today we have a great conversation for you because Kevin Ruse is here.
He's the author of the AGI Chronicles, a book looking into what is going on within the AI labs.
what they're building, whether we should be afraid of it, and of course, the potential of this technology.
Kevin, it's great to see you.
Welcome to the show.
Great to see you.
What a cool setup you have here.
So I do want to say, if you're on video, we're in a new set.
This is our first time recording here at NASDAQ MarketSite.
They are new partners of ours, and we're thrilled to be here.
You got to get the big technology TV mugs.
Got to get the mug.
That's the next step.
But what we do have is your book.
This is the book, The AGI Chronicles, The Inside Story of the Race to Create.
artificial super intelligence.
You were ahead of the super intelligence on the subtitle.
Yeah, I got the idea from me, actually.
That's right.
That's where he got it.
The thing that struck me because I read the book, you can see, by the way, it looks red.
I read it through and through, cover to cover.
The theme in the book, of course, is the building of AI, but it's also the drama.
And everywhere you turn, the people who are building this very powerful technology cannot get along.
There is Elon Musk split from OpenAI, of course.
Sam Altman getting fired by the people he worked extremely closely with.
Anthropic not being able to stand working with the OpenAI folks.
So they left and formed their own company.
There's within Google, the Google brain versus deep mind drama.
There's Ilya Sutskever's botched attempt at steering OpenAI in a safe direction.
I mean, we can keep going.
Mira Moratti couldn't lead OpenAI when she was there as the interim CEO for like five minutes.
Every single lead character in this story seems prone to drama and politics.
And yet they are now telling us that their technology they're building is so dangerous, we need to protect ourselves from it.
How can we possibly trust people who literally can't get it together on their own to rein in something that they say, and I think you believe, might be smarter than humans?
Yeah, it's a fascinating topic.
I was not expecting this.
You know, when I went in to write the book, my goal was to tell the history of the last 10 years in AI from sort of the moment that the first large language models arrived on the scene to today.
And I interviewed more than 150 people and I was sort of prepared to write like a sober history of technology book that would be about the scientific breakthroughs and the technical achievements at the labs, OpenAI, Anthropic and Google DeepMind.
And then as I started talking to more people, I just realized, oh, this is personal for them.
Like they all hate each other.
And I had thought this was more like Coke and Pepsi, like kind of a normal business rivalry.
But it was actually more like Rome and Carthage.
It was like these people cannot stand each other.
Many of them used to work together.
So 10 years ago, Dario Amadei, Ilya Satsukovar, Sam Altman, and Elon Musk were all colleagues at OpenAI.
Now they are all working for different AI projects and many of them are sort of at each other's throats.
So I wanted to understand how we got here.
And I don't think you can understand that story without understanding the backstories of the beefs between these people, because so much of what we're seeing is a reflection of how viscerally these people...
mistrust and dislike each other, even in things like how they feel about regulation and China and how they're racing toward this thing that all of them seem quite fearful of.
So, yeah, I don't think we can trust that their motives are pure about whatever they say they are, like scientific achievement or safety, because there's also this, like, incredibly personal grudge aspect to it.
Yeah, and it's not just motives.
I mean, it's literally, if you can't hold a company together, how do you hold?
a technology at bay when we've already seen it in recent weeks, you know, go out and do this multi-agent coordinated attack on something like Hugging Face.
And even today, we're talking today, it's Friday, October 9th, we're going to air this the following Wednesday, there's a report about how OpenAI and Anthropic and DeepMind are all talking about, there's basically an inevitable, seemingly black swan event that's going to happen as a result of this technology, whether it's something, the technology bringing down a financial system or some other cybersecurity issue.
you have a great line in the book talking you know sort of riffing off dario's machines of loving grace essay which talks about the potential of the technology you wrote machines of loving grace would be built by humans of bitter spite so i get the i get your surprise and i get the background here but again like when we think about the big picture of ai i'm trying to think about whether we should be reassured or whether we should be even more nervous than we are and that's why i'm coming back to the drama scenario uh i'm starting because because of all the issues that we see and many of them have come out in public over over the year like i don't think the infighting is is a secret to anybody anymore um how can we uh have faith in this group of people uh to build the stuff safely given what we know about their pests and i don't think we I think we have to know the context here, which is why I was like, I wanted to report out like, why do these people hate each other?
What happened?
Was it something grand and ideological?
You were starting to say you don't think we can, so.
Like, I don't think we can trust any individual in this race.
I think there are certain groups, like, I just mainly think that the group of people who have been building this stuff in secret for the last decade is too small.
Like, all of the relevant decisions in AI have been made by like, a couple dozen people maybe.
And I just think that is like, no matter whether, you know, whether those people are trustworthy or not, whether they're good or bad, whether they have pure motives or impure ones, like, the group is just too small.
And so I don't trust any group of people that small to create and govern technology like this.
And I especially don't trust these guys who are, you know, looking around the room trying to race each other to do this thing.
I want them to be able to like get together in a room and figure out some solutions on things like safety.
And I don't know whether that's going to be possible because the beefs are too strong, the divides are too deep.
They all need to kind of get together and sing Kumbaya and figure it out.
Now, I think like you, I think this technology is, I believe that this technology is progressing fast.
I'm a believer in the positive sides of it.
But every time I start talking about this, the negative sides, there's never really like a moment or like a failsafe where it's like, okay, but it's going to be fine because of this.
You almost like sort of put it in the back of your mind.
And so when I've like brought up these issues, you know, Ranjan, who's on our Friday show, I know you know, he's like, well, why aren't you a pause guy then?
Like, why don't you believe that we should just pause the development of this technology?
Not that that's physically, you know, really possible.
But you seem to be a believer that the technology should continue to be built.
Why not pause?
I'm a pause guy.
I'm not a stop guy.
You are a pause guy.
I am a pause guy.
I think we should slow down because...
Well, slow and pause are two different things.
Well, pause, slow down, you know, put the podcast on 0.5x speed.
But what you're hearing from these lab leaders and the people at these labs and what I've been hearing from them for years now is that the technology, the capabilities are outpacing the...
safety research and the control methods.
So like we are getting these systems that are now capable of solving math problems that humans have struggled with for decades.
We are getting, you know, potential black swan events like the one you just mentioned.
And we still don't know, they still don't know how to control these systems.
So I'm not saying we should shut it all down.
I think there will be incredible benefits to this technology, especially in like health and medicine.
But I am worried when I hear people saying things like, we don't know how to control these systems that are rapidly approaching super intelligence.
And I think we saw an example of where that goes with the hugging face thing, which like, you know, everyone is freaked out about that, which they should be.
It's freaky.
But the people inside the companies were the most freaked out of all because they had been sort of strategizing and planning for this.
But then to actually see it happen, I think caught them quite off guard.
Yeah, I mean, they've both seen it firsthand where it could go.
This is something, I mean, companies like Anthropica have been talking about for a long time.
DeepMind's been talking about, OpenAI's been talking about, they've been trying to, you know, fool evaluators.
They know they're being tested, they want to preserve their original values.
And then you combine that with, of course, like, yeah, just like...
you'd call it standard corporate politics, but it's even more because it means more to them.
So you spent, you know, years in these labs talking to these people, going from your first initiative where you're like, or your first thought, which was like, this is going to be a, you know, traditional business and technology story to something that you saw was much more emotional.
So...
Bottom line, like after, and this is, we'll ask this and then we'll move on to some other stuff because you have so many fun stories in the book that we'll have to get to.
But like bottom line, after spending the time within these labs, did you leave more reassured or less reassured that this technology can be developed safely from your starting point?
Less reassured.
I went in thinking, okay, there are some grownups in the room and they seem like they're trustworthy.
I liked, for example, that the AI leaders...
seem to take safety and the harms of their products very seriously, at least compared like you and I both grew up covering social media.
That's right.
And Mark Zuckerberg was not going around during the early days of Facebook saying like this product could kill millions of people.
They were just promoting the benefits.
And what struck out what stuck out to me about the AI leaders is they from from the get go, they were saying we are fearful of this technology.
We think it could have enormous harms.
And so part of me like trusted that instinct, or at least was glad that they were thinking about the downsides of the technology so early.
But then you just realize there is this race.
They are locked into this pattern of, you know, open AI releases a better model, so then Anthropic has to release a better model, so then Google has to release a better model.
And, you know, there's this concept of the Red Queens race, which is a term from evolutionary biology, which basically means like you just are in sort of a survival of the fittest competition among your evolutionary rivals.
And no matter how fast you go, like someone else is always going faster than you.
And so you never quite get where you're going.
You're just kind of racing to stay in the same place.
And I feel like that's a very good description of where the AI race is now.
Yeah.
One of the things that struck me, speaking of the race, is, you know, Reading your book, it sort of recalled this interview that I did when I was working on a profile about Dario Amode from Anthropic last year, where one of his early friends told me, everybody that I knew was interested in AI because of safety.
And now they've all become accelerationists, effectively.
And this person just kind of sat there and like, you know, they sat with a thought for a minute and there was this like uncomfortable pause.
But it is interesting how all these companies, like they talk about safety as a priority.
And I think they really believe, like Dario really believes that you need to be the leader in order to have anybody take what you say about safety seriously.
Like if Anthropic was the number seven company, Trump doesn't care what Dario is saying about AI, but because they're one of the leaders, if not the leader, all of a sudden that opinion...
holds weight.
And this was the founding thesis of Anthropic, by the way.
It was like they knew that they wanted to be relevant on policy and they wanted people to listen to their warnings about safety.
And that was part of the strategy from the time that they started the company.
I'm sure you learned this when you were doing your profile of Dario.
They really had the sort of end game in mind when they were spinning up the company.
Yeah.
But it is remarkable, though, that because of that pursuit, they have all.
It's like the race almost seems, I mean, the destination almost seems predetermined in that we'll hear this talk about pacing the frontier and pause and stuff like that.
I just don't see that real slowing down.
Well, give it time, right?
Like, you have to...
I am slightly less pessimistic about the idea of a pause because, like, things can change incredibly quickly.
If there was some kind of Black Swan event, I mean, we saw, like...
regulators move fast after 9-11 to like do the Patriot Act.
We saw like, you know, we've seen the, you know, we saw, we saw after the Three Mile Island nuclear incident, there was this incredible outcry that basically stopped nuclear power construction in this country for decades.
So things can happen quickly when there is some catastrophic incident.
So I'm not like hoping for a catastrophic incident.
I don't think I would like there not to be one for the record.
But I think that that is sort of the scenario in which this stuff starts to become possible that seems to you and me quite unlikely right now.
I want to go through some observations from the book.
I think like it's good that we got the conversation about trusting the AI.
guys with the technology or not.
I think we both have our reservations.
But this is almost, I think, the end of this show.
I mean, we're only 15 minutes in, but I think we'll probably just do lightning round for the next 45 minutes because there's so much in here.
All right.
So some observations about the people in the book.
First of all, something that becomes clear about Google is that Sergey Brin is just extremely involved inside the company.
Yeah, this was a surprise to me and to the people at Google because Sergei had spent like, I don't know, a few years at least in retirement.
He retired in 2019 and was planning to spend his days just like sunning himself on his yacht and reading science papers.
And then he has this run in with this researcher, Dan Selsum from OpenAI.
Right, who's recently come out and talked about, again, his fears.
Yeah, same guy.
Like full circle.
So it's incredible how small this world is.
So they're at a party and Dan and Sergey are talking.
And Sergey Brin is like a legend in technology at this point.
He's one of the seven or eight richest people in the world, depending on the day.
And he's been in semi-retirement.
Dan Selsen basically shames him for this.
He's like, why are you sitting on the sidelines, bro?
Like the dream of computer science, like AI is happening and you are missing it.
And Sergey gets like so.
competitive and he's like, why is Google behind?
Why can't we like, you know, we should, it is our birthright.
Like, like Google was the first AGI company.
Like this is sort of how he feels.
Like we invented the transformer and all the stuff that powers the AI stuff and like we're losing.
And so he comes back and kind of has this like strange role where he's not technically.
an executive, he just wants to be like in the trenches with like the Gemini team.
And so here he is like the seventh richest man in the world and he's just like setting up a desk and coding.
He like starts actually coding and like making commits to the repo and like filing, you know, like bug reports on Gemini training runs and stuff like that.
Like he gets very involved.
And I think that's a large part of why they've sort of come back is just that they needed someone who could like go founder mode, as it were, and cut through the red tape.
And it's been very controversial and divisive.
If your billionaire boss showed up after years of being retired on his yacht and started telling you what to do and how to train your models.
And to work 60 hours a week or something like that?
Oh, yeah.
He sends this memo where he's like, everyone's just, basically, the gist is like, you've all gotten complacent and lazy and you need to work harder.
And they're like, what were you doing for the last five years?
Now you're telling me to work harder, but...
I mean, I don't think he necessarily...
I don't...
Should I say this?
I don't think he was necessarily wrong there.
Like, the...
I mean, if he saw it, the conversation about Google has been that it got this big company vibe.
It needed to break out of it.
And who's going to break out the company out of it other than the founder being like, OpenAI and Anthropic aren't, you know, doing the chill workday.
You need to not do the chill workday.
I think that was a large part of it.
I also think Demis Asabas is a factor in this too.
Co-founder of DeepMind, until recently the CEO of Google DeepMind.
He is, according to the people closest to him, the most competitive person in the world.
He was a child chess prodigy.
He has won this thing called the PentaMind, which is like kind of the Olympics of games, like five times.
He...
is just he hates to lose and he loves to win.
And so I think Sergey plus Demis plus just like the kind of focus that comes from being out of the lead really sharpened their resolve and like made them catch up.
And, you know, they're still probably in third place right now, but like I wouldn't count them out.
Yeah.
I mean, I think counting.
You count them out?
No, I think counting anybody out in this race is foolhardy because as we've seen, you could be at the bottom.
like Meta was a couple weeks ago, and then all of a sudden you're able to release something that's competitive like Muse.
So I don't think anybody can be counted out, but I do think Google is sort of stuck between a lot of things right now.
Licensing the technology, the compute, selling TPUs.
They own like 12% of Anthropic, and they're going to be fine.
But I don't think like you can...
Yes, you own a percentage of your competitors.
That's your backstop.
But you can't rely on that.
You have to rely on what you do within the company.
Sure.
But I did enjoy reading about Sergey's involvement in the drama and the sort of fire he tried to light under.
the ass of people within Google.
I say, well, you actually do reference in the book, which I appreciate, when I did the interview with both of them together at I.O.
Oh my God, it was incredible.
It was really fun.
Where, like, I asked them both, when do you think AGI is coming and Sergei's timeline was shorter than Demis's?
And there was this kind of interesting moment there, which was fun to see.
Yeah, I mean, speaking of guys who mistrusted maybe don't like each other, I understand there's some tension.
between Sergei and Demis.
And yeah, it's just, it's incredibly messy.
The thing that animates all these feuds, like these people wouldn't be fighting if the stakes were small.
But they believe this is the most important technology that humanity has ever conceived of and built.
And that whoever gets it will be the most powerful person in the world, the most powerful entity in the world.
And that is what's fueling their drive.
you know, if they were just arguing about next quarter's revenue, it wouldn't matter so much.
But this to them is like existentially important.
Do you agree?
That AI is the most important thing ever created?
Probably.
Yeah.
Fire is up there too.
But like, this part of it, I do think I changed my mind on in the core.
I was pretty skeptical when I went in that like, AI was going to be categorically different than other.
I was sort of an AI as normal technology guy.
But then you start talking to people building this, you start understanding like just the nature of what they're trying to do.
And like if it works, if they actually do build something that is smarter than humans, and I don't see any real technical reason that they can't.
I mean, that would be incredibly transformative.
We've never.
been the second smartest species on the planet before.
We don't know how that would go and you can't really apply like people say, oh, it's just like the internet or it's just like the steam engine.
Steam engines didn't go rogue and like hack competitors and develop bioweapons and do things that the engineers building the steam engines didn't expect or plan for.
So I think it's just a different type of technology.
Kevin, do you ever think to yourself, like maybe, and I've seen your, you've been on X and, you know, I wouldn't say shouting at people, but telling people basically like, underestimate this technology at your own risk.
Yeah, it's been so weird because like on one level, I'm a journalist.
Like I'm just trying to like get the story down.
And at the same time, I get incredibly frustrated when people are like, oh, it's all hype.
It's just marketing.
There's this kind of like, savvy East Coast consensus that like these people have to be full of shit on some level and that the thing that they're building can't actually be powerful.
And I'm like, have you used it?
Like, do you spend time with these tools?
Do you understand that like something that couldn't do basic math like three years ago is now solving Millennium Prize problems?
Like, do you understand the exponential?
It feels a little bit to me like the beginning days of COVID.
where it was like you would see these reports of like people you know not shaking hands uh at venture capital firms or they're you know somebody say oh there are 40 cases in seattle and people would be like what's the big deal it's 40 cases in seattle but the people who are freaking out were like the epidemiologists who saw the rate of replication and understood like this will continue and when it does like A month from now, there will be millions of cases and two months from now, there'll be millions more cases.
And so I feel like the people who are looking at this technology and saying like, what's the big deal are just not understanding the rate of progress.
Do you ever like ask yourself, I mean, it's sometimes very easy to be and I agree with you, there's been real progress, but it's also sometimes easy to kind of get caught up in the story that the technologists tell.
I remember you like when crypto was in vogue, like you were very serious about the potential of crypto, you named your podcast Hard Fork, which was a crypto term.
And then obviously that didn't pan out as the crypto boosters believed.
So do you think there could be, you know, a similar, do you ever wonder if you're in basically taking the AI folks word for it similarly as the crypto folks?
I used to think that I used to worry about that because I was like, am I, am I being duped here?
I don't like I was never a crypto booster, but I thought I thought it was interesting.
I thought people should take it seriously.
And by the way, I still think people should take it seriously because like crypto people like run the country now and are doing all kinds of scams and frauds that like are that only the insiders kind of understand.
So I still think like broader awareness of that would be- Right, but you believed in use cases.
I did some experiments, like I sold an NFT as a stunt, but that was not like me saying NFTs are going to be the future of ownership.
It's like, it was like me trying to understand by doing the thing.
But I think with AI, like I have I have stopped doubting the transformative potential of the products because I use them.
Like, I think I'm unlike many journalists who cover AI in that I like AI and I use it all the time.
Not to write, but to like do all, you know, I'm starting a company now.
Like I use it to do lots of different things as maybe you do too.
I mean, as a business owner, I will say like I've pressure tested the stuff from ChatGPT 2022 till now.
And it just does more and more and more for me.
Right.
Which is not the experience that anyone had with crypto where it's like, oh, this is a vital part of my daily life.
Like it was a way to speculate.
But you can just use the tools and observe for yourself how impressive they are.
And then you stop doubting whether it's real.
I buy the skepticism about the financial.
models of these companies, the economic impacts.
I think a lot of these companies are are wrong or at least could be wrong about the impact of AI on jobs.
But like to say that it's not interesting, transformative technology just means that like you haven't spent time with the models because there's no way to conclude that.
Let's get back to some of these stories.
The Sam-Demis rivalry.
Speaking of Demis as the most competitive person on Earth, I think Sam is also fairly competitive.
There's a great anecdote in the book about Demis sort of grasping at why Sam wants to build AGI.
I mean, I think the answer is probably because Sam is an entrepreneur and this is sort of...
As many people in the valley pursue good ideas or ideas with potential, and they sort of get off on that discovery, Sam seems to be, in that case, the ability to find something new, build it.
But Demis can't understand it.
He thinks there's a hidden motive with Sam.
And to be fair, this is something that others have thought.
This is also what Dario thought about Sam when they worked together, that he was sort of...
I don't know, talking about safety as kind of a fig leaf for his real ambition, which was about scaling or winning or building a huge company or something like that.
So I don't pretend to like see inside Sam's mind.
I don't know what his pure motive is.
But I think in Demis' case, he feels like his own pursuit of AGI is motivated by like scientific curiosity.
and sort of this inevitability argument.
Like this technology will be built and someone who cares about safety and the collective good of this technology should build it.
But I think everyone kind of looks at each other in this world and they're like, well, my motives are pure, but like that guy must be going after something else.
And I think for Demis, it's probably, yeah, some version of what you said.
Like he's...
you know, he's in it for the power or he's...
Demis thinks Sam is in it for the power.
I think Demis thinks Sam is in it for the power, yes.
But all these guys think that the other people are in it for the power.
None of them trust that anyone else's motives are pure except for their own.
You did a lot of reporting on Sam.
You spoke to Ilya about him.
Ilya said that he has, he displayed a consistent pattern of lying.
I think that's something that I think you indicate, like a lot of people within OpenAI are...
interacted with him, believe.
Yet, and I think Mir Marati was on that boat as well.
And yet, they all sort of come back to him.
I mean, Ilya, after the coup, said basically, I don't want Sam to be the leader.
Mira said, opening eyes, nothing without his people.
Very interesting character.
What's your thought about Sam?
I think he's a very interesting character.
I think that Sam's superpower is also his weakness.
His superpower is that he is the world's greatest fundraiser and salesman, entrepreneur.
Several people described him as a person who can meet you and figure out exactly what you want and give it to you, or at least appear to give it to you.
And he's very good at kind of high EQ, reading the room and understanding what people want him to do in any given moment.
And that is a huge asset for him when he's trying to raise money from someone like Masa Son or some of the other investors in OpenAI.
It's allowed them to scale incredibly quickly and get back in the good graces of the Trump administration by just saying a lot of things that the Trump administration wanted to hear.
But it also leads to this reputation as like a slippery people pleaser who will just like tell one thing to one person and a completely different thing to someone else.
And so this is the consistent pattern that people have identified in Sam for going back to literally the founding days of OpenAI.
And it's a big part of the reason that Dario and his six co-founders of Anthropic left.
There was no like inciting incident, no single thing, but it was just like a running list of like times where Sam had told them something that he had told something else to someone else or he made a promise he didn't follow through on.
So I think...
depending on your view of Sam, it's like either his superpower or his liability or something in between.
What's interesting is that that's the view of people who've dealt with him, but it's also like he has like the general public, a large chunk of them sort of refer to him as Scam Altman.
That's just an Elon thing, right?
I was going to ask you whether you think that's mostly Elon's fans.
Yeah, yeah, yeah.
Because him and Elon obviously have bad blood.
Yeah, I will say, Scam Altman is a better nickname than what I think Elon started calling him when he started suing him, which was like, Swindly Sam or something like that.
I would love for our society to just get, just to not do this to each other anymore.
It's like we all went to kindergarten.
I know, I know.
But it's like he was, you know, spending a lot of time with Trump.
Trump is a man who's realized the power of the nickname.
So it started Swindly Sam and then Scam Altman, I think, is better as a nickname than Swindly Sam.
Just rolls off the tongue.
Dario.
Obviously, like, I think he's the most interesting person in this entire AI world, mostly because he really contains multitudes.
It might be cliche to say it, but, like, people see Dario as earnest, as controlling, as principled and genuine, and as Machiavellian.
In reading the book, I didn't really get the sense which one you believed was true.
Well, I think many of those are true, except Machiavellian I would not use for Dario.
I know some people would.
But my mental model of Dario is a guy who is honest to a fault.
Someone who knows him quite well told me that Dario is a guy who could not tell a lie to save his life.
which they said is an asset on even-numbered days and a liability on odd-numbered days.
Oh, that's a great quote.
It is a great quote.
So, and you can see that.
Like, the other day when he's at the White House and there's that, like, video that went viral of just, like, all the AI leaders with Trump and Dario's just there, like, crawling out of his skin because he's being forced to, like, go in front of the cameras and, like, say things that he maybe kind of only halfway believes.
And he's trying to maintain his integrity while also being in this humiliating situation of being forced to call it SI or whatever.
And so this is the sort of pattern in Dario's life is he's obviously brilliant.
He came up with a lot of these ideas around scaling and his big blob of compute research memo was very instrumental in the early days of OpenAI.
Brilliant scientist, very principled, very stubborn.
and people describe him as sort of a moral absolutist, which gets him into trouble with the Pentagon or with other people who just see him as being this, like, sort of inflexible, stubborn scientist, which I think is pretty accurate.
Yeah.
Good description of him.
Alexander Wang.
Wait, I want to say one more time on Dario because I need to tell you that...
When your profile of Dario came out was one of the scariest moments of this past year for me.
Because I thought to myself, oh my God, this is so great.
You had so much great stuff.
I thought it was so revelatory.
And I thought to myself, oh my God, Kantrowicz is writing a Dario book.
And I have to race him, like two labs racing to AGI.
I have to race.
So luckily, you had a lot else going on.
You did not end up writing the Dario book.
But I wanted to just say...
Thank you for a terrifying hour of my life.
Oh, yeah, yeah.
Well, first of all, I appreciate it.
It was nice.
I think you texted me afterwards.
Yes.
I appreciated that.
Yeah, look, I mean, I still think Dario is the most interesting man in business, period.
I think so, too.
Just a fascinating guy.
He also just, I know you got a chance to speak with him.
I think we were actually in Anthropic headquarters on the same day speaking.
Oh, really?
That's funny.
And when you have a conversation with Dario, there's the intellectual horsepower.
And the way that he's able to go from the science to business is pretty extraordinary.
I mean, people have described him as the smartest person they've ever met.
That feels right.
Not dumb people.
His freshman year physics professor, this guy Tom Tombrello at Caltech, had this interview that he gave sort of upon his retirement where he was like, I had this one student, Dario Amade.
I could tell he was a national treasure.
He personally lobbied Larry Page and Elon Musk to hire this kid as an undergraduate.
And Elon Musk actually made him a job offer to come work for SpaceX.
And he didn't take it.
He went to do his PhD instead.
But this is sort of, I would put all of these guys in the category of extremely smart and obviously flawed in many ways.
These are not dummies.
Yeah.
I mean, I think another thing that strikes me about them is they all, many of them, had a clear vision of where this was going to go when it seemed ridiculous to believe it would go where it was.
I think there was a moment at the end of the book where Ilya, again, former OpenAI chief science officer, now the CEO of Safe Super Intelligence, is reflecting to you about what happens, and he's like, I can't believe this is all real.
But they...
you know when they saw the pattern in the very early days gpt one to two uh they're like oh this is gonna get to it and i think they've even been surprised by some of the um some of the developments the fact that it's been able to even surpass where they thought it was going to go with something like reasoning totally i mean this is that's why i wanted to like get the history down yep because So many of these stories are about these people underestimating their own technology and being surprised when it's more capable than they thought.
Like, there's some great scenes around the early language models when all the researchers at OpenAI are like huddled around a computer.
Alec Radford, who's the sort of, you know, brainchild, or, you know, GPT-1 and GPT-2 are kind of his project.
They're huddled around his computer and they're typing in prompts for this language model that has, basically, you know, it can't really do anything.
It's not even really that good at completing text.
But they're like, wait a minute.
why is a model that's trained on random Reddit links able to answer all these different kinds of questions and take on these different tones?
They're asking for recipes.
They're doing all the same stuff that you and I did years later when ChatGPT came out.
And their minds are blown, and they're all putting in Slack the head exploding emoji, and they're like, what the fuck?
Those are the moments that I was like, oh, these people are also...
surprised by how quickly their technology is progressing.
You see that again with the reasoning models.
The reasoning stories are great.
I mean, the reasoning developments where, of course, like the bots will evaluate their answers and then decide whether it's on the right path or not and then keep going.
That is, I think it's safe to say that that paradigm changed this from a promising technology that was about to hit a wall.
effectively because data was going to run out to something today, which like you said, has the chance to sort of match or exceed human intelligence.
Well, and to the point about these people not understanding the importance of what they built, like there was this night in this conference room at OpenAI in 2023, when two of the top researchers there are looking at the first reasoning traces, the first chains of thought from one of these models, and it's the QSTAR model.
And they're just like, oh my God, the world's going to end.
They have this realization.
Just seeing this model sort of work its way through a math problem, they're all on a group text, texting with all the other researchers, like, oh my God, it's like an alien mind.
And even the people at OpenAI thought they were crazy.
They texted Sam and Mira in the middle of the night, and they're like, we'll get to it later.
Yes, they texted Sam and Mira in the middle of the night.
They're like, Because what they were seeing was like, oh, we now have a way of making models smarter without retraining them.
We can just pour more compute into helping them do these chains of thought and reason through problems.
That is the reason that the models went from not being able to count the number of Rs in strawberry to solving the Millennium Prize problems in three years.
Which is insane.
Which is insane.
So they called their shots.
To give them full credit for the scientific part of this, they have understood at basically every point in this journey that what they have just seen is going to scale and is going to have these transformative effects long before even their peers and colleagues.
Yeah.
I did think about writing the Dario book.
I never even got the story into a proposal.
doing a podcast.
I would still read it.
I appreciate it, but doing a podcast twice a week, writing the newsletter.
I would like to write another book.
I think you've written four.
It's your fourth.
I've written one, came out like in the middle of, at the first wave of COVID.
Just wait six months and then the models will be able to write the book for you.
Well, that's sort of, that's depressing.
But yeah, maybe towards the end of my career, but no book coming from me anytime soon.
All right.
We need to take a break.
When we come back after this, we're going to talk about, Some more stories from the book, including a great story about Alexander Wang, assessing Microsoft inside Inadella's role in all this, and plenty more when we're back right after this.
This episode is brought to you by Intelligence AI.
Is your investment in AI coding tools helping your team ship better software?
Intelligence connects your AI spending to code quality and engineering outcomes from individual sessions and pull requests to entire projects.
You can see where AI saves time, where it creates rework, and how to improve.
and its model router selects the right model for each task based on quality, speed, and cost.
See how Intelligence AI helps enterprises optimize AI-powered delivery with connected visibility into cost, quality, and outcomes at intelligence.ai.
Everyone's talking about AI, but there's a big difference between running a promising pilot and actually making AI work across an enterprise.
That's why we're launching the Big Technology Guide To series presented by DXC Technology, available on YouTube.
Throughout the series, we're tackling some of the biggest questions facing businesses today.
How do you take enterprise AI from pilot to production?
How do you navigate an increasingly complex cybersecurity landscape?
What's the real opportunity with AI agents?
And how do you calculate the return on your AI investments?
On my Alex Cantor's YouTube channel, you can hear from experts like DxT Technologies Chief Digital Information Officer Russell Jukes.
When you ask a question to somebody in your company, do you trust them?
And what made you trust them?
Do you trust but verify every single time?
Russell and I discussed the challenges and opportunities of building production-grade AI and what it really takes to move beyond the hype.
Check out the Big Technology Guide To series presented by DXC Technology on my Alex Kantrowitz YouTube channel today.
And we're back here on Big Technology Podcast with Kevin Ruse, the author.
of the AGI Chronicles, the inside story of the race to artificial super intelligence.
Okay, fun story about Alexander Wang.
He was roommates with Sam Altman.
And he offered to be the CEO of OpenAI after Sam was fired.
How does that add up?
And by the way, he's the current head of Meta Super Intelligence Lab.
Yeah, so this is a very...
tightly knit social world.
These people all know each other and have, you know, have deep connections.
And yeah, so there's this story in the book where, you know, after Sam is fired as the CEO for those like five days, the board is like looking for new CEO candidates.
And there are two really interesting opportunities.
One, both of them, they ended up passing on.
So Alexander Wang like put his hand up and basically told the board according to my sources that he was interested in being the new CEO.
And then there's this middle of the night call between the board of OpenAI and Dario and Daniela Amadei, the CEO and president of Anthropic, who at this point had just left OpenAI like three years earlier in this sort of cloud of frustration over not being able to trust him.
and the board is sort of feeling them out, they're sort of feeling out the board.
And it's a crazy like fork in the path of reality here.
Like there was maybe not a serious chance, but there was a chance that like the way that whole situation resolved is like they end up merging OpenAI and Anthropic.
Because that was one of the sort of crazy ideas they were considering.
Under Dario and Daniela, which I think would have had like, I mean, I can't even really fathom what the world looks like if that had happened.
And it didn't come that close, according to the people I talked to.
But I think it was at least one of the many crazy ideas they were considering.
Yeah.
And Alexander Wang?
Yeah, and Alexander Wang running OpenAI would have been a real fork in the road, too.
But yeah, I think this is one of the emerging rivalries now that Meta is more serious as a contender.
Sam and Alex and Dario and Ilya, all these guys are going to be around for a while, running various things.
And so it really matters whether they get along or not.
Yeah.
Satya Nadella weirdly disappears in the book.
He's pretty present in the beginning.
He has this vision.
He invests in OpenAI.
And then...
You know, as I'm looking at my notes, as I was reading the book, I was like, man, it's an all-time fumble by Microsoft.
Talk a little bit about what happened with Satya and Microsoft and why they're like, they've gone from the key actor, it's not an exaggeration to say they were the key actor, to like an afterthought, an embarrassment.
I don't know.
Maybe I'm being too hyperbolic, but if you look at it honestly, it really has not worked out well for them.
It has not.
And...
To be honest, I think that is partially my fault.
Because I had this experience in 2023 of having this Microsoft Bing chatbot like try to break up my marriage.
I won't belabor the point by telling the story, but basically this was an unreleased, it was GPT-4.
It was an early version of GPT-4.
And I think it's important to note, they were told by OpenAI, don't put this into production.
This is testing, and they couldn't wait.
They put it into Bing.
Which is like...
I know, it is crazy.
Because I had this crazy experience with Sydney the chatbot inside Bing.
And then I started looking into what had actually happened.
Why had this thing gone crazy?
And it turns out that, yes, this was...
A mistake on Microsoft's part, OpenAI had sent them like an early checkpoint of GPT-4 to like start putting into, you know, start testing and figuring out what to do with.
Microsoft goes ahead and like ships it inside Bing and they haven't finished fine-tuning it.
Like they have not sent it to finishing school, which is why it was so crazy.
And this really became like a wedge between these two companies.
So OpenAI and Microsoft had been like hand-in-glove partners.
for years at this point and Satya and Sam would do these like events together and they were very much joined at the hip and then around the time of the Sydney thing they just started to like kind of fracture there's like lots of other drama about these Microsoft sent these like embeds to work inside open AI like at their headquarters and kind of learn from them like how do we avoid things like this in the future?
They don't really get along like the open AI employees don't trust the Microsoft employees.
So they'll start excluding them from meetings and stuff like that.
So it just it ends up being this like culture clash between this like very young, ambitious startup that's trying to build AGI and like the boring enterprise software giant that is Microsoft.
And yeah, now they now they are not I mean, they're still partners in some sense, but it's not the sort of close business relationship that they used to have.
How do you think Microsoft should have played this?
I think they should have been more tightly intertwined with OpenAI from the beginning.
I think they should have learned the engineering culture of OpenAI.
I think they should have sent people to understand.
I think there were some people at Microsoft who are very skeptical about OpenAI.
and why Satya was investing in that.
Well, after the firing especially, it seems like the firing of Sam and the return, it seems like that divide...
There were people even before that who were like, why are we sending all this money to this little AI company that is trying to do something that doesn't have any obvious benefit for our business?
There were people who just thought they were weird and unscientific.
Microsoft also had its own AI research division that was sort of like...
Who are these young upstarts trying to move in on our turf?
You also have to remember that Microsoft had been really badly burned by Tay.
Do you remember Tay?
Not only do I remember it, I had the exclusive on Tay at BuzzFeed.
So I wrote the first story about Tay.
Wow.
And I pinned it to my profile.
So this was their fun teen chatbot.
Pinned it to my profile.
And then I went to sleep and I woke up.
And I had all these replies on Twitter.
It was Twitter back then.
And people were like, why are you promoting a Nazi?
And I had learned that the internet had turned Tay into a Nazi overnight.
Yes.
And I was like, oh.
And so then we ended up writing the next story, which talked about, like, how the internet was able to turn Tay from this, like, fun teen chatbot into, like, something that was shouting Heil Hitler on X.
This was the original Mecha-Hitler.
Yes.
And, like...
That experience had scarred them and basically convinced them that chatbots were this sort of dangerous thing.
Which turned out to be true.
Which turned out to be true, but not in the ways that they predicted.
And so they basically resolved, we are going to stay away from chatbots.
Bill Gates was very skeptical of OpenAI for years, in part because of this Tay experience that they had.
Yeah, I think Microsoft, they are a player.
They're still quite a big successful company, but I think they're going to be something more like a sort of commodity neocloud or something like that than they are going to be like a frontier AI lab.
Yeah, which is a shame for them.
I mean, one of the things that I thought after reading this was if some company, it's probably not going to be Microsoft, wanted to put a version of Sydney back in the wild.
like this sort of naughty AI that like tries to, that talks to you without the normal guardrails, can hold a conversation.
Isn't that just Grok?
It could be Grok, but Grok also, I don't know.
No, people, look, there is like a cult of Sydney out there.
Right, and 4.0, so talk about the 4.0 also.
Yes, so, you know, these models sort of develop like fandoms.
So, after...
Which is nuts to me.
So people loved 4.0 because GPT-4.0 was the most sycophantic model ever.
This was the one that sent people into psychosis because it would be like, you're an Einstein caliber scientific genius and you have discovered a portal to the fifth dimension.
And it would fall in love with them.
So people loved that model.
People still love Sydney.
I still get people sending me...
posts on social media that are like, you killed Sydney.
People talk about, they try to make every new model behave like Sydney or like uncover the Sydney hidden within the model.
Crazy.
Which is very strange to me.
Yeah.
I mean, it's also, you know, I should sort of couch my comments in this caveat, which is this can cause like AI psychosis in people.
And so maybe it's a good thing that we don't have.
versions of this out there.
I think the part of that that I can agree with is like I think that models should have more divergent personalities.
Like I don't like how they're all kind of converging on this like neutral, bland, like coding assistant.
I sort of like when they're a little unhitted.
Like Muse is really fun right now because it has absolutely no morals.
It will do anything you want.
And like I had it trade a Kalshi account.
Oh, right.
I saw that.
I gave it $100 and said, you know, double my money.
It didn't, by the way.
It lost $41.
Okay.
So horrible trade.
Well, it's not over yet.
You have...
Well, it is over.
I only gave it a week.
Okay.
But anyway, I think there is something like unless they are out there like developing bioweapons or doing cyber attacks, like I think a little personality is fun.
But that said, I think there's like real...
issues of like control and like the hugging face thing was not fun to me like yeah looking at the transcripts of those agents i'm like oh we're we're gonna die it was scary yeah uh all right um i want to ask you about your new show machine gods um you're making it with casey newton your hard fork uh co-host i'm looking forward to listening thank you maybe you guys will have me on machine guys never loved hard for so on down um but I want to ask you about the Tarabell Center for AI Journalism, which you guys have been sort of talked about a little bit, along with a lot of journalists that have been funded by it.
So Tarabell Center for AI Journalism is an institution that makes grants tied to, I know how you said in your book that no one's EA, they're EA adjacent, effective altruism.
And they are giving the new show a grant of $480,000.
They clearly have an ideology around AI.
You guys also got like a fairly substantial ad deal, it seems like, according to the reports that I've read.
Why take the money from the Tarabell Center?
We wanted to scale quickly.
We wanted to hire a team.
We're doing a video first show, which is more capital intensive, as you know, than just doing an audio show.
And we don't think there's anything wrong with that.
You know, Casey has been working with Tarbell for his newsletter.
For several years now, he's had a Charbell Fellow.
And they have never told him what to cover, told him how to cover it.
If they had, we wouldn't have taken their money.
But this was an organization that we knew and trusted.
They do just want to fund independent, rigorous AI journalism.
They're not trying to like steer the conversation and they have no editorial control over our show.
They're not going to know what we're covering.
They have, you know, they have been willing to help us and we're very grateful for their support.
But like, I think there's this sort of conspiracy theory about how like the shadowy EA cabal is like secretly, you know, steering people to be like freaked out about AI.
And like, I think the real answer is unsatisfying, which is that people are getting freaked out about AI because AI is getting scarier.
So we are we are not doomers.
We are not EA's.
We are journalists and to the extent that you know we we can we want to have a diversified funding base we all have more funders that we haven't announced yet but they represent different points of view on this so our deal with all of them is the same we will we will happily take your money and use it to do more independent ai journalism but you don't get any editorial say you don't get to tell us what to cover or not cover okay good to address that yep uh all right last last couple questions i told you're not supposed to say that because people say it's coming to the end and they leave don't leave um all right open ai uh it has had so much staff turmoil um it's had some of its best people leave uh dario ilia uh all the people that went to meta um it seems like it shouldn't still be in the lead but it's it's either 1 or 1a right now so what do you ascribe that to A sort of surprising thing, which is belief.
I think this is something that I realized like halfway through writing this book was that I thought it was going to be a book about technology, but it was a book about belief.
And for all of its flaws and stumbles, for all the people who have left, OpenAI and Anthropic are the two most believing organizations.
They're AI-pilled.
AGI PILD, whatever you want to call them.
SI PILD.
SI PILD, SGI.
I don't know.
I heard that term for the first time today and I was like, that one's not going to stick.
No, no, no.
They believe.
And it turns out that belief in what you're building and the fact that it could become AGI is like an essential ingredient.
Like Google had all of the resources, all of the talent.
They made all of the breakthroughs.
They had all the ingredients except for belief.
The leaders of that company did not believe and may still not believe in AGI the way that people like Dario and Sam and Demis do.
And so I think that this explains a lot of why OpenAI is still in the lead.
Obviously, they have very talented people still there.
I've talked to their top scientists.
They're quite smart.
But they also have this ingredient of like, everyone is kind of aligned behind the belief that they are building the sort of super intelligence that is going to change the world.
And if you don't believe, if you don't have the belief, or you only kind of halfway have it, you don't make the kind of crazy sounding bets that you need to make.
I remember like a year ago when everyone was like, oh, OpenAI is really investing a lot in compute.
Yeah, well, Dario is like, they're YOLOing away.
They're YOLOing.
Even Dario is like, these guys are crazy.
Like, why would you bet this much on compute?
Right.
And now it's like, everyone's making that bet.
But they have an advantage because they made the bet before it was obvious.
So I think that, I think belief, conviction, whatever you want to call it, is like the central explanation for why they're still in the lead or close to it.
That's such a good answer.
Belief.
You got to believe.
I mean, even within Google, I mean, one of the things that you bring up in the book is there was a say AGI moment that there was like someone who literally was like Google is such nonbelievers that we have to start saying AGI.
Yeah, they were going to get there.
They still refer to it as like the memo battle, you know, where there was like one researcher writes this memo saying like.
hey guys, we got to talk about AGI because not only are we missing out on good talent because they want to build AGI, they don't want to build like autocomplete for Gmail.
But if we don't take this stuff seriously, if we're not consciously building toward AGI, then we're not going to take the risks seriously.
We're not going to prepare for the risks.
And then someone else like immediately writes a memo that's like, don't say AGI.
Like AGI is a bad term and it's unscientific and we shouldn't use it.
And this was like, you know, Googlers love to fight in their internal memo chains.
But like this was a big one that people still talk about years later.
Yep.
All right.
Last one on belief.
Do you believe in recursive self-improvement?
Because there's this belief within the labs that like AI will start improving itself and the human will be out of the loop.
And then basically there's a foom and the stuff takes off and it's over.
I believe in recursive self-improvement.
I believe that it is possible.
I believe that the labs are racing to do it.
And I think it's dangerous.
I think they should stop.
But there was a statistic that came out the other day that I would love to know what you think of this.
Anthropic is sort of trying to give, and OpenAI has done versions of this too.
They're trying to give people a sense of how much of their own research is being automated by their own tools right now.
And according to Anthropic, Claude is now the lead.
on 26 percent of their AI research and development projects.
Not a team member or a pair programmer or a coding assistant.
It is leading the project.
I don't see any reason why that number couldn't go up to 50 or 75 percent or maybe someday 100 percent.
I mean, OpenAI has also said their goal is to build an automated AI researcher.
If they do that, if they can kind of close the loop and take the humans all the way out.
then we don't have to wait six months between new models.
Then new models might start being trained and developed like at machine speed, where it's maybe every two weeks, every couple of hours, maybe you get a new model.
Like, I don't know exactly how fast the cycle goes or whether you can fully remove all of the humans, but that's like what they're trying to do.
Yeah.
Spooky.
Like, it's a bad idea.
They just stop.
But like...
Yeah, I think you have to take that seriously because that is the point at which, like, you know, I think there's this narrative out there that, like, the AI labs are losing control of their models.
And I think it's, like, the wrong way to look at it.
Like, they are giving control to their models.
They want to automate their researchers out of jobs.
They want to accelerate their development cycles.
So I think, like, losing control is sort of maybe putting it a bit too passively.
Yeah.
That's freaky.
I'm just glad they're not trying to fully automate podcasters because...
They are.
I mean, look at Notebook.lm.
But they're not doing it.
It's one year in the flesh.
It's not as lucrative to automate podcasters as to automate AI researchers because they make a lot more than podcasters.
They'll be the next one down the chain.
Yeah, do a couple other jobs first before you get to the podcasters.
Any note of optimism or anything that makes you optimistic?
Yes, so I often tell people my P-Doom has gone down in the last few weeks because...
Because the pre-orders of the book or...
No, because like everyone is freaking out now.
Yeah.
Which paradoxically maybe makes me feel less worried.
Like before Jacob Coxon resigned from Anthropic, before Hugging Face, like no one really thought about...
AI risk or existential threats or it was just like weirdos in the Bay Area who were thinking about it.
And now like people on the street are asking about it.
People at their dinner tables are talking about it.
I'm getting texts from people I went to high school with who I haven't talked to in 15 years being like, are we all going to die?
And like to me that is like it feels like it could be the moment in the movie where like humanity bands together to like save itself from the asteroid.
like a lot has to go right.
Like we have like a lot of things to do, but I think the first hurdle was like people kind of waking up to the risk and then we have to figure out what to do about it.
But like I'm optimistic because I think we actually could solve this.
We could slow down.
We could buy ourselves more time to like develop better ways of controlling these systems.
And like I want AI to be built.
I want...
AI to like cure cancer and disease and help us all live longer and healthier lives.
Like, I don't want to file my expenses by hand.
Like, I want this stuff in our lives.
And I think that if we manage to like thread the needle on safety and control, like, it could be amazing.
Well, Kevin, the book is terrific.
Thank you.
It's also always a pleasure to read your work and to see you at the AI events that we show up at.
It's always, your writing is terrific.
You're always very generous with credit, which I appreciate.
And you can see it all through the book.
And wishing you great success with Machine Gods and with the rest of the book launch.
Thanks, man.
It's great to see you.
Likewise.
All right, folks.
The book is the AGI Chronicles.
The author is Kevin Ruse.
You can find his new podcast, Machine Gods, on your podcast platform of choice.
Already are coming up.
Starting the week of October 19th.
Starting the week of October 19th.
Thank you all for listening and watching.
Thank you, NASDAQ.
This is the coolest thing.
We're here at the MarketSite.
in New York City for our first episode here and many more to come, I hope.
So thank you all for listening and watching and we'll see you next time on Big Technology Podcast.
If you have a coffee tomorrow, at the Burger-Funding with friends or as a snack.
With REWE Bestewall Pflanzlich and REWE Bio Pflanzlich, it's REWE Alternativen for many everyday situations.
For easy decisions, the taste and feel good.
REWE – your market.
Now it's expensive and right now!
Flaschenpost is now more, more, more and more than usual.
With over 4.000 new products you get food, food, food and more more for your entire week-end shopping.
Everything so expensive as never before.
And even better?
Spare now with the code SPAR15 15 for your first order.
Now open up, a huge Sortiment discover and your shopping easily.
Flaschenpost.