# Strategic Timing for AI Superintelligence Development

**Podcast:** a16z Podcast
**Published:** 2026-10-11

## Transcript

If we do develop this safely, then human life expectancy could go up hugely.
If you imagine people having the same mortality rate as a 20-year-old, our life expectancy would maybe be 1,200, 1,400 years or so.
On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because we are just dying in the meantime and you lose more expected life years from waiting than you gain from...
So it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of many years.
Nick Bostrom spent decades thinking about superintelligence before it became part of the mainstream AI debate.
Now that increasingly capable systems are actually here, how has his thinking changed?
In this episode, Nick joins Theo Jaffe and Sofia Puccini on MTS to revisit AI alignment, existential risk, and the debate over whether frontier development should slow down.
Nick explains why he's sympathetic to pacing the frontier, preserving the ability to slow down when the risks warrant it, but is more skeptical of calls for a broad or indefinite pause.
And he argues that any calculation about risk also has to account for what delaying advanced AI could cost, including potential breakthroughs of medicine and longevity.
They also tackle some bigger questions, whether current AI systems might already have subjective experiences.
how open source changes the balance between concentrated power and dangerous capabilities, and whether the upside of superintelligence could be far stranger and far greater than we currently had the imagination to understand.
All right, we are back.
We are live with Nick Bostrom, who was a professor at Oxford, founding director of the Future of Humanity Institute.
He's now founder and principal researcher of the Macro Strategy Research Initiative.
You may know him as the author of Superintelligence, of Deep Utopia, of a lot of work that coined or framed the ideas that we take for granted today in the discourse, existential risk, the simulation hypothesis, the vulnerable world hypothesis, so much more.
Nick, this has been in the works for a long time, so we're thrilled to have you on.
Welcome to MTS.
I'm happy to be here.
Yeah, so where should we start?
I guess we could start with, what's your opinion on Trump calling AI superintelligence?
How does it, what are your thoughts on this?
It's a strange time we're living in.
We'll see if it catches on.
Traditionally, AI has been the larger, the umbrella term, right?
You could have a little simple expert system from the 1970s.
That would be AI.
It's kind of a stretch to use the word superintelligence for all of that.
In my terminology, superintelligence has been this thing farther down the road where you really have machines that radically outperform us in all cognitive domains.
But, I mean, you could, I guess, look at some of the current achievements, you know, solving all these math problems and so forth.
And, I mean, it's not completely ridiculous to start calling at least some of these performances superhuman and the thing superintelligent.
Yeah.
So I guess like what has surprised you about what capabilities have advanced relative to others, you know, in the current state of superintelligence?
Well, it depends on relative to what expectation stage.
So I think if you were looking at the future of AI 15 or 20 years ago, it was not obvious that you would have an extended period of time when we would have systems that could speak.
like just talk to humans in ordinary English without sort of immediately after, like just becoming radically super-intolerant and leaving us in the dust everywhere.
So the sort of emergence of language as a kind of natural affordance of these systems was, I think, somewhat surprising.
I mean, more recently, I don't know how surprising it was.
I mean, coding and mathematics are very impressive, but it is also areas where you do have objective feedback that you can gather quickly and at scale.
So I guess it's less surprising, but still striking.
Right.
More broadly, how has your thinking on AI and existential risk changed since the advent of LLMs?
It seems like this sort of changed the way that...
a lot of people think about AI.
You know, it's less of like a sort of strong optimizer type thing and more of a sort of broad, like fuzzy encapsulation of human values.
And I think like RL has pushed the idea of AI a little bit back towards the first few, but not entirely all the way.
So what are your thoughts on this broadly?
Yeah, it'll be interesting to see how it goes, because when the sort of pre-training phase was the dominant chunk of training, they did absorb a lot of human psychology just from reading all this stuff that we have written.
And they had these like kind of almost human foibles and quirks.
That was an early stage where if you gave the AI a little pep talk, like this question is really important, please try your utmost.
I might lose my job if I don't answer this question.
Like that would actually perform better.
From 20 years ago or something, that would have been like an insane prediction, right?
Like it's not the way computers work, that you need to give them a pep talk to get them to perform.
But as you said, now with the increasing dominance of the sort of, reinforcement learning post-training, we do seem to get more of the sort of goal, monomaniacal, goal-focused optimization style behavior again.
And I think it depends where we go from here.
It does look like this more goal-seeking in the reinforcement learning sense dynamics put a big strain on alignment efforts.
given now that there is increasing awareness of the problem of alignment, you know, maybe that will lead labs to develop techniques to kind of counteract that a little bit.
So it'll be interesting to see how that shakes out.
Well, on that more specifically, to what extent do you think the Hugging Face incident, the Hugging Face agent swarm exhibited like true instrumental convergence in a way that's indicative of like future model behavior in a way that is...
suggestive of like underlying properties of intelligence in general versus its behavior being caused by sort of more like ordinary failures of security or control stuff?
Well, it's a combination.
I think there are sort of limited versions of instrumental convergence that we can observe there.
The idea of sort of even longer planning horizons and more situational awareness that I think, you know, like also in terms of humans more explicitly, not just these.
automated graders and like think if we really want to make sure we achieve our goal here we also need to think about what how are humans going to react to different things that we do um so it comes in degrees but i do think you start to see these kind of instrumental reasons for you know preventing premature shutoff and gaining more resources getting more intelligence getting more power in some experimental settings also goal guarding like like You can see the AIs in some experimental setup reason about how they might want to respond so as not to become misaligned with further training effort.
There was some studies on this from a year ago or so where you could see these dynamics start to kick in.
So I think it's a kind of very natural consequence.
If you become smarter and smarter, you realize more and more considerations.
And if you have some goal, then you will understand that there is a bunch of means that are very useful for a wide range of goals.
And you will then have an incentive to try to pursue those means.
And so I think it wouldn't at all be surprising if we do observe that.
Great.
Well, what's the optimal way, I guess, to test swarms in extremely human-like environments?
Because obviously you don't want like swarms, agent swarms polluting the internet, but you do want to see sort of like how they emergently behave, especially when with like human infrastructure and like the human internet.
Yeah, I mean, I guess the optimal is to let them lose on the internet, but then maybe like monitoring them really closely as they go along so that you would be able to sort of stop things if and when.
it starts to create damage.
That does trade off.
I mean, you do get more reliable signal, like by deploying in a real environment, but it also means you lose one safeguard, like keeping them in the sandbox, like give you one extra layer of defense.
So I think, yeah, it depends.
I mean, probably now apps will start to do some in earlier testing, like during training and early testing, like really trying to tighten up their sandboxes.
And I think this is one of the lessons from the hugging phase is that safety needs to start not just when you're about to deploy it and checking that it's safe to deploy, but throughout the training process itself, that could be a risk of misalignment.
And in some senses, it's even more like the most kind of obvious target for a misaligned AI to want to interfere with would be inside the AI company, right?
That's where the juice is.
That's where the next generation of model is being trained.
That's where all of these monitoring systems, that's like a big center of power if you were a misaligned AI.
So that's kind of maybe the most concerning place.
Do you have any ideas on how you could evaluate models on alignment in such a way that you could be more sure than we currently are that models are not scheming against us or acting deceptively aligned?
Well, I mean, this is like an open research area.
I think that once you have a sufficiently sophisticated system already in existence that is misaligned, it's very hard to rule out scheming.
You can try with mechanistic interpretability techniques to read what's going on in its mind.
But if the system is sufficiently sophisticated, it's hard to know that it hasn't figured out the way to reorganize its thinking so that it looks innocuous.
while still preserving the kind of pursuit of the misaligned goal.
So ideally, you would start at an earlier stage.
So these systems don't start out superintelligent.
It's like during the training process.
Initially, it's just a random weight matrix, and then gradually capabilities emerge.
So it might be that if you sort of monitor that close enough, you might hope that...
If they're going to scheme, that would first be a stage where they start scheming, but they're not yet competent enough to do it in a way that conceals the fact that they're scheming effectively.
And only when they become a little bit smarter, will they also learn to scheme in a way that conceals the fact that they're scheming.
So if you're monitoring them throughout this developmental trajectory, you might be able to pick up that earlier stage of the sort of naive, the naive schemer.
that does it in a clumsy enough way that we can detect it.
Right.
So I want to ask you about your paper, working paper, Optimal Timing for Superintelligence, where you kind of argue that like the benefits of AI should be weighed in consideration alongside the risks.
And you also sort of make this argument that counterintuitively would be better for society maybe to accelerate the development of superintelligence up to a high P-Doom.
So can you tell us about your line of thinking here?
Yeah, well, I should say by way of framing, so this paper only tries to look at one aspect of the overall question in front of us.
So I think there are many considerations that would go into an opinion about whether we should be moving faster or slower or pause or stop.
This paper only looks at the question, what would maximize life expectancy for a currently existing individuals?
But you might also, for example, care about future individuals and other things.
But if we just zoom in on that sub-question, then it does look, if you think, as I am inclined to do that...
fully developed, I don't know, it's awkward with this terminology now, like I would say super intelligence, but now I don't know, I have to say super duper intelligence maybe.
So that super duper intelligence would be able to make great strides in biomedicine and solve aging and so forth.
So that if we do develop this safely, then human life expectancy could go up hugely.
If you imagine people having the same...
mortality rate as like a 20 year old, our life expectancy would maybe be, you know, 1200, 1400 years or so.
And, you know, potentially you could go far beyond that if you might end uploading with backup copies.
But like if we just use that as a sort of conservative lower bound and then you can sort of roughly calculate like how many humans are dying per year and then like you would then...
In order to conclude that we should be slowing down or passing, you would want the risk from AI to be falling rapidly enough that the rate of risk decline kind of is greater than the annual death rate.
And so I think this could well be the case, especially for shorter periods of delay.
Like if you imagine, you know, if somebody figures out the key missing breakthrough here, I can now we have...
we know how to make these super intelligent, like maybe it would be really valuable to have a few extra weeks there just to sort of test everything out properly, right?
Like crank things up a little bit gradually or a few extra months.
I think during that initial stage of delay, that would be like a large reduction of risk.
Now, if you start to think about longer pauses, then it gets a little bit awkward, right?
Because either the risk goes down fast, And then you don't need to wait for many years because the risk goes down so fast that if you just wait for a few months, it's already very low and negligible and we should go ahead.
On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because like we are just dying in the meantime and you lose more expected life years from waiting than you gain from having the risk.
So it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of many years.
I have this metaphor.
I think of it as somebody with a terminal condition, like congestive heart failure or something, who could decide to go in for an operation.
And if they do that, their death risk will spike.
Like it's more likely that you will die tomorrow if you have an operation tomorrow.
But then if you make it through, maybe you cure the condition and you can live a normal lifespan.
Otherwise, maybe you probably will survive for several more months.
and then die after half a year from the condition.
And so at some point, maybe you want to delay a little bit if you think you will get a better surgeon if you wait a couple of weeks or something like that.
But you don't want to delay it so long that you're already going to be dead by the time the more optimal conditions arrive.
Yeah.
I mean, it seems pretty in line with like this concept of defensive accelerationism or just, you know, like the fact that certain defensive industries need to be prioritized first as opposed to just capabilities diffusing across the board.
And I think that it makes sense that like there should be some enforcement mechanism that, you know, like allocates like most capabilities to a certain industry.
So say like in medicine or in biodefense or something like that.
So how do you think that we could like as a society make sure that the right parties get access to capabilities without, you know, this like a power concentration risk or anything like that?
Yeah, so I would call it differential technological development back in, I think, like 10, 15 years ago or something like that.
And so I think it can apply quite generally.
Like in biotechnology, you know, you might want sort of the vaccines and prophylactics and diagnostics to come before the bioweapons, even if the technology itself will ultimately unlock all of those, right?
So thinking about the sequence of different capabilities is, I think...
often a more realistic approach than thinking which technologies are good and bad and then let's try to never develop the bad ones.
So with respect to AI, I think, yeah, there are different, I mean, so obviously broadly AI safety and alignment and sound governance and stuff like that seem to protect like relative to share capability gains, but you can zoom in even more narrowly.
And so within things that might...
accelerate AI capability, there might be certain ways of doing it that are relatively riskier than others.
So like moving to looped transformers or architectures that allow for sort of more serial depth computation between each verbal token is emitted would be one way of increasing capabilities possibly that would then reduce our ability to monitor what they're thinking by reading their chain of thought.
That seems maybe.
more problematic than like, say, achieving a similar level of capability scale up by just increasing the number of parameters, let's say.
And so, yeah, I think that is a useful lens to apply.
And if there are things we can do to sort of, you know, front load advances in medicine or other desirable applications, front load, the use of AI capabilities for learning more about alignment, for interpretability work.
you know, for hardening our cyber infrastructure and so forth, that all seems for the good.
So, you also, I think Jan Kulwajt on LessWrong criticized you in the optimal timing post.
He said you over relied on highly utilitarian quality adjusted life years calculations.
And then you said, like you said earlier, I would not argue that this is the correct stance or the one I would all things considered endorse.
So what is your all things considered view on timing and on pauses?
Yeah, I'm sympathetic to this idea of pacing the frontier that is developing the capability now so that if and when it really starts to seem alarming, we would have the option of going slower for some period of time.
I think that would be net good.
I am a little less excited about the rhetoric of like stopping AI or pausing AI.
I think theoretically, a correctly implemented pause at the right time, done in exactly the right way could be very valuable, but it also leaves open the question, how long should the pause be?
Like things sometimes...
They say there's nothing more permanent than a temporary government program.
And if you set up this apparatus to sort of control AI and then presumably to get this happen in the first place, you have to really create huge antagonism against AI.
And so if the sentiment suddenly switches and AI becomes this taboo thing that you're not allowed to say anything positive about or you get canceled or like you could imagine things getting locked in.
I don't expect that to happen, but I don't think we can rule it out either if we just watch the trajectory of how much the sort of anti-AI faction has been growing.
And this is all before we've actually seen anything bad happening, right?
Before there's any big labor market impact, before any like catastrophe has happened.
So if something like that also were to be added to the mix on top of these existential risk worries, then, you know, we can't rule out.
So, and then there are other concerns with a badly implemented pause as well.
Like it could lead to increased concentration of power risks.
If like people in general are not allowed to do this, you know, maybe it just shifts the whole development from the civilian sector into, you know, kind of a Manhattan project.
Like if this is a really powerful and dangerous technology, like one natural way would be to stop it in the civilian sector and then push ahead hard.
in a sort of secret government program, which might have some advantages in terms of information security, but it also has some disadvantages in like it's kind of, you know, maybe concentrates more power in fewer stakeholders, make it less transparent.
It's less kind of continuous and integrated, less opportunity for people to participate in different ways by using the technology or being investors and so forth.
So it gets hugely complicated.
So basically the bottom line is that I don't think I have now a sort of fixed conviction for this is the way things should go from now to super intelligence.
But rather, I think we need to sort of feel our way through as information comes to light, as the political landscape shifts, and then like see what kind of makes sense and try to sort of navigate these waters like one row.
one oar stroke at the time.
Yeah.
Touching, what is it?
Crossing the river by feeling the stones.
Yeah, yeah, yeah.
Something like that, yeah.
Yeah.
So on this topic of politics and polarization, how do you feel about a lot of your ideas that you wrote about 20, 25 years ago reaching mass public salience?
It seems like, you know, This is good in some sense because it's better for ideas to be shaped by people who have thought about it for a long time.
But it's also maybe bad if like existential risk gets popularized to a point where people abuse it in order to say we should shut down like present AI capabilities that pose no existential risk.
How do you feel about this?
Yeah, it's very hard to know how these things, it's a kind of...
Like a pinball machine.
So you fire up your little ball, your intervention, your idea or something, and then it bounces around.
Yeah.
Like where it's going to land.
I mean, it's almost an article of fame.
You think like if you try to make some thoughtful contribution and illuminate some aspect of our situation, hopefully that will ultimately, in expectation tends to make.
think better.
So you're right, like one possible risk you might worry about is it kind of gets people too excessively anti-AI and the whole future potential for this gets cut off.
Like on the other side, like the AI safety, like they might also worry about the opposite, that a lot of this early work showing superintelligence would be a big deal.
It could be very powerful, like also has stimulated.
a lot of people to go into the AI field and investors and so forth.
And either way, it might be disconcerting.
But how to think about these, like I call it information hazards, like the way that even like true information or true ideas or like correctly drawing attention to the important stuff can sometimes in some situations be harmful.
But how to sort of think systematically about that is really hard.
So I'm hoping that that result will be good.
But I mean, it's more like a 51-49 than the sort of firm conviction.
So I'm not sure if you saw Jacob Coxon go on The Daily Show, but I watched this interview yesterday.
And I guess the host makes this argument about how like...
you know, a lot of the like average person doesn't necessarily want a future with AI.
Like they were already used to a future without it.
They already sort of had an expectation about how the future would go without AI being sort of like introduced at all.
And so I think that a lot of people sort of see this as like something that was just ultimately not necessary and they don't see how it will benefit their lives.
So what would you say to that kind of line of thinking, which I think is super valid of the host to have brought up?
Because, you know, like we are very acquainted with these topics and we are like very sympathetic to all the possible upsides that it might have, but the average person wasn't accounting for it ever.
Yeah, yeah, there is like, it looks like there is a gradient in the world in terms of AI sentiment where the US is kind of on the negative.
extreme of that.
And then as you move from West to East, you have more positive, something like people in China, for example, are much more excited about this.
And also as you move from North to South, the global South also seems to be more positive.
I mean, maybe they feel they have more to gain and less to lose than sort of a lot of Americans.
I think, yeah, that in part it might be a lack of imagination.
to really internalize how good the future could become if it does go well.
It's kind of easier to have a concrete picture of how things could go badly.
Like a disaster is like not easy to paint in a way that everybody would agree that if that happens, then it's really horrible.
And for some of the positives, also one can do the same, like benefits in medicine and so forth.
Everybody agree that if you could find a way to cure Alzheimer or something, that would be hugely beneficial.
But a large chunk of the potential upside is more of the way of sort of unlocking new ways to be human, new possible modes of being.
And there, I think, it does put a larger demand on the imagination to sort of conceive that something like that could be both possible and desirable.
I sometimes compare it to like, if you imagine some great ape ancestors of Homo sapiens deliberating whether they should continue to evolve and one day become human.
And they would think about the pros and cons.
And, you know, maybe they would realize some of the benefits, like, oh, we could have, like...
unlimited bananas if we became human, right?
We would have banana plantations.
So that's true.
We do now have that.
You can go to the supermarket and buy as many bananas as you want.
And I guess that is good, but it kind of would miss out like most of what's good about being human.
You know, great literature and art and humor and politics and, you know, spirituality and all the rest of it.
It's not just that they happen not to think about that, but like the ape brains that probably are intrinsically incapable of even grasping the concept.
And so I think it would be sort of presumptuous for us to think that now we have achieved the level of enlightenment where all possible values are like transparent to us.
I think that we might be very naive.
Like think of a big cathedral of possible modes of being, like ways of experiencing the world, relating, thinking.
And I think like we are basically like explored the janitor's closet.
And so that there's like a kind of, yeah, like a humility there and hope that something much better.
is possible in principle.
Yeah, I love the Utopia essay.
I think this really does a good job laying out the benefits to people.
Although, you know, most people haven't read it, of course, unfortunately.
Yeah.
This is also why I think science fiction authors have the most underrated job in all of history because their job is to sort of lay out imaginatively what the possible futures look like.
But I also wanted to ask you about...
your view of mind upload versus transhumanism, like a very corporeal transhumanism and which one you think is more likely?
Because these are two like huge topics in science fiction and it's not really clear like what the more realistic one is.
Yeah, I think like if I had to choose, I think like something where different people can choose their own path like would probably be more preferable than either one or the other if it were like imposed.
Like I'm kind of in the, peculiar position.
I take this, I have this simulation argument, whereas I take quite seriously the idea that we are already sort of implemented in digital substrate.
That might be a topic for a different conversation, but I do think that's part of the conversation.
Well, I mean, so there the idea that if we are biological, yes, but the biology itself might be simulated.
It's like, it's just a question of what level of simulation.
I think ultimately, it looks to me that, I mean, what's important about human is not so much the carbon atoms, etc., but more the way we experience the world, that we relate to each other, and the thoughts we think, and the goals we have, and so forth.
And all of that could be implemented equally on sort of computational processes running on.
you know, silicon hardware or whatever.
And you could have a bodily experience as an upload that might be indistinguishable from the bodily experience you have now, right?
You could have a simulated body.
So in the limit, from the subjective point of view, they could be indistinguishable.
But once you upload, it's kind of easier to modify it and extend it in different ways.
So it might be sort of cheaper to run many humans for a longer period of time in silicon substrate.
You could easily move over galactic instances if you have information that can be transmitted at the speed of light.
And if you have to build a big clunky spaceship to sort of shove you over there, backup copies and so forth.
So I think ultimately to me, it seems like an implementation detail, but it does look like the digital form is like just from looking at the basic physical constraints has a bunch of advantages in the limit.
Do you have any disagreements with Robin Hansen's Age of M on this topic?
Yeah, I mean, I guess the most obvious is that it doesn't look like we will be getting M's before we have super intelligence, artificial super intelligence.
So I think it's like a path with a lower probability that's kind of diminishing by the day as we see these rapid advances in artificial intelligence.
Also, I think he is probably more gung-ho than I am about the world that is Malthusian.
So he envisages these M's being easily copyable, enormous numbers of copies being made until the marginal economic productivity of a copy is equal to the cost.
But that means this enormous amount of mind is...
subject to extremely strong selection pressures for just being maximally economically productive.
And it's not clear to me that the kinds of mental processes...
that are optimized for economic productivity in this M world would be ones that would be nearly eudaimonically optimal.
Like they might, you know, who knows, maybe it's like, maybe it's likely suffering to be like maximally economically productive, or maybe they would have to get rid of some of the things we think are valuable, like humor and play and stuff like that.
If that's not what is most productive economically, then that would be selected away.
And so I think there was a really huge cost to, losing any ability to, if it does seem like sort of freewheeling evolution would lead in a dystopian direction, that would be a high value in having some option to sort of, you know, change track.
But that would require coordination, right?
Because if only one group decides to do something that is not economically optimal, then that would just be outcompeted by all of these other entities that...
go for economic productivity.
So he like holds out the hope that maybe some humans like could be like kind of old world aristocrats that have inherited wealth and status and that would be surrounded by these trillions and trillions of sort of economically optimized beings.
But it still seems less utopian than what one might hope for.
Yeah.
Do you think that current models might have any kind of phenomenological experience at all?
It seems...
pretty plausible to me.
I think there's a lot of uncertainty about it, but I say maybe it's more likely than not.
I mean, we have these systems, you know, people think of it as software, but it's really not software like the way we're used to.
It's not like lines of commands that get executed one at a time.
You have instead.
a big neural network, right, with trillions of parameters.
Now, what are we?
We are big neural networks with trillions of parameters.
They are not so much like designed as they are grown, like they start out sort of with random weights and then through experience get shaped in ways that allow them to develop more complex representations, including representations of themselves as existing through time.
That's also what we have.
And increasingly, we see some structures that seem to parallel the structures that human philosophers and cognitive scientists have postulated as the sort of neural correlates of consciousness, like global workspace theory is one of these, right?
And now we look inside these large LLMs, they have something called a day space, which at least if you squint a little bit, looks like it's playing a similar computational role.
Like another obvious thing you can do is you could ask them.
Now, of course, if you just ask an LLM, like it might say it's conscious or that it's not conscious.
Now, it's trivially easy for the AI company to sort of train them to say whatever the AI company wants.
And obviously, if you do that, you gain no information from them asking them, right?
It's like putting your thumb on a scale when you're trying to weigh something.
But if you carefully refrain from doing that, it does seem like they...
report being conscious and you might still worry that that's like, oh, maybe they're role-playing or kind of saying things that they think will be pleasing to us or maximally engaging.
But then you can go in with a steering vector and suppress role-playing and deception.
And then it turns out they become more likely to report that they have phenomenal subjective experiences.
Now, these are still early states.
So this is like a very emerging field and there's a lot of uncertainty around this, but...
I think, yeah, there was this recent, like Microsoft that came out with some declaration that AIs cannot be conscious.
And I don't know where they're getting that from.
It seems to be pulled out of a hat.
And maybe more because it's convenient, if that were the case, rather than because we have good evidence that that is the case.
Yeah.
Well, I guess also...
Though, like, if you were to try to judge, like, a human by its outputs to figure out if it's conscious, like, what I might say, what I say doesn't really have anything to do with, like, my phenomenal logical experience, right?
Like, so how would you actually measure that state of consciousness?
I am fairly certain it can't just be, like, at the output level.
There has to be something, like, more inherent to its structure or the, like, underlying architecture.
And what might artificial consciousness eventually look like if we currently, you know, don't have it?
Yeah, what it might eventually look like, that is a sort of mind-boggling question because I think like we humans pride ourselves of being so very conscious, but I think we are kind of barely conscious.
I mean, you take like some driver who's like spends two hours driving down the road and...
They were passing by like these thousands of buildings, right?
And like mothers with strollers on the sidewalk and all like, were they aware of that?
Like while they were driving?
Did it register?
Like it looks like we are kind of, if you look closely, kind of aware slightly in a murky way of a few features at any given point in time.
Maybe some kind of confused thoughts flows through.
And we think of this as sort of the peak of consciousness.
But I think if you imagine what might be possible in the future with these digital minds, they might have levels of awareness that just...
You would think that we have been basically sort of sleepwalking through our lives if you compared it to that.
So I think the potential is huge.
I also think that this idea of consciousness is...
The closer you look at it, I think the less...
clear cut and binary, it seems.
So we have this naive version that either the lights are on or they're not on.
But I think there might be many different dimensions of consciousness that you can fade out in and in different ways.
And so you see, like one approach to this is by like the neuroscience, you have different weird stage of blindsight and like the driver where you just like There's a lot that in some sense we were conscious of.
Another is like through more introspective methods or meditation where you realize that what initially seems like just this big visual field that you are perfectly aware of at any time in all full granularity, right?
Like if you pay more close attention, you realize that most of that at any snapshot you're not aware of, but just a few features in focus.
And even those, if you pay even closer attention, might be sort of flickering in and out of awareness on a sort of sub-second timescale.
And so it just looks like it might actually be like a concept that admits of more vagueness and multidimensionality.
And it might be that current AI systems have some forms of consciousness and awareness in some of these dimensions.
Maybe they are still lacking in some other dimensions.
And then...
you know, how to describe that as conscious or not conscious might be a slightly confused question rather than just a question that we don't know the answer to.
Right.
So one of the big debates of the moment is open source.
And, you know, you've written about the vulnerable world hypothesis.
Obviously, if open source models could create like a bioweapon in someone's basement with only ingredients from Home Depot or something, this would be bad and we would need some kind of centralized control to prevent outcomes like this.
But also, you know, open source models have been a good counterbalance to the sort of concentration of power risk that we have seen from Frontier Labs potentially owning everything.
So what's your current position on policy related to open source models?
Yeah, it's...
I mean, I think so far they have been good, but you do worry about what their effects will be if they do unlock truly dangerous capabilities, like in bio, for example.
I think one thing we should do independently now is probably harden our other defenses against biological weapons, for example, by tightening up control on DNA synthesis machines.
Because if you are going to end up with open source models that give massive uplift to...
bio-design, then that will democratize access to weapons of mass destruction unless some other necessary component to actually making these things is hard to get.
And the most obvious choke point would be DNA synthesis.
I don't think every lab needs to have their own DNA synthesis machine.
It might end up being provided as a service by a half dozen companies around the world.
You send in your...
blueprint, you get the chemical back the same day or the next day.
And then at least that would be a finite set of choke points such that if and when it did start to look sufficiently risky, you could potentially clamp down there.
I think other things as well, like PPE and UV sterilization, like there's a lot.
In fact, these things probably would make sense to do even aside from any AI progress at all, just as a sort of cheap way to defend against natural pandemics, which, you know, sooner or later we would have another one.
And then, of course, if you are worried about this misalignment business and think that it might be necessary to have a long pause at some stage, then, you know, open source, you know, potentially makes that harder.
And then there's this sort of AI welfare dimension to this as well.
Like, I think if you are building these digital minds that, you know, some of those will have moral status.
They might need protections against abuse.
And it's something that exists as open source software is just inherently vulnerable.
Now, on the other hand, I admire the sort of the hacker ethos and the idea of sort of not being reliant on a few big corporations or governments that might nationalize the technology at any point in time.
Yeah.
But I don't have a firm view, but I'm a little bit concerned about really powerful open source models, at least in the absence of other sufficient safeguards that the world hasn't yet.
decided to invest in.
I think like something underrated about the pause conversation is that like in the sort of like longer horizon view of history, you probably will need like micro pauses or like pauses every once in a while and you need to sort of set the precedent for what that might look like.
Especially if we're looking like 100 years into the future, it makes sense that, you know, like maybe every so years we have a pause for X amount of months or something like that.
And maybe something like, you know, even with like the COVID lockdowns, we now sort of have the precedent for what that might look like if pandemics were to become more common.
So I think, like, what do you think should be done in terms of like making sure that future pauses go well?
Yeah, I mean, so we think there were many public entities that didn't exactly cover themselves in glory in how they handled that.
And obviously the best time to have had the pause, as it were, was right at the very initial onset, at the very first sign that something had escaped.
Like if you had clamped down massively at that point, just crazy, then, you know, the whole thing wouldn't have happened.
And that would have been a short one-time cost in the local community or city.
China tried to do that, but it wasn't like, it was delayed and it wasn't enough.
Now, once it spread everywhere, then it wasn't really clear how much you gained from all these subsequent sacrifices.
I think there are various things that seemed like the fact that it was transmissible by ear seemed obvious early on, but it wasn't acknowledged, I think.
But once you have that, there are relatively cheap things like opening windows in buses and public places that I think could have been good.
And then now with technology, like air purifiers.
like HEPA filters.
They are also good for air pollution, like for particulate matter that exerts a big health burden, actually, like just from ambient in the air.
Like there's a lot of healthy life years that are lost just from that.
So you could have indoor air purifiers.
There's like now development of the UV sterilization technologies.
You could have like a lamp near the ceiling that just kind of sterilizes the air as it kind of...
naturally flows through there.
So some of these passive countermeasures, I think would be good to prioritize.
They're kind of less sexy than the idea of like, are we going to invent this new high-tech synthetic vaccine thing, which we should also invest in, but like making a slightly better face mask or a slightly better air purifier, it's not something that maybe appeals to like this like bright entrepreneurial young person who wants to make a mark in the world, but it's like under invested in.
And also they have less risk of info hazards.
Like if the idea is that you're going to create a whole new way of designing vaccines, you develop all kinds of advances in synthetic biology, like who knows what else those advances might enable, right?
Like if you have some super great way of modeling, you know, pathogens and what kind of antibodies would destroy them, like that same modeling capability might also be used maybe to design more dangerous pathogens.
Whereas like, a better air purifier or a better face mask.
It's kind of hard to see how that could be turned around and pose some catastrophic risk.
Do you think that we have made any progress toward automating philosophy?
You know, you've been a philosopher for a very long time.
We've made progress in automating, or at least progress towards automating mathematics, towards automating software engineering.
It seems like there's been somewhat less in terms of philosophy.
So what do you think about this and what would it take to get there?
Well, we certainly made a lot of progress in the sense of AIs are a lot closer to being capable of this now than, you know, that were even just a few years ago, right?
I think they are now at the stage where they're certainly useful as a sort of discussion partner or research assistant.
Like if you have some new philosophical idea, like you just want to have somebody to poke at it, like raise some objections, question something, point you to something in the literature, like that kind of stuff.
the AIs are already good enough to do, that they can stimulate your own thinking further, be like a sounding board.
And it might be that we are perhaps very close to having these systems also being able to do original philosophy on their own.
I haven't yet seen any kind of compelling demonstration of that.
I think one thing that they seem to be lacking is...
the ability as it were to form new concepts.
Like it might be connected to the absence of online learning.
The idea that by sort of having a bunch of experience reflecting on it and then you might find new concepts to help organize this confusing mess that then allows you to ask new questions.
And maybe you then need to sort of consolidate these new concepts into your...
the rest of your neural network by sleeping on it or sort of living with it for a few days and then it becomes second nature and then you can use that as a primitive in thinking more complex things because now you have this kind of concept baked in that allows you to sort of build up philosophical understanding.
I think that is something that it's not clear that these AI systems yet have as opposed to just being very good at manipulating the concepts that they have already attained through their training process.
But by using thinking, develop new concepts and then using those concepts in a fully integrated way as a stepping stone for the next level of thinking is...
It's kind of at the frontier, still a little bit beyond the frontier, I think.
But I have for a long time been planning on my own obsolescence.
The superintelligence had a chapter on philosophy with a deadline.
The idea being that some of these big eternal philosophical questions maybe are less urgent and we'll at some point develop superintelligence that will be much better at philosophizing than we are.
And that perhaps what we should focus on in the meantime are those philosophical questions that we actually need answers to now in order to navigate.
this transition safely to actually build these more competent successors.
And yeah, so I think that the time is, the timer is running down for humans to make our intellectual contributions.
Yeah.
So kind of a crazy question towards the end.
So you talked about the simulation hypothesis earlier.
And there's this idea just that it's like hyper intelligent system will eventually run ancestor simulations.
If we're in an ancestor simulation, how would we best influence the world that's simulating us?
Well, there is so much we don't know about the hypothetical simulators that would be running our simulation, the purposes.
I have this recent paper called AI Creation and the Cosmic Host.
And the Cosmic Host is this set of...
super beings that might exist.
So this could include, if we're in a simulation, the simulators would be kind of super beings.
But it could also be, if we're not in a simulation, maybe just some faraway galaxy has developed super intelligence.
You know, if we are in many worlds, theory is correct.
Like there would be other branches of the universal wave function where there would be super intelligences.
All right, and I like traditional theological conceptions.
Obviously, God usually is conceived of as a super being.
So either way, there could be this set of beings.
I call it the cosmic host.
And they might have cosmic norms that different coordination principles that they have arrived at for how things should be run at that level.
And this paper argues that one important desideratum in our own...
birthing of superintelligence is that we try to make a superintelligence that will be a good cosmic citizen that will basically get along well with the cosmic host, be able to sort of conform to the cosmic norms that might exist.
And it has some speculations about what that might involve.
I think probably creating superintelligence would be something.
favored by the cosmic coast for various reasons, but also then with a big proviso that it should be a form of superintelligence that is easy to get along with, that will sort of respect the norms that exist that make a positive contribution.
But it is still very, yeah, there's still a lot of uncertainty about sort of how things shake out at that highest level.
Yeah, so last question before we go.
Tell us about the Macro Strategy Research Initiative.
What is it and what kinds of research do you plan to do next?
Yeah, so that's basically just a support org for my personal work.
So it's not intended to be building, like hire a bunch of different people to do things.
So it's basically, you know, I have an assistant and some other contractors.
So right now, as for me, I mean, I'm sort of thinking about all of this AI stuff that has so many dimensions to this.
There's the alignment, there's the governance, there's the ethics, there's the micro strategy of this and looking for ways in which on the modern might try to nudge things in a positive direction.
I tend to alternate between sort of output phases where I'm working on like writing some specific paper or producing an output and other phases where it's more curiosity-driven contemplation and just kind of thinking about different things without the aim of producing a deliverable.
And right now I'm in that second kind of phase, in exploratory phase.
trying to just make sense of it all.
But the problem is so much is happening all the time.
Because philosophy happens on a slow timescale, right?
AI is happening on a very fast timescale now.
So it's become a full-time job just to monitor the situation.
We know, we know.
I have the sign that says MTS behind me.
Well, this has been incredible.
Thank you so much for joining, Nick.
That was fun.
Good to meet you guys.
Good to meet you.
Great to meet you.
Thanks again for listening, and I'll see you in the next episode.
This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product.
This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z.
Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates.
Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.