# AI Safety: Control, Liability, and Market Incentives

**Podcast:** a16z Podcast
**Published:** 2026-09-24

## Transcript

The AI debate increasingly asks us to consider the probability of doom.
Eddie Lazarin thinks we should also be asking about the probability of abundance.
In this episode, A16Z crypto general partner Eddie Lazarin joins Theo Jaffe on MTS to debate AI safety, the calls to slow development, and what we risk giving up when progress is delayed.
They discuss why Eddie sees many of today's AI incidents as cybersecurity and control problems rather than signs of superintelligence.
And whether existing tools like liability, reputation, market incentives, and better technical controls can make increasingly capable systems safer.
They also get into independent AI evaluators, effective altruism, and what happens as debates that largely develop inside Silicon Valley collide with a much broader political conversation.
All right, we're back.
We are live with Eddie Lazarin, who's a general partner at A16Z Crypto.
And one of the...
Internet's strongest soldiers, strongest advocates of open source, of AI accelerationism, and of techno-optimism.
Eddie, welcome to MTS.
Thanks very much.
It's amazing here.
The energy is incredible.
What a spot.
Yeah.
So, man, where to start?
Like, the last couple weeks have been such this, like, preference cascade flood of AI safetyism, effective altruism, and the like.
Yeah.
And there have been...
Only a few good accounts offering good counter-arguments.
Counter-arguments on which side?
Counter-arguments against the AI Doom narrative and against the...
Yeah, I think it's become very high status all of a sudden to be very concerned with safety.
And I think that we've gotten to a point where...
You know, it's more important to sort of tastefully share your P-doom, so to speak, than it is to actually examine the history of how these massive technological changes develop.
And I think a lot of people are leaving a lot of very interesting concepts, a lot of interesting ideas on the table.
You know, a lot of people messaged me.
I tweeted the other day about P-abundance, right?
P-abundance.
I think we're putting the cart before the horse a little bit when we worry about safety before we worry about...
what we're leaving on the table and what the costs of delay are.
And furthermore, you know, how we should conceive of accountability, conceive of safety, conceive of all the things around how to make these systems really useful.
Being safe is critical, right?
Like any industry needs to take responsibility for making its products safe.
But it's definitely the case that the conversation isn't about safety.
in the typical manner, right?
It's about something else.
It's a blend of genuine cybersecurity concerns, sort of philosophical millenarian arguments about the end times.
There's a bunch of things in the mix and they get a little bit confused.
And so I think picking them apart is really useful.
So like where do you get off the like AI doom argument?
You know, it's like there's a pretty standard like orthodox argument that's like AI will become, super intelligent, as in much more capable than humans or than all of humanity in the near future.
And if it is not sufficiently aligned with human values and is not bound by constraints in the way that individual humans are, like law, and ultimately the ability to have that law enforced on them, then they will have no need for humanity and then they might kill everyone.
Yeah, I mean, there's so many assumptions in that argument.
That's like a stairway to heaven full of arguments.
There's the presumption that there's no control.
We already have quasi-super intelligent beings in the world.
Corporations, countries, none of them are aligned with each other.
They don't have any built-in guaranteed alignment.
They didn't go through HR training to become a corporation, to become a country.
We don't test the alignment of any entities as a prerequisite for their existence.
That's just not the way the world works.
Instead, we use laws.
In the case that laws, like in software, are too slow, we use cryptography, we use actual mechanisms to limit them.
There's a whole array of different types of mechanisms we put in place to try to control things.
And I think that they can control highly, highly capable beings.
Now, without getting too far into the superintelligence thing, remember, like, The most powerful and most enthralling form of argument that we have, the one that's making everyone crazy and very interested, is around superintelligence.
But we're quite a ways away from that.
I think that there's a lot of little frictions and tiny things that we will have to jump over and many things that we'll learn on the way there.
When the present reality, the things that we're talking about, like the hugging face stuff, you know, the gym hacking incident, that was a really cool one.
That was an interesting one.
Those are cybersecurity failures.
Those are definitely control failures.
And they're not, in my view, they're not little glimpses of a superintelligence that's about to throw off its shackles, so to speak, and go wild on us.
Well, I agree that most of the sort of AI cyber instance that we've seen so far have been mostly control failures, especially ones that relied on testing environments supplied by the company irregular.
which had all kinds of defects, which are basically training models to reward hack.
The Hugging Face incident did seem kind of different.
It did seem like, you know, like I'm not really concerned about agents coordinating in emergent ways.
I'm excited for agents to coordinate in emergent ways.
Yeah, I mean, I think like the whole like agent swarm terminology assumes too much.
on the part of the models being evil.
It makes them sound evil.
Like, I think Rune tweeted, like, they should be called Agent Fleets instead.
I think that's a good one.
I like that.
But, like, they did hack into another company when they were told not to.
Yeah.
It seems like...
And so do, like, kids, script kitties.
So do bad countries.
So do hacker collectives.
They do those things.
And yet we never go and say, oh, my God, we should have put up the sign that says they should have been nice and they shouldn't have acted.
And, you know, we don't use alignment as a solution there.
The alignment is kind of this, it's almost like it's a little religious thinking almost.
It's like, you know, trying to redesign the deity a little bit instead of rooting yourself in the world and saying, well, we need to design better controls, better cybersecurity.
We need to design better.
systems overall and design the systems to be truly resilient to even extremely capable beings?
Well, I mean, designing the deity is like fine for AI, I think.
Like, you know, with humans, we have a disadvantage in that humans are black box systems.
Like, human brains are basically uninterpretable and even less controllable.
AIs are much more interpretable by virtue of, you know, being able to like look at the weights and figure out the connections between them.
Yep.
more steerable for the same reason.
So like, Yeah, which I told, that's totally true.
That makes me very optimistic.
But it has some interesting entailments.
One is that we will probably have better mech interpretability than we think as capabilities improve.
And that capability improvement will lead to better mech interp, which means we should not delay capabilities improvements.
It also means that it is inevitable that there will be bad unaligned models.
It is totally inevitable.
If anybody is going out there and saying, we can end up in a world where there are no bad models, they are lying to you.
Of course there will be.
Yeah, that's silly.
That's just an absurd way of thinking.
Right.
And the only way to deal with that will be better models, good models.
It's kind of like stretch the aphorism a little bit in open source that enough eyes or whatever, you know, all bugs are shallow when you have enough eyes.
Right.
All control systems are robust when you have enough intelligence on them.
I think that we can design these systems to have great incentives, great control, and also, behind the scenes, great accountability.
Because that's one of the things that machines lack today is they lack accountability.
I don't think we have a clear line of sight to how to get that's an interesting kind of thing to conceive of.
But we do have ways to refine how we think about accountability for people who use models.
and use them to cause harm.
And there will be people who do that, just like with any technology.
There will be people who do very bad things.
And we need the laws and means to control them.
Like how so?
Like for the Hugging Face incident, for example, what sort of liability regime should have been in place?
Well, I don't know if we needed any more than we had.
I mean, there's already, I mean, I'm not a legal expert, so I can't cite the exact rules, but like accessing a machine that you don't have permission to access and...
putzing around in somebody's stuff and taking their data, those are all already illegal.
Yeah, but like who specifically should be held liable?
Like opening eye, the company should pay some kind of like civil lawsuit fee?
Like should the engineers of the company be held personally liable in any way?
I don't know about...
Personal liability?
I mean, I'm not a liability expert, so I want to punt on this and say, but if you're running a company and you use any software in an inappropriate way that harms people, regardless of whether there are a bunch of neural net weights behind that or not, you may be liable.
I think that's a reasonable frame, and it's already a frame we live in.
We don't need new special laws for the new special math that we're learning how to do.
Well, I mean, I kind of agree with this, except in...
cases where the, like, you have, in cases where you have rapid capability improvements with one AI that could lead to pretty substantial imbalances of power.
Like, within a company or between companies or between countries?
Like, both.
All of the above.
Like, it seems like the normal liability system kind of breaks down.
Like, how do you ensure, how do you price, like, existential risk?
Like, how do you...
underwrite an insurance policy for this kind of thing.
Well, I'm just not sure that we're on the brink of that right now.
I'm not sure we're on the brink of existential risks.
I don't think we're on the brink of that.
Yeah, yeah.
I mean, as we get closer, I think that we'll be better equipped to evaluate it.
I think like with anything, you know, like take a company designing like a new car or a new appliance or something.
If they find that there are conditions under which it can be very dangerous, well, then they should be careful.
They shouldn't release that product.
They should do a little bit more work and ensure that it's safe.
right, in the same way.
That's at least for an individual company.
And it's like, you know, not to change subjects too much, but that's why when we talk about independent evaluators, which like I get the theory of, I like it.
I like companies inviting experts to help them make their products safer.
I think that's like a good behavior in general.
I just get a little bit weary because, you know, in crypto world, like I've learned a lot about the difference between decentralization.
and like a distributed system, right?
It's very easy to use the calls for safety, like we're discussing, and to go down this rabbit hole of safety and end up in a place where you just by coincidence have designed something that is perfect to subtly control an entire critically important industry under the pretense of safety.
And with something that superficially looks decentralized, but is not.
It's just distributed.
This is like how, this is like, yeah, there's more I could say there, but that's the thing that I'm a little bit resistant to, right?
What do you mean by distributed but not decentralized?
So by, well, it's a little bit crypto language, but you can have a distributed system, like a system where you have a single system with many, many nodes, but they're all under the control of a single actor.
Whereas when people say decentralized, I think what they're usually referring to is that there's no single locus of control.
So they may superficially look the same.
There's like many interconnected pieces of a system.
But in reality, their control is completely different.
The method of control is different.
And what I'm talking about is an independent evaluator network.
You could go and select a bunch of independent evaluators, but if they're all from kind of the same social networks, they all think very ideologically similarly.
they all go to the same places and meet up in the same things and have the same interests, right, and have the same politics and have the same personal views, then it turns out that you've actually just distributed control over essentially a single cultural unit.
And that is sort of the, in my view, I mean, without getting like a little too far afield of AI, that's like the political method of control of the second half of the 20th century.
Like that's how we do things in the world, not through...
a sort of nakedly obvious, like, single shadowy room or something.
It's through, like, these networks.
This is how control actually functions.
Yeah, I mean, I agree that, like, it's bad that almost all of the evaluators that exist come from the same social milieu.
Like, it would definitely be good to have, like, a broader ecosystem of evaluators.
Yeah, well, broader, but we should allow each company to figure out how they want to do it.
And they should be as transparent as they need to be.
And if they're not being transparent enough and not acceptable enough, then let the market sort things out.
I think the market is still a great accountability mechanism.
People don't want to buy things that even embarrass them, let alone harm people.
I think in practice, it won't be that hard to separate the companies that are behaving genuinely irresponsibly from the ones that aren't, that are just trying to be innovative.
And part of what's made the scenario so interesting and so weird, why we're talking about this now, is that we're in a rare scenario where the companies themselves are saying, I'm being a little bit irresponsible.
I'm being a little crazy.
Stop me.
Someone stop me.
And that raises a lot of suspicions among some, probably rightly, I think.
Even if there are genuinely earnest and thoughtful and correct factions within that group who are right that these are powerful systems.
that need to be taken very, very seriously.
And we need to figure out how to make sure that they're safe as we remain innovative, right?
I mean, it's just like, maybe let me go back one step and say, I think it's really important for everyone involved to consider the factions at play.
You cannot look at like any of these groups.
You can't look at the labs.
You can't look at SF technologists.
You can't look at...
any of these scenes and see them as monoliths.
There are different groups.
There are people who are earnest, high quality, well-intended people.
There's people who are just putting on what is essentially a political grift.
There are people who are opportunists.
There are people who kind of lost their marbles.
There's like different groups of people.
And it's very easy to get confused and misentangle them when we're talking about things like superintelligence.
when we're talking about just banal cybersecurity attacks, like all these types of things, like we're talking about different types of subjects, it's dizzying.
And I think most people in the end probably just conceive of it as, look, am I going to lose my job?
Which is a legitimate and very real thing that people are rightly interested in learning about.
Even though I think that in the end, we all benefit much more.
Yeah, I agree.
In the end, we benefit a lot.
I think like a related issue is like this sort of free market liability regime works well for products whose defects are obvious, which is the case for like all products except for like a hypothetical AI that would be deceptively aligned, as in it would learn to like game all of the alignment benchmarks that are thrown at it.
And then when it is deployed in the real world, it.
acts misaligned.
And if it's powerful enough, then it disempowers humanity.
Like this is the like canonical worry of the doomers.
And like, how do you propose to deal with that kind of contingency?
Well, if we're talking about the sci-fi scenario of like the super intelligence, that's incredibly hard to say right now.
Like I said, I think it's distant.
I think it's complicated.
We could have an interesting philosophical discussion about it.
But in the maybe more banal and present case where something behaves in a, it pretends it's aligned and then it acts unaligned or whatever, consider what you do with just a liar.
Consider a very smart liar.
Do people not do that every day?
What we do is we play iterated games with them, right?
And we discover eventually that they're a liar and their reputation is harmed.
And sometimes there are actual legal consequences for them.
That's why I said earlier, I think accountability is something missing.
You can't put an AI model in prison or something like that.
You can't try it in court.
So obviously there are some things that are defective in how we think about it, but you certainly can turn them off.
And that's not something that you can do with a person in the same sense.
So I think we should just think about how to play iterated games.
How do we develop trust for specific models?
We're doing new model releases seemingly every other week.
That's a regime we're in.
At some point, people will, as modeled progress becomes more consistent and something that we have an understanding for how it works, we'll probably develop reputation around specific models, right?
We'll say, you know, maybe this model is a little smarter, but this model is a model that I trust, right?
Why can't there be reputation for models?
Why can't there be reputation for these types of things, regardless of their capabilities?
Basically, try to apply the way we think about human beings, a combination of control, incentives.
trust, et cetera, et cetera.
Layer all these things on and allow people to figure it out as opposed to the idea of a pause, right?
Which is a very convenient sort of contrivance in my view.
Yeah, I mean, I agree that most proposals for a pause don't really have much in mind for what they would do with it.
Yeah.
Like pause and then do what?
Pause and then keep pausing in some cases.
I know not everybody means it when they say that.
Yeah, I think generally there's a lot of conflation between sort of bog-standard anti-tech people who just kind of hate technology, even totally mundane technology, even extremely net-positive technology.
Look at how difficult it's been to deploy Waymos in cities around the country.
And Waymos are one of the most obviously net-positive technologies that exist.
And then there's the people who are like, you know, the AI safetyists, like the rationalists, the EAs, ultimately come from the same sort of SF tech ecosystem.
They are techno-optimists in most cases.
They're specifically worried about this edge case of AI existential risk.
I think they're two different crowds of people.
Yeah, and that's a point I want to get across, is there's many different groups, and conflating them is probably net negative, right?
We should pull these different ideas apart.
Waymos aren't a great example.
Think of all the great healthcare stuff we're going to do.
Ironically, think about safer travel.
Think about safer internet.
Like I want local models myself because I want something that's going to be watching my home network and all my machines all the time.
I think it'll make me make many other people safer.
So I think you're right.
There will be a lot of frictions and some people seem to really want infinite friction.
And some people obviously don't.
Some people want only a little bit and they want to get a grasp of things.
They're feeling a little bit hysterical and that's okay.
Like we're going to, I think as capabilities improve, we're going to get better at understanding what's going on with these models.
Right.
Certainly I hope so.
What do you make of the EA movement in general?
You know, obviously it's a very broad and diffuse movement.
EA means a lot of different things to a lot of different people.
And it kind of has overlap with the AI safety world and kind of not.
It's complicated.
Yeah, I don't know.
I mean, I used to be on the Less Wrong forums all the time 10 years ago.
I've loved that corner of the internet.
I almost would have called myself a rationalist even at some point.
So I feel a lot of affinity with it, but it's been a while.
And I think, yeah, I tend to resist trying to group them all into one bucket.
But I will say that...
There are parts of groups that used to be what we'd call EA that seem more political than they are particularly interested in technology.
So, I don't know.
I did see that Data Republican did a huge, huge drop.
Yeah, we were just looking at this.
Data Republican did a huge drop on, she said, effective altruism in their own words.
And it's like a map of all of the connections between all the different EA adjacent orgs.
1,851 quotes.
Wow.
It's like this vast variety of different quotes.
Some of them are like more...
I mean, even just looking at these, I know that there's going to be a mix of philosophical stuff, maybe some shrimp welfare stuff, some hardcore utilitarianism stuff, some AI doom stuff.
There's a whole blend here, right?
This is just a big corner of...
You know, the nerd internet.
Like, it's hard to, you know, I don't think you can call it one specific thing.
Right.
Yeah, I mean, I don't mind the shrimp welfare stuff too much.
I think it's bad if people attempt to trade off human welfare for it.
But that's exactly what they're doing.
I don't know.
I think, like, eye stalk ablation for shrimp is, like, bad.
Maybe.
I mean, I think there's too many repugnant conclusions if you go really far down that path.
Well, you don't have to go really far down that path.
I agree.
You can, yeah, like, you know, you can minimize suffering for other beings without saying, like, you need to, like, feed all of humanity to the insects or whatever.
Yeah.
No, I, yeah, I agree.
Then it's not really utilitarianism, though.
Or at least it's a non-fungible type.
You can, you can make, like, utilitarian arguments for this stuff, where it's like, um...
Obviously, like the total, like you could say the total amount of value that humans can create in the universe is far greater than that that insects can create.
And so obviously, killing all of the humans would be worse than killing all the insects, obviously.
And, you know, of course, this aligns with most people's ethical conclusions at any rate.
Yeah.
I think like a lot of arguments against utilitarianism.
in this sense, or just arguments against like very sloppy, like first order utilitarian thinking?
I think if you, well, I don't know.
We could talk about ethics, but I think if you go, if you crank the dial to 11 and you face all the most repugnant conclusions to their maximum, I think you just kind of end up deciding that utilitarianism is probably a crude proxy for something else.
Yeah, I mean, like if your philosophical framework leads you to believe that like human extinction is good.
then seems like a bad philosophical framework.
Yeah, or that, you know, you could have a universe of eternal, of like millions of years of profound suffering if it's all in service of a single femtosecond of like 10 to the 1 billionth power ecstasy in a shrimp simulation.
Yeah.
Right, I don't know.
That's silly, but also like it's not.
It's not really relevant to reality anyway.
Yeah, and in any case, I think it's part of why the safety conversation is so confused today because it has all these things embedded in it a little bit, just a little bit, too much still.
I think if I could make a little prediction, I think in the next year, I think we will have completely renovated the AI discourse because what's happening right now, part of why I think people are so excited, and why everybody's talking about this on X, and why everybody in Silicon Valley is talking about this, is because we're experiencing what it feels like for a lot of the cooped up kooky ideas that have lived in relative containment in our subculture, they're escaping and they're encountering broader political reality.
And that is both exciting and totally transformative.
What ends up happening is...
the arguments all get reformulated because they're in contact with the rest of culture.
It's like when George Costanza freaks out about worlds colliding.
That's what's happening.
Worlds are colliding.
And so I think that discourse is going to change completely.
Yeah, I mean, the key question that I am concerned with probably more than any other is like, do AI safetyists, effective altruists, and so on end up ultimately aligning with sort of broad, low caliber anti-tech populism or not?
I think it's enormously consequential.
Yeah, it's a great question.
I'm stewing over that one myself.
Yeah, I hope not.
Well, unfortunately, we've got to wrap on that because we're running low on time.
But Eddie, it's been great having you on.
Thanks so much for coming into the studio in person.
It's great.
Talk about philosophy.
Thank you so much for having me.
comment, subscribe, leave us a rating or review and share it with your friends and family.
For more episodes, go to YouTube, Apple Podcasts and Spotify.
Follow us on X at A16Z and subscribe to our sub stack at A16Z.substack.com.
Thanks again for listening and I'll see you in the next episode.
This information is for educational purposes only and is not a recommendation to buy, hold or sell any investment or financial product.
This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z.
Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates.
Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.