# Reframing AI Safety as Software Engineering

**Podcast:** a16z Podcast
**Published:** 2026-09-21

## Transcript

The AI people are making it impossible for anybody to understand what they've done.
And they're using words like, well, the AI failed to be aligned.
Okay, what does that mean?
What it means is there was a bug in the software.
When Word ate your file and deleted all your content, we didn't think that demons had taken over Word and made it do things against the will of man.
When AI does something we didn't expect, is it misaligned?
Or does the software just have a bug?
A16Z board partner and former Microsoft Windows president Stephen Sinofsky joins Theo Jaffe and Sofia Buccini on MTS to argue that the language around AI safety is making the technology harder to understand.
Drawing on decades of building software, Stephen explains why AI labs need better telemetry, and what today's industry can learn from earlier software failures, computer viruses, and Y2K.
They also discuss why terms like alignment, goal-seeking, and rogue agents can distort the policy debate, and why treating AI more like software could lead to clearer conversations about both safety and regulation.
We are live with Stephen Sinofsky, who is a board partner at Andreessen Horowitz.
He's a seed investor, author of the Substack and the book, Hardcore Software.
Previously, he was the president of Microsoft's Windows division, engineering and marketing across Windows, Windows Live and Internet Explorer.
And he posts on Twitter at SteveSI.
Stephen, welcome back.
Howdy.
So we were just talking about, we're in DC right now at the FAI office.
How about FAI?
Yeah, we're at FAI.
And we've been talking about all of the new regulation and new laws that have been proposed for the AI industry over the last few weeks, including the Stop Rogue AI Act, which was proposed by Josh Gottheimer of New Jersey.
And you're offering some thoughts on this.
What's your take?
Well, sure.
Well, right now we're in the maximal phase of like, let's just throw tons of legislation out there.
So real quick, I'm just going to take you back.
I can't believe we're going to take you this far back.
This is going to seem like infinity years ago in another dimension of time and space.
But in 1983, I was a freshman in college.
And one day in the fall, like, and there were two people in our 100-person dorm that had computers, me and another guy.
And that was it.
And there was nothing to do with them.
Nobody knew why we had them, blah, blah, blah.
And one day, literally, G-men in suits crashed our dorm.
confiscated the other guy's computer, and arrested him.
Wow.
And it turns out, fast forward a little bit, he was part of a hacker group, one of the very, very, very first hacker groups called the Inner Circle or the Inner Ring.
And they were hacking this thing called GTE TeleMail.
GTE was a big communications company.
And it was one of the very first online mail services that you used a modem and you dialed up and you could just mail other people without telemail.
Who were some of the clients of it?
The United States government and NASA in particular.
And immediately after this series of arrests, they arrested people all over the country in this coordinated sting operation.
I mean, imagine, I'm a freshman in college and they just carted this guy off with his IBM PC XT that weighed 50 pounds at a giant monitor.
And it was like a freshman in the same, we were the computer science majors together.
And it turns out that there was no crime committed.
It wasn't illegal to do what they were doing.
There was no law that they were violating.
Oh, wow.
Suddenly there were all of these hearings.
And in the hearings in Congress, they're literally playing clips of the new movie War Games and explaining like what this is happening.
Like these kids in high school.
are going to take over the government and launch all the nuclear weapons.
And that's what really happened.
And it took several years before the first federal crime statute for hacking.
And Ronald Reagan signed that.
And that's what became illegal.
And then five years later, four years after that, I'm at my first homecoming.
And I go upstairs to the computer science building to go see my old advisor, like the day before homecoming.
And there's a press conference going on.
It was the weirdest thing I'd ever seen, a press conference in the middle of Ithaca, New York.
And it turns out that was the press conference to announce that they've found a computer worm called the Morris computer worm that had originated from Cornell.
And that was the very first prosecution under this criminal act.
And so this stuff for me goes way, way back.
So I'm very comfortable with all the congresspeople posing legislation and having hearings, being hysterical.
But right now we have this huge problem.
And well-intentioned people like Congressman Goheim are getting on TV in the morning and saying, we need a bill to stop rogue AI agents.
And the Stop Rogue AI Act.
And you listen to all of this and you realize we have a huge problem.
And this huge problem is just making my head explode.
which is the AI people are making it impossible for anybody to understand what they've done.
And they're using words like, well, the AI failed to be aligned.
Okay, what does that mean?
What it means is there was a bug in the software.
The software did not do what we thought it would do.
But when you say misaligned, it puts all these characteristics on AI that it was...
thinking of doing something, that it had to make a moral judgment, that it had roles and guidelines to do things.
And it's like, wow, we had bugs in spreadsheets where they didn't do what we thought that they were supposed to do.
And we didn't sit around like in a circle and pray to the gods of alignment that the spreadsheet would do math correctly.
Like, you know, when Word ate your file.
and deleted all your content, we didn't think that demons had taken over Word and made it do things against the will of man.
And this is a huge, huge problem because it's literally nonsense what they're saying.
Like there was a bug.
Like when you, when software doesn't do what it's supposed to do, it's a bug.
Now there's different kinds of software.
There's simulations and there's models and there's emulation.
There are all these ways to decide what to do.
And just because it decides to do it, not based on step one, step two, step three, if A or B do step five, but just based on statistics, that doesn't mean you eliminate the idea that it should do what you want it to do.
It means your statistics are wrong.
It means that you guess a word that should come next that shouldn't have come next.
And that's just a bug in your product.
And we've got to get to a place where people are talking about software like it's software.
You know, in full self-driving, blows through a stop sign because the software interpreted the stop sign as like a Christmas tree.
We don't sit around and say, oh, it was not aligned with the idea of, you know, December holiday seasons and road signs.
We actually say it's a bug and it's a very dangerous bug because that stop sign means something.
And Tesla...
Built in all of these tools and Waymo, they have all of these, all this telemetry, all these diagnostics, all these extra cameras, all of these things that you don't normally put in a spreadsheet, but you put in software that life depends on it and it's making mathematical statistical decisions on things.
What I see is the AI models are still in the research project phase.
They don't.
They don't have all the tools and all the telemetry and all of the things that you would normally put in software doing important things.
And I lived through this.
We went through a whole 20 years of building software where we actually never really knew why it crashed.
It just crashed and we went, oh, software crashes.
Let's make a list of all the places it crashes and fix the top 10.
And then one day we said, well, we literally did say this.
Someone said, why don't we just write a program?
such that when it crashes, it uses this new internet thing to tell us that it crashed.
And suddenly we had telemetry and suddenly we had more bugs to fix than we knew it ever existed.
Like we literally went from like, we're shipping, we have no bugs, we're ready to go to, oh my God, we have enough bugs that if we just stopped work, we would just fix bugs for the rest of time.
Yeah.
So when you talk about an AI pause, Of course they should slow down.
They should stop adding things and go and add the telemetry, the tools, the logging, the step-by-step debugging, all of the stuff that you would add if your software did anything at all important and you cared if it did it right.
And you read all the bug reports from OpenAI that they just put out yesterday.
And it's super clear that they just, they don't know.
And so they, like if somebody had handed me a bug report and this is, here's a real bug.
In 1987, the first version of Excel for Windows had a bug in it that somewhere on the disk, it wrote the word send dogs.
Just S-I-N-D-O-G-S in a particular Windows settings file.
It was like Excel got taken over by demons and it wrote this word to the file.
Nobody knew where that word came from.
Everybody was looking through all the Excel source code to find the word send dogs and to figure out.
And no one could find it.
And so somebody said, you know what we're going to do?
We're just going to run Excel brutally nonstop doing a million different things for days on end.
And so one tester wrote a script that created graph after graph after graph and printed it on an old school dot matrix printer.
And it turns out if you did that for like two days, it would write send dogs to that one spot in the file.
Yeah, and I thought it was all contextual and it was based on how much memory the computer had, how long it ran for, and what printer and device driver was all in use.
We could have very easily just concluded, it did this.
The Excel brain decided to print Sendog somewhere.
But we didn't.
We actually went and debugged it and then we put in all of this telemetry.
The first thing I learned when I got to Microsoft was how to...
figure out buffer overruns and memory allocation and to put that code into every product.
And then there was a version, you know, what happens is there's a version of Excel that runs that has all this extra logging.
And we ran that internally for the beta tests.
And so all of that infrastructure, I have no idea in these models how much is there.
I can just tell you reading the reports that OpenAI put out, it's not enough.
They're just, they're not grown up yet as platforms for doing what they do.
And they've got to figure that out.
And they should just do that.
They should just wake up and say, we have to go do that.
And they shouldn't be like saying, okay, well, Congress needs to meet and order us through legislation to fix our software.
Like literally this is their, it's like you have one job making software that works.
That's it.
That's the job.
Then that turns out to be a really important part of your job.
Again, done ranting.
Ask me a question.
No, I really like the like historical perspective that you bring to it.
I also think like, you know, the tendency to talk about alignment as if it's impossible or as if it's this like mythical impossible task is really damaging.
We have to solve alignment.
Yeah.
Whatever that means.
I mean, I think that that will happen.
Like, I don't think that it's like this unanswerable question or like domain that's impossible.
But I do think that what's different, even if you use the bug analogy, is that the total addressable surface area of the bug.
is obviously more catastrophic in theory, or like this is the least surface area that the bug will like ever have.
And then in the future, you know, it just becomes like wider and wider.
One of the things that, I mean, again, all of these things, I was young once, and I remember sitting in a room going, yeah, our software, it's not like mainframe software.
And all the little IBM people were looking at us saying, you know, your software is really crappy.
Like all of your, like Windows is just like crap compared to these IBM things that run all the banks and all the airlines and all the power plants.
And for a long time, we kept looking at them and saying, no, you don't understand.
We're different.
This is all new and cool.
And we run on 64K, not one megabyte.
And we cost $5,000, not $5 million.
And we had all of these reasons to claim that our bugs were okay.
in terms of bugs.
And it turns out all we had to do was grow up and watch people use Excel to decide how do planes fly, use Word to file briefs in the Supreme Court, use PowerPoint to present the Challenger space shuttle failure.
But like once you use your tools for that, and so one of the things that happened was the very first internet worm happened.
So the internet then is like people using America Online and sending email and stuff like that.
There's very little corporate email, nobody to eat.
But it turns out there was just enough that if somebody used Outlook, Word, and internet email all together, they could grind the entire United States and world economy to a halt.
And that's literally what happened in March of 1998.
I wake up and there's a reporter hyperventilating on the phone to me saying, I love you, I love you, I love you.
And it turns out that was a virus that got created that then spread over the internet as a worm.
And the next day, the headlines were, $12 billion worth of damage to the U.S.
economy in one day.
And so I did that.
And we had to have a meeting and go, we're going to pause the development of Outlook right now until we figure this out.
Nobody had imagined you could create a bug that in 12 hours could do $12 billion worth of damage.
But we did it.
Woo-hoo!
Yay, team.
And so, of course, the more people use a system, the more...
damage you can do if it goes awry.
But that's like table stakes.
That's like what you literally signed up to do.
The fact that it caught you off guard or that, wow, we're so popular, we can't figure it out so quickly because we're still competing.
That's like too bad, tough luck.
One of the most famous tech journalists, every time there was a bug in our software, and I would do what, he would just make fun of me because I would say, well, it's not so straightforward or it's hard or it's complicated.
He would just look at me, and just say, Sanofsky, that's why you're paid the big bucks.
You don't, it's not an option.
It's not a thing that you get to appeal to some higher authority, whether it's the United States Congress or, you know, embedded testers or some other third party evaluators or just like laws of nature.
That's not any option.
Like this is a tool that human beings are building, that other humans are using.
You've got to figure out how to make it work or not sell it.
Like those are your two choices.
And since it's made by people, we're fully capable of figuring out how to make it work.
It wasn't handed down.
It didn't show up like an obelisk of one by four by nine that we're supposed to figure out how does it do what it does and what's its source of power.
It like literally it had no code.
Then it had a little bit of code.
and a few users, and a lot of code, and a lot of users.
We've all been there through this.
It is pure insanity to just sit and argue that higher powers need to be summoned in order to fix this.
And alignment has this very specific problem, which is the only way to be aligned is to have a very large set of rules that say what alignment is.
You can do this, you can't do that.
This is safe, this is undaided.
And it turns out software, it's well understood.
that if you don't put too many constraints on a system, then its system won't work.
And you'll have an infinite number of unintended side effects.
And so you can't really just say we're going to align because you're going to have to come up with a rule.
And then when you come up with this rule, someone's going to say, yeah, but what about this case?
Then you're going to add another rule.
And you know what that starts to look like?
It starts to look like Google and all the work that they constantly have to do to decide how to present search results.
And it turns out that's really, really, really hard.
And they have spent 20 years and thousands of people full-time forever working on that.
And so that's what they've just signed up for the next level version of it because they're also making up the result, not just searching through the node club entities for the results, but they're also synthesizing it.
So they're going to have to invent a whole bunch of stuff, but none of it has to do, there's no law that's going to make it invented.
Like it's just all on them.
I'm a little wound up.
How similar do you think AI alignment is to like Y2K in that it's, you know, it's something of a real problem.
It was addressed proactively by companies operating under their own incentives.
And then it ended up not becoming a global catastrophic risk.
Well, there's a cause and effect there.
There was a lot of worry by professionals.
And then there was a lot of taking of responsibility by professionals.
And then nothing bad happened.
That's, There's a direct causal relationship.
Like the fact that we rented generators and campers and moved computers into secure bunkers, like that led to the fact that nothing happened.
But nobody legislated.
Actually, there were actually weird regulations about Y2K compliance, but most of it was industry drafted.
Like we all, there was a cross consortium of banks and of insurance companies and all sorts of people to generate the rules and what meant.
Something that OpenAI needs to get together with all the model frontier people and the labs.
And they just need to have much better reporting.
The reporting that they did yesterday, sure, it's a great first step.
But that is not anywhere near the kind of reporting that you need as a third party to understand what went on.
You see all of this information, what versions of software, what are the context of it, what are the steps required, what other software is impacted, what other bugs are related.
What is it a derivative?
Like you see all of this information.
So we need a version of that for AI.
And maybe yesterday, giving them the benefit of the doubt was a first step.
It still felt more like marketing covering for the event because it still said a lot of stuff like, well, we are looking into this, but we presume at this point.
Okay, well then don't write anything.
Like go figure out, like the Federal Aviation Administration does not like announce on the day of some awful event.
We presume what happened.
They just keep their mouths shut until they know.
And then when they know, they really, really, really tell you with a tick-tock down to the partial second with all of the telemetry of the instruments.
And that's what CVEs are.
And they need to do way, way more because these are defects in the software.
And it's just so critical to wrap our heads around the fact that that's what's going on.
It's not divine intervention.
It's not, you know, oh my God, we need to spark up more of the left brain of this thing to make it more rational.
It's silly.
Yeah.
So I guess with the Stop Rogue AI Act, for example, do you think that these are people in Congress having like imperfect information about what's going on in the labs and so that they tend to use this like more anthropomorphic language?
Or do you think it's more intentional?
they know that this is what will spark a reaction in the people or will get some sort of populist support.
Well, it's really important right now to not ascribe ill motives to people when they're doing stuff because that's not going to bring everybody together to solve the problem.
I have no idea why anybody is doing anything.
I actually think, look, AI was, you know, in a Dartmouth summer conference, they started talking about human brains and consciousness and all this stuff.
And there's a long history in all fields of academic research to basically be too cutesy about things.
Like, because that's how you get attention in academia.
Your paper has a cool title and all that.
So it's no surprise that something that had its roots in academia carried forth all of this terminology.
But it's just time to stop.
Like, the terminology is driving people apart because people presume a vast amount of stuff when you use...
metaphor or allegory or anthropomorphism.
And it's just not true.
Like, this stuff is not acting with any of the verbs that it says.
It's not, it's goal-seeking in this old-school technical sense that it, oh, well, there's a curve and it's trying to get to the bin or to the max.
And that's goal-seeking in those terms.
But it's not when a normal person like a congressman hears goal-seeking.
They think of a person trying to get an A in college.
And then when they hear cheating, they hear that they did the thing you're not allowed to do.
And when they hear secretly coordinating, they think of spies invading a country.
They don't think of two pieces of software with a semaphore, which is just another form of secretly coordinating.
Yeah, all of the language is absolutely destroying this dialogue.
And look, the very beginning of the word computer virus was kind of a fluke.
But it turns out it happened at exactly the wrong time.
It happened right when AIDS and HIV were a thing.
So virus was like in the air everywhere.
So if you listen to the computer hearings from the 1980s about the first viruses, they're scary in that context because people are literally thinking about death.
But that's not at all what was happening.
It was a metaphor that one person in graduate school picked up on from a friend.
Right.
And in fact, they were smart enough back then, a little trivia, he wrote his thesis on computer viruses and it basically proved there's nothing you can do about them.
They're here forever, which of course is a metaphor to how people thought about HIV at the same time.
But he also couldn't prove it because his university wouldn't let him run the test because he was writing a paper about how dangerous all this software was and they wouldn't let him use the computer to run the test.
Right.
Yeah, I mean, it was...
Like, it seems like viruses are actually like kind of avoidable.
I mean, obviously, like there will always be some viruses out there at all times.
But, you know, we still get to use computers normally almost all the time without worrying about everything we have getting hacked.
Yes.
And why is that?
Because we have defensive cybersecurity.
Because, yes, because the industry said, well, this is bad.
We can't allow this to happen.
And so Apple did a fantastic, they did such a good job.
that they made up TV commercials on a Mac and I'm a PC.
And they told people that Macs are better because they get fewer viruses.
Not by some fluke of nature, but because they did a whole bunch of work on the Mac to prevent viruses from happening.
And it just so happened, it was work that was very, very difficult for Windows to do because of the business model of Windows and all this other stuff.
Yes.
Yeah, I remember the like, I'm a Mac, I'm a PC, like video ad thing.
Yes.
Where the PC guy was like sneezing and coughing.
He's like, oh, I have this terrible virus going around.
And then the Mac guy's like, oh, I think I'll be good.
You know, Macs don't get viruses.
I think what's funny is that...
You know, this hurts.
Those hurt me physically, thinking about those commercials.
But yes.
Yeah.
Yeah.
Like alignment is like just a really advanced capability eventually.
Or alignment will mean like the same thing as like very deeply competent team that's able to tackle very, very like high level issues.
Oh, absolutely.
You can think of it like...
We went through a phase where we couldn't figure out what a bug was.
I know that sounds crazy, but like some people wanted a bug to only mean like you lost your data.
Other people were like, well, maybe your data is fine, but the computer still crashes.
And we eventually decided a bug just means the software and the computer didn't do what you wanted it to do.
You weren't happy with the result.
And we allowed customers to tell us any bug in the world.
Put them all in a database.
And so suddenly we went from only the bugs we could find to the bugs of the universe.
And we went from thousands to hundreds of thousands.
But then we put both a severity and a priority.
And so a severity meant like at one, you lost your data.
And at three, well, it was flaky.
And a priority meant must fix this.
And so we walked around the whole lingo of our hallway with Sev1, Pry1.
And IBM, the old people at IBM looked at us and said, we've been doing this since 1965.
Like, where have you guys been?
And we were all proud of ourselves for inventing it.
And so, so much of what's going on now is we're in the throes of inventing this way to talk about what is a bug in a stochastic statistical system.
But it turns out software has been doing that for years.
The weather forecast is exactly this.
It's a statistical model of what's going on, and it depends on a bunch of inputs and a bunch of outputs, and it's wrong.
And when the weather forecast is wrong, like when they predict a hurricane path, and then the hurricane goes through in a different way, the weather people all get together and look at the model and talk about the bugs in the model that led to the incorrect forecast.
They don't appeal to Zeus and say, please, Zeus, tell us why our weather was wrong.
Right.
Well, we're coming up on time, but it's been great having you on.
Yeah, we'll need more sort of fresh perspectives like this in AI world.
Especially from like software veterans.
Yeah, I'm quite sympathetic to this view.
We need software, but we also need like the AI people to listen to the operational security people because everything that happened with Hugging Face and OpenAI, it was an OPSEC failure.
There was no intelligence, no consciousness, nothing but...
a pure OPSEC failure.
And they need to treat it like that and they need to talk about it like that.
Because otherwise we're going to get legislation no one is going to be happy with because they don't understand what our industry is saying.
Thanks, guys.
Seems like we're on our way there.
Thanks so much, Stephen.
Stephen, thanks so much for coming on MTS.
Apple Podcasts and Spotify.
Follow us on X and A16Z and subscribe to our Substack at a16z.substack.com.
Thanks again for listening and I'll see you in the next episode.
This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product.
This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z.
Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates.
Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.