# Performance Architecture and the AI Coding Shift

**Podcast:** The Pragmatic Engineer Podcast
**Published:** 2026-08-26

## Transcript

Why do most devs not care about writing performance software?
And should we?
Today's guest, Casey Moratori, spent the last decade arguing that we should.
He also says that most software out there runs tens to hundreds of times slower than it needs to.
Today we discuss why the focus on performance took a backseat across the industry and why Casey thinks the tide is finally turning.
Why you'll want to learn reading assembly if you're serious about high performance code and why it's less scary than it sounds.
The saying, premature optimization is the root of all evil, while Casey says that the majority of people use it to avoid thinking about performance when they really should.
And many more.
If you want to get better at writing faster software and become a better engineer while doing so, then this episode is for you.
And if you're one of the people who hit a like on this post by Ryan asking for this episode to be more about programming than about AI, this episode is also for you.
This episode is presented by Antisysys.
If you work with agents, your job is no longer just writing code, it's also specifying and testing it.
and Antistasis is the most effective method of verifying agentic code today.
This episode is brought to you by Sentry.
You probably already know what Sentry is because you're a developer.
If not, just ask a dev and they'll tell you.
I use Sentry to monitor the backend of the Pragmatic Engineer for any and all events and errors.
Of course, Sentry doesn't only do errors.
They also have logs, replays, spans, profiles, metrics, and more because they're all connected to the same trace.
One new capability Sentry has I'm really liking is its ability to fix errors.
Let me show you.
Here's a list of errors on my admin backend.
There's a recent error on auth that I want to check out.
Let's have Sear run an autofix for us.
Sear is Sentry's AI debugging tool.
First, it generates a root cause analysis.
It's finding some problem with HTTP versus HTTPS URLs.
Cool.
Now that we know what's going wrong, Sear can create a plan on how to go about fixing it.
I could go and edit this plan, but I'm happy with it, so let's create the actual code fix.
Here's a code fix that Sear generated.
Assuming it looks good, and in my case it does, let's draft the pull request.
And boom, the PR is created, ready to merge.
What I love about autofix is how Sentry went from showing a list of errors inside my application to offering me a fast way to fix it and close the loop while I stay in charge of this bug fix the whole time.
Debugging got a whole lot faster and a whole lot easier.
Check out Sentry at sentry.io slash pragmatic and start detecting errors, diagnosing your root causes and fixing issues and regressions today.
All right, Casey, welcome to the podcast.
It's so nice to have you here.
Thank you so much.
It's great to be here.
Thank you for the invitation.
Now, I want to go back when we start to the beginning.
How did you get into tech, programming, computers?
Well, I guess computers, it's like very, very early on.
My dad was a programmer at Digital Equipment Corporation, which is a company that people will know if they studied computer history, but would not know if you just looked at the landscape today.
They're completely gone, right?
They got absorbed partly by Intel, partly by Compaq, I think.
There was, you know, they kind of got broken up.
At that time, it was kind of a really big computer manufacturer, you know, computers like the PDP-11.
That's a Digital Equipment Corporation computer.
The VAX, like things that you may have heard of in computer history.
Oh, these were these massive mainframes.
Yeah, mini computers as well.
So like smaller also sometimes than mainframes, like the kind of next step down, right?
And so in general, that...
That era, my dad was a programmer there.
He would later end up at Intel because, you know, like I said, parts got acquired.
He never actually left his job.
He just ended up at Intel through kind of digital's eventual demise.
But as a result, we always had computers at home, even though at that time, you know, that might have been a little bit odd.
You know, I learned a program when I was seven, which would have been in like, you know, 1982 or something like that.
And so at that time, you know, maybe you might have an Apple or Commodore kind of computer at home, like some kind of early computer.
I don't remember the exact dates of those computers, but most people didn't.
And it was only until a little bit later that you would.
And you probably wouldn't have had a programmer in your household to teach you, more importantly, right?
So so I learned really early on and that's when I got into computers.
How I got into games was I ended up randomly interning at Microsoft and I met people there.
And like I went and, you know, sort of went off into games through them.
That was that was how that happened.
But wait, how is the Microsoft slash games relationship?
That's not kind of a given, right?
Microsoft is not.
They have one game, right?
It's Flight Simulator.
Yes, at that time, it would have been very weird.
The reason that happened was a guy called Chris Hecker, who a lot of people don't really know his history because there was sort of two waves of bringing games to Microsoft Windows.
Because, I mean, now it's funny to think about now because people think of like, what platform are you going to play games on if you have a PC?
Windows is like the default.
Linux is now an insurgent to that, but Windows is the default.
And so you think about.
How did that happen?
You know, because that wasn't the case.
If you were back in the early days, Microsoft Windows, not a gaming platform.
It's almost nothing on it.
It's like Solitaire and Minesweeper and a few other sort of games.
How did it happen?
So what happened, the reason for this, this kind of gets into a technological reason.
The reason for this is that it was very hard to actually produce images that could be displayed on the screen quickly.
And to understand why this is is like its own kind of topic.
But in general, you can just imagine you have these this operating system running, which is Microsoft Windows.
It's controlling the graphics card.
It's often running at a fairly high resolution compared to what a game might want to run at.
You have to negotiate with it in order to display your bitmap in some way that won't destroy all the things that it's trying to display.
So on and so forth.
Early versions of Windows.
up through like Windows 3, Windows for Workgroups, let's say, if anyone remembers that name.
Was it after 3.1?
It was 3.51, I think is what it was called.
Or maybe just 3.1.
Yeah, there's NT351, no.
So it's like, I think you're right, 3.1, 3.1, I don't know, something like this.
Windows for Workgroups was, you know, around that time.
That version of Windows, which is in the 3 series, didn't really have a way.
to quickly use the CPU to fill pixels, which is what games need to do, right?
There's no GPU acceleration really at this point.
There's a little bit we could talk about, but it's not mainstream.
It's not in consumer.
So they need to be able to do this sort of thing.
They need to do it in a double buffered way so they can draw to a back buffer and then show it to the screen.
And that has to happen very quickly.
And there just wasn't a way to do this in Windows.
In Windows, you had to kind of go through this API where you would produce sort of a bitmap that wasn't necessarily in the right format for the display you were using.
And then it had to do a translation from that bitmap to the other one when it displayed it.
All this sort of stuff.
So games on Windows, like things like Doom, they're not coming to Windows, right?
That kind of future was not in the works for Windows.
And that was kind of what some people in Microsoft wanted to change.
And one of the people who brought this change about was a guy named Chris Hecker.
And he was like, OK, we could actually just make a library that did the fast blitz to the screen so that we could have a way that people could do these draws and get them on the screen quick enough.
to make gaming viable.
Would it be 100% as fast as DOS?
Probably not, but could you actually run, you know, some of these new games?
You know, Wolfenstein 3D, I think, would have been out at this time.
Doom was kind of on the horizon and that sort of thing, right?
And so he started this project that he was not supposed to do.
He did not have the authority to do this, called WinG, which I did at least stand for like WinGraphics or something like that.
Total Skunk Works Project.
He had cover from his manager.
I'm assuming I can say all this stuff now because it's ancient history.
He had cover from his manager.
His name guy's name was Michael Edwards.
And Michael Edwards basically just kind of ran cover for this, which is a thing that probably wasn't going to fly because they were in a division at that time, which would have been Microsoft Research today.
Kind of it was called AT Advanced Technology was the early version of Microsoft Research.
You were not supposed to be shipping core libraries for Windows like it had nothing to do with that.
Long story long.
What ended up happening is that product did ship.
WinG, I guess you wouldn't call it a product.
It's an add-on for Windows.
It did ship, and it was the first step towards DirectX.
People forget that.
WinG was the first way you did this.
And then eventually we had DirectX and dib sections and Win95 and all that sort of stuff.
When I went to Microsoft, amusingly, the person I was supposed to be reporting to as an intern was Michael Edwards.
When I showed up...
First day, along with the other interns, a guy named Rudy, a guy named Rajiv, we were all supposed to report to him, right?
Because you get a couple interns in under one, you know, sort of manager.
We show up and we're just taken aside, you know, after some kind of, you know, stupid HR orientation thing that was, you know, lame as it always is.
And we get taken aside by somebody.
I don't remember the lyrics.
They're like, look, bit of a problem.
The person you interviewed with and we're supposed to like report to.
They they aren't here.
Like, I don't remember exactly how they put it.
So you're going to go talk to this other guy and they'll find something for you to do.
Turns out the WING thing had blown up the previous week and Mike Edwards had stormed out of the building and had not been seen since.
That is what actually happened.
So that's I arrive at Microsoft, the person I was interned with.
He just he just flamed out and left.
No one knows when he'll be back.
He did end up coming back like a week after I got there.
But he was kind of moved over to a different division.
There was a bunch of triage work done there to figure out what's going on.
So that was my experience.
But suffice to say, it meant that I was right in sort of like ground zero of games on Windows.
So I met Chris Hecker.
I got to talk to a bunch of people there.
He actually took me out to see one of my childhood heroes, Ron Gilbert, who is the guy who did the Scum Engine.
Yeah.
He knew all these guys because working on WinG, he had gone out to see a bunch of game developers and like work with them and this sort of thing.
So I got to go to Humongous Entertainment, which was Ron's new company.
He gave me a secret of Monkey Island mouse pad, I remember.
So anyway, it was really cool.
And that's how I ended up getting into the game industry was through Chris Hecker.
And he's kind of an unsung hero of getting games on Windows because, you know, he just he wasn't out there making his story known.
But, you know.
I am now, I guess.
But it's so interesting to hear these stories.
Obviously, now you can share it, I'm sure.
You know, like for a while, this would have been like only within the inner circle.
But the fact that, you know, of course, DirecX was a huge success.
And it did like as far as, you know, from my vantage point, huge reason why games are big on Windows.
But now here's someone who just ignored, wasn't asked anything, just was doing something, got into conflict, fights and just like pushed an idea.
It's more than you think, because There were three people who really were the core people who pushed DirectX, meaning institutionally pushed it.
There are tons of programmers like Todd Laney who did, you know, really important core work.
It never would have shipped without people like him.
So not on the programmer side.
I'm talking about institutional side.
It's Eisler, Engstrom, and Alex St.
John.
I don't remember which one.
It was either Eisler or Engstrom was the tester on WinG.
So he came from that team.
So the start of DirectX, one of the core members of DirectX was on the WinG team.
So it's a direct lineage.
It's not even like an unrelated push.
So WinG really was the start of it.
And then DirectX was kind of the actual full blossoming into an org with Microsoft's blessing at that point, you know, that actually became powerful internally.
And then after after his Microsoft internship and getting exposed to all these folks, you actually went and.
build games tooling.
You then started your own studio as well, right?
So like how did that sequence?
I guess there's a couple of steps in there.
I worked at a startup with Chris Hecker that didn't end up doing anything interesting.
Then I went to a company called Gas Powered Games, which was actually a Microsoft.
I guess they're not a Microsoft studio, but their publishing deal with this Microsoft.
And they did a game called Dungeon Siege, which is kind of a, you know, I don't know.
It's not a particularly well-known title.
From there, I went to Rad Game Tools, and that's where I stayed for quite some time.
I did their character animation system.
That was a very popular product that ended up getting used in lots and lots of games.
It's still used to this day, much to my surprise, because that's a very, very...
I haven't worked on it since 2004.
But I guess other people had maintained it, and some studios just kind of integrated into their...
You could get source licenses, so I guess some studios just integrated it into their pipelines that have never removed it, and they just maybe keep updating it.
to keep it working the way that they want.
So anyway, that's what I did there.
And then afterwards, I've been independent since then.
I just have a company called Mally Rocket where we do various stuff.
I've done contract work for people through that.
We now do the substack through that where we do educational materials.
So I've kind of just done random stuff since then.
Although I have done some work on games.
I noticed your checklist.
You're talking about The Witness.
Obviously, that one was an actual specific title.
that I worked on.
But that was mostly just because it was a very big project.
And I was, you know, I'm friends with John.
So I was just trying to do some helpful programming on the side.
I did some stuff on how the movement system worked.
I thought there were some interesting problems that we could solve there for games.
And so that was a really fun project to work on.
Yeah.
And then today you're you're doing educational stuff on a bunch of subprogramming performance on your sub stack.
And what else are you busy with?
So we do actually have an unannounced project that we've been working on uses up sort of the rest of the time that I have, if that makes sense, which is not always so much.
And that we will we're hoping to announce it sometime soon.
But it's not quite out yet.
Believe me, I will.
I will.
You will hear me.
I will send you as soon as we have.
an actual announcement.
But given the fact that it is kind of like a split time sort of thing for us, because we're pretty focused on making sure the sub stack is good and all that sort of stuff, we're trying to keep it fairly tight lipped until we actually know we're mostly done because we don't know how much time we can always devote to it, if that makes sense.
Yeah, no, it's pretty typical games, really things, right?
Like tight lipped until you have something.
for good reasons.
Well, sometimes people play the other game.
They go like, look, we're going to day one, we're going to be very loud about this and try to build a community around the development of the game and all that sort of stuff.
And that's great.
So, you know, that's another route you can go.
But if it's not your full time thing, if you have, you know, other responsibilities, that doesn't seem great.
Right.
Because you don't have any insight into how much time you will actually be able to devote to it.
Right.
Yeah.
And on Substack, it's called Computer Enhance, and you started it with performance-related topics.
And that's how we started to talk, I think, about three years ago, when you already had a Substack.
And we had a direct message conversation.
I remember you messaged me saying, hey, Gergay, why do you think in the industry, people, software developers, just don't really focus on performance?
I think you specifically wrote...
I did.
You were saying how there's...
There's little emphasis on performance.
There's even though there seems to be overwhelming evidence that performance is critical to the bottom line of most software.
And I wanted to ask you, we've had a good back and forth on this.
And actually, I think initially I told you like, oh, here's why you don't need to care about performance when you're building, I don't know, distributed systems or like.
But since then, what have you learned?
Why do most developers don't care or even most companies, teams, engineering teams not care about performance all that much?
It's a really good question.
And I think.
I do think your answer at the time, if I remember it correctly, is certainly an accurate one for some subset of industries, which, you know, you said something along the lines of, look, a lot of these pieces of software that you're seeing, the user isn't the purchaser, right?
Like you were like, this is some kind of thing where, you know, somebody very high up is going to look and say, we need software for managing HR.
They're going to look at the cost of the software.
They're going to look at the compliance terms of the software, the legal liability, whatever.
Right.
And then they're just going to make a purchase decision on that sheet.
They're not in there looking to see whether it takes like, you know, whether there's a 30 second pause every time you want to try and access somebody's record.
Right.
And I think that's very true.
Like, unfortunately, the.
situation for a lot of enterprise software probably is that way.
So maybe an individual might well be upset about the performance of that software.
And I certainly hear from people all the time who are upset about the software that they use.
They might not be any position to change it.
I think that's one thing.
There's thing two, which is that in a lot of cases, you simply have monopoly effects.
You know, people aren't right now.
realistically going to challenge the social networks that currently exist, for example.
People have tried.
It's very hard.
You know, Blue Sky and Threads have tried to assail X.
And, you know, you've got Facebook and Instagram and TikTok, and they kind of just own those spaces.
Right.
And it's very hard to push into those because of these like network effects.
maybe performance could be part of a package where you try to take on one of those players.
Like, hey, look at how much more responsive our thing is than theirs.
Might be a nice plus, but that's not going to be sufficient.
If you just show up with no plan for how you get adoption, no plan how you get big influences over there, all that sort of stuff, it can't sell a product on its own into a monopoly space, right?
If you're just talking about apps that someone can choose to download, maybe you've got a shot there, but those are just...
they're forming a smaller and smaller subset of what software is, right?
And this bigger and bigger subset is like these monopoly platforms you go on to sort of work with.
So I'd say that's another thing.
Thing number three is I think now people sort of are caring about performance more.
I think over the past decade, the people, including myself, but many, many other people who have been saying that this is a problem have actually had some effect.
Like, I don't think that it was a waste of time.
I'm seeing a lot of new...
emphasis on performance, people talking about performance, people posting benchmarks on things.
And so I actually think that the third thing is, well, actually, it kind of does seem that pointing at this issue and saying this is something we should be doing better has not been completely a waste of time.
I do see things as sort of starting to turn around a little bit.
I also see people attacking major product categories now with performance based pitches.
Things like FilePilot or the Blick Video Editor, like things like this that have been coming out lately, where it's like, oh, really performance software to try to take on incumbents in a space.
And they've been getting traction.
So I think that's also a really good sign.
Yeah, and I guess on this last category, a really good example in the developer community is BUN versus NPM, where BUN just said like, OK, we're like 10x or 20x or 50x faster.
And devs are like, what is possible?
And then it was so they.
there was this outrageous claim.
I wonder if you need to have these outrageous claims because dev started to pay attention because it was 10x faster in many categories.
Linear versus Gyra is also a good example where Linear have this benchmark of, okay, they have 300 milliseconds for any action and Gyra, of course, we know is...
It's just slow because they have a bunch of complexity.
We can explain why, but it's slow.
It was never built for that.
Yeah.
And, you know, if you think about something like a 300 millisecond budget for an operation, 300 milliseconds is like an eternity in computing.
Right.
And so if you're talking about like our pitches that were not more than 300 milliseconds, that just shows you how the bar was so.
far past where it probably should have been for something.
And you see it everywhere.
You know, you go on to programs and you're waiting sometimes seconds for an operation.
And I don't think people realize just what an eternity a second is in modern computing, especially when you're sitting on networking like that has, you know, sub 10 millisecond ping times.
Sometimes you're talking about this.
you know, the actual packets had to travel physical distance to get to this data center.
And that was being done far faster than this very simple operation that you were failing to do in a reasonable amount of time.
It's just like, we are massively underperforming.
And people don't believe it when you say 10x, 100x, but it's actually true.
And we've seen a lot of proof of it, as you point out.
I do wonder if one part of the not really much focus on performance is that a lot of developers don't know.
the baseline thing.
And I'm reminded by Simon Erickson, the founder of TurboPuffer, has this project called Napkin Math, where he did a list of mostly networking operations.
How long does it take to transfer one byte between two AWS data centers?
How much does a gigabyte take?
How much does a terabyte take?
How much does it take to write an SSD to an NVMe and so on?
And so he had these numbers and he said that what he found is whenever in Shopify they were deciding do we choose vendor A or vendor B as a database, they would just run a benchmark that they would write themselves and they would get like, okay, like, I don't know, storing this and this, it takes two seconds on this one, 10 seconds on that one, we will choose a two second one.
And he looked at it and said like, hey, like, this doesn't make sense.
Like the amount to store in a file system, like here's a theoretical limit, which is, I don't know, 100 milliseconds.
Like, there's no way that's going to be 10 seconds.
And it often turns out that he found that the benchmark was just wrong.
They were benchmarking the wrong thing and they were making decisions.
So I wonder if there's a thing where many engineers, developers are maybe just not aware of how truly devastatingly slow this thing is versus the resources you have.
That is the entire point of like my sub stack, right?
So what you just said is exactly true.
And it is the thing that I hammer home on the Substack through all the parts of like the courses on there, which is that in general, there's a misconception about the way that you approach optimization in like in computer science or in whatever software engineering, let's say.
And that misconception is that what you do is you run a profile, you identify where the like, you know, big parts of the profile are.
You make some changes to those and you measure like statistics, the better the statistics, you know, the more statistics you can get, the better.
And you look to see if those statistics improved, if they have, that was a good change and you proceed as such.
And this is completely not correct.
That is not how anyone has ever, you know, I've worked with many extremely good optimization people and that is not how it is done.
The correct way to do optimization is very much like what you just said.
You first go, what are the operations that this system has to perform?
What is the underlying hardware capable of doing at its theoretical peak?
And then you measure the delta between that theoretical maximum and what you have achieved.
And then your goal during optimization is to shrink that gap to something that you think could be plausibly explained and hopefully come up with explanations of why you aren't at theoretical.
Because oftentimes you can't hit theoretical.
That's why we call it theoretical, right?
And it's crucial that you do this because otherwise all you're doing with that other method is finding a, you know, with the, I'm just going to, you know, make something I think might be an optimization and look if my statistics improved.
All you're doing there is finding a local minima.
That's all you're doing.
You're just, you know, you've got this shape of your performance and you're finding some little spot and you're sitting in it.
That's not optimization.
That's improvement.
But optimization means to make optimal, right?
It means we're going to find what we actually should be able to get this machine to do.
And so, you know, that's why I emphasize that approach, because it's the one that I've always seen great optimizers take.
That is how they get good performances by knowing what the maximum could be.
In addition to that, it also is what lets you become better at optimization, because no matter who you are and no matter how much you already know.
When you go to tackle an optimization problem, there may well be some things in the new way that the system is laid out that you don't know about.
New things that people have not figured out about modern CPUs, new things that are different about the network backplane, new things that are different about the GPU drivers.
Who knows, right?
And if you don't have some theoretical maximum to look at and to measure your delta from, you don't know if there's some serious anomaly there.
And you would be surprised at how many times we find anomalies like this.
Things in CPUs that no one knew about.
And I've literally had them in the course of making the substack.
I've been like, what is this thing?
And I look into it, it's like, oh, there's this new renaming, this new rat table thing that Intel chips seem to be able to do.
We didn't know about that.
And that's like a new thing we have to model when we talk about how to do performance.
And so that's the other crucial part of, I guess, what you were calling napkin math.
I also call it back of the envelope.
That's the term I've heard used for it.
Oftentimes they're kind of interchangeable, right?
Knowing what the theoretical is, is how you learn as well, how you learn about new hardware and new performance options.
Interesting.
Plus, by doing this, you're just learning.
You're becoming better professional.
You understand more about given hardware or the inner workings of your computer or software stack or kernel, all the stuff that I guess goes way beyond the...
the vanilla programming language.
Like, because you can say I'm an engineer, I'm a software engineer because I know how to use this programming language.
But I'd argue you're probably an engineer if you can go down the stack and you have that ability and you have like a good understanding of some of it at least, right?
And then you can learn the rest.
And I would also say that one of the other things that we do in the class is teach how to read assembly language.
And people often ask, like, you know, assembly language, what would I ever need that for?
There's a very good reason for it.
And that is that everything else that you might use, doesn't tell you anything about what the CPU is actually receiving.
You know, if I look at a Java program, if I look at a C program, if I look at Haskell, OCaml, whatever, right, Rust, all I'm seeing is input to a compiler.
I have no idea what the CPU is actually going to be asked to do.
If I look at the assembly language output from that compiler, I know exactly what the CPU is being asked to do.
And it's not that hard to be able to learn to read a semi-language so that you can see very quickly, is the CPU being asked to do the things that I think it should be asked to do them and in that way, right?
You don't have to write it hardly ever.
It's very rare that you have to write assembly language to do anything other than sometimes for test purposes, it's easier to do that.
So you don't have to try and convince a compiler to output something.
So if you're just testing something, sometimes it helps to be able to write some assembly language.
But if you're just talking about the vast majority of tasks you might do in optimization, writing it, no, reading it, essential.
And it also unlocks this sort of huge world of possibilities to you because once you know assembly language.
You can now do things like read those CPU diagrams.
Like, you know, when they announce a new processor, they put up a little diagram.
That diagram tells you stuff like the fastest this thing could do multiplication and stuff like that.
It tells you that if you know assembly language, you can read it right off the chart, right?
If you don't know assembly language, you look at that chart and like, I have no idea to do what I'm looking at, right?
Like it's just this weird flow chart that doesn't really tell me anything, right?
And so one of the really great things about assembly language is it unlocks all of this knowledge for you because it's the actual input language to the machine.
And it allows you to figure out how it's operating.
Plus, I guess we should add that assembly language is not all that complicated, right?
Just by nature.
It's a far simpler language.
Okay, it's harder to read if you've never seen it, but it's in terms of the number of operations, it's so bare bones because that's what assembly is.
Like every single higher level language will have way more keywords, structure, whatever, you name it, right?
Than assembly.
Massively more.
And especially when you consider the subset that are actually used.
If you look at the subset of constructs that you would need to understand to be able to understand, say, just a website from today, all of the JavaScript libraries, all of the JavaScript syntax, all of the DOM, you know, all of the behavior that's going to go on the CSS.
React, CSS, right?
All of that.
Assembly language, you know, maybe there's 20, 30 instructions you might have to learn total.
Because most assembly, most things in legacy assembly like x64, most of them are hardly ever output by the compiler.
So you only need to learn a very small subset.
That's the ones that are actually going to be, that you're going to be seeing in 90% of the cases.
It's so much simpler.
And also when you're looking at performance, you're typically only looking at a very small part, right?
You've kind of understood roughly what's going on.
You've seen the basics layouts of your program.
You've identified what's supposed to be happening.
And you're just looking to see like, wait.
Why is this part, which I don't think should be running this slowly, why is it running this slowly?
It's just a very small piece you typically end up having to look at as well.
So it's really much easier if you can understand how to center a div, as they say.
If you can vertically center a div in HTML, then you can probably learn assembly language, I would say.
Okay.
You're super passionate about performance optimization.
You also have really good educational materials, both free videos, your paid sub stack, the free parts of it, etc.
But let me just play devil's advocate.
There's a saying that premature optimization is the root of all evil.
And we typically use it or I typically use it so many times.
We're like, oh, should we make this performance?
Should we optimize this thing?
And like, nah, let's not do that.
Let's first build it.
Let's see if it's good enough for our customers, for ourselves.
And if we need to, we can always optimize it.
I mean, you know, like it's not the hardest thing in the world.
OK, maybe not as good as how you mentioned, because maybe I don't really assembly.
And that's kind of thinking of building, you know, like a SaaS software, building software at Big Tech.
What is your reply to that?
Because I feel really good that I made a really good argument here.
So I guess what I would say is the important part about that, and I guess I'll divorce it a little bit from the saying.
I have an entire lecture on that saying, by the way.
It's like two hours long.
And I gave it at Better Software Conference this year.
I believe the VOD will be coming up.
We're linking that in the show notes below.
Okay, it'll be like a week or two, I think, till it's up.
So it may be right at the same time as this.
But so if you want to find out the history of that phrase, you can go look at that.
But I wanted to talk about the idea behind it because I think there's, I don't want to dismiss it entirely because it's not entirely false.
And the idea is that, well, I'm just going to delay optimization work.
I'm not going to think about that.
And then I'm just going to make whatever I'm going to make.
And then, you know, either myself or maybe I'll just hire some performance person to come in and.
Clean up the mess later, right?
So here's the positive side of that first.
The positive side of that first is for some types of code that will work.
If you happen to have written some operation poorly, where the optimized version of that operation just looks like someone taking a loop and changing the loop from your really like, you know, naive version to a really well optimized version.
The typical of like, I wrote a bubble start, we can later optimize that.
Who knows, right?
Anything of that form.
Okay, maybe we can just do that.
So there are certain times where you do in your head want to be doing this, where you want to say, okay, I could go spend a week researching the fastest hash table implementation here.
But part of software engineering is being smart enough to know it won't matter if I do that now or later.
The architecture around this piece won't have to change.
I'm quite certain because I understand the problem well enough.
It's okay.
I can defer that to later.
Maybe it's never too slow with the naive one I put in there and then we don't have to do any work.
Maybe it's too slow later.
That's okay.
I just target this one hash table implementation and we'll get as fast as we need, right?
If you're doing that, if you're applying that true engineering mentality to it, you don't have a problem.
The problem comes when you don't know if the choice that you're making produces that kind of optimizable hotspot.
And I'll give you a very simple example that usually people have had experience with.
A very simple example would be we write our entire software thing, like we just, whatever this massive thing that we're imagining doing where we're going to ignore optimization.
We sit down and we write it and we use a paradigm where we ask the server for something.
We have like some API, you know, that we've built for asking servers for things.
We ask the server and it returns to us what the server's response was.
And that's like kind of how we architect this thing.
So everyone writes, you know, hundreds of thousands or millions of lines of code and they all look like ask the server something, do some calculations, ask the server for the next thing, do some calculations, right?
Then at the end, you find this is way too slow, but that's okay.
You weren't worried about that because you're like, we're an optimized at the end.
You call in some performance experts, they look at it and they go, there's nothing we can do for you, sorry.
Why?
Well, the reason is because you created a serial dependency chain.
All of your code looks like wait for a network request to come back, do something.
Wait for a network request to come back, do something.
And that serial dependency chain can't really be shortened without just rewriting it.
If instead you had made the paradigm and told your programmers, look, here's what you need to do.
At the top of every operation, you need to figure out all the things you might want to ask the server for.
You ask them for all of those things, right?
And then you do all of your processing there and you only create a chain of dependencies if you absolutely couldn't have determined what it was you needed to ask the server for.
Now you're just in this situation because you didn't tell them to do that.
You have to rewrite all your code.
Everyone is now going out, rewriting all the code, if they even can, if it's even possible to really do that in a way that's not slower than just rewriting the thing, right?
So what happened there?
Well, again, we talk about this a lot on the Substack, but there's this idea of a serial dependency chain.
It's when you stack things in order, right?
And the performance of your software is generally determined by the longest serial dependency chain because it's something that cannot be parallelized.
If I have thing A that then B depends on, that then C depends on that.
We cannot shorten that because it has to go in order.
Everything waits for it.
We can't multi-thread it because it's dependent.
We can't make it run wide.
We can't amortize the network request, whatever.
That kind of thing can be pervasive in the programming and we can't cheaply remove it because it's not a hotspot.
It's a way that you did things.
That's the part where that kind of thinking breaks down.
If every software engineer knew to watch out for false serial dependency chains, things where they were creating series of dependent operations that could not be optimized the way or other sorts of architectural problems like that that cannot be easily fixed, then the world would look more like just wait and optimize the hotspot, right?
Yeah, so this is the architecture, the planning, right?
Like if in that phase you're like, okay, like as this thing grows, like what would get in the way of performance?
What would slow it down?
Or you can ask all these questions or like different flavors of these questions, you know, or from the other side and so on.
Another way to think of it is because hotspots is the way that people talk about that.
Like, oh, it's going to be hotspot optimization.
We just got a few spikes.
Someone will come in and clean up those spikes and we're done.
Right.
The way to think about it is your code base will not end up that way by accident.
In most cases anymore, you have to engineer up front for a hotspot code base that people can then optimize.
Right.
And so that's the crucial takeaway is everybody on your team who is making architectural decisions.
Those people must know performance and they must make decisions that will allow the other people downstream of them to use an architecture which can be optimized later.
If you don't do that, you're just rolling the dice.
Casey just talked about how engineers making architectural decisions should know about performance.
This is also true when choosing your dependencies like which database to use.
And this is where I want to mention our season sponsor, TurboBuffer.
You already know how TurboBuffer is a vector and full-text search engine.
But here's an interesting story from Linear on what happens when you stop thinking of TurboPuffer as a search engine and start using it as a primitive to reduce latency.
As context, Linear is a local first app.
So each client keeps a local database, and when that client goes back online, it needs to catch up with what happened and do it fast.
Their biggest workspaces generate around a million sync actions per day.
Doing catch-ups by reading from Postgres was getting slow for large reads, so the tail latency got too large.
and adding more replicas did not help either.
Linear solved the problem cleverly.
They started using TurboPuffer as a serving index for each client.
This is because TurboPuffer itself is built on top of inverted indexes.
So for every index value it stores the documents that that value can be found in.
The lookup cost for such an index is constant.
So Linear took this structure and had each client's index point to the changes that they needed to sync.
As a result, not only did they reduce latency, but they kept it constantly low no matter how long the change list is synced to the client is.
Linear published a blog post about this refactors titled Rebuilding Linear's Delta Sync Read Path.
Check it out.
I love this story because it shows how important it is to choose the right primitives and how good primitives can improve your system.
If you're building systems where you store a lot of data or serve a lot of data, TurboPuffer can probably speed things up or save on your costs.
Learn more at turbopuffer.com slash pragmatic.
I'd also like to talk about our presenting sponsor, Antisysys.
While Casey deliberately does not use AI coding agents for his work, most of us do.
And when you work with coding agents, your job is no longer writing code, it's specifying and testing it.
Antisysys is the most effective method for verifying a genetic code today.
Let me explain how it works.
Antisysys runs your whole system in a hostile simulation.
By doing so, it finds every bug before your users do.
And because the simulation is fully deterministic, Antisysys doesn't only find bugs, it gives you a perfect reproduction of every issue.
To create such a tool, the Antisysys team needed to invent new kinds of debugging tools as well.
For example, here's what's called a bug probability graph.
The x-axis is virtual time and the y-axis is probability.
As Antisysys runs a hostile simulation, it plots timeframes when the bug probability increases, which greatly helps us find the root cause of the bugs.
And Antisysys also has a log visualizer.
Vertical lines going down represent events branching off from the same state.
And the purple dots are where the bug happens.
Antisysys is as good as it gets in being able to ship agent-written code.
It's what teams at Janestreet, Fly.io, and the etcd community use to ship with full confidence.
Head to antisysys.com slash primatic to learn more.
And with this, let's get back to Casey and how, if you don't design an architecture that can be performance optimized later, you're just rolling the dice.
And we've seen so many projects.
I have an entire video where I go through like, look at all these blog posts of people who like say, you know, it's Facebook, it's Uber, it's everybody.
They've got blog posts of we had to rewrite this whole thing because the performance is bad.
If it was always hotspots that made your performance bad, you'd never have to rewrite the whole thing.
So we know that that doesn't work anymore.
Why?
Because of the things I just said.
I was at Uber where I was not making the decision, but the teams next to me were and I saw or I kind of understood why they were making.
But typically, and right now it's happening with AI companies.
Oftentimes, it's like we chose this technology, which is Python, and it's single threaded.
And it made sense at the time on the web server.
But now we're big.
And this happened at Uber.
It was Python and Node.js.
And then they went to Go and Java on the back end.
And now with AI companies, it was Python, OpenAI, and Entropic are both going through this right now.
They're both either public about it or everything with Entropic.
They share with me, but I put it out there.
They use Python because...
data scientists or AI or machine learning engineers, new Python.
They put in a bunch of web servers.
They had their API run on it.
Initially, they just, you know, scaled horizontally.
But now they're like, well, if we move over to Rust, they right now they're choosing Rust or Go, but I think it's Rust.
Well, we can actually have multi-threaded and the same machine can actually handle more connections.
So cool.
I came across a lot of that because I think that's easy and safe to communicate because it doesn't look bad on you.
But you're right.
A lot of times.
I don't think on enduring blog posts, you'll get the real reason that these companies put out there.
Like when it's kind of a very kind of, you know, easy to own mistake or not mistake, but just the decision, which made sense, they'll tell you.
But if it's something that was an oversight, you're not really going to get that on a public facing enduring blog post, except for maybe some startups who are really there.
But don't forget, like a lot of those blog posts are going to help someone get promoted or get recognized.
They will always be way more positive, especially when there's a content writer team, which large companies do have.
So it's not quite PR, but it's somewhere midway in between.
And I mean, yes.
And also, I would just point out the fact that like the fact that these things are happening, though, is all we really need to know for the signal.
Right.
Because in general, this should not be happening.
If the ideas about optimization were true, you'd never have to rewrite something in a language in a different language unless you just preferred that language.
It would just the story would just be we rewrote it in this language because we wanted to use this new language.
It would never be or for Rust, it might be just memory safety.
We see those blog posts, right?
It's like, why do we write in Rust?
It wasn't performance.
It was just we wanted the memory safety or something like that.
If I'm a software engineer programmer.
And I'd like to just get better at writing performance code.
I'm interested, you know, maybe after this podcast or looking to some of the things that you did, what is a learning path you would follow outside of your sub stack where you cover a lot of these things?
But what are areas that you think are kind of like you need to understand these things to like get better at writing performance code?
I think it's actually very simple and perhaps a little bit counterintuitive.
So I'll start with the very good news about learning to write performance software.
The good news is that optimization of the kind that we sort of talked about, the like hotspot kind, where it's like somebody is going to go in here, maybe they're going to even rewrite this routine in hand coded assembly or something crazy like this, right?
That's very rarely necessary these days.
One of the reasons that you...
don't see hotspot optimization as a thing that really matters that much anymore.
And one of the reasons I advise that architecture and not making bad decisions is much more important is because a lot of libraries already have been optimized for you that you might use.
CPUs are incredibly good at taking bad code and running it quickly and so on.
So typically when we're talking about the causes of performance, negative performance, that aren't squeezing every last little thing out of the hardware, but rather just making sure this thing isn't running like a hundred times slower than it should be, usually it's more just about having an awareness again of what the computer should be able to do and making sure you're making software architecture choices that allow it to do that.
And if you do those things, you will generally be within, you know, 2x or something, which is 50x better than the people who are 100x away, right?
So the good news is in order to write software that's much better than a lot of the software you use today, you don't have to know that much.
So what do you have to know?
What I argue and what we focus on the sub stack is I think you just have to go through the experience once of learning, reading the assembly language, seeing how the CPU works.
Seeing the difference, seeing why Python is slow, which we show on the subs.
So one of the first things I show is I walk you through the assembly language necessary to execute A plus B in Python.
And it's so vast that, you know, I have to skip most of it.
It's massive, right?
It's like this huge.
And whereas, you know, if you have the equivalent function in C, it's one instruction, add, right?
So, you know.
understanding basic things like that.
If you go through, learn to read a semi language, learn to look at some code, learn to do some CPU timings, and you just have that experience, just spend, you know, a month or two of nights or whatever you want, just understanding some performance stuff and going through a few examples where you play with it and you see the difference.
And just so I understand, you're saying do this not because let's say you're doing iOS development or web development, read what React, like you will not look at the assembly that the React does.
But if you do this on a project, you will be able to conceptualize what is likely happening, what the layers are.
And you might be able to decide, like, do I want this layer or do I want to use, let's say, WebGL, which if you're a React engineer, you probably haven't touched it.
Again, you can skip a bunch of those things and it comes to trade-offs with maintainability, yada, yada.
But that now you will know like kind of what you say by keeping this layer or not keeping it and so on.
Do I get that right?
Essentially, yes.
And like, you know, the simplest example is the Python example.
Most people have never internalized the fact that it takes, you know, maybe on the order of 100 more CPU instructions to do an ad in Python than it does to do it.
in an equivalent language like C for the same piece of text, just A plus B compiled in two different, you know, in two different languages, right?
And so just understanding even just that is enough for you to kind of go like, oh, okay, A, now I kind of understand why if I'm using Python, I kind of have to use libraries to do things.
And those libraries were written in C because it's like if I'm ever going to do any operations on a large number of things, I can't do it in this language because the amplification factor is so high on each operation that it just kills the performance immediately, whereas these other languages don't have that.
Right.
And so understanding those orders of magnitude and what's actually going on, I think that allows the programmer to know, OK.
If I think through what I'm doing right now, can I afford the super slowdown that I'm going to take?
And usually I don't think you have to be a performance expert to make that decision.
You could usually know like, okay, is this a part of the code that can afford to be a hundred times slower than it should be or not?
Right.
And, you know, most people can, I think, make that decision fairly logically.
And if it's not, then now, you know, like, oh, okay, if I'm in Python, then what I got to do is.
Either I got to go find a library call that will do these sorts of things and structure around how that library works, or I should maybe get something like Cython or something where I can do compiled stuff inside my Python and make my code work around calls out to that kind of code.
You know, you now have the tools you need up front to make sure that when you write the program, you've put the parts that needed this.
And you've structured the code in such a way that you are only paying the 100x on things that you know are very infrequent or happen like only, you know, once per every so often, things like that, right?
That's, I think, the biggest thing is just the knowledge.
And once you know, you can start to make much better decisions in any language because it doesn't take you very long.
You know, a simple search or, you know, asking an AI or whatever is the common practice that you're going to do.
A simple...
A bit of that, once you know what you're asking for, will get you this information back very quickly, right?
You just have to know that you should have been thinking about it.
Now, as a software, and you mentioned it's good to understand how the CPU works.
As a software engineer who is not a games developer, I'm not doing low-level stuff.
What does that give me?
Because for the most part, even in academia or in computer science, there is some level of some basic CPU theory taught.
But usually we just kind of, we stop at the code.
Okay, maybe you look at the assembly, but you rarely go further than that.
The folks who, you know, you've taught and they learn these things, what you see them get out of this that they wouldn't otherwise.
So you're talking about specifically the knowing the CPU part.
Knowing about the CPU, knowing about the details about the CPU, because you mentioned that that's also part of it, right?
It's not necessarily just stopping at assembly.
So the reason for that is more the other way around.
the reason to learn the assembly language is so that you know what the CPU is doing.
So it's the CPU part that's actually important.
And the part that's important about it is that the CPU is basically, you can think of it as a little machine whose internal gearings we are not privy to because for the CPUs that we care about, so, you know, an M series CPU in a Mac, a Zen core CPU in a server.
or in a laptop or an Intel core series, those sorts of things.
These CPUs are not documented at the level where you're going to be thinking about how each little individual part works.
And to that end, it's unclear that you would have time to do so anyway, because these are massive, very complicated machines that we're talking about, right?
But from a high level, from a more black box perspective, they are machines that we can think of in relatively straightforward ways once we know kind of what their core instructions are that they tend to execute.
And they break down into a couple of different categories.
There's how does data move into and out of a core?
And this is basically how like load store units work, how the cache levels work, L1, L2, L3.
So we have like we have L0 now sometimes, things like this.
How does that work and why?
What is the granularity of it?
What is the policy?
How does the CPU go about actually working with those things?
Understanding that part of the machine is crucial because when you're working with a lot of data, the difference can be massive if you structure it in one way versus structure it in another way, right?
Again, architectural decisions that have nothing to do with hotspots.
They're how all the data is laid out and what the access pattern is, right?
Things that are very hard to change sometimes.
So that's one part of the machine you want to understand.
Another part of the machine you want to understand is how the instructions flow through it.
And, you know, a lot of people have heard about like...
branch misprediction or things like this.
I cache misses.
There's words that you might hear, but you're not really sure what they mean.
They're all actually pretty simple to conceptualize.
Sometimes they're harder to pin down exactly how they work because branch predictors are, you know, getting more and more complicated and so on.
But you can still categorize the behavior of them and understand how code flows through it and when you might care and when you won't.
And then finally, there's the execution unit scheduling part, which is about knowing.
what's the raw sort of throughput for any particular type of operation?
Floating point multiplies, integer additions, division, whatever it is that you might want to know, right?
Once you learn a little bit of assembly language, you understand what it's reading.
You understand how it turns those assembly language instructions into micro operations, which it actually does, and how they get distributed through that machine.
That flow chart that they put, like basically up on the, we've announced the new Zen core, that flow chart, you can look at it and go, I know the performance of this machine roughly, right?
Not exactly, because like I said, there's all these little edge cases that if you really want to be a crazy optimizer, which again, I don't really advocate people do.
I don't think it's important that they be like crazy hyper optimizers.
If you like to do it, great.
It's a lot of fun, but it's not the important part.
Just look at that.
You're like, okay, I see what the CPU should be getting in terms of like what I could do with this size data load, that size data load.
This is what I could probably get out of it for if I was doing a bunch of like, how to do a bunch of like math ops on it, you know?
And I think that's just something that should be kind of par for the car.
So software engineering, you go to school for four years to learn this.
There's no reason you can't learn this in a few months.
It's not that hard.
Yeah, plus I guess just from a craftsmanship perspective, like we should know our tools.
We should know the machines that we're programming.
Obviously, we know that our code will be, if you're doing web, it'll be running on all these different things or if it's mobile on all these different phones.
But from a conceptual point of view, like we should be able to know what's going on.
So I feel there's a bit of a pride as well.
Like if nothing else, you would learn a bunch of stuff.
Like I know some of it, but I'm now getting motivation to learn more about it.
I do think there's a craft and shit angle.
I think there's a large number of people who maybe don't feel fulfilled when they, I've certainly heard from lots of people who, when they write something and it's just kind of this amorphous high level thing, they don't get as much satisfaction out of it.
And then when they learn how they can look more deeply at what's going on, they feel much more satisfied, even if they didn't change what level they were programming at.
they feel much more satisfied that now they know what they're doing, right?
And it's like, oh, I see.
And I understand why this thing was happening this way and this thing was happening that way.
That's very satisfying, right?
So there's an aspect of that.
I also want to emphasize another part, which is that it's a percentages game.
If we convinced enough library maintainers that this stuff was important and the libraries all get a lot faster.
all of a sudden, all the people using the libraries code gets a lot faster and so on and so forth.
If the API is start changing to make it easier to optimize the libraries, because people now thought that through, right?
Like it's infectious.
The more people are doing performance, the less people need to do performance.
If that makes it kind of paradoxical, right?
Well, plus, plus I do think right now there's still an edge in inches being performant again.
And you mentioned, but there are categories of software that is just winning by being much faster.
And to do that, you need to do this.
And if you know how to do this, maybe you're going to spot opportunities as a software engineer.
Maybe right now you're not as happy in your position, but maybe start something or do a side project that turned into a full time thing and so on and so forth.
So like I feel there's like, worst case, you just learn like net new knowledge, which will probably not be as outdated with AI, which we'll get into later.
But this stuff, it feels it just feels very interesting, right?
Like kind of it moves your brain.
It does.
And thankfully, like.
It's also not that hard to update your knowledge because, again, you get these presentations that the CPU companies give and they're like, here's the changes we made.
So you kind of are aware.
Every time a new thing gets, you know, there's little tiny things that creep in that you don't that, you know.
But again, there's people out there who are running lots of micro benchmarks that you will find out about.
And they often uncover these for you as well.
Right.
So I want to talk about games.
You've been you've built games for decades at this point.
Can you give an overview for those of us who are not in the games industry?
How is a game typically built?
from the games that you know of, that you've observed or you worked on, especially trying to compare for, you know, like a typical SaaS or distributed system or something, we're building a website or services kind of, you plan this stuff, you, you know, we'll do an estimate, we'll build it in a few months or a few weeks, we deploy it, and then we monitor it, and then we keep tweaking it.
And then, you know, fast forward five years later, it's now this like gigantic thing with microservices, but it keeps evolving, right?
Like it's, we do this like a lot of prototyping thing.
For games, it's pretty obvious, right?
From the get-go, as we're talking, like there will be a launch.
But can you, when you're inside or when you join a game studio, like what would you observe there in terms of what the process is like and how it's different or how it feels weird compared to like this, I guess, I don't know, traditional software, SaaS software, whatever development that is.
So I guess what I would say is, unfortunately, I'm probably the wrong one to ask because my knowledge is outdated at this point.
Because one of the things that has happened to games recently is I feel like they've moved closer.
I don't want to necessarily say entirely in development practices, but at least in terms of the nature of the product has changed somewhat dramatically to be more like something like Sass.
where, you know, if you take some of the most popular things that are being in terms of dollars, let's say.
So I guess maybe popular is kind of might be hard to say specifically, but let's just say revenue generating.
So if we were to measure the total games industry revenue and you look at what are the largest slices of that, you're seeing things like Fortnite, like Roblox, like Grand Theft Auto 5 Online, et cetera, et cetera.
Minecraft.
these things are starting to look a lot more like an always on live service kind of we ship incremental features to our customers kinds of things.
And so I would actually say that, you know, I'm the wrong one to ask about what that actually looks like from the inside because I haven't actually gone and worked at one of those companies.
I have friends there, so I hear things, but I'm not the right one to like give it an accurate picture of it.
But I just would point out.
From my perspective, the game industry practices look different today than they did when I have a more intimate sort of experience with what we were actually building.
But can we talk about it when you were building games, which was, you know, 10 plus years ago, when I understand these were the games where they were built, they were launched, you know, maybe they got a patch or two, and then they were kind of, you know, the team moved on, they were disbanded.
It was a time box thing that was a lot of development, a big launch, and either it...
It went big, huge hit, or a huge failure, right?
And the studio goes bankrupt.
So how did that work?
Because I feel that's a world where, okay, today a bunch of games don't have those constraints, but it has a bunch of constraints that I'm interested in what worked in those constraints.
So...
In the early days, you didn't have licensable engines.
So up until sort of the point, like nowadays, like this is why I say it's a lot different now than it used to be.
You know, nowadays you think of web development, like I'm gonna go grab like a thing like React and I'm gonna make this thing or whatever.
I'm gonna go grab an off-the-shelf database thing, Postgres or Oracle, or I don't know what would be the thing of choice, right?
But by the way, just definitely Postgres and not Oracle for most people.
Okay, sorry, sorry.
Postgres, I didn't want to slight anybody there, so I apologize.
So, okay, definitely Postgres.
Sorry, Oracle.
So you're going to go use some kind of a variant of one of these off-shelf databases and so on.
That's more, like I said, what people might be doing nowadays, too.
They'll grab the Unreal Engine.
They're not going to develop an engine on their own.
They'll grab back-end server stuff from people.
It might even be some Postgres in there, right?
Like, who knows?
In the earlier days, none of this stuff existed for games.
I actually worked, like I said, in middleware at the time.
So I was actually sort of one of the people who was working at the time on maybe changing that a little, like producing code that would get reused throughout games, which was actually fairly rare.
And so back in the day, every game built their own rendering engine, for example.
Correct.
And so, yeah, yeah, yeah.
And so really early on, right, if you rewind the clock far enough, then yeah, the degree to which people were reusing code, for their thing for like their rendering engine it'd be like because i got some code from like dave or whatever who was or we were both at atari and somebody wrote this good routine so we used it right there was that kind of thing but there wasn't like this sort of set uh engine and the time when that sort of maybe you could say first started happening a little more widespread was with things like what id software did where they sort of started having like, oh, you know, like someone's going to build something with the Doom engine or someone's going to build something with the Quake engine.
There was also the build engine at the time made by Ken Silverman and some things like that.
So there were some early cases where a few people would make a game, but they were making a game very much like that.
Like if we did the Doom engine, we're going to make a game very much like Doom.
So it really was the case that for most games, people were rebuilding most of the things from scratch, at least for their studio.
And studios often their existing code base was kind of part of the value of the studio too.
Like if you are, think Blizzard and we just built Warcraft 1, well, rolling all of that knowledge and code into Warcraft 2 is a huge advantage for us because everybody else who wants to build a competitor to Warcraft 1 has to do all of that from scratch.
They have to make the pathfinder, they have to make the level editing tools, they have to make the rendering, they have to make whatever.
And so, you know, that was how things were traditionally done.
No wonder the games industry is so secretive compared to the rest of the software engineering industry.
Like, seriously.
Used to be.
Used to be, at least.
Yeah, maybe now it's changing.
And so there were two really big risks, typically, when you started a game project in those days.
One was the engine risk.
Would we be able to make something that would be technically able to do what we need to do for this game?
And that risk comes in a lot of flavors.
One, will it happen at all?
Two, will it happen fast enough for us to actually reliably build the game on it, right?
One of the things, I mean, I don't know how detailed an answer you're looking for for this question, so stop me if I'm going down too many tangents.
But one of the things you also have to remember is that at that time, and this is sort of still true today, but at that time, it was very important, there was no way to really buy something all that much faster than what you had.
There was not a huge strata of PCs that you could buy or anything like that.
So the rendering engine, there wasn't like a way your level designers could like be playing on a faster thing than the consumer would have.
Really?
You can only have the machine that you have now.
And if this game comes out in a year, that's sort of roughly what the consumers might have or a little bit.
So there are people who started doing things like buying SGI workstations because those were actually faster and up, right?
And things like that is what, you know, you kind of had to do and so on.
So anyway.
So there was a huge engine risk and some games just failed because they couldn't produce a thing that could technologically do what they needed.
You saw houses who survived on technological prowess.
You had id Software that was kind of unrivaled at making those kind of first person engines.
You had Bullfrog who had this engine, those Pseudo 3 engine that they did for like the racing games, Magic Carpet, Dungeon Keeper.
Like they were all based on this, you know, one core tech and all the sorts of things.
There's that engine risk.
That was huge.
And how do you mitigate it?
You didn't.
Right?
Because there wasn't a way to buy one off the shelf.
So you were just kind of gritting your teeth.
The other big risk, and this one is still somewhat true today, but it's just much less because you can start, you could do prototyping early.
The risk is, is the game any good?
Like, what are we building?
Is it interesting?
Is it fun?
And when you think about this problem of we can't even really run the game as it will be because we're just building this engine and we don't even really have a way to test the game super well.
We can't really build much of a final level because we don't have level editing tools yet.
Those are just coming online.
Trying to guess what you are actually going to be shipping in terms of gameplay is incredibly hard.
And there are games, famous games.
I want to say like Thief the Dark Project, a very famous game, looking glass game, was formative in the stealth genre, launched a franchise, which was Thief.
You know, I want to say everything I heard from people on that team was that like the final core gameplay only sort of came together like right at the end.
Right.
And so the game just could have been a lot more, could have been polished a lot more.
But it's just the timing of these things coming together was so hard.
And so it really was an incredibly different thing.
And nobody really had a way around it.
Eventually, there was sort of this push towards something.
Well, it's basically early vertical slice prototyping where as games started getting bigger and people were like, we can't keep doing this, especially if we're going to be putting millions of dollars on.
Like this is not like an option.
Right.
They started to move towards this thing about, look, what we're going to do is we're going to focus the entire studio on building one vertical gameplay slice as fast as we can, as hacky as we can.
Whatever we have to do, do that.
Prove that that is engaging to play.
And then start building out everything else.
Because we simply can't afford to not know.
what that thing is.
And then we can start building like spreadsheets that'll schedule what are the assets we need because now we actually believe in the thing.
We can see it running and it fills in all those details, right?
And that I believe, you know, I'm not a game historian.
So take what I'm saying with a huge grain of salt.
That I believe was a pretty big paradigm shift for the industry when they started going, okay, we got to actually know.
And it became much less seat of the pants after that, if you will.
Now I'm interested in your observation.
I know you're not a game historian, but.
You were in the industry and you still remain connected to it.
What happened when game engines became widespread?
They became not only licensable, like we're talking about like Unreal Engine for larger studios, but ones like Unity or Godot, which now amateurs could also afford.
I mean, amateurs in the sense that you're a college kid or...
or you do some side project, you can already afford the license and you can build stuff because now that risk has gone for clear the studio.
So that risk is eliminated.
And it also, I guess it now opened up so much more people who can now have a shot at creating a game because you no longer have to either have this massive amount to license, this super expensive game engine.
You no longer have that risk.
The only risk is, is it fun?
what have you observed happen in terms of both for the industry, for development, pace, those kind of things?
And the reason I'm asking, because I wonder if there's going to be a parallel with AI where, okay, you know, like you needed to have an engineer who is...
You nailed it.
Right?
I feel games might give us a bit of a hint of what we might expect at the broader industry.
So that is actually, I would say that's a brilliant analysis of the situation.
For not having lived through games and for noticing that, that's...
That's impressive.
I'll say that first.
And I totally agree with that.
I've said to people in the past who have asked about sort of AI impact on games in that sense.
And I've sort of said as much.
I've said like the licensable engine thing kind of was our AI transition already, unfortunately.
And I regret to inform you that the news is not probably that positive.
So there are some...
definite positive things that happen early on because, as you say, it opens up the ability to make games to people who could not have marshaled the technical sort of the sort of technical staff necessary to produce a competitive engine.
And so giving them the ability to make games is a pretty important thing.
And it allows a bunch of people to make sort of some artistic expressions that made they just wouldn't have been able to do.
early on that tends to be a net positive because you just have some more games coming out.
Maybe some of them aren't that good, but some of the existing games aren't that good.
That's not that different.
But then you get some really cool games coming from some sources that just simply wouldn't have been able to do it.
Thumbs up.
Problem is, it rapidly kind of accelerates into this kind of a nasty scenario where you just have massive numbers of releases.
And I think at this point, we're at the point where I want to say Steam games are in the like tens of thousands or a hundred thousand per year or something like that.
It's so massive that there is no way that your game will be organically noticed anymore, pretty much period.
So essentially it's this really nasty problem where you just have the market flooded with products.
And there, you know, it used to be that if you made a quality game, if it was fun, people would find it.
Because there were so few games that someone would play the fun one and tell people about it and it would get purchased, right?
Like word of mouth or just exposure on a storefront would be all you really needed to get, you know, sort of the word out about a game.
You didn't need a huge marketing budget or anything like that.
Fast forward to today where we have this sort of massive influx of games, again, pre-AI.
It's just because now the barrier to entry is very low and you really need a strategy.
to make sure your game gets found.
Is it possible that sometimes, you know, a small indie game with no marketing plan or nothing will get discovered and become a huge hit?
Absolutely.
It does still happen once in a while.
The chances that you will be that game are like zero.
So you kind of now need a marketing strategy, a real marketing strategy, and going into the market for games without one, and expecting to sell any significant number of copies above, you know, maybe a few thousand at best is really unwise.
If you want to hit reasonable numbers of sales of a game, you have to have an idea of how people will find out about this game.
So if I'm getting this right, it sounds like the game itself being good as table stakes, but not enough on its own, right?
That distribution, marketing, getting people to hear about the game is much more of a differentiator because...
There's just too many good games out there, and now they're easier to create.
I think that's exactly right.
And that's just the unfortunate reality of it now.
Was that a good trade?
I don't know.
But that's what happened.
And so that's where we are in the industry now.
There's this other thing that I heard about, which is how new games not only compete with other new games, but with old games as well, right?
Like the other day, I spent a few hours playing Death Rally, which is a game from the 90s.
And every year there's more and more good games to play.
They all take away from the time that the new games have.
Yes, and that problem will only get worse because one of the things that the game industry could rely on in the past that is much harder to rely on now is that older games would look dated technologically in ways that consumers cared about.
And we have now kind of also crossed the threshold where...
There is a segment of the market where people really do care about the latest like ray traced lighting and all these sorts of things and, you know, more photorealistic rendering or whatever it is.
But a large portion of the gaming market by revenue doesn't really care what the game looked like all that much in a sense that whatever we're doing today is good enough.
So 10 years from now, if the games look much better for some reason.
No one will really think of that as a huge differentiator in terms of sales.
You go back to 1995 and technological advances were a huge differentiator in terms of sales.
You come out with something that looks good, that takes advantage of the hardware of that day, and boy, did it look cooler and feel more responsive and all these other things as compared to earlier titles, right?
And so...
That's also going to increase the degree to which the thing that you're talking about will happen.
I can go play an older game because it doesn't feel obviously dated in an audiovisual way.
I don't have to be an appreciator of retro gaming to go play something from 2017.
It just looks fine, probably, right?
So there's that.
The other thing that I'll just mention, which we kind of already touched on, but that ties directly into your point, is that also live service is such a prominent thing now.
People are just logging on and playing Fortnite for several hours or something.
That's also taking away from the possible revenue that might be spent on buying some indie game or some new AAA game even.
So you have these sort of incumbents, people playing Minecraft, spending their time playing Minecraft, spending their time playing League of Legends or Dota, and that's taking up a huge amount of their time.
That's...
It's zero sum, right?
They can only spend their hours in certain places, just like Netflix or anywhere else.
They have to start thinking about, you know, they're competing with everyone else for entertainment hours.
Okay, so I need to ask you this.
GTA 6, how is that in 2026 at a time when we have better tools than we have ever before and we can build software and games faster than before?
Like, how do games take 10 plus years to develop?
Is this some kind of outlier or has AAA game development taking many, many years just not changed at all?
What do you think is going on here?
So from a player's perspective, I can understand why someone would look at it and go, wow, Grand Theft Auto 6 has been in development a long time.
How does that make sense?
Or, you know, something like this.
From a business perspective, you have to understand that Grand Theft Auto 6 is not a game that they are selling to players who are going to play the game.
That's not what it is from a product standpoint, right?
What Grand Theft Auto 6 is from a product standpoint is a replacement of Grand Theft Auto 5.
Grand Theft Auto 5 at the time was, if I'm not mistaken, by far the most revenue generating entertainment product in existence.
The online part of that game was generating like billions of dollars.
And like I said, not a game industry historian.
So, you know, take what I have to say with a huge grain of salt.
But Grand Theft Auto 5 was kind of like Fortnite before Fortnite, if you will.
They were a huge, huge live service revenue generating product.
So from Rockstar or Take Two's perspective, right?
Grand Theft Auto 6 is not just let's try to get out the next Grand Theft Auto as soon as we can because we'll make money selling that title.
It's a...
We are going to replace the most profitable thing we have ever built, which is still generating a ton of money for us, with a new thing.
And you can better believe that they want to make sure that they are going to do that right.
Because the last thing you want to do is ship a new product that cannibalizes something from your old product and then is less revenue generating, right?
So I'm sure that their planning around Grand Theft Auto 6 is not just about trying to produce a Grand Theft Auto that their fans will love and will buy as the original single player gaming experience that it was.
I'm sure they care very deeply about that just from a reputational and from an artistic standpoint.
I'm sure there's a lot of people on those teams who care about that.
But from a business standpoint, I am sure there's also been a tremendous amount of thought and work put into what does the live part look like?
And that's a huge undertaking, you know, that I'm sure that they've been planning for quite some time as well.
So it's a massive, massive thing that they're doing here.
How well will it succeed?
I have no idea, but it is not just a new Grand Theft Auto is, I guess, the way that I would look at it.
Grand Theft Auto 5, I think, was somewhat of a surprise to them.
I don't think they knew it was going to generate that kind of online revenue.
I mean, maybe they had hopes, but I don't think they knew that it would be that kind of a massive moneymaker that it was.
And so this is the first product, really, where they know they will have the audience.
For Red Dead Redemption, they kind of tried to do Red Dead Redemption 2.
They did a similar thing where they tried to have the online thing.
It didn't.
I don't think it hit nearly as big as Grand Theft Auto.
Grand Theft Auto 6 is the first time they're shipping a true update to what is their flagship.
And so, you know, it's equivalent to like a relaunch of Google search or something like that is what they are doing here.
I, you know, I'm sure if I was in charge of that project, I would be sweating bullets.
So I'm sure that they are putting a lot of thought into it.
And it's a very massive undertaking, I'm sure.
I'd like to switch gears to SoftwareCraft.
You made this video titled Clean Code, Horrible Performance, an essay slash video showing how Uncle Bob Martin's polymorphism-based refactoring pattern runs about 1.5 to 15 times lower than a plain table switch version.
Can we talk about the responses to this piece?
Well, I guess I can put that in context.
So that is sort of from that course on the Substack.
So it kind of goes with a bunch of other videos that are part of like the Substack thing.
I guess the first thing I'd say is I feel like the response to it was very positive.
I was kind of surprised.
There are plenty of people who didn't like it.
Don't get me wrong.
It's controversial to be sure.
But I was surprised at just how many people were enthusiastic about it as well.
But what I would say is it's really I.
I don't really think it should be so controversial because there's one thing where people want to just use the term clean, clean code to mean code that they like or think think is written properly.
And that's not something you can argue against.
Right.
Because that's just, you know, I probably have a version of what I think is clean code.
And obviously, I don't think that's bad.
Right.
Like it's it's my idea of what good code looks like.
So if your idea of clean code is just whatever you want it.
you know, whatever you happen to think are good programming practices, I might agree with those program practices, I don't know.
So in this particular video, I was talking specifically about the things that were advocated that are like very specific things that are said, like don't have functions over a particular length or these sorts of things, right?
Things should not know the type at runtime or whatever, right?
There's all these like kind of rules about it, preferring polymorphism always, right?
If you look at those things, they're kind of just...
bad programming practices.
I mean, I don't really know how else to say them.
They don't mesh well when you put them together.
In isolation, some of them might be fine.
So for example, if you really prefer lots of small functions, that's actually fine if the compiler can see all those functions and know that it can safely inline them and collapse them as necessary.
This is a part a lot of people missed about the video, I guess, because it's a pretty short video, so I didn't explain anything in detail, but...
A lot of redundant code happens when you have lots of tiny, you know, little these little tiny functions.
And if they're all virtual functions in C++, let's say the compiler can't know for sure which ones of them are being called and so on.
Even if you put things like final in them and there's all people have a lot of weird beliefs about how the code works.
You can just go do this testing.
When you have lots of these little functions, if they're all like statically defined and aren't virtual calls, if they're just known calls, like or just member functions, when I say static, I kind of mean just known to the translation unit, not external.
The compiler can put those together, collapse all of the redundant code and actually produce something reasonable that will run pretty fast out of that.
It can also do things like widen the code path.
If it needs to vectorize to like run in SIMD and stuff, the compiler has all these options to take what is fundamentally not particularly great code in terms of how you would want it to run at runtime, but it might be able to turn it into that because, you know, compiler optimizing, optimizing compilers are pretty heroic these days and the sorts of things they can do.
If instead you use all of these, you know, things that were recommended, you completely block out the compiler from being able to do those things.
Because if it can't tell what it's doing at runtime, if it has to leave open the possibility that you substitute it in a different class here or something like that.
then you end up in a situation where the compiler can't do any of that work.
And people mistakenly think that this is just because like virtual function calls cost too much or something like that.
That's not what it is.
It's not the cost of the virtual function call.
We could talk about that as a separate thing because you can analyze that cost as well.
It doesn't have much to do with specifically whether it's virtual or not.
It has to do with a lot of things like branch prediction and how much stuff was getting pushed in the stack and whatever else, right?
But...
It's the cost of the compiler not being able to do any optimizations.
That's the actual cost.
And that cost can be severe.
I showed only, I think, a pretty mild degradation compared to what you would actually see in production if you really had a huge number of things doing this.
And I think it landed pretty well.
It's a very widely viewed video and a lot of people seem to really like it.
I thought it would be probably even more controversial than it was.
So I was pleased with that.
But yeah, I mean, all that stuff remains true today, I guess is what I'd say.
And I think it's good for people to hear because they need to hear opposing viewpoints.
I think you can write code that is maintainable and easy to read that doesn't follow those principles in that way and that doesn't have those problems.
I don't think you have to do those things.
So I think it's worth exploring other options that are still maintainable, that are good code, but that allow the compiler to do the right thing.
What is your take on test driven development?
You know, when you write the test first, then you write the business logic.
You've talked a little bit about this as well, because it's a practice that used to be super popular in the, you know, like especially when you're building services, some of those things, especially in the 2000s, kind of got a little bit out of fashion.
And now it's unclear if it'll come back or not with agents or not.
I don't have that much of a spicy take on that one.
My take is very pragmatic, which is that if you can identify tests that will save time in total.
That's usually what I try to emphasize.
In other words, if the amount of time it takes to create and maintain the tests will actually save us total development time because they will identify bugs that would be hard for us to find in production or in.
or would be very costly to get to if they got out, then great.
And I've used them before.
Like I talked about working at Rad Game Tools.
I had a regression tester that I ran on like the core libraries there that I had written for, you know, they're not called libraries, but the core like routines to make sure that, you know, anything that I could be testing for our customers, I sort of was.
And so I think there's good times for testing.
I would say the part that I don't like about test driven development is the test.
driven part.
I don't think development should ever be driven by tests.
I think tests are a thing that you should be aware of.
You should know what your options are for testing and you should make intelligent engineering decisions about tests.
Now, could that decision be that for this particular project, we are going to drive it primarily from the tests?
Yes, that could be a decision that you make, but you shouldn't really.
think of development as something that is primarily test driven, like by default, because that might be a very bad decision for some other project where it just ends up costing you more to have done it that way.
So like with most things, I would advocate for a pragmatic approach to testing.
You should understand the cost of testing, the cost of developing, maintaining the test and the cost to your code base if it makes it harder to change your code base because tests have to be rewritten and you therefore don't make changes you should make.
All of that stuff should be in your brain and you should make an intelligent decision about what your testing strategy is.
If that decision intelligently made turns out to be we are going to have a lot of testing on this project, that may well be a good decision.
I don't think there's an absolute thing you can say about how many tests there should be.
Some projects probably shouldn't have very much.
Maybe some projects should have a lot.
And I think knowing which of those you are doing.
is part of being a good software engineer is I guess what I would say.
You mentioned being a good software engineer.
But before we get into what is a good software engineer, what does good code mean to you specifically?
So good code to me usually means that you have written a something that is as straightforward to what the machine actually needs to do to solve the problem as it can be.
And also hopefully that you have I guess I'll say, properly identified ways of breaking it into easily digestible pieces and named those pieces in ways that are easy for someone to understand, especially yourself, because you are very likely to be someone who's going to have to modify it.
So that's the way I tend to code.
I try to identify what do I actually need the computer to do?
I try to write as simple as possible the thing that will do that.
And then I try to.
put that in terms that are, you know, I would say least redundant.
So, you know, I don't want to see the, you know, the equation for Euclidean distance scattered throughout my code.
I want to have a function that's like compute that distance and I want to use it, right?
I want, I want it to then be nicely broken into the pieces that it represents.
And I want those pieces to be reassemblable properly by the compiler in a way that will produce code that runs very efficiently.
Right.
And so that's what I'm usually trying to do when I'm trying to program.
And For me, I have never really understood the sort of mentality of there's a difference between code that is like well architected by some principles and code that runs quickly.
Because in my experience, usually the code that is architected properly is also the code that runs quickly.
And yes, there is a point where if we decide that something absolutely has to get.
as close to theoretical maximum as it possibly can, yes, we will start to make that code harder to read and modify because we are now like really over-specializing it for this piece of hardware or whatever.
That's true.
But that point is like, you know, way out on the curve.
It's not the common case.
Most of the time, assuming you just want code that runs pretty darn well on most hardware.
The simple, readable version of the code is actually very fast.
It's only once you think you need to have 27 factories and 8,000 microservices and all these things running that it starts to be this thing that's like good architecture, but also like hard to modify, hard to read, run slowly, right?
All these things.
So I tend to think of like good code.
There's like this nice nexus of runs pretty darn well, easy to read, easy to maintain.
as close to theoretical maximum as it could be, but it's close enough.
And the paths to theoretical maximum have not been foreclosed.
We left the door open with the way that we wrote it.
So if someone really needs to come along and boost its performance, it's set up to do that, right?
And related to this, what is a good software engineer to you?
Is it just someone who writes good code or it goes beyond that?
I would say it really depends on the environment a little bit.
because I think I've seen a lot of different kinds of good software engineers.
And so I would liken it more to a, you know, if you want a sports analogy, you'd imagine something more like a baseball team where it's like, what's a good baseball player?
Well, it's like, are we talking about a pitcher or a designated hitter?
Right.
And it changes quite dramatically.
So there might be some things like, hey, if someone's pleasant to work with.
and doesn't goof off all the time and actually gets their work done.
Those are obviously things that we would say are true of any software engineer.
There are some general personality traits that might be positive.
But when you're talking about things that are more specific to just software engineering and not just being a good employee or something like that, I would say I've seen a couple of different kinds.
I've seen people who are like the utility infielder.
There are people who just like...
They can identify and go and try to fix a problem and succeed.
Even if the code base is kind of wacky and out there, they're good at getting the lay of the land very quickly of identifying something that's going on.
And they're not afraid to go in and like, okay, this is kind of, this code base is kind of ugly here.
It's okay.
I'm going to patch around.
I'm going to do what I need to do and get things done.
That's a great engineer to have around.
I've also seen great engineers who are the exact opposite of that.
They are just like, I take...
This one particular problem that we have.
And eight months later, I have ground out every last thing there is to know about this.
And sometimes to the point of like producing new algorithms that no one's even known before, right?
That are like these breakthrough things, right?
And that's a great software engineer to have on a project if you're going to be having that kind of thing.
And so I've seen a lot of different people that I would consider great software engineers and they aren't all the same person.
Right.
So I think that it's kind of important if you're asking it from the standpoint of like, hey, you need to put together a team to go build this project.
What's a great software engineer?
I would say the best advice you could give someone in that position is think about the roles.
Think about what kinds of roles there are going to be here.
And don't think great software engineer.
Think great.
that role, right?
Who is going to be a great pitcher?
Who's going to be a great first baseman?
Who's going to be a great outfielder?
Who's going to be a great this, that, the other thing?
Great third base coach, whatever it is, right?
And that's what you're trying to put together if you're trying to build a team, to me.
Yeah, so like, it's just not one size fits all.
But I still want to push you a little bit.
Like, what are things that you think are non-negotiable for someone to be a great software engineer?
I mean, we talked about the things that we talked about, which is a recurring theme.
with you is just going deeper and deeper and understanding the next and next layer you know like understand if if you're doing web development understand react once you understand react understand what's going on in the dom go all the way to assembly once you've done there understand how the cpu is doing operations and branch predictions and some of those things like to me that's a skill of like curiosity driving deeper crafts whatever you call it uh there are different ways we could do it but Along these lines, what are those traits that you think, no matter what kind of role we're talking about, but if you think back of some of the different types of roles that you work with, do you see some overlap that they all had something?
I would say that it's pretty unusual, I guess, that I can't think of someone I would think of as a great software engineer who like...
didn't know how to like read assembly or something.
That is true.
It might be that having that curiosity about how things work and knowing at some level what's going on is kind of maybe something that's going to be very common to a great software engineer.
But I would just underscore the point, the degree to which they are employing that knowledge may vary quite a bit.
For some of them, that may be their bread and butter and they're doing that all day.
For others, it's just really a thing where Because they know how a computer works, they're not making those stupid architectural decisions that come back to bite us later, right?
And that's great, but they may not really be doing all that much actually at that kind of level or thinking about at that level.
They're just going like, yeah, I know we got to kind of push.
Okay, this stuff's going to have to be done in batch because I just kind of know that that's how the machine's going to have to handle it.
So I'll make sure I write the code that way or whatever.
Yes, but so there's a little bit of that.
The other thing that I would say maybe.
is actually like not being dogmatic about things that they haven't actually themselves proved out is probably a huge one.
I find there's a lot of like received programming wisdom that's just nonsense.
Like clearly no one's ever tested it.
And if they did, they would have found out that there's no actual basis for it.
It doesn't necessarily mean it's false.
It's just, there's no like, there's no actual tangible way you can demonstrate.
And sometimes it is like you could demonstrate that there are actual.
concrete downsides to this received wisdom, right?
And so in order for it to be received wisdom, you should have to be able to at least demonstrate concrete upsides, which oftentimes cannot be done.
So I would say people who actually focus on what works in practice is a huge plus.
And you can apply that anywhere and to anything, right?
Not just saying, oh, the flavor of the month is that we're writing everything with classes and virtual functions and hierarchies or whatever.
It's like, Did you actually determine that that results in less code or that the code actually is more like that?
We do any testing to figure out if this is helping us rather than hurting us.
And the answer oftentimes is no.
Or if if it was at all, it was extremely shoddily done and you would not take those results as conclusive in any way.
And so it's like being more skeptical about coding practices and actually trying to focus on what is working in practice and what we can demonstrate and measure in some kind of a repeatable way.
is I think a really great thing for a software engineer to have as well.
So people who don't tend to fall prey to that, just like I watched some presentation and someone at Google says always call Memset or never use if statements or whatever it is.
Like if that's the level that you're thinking at, then I probably am not going to put you in that category of really good software engineer because that's not how it works.
Plus, it's not that hard to try these things out or set up or run an experiment.
Now, the final topic I want to touch on, which I deliberately didn't get into until now, is AI and how it's changing your work.
And I'd like to start with that.
Like in the work that you're doing at Molly Rocket with this project that is yet unreleased, how are you using AI tools if you're using them at all?
We are not using them at all.
So you're doing it just like before.
You're writing your code.
What made you decide to?
to take this path?
Well, we're a little bit different, obviously, in the sense for two reasons.
One is that we, like I said, we are kind of, we have like sort of two projects here and the substack is our primary focus.
And this other one is a thing that we're doing because we want to do it.
And when you think about that perspective, it's like, well, why did you want to do it?
Well, the reason that I want to program like things in a game is because I want to program them.
If I just wanted an AI to program them, I probably, you know, first of all, we'd probably just go use a licensed engine, right?
Like I wouldn't even bother asking AI to do it.
I'd just go get the Unreal Engine.
Get the engine, yeah.
Right?
Or something like that and so on.
So I think a little bit of that decision is probably not that relevant to your audience because it's more about what do you want to do?
Like why are you spending this time, right?
It's a philosophical question, not a productivity question.
So it's not like I evaluated it and said, I don't think this will save us time or I have questions about the copyrightability of it or the ethics of it or all the sorts of things that you could rightfully evaluate AI on.
It wasn't necessarily that.
It's more just like this does not further the goals of the project to use it.
So it kind of was a non-issue at that point, right?
Stepping out a little bit more to a broader philosophical framework about AI, I guess what I would also say is I think that if you, regardless of what you think will happen with AI in the future, because obviously we don't really have any way to predict what it will look like 10 years from now, it's anyone's guess, really.
I think there will probably also be at some point a notion of like traditional hand crafting that will come into play because we've seen this in most other times when you automate something.
So.
If you automate making furniture and you have like Ikea or whatever, that doesn't mean that there isn't some weird guy down in the industrial district of your city making crazy wood tables with iron and welding and something.
And that that's just a thing that people are still doing.
And some people want that table.
I don't necessarily have an explanation for it.
And I'm not trying to argue that it has more or less value, but it's just something that happens, right?
And so if I imagine what I love about computers and what I want to do with computers, and you asked me, move that into some other context, which of these people would you be?
My answer is always, I'd be the organic farming guy.
I'd be the guy who's making the weird table in the industrial district.
I have no interest in managing a division at IKEA.
I literally couldn't care less about that, right?
And so I think for me, another reason why I'm not that interested in pursuing AI is because I would like to be part of whatever the set of people are who are going to keep this traditional craft alive just because that's something humans do, not because we're trying to say that that's the right business case, right?
If that makes sense.
And at this point, there is a bit of a tradition, if you will, even if we assume that these machines will do as good or better than.
humans it for like 60 plus years.
We've we've all only exclusively handwritten software because that's how it got done.
Right.
Like a lot of us, anyone who started coding before 2023 or the end of 2022 or probably honestly 2024 when these things have gotten like decently good.
You just wrote it by hand.
A lot of it or tap complete still counts.
Yeah.
And and I guess I would say like, again, it's just, you know, why why do that?
Right.
Is the question.
It's like, I don't know why humans do that.
humans do that because it's something humans do right humans like to do things themselves sometimes you know people can buy a hat they can buy a wool hat trivially or they can buy whatever.
And then someone's out there knitting a hat right now.
That's just, it's just something humans do.
They like to make things by hand sometimes and at varying levels of handmade-ness.
You know, there's some people just buy the wool or whatever, or buy the pre-made yard.
Some people raise the sheep or whatever and shear it, right?
Like you can go arbitrarily for it and you can find somebody.
who's going all the way, right?
Even further than probably I would ever go if I was in that thing.
So, you know, you could imagine someone making their own hardware these days, right?
I'm not doing that.
And so that's kind of my take on it.
So I'm kind of the last one to ask about, you know, AI coding or what you might want to do with it.
I really have nothing of value to add.
Yeah, but I am interested in asking you through the lens of the games industry.
And we touched on...
games engines arriving and now so many more people can make games, not everyone, but it's a lot easier to enter.
What are you observing in terms of most people outside of who are still handcrafting code because they want to are using these AI coding agents for two reasons.
Either it just makes sense and they realize, well, this thing can now generate code as good as I did, which was a turning point in January.
I had the turning point actually myself.
Some are actually just pushed with corporate mandates of like, you need to use these tools.
And eventually they kind of get on board, whether willingly or unwillingly.
But so many folks are having AI write the code for them.
They're prompting it, but they're doing it.
What do you observe of the effect having from your vantage point?
May that be on quality, craftsmanship, on just output, speed, etc.
What are you seeing?
I think it's a little too early to assess, to be honest, because kind of as you pointed out, obviously there's been people who maybe, you know, we might derogatorily call AI shills who have been saying that it was producing as good a code as humans for, you know, two years now or something like that.
Right.
Yeah.
But in reality, the people whose opinion I would trust more.
None of them thought it was really all that usable until much more recently.
And so we really haven't, they haven't had very many months to actually be figuring out how to use this thing or to determine to what extent they can use it and how, what it's best at, what the workflow looks like that makes it produce the best results.
It seems like at the moment, I would say probably need to give it at least another six months, if not another year or something to let everyone kind of shake out like, What are actually the best ways to use this thing?
I know tons of people in the game industry are using it.
So I know that they are doing various things with it.
Whether those things are the same sorts of things they will eventually think are the way they like, you know, like the things that they're doing right now may be like, oh, that was kind of dumb.
Like you shouldn't have used it that way.
You should do this other thing with it and it's way more productive or something.
So I feel like it's probably too early to assess.
We haven't seen any real like obvious like, oh, wow, like, you know, the Fortnite ships once a week now and it's bug free.
Like nothing particularly interesting has happened in terms of output there.
But again.
It's been what, like five months or something.
So it's just it's way it's way too too early to see how it actually gets integrated into a reliable process.
Right.
Yeah.
And there are I know there are some companies who are now tying up, let's say, agents fixing bugs, but that's only a few months old.
The oldest software that's widespread that is written.
close to 100% by agents is from the labs, OpenAI's codecs and Entropics' cloud code.
But even there, it's been since November or some parts of it, December.
So like maybe six months.
And it's different, right?
That is a product they're selling.
So there's, I'm not sure we'll know for sure.
Like, is it truly 100%?
How much, you know, there's a marketing angle or not, but there's a self-bias there.
So like.
I would put those aside in terms of trustworthiness.
And you're right that the rest, we just don't really have the information.
It'll be, I'm sure there's so much experimentation, but to your point, it takes time to bake, right?
To see the impact.
Most of these things are currently presented as tools, meaning a human has to operate them at least in some way, like at least setting it up to do what it's going to do.
And therefore, you have to give it some time You know, nobody currently is selling a product where it's just like, oh, just turn this thing on and it will just ship Fortnite by itself forever.
And you can just get rid of all your engineers.
Like no one's actually selling that product yet, right?
We could evaluate that product because we'd be like, did anyone do it?
Did it start shipping Fortnite on its own, right?
So if it's still something where humans have to kind of figure out how they want to like slot it into what they're doing, then it's entirely possible that the reason that we haven't seen some big uptick in productivity that would be obvious to an external observer is because it's going to take a while for people to like shake that out.
Or maybe the AIs need to get a little bit better.
Maybe like we've got to go through some more update steps or, you know, whatever.
I'm not sure.
So there's all that on the table.
Then there's another possibility, which is that it actually already has worked, but just the productivity boost isn't as big as would be obvious.
If people got 10% more productive, that would still be pretty.
impressive because it's hard to get a 10% across the board uplift.
I've said this before on podcasts.
I'm like, if you have a tool that can give everyone 10% up, that's great.
Almost no one would know, right?
It's like you can't, it's not really externally observable that clearly if that's what you got, but it may have happened, right?
So it's really hard for all of those reasons.
At some point, if the AIs are really fantastic and people figure out how to use them really well.
It should be obvious.
It should be like five people are now shipping four at night instead of 5,000 or whatever, right?
But until that point, it's really hard to know because it's just like, especially if it was small, it'd be hard for us to see.
Well, this is anecdotal, but I'm getting a lot of data points and messages from software engineers and managers.
One impact it's having is there's this kind of like AI fatigue slash burnout from software developers who are like, look.
I am good at coding.
I've always been good at it.
I enjoyed the work to various extents.
But since this AI thing happened since the end of the year, beginning of the year, since it's actually I'm now prompting and now all my code is generated, whether that's corporate mandates or is just faster, I'm starting to lose my drive.
Like, why am I here?
Like, anyone could do this.
And I think there's a sense of like, I'm using a lot less of.
what I'm capable of.
There's all this pressure from above to be more productive with it.
And it's, I think we should, like, I'm seeing more and more signs that it's, what do we call it?
Burnout, AI fatigue, et cetera, loss of motivation.
I haven't seen a technology, I don't remember technology having this widespread impact, like everywhere.
I'm hearing from folks at some of the leading, like kind of not AI companies per se, but like, you know, big enough, like database providers who are now.
hugely into ai and they're powering a lot of the things traditional companies modern companies everywhere have you observed some of this thing and would you have any any advice or any pointers to folks who are feeling like this right now i guess i would say observed no uh heard about yes uh i guess is what i would say like i have talked to people who have been like such and such has been having a really hard time with this or such and such has been having like there's there I've definitely heard that interacted directly with someone not currently know and part of that is is probably largely because most of the people I talk to have a fair amount of latitude with what they do and how they do it a lot of the people that I talk to on a daily basis are able to make their own decisions about what they want to do with AI and so on.
And so I don't necessarily hear from as many people who are going to be in a position where some manager told them, this is just what you have to do.
This is very interesting because one thing that keeps coming back and Armin Ronitur was telling me the same thing on the podcast is he's observed that autonomy, like at your work, how autonomous you are at your work, like how many decisions you can make on what you work on, how you do your work.
The people who have a lot of that are typically like, oh, great, I can use this for this.
I can use this tool.
But the people who are told, you know, like beforehand you're given a ticket or the PM tells you this, they don't have much wiggle room.
And now those folks are seeing it way more as a threat because, of course, subconsciously or consciously, they're thinking, well, this thing could automate my job.
It's now or it made it made from that little effort I had to do that.
It took it away as well.
So.
I wonder if there's a connection here.
I mean, that sounds totally logical, right?
If you're somebody with a high degree of autonomy, then when are you going to reach for an AI?
Well, whenever there's something that you didn't want to do, right?
So kind of by definition, I think at that point, you're going to have a much more positive experience with it because worst cases, it just doesn't work.
In which case, I guess that's not great.
You're going to be like, ah, this thing was kind of crappy.
But assuming that it's able to accelerate some part of that, that was great.
It's like, hey, I didn't want to do this thing already.
I had this AI do it for me.
And now I have the thing.
That's just a positive experience for them, right?
Whereas, yeah, if you're just told like you had this thing that you wanted to do and you were told you can't just do it yourself, you have to do it with the AI, you know, and by the way, we just had layoffs or whatever, you know, a lot of that stuff obviously could totally change your mental reaction thing because now it's not you deciding to use an AI because there's something you didn't want to do that you thought the AI could do for you.
Now it's you just being told that you're supposed to be using this AI to automate whatever your job used to be.
You can see pretty obviously why that would have different psychological effects on people, right?
So I guess it might be just an idea for folks in this situation that now you might want to evaluate your current position or if you're interviewing your next position based on how much autonomy will you have?
Because the more autonomy you'll have, the more likely you're going to have control over how you're using this stuff.
how much you can experiment versus being given a mandate that I don't know, we're expecting you to have this output increase or output change, whatever that is.
I wonder if this will reevaluate some of, you know, like what is considered an attractive position.
Because like, for example, big tech was considered a great place to work because high compensation, pretty clear expectations, like easy to understand career advancement.
But now they're the ones who are starting to measure.
Your AI usage, which is gonna like giving a kind of a bit of a handcuff of what we're expecting you to do or There's where you might have There might be mass layoffs which again you have no control over right like it's getting one more or or inside of metaphors reassignments of like you will now do labeling for X months I mean you could sort of think of you know, could we coin the phrase?
Are you using an AI to do your job or is an AI using you to do to do your job?
Right.
Like because at some point it definitely it definitely felt like meta, for example, from from your reports on it.
And I have seen the same thing said by other people.
So it does not sound like a one source kind of thing.
It sounds like this was kind of just accepted as fact that they kind of just were using you as AI training.
Right.
Like that's what you were kind of, you know, you're just there to train the eye to do it so that we don't need you.
anymore.
Right.
And so thinking about that from a, from a perspective of choosing your job, it does make some sense if you do, if you have any latitude, right.
But yeah.
As closing, what are one or two books that you would recommend that had an impact on you?
I'm going to have a hot take here if I, if I might, because it's sort of a, it's sort of a push I've been on recently.
I don't think people should necessarily take a book recommendation from me.
I want to recommend that people read a paper.
I'm trying to get more people to just read papers because I realized I read a ton of papers.
Like I am constantly reading papers on things that I am interested in.
Like if I'm going to go do some programming in an area that I haven't done before, I will read a ton of papers.
I'll crawl the references on papers.
I'll read a survey and go gather all those references and read those references and crawl them back.
And I find that I learn a ton that way.
And I feel like a lot of programmers just don't do that.
And so my recommendation would, you don't have to read a specific paper.
I'm not going to give you a specific paper to read this one.
Just think about the domain you're programming in.
Do a search on Google Scholar for some part of that that you're interested in.
Try reading a paper, following the references, see what you think.
I think it's a great thing to do.
And I get a tremendous amount of...
not just enjoyment from the education of it, but also just like more knowledge about what I'm doing pretty much every time I do this, even if it's just to learn a little bit more about the historical record of how things got discovered.
But a lot of times it's just like I learned about whole new techniques I just was not aware of because there's way too much out there for any one person to know.
And I don't know to what I, again, since I don't currently use AI in my workflow, I couldn't say, but my assumption would be that AIs would also be very good at helping you find some papers to read if you were interested as well, because that's, you know, chewing through a lot of the technical record is something that they do.
And so maybe you could even ask your favorite AI to suggest a paper that you might like based on some things that you tell it.
I don't know if they're good at that, but I'm guessing that's something they could do.
Casey, thanks a bunch for this conversation.
This was great.
Thanks so much for having me.
It's been a pleasure.
I've been wanting to talk about performance with Casey for such a long time, and I'm glad that we finally made it happen.
I kind of wish the industry had more people as excited and interested in high-performance code as Casey is.
If you made it to the end of this episode, you might just be one of them.
I appreciate that Casey did not beat around the bush.
If you care about performance, you want to be able to read assembly and spend some time reading it.
Reading assembly is several times easier than writing it.
If you can read assembly, you can see what's happening at the machine level.
And it's a lot easier to understand, for example, why a programming language like Python is much slower than something like Rust or C when you see the assembly code for simple operations.
I was chuckling when Casey talked about these blog posts about how we rewrote our services in a new language and got 10x performance improvement and how those rewrites are usually not about the new language with fixing the architecture that caused the performance issues to start with.
And although we did not talk much about AI, I found it amusing for Casey to say that the games industry had its AI moment years ago, when game engines became accessible to pretty much anyone wanting to build a game.
Before, large teams were needed to build both a game engine and a game, and now teams of one or two can create full-blown games.
After a brief spike of positive effects, with lots of new, good games released, games have flooded the market in such great numbers that it's now impossible for a new game to become hit organically.
so marketing and distribution becomes mandatory even for great games.
For more deep dives related to game development and performance software, check out the link to Pragmatic Engineer Deep Dives on these topics.
If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube.
And a big thank you if you also leave a rating on the show.
Appreciate it, and see you in the next one.