# Resilient Systems, Microservices, and AI Strategy

**Podcast:** The Pragmatic Engineer Podcast
**Published:** 2026-10-07

## Transcript

Sam Newman is the author who wrote probably the most widely read book on microservices.
And yet, as you heard, he said that microservices are an architecture of the last resort.
But why?
Sam was in the room when the term microservices was coined.
I find him to be one of the most thoughtful engineers on this topic, as well as on resilient systems.
And his latest book, Building Resilient Distributed Systems, is out now.
Today we cover specs versus code as a source of truth and why Sam says that production is truth.
idempotency keys versus fingerprints, and why you should avoid idempotency keys when possible.
The four dimensions of resilience, robustness, rebound, graceful extensibility, and sustained adaptability.
Cognitive depth and cognitive surrender, and Sam's advice on how to better use AI to avoid these, and many more.
If you want to understand what it takes to build resilient systems, from an expert who has been knee-deep in this topic for 10 plus years, this episode is for you.
This episode is presented by TurboBuffer, the AI retrieval layer built on object storage.
It's fast, cheap, and extremely scalable.
Antropic, Cognition, Notion, and many others are happy customers.
Before we start, let me mention Tire, our seasoned sponsor and our product and team I'm very bullish on.
You probably already know that Git hosting has a problem.
AI agents.
Git is increasingly becoming a bottleneck for modern agent-heavy software development.
Engines like you or me are creating more code with agents.
Those agents push more code.
And most of us are running parallel agents, which push way more code.
GitHub is clearly struggling to keep up and has frequent outages.
So what's the solution?
Entire was founded by GitHub's last CEO, Tomas Dumbka, who by the way was a guest on a podcast when he was still at GitHub.
And Tomas revealed Git hosting for the AgendaC era from scratch.
Entire was built to be very fast and to have your repos regionally close to you to reduce latency.
And this allows for fleets of agents to push in parallel.
And Entire's performance is next level.
The repo can handle 418 pushes per second, which is up to 89 times faster than every competitor on the market.
When GitHub is down, you can still keep working and you don't even need to migrate away from GitHub.
You just sign up to Entire and the platform mirrors your repo.
And here's a neat thing.
Have you ever wondered what prompt resulted in this specific code being generated?
I find that the prompt and conversation with the agent carries more information than the PR itself oftentimes.
Entire captures all the prompt history with your agent right inside of the repo, easy to check back.
And it's got a pretty innovative UI to show all of this.
If you're looking for Git hosting that works even when GitHub is down, head to entire.io slash pragmatic, install the CLI and mirror your repo with a click.
I've already done it and I love it.
Oh, and did I mention that it works with any agent and it's open source?
Sam, it's great to have you on the podcast.
Thank you so much for having me.
Happy to be here.
Really happy to be here.
Big fan of the show, as they say.
I'm not going to lie, I've become a bit of a millennial and I think I've watched a lot of shorts rather than watching the whole things now and becoming my son.
Yeah, a millennial is a new boomer apparently I've learned.
I was happy enough being Gen Y, now I'm wherever I am now.
Well, so let's go back to the beginning.
You've been known for a lot of your books, your takes on resilience system, microservices.
Before that you worked at ThoughtWorks, but let's go into the very beginning.
How did you get interested in tech and computers?
I'm quite fortunate, living in the UK.
We had at places an access to computers quite early on.
We had an Acorn Electron at home, which was sort of one of the generations of the BBC computers that came after the BBC B.
And then I got a Spectrum 128 3, which is the Spectrum that had the built-in tape drive.
This is way before the Commodores.
I think some of this was the BBC.
I think I'm pretty sure that the Acorn Electron was pre-Commodore, but the Spectrum would have been...
Around the same time.
The UK was a bit more of a spectrum and BBC market, whereas the US, the Commodores and the Ataris and things were a bit bigger.
So that sort of got me a bit interested.
And then I always sort of stayed interested in it.
Dad had a computer.
I did the very boring thing.
I went to uni.
I did a software engineering degree at a place called De Montfort University, which the running joke in my family is it's in the top five universities in the Leicester area.
There are only two universities in Leicester.
I mean, it was, I loved it.
Actually, it was a great class.
It's basically my T syllabus.
And then I just did work after.
I got a great, back then we did these things called sandwich courses.
Sandwich course.
Yeah.
And so basically the idea was you would do two years of study.
You do a year in industry that was paid, but you effectively going in as a student.
And then you did your final year.
Oh, nice.
Which is actually fantastic because I kind of was drifting a bit.
And when I did my year in industry, so many things clicked.
It meant there was much focus going in.
And I got to do work for a company called, back then they were called GC Alstom.
They were called Alstom.
And we did software for the European Space Agency and gas and steam turbines.
Yeah, it was really cool.
I joined about the first time at that, and then I went back after.
And when I went back, it was just around the time of the first dot-com boom.
And so everybody left.
So I went from being the person that was doing like nothing to the person who did everything.
I remember my first week there, I was given a Fortran 77 code base.
So Fortran 77 was released in the year I was born.
And every single for loop had a break condition because we were basically doing analysis on these Unix machines.
If the models and the time on the machines was expensive and we were worried about accidental infinite loops, this is us trying to solve the halting problem basically.
So every for loop crosses 8,000 files.
had a break condition of, I think it was 99,999 loops.
And if it went over that, the program basically broke and aborted.
The models had got bigger.
The computer got more powerful.
So it had to go through and change them more.
By hand back then?
By hand.
Well, this is one of the nicest things that anyone's ever done to me in a professional career because Fortran 77 is based on the old punch card limitations to an extent.
So your lines can't go very long.
So it's all a bit fiddly.
You can't just go and put a nine.
And my boss said, I reckon it would take about two weeks if you do it by hand.
Or, and he slid a book across to me, you can read this.
And he gave me a copy of the O'Reilly Set and Orc book.
And so I spent two days learning how to do regular expressions.
And I wrote a refactoring using regular expressions, accounting for line lengths and everything else.
And it's, I think, the closest I've ever come to feeling like a god.
I did it and I thought, oh, that's the thrill.
That's the absolute truth.
I'm not sure if it's a joke or a story, but, you know, the two builders who need to build a wall, and one of them starts to build the wall one by one by one, and the other one is kind of drawing, making plans, and, like, I don't know, like, it takes three days to build the wall.
And the first two days, this guy does nothing, and the builder building it feels really good.
And then on day three, this builder, like, builds a machine that builds, or, like, puts together a machine that builds the wall for them, and then...
Yeah, but sometimes, sometimes, of course, the wall falls down the moment you said it was 15.
The thing is, I love that.
That story is all true.
It's one of the earliest times I can remember learning something and having such a visceral reaction.
And I remember, just towards one of the end of the thought works, I had some grads in, and they were asked to tell some stories about what had happened in my previous career.
And I told this story.
I said, I added the nine, made the models bigger.
And then one of the grads said, why didn't you just...
replace it all with a constant so you could just change the number once any time in the future.
And I was like, I thought, I'm learning once again.
Yeah.
So that's also why I love it.
It's like you get humbled all the time in this industry.
You get to learn stuff.
Yes, there's reinvention of the wheel going on, but we're inventing better wheels the whole time.
And I think it's just nice to remind us every now and then I'm sure everyone who's built.
code and programs, like at some point you have the realization that you do feel like a god, that you can do things with these things, which is just so difficult to do by hand, right?
Like there's, I think we forget about that when it's kind of day to day and we're just like doing our stuff or even like right now we're using LLMs, which also do incredible magical stuff that would have been impossible.
But like, it's kind of cool that we are in charge.
We can do amazing stuff.
You get that little fizzy dopamine here.
And I think that's something we're struggling with a bit now is as people are getting further away from the code, they're trying to work out like, how do I still get that kind of dopamine in the same way?
It's not as much an issue for me because later on, you know, I've been doing this for almost 30 years now and I've already been mostly working at the area of architecture and sort of architecture and infrastructure for a while.
So I feel a little bit more removed, but it's, yeah, talking to my son who's a programmer, he's got into it around an interesting time right now.
And then you eventually made your way to ThoughtWorks where you spent a bunch of time.
What kinds of things did you do at ThoughtWorks and what are things that you learned there?
I did anything.
I mean, I joined when there were probably just under 400 people left with about 4,000.
And I worked and got to go all over the place.
I joined at a good time because a lot of the...
I mean, we're very much an XP shop.
And I think that the core of the business still kind of primarily is when I speak to people there.
By the way, XP, like, was that influenced by Kent Beck's XP?
Yeah, no, that extreme programming is very much, it's Kent's extreme programming.
There was a bit of a London flavor.
So the kind of London School of Test-Driven Development and the Extreme Tuesday Club, XTC, around here.
So what's the London flavour of XP and TV?
It's sort of like, if you want to know, you probably need to read Growing Object-Oriented Software Guided by Tests.
One of the worst titled books in existence, written by a couple of friends of mine, Steve Freeman and Nat Price.
And I think that was a lot.
There's a lot of people in the community around here that knew that.
There's a rich meet-up scene.
And so I kind of got introduced some people through there and came in.
I was, same time I was offered a job by somebody else.
It was a bank.
The bank could tell me exactly what I was going to be doing.
And ThoughtWorks, basically, the recruiter at the time is now one of my very good friends, one of my oldest friends now.
She said to me, I don't really know what you'll be doing.
And I'm like, that sounds like the job for me.
I went for that one.
And this is at a time when, like, you know, we talk about continuous integration, but the tooling wasn't there.
And talks about TDD and the tooling wasn't there.
It was in, you know, it was around that sort of time that we started work.
I started building the built pipeline stuff in about 2004.
And then I worked, Dave Farley was my first tech lead.
Dave Farley?
Yeah, yeah.
He's my first tech lead at ThoughtWorks.
I was one of the people that interviewed Jez Humble to backfill for me on a project.
No way.
A project at AOL of all places.
And we'd use the backfill.
So I was like, I'm sort of half his sysadmin, half dev really.
So I see myself.
So we needed someone to take over my job.
And so I interviewed Jez.
And so I was around when a lot of interesting things were happening.
So I was fortunate enough to be there while we were sort of kicking those ideas around.
And obviously people like Dave and Jez put those ideas together really, really well.
You're sort of surrounded by everyone talking and thinking about these ideas.
I think you got that.
I got the idea very early on that this stuff's all up for grabs.
These things aren't set in stone that we can push.
And when you're saying these things, like you mentioned continuous delivery, you mentioned what other?
Test automation, build pipeline optimization, infrastructure automation.
I went over to the US in 2007, actually to work in a big, I've actually worked at Google for like a year and a half as a ThoughtWorks employee.
Wow, that's quite rare.
We were there doing a project.
This has all been talked about publicly.
It's a great write-up by Mike Bland all about our experiences.
It was me, a few of my Thoris colleagues, some people from Industrial Logic and some people from Pivotal, the old Pivotal, pre-acquisition.
They were known for their really excellent engineering culture.
Back when it was just Rubby's company or anything else.
And we were there to teach Googlers primarily how to write automated tests and how to write code that was testable.
And so I got out there and I was talking to people out in the Valley and stuff.
It's like, we were going into these enterprise companies doing test automation, using stuff like Selenium.
And you go around talking to people and it's like, people didn't, I remember having conversations with people at like Yahoo and even big chunks of Google who did not think you could do functional automated testing of like websites and stuff.
I guess because they just never did it, right?
Yeah, absolutely.
And for me, like, I hate it whenever I haven't pushed myself or like, found a better way of doing something.
And I was lucky enough that early days in ThoughtWorks, that was just a constant churn of ideas and reinvention.
I mean, goodness knows how many mocking frameworks and inversion of control things I was.
I built two different replacements for cruise control myself, right?
And so, but like, you know, it was at that time I created a thing called DB Deploy.
My friends Nick Ashley and Graham Tackley, which was like the first database refactoring tool and all these sorts of things.
It didn't exist.
We needed them.
And it was like, if you want to do it, just do it.
That was their mindset.
And so I think there was always this idea working at ThoughtWorks was always like, what's next?
And I was quite fortunate that because ThoughtWorks was growing so much, there was always things needed to be done.
And my default was normally just to say yes to everything.
So I got to travel all over the world.
It was fantastic.
Met my wife at ThoughtWorks.
Most of my best friends will come from ThoughtWorks now still to stay.
So although it's not been there for a long time, it was kind of really formative.
I think, in terms of that.
And it is really interesting because ThoughtWorks has an alumni pipeline, people who used to work there, who are now really well-known in the industry for their ideas, for thinking differently.
Some of them educate people.
Others have created businesses to build software in better ways in certain dimensions.
And it's just interesting because usually I wouldn't think of...
I'll just like put it in a box.
Like, you know, you have product companies like Google, like Meta, like they have, they build their own products and you have like different tiers of these companies.
Some can afford the best people and they have an excellent engineering culture.
Again, Google was known for this.
Meta to some extent.
And there's some companies maybe who are like lower on tier.
And then you have the so-called consultancies who are, you know, they don't build their own stuff or they don't do too much of it.
They come in and they help companies build or you have a problem when you hire a group of experts.
And ThoughtWorks technically is in this group.
And usually, you would not think that there's too much innovation going on in these places.
But with ThoughtWorks proves to me, yourself and other people, it's actually, you can have, like, when you are the best in what you do globally, you can actually have a lot of innovation happening there as well, right?
Yeah, and it was innovation with a different set of constraints.
I mean, I would say there aren't that many people that have come from ThoughtWorks that were really great startup product people.
I mean, a lot of...
You know, a lot of thought workers make X and scale up people once you're all together.
There are actually very few real entrepreneurs that came out of the ThoughtWorks machine.
I know a few of them.
There aren't many of them.
But I think there was always the opportunity to innovate even within a constrained enterprise environment.
So much of our working conditions as a consultancy, who mostly did co-source delivery at that time, we were doing kind of co-source delivery, a bit of enablement through delivery.
Your working conditions are often based on what the client does.
And unless you're there explicitly to change how the client builds software, so what you worked, what you got.
But that was self-perpetuating because we were already used to sort of challenging and pushing and trying to do new things.
That spirit carried on.
And so I think we grew quite aggressively, but not that aggressively.
Rotating people through and that spirit kind of carried on.
I mean, look, we had to start most engagements with convincing people why testing was a good, why pairing was a good idea, let alone TDD.
Yeah, and I imagine how the conversation went, like a manager going like, well, what would you pair?
Then I have like, you know, like eight engineers and now you're making it four, which sounds like a terrible thing to do.
Which, and this is, I mean, I had good fortune because of the people, because of the work at ThoughtWorks.
I got to know Martin Fowler quite well.
And, you know, he supported me a lot when I wrote my first book.
I still remember when he said, well, programming is not typing.
I said fundamentally it made it click for me was having those conversations with clients.
If the client thinks that programming is typing and that's why they don't want you to pair program, well then you've got a fundamental problem that needs to be dealt with and it's not a conversation about pairing.
The conversation is what do you think programming actually is?
Well, but let's bring this a little bit forward because right now we're having the same conversation, right?
For a lot of CEOs, even founders are thinking who have been building tech companies, you know, they all look at what is the most time-consuming part of what we do in building software.
50% of the time has historically been writing the code.
So a lot of people think, you know, like programming is code and LMs are really good at churning out code.
So what is programming?
Okay, there's programming, I think it's Peter Knorr's article, the guy who came up with Barker's Knorr form, well, he's Knorr in Barker's Knorr form.
He wrote his paper years ago.
we should run the Turing Award for.
And he sort of describes the program as being a theory that lives in the mind of the developers.
That it's actually us together, working together collectively, who builds a program.
The fact that we happen to make that program work through code, the code is really a side effect of how we work together.
And I think product development is just that extended, right?
What we are building is collectively.
We are building a system together.
And the code is one of the mechanisms we use to deliver the system.
It's not the only thing.
So I think holistically, that's what it's all about.
It's just that we focus on the code because that's historically been where we sort of spend the time.
But I mean, this is also a classic theory of constraint stuff, though, right?
If you speed one bit up, you get constraints elsewhere.
And it exposes friction points elsewhere.
I would also argue that a lot of the time coding was not.
problem or not the bottleneck it was how the coding was done delayed feedback cycles and things but i do come back to that peter law piece again because it's like margaret ann story referenced it recently in her term pay for uncalculative debt but it's like building software is a collective it's a collective endeavor it's people working together who build software uh kent put it in his tiny first book that software design is an exercise in human relationships right ai doesn't change that And I think all that's really happened over, say, the last 15 years and my experience is that we've got a bit better at understanding that we need better working relationships, like people at product owners, for example.
But that's what the programming is.
The art of creating, it's the collective art of creating software.
And it's not just developers that do that.
Never has been.
And so after ThoughtWorks, you've gone independent, right?
I did three startups.
None of them worked out for different reasons.
So don't ask me for advice on how to make a successful startup.
And then I went independent.
Yeah, I moved back.
I was living in Australia at the time and moved back from Australia.
And yeah, I've been independent ever since.
So doing kind of a mix of stuff really ever since.
I'd written the first book about microservices in my spare time at ThoughtWorks.
And then that book came out and I'd been at ThoughtWorks a long time.
And I realized that there's always a danger of becoming institutionalized.
You get to a point where the company is like, I think there's things that companies do to change.
I've tried changing those things and they don't want me to change them.
So either I'm bad at changing them, which is probably true, or they don't think it needs to be changed, which is also probably true.
It's like, I could hang around and get bitter.
I'll just go now.
I'll just go now.
So I went off and I thought, I went independent.
It's been really interesting ever since because it's sort of, on one level, the shackles are off.
The problem of working for yourself, though, is that the boss is frequently an asshole.
So current Sam frequently.
is cursing past Sam for the commitments he's made.
That's one lesson, I guess.
Yeah, yeah, yeah, yeah.
I mean, as I say, I'm almost 50.
I'm still not learning the lessons of that, right?
And I've made those mistakes for a few years now.
So I want to talk about microservices.
You have written the book, Building Microservices.
In 2015, you started to write it in 2014.
And as reference, Uber started to talk about how they have thousands of microservices in 2015.
So you were kind of writing this book right as...
The concept of microservices was starting to go mainstream across startups.
Do you remember how and when microservices took off and where was it even coined?
Was it around Martin Fowler?
No, it was James Lewis.
So it was going back in the day.
So you got to remember, again, it's like something in the water, something of a bushfire.
There was a lot of buildup before the bushfire kicked off.
And so this would have been 2011 to 2012.
James was talking to a bunch of different companies.
who were building what would have classed back then as service-oriented architectures.
And earlier there'd been this kind of, what we refer to as the guerrilla SOA movement, which is an aim to simplify service-oriented architecture, you know, shift over to REST, work done by people like...
Yeah, the original service-oriented architecture had like this XML format.
What was the name?
Oh, WSDL and simple object access protocol.
Never has the word simple been less well used.
Soap.
Soap.
Yeah, soap.
Soap.
Yeah.
I start with simple.
It was the most complex thing ever.
Exactly.
So you had this whole heavyweight and then people like Ian Webber and Ian Robinson and Jim Webber had like been pushing centralized.
Had mantras like make the endpoints smart and the pipes dumb to keep things simple, put all the smarts in it.
So this is all the background.
We were also at this time doing a lot of work around like I was running the continuous labor practice for Europe at the time.
And so a lot of my engagements were like optimizing the final mile.
you've written the code, and it's like, how do I get it here into production as quick and effectively as possible?
So basically taking the idea as a continuous delivery.
So I was around when Jez and Dave started working on those in 2006.
I'd had the conversations with them.
So that was all happening.
James was talking to companies that were creating service-oriented architectures where they could replace the services quite quickly.
And so basically you'd look at it and say it's a finer-grained form of SOA.
So that was all happening.
So James went, we had this architectural symposium.
up in the Lake District.
And James pitched this idea.
He was called the micro apps at the time in this room.
And somebody else in the room said, they're more like services.
Don't remember who said it.
There was about 10 of us in the room.
Most of us were hung over.
And then James and Ian, so James and Martin wrote the first post.
So the first paper on Martin's website was written by, it was sort of James's original kind of discovery of this pattern that he called microservices.
At the same time, Martin had a lot of stuff around infrastructure automation.
And so I was coming at it from a different lens.
So I think James was coming at it more from an architectural lens and I was coming into it from a continuous delivery lens.
I spent at this point almost 10 years with that being my focus.
And so I was starting to see the architectural impediment.
So if I want to change things frequently, but the things I want to change are integrated with other things.
Yeah, like a monolith.
Now that's how we would describe it.
Exactly right.
I mean, I did a talk called Designing for Rapid Release in 2012, which is sort of touching this.
So basically, James had this sort of, had put a name to this stuff and put some patterns on it.
And I sort of thought, hang on a minute, that stuff I was talking about over here seems, these things seem quite sensible.
And I'd always wanted to write a book.
I mean, honestly, I wanted to write an animal book.
I wanted to write an O'Reilly book because that said an orc book I was given.
And so I just started writing.
I've always found writing quite...
I'm quite lucky that I find writing enjoyable and quite easy.
So I just sat down and wrote the book and that's it.
But for me, it was an extension of continuous delivery.
And it was a style of architecture that was only possible because of continuous delivery and because of DevOps.
We'd had these ideas before.
I mean, Erlang is a representation of these ideas, but making it real, practical and attractable thing.
But I mean, really fundamentally, it's just the whole thing has been an exercise in smuggling through concepts of the 1970s, which is fine by me, right?
If I get people to pay attention to information hiding and coupling and cohesion, that's all right.
But then as you said, like this was literally a spark before the bushfire and then the bushfire did start.
Like suddenly, again, I think Uber was a very notable company starting to talk in public about having thousands of microservices on conferences like InfoQ and it got people's attention.
And then...
it kind of seemed that overnight, a lot of startups started to say like, you know what, let's just do microservices.
There was a bit of rebadging going on.
I mean, I was lucky when I wrote the first book that I met Ben Christensen fairly early on through the Yale conference.
Ben was looking after performance and sort of reliability at Netflix.
And I also knew people at REA.
And these were two early REA, realestate.com.au in Australia.
And these were companies that were very much pushing the envelope in different ways.
And so I had two very good real world case studies to draw upon for my book.
But most other people weren't talking as publicly around that.
And also, before Netflix referred to their architectures, microservices, they just referred to it as fine grained SOA.
And to an extent, you know, I have the mind's understanding, that's what happened at Uber and elsewhere.
We didn't have a different word for it.
Then, of course, what happened was that everyone just called their SOA architectures, microservices, which is sort of...
semantic diffusion is a thing.
It's to be expected.
So I just kept fighting the good fight for like, it's just independent deployability.
I've always positioned it as it's an opinionated style of SOA.
You may not agree with the opinions, that's fine.
Do something else then.
But there are some opinions.
So then what is the difference between microservices and what was the other?
Service-oriented architecture.
Service-oriented architecture.
What are the differences?
In your very strict...
Now, to be clear, right, I have my definition.
You might have your own definition, but I'm the one that wrote the books on it.
So my definition wins.
That's always been my argument, right?
Write your own damn book.
I've always presented it as it's...
There were two kind of very strong opinions I think we hold with microservices.
The first and most important is that your microservices need to be independently deployable.
So your unit of deployment, unit of change needs to be a single microservice boundary, a single service boundary.
And, you know, rather than I'm deploying everything together, that's a monolith application.
Three services at the same time.
Yeah, even a mini monolith, right?
And it's a single service at the time.
That's clear.
That's it.
And the second one, which I think there are going to be exceptions to, is that your kind of default way for defining your service boundaries is around your business domain.
This is taking the concepts of, you can use the management, you don't have to.
But rather than a lot of SOA architectures were dominated to an extent by technical decomposition.
The classical N-tier architectures, layered architectures, where you've picked your service decomposition based on the nature of tech, rather than thinking about the business functionality.
That's a bit softer.
Independent deployability is the main thing, though.
Okay, well, because I'm not thinking back at Uber.
I was at Uber at the time when your book was starting to become really popular.
I was there from 2016, and we had independently deployable services.
We didn't really think too much about the businesses or domain.
They had, obviously, their interfaces.
They had contracts that they offered increasingly strongly typed and we used different ways to do it.
But it was just more like, I mean, as you can imagine, there was just devs, oftentimes not very experienced devs who were like, all right, let's build a new service.
Here's the interface.
And then anyone can use it, which actually, there were these things where a dev or even a not very experienced dev deployed a microservice and then others started to use it and suddenly it became...
you know, years later, there will be these exercises to figure out the hot path and label services.
And then this like loan service was on the path of like some really, it needed to be requested as a really high tier service because a lot of others were using it.
I mean, there was always, I mean, also in those sorts of environments, I was checking to people at SoundCloud, similar about that.
And this is also like about a corporate culture and like what was acceptable.
Like in that kind of environment, just go and do stuff.
It's absolutely fine.
And I think that's why Uber, for example, When did I just talk with Twan Palm, the C2 at the time, about like what was the real deal about microservices?
And he said Uber, like he never really wanted that many services, but it was just more of a constraint because Uber had a monolith.
The story was like, which he told in a podcast, we'll link it at the show notes below, that Uber had this big monolith and it was really slow.
It was slowing delivery down and they needed to move fast because all the other competitors were out there.
So they said, all right, like they wanted to get rid of the monolith and put together smaller services, but they knew unless they had a strong mandate that you cannot add new stuff to it, this will never happen.
So they said the mandate was everything new needs to be your own service.
You take care of it.
We don't care.
And suddenly so many independent services were born, which was not the goal.
And years after the fact that when Uber decomposed the monolith, they're now looking at maybe sensitively merging based on business domain, et cetera.
But this story wasn't really told at the time.
It was just, oh, look, like Uber has thousands of microservices, almost one or two per engineer, right?
And this is what the world saw.
I mean, it's also, though, but this is the reality of microservice adoption.
It's like, it's just a taller toolbox.
I mean, I've described microservices as being an architecture of last resort.
Really?
Yeah, yeah, absolutely.
Because it pushes you quite naturally to a more aggressive distribution of your state and everything else.
And you have to have a good reason.
One of the best reasons to use microservices is if what you actually want is an organization with a higher degree of autonomy.
If what you want is teams who are able to act independently from each other, the microservice architecture is a fantastically good fit.
But then really it's a means to an end, right?
So that's often what a lot of people get wrong, I think, is they adopt microservices and they think this other magic stuff's going to happen.
If you want organizational autonomy, that's what you're working towards.
And this is just one of the mechanisms to get you there.
And most of the enterprise organizations, the vast majority of ones I've worked with in this space, that's what they're after.
They've got huge numbers of developers, lots of middle management.
They're not getting much work done.
They're thinking, what do we do?
Organizational autonomy becomes, you know, the solution to all problems.
Again, I don't always get to have to say whether or not it's a good idea.
And I think that's where microservices can be especially useful.
And there are other exceptions as well.
And so an organization like, you know, Uber.
I mean, I know at Netflix as well, things like Domain Driven Design weren't talked about at Netflix, but the teams were already fairly well scoped around areas of products already.
And so you had those natural high-level boundaries around the business domain.
So those things flowed naturally.
And so that for me was a logical kind of stepping stone.
You also have the difference, and this is something I realized quite early on, when you go and do the work in the dot-coms.
versus the enterprise organizations.
When you say dot-coms, you mean the digital companies?
Tech companies.
I'm old enough that I still talk about dot-coms.
But the tech companies, I mean, fundamentally, algorithm, you know, from a domain concept, they're incredibly simple.
The domains are very, very simple compared to enterprise organizations by and large.
Domains in there are much, much more complicated because they're based on business processes that have existed for 10, 20, 30, 40, 50 years.
Oh, yeah.
So what you mean in a lot of tech companies, a product is pretty much like, here's our product, here's how it makes money.
And that's kind of it.
And we have another product in Uber.
Inside Uber, like a simple thing is there's rides, which it makes money like this.
And then there's Uber Eats, which is a separate business.
Overtime, by the way, does get more complex.
But when you start, it's like really, really simple.
And it's really separate.
And it's clear where one ends versus where the other begins.
And then you do some hacks when you need to launch Uber Eats really quickly.
And you build it on the rides infrastructure.
And then Uber Eats order has like two.
trips attached to it because anyway, but as you say, like early on, it starts very simple.
But then also in those environments, your business processes and the digital implementation of those business processes is evolving together.
A lot of enterprise organizations didn't have that.
I mean, their first generation of digital products were codifications of paper-based processes.
Oh, so you're talking that these companies before digital, let's say, Adidas or let's say a shoe company, virtually all banks.
Virtual banks.
And now they're like, we have all these processes and now we need to build a software that mimics what we do, which of course is...
That's how you start off.
I mean, that's what the whole Domain Duranesign book came out in 2004.
And it was a reflection of the fact that the vast majority of software development was the process of us trying to model existing non-digital processes and how you do that.
I wonder if we will have a different but similar problem of the same thing now that...
You know, we know AI tools are here.
We know that, for example, they're great at generating code.
And we have two types of companies.
We have a company, a digital company that has pre-AI.
They're used to, you know, writing product, programming documents, deploying developers, product managers, designers, QA.
They all have their roles and it works like a really well old machine.
And now you have these so-called AI native startups who just, again, they have a few people.
They're just, from the start, they're using agents for everything.
They have a very different process.
And some way when this whole thing will mature and we will have best practices, we'll have the existing companies who, like the previous one, they will try to retrofit, you know, this tooling to how they're working today.
It's already happening.
I mean, and the thing is, enterprise companies will rarely be successful doing those first-gen things because ultimately the corporate DNA would just crush that stuff from within, right?
And it'll take a while.
But I think the whole industry is grappling with the idea of, like, are we now moving to a new level of abstraction?
Right.
So software is all about abstractions, right?
But, you know, we went from doing machine code to doing assembly, went from assembly to general purpose programming languages.
And you're coming back to the idea of the program being the mental model.
We would evolve that mental model at the code point of view.
That was the most effective place was us working at Java, you know, Java, C Sharp, whatever it is, rather than working at assembly level.
Right.
And it's like, OK, well.
is actually what we're talking about, really.
We're just stepping that level up.
We're still actually building a product together.
We're still thinking about what the software is.
We're still building that mental model, maybe now more collaboratively involving, you know, more product people, for example, in that conversation when we're writing our specs.
And so the question for me is, are we ready to make that step?
And the answer is no, not everywhere.
And this is a really good point because if you think about it, like, you know, like one of the mantras inside of Facebook used to be that I think, something along the line that code is truth or something like that.
But in general, in a lot of tech companies, if you want to know how things work, like where the truth lives, is it in documents or is it in the coded business logic?
It's in the code in the business logic.
And if it's well-maintained code, often well-commented or structured, you can actually see the business logic and the rules for, I don't know, for example.
See, I feel like I've got to channel my friend Charity at this point and say the truth is production.
Everything else is, is the lies we tell ourselves.
Production is truth.
But I think in terms of like a definition of what the system should be doing, there is this ongoing debate about where is that definition?
Is it the code or is it the spec?
Brigitte Brockley did a nice article on Martin Fallow's website recently about spec-driven development.
And she talks about the idea of some companies being spec-first or spec-anchored.
And a lot of it comes down to where they see the source of truth being.
So a lot of companies are using things like Kiro or SpecKit to define a feature.
And then the LLM executes that spec and you can iterate on that process, right?
And so a lot of people, they start off, they use the spec to write the code and then they throw the spec away.
And then you get other companies that keep the spec.
And then when the feature changes, they just update the spec.
And so they're sort of seeing a bit the code and a bit the spec as a source of truth in terms of what the system should do.
And then you're now starting to get companies where they didn't even ever look at the code and they just look at the spec.
And so we're kind of going on this spectrum.
She had some great terms for these.
And I think for me, it's like, is that the right path?
I don't know.
I think there's a lot of questions about what we would need to do to get that.
We have a degree of confidence about what happens when I execute a line of Java code beyond the safety of the compiler and the runtime.
you know, all that stuff.
There's a lot more questions to answer, I think, before the whole world is ready to say the spec is the source of truth of what our system is supposed to do.
Production is always going to be the real truth, though, just to be clear.
But I think it's a good question to just ask yourself, however you're building, however you're using agents, of where is the truth of your system right now?
Is it more in the code?
Do you have a spec?
Is it scattered between the minds of people?
which goes back to the collaboration.
Because again, the reality of a lot of startups are is a few people hold these things in their head, right?
Which is okay in the beginning.
But of course, as you scale, one of the reasons we used to love code review, which is now going out of fashion, but we used to...
share information and when someone left the company or got ill or, you know, the famous bus factor, it was okay because other people understood because they reviewed, they were taking part in that part of it.
But now with AI code review, some companies are like, well, we're not going to do it because it's too much code.
That was always one of the superpowers of pairing.
Right, pairing, if you do promiscuous pairing, you rotate every day.
When someone leaves, everyone's got the same shared context.
That was always something people missed around one of the benefits of pairing.
I think there's a lot to be said around this.
I was chatting to Chris Ford.
He's sort of head of software, head of technology for ThoughtWorks in Spain.
And he said the biggest problem we've got around AI-driven software delivery is tacit knowledge.
There's a lot of knowledge in our heads that we've built up that we don't necessarily know that we have.
When you use something like a code review, you've probably had looks at that and said, that doesn't really feel right.
I wouldn't do it like that.
And sometimes it's subjectivity, but sometimes there's something about that that you can't quite pinpoint.
Maybe you can unpack it, maybe you can't.
But it's also like, you have bunches of things like that in your head about how software delivery should be done, about how the business works and everything else.
If we want to step up that level of abstraction to work at the spec level, we actually need as human beings to be able to unpack that tacit knowledge that we have and make sure it's in the spec if we really want to be hands off.
I also think there's a fundamental problem that we've got in the current way a lot of people are doing software deliveries.
You've got, and this comes back to the cognitive debt stuff, which is you've got a lot of people running off in different directions.
I've seen a few people talking about pairing with their AI assistant.
Not a human.
Stop pretending it is.
And so you're all going off.
You're doing more in a different direction.
That shared mental model of the program is breaking down and there's gaps in between.
AI was supposed to free us from drudgery.
Right?
It isn't for most software developers because we're doing more work, we've got more context switching, we're losing that big picture.
So I think if we can work collectively on the specs, we can rebuild that model, then that absolutely could be the source of truth.
I still think it's going to be parts our brains and part the spec, right?
But for that to work, there does have to be the collaboration piece.
Whether or not that's our dev teams go from being 15 people all in to about five people or six people all in, don't know.
But collaboration is really important.
You know, Chain Guard are a proper company, right?
They do proper stuff.
They're excellent for them.
They recently announced they were getting rid of any human-based code reviews.
And they said, we're going to spend more time working on things like, effectively working at design doc level.
And actually, I'm okay with that because firstly, I trust that they know what they're doing.
But also, there's something about, I mean, I think a lot of companies are making the mistake of not doing the thinking.
Whereas Chain Guard is saying, we're going to spend that time doing more important thinking.
Right.
And I think that's sort of how we should view this.
I'm okay, you know, not looking at the code as long as you're doing other things to make sure the system is operating correctly and that you're thinking critically about how the system needs to change and evolve.
If you're just not looking at the code and you're not caring about those other things, well, then you deserve everything that's going to happen to you.
Which also goes back a little bit to what Charity was talking about on how you want to have systems that value, that works.
about having empathy, for example, for the work that QAs and SRE folks have had to own code in production that they did not write, they don't necessarily read or even understand, but they know how to operate.
They know what good looks like in production.
They know what it looks like when it goes poorly.
I mean, that's also a bigger thing that now more developers are having to struggle with because if you want to do spec-driven development, you have to know what good looks like.
I did a...
I run these like sort of half day conferences for O'Reilly online.
We did one recently on software delivery.
A load of speakers and about six speakers and four of them were talking specifically around different varieties of spectrum development effectively.
And all of them came back and said, you can't do this unless you can define what good looks like.
And if you can't, don't bother.
And, you know, some of those people were much more bullish, like Addy, also very bullish on all of this stuff.
But one of the presenters who was doing, they've got like a small software fracture, only 15 or three people.
But like they're actively pulling in their observability data, their O-Tale stuff, right, into their agent to understand how it's changing.
And I think that's exposing, I think, more developers into understanding.
I mean, this comes back to the acceptance criteria stuff people do on stories.
They're always fairly narrow and functional requirement stuff.
So now it's like, I think we'd have to be more grown up about those things.
The discipline is...
absolutely important and the expertise is still needed.
It's just needed at a different place.
Sam made the really good point that you cannot hand your work over to agents unless it does these two things.
One, define what good looks like.
Two, prove that your system does what this good definition looks like.
You already know who I need to mention when it comes to proving that your system works as expected.
Antithesis, our season sponsor.
Agents are writing more of the code, so how can we trust that our system works correctly without bugs?
Code review is increasingly naive and ineffective approach to do so, which is why Antisysys goes well beyond this practice.
Antisysys runs your whole system in a hostile simulation.
This simulation includes both targeted testing and fast testing.
By running this simulation, it finds every bug before your users do.
And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it a lot easier to fix issues.
The first thing I thought when I heard about Antisysys and that it's automated bug discovery and fully deterministic testing, that it sounds like science fiction.
But actually, it's hardcore engineering when you look under the hood.
JaneStreet, Fly.io, and the etcd community ship agent-written code with full confidence because they know it's been verified by Antistisys.
To see more case studies and details, head to antistisys.com slash pragmatic.
I'd also like to mention our presenting sponsor, TurboPuffer.
We know it's a database that's ridiculously fast and extremely cheap.
And they are doing something amazing.
They are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale.
If you follow TurboBuffer, you know that their storage architecture was a massive part of their early success.
Redesigning a winning architecture is a big deal.
It's one thing for query plans to pass unit tests, which is already happening.
It's another to bring each query plan to performance parity or better while maintaining correctness and reliability in production.
And the team is sharing the messy reality of the rewrite and how the v3 version is currently a lot slower than the current production one.
For example, we can see how cold full-text search is 11 times slower than prod and hot full-text search is 126 times slower.
But we can already see some wins like hot and cold vector search is at parity or better.
I really appreciate this work-in-progress view and the real-time logs and what it takes to do a successful storage architecture rewrite.
The TurboPuffer team is keeping on the work log of their journey, and you can follow along at turbopuffer.com slash v3.
I definitely will do.
And with this, let's get back to Sam and what QA and S3 folks already know about running code that they didn't write.
So I want to talk to you about your latest book that is just coming out, Building Resilient Distributed Systems.
Yeah.
What is the book about and why did you feel?
this is a good time to write it.
Years and years and years ago, I went to Velocity, which was quite a well-known conference.
It's a conference.
It's still on?
No, no.
It was the O'Reilly conference and it was like the conference where a lot of the early dot-com people went to talk about how they made their systems scalable and stuff.
I got to go.
It had been like one of those career ambitions to go to.
I got to go.
I was after I'd written the first book and I got chatting to John Allsport there, who I still chat to for now and then.
He now works in his resilience engineering space.
pointed me, I don't know if it was him or someone else, but it pointed me at this paper by David Woods that was all about resilience engineering.
And the paper was utterly baffling to me, in absolutely no sense to me.
But it was all, because it's not, yeah, because it's just, it's not written for computer scientists to read.
And I'm not even a computer scientist, so I was really in trouble.
But it was really fascinating seeing this whole other world about how we make, how people view resiliency.
And then what we do in the software world.
And people like John were trying to take some of those ideas in a little bit.
But I found it very difficult to take some of those ideas from sort of safety critical systems and that type of thinking and make in any way, shape or form palatable for normal people like me.
And so I kind of, this would have been back, what, 2016 maybe, 2015.
And so I'd go back and try and reread that paper every six months or so.
I'd read bits around the edges and see other smarter people than me that summarized it data better.
And then I was thinking, okay, these ideas now are starting to make sense.
And then also I kind of realized that despite my best efforts, lots of people seem to be building microservice architectures, even though if you've read my books, most of my book is about telling you not to do it.
But you have to read the book, which is called microservices.
Let's be clear.
As long as they buy the book, that's the most important thing.
Whether they read it or not, that's just, you know.
And so then I realized that there were more people exposed to the...
the horror, pain and suffering of a distributed system.
And so it was like, okay, so I've got this stuff happening.
I've got this, like my, you know, my son's developed for now.
And I almost thought that if somebody found themselves building a distributed system, working as part of building a distributed system, what was like the kind of equipment of a rubber ring I could give them, say, stay afloat and don't drown.
And so the goal of the book is to start you off with really easy, foundational, simple kind of engineering concepts about how you can start making dealing with the problems of disability system, but to also take you on a journey into kind of the whole concept of resilience engineering.
And so it starts off with really easy ideas and then you slowly get more and more complicated.
And, you know, so I introduced kind of concepts like what a socio-technical system is, the difference between safety management and resilience engineering.
These are all like big concepts.
But the second chapter after the introduction is what's observability to talk about OTEL.
The third chapter is what's a timeout, right?
And these are all, I think, quite tractable things to talk to people about.
But they're building blocks, right?
The primitives for these systems and the vocabulary as well.
Yeah, absolutely.
And to give, you know, to go through and sort of distill some of that knowledge down and also to be a bit of, I'm quite happy being a bit opinionated on these things, but I'm also lucky now because partly because of my early work, I get to chat to more people so I can actually talk to people who this is their full-time job and try and distill down some of that knowledge a bit.
And so I wanted that book partly as about being evergreen.
But it was also about saying there are lots of people building a distributed system.
Let's put what type of distributed system you've got off to one side for a moment.
There are lots of people building them, whether they should or not.
And they're really important to us now.
So how do I help people start and grow?
And that's what I wanted to do.
I mean, I enjoy the process of writing anyway, but this is a book I wanted to be able to give to people.
That's always been why I wrote a book.
I want the books of that book.
Every now and then I've gone sat down to read a book, to write a book, and I found someone else has done.
I know everybody's like, well, that exists now.
So this for me was a book that I wanted to exist but didn't.
That's kind of why I wrote it.
Yeah, and I think it's a book I certainly could have used.
Like, it got dropped into a place like Uber where we did have what I realized were distributed systems.
It's just large systems handling a bunch of stuff, lots of services depending on one another because even a simple thing of like knowing like, oh, you should pay attention to observability that, I mean, you figure it out after a while.
Yeah, yeah, yeah.
But it takes a while on your own.
I mean, again, I mean, maybe this is my attempt to unpack some of that tacit knowledge to an extent.
I've made so many mistakes and I keep making mistakes.
I think making mistakes is useful, but I think also creating opportunities where people can learn from other people's mistakes first is a good idea too.
And so in the book, I also try and pull in real world examples and sometimes from big companies, sometimes from my mistakes to kind of make this stuff seem real.
And I think the other thing I think I really try and get across is that things go wrong.
Right.
I mean, engineering is all about making sure as many things go right as possible.
It's not about making sure everything goes right.
And there's also the only way you can build resilient systems in general is also by being honest about mistakes that have been made and talking about them and sharing them because that's how everyone gets better.
And if you don't do that, you can't build a resilient system in your company.
And so this, again, is my attempt to sort of highlight.
all the awesome work that's gone before, the fantastic post-incident reports of people that some companies share.
Like when there's a system that we all use or many of us use has an outage.
I mean, it's annoying, especially when you're using it.
But I always greatly appreciate when the companies, because they don't have to do this, but when they share a public post-mortem or incident recap, whatever that is, they go through what happened.
For example, GitHub had an outage and they share a really interesting one on what all the things that went wrong and in what order.
And then you also have companies where something like this happens and they don't do anything.
So, of course, you're not obliged to in any way, but I always have a bit of a lesser of...
They are obliged to now, a lot of them.
Regulations are now making them.
Which I think is great.
Yeah, things like DORA and stuff in the UK, in Europe rather.
And I think there's also, we should also separate around that post-instant report stuff that there's marketing and PR versus have you done a proper post-instant review?
Now, I do think you've got duty of care really to let users know it's part of a proper post-incident review, but those companies that maybe say nothing, they might do a fantastic process internally.
There's also the fact that some companies that put those things out, you read them and you're like, this does feel more marketing.
I think a great, if you want a good, a company that puts out some of the best self-flagellating post-incident review write-ups, go for Cloudflare stuff, they absolutely go for themselves in those.
And it's like, I feel like, guys.
go easy on yourselves, right?
Everyone makes some mistakes, but I quite like.
And they did it rapidly within 24 hours and they're a publicly traded company.
Yeah.
Which I think just goes back to the myth, like some of them are like, we're publicly traded, we don't want to do it because it would give away information for shareholders.
No, like Cloudflare does it and they're fine.
Exactly.
And also it's...
And the belt of trust.
It is.
I mean, a lot of how an organization is perceived is down to how you deal with mistakes when they occur, right?
I mean...
Nothing ever goes perfectly for every company.
So in your book, you talk about the three rules of distributed systems.
What are these three rules?
So I really tried to distill this down because I could never remember the eight, the rule of eight that came out of Sun.
The fallacies distributed computing.
I always forget what those eight were.
There's loads of complicated stuff.
But I realized when you strip it all down, there's really only three things that really cause all the other problems.
The first is that you can't send information instantaneously from point A to point B.
It's physics.
I mean, now it turns out when you get into it and start talking to people about quantum physics, they say, aha.
And there you go, there's a thing called quantum entanglement, which does allow for the theoretical instantaneous read of electron spin states across arbitrary distances.
At which point I point out that A, that only works for reads as far as I'm aware, and B, not a networking protocol.
So like, but it takes time, right?
So I'm going to get from A to B.
And I, that.
I can't control all of that.
I can control bits of that as a developer.
I could control how much data I send.
I could control the payload size, the technology I use to serialize and deserialize.
I'm unlikely to be specifying the network cabling, the equipment we use, or the distances between them.
So I'm not really under my control.
And the moment you go over the public internet, BGP has a bad day and your packet has gone down the wrong trouser leg of the internet, right?
Yep.
So that's the rule one.
It takes time.
And that's important to recognize because sometimes it looks instantaneous.
It's not.
Second thing, sometimes the thing you want to talk to isn't there.
Sometimes that's under, and we do things to make it under our control, right?
Like we would run, say, multiple copies of a service instance behind a load balancer.
What happens if the load balancer fails?
But I mean, like you can't control for everything, right?
I've had services taken out by a fire in a data center, a sand caught fire.
And specifically one bit of a SAN.
The SAN had a caching storage layer with SSDs.
These two pairs of SSDs.
One of the pairs of SSDs caught fire.
The other pair, the SSDs were working fine.
And so we had these weird latency graphs coming out of the SAN.
Like, perfectly fine if your load went through the ones that weren't on fire.
But the failovers in the other one were routing.
To be fair, it was routing around the bits that were on fire.
I had a system where we had...
ducting between the office buildings and rabbits moved into the ducts and ate the networking cable.
I can't control for rabbits, right?
And sure, these things are like, they're real.
These are true things, right?
Yes, you can control that and you can reduce the chances, but you can't eliminate those chances, right?
So sometimes, rule number two is sometimes the thing you want to talk to isn't there.
And the third rule, I'm very grateful to an old colleague of mine, Ben Butler-Cole, for saying I'm missing, is that resource pools are not infinite.
Meaning?
You can run out of computing resources.
CPU, memory, IO, whatever.
Basically, your load balancer could be overloaded and then just not respond, even though it's there and we know the latency, but maybe its queue fills up.
And look, the vast majority of the time when you have a system outage, it's down to that.
Some resource has been totally saturated somewhere.
And it might have been that saturation might in turn have been caused by the fact that something is down and you're in a retry storm, which is in turn causing...
you know, connection starvation, which is, of course, because of resource saturation, which is in turn, yada, yada, yada.
But that's the killer.
And again, I think we understand that, you know, sort of intellectually, all of these things.
But you can go to all the concept, complicated concepts in distributed systems and talk about why is Bison-Tine consensus protocol a thing?
Well, let's just go all the way down to this bit.
What about CAP and PAC-HELC?
Let's put ourselves on the side.
Remember these three rules first, and then most problems end up becoming a little bit easier to deal with.
And in the book, after you introduce these rules, you start, as we mentioned, the second chapter is observability.
To me, that kind of suggests that this is a really, really important part for distributed systems.
Why is observability so important?
I mean, the way to think about observability is it's a property of a system, right?
So the more observable something is, the easier it is for me to understand how that thing is behaving based on the external signals.
When you run a system, which is a single program, you actually get some pretty high quality signals coming out of that program, therefore give you a high degree of observability.
I can get a lot of information out of that.
I could attach a debugger to a process.
I could generate a core dump and run that in.
I could do a thread dump on the JVM and work out where my semaphores are blocked.
If this is my whole system, like knowing the CPU is stuck at 100%, it's actually really useful information because that's probably a bad, that's a problem, right?
So this basic information that we pretty much get for free with a single program gives us a high degree of observability.
It's a single program running on your machine that you have access to.
Exactly right.
But now we go to a distributed system, you don't have access to the computer in the same way.
You've got multiple computers.
And so those basic signals are harder to get.
You can't correlate them unless you do extra work.
And those individual signals by themselves don't really give you a proper sense of how the system is behaving.
So you have to do more work.
to make the system observable.
And as annoying with these sorts of things, it's much easier to do that from the beginning than it is to retrofit it.
And then, you know, luckily we've got things like OTEL now, the Open Telemetry Protocol and specification, because that at least gives us an open framework.
A vendor agnostic.
Oh, Honeycomb are the best.
Charity pays me in stickers, just to be really clear.
I always think, I work independently.
I have to be very clear.
Charity pays me in stickers.
But like...
We get a vendor agnostic open standard for this now, so it makes it so much easier for me to have the conversations about this.
But if I don't get the signals coming out of the system, if I don't get that stream of structured events telling me what the hell's going on with my system, I'm flying blind.
Like, how do I know what appropriate timeout is from a client point of view talking to a service?
Well, I'm only going to know that if I have a sense of, well, what is its normal response time distribution?
So I'm going to need latencies, know the success or failure of those latencies, probably want to be outplotting a histogram maybe to take a look at it.
If I don't have that information coming out, I'm not going to have to specify timeout correctly.
And then we're talking about kind of low-level things like latency and things that we can measure at the machine level.
But of course, is this observability when I'm talking about, like, does it work for the user or the customer?
Does it do what they do when they click the button to submit?
Does it submit?
Do you look at that?
Because we know observability is usually we talk about these kind of...
the statuses, the 500 responses, that's kind of bad, that means it's down.
But there's always a player that I always wondered, and I found it hard even when we're working in Teams to, okay, like we have the easy observability.
Like this is kind of text, not textbook.
It's the metrics.
It's the metrics.
And the logs, maybe.
But then there's the stuff of, does it work?
Does it do what people care about?
Like again, in the case of like an Uber.
Like when people hit a button to request to write, like does it actually request within usually five seconds or something like that?
Or is it stuck in an infinite spinner, which is bad?
Your locks can be as healthy as it can be, but people are upset, right?
You know, you treat the raw data of making the system observable as a stream of structured events.
And then from that, you build up the higher level abstractions and abuse, right?
So uses as primitives.
So a trace, right, where we get a sense of for a given...
click of a button, what happened in the production system?
And this is a big blind spot, right?
Most organizations had to build their own tracing systems.
So this is where things like the open tracing stuff came from and things like Zipkin and things like that were built as an attempt to recreate.
Like, I mean, most people were just normally trying to recreate what Google had already built there.
And this was an attempt to say, okay, can we at least look at a flow of calls?
What happened during that flow of calls?
Where did these things go?
Well, that requires me to get a bunch of spans across from all these different machines.
And a span meaning?
A span meaning an activity.
It's something I did.
And then a span can begat more spans.
So I can say my first span might be a call has hit my server.
So I've now got a span that starts.
Within that span, I might talk to my database and then do some other operations and I can put additional information on those spans.
And then I can go up to another service.
And then you get a trace ID that's shared across those spans and you've got timing.
So now you get...
effectively a view of what's happened across those calls.
But that's all just based on individual events that have been stitched together.
And then you also start getting into the situation which is, is my system behaving correctly?
And, you know, this is going away from, again, I think half the time I'm just saying what charity would say.
You know, again, going away from the vanity dashboards.
This is moving into the sort of the world we call the SLOs, right?
Service level objectives.
And SLIs.
And SLIs.
So an SLI, service level indicator, that's our way of saying, at this given point in time for this thing, is this thing good or is this thing bad?
And we use a collection of those.
So one example of that might be a successful percentile response time, right?
So that would be, is it a good one or a bad one?
It could be a bad one because it's an error.
It could be a bad one because it's slow, good or bad.
And I collect those together and that helps me know if I'm meeting my SLO.
So I might have an overarching SLO, which is I should be able to request a ride.
the 99th percentile successful ride request within 300 minutes.
And that would be your statement and that would be a commitment.
The SLO is supposed to be a commitment.
That's something our team is working towards.
You then get this raw data, you calculate your SLIs on the individual things and then you use that to work out what your SLO is looking like.
If it's green, everything's a good time.
If it's red, people are unhappy.
I mean, you get a bit of red.
There's way more nuance to it.
I'm taking a very bad introduction to this stuff, I think.
Without those building blocks, because I talk about the first chapter of the book, nobody says, I don't want my system to be resilient.
No, no one.
But at the same time, resiliency is not a binary property.
It's not like off or on.
It is a bit like observability.
It's a degree.
So then the question is, how much do you need?
Because am I having a conversation with you about multi-cloud and, you know, doing complete rewrites of systems for critical standings like Monzo do, for example?
Yeah, they have a standing bank when they go down.
So it's sitting there, it's costing them money for the most part.
But then just a few weeks ago, their main application went down for a short period of time and they activated that bank, which I'm assuming it costs millions of pounds or dollars per year just to...
keep it there?
It's not even a slight sidebar, but so firstly, their standby bank has a real stripped down list of features.
It's much more bare bones, but it's all the core ones.
Like you can freeze cards, you can take money out, you can check your balance.
It's a complete rewrite.
So it's no shared code because they're worried about a bug.
Different cloud vendor.
It's also, by design, significantly more cheap to run.
So they've got thousands of microsomes in the full stack.
This is about a tenth the cost of the cloud costs.
They also run it constantly.
So that is always there.
But like Monzo, no, they're Monzo, right?
That's how that works.
But like, there's no point you starting there.
So actually, the reason I put the observatory stuff in is like, you need the raw data, but also you need to make decisions about, this comes back to what good looks like.
Then you need to know, okay, do I need to worry about multi-cloud vendor, VR strategies, because that's where I need to be going?
Or am I somewhere else?
And so I think without that kind of sense of your North Star of how resilient do we need to be, without that defined, you don't know how deep you need to get.
This is why I'm trying to say how secure does my system need to be?
Have you done a threat model?
Have you done a threat model?
I don't know.
And I guess to answer this question of how resilient do I need to be, it greatly helps.
And maybe it's actually a requirement to do that.
The foreign decision is...
how much will it cost me in terms of operating costs, in terms of time to build it, in terms of, for example, latency.
Just a simple example is if you say like, okay, you know, the business is like, oh, would you like multi-cloud?
And this is like, yeah, we'd love multi-cloud.
Like if Amazon goes down because we heard they had an outage last week or a few months ago, I actually want to be able to run Google Cloud.
But then you'll be like, okay, well, first of all, this will mean we're paying two cloud bills.
And if we want full replication, it might increase our latency because we need to wait for it to write for.
Both of them, which might mean slower, you know, customers might see things slower.
Anyway, there's all of these trade-offs.
And in my experience, like oftentimes, like you might retrofit that, but if you can have a discussion upfront, that's great.
But to have this discussion, you go back to the basics, you need to know what the trade-offs are.
You need to know what's happening in all these dimensions that, you know, you care about, like how long will it take to build?
How complex will it be?
How much money will it cost to operate?
But also as technologists, We can't just say, oh, we have also got to get good at communicating with our business stakeholders, with our end users.
What percentile latency do you want?
Like some developers know what I say when I say that.
Not all, but some.
How many, you know, experts in their field who are great product owners or subject matter experts, but are not software developers would know what I meant by that.
And so we also have to understand the business domain.
the product domain well enough to do the interpretation and help and to bridge that gap.
And it is a dynamic thing as well.
I mean, this is the whole thing about resiliency in general, right?
A resilient system is one that changes.
It changes and it evolves.
If you don't change, it evolves.
You're a brittle system.
You're not a resilient system.
And so even those targets, we say, that conversation we have to have about what is good enough does change.
But like think about something like in the case of, say, the US East one outage towards the end of last year.
There were lots of...
On AWS.
On AWS.
So AWS, US East One region went down.
It was the oldest cloud region.
Big region.
Lots of people were there first because that's often where new services got launched historically.
When that went down, it caused a lot of people to say, do we need to be multi-region?
And some of those companies would have had the conversation before and decided, no, we don't.
And then they'd seen the fallout and they may have changed their minds.
And that is fine.
This is what you mean by...
a resilient system evolves.
Yeah.
That's the real world.
But also, like, it is about what problems you're not going to try and solve.
There were some big-name companies that got taken out when US East 1 went down and got severely, had severe degradation in terms of what they could offer.
Companies like Zoom, for example, they weren't taken off entirely, but they were severely affected.
Do we think that people working at Zoom are idiots?
No.
Do we think they knew that if a cloud region went down, they'd be impacted?
Yes.
They probably looked at that and said, well, For us to come out with a multi-region approach, it's going to have this impact.
It's going to cost us this much money.
It's not worth it.
Let alone talking multi-cloud.
Because multi-cloud versus multi-region is a whole different conversation.
And that's okay.
And I think as long as you've had the conversation, and as long as, this is the other secret bit of it, of course, is as long as what you think you need to do is in tune with what your users think you need to be doing.
This is why I make the point in the book when talking about the difference between SLAs and SLOs.
The SLA is what you're going to get sued for if you breach.
If you only ever deliver to the SLA, your customers are rarely ever happy.
You've got to do more than that, right?
So there is the social contract there as well.
Now, with building distributed systems, building resilient systems, it really helps to understand concepts that are not as straightforward.
I'm not going to go through all the things.
You explain a lot of things in the book, but I do want to touch on a concept that I found a bit tricky to understand and then appreciate and you really appreciate distributed system, which is idempotency.
Yeah.
What is idempotency?
So there's a mathematical concept behind idempotency that I don't understand.
But basically, an idempotent operation is one that you can, you know, you can apply that operation multiple times without it actually having an effect multiple times.
So it has an effect once, first time.
A really good example of an idempotent operation would be pressing a lift button.
So you want to go...
down in the lift you press the down button it lights up if i press that down button again nothing's changed the lift is still coming well we hope the lift is still coming right yeah right that's an idempotent operation some light switches are actually toggles i toggle it light goes on toggle it light goes off right that's not an idempotent operation in computing we have some operations which are naturally idempotent reads are a delete is typically idempotent right But the example I think I use in my introduces concept is the one of payment.
I was about to say that I have a lot better item potent example for you.
Yeah, payment is a good one.
So I'm going to pay you money.
So I'm a service that wants to issue a payment and I send you, please pay Sam a hundred pounds.
This is, I'm NLPing people back at home.
Please say pay Sam a hundred pounds.
And I don't get a confirmation back.
I don't get back a response.
The problem is from my point of view, there are two possibilities.
The first possibility is that you received that payment and you've done the payment and it's been completed and the result has got lost to me.
For some reason, I didn't get the response.
That's one possibility.
The other possibility is that you never got that.
You never carried out the operation.
maybe didn't get to you, maybe you crashed halfway through.
I think like a good example that I always, I've seen it in the past, maybe it's over, but airlines, you're about to buy this airline ticket for $1,000, $2,000, you've checked all of your details and it says like play or reserve.
And then often some airlines would take you to an intermediate page which says like, please do not refresh this page or submit it again because you might be double charged.
And what you really want is you pay, you know, like pay.
And if you, you get the spinner and it's not idempotent if you do it again and you get charged twice.
Yeah.
But it is idempotent if you refresh, you copy the same thing, you submit the exact same form.
Oh, that's interesting.
So then the question then becomes, right?
So I sent you that payment, book my airline ticket, whatever it is, and I haven't had that response back, but I really want to make sure that the airline ticket booking goes through.
So this is you with the browser or the people at home.
sending Sam 100 pounds, right?
You really want to make sure Sam gets his 100 pounds, right?
Now, when something doesn't work in a distributed system because of rule number two.
Rule number two meaning the thing might not be there.
Rule number one, sometimes things take ages.
Yeah.
And accepting this thing is, you just try it again.
You retry.
Because you really want it to happen.
And in retrying, there's the danger that if the first operation had been processed successfully and you didn't know about it, or the first operation had been applied successfully, and you do it again, you're not getting paid twice and it would too.
two tickets.
So that's the problem with item potency.
You want to make it safe to retry important operations.
So to do that, we need to make item potent.
There's two ways to do it, really.
One that's good, and that is one you should do, but you kind of have to retrofitting is paint.
And one, which is one you just explained, which is useful, but has problems, right?
The first one is item potency keys.
So you generate, you have a unique ID.
You'd normally use a UUID for this.
And so when you say...
The unique ID which you generate and when you generate two of them, they should be different.
There's like this tiny, tiny, tiny, tiny, tiny, tiny likelihood of clash, which again, what happened, I think.
I know, but like, I think with UUID 7, I think it's like a 50 nanosecond chance.
But it's something you can generate locally without an additional roundtrip.
Your collision chance is extremely low.
And there's ways to modify that.
You can work with that for the most part.
Unless you're Amazon, probably.
Well, this is, I mean, The AWS APIs use the Stripe, Adium, PayPal.
They all use the same key scheme.
So you send this to your UID.
So then if you send that to me, you say, pay Sam, and here's the idempotency key.
I don't send you back a response for whatever reason.
You retry it.
You pass the idempotency key.
If I had carried out the processing, I could look at it and say, well, have I processed this?
Yes, I have.
Okay, great.
If I have processed it, even if I get a second or a third or a fourth, have you processed this?
Processed it.
I just keep going, yeah, processed it.
It's processed.
It's fine.
That's a nice, simple thing to do.
The issue with making things idempotency key is it requires both the client and server to support it.
So it has to be a required field to go through, and it has to be sent to server support.
So if you're trying to retrofit this in, it's a pain.
But if you're building it, use an idempotency key.
Use it, exactly.
If anything needs to be idempotent, do that from the beginning.
The one you sort of mentioned was, I go back, I hit refresh, I fill in all the same fields again, and I submit the airline booking again.
You should know it's a...
duplicate.
That's another thing you can do.
And this was as good if you want to retrofit item potency in.
So if I've sent the same fields, so the same person, Sam, the same amount, 100 pounds.
And I'm sending this through again.
What you can do on the server side is generate was a fingerprint.
So I take these fields.
Yep.
I then generate a hash that represents those fields.
So if you do a second booking, it looks the same.
And you're using that as item potency, as a retrofitted item potency key.
Effectively, yeah.
There's a couple of downsides with that.
You do get false negatives with that.
The false negative meaning that?
It might get legitimate second requests and get rejected.
Got it.
So a real world example, this happens in, I know this happens in Denmark.
It might happen elsewhere.
I'm speaking to people in Denmark who works on a payment system.
You go in, you buy something for 40 kroner with this particular payment system and give the car, scan the car for 40 kroner or whatever.
If within the next sort of five minutes, you buy something else for 40 kroner in the same shop.
The tilt handset rejects it.
If it was 41 kron, it'd be fine.
If it was 39 kron, it'd be fine.
But that's because the chances of you buying something for the exact same money is so remote.
They're doing some basic idempotency to top double taps.
And so when you get situations where operations can legitimately look the same, but actually be different and carried out at the same time.
So a really good example would be the AWS APIs.
It is completely legitimate for me in a for loop.
to provision, say, 20 EC2 instances.
Now, there's better ways of doing it than sticking it in a for loop, but I could.
So I could iterate through the for loop and spin up 20 identical EC2 instances.
If I just relied on there being a fingerprint, they'd all look the same.
So that's where this stuff breaks down.
And so if you've got situations where legitimately you have operations that look the same, but are actually different, that can be done close to each other, you have an issue.
If you've got a bit of space around them, what you can do is hold on to those fingerprints for a period of time, five minutes, 10 minutes, an hour.
So there's still some potentials you can still let duplicates stick through.
So it's not ideal.
Item potency key is the way to go.
Fingerprints when you have to register a fit.
If you use them both together, that can be really useful.
So you can pick up situations where people change the parameters between the retry attempts, which leads systems into confusing.
It's a bit more of a complex topic, but...
a lot of public-facing APIs will still generate a fingerprint.
And so if you send, carry out this operation, and you give the same idempotentcq as you gave before, but you've changed the parameters, the server can reject it as being effectively an invalid request.
So that's how those things work together.
The ideas are really simple.
There's some nuances about implementing them.
There's some interesting stuff that crops up in terms of how you store them on the server side and things, but the fundamental ideas are...
pretty straightforward, I think.
Another interesting topic that comes up often times these days, it feels more often a thundering herd.
Yes.
It's when a system restarts and suddenly it faces this big load.
Specifically, GitHub had an outage now a few weeks ago where it was networking related and recovery took a lot longer time because there were retries going on and it created the thundering herd.
What is a thundering herd?
Genetically, I talk about it in any situation where you've got mismatched capacities.
You've got too many requests for too few resources, right?
Resource pool's not infinite.
Coming back to rule number three.
Yeah, rule number three.
Yeah, so in the chapter, I talk about a whole bunch of different ones from different types of scenarios, right?
So you've got everything from the malicious thundering herd, like a denial of service attack, right?
A digital denial of service attack, malicious attack, often short notice, and there's not that much you can do about them.
The answer is use Fastit or Cloudflare, but I go into details about how that happens.
You get things like a kind of internal self-inflicted ones.
So a good example would be something like a cache collapse, right?
A system relies on the cache not only for speed of operation, but also for reducing load on other components.
Cache gets restarted.
You then have a lot of cache misses.
You're hitting the origin and the origin goes down.
or was not expecting such load.
Yeah, so a common thing there would be, first you'd flush those cache snapshots to disk, reduces the chance of you being out or having high cache misses over a restart.
But also a lot of companies I know will just turn their services off until the cache refills.
And so that's kind of another example of one.
Retry Storm's another one.
Back in 2017, Square had a big outage.
They'd restarted Multipass, which was their multifactor authentication system.
MultiPars in turn pulled its data from Redis that they're using as a database because you can do that apparently.
I don't think it's a database.
Well, they're using it as a cache basically, right?
Well, no, I think they were using it as a database.
There are some people that think Redis is a database.
This is a different podcast.
I'm thinking of starting a stupid technology podcast where we talk about using technology in a stupid way.
So that's a separate conversation.
Anyway, so they were putting this data back from Redis and they had a problem that the clients in MultiPars were going, basically, if the Redis calls failed, they'd try again.
but without any delay between the retries.
And they had a hard-coded limit of 500 retries.
So they just basically smashed Redis and they kept restarting Redis and then they'd restart multipass.
And again, this is a long time ago.
And they fixed it, right?
They fixed it by having a more sensible retry policy and externalizing those values.
That's another thundering herd.
The last kind of sections or ones I talked about were ones caused by like products being too successful, like Twitter and healthcare.gov, right?
This is where...
You have an external, real, legitimate load that takes you out.
And the thing is, you look at all of these as solutions, although what's happened is the same, the kind of solutions are a bit different.
So like when I've got a thundering herd that's coming from externally that's legitimate, well, I've got different things I'll look at.
Like I will have more, I'll have a more constrained sign up.
I'll have fewer invite codes.
So with healthcare.gov, one of the issues, there was many others, but was that they went live in like 36 states at once and they were massively overloaded.
So in hindsight, maybe going live in one state would have been a good idea first.
For example, with Twitter, they were going down all the time.
So they could have constrained signups or put tweet limits on you and things like that.
Those are things.
And they did those things as well, right?
And so although they look the same, the mitigations for Eton was a little bit different.
So I kind of quite like talking about that because they were all great stories I found.
And of course, I think we're talking about solving it with...
when you have same resources, but then we get into things like scaling.
Like, can you scale horizontally, add more resources?
Because there's two sides of this, right?
Like whenever you have it.
And so what point can you do it?
Or are you going to face a bottleneck because you still have a central whatever?
But like, that's a great example of where sometimes mitigations work and sometimes they don't.
So an example of something like your product is successful, if you're a startup or whatever, and you suddenly got successful, that success comes like a wave, right?
The wave of success hits you.
and then a lot of it recedes, you'll get a thousand customers in and maybe you'll keep a hundred.
Maybe, right?
That's just how things work.
So you need to be up during the peak, but then you probably can shrink down again to keep the cost low.
So if you've built your application to have some dynamic scalability in it, and if you're on the public cloud or similar place where you can spin up and spin down, that's a viable mechanism, right?
Okay, we can scale up to handle this and that's fine.
Would I use that as defense against a distributed denial of service attack?
No, because then your distributed denial of service attack becomes a distributed denial of cash attack, right?
You go from to not only having no service, but also now no money because you hit spinning up, you know, nodes on your Kubernetes cluster and Amazon just keep taking the money.
It's great.
And you still can't keep up.
So that's a perfectly valid mechanism for some things.
But you kind of have to know the context in which to say, you can't just get yourself out of trouble.
I think we're going back to a lot of times as an engineer, you want to know about the business.
You want to know what creates a value, what creates revenue, what is important for the company versus what is not.
And then where we want to maybe invest to with our infrastructure, with our reliability, versus where it doesn't matter.
It's not worth doing.
It's all about trade-offs.
And I think it's about, you don't know, unless you get information out of your production systems, you don't know how close you are to the red lines.
So that's why the observability is key, but it is all about trade-offs.
And that's why I do a whole chapter on like formal methods.
I talk about cap and pack elk and I say, look, these things are complicated, but actually really all comes down to a conversation with understanding the business context.
So the example I give there is like if an e-commerce company, if I want to buy an item online, right, physical item online.
And if at that moment I want to purchase it, if you're not sure if you've got it or not, most e-commerce companies will say, yeah, sure, you can have it.
Because they know there's a good chance they do actually have it in stock.
And if they don't have it in stock, they can get it in and back order it.
And the worst case scenario, they can email and say, sorry, we fund you the money, right?
So that's not great.
It's not ideal.
Now, if they said, we don't know if we've got this item, therefore we're not going to sell it to you, they're missing out on a sale.
And I could go buy that somewhere else probably.
And they might have had it.
So e-commerce companies in that situation, because of that sort of understanding of the relationship with the customer and the impacts if they get it wrong, they're going to try selling the item.
Keep that problem exactly the same, but now change the business context.
Now we're selling concert tickets.
So you're getting tickets to go see Tay-Tay, right?
You're going to go see Taylor Swift.
And we're not sure if you've got the Taylor Swift ticket or not because our link to that system is down.
But those Taylor Swift tickets are yours.
You're taking your kids, taking your wife.
It's all going to be yours.
we're going to Portugal, we're going to go see Taylor Swift, it's going to be amazing.
And then we email you guys, sorry, we don't actually have those tickets.
Now, the problem with that is, firstly, the devastation you're going to feel is a bit different.
They can't back order a ticket.
And so in that situation, it's like the impact of the customer is so much greater.
And then they'll book the flight.
Well, you might book the flight.
So what they do is they make a different decision.
So in that situation, if they don't know if they've got the ticket, they don't sell the ticket.
I'm not sure if you cover in the book, but this is the fail open versus fail close, right?
When you have an error or you're unsure, are you going to go through or not?
And as you say, this can be the decision, the same problem.
Two companies can take a decision.
A company years down the road could take a decision.
Uber had this classic example of when payment, we weren't sure if the payment went through.
Do we allow the ride to continue or not?
And early in the company's years, we always failed open.
Yeah, just have the ride.
We might have the money, and if not, we'll deal with it.
As the company starts to go for profitability, we feel closed.
Like, we're not sure we have the money.
If the charge went through, you're not going to get the ride.
And again, same company, different focus.
And if you're a developer, you needed to pay attention to what is important right now for the business.
It was growth initially, and it's okay if we lose money.
And then now it's, I don't know, unit economics, profitability.
But also that open close versus open, that's a failed close versus a failed open discussion.
isn't one size fits all for the whole system.
It's on a feature by feature basis.
And even then, you could say this taxi ride is 10 euros or it's a thousand euros.
You can make different decisions.
Now, if you approach this from a mathematical and from a computer science viewpoint, we just talked about CAP theory.
That was a conversation about CAP theory.
Right, so in the chapter I talk about, here's CAP, here's Pac-El, which is an extension of CAP.
I explain what those models are because when you get deeper into the systems, you'll hear about them.
But fundamentally, that's what this is about.
When you have a partition, that's what P in Cap really defines.
I can't talk to one part of my system.
Remember, it's rule number two.
When I can't talk to another part of my system, what do I do?
And all you need to know as a developer, I can't talk to that.
What do I do?
Sometimes we can make technical decisions.
I'll try again.
I hope I don't bring my systems down for the retry storm, but I'll try again.
If that doesn't work, what do I do?
Well, actually, often the answer is, Yeah, sure, you can look at CAP and Pac-Helk or whatever else, but go have a conversation with a product owner.
What should we do?
The resolution of that often ends up being a decision about are we trading off consistency versus availability in Pac-Helk or harvest and yield if you're going into that version of Brewer's Conjecture.
But don't worry about that.
That's in the book.
That's why it's a later chapter.
That's chapter like 12, I think, or 11.
Don't start there.
Start with having the conversation.
Rule number two, I can't talk to this part of the system right now.
What should we do?
That starts with having the chat.
That starts having the chat.
When we're talking about resilience, you talk about the dimensions to think about resilience.
What are these dimensions?
I start in the first chapter, coming back to that paper by David Woods I mentioned, that was impenetrable to me at the start.
And he comes up with four concepts for resilience engineering.
And I like the four concepts he comes up with because he helps us think broadly, right?
So it starts off with robustness.
This is where we can basically...
I think in his words, absorb known perturbation.
So we model a system and something goes wrong and we can kind of just deal with it in a way that no one cares about, doesn't even know.
So a very simple example would be in Kubernetes, a pod fails and Kubernetes means a replacement.
Now the pod failing, that's a perturbation.
It's gone a bit wrong, but we dealt with it.
Great.
So that's the start.
And the thing is, most engineers I speak to, they focus entirely on this.
They don't look at the other three.
They focus entirely on this.
One of the really nasty things about robustness is the things we often do to make our systems more robust in turn increases the complexity of our system, which in turn increases the surface area for new types of failure.
I bring Kubernetes in to deal with the failure of a container.
Now I've got Kubernetes and that becomes, and that's not solvable by the way, that's just something to be aware of.
That is also why sometimes people don't go multi-region or multi-cloud because it's increasing the surface area of other failure types.
So that's number one, it's robustness.
Right.
Can we just do one of those things?
And for things to be robust, we have to know about them ahead of time.
This is predictable.
We deal with it.
Then there's rebound, which is when something does happen and goes wrong.
How quickly can we recover?
So this is typically the time to recovery, well, time to detection and time to recovery.
Yeah, and there's different ways of measuring that.
And I talk about in the DR chapter, I talk about, you know, data loss.
Disaster recovery.
Exactly, yeah.
But that's like, how quickly can we get back up and running?
So we've gone into a degraded functionality, how quickly can we get up and running?
So that's thinking about the rebound.
And again, for that, you have to be very humble as a developer because the robustness is, the system will never fail.
And it's like, but what if we got it wrong?
Okay, well, we have to admit things go outside of our control.
And I guess for recovery on how you can recover, this is where, I mean, if you want to do it well, teams that are serious about this, they do.
simulations of the system going down.
This is where Netflix's very famous chaos monkey comes in, where it goes and shuts off parts of the system to test this live.
And there's drills that are being done on how our on-call process works, how our decision-making, etc.
This is actually a hard problem.
And then you did it once, that's not enough because you need to do it regularly because the system keeps changing.
I'd like to give a shout out to Uptime Labs, who are based here partly in London.
They do excellent.
instant training that you can do online in a web browser.
It's fantastic.
We'll add them to the show notes below.
They're nice people.
So you've got that recovery process, like how do we get up and running?
And I think kind of techies are with me.
And then we go into the next two, which the third one is graceful extensibility.
And so graceful extensibility, again, it's a slightly weird name, but it's basically dealing with surprise.
When we make our system robust, we're dealing with the known problems.
We know what to expect.
What about things we don't expect?
If we don't expect it, this is now not about computers, it's about humans.
This is why distributed systems are fundamentally associated with technical systems, people and technology are working together.
We couldn't expect the rabbits.
How do we deal with the rabbits?
How do we deal with the farm and the data center?
It turns out badly is the answer.
But if you're in a corporate environment, which is very command and control, quite hierarchical, where you've got a very narrow job description, you're often constrained in what you're able to do.
Those organizations tend to deal with surprise badly because the corporate structure is structured around the known.
And so often structurally, you'll find that difficult.
You mentioned drills earlier.
Drills are a great way of helping with surprise.
Coming in and having people often from outside saying, this has happened, how will you deal with it?
Google had a thing, I don't know if they still do it.
SREs would pop around and they'd do a thing called the wheel of misfortune.
They had a physical wheel.
Today, your system has suffered a...
the spin the wheel, DNS poisoning attack by a state in the name of rogue state here.
Here's what's happening.
How do you deal with it?
That is all about people and experiences and trying new things and constantly pushing yourselves.
I guess learning also from others.
I remember this very famous outage happened a few years ago at Meta where their DNS system through BGP, it just all went down.
WhatsApp, Meta, everything.
to fix this because they pushed out an update to their BGP table, which is inside DNS to their servers.
Now to fix this, they needed to get access to the computer that physically had to get access for whatever reason, because they couldn't reach it remotely.
So they need to go into the building.
Now in going to the building, they had to go to a door that protected this system, which was running on their own domain, which was down.
So they had to break physically into, but again, this is a thing where they share the details, which was great because now everyone else could think about like, do we have any of our physical security, for example, tied to our own domain?
Because if you have it, you want to change that learning from what Meta have learned about.
And this learning takes us from that grace of extensibility, which is dealing with surprise.
That's like sort of almost in the moment.
Are we ready to deal with surprise?
Into the fourth one, which is sustained adaptability, which is how can we learn from what's happened?
Not just the learning.
So do we have the environment in which we can learn and share and grow, but then also adapt the system based on that learning?
And it's called sustained adaptability because that is a completely continual process.
That's not just today.
That's not just when you have an incident.
That's when you don't have an incident.
What are we doing to learn?
And I think your point you raised there was excellent, which is I like reading other people's incident reports.
There's a little bit of schadenfreude with companies I don't like, but there's also a little bit of it is like they've learned something.
Can I learn from them and do something that resolves my system?
So when you don't have an incident, read other people's incident reports.
There's a place called The Void.
It hasn't been updated for a bit.
There's hundreds and hundreds of incident reports there which is worth going through.
So that idea of sustained adaptability is all about us adapting and changing over longer timescales.
For those last two things, sustained adaptability and graceful extensibility, this all comes down to people and culture.
Having proper environment psychological safety is just...
Absolutely required.
If you can't raise issues, if you can't challenge in a safe way, you will deal with surprise badly because things won't get raised and you won't learn because things will be hidden.
And so that all comes down to behavior stuff.
And so the last three chapters of the book, we focus on that is looking at some of the people and the social aspects of that.
So I've got like 14, 13 chapters, or 12 chapters, sorry, about the other technical.
And then the last three chapters are all around that side of things.
So like, what can we do to make sure that we...
have that environment of psychological safety because without it, you can't actually be resilient.
That's pretty amazing because so far throughout this whole conversation, we talked about building resilient systems.
We talked a lot about people, about how to think about resiliency.
One thing that we didn't really talk about just passively is AI, even though this book is coming out now.
And I know it has a section on AI as well, but it's just interesting to reflect that even though AI is changing how we're...
building software, it seems like some of these ground truths of how to operate reliable software are a little bit just outside of this realm, right?
Well, I mean...
Or is it?
No, it's not.
I mean, firstly, there's some stuff, there's some AI stuff in there that can help.
I detail quite a few examples where around sort of correlation analysis and, you know, help support around post-instant.
There's some instant.io doing some amazing stuff in this area.
So you're saying we can use tools to do a lot?
But really, there's more fundamental kind of things here.
And the first thing is the macro and the micro.
At the micro level, there's some useful stuff we can do here.
And I talk about that, but the macro level is that none of it stacks up financially, right?
So if you're taking a longer view about resilience, if I'm building my products based on a market sector that is fundamentally flawed, these companies don't have a viable path to sustainability.
When you say which companies are we talking about?
OpenAI and Anthropik and the hypervisors, right?
Conservative analysis by The Economist are showing that they need to be generating 2.6 trillion in revenue to justify their capex expenditure by 2030.
The entire software market is 1.4 trillion.
So macro level, I don't know how that gets sorted.
I'm not smart enough.
I also don't have any shares of those companies.
But, you know, that's a separate conversation.
I can't solve that.
What I can start thinking about is, OK, if that's what I'm operating in.
How do I deal with that?
I can't deal with all that.
What I can do if I'm taking a view to resiliency is being very careful about how I use AI.
For example, you know, if I'm thinking taking a longer view, being more flexible about model choices, that's going to help me because I've got the opportunity to use open-weight models from different vendors.
It's even just basic things like open AI and Anthropic are pretty terrible when it comes to availability of their core models from their services, which is why everyone I know also will use like, they'll use Google or Amazon or Microsoft as, either their primary place they go to as models or their fallbacks.
They have better uptimes and things.
So there's all that stuff we've got to deal with at that level.
There's also the conversation about non-determinism and Charity pulled me up on this a bit.
And I said, the thing is that, you know, we're going from deterministic software to non-determinism.
You know, that's a challenge for us.
We don't know if things work.
And then she said to me, our distributed systems were never deterministic.
And now they're really not deterministic.
And of course she's right because when we take our distributed system and we add the people.
beyond the rabbits, right?
And the fires and the people.
We always had the degree of non-determinism.
We're just amping that up a bit.
So I think that's a continuation.
So for me, I think it's a confusing landscape.
It's not good or bad.
I think at the micro level, there's some really great things that we can do.
When we say micro level, that's the tactical things that we can do, right?
I think really what I mean by that is I'm taking the macroeconomic policy situation out of it.
I'm saying as a developer, there's things that we can make use of, right?
There's some good stuff in there that we can make use of.
But I also think there's some hedging we need to do.
Longer term, if you're building AI into your product, there's some simple hedging things.
So being multi-vendor, being multi-model, for example, breaking rather than having a single agent, breaking that down into more workflow models.
Therefore, you can take bits of your workflow and realize actually this can now be swapped out for a bit of deterministic code that I write that could run more efficiently.
I think these are all sensible things that you can do.
We can't fix this.
We can at least deal with our world.
And I think there's some sensible things that we can do in that space.
What are the things that we know that we do not know about AI?
I think the tech world in general is quite naive about AI and fundamentally misunderstands what an LLM is.
Let's be real.
Why did the LLM delete my database?
Well, because it has no concept of causality.
That's what it comes down to.
It has no concept of causality.
There's a glass here.
push that glass onto the floor, you'll be working out as a carpet down there.
If I ask you what's going to happen, you'll say, well, glass is breaking me, fine.
If I push it onto that floor, you're looking at it saying it's a hard floor, that might break.
You're carrying out a bunch of calculations in your head.
You've got an idea about the physical properties of the universe.
You've got a sense if I push the glass, this is going to happen.
LMs don't do any of that.
They have no concept that if I do, A, B happens.
Well, there's a push for world models, but...
These LLMs are not world models.
They're not world models.
And world models have been around for a long time.
They arguably predate LLMs.
They can do stuff like that.
So they could make decisions like if you had the appropriate model and the appropriate inputs, they could at least get to the point of, if I do X, then Y happens and I've been told not to do Y.
This is why all the guardrails around LLMs are really not long-term going to be the right solution.
And so I think we expect LLMs to do more than they can actually do because they seem so smart.
And so I think that's the problem and that leads us to overestimate their capabilities a bit too much, I think.
Or maybe over rely on them without knowing and we might have some incidents later.
Absolutely, absolutely.
And I think we're knowing where and when to kind of to use those things.
And I think what we don't know right now is where should I be having, and again, this is wider, where should I have the LLM do it and where should I just write some deterministic code?
There's a lot to be said.
But just having your agent call a script and you know that script works rather than having it implement the script in its magical brain for you.
Well, there's something to be said that it seems both labs, both leading labs, Entropic and OpenAI have realized that coding is so important.
So oftentimes when you ask the alum to do something, it will start to generate code because it writes a deterministic program, because that part will be deterministic.
It's very interesting when you ask it, it will often just pump out a Python or TypeScript or whatever, especially when it has to do with involving compare this or that.
And they have trained these models to like default to coding.
I'm not sure how this came, but it seems to be working to some extent.
It's also because it's very tractable, right?
They are there to generate plausible code.
They're there to generate plausible words, right?
And in the context of code...
They've generated some plausible code that we can now validate.
Does it execute?
And so that's a really nice, tractable aspect of it.
And there are some places where it works really, really well.
But there are other areas where it's terrible.
Like when I wrote my book, I use very little AI at all.
Well, you didn't use it for the writing.
You used it for research.
I don't use it for the writing at all.
I used it for the research.
But in the context of the research, I used notebook.lm, basically.
But in that context, it's plausible.
But then I just go and click all the links it gives me.
And I say, was that there or not?
I am checking the work.
It's done way more work than I could have done going through all those papers.
And God knows there's only so many NIST papers you can read through, right?
But in that context, I feel completely fine and able to do that.
In other settings, you're getting now this problem called cognitive surrender, which is where you're using AI to write documents that you're never going to read.
That's not really around.
a problem with the LLM technology.
That's the problem with us.
Okay, can we talk about this cognitive surrender and cognitive depth?
Yeah, cognitive depth is a kind of term that's been around for a while and that sort of explains the erosion in our ability to think critically and to reason that can occur when we're using AI tools.
And there's been a bunch of research into this that's shown, there's been some studies that's shown that people's ability to think critically is impacted the more they use chatbots and things.
And I think we've sort of seen other examples that margaret and story's got a great um article about this and she cited some some of the research around it as well and that sort of explaining that i mean her her thing is you know she's talking we talk about technical debt building up in a system we have to be careful that our use of ai's it's that cognitive debt we need to have a shared mental model of what this system is supposed to do thinking about it narrowly the lens of a program that we're building and if our use of ai as an example we all go off in different directions that shared review breaks down that becomes a cop a debt cognitively.
Cognitive surrender is a different thing, which is like, oh yeah, it's fine.
It's basically LGTM, right?
But for whatever the AI tells me.
Yeah, LGTM was typical, looks good to me, approving large pull requests where you just couldn't be bothered or you try to look at it and for whatever reason you didn't want to go through or it came from someone who you trusted.
I worked on the Google tool team on all that stuff, right?
And so LGTM.
was literally a something with pattern match on and takes that as an approval.
So if you saw LGTM, that person's approved, right?
And it was just, that was where, I think that may even have been where the acronym originally came from, I'm not sure.
I think it came from Google, yeah.
Yeah, but it is this idea that, and it's become shorthand for, I didn't really look at it.
And I think that's, that cognitive surrender thing is coming out in lots of ways.
You know, we're getting AI to write documents that no one's going to read.
And we've got AI that's reading documents that weren't written by other people.
So the cognitive surrender piece is really that the humans, this comes back to the human being in the loop.
There's very subtle differences in those workflows and to the kinds of differences it changes.
Like there's a big difference, like Cory Docter talks about it.
It's a big difference, like if you're, say, a cancer, an oncologist, or a cancer specialist.
There's a big difference to you as an oncologist working and an LLM doing sort of detection of some of the scans that you're looking at going, oh, you should look over there, by the way, you might have missed that.
That's actually quite useful, right?
versus the LLM-powered AI going, here's all the scans I've looked through, here's the ones I need you to sign off.
The last one, the second one, takes the expert out of the loop and makes them an approver.
And often in that context, they've got rid of a bunch of oncologists.
You've got one person in the funnel that's going to lead into issues.
The other model is we're making people better.
I think that's to come back to the original idea of AI or some of the pitch, right, was it was going to free us from drudgery.
just like the Industrial Revolution was going to free us from drudgery, but it would elevate us, we'd have less toil, we can focus more on critical thinking and smarts, and so much of our current use of AI is not freeing us up from critical thinking.
It's causing more context switching.
It's causing less time for thinking.
People are working longer hours.
We've got studies that show this now.
That's not necessarily AI's fault.
It's just how we're using it.
It's like we need to re-address that a little bit.
Can we talk about complexity and AI?
maybe adding it.
I'm going to quote you from a few months back.
You wrote, I worry AI could just become an exercise in doubling down on current complexity rather than stripping things back to simplify, at least in the platform space.
Yeah, I can't remember the exact context in which we had that conversation.
But I mean, the platform space was all, I mean, the platform space is kind of interesting, right?
Because that was an attempt to make things simpler for developers because Kubernetes became popular and it's not developer friendly.
That's not what it's designed for, right?
But most developers never have to touch it.
No.
I mean, well, If you have a platform, if you don't have a platform, yeah, there's a platform team's problem, right?
And that's why they get paid.
And so I think there was this idea that I was sort of hoping that there would be a bit of a complexity collapse around that space.
I had this hope for things like WSGI, for example, simplifying down those programming models a little bit.
And I think there was starting to be a bit of a conversation starting to be had about some of that stuff going on.
What then happens, of course, is AI has come out and everyone's like, I've got to AI all the things.
And so rather than thinking about what we need and trying to strip that complexity back, we've now just got another piece that requires us to still have all the old complexity of adding new complexity on top.
And somebody's got to look after all that stuff.
Yeah, but I think it goes back to a little bit of like, we seem to have more work with AI.
And probably it doesn't help that it can add more complexity, especially when we're not.
being clear of what we want, we're not being critical of the output when we have cognitive depth or cognitive surrender, especially not with cognitive surrender, you know, just right.
So these things might be amplifying the fact that it's just more complexity, right?
Yeah, and I think there's also that connection through to what the AI is doing.
It still comes full circle a little bit, which is the idea of potentially not looking at the code isn't to say that you don't care about what the system is that you're building.
It's to say that you've got a...
better way of understanding what you're building.
So one of the concerns is if I'm using AI to create code, this is going to create awful loads of debt and a complicated system.
No one's going to understand.
Well, we can't let that happen.
We still need to remain connected to the system that we're building, which is what I'm a big fan of, like modularizing system.
Whether or not you put those modules in different computers, I don't actually care.
Getting AI to work within the modules, having the humans think carefully about what those modules are and the connections between them.
let the AI go hog wild inside those boundaries that are easy to change.
The gaps between them, that's where the human beings spend to think about the mental mold of how things fit and slots together.
That feels a very sane way for us to balance our time and our energies appropriately.
So do I understand correctly that you're saying that it's probably a sensible approach is have the architecture in your head, know how your system works, know how the boxes work, know where there's AI inside it.
Maybe it's generating code.
Maybe it's running your incidence response in isolated ways, and then spend your time talking with other people in the business, on your team, with your customers, because you're, as you said, systems like software is people's ideas connected and you have things in your head or you write it down, but now you're using it as an AI as a tool to enable you to maybe take on larger chunks of the business and maybe make better decisions because maybe you have more time to.
as an engineer, understand more on the business, on what is working, what is important, those kind of things.
Yeah, and also to make the structure on the model a bit more explicit as well.
I mean, that's one thing a good modular structure does.
I mean, this is why Shopify have been so successful keeping their thing as a monolith, right?
It's because they spend a lot of time building their monolithic application as a modular domain-driven architecture, right?
Because it makes the structure easier to understand.
You can zoom out and see how those things fit together.
But also in the short term, while we're having these debates about are we going full software factory or not, we don't have to make a one-size-fits-all approach.
We can say, look, this is our architecture, being a modular monolith or a microservice architecture doesn't really matter.
And over here, this area here is where we're trying a full software factory approach where we don't look at the code anymore, but we specify the interface.
We have verification around that.
We know what good looks like for that module, right?
We've got functional tests, tests that look at, say, some of the...
operational requirements as well.
These areas over here, we're still grappling.
These are really key and really important to us.
We're not ready to give up on those yet because we don't have the trust level yet.
We don't want to take that risk just yet.
And so over here, we're going to work with the AI more of a collaborative way while we work together.
And we're still going to be intimately involved with the code because we're not ready to make that step here.
But if we approach our whole system with one big black box, then we're forced into making one or either decision.
I don't think that's healthy.
Do I understand that you're saying that you think, that the software factory, the dark factory, you giving it a spec or the AI either building it or operating it, you see that as viable.
There are some companies I know that are already all in on this.
Not everyone should be doing it whatsoever.
I think if that is our future, if that is our future, I think there's a lot of ifs there.
The question is about how you go about adopting and trying.
So what I'm trying to do is find ways in which we can lower the barrier to entry to try these ideas out in a safe way.
And like low-risk environments and maybe simpler domains, those kind of things, right?
But also like a lot of dark factories don't create finished products.
Like real world, beyond the fact that real world dark factories, there aren't many of them, most of them create components.
They don't create the finished product.
So in the same way, if you think about a module being...
a piece of the overarching system right well let's have a factory around these bits here and actually looking at it like if we had to rewrite that from scratch it might take us like with the collaborative way with ai it might take us a few weeks to have to rewrite that if we had to so actually we're quite relaxed about leaving that to go as is and it's been working fine and it's a it's a module around which we've got great test coverage already for example where it's not on the critical path from a latency point of view or it's a service where we've got really good observability data and really well-defined slo's and we've got good steady state well we can start running it as a factory now and seeing what's happening in production we can have a look at the slo's we maybe add some special slo's around there i'm hearing correctly that you're saying that as a professional as someone who wants to be up to date with the new techniques that we can use it's probably if you have the opportunity to start to build software factory, a somewhat autonomous place in a component in something low risk, it's probably worth trying it out to see how it works, to see what you learn, to see if it even works or where it breaks down.
But do it in a way where you're minimizing the last rate.
It's a little bit like when, you know, when microservices came out, the concept, you didn't start by moving your biggest thing.
You started to build a small service that didn't really matter all that much.
And you were okay to learn from it when it goes down and it wasn't on the half path for the business.
Yeah, I think it's the thing to start off with is getting your head around spec.
So looking into spectrum development, whether that's Kiro or TESOL or SpecKit, whatever that is, I mean, they're all slightly different.
Have a look at Brigitte's article on that as a good on-ramp to get ahead of it.
Start off by playing around with that because that's the stepping stone.
That's like, you have to define what good looks like as an input to your factory.
Start with a spec.
Good input and also for the output.
Yes.
And then it's like, how are you going to go about verifying the output from the factory?
Have a think about that.
And, you know, like not just in terms of the kind of normal sort of behavioral aspects of it, but I agree with Kevin Hedy on this.
He said he hates non-functional requirements because if a functional, if something is non-functional, it means it doesn't work.
So he breaks the world down into like development requirements, operational requirements, and sort of business requirements.
So I think it's a nice way of doing that.
So along those three axes, how do I verify that the software being created by this module is, has...
matches my developer requirements, so for, say, maintainability.
My operational requirements, latency, uptime.
And my business requirement, is it working?
Features working.
So as long as you've got a way of validating your software around that, and if you don't know what the good looks like for those three things, guess what?
This isn't for you.
But you can still do specs, by the way.
But you can't do a factory.
But you can do specs, and that's, you're edging.
And I think...
it might not work.
There may well be some blind alleys.
This could be model-driven development all over again, which we do not have time to get into because we get very angry about it.
But it could be that.
But I have a hope it might actually work this time, unlike MDM.
And then as closing, I think a lot of what we talked about is it's important to...
pull the architecture in your head to think about the architecture thing, how to modularize now for engineers who want to get better at this kind of high-level thinking, strategic thinking, architectural thinking, outside of reading your book, which I think brings a lot of really good ideas on architecture as well and not just really insisting.
But outside of that, what would your recommendation be for a tech professional, for an engineer who's like, I'd like to get better at architecting systems to know what good looks like?
I think there's a few books I can point people at.
I think the, Software Architecture Fundamentals by Neil Ford and Mark Richards is a good read.
Just to be clear, they're friends of mine, so I'm not unbiased in recommending it.
I would say around the modularity, Vlad Kononov's book on balanced coupling is fantastic.
His book on DDD is also very good.
But that balanced coupling book is really great.
He's also developed some skills in Claude to take some of those ideas.
So definitely, definitely take a look at those.
I think if you're looking at things architecture and you think in terms of modules.
For me, I don't agree with everything in Mark and Neil's book.
There's lots of great stuff in there.
But for me, it actually all starts.
I sort of mentioned earlier that writing the second mission of building microservices was really all about smuggling in, getting people thinking about single information hiding.
It's a very hard idea from the 1970s.
It still comes back there to me.
Modules are still the whole thing for me.
So I would read Vlad's book.
And if you want to go read the original Parnas papers from 1971, 1972, they're still worth reading.
But those would be good jumping off points, I would say.
You can still, I think the nice thing about thinking in terms of modules is even within an existing code base, you can start to modularize it in some small ways.
You can start to think about boundaries, you can think about clear separation, about using encapsulation as a way of information hiding.
And I guess it's fair to say, like, look, if you have an idea on how you can make your existing code base a bit more modular, you have zero excuse not to do it.
I mean, we don't trust agents fully, but this is a textbook example of like, if you can say, here's my goal, refactor this, like, If it would have taken by hand X amount of time, it will be faster.
So you should be empowered to at least experiment with these, right?
Especially if you've got a decent test suite.
And if not, start with that.
Start with that.
If you've got a statically typed language, you don't even need to, just a good old fashioned IDE does an amazing job, right?
Dynamically typed language now don't have the excuse, especially if it's a modern one that's well understood by the NLM.
So if you're using, you know, Python or Ruby, it's so much better now than it used to be for refraction as code bases.
Absolutely.
You need to get a test suite in around that.
But yeah, you can start modularizing.
And that's about building.
And that's also a little bit coming back again to taking, we don't want the architecture only in your head, right?
We want the code to sort of represent that mental model.
Because we communicate through the code.
Maybe in the future, we're communicating through the specs.
And when you make those things more explicit.
then I think that helps everyone stay on the same page.
There's George Fairbanks wrote a book.
I've forgotten the name of the book now.
But in it, he has a pattern called an architecturally evident coding style.
And that we should code in such a way that it clearly communicates the architecture of the system.
And I think our programming languages are quite bad at having a good concept of modules in general.
So I think we have to go a bit above and beyond ourselves.
But we could do more to make the architecture of our systems that little bit more explicit.
This is also why I like spending time production, because especially if your modular structure ends up being a process structure, in other words, it's process boundaries.
I can go look at the networks to find out my architecture.
I shouldn't have to, but at least, you know, I know production is truth.
It's like football is life, production is truth.
So where can people get your book?
You can go and order it now.
If you can, go and buy it from a local bookshop.
That's always a great idea.
It's also going to be available for people on the O'Reilly platform in sort of digital form.
So you can go and read it as part.
If you're a subscriber at O'Reilly, you can get it there as well.
But lots of people love having the physical copy.
And we have these in the show notes below.
So check them out to get the book directly.
And also, you've been doing some videos, right?
Yeah, well, I've been doing some stuff on Dave Farley's Modern Software Engineering channel.
It's a kind of a good excuse to...
do some sort of semi-ranty, you know, monologues to video and have some fun chats.
So yeah, go for the Modern Software Engineering channel on YouTube and you can find videos from me and Dave Farley and Kevin Henney's there.
We've got stuff from Kent, Trisha G, Emily Borch, Dan North is there as well.
I'm pretty sure I've forgotten somebody else.
There's almost a couple of people there as well, but we try and put videos out once a week and it's a bit of a fun place.
And we're absolutely looking at that one.
And if people would want to work with you, where can they find you?
How can they find you?
What kind of stuff can they get help from you for?
I do like internal talks with people, like internal events.
I do like customized training courses and a whole variety of things around resilient systems and security and microservices.
I also do a lot of consulting for clients, helping them around sort of re-architecting their systems and stuff like that.
If you know more about that, actually go to my website, samnewman.io, and you can contact me there.
You can also find me on LinkedIn.
But if you come straight to the website, use email.
I'm still a bit old-fashioned like that.
And I work with clients all over the world.
So if I'm able to help, yeah, head over to samnewman.io.
My contact details are there.
Wonderful.
Well, Sam, thanks again.
This was great.
Thank you.
Thanks very much to Sam for this conversation.
I really like Sam's three rules of distributed systems.
One, it takes time.
Two, sometimes the thing you want to talk to isn't there.
Three, resource pools are not infinite.
Sam said how most of the outages happen because of this last one.
resource polls not being infinite, and services are getting saturated.
But I was chuckling when he was talking about rabbits stirring through networking cables between office buildings.
Then again a few weeks ago, with thousands of flights stranded on the east coast of the US, it was because workers accidentally cut a telecom cable in New Jersey used by the FAA.
Another food for thought from Sam is how failure should be handled differently depending on the type of business.
An e-commerce shop might decide to show you the item when the inventory service is down.
Worst case, after you order and pay, they just refund you because we're like, oh, we don't have the item.
Basically, failing open is fine in this case.
But on a concert ticketing website, you don't want to sell a ticket that might or might not be there.
And so failing close is the right call to make.
Because again, as we discussed in the episode, if you sell the ticket, people might book flights, they might get upset, you might lose them as customers.
Finally, I appreciated how level-headed perhaps even cautious Sam is about AI.
He's got no doubts that AI is very useful at the micro level, basically a tactical, things that we can already improve.
But he is worried about the economics of the model providers.
His suggestion is pretty pragmatic.
Hedge your vendors and your models.
Be multi-vendor and multi-tool and wherever you can, swap non-deterministic AI agent workflows to deterministic ones.
Basically have LLMs generate deterministic code, which I find very, very...
refreshing thinking.
Check the show notes below for the story about scaling Uber in the early days with Uber's first CTO, Twan Palm, that we discussed and it's an awesome story.
And for more, the pragmatic engineering deep ties on resilient systems.
If you enjoyed this show, do subscribe to your favorite podcast player.
And thank you very much if you recommend the podcast to your friends or your colleagues.
Thanks, and I'll see you in the next one.