# Durable Computing: Resilience for Distributed Systems

**Podcast:** Thoughtworks Technology Podcast
**Published:** 2026-03-05

## Transcript

Hello and welcome to the Pulp Perks Technology Podcast.
My name is Alexei.
I am one of your regular hosts, and I'm speaking to you from São Paulo in Brazil.
And this time around, uh, we're here to talk about durable computing.
And I am thrilled to have with us uh Brandon Cook and John Coleman to help us navigate through this very interesting topic.
Hello to both of you.
Brendan, maybe you would you mind introducing yourself?
Yeah, I'm Brendan Cook, principal software engineer at ThoughtWorks, um, based out of New York.
Excited to chat today.
It's amazing to have you with us.
Thank you so much for joining.
And how about you, John?
Hi, yeah, my name's John Coleman.
I'm from the Bangkok uh works office, and uh I'm a league consultant.
Amazing.
Thank you so much for being here with us to talk about this.
And so it's maybe we can get started by talking about some some motivation if you if you don't mind.
So I mean, the foundations uh of the topic go back to the 70s, right?
So acid properties to face commits.
Uh why are we discussing these things now in the context of uh distributed systems?
So what has changed or or what is relevant to the topic at the moment?
I got started in the sort of durable computing space after uh uh an assessment with a client.
Um, they were focused on oh, are we building the right event-driven architecture patterns in our system, right?
Right.
And what we were finding is that they had all the good patterns and principles nailed down for like more or less the happy path, but we're really lacking in sort of the the sad paths or the failure paths, and then that's kind of where a lot of the durable computing comes into play, sort of building in those sort of complex event driven patterns like event replay, recovering from different failures gracefully, um, being able to sort of continue things um in your distributed system if something goes down.
That's kind of like the the main key outcomes that you get from durable computing, offloading all of that operational burden off of your individual teams so they can focus on the design of the actual like the business domain, business functions rather than having to deal with known patterns across the industry, right?
Yeah, that's cool.
That makes a lot of sense.
Yeah.
So well, when and the architectures are becoming quote unquote more dig just more distributed in a way.
So more microservices and and and the scale of those kinds of things.
So but maybe before we go any further, maybe we can try to uh explain to the audience or define uh uh the concept itself.
Uh if we go back to the question, what is durable computing, how how how can we explain that in a in in you know in a in um in a simple way, maybe.
I think there's some slightly different angles on it.
When we had some discussion before about this.
For me, uh I I tend to focus on the state part of it.
So it's this ability for a program to recover its state and continue uh from where it left off.
Um and that it it it depends on the implementation.
There's different ways that that's achieved.
Um, but essentially it's that ability to recover the process and continue from where it left off.
Um usually to guarantee the process completes.
Um the typical uh sort of application, if you think of a workflow, is that you might want to um call various systems.
So you you might have two or three other APIs that you want to call, something like that in a sequence or a chain, um, and you want to guarantee you get to the end, and that uh also the chain of events is um correctly handled at each step.
So it's a lot about uh you know coordination and making sure that something either happened or doesn't do any harm, uh, and that you can have those kinds of guarantees, right?
In a distributed context.
Yeah, and it it can be more fine-grained than that.
Um, some of the platforms have more sort of superpowers.
So when we talk about the state, it can be there's there's a levels of granularity that you can have around that.
So some of the platforms are much more granular and can really recover the program and its internal memory uh stuff um i more precisely.
So they're much more fine-grained.
Some of them are a bit more uh basic, and uh they just record things like the the effects and what were the results of the calls, and then they just start the whole process again, but would give you the effects of the calls, something like that.
So there's a there's a uh a broad array of how it's done.
But it's all aimed towards the same kind of uh of outcome.
Yeah, that's that's that's great.
Thanks, John.
And I mean, just just for uh more for the sake of of clarity, what what kinds of guarantees uh are we talking about?
How how how is this different from you know a simple orchestration, for example, when we think about a workflow?
And or when we compare to sagas, for example, how how is it different from from those kinds of patterns?
Well, I think there's some overlap.
You'll you'll find similar to a terminology, like uh you get assured delivery, um, and the sort of the once-only uh you'll you'll find similar terms bounced about.
So I think the the concepts are nothing conceptually particularly new about this, I think.
It's just the power of the platform to take away the pains of of how you handle failure and retries um and recovery.
It's basically taking the the pain of those uh typically programmatic solutions and putting them on the platform.
It's essentially like all these teams and orgs that have started these sort of their durable computing platforms, having the realization that in all distributed systems, we have to solve these type of problems.
And they're essentially just focused on uh extracting that away for users.
Yeah, nothing new, right?
Like these are known problems, known things that you have to solve in distributed systems.
It's not a matter of if you're gonna face that problem or not.
It's more of a when.
And that's where these these platforms are really focused on saying, okay, this is going to happen, this failure is going to occur.
But we're here to help you recover from that.
Yeah, if you think about the history of like how people are doing these kind of distributed systems, quite often if you're if you're writing the code, you actually don't really know what to do when something fails at a particular point.
So you you might just cancel the whole process and the user gets a 500 error on on their API or whatever.
It just it just breaks.
And it's quite challenging as a programmer to figure out what exactly you have to do.
You may not necessarily know exactly how to solve a failure at a particular point.
But if you could sort of have more assurance of of the delivery, um, that that makes your life easier as a coder as well.
So if we're developing these kind of fallible systems, um you're you're you're taking a lot of the coding headache away potentially.
Yeah, and it's interesting.
So, Brendan, you were mentioning that you came across it when you were doing uh an architectural assessment that looked at platforms.
And uh John, you were talking about some of the origins.
It looks like, I mean, it was an emerging challenge in the industry and in based on the technologies we're using these days.
So we saw platforms coming out of Uber, Airbnb, Netflix, uh, and then Project by Apache.
So it's interesting to see you know that uh conjunction of factors that led to that, isn't that right?
So yeah.
And funny enough, after that assessment, we were able to sort of move on and sort of create a platform team that was focused on building out a lot of these common capabilities.
Yeah, one of the first things that we did was basically try to assess a variety of these platforms.
Um, and there is there's quite a few and a lot of considerations that need to be taken into um into account.
But yeah, like it is essentially a way for you to not need to have a set, maybe you still have a platform team that knows how to host these tools or um help tea enable teams on how to use those tools in in the most effective way, but then they don't also have to go build all of the platform and orchestration components that are are in the in these tools.
So as before, you can see like all these c organizations who like who came out with these platforms.
If you take like temporal for example, I think that came out of Uber's sort of cadence um platform, right?
These are all startups or organizations that realize this built-in resiliency into distributed systems was needed, and so they built it, and then now they had to maintain it and run it.
And that in and of itself, it's its own product, and it becomes its own beast of itself, just to say, okay, we're building resiliency into the system.
So now it's like, okay, how do we take these platforms and and I guess in a way democratize it and share it with with the rest of the industry.
I worked on uh ACA actors uh quite uh quite a few years ago.
They have they have this sort of replay capability, so you can persist the state as a sort of a stream and a log, and then they can they can replay that to recover.
So I think I think that's something that's been around quite a long time now.
Um, but it was never I I don't think anyone called it doable computing at the time.
Because it it isn't quite that, perhaps.
Um but yeah, there's there's been these ideas, and like you had we mentioned the database before as well, you know, your acid transaction, you can commit and roll back all of that.
So this there's this that there's I think there's been a progression uh uh to some extent of these developments, and now we've we've kindly kind of arrived at the point where it's pulled together.
You don't have to sort of conform to some sort of model or before it was like various pieces, but now it's just like we can do the computation now, we can just write some business application code and we get the guarantees.
We don't have to be thinking about these different bits and pieces and and making them work together sort of incoherent before, and now it's it's all formed into one easier to use uh way of working.
Yeah, it's funny though that you mentioned like Aka, because I think it highlights a lot of key context around some of these platforms, whether they like are like more heavily opinionated and heavyweight or more lighter weight.
So I would say like ACA probably is more in the lighter weight area of the world because it is more of a framework that you can build into your code base.
But I think the the organization that's called Light Ben that builds Aka, they also had like this more heavily opinionated system called Calyx as well that does this sort of they don't brand it as a durable computing platform, but it does kind of the the same principles.
But it was like once you're locked in and you're building in Calyx, that's the way you need to do it.
Similar with other kind of like event-driven platform y type experiences that are trying to build resiliency and like axon or anything like that, like some of those are like once you start building in those, you're kind of like stuck into into that way of of working and that way of designing the system.
So you essentially got to become expert in that in that framework, whereas the newer ones are are starting to become a little more flexible in terms of how you want to work and what language you want to work in and and being a little more lighter lighter weight in terms of that but still providing kind of that key um durable execution back end for you.
Yeah the lock in aspect is is interesting.
I mean even even uh cloud providers also have some of their platforms like AWS have step functions and Azure also has some flavor of that so that is definitely one key trade-off to to keep in mind right yeah like for the the org that I was working with on the assessment like they would like they were really itching for AWS to kind of build like durable lambdas.
So like they were like we really want this it wasn't out yet but like I think it was like last year they announced those durable uh lambdas at uh I could probably uh reinvent or something like that.
So I'm I imagine that that org is probably looking looking at those now because they were very much very much embedded and locked into AWS.
But that was a decision they've made from a tech technology strategy.
And they were quite effective at at deploying and and working with AWS.
So it was kind of they were like they saw AWS as their platform of choice.
Yeah.
And I mean, that's always a trade-off, right?
You're you're you're deeper into a platform and you can leverage more of the resources the platform will offer you and be more efficient.
But maybe uh I'll ask you this, friend, and then John, feel free to to to add.
But uh building on on the experience you had doing an assessment and looking at this from you know, making those architectural decisions and and considering trade-offs and those kinds of things.
So, what should teams evaluate when looking at those technologies?
Uh so what are some of the you know the key dimensions that are relevant, some of the questions uh you you need to ask yourself when thinking about these kinds of things and making those architectural decisions?
First, I always look at like the the hosting model.
So, like where are you essentially gonna host this this durable computing platform?
Because it is literally going to store everything that you execute.
Like some of them will literally store everything that you're executing.
Enterprises, like I don't know, financial systems and things like that, they're not going to want to use the the SaaS version of these tools.
They're gonna want to self-host.
So the operational burden of yes, you don't have to build it, but now you still have to host it and make sure that it's running.
There's some operational cost to running that infrastructure.
So looking at the hosting model and understanding what you need to do.
So if you're a startup, you're trying to get something going, you want to build a resilient system, maybe that SaaS product is something you would reach for quickly.
But if you're trying to maybe internalize looking at the hosting models, so like looking at like I guess a temporal versus a restate from a platform perspective, maybe those folks are kind of reaching more for restate because it's like a single binary, it's easier to deploy.
Now I think temporal is making strides to make, I guess, their stuff holstered choice a little easier to host, but I'm not sure what the developments are there.
But it's it is part of the decision criteria that you would think through.
Another thing I would sort of assess is like what languages do your teams know?
Like how do they actually develop on the day to day and what's supported by the various platforms?
Because at the end of the day, you're going to still need to understand like what are these SDKs look like that that actually integrate with those platforms and what do they look like?
Are your teams going to be able to understand them?
Like, are they going to be able to sort of understand, I guess, the web workflows and all these different um facets of like the idiomatics aspects of all these different SDKs and languages?
So I think that's a probably key key aspect to consider.
And then yeah, like what workflows do you have?
I think really understanding the business domain, obviously, is as a first quest principle around like sort of just still doing domain-driven design and saying, okay, what workflows do we have?
Do we have a bunch of long-running processes?
Do we have a clear district distinct workflow?
Do we have workflows that kind of fan out or fan in to certain things?
I think all of those have to be considered when sort of choosing which one of these durable computing platforms to reach for.
Yeah, and then there's you can dig into the like details of uh you mentioned the idioms.
So although there's a lot of overlap between them, uh, they can have slightly different idioms and slightly different applications.
Some of the platforms cross over a lot.
So you you've definitely got like things you can choose from.
And some of them are a little bit more distinct and have particular use cases that they might be better for uh versus other ones.
So you you have to dig in and understand uh what it is the the platforms are offering and how how that fits your use case.
They're not all gonna be sort of equal.
In fact, they're all quite different on the whole.
Yeah, that's great.
That's great.
And uh is there uh obvious scenarios in which you should not be using durable computing?
Are there a couple of factors to consider to say, hey, they're not needed in this case?
So yeah, I think like you gotta think about what what your scale looks like, what's your uptime that you need to deal with?
Can you deal with some failures and recover without hurting the business?
Right.
I think these are like I know we always like put these in like the heavy technical bucket, but at the end of the day, they're very much business driven decisions.
So if you need high like high scalability, high recoverability, all these different aspects, maybe the durable computing platforms is something you need to reach for.
Um, if not, maybe it's a little overkill for your system, right?
Um, and you're you're spending a bunch of money that you don't need to spend.
Yeah, cool.
And any considerations to testing.
So I mean, it's uh it's obviously we're uh the platform is uh we're leveraging the platform to do uh to bring a lot of resilience to the workflow and those kinds of things, but anything that we still need to be mindful of, uh, or other scenarios we need to consider.
So how how how does uh that change uh the testing strategy overall?
Yeah, I mean the testing strategy I think changes significantly, particularly like when if you're just like uh trying to think about what does a normal unit test look like, or what does a what does like a full test of the whole program look like?
Because are you gonna spin up all the infrastructure for your tests, right?
To get the feedback.
Um what testing support do these platforms provide you is also probably a key decision criteria.
Uh last thing I last thing I remember is temporals, like testing support was was was growing significantly.
Um so like yeah, getting that fast feedback, so you don't have to spin up the whole infrastructure to really make sure something is working.
Another key aspect is they made these these systems may look like they're like when you're writing the code, it may feel like you're writing synchronous code, but like at the end of the day, it's still distributed.
It's all async, it's all event driven underneath the hood.
Maybe you don't need to deal with the event-driven pieces a lot.
So that also has the an effect on on how you wanna test or how you approach testing as well, because it is a definite a mental mind shift to test synchronous code versus uh more event-based or asynchronous code.
Yeah, I'm just thinking back, I I did a sort of proof of concept with uh temporal, and the way I did that is is use it just using doc containers.
So I ended up having to have a container for all of the external services that I was um linking together in the workflow.
Um and you can imagine that quite easily becoming uh unmanageable.
If you had a substantial or you had an external provider.
Um I mean, those are problems you you're gonna face anyway, if you're doing integration testing.
Um, but now you you've added a bit more complexity to that story as well.
Yeah, good, good, good.
Yeah, well, well, you'll put the there there's there's the accidental complexity, but there's always the essential complexity as well that you can't remove, right?
So and that's that's part of the the problem itself and uh non sellsolid technologies we're using.
Once you start testing locally and faking things, you're in a sort of false world as well.
You you can't guarantee that you know at some point you really want to get to the production thing.
Um, but you know, you always have to to be about like progressive with your tests, don't you?
You you start you start with something local just to check the contracts and the behaviors, and then you sort of wishfully think it's gonna work like that in production as well.
You know, you hope for it it'll be the same.
But at some point you're gonna need those other services sandboxed or something like that.
Um yeah, you face a lot of the same challenges as regular end-to-end testing.
Yeah, good.
And I I mean, moving beyond testing, I'm also sure that uh, you know, that resilience, those things the platform brings don't they don't come free.
So there are probably you know uh things you need to be able to implement uh in in the code and and some gotchas that you need to be be mindful of.
What what are some of those things you in your experience uh you know yeah one needs to be very pay close attention to when when developing uh under uh those platforms?
One of the ones that that I've been looking at and considering is like latency.
So if you there's various points at which it might fail.
So some of the some of the ways that the dual compute platform works are like a full recovery.
So your process is gonna start from the beginning again now it it might be that your process um in theory all the effects the the async stuff should happen quickly when it's being replayed because it's being re replayed out of of some durable state some memory or disk somewhere so it it should progress a lot faster because it's not doing the IO anymore but that's that latency is still there so if you potentially had a very long or complex process or something with a lot of computation inside it that had to be replayed you could still have like uh an additive effect to your latency that could be quite significant.
Some of the platforms the more granular ones will recover like more of the memory state so they will actually genuinely continue the process uh where it left off so you wouldn't have that sort of replay latency coming into play.
The other kind of latency you can get is when the a process itself fails, a durable platform, a node fails, and then that that node has to be recovered um and given its state.
And that recovery process itself can take a while.
So that that would depend on uh like the the op log or whatever it is that's behind helping to restore the state, that that can take a while to replay as well.
And there's there's various strategies you can use um to try and tweak that on some of these platforms.
Um, but you are gonna get some latency side effects potentially, and of course, there's um resource overhead involved with that as well.
Another key thing is idempotency.
All of these platforms rely heavily on determinism.
So just how John was saying, right?
If something spins back up and is trying to replay, it's very important to have that idem potency built into the domain that you're building, um, as well as idempotency with any third parties or things of that nature.
Because the last thing you want to do is have something fail, it spins back up, and maybe a financial transaction gets processed twice because now you're you don't have any idempotency because the platform spun back up and replayed the replayed sort of the last sort of step that it was in action of doing.
So yeah, I think that's that's really key.
And it's very similar to like any other event-driven architecture, right?
Having idempotency as a as a focus, but I've seen in a lot of areas where like ident potency isn't isn't like considered, right?
People kind of just assume that, okay, yeah, it should be fine.
Happy path is working.
Oh, I didn't receive the event twice, it's been working, and then okay, when it does happen, trying to figure out or debug becomes a very sort of stressful, stressful nightmare there.
So it's like understanding what idempotency is, understanding it primarily on the consumer side.
Uh I've seen teams also focus on the producer side from the event, like from an event standpoint, be like, oh yeah, we built idon potency into our producer, and then the consumers downstream are just like, oh yeah, well, they built idempotency up there, so we don't have to worry about it, right?
So but like we all know that's probably not gonna happen, especially with events being able to sort of fire off multiple times um just off basis of those systems.
So really focusing on sort of consumer item potency and and and guarding yourself against sort of that maybe multiple events or multiple replays of the same thing happening and ensuring that the same the same thing happens in the state and the same thing occurs so the state maintains uh throughout the system.
Yeah, I mean, as as far as I know, you can also have uh long running workflows with a state, like uh months long.
And what do you need specifically regarding you know, perhaps backboard compatibility when a new version of a service uh uh comes up uh and and that workflow is faced with different versions of a service available and those kind of things?
How how how does that kind of thing work?
Yeah, that's I think this is where the these little platforms they're they haven't really built that out for you.
So yeah.
Really thinking about what that versioning looks like.
Because once you start deploying a new version out there, uh and then maybe some long run running process, like you're saying, is running, and now you have a new version, and then there's some obviously some some technical failure that occurs there, that that can be a problem.
Uh another thing where these durable computing platforms aren't going to save you is if you also have like a business failure as well.
Like if maybe how you're how you're doing how you're how you like perceived you were doing something is incorrect and now you need to compensate for those down the line, that goes hand in hand with the with the versioning aspect, right?
Like you have a bug in the system that we need to fix, or someone inputted data that was incorrect that we need to fix that uh affected downstream systems.
Yeah, the the resiliency of the platforms isn't going to save you from that.
So really considering that versioning strategy and how you're going to migrate over and understanding what processes are running in that workflow and how long they've been running to ensure that um everything sort of continues and works as expected.
And uh just curious, so what and from uh from more of um you know a developer mindset perspective, I mean uh uh I uh I I know many, maybe most developers are used to uh, you know, a request response style of developing.
You you you you get a request, you produce a response, and then that's it.
Uh everything that's related to the transaction happened within the the production of that response, and you see that as a unit, uh, you don't have to worry about uh other things happening in parallel.
But when we're talking about uh these long-running processes stateful uh with retries and those kind of things, the the the way of approaching and thinking about that shifts, doesn't it?
So what have you seen uh you know, uh both how how how had how has that shift been for you and what have you seen in in teams and and the way developers approach it?
It's definitely a mental like it's a mental mental model shift, obviously, from that more request and response type function to sort of understanding okay, this is event-based, this event triggered this, maybe now it's been long running for this process this long, then it kicks off something else.
So it's really focused in on rather than I guess looking at I guess a stack trace and more focused on okay, what what's kind of the event log or the history of events that have occurred, and that will translate into how you design the system, right?
So really fundamentally trying to understand sort of the the workflow interactions, um understanding okay, this is we can kick this off let it run for this time period and know that we're gonna still continue on at some point once it completes rather than always having to just wait for that instant feedback getting into debugging as well because though the debugging does feel quite different right trying to try to understand when a failure does occur or when something goes wrong how do I how do I actually debug it debug the system is is quite different as well.
Great great and maybe maybe the last the last topic I I'm quite curious about and wanted to hear your thoughts on this is uh so we'd be people have been talking about uh durable computing some of these platforms connected to uh agentic development and the use of agents what's the connection there why why are people talking about these platforms as uh you know enabling ai orchestration and those kinds of things so what's what's the connection what what what have you seen related to that yeah there's I guess a new I guess a new term, new technique uh obviously, with AI, there's a million new terms and techniques popping up every second.
But I guess they're calling it durable agents.
So you can see like um temporal restate.
I think even Vercell has their own durable workflow thing that is focused on on durable agents that they've released.
I think it's it's almost it's a convergence of the two technologies coming together while people are building sort of agentic architectures and making them also distributed potentially with multi-agent architectures and orchestration.
They're starting to realize that okay, what if what if I can't reach that LLM provider?
Or what if I can't search um the database for some rag operation?
How do I actually then recover when those things are available again?
I think any of the human in the loop kind of interactions uh that you have in those systems as well, because maybe it's in the middle of a workflow, but you're they're waiting on a human response.
That thing could sit for days.
Someone doesn't respond for days, but you don't want to have like that agent up and running just waiting and listening.
So with these platforms, you can just have it tear down, and then once someone responds, it will kick off that workflow again and spin everything back up.
So yeah, it's it's a very interesting space.
Uh, and it's definitely definitely emerging.
So yeah, it's exciting to see where where this goes.
But it should make the life of a developer uh a a lot easier around these kinds of uh agent-based solutions.
You know, you want to interact with those different systems.
Maybe you want to call a lambda function, and then you want to call the database um or whatever, and then you've got some APIs you need to call.
I would thought it's it fits very nicely into you know it's come along in a timely fashion for for the AI solutions.
Yeah, amazing, amazing.
Let's let's keep an eye on that.
It's definitely an exciting field.
So let's see, let's see how it evolves.
But coming to to the end of the episode, any any parting thoughts uh you want to share?
I mean, any uh, you know, ideas for the future, where is this headed, or if someone wants to learn more about these platforms, where to start?
So any anything you want to share before we before we close.
Yeah, I think it's I guess the the best way to get started is just to look up one of these platforms and start playing with them and and seeing how maybe they can fit into your systems, especially if you're building a distributed system and your team is struggling with a bunch of failures and recoveries, and starting to look and see if this is a viable option for you to maybe start incorporating into your system.
So I mean, we listed off a bunch of them.
There's free state, temporal, golem.
It it can be, I think it's quite overwhelming at the moment, definitely, but it's definitely a uh the explosion of these platforms is is warranted because they do they do fit a need in the industry, particularly around building distributed systems.
And is is there any one of those platforms, Brendan, that would be, you know, if you want to start start here, or they just they just fit different needs and they have each of them has their you know applicability and uh I think maybe the easiest thing, maybe the easy ones to get your head around, and maybe the most accessible, maybe one of the cloud platform ones, like with the Azure durable functions, or obviously the AWS um durable lambdas that we mentioned before.
Those may be the most accessible for you to play with and spin up and test out.
I think the other platforms, like the ones that we listed I listed before, provide a lot more of the bells and whistles that you you might need in terms of like observability and all these other things that um the cloud of providers haven't really focused on.
I mean, there's some stuff there, but there's a lot more tooling with with these other platforms.
So maybe it's it's worthwhile starting with those to get your get your feet wet and understand them.
But then once you wanna start adopting, start considering some of these other platforms.
Yeah, you can join us on the durable computing space uh as well if you wanna have questions or to talk about these things.
We we're trying to gather info together to help with these kind of processes of of how you might select the right technology, what the features are.
And I also put together a uh GitHub uh project which you can check out.
I did uh like a POC with temporal, so you can you can pull that and play with that.
It's quite simple.
Um I also had experimentation with Gollum.
Uh Gollum is is quite fresh and not I I wouldn't say it's particularly production ready yet, but it's a very interesting new player on the scene.
It takes quite a radical and different approach to durable computing than the more established solutions.
Um yeah, so you can have a look at have a look at the code and and play with it.
Um but yeah, of course you can always check the the websites and have a look at the platforms themselves on the websites they're gonna tell you everything you need to know.
All right, then I guess this brings us to to the end of this episode.
Brandon, John, thank you very much for joining.
It's been an amazing conversation, lots of fun.
Thank you very much.
Bye.