# Long-Running Agents: Infrastructure for the Agentic Era

**Podcast:** Chain of Thought | AI Agents, Infrastructure & Engineering
**Published:** 2026-11-12

## Transcript

This is, I think, the biggest directional shift in AI that we've had since, like, maybe TOS.
The definition of a long-running agent is they will live for months.
Building up their own context, every user is going to hammer your API with thousands of requests just to get the context it needs for the agent.
Geniuses with absolutely no memory.
And so it still means that what you build around AI models themselves is extremely important.
Welcome to Chain of Thought.
I am your host, Conor Bronsden, and today we are talking about long-running agents and the execution layer that's underneath them.
Retries, states, waiting on people, and what it takes to leave a workflow running for days, if not weeks, and doing that across devices.
There's so many challenges that come in as you get to enterprise scale with agents, as you look to have your agents perform more real-world tasks.
And we're here today at the offices of Ingest.
They are presenting sponsor of this podcast and of this episode.
Joining me in person today in San Francisco is Tony Holtstock-Brown, CEO and founder of Ingest.
Tony, it's great to see you.
Thank you so much for having me.
Really excited to be on.
I am super excited for this conversation because I think there is this sense that agents are changing the world, but the specifics of them can be very nitty gritty.
So Tony, you built Ingest before the agent wave started.
doing background jobs and serverless functions.
What did agents change around your approach and how are you thinking about the future?
Yeah, that's super interesting and a really, really, really good call out because I think Ingest started, like you say, before agents and it was all about workflows, a series of steps that run, making them durable, making it work so that you didn't really need to think about the infrastructure as you were writing your functions.
And it turns out that translated really, really well to agents because agents are just a series of steps.
And it might be a series of steps that recursively calls the same.
It might be a series of steps that calls an LLM delegates to sub-agents, which then run a series of tasks and propagate that data back to the parent.
But it's essentially the same thing.
So the mental model that we had prior to agents almost mapped pretty directly to agents.
And it meant that the transition was relatively smooth.
And we didn't really need to re-architect or build too many different things because, again, like essentially one long chain of a function.
Breaking that up into a series of steps was really good for us because then you were able to see sort of deterministically what your agents are doing at any point in time with the traces that you get.
And if you wanted to replay an agent's trajectory from, say, step 20 of a thousand step function, it's totally possible.
But the interesting thing is like the way that people used us massively changed.
And it turns out that like, obviously, if you build something for workflows, background stuff, it's typically really async.
If people are building agents, the first wave of agents, which are prior to the long run agent stuff we're talking about, very, very, very synchronous, very chat focused, less like asking agents to do something, it has its own trajectory or figure it out because the models weren't as good back then.
And so you'll be...
coaching the agent a lot.
And that was very, very synchronous.
So speed and latency was really important.
We had to put in some changes to make sure that latency was really, really low to make sure that workflows are running as fast as possible.
But apart from that, like the mapping carries over pretty directly.
And this is maybe one observation as well.
I think along the way of building with AI, there's a lot of new patterns and interestingly, a lot of old patterns that still map directly to building with AI that people may have forgotten.
which is really, really cool to see and cool to re-explore.
Good systems engineering is still good systems engineering.
Yeah, exactly that.
So we were pretty lucky.
I think it more is that you built the right primitives in advance and now you're continuing to be able to build on top of that.
Because I had a similar conversation with the chief product officer of Slack recently, Jamie DeLange.
And Jamie and I talked about the fact that Slack is a primitive for context storage is really excellent for coding agents.
And that's how this coding channel thing has spun up for them.
And that's why Anthropix ships more than 50% of their code in Slack.
And I think we're seeing that same thing with platforms like Ingest, where because you have set up to understand and ingest data the way that people need to in their day-to-day lives, when you see an agent fleet start to take that on as well, you're prepared and just simply need to scale up what you're able to do with it.
Crucially, though, I'd want to define one term you're using because I think it's really interesting and maybe it'll help level set the conversation.
Long running agents.
Where's the line?
It's a really, really, really interesting question.
And I think it's really easy to talk about two extremes.
You've got your sort of Google-esque type in something in the search bar and now Google will give you an AI response.
Absolutely not running, single shot.
The stuff that you've been doing since the early days of like GPT-3.
And then recently, we're recording this and yesterday it was Antropics duet that was announced.
You've got Grokbot, duet, Muse and so on.
And Dots.
Instinct, there's so many of them.
Yeah, Instinct like being one of the early ones, but it was like very invite only.
And I say early, that was like two months ago.
These are like personal agents that run 24-7, almost like Open Claw.
And these are like purely long-running agents that maintain state.
orchestrate sub-agents, basically go through, figure out which tasks you want done, whether that's your feedback or things you've set up for it.
And the definition of a long-running agent is they will live for months, building up their own context.
And this is super interesting because I think more and more, especially over the last month, we've seen that.
And the power is crazy.
One of our folks internally uses stuff like this to do a lot of his work.
And it's incredibly interesting just how fast you can go with this.
Let's talk a bit about how long-running agents are being leveraged today because I spent a lot of time in the last year engineering my contact system so that agents could hand off to each other easily.
And I think that a lot of engineers spent time doing this.
And now we have agents that are seemingly able to handle so much of this process for us.
What does it look like to use long-running agents today?
when it comes to the workplace versus consumer side.
I think like a few interesting threads to pull.
Number one, yeah, I think we did the same thing originally way back when we had like a deterministic layer over agents.
Turns out a bit of lesson gets us all, you know, like agents just get better.
So that's crazy and that's super interesting.
For us, I think like we see a few different things in terms of customer base and then I'll talk about architecture and then also just the market.
You've got people that are using agents internally, personally for their own sort of like personal projects or their own useful life.
And we've got a bunch of companies that are building long running agents on top of ingests.
And you've also got like instinct, news, that sort of stuff that's like come out and really kind of change the game for a lot of people personally.
And I'm going to, I'm going to position this as a difference to what used to happen with Claude locally in Claude code.
Claude code, you would run things on your one machine and you could ask it to spin up a sub agent.
Pretty cool.
And you'd be like, use as many sub agents as you want to review.
And it would do everything on your machine.
Pretty sick.
But if you're building something like DOTS or you want to build something like Duet, you end up with the central orchestrating system that's doing this for every single one of your customers.
And it's spinning up agents, managing context.
It has to do things in a sandbox.
So it's also orchestrating sandboxes, providing all of the context to and from each agent and its subagents.
And that orchestration is super important.
There's no way you can do this without the central orchestration system.
And that becomes, in a way, your product.
And I think like we see this from the leading labs.
It's kind of inevitable that this is the way that the world moves.
And so a guess is what happens with every other product.
Like Datadog is a really classic example.
There was this really interesting post on Hacker News this week about agents for monitoring.
And why could you not have a long running agent that also lives in Datadog that takes a look at every SLO you've defined and it takes a look at every one of your systems and it proactively reports, it proactively suggests different things.
That would be sick.
And at that point, you've got your personal agents that run on whatever system you use.
You've got the product you use that have agents.
Maybe they just chat with each other.
And the orchestration of these is like incredibly important.
I think a backdrop to all of this is how much growth we've seen in the agentic economy this year alone.
I saw a stat that in the last year, it's been over 34X in token usage for agents.
And that just seems to be accelerating as we roll out personal agents.
everywhere.
Are you expecting the way that companies are working with the NGIS platform to build long running agents to continue to change in the coming months?
And where do you see this Identic Era going over the next, I don't know, call it a year?
Yeah, yeah.
So on a personal note, we've seen a big shift away from UI to MZP and API CLI.
And that's like the majority of our traffic now.
Most of our customers actually don't log into the UI.
They use agents to check on whatever it is they're built, which is insane.
And the stats are that people that build with agents are far more likely to activate.
Like 93% of them activate versus a lower activation rate.
And even then our activation rate was super high at like 40% of people.
But it's insane.
So I think it's number one indicator that that's the way the world is moving.
It's much faster, much easier.
So I think over time, yeah, like what we'll see is a lot more people building number one this way.
And number two, just taking a look at the way people work with dots, the way that we've seen our internal work, the way that we've seen our customers work, the way that we've seen people build.
It's sort of inevitable that over time, you end up sort of abstracting the meaning of your tasks and becoming much more higher level.
And a bunch of people have talked about this on different podcasts, like you almost become a product manager.
of what you do.
I've said that.
Exactly, exactly.
Which is great.
And it's like so true.
Like I can't see it going back to the way it was ever.
And that means that over time, yeah, kind of expect that people build much more in terms of these agensic systems and it will only get better, which is really cool.
I've, I talked to Laurie Voss recently, who is head of developer relations at Arise and he is a co-founder of NPM and he calls it product engineering.
And I really liked that term where you are, the software engineering has changed.
You're still solving problems, but you're not a programmer anymore.
You are now engineering the product and the systems around it.
It's like a systems analyst from the 50s and 60s.
And to your point earlier about these ideas coming back, it's saying, yeah, we still have to engineer the problems.
We have to engineer the systems.
We're problem solving.
Just the abstraction layer has changed.
And I know you have surveyed some engineers and customers around this as well.
And a few interesting trends that I saw in some of the data you shared.
One is that There are quite a few incidents happening and people haven't quite solved these systems.
I think it was 74% of AI teams that you surveyed had a customer visible incident in the past 90 days.
That is concerning.
I think where that's, people see the wave of like cyber incidents and worry, oh gosh, are we exposing ourselves too much?
How do we solve the reliability problem?
Yeah, it's really tough and there's a lot to say about this one.
So I'm going to go from the engineering point of view and then we're going to go for the tracking point of view.
And then we're going to go for like a meta point of view afterwards.
Engineering point of view is really hard, especially if you talk about a central orchestrator for many personal agents that are long running.
Long running by nature means that things are stateful.
And it means that if you're doing a central orchestrator for your product, which you must, otherwise it's very hard for you to orchestrate long running agents for tens or hundreds or millions of users, reliability becomes a big concern.
So long running state orchestration.
typically means that you're going to be building with a lot of asynchronous systems, queues, state, durable execution, a lot of databases that are high throughput to make things work.
Observability becomes a pain and that's sort of our central underpinning of what we tried to tackle and just essentially state manage for you, queues, orchestration and so on just out of the box and you write a few steps, everything is done.
That is tough and it's even tougher when you consider you're going to need to spin up sandboxes for each of your users because if you're say for example Crogbot, you've got 100 million users on your platform, half of them are writing code because of cursor, and you want to be able to spin on things like clawed code.
That needs to happen in a VM, so then you're orchestrating infrastructure.
And infrastructure is like maybe the one immutable piece that doesn't really change because that's always going to exist.
Plug something into a DC.
So reliability is quite tough with this aspect.
And as things scale, I think we haven't quite understood the compounding nature of agents.
I think like, It would be very hard for one user to go through and make thousands of API requests on a daily basis without scripts.
And you would only have a few users that would go ahead and make those scripts in order to automate their data life.
But now every user is automatically doing that and every user is going to hammer your API with thousands of requests just to get the context it needs for the agent.
So reliability from the infrastructure point becomes a little bit more difficult because the extremes just compound.
That's quite challenging.
And therefore you need things like flow control, rate limiting, just to make sure that you're good.
And agents can deal with that pretty well.
If you rate limit your API, it'll just retry.
You know, it understands everything generally.
So you really need to bake in protections.
That's tough.
From just a what an agent does point of view, and this is both from a lab point of view and also you building your product, it's very hard to track outcomes.
And I think the classic way that people have been doing this for quite some time, and it's not wrong, but it's tough, is evals.
which are unit tests and a non-deterministic unit test over maybe a subset of all of your data in production and a non-deterministic user test over a non-comprehensive portion of your user base is very very tough to understand whether or not your agents are doing the right thing.
And that's not to say it's wrong, it's absolutely right because why would you not do unit tests but it doesn't track outcomes in production.
So one of the principles and things that we've learned from It was Kissmetrics back in the day who I don't think really do too much nowadays.
It's amplitude.
It's the folks, but like Kissmetrics learn this and AB testing learn this.
Track events and then track outcomes based off of what happens.
So for example, if you're at Amazon back in 2004, does the purchase button being 30% larger increase conversion rates?
Yeah, probably.
Track the events, see what happens and then track outcomes based off of what's happening.
That's like something that...
we ended up building directly into the platform because we have stepped up wait for event.
If you're building a coding platform and you're doing some sort of software factory, it would be good to track events to see if the review passed, to see if a user merged within some certain amount of time.
And if they didn't, you can use these events as a heuristic to say the agent did a good or a bad job because they're indicative of the quality of the work.
And tracking events is super cheap, dude.
It's existed from Kafka so easy.
So you can literally just consume all of the data from how your product works.
And then if you've got some foresight about what the agent is doing or what the intent is, track based off the events to see whether or not it did an acceptable job.
And that gives you a pretty good deterministic view on how agents are performing.
And it doesn't have to be sampled.
100% of your agents can be monitored, which is very different to...
taking an entire trajectory, throwing it through another expensive LLM as a judge and having another agent guess, did it do a good or a bad job?
Which is kind of biased because you're asking an agent that had the same weights to judge itself.
So that one's a tough one.
And I think that's an interesting point of view to improve AI reliability and to tell you whether or not generically things are performing, which is just like we've been doing for products.
management in year for the past few decades.
You've given me so many interesting threads to pull on here.
No, this is fantastic.
Making my job very easy.
I want to go to one very niche part of what you said, which is statefulness.
Obviously, one of the challenges that we are orchestrating around with LMS is that the models themselves are not stateful.
You know, they are as you know, when from GenSpark told us geniuses with goldfish memories.
And so we are building the system around them, the agents, the orchestration, the context, the memories to ensure that they can keep working for us in the long run and actually be more deterministic and solve the things we want to.
How do you think about the stateful side of ensuring these long-ring agents are more successful over time?
And what are the boundaries today of you know, how long can a long-running agent exist for?
What are the context challenges you're having?
Yeah, it's super interesting.
And I think it'll only compound the problem that you stated with context and state as long-running agents and personal agents continue to expand.
So firstly, I just want to quickly note that I think it's really, really, really cool that LLMs are essentially stateless and what you put in at any time can just continue as long as you have that state yourself.
So that's exciting.
I think it's like really tough.
And I think context management is one of the biggest problems in AI engineering, as everyone knows, it's no surprise.
So firstly, if you're having one particular conversation, it's pretty cool if you can do something like Dribbble execution, where every step is recorded, the inputs and the outputs are tracked, and every tool recording, every tool use is recorded.
So you can see which tools were generated by the LLM's response, what their execution was, what results they had, and so on.
If you get a fully deterministic system, it's pretty sick because then the state for one function is entirely captured.
And that means that if you didn't like what happened in one particular conversation, you can replay to a specific point in that conversation, maybe adjust some of the inputs and see if it would have taken a different path afterwards.
Quite fun for testing.
But then in the world of personal agents, you have this compounding problem of sub-agents and these long-running stateful tasks.
which I think you've seen with Claude code for the past few months.
Like it writes memories, there's a huge markdown file, files are memory and files are great.
And there's like maybe this balance between do we use something like a vector store or do we just save everything in files and have an agent grab for it?
Legitimately, I think for us, we see people write one parent orchestration agent, which uses essentially step.invoke to run subagents.
Subagents have the entire context of their own thread.
which they summarize and pass back up to the parent agent as you'd expect.
And from there, it's really important to be able to take the content from that particular conversation and extract what might be good to save that somewhere so that you can load that in the future.
I think like the tough part here is knowing what's relevant.
And so this just honestly means that you end up spending more and more tokens to figure out what was relevant and what happened in the past that might be relevant.
Then, I don't know, like this is more theoretical now.
you get into a problem of like freshness.
What was learned a year ago may not now be true.
Yes.
Which is really, really, really challenging.
How do you forget memories?
Yeah.
Yeah.
Yeah.
And that's really challenging to deal with.
So I think like there's still a few unsolved problems and this is like something that we're taking a look at because it's, it's really, really fun to think about.
Um, and my, my personal belief is that like a lot of people are taking a look at this from both the front of lab perspective, a product perspective internally, like us, um, personal agents are incredibly useful.
Very cool.
Extremely scary.
Everyone in Hacker News is going to talk about the security implications and I agree with them so very much.
But they're still really cool.
And so like what happens and what happens overall with a long trajectory of personal agents and what technology does that bring?
I think it's going to be like very quickly unfolding over the next like three to six months, which is like, I can't wait to see it.
Yeah.
I also think there's a marketing challenge here too, where We are at a state where I think most people in tech are going, oh yeah, personal agent makes sense for me, have these things I want to do.
But it hasn't necessarily broadened out and there is a need to communicate the value of a long-running agent to people's lives.
And they're starting to see that.
You're seeing ads for like, okay, let me help manage your family's calendar, that kind of thing.
But it feels like there is a potential wave here if the broader population starts to unlock and go, oh, I do want that personal agent long-term.
Yeah, yeah, totally.
And like, I guess this is one interesting thing.
And then like, you've had companies like Apple with Siri, take their time to roll out a lot of AI.
Take their time is a nice way to put that.
I think that's like, actually, both really good and really bad.
But if personal agents get to the point where they're unilateral success, which is possible, and it must be the way that AI progresses.
then like it does become just a branding thing and an accessibility and distribution thing.
I think like a lot of the times companies are really a distribution game and technology is a lot of distribution.
If you've already got distribution, like for example, WhatsApp or Siri, do we really even need to brand personal agents or is it just like, hey, you can use Siri for everything now?
And that's like a general question I don't know the answer to.
I think like this is where tech just sort of builds on itself.
It is very much a branding thing for people that aren't in our circles.
Like, I can't see my mom going on like GrobBots and being like, this looks so good.
Let me play around with this.
It's going to change my entire life in her seventies, you know, like that's not going to happen.
But if she were to use Siri and WhatsApp, like that totally could.
And I actually find the hardware push that Meta's doing with Muse really interesting here too, where they're saying, we're going to be cute or going to give these interesting hardware devices.
You can have a Tamagotchi style Muse experience.
I'm very curious to see how that lands with older populations and younger folks.
There's obviously a lot of AI pushback outside of the bubble that I think we're in.
So it's going to be a fascinating challenge for companies like Meta to market this to the broader population.
But there is a massive opportunity, as you point out.
And I think we're seeing with the huge increase in agentic usage for not just coding, but now like motion graphics, all sorts of use cases, that we are going to continue to just solve things in a way where agents can now do so much more than a year ago, let alone six months ago.
The leaps are massive.
And so I have no reason to expect that we won't solve this in a way where everyone starts to get access to an agent.
I think the trajectory of AI is such that it keeps getting better.
As like the orchestration systems around context and the building of engineering that we do around AI becomes better and the models get better themselves, what we can do with AI just kind of compounds.
And that's really, really cool.
I can animate these episodes now that I have no animation background.
Insane, like absolutely insane.
And engineering becomes much more of a code factory than it does like engineers writing code.
I was on a screenshot of some of my engineers yesterday and I was writing the interfaces manually and then asking Claude to kind of help me once I'd got it all done.
And they were like, what are you doing?
This is insane.
Yeah, like, dude, grow up.
But like, as we build better systems, and that is a lot of engineering around the AI system and context overall, it just becomes really great.
And then I wonder like, yeah, there's, AI pushback.
But once AI is so good that it just does the thing, I don't even think we're going to be branding it as AI anymore because it just becomes the norm.
And I think people's pushback just disappears because the expectations rise.
Like you go back to the 90s and you were on Dialog, you opened up Windows 3.1 on 95 and it was just taking forever.
You could hear the hard disk spin up.
That was normal.
In turning on your computer, you would be okay to wait 120 seconds.
I would never do that today.
Never.
Frustrated.
Like if my MacBook doesn't turn on, I never turn it off.
But if it didn't turn on in two seconds, I'd be so annoyed, which is crazy.
But the collective experience of everyone, expectations are going to change.
And people could theoretically push back on AI.
But I think like as it gets better, people are just going to get so much more used to it.
And I think that's ultimately why I think like the engineering behind AI and how you build product is so important.
Because if you don't meet that expectation that will be there in a year.
A question is like, it becomes like the whole Salesforce versus CRMs back in the day in the nineties.
Like, are you building something that people just no longer want to use because they're not going to?
I think this is why one of my, my favorite podcast today is how I AI by Claire Vo, who's a past guest on here, but just explaining, okay, here's how the people I'm talking to are using AI.
Here's how we can leverage it.
Here's how I'm leveraging in my day to day.
And I think there's a lot of work to be done on that front of just, helping encourage people to be ambitious because the model capabilities are advancing so fast.
The orchestration, the work from folks like yourselves around enabling those models to be long running and stateful have massive compounding advantages.
And we can just simply do more than we could a week ago.
Can we cut a month ago?
Then we cut two years ago.
I mean, like this takeoff has really only been like what, four years now ish.
And.
I mean, I look at the like six months ago AI posts and it's just insane what's happened.
Cloud Code came out what last year?
Codex was February of this year.
And I just don't see a world where barring a solar flare, wiping out all technology, we are going back from this.
The tools are just better than they used to be.
And there are tipping points.
We're seeing it with engineers where engineers who were like, I'm not going to use AI, I don't like this, have.
gone.
Actually, this is really good now.
We're seeing it with animation.
We're seeing it with people, with marketing.
I don't think like writing, solve or something.
We're seeing with math, where math has just completely changed.
I think we're going to start to see it with drug discovery and bioscience, though there are still major regulatory hurdles there.
And I would caution folks to think carefully about how fast can that actually move.
It just seems so clear that the better lesson is going to be applied over and over and over again.
And that's super interesting because to bring it back to one of your points, geniuses with absolutely no memory.
And so it still means that what you build around AI models themselves is extremely important.
And so you might argue, and I've heard this from different people, where's the edge?
Building a company and everyone has access to frontier models.
Why is my company going to be better than somebody else's company in the same category?
And it turns out that like, yes, you have access to the same models.
But the way that you manipulate that model, the way that you store context, the way that you guide it could be very different.
The experience you create, even if it is a set of APIs, can still be fundamentally different.
So I think like the engineering around AI still becomes the most important thing, even if you're using Claude Code to do that.
Like it doesn't change the fact that you still have a bunch to differentiate on.
Let's get into some of the nitty gritty of those decisions.
I've seen you argue.
for explicit step APIs over compiler directives like for cells.
How are you thinking about some of the specific infrastructure decisions around APIs and other aspects?
Yeah, I'm going to go back to our principles and then I'm going to talk about the future of how things might work, especially with confounding AI.
Our principles when we started the company were meet you where you were at.
And preface this by saying I very much like Cloudflare.
It's the opposite of Cloudflare.
Cloudflare, like, we've built some cool systems, you're going to have to re-architect to build on workers, and that will solve all your problems.
Which is great, workers' platform is pretty sick.
We were very different in that you could drop your SDK in Python Go, TypeScript, run it on Lambda, run it on GCP, run it on EC2, it doesn't matter.
And it would just work.
So meet you where you're at, be very flexible, and have it do the right thing based off of an SDK.
And that's quite cool, because then it does just work anyway.
So our principles are...
If it's simple and amiche where you're at, then generally speaking, it's a better experience for both humans and agents.
And that's super important going forward.
Secondly, it shouldn't do fancy stuff and it shouldn't be surprising because the more surprising it is, the more thought both you and your agents have to put in to understand things, which might be short-circuited with a lot of the training that happens with models, but it's easier to be simple.
For us, I think the Step API is good because...
It gives you a clear definition between transactional boundaries in your code.
That's great.
But it's also important because, for example, one of the important things in our system is step.waitforEvent.
And that's actually used under the hood a ton.
For example, if you've got one parent agent and it orchestrates four sub-agents, each of the sub-agents need to tell the parent agent when it's finished.
And in order to do that, there needs to be some mechanism.
The typical way of doing this every time you are building a system, was to store an entry in a database, like a signal, pass that signal down to the sub-agents and say, tell me when I'm done.
And that's terrible from an engineering point of view because it's just direct coupling that sucks.
It's very hard to change.
Step.way for event inverts the model where an agent just says, yo, I'm done.
And you set up a signal to say, anytime an agent is done, if it matches this particular pattern, if it's like this agent ID, let me know and I'll automatically resume.
That's really cool because if you're doing a classic sort of RPC thing.
When you are doing an e-commerce payment, you don't need to know the 10 downstream workflows that need to run when that payment finishes.
You just need to say this payment was done.
And so the primitives around how we built our thing are very, very different to say, like, put it in a compiler.
The primitives carry over from any language on any system.
And it means that if you rewrite something from TypeScript to Python because you decide you want to be more AI forward, your functions will pick up where they left off because it's the same step IDs.
You switch from AWS to GCP, you do a cloud migration across regions.
Cool, it'll pick up where it left off.
So less surprising, maybe a little bit more powerful, but less magic.
The magic is cool.
I just, I don't know, when you've been doing this for so long, I don't really love magic anymore, you know?
What about the human element?
So you mentioned trying to build simplicity for both agents and humans.
And I think that's smart, given that agents are increasingly the drivers of many apps, the people who are using many of our tools.
How do we ensure that human judgment is being ingested and pointed at the right directions?
Yeah.
Yeah.
I think this is like very, very, very tough overall.
There's a few different levels to this, the engineering level or the architecture level and the overall direction from an engineering level.
I think like some people care less and less, we care more and more.
Some people care less and less, and that's totally fine.
You know, I think there's a view if you're, especially if you're building something for yourself.
If it does the task, does it matter?
For us, if we're an infrastructure company, well, yes.
But if you're building your own personal project, maybe no.
That's arguable, but maybe no.
From an engineering point of view, though, the simpler, the better, because it's just easy to understand.
Like you're saying, general software architecture principles.
And like the stuff from Clean Code and whatnot back in the day, good book.
I think it actually got some flack over the past four years.
But still, interesting book back in the day.
All that stuff is still relevant.
From a directional point of view, my biggest problem with agents are that agents will nitpick on everything and give you an infinite amount of work to do.
And this is the toughest part.
You go from something like a reptile or a code reviewing bot, nine times out of 10, you're getting a four out of five.
And the four out of five is because of some crazy edge case that absolutely won't hit.
And it's maybe because like you were writing something, you had a database migration, you adjusted the database migration in that same PR.
And then it was like, this is going to fail because once you deploy this, all of the previous stuff is going to change.
I'm like, you haven't deployed this.
Let me push back on this a little bit.
So we are seeing that AI is so much better at chaining vulnerabilities than ever before.
Should we not be much more careful about these bugs?
I totally agree.
And I think maybe the framing of like the nitpicks, the edge cases that would never happen was maybe bad.
I think it's like, if you would like to improve this API, here are eight next steps you can do rather than like, Here's an edge case.
You have to do this.
Yeah, exactly.
It needs to be, here's our...
Bad about that.
Yeah, I totally agree with you.
It's just, I think like, yeah, sure.
Handle the edge cases, make sure that things are good.
Handle item potency, make sure there are no conflicts, you know, do things transactionally.
All the standard stuff applies with software engineering.
But once you've built an API, it is like, here are eight things that you could follow up with.
You have to have an endpoint.
You have to have an endpoint and agents are really good at just layering on more and more.
I think like, if you take a look at, I don't know if you've used like, chat or Gemini on your phone.
Every time you ask a question, it will follow up like, would you like to learn more about that?
It's like their training where they're like, oh, here are eight more things you can do.
And to answer your first question about like the specifics about like judgment and so on.
I think we need to be really, really good at figuring out the intent of what we're asking and when we're done and the direction.
And that, uh, I think that taste the intuition, um, intuition is a really big thing in our company.
The intuition of what to do is extremely important.
I don't think that I'll ever go.
I tend to agree.
We are at a point where agents can come up with good ideas based off of the corpus of information around them.
But human expertise is so important to drive that and to find the areas where they need to stop.
Yeah, yeah, yeah, yeah, yeah, yeah, definitely.
And on your note about security, like this week was crazy, like crazy.
If we're talking about long running agents, I think it's absolutely critical that we do this from a security point of view as well.
You know, like one of the easiest wins for people would just have would be to have a long running agent that supersedes dependable.
Please supersede dependable for the love of all that's holy.
Get rid of dependable and make it better and have something that just continues to improve the security posture of your app because the educators that you mentioned are like absolutely critical.
And if we don't do that, I think like the discourse around Hacker News and everything, everyone on Twitter is like, it's much easier to be red team than blue team.
Obviously entropy and whatnot.
Agree, get help, get help with agents.
Like it's, it's good and bad all in one because maybe they'll be misguided.
But honestly, if you're red team with an agent and it finds stuff, you can obviously red team as a blue team and then.
and help.
Well, that is just the blue team.
I've been experimenting with this just on my personal code base, which is not that production bearing beyond this show.
But having, in my case, I'm having like a DeepSeq V4.1 Flash agent that is just...
continually checking every day like where are their potential vulnerabilities and then passing off issues based off of that to coding agents to go solve and i think we're seeing that happen at scale at some companies anthropic released a great article recently about their code review process and they talk about turning bugs into evals and they talk about turning on these like security loops and i absolutely think every company needs to be thinking about like what is my security loop because we are going to continue to have so much more code And we have to ensure that we are beating attackers to the punch on it.
Yeah, exactly.
Exactly.
And then to bring that to the point of building long running agents, it becomes once again, the challenge between some event happened, code was pushed, we need to orchestrate a bunch of agents around that code, similar to code review, just to make sure that things are good.
And this needs to be a continual ongoing process, like the memory that you talked about, what is valid now, and goes in now, may turn out have a bunch of security issues if you have a dependency somewhere in your supply chain and the long-running agents come into play because you need to make sure that what was good now might not be good in three months time and that needs to be fixed even if it's not relevant to code that was being pushed today one of my favorite loops i've seen recently was i think reese sullivan on x posted about it it was hey, I want you to look at my app every day and figure out how to make it faster.
I just want to have an everyday loop of like, how can my app be faster?
How can it move better for both agents and humans?
And I think there's so many opportunities that we just need to have a long running agent that is checking regularly and saying, okay, this is my one job.
I'm going to keep solving this.
And then orchestration around that.
Yeah, exactly that.
And I think like to talk about the principles of how we build our stuff in steps one more time versus directives and so on.
A step is really just souped up tracing, to be frank.
Tracing is a step in sort of open telemetry that stores the output, then it reruns your function if it crashes, and each of the traces just re-injects the state that was saved previously.
That's an easy way to think of it, memoization.
But because you're breaking things down into steps, you get observability across every single facet of your function.
And if things were taking a long time and one step was slow, tick, totally easy to find out.
You can use insights to query for slow steps and then you can use an AI agent to go through and then improve the perf of your functions.
And it really comes down to, as everyone knows with AI, the data in, data out, just how much data are you getting from the things that you've built and how easy is it to continually improve?
And to that effect, I would love to add, we have this thing called like DruidLenpoints, essentially workflows, but in an API endpoint.
And if your API fails, it redirects you to another URL that blocks and waits for the retry.
We go and retry your API endpoint, pick up where it left off, finish the API result, grab that, send that to a user on the redirected URL.
But it would be sick if you could just add SLOs to your API endpoint.
And if it didn't hit SLOs, triggered something, goes through, looks at the stack of traces that didn't meet the SLOs, investigated the reason, and then automatically push fixes to your entire code base to make it faster.
Like that is the stuff that's totally possible.
No human really wants to do that.
Like no human loves the idea of, and there are really good SRE teams that are really good at this.
People logging onto dashboards every day, looking at an SLO and then being like, I need to make this faster.
And then going on an eight hour investigation to find the root cause is an entire job that is draining.
That an agent can just do this.
Like that entire thing around continual improvement is like totally possible.
And I think for anyone who is interested in problem solving today, there is so much that you can do.
And this is where I would encourage our listeners be more ambitious with what you try to achieve with agents.
You can do more today than you ever can.
And I want to come back to a point you made very early on in this conversation about using product signals to identify the success of what you're building.
And to me, this just speaks to the value alignment there of if you can build great loops, you will get these product signals as long as you can trace what's happening, run some evals, maybe not all of them in production.
I think you did a survey where you found what, 35% of teams building AI in production weren't actually using evals at all.
Just yellowing it.
Okay, models are pretty good nowadays.
That's a reasonable thing to get something out the door.
But you want to make sure that it's meeting the need of your customers.
Speaking of customers, I want to ask about platform scale.
I think something that people run into is the constraints of compute and the constraints of the systems you've built.
How are you handling customer queuing for these long-running tasks as your platform has scaled out to rapidly?
Oh, can I tell you a secret?
Please.
We ended up running out of compute.
I'm not surprised.
Yeah, yeah, yeah.
We run on multiple different clouds and we also have bare metal.
We ended up acquiring a bunch of new bare metal machines and then racking them ourselves with switches because computers just crazy hard.
Bandwidth in our game is really, really tough because we're doing a lot of model calls for our customers.
We're storing a lot of state and it's just public clouds have got this like hold on you with bandwidth that is very hard to break through.
So we did a bunch of stuff on bare metal and that's great.
We ended up building a multi-tenant system that works in cells essentially.
And that's really, really cool.
because it basically may stay cheaper for everyone.
Instead of spinning up like an NVMe disk per customer and then storing a bunch of state because state is essentially the key value store for each particular step that runs.
When a function run finishes, that state gets deleted.
Super high throughput, super high read, super high write, super high delete.
Worst case for all database systems.
Rinses NVMe's, goes through the entire drive write per day.
It's not fun.
Doing that on a per customer basis and single tenant is hard, possible.
And it's just easier for everyone involved, but it comes with risks because then you have multi-tenant architecture, a lot of issues around that.
We've been investing a lot in cell-based architecture, queuing.
We actually built on FoundationDB when I started the company and then switched to something Redis compatible, which in my opinion was awful.
But it was the right decision because we wanted people to be able to run it open source.
and FoundationDB isn't something that people want to run open source just to develop locally.
So our entire view of meeting you where you're at is that you can run in just in one command and bring up the entire production-like infrastructure in one command and then have everything run locally in exactly the same way it would when you push it to the cloud.
In order to do so, you really need not just a fake copy of FoundationDB, but the actual systems running on your service, on your machine.
Hence, open source compatibility with everything.
Even if FoundationDB is open source, having a prerequisite to install that wasn't going to work.
The cloud does work a little differently now behind interfaces.
There's a bunch of different changes.
FoundationDB is back because scale.
It's quite a challenge.
It's quite a challenge, but very fun.
Are there any particular learnings that you've had as you've gone through the scaling process that you would want to share with the audience?
Yeah.
And actually, The folks at PlanetScale talked about this with Neki just last week.
PlanetScale is just so good.
Yeah.
They talked about something with Neki and the things that they do with SQL passing in exactly the same way we talk about it with events.
Turns out that not doing work is the best if you're trying to hit scale.
Just don't do needless work.
So for us, let's take step.waitforEvent, which is so good as a user and so awful for us to run as a company.
Let's say you run a million functions, which is not that many.
And you do step that wait for event.
You're waiting for a million unique events.
Maybe a million unique users are doing something and each of those is a different expression.
Anytime you send one event to us, we need to check all million events to see if it worked.
And because you've got a million wait for events, you could assume that we're going to get a million events.
That's a million times a million, which means we're now doing trillions of quadratic expression matchings, which has to happen in real time.
Otherwise, you get huge delays on your function run resuming from step.waverevent, which is not what we want.
So the answer really is to break down the expression into a bunch of inverted indexes, see the expressions that come in, see which might match, filter out all of the expressions that might not be valid before you even evaluate them.
And then you can go to evaluating in what would take hours.
in a millisecond.
And it turns out that exactly the same as Plan to Scale, don't do the work you don't need to.
And then the standard engineering, you know, SLOs, take care of things that you really need to take care of from a memory point of view, make sure that you're not doing extra in your CPU processing, which is the same principle.
But Scale is tough.
The other thing I suppose would be like, even if you want to be extremely generous to your users, which we do, you still need limits in everything.
AWS has limits in everything.
You need to ask them to raise limits on Lambda.
You ask them to raise limits on SES.
Just everything should have limits lower than you expect and you raise them over time once you're safe.
And I think that this is even more important, bringing it back to AI, because agents are going to do things 10 times more than your customers would have two years ago.
Yes.
We simply have to scale more than we used to.
And Tony, I really appreciate you taking me through the approach that Injustice had to this scaling.
I think it's fascinating to hear about some of the perspectives and approaches you're using.
As we near the end of this conversation, what are the things that you would say to engineers, founders, and leaders listening about the best practices they should take forward as they also look to scale for the agentic era?
Yeah.
I think like this goes back to the conversation and theme of long-running agents.
The long-running personal agents are relatively new, but it becomes interesting and apparent that this is the way that people are going to consume other products.
So thinking about how your product works, not just with an agentic experience, like how do agents access it, but literally how does it work if your user never sees your product overall?
That's incredibly important.
How you orchestrate your own agents to solve your users' problems, long running as they may be, is super important.
And the infrastructure around that is incredibly difficult because not only are you building your original product, you're now building a system of long running agents around your product to empower your users.
And the choices you make around that are super important and really, really tough.
Let me throw maybe one more thing at you here real quick.
If I was trying to build my first long running agent, And I said, you know, I don't want it to be a personal agent through Muse or Instinct or whatever.
I want to build my own.
How should I get started?
Yeah, this is like how to draw an L.
You know, draw the circle, draw the rest of the L, dude.
It's really tough.
And I guess like it depends on the overall objective.
I think like, firstly, it's pretty clear that you can't have one agent do everything and that subagents are pretty good.
So figuring out some architecture where you have a request go in, create an agent, and the agent can delegate to other sub-agents as necessary is super good.
And that's really good because then you get almost unlimited tries because each sub-agent has its own context that can go completely haywire, report back to the parent agent, and it doesn't derail what's trying to happen.
So architecture around one input, series of steps, sub-agent delegation, particularly important.
It depends on the experience that you're providing to the user.
Do you need real-time UI updates?
Do you need chat?
Do you need steering?
At which point then you need to interject some state into those functions, which is a choice for you to make overall about the product experience.
Overall, you need to take a look at queues, you need to take a look at workflows, and you need to take a look at state so that you can track what your agents are doing and come back to it to see if it were doing the right thing.
tons of different frameworks out there for building with agents.
There are tons of different LLM providers.
Honestly, at this point, I think the agents, LLM and models are so good that you can just straight up use an open AI or Anthropic request and get everything that you need nowadays.
But the tracking around what you're building and how the agents work is the most important thing.
I think secondly, tracking the outcome is so important.
I will sort of take a hypothetical point of view, which is We talked about security.
We talked about like the Datadog view of like, make my app faster.
If you had a long running agent that you wanted on your infrastructure to consume something from say like Datadog or like some other APM, you would then have a goal of a PR that wanted to be merged.
And then you would have the agent pause after the PR was created and you would have the agent resume once the code was merged.
to check whether or not that PR had an impact on said SLO.
So you're going to be storing a lot of state.
You're going to be trying to revive this state once the PR is merged, which means a lot of events and event-driven systems to make sure that your agent is revived in the right way at the right time.
Orchestration is like, orchestration is the most important thing there.
Legitimately.
Still early on the long-running agents, but like when you have that talk in the future, I can't wait to watch it.
This is, I think, the biggest directional shift in AI that we've had since.
like maybe tools.
Yeah.
I can't wait for it.
I think it's going to be fascinating.
And I know we're going to be writing about long running agents on our newsletter at newsletter.chainofthought.show.
And so I'm very excited to get more thoughts from you and more feedback because I think there's a huge opportunity to explore this topic more with our audience.
Like, I just want people to, I want to see people's use cases, man.
I want to see people's thoughts.
Like, I want to see how it happens.
It would be so interesting for you to demo if you had personal agents doing things like.
just how it works.
Do it is going to be good.
We were trying some, some folks in our company were trying, uh, trying, uh, opening eyes version.
Um, but they were like, you can only run one agent at a time, no handoff, relatively weak.
But then you go to dot, like you go to a crop bots and it's like an entire, like a lot.
And I I'm really impressed by how much they're scaling it out.
I need to spend more time with that.
It's crazy.
So I think like one of our goals is like to, to make it easy for people to build personal agents overall.
inside their product in a similar way.
And the data advantage you get from something like having access to all that data at Rockpot is so huge too.
Yeah, exactly.
Exactly.
And like specifically, honestly, outcome tracking, like we released it and then a bunch of people just started naturally using it and we've not really talked about it, which is fine.
But the outcome tracking of 100% of your agent trajectory is like great because one of our longtime users, you can take all of that data and if it's known good because it had a good outcome.
you can use that to help improve future area, whether or not that's fine tuning or open weights models, like it doesn't matter.
How can folks use that with Injust?
Yeah.
The core premise right now is you write your entire AI agent and then you score it based off of events.
So for example, you're doing a coding agent or a code review agent.
If people thumbs up, thumbs down, if they resolved the comment without pushing a commit, then you know it wasn't really a good comment.
So you take all of those events, you take a look at the state of set events and what happened around them.
And then you get like a good score.
Yes, no, good, bad.
You can give yourself like a score between zero and one if it's not quite Boolean.
And that results in every single trajectory having a group of scores that tells you how good the outcome was.
And you literally just group them.
You just search for all of your runs and you group them into data sets as good or bad.
And you take all the bad ones and you improve them in a very similar way to evals.
You take the good ones, you download the trajectories, throw it through something like Tinker, fine-tune your own open weights model, throw it up.
We have step.experiment, which is nothing novel.
It's A-B testing, dude.
Optimize it.
We're all doing it.
Dude, it's just exactly.
It's like, people will be like, it's the most world-changing thing.
You can A-B test models.
No, dude, it's like the same stuff we've been doing in 2013.
It's 13 years old.
But you can like A-B tests.
whatever you want inside your workflows and then see if it did the right thing.
So like, I'm really excited about the future of that.
Yeah.
Amazing.
And I'll say I highly encourage everyone listening.
If you have found this interesting, check out the work being done at ingest.com.
There's a ton there from their team and a lot of learnings from Tony and everyone else at ingest.
And if you enjoyed this conversation and you've sat with us through it, thank you.
First of all.
but please just drop a quick comment.
Let us know what you enjoyed.
Let us know what questions we didn't get to you that you'd want to hear from Tony next time.
And thank you so much for having us down to your offices here in Saramse.
It's been a blast.
Thank you so much for having us.
It's really, really great to chat with you.
Super fun.
So fun.
Yeah.
Thank you all for listening and we'll see you next week.