# Agent Fleets and the Shift to Systemic AI Leverage

**Podcast:** Chain of Thought | AI Agents, Infrastructure & Engineering
**Published:** 2026-10-25

## Transcript

The models are frankly all good enough to do most tasks that we need to do in our day-to-day work.
The vast majority of things you'd ask an agent to do are not intelligence-limited.
They are context-limited.
Biggest change we see is moving from solving one problem at a time with an agent to building systems to solve whole classes of problems.
I think we find the companies that see the biggest leverage are instead of asking the agent to build an integration, they say, hey, build me a system.
Joining me today on Chain of Thought to talk about how companies can effectively leverage agent fleets and what breaks in their infrastructure is Andy Treidman, partner at Theory Ventures.
Andy, great to see you.
How is the current AI acceleration treating you?
Nice to be here, Connor.
It is one exciting time.
I think the pace of progress, not just at the frontier of models, but with exciting new things, models like JEV, new post-training techniques, it is like a full-time job just to be reading Twitter and each each new post is like sparks a bunch of exciting ideas of how to deploy stuff in applications you're building.
So that's really fun time.
And I would be remiss if I didn't give a quick thank you to our sponsors this episode.
Vix, Wall West Memory, Ingest, our season presenting sponsor of Chain of Thought, G2i.
Yeah, I mean, speaking of Jev, we saw Jev come out.
It's a new hotness for a few days and suddenly there are so many copycats.
It really drives home how hard it is to establish a durable moat.
as a new startup and a new product today.
Yeah, I think increasingly and, you know, I've been a bit or less until from the early days, but I like continue to be very model architecture and technique agnostic.
I think of ML generally as just, you know, increasingly efficient ways to apply compute and data to learning distributions.
And so I think and, you know, all these people are moving around to different labs.
And I think you see great ideas are diffused very quickly throughout the ecosystem, which I think is great as builders with the technology.
But it makes it hard to interpret what will be like meaningful defensibility, especially at the model layer outside of capital data compute.
Yeah, absolutely.
I think there's something to be said for building distribution brand behind the distribution.
And we're seeing that come into play with.
certain companies but it's hard to get to that scale without leveraging those those first few uh pieces that you mentioned and i think one of the interesting things we're also seeing as part of this transition is we're moving from an era where teams were using agents to help employees and we're doing a lot more steering and now we have agents taking on whole tasks we have these agent fleets that are i mean solving millennium prices and uh It's just very clear that the way work happens is not only changing rapidly today, but is going to continue to do so over the coming months and years.
What do you view as the implications of this current shift?
And how are you communicating or studying this with companies that you're working with?
Yeah, absolutely.
So, you know, I think one, just observing the.
progress in models and the implications on what that means.
I think we see for almost any task, there is a really non-linear function of how useful AI is to do something where the model gets better and better, but it's just not quite good enough to really meaningfully impact the job you're trying to do.
And then all of a sudden it passes a threshold and it goes from basically not useful at all to, you know, doing it as well or better than you can do.
And you basically effectively entirely outsource a task to that AI system.
And I think we are, seeing that consume more and more valuable tasks.
And then I think thinking about the implication of that, we are considering a lot about which functions are competitive or core differentiator versus not.
And so, for example, in software engineering, if a team can move twice as quickly and build twice as much software, they will blow all of their competitors out of the water versus maybe you look at a back office function, something like legal.
There's only so much legal work to be done in a business.
And if a company can do two or three times as much, they're not going to crush all their competitors for having a more efficient cost structure legal process.
And so I think in domains where agents can do things that directly implicate a product or company's value, you will see a forced, very rapid acceleration of use of agents to drive the business forward in a way that any company that doesn't do this effectively will be left behind.
Yeah, I think we're seeing some of this in software engineering for sure, where, you know, if you follow at potato with a POT ETO from SpaceX, Lauren over on Twitter, she is talking about shipping, you know, 200 PRs a day.
Sometimes now there's like a leaderboard competition within SpaceX where they're just trying to accelerate the pace of those PRs.
And we did talk to the Sonar CTO recently about this, like obviously.
a PR on its own does not mean it's a measure of value.
One PR could be more valuable than 20 other PRs in certain circumstances.
But it does give an approximation of the velocity and the pace that's being able to be leveraged here.
And to your point, if an engineering team is able to hit a level of velocity and continue iterating off of it, even if an individual unit of work is not as valuable as another engineer's unit of work, the pace of iteration they can can simply achieve so much over time.
And I wonder, you know, what what are the tasks that you feel like are still needing human approval versus can be fully offloaded to these autonomous agent loops?
And where do you see that changing over the coming months and years as that continues to impact startups in particular, I think?
Yeah.
Yeah, it's interesting.
So, I mean, I think back to your previous point, it is hard to accurately assess the current state of AI capabilities just because they're moving so quickly and they take chunks out of so many tasks in this kind of binary way where you say, hey, you know, this just isn't useful for this kind of thing.
And then all of a sudden, like you snap your fingers and it's better than you ever were.
And so I think to your point, you know, like obviously PRs are not a perfect representation of product progress, but I think they're, you know, quite a good...
proxy and i think the the concept you know we talked in the early very early days of llms people were always worried about hallucinations and to me that seemed like just like a point in time issue that the llms were going to blow past and now by and large they have you know they'll do web search they'll interpret data and i don't think anyone's really worried about a model hallucinating unless it's a like a particularly niche uh question without good information on the internet and i think we see a similar thing in in software engineering work And to a certain point, there's some selection bias as well.
I think the companies that we see that are shipping a huge number of PRs per person or per person enabled by agents, they are driving like extremely fast product velocity.
And you look at the, you know, we look at some of the companies we work with that are very agent native in how they develop software.
And we'll go into a quarterly board meeting and they have what looks like an annual plan.
And they've done all of that in the last...
three months in terms of new features, new capabilities, etc.
And so I think the teams that are most sophisticated at driving agents towards producing a lot of PRs are ones that are driving them towards PRs and helping shape PRs that help drive the product forward.
Even though, to your point, like not every PR is a huge, meaningful feature.
In terms of like what can effectively versus can't be outsourced, you know, I have experience both working with a number of portfolio companies.
So Theory Ventures, we work primarily with AI native companies building AI infrastructure and application platforms.
And I think the biggest change we see is moving from solving one problem at a time with an agents to building systems to solve whole classes of problems.
And so a couple examples of kinds of things that we see, one would be building integration.
So we work with a number of cybersecurity companies that need to integrate with 50, 80, 100 tools.
The sort of natural first thing to do would be to ask an agent, say, hey, build me an integration for this CrowdStrike functionality.
Now build me one for this Splunk connector, et cetera.
But I think we find the companies that see the biggest leverage are instead of asking the agent to build an integration, they say, hey, build me a system to develop and test an integration.
And then now when we have stuff in the future, I can say, hey, here's these 20 new integrations and they'll go basically run through this process to build them.
And so that's, I think, one example where there's just really clearly you can encapsulate all of the core product judgment into a system that then enables agents to go and execute.
And I think similarly in anything where you have like quantitative performance indicators.
So, you know, what is the accuracy of some agent native workflow?
What is the cost it takes to run it, et cetera.
You can similarly build like really nice systems that help you.
deploy agents to the task, either you trigger them or maybe they run autonomously to say, hey, we collected a new week of user customer data, we're observing these failure modes, we're going to make some iterations to our agent instructions, agent harness, and we're going to back test it in this and we see that it drove these metrics in the way we care about maybe, you know, had some set of like human defined constraints, and we're just going to deploy this because it clearly worked.
And so I think those are two kinds of things that are extremely well suited for agents.
And I think the more creative you are and as the agents get better and better, you can actually like put more of that product judgment into these systems.
And so more and more things get covered by this characteristic where you can sort of have an agent run at it and not worry about it, you know, generating bad outputs.
Yeah, I think finding these loops just gives you such an opportunity to accelerate.
And I'd point to one I saw on Twitter yesterday from Reese Sullivan, who's a mutual of mind, who is building Executor at YC right now.
And what he has set up is a prompt where every six hours his agents will go and say, OK, how can we make my application faster?
And so he's got a core application and it's just like every six hours he'll get PRs that are improving the performance of the application.
They're validated against a staged version of production and they've passed multiple layers of agent review.
And he just has a dashboard that basically shows, hey, what's the improvement we've made on this over time?
and to just keep cycling through because like this is an obvious thing he's always going to care about as the product continues to get broader and they add more features they're going to have to keep tuning and improving performance there and so it's just an obvious iteration or i'd look at like jen spark who we had on the podcast a few weeks back at this point their ceo when saying talked about how the fact that they have an agent that just runs across all customer requests and puts them into an automated development queue.
So similar to the integrations point where bugs are getting automatically pushed up to coding agents, they're spitting out a PR.
Often those PRs are merging, depending on the part of the code base, without any human review.
And then they have a human that will pop in to review certain key parts.
And it's so clear that companies that are able to do this are just accelerating because there is just an opportunity on the edges here to to move faster and faster.
Are there particular companies that you're working with that you're seeing do this especially well?
Yeah, absolutely.
So one example of a company that we work with on the portfolio side is called Artemis Security.
They're based in New York.
They're building an AI native security data platform.
So it replaces what's called the SIM, which is a core data platform, and then builds an entire suite of agents on top of this.
And I think this is really a like a perfect.
crystallization of these kinds of problems where they need to have a very broad set of functionality and connectors to connect throughout the security data ecosystem.
So that fits really nicely into the sort of first category I talked about where you can build effectively like an integration or connector building suite.
And then they're basically building and running a huge number of sophisticated models, which each of them obviously generates quantitative.
signals, they have false positives, they have false negatives, etc.
And so those I think that also does really well.
And then I'd say the third layer on top of that, which sort of to the point that you mentioned is they're trying to be extremely fast with their UX iteration and customer responsiveness.
And so those are maybe a little bit softer.
But you can still drive really rapid iteration loops by having agents that are, for example, taking in customer feedback and generating ideas for how to do it.
And then I think, you know, on the personal side, I've been building a personal app, which is like iOS consumer fitness app.
And I've been doing it in large part just to get experience with building a probably built a very overly heavy and sophisticated software factory for like a one person weekend app.
But that's given me a lot of room to play around with understanding sort of the frontier of what's possible.
I think unlike my portfolio companies I work with, I'm probably a little more like laissez-faire and willing to like push the gas on the agent automation, which gives me a good sense of like where it starts to break and where it's really strong.
Season four of Chain of Thought is delivered by Sphix.
We spend a lot of time talking about what agents need in production, and one of the least glamorous answers is events.
Your customers want agent workflows that react to things happening inside your system, which means your API needs webhooks that actually work.
Not just a post request and a prayer.
Retries, ordering, idempotency, replay protection.
Sphix does that as a service, and they wrote standard webhooks.
The spec that Anthropic, OpenAI, and Google bailed against.
So if your API doesn't have reliable webhooks, that's turning into a lost deal.
Join Brex, Drada, Daytona, and many others on Sphix.
Get started at link.svix.com slash C-O-T, or go to the show notes to grab the link.
Qualified startups will get $12,000 in credits, $50,000 for YC companies.
I can't recommend Sphix enough.
I'm a huge fan of their open source project.
I've actually contributed a bit myself and they're so easy to integrate with.
I think you'll really enjoy it.
Check out Sphix at link.svix.com slash C-O-T.
Yeah, I'd love to get your take on where you are seeing agent automation breakdown today.
Yeah, so I think...
You know, it's really clear that anything with a verifiable objective, so like, hey, did we build an integration, did it work?
Or quantitative signal, like those are pretty clearly, I think, it can be challenging to build them in the right way, but like that is 100% doable by agents today.
And then I think, to your point, like small, you know, bugs and small pieces of customer feedback, that is also pretty clearly you can have like a fully lights out autonomous agent loop.
that works pretty effectively.
I think like when I think through, you know, new product features or capabilities, user experiences, etc.
I guess I think through sort of like hierarchically, you know, you might have the one sentence or, you know, one paragraph description of, hey, I want to build a new feature and I want it to do XYZ.
If you ask like Cloud Code or Chattypt today, you give it that, it'll give you maybe three or four follow-up questions.
Hey, do you want it to do it this way or that way?
Do you want to have it in the UX, live versus asynchronous, et cetera?
And I think a lot of people might like stop there, take the plan and just run it.
And I find that that is like one level too high where there is maybe like the next eight to 10 questions that would come out of that can be really meaningful from a user experience perspective.
And if you just do the one paragraph version and answer the three questions, you might end up with a lot of assumptions that the agent system makes that actually have a meaningful impact on your customer user experience.
And from personal experience where again, I'm being a little more laissez faire and I might just like some of these let them rip and run through the plan.
You end up build it with it building stuff, not that is broken, but that changes the customer experience in a way that I might a couple days later be using it be like, oh, I actually didn't want it to work like that.
And so I think like what I've found is just going from those three or four questions down to the next eight to 12.
And actually the agent I think does a pretty good job at proposing most of these and then maybe reading them, you might come up with a couple others.
But you just add that like additional layer of specificity where you have characterized 90% of the 95% of the edge cases and the ways you do or don't want it to accomplish something, it can then run pretty effectively from there, even if it's generating 10 or 15 tickets.
And even if in those tickets, there still is like another layer of, you know, hey, we found that actually this data didn't support this thing.
So we're going to close this ticket out.
Or, hey, actually, we found that we're not able to do it this way.
So we're going to do it this slightly different way, et cetera.
More of like an engineering execution decision.
And I think the agents can do that really well.
But making sure that you've...
fully understood the scope of what you want the functionality to be and how it impacts the customer experience, I think is the critical thing that distinguishes between sort of the magical experience of like pushing out an Epic, having it run through the factory and it just comes out the other end perfect versus something that comes out and you say, oh, this wasn't exactly what I was thinking.
I know you have a background in neuroscience and obviously you're a very bitter lesson pilled.
Is that impacting your perspective on where things are going and where the constraints are today?
Yeah, I think that certainly, and I'd say in two ways, I think from a core model capability perspective, it causes me to one be very long that like core foundation labs.
I'm not sure that they will monetize and build the most effective businesses.
But certainly in terms of driving model capabilities, I think it is clear to me that it is a matter of like dollars.
that turn into compute and data.
And from a consumer of those, I think we have a very strong belief that at least for the foreseeable future, two, three, four years, these models will just continue to get much faster and much cheaper.
And so you can sort of like build applications predicated on that assumption and all of the downstream things like, again, you know, I don't really worry about agents generating, you know, sloppy PRs if the specs are well-defined.
And then I think from an application building perspective, what's really interesting to me is that to a certain extent, I think the software building role is becoming much more like a researcher or scientist type role where everything we're building is now, to a certain extent, non-deterministic.
And so that means whereas previously you say, hey, I'm building this feature, it's going to take...
it's this many points, we'll have this much stuff done during a sprint, you need to sort of entirely shift how you spend your time, design your roadmap, et cetera, to say, hey, we're going to build this thing.
It's going to take a variable amount of time to get to the first iteration of the product.
It's going to either work or not work or kind of work.
And that will then inform what we do next.
And so I think just that like mentality and way of working and planning how you're going to.
build stuff and where you spend time shifts from a very like execution orientation to a much more like thinking and experimentation.
I want to dig into this change in orientation because I think it is a really crucial and somewhat underrated aspect of our current moment is the shift between how engineers in particular have thought about their roles, even though engineers have been problem solvers.
We had charity majors on the show last year to very much make that point for us.
But now they're really transforming into more of a research or a product engineering role in many cases.
We're not putting hands at keyboard to code our way through a problem.
We are instructing our digital minions to go and solve them for us.
And I think another aspect of this, before we talk more about the theory piece, is parallelization, where previously you typically had to try to solve the problem yourself.
Now you're able to parallelize multiple problems.
But where are you seeing this parallelization of work either change how roles are operating or break the current operating system?
Because something I have observed has been that it is at times causing us to lose focus because there's so many different options to be working through.
I think we're seeing with things like cursor projects and other changes to the level of abstraction that we're operating at for engineers that when we're able to take on that, hey, I work with a particular team lead agent and that it's kind of driving the factory for me, we're seeing more success.
But that does, to your point, require a lot more architectural design problem solving.
Yeah, absolutely.
Yeah, so I think about this a lot and we can talk about it both from an organizational lens and how you're spending time and also the actual implementation and what starts to break with parallelization.
But starting on the first category, it's clear that to your point, almost everyone will need to up level where they're spending time.
I think you already see the best engineers are already extremely customer centric.
and need to really deeply understand not just, hey, we're building this feature to do this task or do this function, but our customer is trying to accomplish this goal and here are their priorities and here are their constraints.
And so we are building this feature to help them do that to achieve some outcome that they care about.
And I think that sort of shift in framing is really critical because there are so many both spoken and unspoken assumptions about how you want to product or platform to work and how you want to serve your customers both now and in the future.
And if you just give agents a list of features to build or tasks to do, they will probably do that very effectively.
But maybe without that broader context of, hey, we know that we want to expand to support this new customer segment in the future, or we know we really want to prioritize this kind of usage over that kind of usage, etc.
And so I think it forces people to move into the why and build a level of customer empathy.
to then be able to direct the agents on what to build, what constraints, how to architect it, etc.
And then you sort of relinquish control over what, you know, maybe we used to spend most of our time on, which is the actual like implementation and delivery.
Season four of Chain of Thought is presented by Walrus Memory, the portable memory layer for AI agents.
Your agent has learned your code base and how you work.
Now you want to use that context in another tool.
Exporting a file gives you a snapshot, but what happens when the context changes?
Walrus memory is a portable memory layer that lets your agents store context for later use across tools.
Python and TypeScript SDKs plus native MCP support let you connect it to your agents no matter where they are.
You said who can read and write, so sharing context doesn't mean opening up your whole memory store.
If you are building across tools or model families, take a look at Walrus memory.
Learn more at walrus.xyz slash cot.
That's walrus.xyz.
slash C-O-T.
And an aspect of this is also providing our agent fleets enough context and memory to be successful, including I've seen you propose storing working states and reasoning alongside changes.
Obviously, we're seeing entire and others take this approach.
How are you thinking about this aspect of ensuring the agent fleets have all the data and information they need to be successful?
Yeah, it's a good question.
So I think you know, talking through like my personal journey, which I think probably reflects most people's and companies, you like start with a, you know, single or a couple cloud code sessions or something like that.
And you're sort of like managing multiple threads.
And then you maybe start to like move up the abstraction ladder.
You like build things that are, you know, taking in customer feedback and automatically like running something in the cloud to go triage that, investigate it, deliver.
And as you move from one to three, you know, couple parallel threads at a time up to, you know, you're working on six, 10 things at once.
And you maybe are coming back and like circling back to a feature you built a month ago to make an additional improvement, etc.
It becomes a really tricky sort of second order question of like, how do I manage context?
Clearly, the code itself is non-sufficient, even though these coding agents love to put very verbose comments everywhere.
And you get some value out of, I think the immediate first step is like, hey, we're going to have our agents write documentation.
So for my personal repo, I started with running parallel threads.
And I said, hey, everything we're doing, we're documenting in Markdown files, we're putting them in the repo, we're versioning them and sort of like built some systems to like clean up old docs over time.
And that works pretty well in the like handful, you know, five to 10.
agents at a time when you're working on a feature, you might go back once or twice over a couple months, etc.
So that is sort of like a first step.
And I think the agents are very naturally capable at like writing and documentation and then gripping it.
But I think as you move to sort of a company scale, and you imagine, you know, 10, 20, 50 engineers working with hundreds of agents, that approach of just like markdown files in your repo starts to break down as well.
You run into just sort of like decay and rot of those documents.
You have a whole bunch of other stuff that exists in Slack conversations or that certain videos in the call, et cetera.
And so I think that we will certainly need more sophisticated systems or probably, again, agent systems to be continuously monitoring everything that's going on in an organization and making sure that there is a canonical living source of truth for...
what the company is doing and why and what decisions they've made.
Because for agents, you know, for humans, like an old PRD, they just ignore it.
They say, hey, this is out of date.
I know we talked about it.
We're doing something else now.
An agent's going to take everything and they sort of take everything as true.
And so it becomes extremely important to make sure that all of the information that the agent can access is up to date with the latest things that people have been saying.
All of this has a cost.
Obviously, inference can be costly.
How are you thinking about ROI and particularly as we have these iterative loops that are spending a ton of time on ever expanding efforts, it's easy to spend a lot of inference.
How do we make sure we're getting value out of it?
Yeah, I think, you know, first, I am very convinced that agents can and do create software that drives forward, you know, again, like core product roadmap and customer value.
Two, because software engineering for all the companies we work with is sort of the core differentiator.
It's their core product.
We think that competitive dynamics mean that companies will need to invest very aggressively in doing this and that you basically get very strongly positive returns to investment in the software factory just by virtue of being able to move faster, be able to serve more customers, be able to generate more revenue and grow as a startup, particularly.
I think that there are certainly like we will come to a point Maybe we are starting to come to a point where cost optimization becomes more important.
I think we see a few different levers.
One, you know, open source or just cheaper coding, closed source coding models are becoming extremely good.
You know, I think they are good enough for the vast majority of tasks.
And actually an interesting observation for my personal stuff is like the better your software factory is and the better specified the tasks are, that makes it much easier to then.
push most stuff to cheaper models.
And so you can, you know, have a very well-scoped epic of work.
You generate 15 tickets, you say, hey, actually, like, seven of these can go to Luna, a couple go to Sonnet, and only a couple need the most intelligent models.
There is an interesting additional layer, which is just, like, the infrastructure cost.
That's something that, you know, I've certainly noticed in companies we work with, like, you pretty quickly start to run up a pretty insane CI bill.
And so there may be some other stuff there.
But to be honest, it mostly feels on the margin.
I think if you say, hey, you can ship your product roadmap twice as quickly and you don't need to hire 20 more engineers, I think you got a pretty safe cushion of ROI there for most business leaders.
I tend to agree.
Obviously, at scale, I think as you hit the enterprise and there are so many.
coordination taxes that come into play.
And I think it's more important to start to optimize costs because those bills can get really crazy.
But for smaller startups and teams under a few hundred people, I mean, just lean into the velocity right now.
There's such an opportunity.
And frankly, you have competitors that are.
If you aren't leaning into shipping faster and iterating, I worry for your positioning, just to be frank.
So.
As all of our human time starts to abstract away from the layer of code and we're spending more time on understanding intent, understanding customers, customer empathy, I think you called it design, product engineering skills, as Laurie Voss called it on our show.
What has to be true to set up our agent factories to act on our behalves with low levels of review?
Yeah.
Again, I've been certainly pushing the frontier of this in my personal stuff just because I am busy with my regular job.
And so I've had a nice natural experiment of where is too far.
I think first, constructing the factory in a way that gives you the right paths to escalate the right things to humans at different time and understanding really where you need different levels and types of feedback.
has been really useful.
So for example, you know, I mentioned properly speccing out, you know, product requirements and making sure that you've had multiple iterations.
And so, you know, each product requirement I end up building might have like a 20 to 30 question Q&A at the end.
And we have specifically like a control loop that says, hey, we're not going to build anything.
We're not even going to like spec out the.
uh tickets of engineering work until we have like fully exhausted this list to make sure we covered every edge cases um and i think one thing that i observed before is like when we didn't have that i would then actually create a lot more work for myself because we'd have more agents that would discover stuff that they correctly realized was a significant change during building and so instead of like reviewing one big document one time i'd sit down on a you know weekend morning and like crank through it all i'd be getting like pinged all the time with 10 or 15 different requests so i think a lot of it is more of like a process perspective of understanding where in the process and at what level of detail you need feedback in to then enable yourself to like go lights out in the execution so that's you know and generally we see that's a lot of like shifting stuff up front um And then I think one other very important aspect of Observed is making sure that the factor in the agents are themselves very oriented around quantitative results, data, and experimentation.
And so, for example, any change you'd make to an agent system or even sometimes not to an agent system.
you might construct and run an eval to measure that.
And you might typically want to agree on that upfront.
And so you're saying, hey, we're like thinking we're going to build this feature.
But before we build this, you know, I'm trying to decide if I move from a rules-based to an L1-based classifier for something.
Before even building that, we're going to run this very quick, low-cost data evaluation to understand, like, is this even a problem that we need to solve?
And then we are going to test options A, B, and C.
We're going to put together this golden evaluation set.
Here's how we're going to evaluate it.
And the agent's going to go run that experiment and deliver the results to me.
And you might even pre-specify, you know, if the LLM is better than the historical classifier in 95% of cases, just ship it.
If it's better less than 50% of the time, we're just going to close the ticket.
And if it's in the middle, then give it to me, give me some examples and let me choose it.
And so I think by taking that kind of approach, you can...
enable the system to run autonomously with much higher confidence, knowing that anything in the sort of gray area will come up for a human review.
This episode is sponsored by G2i.
I've said before on Chain of Thought that most teams still treat evals like unit tests.
Write them once, check a box, and move on.
That doesn't hold up once an agent is making decisions.
G2i is about to close that gap.
For over a decade, they vetted and placed engineers at other companies, from startups to FANG.
Two years ago, they turned that same judgment inwards, building their own bench to review our own environments, evals, and training data that models are trained on.
These reviewers know the difference between code that runs and code that's actually good because they've shitted it themselves.
If your team needs that kind of work and doesn't have the engineers for it, check the link in the show notes to bring them in.
One of the key pieces here of hill climbing these evals is also understanding the guardrails or sandboxes within which you let your agents operate.
I know you wrote about this and your perspective on it earlier this year.
I think you called them isolated worlds for agents.
How are you thinking about this aspect of both breaking work down and providing the right level of context, access and information to agents?
Because I think we've seen enterprises take a much more conservative approach on a lot of this.
And that is very different for an individual builder or a small startup that needs to move faster.
Yeah, absolutely.
So it certainly depends on the domain.
On one side of the spectrum, for my personal app, I say, hey, just give agents read-only access to the database.
I can run an iOS simulator on my device and sort of just having them effectively be able to run everything.
They're taking screenshots, et cetera.
And that works quite well.
Now going down to the other end of the spectrum to, you know, for example, Artemis that we chatted about in cybersecurity, they need to simulate very complex enterprise environments involving, you know, 10 or 15 different like closed sourced production systems and including huge amounts of simulated behavior logs, et cetera, all of which is quite sensitive since it's in cybersecurity.
And so I think These agents are really hamstrung if they cannot take actions in these systems.
And they are getting much, much better at spinning up an application, clicking through a UX, taking screenshots, making sure it works, and using that to build this sort of closed loop iteration.
And so I think we see that it is an absolutely necessary prerequisite to build a software factory effectively.
And the only thing that changes, I think, is the amount of engineering effort that goes in.
And so we see the...
Most sophisticated companies that we work with, they have entire teams that are spending basically all of their time just building these simulation environments and the capabilities to spin up and configure them to look like different kinds of customers and being able to run their agents or their application in effectively a perfect facsimile of what it looks like in production.
And that usually lets agents be very effective and then transfer to the real world.
It's a great way of avoiding shipping an incoherent product by actually doing these simulations and ensuring that the different features your parallelized agents are working on come together and actually solving your roadmap.
Yeah, absolutely.
Well, let's say that I am earlier on my journey here.
I think most of the people who listen to this show are more AI pilled, but the level varies.
There are plenty of folks who come in and listen to some of our episodes and are saying, hey, I am looking to build my startup to a point where it can be a better AI factory.
I want to level up what I'm doing here.
Maybe I'm just using coding agents at a very basic level.
I don't have them externalized.
I don't have them operating in loops.
You know, I know I need to accelerate.
What advice are you giving to founders you talk to who are?
looking to embrace this acceleration and velocity so that they don't fall behind?
Yeah.
The first couple of things, I guess I would suggest, first, I think is just like skills, building and using skills.
A factory, if you think about it as basically just a collection of, you know, 10 or 15 different skills that run in some environment.
And so I think like the first step is anything that you do on a repeated basis.
take the extra 30 or 60 minutes to encapsulate that into a skill and invest the time upfront to both build that and then continue to improve that.
I think that that shift in mentality, both doing it yourself and like encouraging all of your engineers to do that is the most effective first step.
And we see a lot of companies that were sort of in this like, hey, all my engineers are using, you know, an AI coding assistant, but they're sort of doing it on their own.
Shifting that, mentality to like, hey, Connor did this awesome thing.
You know, we're always reviewing PRs in this way.
And Connor took two hours over the weekend and he built this skill that does it.
And then we all start using the skill, et cetera.
I think that's like the first way to like shift the mindset towards a factory building mindset instead of an agent for software execution mindset.
And then I think like a key enabler from that, from an infrastructure perspective is just making sure that you have the ability to run agents against your application in a sort of like infrastructure agnostic way.
So like setting up your cloud environment so I can have an agent that runs and does some stuff on a, you know, century alert or on a piece of customer feedback or on a like epic that we have in GitHub or Jira or linear and run that in the cloud.
I can look at it on my phone.
I can look at it on my computer.
It has access to all my data, et cetera.
Just making sure that that's really frictionless for people, I think is the key enabler.
And then I think that two of those things, honestly, like naturally start to evolve into a factory.
And then I think on top of that, you can start to put more structure.
Okay, how do we want to set this up?
How do you want to run triage, et cetera?
But I think those are the two most important first steps.
And as you start to move up this maturity curve, how are you protecting customer data and isolating agents?
to ensure they're not breaking things because, yes, we want to move fast and break things.
But also, if you cause problems to your customers, that has metered downstream implications and can kill the trust you need as an early stage startup.
Yeah, 100 percent.
And especially, you know, all the companies we work with are enterprise AI companies, and so their customers are very sensitive about their data and environments.
I suppose I think about that in two ways.
So first is from the customer data and system side.
I think that you need to obviously be very careful.
I think first should probably go without saying, but like agents should never have right access to any data you have, both for, you know, customer reasons and just for you don't want it to like overwrite something in database.
And so like making sure to have the right permissions for the right set of data and particularly in an enterprise setting or whenever there's any sensitive data using sort of like simulated copies or using like an internal mock customer data, etc.
is basically as effective and mitigates any risk you have there.
The second is sort of on the shipping and delivery side and sort of how do you implement the right controls to prevent the agents shipping something that is either just a sloppy feature or worse, something that actually takes down a production system or impacts customer experience, et cetera.
I think what we see as most effective is clearly to start as you are.
building skills to do the thing and you have a human in all cases, whether it's at the PR stage or somewhere else, you have a human as a final person that's like clicking, clicking merge at the end.
And then you probably have a person that's like running QA on the feature in production or something like that.
And I think like starting with that gate default on and then very selectively taking it off when you feel comfortable that, you know, a certain kind of task can be done.
with a very high level of precision and then maybe also adding on some sort of like monitoring a learning system, making sure that's really effectively is the best way.
So sort of like you start from a default safe mode and then you can very selectively like take your gates out when you need to.
A couple of big through lines on chain of thought recently have been providing the right context for agents and providing agents with the memories they need to be successful, especially for long running agents.
What's your perspective on context and memory for AI agents.
Yeah.
I've for a very long time been quite context-pilled and believing that context is by far the most important component of almost any agent system.
I think particularly as we see intelligence becoming more and more commoditized, the models are frankly all good enough to do most tasks that we need to do in our day-to-day work.
the vast majority of things you'd ask an agent to do are not intelligence limited.
They are context limited.
And it's been really interesting to see the evolution of that.
And I think as intelligence becomes increasingly a commodity, we see more and more value to the applications surrounding them, primarily for the function of delivering context.
Even as the agents themselves get better at, you know, reading and writing to files, there's some people saying, hey, all you need is a...
file system, our agent's going to write its own notes and they can do their own context management.
For most applications, there is so much domain and workflow knowledge required to do the job effectively that we see a huge opportunity for startups with deep expertise in a domain, whether that is operations or finance or cybersecurity or legal, and a deep empathy for a customer in their workflow that you can build systems that very effectively generate, transform, compress and deliver the right information to the right models at the right time, which just lets you do almost any kind of work much more reliably, much more cost effectively, and to build a like compounding advantage, sort of like traditional data network effects, both within and across customers that we think will make applications more and more valuable over time.
Season four of Chain of Thought is presented by Ingest.
Long running agents need endurance.
They must endure waiting for users and weather tool failures.
They must take each new jump in model intelligence in stride.
Ingest makes long running agents durable, observable, and improvable over time.
Check out their generous free tier and a sleek local dev server for testing your workflows.
Grab the Ingest link in the description or visit INNGEST.com.
dot link slash C O T dash pod to get started.
Build for the long run with ingest.
What about memory for these agents?
Obviously it's closely interrelated with context.
How do you see agent memory evolving and what's your perspective on how it should be used over time, particularly across agent swarms?
Yeah.
So I think, uh, you know, to your point, memory and context are very closely related.
I think at their core, it is like at inference time.
what information is delivered to the model along with a request.
And then when you think about what that means from an application perspective, there is clearly a huge hierarchy of different kinds of memory.
There's memory that might be like stuff in a previous thread you just mentioned.
There is memory that might be a shared vision doc for your company or related to previous PRs you shipped or maybe context that was like in a video conversation that you and I had in a sprint planning meeting two months ago.
And so I think like thinking about memory and context, not as one pool of information, but as a hierarchical set of information, Google actually released a really interesting paper here.
I can find the name and send it to you about hierarchical memory.
But I think about it as, you know, you have like recent session traces and information that might then ladder up into, you know, hey, here's our weekly priorities.
Here's things we're working on.
Here's decisions we made.
And then that might ladder up into higher, you know, here's a canonical doc that describes the architecture of this system.
Here's a canonical doc that describes the priority of this business and that you probably have different systems and mechanisms for moving or updating information from that sort of like recent, hey, we talked about this yesterday up to, hey, here's something we're working on this quarter up to here's that core pillar of our company.
And you probably need both different kinds of systems and different mechanisms for each one of those.
Andy, thank you so much for spending the time with me today.
I'd love to close out by giving listeners an opportunity to understand more about theory.
And for those who might want to reach out and pitch you, what should they be doing there?
What are you looking for?
Because we do have quite a few founders who listen to the show.
We have a lot of engineers who are thinking about founding their own company as well.
So we'd love to open the floor to you on that.
Yeah, absolutely.
So Theory Ventures, we are an early stage firm that, as the name would suggest, is extremely.
focused and thesis driven.
We all have a technical background.
We're investing broadly in the AI and data stack from infrastructure and applications.
And we look for businesses in what we call technology enabled new category creation.
I think this is the most exciting time in many decades in terms of new technology, particularly around AI capabilities, entirely transforming not just existing software tools and categories, but creating brand new opportunities to rethink how we build.
software systems.
And as part of that, I think, you know, we talked about this intelligence explosion and commoditization, we see a huge opportunity to build both infrastructure and applications for this next generation of system where we assume that we have effectively unlimited human level intelligence, and then are building the infrastructure and tooling to help people direct that intelligence to accomplish tasks that they want to do.
And so we're looking basically across every enterprise function from software development to cybersecurity to finance and operations and thinking through what an AI native organization looks like, how they'll operate, and then what kinds of software and tooling they'll need to do that effectively.
And so any founders working on that from the early stage, from day one of a new idea to starting to scale the business would be a great fit for us.
We'd be happy to chat with them.
What's the best way for them to get in touch with you or to follow your work?
Yeah, they can follow us at theoryvc.com or check us out at Theory Ventures on LinkedIn.
Amazing.
Andy, thank you so much for the time.
It's been great chatting with you and lovely having you on Chain of Thought.
Thanks, Connor.
It's really fun.