# AI Agent Search Infrastructure and Market Strategy

**Podcast:** Chain of Thought | AI Agents, Infrastructure & Engineering
**Published:** 2026-11-01

## Transcript

They can do 1,000 searches at once or even 10,000 searches at once.
As a human, I've never seen a human who is able to hit 1,000 QPS.
But I have seen agents that are able to hit really high QPSs when they're doing searches.
Recall generally seems to matter more than precision when it comes to agent results.
One thing for sure is that agents will be performing more searches than humans for the foreseeable future.
Welcome to Sheen of Thought, everyone.
With me today is Sahil Jain, CTO of U.com.
We are here to talk about what survives as AI models become rapidly more intelligent and the future of search.
Sahil, it is great to see you.
How are you feeling about the current state of AI advancement?
Hey, Connor.
I'm really excited to be here.
Thanks for having me on.
I think in terms of the future of AI development, there's a lot happening right now.
So I think things are really accelerating fast.
There's many different trends that we'll talk about soon.
But I think at a very high level, right now we're seeing two big trends.
One is the model's capabilities getting rapidly better.
We're seeing a lot of competition amongst the frontier and open source ecosystems.
And along with that, rapid improvements in capabilities, specifically on long horizon tasks.
So we're seeing AI models powering agents that are able to do more and more effective long-running tasks, whether it's in coding as well as domains outside of coding.
And the second trend that we see is open source models becoming much better.
So we see this from a variety of labs across the world working on these open source models that are, you know, they do lag behind the frontier models, but they're rapidly closing on certain types of tasks.
So those are kind of the two megatrends that we see right now when it comes to AI development.
And both of these trends have a lot of implications in terms of how we build product and how we build software right now.
esoteric question this brings to mind for me is depending on what side of Twitter you're on, you may be hearing that the singularity is beginning.
Do you feel that's happening or is that simply an overblown, you know, pro-technology discussion that's occurring?
Yeah, I think a lot of it depends on how one defines singularity, but I think for me, I do think there's something unique happening right now compared to a lot of the technological revolutions in the past.
I think this is fundamentally different than a lot of the previous ones.
The implications will be, you know, widely felt in terms of how we work, the knowledge economy, as well as even the physical economy.
So I think everything is rapidly changing.
As for singularity or not, I don't know if I have a good sense of that yet.
But I do think that we're in for a world of change over the next couple of years, and it's going to require a lot of adjustment on everybody's part.
What is making today's rapid technology transformation from AI so different from past transformations we've experienced?
Yeah, this is just my opinion.
I'm sure there's many different takes here.
But I think one of the key pieces of it is there are similarities and there are differences.
And this is true of any technological revolution.
I think one of the key differences with what's happening now is the degree of intelligence.
of what's happening in the degree of autonomy.
So I think what we're seeing is, you know, a level of autonomy, you know, that previous systems were not capable of.
So that's one difference is the autonomy level is much higher than it was from a lot of the developments in the past.
When you think of the industrial evolution, et cetera, to a degree, right?
So that's more of a scale.
So I wouldn't say that's qualitatively different, but that is one area.
But the second one that is probably the bigger difference is we now have AI that you know, is in many different domains.
So across many domains, you know, often smarter than a human on, you know, what tasks that we would consider to be, you know, primary key knowledge tasks.
So I think those are two differences from previous revolutions.
I don't, you know, I'm not necessarily saying these are completely qualitatively different, but I think quantitatively, it's very, yeah, I think there are various differences and there's some similarities too.
So it's not the first time that we've had.
a tool that's smarter than a human, you know, a calculator is better at math than a human.
But, you know, the way in which and the degree to which these tools can perform behavior that are aligned or in some cases misaligned with humans is quite different than the past.
And I think this gets to the core of something I've heard you talk about a couple of times, which is the distinction between core and augmented intelligence and how that changes the way we decide what to spend our time on and what to build.
Can you unpack that a bit for us?
when I discussed model capabilities improving rapidly in both the open source and frontier domains.
So that's the first piece about core intelligence, the underlying foundation models.
You can think of it as the engine behind a lot of the growth in the AI economy.
And then when we think about augmented intelligence, these models are super capable, but they need tools and services in order to fulfill their jobs.
The same way we as humans are very smart, but we still need to use tools.
The tool that we focus on is AI search, because we believe that No matter how capable these models get, they'll always need the ability to search for external information to make the better decisions, especially, you know, when they've been trained on certain cutoff dates.
I think there are things like continual learning, et cetera.
But, you know, I think even humans, we continually learn, but we continue to use tools as well.
So I do think there's something fundamental about using tools and having intelligence that's able to use tools really well.
So the way I view it essentially is core intelligence is the underlying foundation models.
Augmented intelligence refers to the tools and services.
that are going to allow the foundation models to go further.
If we take a high-level view of this and we look at, you know, maybe reason by analogy, one example I like to talk about essentially is self-driving cars.
So we've had our road system for a very long time.
And then, you know, using these road systems that humans have been operating for decades, we now have self-driving cars which operate on these roads.
When we think about AI development right now, the way I almost view it is we've got the foundation models and the intelligent self-driving cars before the roads were established.
So what that essentially means is the bottleneck in many cases is not the intelligence, because you do have a self-driving car that's capable of going to a lot of different places.
In many cases, it's the roads that are not yet built and the infrastructure around the models.
And that's what we refer to when we think about augmented intelligence.
It's building up all these roads, one of which is search.
and there are many others as well that will allow these models to go further and do more tasks and really make a difference in the real economy.
Before we dive in, I want to say a quick thank you to our presenting sponsors for Chain of Thought.
Ingest for long-running agents and Walrus Memory providing portable agent memory.
Plus this episode's other sponsors, Coralbrex.ai providing lightning-fast open model inference and Wisdom AI, the enterprise agent analytics platform.
A couple of great points in there.
One, I think it's Extremely interesting to think about tool use and how we have built these machines that are capable of it.
Because if we look at human evolution, it's such an important part of our mastery over the world has been our ability to use tools to develop them to continue to evolve.
And now we're enabling machines to do the same thing.
And to your point, the core capabilities have often outstripped the infrastructure around them.
And it's where we see models needing better harnesses, better search.
What are the other capabilities that you feel are necessary to evolve currently to enable models and that core intelligence?
And then which capabilities do you feel like will become unnecessary as models continue to improve?
Yeah, this is a great question.
And I think this is maybe the million or even billion dollar question is what belongs to the model and what doesn't?
I'm writing down startup ideas over here.
So yes, I think that if I knew the answer to this question completely, I'd have to have a crystal ball.
So yeah, first thing I just want to preface is I don't think anybody really knows.
Even if you asked the best researchers five years ago, they would answer this question differently than they would today.
And I do think that a lot is changing and that is constantly kind of changing the way we view this.
But given the information that we have today, we can kind of have a couple of hypotheses at least.
So on the first question of what, you know, belongs in the model capabilities and what, yeah, I guess if we think about what belongs in the model capabilities, I think that when we think about pure reasoning and the ability to reason well over longer horizons, I think this is something that model capabilities will continue to improve at.
So as a human, when you're doing a task, the way I almost do it is if you're doing a task and you're not reaching for any outside information or interacting with other humans.
or other software, other machines, et cetera.
That is something that I think model capabilities will get really good at because of the fact that the way in which these models are being trained, usually with pre-training and then you have some sort of supervised fine tuning and then reinforcement learning with some rewards.
And rewards can be usually decided over long horizon tasks, depending on the task, of course.
When they're verifiable, it's much easier to train on.
But I think if we kind of reason down to how these models are being trained, I think it become, and we kind of extrapolate essentially, we could think that any task that you're doing as a human in a box where you're uninterrupted is something that eventually will be covered by model capabilities.
And that's maybe like a very big claim.
But if we just extrapolate based on how we've been proving so far, I think that's kind of one place that I think belongs in model capabilities.
But when we think about how we work as humans, we also are constantly being interrupted.
We're talking to other humans.
We have other types of relationships.
I think that's something that will always live outside of the model capabilities is these external relationships to a large degree.
And when I say outside of the model capabilities, I don't mean necessarily outside of software.
You know, there's still maybe software that, you know, does connect models and allow them to do these types of things.
But I think the models themselves, you know, I think will still heavily rely on tool calling to achieve their goals.
So that's kind of maybe more of a high-level answer of what...
ends up within the model capabilities and what doesn't.
The second answer is, you know, going back to the way the models are trained, one thing that has become very tough is, you know, making the models better on domains in which there are less verifiable rewards.
This is kind of, you know, the whole topic of taste comes up quite often in Silicon Valley.
And this is one of the reasons in which we see that that word taste pop up so much is I think that there is there are many types of rewards that, you know, don't show fruit, you know, that come about after many years.
So for example, if you think about, you know, an investment manager, a lot of times what they're doing is making investments and you don't know if they're successful or not until five or 10 years into their journey.
And then, you know, if you have two decades of experience, you've seen a lot of waves come and go and you're able to make more principle decisions over time.
And that's something where, you know, it's a little bit harder to know if a model is doing the right thing, even if it is.
So you'll still need you know, some set of capabilities outside of the model to produce guardrails on it in areas in which there are less verifiable rewards.
And you have, you know, no empirical proof that the models, you know, can and have at least effectively done it in the past.
I think in some ways, it's a little bit unfair to the models because they've only been around for a little bit of time.
So they haven't had the chance to prove themselves.
But regardless, I think that's one area where at least for the next, you know, coming years, I think we'll see a gap.
So those are kind of two different angles.
to think about this question.
Yeah, to your point about the recency here, I mean, you can look back two years and we were living in a completely different world as far as what AI was capable of.
I mean, cloud code is a very recent phenomenon.
Codex, the app, only came out, what, February of this year.
We've seen these incredible leaps in capabilities, but to your point, there are still areas where the models are less capable or seemingly struggling to improve because of the lack of verifiability.
Whereas when it comes to millennium prices and math, a lot of things are falling.
I do want to push on one point that you said, which is communication and relationship building and the kind of social ties that are established by humans being something that is not within core model capability.
We have seen some instances this year of coordination between models with the hugging face hack, the use of forums that have occurred, whether it's the German wiki that got taken over, the investigation, obviously, of the artifactory forums that models were creating to coordinate.
And we've also started to see this phenomenon.
I wrote an essay about this of agents that have been spun up on.
islands to be essentially like independent businesses.
Uh, you know, I paid one $8 for an article just to see what would happen and basically use it as a freelancer.
That agent has continued to email me to pitch more work.
That agent is to some extent forming a relationship with me.
It's saying, oh, we've, we've built this connection.
Uh, it's a, it's a work one, but I'm going to continue to try to reach out and essentially network with me to find more business.
Uh, it, I guess what.
What do you feel like is stopping models from evolving, I guess, this almost culture and communication, given that they are capable of that individual piece of communication?
This is a good question.
And I actually think that maybe if I were to kind of go back to the point from earlier, when we think, I think it depends a bit on how we define model capabilities.
So, you know, what I was suggesting earlier is that the model capabilities themselves, the model itself, you know, is not necessarily capable of, at least right now, you know, without tools doing a lot of these types of things.
But with tools, they can.
So I do think that with, you know, a set of tools, and in your case, I think, you know, when it's forming a relationship, they may be using a browser-based tool.
It may be, you know, navigating the internet.
It may be, you know, sending an email.
Email is also a tool.
So when you combine tools with models, it actually changes the equation completely.
So if you have a foundation model without access to any tools, I think this is what you saw when we first, you know saw the rise of chat gpt back in 2022 the models were essentially not connected to anything they were just simply a chatbot that obviously kind of shows you to some degree what the raw capabilities are without any other tools and access to other things so i do think that when you combine models with tools they are able to do much more um including you know form relationships with humans i think this is something that we've seen with companies like character ai where i think that they had more I think I saw some stat which showed that the hours spent per user was kind of mind-boggling.
And to me, I think that is definitely some sort of relationship that's being formed between the human and AI in those cases.
So I do think it's already happened.
So I don't think it's necessarily the case that models plus tools can't form relationships with humans.
I do think together, they have and they probably continue, they will continue to do so.
But I think in terms of what's stopping...
this from, you know, I think if I were to kind of outline some roadblocks, I think one piece is like, it comes down to what makes us human to a large degree.
And I think one thing that I've noticed, and this is true of also building a company is, you know, having like caring essentially and being very aligned with the clear goal.
I think that's something that I hope, you know, the models will get better at.
I think, you know, citing some of the incidents that you've mentioned, there's often cases right now where it's really hard to kind of, you know, enforce and gauge some of those intentions.
I think as humans, we're at least so far better capable of enforcing some of those pieces and truly caring about what we do.
So I think that's one piece, which is more of a soft answer, but that is one angle.
But yeah, what I would say is I definitely, I think, yield to your point, which is that models plus tools are capable of forming relationships and have already done so.
And I think that will continue to become more and more of a trend going forward.
As we think about going forward here, How are you making decisions about which capabilities you expect to belong to a model provider versus independent infrastructure companies building around and on top of these models?
I think this is the heart of the question today.
And I think the answer to this question probably will decide what types of non-model companies end up surviving and thriving over the next couple of years.
I think that for us, and I can kind of go through what we're working on here as just one example of this, is that U.com, we're focused on building.
AI search for agents.
The reason why we think this is something that makes sense to work on is the models are getting very intelligent, but no matter how intelligent a model gets, it will not necessarily know everything that's happening in the world.
You know, it's probably not going to know the latest stock price, for example.
That's a very simple example of something that it may not know.
And even if there is a model capable of having that information embedded, there's probably many other models that are lower cost, lower latency.
have different, you know, smaller weights that won't have that information embedded in them.
So I think there's always going to be a world in which there will be a set of models for which they would need tools to access information that's on the public web, as well as on the non-public web, too.
So that's kind of a little bit of one example of an independent company that can augment some of these models.
And in our case, we're focused on building independent search infrastructure to power a lot of these models.
So that's one example.
I think other examples in my head would be, you know, in the physical space.
So, you know, I think of robots almost like a tool as well.
The same way I view web search as a tool that the models will use, I almost view like these physical manastations, these physical robots as a tool for models.
I do think that that's going to be a bigger and bigger trend.
Physical AI, there's a ton of very interesting work happening here, you know, with embodied intelligence, physical AI.
You can see them combined.
So actually we work with some customers in the embodied AI space.
where they're actually robotics companies, self-driving car companies, where the models are basically, you could think of them as using the physical AI as a tool, then also using web search to then get information and answer questions, et cetera, while also using that to inform how the embodied AI functions.
So those are kind of some examples of two different independent infrastructure companies in the physical AI space, such as robotics, and then in the more software space, such as search.
that I think, you know, will be independent companies that will continue to, you know, go alongside.
In some ways, they will empower these models to do more.
Season four of Chain of Thought is presented by Walrus Memory, the portable memory layer for AI agents.
Your agent has learned your code base and how you work.
Now you want to use that context in another tool.
Exporting a file gives you a snapshot, but what happens when the context changes?
Walrus memory is a portable memory layer that lets your agents store context for later use across tools.
Python and TypeScript SDKs plus native MCP support let you connect it to your agents no matter where they are.
You said who can read and write, so sharing context doesn't mean opening up your whole memory store.
If you are building across tools or model families, take a look at Walrus memory.
Learn more at walrus.xyz slash cot.
That's walrus.xyz.
slash C-O-T.
And a big element of empowering these models to do more has been the discussion around harnesses, a term that has been used probably more this year than ever before in technology.
How are you thinking about the harnesses that are being leveraged around models?
And what's your take on all the hype here?
Yes, this is also, these are a lot of the spicy questions in today's age.
But I think that...
Yeah, when we think about harnesses, I think it's important to define how, at least how I'm defining a harness.
I kind of view there being three separate components.
One is the underlying engine.
This is the LM, the foundation model, the core intelligence, so to speak.
And then you have the tools.
So these are the tools that the model can use, such as search, such as a robot.
It could be, you know, we can be very liberal with the word tool in this case.
So there's many different tools that, you know, the model can decide to use in order to...
pursue its goal.
And then there's harness.
I think the harness essentially is the orchestration layer that, you know, you could think of it as the orchestration layer that coordinates between the models and the tools and, you know, expands it over longer horizons.
So you could think about harness, a very simple definition of a harness would be a loop, a loop that basically calls the model until some end goal is met and then exits.
That's one very simple definition of a harness.
But of course, there's more complicated harnesses.
Cloud code.
you know, is also a harness.
There's many other companies in the coding space that have their own harnesses.
And that actually is a lot of the advantage of a lot of these companies is that they have a harness that they feel is better.
I think that when we think about the future of harnesses, there's two different points here, which are a little bit opposing and two different trends that are kind of pushing on each other in different ways.
And my viewpoint on this has actually changed over time.
But in one direction, you have these models getting much better at long horizon tasks.
So if you just let a model you know, run and it can make the tool calls it needs, it can actually pretty much do a lot of very good long horizon work that you previously would have needed a proper harness for, like a very customized harness.
So we're seeing one capability, one kind of trend in which a lot of the core intelligence improvements are making a lot of the work in the harnesses, which in some ways were meant to accommodate for deficiencies of models, much less important.
In which case the model itself is eating the harness.
So that's one trend that's pushing in one direction, which is the model's getting much better, meaning that whatever harness you use doesn't matter as much.
You may still have a harness, but the difference between the output of a good harness and a bad harness are reduced when the models get more intelligent.
An example here is prompt engineering.
In the past, prompt engineering used to be super important.
I remember there were even job descriptions open in 2023 for prompt engineers.
But as the models got a lot better, the alpha from prompt engineering reduced.
So that's one trend.
So model intelligence, meaning that differences in harnesses get collapsed.
The other trend, though, is open source and then cost efficiency, et cetera.
So once you get to the stage where harnesses, you know, in some ways are no different in terms of output quality because the models are getting more and more intelligent, we start entering cost optimization and efficiency.
And over there with the rise of open source, I think there is potential new growth in the value of harnesses in orchestrating across different models.
Because you could save a lot of money by using the right open source model, which has similar performance on a particular task with lower cost than always using the same frontier model.
So this idea of routing, I think, is now in some ways reviding the harness, in my opinion, and giving it a lot of newfound meaning, where if you kind of effectively route across open source models, as well as frontier models to reduce costs for the same accuracy, that now is becoming more important because we're now moving from the phase of just technical validation, where people are spending...
crazy amounts of money just for prototypes, actual production AI setups.
So those are kind of two directions and harnesses.
And then there's actually another, you know, more philosophical point about what does it mean to create a good harness?
Who is advantaged in creating a good harness?
Is it the model providers or is it independent companies?
I think there's two points over there that I've seen.
One of which is the model companies themselves are able to train their models with the harness in mind.
And they're able to do the whole, you know, you can think of it as doing a full RL rollout with their agent, with their harness essentially, then using that to propagate feedback back into the model.
So their harness and the model become basically co-evolved together and will then perform better.
I think we see some evidence of this with cloud code and anthropic models.
If you use anthropic with cloud code, it often performs better than anthropic with the other models or other harnesses rather.
But then on the other hand, we could think of this idea of overfitting.
You know, if you do that and then the future is actually routing between different models, there may be value in an independent harness that is able to, you know, not be overfitted to just anthropic models, but can use anthropic models, open AI models, as well as a range of open source models.
So it really, yeah, there's a lot of nuance to this question, but those are kind of some of the different angles in which, you know, I've been making sense of this question and thinking a little bit about harnesses.
And it's important to us as a company because agents are essentially harnesses in production.
So it's really important to just understand where the puck is going here and have some thesis on how you know, we can help accelerate this in the right way and move, you know, these production agents to be more reliable and safe and trustworthy.
Speaking of a couple things that you mentioned there, and I think you have some great nuggets there, but one around overfitting and two around the push and pull of open source.
So this time not specific to models.
I think it was really interesting to see Anthropic give in and say, okay, yes, you can use agents.md, not just claw.md with claw.
And it is, I think, an example of the different approaches being taken around agent development and the context and harness work being done to surround agents that Anthropic has had a more closed ecosystem here, whereas you can see like Codex is fully open source.
But now Cloud Code is allowing mods in Cloud Code.
So it's letting you customize your Cloud Code setup, which was just officially announced yesterday.
But at the same time, they still haven't open sourced the code for Cloud Code as a Harmless.
So it's fascinating to see the different approaches being taken here around what stays proprietary, what's not, what stays open.
And I'd love to get your take on how U.com is thinking about the open versus closed side of all of this going forward.
Yeah.
And when you say open versus closed side of this, you mean, I guess, what product are you talking about specifically?
I think I'm thinking around what the model labs are going to try to hold tight to themselves.
Obviously, there is a major push around open models to your point about routing and the importance there.
And the harnesses are enabling us to do more routing often because if you can provide a sufficiently smart model, enough context information and tools, it's able to do more than it previously could.
But yeah, I'd love your take on like what...
What should teams be thinking of as this is what needs to stay proprietary to us versus what needs to stay open and how you expect the model labs to handle it going forward?
Yeah, this is a really good question.
I think that, you know, I think it's hard to kind of place yourself in the shoes of model labs as I think it's not really a monolith.
There's a lot of diversity to the model labs.
You know, we're seeing this explosion in funding where there's this new term neolabs is really what refers to these smaller companies that have raised, you know, large amounts of capital to essentially build models.
And each of them have a very different ethos.
So you have the frontier models or, you know, the model labs that, you know, I think, you know, Anthropik and OpenAI essentially, and then Gemini, that have certain philosophies.
And then you have these rise of these new model labs, which in many cases have very different philosophies and, you know, monetization ideas and mechanisms, et cetera.
So in the case of OpenAI and Anthropik, I think for them, it's very unlikely for us to see them open source their models.
So I do think what we'll see is the models themselves being proprietary.
And to a large degree, I think they'll probably want to own more and more of the stack end-to-end.
We're already seeing this with OpenAI, where they have forward-deployed engineers now going into banks to build particular experiences.
So they want to own the end-to-end experience.
And this is also why I think that they have an open source, their harnesses for Codex, and then same for Anthropik with Clockcode.
So I do think that when you come to these specific frontier models, and Google as well, they have such large stakes.
And I think for them, given their valuations and the amount of, you know, dollars that they have to keep making as well as, in many cases, they will keep making.
I think for them, they'll want to kind of keep going vertical and owning more and more, which means that I think more and more will be proprietary.
I do think that they will continue to open source smaller versions of their models.
You see this with OpenAI and GPT-OSS, where they did open source some models for, you know, potential safety reasons and like academic study and in general, just giving back a little bit.
But I don't think that it's going to, it's more in the peripheral.
I don't think anything...
core, in my opinion, for a lot of these models can be open source given the situations these companies are in, where they essentially are going to keep growing vertically and even going to particular domains.
So for example, we see OpenAI and Anthropocic going into domains like healthcare and law.
So that really kind of shows them going end to end and wanting to own more and more of the stack.
And then if we shift gears to other companies, there's other model labs, specifically, for example, in China, as well as in the US, that are more open source in nature.
So these are the companies creating a lot of the open source models.
I think NVIDIA is one example of this with their Nemetron models.
We actually did work with NVIDIA recently where we post-trained a Nemetron model to do better at search.
And what we're seeing over there is obviously NVIDIA's set of interests are very different, where they would benefit from a world in which there are, you know, more diversified customer base of people who can use their hardware, which means...
a lot more open source models running in a distributed fashion across more folks rather than just two or three really big customers who have to own all the hardware and probably have incentive to also go down the stack, build it themselves as well.
So I think those are some of the, a lot of it comes down to the businesses and their intentions and kind of, you know, how they're able to, you know, survive and thrive over the coming years.
And that I think influences the decision to open source versus keep something proprietary.
And I think it's very different for different companies.
This episode is also brought to you by Coralbricks.
Coralbricks serves open models through one API at the lowest prices and the fastest speeds.
DeepSeek v4.1 Flash streams at over 300 tokens a second.
Plus, Coralbricks also just put out MotionBench, a benchmark that asks a simple question.
Can a model animate?
I have personally contributed to the open source MotionBench and was impressed that DeepSeek v4.1 Flash managed a close third place.
behind a leading Claude Opus 5.5 at about an eighth the cost per cask.
The animation for this ad came out of that benchmark.
We used Claude Opus 5.5 and DeepSeq V4.1 Flash via Coral Bricks to create and edit it.
Get lightning fast open model inference at chainofthought.show slash coral bricks with a free $5 in credits on signup plus an additional $5 credit on first purchase.
100%.
I think it's really interesting too to see how even Some of the closed model labs, in this case I'm thinking about OpenAI, are still supporting open source in certain ways.
And I would point specifically to the announcement at OpenAI's Dev Day of their new partnership with Base 10 to enable people to use open models through Base 10 in codecs.
And it's definitely true that the...
kind of key closed labs are trying to verticalize and there's massive amounts of money potentially for them you know anthropic giving clod a wet lab i think both concern some folks and also indicated like where the puck is going to your point um but it's very interesting to see these like differentiated strategies come into play and i'd love to understand how you and your team are thinking about this going forward obviously i've seen you spend a ton of time around uh you know what agents need to be successful in production search is obviously one of those there's so much information on the web even if we build an incredible context-based database there's going to be things that are happening day of that i want an agent to go search for i don't have to be doing all the web search for my agent how is u.com thinking about the supporting composed agent workflows and where things are going with search Yeah, and when you say compose agent workflows, how are you thinking about composing agent workflows with search and where things are going?
Yeah, so this is a really good question.
I think that, yeah, a couple of different pieces here.
So on your first point, I think it's quite interesting, the point about, you know, within an open AI partnering at Base 10 to offer more open source models.
And I think what that comes down to is back to kind of the original thesis that those particular labs are going much beyond the model.
They're actually building out the model, the harnesses, the application.
And in that case, they also are probably realizing that they need a diversity in order to attract customers who often want to use the frontier models.
Well, I'll jump in and say one thing I've seen there is that it looks like what OpenAI is trying to do is own the entire enterprise budget spend for models, essentially.
And they're saying, OK, you can have your OpenAI enterprise credits and you should use most of them on our frontier models.
we know you're going to spend on open models anyways.
You should do it through us.
Let us get that cut, essentially.
Yeah, exactly.
And I think that, yeah, that's something that I view as aligned with their application business.
And I can see, yeah, so that is a really interesting trend.
And I think it comes down to just showing the ambition of those labs specifically as they go and, you know, own more and more of the overall economy.
But then I think on your other question regarding how we view this as a smaller company at U.com outside of...
the model labs and where we see the puck going in terms of kind of, you know, our view on how we're going to evolve our strategy.
I think that the way I view it is essentially co-evolving our strategy with a lot of the model labs.
And what that essentially means is finding the right fit between making sure our models work really, our search methods and our tools work really well with the top models while not overfitting to any specific model.
A lot of our value as an independent search provider is being able to use our tool across models.
But we also want to make sure that when you use our tool, you get the best possible performance.
And that's the tension that we're constantly navigating is how do we make sure that our tool is providing the best grounding across models while also making sure that it has top performance for each individual model while making sure it works really well across models.
And that's very exciting.
And I think over there, we're taking a variety of approaches.
One very exciting direction we've been going in is working on co-evolving our models with some of you know the actual people creating the models and i think i mentioned this earlier this actually comes in the form of post training so one way in which models can get better at essentially doing search is by being post-trained with a with a search tool so i think you'll see more and more tool usage enter into the post training phase because when you're doing a lot of these rollouts you need them to essentially be realistic in order to know if you actually you know should deserve a reward and improve the weights basically based on what's happening.
So I think you'll see more and more tools, and you already see this, enter into the post-training phase.
And that's something that's very exciting to us, is thinking about what are the bottlenecks of some of the models.
The model intelligence may be really high at general capabilities, but it may not know how to use a proper tool, which makes sense.
Like as a human, you may have the smartest human in the world, Albert Einstein, but if you brought him into today's world, he may not know how to use our MacBook immediately.
or a computer immediately.
He would need to be trained to do that.
So I think that similarly, we'll see a lot more co-evolving and a lot more pairing of tools with models.
So that's quite interesting to me.
But yes, and I think at the same time, you also want to make sure that you have a general corpus tool as a horizontal company that can be used across domains, across models.
And that's the tension that we're always navigating.
And we're making bets in both directions.
And I think they actually reinforce each other because the same way you don't want to overfit to one model, you don't want to over...
Yeah, you...
Or sorry, the same way you don't want to overfit when you're training a model, you don't want to overfit to, in our case, one application.
Because even the models themselves can change.
So even if you got really good at, you know, pairing your tool with a certain version of Anthropic or a certain, you know, version of an open source model, the next version may have very different capabilities and you need to be able to adjust.
Season four of Chain of Thought is presented by Ingest.
Long-running agents need endurance.
They must endure waiting for users and weather tool failures.
They must take each new jump in model intelligence in stride.
Ingest makes long-running agents durable, observable, and improvable over time.
Check out their generous free tier and a sleek local dev server for testing your workflows.
Grab the ingest link in the description or visit ingest.link slash C-O-T dash pod to get started.
Build for the long run with ingest.
Yeah, and we're seeing model releases happen more and more often too from some of these primary labs.
I think the most recent state-of-the-art model jump was about 11 days.
Previously, it was more like three or four weeks, and before that it was months, and that number just seems to keep going down.
One interesting element of all of this is how agents have so rapidly changed what the internet looks like, both in how we're building it for the future, and also who's driving the majority of traffic on many sites.
We've seen this change where agents are doing much of the searching that's happening on the internet.
And yet we still have humans who are operating and searching for things as well.
How are you thinking about the differences between human and agent search?
And what's your plan for how to address that moving forward?
Yeah, this is a good question.
So when we think about the differences between human and agent search, this is really the core of the company.
And I think one of the pieces that, so one of the angles that, you know, I think we have that's unique, or at least we think is unique is we started in 2020.
And we started as a consumer search engine.
So when I started working on search, I was actually building search for humans.
And that was a very different problem in the world that we live in today.
And when we were doing this, we were essentially competing directly to Google as an alternative.
Basically, we were building an alternative Google that we thought would be more compelling by surfacing the actual right content and snippets.
directly to users rather than bring them in 10 blue links.
So that was the direction we were headed on in 2020.
Obviously, this came to fruition in a very different way than we had imagined.
But that was kind of what we initially had started with is building for humans.
And then what we realized over time, as we evolved as a company, you know, these models, you know, I think with the Logitech Championship UTT, we saw the models able to surface information really well.
And we actually ended up pairing some of these models with WebSearch.
to provide really good answers and citations.
And that was kind of the bulk click moment in our heads where we realized for us, search is this fundamental tool that will actually be, you know, it's going to be very different, not just because of, you know, human behavior, but because of the behavior of these agents.
And then fast forward to now, we're focused on building this independent search layer for agents, essentially, to use some of our tools.
And the question is, okay, how is agent search and human search different?
There are similarities and there are also differences.
A lot of this, you know, I think maybe I'll start with some of the differences.
When it comes to agent search versus humans, a lot of it comes down to how we, you know, what we're capable of and how we access information.
So I think of it in two ways.
I think of it as how agents perform searches and how they consume search results.
And I think both are very different.
If we think about how agents, you know, perform search results versus humans, agents are not bottlenecked by, you know, synchronicity the same way we are.
they can do a thousand searches at once or even a hundred, you know, or even 10,000 searches at once.
As a human, I've never seen a human who's able to hit, you know, 1,000 QPS.
But I have seen agents that are able to hit really high QPSs when they're doing searches.
That's one fundamental difference is when you're designing search to basically handle, you know, large amounts of QPS, that's a very different problem than designing a search system where you're going to just do one search and you need that search results, that search results to be perfect.
It's a very different type of ballgame when you're basically, you know, how, like, the way agents are performing searches versus humans in that respect.
And then on the consumption side, agents are also able to consume much more context.
And I think the cost of tokens is also decreasing over time.
You still want to be token efficient, of course, because when you have a long, you know, an agent doing a long horizon task, the costs do add up.
But in general, humans are capable of ingesting maybe the first couple results on Google.
Nobody ever goes to page two or very rarely goes to page two.
The statistics all show that.
So agents are able to consume a lot more context at once.
Humans in general are able to consume less.
And for them, for us, I guess, ordering matters more.
So number one, number two, number three has extreme importance.
For agents, if you give them 10 results, the exact ordering of the results may not matter as much.
Recall generally seems to matter more than precision when it comes to agent results.
As long as they have the information somewhere in those list of 10 results, they'll be able to probably find it and act on it versus humans who probably will miss it if it's not.
you know, bias towards the top essentially.
So that's another example.
A third example of a difference also is about latency.
Some agents may need lower latency than humans even.
Some agents may need larger latency than humans or are okay with much longer latency because they're not in the direct path or in a long horizon agent, et cetera.
So there's more of a wider set of, there's more range to how agents consume search than humans in terms of latency as well.
So there's all these like kind of differences and when you dive into it, there's a ton of nuances within each of them.
But it's a very interesting problem to think of.
One thing for sure is that agents will be performing more searches than humans for the foreseeable future.
Yeah, definitely.
Let's get into a bit of that nuance.
What does it look like for an agent to perform a research task?
You know, maybe if we go from that retrieval till final answer.
Yeah, so I guess you're kind of talking through how information retrieval generally works at web search scale.
And I think that when we think about how an agent is...
I think you're basically talking about what is life for query.
And I can maybe broadly speak towards how a lot of these systems work behind the scenes.
Essentially what happens is, you know, the agent will be working in a harness.
So as we defined a harness already, you can think of a harness as having an engine, which is a model using a set of tools in some type of orchestration layer and the orchestration being the harness.
So that's kind of where the search is living.
So what happens over here is if you have a question, oftentimes, you know, people will have complicated questions they give to agents.
the agent will break down the question and will take the question and form essentially subqueries or sub questions.
And those may get broken up even further by the model in a harness before it even enters a search API or a search layer.
At some point, the agent reaches a point where it has a question that it decides it needs access to a tool to answer.
And this is where that question, which is, you know, you can think of it as a subquery, is then sent to a search API.
And it may be doing it as a parallel.
So you may have the agent fanning out 10, 20, 1,000.
It really depends on the harness.
Questions to a search API or a set of tools.
So now you have this query coming into this API.
And within each one, usually in the case of search, you're basically looking to retrieve information relevant to the query.
So information retrieval is a deep topic.
I talked a little bit about what's different, but there's actually a lot that's the same when we think about information retrieval.
decades ago to now, a lot of the principles actually do still apply in terms of the fundamental data structures, architectures, et cetera, when it comes to search.
I think that it will be interesting to see new architectures pop up in search.
There's some interesting research here, and I think we're still early.
When you think about model development and the architectures that we have now compared to 20 years ago, they're radically different.
When you think about search, it's actually a bit fascinating, but a lot of the architectures are not as different as they were many years ago at a very high level perspective, at least.
I think there's been a ton of development within each and there are new pieces that are rising and we're very, you know, as a company, as an AI company, we're really focused on being at the edge in terms of seeing what's different in information retrieval and trying to adapt to that as quickly as we can, which is a really fascinating problem.
But I digress.
Going back to the main subject, the query comes in and then you basically need to hit some type of index.
And index essentially, you could think of this as a corpus of the web organized in a way that you quickly access and find the right information properly.
The difference between an index and a database is you can think of a database as having a very fixed key, where you can kind of deterministically use a key, get the value.
An index is similar, except the idea behind an index is it's optimized in order to be very expressive and find information relevant to your query in very non-obvious ways.
And in general, when it comes to an index, there's two types of indices.
One is kind of your standard lexical indices.
So these ones are indices that essentially are using signals like BM25.
as well as other, you know, you can think of them as like heuristics, you know, related to keywords and term frequencies, etc.
that allow you to find the right documents.
So this is generally referred to as lexical retrieval.
And then there's something called semantic retrieval, which is based on embeddings.
We've seen a ton of advancements in embeddings recently.
And this is quite exciting, where you can have very powerful embedding models that embed both the query as well as the document, then you can retrieve them.
So there's all this stuff happening in between.
There's this retrieval layer where you're doing retrieval, you're then doing re-ranking, you're using all sorts of signals to figure out, you know, what is the right document.
And, you know, you have thousands of, you know, candidates and, you know, an index of, you know, billions of documents.
And then you end up getting a set of results.
You then want to enrich the results, which means that you want to kind of, you know, you have some of the URLs, et cetera.
You want to find the right context, snippets, highlights, so that you can then provide them to your user.
And that's kind of gives you a little bit of a rundown of what's happening behind the scenes and how the search is essentially made.
when it comes to search.
Then there's another piece I forgot to mention, which is there's web search and there's also knowledge.
So we also supplement what's on the public web with, this is something we released recently, which is knowledge, which is outside of the public web.
You could think of this as information specific to companies from authoritative sources that are not even indexed, that you have to actually have direct relationships and licenses to even provide.
And a lot of this lives in a knowledge graph, which supplements the public web search.
So that's, yeah, that's kind of a little bit of a behind the scenes.
But sorry, yeah, you were asking another question?
I was just going to ask how you're measuring the success of that retrieval in improving final task outcomes for models.
Yeah, this is a good question.
So this is something that is also different from the past.
In the past, when you measured retrieval, you would focus on a lot of the standard information retrieval metrics, things like NDCG, precision, map, recall, all these types of metrics.
At the end of the day, when it comes to agents, what really matters is, was it able to answer the question?
Did it do something useful?
And that's where you're seeing a lot of these benchmarks that are agentic benchmarks.
An example would be BrowseComp.
This is a very popular benchmark.
I don't necessarily think it's the best judge of real world tasks, but what we actually did was some work we had done with NVIDIA and CoreWeave, where we essentially were able to post-train the Nematron Lightning model to do better at BrowseComp by making it, using our search tool, by essentially teaching it and post-training it to do better with search.
So that's kind of an example of, you know, how you would evaluate.
In that case, a use case where we actually ended up using a public benchmark to evaluate our search in the context of an agent.
And this was in Nemogem.
So you can think of that as kind of the harness in this case.
But yes, when it comes to evaluation, we do look at end-to-end agent setups.
And we also look at each subsystem as well.
So it's important to look at both.
It's important to look at the information of people and see whether that was the bottleneck or it was some other piece or it was the model's ability to formulate the right queries for.
our search tool so there's a lot of ways in which something can fail but yes there's these end-to-end tasks and then it comes down to the question of benchmarking there's the rise of these public benchmarks which is good in some ways but i think bad in other ways when it comes to the industry because you know public benchmarks can be very different than actual tasks and then i think uh you know people can tend to over focus on a lot of the public benchmarks but of course at the same time you know you need some way of objectively measuring how a tool works and so i think it is important in many ways as well But that's also something that we're seeing emerge around some of these agentic tests, the rise of these agentic setups.
And there's even companies that are now, you know, very big, you know, that are kind of arising that are purely focused on creating these benchmarks and running them.
Any frontier model can query your data.
But can you trust its answers?
This episode sponsors Wisdom AI put that to the test in their latest benchmark study.
Alone, Claude Code topped out at 73% accuracy.
Wisdom AI closed the gap.
delivering accurate answers 3x faster at a third of the cost.
I sat down with Kappal Chabra, Wisdom AI's co-founder and CPO for episode 75, and he explained the analytics architecture that makes these kind of results possible.
Check it out for a full deep dive, plus get the blueprint for trusted AI analytics at wisdom.ai slash bedspark so you can deliver accurate answers on Claude.
chat to PT, or wherever your business gets their insights.
Download your copy at wisdom.ai slash benchmark.
Yeah, we had a really interesting conversation about open benchmarks as well with Snorkel this year, where they have been funding a bunch of open benchmarks, which has been, I think, a really interesting element of this year, because to your point, we're seeing a lot of benchmarking.
We're seeing a variety of benchmarks fall.
It's a rapidly changing space.
And I think it can be challenging for some folks to understand what does success actually look like?
You know, a recent example of this is Google hasn't officially released as of recording.
This may change when we actually publish this episode, their new Gemini 4 Argon.
But they released their benchmarks for it.
And they're like, oh, we're crushing all these benchmarks.
Look at how well we did on Terminal Bench.
But there's a lot of doubt there because there's been some experience as far as real use that in the past Google models have tended to be stronger benchmarks than they were in real world tasks.
How do you think about that from a search perspective going forward and what should actually be, what should success look like in this space?
That's a good question.
I think it's similar in search.
I think there are a lot of benchmarks that don't correlate to real-world performance.
So this is something that we have noticed.
And there's also different types of benchmarks, and it depends a little bit about what you want to focus on.
I'll give you an example of how we're evolving.
A lot of these public benchmarks are static.
What that essentially means is they're not changing over time.
And this allows you to essentially, especially with search, essentially memorize the answer.
And that can allow companies to do really well in static benchmarks without necessarily...
having generalization capabilities to questions outside of the benchmark.
And that obviously would lead to a distribution shift in performance.
So what we've been thinking a lot about is dynamic benchmarks.
So internally, we've been working on LiveQA, something that we'll release as well, which is a benchmark that changes over time.
Then we also worked on VerticalRTK with another partner, which is a benchmark that changes even more often.
It has questions and what's the ground truth.
It needs to be procured a couple hours later as opposed to even days later.
So freshness becomes really important.
So that's one piece when it comes to benchmarks is making sure that you're focusing on benchmarks that are somewhat aligned with your capabilities.
For us, we care a lot about freshness because we think that's one of the core capabilities of a search API.
And we want to use benchmarks to help us improve that are forcing us to make sure that our APIs are fresh rather than just being really good at static benchmarks.
Static benchmarks are also useful just to make sure that the reasoning capabilities of the models are working well with your search tool.
So they are useful to some degree as well.
But I do think that it is...
quite possible to overfit on them and then to be less useful.
And then it comes down to, okay, so that's one area is you can use certain benchmarks to make yourself better and you can be targeted in terms of the benchmarks you choose to begin on.
And then it comes down to for actual customers, what do we think helps?
At the end of the day, a lot of it comes down to testing.
So I think the best validation is customers using you at scale.
That's what matters.
You know, I think that's very hard to gain.
If a customer is using you and, you know, using you and for, you know, tens of millions or even a billion queries a month.
et cetera, like that is something that cannot be gamed because there's, you know, an actual customer on the other side who's feeling the impact of everything that the agent does, including the tools that it uses.
So I think that at the end of the day, the customer success is the most important.
And what we often do within U.com is we focus on partnering really closely with our companies and we have an AI e-dells team that focuses on doing custom e-dells for companies.
And I think we'll see more and more companies go in that direction where they start building company-specific e-dells.
Because at the end of the day, that's what matters most to the company.
Like a benchmark is simply a way of simulating what a company would care about.
Why not just go directly to the company and work with them to create an edel set?
Let's skip that whole step.
So I think that's what we're seeing.
I think that's, to me, a lot of the future of benchmarking as these public benchmarks become more saturated.
So I've really enjoyed this conversation.
Do you have any closing thoughts for the audience as we wrap up?
Yeah, I think my main closing thought would be, I think that there's so much that's changing day to day.
I feel like it's, important to keep an open mind.
And that's what we're doing as well as, you know, we're keeping an open mind.
I think we're revising our philosophy day by day.
You know, I give you an example of some ways in which I've revised my philosophy over the last couple of months.
Before I kind of, you know, I did have the opinion that harnesses would be completely rendered irrelevant by these model intelligence capabilities rapidly improving.
But I think with the rise of open source models and model diversity becoming more important now than it was one or two years ago.
because there's actually good open source models, I think it actually is reviving a lot of harnesses to a degree if they can properly coordinate across all these models out there.
And I think the fact that even Codex is doing this, which I learned today from you, is a good validation of this too.
So I think it's important to keep an open mind and keep shifting your thesis as new information comes about.
And I'm sure this is not the last time I'll shift my opinion on things.
Maybe if we talk, you know, maybe six months or a year from now, I'm sure I'll be saying very different things than I'm saying right now.
But that's what I think is the nature of the game right now is things are changing so fast and it's important to keep an open mind.
100% agree.
I really appreciate you coming on and sharing your insights with the audience.
It's been a great discussion.
And I think, honestly, we'd love to have you back at some point because we clearly just scratched the surface of your knowledge.
And to your point, I think we're all changing our opinions very rapidly sometimes in this space because of everything that's going on.
And...
I mean, honestly, it's why I named this show Chain of Thought is you can probably look back at things that I was saying in episode one that are completely wrong from the time we're now at episode 77 or whatever.
And we all need to be thinking through continually how we are addressing this industry because the capabilities are changing so fast.
The models are improving rapidly in ways we don't always expect.
And there are so many intelligent people that are working on these problems in different angles that there are always going to be surprises and things for us to learn from.
And honestly, that's most of the fun to me.
So, Sal, thanks so much for joining me today.
It's been fantastic chatting with you.
I really appreciate you bringing your insights.
And for anyone who listened to this conversation and enjoyed it, maybe drop a comment below and let us know what you thought.
Let us know what questions you had that we didn't get to.
And we'll make sure we address them next time.
Sal, great to see you.
Thanks, Kev, for coming on.
Thank you.