# Context Quality as the Durable AI Moat

**Podcast:** Chain of Thought | AI Agents, Infrastructure & Engineering
**Published:** 2026-06-03

## Transcript

Think about like any task that like you might have difficulty like solving now in a generalized fashion.
Basically, instead of like over engineering your stack, if you just like wait six months with a more general agent harness, like it'll probably get better and be able to solve the task.
Is the AI framework era over?
Let's talk about it.
Welcome back to Chain of Thought, everyone.
I am your host, Conor Bronsden.
My guest today has argued publicly earlier this year that the AI framework era is over.
the scaffolding layer that he actually helped build is collapsing into the model itself, and that context quality is the durable moat once agent loops get good enough.
In fact, he runs a company that's betting on that thesis.
I'm delighted to have Jerry Liu on the show today.
Jerry is co-founder and CEO of Llama Index.
He built one of the most installed pieces of AI plumbing of the last three years, the indexing and retrieval layer a generation of reg apps were stitched together with.
and then took Llama Index into a new era by pivoting the company into managed document infrastructure with Llama Cloud and Llama Parse.
He's made a really quite aggressive bet that the framework layer he helped popularize is the wrong place to play long term.
You've probably seen him at a conference giving a fantastic talk.
You've likely seen him on Twitter slash X.
Jerry, great to have you on Chain of Thought.
Welcome.
And where are you joining us from?
Yeah, thanks for having me, Connor.
I'm joining from, you know.
beautiful San Francisco.
It is always a pleasure to see you when I'm in the Bay Area.
And I feel like I often just run into you at a conference too.
You are so busy these days with the success of Lama Index.
And one of the ways that you have made the company so successful is with this aggressive bet that what you built originally is no longer the approach.
The AI framework era is over, as you put it.
What specifically died and what survives?
Yeah, so maybe there's probably a little bit of nuances to that statement too, and I can kind of help clarify it to the audience.
But just to kind of trace through an evolution of the company, you know, we started off as kind of a core set of abstractions in open source land to help like developers back in 2023 build these initial applications over large language models.
And at the time, you know, none of the patterns were fully formed yet.
You know, people didn't really quite know what agents were.
And people were just starting to get the hang of this idea of retrieval augmented generation or RAG or feeding your private context into a model.
And so I think basically when there's a space where things are still being figured out and then, you know, there's kind of a lot of flexibility in the end application.
And that's where there's kind of value in a framework layer because the framework provides, you know, the core abstractions to make it easier to build certain things.
And if you think about like life or just like software in general.
It's basically just a series of like abstractions layered on top of each other.
And so we were kind of just the open source abstractions at the base layer around the LLM that made it easier to both connect our data, use different types of LLMs, and then also experiment with different techniques like retrieval, being able to do some sort of like query rewriting, and then eventually some sort of agentic reasoning harness so you can plug in like tool calls and that type of stuff.
And so I think basically...
The reality is over the past three years, this space has obviously evolved quite a bit.
The models themselves have gone exponentially better.
But what that also meant is, you know, the kind of harnesses and the agentic applications themselves have also gone a lot better.
And the core architecture and patterns around certain types of agents have solidified.
And so it's not like, you know, frameworks in the abstract sense are kind of like...
no longer relevant like universally you know obviously there's still web frameworks there's still a bunch of like frameworks in general that even like today like agents want to be able to use like certain types of abstractions on top of others so they don't have to like rewrite everything from scratch but the reality is like a lot of the old patterns of just like you know really trying to figure out the different internals and mechanics of an agent are less relevant today because a lot of those have solidified into these like general agent harnesses If you look at Cloud Code, OpenClaw, you know, Manus, kind of any sort of like deep research agent out there, they all kind of start following general patterns where you start having abstractions appear at a higher level than at the code layer.
So whereas at the code layer, you know, you might need to import like Python classes that wrap like different LLM providers.
At a higher level now it becomes how do you define like a natural language set of skills, an MCP tool.
How do you like program this like engine that's already running?
And so there certainly is room for like a set of abstractions at this layer too.
And, you know, there are plenty of companies kind of building stuff in this space, whether it's like open code or kind of like, you know, these protocols, being able to help support this like new age of agent programming.
I think for us personally as a company, we did start off at the framework layer, but fundamentally, I think our mission was always connecting data without webs.
And so I think throughout the years, you know, we could have operated, I guess, continued to operate at the framework layer as it went up in abstraction.
But I think for us, we decided to really identify and kind of solve an opportunity where we realized, you know, even as the AI agents were getting better, the context layer itself was not solved.
You know, a lot of folks were uploading like unstructured document based data into these systems.
And a lot of that data was just like not represented the right way.
And a lot of the legacy kind of OCR technology themselves, you know, was not actually doing a good job at reading information out of these pages.
And so we kind of basically narrowed our focus as a framework to kind of becoming core infrastructure to help provide, you know, high quality context to these evolving AI agents, which is, yeah, a little bit different than being at the framework layer.
But we do think, especially in our position right now, is something that...
should be durable even as these agents improve.
Now, of course, there's still plenty of space for other types of companies around the stack.
A lot of folks are kind of building various types of context that plug in through MCP or skills.
Systems of record, that's still a thing.
As I mentioned, there's still kind of higher level abstractions in frameworks.
It's just like those are kind of a little bit different than what we started with.
Yeah, I love this idea of context quality as a moat.
Because increasingly it has become like a watchword for folks this last really like six months in particular, I feel like, where the push to have great context has just exploded as coding agents have gotten better and better.
Data has become, you know, we've known it's been differentiated for a long time, but it is just becoming so clear that to get the most out of, you know, the next frontier model.
Being able to fuel it, you need to have great documents, great data for it.
And I look at the way we are approaching enterprise AI today and how much data is still locked away in silos, and both from a tribal knowledge perspective, but very much so from a document perspective.
And it seems incredibly clear that this pivot or focusing, as you put it, has a ton of value for enterprise in particular as they say okay how can i get all of the massive corpus of documents and data that we've established over the you know maybe decades that we've been in existence into extremely useful high quality content that can be fed to our elements of choice whether that's you know an open model or whether that's you know a frontier closed model so i would wonder from your perspective Have you seen a behavior change that either drove your decision to make this pivot for Lomindex or is now driving this kind of next phase of the company?
Yeah, it's a good question.
I think the context need has kind of always been present, even since when we started the framework.
Part of the reason the framework itself got popular is because, you know, literally, you know, you had all these large language models.
They had a 4,000 token context window.
I think the first thing a lot of builders tried to do was, you know, how do I just get this to understand my company knowledge base?
And obviously the company knowledge base was not going to fit within the 4,000 context window, whether it was your Notion database, you know, your kind of set of Jira tickets, you know, your Salesforce, that type of stuff.
Like it just wasn't going to fit.
And so how do you figure out patterns to kind of inject all ones with the right set of information at any given time, which, you know, gave rise to these techniques of RAG in 2023.
And so I think even from that point onwards, there is some, almost like by definition, need to kind of provide context to an LLM at the time in order to make it actually do stuff.
And I think as the AI space has evolved, like the LLM part evolved into some like general agentic harness.
You know, there's Claude, Covert, Claude Code, OpenClaw, Manus, like some of these like general agents implementations that you can basically use from a third party provider.
And then, you know, there's all the things you can plug into the agent harness.
So MCP tools, skills, the actual task that you want to define.
By definition, this is almost like the, this is the context layer, right?
And the Frontier Labs are building these incredible generalized reasoning capabilities that need access to like external services, tools, and context to actually do things.
And so when I think of the context layer, it's literally anything surrounding the model, like the set of services that it has access to in order to do things.
And this extends, this could be anything.
Like it could be an existing piece of software that the agent is able to use.
Like if it has MCB connectors to Confluence or Salesforce, you know, those are providers of context.
That's why I think people think this idea of a system of record is still going to be around because agents need a way to actually store.
act and operate on data.
And then, you know, there's like web context, being able to actually crawl the internet and being able to efficiently search for things so that the agent has access to like live information.
There's structured data, you know, stuff locked up within SQL databases.
And for us, it's like document context, like stuff that's locked up within, you know, these unstructured document containers within your file system in the form of like PDFs or PowerPoints that need advanced technology like new age or cr to actually read information off the page um and so i think uh for any builder or basically any software company today one of the boats is actually just being a provider of like tools and context um to the ai agent or maybe to put it another way i think a lot of software companies are trying to figure out how to make their software and services more agent native and also based on consumption um and so you know instead humans using uh your sasful or humans, you know, reading your documents, or humans doing, you know, searches in Google, you now have agents actually going in and, you know, calling a ton of MCP tools and writing scripts and doing these calls.
And a lot of the world is basically redesigning these patterns to make it suitable so that agents can, like, launch massive volumes of queries through any piece of software to both, like, get information and also do things.
And so I think that's kind of how it's evolved over the years.
But even from the beginning, like almost like, again, by definition, like you need like kind of this idea of specifying some sort of task plus like additional data for AI to do things.
And I think the like functionality of that has only expanded over the years as AI agents have gotten more advanced.
Yeah, the fact that even Mercury for banking has a CLI now, I think really says a lot.
about what we're trying to provide.
We've, not to say that developer experience has gone away, but agent experience is as important, if not, I would argue, much more important, especially not just now, but in the months ahead.
We need to be providing all this information to agents to make them special.
And part of this is because, you know, native function calling has gotten so good.
Construction following has gotten much better.
Agent loops are at least good enough.
And the models are just so much smarter than they were two years ago.
I think you used to be able to see this major gulf between certain frontier models.
And particularly since, you know, DeepSeek really shot onto the scene, but I would look at Kimi and others where open models are not far behind.
Now, I should caveat, I haven't gotten access to mythos yet.
So maybe there is this massive leap coming that I'm not seeing.
But right now, it feels like a lot of the differentiation that you're seeing between companies is how effectively are they able to give an LLM data and context and tools and provide the right governance to just go and run, essentially.
Yeah, yeah, exactly.
I think basically all the generalized frontier labs are converging on kind of similar ideas with like some slight implementation differences.
And the idea is, you know, they're all kind of settling on this idea of coding as the first layer of abstraction that, you know, they're just going to get really good at helping to automate.
And in the process, because coding is basically a proxy for computer use, then, you know, you can kind of use some of the techniques and capabilities there and use that to basically automate any type of knowledge work, not just software engineering.
That is why Clyde...
interfaces today are so popular because they're basically tuned for these like coding agents are really good at writing bash and writing code and actually just enabling them to do that to basically operate any interface.
So I think the frontier models honestly are probably all going to start like they're all going after kind of like the same type of things.
You know, maybe one day it's like Opus is better at like financial knowledge work.
Maybe GPT 5.5 is like a little bit better at certain types of coding.
They're all kind of going after it.
And I think open weight models are catching up.
I mean, I think it's kind of impressive to see kind of the diversity of like parameter sizes are basically almost like simulating what, you know, Opus, like four was, you know, like a few months ago or half a year ago with like a smaller parameter size model.
So the advancements are great.
I think for everybody else in the space, it just means think about like any task that like you might have difficulty like solving now in a generalized fashion.
And like basically instead of like over engineering your stack, if you just like wait six months with a more general agent harness, like it'll probably get better and be able to solve the task, assuming you actually just like specified the parameters on the task, right?
Yeah, I totally agree.
And I think your point about Frontier Labs in particular, solidifying around coding as this first abstraction layer is so apt.
A, it's just a great way to communicate with computers, which enables so much, particularly as we, it becomes so much cheaper to automate.
We now can do differentiated automations, not just linear ones.
But I would look back to a conversation I had with the poolside co-founders on the podcast I've got a little over a year ago now, something like that, where they talked about this idea of coding is the way to AGI.
And, you know, it's a bit of a hype phrase, but I think we're really seeing that come to fruition where it's like, oh yeah, everything is converging around being excellent at coding and then being able to automate tasks because of that.
And we're just seeing that push forward.
Yeah, I think it's definitely a proxy.
I think it's because the way your computer is designed is that, you know, if you like code and program around it, it basically gets you to most of the way of what you want anyways, because there's like the command line, you can like write scripts, you can execute tasks, like every service has like an API that you can just call.
And, you know, like...
With coding, you get a lot of the way there.
I think, yeah, the next part is probably just like vision capabilities.
I think with advanced vision capabilities, you can start looking at things that, you know, not just at the code or like text level, but, you know, actually be able to take like an entire computer screen and just like operate it at a visual level so you can know what to click and, you know, know what you're looking at, zoom in on certain things.
I think that's probably the next step that every Frontier Lab is going for is just like combining.
coding, which is an efficient way of like, you know, writing automation and operating over interfaces with just like vision and even like operating over video streams so it can just like more intelligently take actions.
Yeah, it's interesting to see this all converging as well with robotics and the approaches there of, okay, how can we now get these to take more intelligent actions in the real world?
It's going to be fascinating to see how this all develops the next few months.
But in particular, as you mentioned, The way to make this effective, whether that's trying to have a million door dashers wear a camera so that your robots can get better at delivering food, whether that's getting as much excellent code or bad code that I wrote back in the day into your model for training so that you can be great at coding, or whether it's trying to get your particular business to...
or your particular set of agents to understand your business through the document context and other contexts that you've established over the years.
It all comes back to, you know, training data on context.
And I think this is a bit of a simplification, but I would argue that, you know, we can kind of look at context just like a crucial type of data.
And it's almost silly for us sometimes to not conflate the two terms.
Like, obviously there are reasons not to, but like...
Anytime you say data, you could also think of data as just like context for a model in the right situation.
And you've made this bet that context quality is what's going to compound over the next couple of years.
Can you talk to me about why you're so convinced this is going to be durable instead of just another, you know, commoditized step?
Yeah, for sure.
Well, you know, I can talk a little bit about maybe like documents in particular, but, you know, start with the overall space.
I think in general, you know, if you think about the context layer, I think, as I mentioned, the overall like agent harness is kind of a blank slate.
Like this is the Frontier Labs will give you, you know, you have access to cloud code work.
You have access to like cloud code.
Like there's no kind of starting thing.
Like you basically have to put in the instructions to actually get it to do things.
And I think like once you start like operating with that kind of mindset, then.
You know, there's certain types of tasks where you have to give it a lot of instructions by definition for it to actually do things well, because not only is it specific, like, can you specify that task in like a single sentence of natural language, you now have to give it a lot of context.
Like, for instance, if I'm trying to, you know, like prepare for a customer call that's upcoming on my calendar, it's not like I can just say I need you to prepare for my customer call.
I need to actually feed it the, you know, previous five to 10 transcripts on the customer.
actually look into kind of the email threads of what the recent conversation history has been.
And so there are just like tasks that by definition need like a lookup of kind of external sources of data.
And, you know, you kind of need ways of actually providing that data for the task to kind of be more well specified so you can actually solve it.
And I think the way I think about vertical AI, which we're not doing, but, you know, a lot of these vertical AI companies is basically packaging context in a way that's like a little bit easier to kind of solve certain tasks instead of relying on users to do it themselves.
You know, I think Claude Coburg is such a blank slate and I'm pretty sure if you equipped it with the right like tools and MCPs and skills, you could basically use it to do anything that like is possible within the current like, you know, realm of possibility of like AI agents.
But it's just certain things will require you to put in like you know, a lot of work, like you would need to write and really tune like some giant skills file plus like MCP tools for it to solve like some sort of knowledge task.
And, you know, the reality is most people either don't have time or they don't know how to or some combination of both.
And especially if you look at all these like AI solutions in certain verticals, like whether it's financial research or kind of healthcare, I mean, who knows, like, you know, Anthropics kind of building some of these tools too, but basically, you know, like by providing a tailored interface to solve a task, it just makes it way easier to kind of actually do the thing with AI versus like relying on you actually having to program this like blank slate to actually do everything.
And I think that's kind of where the value of a lot of like vertical AI provides is I almost think of vertical AI as a form of like context to help like solve more domain specific tasks.
Now kind of going into a little bit what we do, because context itself, I think, is like super general.
It's just like literally any set of like inputs you provide to the model, including both like inputs plus like task plus, you know, external data.
We're specifically interested in like document context and turning a PDF into a markdown or some extracted form of information that's like easy and accurate for the agents to understand.
I think.
The way that we get compared is against both kind of legacy OCR tools.
There's like 25 plus years of document OCR tools that their entire job was to convert PDFs to like markdown or some sort of text.
And the other alternative is you feed an image of like the screenshot directly into the frontier model and then basically just use the frontier model as an OCR tool.
So I think like that's kind of like.
that that's kind of the basis of comparison here and i think for us um we kind of see a lot of opportunity in building something that's just like extremely high accuracy um at low cost if we just like focus on both like very fine-tuned models plus like agentic harness techniques um only on document understanding um i think the reality with all the frontier models um All the Frontier models are getting better at like general visual understanding.
You look at Gemini, Opus, GPT, every new release, they're starting to climb the benchmarks on like general visual understanding, but they're also just tuned for coding, reasoning, and a lot of other tasks.
And especially if you're trying to like unlock context or just like parse a million PDFs, you really need something that's both extremely high accuracy, but also really cheap.
And I think the way the frontier models are tuned, they're just not incentivized to actually do that because they're tuned for like high intelligence tasks.
And so what we basically want to provide is, or the opportunity that we see is providing just really deep tech to both like parse and extract documents with extremely, extremely high accuracy, low cost, plus like being able to provide relevant metadata annotations and tags on these documents.
that then provides these AI agents access to almost like audit trailers back to the source document.
If you can trace back to the exact words and draw a box around the words that a legal contract came from when an AI agent generates a response, that's a very powerful tool in a lot of vertical AI stacks, whether it's a big company, a small company.
And so we kind of think of ourselves as building just a general toolkit plus all the surrounding metadata.
That's also really high accuracy for AI agents to access like any pool of documents.
And for us, like hopefully it should be enough of a moat that is differentiated from like users trying to do it with frontier models themselves.
Are you seeing particular verticals adopt your stack, you know, more aggressively?
Yeah, for sure.
You know, I think for us, the typical verticals that we go after reflect industries that.
have a lot of their contacts locked up in documents.
And I totally acknowledge, like, there are a lot of companies that honestly, like, they don't have a lot of documents.
If your entire, you know, like, stack is basically dependent on, like, code, for instance.
Like, we don't typically index, like, code bases.
That's typically the domain of, like, coding agents.
But within legal, within financial services, within insurance, within manufacturing, healthcare, education, government.
And then basically the tech versions of all these, there are a lot of use cases where they're just like either humans or just a lot of like inherent knowledge bases that contain, you know, like millions, if not more of like just unstructured document based data.
And this is kind of like the basis of a lot of projects that if you go into these companies, they're really trying to automate a lot of workflows that typically humans used to do.
and using kind of AI to actually understand all the information within these docs, extract out the right information, and also take actions.
And I imagine accuracy becomes a major differentiator, especially as you're working with regulated industries that are touching customer funds.
Yeah, that's actually super important.
And the reason why I think it's also hard to just like DIY or in stack with a frontier model.
I think what, if you look at Opus 4.7, Jupyte 5.5, you know, they're...
they're like not bad at understanding like medium complexity documents.
But it kind of gives a deception of like, it will get you 80% accuracy over your document corpus.
Well, there'll be like 20% of documents that just like it'll hallucinate some number or, you know, it'll just like represent a table wrong.
The issue in a lot of these use cases is that basically makes it below the accuracy bar for like what you can actually operate with.
Because with 20% hallucination rate, that means if you actually, you know, try to extract out information or answer questions over that data, the downstream AI agent workflow is going to be disrupted.
And kind of any sort of like straight through automation with agents, it makes it a little harder to trust given the kind of gap in accuracy.
And so a lot of these require like really high accuracy, you know, 95% plus, especially in kind of more sensitive use cases.
With human in the loop review on some of the...
extracted data.
So are you seeing a tension between agentic pipelines where the model decides what to extract and then deterministic pipelines that have more on ability?
How is that functioning with these regulated enterprises?
Yeah, I think maybe just to reframe the question a little bit, I think there's kind of two use cases for using AI to automatically extract information from documents, which is basically what we do as a company.
And so I know our kind of mission, our tagline was, you know, you can kind of use our stuff to unlock context from documents and feed it to AI agents.
One practical instantiation of that is you have a bunch of data that's in the form of documents, maybe in your box, Dropbox, Microsoft, SharePoint.
You want to create a knowledge base from those documents.
And so, you know, and put it into a vector database.
That basically means that you use our capabilities to kind of do that first pass OCR.
on top of the documents, convert it into Markdown, and then the downstream operations become like chunking, embedding, putting into some storage systems so you can actually search over it.
So I think that's kind of like one of the use cases where you're structuring context in a way such that you can use it for any downstream agent, whether that agent is like simple, you know, kind of like a low sophistication agent or something that's like super advanced like log code.
And I think...
That's a very common use case.
Basically, any enterprise wants to create some agent-ready knowledge base that can give some agent the ability to do things.
Now, the second use case is kind of related, but a little bit different in that there's a lot of these human workflows that have existed already prior to AI agents, where humans are basically just reviewing massive piles of paperwork and then doing manual data entry into systems.
whether it's like, you know, onboarding, claims, invoices, and, you know, like KYC, that type of stuff.
There's a lot of like existing software where it's really just a helper interface for humans to upload a bunch of documents, help like automatically flag some values, and then enable them to like more easily do data entry and correct like wrong values.
And I think the advantage of a lot of this technology, especially with what we have in comparison to legacy OCR tools, is it just like, massively increases the potential to actually automate the extraction of all these documents with really high accuracy.
It's a little bit less that story of like, I'm going to, you know, structure all these documents into a knowledge base and give it to an AI agent.
It is a little bit more deterministic too, because there's usually like business rules and flags you want to apply and then systems you want it to route to.
But it's still like a super important use case because like a lot of these companies just like have a lot of folks just like kind of doing this task.
That's like relatively repetitive.
And I think if like they use slightly better technology to actually understand these documents and translate it into a digitalized representation, you both be like saving a lot of money and be able to process a lot more volume.
That's actually pretty common in a lot of these industries too.
And that's where we're seeing some, you know, some kind of usage as well.
It's interesting because it seems like from what you're saying, if coding agents are basically the abstraction layer that models are centralizing around and you are enabling this other primitive of the file system to be unlocked is this telling us about what the feature or i guess what does this tell you about the feature of how agents are going to evolve based off of this like clear abstraction layer and you know crucial primitive So I'm not actually convinced the file system plus coding agents is going to be like the feature, so to speak, like a year from now, a year and a half, two years from now.
Honestly, I have no idea.
So I think I'm going to leave that to kind of whoever wants to like drop a prediction.
I do think that's kind of the current state of the art because like coding agents are a proxy for computer use and vision capabilities aren't quite there to help like, you know, these agents massively stream like real time video yet.
And so.
This is one of the closer representations towards just being able to automate a bunch of stuff on your computer.
You combine that with now this abstraction of the file system is like fly ready and it's ready for coding agents to use.
And these agent harnesses are also very good at kind of using file systems to help like navigate existing documents, you know, like retrieve the right context and then like, you know, take actions over them.
And so I think that's a very powerful abstraction.
right now that actually solves like a good chunk of knowledge work um and i think where we come in is you know again like we we basically would be one of the kind of core modules if there are like document um document based like types of data um to help like unlock contacts from those containers um into the file system going into the feature though like honestly i have no idea like i think um the thing is the There's probably going to be some new abstraction that's not just coding.
All the vision stuff that I mentioned, there's probably going to be something there that allows them to more generally see the screen and do actions.
There might be new types of operating systems that actually make it a little bit friendlier for agents to do things on top of.
But I think for me, I reason for first principles, even if the harnesses or even the core primitives change, going from file systems, CLI, and like coding to something a little bit more like vision-based.
This concept of like context, I think still matters quite a bit.
And I think for us, like, you know, I feel reasonably confident in our position of being like a durable layer, even as these like agent patterns evolve.
Yeah, I was wondering about this when you brought up the idea of, oh, the reason why we're converging on code today is because that's the easiest way to just communicate with computers, essentially.
And I was kind of going like, okay, well, we're already seeing changes in our hardware layer.
You know, it's not just GPUs and CPUs anymore.
You know, we've got TPUs, we've got XPUs, we're expanding.
Are we going to see a total rethinking of like what a computer is?
I mean, we've already seen people talking about it.
There's this whole idea of Neuralink, etc.
Everyone's trying things with vision.
I don't know, but it seems like we are at the onset.
of a new hardware wave too though it may take a little longer that is going to match agents maybe more specifically yeah it's possible um i think there might be a new hardware wave and you would know more than i do on this um in terms of just like kind of the underlying um infra and hardware that powers some of these evolving models especially as some of the architectures like evolve and change and that type of stuff i'm not actually sure that there will be like an entirely new software layer though um for for ai because uh the way it works right now with ai is like And part of the reason it's so powerful is instead of trying to like reinvent every part of the software stack, they just, the Frontier Labs just made it really easy to operate on top of existing tools everyone's already using.
So, you know, they're technically not reinventing like Windows or Mac or Photoshop or just like, or even a lot of these like SaaS tools.
The idea of MCP, the idea of like skills, the idea of like, you know, actually just being able to write code, but import like Python libraries is.
operating on top of existing abstractions that have already existed.
And so I do think it's more likely that they're going to try to build something that runs on Mac or Windows, even in a generalized computer use way versus like, you know, and then like later on, there might be a new agent native operating system.
And I'm sure like Microsoft or something will come out with that if they haven't already tried.
But like basically, like I think it's the overall AI trend is to adapt with existing tools and kind of go from there.
Yeah, this brings to mind the framing of kind of this whole episode, which is, hey, Lama Index went from an open source framework to this big pivot.
How did you do it successfully?
Because I think there are a lot of folks listening or reading Twitter who are going, I'm a little worried my startup's going to get killed by Cloud Code.
I'm a little worried that a frontier lab is going to demolish whatever I'm doing because they simply start to do it.
better they are able to automate it away how have you made Lama index so durable it's a good question I mean I wouldn't say we're in the clear yet I'm going to be totally honest I mean we're not like again we're not Microsoft's I do think like at this stage literally anything could happen like you know you look at companies are even much more skilled than we are and I'm pretty sure they're also having constant like existential crises every time there's kind of some some like new model release coming out I do think that what we have done has put us in a bit more of a durable position compared to what we were like three years ago.
And I think, to be honest, it's not like, you know, for any potential founders, like it's not like an easy process.
You like are fundamentally kind of thinking about how to disrupt your own product, your own product strategy.
And then also kind of.
maybe almost like move the audience of who you're serving.
And there's going to be a core group that remains to kind of like some other group that maybe is like a little bit more adjacent to what you were originally serving.
And so I think just like all these questions, you know, it's not an easy task.
But I do think, you know, one thing that's interesting is you listen to all these podcasts of folks with companies that are like much bigger than where we are today.
And they're all kind of like wondering the same things.
Like I think even these like very seasoned business leaders at public SaaS companies are trying to figure out how to disrupt their business or figure out how to like completely reinvent their product.
And so I guess the only thing I'll say is, you know, you're not the only one and pretty much like companies that are a thousand times bigger than you are also trying to do the same thing.
And that's harder than you trying to do it yourself.
And so I think that's one of the things I'll say.
In general, I think...
especially in this like AI landscape, everybody has to kind of be extremely on their toes and just be willing to kind of like reinvent like their ICP.
And I know that has some potential issues for kind of like continuity.
But, you know, at the same time, if you are able to do it, and even for like some period of time, it helps you establish your foothold and just be able to like serve some like emerging need in the AI market.
And the benefit is because like everybody's so interested in AI when something does land, it typically grows pretty quickly.
How did you have to alter your own, I guess, internal company strategy when, as you made this pivot, like, did you have to change your hiring processes or have you changed it in other ways due to what's going on?
Oh, yeah.
I mean, I mean, I think, I mean, this entire journey and, you know, luckily, I think we hired kind of like some core folks that, you know, are like, really invested in just like this, this overall North Star that we had, actually, I don't think has changed too much, even if the product surface and, you know, the ICP itself has changed.
The North Star was always like figuring out how to get data into LLMs.
I mean, I think that basically was how the company started.
And to some extent, that's pretty much like still what we are doing, trying to figure out how to get data into LLMs, just maybe operating out like a little bit different of a layer.
I think people understood that mission.
And I think...
part of the goal was to understand like okay what are the best tools actually serve that mission i think in the beginning it was a framework um because people just didn't really have anything really settled down so the best way to build those tools was something that just made it easier for folks to build like various types of apps and connect their data in various ways i think as those patterns emerged it became clear one of the value that you could uh or the valuable pieces you could provide is basically just like providing really deep tech around a certain piece of context because the abstractions were solidifying, but the need for like high quality production context was still evolving.
Just communicating that to the team and just accept that there's going to be friction.
And then yes, like hiring will obviously change depending on what you actually build.
You know, we need like kind of core applied AI researchers.
If you're interested, let me know to actually kind of work on the frontier of document understanding capabilities and actually you know, like do like deep tech, like ML, not just AI engineering, like actual ML, right?
Like, you know, model tuning, training, that type of stuff.
And so, you know, I think when you're a startup, you just have to adapt.
Jerry, now is when I have to ask you to drop your email or best way for folks to get in touch if they are listening and they do want to apply.
Where should they go?
Yeah, for sure.
There is llamaindex.ai slash careers.
If you're interested, you know, we're looking for a variety of roles.
I think at the time of...
Whenever this comes out, I'm sure the rules may have changed too because, you know, there's always kind of expanded needs across like GoToMarket and also Eng.
I think the other piece is, you know, we're pretty active on socials.
So we're pretty active on LinkedIn.
You can follow us on WamaIndex and also on Twitter.
You know, and I have a Twitter account too.
Some folks may have seen you on Twitter.
That is for sure.
Terry is fairly popular there to say the least.
And I will...
cosign here and say I know a lot of incredible folks at LamaIndex, so it seems like it's a cool place to work, though I haven't actually done it myself.
So definitely check that out if there are any roles that are of interest to you.
I want to bring it back to something you said before we derailed on hiring a little more.
And it's really poignant, but also I'm finding it mildly concerning as I think more about it, which is this idea of, you know, the whole job of LamaIndex is getting data into LLMs.
And it's making me think about...
What a high percentage of my own job today is just that problem.
So I guess really it just brings to mind this point that you made earlier of like, look, this may not be durable forever, but what we're doing today is really valuable because I mean, so much of the value creation happening at companies around the world today is that exact problem of how do we effectively get data into LLMs?
How do we make sure it's accurate?
How do we...
just enable these models to help us run faster if we all think that velocity and throughput is rapidly increasing to lms that we need to do the same to keep up whether it's in our own jobs or for our companies i mean that's a very valuable place to be yeah yeah for sure i mean i think um you know there's this existential question of just like how much are we uh just an outline in general but yes like also i think as um i do think as humans um we basically are kind of responsible for at least like defining the task specifications and an understanding.
It's like similar to eventually like how you would communicate to another human from a junior to eventually a senior employee at their function and be able to give them enough like context to get on board and get started.
And that includes like all the data that might be present that you don't explicitly communicate, but that they have to go and research themselves.
And so they would go in, look at your internal company knowledge base, look at your transcripts, look at your calendar, email.
and be able to synthesize stuff.
And that's basically what's happening with algorithms today.
You give like a task specification, they look at your context, they go do things, and this only gets better as the models get better.
We've talked a lot about context today, but I'm curious to get your take on agent memory, which is something that folks are starting to increasingly talk about.
We had a great episode with Rich Mindalake on it that I totally recommend folks check out.
What's your take on this and how memory and context should be interacting?
Yeah, I think memory is basically just a form of, I guess, like persistent context.
I kind of think of it as like, yeah, kind of.
I'm still kind of forming my opinions about this.
I think there's different types of memory.
I mean, on one hand, there's just like this idea of a system of record, like entries that you update in a database, just because you need like.
a table of things to store things.
I mean, just think about if you make a new entry in a CRM or you make a new entry in Notion, right?
That is a form of memory because you're updating your context within some system of record and then with the ability to store and retrieve from it.
And I think there's a lot of ideas around like how agents update their state that basically revolve around that.
It could be updating like an actual database, updating a SaaS tool.
And with, you know, Karpathy's tweet, it was around like being able to build and update like your own internal wiki, which is basically a system of record on your file system.
I mean, I think that's kind of like just a general applicable practice that literally anyone using a coding agent can do today.
You know, kind of build some internal knowledge base of just like a set of files that you can lock up later.
Oftentimes I find, especially for non-technical people or people that are a little bit, you know, too lazy to code, like myself these days sometimes, you know, like the simpler the abstraction, the better.
And so this concept, just like having memory, just be a pool of files on your file system is kind of like nice to think about because it's pretty easy to represent in store.
I think there are certainly more complicated forms of memory.
There is ways of like being able to actually synthesize like some sort of like graph from your memory or something, form like entity relationships, you know, be able to synthesize summaries.
I personally have not spent a ton of time building those systems.
I'm sure there are folks that have a lot more experience there.
And I do think maybe for some of these frontier labs, when they bake in memory as one of the features, you do a lot of research to kind of understand some of the more advanced primitives that actually enable users to store various forms of state so that they don't have to retype various types of contacts for the next task.
But I think for general users, what I'm interested in...
a lot of times is just like these dead simple abstractions that like literally anyone can just like take today and can still, like people can still use that as an initial instantiation of memory for basically any tasks that they do.
Yeah, I had a conversation, I mean, recorded this morning actually with Tyler Akito, who's the CTO of Red Panda.
And I think it'll come out the week before this episode.
So if it didn't, I apologize to everyone who's watching this and doesn't have that right context.
But he talked about a paper that they're working on where they're working with a psychologist to basically identify trends in agents and think through like, what are we missing as far as how agents think and operate in the world?
And one of the things that came up for us and we talked briefly about was this idea of...
almost like cultural memory, the training memory that we're starting to see emerge across different frontier model families.
And I think the example that is obvious to folks who are maybe too much on Twitter like myself is, oh, OpenAI really loves goblins, evidently.
Like, okay, GPT, like, sure.
Or you can look at like some of the ticks that Claude has picked up along the way and how it is kind of seen by many as treating things a little differently than the...
open AI family of models.
And I think we're starting to see this kind of emerging, almost like model lineage.
You know, you brought that up earlier this conversation, this idea of, look, the models are converging on what they can do, but there's this like different flavors almost of what's happening there.
And I'm curious if you have a perspective around, you know, what do you think is driving this?
Is it related to essentially the training data and context of being provided?
Is it...
more about the alignment approach that the model teams are taking?
Do you expect to see this kind of almost like differentiated diverging cultures occur?
Like what do you expect to happen on the model front?
I think it's a fascinating topic.
I will say I probably don't have that much experience to actually speak to it.
So I mostly speak to it as like a observer.
When I think about memory, like especially because we're kind of like the builders on top of these models.
It's typically the stuff that like is individual user-based to help like store state, to help like solve your task and less like the general memory.
I think a lot of the pre-trained memory is super interesting though, because obviously there's like personality differences between 5.5 and like Opus.
And obviously they all have like these different texts.
They write a little bit differently.
I think that's actually been one of my top complaints is a lot of these models, at least in my opinion, my humble opinion, are absolutely trash at writing.
according to how I want them to actually write.
And it might be a skill issue on my end.
But basically, I think the ability to actually have both this personality characteristics, which actually does flow into your writing too, I think a lot of it's a result of post-training.
And it's kind of interesting.
There's all these RL environments that are being sold to all these Frontier Labs these days.
I almost wonder if it's just like you do some data distribution of like...
all the different environments and sims that like every Frontier Lab is running and then you kind of correlate that to the personality type of like what it actually ends up.
I would be super curious.
I just don't have access to any of this data so I can't actually speak to it intelligently.
If someone wants to send Jerry and I that data that would be fascinating.
I think we would love to check that out.
Yeah, like, you know, let's say this was like 50% tuned on like, you know, financial knowledge work whereas this other Frontier model is like 30% tuned but like 20% more tuned on like, you know, patients or doctor.
Yeah.
doctors talking yeah i think it'd be super interesting that makes a lot of sense to me i interesting yeah i um i will bring up on the writing skills front briefly too as we're wrapping up here uh highly recommend what i've found successful is basically taking three classes of skills to help llms write on my behalf one being like writing voice context so building out like a context document of how i like to write and like building a skill on top of that and then having have a bunch of examples i can reference particularly for like different types of writing two i built a skill that i open sourced called uh avoid ai writing um it's done pretty well it's got like a i don't know 1400 stars something on it now but basically it says okay here are all the texts we see for models please just get rid of that like like think think about running them differently do these things um you might find that useful and then the third one i have is i have like an editor on top of that where i have like a second pass editor it's like okay like I wrote this in your voice.
Then I went through it with the clean AI writing skill and said, okay, like let's get all rid of all the M dashes.
Let's cut down on these like repetitive steps because models, while decent writers in some cases will often like they'll repeat themselves on like, oh, I'm this thing works.
I'm just going to use it 15 times in this essay.
And it's like, good Lord.
And then like have that second pass editor to help it out.
Yeah, it's funny.
You know, I think I tried to create like my own writing skill.
I think it roughly captures some of the stuff that you're talking about, just maybe with a little bit less sophistication.
And it's interesting, you know, we're trying to hack our way around these models, like not being great writers themselves.
I wonder if it will get postering to be better.
Yeah, I think, I mean, I think it will.
But the problem I see is that everyone, I mean, we're seeing convergence in how people write in some areas.
But I feel like anyone who really cares about writing the craft of it wants to have a bit of a differentiated voice too.
So I do think it's crucial to have your own form of post-training through context and how you provision the model and instruct it as well.
For sure.
Yeah, totally.
Well, thank you for following me on this, this brief distraction around writing with AI, because I think it's a fascinating topic that I'll probably talk about more on this podcast at some point.
Maybe I'll write an essay on it.
But Jerry, it's been fantastic catching up with you.
I appreciate you taking the time to come on the show and super excited to get this out.
Do you have any closing thoughts you want to share with the audience?
Yeah, no, thanks so much for having me on the podcast.
It was great to talk about kind of like the general, you know, macroevolutions of like AI itself and just like the concept of the context layer.
I will say, you know, I think I'm not like an expert to like 100% accurate at like predicting what's going to happen in like the next year or two.
But I do think from like first principles, this idea of like...
I think AI is something that is going to fundamentally change and get better.
So it's helpful to think about exponentials in that sense.
But it's also helpful to think about what typically stays constant.
What are just by definition things you need to enable the AI even as it gets exponentially better.
That's basically a thought exercise that we did where we thought about the context layer in general.
And for us, it meant narrowing a little bit down to focusing on document-based data.
I think for like a lot of builders and like verticals or kind of like other industries, it's just like helpful to think about, you know, what are things that these labs are likely going to pursue?
And then, you know, what are things that, you know, they're probably not just not going to have time for or can't because of like the interface.
And those are areas that you could kind of go in and dig into.
And so mine, my thoughts are just one perspective.
There's plenty of successful AI companies out there.
And so, you know, I hope.
at least some of this was like interesting.
I will definitely say it was interesting from my end.
So I hope our listeners loved it as well.
And I know they can find a lot more information from Jerry and LawmindX at lawmindx.ai, at Jerry's Twitter account, on LinkedIn, anywhere else, Jerry, they should go check out besides that careers page you mentioned earlier, which definitely they should.
Yeah, no, for sure.
I mean, I think if you're generally interested, if you do have a lot of docs, you know, and you want to talk about this, we do have a self-serve product called ModelParse.
And then, you know, we also have like a contact form on the site.
But in general, if you also have some of these use cases, you're welcome to DM me on Twitter.
Amazing.
Jerry, thank you so much for coming on the show.
It's been great catching up with you.
I'm going to have to try to write an essay for our sub stack at newsletter.chainofthont.show based off of this discussion about writing and some of the great insights you provided.
So super excited to get that out.
And thanks so much for coming on.
Yeah.
Thanks so much for having me.