# Legora's AI-First Engineering and Legal Automation Strategy

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-10-06

## Transcript

Let's say you're getting sued and there's like multiple strategies and multiple ways that you could handle this case.
That requires a lot of risk judgment, a lot of client collaboration, and I'm sure if you go to different partners at different law firms, they will probably all give you a different answer.
If there's no objective truth, it should be a human who makes that judgment.
The question I ask there is just why?
Why doesn't the AI just choose one?
Let's say I'm a client and I'm trying to get this thing done.
Could I just go with whatever AI says?
Yeah, you probably could.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
Join your hosts, Guy Pagiani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
This is the AI Native Dev.
Yesterday, you were writing code.
Today, you're prompting agents.
Tomorrow, you'll be running a software factory.
Argentic coding is moving fast enough to give anyone whiplash.
This June, AI DevCon London sold out.
Hundreds of developers, engineering leads and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify and GitHub cut through the hype and get to what actually works.
This November, we're bringing it back to New York.
Bigger, bolder and full of people actually building the Argentic future.
Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.
As a podcast listener, you can get 15% off your ticket using promo code POD15.
We have so much planned, and we can't wait to have you join us in New York.
Hello and welcome to another episode of the AI Native Dev.
My name is Simon Maple.
I'm your host for the day and joining me is Jacob Lauritsen who is the CTO of Legora.
Jacob, welcome to the episode.
How are you?
I am doing great.
Thanks for having me.
It's a pleasure.
And Jacob, tell us a little bit about yourself and also for those who haven't heard of Legora, tell us a little bit about what you do.
So yeah, my name is Jacob.
I'm the CTO at Legora.
I've been with Legora for about two years, a bit more.
Legora is a sort of workspace product platform for legal work.
So we help lawyers basically do all of their stuff much faster, much better with AI.
So it's sort of analogous to, in our world, GitHub linear cursor all in one.
That's Legora for lawyers.
Amazing.
And yeah, like, you know, a lot of people who maybe, you know, haven't...
dived into the the legal ai side of things may not it may not be familiar so it's great to it's great to hear that perspective one thing for those who haven't heard is that lagora have hit a hundred million arr across 18 months uh is that is that around correct i didn't misquote there No, you're right.
It took us 18 months to get to 100 million.
And last week, we just announced 200 million.
And that took us six months.
Wow, congratulations.
It's amazing when companies do this.
What does it look like from the perspective of the head of engineering, the CTO?
I mean, I think it's just always exciting that people get value from the product that we're building.
That's sort of the thing for us, right?
And then it's, you know, you can imagine a...
company scaling so quickly as we're scaling that it's a little bit chaotic.
A lot of things are happening, a lot of traffic.
We have so much usage now, but it's super exciting.
I mean, everyone's super thrilled.
Amazing.
And let's look a little bit under the covers because obviously that kind of like speed of growth from a company requires huge flexibility and speed from the point of view of the engineering team, the company underneath.
And we'd love to jump into some of the AI practices that you use.
From what I understand, AI writes over half of Logora's code base.
And I hear...
I think all of it.
Oh, pretty much all of it.
Wow.
Yeah, yeah, yeah.
Now, I heard some engineers can run 10 concurrent agents at the same time.
First of all...
I'd love to understand how quickly did your team get to such a level?
Because, you know, I've played with two or three and it does take time to actually, you know, go beyond a couple or a few.
And then secondly, I'd love to hear maybe some of the pain points along the way.
What were some of the problems or, dare I say, outages maybe that these kind of practices caused?
Yeah.
First question, how quickly did people get here?
And that's very quickly.
I think we've always hired for people that are extremely forward-leaning and people that challenge themselves all the time.
And when you have that type of talent pool, they'll adopt technology that is really quickly.
Going from a few agents to a lot of agents was also relatively quick.
There was a lot of tooling.
I think you're asked about pain points.
That was definitely one of them is like building the tools that let us orchestrate many agents.
We built a custom.
background sort of coding agent that we could launch.
We invested into cloud dev setups so that one agent can have its own cloud setup and run its own, you know, low close and observability and everything and send back videos and screenshots of what it's working on.
But also our own.
I mean, I have, there's a few engineers that have, that are sharing this like TUI that they built, which is like a Kanban board for agents.
And that's one way to orchestrate a bunch of agents.
Another one is we spent some time figuring out how we could have multiple work trees, work on concurrent things, but still sharing a lot of resources for local dev.
So for example, our app requires spinning up Postgres and Redis and all these types of different dependencies and observability stack, et cetera.
And then we built this custom tool that allows us to have one instance of all of those running and then sharing them across all of our workspaces so all agents have their own database in Postgres and their own prefix in Redis and whatever it is so that we can, use the resources on our local laptops efficiently.
Nice.
And I presume the parallelization here is happening across not just a single feature or piece of work, like maybe you would on a work tree, but this is presumably happening.
You're having individual developers running multiple pieces of functionality and running different tickets at the same time.
Yeah, absolutely.
And that's just on the developers that are working on their...
features or whatever they're building on now, but we have full loops that don't even involve developers or developers' machines themselves, right?
We have our buck reporting Slack channel.
When there's a buck that comes in, it kicks off a cloud agent that tries to reproduce it.
And if it can reproduce it, it sends video evidence back into Slack, and then it fixes it and writes a regression test and opens a PR.
And then an engineer, a goalie, will take a look and review and get that merged.
So we have whole sort of loops and work that runs.
outside of any actually engineer.
Yeah.
I'd love to also double down on one thing that you mentioned, which is, you know, the types of engineers that you hire, you know, they absolutely need to be that AI native.
They need to be forward AI thinking.
When we think about senior developers, who are, you know, I guess what we would call old school senior developers versus a young, well, maybe I shouldn't say young, a more, an AI first developer who hasn't been, let's say, tainted by the more traditional styles of software development.
Do you have an inkling to which you would almost like prefer or which you see the better results from?
I think the...
Generally, people that are very fast learners, that's strictly a good thing.
So people that are very curious and super ambitious to learn and try new things, and that's the type of field we have for, regardless of age or background, then there's definitely some advantages to being very senior and having seen a lot of stuff.
especially when it comes to really big scale or distributed systems or databases, where it's just very helpful to have seen some stuff.
But then it's all about finding those people that have seen a lot of stuff and have a lot of experience, but are still extremely forward-leaning.
There's a Venn diagram and some overlap there.
And those people, I find, if you have a lot of experience and you're a really good engineer and architect, and you're then also super AI-pilled and really good at using agents, then you're a force to be reckoned with.
I think that's like it.
Those engineers are just extremely valuable.
Yeah.
It seems to me sometimes you have a lot of people that are experienced but almost are held back from trying to do things a more traditional old way versus trying to fix problems that may not even exist in today's world.
And I even heard someone say that they'll comfortably substitute some of that experience for someone who is hungry and ready to essentially prioritize speed.
Yeah, I think this is really, it comes down to the person, right?
I think both are super valuable.
I think speed is extremely important.
Being pragmatic, solving tomorrow's problems are extremely important.
But I think some part of systems thinking and thinking six months ahead is also really, really important.
If you can look around corners, it's really important, especially as you scale the team.
I think if you're a small team of 10 engineers, then just the only thing that matters is tomorrow.
Like you just have to fight for survival at that point.
But if you're 100 engineers and everyone has that mindset where it's just only about tomorrow, then you're going to end up with a code base that's completely unmanageable and you're going to have a lot of incidents and you can't scale that very well.
Where if you have a few more systems thinkers, you can actually build a system where you can be a lot of people running on it and you can get a lot of gains from the technical investments that you're making.
So I guess also, Jacob, from the point of view of someone in a senior position as part of an engineering, organization, it's really important to be able to know how much your organization has adopted AI.
Now, you've criticized token maxing as a way of measuring that adoption of AI.
First of all, talk us through that a little bit.
Yeah, I think there was probably an era, let's say maybe it was at its peak six months ago, where everyone was talking about token maxing.
People had...
How did you use the tokens?
Obviously, that's not the right solution.
And so I think it's always been a really bad way to encourage AI adoption by telling people that they need to use a lot of tokens.
And I think instead, it's way more important to focus on just setting a high expectation for output and what we can get done.
And then you drive AI adoption through that as a means to get...
more amazing work done faster and highlight all the wins and do some hackathons where people show how you can work now and open up people's eyes a little bit.
I think, yeah, that's how I think about it.
And I think the industry is also coming around that the token maxing doesn't make any sense.
And it's just...
more about setting high expectations.
Yeah, it feels like the new lines of code, you know.
Yeah, exactly.
That's exactly what it is.
And when I do hear people say, oh yeah, if you're not token maxing, you're not using the most out of AI.
Most of the time they come from a frontier lab and obviously it's in their best interest to say that.
So what's on your dashboard at Lagora to understand whether you are, you know, effectively adopting AI technologies?
I'm...
Let's see, I'm not the biggest proponent of trying to measure everything.
The thing that's most important to me is are the teams delivering great stuff really quickly.
That's basically at the end of the day, that's what I care about.
Is the software building great?
Are the products great?
Is the architecture great?
And is it, you know, we're doing it fast.
And that's, you know, honestly, if a team did it without using any AI at all.
That's great.
I don't actually really care, but I don't think that's possible.
I think you need to use AI, and I think AI is an extremely powerful leverage way to get to that result.
So I mostly care about the result, and that's what I'm measuring on.
Are slow, is this nice, great?
Are bugs low?
Whenever you use the system with staff engineers and architects, does it make sense?
And are we delivering projects really fast?
Then we have a team, a developer experience team, a developer platform team that's looking a lot more at AI usage to just make it very efficient.
And so I think we're looking at things like, okay, how many turns does it take us to get to the place where we want to be?
How have we architected our system and our engine to produce code with agents in a way that's really fast and efficient and in a way that reflects how we want to build the system and how we want the system to evolve?
That's like a powerful...
way as a platform for all of engineers.
So that's something that I think a lot about is how can we, you know, invest in the right things as an organization such that all engineers are 5%, 10% more effective.
That's like, I think for many years, that's been what a lot of leaders have thought about.
But now it's actually really possible because you actually have this sort of one knob that you can really invest into, which is AI enablement and sort of AI guardrails and whatever it is you set up that actually has a really meaningful, very quick.
impact on the engineering work that happens.
Yeah.
And I guess there's like AI assistance, AI enablement.
And when we talk about the autonomy of AI and the ability for it to actually replace tasks or replace pieces of work that would normally maybe, you know, we would find less interesting or time consuming.
What's your belief on how ready?
agents and AI models are to actually start replacing pieces of human work.
What have you done in that space at Lagora?
So I think if I look at this from our product and, you know, in legal, AI can do all of the menial work, all of the stuff of like looking for a million documents to find red flags and analyzing huge contracts against.
various laws and doing the legal research that's extremely time consuming and actually very difficult to do.
Yeah, I can do all of that now.
And so it takes away all the menial low value tasks that lawyers used to be doing all the time.
And they can focus on the high value judgment strategic work, which is really why people became lawyers.
You know, no one became a lawyer to sit and look through a million huge documents.
No one's excited about that.
Whether and how we take that, you know, further.
I absolutely believe we can do that, but there's some parts that will always require human judgment, or at least for the next 10 years, say.
There's certain things where, so let's think about it in the way of verification, right?
Certain tasks can be, you can verify them, and that's typically the task that AI will be really good at.
So that could be math, coding, like you can get AI to do something and you very quickly check if it's correct.
With coding you can run tests, that's the way to make code verification.
You see a lot of people now that are optimizing towards a single measure.
Because it's verifiable, AI is really good at it.
I'm sure that you've read a bunch of blog posts and lots of people doing certain things.
I mean, the Bun rewrite is a great example of this, where there was a really good test suite, which meant that we could just do this huge thing, because we could test it and then we'd know if it worked.
With legal...
that's pretty difficult because a lot of the tasks aren't verifiable.
And a lot of them also require strategic thinking and human judgment.
And so some tasks, if it's a strategy in a litigate, let's say you're getting sued and there's multiple strategies and multiple ways that you could handle this case, that requires a lot of risk judgment, a lot of client collaboration.
And I'm sure if you go to different partners at different law firms, they will probably all give you a different answer.
And I think that's one of the tests where if that's the case, there's no objective truth, then AI won't be able to do it.
It will still be able to give you extreme amounts of value, but it can't do it autonomously, basically.
So there's a limit to certain things where you need human judgment.
And then I think our job is, how do we make an interface or a program where it's extremely efficient to sort of bring your human judgment into the system and then agents can run at all of it.
How do we make that interface with all the work that happens down here?
Because it's so much work that all agents are doing, but then make it very easy to provide human judgment into the task where it needs to.
Super interesting.
And I think, obviously, here from a legal case, the ramification of getting something right or wrong is night and day.
I guess this brings me actually to another very recent...
I say very recent, very recent at the time of recording a tool, which is, you know, a decision making model like like Jeff.
And we've actually done a lot of playing with this ourselves and specifically.
Oh, yeah, I'm sure you have as well.
So specifically in and around, I mean, like LLMs.
Yeah, they could be great.
They could be good at that kind of like decision making.
However.
a tool like Jev where, or a model like Jev rather, where it does provide you with, you know, an answer one way or another with a confidence, which is very, very cheap, very, very fast.
This feels like a no brainer that kind of like replaces huge amounts of the, I guess the decision making, a lot of the workflow potentially, and then sharing back to us as humans when confidence is, you know, below a certain threshold of confidence.
Is this going to really change the way we model?
agents, applications and workflows?
Yeah, I think that's absolutely possible.
I think it's almost weird to me now that decision models, even the early ones, the glider, whatever versions you had of this before wasn't much bigger.
Looking back, it's so obvious that a lot of AI stuff today and a lot of application AI is decision making and not writing.
You don't need an autoregressive model to do decisions.
We're using those everywhere, right, to make decisions.
Obviously, it's usually what you need to do, writing or drafting or whatever it is.
But no, it's definitely something that's very top of mind for me is how can we use this?
How can we decompose certain of our tasks?
We're already really good at doing this and introduce something like this in various parts.
We haven't yet found out if the confidence scores that you get from a decision model are trustworthy.
I remember I used to do these big classification models with classifier heads and BERT models, and generally the confidence scores weren't really trustworthy in any meaningful way.
So I don't know if I'll trust those scores too much.
I haven't looked at Jeff specifically on how much I trust those scores, but hopefully they're calibrated really well.
And if they can be calibrated very well, maybe we will fine-tune.
one for our domain that's well calibrated, then that's certainly a very powerful primitive to use across our product.
Yeah, and I was actually talking with someone today in our AI engineering team about something whereby obviously LLMs have been very much made for text and chat, but they also have structured output.
And so we were actually wondering, well, you know, if it has structured output, there is a world where we can actually...
change a model so that it does just actually provide very similar output in terms of a yes-no with a confidence level.
But I presume this is still going to take time.
It's a general model still and so forth.
But it's very interesting.
We haven't run any tests locally here to compare that.
I'm not sure if you have or if you have seen anything like that yet either.
Well, not big enough that I feel comfortable sharing anything.
We actually did have, we have a product where we're actually doing this exact thing, where we basically asked an LLM for, this is like two years back and it's still working really well, where we actually asked it to basically do this.
Like, here's the answer, here's the confidence, here's how it works.
And it's like, you're just, that's just a decision model, right?
Yeah.
So that's right in.
It's interesting because structured output is actually something that isn't new.
It is a couple of years old, right?
Like you say.
So human input, let's talk a little bit about when we get humans involved, the meatbags of our world today.
At what stage should they be involved in decisions?
And we talked about low confidence.
I presume critical decisions as well or critical projects.
So, for example, a legal case where something really does need verification.
How do you decide when autonomy can really be left alone to the agents to decide and when humans need to be involved?
Yeah, this is a great question.
I'm not sure if we've yet nailed it.
I don't know if we ever will.
But it's, I mean, there's obviously user preference.
So we can't just make a universal truth.
There's some users that are maybe new on their journey with AI and they want to confirm everything.
Just like, you know, us in Cloud Code, initially when we had to say yes to every sort of tool they wanted to run, we'd say yes, yes, yes.
And then now there's auto mode, which I think all engineers basically run it on auto mode or maybe they even ran it on YOLO mode before.
So I think we need to bring everyone along for the ride.
And so there's some user preference there where it's like maybe you want to review everything initially.
And then as you get more comfortable, we can do more and more stuff autonomously.
And I really am working towards this future where it's more the agent soliciting impact from the human than the other way around.
And so I think...
We're moving towards having a lot more proactive agents that are always on and always doing work.
And then they will come and they will say, hey, here, I need some input.
You know, we found this red flag in this contract.
How do you want us to handle it?
There are these two paths or whatever it might be.
And I think that's a very effective, you know, speaking about an interface for effective human judgment input.
I think that's an example of a really efficient way to provide human judgment.
Yeah, and I think it's one of those things that is, it's a constantly moving goalpost as well as models change and as we adjust as well and what our users expect from the product.
It's a tough one to think if you nail it, it might not be the same next week, even if you change it.
I think no, exactly.
It changes every six months, basically, right?
Everyone feels more and more comfortable letting AI do more and more and more.
And so, you know, maybe in two years, no one's going to talk about this because they will just...
We'll just always let it rip.
That's it.
And it's funny how every single time we talk about JEV being released and all of a sudden you get a new Luna.
Luna 6 comes out and it's actually in our test it's a little bit slower than 5.6 but it's actually half the price of 5.6.
So it's actually you see these model vendors adjusting constantly on the flight.
It must be so stressful being a bit...
It is.
I think I'm very excited about...
Because I think in the past year, it's been mostly focused on Frontier.
The labs are very much like, can we solve more Millennium price problems?
Can we discover new drugs?
It's very much, which makes sense because they're trying to push AGI.
But I think for a lot of application ad companies like us and loads of others that I talk to that are in a similar position, we're very excited about the entire Pareto Frontier.
We're very excited about a Luna that's faster and cheaper and same level of intelligence.
We can use that a lot.
Not all tasks in coding or in legal or in finance.
require, you know, a country of geniuses in a data center to solve it.
Some of it just requires completely ordinary intelligence, but if it's very fast and cheap, that's great.
And this is really interesting because I know at Luguri you benchmark 10 plus maybe models every single week, and I know you switch between the models pretty much.
So are you using different models for different tasks?
Yeah.
When we do this in various different ways, we decompose tasks into different sub-agents and sub-components.
We have certain different products that will use different models.
There's various ways that we decompose tasks and we use different models.
And yet we change them per task, you know, almost per week, depending on what's best.
Because we're always, basically part of our offering to our clients is that we give them Pareto efficient AI.
Give them easy to use, efficient AI.
abstracted away from them they get the best of what exists and we are really experts at figuring that out um and to do that we have to test these models all the time and always give them what's perrito efficient um and that's also why people you know go to us and law firms trust us and corporates trust us because they know that um not only do they get great ai which was the most important thing a year ago but now it's great ai at a great cost is equally important i remember gosh years now two years ago We had Des Treanor on, who is the founder of or co-founder of Intercom.
And he talked about these things called torture tests, which are just a really ragged set of tests that essentially run testing a model.
based on various different user requests and things like that.
So they essentially replay a whole bunch of some of the trickiest user questions there are.
When you're changing models constantly, is that still a thing where you actually need a really rigid, or not rigid, but a really robust set of kind of like torture style tests to make sure you're not regressing in production or are models not quite so varied these days?
No, no, it actually is.
It's definitely something, but it's not just models.
We need these both for models, but also for the harness stuff we do.
I mean, we might change the way that we decompose the task, or we might change the way we do compaction, or we might change the way that we search in files, or whatever it is.
We might change some prompts or like tool search, whatever it is we change.
You know, we want to iterate super fast on the harness and not just on the model.
And so we need these really extensive eval sets.
to make sure that every time we get something new, a hard thing or something's not working, we add it to the eval set and as a regression, eval our test.
And so we need to run this all the time every time we make changes to make sure that the agent performs at its best possible all the time.
Yeah, super interesting.
I want to come back to something that you've said in the past, which is agents are not humans and we should not constrain them to human language.
I really like this, particularly when we kind of, you know, when we think about the interactions and we talked about Jev that, you know, is now designed to give us an output that isn't supposed to be, you know, in any way like a chat based thing.
And this kind of like lends us, leads us into a kind of like an AI native interface or in a place whereby, you know, we expect.
And agents take in very different inputs and provide very different outputs maybe for consumption by an agent rather than a human.
What does an AI-native interface look like when we think about models?
Or how should we think about it?
I think there's layers to this.
I'm pretty intrigued generally about the way that we're solving intelligence via natural language.
I mean, even in robotics, a lot of robots are humanoids.
Which I think is interesting, right?
Because it's just we're trying to create artificial intelligence to look very much like human intelligence or even look like humans and act like humans.
And maybe that's the right direction, but it seems inefficient to me.
It seems like maybe a necessary first step before we can actually move to something that makes more sense.
I think there's probably a physical build of a robot that's way more efficient than looking like we do.
And the same thing for language models and for intelligence.
But that's sort of at the very lower end of the level.
I think sort of if you look at a product like ours, I think the point that I was getting at back then was that if you're doing really complex work, the most efficient interface, you know, going back to what I said before, for you to review that work and understand that work and guide that work might not be a very long single chat thread.
You know, that is what you and I.
That's how I would collaborate with you.
If we were working on something together, that would essentially be me talking to you, you talking to me, and just a bunch of messages back and forth.
But that's because that's just the human constraint.
This is the only way we have to communicate.
But people have communicated in other ways, right?
They've created slideshows or whatever.
And agents can do all this sort of stuff right off the bat.
So I think when you do a really big due diligence analysis, you're now analyzing.
you know, millions of documents, giving me the output of that huge analysis as like a long chat message is not super efficient.
And so we've built a product, which is like a big Excel, essentially sheet that has all the files as rows and data points as columns.
And that's a very efficient interface for me to review the work because I can very quickly go around and see what's actually happened.
And so I think that's like, that's an example of how we think about certain, when we do really complex agent, agentic work was like a good interface to interact with that.
And this is one example.
There are more.
It's a, you know, constantly moving target.
I don't know what it ends up being the right solution, but I'm not convinced that everything is just going to reach out.
And I presume that extends to your context as well.
in terms of, you know, your context doesn't necessarily need to be human readable or in a form that's, you know, consumable by us.
It kind of makes me think, you know, the black box is turning even more closed as well.
And how do we actually, you know, understand if it's getting the right data?
And we, you know, we rely sometimes on verification or tests, but, you know, it's getting harder.
No, yeah, I think you're right.
But I think that's also solvable in the same system.
Like that needs to be part of the system of interaction.
Part of the system of interaction needs to be auditability and understanding, especially in our domain.
If you're working on an extremely important case and you're a lawyer and you get an output from the core, you need to be able to verify it.
Right now we have multi-hop citations, basically, which is like everything is citable.
You click on it, you see where it came from, and that might be from a different report that was also AI generated, but then you can see the sources down to the source documents or multiple source documents.
So you can easily follow the chain of reasoning.
And I think that's super important.
And no matter how we build our system, we need to ensure that it's auditable and understandable and reliable by that humans know what's happened and can say okay to this.
And staying on context, actually, the way you talk about it there, where there is obviously so much data and there's very strong links and citations between various documents and things like that.
How complex a problem is?
context and keeping your context windows a reasonable size, making sure you have all the data.
Is this a significant pain when you have such a large volume of data to be referencing?
Significant pain?
I don't know.
Everything we're building is a significant pain.
Is this the most significant of your significant pains?
I don't think it's the most significant, but it's definitely up there.
Again, the goalpost keeps moving, right?
Initially, people were using Lagora for very simple things, and now they're using Lagora for multiple hours.
They have agents that run for multiple hours and do extremely complex things.
And then this type of stuff is important, and we need to figure out how to solve that, and we have different approaches and different methodologies to do that.
But yeah, it's a difficult thing to build an agent that can work for the human equivalent of days.
That's a difficult thing to do and very few people have solved that yet.
And that's something that we're continuing to work towards and we're making really good progress.
And I'm very excited to see what the future brings there.
Yeah, it's very interesting.
And you once said, I think, talking about context rot, that the solution for context rot isn't to actually grow the context window bigger, which sounds like it might be a reasonable thing.
Have you been burned by this in, you know?
working through Lagora or is there a better path you found?
You mean using models that have bigger context windows?
To solve the idea of like stale context, context growing too quickly.
Yeah, yeah, yeah.
This is also a great question.
Will we just get models that have 100 million token context window and then there's no problem at all?
Maybe.
It doesn't look like that's the direction that things are going.
I do think...
Yeah, it's been interesting.
If you look back over the past four years or three years or whatever, there was a very big divergence in what AI was.
It was a lot of very different things and different models and could do different things.
And you decompose a lot of inbuilt systems.
And then over the past years, it's conversed a lot.
And it's just LLMs.
And there's three different tiers, maybe.
But they're basically all the same.
And I'm wondering if we're starting to see a little bit of a diversions again with things like decision models and autoregressive models.
And if we are, then...
It opens up for so many new different types of systems to be built and solving really complex problems.
Again, might be decomposing them into having decision models take some decisions, then it's like a tree, and then it does this other thing.
Or maybe decision models end up only being useful in a small sliver of places, and the LLM stuff keeps going, and we keep growing context lengths, and they just get better and better and better, and then all the problems are solved.
I don't know yet.
I think that's...
We just have to see and we have to play both and see what works best for us and where the industry is going.
Hey, everyone.
Hope you're enjoying the episode so far.
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
Whether that's talking about the latest tools, the most efficient workflows or defining best practices.
But for whatever reason, many of you have yet to subscribe to the channel.
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button.
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
Alright, back to the episode.
I don't want to leave this conversation without talking about how we verify answers, how we verify solutions or answers that are given to us.
So we'll talk a little bit about the verifier rule, verifier problems, and then I want to talk about litigations briefly as well.
So we'll talk about the verifier rule.
This argument really states that litigation strategy is basically unverifiable.
So if you have five lawyers, you'll get five answers.
Now, that means, I guess, that you know, there's this ceiling in terms of how well or how good agents can ever be because no matter how good the agents get, they're never going to be able to arrive at an answer if five different lawyers can't get to that same answer anyway.
So what does good look like here and how do you verify against that?
So I should start by just clarifying verifiers rule.
basically states, and I mentioned this a little bit before, basically states if a problem is verifiable, then AI will solve it over time and it will do well.
And verifiable means that it basically means a few different things, but generally it's like there's an objective truth and it's very quick to verify.
It might really take a long time to solve, but once you have a solution, it's very quick to verify it and it's very cheap to verify it.
And so math is a great example of this, where it's very easy.
for an LLM to output something and you can just run the actual math and see if it's right or wrong.
And the same goes for code if you write tests first.
And again, it's very quick to...
And generally this rule has been applied to model training and the actual capabilities of a model.
But I think the same thing goes for just AI systems generally.
If you have an agent and maybe the model itself isn't great at the same math, if you put it in a harness where you just let know it's wrong.
no, it's wrong, then it will converge to the answer, it will get there.
So that's the verifier's rule.
And the way to your question here, will AI ever get so good that it will give us an answer to the things where there's no objective truth?
We can probably get an answer, it's just gonna be one of many answers.
I think what's interesting to us there is that that's the point where you need human judgment to us, right?
That's a human, whoever's like, wants to get this thing done whoever orders this task to run should have that judgment and not an ai if there's no objective truth it should be a human who makes that judgment um and i the question i ask there is just why because if you have five different humans yeah you might as well take an ai's judgment for it you mean very much like you know it it depends which human you ask at that stage if you ask five different humans that are all lawyers you might get five different answers so why doesn't the ai just choose one and and it's a tough it's a tough reality right It is, and I think you make a good point.
It's like, okay, if I was, let's say I'm a client and I'm trying to get this thing done, could I just go with whatever AI says?
Yeah, you probably could.
I think the difficulty is the humans that are there, that you have, that you hire your law firms, they have all the context for your business and what's important to you.
They know you really well.
They know the risks that you're willing to accept and not accept.
They know where your company is going in five years.
previous issues you've had.
Can you put all that data into a system and then have AI do it?
Probably eventually you will, but I think right now that doesn't come in automatically.
If you ask an agent, hey, you know, draft whatever strategy we should have for this litigation case, it won't have that context.
And I think that's why you still need that human to be like, of all these five options or these different paths we can take, I know the market super well.
I know the client really well.
This is what makes the most sense.
Over time, might that change?
Yeah, maybe.
It's an error in knowledge rather than an error in judgment here at this stage.
It's that the agents don't have that level of knowledge that we actually take for granted.
Super interesting.
And I guess when the agent does get it wrong or does pick something that is potentially not what a human would do, what does that then mean?
I guess in certain cases that could be a lawsuit, right?
Have there been any...
you know, lawsuits that have been raised.
Well, I mean, it's interesting, right?
Because it's, you know, who's responsible?
There's a philosophical meta question to this.
And there's like the lived reality today.
The lived reality today is that AI is still a great tool and we make it very easy to verify what it's doing.
We don't guarantee that it's always the right way that the best.
lawyer in the world would do it.
And so all lawyers still look at it.
They're still just like you and me.
I'm still responsible for my PR that I put up, even though I did it with Claude, or even though Cursor wrote all the code.
I'm still responsible that this improves the system.
And it's similar with lawyers that they...
It's their output still.
We help them a lot.
We guide them, but it's still their output.
And they can very easily in our system verify that this is what they wanted.
This is what makes sense.
This reasoning makes sense.
And no, we haven't had any lawsuits.
I expect to ever get one.
I'm glad, I'm glad.
So why don't we wrap?
This has been super, super interesting.
And I think it's very cool that a lot of the time we talk about people who are building applications.
Those applications are perhaps used and embedded into other systems and things like that or used by a user directly.
But this is a really interesting kind of use case where it's ultimately...
I don't want to say a dangerous space, but it's a space that can have vast implications.
Yeah, absolutely.
The consequences.
So let's take it a little step further and say, okay, if you had to bet, what would you say is a genuinely unverifiable, very judgment heavy piece of legal work that today agents would defer to a human to make a decision?
But what would you say is the first, you know, of that type that an agent will actually?
end up doing well?
And one of the first pieces, what would you say is next on that journey?
I actually think there are quite a lot of these where humans are still in the loop now that agents will be able to do it.
One example is when you draft a contract, you might be asked to draft a new contract for X, Y, Z reason.
Then typically lawyers will look at what's called precedent.
What have we drafted before?
And so they'll go and search basically and they'll be like, okay, we have similar clauses.
for similar types of clients in similar situations, in similar jurisdictions.
It's like a multidimensional search, essentially.
You have to find one that makes sense and then you have to apply judgment in like, would I rather take this precedent for a client that's similar, but in a different jurisdiction?
Probably not.
Would I rather have a similar client in a similar size of a company or similar, like a smaller client, a smaller company in the same industry?
Would I rather have one that's, you know, whether the client asks for the same things?
It's like a multidimensional judgment, which one is the right one, and maybe there's three that are right, and then from that you draft.
That's an example where AI will be able to do this very well, because it can apply judgment in like a multidimensional case, or with very little steering from humans.
That's just the first example that I think of, but there are actually, there are many of these.
Yeah, very interesting.
I like the way that you're not actually taking the pure data in there.
It's actually taking historical data on how a human has interacted with it in the past, which is super interesting.
When you previously talked about, you know, it depends on that company, their level of risk and things like that.
Well, once a human's done that several times, you can essentially put that into the system.
Exactly.
That's really exciting.
Yeah, very, very nice.
Jacob, this has been amazingly good fun chatting with you about this.
I really, really appreciate it.
Where can folks who are interested in this kind of legal application, where can they go to learn about it more, to start playing with it, get more information about it?
Go to our website, lagora.com.
Go to lagora.com slash careers if you're interested.
We're looking for really curious, super ambitious engineers in New York, in London, in Stockholm, and in Malamö, which is...
by Copenhagen.
And we help relocate and we help with Visa.
So anywhere in the world, if you're a really ambitious, you're a really great engineer, you should totally apply.
Amazing.
Well, it sounds like a great fun environment to work in.
So it sounds excellent.
Amazing.
Jacob, thank you so, so much.
Really appreciate all your insights.
Appreciate it.
Thanks so much for having me.
Thank you very much.
And I hope you enjoyed that.
A really fun episode on the INA TiffDev.
And tune in to the next episode soon.
Bye for now.
The AI Native Dev is brought to you by TESL, the package manager for skills and context.
Your hosts are Guy Pajani and me, Simon Maple.
Our producer is Tom Dowler.
The AI Native Dev is not just a podcast, it's a community.
And we host monthly meetups at the TESL offices in central London.
Visit tesl.io forward slash community to learn more.
And I hope to see you there.