# Building Trust Layers for Agentic AI Systems

**Podcast:** Product Momentum Podcast
**Published:** 2026-10-15

## Transcript

So I want to start with, you talked about something in your talk about the agent trust layer.
So what is the agent trust layer and why is it important?
Yeah, so in the Q&A, we also talk about like silos.
So today you have silos, human working with humans.
You also have silos, human working with agents.
And the agent many times work in a black box, more or less.
So how do you know agents are doing the right thing?
And how do you know they're getting you the desired outcome?
And it has always been a question for like, you know, ever since AI existed, is how do you design the objective function for the agent?
How do we build that kind of box around something that's probabilistic, right?
Like an autopilot, it's still just it's following like a defined workflow, but it seems way more complex when it's a probabilistic system.
Yeah, absolutely.
I think there is Verification is what we said, you have to design metrics so you know if the system is doing the right thing.
You also design guardrails, which people talk a lot about today, which is like how do you make sure the wrong action gets blocked.
And actually it's very interesting in the early days of AI, like what we did in building the guardrails.
And the last piece people don't think often about is observability.
So after a system that's deployed, you want to constantly track if it's doing the right thing.
As the systems get more complex, how do the people build and maintain the fundamental knowledge and skills to be able to evaluate the machine, right?
How do we make sure as product people, we're maintaining that knowledge?
Yeah, and it's more complex because you write the evals, which is humans trying to use metrics to know what the machine is doing.
The host of verifying the verifier.
And that is really hard.
That's why today you have an expert in the loop.
which is very common if you talk about like ROHF and things like that.
So you have a human expert to look at the result.
Now here is something harder.
There are things that don't have ground truth.
You mentioned something also in your talk that I thought was unique because you used a unique term that I haven't heard a lot about.
So can you tell us what building the harness is?
Yeah, it's basically how do you make sure the system works at scale?
So I think that's the problem.
people's challenge today.
As product managers, we used to build deterministic systems.
I have a set roadmap.
I know exactly what I want to accomplish.
I know exactly how the system behaves.
I know exactly what user care is going to be.
Now with AI, it is populistic.
And it's very difficult.
I see people kind of are on two sides.
One is they just unleash the populistic nature of agent.
And you're just like surprised what you get.
And that may not always be a good surprise.
Dan, we just got done talking to Alexa Zhang from G2.
I'm like winded from that one.
I don't know about you, but like that was one of the like we've made a point about having AI experts on, but this was the first like person that was like in AI from like the ground floor up, right?
She's got 20 years of experience.
What did you take away?
Well, first I took away that 20 years takes us back to the early 2000s and not to 1985.
So that's making me feel a little bad about myself.
I love the framing and the other talking about trust and.
how we put the appropriate constraints upon agentic solutions.
Because they're so complex, things are moving so fast, it's so much more important that we're bounding them the right way.
Really awesome conversation, like a lot to think about coming out of it.
Yeah, take a deep breath and get ready for this one.
Let's do it.
We are here today with Alexis Zhang.
She is the chief product officer at G2, the world's largest software marketplace.
And you've spent like an entire career building AI systems.
You just got off the stage after your keynote and you've done and you did the Q&A afterwards.
So this is like the third part of your media tour for today.
Thank you so much for being with us.
It's always fun to talk about AI.
Right.
So one thing was an interesting contrast between your keynote and some of the other ones we've heard today is you are at like.
You are talking about things that like the cutting edge.
Right.
And so when you are up on stage talking about these ideas, do you feel like you've got to find some way to connect with people that may not be like at this level of like, OK, not everyone has a career building.
Yeah.
Building AI systems that can say that now.
Right.
So how do you find a way to like connect with like people that aren't there yet?
Yeah, I think that's.
Always, sometimes I feel I'm living in my head a little bit.
And you definitely see a different conversation when you talk to people who's building on a cutting edge.
And you're talking to people who's still learning and still getting ramped up.
But I think that gap is going to narrow over time.
I actually see the most impressive are interns.
Actually, there's a thing we have right now at G2 called AI interns.
We hire people from college, fresh out of college, or even...
high school grads and who's doing incredible work on AI.
Like we do these emails that I just talked about.
It used to take our engineering team like, you know, three days to run like a runoff emails.
And we had this AI intern who's an incoming Stanford freshman.
And Eason, if you're listening to the podcast, did this in like eight hours or something for the first time.
So I think increasingly we're going to see a new generation of technology workers who's just like native to AI.
And that's going to close the gap even more.
Okay.
Yeah, I was thinking about when you were talking, I was like, she's definitely talking to the 0.05% of people that Martin was talking about that are actually working with these tools.
So this, essentially what I'm framing is this podcast is going to be an advanced course for everybody, right?
So you might need a dictionary or have Google open while you're listening to this so that you can look up some terms.
Unless you're driving.
Unless you're driving, right?
So I want to start with, you talked about something in your talk about the agent trust layer.
So what is the agent trust layer and why is it important?
Yeah.
So in the Q&A, we also talk about like silos.
So today you have silos, human working with humans.
You also have silos, human working with agents.
And the agent many times work in a black box, more or less.
So how do you know agents are doing the right thing?
And how do you know they're getting you the desired outcome?
And it has always been a question for like, you know, ever since AI existed.
It's how do you design the objective function for the agent?
And there's like multi-objective optimization.
There's also reward hacking, which you probably heard about.
And I heard somebody else talking about this.
And I thought it was very smart.
And they said, if you think about your robotic cleaner, if you tell the robotic cleaner to like clean this room, what they're going to do is they're going to push the trash in the next room.
So that is what evals try to do, is to really understand what the agents are doing in production and to make sure they're achieving the outcome as humans we actually want them to achieve.
Because otherwise, the agent will take the letter of instruction literally, and you may not end up with what you ask them to do.
And you have to do that with metrics.
And this is like kind of age-old thing.
AI always write a metrics.
and you write an algorithm to make sure it's doing the right thing.
And that is becoming more and more important.
And it's actually not that easy to write a matrix.
Is it to force you, number one, to ask yourself, what do I really want to achieve?
How many times you don't really know when you set out to build a product or something like that?
And then you also need to be able to express that.
in a metrics or in an algorithm.
And then you also need to feel like, you know, once you see what actually happens, you need to be able to adjust it.
You have to ask the question, do I have the right lubric?
Do I have the right eval data as well?
And, you know, is the agent behaving as it's expected?
So, like, that's the beauty of, like, writing the trust layer.
Okay.
And so you influence that trust through evals?
You, I think it's less about, you do influence, I think you're right.
You do the evaluation so you know if the agent accomplished the right thing.
And then you also influence, you do the steering.
This is actually another interesting topic.
Today I hear people talk more about rolling versus steering.
Okay.
So we're doing less and less of the rolling.
So agents are doing the work.
But you still need to steer because otherwise you don't know where the agents are going to end up.
So the eval also help you to kind of like set up the steering.
It's like once you do the eval, it becomes kind of the objective function.
This is what I try to achieve.
And then you are able to do better steering.
So.
While you're talking, I was thinking about one of the troubles that people face when managing highly complex automated systems is the complexity building system.
I'm thinking, bear with me for our side quest here, but there's a really famous Air France disaster where the autopilot was throwing a specific alarm, right?
And they misinterpreted what the failure was and handled entirely wrong.
Yeah.
How...
do we build that kind of box around something that's probabilistic, right?
Like an autopilot, it's still just, it's following like a defined workflow, but it seems way more complex when it's a probabilistic system.
Yeah, absolutely.
I think there is verification.
It's what we said, you have the design metrics.
So you know if the system's doing the right thing.
You also design guardrails, which people talk a lot about today, which is like, how do you make sure the wrong action gets blocked?
And actually it's very, interesting in early days of AI, like what we did in building the guardrails.
So a self-driving car, for example, early days, we would block off part of the map and just don't allow the cars to go there because we know they have high risks in those areas, or historically, the car used to make a lot of mistakes.
Or you make sure the car really, really respects school buses.
You know, every time the car sees something yellow, it's like literally going to pause.
It actually creates hilarious problems.
I don't know if you know that in Austin, people have party buses that exactly looks like school bus, but it's painted black.
Have you ever seen that?
And when we were deploying self-driving car in Austin, we used to cause traffic jams because we see a party bus and we just stop.
We're like, oh my God, that looks like a school bus.
So that's kind of early days how you build the guardrail.
And now it's getting smarter and smarter, of course.
And the last piece people don't think often about is observability.
So after a system that's deployed, you want to constantly track.
if it's doing the right thing.
And actually talking about aviation disasters, there's another interesting one.
It's called Duttenberg, if I remember correctly.
I don't know if you know.
It was early days with airlines where you actually could have two planes at the same level of altitude.
Today, you can no longer do that.
I think they have to be a thousand feet apart or something, but, you know, fact check.
We have a resident pilot.
He'll fact check it later.
Yeah, yeah, yeah.
Exactly, exactly.
And what happened was there were two planes and one...
The pilot was trained in kind of Eastern European style, so they trusted the people more than the machine.
They have another plane that was, I think, from Spain or something, and they were trained to respect the machine more than human.
And that was the early days of a system called ACOS.
That is where, like, the machine will coordinate.
And the plane was at the same level.
And then the Eastern European pilot listened to humans who caught it last minute and issued the opposite direction with the machine.
And then the Western European one followed the machine and it just became a disaster and to play hit each other.
So that's the beauty of trust is like, when do we start to trust the machine direction versus the human direction?
What happens when they're different?
Yeah, that's wild.
Yeah, it's interesting, right?
Like how, as the systems get more complex, how do the people build the fund and maintain the fundamental like knowledge and skills?
to be able to evaluate the machine, right?
Like, as opposed to just trusting that even, you know, a GPT response is correct, like, without having the fundamental knowledge.
Like, how do we make sure as product people we're maintaining that knowledge?
Yeah, and it's more complex because you write the evals, which is humans trying to use metrics to know what the machine is doing.
The host of verifying the verifier.
And that is really hard.
That's why today you have an expert in the loop.
which is very common if you talk about like ROHF and things like that.
So you have a human expert to look at the result.
Now here is something harder.
There are things that don't have ground truth.
So we're talking about evaluating sales agent.
So let's say I write an outreach email.
And is the outreach email good or bad?
And what's the ground truth?
Is the other person, would they have responded to it, right?
And how do you know they would have responded without sending out the email?
You can do pattern recognition.
You kind of look at all the successful emails and you see what is the pattern and how expert to score it.
But you may never know, you know, like whether it would have achieved the outcome.
It actually happens in self-driving cars, which is super interesting.
You actually very rarely have an on-road collision in self-driving cars.
If you do, that's a big problem.
So you have near-miss and heartbreaks.
And the question you ask is, if I didn't break, would that have been a collision?
And that is a really, really hard question to answer.
And sometimes we have like, I don't know, 20 safety engineers in a room to debate that very question.
They have 20 different responses to that.
They have 20 different responses.
And they're all 100% certain about it.
Exactly.
I'm always confident with my driving, but if you ever be in a car with me, some of the things don't have a ground truth.
One of the things that I like about that you're describing is like in order to have, in order to really build that trust and excel in this, in designing agentic systems.
you actually need to be aware of the metrics that you're tracking, right?
And you need to have some idea going into that and they'll evolve over time, right?
One of the things that I'm being asked to do more often by clients is we have a product metrics, we have like a success metrics workshop.
And it's almost like in this age of we can build so fast that we are literally like goldfish the second it like hits the market, right?
So like there's no part of the go-to-market strategy that it's like, all right, after 30, 60, 90 days, what are we actually measuring against?
And like, was this investment worth it?
Where do we need?
We're like, you, can't build an agentic system without that sort of plan going into it.
So it's almost like a discipline that I can tie back to something that other clients are asking for.
I love that.
Yeah.
You mentioned something also in your talk that I thought was unique because you used a unique term that I haven't heard a lot about.
So can you tell us what building the harness is?
Yeah.
It's basically how do you make sure the system works at scale?
So I think that's the problem.
people's challenge today.
As product managers, we used to build deterministic systems.
I have a set roadmap.
I know exactly what I want to accomplish.
I know exactly how the system behaves.
I know exactly what user experience is going to be.
Now with AI, it is populistic.
And it's very difficult.
I see people kind of err on two sides.
One is they just unleash the populistic nature of agent, and you're just like surprised what you get, and that may not always be a good surprise.
One is you try to put a deterministic workflow.
on a probabilistic model, that's also not good because you don't unleash the real power of the model underneath.
So you have to find a way to let the model or let the agent be free, but at the same time also manage the unintended outcomes.
And I've covered examples.
This is early days when we did job recommendations.
So when you look at something simple like a recommendation, then it usually has F1 score.
which means how much precision do I get for how much recall.
So precision is how accurate, recall is how much coverage, right?
And if I send a job recommendation to someone fresh out of school, if you think about it, the recall is more important than precision.
I mean, when I was fresh out of college, I learned everything.
I actually wanted to be a tour guide, which I never got a chance to.
But I would be equally happy if you sent me a banking analyst job or a tour guide job.
But if you're sending someone very senior...
then your precision is more important to recall, right?
Because they're not looking to change job all the time.
And if you send them the wrong one, they'll get offended and you lose trust.
Here's another example.
Fraud detection was probably the earliest AMR algorithm that really took off.
And in fraud, I used to joke, it's a problem between dolphins and sharks.
And let me explain it.
If you make the algorithm really, really strict, you're going to catch a lot of bad actors, but you're also going to get good actors, the dolphins.
stuck in the net.
And if you make the algorithm more relaxed, you're going to save all the dolphins experience, but you're probably going to let the sharks swing too short.
During the pandemic, I actually downloaded a grocery app.
And for the first time, I was using a grocery app.
You know, everybody had that experience.
And the first thing I bought was like $300 or something crazy because I just like had bought groceries for a while.
And I was like, as a fraud, they blocked my account.
And eventually I deleted the app.
And that was a very hard question to answer because I feel for whoever was the product manager.
I was a new buyer.
They don't have enough data on me.
And a large amount of debts have certain characteristics of a fraud.
So how should they make that decision?
Because it's not just, in my case, I was a dolphin.
But does that generalize and does that apply to others?
That's the hardest thing of managing probabilistic systems is how can you scale the decision-making?
You've been 20 years in AI, which is like 18 more than I'm aware of.
Yeah, I'm curious, you know, kind of in that journey, how kind of like, you know, what stayed the same versus, you know, what is changing and what are you seeing now?
Like kind of just how has it evolved in, yeah, in less than six hours?
Yeah, I think there are a lot of like the first principle problem solving are still the same.
Like the system is still...
you know, largely probabilistic, you still need to really understand your objective functions.
You can steer the system to where you want.
And a lot of the ways like the, you know, agents, AI, machine learning, whatever you call it, would make a mistake is still very similar.
I think what's changed is things are just updating so fast and it can do so much more than what it used to be.
I remember working on chatbot actually back in.
God, 2012, that really dates me.
And back then, I used to score my answer against whatever the chatbot's answer, so you kind of compare.
And the accuracy used to be like 60%.
If I got 70%, I was like, wow, that's a good day.
And today, you know, like the answers are sometimes like smarter than human answers.
That being said, I think there's an interesting phenomenon.
I heard somebody else talking about this.
They call AI average intelligence.
It's very interesting because AI is the most popular answer in a way that is average.
And our product today are looking more and more like our competitors' product.
The industry sometimes are having a little, you know, coverage to the mean.
And did you know, very interestingly, purple is the most common color of a cloud UI.
Yeah.
So if we do something, you notice that?
It's usually purple or otherwise it's blue.
It's either purple or blue.
So that must have come from the training data.
I think Astropic wrote a blog about this.
I was going to read.
I haven't read it yet.
So excuse me if I didn't quote all the right facts.
We appreciate the honesty.
But most likely, you know, at some point in time, most of our UIs are purple.
Or that's just like what the model associated with like modern, you know, UI UX.
So then everybody's UI becomes purple.
Then where's the creativity?
Yeah, we are seeing that.
We're seeing that phenomenon of like, if you're using AI for competitive research and market research, so are your.
competitors, right?
And you're going to end up with the same type of product or the same type of decision.
We don't want everything to kind of look the same.
So how do we actually influence, you know, how do we actually influence our decisions with what we know about our customers?
We still have to think, apparently.
Yeah, unfortunately.
Yeah, unfortunately, we're stuck there.
So Dan talked about like what's changed.
Where do you see like the, where do you see the AI market going?
Like what should people be preparing for if you are in, you know, the bubble that Martin talked about?
That is a great question.
I think AI is more and more going to be status quo these days.
It's just like kind of internet.
Now I'm old enough.
I've seen the internet revolution, then the mobile revolution, and now it's the AI revolution.
I think, Ian, it's not a cool thing to talk about.
In a few years, this may be just everybody's basic training.
And who you work with is always just going to be a combination of humans and agents.
And you need to, I think that you kind of need to put yourself in a position that you're really intentionally steering, you know, versus you're kind of like mindless using the AI in many ways.
Have you heard the term like centaurs and reverse centaurs?
Yeah.
This is very sci-fi.
Yeah.
So centaurs is like, you know, human head with the horse as a body.
It's like human steering and AI is working.
I think that's probably like where ideally we want to be.
And reverse centaurs, you just get a reverse.
The horse is the top.
Yeah, exactly.
That's not the good one.
That's probably not a good place to be.
So I think for everybody, like the question is like, how do you kind of understand how to leverage AI, but put yourself in a position that you're really steering, you're getting AI to accomplish your vision, your creativity and where you want to be.
You notice how she didn't say Skynet.
So that was really reassured.
That's good.
Yeah.
Because we talked about this being an advanced course, right?
I want to take us back just a little bit.
You have both used terms like probabilistic and deterministic.
Yeah.
What's the difference?
What are the, what does that, what do those mean and what are the difference?
I think the best example would be like early AI when, you know, you have Deep Blue playing chess.
So as humans, we kind of, or, you know, I'm not a super good chess player.
I think about it as like, here's somebody else move, here's my next move.
And AI doesn't think this way.
They think there are many, many possible moves, each with a probability.
of what the other side is going to do and each has a probability of actually winning the chess game.
That's kind of how AI thinks, is everything is probabilistic.
And we talk about just now like a roadmap.
I don't think roadmap is going away.
I think roadmap just becomes probabilistic.
And there is any moment, infinite versions of that roadmap you just created with different tweaks and change and each with different probability of accomplishing your goal.
And as you learn, you get a little bit more data point and that probability changes.
And you have a narrow, narrow selection, which we call converging.
So I think that's kind of like the key of AI today is how fast are you converging?
So instead of like going one set way of accomplish something, now you have many, many ways that AI could create to accomplish something.
But you still eventually need to converge to the most optimal answer.
I think that's also like the challenge today with product management.
There are two ways to build products now.
You can build by experimentation or you can build by design.
In fact, build by design is the old way.
And we used to have to run A-B tests on very specific things.
Now we can do A-B testing on the entire product.
And you can test all your good ideas, but all your bad ones too.
Because the cost is much smaller.
Then what you create is create a lot of random walk.
In AI, we call it explore versus exploit.
And the key is you have to exploit at some point.
You cannot random walk all the time.
Then you, I don't know, end up...
somewhere, like, you just can't come back.
And you create too many local maxima, and you never know where you want to really allocate your resource to hill climb.
Sorry, hill climb is what I meant.
And that is, like, the most challenging thing of probabilistic versus deterministic, is because there are so many versions in probabilistic, you still need to converge, and that's the key.
Yeah, you still need to pick a direction.
Love that we just hit the, like, the probabilistic roadmap, because, like, that's the most realistic, honest one of, like, hey, that thing that...
is next, like, but maybe it's not, right?
Maybe we will learn something that says, like, actually, that's not where we should be spending our resources.
And you'll create a step function change.
Yeah.
If you create the right convergence.
Yeah.
We could just go into math theory here, too.
Yeah, we talk about, like, you know, dishonest roadmaps are linear, right?
Honest roadmaps are, you know, in the shape of the cone of uncertainty, but we want to get to strategic roadmaps that have, like, real decision points that we've had mapped out.
We may not know what all those decisions are, right?
And it seems like those decisions might change day over day in the current world or the current environment that we're living in.
But that's really where we need to get to.
Yeah.
The key is learning.
AI learns from ground truth.
At some point, you need to give it some ground truth.
And we need to keep learning, too.
Right.
Yeah.
Humans need to learn, too.
Yeah.
So this has been an amazing advanced course.
Thank you so much for being here today.
What's next for you and what's next for G2?
And we are working on the next generation of trust layer.
JIT has always built a trust layer for software.
And that has been revealed very much, which gives us great signals of what people enjoy, ease of use, kind of the taste.
And now with AI, what we find is it's actually very important to ground people on the actual success rates with AI.
So if you put an agent to the task.
Are they really going to accomplish the task?
How are they going to accomplish tasks?
Are they going to follow policies?
Are they going to create unintended consequences?
So if a customer call in, they say, hey, I want to post the subscription.
What if the AI cancels the subscription altogether?
And that's what we want to test out.
That's why we create these simulated companies, kind of like self-jumming car with simulators.
And you have to actually drop agents in the simulator and see how they actually act.
So that's, you know.
our vision for the future and that's what we've been working on.
It's actually quite a lot of fun and it's quite challenging too.
There are all kinds of challenges with it.
Yeah.
Awesome.
Thank you so much for being with us.
It's great to be here.