# Defining AI Product Success Beyond Technical Metrics

**Podcast:** All Things Product with Teresa and Petra
**Published:** 2026-10-27

## Transcript

Hi, folks.
This is All Things Product with Petra Wille.
And Teresa Torres.
And we're so happy you're here.
Teresa, you wrote a piece on AI Evolves a few weeks ago.
And I saw there is a part in it which I found particularly interesting, which is the part about what good looks like.
Can we maybe double-click a bit on that?
Because I know AI Evolves, we recorded an episode on that.
It's the catchy title.
But I thought the particularly valuable part is the part on what good looks like.
Yeah, I think this is the part also that hasn't changed.
So I think for product teams, when we're trying to decide what to build, we're defining, like, let's say we're defining a feature.
So even if we fall back to traditional rules, if the product manager is writing a product requirements document, if the team is doing cross-functional discovery and their story mapping and trying to figure out what is it that we're building, a lot of that work is what does good look like?
What does a good version of this feature look like?
What does success look like?
Maybe this is really just outcomes in disguise.
How do we know our feature will be successful?
I think with AI evals, it's really easy to focus on like, this is a new skill and it absolutely is a new skill.
It's how we measure the quality of non-deterministic outputs.
But I think the like heart of the activity, like what is good and how do we put a definition around good isn't new.
So when I wrote my article, some people pushed back and were like, evals measure technical performance.
This is actually really analogous to me to the mistakes we made when we first started instrumenting our products.
So the very first analytics we put in our products were things like uptime, speed, performance.
Like, of course, these things matter, but they're not sufficient.
Right.
Like we need to do more.
We need to measure customer behavior.
Are we actually getting the customer to a value creation moment?
Like when we define what good looks like, good is.
The customer did something that created value for the customer.
When we're talking about analytics, we're measuring, did that happen?
Did we create value for the customer?
And hopefully our knowledge of the customer and what they need is influencing that definition of that value creation moment and those metrics.
And I think with evals, like it's the same thing.
Like, sure, I can just write some evals with no knowledge of the customer whatsoever and say, Was this answer helpful?
And I'm going to make my own judgment of whether it was helpful or not.
But ultimately, the customer should decide if it's helpful.
And so the product team's job is to understand the customer well enough to define a definition of helpful and a measurement of helpful that defines what good looks like and hopefully what good looks like from the point of view of the customer.
And so I find it wildly fascinating how.
In this transformation, we are repeating the same pattern that we saw in past transformations again.
So it is very fruitful these days to have these conversations with people that went through various of the mobile transformation, the user-generated content transformation, so several of these transformations, because folks like you find it...
far easier to spot these like, yeah, I like what you're currently doing with these evals, but you can go one step further.
And I would advise you to do so because I've already been there in the past where we made the exact same.
I would not call it a mistake.
I think it's a bit like an onion model of, first of all, you do the easy things and then the obvious things.
And then you think about, oh, is this actually really helpful?
Inspect and adapt kind of style, which is fine.
So I would not blame the people because maybe it's not there.
fifth IT transformation.
But yeah, I like to see that things are happening.
again and that coming back to the basics and coming back to what has worked as mental models in the last I don't know 20 years is really so helpful so that's why I liked your framing in particular about this what really is the measurable impact that our features and product is having how is it contributing to the bigger company strategy and to the business success ideally so I really like that part in there yeah I think It's easy to get distracted by like, oh, there's new language.
There's terms I don't understand.
There's a new methodology.
There's a new tactic.
And we like glom onto the shiny and new and maybe the hard and complex and like, and we're looking for like, what's the simplest version that we can understand.
And maybe because a lot of this started with machine learning engineers and it felt like a very like engineering topic.
it's natural that we made this mistake of like, oh, let's just measure technology performance.
But I think like this is the argument for cross, a cross-functional approach because of course the engineers are going to think about it from like a technology standpoint.
And nobody can blame them for that, by the way.
No, that's their job, right?
And so hopefully we have the business mindset represented and the user mindset represented.
And we're looking at it as like, When you're building an AI product, just like any feature, when you're building any product, let's just drop the AI.
When you're building any product, we have to look at, is it creating value for the business?
Is it creating value for the customer?
Are those things aligned?
And we can put metrics in place to measure both of those things.
We measure, for example, with feature adoption, we measure like activation metrics.
Did we...
which usually, by the way, is a value creation metric for the customer.
So do we create value for the customer?
And then we measure things like retention, which is, did we create value for the business?
And I think with AI products, we don't always know, like maybe we don't have enough experience with the technology yet to like, say your activation metric should look like this and your retention metric should look like this.
But if you're building an AI product and you don't know what customer value is and you don't at least have a theory of how that's going to create value for your business, that's a problem.
And then I think if we think about evals as just metrics and we think about it as instrumenting our product, to me, it becomes really obvious that we should not just instrument technology performance.
We should tech instrument customer value and business value.
And the eval suite should cover all of those things.
Agreed.
And that might be hard in the beginning, because when I think back to the times when I was working for this big social network here in Germany, which is called Xing, and engagements came up.
So people were liking stuff for the first time.
We had new things to measure because, for example, engagement was not so much a thing prior to that.
period, right?
And I think there will be similar things happening with AI.
There will be new metrics coming up that we're now able to measure or that now the better metrics to keep track of in times of AI.
So I think all of that will happen, but this will not change the fundamentals of what you just said, that it's all about the user value and the business value.
And if those two have enough overlap for us to be able to build something that will be successful.
Yeah, actually, engagement, thinking about the metric engagement is a great analogy for some of this stuff.
So for people that have ever tried to measure engagement, usually what happens is we start with a really crude metric like monthly active users.
And then maybe as we get better instrumented products, we might look at like daily active users.
And then people came up with this idea of a sticky product and they measured daily active users divided by monthly active users.
And so already we see like a metric evolving.
But then the really good teams realize not all engagement is equal.
And we actually should measure highly valuable engagement.
So if I am on Facebook and I just scroll all day, maybe that's less valuable than if I post.
I mean, they have an ad business, so they want you to scroll all day.
But humor me in this example, right?
And so for a lot of products, we...
Or actually, here's a better example.
Let's take something like DocuSign.
You receive a document to sign and you're engaged.
You sign it.
That's good.
But better engagement is that encourages you to now send your own document for signing.
And so you could imagine for that team, they might measure engagement as both actions, any behavior in the product.
And then eventually they realize like the real activation metric is getting you to send a document.
So now we started with these universal metrics, monthly active usage.
We evolved them to be a little better, still universal, daily divided by monthly is still something anybody could track.
And then eventually we realized the best metrics are tailored to the value creation moment of our business.
And I think we're seeing the same thing with evals.
So there's a whole bunch of eval vendors that will provide off the shelf evals.
They'll just come up with these generic definitions of like concise.
The AI was concise in its answer.
The AI was helpful in its answer.
The AI was friendly in its answer.
And helpful is my new NPS by the way.
The AI was helpful in the end.
It's answer is my new NPS thing.
So what's hard about this is like, let's just take conciseness.
If I'm a student, this is the example I used in my article.
If I'm a student and I'm in a physics class and I'm starting to learn astrophysics for the first time, I don't want a concise answer.
I want an answer that like walks me through it and teaches me something.
Yeah.
If I'm an astrophysics professor, I don't want a teaching example.
I want a concise answer.
Right.
So like even conciseness, like the definition of what good looks like.
changes on the context.
Which brings us back to contextualization and hyper-pensalization in times of AI.
We're going through this same learning curve that we went through with regular analytics.
And so I guess my key thing here is we've always had to define what good looks like.
When you're talking about AI products, don't outsource what good looks like.
Really think about in your context what does good look like.
Yeah, so in my context, it's not that it has to do a lot with AI evals, but I often talk about what good looks like as well, because in my leadership coaching, that has a lot to do with...
role definitions so what is a good product manager what is a good product leadership person but product leader or what is a good product organization looks like so is it smooth this is efficient is it low friction is it low conflict all these we talked about conflict just recently so maybe that's a good time to link to that episode um but it's a very familiar concept to me to always think about okay what is the smallest common denominator of goodness across the board when it comes to roles, when it comes to processes, when it comes to organizations, and when it comes to products and features and their success.
So I think it's a very familiar pattern for both you and I in our work, right?
And I think it's even more important now because AI is so much, there's so much punishment if you use AI and automize stuff and look at pattern.
at an organization that is not in a good shape and for a product that is not in a good shape, that is not attributed well, that is not tracked well, where people don't have a clear idea of where we're headed and why, where people don't have an idea of which user group we are serving and so on and so forth.
So it's more important than ever to get your stuff in order, I'd say, on all levels, organization level, people level, product level.
Yeah, you know, what's interesting is that like what's in the back of my head is like, this like intertwined concept of like, this is just the same as setting outcomes.
It's the same as saying, being explicit about our expectations so that we can see when we get it wrong.
So like we teach, like start with an outcome.
What are you trying to achieve?
It's hard to achieve something if you don't set out to achieve it.
Right.
It's the same thing.
Like if we don't define what a good product manager looks like, if we don't define what a good designer looks like, at the organizational level, which by the way is context specific.
A good product manager in one organization may not be a good product manager in another organization.
Like how do we expect people to succeed if we're not explicit about that, if we don't set those expectations?
And I can make the same analogy of like, how do we expect our feature or our product to succeed if we're not explicit about our expectations for it?
And this is like...
It's just different scopes.
This has like always been the work.
How do we define what good looks like, try to achieve that bar and then iterate when we fall short?
And I find that a massively relieving realization over the course of the last, I don't know, 18 months, 12, 24 months is this, in the beginning, all of it seemed so new and so revolutionary, but the last few months, so much of the work that I do.
is the basic work that I've always done.
Yeah.
So it's really defining these what good looks like moment more ruthlessly than ever, I'd say, doubling down on directional clarity with so many of my clients.
And why is it?
Because there is an increasing pain that you're automizing stuff that is not yet in good shape with the use of AI, just quicker and faster.
And yeah.
It gets you not to the output that you actually want to see, or it gets you to the output because you realize you need to optimize your process first before you then jump into the realm of AI experimentation.
Yeah, I saw someone post on this.
It might have been Ethan Mollick.
I can't remember, but the post was something like, right now we're automating a bunch of stuff that shouldn't even be done in the first place.
That's true.
Some of this, I think, is just a natural learning curve.
Yeah.
And I'm going to equate it back to what we were just talking about.
Like, I think that Cal Newport has a concept about this of like, how much of the work that we do doesn't actually matter.
Like it's, wow, this has just got pretty depressing, but like how much of the work that we do, like doesn't actually have an impact.
And like, could we just stop doing those things?
And so already.
already we're not good at asking those questions and like reflecting on like, why am I doing this?
Is this, what's my expectation of this work?
Am I being explicit about that?
So this is how I'm going to tie it back to outcomes and measurements.
Like, what do I expect to happen if I do this?
And am I actually making sure that that's what happens?
And then if we don't do that and then we automate it, now it's like we're automating a bunch of things that have no impact because when we did the thing, it had no impact.
Yeah.
Ooh, I loved it.
Teresa, I'm glad we unpacked this topic, I'd say.
Yeah, I think I'll wrap up by just saying like, I think we just talked about this in the context of products.
We talked about it in the context of organizational rules.
Now we're even talking about it in the context of like the work you're doing moment to moment.
I think like the habit that underlies metrics outcomes is really just like, what do you expect to happen?
can you be explicit about that so that you can understand the world and then the assumptions around it which is just good critical thinking yeah i love that thank you so much teresa thanks petra