# Elevating Product Research: From Data Signals to Strategic Evidence

**Podcast:** All Things Product with Teresa and Petra
**Published:** 2026-07-14

## Transcript

Hi, folks.
This is All Things Product with Petra Wille.
And Teresa Kortz.
And we're so happy you're here.
Teresa, the other day I heard this hallway conversation at one of my clients.
And I was thinking of, there's something for our podcast.
Teresa and I need to discuss that topic.
And what I heard is this.
Do we actually really need to go see users?
Do we actually really need to do user interviews?
Because we have all these feedback coming in all the time.
And isn't that enough user research done?
What's your take on that?
Yeah, so product teams, I think, are drowning in data.
And we have lots of sources, right?
We have behavioral analytics, if we've instrumented our product.
If we have a sales team, they're talking to customers all day.
Almost everybody has a support team where you're getting support tickets.
Some companies even have like feedback forms where people can just submit all types of stuff.
I do think product teams should be using all of this.
But I think there's this concept of like the quality of evidence that teams need to understand.
And I'm actually working on a blog post on this.
So hopefully by the time this episode comes out, we can link to it.
How timely.
Yeah, amazing.
I like that.
I've written about this before.
Like I have an Ask Teresa blog post called something like, what do I do with all the insights that come from sales calls and support tickets and things like that?
And what I've written in the past is that I think about those as signals that tell me what to explore in my interviews.
And the reason for that is that those signals rarely come with enough context to know what to do.
Right?
So if like someone sends in a support ticket and says like, hey, I'm trying to do this thing and I'm stuck.
Like rarely does the customer actually write all the detail of what they're trying to do, what went wrong, why they need to do that.
They just write the symptom.
They'll say like, this feature looks broken.
And you'll be like, I mean, it's not broken.
What's going wrong for you?
And so like you have to, you can't just act on that.
Like, you know, it's a signal something's wrong here, but you don't know enough to know how to fix it.
And what's, dangerous here is sometimes you look at it and you think you know what it means and you think you know how to fix it because you're projecting your own experience as a product expert onto that one sentence of feedback.
And then if you go and talk to that customer, you learn like, oh no, they're trying to do something like total oddball outlier with your product that you never thought they would do with it.
And it's actually really important that we collect that context.
And would you say, because in my coaching, what I usually tend to reflect back on my coachee when they ask a question like that is Henrik Nieberg's triangle, which is like, if you have quantitative data, that could be your app store review, and you have a colleague in the organization or several colleagues, sales, supporting the same idea and verdict and then you can find a piece of qualitative evidence that it is a good idea then you're good to go basically but you always have to make sure that you test for all three so is there an expert in the organization supporting the evidence is there this signal that you are talking about in random quantitative feedback data stuff and can you then generate this qualitative insight on top of it?
I would get more specific than that.
So I'll give an example.
I get exposed to a lot of customer data now because I have my partnership with Vistily where we're doing AI generated interview snapshots and AI generated opportunity solution trees.
And I get to work with a lot of customer interview transcripts.
And what I'm learning is that like we use this term interview for a wide variety of research activities.
Yeah, indeed.
And I mean, no surprise.
And so like, I'm going to give a real example.
A team did an interview where they literally were talking to a customer and the customer shared, I have a usability issue when I rotate my laptop, my iPad from like portrait to landscape.
And here's the issue that I'm having.
Okay.
We probably see evidence of this issue.
in our behavioral analytics.
Maybe we see people switching between portrait and landscape, and then when they're in landscape, they can't find a button, they never push it, right?
We could probably see evidence of a problem in our analytics.
We probably could even conduct a survey and learn people prefer landscape to portrait.
We just did an interview where someone showed us the problem.
Can I now fix it?
Maybe.
Here's what I want to know before I fix that problem.
What were they trying to do when they encountered that problem?
Why is landscape better than portrait?
Like what problem are they trying to solve by switching the orientation of their iPad?
And what specifically is going wrong?
And I'll tell you in this specific interview where I got to see the transcript, the interviewer didn't get into any of that data.
Right?
And so now when they go to build a feature, they're just guessing.
Okay, well, we see people want to go from portrait to landscape.
Let's make sure landscape works better.
Works better how?
Yeah.
I don't know.
Right?
So this is why we teach story-based interviewing.
I want to collect the story.
What were you doing?
When did this come up?
What, like, what were you trying?
What need arose that caused you to flip your iPad?
Did it actually solve your problem?
Is there still an unmet need here?
I want that whole context.
And the challenge we have is like sales calls.
If I'm talking to a prospect and they say, hey, we need a solution that will flip into landscape mode.
Okay, why?
Most of the time the salesperson is like, yeah, product management question.
Yeah, like most of the time the sales rep is like, yeah, we have that feature.
Check the box, move on.
But that's not really appropriate feedback until I get the whole story.
So the whole vibe coded software tools, this work will be more important than ever because it is easier than ever to release all these mediocre software tools.
Yeah.
Where nobody thinks about this exact, okay, is this button really here to stay?
Do we need it?
Can it be elsewhere?
Can it look different?
Do people, why do people need it in the first place?
All these kinds of questions.
So yeah, people get.
better and interviewing.
Yeah.
I mean, like I'm going to say somewhere between 10 and 15 years ago, I wrote, I did, I recorded a little video about the ladder of evidence and it's, it was just this framework I came up with where I tried to communicate to get quality evidence.
We have to move higher up the ladder.
So as you go up the ladder, the effort goes up, but the value also goes up.
And it's just this idea of like, We're inundated with these low value signals all the time, but they rarely carry enough signal strength to actually tell us what we should be building.
But they feel like they do.
And the reason why they feel like they do, we're experts in our product.
So we look at that signal and we go, oh, I know what they mean.
But we don't always know what they mean.
And we often project our own experience onto a low quality signal and use our experience to turn it into a higher quality signal.
But the problem is that translation we often get wrong a lot.
We don't realize like they were using our product for something we never thought they would have used it for.
Yeah.
And we could still, so I could picture a teams challenging each other on jumping to assumptions too quickly.
And nowadays.
Your favorite LLM could do the exact same thing.
So you could have something like the devil's advocate skill.
If you're planning to look at data, then they could always say, have you really understood the data?
Is it just a signal?
Can you kind of act on the data?
That could be something people could be doing and practicing a bit more, right?
Yeah.
So I, okay.
So part of my work with Vistaly, I, because I'm generating AI generated opportunity solution trees.
I have to think a lot about what's a strong enough signal to say this should be an opportunity on your tree, right?
And so when we started, right now we only support two interview types.
It has to be a story-based interview.
That's a strong signal for an opportunity.
Or because most teams aren't very good at story-based interviewing, we also support what we call a general interview.
So a general interview is I just ask you direct questions.
You tell me about your experience.
I don't love those types of interviews.
I actually think that's still a pretty weak signal.
But if we didn't support that format, like almost nobody could use our tool, right?
Yeah.
And long-term, our goal is to help use the tool to teach teams, like show them a signal strength and then teach them how to collect a better signal over time.
I like it.
But what's interesting is...
We're seeing a lot of teams submit transcripts and they don't classify in either of those categories.
They're actually a product demo.
So they're demoing a product to a customer.
How do you like this customer?
Yeah, they're a team meeting.
So this is my guess is it's a stakeholder interview and they're thinking of a stakeholder interview as a customer interview.
We're seeing...
This one is a tricky one.
We see ones where it's not a usability test.
The interview participant isn't giving the participant, I'm sorry, the interviewer isn't giving the participant a task and asking them to think out loud.
It's not a usability test, but they're basically asking for their usability preferences.
So the interview will basically be like, what's...
tell me about your experience with our product, which sounds like a really good open-ended question, but it's not because it's not grounded in specific instances.
It's not grounded in stories.
And so then the participant will be like, well, I don't like landscape mode.
And then they'll be like, okay, well, tell me what you don't like about it.
And they'll enumerate what they don't like about it.
And the interviewer will write all that down.
And they think they're getting like really good quality of evidence.
Yeah, but it would have been a minor issue in the grand scheme of things.
They just collected a bunch of like probably idiosyncratic preferences with no surrounding context.
And so this has really opened my eyes to like just how much misunderstanding there is about what a good interview looks like.
And also just what data is good enough for us to make a decision on.
And then there's this really pragmatic piece.
Like I can't wait for you to be a perfect interviewer before you need to make a product decision.
Yeah.
Right.
Like lots of teams are making decisions with no interviews.
And I can't like go for perfection and be like, we can't use this data because it's not story based.
We can't use this data because you asked unreliable feedback.
We still have to figure out like, what is the signal here, even if it's low quality evidence?
Yeah.
So this has been really fascinating for me to just think through, like, can I come up with a rubric for quality of evidence, which I did.
10, 15 years ago with my ladder of evidence.
But now how do I pair that with like, how much confidence can you have in this signal?
And what types of decisions should you make on it?
Yeah, it's really fascinating.
Yeah, that's really fascinating.
And now you have to bake it into a product.
Yeah.
And by the way, just to mention, I think this is a very good example for a strategic product decision, right?
Yep.
Do you love the decision?
No, you don't like the decision, but it is a very strategic decision to say like, even if we don't think that's the perfect format for an interview, we still take it, we still work with it, but then we educate the user towards what we actually think is a good interview and the better strength of evidence or better evidence quality, however you call it.
And I really like that.
That's just like.
magnifying glass here.
So this is what we usually talk about when we say like that's a strategic product decision.
Well, I think also it's a very practical product decision, right?
Because if I look at the total addressable market, if we limited it to like we're only going to pull opportunities from story based interviews, which in my world, if that is what I would do, we might have 12 customers.
Yeah.
Right.
And all your former, your former trainees.
Yeah.
Even people that have been through training, it doesn't mean they've kept the habit or they've practiced the habit or they've continued to develop the habit.
Story-based interviewing is a skill that takes work to learn.
It takes work to maintain.
It takes discipline to do over and over again.
And the reality is most organizations aren't asking that of their teams.
And so they let that habit fall by the wayside.
And here's the other, here's the kicker.
If you did conduct an interview, it's not a great interview.
It's still better than having never talked to your customer.
So I don't want to give the team feedback like we can't use this because now I just discouraged them from talking to customers.
And that's worse, right?
Yeah.
So it really is a spectrum where like the worst thing you could do is never talk to a customer.
I think the best thing you can do is collect a really rich story about their experience.
And then there's this whole spectrum in between.
And the way that I think about this is if I'm on the story-based end of the spectrum, there's less risk in those opportunities.
I can be more confident that they're real needs.
But it doesn't mean if I'm towards the like kind of crummy end of interviewing that there's no signal.
There is a signal there.
It's just not as strong of a signal.
Yeah.
And so this is something I've been thinking a lot about is like.
And it's like a fun UI challenge of how do you communicate signal strength and how do you communicate and in a way that like doesn't discourage someone from continuing to interview, but motivates them to like move down the spectrum a little bit to get a little bit better at interviewing.
And so this is really fun.
It's fun to like give little nudges.
Like we have the ability now to say, instead of asking this next time, try asking this.
Ah, this is so cool.
Right.
Yeah, this is so cool.
And I love that, as always, you're publishing it on your blog for people to read as well if they want to draw their own conclusions about how they could improve.
You know, I kind of have to, because like if you're going to build really opinionated software, which is what our goal is, and our opinion and story based is better, we have to communicate the why.
And like we have to be really transparent about.
Like if we give you an indicator of signal strength, we have to be really transparent about how we're communicating that signal strength.
Yeah.
So you can't just say this is better.
This is worse.
See ya.
Yeah.
Right.
And then as a coach.
Because we say so.
Yeah.
And then as a coach, the thing that really motivates me is I think this provides.
Now we're looking at we're working in the context of your work.
We're working with your real interview transcripts.
We're working at the moment where you're trying to synthesize them.
And to me, this is a great coaching moment to just give these little nudges of like, next time, try this.
See the difference between this and this.
See how this gives you more context for thinking about a solution.
See how this is very vague.
Like it's a very fun coaching moment in the context of a product.
To think about.
Yeah.
I see why you love it, Teresa.
I see why you love it.
Yeah, it's super fun.
Yeah, so cool.
Yeah, I think we wrap it up.
And I say...
Thank you, Teresa.
Thanks, Petra.