# Observability Fuels AI Agents and Engineering Profit

**Podcast:** Dev Interrupted
**Published:** 2026-05-26

## Transcript

Today, we have a very special guest joining us, Christine Yen, the CEO and co-founder of Honeycomb.
Christine has been at the forefront of the observability movement since before it was even really called that.
And drawing from her early days building analytics at Facebook and more, she spent her career figuring out how to separate signal from noise.
which is the biggest battle that I think we're all fighting right now.
How do we understand these deeply complex systems that we're creating?
So today we're going to dive deep into the state of observability because in an era where AI coding and AI agent is shipping code faster than humans are even really reading it and understanding it.
We're in an impossible battle, and traditional monitoring tools are bursting at the seams.
The techniques that we used yesterday might need to be flipped on their head.
So we'll explore what this means for developers and the pressing conversations the industry needs to be having right now.
But particularly, we're going to explore how the worlds of observability and data science and engineering and product are all colliding into this one big new thing.
Christine, welcome to the show.
Thank you so much for having me.
I'm excited to be here.
We're real excited to have you.
Honeycomb is a big, longtime friend of the Dev Interrupted show as well.
Your co-founder, Charity, has been on here many a time, even predating me.
She's probably been on more episodes than even I, the host, have in some regards.
There was a period of time in my early days where that was certainly the case.
So it's really great to have you here and get your perspective because...
I'm a really big fan of all of the thinking and the leadership that's coming out of Honeycomb right now.
And I really want to dive into the things that you're seeing and your understanding from your role as a CEO of this company of really looking at the landscape and how it's been changing kind of like right underneath you.
You're not an AI native company.
You were.
talking about understanding technology before all of this transformation happened.
So what is like highlighted right now for you in this big wild world and how it's getting shaped by agents and how you think observability is fitting into that?
I think one of the things that strikes me, you started this off by touching on data science.
And when I think about observability, there's many definitions out there.
I just think of it as, trying to make sense of what your systems are doing by using data, right?
Which is a very broad definition.
And when you think about what mechanically that requires, it does sound like a lot of very parallel industries, business intelligence, data science, product analytics.
There's all these adjacent things where we're trying to do very similar things.
And I've always been surprised, you know, historically, how high and thick the walls between these adjacent disciplines tend to be, how little idea sharing there tends to be.
And I'm seeing, and I imagine we can talk a little bit more about that later if you want, but I'm seeing a version of that happening with talking about agents and agentic development, whether it's, I have Claude helping me write code, or I'm trying to build my customer facing agents, like people are really, really trying to talk about this as if it's something entirely new and entirely different.
And we leave all of our traditional software things over here and we're doing all of our AI things over there.
And maybe it is because we've been doing this since before ChatGPT.
But I look at it and I'm like, guys, it's software.
Like it's weird software.
It's non-deterministic software.
It's more autonomous software than we're used to.
There's a lot of the shape of how we serve that software has changed, right?
We don't have clean request response cycles anymore where, you know, you have clients talking to servers and no states shared in between.
The shape has changed, but it is software.
And it's interesting to see different organizations sort of working through that idea.
And figuring out for themselves which ideas to carry over from how they serve software systems, how they think about reliability of that software, how they think about observability, and where they're like, no, truly our practices from the past do not serve us in this new world.
And we need to take this sort of step change evolution in our practices.
That's definitely something we've heard a lot from leaders on the show in terms of really what AI does is it makes more dramatic.
and more visible the problems in an organization that are deeply maybe like around those rituals and those silos and those things that cause the information to not flow correctly.
And so really what it does is it highlights issues that were always there, makes those issues way more stressed and way more strained.
And what you see in these teams are they come together and in this like, you know, the rubber is hitting the road with AI as fast as possible.
building a race car, but it's falling apart on the racetrack.
Why is that?
And they're, for the first time, opening up the hood and fixing that one particular thing that had been, like, really a huge crutch that the organization had found workarounds for, or it wasn't the biggest obstacle of the time.
But now it was what was causing the race car to fall apart.
So because of that, like, you got, like, people shining light on things that always.
should have mattered.
Like what you're saying, like the underlying discipline didn't change.
Just the, the things that we need to carry forward might need to be in a different arrangement.
You might need to repack your, repack your suitcase, but you're still, you're still taking the same thing, you know?
So I think that piece is super interesting, right?
I, one of the, one of my level moments years ago now with AI was realizing that like, it was I was at some event and someone was talking about using AI to generate bedtime stories for their kid.
And how their six-year-old was actually really good at being like, I want a bedtime story about dragons and witches and my left sock.
And I want them to go on an adventure together.
And hearing them talk about this, what has been hard about communicating with humans forever?
Well, it's been about...
communicating expectations and assumptions up front, being explicit about what you need and your intent.
And when I think about engineering practices, when I think about observability, there is something really exciting.
Honestly, when I'm talking to a customer about observability and their practices, one of the most valuable conversations I can have with them and then they can go have with their teams are the conversations outside the tool themselves.
It's...
What does delivering great service to our customers look like?
What are the signals that really matter to the business where we should page an engineer if something looks off?
It's probably not every little, you know, it's probably not CPU hits 70% utilization, CPU hits 75, CPU hits 90, CPU hits 95.
Okay, let's step away from that pattern.
It probably is conversations like, if we are an e-commerce platform and our checkout rate drops and sustains that drop for some amount of time, right?
These conversations that are at a higher level of what matters and what is good are ones that I see teams having all the time now because AI forces those conversations up front.
And it is necessary or important for using AI code generation.
which increasingly, I think, should be, it both should and is happening, part of using tools like Honeycomb.
And so there's, you know, AI taketh, AI one hand taketh, one hand giveth.
There's some really fun shifts happening in how we use our tools and how we do what we are here to do, which is deliver great service to our customers.
Exactly.
Really what the technology has done for many teams is allowed them to close a loop that was always open, where there were people and processes and idea of this is what our North Star is, what we're aligned to.
We're going to push and we're all going to do really hard.
And the product people are going to do the product stuff and the design people are going to do the design stuff.
And we're all going to move this wagon along somehow.
Right.
But at the end, we're all going to then evaluate it and understand it and be like, where did we land?
What's this good?
What's this bad?
Go through our goals as a team.
Now, because all of that can also flow into a system, you know, a probabilistic system that can do a lot of high impact things at scale, because all of that information can flow into that same system.
But that same system needs to be aligned to rewards and like what you're saying, like objectives and like this is.
This machine that can do a thousand X more of this analysis impact than maybe the individual running it could, they now need to know where to point that bazooka very, very accurately in terms of their time and their token costs.
So it really causes what you're calling this transformation that happens within companies where they have to have those conversations a lot sooner, and then they're better for it.
Because everyone is now like, oh, wow, we're really aligned.
And me and all of my agents are also aligned.
And I don't have to exist in this fuzzy space of trying to convince them otherwise.
And so that's really the big unlock, I think, that's happening.
And observability.
is definitely the key to closing the loop.
Observability is really what is handing the information and allowing that loop to be closed.
And so that's why, for me as a go-to-market engineer, I build agents, I work with them constantly, and they're only made possible by observability.
Observability is a key underpinning for how they work because I won't even begin to build them until people can tell me what they have to do.
And so...
And then you work backwards from, you know, observability from there.
So it's like there's really a great advantage that you get to be in to understand and how all of these teams are handing that information across that loop.
Well, thank you for setting me up so well.
Yeah, I mean, people love to talk about AI as like, again, it's this AI problem.
It's so new.
It's so different.
And yes, but.
for that verification piece, for what AI needs to be able to do anything.
You know, there are a lot of people who are like, AIS-REs will be the second coming and I never need to do this work again.
What do those AIS-RE agents need?
Well, they need data.
Everything falls down to a data problem.
Everything falls down to where are those agents getting the information about what is normal, what is not normal, what's happening, what should be happening.
And all of that is driven by observability tooling.
And I think it is both really exciting to just like crack open this world of what is possible and what can we automate and what can we make faster with human striving and all of this.
And also, not to go back to my whole, it's all just software, but every data problem means, or reducing it to a data problem means that there's like two things that really need to be true of the thing that's providing the data to your agents.
And it's the thing needs to be really fast.
especially at scale as we produce more and more telemetry than ever before.
And it needs to be able to support, it needs to be able to provide the agent as much context as the agent can possibly use to figure out where to look in the system and what to investigate.
And that context means really rich schemas or really rich data formats so that you can provide it.
information about, you know, if you're that e-commerce platform, how your checkout process is doing.
Exactly.
It's like all of those things become those little bridges that are closing the loop and they all are existing within this observability space that you work within.
And there's something that also that is happening for these teams.
It's really kind of, it has a side effect that's also then relevant for you in the observability world.
And that's that.
Because, you know, AI agents are doing a lot on the knowledge working side.
AI coding, AI coding agents, AI in the CLI, in the IDE, they are also churning out huge amounts of work, both in like durable.
in the cloud environments that are running 24-7 at scale.
We're all seeing these autonomous coding factories and groups that are doing this at scale, but then also on the machine.
So there's more code than ever before.
GitHub is literally bursting at the seams and they're posting a report about it all the time.
And if you look at the charts of every Monday morning of everyone opening up their laptop and starting clawed at 8 a.m.
Pacific time, it's just like staggering.
And so we're in this world where like the...
There's so much code that that code base itself kind of becomes like not much of a source of truth.
And this is something that I heard you say when you were recently at HumanX and you spoke there briefly about this.
And there was this clip on YouTube that we'll include in the show notes as well.
So our listeners can go check it out.
We've called out this like really great observation that because there's so much more code, like that code is not like as important as what's the impact.
of that code.
And that flip has been like really hard to do testing around to understand, but like, how do we build it and understand our software's impact?
So I wanted to dive more into like what you've thought about that since you've said it.
I stand by it.
Early on in talking about observability, someone used the phrase, someone started talking about the mental model that we have of our systems.
And I was like, that is a great phrase because anytime we are trying to map our code base to what is happening in reality, where it's passing through our mental model of like, I'm going to read the code, put it in my head, and then try to figure out, you know, is what I'm seeing in production aligned with what it's supposed to be doing, or is it different?
And that was always hard.
And that was always something that I think, as systems got more complicated, like just left, increasingly left at the realm of reality.
But now, as you say, or I guess as I said, the rate of change and the volume of code is just so much that any mental model we try to build is going to be immediately obsoleted.
There's a blog post that I loved that I still have open in a tab by Chad Fowler titled, Production is a Compiler Input.
And I love that sentiment, right?
That as we're moving so fast, part of how we...
maybe should have always worked, but now it certainly should be working, is taking these production signals and having that be a part of how the agent is making decisions about how to write code going forward.
What real world conditions are being seen and should be factored into making sure that the next set of changes that are pushed out ensure a great experience for your users.
Yeah, that's the challenge right now.
is compressing that gap, bridging that fuzzy context.
Fuzzy context is a lot of what I call this, where humans like you and I, we go to meetings or we have chats and tasks and threads and emails and a lot of contexts that you could argue in all sorts of different ways that an AI could access.
But could they sophisticatedly understand the relations of those, the timing of them?
this becomes much more of a challenge.
So like, because the consumption of that fuzzy context is really difficult, it becomes like this.
And because unlocking that is the key to scaling like an engineer's impact or a team's impact beyond an individual and to orchestrate further with agents, because of that, the observability tool also then becomes like the learning and teaching mechanism by which the human and the agent.
figure out how to do that dance.
I'm curious, like how you've seen, I know at Honeycomb that y'all are very agentic and you exist in an agentic world, like how you've seen the abilities of working with observability to actually then transfer directly into like engineering competence around being able to orchestrate and work with the tools.
Yeah.
I think that we, I think I take pride in is how thoughtful the Honeycomb team is about where the human goes in the loop, how to not attribute skills, how to, you know, we have some really great processes and actually blog posts about how we do things like incident reviews to really focus on learning rather than, you know, slap a bandaid on it, move on.
And I think that that really carries through how we're trying to build the honeycomb out as a product for our customers.
I think that there, I think I have liked to say since the beginning is that debugging is inherently a collaborative process, even if it is just present you, past you, and future you, right?
Because past you wrote something, made some decisions that present you is trying to figure out, and you're trying to mitigate the problem for- You're hedging for future you.
Yeah.
And there's this question of, again, I used this word in this talk track before AI made it a thing, but- Those different versions of you are passing context between each of you and you're trying to figure out, okay, how did I debug this last time and what signals are interesting?
I think those questions, again, even talking in a pre-AI world, say you're onboarding a new team member, knowledge, there's such a role that observability tools can play in that sort of knowledge transfer.
The naive approach of this is like, hey, you're the new engineer on the team.
Here's my dashboard that I use.
You should use it too.
I think that there's a lot more, and this is something that Honeycomb's always tried to bake into the fabric of our, to mix metaphors, to weave into the fabric of our product, this knowledge of other people on your team and sort of the trails that they've worn into the grass and how to surface those sort of investigative cues.
Now in an agentic world that we have these incredible building blocks that can, try to do some of these investigations for us, it becomes like a really exciting thing to explore.
Again, like a naive approach of this is, oh, document everything that your humans have ever done, put it in a run book and then feed it to the agent to follow.
Like that's possible.
We think much more interesting is like, okay, when two humans?
Or when you're actively in an incident or you're in some sort of investigation, incident implies an urgency and severity that I don't think applies to everything that's interesting worth investigating.
Anyway, so say you're in an investigation.
How can agents connect dots that you might not have been able to?
How can agents help you explore multiple hypotheses at a time?
And then how can a human teammate look at what your agents are doing to maybe spot?
to spot trends or provide some context?
What if they bring their own agent?
What does that look like?
What does the collaboration surface then become if you're all, you know, does it come out of Slack?
Is Slack the primary collaboration surface?
By the time this podcast is released, we will have announced some...
Pretty interesting points of view on what that surface area should look like and what that flow should look like.
And it's sort of just the beginning, but it is off the back of years and years and years of thinking about this collaboration.
And I continue to be bullish on humans.
I think that humans are always going to bring some context and judgment and lateral thinking that LLMs aren't going to be capable of.
But LLMs certainly raise the floor.
And I'm certainly like help write down and document things in a way that a tired version of you can at least come back later and reference.
And I think it is going to be really interesting to see the range of solutions out there over the next few years as vendors explore the surface area of where to encapsulate and shield a user from.
I see a lot of vendors out there making magical promises to their users about never having to do X, Y, Z again.
And where to make those boundaries more porous because there is value to building up knowledge and building up skills in the humans and humans learning how to use these tools maximally effectively.
Right.
There's kind of like a...
A little bit of a ramble, but I love this area.
No, I mean, I love that.
That's really...
Okay, so that's like what you just hit on the very end there.
That's like a lot of the...
the danger and the glossing over of just kind of like mainstream kind of consumerism, I think, around AI.
Like, oh, it's this black box that does these things for you.
And it's just so great.
And you never have to worry about this again.
But that's not really the case.
It's an opportunity to sharpen a really like specific and personalized tool.
But that's going to take work.
And that's going to take understanding what you even want out of it in the first place, because you can't offload that level of engagement.
And so, like, I loved how you called out of it enables this collaboration.
That's what observability does rather.
It enables this collaboration between the developers, between their agents, between the context.
It allows this transfer that is really crucial that not a lot of other tooling allows us to do.
This transfer of context.
Yeah, exactly.
Any observability tool, you're coming in, you're like, what?
What does normal look like?
What does normal look like?
The collaboration piece is not a thing that all observability tools try to answer, and that's something.
No, no, the collaboration piece is not.
Some places are definitely, or some tools are definitely going to be more focused on as much signal, noise for analysis as possible.
It's not really digging into the impact that this hit this objective that we had around it.
That's much more subjective, and that's kind of like it becomes its own evolved domain.
of that, right?
And you're kind of, you're hitting at like something I wanted to ask about, kind of like in this world where there are all of these signals and you have these amazing abilities to transfer context and skills up between the agents and down into the, or up into the humans and down into the agents and sideways, you end up in this world where there's a lot of noise.
There's a lot of data.
There's a lot of information.
And for me, as a builder, as a worker, as a knowledge worker, as somebody in the space, there's a lot, a lot of signals.
So then it becomes like, how do I then cut through that noise and find the highest impact things that I can leverage, that the people around me can leverage?
Or how do I just keep it all from drowning out?
So I'm curious, from your perspective, how you've seen observability where it is more of a fire hose, how people can then...
find the learnable moment from it.
Yeah.
I mean, I think you're asking the, how do I know what matters?
Yes.
And that's, it's true.
When you're like in it, when you're just like handed a giant pile of telemetry and you have to go sift through it, that problem in the abstract sounds terrifying.
And.
For most people, the challenge of investigating a system is not a single contained moment in time.
It is a longer period of time where you have some influence over the system.
You have some ability to say, hey, I'm going to admit this telemetry because this telemetry will be useful for investigating.
You have the ability to do an investigation, and then you have the ability to feed it back.
Again, in the naive case, this has resulted in a lot of people.
building it, you know, emitting some telemetry, building a dashboard, having an incident, having as a postmortem, build more dashboards, and then you end up in a sort of vicious cycle where you just have too many dashboards.
I always try to reframe it for our users and customers about having a virtuous cycle around improving your telemetry instead of building dashboards and artifacts, right?
Dashboards are just, I think dashboards are, frankly, also going to become obsolete in an AI world.
If you interact with most of these tools via your own MCP or via their MCPs and your own agent, like who cares what dashboards are pre-canned?
Like you're always going to ask possible questions, which again, points it back to a telemetry problem.
Cool.
Now that we're all looking at the telemetry, what fields can we add?
How will we know what matters?
Well, if you start thinking about it from a data point of view and you think about what does it mean to deliver great software?
You know, your boss or your boss's boss is probably going to start saying something like, well, you know, our high priority customers need to have great experience in the platform.
Great.
How do you define high priority customers?
How do we capture those bits and store it in our telemetry?
What does it mean for them to have a great experience?
What parts of the product are really important to be, you know, low error, low latency things?
What are the contracts?
that have been implicit in how we're building our system that might not be expressed as a simple red metric.
An example with Honeycomb, we are, you know, at our core, a data processing system.
We, an internal metric that we look at really closely and have always looked at really closely is how long does it take between the time that some blob of telemetry hits our API to when it is queryable.
That's something that touches across the whole system.
That is something that directly impacts your user experience if you're trying to query something that you just sent in and you want a really real-time response.
These questions around what matters then drive how do we capture what matters, store it to query, and then build this flywheel of making sure that we can capture all the bits that might be useful.
One last caveat here.
I imagine a lot of your listeners right now are rolling their eyes.
Ah, another observability vendor telling me to capture all the data that serves their business model so well.
You know, I've experienced, I've tried to throw, I've accidentally thrown an IP address into my metrics and seen things explode and had people get mad at me.
Like, you're not going to catch me again, Christine.
And for those folks, if you're feeling that, I really encourage you to interrogate how that observability vendor stores your data because, again, to reduce everything to a data problem for an engineering audience, it is not so hard to imagine that any data tool is only as good as the data store.
And we are really in the middle of, I think, sort of an inevitable transition from the previous generation of ways that we used to store telemetry data in like a log store over here and a metric store over here and a trace store over here towards sort of the next generation of columnar stores that are meant for really fast analysis that are meant for you to send lots of metadata that you might need in a way that is really cost effective in Honeycomb, effectively free.
And this is the shift from the tools that have those silos that rely on pre-ergregation, cardinality limits that you've run into in practice, and then you read about them in documentation after the fact when you're trying to figure out why your cluster fell over.
A lot of the problems in the previous generation around data fidelity and data quality and context are things that this next generation, Honeycomb included, have.
really taken into account and built for.
So for anyone in the audience, rolling your eyes, feeling deeply skeptical about observability vendors telling you to capture richer data, take another look because things are changing.
Things have changed.
At the top there, you gave us a really powerful strategy for when you're drowning in data and signals and you're trying to understand what am I supposed to be learning from this?
You need to interrogate.
You need to flip.
the script actually.
And you need to start asking questions because by asking all of these uncomfortable and investigative and really deeply like core to the consumer and the product and the audience of your product, really asking those questions is the only way that you're going to then figure out what is the signal that I should be measuring to finally find something in the sea.
of noise that you and other people can align to that then you can turn into, you know, nom nom nom delicious agent food and feed to them and get them on the same page.
And that's all going to be possible through treating your relationship with observability and telemetry as like the first class citizen of your organization as the primary highway through which all of the other areas and developing things are going to get built and through all of this is going to be dispersed because.
Even when I was at HumanX, I hosted a panel with some really great engineering leaders, and it was about taking AI prototypes from demo to production, particularly around manufacturing and factories and robotics and incidents where it's like a 10% failure rate is nowhere going to cut it.
We're talking about serious levels of nines of security and uptime.
Right at the top, one of the opening quotes that came from Robert Nishihara, he's the co-founder of AnyScale, he said, I've never heard anyone say they over-invested in observability.
And that set the tone for the whole rest of the talk because everyone was framing as engineering leaders about how they got to this level of, you know, nine nines of accuracy on this particular workflow or dependability in these factory floors by having a lot of these smart, protective ways of understanding and engaging with how the problem was solving or how the product was solving problems for their customers.
And they all were underpinned by observability, by having this really rich source of telemetry that just like what you said, that was performant, that was modern, that was aggregated, that was kind of like cut.
down a lot of the traditional baggage that was preventing them from accessing that information.
And so with all of that in one place, it becomes this super tool for product leaders to investigate and understand about what matters to their customers and what the business should care about.
But then this also becomes amazing things that you can align your agents or the workflows or the tools or all of the undercurrents of the modern software engineering organization.
You can align them to this data as well.
And by asking those uncomfortable questions and surfacing those signals that are doing you good over there, you're also doing them a favor over here because now you have everything you need to map.
their objectives to impact that they can measure and trace and sift out of that data way better than you ever could.
And now you have a virtuous cycle.
So it's like really like the whole, the whole strategy on a plate and going back to the whole idea that obviously observability is the key underpinning of this.
Absolutely.
This ROI question of like investing in observability.
There are probably many CFOs that would say that their teams have over-invested in observability, right?
But that's because they're looking at it from a single, you know, one lens.
My co-founder, Charity, has a lot of posts on our blog around how to think about the cost and the ROI on observability.
I think she has a whole chapter in the upcoming engineering book.
Well, she's rewritten it for like...
business leaders and talking about this.
Very cool.
But one of the phrases that has always stuck with me is, at the core, do you talk about engineering as a cost center or a profit center?
Right?
Is for, especially a lot of companies in tech, the answer is like, obviously engineering builds the product that we sell.
And so obviously there's a tie between the work that we do and profit.
And yet there are some There's some classes of tools that are viewed as risk mitigation or insurance almost, and so put in the cost category.
And there's a lot of arguments to be made that tools like observability tools that enable faster feedback loops, that enable more confident development, that enable more change to be made more frequently, more confidently, actually those tools should be moved.
into the profit center.
They should be moved into accelerants.
And when I think about the customers that I've seen, you know, use Honeycomb really well, they're tracking things like, you know, they've moved past the, have we reduced our downtime?
And they've moved towards things like how many of our pull requests or GitHub issues reference Honeycomb URLs?
Because that becomes a signal that...
Our engineers are thinking ahead.
They are updating their mental models of what's happening in production.
They are building to the reality we can see right now.
And that is like a really cool way to sort of turn that measurement or think about impact in a way that aligns with velocity and positive investment in an engineering org.
So I have to ask you as well.
I'm curious how you think about the, the, the kind of skills that you think definitely engineers should be adapting, but also the skills that the engineering leaders, you think they will be picking up and, and around like the roles that we're all in, you, you get this broadening of competency.
where you can do a lot more in a lot broader ways than you could.
And you start to get folks that are able to bridge across disciplines, like a designer who can ship and an engineer who can update the website or, you know, a marketing copy or an engineer who's going into those customer conversations.
That's actually, I think, one of the biggest unlocks recently that people have been doing that we all should have been doing all along is.
Putting your engineers in the call with your customers and letting them understand how what their building is getting used.
We're using that now because it's a shortcut to kind of getting like, you know, cheap and easy and raw context is right out of the person's head.
And then you just go deal with how you're going to turn that messy data into action.
It's not like the highway of observability, but it is a bandage.
You're seeing like skills.
broaden and the kinds of folks that are kind of like entering in the field are much more diverse.
I'm just curious, like what you think about how you like your own skill set has changed, how you've seen the skills of engineering leaders, product leaders at Honeycomb change, if and what you think that has done for the whole organization.
I'm going to start with something that is very classic Honeycomb to say, but I promise I'll bring it into the AI era.
Our whole origin story is Charity was an ops person.
I was a developer.
I broke production.
Charity would be like, what did you do to production?
And I wouldn't understand the graphs that she was putting in front of me because they weren't built for developers.
And so you could say all of Honeycomb is about reducing this gap between development and production.
Building a shared language, building a shared understanding of what is actually, how are changes we're making to the system impacting our users?
And so I think that the...
change we've always been trying to drive, that AI in some ways is accelerating and making easier, and what you're sort of describing, is people being more aware of production, thinking of production and what your users are actually seeing as like a core input into what all of us are doing.
Certainly what you're describing, I mean, the example of an engineer being on a customer call is really just the qualitative version of what they should be doing with production data already, right?
Precisely.
With mouth words instead of telemetry.
Yeah.
Raw and just like you figure it out kind of context.
Here you go.
The other really exciting and interesting shift I'm seeing, though, that is a little bit sort of outside what we are trying to drive, but is a happy effect.
It has always been a below the line measure of success in an account for Honeycomb.
If we see people outside the engineering org reaching for an observability tool.
Right.
Because we aspire to be, you know, the truth about what's happening with your users in production.
How are people actually using the software?
When we used to see support engineers or product managers start to be able to ask, how's the user using this?
What does normal look like before, you know, identifying workers spinning off a ticket?
Awesome.
Today, though, now that MCPs and LLMs have just eliminated the barrier to entry to using these tools, We have one of the heaviest users of our dog food MCP inside Honeycomb is our head of sales.
Because who else cares about how our users are using our software?
Well, it's the people who are driving customer conversations.
We actually have a customer, sort of a leading inference provider, I can't name, but you recognize the name.
Yes, their engineering team uses Honeycomb, and their whole sales team does.
Before they go into a conversation with a customer, they're using Honeycomb.
Hey, how is this customer?
using our tool.
And they don't have to figure out how to use, they don't have to figure out how to interpret the graph, they don't have to figure out how to navigate the list of SLOs.
They can just ask their CLOD or their codex or their whatever to hook in the Honeycomb MCP, pass them the name of the customer or ID or whatever they use, and just say, like, tell me how their traffic has changed.
And this...
You talked about, you know, all these disciplines collapsing.
Like, it is so cool to be this source of truth for what is happening with your software, which more and more is just like the rate, like, what is our reality?
And, you know, Honeycomb's not going to turn around and sell to sales leaders anytime soon.
But it does feel really natural and exciting and right to have, you know, marketing teams.
doing gut checks of how healthy is the customer or, you know, hey, product, how is this feature being used?
Or there's all sorts of conversations around the org that can now be served by this source of truth.
And it's so fun to think about how relationships and roles continue to evolve when there's no longer a skilling up necessary to use a certain set of tools.
Exactly.
When you can be skilled by the harness or the tooling around you that you're using that meets you on your level.
Because context belongs to everybody.
And the context that matters is inside of a lot of different people who are all at different levels of investment in the process, including some of them completely outside of the process.
They're external.
And you're right to call out that that actually puts more of the burden on the vendor to make sure that what we are returning.
is good even with a less perfectly formed question coming in.
Exactly.
So really now every company's job and every employee's job is to become a data scientist.
To really understand all of these signals, what matters to me, breaking that down into when you start thinking about inputs and outputs at that level, at that level of granularity and capture and record, you start to be able to really actually form almost like more scientific.
like hypotheses.
You can run your business, you can run what matters to you and your customers, like business experiments that have just as much of a rigid methodology and research data underpinning them as anything else that you would call data science.
And so you're just applying it towards your particular.
business.
And everyone, I think, has a responsibility to kind of figure out what that means for them.
It doesn't mean that everyone now has to be some very seasoned data researcher, but everyone needs to be able to be fluent in the data that they're swimming in.
I would suggest, I think saying that it's everyone's job to become a data scientist carries a lot of like, well, what does that mean?
And all of that.
I would turn that, again, maybe 10 degrees and just say, it's our job to ask good questions.
In a way, that's always been our job.
But now we have more tools and more powerful things and more access to more data to ask those questions on of ourselves, of our systems, of each other.
So true.
You're saying that, too.
I'm like sitting here.
I'm like, this is the host of the show.
I'm like, my job is to literally ask questions.
I agree with this so much.
But then I also agree on dual.
And it's like I find myself now in an agentic world and the enablement that I've been able to kind of.
provided myself and my team, I find myself now asking way more questions than I ever did before.
And it's not even that because my level of involvement or commitment or dedication changed.
It was just that my ability to leverage those questions into solutions has dramatically shifted.
And my ability to then identify what matters to us most right now, that's all that matters, is figuring out what matters to us the most right.
And then I know I can trust that I have this system I can take immediate action on.
Because you eliminate that lag and execution from an idea, you become so much more bolder and brave to think outside of the box and to try new things.
But you can only do that if you have this highway of data, this support, this platform that you can trust and that's verifiable.
And that, frankly, is surfacing signals from everywhere so that you can run all sorts of crazy experiments.
Yeah.
When the cost of answering a question drops, you can ask more questions, thus building up the reps to ask better questions, which then you just have this beautiful flywheel of being able to go in a bunch of different directions and think more and do more, hopefully build up your intuition of what hypotheses are worth exploring.
Absolutely.
I'm really excited to see.
where this all takes us.
I've had an amazing chat with you learning about your perspective on how observability is driving all this transformation and the opportunities that are available to everyone right now with the data that's at their fingertips.
It aligns so much with things that we talk about on the show, and I know I'm going to continue thinking about it and sharing things from what I learned today.
But just as we wrap up, Christine, where can folks go to learn more about you and your work at Honeycomb and all the stuff that we chatted about today?
I am not terribly active on the socials anymore.
So if someone is interested in finding me, you can probably find me on LinkedIn.
But the Honeycomb website and our blog, we have a lot of our engineers who are incredibly thoughtful and also great writers.
It's kind of unfair.
And there's just a lot of really great posts about...
what does a healthy SLO process look like?
What is, or like, how do you determine great SLOs?
How do you build a process around it?
How do you do instant reviews?
A lot of the socio part of the socio technical systems we're all trying to build.
Highly recommend that.
I also, by the time this podcast is announced, we will have be on the other side of our innovation week where, and I believe if you go to honeycomb.io slash innovation week, we'll drop the actual link in the show notes.
But.
There will be a whole bunch of announcements that, again, I think nod to how we think engineering teams should work together with AI agents to do these sorts of investigations and ask the sorts of questions that we've talked about so far, as well as how to use observability tools to make sense of the new shape of software that we're building today.
as we move away from request response client server to something a little more complicated.
So there's a lot of cool stuff I'm excited to share with the world.
Well, we love a good engineering blog here at Dev Interrupted.
So we are going to be sharing it and continuing to showcase all of the great thinking coming out of Honeycomb.
And to those listening, if you are not following Dev Interrupted, definitely be sure to follow us wherever you're listening to us today.
Don't forget to check us out on LinkedIn and Substack, where we publish a weekly newsletter that will be accompanying this episode with Christine.
And you can find both of us on LinkedIn to continue the conversation.
If anything today sparked a thought or a question or tickled you.
you're fancy, or you really disagreed with it, we would still want to know and come find us and let us know.
And while you're at it, definitely be sure to follow Christine and check out the Honeycomb blog, which we're going to include.
So Christine, thank you again for coming on the show.
It was super fun to have you.
We'll have to have you back sometime.
Of course.
Thanks again.
Have a great day.
Are you looking for a trusted way to evaluate engineering productivity platforms in the AI era?
Gartner just released the first ever Magic Quadrant for developer productivity insight platforms, and Linear B was named a leader.
As AI changes how software gets built, engineering leaders need better visibility into productivity, bottlenecks, and AI ROI.
Download your complimentary copy of the Magic Quadrant to see why this category matters now, how the market is evolving, and why Linear B is recognized for its vision, execution, and workflow automation.
Check the show notes for the link.