# Deutsche Telekom's Observability Strategy for High-Stakes Reliability

**Podcast:** Dev Interrupted
**Published:** 2026-11-13

## Transcript

Hey, everyone.
Andrew here.
Quick bit of housekeeping before we get into it.
If you've been listening a while, you know Tuesdays are where our regular interviews live.
So a Friday interview landing in your feed is something new.
And that's on purpose.
We're bringing back our Friday episodes, but this time with a twist.
We take Dev Interrupted on the road.
We're going to conferences, we set up on site, and we're sitting down with the builders and engineering leaders who are actually there.
Today's episode was recorded at ObservabilityCon, Grafana Labs conference.
And we got more from the floor coming your way too.
The conversations are a blast and they look fantastic.
So if you want the full experience, be sure to watch them on YouTube.
And one last thing, if you like Dev Interrupted and you want it at your conference, let us know.
We'd love to make it happen.
All right, onto the episode.
Welcome to Dev Interrupted, brought to you by Linear V.
I'm your host, Andrew Ziegler, and today we're sitting down for a fascinating story on site here at ObservabilityCon by Grafana here in San Francisco.
I have Felix and Neha from Deutsche Telekom IT, and they're here today to tell an incredible story about how observability allowed them to deliver a little thing over the summer that you might know as the World Cup.
We're talking about some of the highest stakes in television around, and we're going to be diving in today about how observability is the hero and champion of it all.
And Felix, Neha, thank you so much for joining me for the show today.
I am so excited to dig into this one.
We are all so excited.
Thanks for having us here.
Of course.
I mean, I guess I got to start by saying y'all traveled quite a distance.
You're both here from Germany today, here in San Francisco.
You know, how often do you come to the Bay Area?
What's it been like to be here?
Every 15 years.
Never before.
So it's my very first time in the U.S.
Wow, welcome.
Yeah.
Are you having a great conference?
Yeah, definitely.
It's cool here.
And so many global players over here.
And we as well.
Yeah, absolutely.
Y'all are a global player.
Y'all were just up on the stage and y'all were sharing your story.
Maybe we want to start there.
I think that your story is pretty incredible and I want to give you an opportunity to share it to our audience as well.
So maybe you want to walk us through a little bit about what your roles are and what a day-to-day looks like and then how things start.
What a typical day for us looks like.
Maybe with the role first.
With the role.
Oh, we start with the role first.
So I'm head of observability at Deutsche Telekom and Neha is in my team at Das.
And I'm a consultancy team lead in Felix die Pa...
department.
So I'm kind of leading a team and only focusing on driving the adoption and scaling of the observability stack.
And so the both of you all work together on the challenges of observability within your company.
Yeah, we meet each day for half an hour to discuss those challenges.
Even more actually, not even half an hour.
Yeah, I would say even twice.
In the morning we need.
In the morning half an hour and then in the evening half an hour.
So there's a...
In between half an hour.
In half an hour?
And in your role, like, obviously, the World Cup is a high stakes situation.
There's a lot going on.
But in a typical day to day and in making sure and delivering, you know, the reliability of your platform, like, what are some of the major things that have changed for you even just this year leading up to the World Cup?
So the main change was definitely that we have now one single observability stack.
So this is the biggest change because we are kind of operating several observability stacks.
still, yeah, and we are trying to kind of combine them in one observability stack.
So that's the biggest challenge.
And I mean, we have really many applications running in Deutsche Telekom IT.
So I would say over 600 to 800 different applications.
internally and we are kind of trying to onboard every application onto our observability stack and really getting the telemetry data onto our stack so that we can actually reduce the meantime to detect and meantime to repair.
So that's the biggest challenge and that's really what we are working on right now.
Yep.
And you agree?
What do you think?
I'm giving her like what she needs to do.
So and I'm the visionary guy.
So to say all the visions that I have, she has to execute together with.
with Basi who's now also taking pictures of us.
What an incredible duo.
And Felix has always big visions.
Yeah.
So always he's like talking like the biggest, the vision is for me already complete.
You know, the work is already done.
It's just details about him.
These are just details what we are doing right now.
And he's always living in the future while we are living in the present.
Yeah.
And then someone else has to do it, right?
Because the person living in the future is too busy seeing the future.
I love this because this is really the heart of the challenge, I think, for teams right now is y'all are literally...
creating the ground in front of your company as they're stepping onto it.
And that big challenge of like, we have this huge ecosystem of tools and systems that we use that we have to, the stakes are so high, we have to bring them into one environment where we can look at them, understand them all, and then be able to provide that like premier kind of service to folks.
And I think along the way, like setting some North Star, some goals like a mean time to resolution and like reducing number of incidents, all of these things are really important.
And also really a great conversation.
to bring like non-technical stakeholders to the table because that's what they care about too because they're the ones that have to like deliver value to the customers on the more front lines.
And so- observability becomes the way by which customers and those non-technical executives and the technical teams can actually communicate across to each other and close those gaps and fix problems before they happen and keep problems from happening again.
You know, when you think about like organizing your teams and making them successful, bringing all of the data, all the stuff into one place is a big, strong bet you have to operate on.
Are there other big, strong bets or different ways that y'all operate now?
I would like you rang a bell a little bit with saying avoiding incidents or fixing incidents.
I personally accept that there will be always incidents.
So that will never change.
We will always have incidents because we have IT and the IT is so complex these days.
So we are stacking and stacking and stacking things above each other.
So there will be always incidents, always.
And if you accept that fact, then observability is kind of your safety net.
Let me put it like so.
So if something goes south, then you have the observability to alarm you.
And that is, since you were talking about the road ahead, that is what we will do this year, hopefully next year.
No, it's 2026.
So next year, we will change the organization and say, we accepted their incidents.
What can we do to enable you to, if it happens, to work with that, with alarms?
Maybe also as part of the software development lifecycle, but actually it's still Neha's job next year as well.
So we have 200 applications connected or 240 it was in the morning, right?
237 actually.
Yes, because you probably have a dashboard around.
We have a dashboard.
We are tracking day by day, even hour by hour, actually.
Hourly.
Yeah, hourly, actually, how many applications have already onboarded onto our observability sec.
So Felix, again, is living in the future, but let's be on the present.
You're living in the real world now.
So we are right now doing the fundamental work, I would say, like really creating the baseline because we just started.
kind of sending the metrics to our observability stack.
Okay, so there's a lot coming.
So we still need to do logs and traces and profiles.
This is going to be the harder work.
Some of the applications are already sending these kind of data towards us already.
But let's start with the metrics.
Yeah, so this is definitely the work we need to do.
And from...
I think how many applications we would like to onboard, it's around 600.
And we said it's 600 in total?
Exactly.
It's in total 600.
And we just onboarded 237.
So we said our timeline is like that, kind of strict.
Like by end of the first quarter, 27, we want to be done with the metrics.
Everybody, 600 vacations, everything on it.
So every application has to onboard to our observability stack.
Wow, that's like a huge challenge.
Like, how do you organize everybody around it?
Like, what are some of the places you do it?
Yeah, so what we actually did, one of the great things, and I think that was really worth it because you were also pointing out that we need to kind of also address the non-technical executives, right?
Because they need to enable their teams, which are, again, then the technical people.
Enable is a nice word for pushing them.
Okay, if you say so.
By the way, but in Germany, we are divided in Deutsche Telekom IT in eight regions.
So what we did is kind of a roadshow.
We went to every single region within Germany, also together with our Grafana counterpart, who is Dennis Dürmann.
So we went with him together in each and every region and really kind of sell the story and do hands-on.
hands-on lab session, what we were also doing.
How do we actually call it?
Yeah, hands-on lab session.
Hands-on lab session.
Yeah, exactly.
Always in the afternoon.
So Grafana was actually providing a Grafana environment and the people really had to bring their laptop.
Yeah, open the laptop, do it.
Exactly.
And then kind of we trained the people to kind of work on our observability stack.
So kind of you need to convince the people to onboard with their applications onto our observability stack.
Yeah, that is an amazing way to tackle the challenge.
Because as you describe like this staggering number of services that you need to migrate over, then my mind immediately goes to.
all of the data and all of the stuff they produce and immediately how there's going to be so much volume of stuff and how there's going to be such an engineering challenge around managing and just housing the data, much less making it actionable, useful, and real time.
So that whole challenge of bringing that all together is going to require standardization, going building to building, person to person, sitting down with each team, almost all 600 of them, and getting them on board and getting them bought in.
But then more importantly, get them all doing it the same way.
Because otherwise, you're just going to take 600 things that are boxed off from each other, and you're just going to put them inside of a big box.
But they're also going to be boxed up from each other.
So you have to also do the work while you're in there of standardizing, cleaning up.
Do you need to be putting all of that in our observability?
Or where is this blind spot that I think would be critical to your service?
So those become those opportunities for you all to come in as observability leaders.
And you also need to start selling the story from zero again.
Yeah.
Each time.
Each and every time.
I mean.
We have maybe sold the story like 100 times already and it still doesn't feel enough, but it's definitely worth it to repeat it again and again and again.
Because only then the adoption is successful at the end of the day.
It is a struggle.
I used to be a classroom teacher.
You have to say something three or four or five times just because you have to keep saying it doesn't.
It doesn't.
That's how people learn.
You need to be patient over and over again and be patient and just create the space like creating labs or creating an environment.
Instead of it just being an extra burden of, oh, I have to migrate this thing I already did.
That's so annoying.
No, let's carve out the time and the space for you to...
But you know, also in this virtual era, it's really difficult because whenever you try to do a virtual session, it's always like, you know, some email is popping up.
People are not listening.
So that's why you really need to go to the people to invest your time and resources and sell the story on site.
But to be honest, after I gave the presentation for the fifth time in a week, I was also quite bored.
So even though I like my visionary...
But the audience is actually always new.
It's nice actually to interact with them because they're always dealing with another application.
Yeah, yeah.
And for them, it's always the first time that they read the story.
Yes, yes.
Or you think like, okay, fifth time in a week.
I get that too.
Like I said, as a teacher, like by the time the fifth, sixth period rolls around and you've given the same lesson now, like all 45 minutes, all like...
10 times it's like you know it's like you kind of start feeling like you're droning on droning on but what you're doing is you're creating a standardized way that they're all going to learn and build together but i still still like customize my slides each time a little bit just to spice it up for me yeah for you because we are also learning from day by day with our own observability stack So we are kind of always updating our slides as well.
Not only the numbers that are actually onboarding to our observability stack, but also the storyline, definitely.
Because you need to address the right audience.
And every region is different.
And we are always analyzing, okay, who's coming to that kind of workshop right now.
So we need to address the right people and give them the right resources also.
This episode is brought to you by Linear B, the engineering productivity platform.
Your agents now ship more code in a week than your team has hours to review.
And on November 18th, we go live with a walkthrough of Linear B's new agentic studio built to deliver outcomes when human attention can't keep up.
Co-founder Dan Lyons opens with why we rebuilt Linear B around context.
Then the product's team shows it working.
I'm hosting with Jamie Burse, and we're saving the last 10 minutes for your questions.
So be sure to save your seat.
And if you can't make it live, register anyways, and we'll send you the recording and the assets afterwards.
Link is in the show notes.
This sounds like an incredible challenge, just also just because of the scale, but also the distribution and the stakes.
And also just the fact that your users, your consumers are effectively everybody.
So you're very much on, you're literally on the stage because you are the stage.
And so in that world, like...
Stakes can be really high.
And like I said at the beginning of the show, obviously, the World Cup happened very recently.
Put a lot of strain on a lot of services all across the world.
Your own included.
Would y'all like maybe share your story about what it was like to be a Deutsche Telekom IT during the FIFA World Cup?
Felix, you start.
My journey.
So we had the Situation Room in Bonn or Situation Room, that's how it was called.
Control Room, whatever.
Control Room, whatever.
So we had a big...
like eight screens we had big screens and they asked us like we want to have dashboards on those months and see where the customer comes in and then it fails or doesn't fail or if he hops over to the next to the to the booking to the cache to the to whatever we want to have really really cool dashboards and we had three weeks no three weeks three months so I said well you need to build that and we just signed the contract with Grafana And well, we knew that it usually takes four months to build those dashboards.
And then we were using the AI assistant and just said, let's just build it and see how it goes.
And went pretty smooth with the 80% version, just in the first shot in the first half an hour.
And then we spent two more days on refining.
Also, we had to do some stuff in the backend.
And then we were done.
And everybody liked it.
Everybody went on LinkedIn.
Not on LinkedIn, but our senior management went on LinkedIn, took pictures.
Yeah, this is exactly what we wanted.
Because they were proud.
Because they were excited.
And y'all got it done so fast.
Yeah, the scope of that project before would have been huge.
But now with the tools that Grafonic gave you, you're able to like instantly start.
And because you're building on top of your own context, so it already kind of knew the finish line you were trying to run towards.
Kind of.
Yeah, kind of.
Kind of.
Kind of.
There are always details which you have.
which you always have to work on which you will always have to work on also with the AI but yeah so that was at stake and Yeah, we managed.
We definitely managed and we were dealing with massive volumes of data, right?
So during the World Cup, you need to be really fast to respond to any incident.
And we had this kind of AI Grafana assistant, which was really helpful because there was one day, we were also saying that on the stage actually, but there was one day when we saw an increase of outages in the payment area.
And the Grafana assistant could quickly narrow down, okay, what is really affected?
external credit card provider service is down, it's not available.
So we could isolate the issue and also identify the root cause even before the customer actually noticed it.
Right.
And you have to do it.
I mean, the customers want to pay, which is a good thing, which is fortunately for us.
But if you don't have the option, then the whole booking process stops immediately at that point in time, which we wanted to avoid.
And luckily, we were able to identify it very fast, I think within minutes.
Yeah, and I think without this kind of situation room, that wouldn't have been possible.
Yeah, I think that like that's a huge opportunity is the triage that can instantly bring all the right people together.
Like, hey, this has happened.
I've noticed an abnormality in the data.
And as a result, like you get that way faster resolution time, which matters a lot to y'all.
But then also, too, you just get like.
higher signal on fixing the problem because then you can fix it in a more permanent way.
And obviously an external service going down is harder to fix in a permanent way, but it's still an engineering problem you could design.
But at least you can call them and say, hey, look, there's something down.
I know it's down before you know it's down because I'm using this tool.
Which is generally our story.
I mean, we had that also on stage when somebody was asking us, so how do you know that sometimes or very often still until this day?
we get the call from our business saying, I don't know, well, what's down?
We cannot pay anymore.
Or the call centers get increased volumes saying something is going wrong.
Then they call us IT and then we start investigating.
And this is what we want to turn around and say.
We want to be predicting it.
We want to be predicting it and finding it and being the responder.
Hey, there is a problem.
Not someone calls you.
Hey, there is a problem.
What is it?
And even fix the issue before the customer actually knows it.
So the best scenario would be the customer is not even...
Doesn't even notice.
Exactly.
But we are talking at the moment to our call center agents to say, until now you called us if we have a problem.
In the future, we will call you and tell you, I don't know, whatever system is down.
Please expect that more customers will call you and you can give that answer.
And it should be even possible in the era of AI.
It is.
It is.
So that will help them.
And that's exactly where.
So we are coming from a technical problem, but we turn into the business solution here.
That's important for me.
The visionary part.
Yeah, I mean, like a lot of the challenges y'all face, like they're partly engineering, but a huge amount of them are communication, interpersonal, you know, like just being able to collaborate effectively.
And that's like the thing about agents is they're a magnifier, they're a multiplier of both good and bad.
So like this is your teams, there's been doing a lot of going back to fundamentals and about like making sure we have really good and strong practices about how we think about features and how we ship things in our SDLC from beginning to end.
And in y'all's case, like how we even think about housing the data together because there's this, There's this opportunity to create a layer of context that is so useful that it can tell people that there's problems.
effectively before their problems, solve them before people encounter the problems.
And this becomes something really critical, I think, for like a high stakes kind of service with like, especially like in your kind of industry.
That is like the challenge, I think, for engineering leaders.
Definitely.
And also like how to manage the improvements, because there are always new, new features coming in.
So we also just don't need to scale this kind of story.
Just, OK, we have our observability stack here and there and you need to on.
but also to scale the best practices that we are learning from team to team.
So that's also a big challenge.
That will be our challenge next year.
It's also the challenge too of just like having the right ratio of like, you have like, you have the budget of everything and all of the world of everything you manage and then the budget of the observability and understanding that world and they're uniquely coupled and there's kind of like a ratio and as...
like you said, new things get expanded and added over here.
It's going to break and change this system.
They have to grow and flex with each other.
What are some of the big strong bets that y'all are making beyond, you said, this is the initial stage, is moving them all together.
And then once you're there, what are some opportunities maybe you feel like you want to take since you live in the future about once Neha's done all of the legwork of organizing?
She will have to do more.
I love challenges.
Yeah, the organizational challenge will be our first one.
So at the moment we have people staring at dashboards, how I always phrase it.
And I want to change the whole organization into an alarm.
or event-based organization.
So there is nobody in the night watching it, dashboards, you just get an automatic call if something goes wrong.
You get the event, you get the event in a context.
That's why we have AI.
And then you know why you are here and what you need to fix.
Ideally, in two years from now, visionary part again, in two years, we will do that auto-remediation, which everybody is talking about.
But that's a cultural change.
It is a cultural change.
It is a cultural change.
A shift in mindset, definitely.
Technology is there.
It's a cultural mindset.
How do you bring your product and engineering leaders to the table to have those conversations with you?
Or do you go to their table?
I mean, if you talk about restarting a central CRM system, for example, if you restart a CRM system, which may not be in five minutes, but half an hour, So that's like, do you restart it or do you try to still fix it?
So how much time do you spend where that and how much does it cost you?
We don't know.
But once we have that, I'm sure Neha will execute everything.
Yippee!
What was there for you?
Amazing.
So obviously during the World Cup, it was high stakes, high volume of data.
There were things happening, things that you're resolving and triaging in real time.
But then...
eventually, you know, the madness fades and you kind of come out of that and you're like a little daze, like, whoa, what just happened?
Like that was...
Nights.
Nights, right?
It's like, I haven't slept in a while.
It's like, maybe this is a sleep deprivation, but like, what am I supposed to be learning from this?
What am I taking away?
What were ultimately the things you unpacked and took away from the whole experience of going through that that like has strengthened your organization since?
Oh, difficult.
What did we take from the World Cup?
Oh, difficult.
Basically, that's that if everyone is pulling in the same direction, it's definitely possible.
Everything is possible.
Everything is possible.
Like even the onboarding, even if it's taking a lot of time right now.
But we had this kind of time frame that was really tight.
I think we said three months, three, four months.
Everyone was able to really onboard the application and to really create.
those dashboards we really needed to identify the incidents even before the customer actually knows it.
So that's definitely possible.
So we realized if there's a kind of tight timeline, everything is kind of possible.
We can do it.
We can really do it.
But that is also the problem.
Like, well, what do you do if that pressure is not there?
Exactly.
How do you pull the people?
Yeah.
I think a lot of engineering leaders, maybe even some of our listeners, are in that position where it is a push-pull.
Sometimes there's a vacuum and it's like everything's getting pulled one direction.
But sometimes it's completely inert and it just doesn't seem like you can get that kind of action.
So my advice to the listeners and to the guys who are watching, actually choose a tool, if it is Grafano or anything else, choose a tool which the engineers like and love.
then it is easier to convince them to work on the hard part.
That is, I think, one of our...
Definitely, because people need to like the tool.
Otherwise, they're not going to use it.
That's exactly right.
And then the adoption becomes a struggle.
Yeah.
We had Tanya Janka on the show.
She's a Wasp Hacker of the Year.
And she talked about how developers are like water.
They'll flow around anything you set up, any kind of system you create.
And so the golden path has to be the path of least resistance.
It has to be the easiest, the best path.
That's what we just discussed in the morning of another topic.
It's like it's like it's like I think in y'all's roles, it's like you become those you're curating that that golden path and you're making it easy.
You're reducing the friction that requires a lot of like, you know, a lot of rollout, a lot of sitting down with leaders and selling the future.
But ultimately, if everyone's pulling the same direction, it's possible.
And that's what all of this is starting to show us.
And while we've been sitting here, I've just I'm one.
I'm just so.
impressed and blown away by both of y'all.
The way that you can collaborate and talk about the work that you do together is just really inspiring to me.
But I'm also just really intrigued by the way that you talk about observability because it's predictive.
You're talking about this world that you're moving into where you're able to know the problems before they happen or completely avoid them because of this observability.
It's almost like the observability reaches into the future somehow.
And y'all can see that.
And I really see this bet from y'all.
And so like...
How do you think about that?
Where do you think observability is going to go?
Oh, difficult.
So I just know my vision.
I would be interested to know that.
You know that.
So we are already seeing nowadays that we get approached by security, by data privacy, and by business because we have a huge data pod which hosts a lot of data which we...
which they can serve from.
And in Germany, we have several restrictions as well.
Exactly.
So that's, so Obstermability can lead or can improve security and data privacy in a sense.
And it will hopefully also enable marketing, for example.
So if you know where on the journey customer is.
breaking off because the website is loading too slow, the app is wrong or he doesn't get the contract, whatever.
If you get that kind of data, which you have with observability, you can turn that into business capability and say, hey guys, improve that website or I don't know, put a button somewhere else.
So that is what I would wish for in two years.
What are you wishing for in two years, Mia?
Maybe getting away from dashboards?
Yes.
Because you can never have that perfect dashboard.
Yep.
Never ever.
And you can never have that kind of human power or, let's say, resources to kind of really staring at those kind of dashboards and then reacting later.
So I would be actually happy as an employee to kind of have a great good night's sleep.
And then, yeah, really.
And only get cold if it's really needed.
If it's really needed.
If it's not able to fix it on their own.
If the AI is fixing everything on themselves, but that is not only observability that is already...
Because I'm actually coming from operations world, so I know...
Yeah, you come from operations, yeah.
Valuable sleep actually is...
It's really cool if you can sleep, yeah.
Incredible.
Well, this has been an amazing, like...
glimpse into just how your day-to-day works.
Like Neha, Felix, just two incredible minds, I think, in the observability space.
I've had such a pleasure hosting y'all both on the show and learning about the work that you do, both thinking about the future, but also down on the ground.
But then also just like really working in these high-stakes scenarios that are really, really close to folks, like really close to folks.
One of our biggest listenerships actually on the show is in Berlin.
So, I mean, if you're listening to this and you watch the World Cup and you were in Germany, you can thank both of them.
for being able to do it.
And so thank you for joining us on the show.
And it was great to host y'all.
Thank you so much.
We enjoyed this chit chit.
So much fun.
It was fun.
Thank you so much.