Deutsche Telekom uses AI to detect payment outages during World Cup

Deutsche Telekom IT operated a control room in Bonn during the FIFA World Cup, using eight large screens to monitor customer journeys and system failures. The team built dashboards for this operation in approximately two days, leveraging an AI assistant to generate an 80% complete version in the first half-hour, a task that typically takes four months. During the event, the Grafana assistant helped identify an increase in outages in the payment area, quickly narrowing down the root cause to a downed external credit card provider service. This allowed the team to isolate the issue and identify the root cause within minutes, before customers noticed the failure in the booking process. The company aims to shift from reactive incident response to predictive monitoring, where IT proactively informs call center agents of system issues to prepare for increased customer inquiries. Beyond event-specific monitoring, Deutsche Telekom is consolidating its observability infrastructure into a single stack to reduce mean time to detect and repair incidents. The company operates between 600 and 800 internal applications, with a target of onboarding all 600 to the new stack. As of the latest report, 237 applications had been onboarded, with a strict timeline to complete metrics onboarding for all applications by the end of the first quarter of 2027. To drive adoption, the observability team conducted a roadshow across eight regions in Germany, partnering with Grafana representative Dennis Dürmann to provide hands-on lab sessions. The team emphasized that standardization and repeated in-person training are necessary to ensure all teams adopt the same practices, noting that virtual sessions are less effective than workshops for driving behavioral change.