New compass maps 120 metrics to guide AI ROI measurement

Andre Meyer, a senior researcher and lecturer, has published a paper in ACMQ titled "Too Many DevX Metrics, Too Little Guidance." The work introduces the DevX Metrics Compass, a tool that systematically maps over 120 metrics from more than 40 companies and research sources. The compass was created to help engineering leaders navigate fragmented frameworks such as SPACE, DORA, and Eng Thrive. Data analysis revealed that most company dashboards focus on AI use and output metrics, such as daily active users, acceptance rates, and lines of code, rather than value or outcomes. For instance, Google and Microsoft publicly shared between 20 and 24 metrics each, yet only shared around six measures in common. Meyer argues that traditional output metrics are easily gamed and inflated by AI, which makes code production cheap.

The compass allows users to benchmark against similar companies and receive actionable reports to select metrics tailored to specific goals, such as quality or developer enablement. Analysis of the data also reveals a significant gap in collaboration metrics. Most engineering metrics focus on development activities, with almost none addressing collaboration, despite developers spending 30 to 50 percent of their time in collaboration and communication. Meyer notes that while collaboration is a dimension of the SPACE framework, it is often the hardest to measure. Existing metrics rarely capture the quality of meetings or the balance between individual work and team interaction. Meyer’s earlier "Good Day" study at Microsoft Research, conducted with Tom Zimmerman, Chris Bird, and Earl Barr, found that good workdays involve creating value and efficient time use with few blockers. In the AI era, Meyer suggests that protecting focused time and structuring collaboration is critical to managing cognitive load and preventing burnout.