# Measuring AI ROI Beyond Activity Metrics

**Podcast:** Engineering Enablement by DX
**Published:** 2026-11-12

## Transcript

I want to start with an example from comparing Google to Microsoft because they are both in the data set and they both publicly shared between 20 and 24 metrics, yet they only have around six measures that are shared among the two.
And of course, there's the caveat that we can only compare what they publicly shared.
But it also, if you think about it, makes sense to not just use a fixed set of metrics and apply it to your company, but to actually tailor it to your own needs, to your measurement goals or to your initiatives.
Welcome to the Engineering Enablement Podcast.
I'm your host this week, Brian Houck.
Before we jump into the episode, I want to tell you about the CAFES framework, a new research paper co-authored by researchers from DX, Capital One, GitHub, University of Victoria, and Google.
It explores why context quality has become the central bottleneck in agentic workflows and introduces five properties to help teams make their AI agents more effective.
When teams provide better context, they build more effective agents and achieve far better engineering outcomes.
To learn more and read the paper, head over to getdx.com forward slash cafes.
Welcome to the Engineering Enablement Podcast.
I'm your host, Brian Haug.
Today, I'm joined by Andre Meyer, a senior researcher, lecturer, and software engineering practitioner whose work focuses on the...
human side of software development.
His work explores developer experience, flow, and many aspects of productivity.
His latest paper, Too Many DevX Metrics, Too Little Guidance, is out now in ACMQ.
It looks at how teams can choose metrics that help them understand and improve their experience.
With AI changing how we build software, we're going to be discussing today what's worth measuring, what makes the work rewarding, and whether we're automating the right things to begin with.
On a personal note, I first met Andre when he was doing an internship with Microsoft Research, working on a paper called Today Was a Good Day.
That paper was foundational in how I think about developer productivity and understanding and measuring sort of developer experience broadly.
I think I could say that that paper sort of kicked off my entire career.
And so it is with tremendous pleasure that I introduce my good friend and peer researcher, Andre Meyer.
Andre, welcome to the show.
Hi, Brian.
Thanks so much for having me.
It's always super exciting to talk to you about good developer experience.
Awesome.
So let's dive into your new paper.
Your paper opens with an engineering leader being asked, We've invested in AI, but has it helped?
And leaders today frequently have dashboards, but they rarely have clear answers from those dashboards.
Like, why is it such a difficult question to answer?
Like, you know, has AI helped?
Like, has our investment in AI helped?
Yeah, from talking to developers and building the data set around the compass, which, by the way, includes metrics from around 40 companies.
We've noticed that most dashboards are only covering AI use and AI output.
So instead of actual outcomes and value delivered.
So they would look at metrics around AI adoption, daily active users, acceptance rates, in some cases, even lines of code or token spend.
And those are really easy to collect, which is why they are so commonly used.
But it only shows if AI is being used and it doesn't really show how it helps to create more value or even true return on investments.
And also we've noticed that they use a lot of sort of classic output related measures like throughput and number of pull requests and deployment frequency and things like that.
And they were already a challenge before AI to like compare between teams and divisions and so on.
But this got even more ambiguous with AI.
The problem is that they were designed for a world where producing code was really costly.
And now it's almost free.
It's super cheap.
And so those metrics inflate a lot and they're also very easy to game.
And shipping more doesn't really mean that you're creating anything of value, right?
There is one positive thing that I've seen, though, from like AI measures, which is that most companies still capture qualitative feedback around developer experience and the impact that AI has with things such as satisfaction and friction and AI time savings.
So that's really interesting.
Sort of the idea that whether or not AI is helping is a difficult question to answer because the easiest metrics to measure are the ones that tell us sort of the least about outcomes.
And to try to help us reason about measures better, in your paper, you mapped out more than 120 different metrics, as you say, across over 40 companies.
And you used that to create what you called the DevX Metrics Compass.
Could you briefly explain what the compass is and sort of why you decided to create it?
So as you've mentioned, we've looked into all the measures that we could find in the engineering space, so what companies are actually applying, but also from the research space.
And we've seen that there are a lot of metrics and we've started to collect them in a systematic way and we've gathered the 120 or more measures.
And we've also seen that there are more and more frameworks popping up such as like space framework and dora and eng thrive and so on but it's super hard to really navigate all those frameworks and all those metrics and really identify which metric does really matter for my own team for my context or for my goals and It's especially hard to get started with defects measurement if you have no metrics at all or new measurement system at all, no dashboards.
So the compass really helps you to navigate this fragmented landscape by exploring metrics, by comparing or benchmarking against other companies that are maybe similar to you, and also by allowing you to receive an actionable report at the end to sort of know where to get started and what to do next.
For those who have not seen...
the DevX Metrics Compass.
I like cannot encourage you enough.
It is beautiful and just like awe-inspiring how many different metrics are sort of mapped together to different frameworks.
And it's like, it's a lot of fun to go and dive into.
And so as you were creating it, as you were diving into all of these metrics and use across the industry, what surprised you most about what companies choose to measure?
Before I answer one quick addition to my previous response, I wanted to give a shout out to also my collaborators on the compass briefly, Thomas Fritz, Gail Murphy and Patrick Meyer, who helped me with all of that and setting it up and giving feedback and so on.
And to come back to your question about what surprised me the most is that most of those metrics are focusing around development related activities and almost none of the metrics around collaboration.
challenge around that is that developers spend maybe 30 40 50 percent of their time in collaboration and communication but with those dashboards we don't really understand how this part is impacting their work lives and their developer experience i think collaboration is part of the very definition of developer productivity it is one of the dimensions of space and has been often the hardest to understand because collaboration is incredibly important to delivery and innovation, but it can also be an incredibly large source of toil.
And I'm curious, like, is there one collaboration metric or a couple of collaboration metrics that you've seen there?
Like, oh, that's a really cool way to look at it.
Not that I've seen, to be honest.
And there are only very few collaboration metrics that we actually have to be able to find.
That's the opportunity, I guess.
Exactly.
That's an opportunity for researchers, for DX and many others to contribute.
I think that some good measures would evolve around finding a balance between collaboration versus individual work time, meeting quality, meetings being scattered throughout the day, but also some of the metrics that we have seen, like focus or flow and interruptions, but also yet they are really...
hard to measure, but they are also very hard to act upon oftentimes.
And one of my more preferred ones are also to look into after work emails or communication because that's often a sign that something is amiss.
Just like a huge shout out to the idea of measuring sort of meeting like clustering.
And do you have sufficiently large gaps between meetings to do deep focused work?
found that that can be substantially more predictive of how much work devs get done than just like the total number of meetings.
So as you look across all these metrics, I suspect, you know, you found that even very similar companies choose wildly different metrics.
And, you know, how should a listener to today's episode Think about deciding what's worth borrowing versus not.
That's a fantastic question.
And I want to start with an example from comparing Google to Microsoft because they are both in the data set and they both publicly shared between 20 and 24 metrics.
Yet they only have around six measures that are shared among the two.
And of course, there's the caveat that we can only compare what they publicly shared, what we know that what they thought was mature enough and so on to actually share.
But it also, if you think about it, makes sense to not just use a fixed set of metrics and apply it to your company, but to actually tailor it to your own needs, to your measurement goals or to your initiatives.
So maybe in the beginning, you might want to start with more generic measures if you're new to.
defects or you want to start observing defects in some ways.
But at some point when you start to mature more and more, you want to look more into specific ones, maybe around a certain friction that you identified or that you want to tackle.
And then you want to create specific measures around that particular friction to see if your initiatives or your rollout actually makes developers' experience and work lives better.
In the paper.
You say that having more metrics doesn't equal more insights.
Does your compass run the risk of exacerbating that problem?
I think it may, because starting with the compass and seeing the 120 plus metrics could also be a bit overwhelming.
And you might just want to start to select everything that remotely sounds interesting.
This is why we recommend...
defects leaders or engineering leads or even team leads who want to start capturing metrics to take a step back first and think about what is the goal that I want to achieve with my defects measurement?
Is it to increase what value or is it to increase quality or developer enablement and things like that?
And based on that, start to filter down into the compass and start to choose metrics that are really specific to the objectives that you You had sort of hinted already that there's maybe a little bit of selection bias at play and that your data set captures what companies publicly report measuring.
How do you distinguish a real gap in measurement and something, say, like collaboration versus something that companies simply don't want to talk about publicly?
I do think that most of the gaps that we found or visualize in the compass are real with some caveats.
And of course, we can only capture what they publicly share before or what they admit to.
And I would assume that they basically share metrics that are mature enough or also they might omit some sensitive topics around like burnout or attrition and things like that.
So what we are doing is we are triangulating the gaps with other signals.
And one of those measures is that we have conducted interviews with 20 plus engineering leaders in the space to talk to them about their measurement practice.
And some of them are also included in the compass.
And they basically confirmed that the top most used metrics are also the ones that they publicly disclosed.
And those are basically the ones that are most actionable to them.
There is some nuance, though, like some composite or higher level metrics that often look very nice, like in a paper or presentation.
They are often really hard to act upon and might be eventually removed.
The other thing that we are doing is we are looking at research as well.
So we're not looking only at what...
companies share, but also what researchers found in industry papers, but also more scientific ones.
And there we see that there is a trend of looking way more at well-being related measures such as cognitive load and stress and burnout and so on in research compared to the industry.
And then finally, as soon as more and more people are using the compass, we can use anonymous usage data to see what people are searching for and what they don't.
necessarily find.
One early finding is that some people started to look for metrics around joy and meaning, which is not something we've really seen in any other paper or shared blog post yet.
That is like so heartwarming that the way that you're seeing the compass actually be used is to find some of these more humanistic measures.
And so I see sort of multiple sources of tension in the compass where practitioners might be more reluctant to talk about metrics that have not been fully vetted and, you know, they're still in progress versus researchers.
We want to see the cutting edge metrics to try to help refine them and think about them.
And I'm curious, do you have this notion of not only which metrics, you know, may or may not be harder to act on, but how difficult are they to implement in practice?
To admit, I haven't really talked to them too much about measures that are actionable or not, other than like a few occasional inputs that they have given us.
Very often, I think it depends a bit on how abstract the measure is and how it is actually analyzed.
So if it's, let's say, a flow measure, because this wasn't a concrete example that I've seen at a company, looking at flow from a very specific team level perspective or even a couple of teams, this is usually still relatively helpful because they can say, we have a small initiative.
Let's say we group meetings and we don't scatter them throughout the day.
Does this increase our flow?
On a larger scale, a couple of companies mentioned that it was really hard for them to make this actionable because people's work is so individual.
And what makes, in that case, good flow or what improves their ability to focus was just super hard to generalize and come up with initiatives that work for everyone the same way.
That's very interesting.
But I could imagine a world where even if conceptually a metric is easy to sort of action on, you know, it helps guide decisions.
From a technical standpoint, it is very hard to measure because of the required instrumentation and things of that nature.
And was that...
part of sort of coming up with the compass or thinking about metrics like looking at the feasibility of implementing some of these things from a technical standpoint?
Not necessarily.
I think we wanted to display what is out there as accurately as possible, but we did see besides some of the higher level or harder to measure ones that we've only seen at basically the bigger companies that have entire FX teams and good instrumentation to try this out.
The other thing that we have seen is that outcome measures or value-based measures are really, really rare and hard to get right.
And we also prompted people to talk a bit more about that in the sense of, is this important to you?
And how would you want to measure it?
And a lot really said that they struggle a lot in finding good definitions or good designs for metrics.
And then they resort to basically relatively simplistic ones, like on the customer side, maybe on things such as deployment frequency or if customers reported bugs or if they're happy and that promoter score or in a business side, mostly around AI spend and revenue.
And they are not really satisfied with that status quo.
But the key problem there is that they really have to not just looking to engineering data, but also triangulate data from project management software or customer support data and finances and so on, which is really hard even to get access to from like a privacy perspective sometimes or from a technical point of view.
So then like, let's get specific.
If an engineering organization wanted to measure sort of customer or business value for a specific engineering initiative, like what?
metrics should they be looking at?
So in the compass, we try to be very neutral in not making specific recommendations, but I can share you my personal point of view, basically.
Perfect, please.
I think while having to admit that it's really hard and it would be an amazing research project to do at some point, I think it would be interesting to look at it from...
an innovation side of view or innovation point of view where we look at how much time can developers actually spend implementing new features and pair that with user adoption data or customer satisfaction feedback so releasing this particular new feature that lots of customers ask for does this really make people happier does customer retention increase and things like that and i think around that we could also look into what the Microsoft Engineering Thrive framework does, ID to customer time.
Like how long does it really take us to ship innovation?
Because if you get customer feedback on what would be valuable to them in their daily lives or making their job safer or using the software more fun and it takes a year to ship, they will have forgotten.
already about it or switched the product.
I love that.
I think especially as organizations start trying to look beyond just the SDLC to the broader PDLC, things like time spent on new innovation and ultimately the velocity of that new innovation will become incredibly important and will matter to lots of different disciplines, not just software engineering.
So you argue that a healthy measurement system includes the concept of retiring metrics.
Like what tells you that a metric has outlived its usefulness, even if leadership may still like seeing it?
So there is this myth around creating defects measurement once and be done with it.
And many treat it as a simple project and not a continuous practice.
But the reality is that problems are changing, tools are changing, teams are changing and so on.
if we just create something once and don't evolve it that's bad but also if we just keep on adding metrics that's bad too because some metrics might no longer be serving a specific purpose and if we have too many metrics there will be a lot of noise or waste and dashboards that will be cluttered and it will be hard to know what is really relevant or what even changed and also i think metrics that have been used for a long time have a bigger chance of being misused for gaming and might be used as a target at some point, for example, for reviewing people or rewarding people.
Once people start to learn about a specific metric being a target to greater success, they might start to optimize for them.
So that's basically Goodhart's law, right?
And what I would recommend companies to look for is whether a metric is still actively being used.
So is somebody actually monitoring it and using it?
Did this impact any decision in the past few months or not?
If not, you can probably get rid of it.
Is it actionable?
Is it something we actually know what to act on if a metric gets better or worse?
And then also a question around should we keep monitoring it?
because it might have solved a problem in the past because of a certain initiative or a tool or a process that we've rolled out.
But if this is something that we have established well, in theory, we could remove it from the dashboard.
And if this eventually becomes a friction again, our quarterly survey, for example, would raise that again.
And then finally, I think we should also ask ourselves the question, Do we still trust the measure or are there any signs of like gaming it or being misused in benchmarking and things like that?
So that's really interesting because, again, there are sources of tension here where it's you don't want a million metric because if you focus on everything, you'll accomplish nothing.
But you also need to be able to see if you've regressed.
You know, there might be a problem.
So let's take an example.
Time to first pull request, you know, how long it takes a new hire to.
merge their first PR.
And maybe in a given organization, it's really bad.
It takes many, many weeks.
They focus on it.
They bring it down to, you know, within a few days.
I could imagine a world where there's this desire to say, okay, we've solved this problem.
Let's stop measuring it.
But then it gets worse over time and no one sees it because that measure has been retired.
And so how do you balance that tension?
And is your answer just...
We'll get it in this survey.
I think it really depends on the specific measure and again, the goals and so on that you're pursuing.
And I do think there should be a base set of metrics that you know how to apply them.
You know how to review them and you're really looking at it over the long time and almost like guardrail measures that you just observe for a while.
And I think those you shouldn't change and throw out or retire.
We are seeing some companies or also in the interviews, this was the topic of discussion where people said, yeah, actually we realized that nobody has ever looked at this metric anymore, but it just keeps being there.
So just a simple review once a year or so can help to see if a...
metric is still relevant or if it still serves a purpose.
But I do agree that maybe a baseline of metrics should always be around.
I remember when we first released space, people would eagerly show me their dashboards that would have every dimension of space and would have multiple metrics under each of those dimensions.
And even though in the paper we explicitly said like.
no, no, no, don't measure every dimension, like choose maybe three that are highly relevant to you, focus on those and then move to meet the pressing needs of the moment.
I'm curious from your experience looking into this, when thinking about, you know, having this set of baseline metrics, like what's the right number?
How many is too few?
How many is too many?
What do you think?
What have you seen?
That's a very interesting question, Brian.
I think there is not really a good answer to that because it's very individual to your measurement maturity, for example, or your ability to act on those measures, or if you're splitting certain measures or showing certain measures to only a specific subset of users and so on.
So I can't really give you an exact number, but I'm also curious to hear what you have been doing or what you would recommend to companies.
So I think from my experience, You need certainly, you know, more than one or two because metrics have to act as checks and balances.
And so you want these guardrails.
And again, I mentioned, you know, if you focus on everything, you'll accomplish nothing.
And so I think sort of three to five top level metrics is what I have seen make sense for a lot of engineering teams.
But I think of sort of a taxonomy of metrics or a hierarchy of metrics where you might have three to five top level metrics.
But then you have lots of data points underneath that help you understand those patterns.
And like there, you might have dozens of them, but those are like the drill down details for actually guiding the initiatives.
And that sort of makes me think, you know, if you looked at the metrics compass, did you see other engineering leaders or other mad scientists like me thinking of sort of hierarchies of metrics?
Actually, no.
And I'm surprised by that because I think it makes a lot of sense.
I mean, we did talk a bit about abstractions and having some very specific measures that only serve as a backup essentially or as an additional context to another metric.
But I do think it can be valuable to start with a couple of high-level ones.
Those are probably also the ones that you will not retire at any point and you will keep on measuring.
And then if you need more specifics, you can drill.
into those metrics to see other telemetry metrics or even to see qualitative inputs from interviews or self-reports and things like that?
I mean, I think activity metrics are a good example of some of the things that I think are sort of at that lower diagnostic level.
And we had opened this conversation talking about, you know, how do you measure the ROI of AI?
And as you mentioned, So many engineering leaders jump to activity metrics because they're easy to collect.
And I think those are really useful.
It is interesting to be able to explain how we do our work.
But sort of returning back to that question, then if you were giving recommendations to engineering leaders about specific measures of AI ROI beyond just activity metrics, what do you think you might recommend?
I think there are a couple of.
steps that I would recommend engineering leaders to take.
And the first one would be to define the objective, to think about why would they want to measure it and what do they want to optimize for?
So is it for improving the effects?
Is it to create more features or values or is it to improve quality?
Because this will shape the metrics that they end up selecting in the end.
And then secondly, they also run service.
And even though they would have very elaborated telemetry systems and good teams that can do proper statistics and so on, but they still want this additional layer of qualitative inputs.
And so on a qualitative side, I think I would look into the typical things that we already have done before AI, such as satisfaction, ease of delivery.
overall time savings, productivity, and things like that.
And then, of course, always pair it with a qualitative feedback on friction points and taking those friction points serious and actually acting on them and analyzing them.
And on the telemetry side, I think I would look for an established set of solid metrics that are measured across several dimensions, so not just output or activity or pull requests or lines of code.
all of that, and also measures that are observed over time.
So not just looking at them on like a single, as a single value or single snapshot, but looking at trends and relative changes and the like.
So ideally creating a baseline of some sorts.
I'm curious on that.
Is there a dimension that you think gets missed most often?
So we've already talked about value-based metrics that are very missed a lot and the collaboration aspects.
But also I think quality remains very hard to measure and the impact on quality.
But I think there are a couple of metrics that are helpful in that regard, such as rolling back releases afterwards and failures that are found after releases and so on.
Yeah, I mean, change failure rate is something that I'm...
really interested in looking at those patterns relative to AI adoption and proving the point on why looking across dimensions is so necessary.
We see across hundreds of organizations, change failure rate doesn't seem to be getting better as we're using AI more.
And like, that's something we need to keep our eyes on.
So shifting gears a little bit, as I mentioned in the intro, when we first met, you were working on your good day study.
Could you just explain very briefly to the audience what that study was, what the questions you were trying to answer?
So this was a project that I've done in one of my internships at Microsoft Research together with Tom Zimmerman and Chris Bird and Earl Barr as well.
He was a visiting professor at the time.
And that's also where I've met you.
And we wanted to better understand what makes developers' workdays good and what makes them bad because Potentially, this better understanding could help us to better understand the very individual factors and reasons that actually make it day good or bad, and it could help us to fix the processes and tools that can help us to optimize happiness.
At the end, happy developers solve problems better.
They are more motivated.
Retention is better.
They have less sick days, as you have shown in a study, and they will just do better work overall.
And unfortunately, what we're seeing is this tension now with AI between developer sentiment and sort of the amount of work we're producing.
Annie Bella calls it the productivity experience paradox.
And I find that just like a very interesting phenomena that is terrifying it in many ways.
So you were mentioning.
sort of this idea of mixed methods, right?
Like don't rely only on telemetry, don't rely only on surveys, but when you can blend them, that's even better.
And with the Good Day study, before the survey, you added yet another research type where you did observational work.
You actually like went and sat with developers.
And I'm curious, what did you learn from that that you wouldn't have ever been able to see in the telemetry itself?
Yeah, it's so much fun to reflect back on the work that I've done initially when I started my research in the space.
And the very first study that I've done were those observational studies.
And it was super weird, actually, because I had to sit behind developers for several hours.
I had to sit very quietly to not interrupt them and to...
hope that they forget about me while trying to really grasp what they were doing, the tasks they were working on and the context and so on.
And it was even weirder for the developers because they had somebody who took notes of everything they were doing, all their activities and bio breaks and meetings.
But it was an interesting experience to really understand, especially the crazy amounts of context switching that...
impacts developers work.
We later did some studies with automated tool usage data and there you get a lot of very rich data around activity switches.
So is a person switching between the ID and the browser and the chat?
But also they are not all bad, right?
So if you're switching from the ID to the browser and read up documentation, this might not be a costly contact switch.
But from just automated tool usage data, you wouldn't know about that.
So when we've done observations, we were able to distinguish between the different tasks that developers were working on.
And this sort of gave us this more nuanced understanding of the cost of task switching.
And we've seen there that some task switches are, of course, very expensive, especially when developers have to rebuild their mental model of a task.
And we're talking about very different aspects and topics of work or even work unrelated things but there were also tasks that developers reported as not being costly so for example when they were waiting for a build to complete or nowadays it would probably be an agent to complete the work they would switch to email or do a quick code review and then later report that this was actually a very good use of their time and not a costly contact switch but this is something we would have never learned from just telemetry data Yeah, they're like maximizing the gaps between the core thing they're trying to accomplish.
Yes.
So now AI is completely changing the way in which developers work and task switching, you know, this idea that we're all going to be managers of agents.
And so we're constantly switching between tasks.
I'm curious, if you were to rerun this today, what do you think would change around?
How, like what makes a developer feel like this was a good day versus this was not a good day?
So one of the core findings of the study was that developers said that on a good work day, they were able to create something of value, which means they were able to make progress and work on something that is meaningful.
And I think this hasn't changed.
The other thing that we've also seen is that on good days, they spend, they are able to spend time.
efficiently.
So this means they have very few blockers and no infrastructure issues and few interruptions and things like that.
Sadly, this also has not changed yet.
And I would argue that AI in some cases even creates more interruptions.
And then finally, they also said that on good days, they spend more time coding and they have more time to actually write code, which is pretty cool.
On good days, this was around 30% of their time.
Yet, It wasn't really about the act of typing code or like really writing it down.
It was again really about being able to build something that benefits somebody else at some point.
And this is also why many of the developers we are talking to today with AI, they are still pretty happy, even though AI essentially takes away that part of their job, which is the writing of the code.
But as long as they remain in the driver's seat and can...
choose what to build and design a software or design a solution, they are oftentimes still pretty happy about the creation part of their work.
Now, I do think that for many developers, having a sort of deep understanding of how the system works was also like a sense of joy.
And they liked to get deep into the technical details.
And, you know, you say that as we move to sort of higher levels of abstraction, you're still dictating what is what it is we're trying to accomplish, but maybe have less hands on saying the way in which it gets accomplished.
Like, does that run the risk of sort of complicating this good day, bad day relationship and exacerbating what Peggy's story calls sort of like intent and cognitive debt as well?
I would say yes or no.
So yes, there is this switch off.
from writing code to orchestrating agents more or supervising agents more and then reviewing code as well.
But I think it really depends on whether the developer can decide what they delegate and what they still want to keep close to their rest and whether they still understand the result in the end.
And I think it's a loss if people get or developers get.
bigger and more frequent pull requests that they need to review and they have a harder and harder time to understand the code base that they don't really understand at all because of intent and cognitive depth and so on.
But it's also a win when they remain able to set the intent and own the key decisions.
And AI basically just removes the tedious part of writing code.
And I think the reality is probably somewhere in the middle.
So we've talked quite a bit about collaboration.
So far.
And, you know, in your good day paper, you sort of like look at that tension a little bit between collaboration can be incredibly helpful, particularly around things like planning, spec generation, but it's also a source of interruptions.
And I know a lot of your work has looked into sort of focus time and flow state and things of that nature.
And so in practical terms, like how should we be thinking about protecting our focus time without treating all collaboration and sort of interruption and toil?
The short answer is it is very individual how you can find a balance between the two, essentially.
But meetings and interruptions being the number one impediment to productive work is something we've been hearing in all the studies that we've run so far.
But we have to remind ourselves that even on good days, In our study, we've shown that people still spend about a third of their time collaborating.
So collaborating is not necessarily bad, but it just depends on how much of it and when you have to do it.
And then, of course, it depends on the quality.
And developers talked quite a bit about meetings being bad in general.
But when you then push them a bit to explain in more detail, they talk mostly about the timing of meetings.
So if they fragment their time, there at the workday leaving very little time for actually focused coding especially and also when people are not prepared or when meetings are not moderated well and things like that and then there are also specific project phases where it's more common to actually have more collaboration such as around kickoff of a project or towards the end when you need to fine-tune certain things and so on this is a nuance that most developers understand.
So they don't really just want to get rid of all the meetings, but they want to structure their workday a little bit more.
And I think it's in the end really about, as you mentioned, protecting for focused work, where maybe for an hour or two, you can just focus on your own things and sort of have your focus head on.
And then...
You switch to collaboration, maybe you have a daily stand-up, and then you have a couple of follow-up meetings, and then you can go into focus again, where it's like not super split up and split out.
And we're seeing that very few teams actually talk about when do we want to spend time for collaboration and when do we want to spend time for focus.
And when it just naturally happens, everything is fragmented and everything is scattered across the field.
the week.
And this is especially hard for individuals that are more introverted when they have to switch between their collaboration head and focus head all the time and rarely have a few hours where they can just focus on their own.
Like clustering is so important.
Yes.
Or like batching of emails and chat messages and so on.
And I mean, one thing that we hear every now and then is that companies roll out no meeting days, for example.
I don't really think this works very well in practice because on most days you will still need to ask the occasional question or maybe want to do a quick sync.
So I think what works much better is if teams say, okay, let's, for example, start our day with focused work, then do a daily stand-up, then do a bit of collaboration, and then maybe in the afternoon we also, all of us again, look some time for focused work.
Really like structuring...
collaboration time, focus time together and not on an individual basis.
And consistently across days, not just like one given day, day of the week.
Yeah, that's just if I then have a question, but I know that my team will be around later for a collaboration time, I might be willing to wait for half an hour or an hour to ask that question instead of just dropping by my colleague and interrupting them all the time.
So sort of...
Final question for you, Andre.
If a team wanted to make more of their work days, quote unquote, good over the next month, what's one change that you would recommend for them to try?
And how would you tell whether it helped both the developers themselves, but also ultimately that result in good outcomes for the customer?
So my number one suggestion would be to optimize for focused versus collaboration work.
But another thing I have been thinking of more recently is that PR review load is increasing a lot.
And for developers, it's a huge challenge to deal with this growing review load.
And either they just rubber stamp reviews or they do it properly, which stresses them, increases cognitive load and so on.
So what I would do is I would...
gather with the team and agree on something like a PR etiquette for AI assisted work, basically saying, okay, you as an author, you own the change.
So this means you actually still need to read and understand your AI generated change before asking anyone else to spend time on reviewing your potential AI slope.
And then the author might also need to rate something like the risk level or like a confidence score or something because if it's a low risk change or like high confidence in that change we can maybe just trust our tests so we may not even need to ask someone else to or like another human to review it and we can just rely on an AI review but for the high risk ones or the low confidence changes we need to ask someone else to to review even not just for finding bugs but also to spread knowledge and to ensure that things such as cognitive debt are not growing right so for those we should probably i would recommend the team to them think about or agree on what is a size of a pr that is still reviewable and reasonable and also what is a good review turnaround time so if i understand it's like a mix of both having the time to do deep focused work and then having clearly agreed to sort of accountabilities and boundaries on where everyone's responsibilities lie on that.
And you think like doing those two things will lead to the better days?
Absolutely.
Yeah.
I think if we can reduce the number one thing that we still keep hearing in the age of AI, which are meetings and too many interruptions, that's actionable and simple thing to work on and explore.
And the other thing really is the growing cognitive load that developers keep talking about and the growing burnout and stress that they need to tackle.
And I think managing this a bit better or finding ways, very specific team specific ways to deal with that better is something that can have short term positive impacts already.
Awesome.
I think that's a great place to leave it.
I'd like to remind the audience, too many DevEx metrics, too little guidance is the paper out now.
ACMQ could not encourage people to read it anymore.
It is phenomenal.
Andre, thank you so much for joining us.
It was my pleasure.
Thanks so much for the good discussion, Brian.
Cheers.