# Measuring AI Impact on Engineering Productivity

**Podcast:** Engineering Enablement by DX
**Published:** 2026-10-02

## Transcript

I would say until quite recently, the currency at Meta was code.
So diffs played an extremely central element at the company.
And so everyone talks about this all the time.
And it is then natural that the business wants to know how much time are we actually spending on that very central thing that we're talking about all the time, which is this.
And thus diff authoring time was born.
Welcome to the Engineering Enablement Podcast.
I'm your host this week, Brian Hook.
Before we dive into the episode, I want to tell you about our Q2 State of AI and Engineering Report.
It looks at data from over 500 engineering organizations and examines where AI is generating measurable gains in throughput and time savings, where those gains are not translating into shipped software or improved developer experience, and what the data suggests about sustainable AI investment.
To read the full report, head over to getdx.com forward slash reports.
Hello, welcome to the Engineering Enablement Podcast.
I'm your host, Brian Hauck.
Today's guest is Moritz Beller, a software engineering researcher at Meta whose work sits at the intersection of empirical software engineering and real-world developer productivity.
Over the years, Moritz has helped advance our understanding of software testing, developer workflows, continuous integration, and more recently, not surprisingly, AI-assisted software engineering.
Today, we're going to dig into some of the work he's been leading at Meta, including new ways of measuring engineering productivity and what those findings tell us about the future of software development.
Now, on a more personal note, I will say that Moritz did an internship with Microsoft Research many, many years ago.
And I first met him as my team helped on one of his studies, which was foundational in my transition into a more formal software engineering researcher.
And so Moritz has played a huge role in shaping my career.
And so I am incredibly excited to have him on the podcast this week.
Moritz, welcome.
Thank you, Brian.
Well, it's an honor to talk to you.
So just to help set some background context, could you tell us a little bit about your role at Meta?
Yeah, like you were saying, I'm basically doing sort of wearing three different hats in a sense.
I have this academic background on doing software engineering research.
So naturally, a lot of the problems that we then tackle at Meta also have a research component to them.
So it's not...
like just taking a requirement right and seeing it end to end through to production but actually usually you know kind of ironing out what it actually is that that we are interested in and then designing some studies to further identify that that typically then yields some empirical results we do that If we think, right, we can do it ourselves and then we do that.
But if we think that some more rigor is needed, we maybe pull in folks like you to actually get us good statistics.
And once we have that, then with a team of actual engineers, we then decide based on the data, okay, what should be the best approach to implementing this or just kind of surfacing it.
as a dashboard, be it for leadership or for direct personal consumption.
It's really like basically a research head, a kind of software engineer head, and then we'll kind of take leads and kind of orchestrating between the different roles.
I love it.
We've sort of overloaded the term applied scientist.
Typically now it means people actually like building ML models, but like in my mind, like that's what real applied science is, right?
Like you're doing actual research and then.
You're doing it not just for the sake of advancing sort of knowledge, which is a valid cause, but how do you put it into practice?
And so speaking of that, some of the work that I've seen from you, you know, over the last couple of years, talks about a metric that I think is incredibly interesting called diff authoring time.
And so just like to help us start with the basics, what exactly is diff authoring time?
diff authoring time we should maybe prefix this by asking the question what is a diff for those that are not super familiar with terminology so it's essentially the equivalent of a pull request right so it's a self-contained code change together with a summary of the changes that you're making or the agent is making in today's world with a test plan and then reviewers have the possibility to leave comments on this and you may have multiple rounds right multiple versions of that code change and so basically i would say until quite recently the currency at meta was code so diffs played an extremely central element at the company not just you know for kind of performance reasons but simply because this is how we were talking about things and this is what moved the company forward from an engineering perspective and so everyone talks about this all all the time basically and it is then natural that the business wants to know hey how much time are we actually spending on that very central thing that we're talking about all the time which is this and thus the authoring time was born which basically measures the active time that humans spend engineering, reviewing, maybe even preparing, testing, diff.
That's super interesting to me.
A lot of my work has looked into how are developers spending their time, which of those activities are sort of productive, what aren't, what are the, you know, how do they relate to each other?
And, you know, I definitely have looked at, well, like how much time are developers spending sort of like actively writing code.
And I know that it is you know, maybe changing today because of AI, but in general, like that's the activity developers want to do the most.
Now, are you able to tie that sort of authoring time to specific diffs?
Are you just like averaging out over all of the diffs that someone may participate in?
It started out as practically being an average.
This was a kind of cruder metric that predated DAT to fill the immediate business need.
But then we realized, hey, this leaves a lot of gaps right there are a lot of things that we can't do one of the more important ones is that we can't actually run controlled a b experiments much in the same way that we do in product so of course um any anyone who's interested in software engineering knows this that sites like facebook instagram but also others run lots of a b experiments um all the time to find out, hey, which of these features gives a better user experience, which of these features optimizes for certain other metrics, right?
So performance could be other metrics as well.
But internally, we weren't quite at that point because we didn't quite have that metric for our own internal development and how we could assess basically the development experience of our own engineers at Meta.
So these cruder metrics wouldn't allow us to do that because say if we had introduced a certain feature or people were using a certain feature in a diff but were also not using that feature on say on the same day or the same period where we were averaging over we wouldn't get clean signal.
And so it was pretty apparent that if we wanted to make our internal development more scientific and basically bring it on par with the standards that we had in product, we would need the fidelity that diff authoring tab allows us to do.
And so that's exactly what you say.
Pretty much on a millisecond basis, we can tell sort of what diff you were working on at that point in time, if you were working on a diff.
So that's like...
That's wild to me, sort of the level of precision there.
But you said something there that like really piqued my curiosity around like using this for A-B testing.
And so I think probably like the natural reaction to that is, oh, well, like using different development tools.
And it's like, oh, like development tool, you know, if I'm using VS Code versus Emacs, like what does that do to my default during time?
If we do different sorts of like build optimizations, all of these sorts of things.
But I'm curious, like, Do you ever run wild AB experiments like playing with our environment?
Like, oh, well, what happens if we put a bunch of plants in?
What happens if we put like UV replacement?
Red walls versus blue walls versus cream walls.
Like what sorts of AB tests might you be running?
Yeah, I think it's closer to your first.
I suspect it.
But then the second one, because there is a cost, I don't think we have a team, right, that optimizes some like wall color in development environments, although that's an interesting thought.
But so one example would be, and actually I basically have two really good examples that were controlled A-B experiments.
And one of them is using auto-memorization in the hack compiler, where basically people had to previously hand roll their own kind of if you will, caching mechanism in the UI.
So if the state of something changed, then you would need to kind of, you know, recalculate for all of its children whether their state also had to change and then possibly re-render this.
That is both error-prone and pretty costly to do because also to be redone for every single UI element that you wanted to have caching enabled.
And so then the React compiler team set out to basically make this a supported default in the React forget compiler is the name of that.
And so we were able to compare diffs that were using this feature of the new React forget compiler versus diffs that were hand rolling their own.
memorization slash caching in the past.
And so we didn't even look at the sort of, you know, quality benefits that you would get from that.
But just on the on a diff authoring scale, we saw quite massive improvements.
It was like, I think, around 30 percent fewer DAT for the diffs that were using, you know, the memorization.
Of course, it was a big lift from the React team to get that in.
So you would have to eventually, you know, trade off those costs against the one-time investment.
But if you have enough reuse of that feature later on, it's certainly paying off.
I mean, I certainly acknowledge that.
running A-B experiments, using diff authoring time as sort of the outcome variable, to look at practical changes in engineering tools and engineering workflows.
That makes sense.
That is, I think, probably a more useful use case.
But the mad scientist in me wants you to run things like, hey, if we take a bunch of people who work remotely, what happens if we drop a dog into their environment versus a cat?
What's more disruptive to their diff authoring time?
But maybe we'll save the mad sciency experiments for some future collaboration, you and I.
Yeah, I know Microsoft Research had some interesting stuff on that and I think are still doing it, right?
I think we are a little bit more conservative in that approach, but I do agree with you.
It would be interesting.
And I think there's certainly factors that are maybe much more influential than what we're currently measuring.
We are products of complex environments.
Absolutely.
Exactly.
Yes.
So everyone...
is passionate about their own developer experience metrics, right?
Like there's lots of opinions on what metrics are best for capturing developer productivity.
And I'm curious, you know, what do you think was missing from the metrics that meta was already using that diff authoring time sort of solves that makes it so suitable for running experiments like you just mentioned?
So it's a velocity metric, right?
In a sense, it has to be embedded in, if you're interested in kind of drawing a relatively complete picture, it has to be counterbalanced against some measure of quality if you care about that.
And of course, different businesses would associate different weight to that.
And also through basically the throughput.
And I think this is basically the kind of triangle that it works in.
So we have another metric called DDM.
uh diffs per developer per month that was basically pioneered by my good friend karim nakad um who you should maybe also invite him our our shared good friend karim ddm is interesting because it gives you it gives you uh basically a throughput metric in a given time frame that's hopefully large enough to you know factor out some minor um changes in like day-to-day life if you take a PTO or something like that.
And then I think there's a quality aspect, but then you could, like, if you want to stay sort of in the diff ecosystem, another thing that would be interesting is to look at like how large are those diffs, right?
And those three kind of factors, like throughput, size, and duration, they would naturally balance each other out, say, if you wanted to gain one of these.
then you would see movement in the others.
So, but I think maybe coming back a bit to your question, what were we missing in that landscape that we had was that we had this relatively crude metric that averaged on a daily basis.
And we found that it just didn't provide the fidelity that we needed.
And in fact, now that DAT has existed for some three years, we actually can, we still benefit from the investments that we were making because we see that it, applies much better in an AI world than our previously averaged metrics.
It's much more sensitive to movements that we would expect than these averaged metrics.
So there's actually continued investment in it.
That's very interesting to me because I suspect when people hear something like diffs per developer or even diff authoring time, there might be some concern on, well, like...
not every diff is created equal.
Like there are different expectations for, you know, more, more senior developers versus more junior developers.
So if these things are being used for performance assessment, like it might not be truly capturing what, what I'm doing.
And I can understand teams like yours, where you have a rich background in rigorous research, like using this responsibly for things like running A, B tests.
I'm curious, like how you actually see this used on, you know, development, you know, frontline development teams in the wild?
Like, what are the different real-world use cases for this?
Right.
So I think we covered, I think, the scientifically most exciting one already, which is running IB experiments.
Even if you had an intuition, right, of like, hey, this helped our developers, it really is beneficial to be able to quantify it.
And that then allows, you know, organizations to plan for it and to prioritize certain changes that they might otherwise not have.
So, for example, the magnitude of those DAT savings that we saw in this experiment, but also in the second one that I was briefly referring to earlier, were unexpectedly large.
And so it seemed to us that we needed to invest more into kind of foundational changes in frameworks and in tooling rather than maybe micro-optimizing UIS a little bit, right?
Which is a bit different than what you see in the product landscape, where anecdotally, I think there's paper by Airbnb where they, did they change the button by a couple of pixels or they changed the color, right?
And it yielded a statsic, huge improvement for them.
We haven't actually seen this with DAT, right?
Maybe it's...
a different population.
But this is interesting.
So it's basically experiments and then they informed how we as an organization think about planning and which changes are being funded.
And then I think even more so it also informs the way of developers for even if they're not using DAT, which they really don't have to for every single feature, how to how to think about designing their products and their changes.
So it's still so different, I would say, state of mind when you're developing more like a scientist.
And then this is one class.
And then I think the second one is for basically leadership reporting, right?
Making sure that basically the team or again, then company level, how like how are our metrics doing?
Do we see kind of unexpected crazy regressions in something?
Are we hitting our goals if we set certain goals because they're associated with different initiatives?
I mean, thinking about goals as it relates to diff authoring time is really interesting too, because not all diffs, certainly not all sort of class of activities are created the same.
I'm curious, do you try to sort of bucketize diffs by like, is this new innovation versus sort of running the business versus some sort of like, you know, paying down some sort of tech debt?
How do you think about that?
That's exactly what we're doing.
We're categorizing this a certain way there.
And what's also interesting is you would expect different distribution of those categories per org or team, right?
A team that's an infrastructure, if most of their diffs are internal, that's totally expected and fine.
However, if you're a product feature team and most of your diffs are internal, something's strange, right?
It could be iClassify.
It could be like...
the work you're doing.
Research is pretty clear, like the literature is pretty clear that developers want to be spending time actually building, right?
Like they want to be spending time building.
We certainly as an industry want engineers, and we talk about this as part of the DX core four, like we want to be spending more time delivering innovation.
And so how do we spend more time on delivering innovation?
And like diff authoring time, especially if you can categorize the diffs, like I see how it's a good measure for that.
But it's not just a velocity metric in my mind, right?
Like there's also sort of an efficiency standpoint.
So is it really about can we drive total diff authoring time up or down?
Or do you actually look at it as like time per diff?
And are we getting more?
Are we creating more diffs per unit of time that we're spending?
Yeah, so we've never used it for performance reasons, right?
Unlike other metrics that were surfaced in the tools that were used to assess performance, diff authoring time never appeared in there and that was a conscious choice in our part.
That said, especially now, right, but also for meta, the year of efficiency is 2022, which is when diff authoring...
time development started.
So that was not a coincidence, obviously.
So we are interested in that, even if it's not maybe on an individual basis or on a performance base.
But yes, so we basically look at, okay, how many tests are being created in a certain time period, like per month, and what was the...
DAT for that, right?
And the reason for that pair is basically, well, you can trivially, you know, 2x your DDN if you half your DAT, but the end product is the same.
We want to avoid that.
Probably it's not exactly that relation because there's some overhead associated with creating more diffs.
So actually, like...
the incentive is not to do that for yourself, right?
For your own review, because in theory, what you're evaluated on is the impact you shipped and whether that's done in 10 or 20 discs in isolation would make a difference.
Interesting.
So like that certainly looks at the, you know, the game ability of it as like, what can individuals do to game it?
But I was thinking even like more, more high level, which is just like, if you want to look at system efficiency, you know, it's, Even if you sort of normalize the diffs in some way for size and complexity and all of these things, it's if I can create 10 diffs with an hour of diff authoring time versus I can create 20, and maybe that comes from like my build tools are faster or more predictable.
So I'm not spending as much time waiting around.
Our code review processes are better because we're like automating code review with, you know, radar or something.
I'm curious if it's used in that way at all.
Oh, certainly.
And I mean, the biggest game changer that we've seen there obviously was AI, right?
And so what we've seen there is basically our year over year DAT is down by over 40%.
At the same time, DDM has increased substantially.
And now you could say, well, maybe you just made your diff smaller, right?
That would explain that, but actually diffs are getting larger as well.
So we've never had, and this is at the company level, right?
Which is obviously the strongest indication that you could see for this.
So we were, you know, occasionally observing something like this for individual teams when they had, you know, different initiatives.
Sometimes also, to be honest, I think some of those are just...
noise because at the end of the day it's a very large company with so many teams you're always going to have some outliers so you need to be careful not to draw conclusions from this but if you get such a clean and strong signal like we've seen with ai adoption where we can also ascribe certain events in time to certain reactions in the graphs it is a it is very clear so this is all wild to me it's First off, I saw that, in fact, I recently published a newsletter where I talked about a study from Microsoft, from Alex Heilman at Microsoft, that similarly found that the number of pull requests per hour of active coding had increased about 40% in Microsoft.
And it seems like, you know, we're seeing very similar orders of magnitude.
And I'm curious, you know, if we're having more and more code.
but having to spend less and less time actively authoring it.
Do you have some idea of where does that time go?
Like, is it going to other things that are still building activities, but we don't really think of them as diff offering time because it's not actually writing code.
It's generating specs.
Is it going to, you know, validation time?
Like, where is that time going?
Yeah, this is only an informed opinion that I'm going to usher now.
It's not actually scientific.
We're still looking into this, right?
I think it's context gathering.
I think you're also right in pointing out that a lot of the activity that we traditionally considered not coding are now perhaps, or should now perhaps be considered as kind of adjacent part to coding.
I think it's also pretty obvious that not a lot of time goes into prompt writing.
I mean, yes, some sometime goes into it, right?
But I think it's more so like, I think initially we had a point where like, oh yeah, I'm just going to stuff my prompt, right?
With like all of the information I can get.
But now it's, I think, more efficient to just link the information because the agent can then go ahead and retrieve that itself.
And so I think there's some of that going on for sure.
But I don't think we have quite found where all of that you know, time is exactly spent.
I think there's also a narrative that you could say, well, maybe we need to spend more time reviewing now, right?
And I've seen that.
I think GitHub has made some similar findings.
On the other hand, I wonder, do you think that a line of code still has the same value?
Do we even need to review it that thoroughly?
Well, I mean, I think there's two questions in there, right?
Like, does a line of code have the same value?
And I mean, to the...
business maybe, but to sort of developer platform James Peeblestone developer productivity will know because the cost of generating that line is so much less.
I mean, you said at the very, very beginning, you know, historically code was the currency of software engineers, the currency you cared about and now really intent.
is the currency we care about like what is the throughput of our our ideas and i think how do we measure those things are like the going to be increasingly the important thing yes yes and it's it's the idea that we put into the system actually what's coming out in the in the diff right or is there some drift between the two going on Yes.
And so much around, you mentioned so much time now is sort of going into gathering context.
And like, how do we think about the quality of that context, which would include all of the prompts you are, you know, it's not just the data you're feeding in, but like, are you able to clearly express your ideas?
And then, you know, from, from harness and model capabilities, does that translate all the way through to finished product?
And there's lots of places where the human agent collaboration can break down as it, as it turns out, I think.
So, you know, as we look at, like, you have this rich data set on how much time are developers actually spending to create these atomic units of work?
And maybe these are now the atomic units of work of agents moving forward and sort of intent is the atomic unit of work for developers.
And you're seeing how patterns are changing sort of broadly.
Do you see differences as you slice into different cohorts?
Like, for example, are junior devs seeing their diff authoring time drop faster than you know because of using ai than you know very senior devs are we have um initially observed the opposite of that much to our surprise as well um where it's definitely to my surprise yeah um and i think it's sort of this finding's been corroborated in in the industry um where it seemed like you know seniors were able to more effectively use these ai tools But I think that has evened out some over time now.
So if you, at least from a metric perspective.
As I think about all of this, it's, you know, I'm struggling with this idea that traditional telemetry misses a lot of the invisible work that goes into software engineering, sort of whiteboarding and...
brainstorming and talking with colleagues to sort of do architectural alignment.
And I'm curious, can we start leveraging LLMs to sort of like synthesize our meeting transcripts or our meeting audio to capture some of that missing piece that is going to become increasingly important and have that sort of feedback into the definition of diff authoring time?
Yeah, I played around with this idea at the very beginning of when we started at Meta to record our meetings internally.
Now we're basically auto transcribing everything if you want it to be auto described, right?
So then it's pretty easy to do a similarity match, I think, between this meeting and the diffs that are out.
I think that that would be very interesting to kind of strengthen these links that are latent but do exist right now.
I also think it's, you know, there's also an interesting shift.
So on the one hand, we say, well, it's much cheaper to generate the code.
Therefore, maybe we don't need to be as careful in architecting it.
But you could have the counter argument, actually, for a very recent iteration of DAT.
We are now at version 8.
The person who implemented it didn't have a design doc.
But then what that...
cost is that I needed to spend a lot more time reviewing this and basically extracting the design from the details of the code where I didn't need to look at the code where I'm thinking, well, if you trust the agent to implement your architecture, then it would have been much easier to just get the architecture because that's the piece maybe we care about.
And so maybe just because we can skip the architecture step, we shouldn't, right?
Yeah, I think Dr.
Peggy Storey's recent triple debt framework talking about the rise of not just technical debt, but cognitive debt and intent debt would argue pretty strongly that we shouldn't just skip the architecture phase.
So this notion of how developers are spending their time and how it's changing sort of makes me reflect back on where you and I first met, where you were working on a study called Mind the Gap, which was hugely influential in my own work.
And multiple of my feature studies relied on sort of the concepts of Mind Your Gap very, very heavily.
And so could you give us like a two minute version of what you were trying to study and what you learned from that project?
Thanks, Brian.
Pretty simple.
This is about developer productivity way before AI.
Basically, we had these traditional productivity measures such as lines of code per our worked number of features, maybe even something like function points on the one hand.
And then there was a relatively nascent research trend to look at perceived or self-perceived productivity.
But those two lived in isolation, basically.
And what we wanted to do in that study with Thomas Zimmerman, who was my mentor at Microsoft Research, was to basically bridge the gap between the two.
And so we just wanted to assess both at the same time and basically counterbalance the drawbacks that each of them has with the strengths of the other.
And so, yeah, this is like, I was super excited about doing that.
I mean, I am a huge proponent of...
mixed methods research and I think measuring large scale telemetry and blending it with sort of developer perception is so valuable.
And I think one of the, for me, really important findings of your study is that there is a relationship between those two.
The developers actually are, when they are assessing their own productivity.
like that has a relationship in the actual activities they are doing that you can measure with telemetry.
That's right.
The number one thing we saw there, which you, I think, mentioned here and there in this episode already was time spent coding.
And we were also, lo and behold, it was pretty low back then at Microsoft, right?
So it was like, you're a software developer, so like most of your time should be spent.
But it's actually not the case.
Like the core development time was not the number one activity in the day.
Much to the dismay of many developers.
And now lots of studies have sort of come along and reconfirmed that, yeah, time-semit coding may only be 13%, 14%, 15% of the day for many developers.
That's right.
Yeah.
What's also interesting, we did find that I think overall this was a large driver, but we also saw things like whether you're on call or I think Microsoft calls the DRI designated response individual, I believe.
It's been a while.
That had an influence, but things such as like how many interruptions you had are also counted.
So some of these were like attributes that people had to enter into the system manually with all of the telemetry.
that we have in place at Meta now, we would be able to automate this, which you could say is a more truthy signal in that case.
I mean, if I remember correctly, one of the strongest predictors you had of someone's self-reported productivity is whether or not they had a good or a bad night's sleep.
before, which has influenced a future study of mine as well.
It's just like these sorts of things matter, which like harkens back to the crazy experiments we should be running, the crazy.
Yes, that's awesome.
How did you measure whether someone had a good night's sleep or not?
So in Mind the Gap, I know that it was self-reported and this study that hopefully we will be publishing soon, we used fitness trackers.
We used...
Fitbit first of four is to actually quantify sleep quality.
So I'm curious, like the world, you know, Mind the Gap at this point in time has got to be six or seven years old, right?
And it's the world has changed so much since then.
And so if you were to rerun it today, what activities would you add?
What changes do you think you would see in the behaviors you measured back then?
Like, how would it be different?
How would Mind the Gap be different in 2026?
I'll give you an answer to a question you didn't raise for a second, and then I'll actually answer your question.
If I were to rerun it then, I think the one thing that we were really missing was the lines of code.
So we had the number of pull requests, but it was more binary feature because it was in the Windows devices group where we launched this and people were basically at max, like, putting out one pull request a day because, you know, kernel complicated and stuff.
So I would have like this, I think would have really, you know, put the silver lining on the paper, but we, Microsoft had just changed to GitHub.
And so this feature wasn't available.
I mean, do you think that lines of code would have had a measurable relationship to self-reported productivity?
Or do you think you would have found that like, no, lines of code doesn't actually like that?
Yeah, I'm pretty certain it would have had one.
I don't know just how strongly, right.
And the reason for that is, is like the more time you spend coding, the more code you write.
And so therefore, that's true.
Yes, of course.
But yeah, how strong that would be interesting.
Like in general, I think of lines of code is a pretty garbage measurement, but you are right.
It's like, as you code more like time spent coding matters and there's got to be a relationship between time spent coding and the lines of code you generate.
Now as to the actual question, what would we be doing different today?
We would have to incorporate a agentic behavior much more.
Some measure, some form of How well are you able to interact or even operate in today's world, right?
Which is very parallel and where I don't think there are very clear patterns yet on what really works and what doesn't.
I think I would also like to have something like a sort of more opinionated question on like, hey, what's your opinion on agents, right?
And so...
I would be very interested on people that are like on different ends of the spectrum of like, hey, I'm trusting this a lot or I'm not trusting that so much.
And because obviously if you trust everything, you're going to commit stuff that doesn't work and worst case causes problems.
And we've seen, you know, these outages are already, I think pretty much every large tech company has had one of one agent cost outage yet.
So I wonder if there's, you know, some of the burrito frontier or somewhere where like hey i'm like i'm 80 uh trusting this uh so i wonder if there's something like that i think other than that i might be very curious to read your study on the health aspects of uh job performance well this like makes me think of a recent study from annie vella that showed that sort of productivity and developer experience may be decoupling.
And DX found in our most recent AI impact report that even though the amount of work that we are producing is skyrocketing because of using AI and agentic experiences, our sort of developer experience index is flat at best.
And so I am curious also then if we were to rerun Mind Your Gap, if...
the relationship between things like diff authoring time, coding time, and self-reported productivity, if that is even weakening now, what would you expect there?
I only have opinions right at this point.
I think it may not weaken.
I probably suspect that if you run too much in parallel, that that could...
be weakening.
I think the same influences for kind of, you know, why you might have a bad day are still true, which is on the one hand, all of those health reasons, right?
And then the other ones, kind of environmental ones.
And I think the third one are still pretty much like, I don't see those change at all, which is like, hey, I have too many meetings.
We're in endless alignment meetings.
We're not like, we're getting pulled in one direction and then, you know, the following week and the other one.
So I don't think that changes at all.
But I could see that there's now like a fourth category, if you will, which is like how satisfied and how cognitively not overloaded am I with my setup of using agents?
And I think that's actually, I think that's a relative benchmark to your peers, right?
You see what they produce.
And so I think that's an interesting aspect, yeah.
We've talked a lot about sort of coding and active coding, but that the value of a line of code may be dropping because it's so much easier to produce.
And lots of people across the industry are talking about sort of shifting bottlenecks.
And now, you know, planning's a bottleneck and validation's a bottleneck.
And you have another, you have many incredible papers, but another one of them is when, how, and why developers do not test in their IDEs.
And if I recall correctly, you're a A big finding is that a lot of developers just don't do any testing.
And I'm curious, are you breathing a sigh of relief at this point in time?
Because it's like, oh, well now AI can fill that gap.
Or are you even more worried because it's like, we weren't great at validation before and now we are going to understand it even less.
I think both are true for different reasons.
I think it's very easy to add tests now.
you can have unit tests you can have in some cases even end-to-end tests because you have a harness that allows you to screenshot the result and then you know the agent can either operate on that and improve or include it as part of the test plan so at least it's easy right if for nothing else then we can either include it in our system prompt or we just um if there are no tests we just you know tell the agent to add a couple tests and um people understand that testing is more important now because we don't quite trust what the agents are doing.
On the other hand, I also think that the presence of tests, something that's green, right?
That might, you know, lure you into false sense of security of like, hey, the agent actually did what we are asking, but there really was a drift in what it did versus what we asked it to do.
Or maybe not even a drift, right?
But just underspecified on our end and you can have different interpretations of what was asked.
Yeah, that's like, it's so interesting.
Now I think I'm more worried than before I asked the question on this, like notions, like, yeah, it's so cheap to, I mean, it's like, that's why we're having these conversations.
It's so cheap to generate tests.
But if you don't really care about testing before, then you're not like really going to know what you should be validating that the agent actually like wrote the right tests.
I wonder if that feeds into this sort of triple debt notion that I mentioned before.
Yeah, maybe we need a, you know, some sort of resurrection of TDD in the age of AI.
I like, I'll be honest, I loved TDD.
And like, I'm curious if like, you know, what is sort of like, is spec driven development?
Is that sort of like conceptually very similar?
And it's as we think about generating our context, how much do we think about, well, what validates the correctness of our, our, the action, you know, our goals and how much, how do we specify that ahead of time?
Like, is that the new version of TDD?
Yeah.
And so far, right.
What we see often is all, we'll just throw, you know, another agent with maybe different agent harness and a different model at it.
But, you know, I like don't quite think that that's the answer.
I also, I don't think that ultimately what's going to be holding us back in many cases is.
sort of the capabilities of the models themselves.
What is one aspect of software engineering that you still think we lack a good way to measure?
I think it's how you interact with your colleagues, right?
How you can argue for certain projects, which is like all of these interpersonal dynamics, assuming that there are still people working, right?
Apart from that, and somewhat related to it though is basically arguing or being able to see through the changes that you make.
I think what we're seeing now basically is that so many new features get implemented, so much new stuff is happening.
It's very hard, even if the functionality is there and we're perfect, which I actually doubt is always the case, but for the business to think harness.
what is there right because people move on so quickly so there's no real sort of consumption of of what has been implemented unless it's very obvious and you know very high priority so i think for this latter part and then also feeds into hey let's not redo the same thing multiple times because It's so easy to do that now, right?
And everybody's kind of working in the same environment.
It's easy for people to redo the same thing again.
And I think we need a solution, at least at a company level, for flagging this or avoiding it if we want.
Not maybe for the sake of avoiding, but at least making other people aware of, like, hey, Brian is also working on productivity metrics.
Maybe Brian and Moritz should talk to each other, at least.
I love that concept.
You know, in the space framework, sort of communication and collaboration is explicitly part of the very definition of productivity.
Like, how well are we actually collaborating with our colleagues?
And yet it is by far the least measured and the least understood.
Like, we can see interesting patterns between, you know, collaboration and other outcomes.
And I've recently written about a case study where we saw that, like, a organization, effectively, all they did was cancel their low quality meetings and they had a bigger throughput gain from that than they did from adopting AI.
And so we know that these things are incredibly important, but we don't really know how to measure like, are they good or not?
Like we can see after the fact, like, was this bad if we got rid of it?
But like, if we could measure that more effectively and then use it to increase innovation velocity, I think it would be so powerful.
That's right.
I think this kind of meeting canceling, I think that's an interesting point.
A colleague of mine wrote about this during COVID, where I think initially we saw a huge productivity boost because I think basically people's time got freed up so that they could implement all of these things that were on their backlog.
But at some point, that backlog...
was drained right and so then they argued that we were in a situation where we were actually you know um basically suffering from the lack of meetings or lack of alignment um so i'm not sure if i you know buy 100 into this but i think it's a it's a interesting thought yeah well there needs to be balance i found that canceling low quality meetings had a bigger positive impact to throughput than ai did and so it's like oh let's cancel all meetings now on the flip side again, sort of going back to COVID days, where it's like, oh yeah, we found sort of the height of the COVID pandemic days.
You know, we found that, oh, having more uninterrupted time, like led to more throughput.
Now on the flip side of that, I worked on a study, Tale of Two Cities, where we found that The best predictor of a developer's self-reported productivity was whether or not they frequently engaged in informal small talk before or after meetings.
And so like those small human moments of like, oh, hey, how are you doing?
Like, how was your weekend?
Like that was more predictive of a developer's productivity than the quality of their internet connections.
And so like we need these human interactions, but it has to be balanced.
And so it's like, how do we understand that?
I think it would be so valuable.
relation the more they checked in with their colleagues the higher their productivity was oh that's cool fascinating yeah yeah like more the more they checked in on just like sort of just like trivial like human moments and so it's like these things matter and so it's like not all collaboration is bad bad collaboration is bad and so how do we understand that more like i think that would be that would be awesome So I think with that, as we look to the future and all the future things to discover, that is a phenomenal closing note for us.
Moritz, this was a phenomenal conversation.
Thank you so much for joining us.
Thank you, Brian.
I've had a blast.
Cheers.