# Beyond Token Maxing: AI ROI and Software Factories

**Podcast:** Dev Interrupted
**Published:** 2026-08-21

## Transcript

All right, Angie, another week, another life beyond token maxing story.
I love it.
This time it's from a Linear B customer, which I love even more.
In fact, past dev interrupted guests as well.
So super.com.
We were all really excited internally about this really awesome article from their head of AI over at super.com.
Just talking about how they stopped counting tokens.
So yeah, I guess chalk it up.
Yet another company out there that...
You know, they start with those token maxing leaderboards.
They have fun for a day or two, maybe.
And then they immediately start thinking like, well, how do we actually make sure we're getting value out of this?
So it just, yeah, what do you think about it, Andrew?
It warms my heart reading this story.
Yes, well, it just keeps happening and I think it's going to keep happening.
And this article from Luis Morales, their head of AI, he does a really great job of explaining that token maxing and the leaderboard is just the first of many questions that you need to be asking about AI adoption within your engineering org.
Likens it to a...
like rungs on a ladder, right?
And he calls out that a lot of these big industry players like Meta and Uber and the like, they're all still stuck on rung one.
And while it's undoubtedly impressive, which he agrees as well, that companies of this scale can get to a point where they are using that many tokens and there isn't that much widespread adoption.
But if you don't grow beyond then asking what's the next question, which then he further defines as asking about the outcomes.
the business outcomes that you're getting from your AI adoption until you move to asking that question, you're just going to be stuck in this token maxing world.
So I think we're going to see lots, lots, lots more token maxing abandonment stories from leaders.
Yeah.
And of course we'll link to the, to the article in the show notes for this.
My opinion is like, yeah, make the token maxing leaderboard, like set a hard limit on the budget.
Just set it up, let your engineers go wild for like however long it takes them to spend that budget.
And then, you know, start to reevaluate how you can actually capture some gains out of that.
So, yeah, really great story.
And these are the kind of stories that we cover here on the Friday Deploy, brought to you by Linear B.
I'm your host, Ben Lloyd Pearson.
And I'm your host, Andrew Ziegler.
And this week, we are covering why GitHub is down again.
All of the new competition that's appearing in the...
cloud code hosting space, the latest in software factories and how to end projects.
So Andrew, let's talk about the thing that, well, for anyone who's a GitLab user, they're like, man, I wish our service would go down every once in a while.
So I could just, I mean, maybe it does.
Well, this is so from GitHub here, this story is honestly.
Interesting repeat that's been happening throughout a lot of the year.
I mean, obviously, as the world has, engineers have become increasingly agentic and their agents are slamming GitHub, which up until now has been like the de facto code repo, the place you go to throw your code.
GitHub is effectively one, just the general adoption of like code hosting for repositories.
And naturally, it became the place where people were chucking all of their agentic requests.
And GitHub, try as it made, is definitely having trouble scaling and keeping up with the uptime needed to handle these kinds of requests.
And GitHub has joined the many companies this year whose status pages have started to look like, you know, completely orange or a mix of orange and yellow.
It's starting to look like a bell pepper mix.
It's a meme at this point, like how status pages have changed dramatically at some companies.
Yeah, just like a year or two ago before everyone was orchestrating agents, like pretty standard for status pages for engineering platforms and services to be fully green with lots of nine develop times and lots of SREs around it.
Now it's typically more like a blend of a whole bunch of outages.
And this was something that hit GitHub over the last week.
They had an unfortunate outage that impacted a large amount of their customers and their ability to push code, which in our agentic world caused a lot of problems.
So naturally.
what happened in this environment, an environment where people had all of the tools to move fast and where they had an opportunity or see an option to maybe move and build their own moat, house their own code, because they've been doing this with a lot of software, right?
Everyone's been going down this build versus buy story.
So now suddenly GitHub goes down and people get these shiny eyes that they never had before of what if I built GitHub?
What if I had my own GitHub?
And so really effectively, it's kind of like...
GitHub is like Caesar in the Senate and like everyone's coming up and doing like one stab, right?
It's like you and he's like YouTube, Brutus.
And that's, I don't know, one of the other open source hosting platforms.
So everyone's spinning one up.
There's a really fascinating one that came from Cursor as well.
So you see AI native companies, AI model delivering, inference delivering companies also taking a piece out of the code hosting pie.
Yeah, I wanted to save this point for a little later, but yeah, code hosting is like that.
It's like one of those things that I feel like every developer feels like they can DIY it themselves.
It's like, I have ideas about how code should be version controlled and hosted and shared with my team and all of that.
I could surely do that and build it myself.
So it doesn't surprise me to see companies taking advantage of this situation and just more like a larger proliferation of these companies, particularly.
As the, you know, the timing of this with, with between cursor releasing their own hosting platform and the outage of GitHub is just like very unfortunate for GitHub.
But yeah, I mean, getting into the details on this, like, you know, GitHub had degraded service for like over seven hours, actually.
And it affected a lot of, a lot of different systems.
They had a cascading series of issues that caused just failure after failure.
And then.
The attempts to recover from the failures actually increased the burden on the infrastructure that was already failing.
And, you know, it feels kind of crazy to think, you know, we're talking about some of these new hosting platforms that are coming out.
It's really kind of crazy to think or to be speaking of GitHub as an incumbent.
But I kind of get the impression that they are more of an incumbent company in the Git space that is really struggling to keep up with all of this disruption.
AI is causing.
You know, I mentioned like, you know, how a while back I saw this meme of like, of GitHub before Copilot came out versus GitHub after and how the status page like was just a completely different assortment of colors.
And it's a very stark difference.
And it's a real one that we're all feeling through explicitly through outages.
I don't know if there's, if there's been another outage of this magnitude for as long for from GitHub in like quite some time, like.
This really does feel like it was a pretty major outage for them.
And I don't really want to like rib GitHub too much because I think they still have built a really great platform and continue to build a great platform.
But I really wanted to draw just attention to how this really kind of is representative of this broader trend that we're seeing where, you know, AI is starting to magnify quality risks within organizations.
I don't really want to call GitHub like legacy, but...
For older code bases, they may be more difficult to integrate with these new agentic workflows that are coming out.
But it's just putting more and more pressure on engineering teams to both respond to the incidents, but just generally speaking, to move faster at everything with AI.
GitHub has got to be feeling the pressure from all of these new companies entering into their space.
We mentioned Cursor.
And, you know, a lot of these fundamental fabric layers of software development, you know, the primitives, they are changing to accommodate this world where humans have to interact with other humans, but then also agents are involved to some degree as well.
And not all that collaboration, some of that collaboration happens in like your, or an increasing amount is happening in your IDE with the AI itself.
But there's still a high degree that needs that sort of like interaction between multiple humans and agents and AI.
Yeah.
I see this as a really natural progression, actually, of the build versus buy story that we've been covering a lot, where you see a service or a platform and you maybe don't utilize or need all of it.
GitHub is a large platform, is practically a social media network for engineers, and it has a lot of tooling and services on it.
So you also get this environment where people are taking more control over their infrastructure and the code that goes into it, and there's more appeal in owning that part of your stack.
just building exactly the slice of what you need.
So it's a really precarious situation for a company like GitHub.
And I think it's a really important bellwether to watch for other incumbents and for other industries and parts of the coding process that we take for granted, because the same kind of divvying up for parts could happen really to anything that we take for granted.
Yeah.
And I want to point out that this is not just happening to engineering.
either.
Like this is happening to a lot of areas of knowledge work.
You know, we've been seeing this a ton with content production here at Dab Interrupted.
You know, our velocity has increased so much because like all of the fundamental artifacts that we have to generate as part of our content production process is, you know, a lot of them are like fully automated at this point or very close to it at the very least.
And a lot of the tools that we built around how we used to do content.
These collaborative editing tools, like we use things like Google Docs and comments on documents and edits and like all of that stuff.
That was all designed in this world where it was just humans working with other humans.
And there's this feedback loop that we have to get into our AI workflows that has been difficult for us to capture.
So what we've had to do is sort of like hack together all of this like custom infrastructure to work around like just the complete lack of tooling.
in this space.
And I think that's, that's really what's happening to software engineers right now.
Like the velocity of, of teams right now is outpacing the tooling that's being built for them.
So they're having to build a lot of that infrastructure themselves.
And, you know, it used to be, everyone was talking about it harnesses like six months ago.
Now there's like a dozen that are out there that you can just go get that are open source and like have your own harness.
Or you've rolled your own because many folks have built their own.
Really, the new territory is the loops and the systems right immediately outside of that harness.
And, you know, it makes me even think of how folks are taking to this story on Hacker News exactly to your point, Ben, of like you have folks dropping advice in the comments about like what size hard drive that you need to run this kind of like a Git repo locally for X size team.
You have folks like arguing against.
like the oh, like you can't just say just like there's a lot involved and you have like it's like divided into camps very predictably with folks even offering up their favorite open source alternatives.
And so the reality is, is that, yeah, teams are strapping together with duct tape these really almost unique and very lightning fast solutions to problems.
Developers have always been like water.
They'll flow around any problem.
We learned that from Tanya Janka when she was on the show.
She rightfully called that out about security practices.
They are like water.
They'll go anywhere.
Well, in this case, they're like a waterfall.
They're going right over the edge, right?
And there's no stopping them from being able to host their code or run their platforms however makes sense.
So it's important to pay attention to that open flexibility.
Yeah.
And so, you know, we've been talking about like the agent harnesses now.
We've like.
solves that.
And then all of a sudden everyone's talking and we'll get into this in a moment, but now everyone's talking about software factories.
So it's like the moment that you feel like you've, you've achieved the new velocity, it's like this new thing comes along, but you know, I'll get off my soapbox on, on all of this.
Like I have a lot of empathy for the SREs who had to respond to the GitHub incident.
Like, I hope you're getting some time to recover and relax from it.
Cause I know how stressful that can be.
And, you know, and I guess the general advice is that that I would provide is that just make sure that if you're achieving higher velocity with AI, just make sure you're not sacrificing quality along the way.
You know, people can be forgiving of outages, but if that starts to become the norm and the expectation with your company, then, you know, that's going to cause a lot of downstream problems with retention and all of that.
So, you know, personally, I'm not I'm not the kind of person to just go out and like try to like.
roll it myself on my own infrastructure or like try out the latest tool that just got released.
Like I like to have more mature platforms for this kind of thing.
But at the same time, you know, I think it's just all important for us to be aware that like we need to start thinking about how just fundamental components of the SDLC like Git or like code reviews or how we generate code.
All of that is starting to change.
It is changing.
And we just need to be adaptive to it.
Exactly.
All right, Andrew, let's get into software factories.
And it sounds like everyone's talking about it.
And I guess I probably say that because you and I just wrote an article about it.
Yes, yes.
We might be a little biased.
But what Dev Interrupt right now is at the center of a zeitgeist of discussion.
around software factories.
And we've been hard at work trying to figure out what it means for you, our listeners and the teams and the opportunities that are ahead in organizing software delivery in this way.
So taking a step back for a moment, like what is a software factory?
Most folks, I would say, aren't calling their current SDLC or their AI adoption a software factory.
But the reality is, is that we're optimizing towards many parts of that state.
That includes getting to a point where co-generation and agentic P PRs have their own kind of creation process and come into a review cycle.
This is where many teams are encountering bottlenecks, which we talk about a ton on the show, where they build up and review and these agentic PRs don't merge.
So then you have to implement new guards and new ways to direct this kind of attention so that the human operators.
can keep the stuff moving through.
That's a lot about how a factory works too in the real world.
So there's a lot of lessons that we can extract out of that, but also too, a lot of opportunity for us to learn what it means for how we can ship code safely and securely and at scale.
So this article dove into some really emerging domain experts that we've found on this topic.
Some of them are also past guests on Dev Interrupted.
So it was a real pleasure to be able to focus and spotlight their expertise.
I want to call two of them out because one of them is Dex Horthy.
He's the CEO, co-founder of Human Layer.
And the other one is Zach Lloyd.
He's the CEO and founder of Warp.
Warp actually just dropped a software factory system that we're going to be talking about some in an upcoming workshop that we have as well next week.
But Dex is also attending this workshop because he's been writing a lot about how to watch for drift and rot and the problems that happen after you turn the lights off and you automate a software factory.
So they both bring a really valuable perspective on how to not only operationalize it, but then what to watch for.
there's a lot of opportunities for you to learn about maybe pain points in your own process.
So that's like a high level view of this story.
So if you haven't checked out the software factory coverage that we've been doing, I highly recommend that you start with that article from last week, dive into some of the latest announcements from our guests and be sure to RSVP for our round table that I'm hosting next week with both of them and Dan Lyons, my co-host here on Dev Interrupted.
and the co-founder of Linear B.
We're going to have a discussion about what it takes to know if a software factory is successful.
Ben, what do you think about this topic?
It really excites me, obviously, but what goes through your head when people talk about this?
Yeah, well, first of all, I'm glad we can get in on the ground floor of this phrase because I feel like within a month or two, it's going to be a catchphrase that everyone's kind of sick of because I think it's going to be what everyone's talking about.
So, yeah, I really do think it's sort of like the logical next step to you've started to figure out how to harness your agents.
Now you need those agents to sort of be orchestrated in a way that you can offload significant chunks of work.
And we've really started to think about our work sort of as like assembly lines.
Like there are there are components of the factory, the software that we produce that we can put on an assembly line.
and sort of make sure that it always comes out in a consistent manner.
And if we just chunk off work like that and start building assembly lines everywhere, eventually it starts to feel like a factory.
Yeah, so this is, I think, definitely the topic du jour right now.
You know, I've really had a lot of fun watching both Dex and Zach go off on kind of the deep ends on this concept.
And yeah, I think we're going to have a really great session.
next week on this topic.
And of course, Dan Lines, the co-founder of Linear B will also be there.
So we'll have a really great perspective on, you know, if you're, if you're still trying to unpack what is a software factory, like, do I have one in my organization today?
And then how do I scale that software factory and make sure that we're getting like measurable improvements from it?
Like, this is definitely going to be something that will help arm you for what I think is going to be like, the next big thing that all the executives start to ask engineering leaders about.
Your SDLC looks more like a software factory every day.
How do you get ahead of that transformation and how do you prove what it cost and what it delivered?
On August 27th, Dev Interrupted hosts a live roundtable on this very topic, proving AI ROI from software factories.
To learn, we've invited two industry experts and past dem-interrupted guests.
It's Dex Horthy of HumanLayer, who ran a fully automated factory and then shut it down.
And Zach Lloyd of Warp, who publishes frequently about how he measures what his factory pays for itself.
Linear B co-founder Dan Lyons will join them to discuss the power of the context layer that will make all of this possible.
Save your seat on Luma.
All right, Andrew, I just kind of wanted to cover a sort of a just a closing statement because, you know, I always like I always like closing out on more deep philosophical sort of things and less of the like, here's how AI is just taking over everything.
Yeah.
So we got this article from a friend of a show who we've been covering for a bit, James Stanier, titled Landing the Plane.
So why don't you explain what's in this one for us?
Okay, well, everyone who's ever started a project is familiar with the idea of scope creep.
Landing the plane is all about the ways that you just never really end up finishing a project and the real difficulty of finishing that last 10% of anything.
It's really easy to think of any kind of project as a linear task.
You go from 0% to 100% and then you're done.
But the reality is, is that you can get that first 90% very quickly a lot of the time.
And what takes most of the time is getting that last 10%.
There's a quote that Stanier includes in here that I really love that once you get the 95% of the way done, you're not even halfway there in terms of everything that you'd have to do to make it 100%.
And what this is.
really rooted in is there's two problems in why 100% is so inescapable or inachievable for many.
And one is just adding new things to do.
One more thing, one more possibility, never really closing the door on saying that something is complete.
I think even just all modern software 100% falls into this guilty as charged category.
I think back to like.
infrastructure level, things that are open source like and that are considered feature complete where the code base is done and it's closed.
So we're not adding anything new.
Like that's not really a state that code bases of today are in or trending towards.
Everything stays open as like an open ended thing.
You're constantly chasing that elusive product market fit.
You could never dare close the door on adding new stuff to your project.
And so that's something that causes a lot of things that just never get finished.
And another one is also just knowing how the cadences of how stuff gets done.
Somebody who's really good at getting something from zero to 90% is usually not the person who's good at getting it from 90% to 100%.
I've experienced this pretty much in every major project I've ever worked on.
you get the idea or like the delivery person and then you get the one person or the leader handoff that perfects it and gets it absolutely hardened for every kind of use case.
And those are very different types of skills.
So for solo builders or small teams, they just might be also missing that perspective needed to fill in those last three or 4%.
What do you think of this one, Ben?
Well, first of all, I feel a little bit called out just because.
It makes me really reflect on all of the things sitting around me that are at like 80% or more complete.
But, you know, we'll get to that.
And I don't want to spoil too much in the article because I do think it's very succinct and very well written and it's just packed full with tips and references to other resources that can help you develop your first brain on this.
You know, I think that...
It's really great to have a reminder at this time when it's easier than ever to start a new project.
It is so easy to start something new with AI, particularly if it's like a greenfield thing.
The hardest part is always going to be finishing it at the end.
So I think the saying is something like the first 90% takes 90% of the time, the second 90% or the last 10% takes the other 90%.
Just because AI is letting you create more than ever before doesn't mean all of that more is going to turn into something valuable unless you see it all the way through to the end.
So yeah, you mentioned a few of them.
There's a lot of classic tips in this, like protecting your scope, over-communicate with stakeholders, shield your team from new requests.
Run retrospectives to find ways to get better and just a lot of other things.
So yeah, just a lot of great general advice about how to be a leader in this very chaotic time.
Yeah, you know how there's like the ideas guy, the person who has the idea for like the game or the thing they want to build, but they don't want to do it.
If only I had a code or somebody who could build it right and that persona exists.
Well, now the ideas guy can just sling his idea into code that gets 90% of the way there.
But there's still the moat for the person who actually hardens the craft and delivers it.
And I think that's what this post really celebrates.
Absolutely.
Well, that's what we have for you today on the Friday Deploy brought to you by Linear B.
Thank you to our dear listeners for following us all the way till the end.
If you've stuck around, you know, the best thing that you could do at this moment is just to help us grow the show.
Give us a thumbs up, comments wherever you're consuming our media or.
engage with us over on LinkedIn and Substack.
You know, if you're only listening to us here on the podcast or watching this on YouTube, you're actually missing like a lot of things that we share on a weekly basis over on LinkedIn and Substack.
So make sure you go check out Dev Interrupted over there as well.
So thank you again for listening to us and we'll see you next week.
See you next time.