Aug 21, 2026 · Dev Interrupted · 5 min read

Beyond Token Maxing: AI ROI and Software Factories

Engineering leaders are moving past vanity metrics to measure true AI business outcomes. This analysis covers the shift from token usage to value capture, the rise of software factories, and the infrastructure challenges posed by agentic workflows.

Technology Business Strategy Engineering Management

The takeaways

Shift Metrics From Tokens To Outcomes. Stop tracking token consumption as a primary KPI. Instead, measure the specific business value and efficiency gains derived from AI integration to ensure sustainable ROI.
Adopt Software Factory Assembly Line Models. Structure development workflows like manufacturing lines to standardize agentic code generation. This approach reduces variability and enables scalable, consistent delivery of software components.
Mitigate Agentic Infrastructure Strain. Proactively upgrade hosting and repository infrastructure to handle high-volume automated requests. Relying on legacy platforms without scaling for agent traffic leads to critical outages and productivity loss.
Prioritize Project Completion Over Initiation. Allocate dedicated resources to the final 10% of project delivery. The skills required to harden and finish software differ significantly from those needed to build the initial 90%.
Build Custom AI Collaboration Tooling. Develop internal infrastructure to bridge the gap between human collaboration tools and AI workflows. Existing platforms are often ill-equipped for hybrid human-agent feedback loops.
Monitor For Post-Automation Drift. Implement continuous monitoring for quality degradation in automated pipelines. Without active oversight, software factories risk accumulating technical debt and functional rot over time.

The End of Vanity Metrics in AI Adoption

The engineering industry is undergoing a critical maturation phase, moving away from superficial "token maxing" leaderboards toward rigorous measurement of business outcomes. While early adopters celebrated high token consumption as a sign of AI penetration, leaders like Luis Morales of Super.com argue that this metric is merely the first rung on the ladder. The strategic imperative now is to quantify the actual value generated by AI integration. Companies that remain stuck on usage metrics risk misallocating resources and failing to demonstrate return on investment to stakeholders. The shift requires a fundamental change in how engineering leaders define success, prioritizing efficiency gains and feature delivery speed over raw computational volume.

Infrastructure Strain and the Rise of Alternatives

The surge in agentic workflows has placed unprecedented pressure on legacy infrastructure, exemplified by recent GitHub outages. As AI agents generate and push code at high velocity, traditional code hosting platforms are struggling to maintain uptime, leading to cascading failures. This instability has accelerated the "build versus buy" debate, prompting organizations to consider self-hosted solutions or emerging competitors like Cursor. The incident highlights a broader trend: the fundamental primitives of software development are being disrupted by AI, forcing teams to re-evaluate their dependency on incumbent platforms. Organizations must now view infrastructure resilience as a strategic risk factor, not just an operational concern.

The Software Factory Paradigm

The concept of the "software factory" is emerging as the next major framework for organizing software delivery. This model treats code generation as an assembly line, where agentic PRs move through standardized creation and review cycles. However, this approach introduces new bottlenecks, particularly in the review phase, where human operators must manage a high volume of automated changes. Experts emphasize the need for new guardrails and monitoring systems to prevent quality drift. The success of a software factory depends on its ability to maintain consistency and security while scaling output, requiring a shift from individual craftsmanship to systemic orchestration.

The Final 10% Challenge

Despite the ease of starting projects with AI, finishing them remains the primary bottleneck. The "last 10%" of development requires different skills than the initial build, focusing on hardening, edge-case handling, and stakeholder communication. Leaders must protect scope and manage expectations to ensure that AI-accelerated initiation translates into completed, valuable products. The ability to close projects effectively is becoming a distinct competitive advantage in an era where starting new initiatives is trivially easy.

Key insights

Token usage is a vanity metric that does not correlate with business value. Leaders must transition to outcome-based metrics to validate AI investment.

Strategy →

Impact: Prevents misallocation of budget and aligns engineering efforts with corporate financial goals.
Agentic workflows are overloading legacy infrastructure like GitHub, causing significant outages. This instability is driving adoption of alternative hosting solutions.

Infrastructure →

Impact: Increases operational risk for companies reliant on single-vendor platforms and accelerates market fragmentation.
The "software factory" model standardizes AI code generation but creates new bottlenecks in human review cycles. Effective orchestration requires new guardrails.

Process →

Impact: Enables scalable delivery but demands significant investment in workflow automation and quality control mechanisms.
AI lowers the barrier to starting projects but does not solve the difficulty of finishing them. The final 10% requires distinct skills in hardening and scope management.

Execution →

Impact: Identifies a critical skill gap in engineering teams and highlights the need for dedicated completion phases.
Existing collaboration tools are ill-suited for hybrid human-agent workflows. Teams are forced to build custom infrastructure to manage AI feedback loops.

Tooling →

Impact: Creates an opportunity for new tooling vendors and increases the technical debt burden for engineering teams.

Action items

Replace token usage dashboards with outcome-based KPIs such as feature delivery speed and error reduction rates. Define clear business value metrics for AI initiatives.

Impact: Ensures AI investments are tied to measurable business results rather than activity volume.
Audit current code hosting infrastructure for resilience against high-volume agentic traffic. Evaluate self-hosted or alternative platforms to mitigate outage risks.

Impact: Reduces dependency on single points of failure and improves operational continuity during peak AI usage.
Implement a "software factory" workflow by standardizing agentic PR creation and review processes. Introduce automated guardrails to manage review bottlenecks.

Impact: Scales code generation while maintaining quality and security standards through systematic oversight.
Dedicate specific resources and time to the final 10% of project delivery. Establish clear handoff protocols between builders and finishers.

Impact: Increases project completion rates and ensures that AI-generated code is hardened for production use.
Develop custom internal tools to bridge the gap between human collaboration platforms and AI workflows. Focus on capturing feedback loops for agent interactions.

Impact: Improves the efficiency of hybrid human-agent teams and reduces friction in collaborative development processes.

Quotes

“token maxing and the leaderboard is just the first of many questions that you need to be asking about AI adoption within your engineering org”

“GitHub is effectively one, just the general adoption of like code hosting for repositories”

“once you get the 95% of the way done, you're not even halfway there in terms of everything that you'd have to do to make it 100%”