AI Engineering Bottlenecks and the Dark Factory
LaunchDarkly CTO Cameron Ettezzati explains how AI shifts engineering bottlenecks from coding to review and testing. Learn how to implement probabilistic guardrails, optimize agent fleets, and restructure teams for the new deterministic-probabilistic hybrid workflow.
The New Engineering Bottleneck
The rapid adoption of AI coding assistants has fundamentally altered the software delivery lifecycle. While code generation has accelerated, the bottleneck has shifted downstream to code review, testing, and deployment. Organizations are facing a crisis of volume: pull requests are stacking up faster than humans can review them, and testing costs are becoming prohibitive. This shift demands a re-evaluation of engineering infrastructure to handle the new pace and uncertainty.
From Deterministic to Probabilistic
Traditional software engineering relies on determinism: the same input yields the same output. AI introduces probabilistic behavior, where identical prompts can produce varying results. This breaks existing guardrails and testing frameworks. To manage this, companies must implement runtime control mechanisms that can steer agent behavior in real-time. LaunchDarkly’s new agent control framework exemplifies this approach, allowing teams to optimize agent paths and adjust models or prompts at runtime to ensure desired outcomes.
The Dark Factory Model
The future of software delivery is the "dark factory," an automated assembly line where AI agents generate code, wrap it in feature flags, instrument it for experimentation, and monitor it in production. This model decouples deployment from release, enabling safe, rapid iteration. By treating software development as a mass-production process, organizations can achieve higher reliability and faster time-to-market. The goal is to make software delivery as invisible and reliable as a physical assembly line.
Strategic Implications for Leadership
Engineering leaders must adapt their hiring and management strategies. The traditional "code monkey" role is obsolete; engineers are now frontline managers directing agent fleets. Success requires strategic thinking, product management skills, and the ability to define clear goals for AI systems. Additionally, the build-versus-buy calculus is shifting. AI makes it cheaper to build simple features in-house, but vendors with economies of scale and high reliability remain essential for critical infrastructure. Leaders must focus on core business outcomes and leverage AI to amplify creativity and efficiency, rather than just speed.
Key insights
-
AI accelerates code generation, shifting the primary bottleneck from writing code to reviewing and testing it. This creates a volume problem that human teams cannot sustainably manage.
Impact: Organizations must invest in automated review and testing tools to prevent delivery delays and maintain competitive speed.
-
Probabilistic AI outputs break deterministic testing and guardrails. Traditional quality assurance methods are insufficient for managing non-deterministic agent behavior in production.
Impact: Companies need runtime control frameworks to steer AI agents and ensure consistent, reliable outcomes despite inherent randomness.
-
The role of the software engineer is evolving from a code writer to a frontline manager of agent fleets. Engineers must possess strategic, product, and design skills to direct AI effectively.
Impact: Hiring criteria must shift toward strategic problem-solving and goal-setting abilities, reducing reliance on pure coding proficiency.
-
Model Context Protocol (MCP) servers are becoming the new user experience for internal tool integration, replacing traditional UIs and dashboards.
Impact: Centralizing MCP servers improves data flow and reduces context switching, enabling more efficient collaboration between teams and AI agents.
-
The build-versus-buy decision is being recalibrated by AI. Simple features are cheaper to build in-house, but vendors with economies of scale remain critical for high-reliability infrastructure.
Impact: CTOs must carefully evaluate vendor value, focusing on scalability and reliability rather than just feature sets, to optimize costs and performance.
Action items
-
Implement runtime control frameworks for AI agents to manage non-deterministic behavior. Use tools that allow real-time adjustment of prompts and models to ensure desired outcomes.
Impact: Reduces the risk of AI-generated errors in production and improves the reliability of agentic workflows.
-
Adopt a "dark factory" model for software delivery. Automate the pipeline from code generation to feature flagging, experimentation, and production monitoring.
Impact: Accelerates time-to-market and improves software reliability by treating development as a standardized, automated process.
-
Restructure engineering teams to focus on strategic management of AI agents. Hire for product management and design skills, and train engineers to direct agent fleets.
Impact: Enhances team productivity and ensures that AI tools are used effectively to achieve business goals rather than just generating code.
-
Centralize Model Context Protocol (MCP) servers to streamline internal tool integration. Use MCP as the primary interface for AI agents to access data and tools.
Impact: Improves data consistency and reduces the cognitive load on developers, leading to more efficient and error-free workflows.
-
Re-evaluate the build-versus-buy strategy for software tools. Build simple features in-house using AI, but retain vendors for infrastructure requiring high reliability and economies of scale.
Impact: Optimizes costs and ensures that critical systems are supported by robust, scalable infrastructure, reducing operational risk.
Quotes
“The PR volume is more than humans can get through. The testing volume, the amount of tokens I spend on testing versus actually writing, it's borderline obscene.”
“I fundamentally think engineering has changed, not as we move the bottleneck, but the folks that took Doritos and Mountain Dew and turned spec into execution, which was me in my early days, no longer exist.”
“We're no longer hiring code monkeys. We're hiring frontline managers at every level. It's just you're managing agents.”