Retrofitting AI Into High-Performance Products
Superhuman CTO Loic Goussier discusses the architectural and organizational challenges of integrating LLMs into a product defined by sub-100ms latency. Key strategies include a 'smart-to-cheap' model optimization pipeline, dimension-based evaluation frameworks, and maintaining human-in-the-loop controls to mitigate agent risk.
The Latency-AI Tension
Superhuman’s brand is built on sub-100-millisecond interactions, a constraint that fundamentally conflicts with the inherent latency of Large Language Model (LLM) inference. CTO Loic Goussier explains that retrofitting AI into such a high-performance product requires architectural adaptation rather than simple feature addition. The core challenge is maintaining the user’s perception of speed while leveraging probabilistic AI models. This tension forces a rethinking of where computation occurs, shifting from purely server-side processing to hybrid on-device and pre-cached architectures.
Architectural Strategies for Speed
To reconcile AI with speed, Superhuman employs aggressive pre-caching. For voice features, the system pre-loads context and model states before the user speaks, reducing the real-time payload to just the transcription. This allows the LLM to process intent and tone instantly. Additionally, the team uses a "smart-to-cheap" optimization pipeline. Features are prototyped with expensive frontier models to ensure high-quality outputs, then iteratively optimized by fine-tuning smaller, open-source models for specific use cases. This approach balances development speed with production cost efficiency, enabling dedicated infrastructure for high-volume, low-complexity tasks like email classification.
Organizational Structure and Risk Management
Organizationally, Superhuman operates on "aligned autonomy." Engineering pods own their specific AI experiences, supported by a central platform team that manages model routing and evaluation infrastructure. A key risk management strategy is the strict limitation of agent agency. The system drafts emails but does not send them, preventing irreversible errors. The team defines "looking foolish" as a P0 bug, prioritizing user trust over automation convenience. This cautious approach acknowledges that while AI can handle routine tasks, the stakes of email communication require human oversight.
Leadership in a Rapidly Changing Landscape
Goussier emphasizes that leadership in the AI era relies on pattern detection and people management rather than tool mastery. The pace of change creates cognitive load for engineers, necessitating internal guilds for knowledge sharing and external engagement with startups to maintain market awareness. By focusing on core management skills and empowering teams to self-correct, leaders can navigate the uncertainty of AI integration without falling behind. The ultimate goal is to build systems that are not only fast and efficient but also trustworthy, ensuring that AI enhances rather than undermines the user experience.
Key insights
-
Integrating LLMs into low-latency products requires decoupling context preparation from real-time inference. Pre-caching model states and context allows for sub-second response times despite the inherent slowness of LLM processing.
Impact: Enables AI features in performance-critical applications without compromising the core user experience or brand promise of speed.
-
A "smart-to-cheap" development pipeline optimizes both quality and cost. Starting with frontier models ensures feature viability, while subsequent fine-tuning of smaller models reduces operational expenses for high-volume tasks.
Impact: Significantly lowers the total cost of ownership for AI features, making them scalable for large user bases without sacrificing initial quality benchmarks.
-
Evaluation frameworks must focus on specific dimensions (e.g., date parsing, tone) rather than broad example sets. This approach allows teams to systematically cover the vast space of possible AI behaviors and identify edge cases.
Impact: Improves the reliability of AI outputs by ensuring that critical functional aspects are rigorously tested, reducing the likelihood of user-facing errors.
-
Limiting AI agency to drafting rather than executing actions (like sending emails) is a critical risk mitigation strategy. This human-in-the-loop approach prevents irreversible mistakes and maintains user trust.
Impact: Protects the brand from reputational damage caused by AI errors, ensuring that users retain control over high-stakes communications.
-
Effective AI leadership relies on pattern detection and people management rather than mastering every new tool. Leaders should focus on empowering teams to self-correct and adapt to rapid technological changes.
Impact: Reduces cognitive load on engineering teams and fosters a culture of continuous learning, enabling organizations to stay agile in a fast-moving market.
Action items
-
Implement a pre-caching strategy for AI features by separating context loading from user input processing. This allows for faster response times in latency-sensitive applications.
Impact: Improves user perception of speed and enables the integration of AI into products with strict performance requirements.
-
Adopt a "smart-to-cheap" model optimization pipeline. Prototype with frontier models to establish quality, then fine-tune smaller models for production to reduce costs.
Impact: Lowers operational costs for AI features while maintaining high-quality outputs, improving the overall ROI of AI integration.
-
Develop dimension-based evaluation frameworks for AI features. Identify specific aspects of output quality (e.g., tone, accuracy) and create targeted tests for each dimension.
Impact: Enhances the reliability of AI systems by ensuring that critical functional aspects are rigorously tested, reducing the risk of user-facing errors.
-
Restrict AI agent actions to drafting or suggesting, requiring human approval for execution. This human-in-the-loop approach mitigates the risk of irreversible errors.
Impact: Builds user trust and protects the brand from reputational damage caused by AI mistakes, ensuring that users retain control over high-stakes decisions.
-
Foster a culture of "aligned autonomy" by empowering engineering teams with ownership of their AI stack while providing shared platform infrastructure and regular knowledge-sharing sessions.
Impact: Increases team agility and innovation, enabling faster adaptation to new AI technologies and market trends.
Quotes
“The speed is coming with some heavy architectural decisions. 100 milliseconds, you cannot rely on network for everything that you do, which means that everything needs to work offline by default”
“We always go with them. one expensive model, like the best of the models, let's try if the feature can even work. But that's part of the process of the development”
“Something that makes a user look foolish is a P0 bug for us. So it's a category of QA bugs that look foolish because we archive an email and they missed it”