Google AI Strategy: Frontier Models and Product Management
Tulsi from Google DeepMind discusses the unpredictable nature of AI model development, the resurgence of product management in the AI era, and Google's full-stack strategy. Learn actionable frameworks for building AI products, including the 60-20-20 rule and the importance of model vibes over rigid roadmaps.
The Unpredictable Nature of Frontier AI
Tulsi, Head of Product for Google's Frontier Models, challenges the traditional view of product management in AI. Unlike deterministic software, AI models are unpredictable; while researchers set specific goals, the resulting capabilities often surprise even the developers. The PM's role shifts from strict requirement enforcement to identifying where the model "shines" and shaping those emergent behaviors into user value. This requires a deep understanding of "model vibes"—the intuitive feel of a model's persona and behavior—which is now a critical hiring criterion for product teams.
Google's Full-Stack Strategic Advantage
Contrary to perceptions that Google lags behind OpenAI and Anthropic, the company leverages a full-stack advantage spanning hardware, modeling, and end-user applications. With 14 products serving over a billion users, Google aims to build a Pareto curve of models that serve diverse use cases, from search to cyber defense. The strategy is not just about winning the frontier conversation but ensuring models are accessible and effective across a wide spectrum of user needs and cost structures.
Actionable Frameworks for AI Product Teams
The discussion introduces the 60-20-20 rule, a framework for balancing current functionality with future potential. Teams should build 60% of features that work today, 20% that are near-term improvements, and 20% that are ambitious bets on future model capabilities. This approach prevents products from becoming obsolete as models improve. Additionally, the role of the PM is evolving to include direct data analysis and prototyping, moving beyond writing requirements to actively validating hypotheses through small-scale evaluations and hands-on model interaction.
Safety and Operational Resilience
AI safety is embedded in the development process through continuous monitoring of model alignment and chain-of-thought. Google emphasizes using AI to harden systems, such as partnering with cyber defenders to patch vulnerabilities. For teams, this means building robust monitoring structures that allow for intervention and improvement, ensuring that as models grow in capability, they remain aligned with intended actions and safe for deployment.
Key insights
-
AI model development is inherently unpredictable, requiring PMs to focus on identifying emergent capabilities rather than strictly enforcing initial goals. The 'magic' of the model often lies in unexpected behaviors that can be shaped into unique user value.
Impact: Shifts PM focus from rigid planning to adaptive discovery, leading to more innovative and differentiated AI products.
-
The 60-20-20 rule provides a framework for building products that balance current functionality with future model capabilities. This ensures products remain relevant and grow in value as underlying models improve.
Impact: Reduces the risk of building obsolete features and aligns product roadmaps with the rapid pace of AI advancement.
-
Hiring for 'model vibes' and daily hands-on interaction with AI tools is becoming more important than traditional technical skills. PMs need intuition for model behavior to effectively guide development.
Impact: Improves the quality of product decisions by ensuring PMs have a deep, intuitive understanding of AI capabilities and limitations.
-
Small-scale evaluations, starting with just five to twenty prompts, are more effective for initial signal detection than large, complex benchmarks. This approach builds confidence and clarity before scaling up.
Impact: Accelerates the feedback loop for AI products, allowing teams to iterate quickly and identify issues early in the development process.
-
AI safety is best managed through continuous monitoring of model alignment and chain-of-thought, rather than just pre-deployment testing. This allows for real-time intervention and improvement of model behavior.
Impact: Enhances trust in AI systems by ensuring they remain aligned with intended actions, reducing the risk of harmful or misaligned outputs.
Action items
-
Implement the 60-20-20 rule in your product roadmap, allocating 60% of effort to current capabilities, 20% to near-term improvements, and 20% to future bets.
Impact: Ensures your product remains competitive and relevant as AI models evolve, balancing immediate value with long-term growth.
-
Start your evaluation process with a small set of five to twenty prompts to identify signal and build confidence before scaling to larger benchmarks.
Impact: Reduces the complexity and cost of initial testing, allowing for faster iteration and clearer insights into model performance.
-
Hire product managers who actively use and stress-test AI models daily, prioritizing intuition for model behavior over traditional technical skills.
Impact: Brings a deeper, more intuitive understanding of AI capabilities to your product team, leading to better-informed decisions and more innovative features.
-
Design distinct user experiences for different user segments, such as accessible interfaces for mass-market users and power-user tools for developers.
Impact: Maximizes engagement and satisfaction across your user base by tailoring the UX to the specific needs and capabilities of each segment.
-
Use AI to aggregate data and generate reports, but reserve human effort for providing strategic insight and creative direction in documentation and communication.
Impact: Avoids the pitfalls of AI-generated slop, ensuring that your content is insightful, valuable, and reflective of your team's unique perspective.
Quotes
“The model isn't predictable. It's kind of like Snape's potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic.”
“I actually think we do want to win the frontier conversation. I don't think we should say that Google is not participating in the frontier.”
“I think you want to actually build features that aren't necessarily always working today. It feels like every AI assistant launching these days is taking all this juicy Google data.”