Sep 09, 2026 · a16z Podcast · 6 min read

Independent AI Evaluation Drives Enterprise ROI

VALS founder Rayem Krishnan discusses the critical need for independent AI benchmarks to validate model capabilities. The episode explores how public metrics often mislead, the rise of enterprise-focused evaluation tools like ValSmith, and the strategic shift toward measuring real-world ROI in AI adoption.

Technology Business Strategy Entrepreneurship

The takeaways

Public Benchmarks Mask True Model Performance. Self-reported metrics on open-source benchmarks often overstate capabilities, creating a disconnect between advertised performance and real-world utility. Independent, held-out private benchmarks are essential to verify actual model proficiency before enterprise deployment.
Token Spend May Eclipse Salary Costs. AI usage costs are rapidly becoming a primary operational expense, potentially exceeding employee salaries in high-adoption firms. Companies must rigorously justify AI ROI to avoid unsustainable token burn rates and inefficient resource allocation.
Independent Evaluation Mirrors Financial Auditing. The AI industry requires third-party evaluators to function as audit firms, providing neutral verification of model capabilities. This separation of duties prevents conflicts of interest inherent in self-reported lab metrics and builds market trust.
Benchmarks Must Evolve With Model Capabilities. Static evaluation metrics become obsolete as models improve, requiring continuous updates to reflect current world knowledge and complex agentic tasks. Deprecating saturated benchmarks ensures that evaluations remain relevant and predictive of future performance.
Enterprise Needs Custom Coding Benchmarks. Generic public benchmarks fail to predict performance on specific codebases, leading to suboptimal model selection. Tools like ValSmith allow enterprises to build internal benchmarks from their own GitHub repositories to identify the most cost-effective AI agents.
Evals Become Shared Policy Language. Standardized evaluations provide the empirical data needed for effective AI regulation and geopolitical verification. A shared language for model capabilities enables governments to set enforceable rules without stifling innovation or lagging behind technological progress.

The Crisis of Self-Reported AI Metrics

The rapid expansion of the AI industry has exposed a critical gap in market infrastructure: the lack of independent, high-fidelity evaluation standards. While model labs report impressive results on public, open-source benchmarks, these metrics are increasingly susceptible to overfitting and do not reflect real-world performance. VALS, a third-party evaluation firm, highlights a stark disconnect where models like Meta’s Llama 4 showed strong public scores but underperformed on private, held-out benchmarks. This discrepancy underscores the necessity for an independent testing ecosystem, analogous to financial audit firms, to validate model capabilities and ensure that investment decisions are based on verified data rather than self-reported claims.

Enterprise ROI and the Token Economy

For enterprises, the challenge has shifted from mere adoption to efficient utilization. AI token spend is emerging as a significant line item, potentially eclipsing traditional salary costs in some organizations. However, current adoption strategies are often arbitrary, with companies setting flat usage limits without understanding the true return on investment. This misvaluation of intelligence leads to inefficient workflows, such as engineers idling during rate-limit resets. To address this, enterprises are moving toward custom evaluation frameworks that measure performance against specific internal tasks, such as coding within proprietary codebases. Tools like ValSmith enable companies to build internal benchmarks, allowing them to select the most cost-effective models and agents for their unique operational needs, thereby optimizing the Pareto frontier of cost versus capability.

Strategic Implications for Policy and Market Structure

The evolution of AI evaluation extends beyond commercial applications into the realm of policy and geopolitical strategy. As models gain agentic capabilities, the need for standardized metrics to assess risk and alignment becomes paramount. Independent evaluators provide the empirical grounding necessary for policymakers to craft effective regulations, moving away from abstract concerns to data-driven standards. Furthermore, in a global context, shared evaluation languages are essential for verifying sovereign AI capabilities and managing risks related to recursive self-improvement. The industry is transitioning from a phase of rapid, unverified growth to one of structured accountability, where independent evaluation serves as the bridge between technological innovation and responsible deployment. This shift ensures that the market can rationalize the value of AI, leading to more sustainable and efficient integration across industries.

Key insights

Public benchmarks are often gamed or saturated, leading to a significant divergence between advertised model capabilities and actual performance on private, held-out tests. This creates a market inefficiency where buyers cannot rely on standard leaderboards to make purchasing decisions.

Market Dynamics →

Impact: Enterprises face higher risks of deploying underperforming models, necessitating a shift toward independent verification to protect investment returns.
AI token expenditure is becoming a major operational cost, with some companies spending multiples of employee salaries on AI usage. Without precise evaluation of which models deliver the best ROI for specific tasks, companies risk unsustainable burn rates.

Operational Efficiency →

Impact: Firms must adopt granular, task-specific evaluation methods to optimize AI spend and ensure that token costs align with tangible productivity gains.
The role of independent evaluators is analogous to that of audit firms in finance, providing neutral verification that prevents conflicts of interest inherent in self-reported lab metrics. This separation is crucial for building a rational buying market for AI capabilities.

Industry Structure →

Impact: The emergence of a robust third-party evaluation sector will standardize model quality, reducing information asymmetry and fostering trust among investors and enterprises.
Static benchmarks become obsolete as models improve, requiring continuous updates to reflect current world knowledge and complex agentic behaviors. Deprecating saturated metrics ensures that evaluations remain predictive of future model performance.

Methodology →

Impact: Evaluation providers must adopt a dynamic approach to benchmarking, ensuring that their metrics evolve in lockstep with model capabilities to maintain relevance and accuracy.
Standardized evaluations are becoming a critical tool for policy and geopolitical verification, providing the empirical data needed to assess risks and capabilities across borders. A shared language for AI metrics facilitates international cooperation and regulatory alignment.

Policy & Regulation →

Impact: Governments and international bodies can leverage independent evaluations to craft effective regulations and verify sovereign AI capabilities, enhancing global security and stability.

Action items

Implement independent, private benchmarks to validate model performance before enterprise deployment, rather than relying solely on public leaderboards. This involves creating held-out test sets that reflect specific business use cases.

Impact: Reduces the risk of deploying underperforming models and ensures that AI investments are based on verified capabilities, leading to better ROI.
Develop internal evaluation frameworks using tools like ValSmith to measure AI performance on proprietary codebases and specific tasks. This allows for the identification of the most cost-effective models and agents for unique operational needs.

Impact: Optimizes AI spend by aligning model selection with actual task requirements, avoiding the pitfalls of generic, one-size-fits-all model choices.
Monitor and analyze AI token usage patterns to identify inefficiencies and optimize resource allocation. This includes tracking which models and agents deliver the highest productivity per dollar spent.

Impact: Prevents unsustainable token burn rates and ensures that AI costs are justified by tangible productivity gains, supporting long-term financial sustainability.
Engage with independent evaluation providers to stay updated on emerging risks and capabilities, particularly in areas like cybersecurity and biosecurity. This helps in anticipating regulatory changes and aligning internal policies with best practices.

Impact: Enhances risk management and compliance, ensuring that the organization is prepared for evolving regulatory landscapes and potential security threats.
Advocate for and participate in the development of standardized evaluation metrics within the industry. This contributes to a shared language for AI capabilities and risks, facilitating better communication with policymakers and partners.

Impact: Supports the creation of a more transparent and accountable AI market, which can lead to more effective regulations and greater trust among stakeholders.

Quotes

“When Meta released Lama 4 on our held-out private benchmarks, the model was actually underperforming. But on all of the major public benchmarks, it was showing incredible capabilities.”

“I think that you see that there is a misvaluing of intelligence happening at every layer of the stack.”

“The government kind of has an inclination of what it's afraid of, be it biohacking or cyber hacking. But then there becomes the question of, can the model do it?”