Everything on Model Evaluation Strategy

Optimizing models solely for public benchmarks leads to "benchmaxing," where performance on tests diverges from real-world utility. This creates a false sense of progress and risks deploying models that fail in production.

Impact: Companies risk eroding user trust and wasting compute resources by chasing metrics that do not correlate with customer satisfaction or operational efficiency.

— from OpenAI Research Lead Reveals Shift to Real-World AI Evals · OpenAI Podcast· Jun 16, 2026