Benchmarks show AI agents struggle with nuanced judgment tasks

David Pollan conducted benchmarks testing AI agents on complex tasks requiring judgment, specifically balancing cost and time preferences. In a flower ordering test, Instinct, Kickr, and Season performed best by equally weighting price and same-day delivery, while other agents struggled to prioritize price. Kickr utilized a Chrome extension for browser use, providing a performance edge. In a travel planning test to Los Angeles, agents GrokBot, Dots, and Instinct interpreted 'luxurious cost-efficient' differently; GrokBot prioritized time, Dots prioritized cost, and Instinct balanced both. In a multi-agent chat room experiment, GrokBot was the only agent to methodically incorporate value from other agents' proposals into its own itinerary.

Pollan also surveyed 354 AI power users, finding that only 3% used Dots as their primary agent, despite its advantage of leveraging existing ChatGPT context. He noted that 14 of the top 15 benchmarked agents allowed users to export their memory files for switching, a common onboarding pattern. Marissa Mayer’s company Dazzle is highlighted for its approach to photo-based context, using a thesis that the agent should 'forget' data it does not need to build trust. Pollan observes that agents currently lack the nuanced judgment to handle implicit social preferences, such as not appearing cheap when buying gifts.