New benchmark shows AI agents struggle with intermediate decision-making

Researchers from Microsoft and the City University of Hong Kong developed TasteBench, a benchmark designed to measure the quality of intermediate decision-making in AI agents rather than just final outcomes. The test uses thousands of recorded agent runs from programming and research tasks, pausing at decision points to see if the model can choose the better of two paths. The best of 14 tested models, GPT 5.6, was correct in only just under 60% of cases, with GPT 5.5 following closely. When the better path was only apparent late in the process, accuracy dropped to 21%, near chance level. Increased reasoning effort did not improve accuracy; models spent the most time on the hardest questions but were most often wrong. However, training a smaller open model to provide advice at decision points allowed a coding agent to solve one-third of software tasks, up from 15% previously. The researchers argue that current benchmarks fail to capture the 'taste' or quality of an agent's intermediate steps, which is crucial for long-running tasks.