Atlas benchmark reveals agents fail high-stakes banking tasks

Handshake AI introduced Atlas, a finance benchmark comprising 100 tasks across 13 simulated bank environments. Models currently score sub-20% on the benchmark, rendering them 'virtually unusable' for professional bankers who follow the rule that 'if it's 95% right, it's 100% wrong.' In one example, an agent building an LBO model with over 2,600 formulas made a single error in tax treatment, specifically regarding the 338H10 election, that poisoned the entire valuation. The model did not flag its uncertainty, highlighting a critical gap in reliability.

Handshake developed a grader called Gandalf, published at NeurIPS, which achieves over 90% agreement with human graders by providing better context to verify complex Excel outputs. The company highlights that while models can handle context retrieval quickly, they fail at the nuanced financial logic and scope completeness required for trustworthy deployment. Handshake AI contends that current AI agents lack the reliability and interpretability needed for finance, where small errors in complex models can lead to significant financial losses and loss of client trust.