Vials benchmark shows frontier models fail basic lab tasks

Handshake AI released Vials, a life sciences benchmark that tests AI agents on interpreting lab assays, including molecular diagrams, plasmid maps, Western blots, and histology images. Frontier models perform poorly on the benchmark, with most scoring under 50% and some sub-20%. For example, models struggled to identify hydrogen bonds in a GLP-1 structure or correctly interpret a Western blot lane, tasks where a human expert would take under a minute. Handshake notes that even with tool use, such as Python and OpenCV, accuracy remains under 60%.

The company argues that models lack the 'human taste' and visual consistency required for reliable scientific work, with errors in these tasks having significant real-world implications for drug discovery and diagnostics. Handshake is generating thousands of training data points to help models climb this benchmark, using post-training on open-source models to identify where learning stalls. The company contends that AI cannot cure cancer or facilitate drug discovery until it can reliably interpret basic scientific images, and that current models are not yet safe or accurate enough for lab-in-the-loop workflows.