Anthropic rewrites Messages API in Rust to cut CPU costs
Anthropic replaced its Python-based Messages API with a Rust version to address critical CPU bottlenecks. The migration reduced CPU usage by 40 to 50 percent and improved time-to-first-token latency by 10 to 20 percent.
Anthropic has fully rolled out its Messages API written in Rust, replacing a previous Python monolith that operated within a single Google Cloud Platform region. The rewrite was driven by CPU constraints, as the company faces a zero-sum allocation of CPU resources across product services, sandboxing, and continuous integration runs. The engineering team, which initially lacked Rust developers, used Claude to write the code while human engineers focused on system design and output verification. They constructed a fork of the entire API in Rust and executed shadow traffic to confirm identical outputs before switching over. The migration resulted in an order-of-magnitude savings on the CPU side, specifically a 40 to 50 percent reduction in CPU usage and a 10 to 20 percent improvement in time-to-first-token latency. The company noted that CPU is now a significant bottleneck comparable to GPU accelerators, requiring advance capacity planning rather than on-demand scaling. The engineering team views the Rust rewrite as a demonstration of the agentic coding era's capabilities, where AI handles implementation while humans focus on system design and verification.