Slack replaces headless Chrome with Go processes for load testing
Slack abandoned expensive headless Chrome simulations in favor of lightweight Go processes for its Koi Pond load testing system. The new approach reduces costs and allows for targeted testing of specific failure modes like thundering herds.
Slack developed a load testing system called Koi Pond to prepare for a customer with 500,000 users, a deadline accelerated by the pandemic. Initial attempts used headless Chrome instances in Kubernetes pods to simulate users, but this proved too resource-intensive. A single five-hour test run cost $150,000 due to EC2 instance costs, with each headless Chrome instance costing 37 cents per five-hour tranche. The team abandoned this approach in favor of lightweight Go processes that establish active WebSocket connections and execute recorded API request sequences. This method allows for targeted testing of specific failure modes, such as 'thundering herds,' where 100,000 users simultaneously react to a message, causing massive WebSocket fan-out. The system includes a recorder tool that captures API and WebSocket traffic from real user flows to generate test scripts. This enables product managers to define specific use cases for automated load testing, providing a more cost-effective and precise alternative to browser-based simulation.