Roblox pilots autonomous production deployments using AI risk assessment

Roblox is piloting a system that allows certain pull requests to merge into production autonomously without human code review. The system relies on a programmatic check to ensure the AI code review agent found no issues, verification that the repository has opted in to autonomous deployment, and a final LLM-based risk assessment. Currently, the system is conservative, primarily allowing autonomous deployment for low-risk changes such as documentation and test updates. This approach is similar to Meta's "diff risk score" system. To build trust in this autonomy, Roblox invested in traditional engineering guardrails such as unit test coverage, integration testing, canaries, and rollback capabilities. The company also conducted evaluations and injected faults into the system to demonstrate safety to engineers, compliance teams, and leadership.

Roblox has deployed an AI code review agent that achieves a 70% acceptance rate for its feedback, compared to the 50-55% acceptance rate observed for human-written comments. To improve accuracy, the company extracts engineering rules, termed "exemplars," from historical code review comments and stores them as YAML files in repositories. Andrew Swerdlow argues that autonomy must be earned through trust and safety. Roblox is also redefining software engineering metrics, using LLMs to tag every pull request to calculate a normalized feature velocity rate per engineer. Swerdlow identifies feature velocity as the closest proxy to innovation. "Stop counting tokens. That metric doesn't matter. Start counting what code gets accepted, what got rolled back and what shipped with no human intervention," Swerdlow said.