LangChain argues model routing belongs in harness, not gateway
LangChain founder Harrison Chase contends that intelligent model routing must occur at the harness layer to optimize vertical agents for cost and latency. The company also disclosed that it spends roughly 30% of its engineering salary budget on coding tools, a figure it manages through strict visibility and budget controls.
Harrison Chase, founder of LangChain, argues that intelligent model routing is most effective when implemented at the harness layer rather than the gateway level. Chase maintains that harness-level routing provides superior awareness of system state, prompt cache status, and the specific prompting strategies required by different models. While he acknowledges that gateway-level routers like OpenRouter are useful for general-purpose chat or coding tasks, he believes domain-specific vertical agents require harness-aware routing to optimize for quality, latency, and cost. Chase emphasized that routing is not a commodity and remains difficult to build due to rapid model changes, noting that the value of the harness lies in data portability, which allows users to switch between model providers without being locked into specific context or memory structures.
LangChain’s product suite includes an open-source framework for connecting models to tools and data, along with LangSmith for observability and evaluations. The company recently launched Engine, an agent within LangSmith that analyzes traces to update harness code, and SmithDB, a proprietary database for storing agent traces. Chase stated, "I think right now, I pretty strongly believe that routing, you have to be harness aware."
The company also detailed its internal approach to managing AI costs, revealing that it spends approximately 30% of its engineering salary budget on coding tools. This figure has decreased over the last two months despite an increase in total token usage. LangChain manages this spend through a three-step process: tracking all coding agent usage in LangSmith for visibility, setting budgets that require VP approval to exceed, and optimizing the harness. The company developed an internal cloud-based coding agent called OpenSwee, which Chase uses exclusively via Slack. This internal harness incorporates prompt caching and routing to reduce costs. Chase noted that the team identified a specific inefficiency where a plugin was taking DOM snapshots every turn, resulting in a $40,000 weekly spend on a single task. While some high spenders are highly effective senior engineers, others incur costs due to improper setup or mistakes. Chase predicts that organizational design will evolve, potentially leading to smaller teams driving large numbers of agents, though he believes human collaboration remains a significant component of current workflows.