Datadog Lessons For Scaling Agentic Coding
The Datadog AI developer experience program shows how agentic coding scales from tool adoption to governed workflow. The company used evals, context hygiene, and team ownership to reduce risk in code review and model selection. The result is a practical framework for engineering leaders who need measurable, cost-aware AI development.
The Datadog AI developer experience program shows that scaling agentic coding is less about tool access and more about measurement, context hygiene, and workflow redesign. The company began with a Cursor proof of concept in early 2025, quickly moved from a few hundred users to about 1,000 daily users, and later added a terminal-based agent tool for large monorepos. That adoption wave forced Datadog to create a dedicated AI DevX function, later split into a Signal team for metrics, cost, and evals, and a Flow team for developer workflow and SDLC bottlenecks.
Evals as the decision layer
Datadog built a custom eval platform in Go and connected it to its metrics stack. The first high-value use case was code review: replaying pull requests that had caused incidents and checking whether an agent could flag the likely failure. This approach works because the ground truth is known. The team now runs nightly evals and ad hoc runs when steering documents change. It also uses evals to compare frontier models and open-weight models, finding that open models would need roughly 50 percent better performance to be fully viable for the current bar.
Context is a liability when it rots
A key counterintuitive finding came from a front-end monorepo. A large context file written for older models and older tooling was removed, and eval performance improved. The lesson is that context is scarce. Old instructions, such as basic package-manager guidance, can consume attention and push agents toward outdated behavior. Datadog is shifting from broad upfront context to narrower, team-owned marketplaces, clearer error messages, and next-step guidance only when needed.
Cost, ownership, and hiring
The organization is not using token quotas as a blunt control. Instead, it is tracking token spend, compressing output, and assigning ownership of skills and evals to platform teams. It is also exploring meta-harnesses that route work to agents with the right preinstalled skills. Hiring changed as well. Live coding interviews are being replaced with AI-assisted tasks on large codebases, where candidates must understand context, review a diff, and explain trade-offs. Career expectations now emphasize judgment, POC-driven validation, and the ability to own a full work stream with AI support.
The strategic takeaway is that AI development tooling becomes a business capability only when it is governed by data. Evals turn adoption into a measurable system, context hygiene prevents silent degradation, and workflow redesign targets the real bottleneck in code review. For engineering leaders, the priority is not simply buying more AI tools. It is building the measurement, ownership, and feedback loops that make agent output safe, cost-aware, and useful at scale.
Key insights
-
Agent adoption at scale requires a measurement layer, not just tool access. Datadog built evals to replay incident-causing pull requests and validate whether agents could catch likely failures.
Impact: Engineering leaders can reduce production risk by anchoring agent use cases to known failure modes. This creates a defensible baseline for model and tool selection.
-
Context files can degrade agent performance when they become stale or overly broad. Datadog found that deleting a large context file improved eval results in a front-end monorepo.
Impact: Teams should treat context as a scarce resource and audit it with evals. This prevents silent quality loss as models and repositories evolve.
-
The SDLC bottleneck shifted to code review as AI increased pull request volume. Datadog is using agents for linting, formatting, and incident-prevention review where correctness is measurable.
Impact: Organizations can unlock AI productivity by targeting high-frequency, low-ambiguity review tasks first. This reduces rework and shortens merge cycles.
-
Open-weight models are not yet fully viable for Datadog current bar, but they are useful for cheaper regression checks. The company estimates they would need roughly 50 percent better performance to match frontier models.
Impact: Businesses can lower AI spend by routing simple tasks to cheaper models while keeping frontier models for judgment-heavy work. This improves cost efficiency without sacrificing quality.
-
Hiring and career expectations are changing around AI-assisted codebase understanding. Candidates are assessed on judgment, trade-offs, and the ability to use AI to navigate large codebases.
Impact: Leaders can build stronger engineering teams by testing real-world agent fluency rather than isolated coding speed. This aligns hiring with the skills that drive AI-era delivery.
Action items
-
Create an eval suite from historical incidents and repeatable pull requests. Use an agent judge to check whether the review output identifies the known failure.
Impact: This gives engineering teams a measurable guardrail for AI code review. It reduces the risk of shipping agent-generated or agent-reviewed changes.
-
Audit all steering documents and context files for stale instructions. Remove content that no longer matches current models, tooling, or repository behavior.
Impact: Context hygiene can improve agent performance and reduce token waste. It also makes agent behavior easier to explain and debug.
-
Split AI development ownership into signal and flow teams. Assign one team to metrics, cost, and evals, and another to workflow redesign and developer experience.
Impact: This prevents governance work from crowding out workflow innovation. It creates clear accountability for both measurement and adoption.
-
Route simple regression checks to low-cost open models in CI. Reserve frontier models for tasks that require judgment, creativity, or complex code understanding.
Impact: This lowers AI spend while preserving quality where it matters. It also creates a practical path for open-model adoption as performance improves.
-
Update hiring and promotion criteria to test AI-assisted codebase review, trade-off analysis, and POC-driven validation. Move away from low-signal live coding exercises.
Impact: This aligns talent decisions with the skills that drive AI-era engineering output. It helps teams hire and promote people who can lead with agents, not just write code.
Quotes
“what if we delete the whole context from the repo? Like what would happen?”
“the evals started to perform like much better, which was counterintuitive”
“we wanted to put AI in the loop for code review to really like prevent an incident or like kind of major bug”