Controlled Agents For Enterprise Data Security
Sierra built an internal agent platform that combines whitelisted tools, citation based output, and a knowledge graph. The system supports operations, support, and product workflows while limiting data leakage risk. The case study offers a practical framework for scaling AI agents in regulated environments.
The Controlled Agent Opportunity
Sierra's internal agent platform shows that enterprise AI adoption is less about raw model capability and more about containment, provenance, and ownership. The company built a shared agent harness around whitelisted tools, structured citations, and a knowledge graph, allowing hundreds of users to run roughly thirty agents for operations, support, and product work.
Containment As A Product Feature
The core design choice is to avoid open-ended agent access. Agents receive a bounded toolset, deterministic triggers, and an escape hatch for missing capabilities. This approach limits blast radius while still allowing useful autonomy. For data security and regulated environments, that tradeoff is commercially important because it makes agent deployment defensible.
Citations Reduce Trust Friction
The platform forces claims to carry source references. Unverified reasoning is marked as opinion. A second model can then verify each claim against clean context. This pattern lowers hallucination risk and gives users a practical audit trail. It also creates a feedback loop that improves future responses.
Knowledge Graphs Improve Context Efficiency
Instead of dumping large corpora into prompts, Sierra indexes entities, episodes, and relations. Agents use exact search, semantic search, and relation walking to find relevant data. This reduces noise, improves consistency, and lowers token spend. The main cost is upfront data cleaning, which is a one-time infrastructure investment.
Adoption Through Ownership
The company used a platform engineering approach. Engineers create named agents, customize prompts, and inherit shared guardrails. This lowers friction and makes the system feel personal. A small AI captains group helped spread the practice, while shared tools and scheduled self-review loops reduced maintenance burden.
Strategic Takeaway
The lesson is that reliable enterprise agents require deterministic scaffolding, not just better models. Companies that invest in provenance, controlled tooling, and internal ownership can scale agents into high-value workflows while keeping risk manageable.
Key insights
-
Sierra treats agent containment as a core product requirement rather than an afterthought. The platform gives agents a bounded toolset, deterministic triggers, and an escape hatch for missing capabilities. This design makes failure modes easier to predict and audit.
Impact: Enterprises can deploy agents in sensitive environments while reducing blast radius. It also supports compliance by making agent actions easier to review.
-
Citation based output turns model confidence into a verifiable signal. Each claim carries a source reference, and unverified reasoning is marked as opinion. A second model can then check claims against clean context.
Impact: Teams can reduce hallucination risk and lower manual review effort. It also creates a feedback loop that improves future agent responses.
-
A knowledge graph gives agents a structured path through organizational data. Instead of scanning noisy corpora, agents use exact search, semantic search, and relation walking. This improves consistency and reduces token spend.
Impact: Companies can lower inference costs while improving answer reliability. It also makes context engineering a reusable platform capability.
-
Agent ownership drives adoption. Engineers create named agents, customize prompts, and inherit shared guardrails. A small champion group and low friction templates spread the practice across the organization.
Impact: Businesses can scale internal AI use without central bottlenecks. It also encourages local innovation while keeping safety controls in place.
Action items
-
Audit existing agent tool access and remove broad shell or file system permissions. Replace them with task specific tools and an explicit escape hatch for missing capabilities. This makes the agent boundary easier to reason about.
Impact: It reduces the chance of catastrophic actions and supports security review. Teams can deploy agents with clearer accountability.
-
Require structured output where each claim includes a source reference. Mark unsupported reasoning as opinion and route high risk claims to a verification model. This creates an audit trail for every response.
Impact: It improves trust and reduces hallucination related rework. It also gives product teams a measurable reliability signal.
-
Build a knowledge graph for high value operational data. Index entities, episodes, and relations, then provide agents with exact search, semantic search, and relation walking tools. Clean the noisiest sources first.
Impact: It improves retrieval consistency and lowers token spend. It also makes context a reusable company asset.
-
Create a paved road agent platform with shared guardrails, templates, and named agent ownership. Let engineers customize prompts and tools while keeping core safety controls centralized. Use a small champion group to drive adoption.
Impact: It lowers the barrier to building agents and increases internal usage. It also preserves safety while allowing local experimentation.
-
For confidential data, run privileged agents inside a controlled perimeter and allow only validated vocabulary to leave. Use a second agent to translate safe outputs into human readable explanations. Add a human reviewed escape path for unexpected cases.
Impact: It enables troubleshooting without leaking sensitive data. It also expands agent use into regulated and customer committed environments.
Quotes
“LLMs in an agent harness, if they have an open check to do everything, then it's inevitable that they will eventually do something terrible.”
“I wanted to make it more obvious to me when that's happening.”
“I would recommend trying to do it with a platform engineering approach.”