# Enterprise AI Control Plane and Harness Strategy

**Podcast:** AI Engineering Podcast
**Published:** 2026-09-19

## Transcript

Hello, and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems.
Your host is Tobias Macy, and today I'm interviewing Nakunj Bajaj about the challenges of keeping your agents reliable when the underlying models aren't and how your harness can help.
So, Nakunj, can you start by introducing yourself?
Thank you so much for having me here, Tobias.
I am Nakunj, co-founder and CEO at True Foundry.
My background is in machine learning, used to lead one of the conversational AI teams at Meta.
And prior to that, have worked at the intersection of building ML models and ML infra at a couple of other startups as well.
And do you remember Hapers got started working in the ML and AI space?
Sure, I do.
My absolute first exposure to ML and AI happened during undergrad.
And this was not through ML and AI, but the underlying concepts in ML and AI that was through linear algebra.
okay in my freshman year during undergrad i got a chance to implement some of these matrix decomposition algorithms on top of fbgas which are like free field programmable gate arrays and when you have to really break down these complex algorithms into their unit of operation to be able to run them efficiently on fbgas you understand what's happening under the hood And I think that learning really exposed me to machine learning.
And then it continued through my undergrad and the rest of my career.
In terms of what you're building now, you mentioned that you're at True Foundry, you've built the True Forge harness.
I'm wondering if you can just start by giving a bit of an overview of what is your current area of focus and how did you end up working in this space?
Absolutely.
Yeah.
So very quickly.
I will describe what is True Foundry and what is True Forge and I will talk a little bit about how I personally got interested in this entire area.
So True Foundry is the control plane for enterprise AI.
This is one pipe through which all the enterprise token traffic is supposed to flow.
And when I say token traffic, I mean traffic including calls to models, calls to MCPs and calls to agents.
Once all of the organizations traffic flows through a common pipe, a common gateway, then you get observability unlocked, you get the reliability unlocked, you get cost optimization unlocked, you get security completely taken care of, and lastly you get your identity management taken care of.
So it basically solves for these five pillars, observability, reliability, cost management, security, and identity.
Okay, so that's what True Foundry is about.
It's a control plane to enterprise AI implemented through an AI gateway.
And True Forge, on the other hand, is actually an open source agent harness.
And the way we think about this is if you think about building an agent as a car, we think of the model as the engine, but everything else around the model, the engine, is essentially the agent harness, right?
So it means how do you steer it?
How do you put brakes on it?
How do you have a dashboard where you can see everything that's happening?
Those are all components of an agent harness.
And technically speaking, things that you include in the agent harness are how does the model get a sandbox to execute some code that it might be writing?
How does the model does context and memory management and the token optimization?
which tools, which MCP servers, which sub-agents that you call.
All of these concepts are handled by the agent harness, and that's what TrueForge is.
It's a neutral, open-source harness that we actually launched recently.
Just a little bit about my personal background that led to us working on this.
So when I was at Meta, I got a chance to lead some of the conversational AI teams, and what stood out to me was the internally developed ML platform at Meta.
And the core design principle that really hit home with me was this notion of federated execution that I as a developer can pick the tools, the models, the frameworks that I really care about and be able to move as fast as I would like.
So I was enabled with the right tooling to just execute.
But at the same time, it never left the site of centralized governance.
So I didn't have to worry about instrumenting my code.
Like it would always log things that I care about.
And similarly, a central platform team, if they had to figure out any erroneous or malicious activity, they always had the control to check what's happening and be able to take care of that.
So this entire dichotomy of whether there's a trade-off between federated execution and centralized governance turned out to be false by looking at Meta's internal ML platform.
And we want to take that learning and bring it to the rest of the organizations out there.
So the whole space of agent gateways and agent harnesses kind of spans at least two of the major hype trends over the past year or two, where maybe a year or so ago, AI gateways were gaining a lot of attention.
There were a new gateway probably every other day.
Some of those have been consolidated into other systems.
Some of them have...
ceased operation.
Some of them are continuing to grow.
And then the past six to eight months has seen a lot of focus on the idea of harness engineering and that being one of the key differentiators beyond the models themselves.
And I'm wondering if you can just talk through some of the ways that you're navigating some of the challenges and complexities of being in such a crowded space and what you see as some of the benefits of working across both of those and some of the ways that having a harness and a gateway that work well together can be a force multiplier.
Absolutely.
Happy to zoom in.
And by the way, I cannot agree more with the pace that everything else is changing.
In the startup ecosystem now, there's this new saying, right?
That earlier people had to find PMF.
or the product market fit once now you are fundamentally finding product market fit every six months right the space is changing that fast but also the interesting thing around this is that this is what is creating the opportunity as well so this is certainly a challenge for building but this is an opportunity for operating right in the startup space and the reason being that think about what's happening with every consumer of call it like an ai gateway or a control plane layer right an organization builds out a centralized platform engineering team that's going to develop this AI gateway because they care about solving those five problems that I talked about earlier, right?
They start with doing this, they build a thin proxy layer and they're like, okay, we are sorted.
And then soon enough, the model proxy layer that they have developed, all the API signatures start changing out there in the market.
A couple of months later, MCP is launched.
And now they're like, oh, I need to bring my MCPs in the same proxy layer.
And they have to solve for authentication and authorization and all that, right?
And then A to A is launched.
So they have to figure out this agent to agent communication protocol through the platform, right?
Then multiple agents that their teams have developed hit production.
So they care about ensuring uptime, reliability, latency, optimization.
And then agent identity becomes a real thing, right?
So it's an ever-changing space from their standpoint.
new models keep getting launched, new protocols keep getting launched, and the complexity and the requirement from this platform keeps on changing every single day, right?
So for thousands of teams out there whose main job is not building and maintaining a platform like this, it becomes a completely intractable problem to solve, right?
So we have seen that organizations went from what we call as this build, break, and then buy cycle.
right and that has created an opportunity for a startup like us to build a platform like this now of course at our end the complexity does not reduce it only increases because now we have to deal with the stack of all of those thousands of enterprises right while the space is ever changing so so we have done a lot of optimizations at our end where we have built out internal agentic solutions that keeps polling what are the new things that are getting launched in the market what's breaking in our platform and how do we continuously keep optimizing this besides of course like the fact that that there's a team that really cares about solving this problem statement same thing for agent harness i feel like this is relatively newer but we have seen a major demand from organizations that they care about being able to take control of the agents that are being developed and you're just not able to take the control of the agents that you're developing if you're using a closed proprietary agent development platform that you just don't have any control over.
So that's where people want more of a vendor-neutral harness and it's been fun to work on this.
The reason this becomes extremely interesting is vendor-neutral agent harness by design is open to models from multiple providers.
So you are able to access frontier models from the labs or you're able to work with frontier models that are open source.
All of those models for all of the organizations that you work with are available through the gateway layer.
So people can bring their own model.
Extremely straightforward, right?
And this is extremely straightforward to bring their own models, basically.
Similarly, people care about being able to bring their own MCP servers.
Now, the good news is that a lot of MCP servers that people want to access are these public MCP servers, remotely hosted, centrally managed.
So they don't have to put in a lot of effort to try out different MCP servers.
It's all connected through the platform.
of the gateway right but then these mcp servers can be easily plugged into the uh the agent harness there and help them invoke these tools and same thing you can extend this to agents as well agents are all accessible through the uh through the gateway layer right now across any of these models mcps and agents if you wanted to apply any kind of guardrails your guardrails are also available in your central control plane layer that gets implemented through the gateway right so fundamentally what's happening is when you're building your agents All the raw building blocks that you need and the corresponding registries are available as part of your control plane.
The logic to build these agents, the engine that operates the model and the tool and the agent loop that sits within the harness and it's able to help you orchestrate your calls better.
And the two together actually becomes a powerful force because now you're able to build agents that are accurate, but cost efficient and centrally governed.
So that's roughly the unlock that we're getting by bringing the pieces together, if that makes sense.
Yeah, it definitely does.
And one of the other interesting pieces of the gateway is that one of the, I think, motivating factors, at least in the initial push, was to give a unified interface to all the different models.
But as the different providers have...
changed their APIs and as OpenAI itself has evolved their API, that became a much more difficult moving target, as well as the fact that by forcing everything into that one shape, you potentially lose out on capabilities of Anthropic or DeepSeek or whomever else.
And so I'm wondering if you can just talk to some of the tensions there of having an easier onboarding where you just have one API contract to code against versus...
being able to actually take full advantage of the underlying providers?
That's a very interesting question.
There are two aspects to it that I would like to call out.
Number one, that of course, with the API signatures of the model providers continuously changing, this becomes relatively difficult to implement this.
But to that end, what we have done is, for the most part, we do the job of unifying the API signatures across the diverging API signatures on the right side and bring them all with a unified access for our end developers.
So we don't lose the capability of different model providers, but at the same time, we try to solve for bringing them all into a more unified API signature for our end users.
So they don't have to worry about these waning signatures.
But at the same time, you're right that this is not possible to execute every single time.
And for that, what we have done is we have also implemented what we call as a custom endpoint that you can invoke through the gateway.
And in that case, like you do not have to be compliant with the gateway specific signature and you are able to run them through the gateway.
You're able to get a lot of benefits of the gateway to the end customers.
You obviously get full benefits of the model in that case.
but you do lose some out-of-the-box capability that the gateway is able to do so long as it understands underlying model signatures.
So I guess we have divided the platform into those two parts.
One where you get complete benefit of the gateway and unified API signature.
And the other case, you bring a custom signature and you don't get complete benefit of the gateway.
Digging more into the harness itself, as we've noted, that's, I think, the most recent.
hype cycle that we're in where everybody's building a harness.
It's become a very complex market.
There are a few that have gained a decent amount of traction.
I think Pi is probably the most notable one, largely due to the success of OpenClaw.
And I'm wondering if you can just talk through some of the key challenges and components of building a harness that actually works well.
Absolutely.
Absolutely.
Our team has been having a fun time building out this harness.
And the first thing is getting the accuracy right, right?
Getting the agent's accuracy right, ensuring that you are able to get the quality of responses that you get from some of these out-of-the-box harnesses, managed harnesses, is actually a tough problem to solve, right?
You don't quickly implement a harness and get there, right?
But after that, once you have gotten there, the challenging aspect of Harness is a lot of the components that, additional components that you have to deal with to ensure that all of this is happening, right?
So for example, how do you make sure that you give access to a sandbox to an environment?
And by the way, like, of course, along with all the recent incidents, how do you check that the agent is truly sandboxed, right?
It's not able to get out of it.
How do you check that the agent, when it's executing the code, it's actually executing it efficiently and in the right environment, right?
How do you make sure that which of the tasks should actually be done through coding and you do the memory management around that done through coding versus it's done through the tokens directly and you have to build the memory management around that.
So then some of those complexities are actually extremely difficult.
Then anything around durable execution, human in the loop.
that you have to ensure that certain tools you get feedback from the human beings and then only you let the agent proceed forward is complex.
And lastly, the one thing that is extremely challenging with the harness layer is how do you manage permissions and access control through agent chaining, right?
Because what's happening now is agents are often acting on behalf of users, right?
And what I mean by that is as a user, I can send a Slack message to Tobias an agent acting on my behalf can send a slack message to the bias right how does the bias know whether this is directly coming from nekunj or an agent acting on nekunj's behalf right and what happens if this agent invokes a sub-agent invokes another sub-agent in the chain basically right and each of these different agents had their own identities and had their own access control how do you ensure that the least privilege is managed throughout that entire chain right so all of these uh like you know complexities are actually fairly difficult to handle.
And what lives in the harness layer, what has to live in a more centralized governance layer, the right thought process around that is also something that made things very complex implement.
The other interesting aspect of harnesses is that that term has become a little bit conflated because most people are thinking about it as far as their cloud code or codex or pi agent harness or any of the numerous other harnesses that are out there specifically for coding agents.
But then there are also harnesses that are focused more on deployed production agents that are interacting with various other systems.
And you want to make sure that those are more constrained than your local terminal agent might be and have different capabilities.
And so I'm wondering if you can talk to some of the aspects of how to build that harness in a way that is layered and composable so that you can have that.
core functionality that's shared across all the different use cases, but be able to actually exclude functionality or exclude permissions from the harness for cases where you want it to be operating in a more constrained environment.
That makes a lot of sense.
And you're exactly right.
First of all, there's actually a lot of noise in the market.
People actually are not able to understand the difference between all of these different products and the corresponding intended use cases with which they were developed.
From our standpoint, the stance that we have taken here is that we are focused on building a more general purpose.
agent harness right so like general productivity agent harness is what we are uh like you know what we have taken a stance on building but also like you know as you mentioned that we are developing this with an intent that we start with like this entire personal productivity type of an agent harness right and eventually if we want to focus a little bit more on becoming a coding harness, we are able to steer the harness in that direction essentially.
The fundamental building blocks don't necessarily change between these different harnesses, but the way these components come together and how in one case you're always dealing with a constant context layer let's say from a coding side versus in the other case like you know you get a lot of the context from your mcps and stuff that you're connecting on the harness right so how do you how does the harness handle those nuances is something that like you know we have basically kept it configurable from our standpoint one thing though that i would mention that that has been uh that has been a big win by the way in exactly differentiating these between different types of harnesses is by not trying to be all in one you actually gain a lot of efficiency.
And what I mean by that is if you try to solve all the problems in one harness from the get-go, then essentially, like, if you think about all of your system prompt has to become massive, right?
And because a harness at any point in time might have to be dealing with any of the underlying, call it, tasks that you care about it solving.
And because of that, and that system prompt, by the way, goes in every turn of the agent conversation.
Like, you cannot avoid that.
and when that starts to happen like you start paying taxes for that right so uh you are basically consuming more tokens in every task that you are uh that you are doing and you are spending more dollars uh on every task that's that you're doing and that has been actually one of the design decisions that we have made that the way we have designed the harness that we we are able to specialize this for different tasks and like no you don't have to do all in one at the same time that said i also wanted to share that Because of this being one of the reasons, and actually this is one of the reasons not everything, we have seen that when we ran the benchmark of our harness against other tools out there like cloud managed agents and all, we were able to get over 30% cost reduction for a maintained accuracy while using this exact same model.
And as much as 70% cost reduction, when we let the harness choose the models, like, you know, choose its own model essentially, right?
So it also had that flexibility of being able to choose the right model from different providers and stuff.
So that's how much range we have been able to achieve by, like, you know, focusing on the tasks that the harness is accomplishing as well.
On that concept of model routing, there's also been a lot of focus there.
The project that I'm most familiar with in that space is the VLLM router that...
lets you dynamically choose which model to route to for a particular request or a set of requests.
And I'm curious whether that's something that you see is belonging in the harness itself, in the gateway, something in between, and just some of the aspects of how to make sure that you are effectively choosing which model is going to have the right result without burning a bunch of tokens on a failed result and then having to retry it with a different model.
The way I see it, you have to have this kind of routing capability at this point in both places, at least the way the industry is going right now, you have to have this in both places and they serve a different purpose in the two places.
Right.
A harness has to be able to figure out smartness when it's running the loop.
Right.
That which agent I'm calling, which model am I calling, which tool am I calling.
Right.
Around that time, it also has to have certain level of smartness that I know that certain models are really good at coding tasks.
So when I'm getting a coding task, I route my query to those models.
In this task, I have such a large token context window that I need to use a model accordingly.
So it has to have a certain level of that intelligence built into it.
And that's the routing logic that I believe the harness will continue to implement and own.
That said, you can also imagine that when you're using uh when you're making calls to the models many of these calls are coming from the harness but many of these calls are directly coming from end users where there isn't an intelligence layer uh built into the caller to begin with right so so from a and all of those calls are fielded by a gateway layer right so a gateway has to be also smart by looking at the prompt uh right and figure out which model So should it route it to?
And of course, gateway solves the problem from a different lens as well, right?
Because the gateway has the awareness, the real-time awareness of what's the status of different model providers?
Like is one of them serving high latency and I may want to route to a similar model, but to a different region or the same model from a different provider, right?
And it also, of course, has a lot of context around.
how much overall budget and rate limits have been consumed so far.
So it has a lot of real-time awareness and it would do the routing based on complexity, but also all of this awareness as well.
And that layer sits rightly within the gateway.
And the task contextual awareness routing sits more within the harness and both of them need to coexist.
The other interesting aspect of harnesses is because you're working, with these models so closely and because testing anything that touches a model is so challenging due to their probabilistic nature, what are some of the validation methods that you use when iterating on a harness to make sure that you're not regressing and that it is able to actually work effectively across such a wide swath of different models with different capabilities?
I think that is certainly a challenge.
for the exact reasons that you mentioned, right?
And I wouldn't say that we have fully solved the problem where we have absolutely contained the test cases and regressions and ensure that this is always fully solved.
But of course, we try as much as possible and the community understands this problem statement, right?
So now there are a bunch of benchmark data sets against which you can always keep running.
uh your harness to ensure that you are not regressing on your quality latency and cost uh benchmarks right so we keep running that and then internally for ourselves like you know think i think of these uh benchmark data sets as more integration tests from the from the old software world right and then you still need your functional test and unit test as well so what we have done is for different modules that we are building within the harness we have actually designed some of these test cases as well right so the independent modules don't regress but meaningfully with like you know so many variables and the stochastic nature of the models and the harness the only true way of testing this out is against the benchmark data from our standpoint.
But even then, so it so happens that the end users of Harness should also actually like depending on the use cases, they should have at least some coverage of these test cases at their end as well.
That's a final litmus test to ensure that this doesn't regress.
As people are designing and implementing agents of various shapes, And given the wide variety of harnesses, as we've discussed, what are some of the ways that people should be figuring out what are the differentiating factors across these different harnesses and how to identify which one is going to suit their particular use case?
It's a very interesting question, Tobias, because the way I'm thinking about this, and I would love to brainstorm with you on this one, like hear your thoughts as well, right?
But of course, like, you know, while...
We see that there's a bunch of harnesses out there in the market today that you're able to leverage to build some of your agents.
The divide in the industry is at a very macro level, right?
And actually, we are seeing our customers choose their harnesses based on those macro divides at the moment, rather than trying to get super micro with it, with a bunch of lower level capabilities of different harnesses.
Or at least that's what we're seeing at the moment.
What do I mean by that?
At the first level, you can imagine that you have harnesses that are SaaS, like managed harnesses that you get to use out of the box.
There are some of the harnesses that are more vendor neutral, more open harnesses, right?
And people are able to take a call based on their own needs that am I happy to be locked into this ecosystem and get the advantage of an integrated ecosystem, right?
Which could come at a price or do I want to do a little bit more work and get into this open source neutral type of a harness, right?
So that's one type of a thing.
The second here is that the way some of these harnesses are implemented, they're intended for certain audience.
Like some harnesses assume that you're a developer, you understand, like you're able to set up a bunch of configurations.
Even if need be, you're able to write some code, right?
Or micromanage how the harness is working, right?
So it's intended for a more technical audience.
where some harnesses actually are just like you know built for a knowledge worker who does not necessarily come from a coding background and does not want to manage complex configurations uh overall right so today i'm seeing that like you know there's a few handful of tools that are out there that like you know people for the most part are leveraging can be divided based on based on this itself right we people don't have to go too low level that oh like in this case they harness does uh very well on memory management and in this case the harness is very well on a sandbox execution so and so forth and we are not seeing people take calls based on that are you seeing a different trend by any chance so yeah i think that in the harness space one of the interesting pieces just as an anecdote is that a lot of them seem to be written in typescript and running on the node ecosystem which As a Python developer, it is curious to me, but I understand that JavaScript is taking over the world.
But yeah, the key factors that I see when I'm looking at different harnesses are, yeah, what is the model support, which is generally fairly broad, but also how does it integrate with some of the different providers?
So I think Claude and Codex are probably the more interesting ones, Claude specifically because they've been having a challenging time of figuring out how, their pricing that when you're going through their cloud desktop and cloud code work versus if you're using a third party harness where you don't get the benefits of their flat rate pricing.
So that's one of the considerations when I'm looking at some of the harnesses as a software engineer.
But the other pieces are the MCP and A2A support.
So how quickly they're adopting some of these different protocols, particularly as the protocols themselves change.
So MCP just released their latest revision maybe a month ago.
And then what is the footprint of the harness?
How much resource does it take?
How does that factor into deployability, particularly for if I'm using this as a deployed agent that I want to run?
How does it fit into things like instrumentation and observability?
Because I want to make sure that I have a full trace of what the agent is doing so that I can figure out.
when things go wrong, why do they go wrong?
How do they go wrong?
Because they're moving at machine speed, you have to have other agents that are monitoring those outputs to make sure that you can catch those failure modes versus just if you just have a harness that says, yes, I can run the model, I can plug into all these protocols, but it's a black box and opaque and you have no introspection into the system.
That's definitely a non-starter for me as an operations engineer.
So those are some of the things that I look at when I'm thinking about different harnesses.
That makes a lot of sense.
That makes a lot of sense, Tobias.
For sure, we definitely see our users considering these, but a lot of the times, at least from our lens, what we are seeing is...
We work with organizations and organization level.
I think the decision matrix is broader than this because people really care about being able to suit some harness to their own like broad scope needs and they choose based on that.
But we'll get exposed to more and more of these nuances like, you know, as our open source agent harness gets like a broader adoption as well.
So I think that would definitely be interesting learning for us.
So digging now.
further into the true forge harness itself can you talk through some of the key design and architectural elements of it and some of the ways that the shape and scope have evolved from when you first started working on it so so in terms of like just generally the way we have designed the first of all i can just let me give you a little bit of a background on how this entire uh concept of harness started from from our standpoint right and then i will describe how some of the architectural choices have evolved for us as a team as well.
So there were two main polling factors around this.
We wanted to build some capabilities on our own product like True Foundry AI Gateway product where people are able to consume a lot of the information through the product but in a more agentic way.
I'll give an example.
True Foundry AI Gateway sees a bunch of traces, agentic traces.
And we want to be able to reason on the traces, right?
That which of my agents are doing well, which ones are prone to any guardrail prompt injection attacks, or where is PII leaking that I don't want it to, how can I optimize for cost, and stuff like that, right?
So we wanted our users to have that capability where they can more naturally improve the agents that they are invoking through our gateway, right?
So fundamentally, this means that we wanted to...
bake in a first-class agent as part of our own product so that people are able to leverage this.
And to that end, fundamentally, everything that we needed this agent to access, all the context, all the traces, all the models and MCPs that the user had permission to was available within the product.
And we had all of these things available as context and as MCPs.
So from our standpoint, this This felt like it should have been as simple as that we are able to connect a bunch of these contexts and use a harness to quickly spin up this agent that we are able to ship with the product.
But when we try to use the tools out there today, like back then, we realized that you are either locked into a bunch of these closed harnesses, which does not allow you this capability of shipping agents as part of the product itself, or you would have to fundamentally leverage more pro-code frameworks, which require you to micromanage the performance of your agents and then quickly things diverge and you keep upgrading that essentially.
So we really wanted to strike that balance where we are able to optimize for speed, ship things fast, and they cannot get locked into any ecosystem.
So that was one of the motivation that led us to kick things off.
And at the same time, we started seeing that a lot of our customers started caring about like not being locked into a certain vendor that way.
So when we started building this out, the neutrality, the openness, the speed at which you are able to ship, these were some of the design principles with which we had started building out the harness.
Over a period of time, when we started actually implementing this, and this actually changed a lot in how we thought about the overall architecture, we realized that while we are able to make building of the agent relatively straightforward, being able to run this, right?
And being able to run this at times and in environments that we care about is another challenge.
Like there's a lot of things that you want this agent to do that you want to be able to schedule, that you want to run every day or...
every few hours, right?
Or you want to run this on a trigger.
How do you expose or sometimes you want the agents that you're developing, you want this agent to run as part of another ecosystem, right?
So how do you build out that entire runtime of the agent is another major problem that we realized that we needed to solve, right?
So we continue to evolve the harness from primarily being neutral to this entire runtime.
Now, two different types of runtimes basically, right?
Similarly, Some of the design choices that we needed to change was how do you think about sandboxing overall?
And what we thought was like, you know, sandbox should be exposed to the agents of the tool.
Like that's the other design choice that we had to make.
Like most of the hardasses that we know run the entire agent itself in a sandbox.
We actually ended up flipping that design principle over a period of time.
So now what we do is we run the agent loop in the server itself.
And we provision a sandbox only when the agent actually needs to execute that code, right?
So the advantage that we got by doing this was one server is now able to run many of our agents concurrently.
And it turns out that when you are not using writing code, basically, it makes those turns go much cheaper and much faster.
And you're ensuring that things like your entire secrets and all are never leaving the...
the layer that the harness can touch at all, right?
So a lot of these things that we started editing in our architecture came as evolution of what we saw as this accuracy, cost, and efficiency trade-off that we really, really wanted to nail down.
And over a period of time, we saw things evolve.
And as you have been working on the harness and using that to validate the gateway, what are some of the most complex or challenging aspects of building these two different systems and using them to play off against each other?
And just some of the discoveries that you've run into that were kind of unanticipated roadblocks on your path to actually building these products.
Yeah.
So.
Like in all honesty, I think one of the problems that you always have to solve for in such settings is organizational, right?
Because the way you organize your teams building out the problem, by the way, these are problems that everyone needs to solve for, right?
Like people have to figure out the solution.
So I would like to share this with folks as well listening to the podcast.
When you think about...
building out the gateway itself.
You are building for one intended audience, which is getting used in a certain way, right?
Organizations buy a gateway for observability, for control, for being able to access the rich ecosystem relatively fast, right?
When you're building out an open source harness instead, you are building for the capabilities to empower the developers with a bunch of these capabilities so that they are able to, developers or even knowledge workers, to take up a harness and they're going to build out these agents.
Now, because the user is slightly different in the two cases, the team that's building it has to always continuously empathize with the end user that they're building towards.
And the pace at which the team is developing different components also needs to change because one is serving more production systems.
The other one is serving like in a more building phase of the ecosystem as well, right?
So from our standpoint, how do we allocate the bandwidth?
How do we divide the team?
How do we ensure that we are able to meaningfully make progress on both the two fronts was actually a relatively big problem for us to solve for, right?
The way we came up with, the approach that we came up with to solve this problem is we actually combined the part of our gateway that was leveraged to build a bunch of these agentic solutions directly.
Like we had a component in our gateway where people are able to register registries, agent to a registry.
And from there, they would...
always run into some roadblocks into how they have built out an agent using another closed source product or some other framework.
They want to register to the registry and then access it centrally through the gateway.
They would keep running into those roadblocks.
And there were a lot of learnings that were coming out about how the agent development platform or the agent development hardens itself should solve that problem.
So we combined the team that was always getting that learning from this part of the product.
to also work on the agent harness layer, right?
So the top of the funnel, like the core pain points that people want to leverage and outsource open source harness was always coming to this team, right?
So the empathy for the user, both from a gateway standpoint and from a harness standpoint is how we solve for.
And a lot of the capabilities that we ended up building within the harness by design, the roadmap of the harness actually came from our early users of the gateway directly, right?
So that way, a lot of the downstream challenges that we would have needed to solve for actually got front loaded by working with the right set of users using the product as a design partner, I guess.
You touched a little bit on some of the cost aspect of the harness being a contributing factor where that's largely the system prompt and trimming that down can have a substantial savings.
But as you have been building the system, what are some of the other key elements of the harness itself that contribute the most to the cost efficiency, the accuracy of the outputs and the reliability of the agent?
operating within that harness?
So there are three major factors to this entire cost optimization and the accuracy and reliability trade-off.
Like one we covered, which is your system prompt, right?
The second one is generally your harness efficiency.
And what I mean by that is the number of tool calls that you are making.
the order in which you are making the tool calls, the order in which you are applying any kind of policies that you may want to do, right?
Like, you know, which actually has a significant impact on how the overall harness is working, right?
And to this end, there are certain, like, you know, more technical concepts that you also bake into the harness.
For example, you...
You do deferred loading of tools, for example.
When you do not need access to certain tools at a certain point, you defer that, right?
So how do you bake in that intelligence that for any given task that the harness is accomplishing, right?
You always pick the right combination of whether you are loading a certain tool, whether you are using the actual persistence to be able to store some context and only...
get a part of it when you need to answer a certain question.
What do you solve for using tokens of the model directly versus where you think that being able to write code and execute this would be the right way to solve this problem?
So this is what I call as a harness efficiency.
And the right choices that you make in this one is extremely critical to solve for the cost and accuracy trade-off.
And the last one.
which is the overall model benchmarks overall, right?
Like, you know, that's the other one that's extremely critical.
Many times, like for the same task, like different models can have an order or even two orders of magnitude cost difference.
But like their performance could be fairly comparable at that point, right?
So how do you choose the appropriate model for different types of tasks that you are doing?
And how do you train the harness so that...
it's able to make those design choices.
I think these are the three key factors that we have worked to optimize on.
Obviously, a lot of effort that has gone into is problem number two, where different components of the harness, how do you optimize against those?
That's where a lot of efficiency is.
And citing some numbers here, there have been tasks where we have seen some other harnesses which are taking more than 75,000 tokens to accomplish the task.
Our harness was able to do within 23,000 tokens, basically.
So we have seen anywhere from 50, even 70, 75% of token reduction in similar kind of tasks by getting this part right.
And then going back to what we touched on briefly of different models behaving differently.
given even the same prompts, the general wisdom that has built up over the past couple of years, and of course, that general wisdom is frequently invalidated because everything moves so quickly, is that just changing a model, given all of the same inputs, can have wildly different outcomes.
And given the current generation of models that we are operating, is that...
assumption still accurate?
And what are some of the ways that people should be testing and thinking about their preconceived notions, given what they've learned from the past two years of evolution, where I think the hallucination rate was one of the first ones to gain a lot of ground.
And that has, it's still a factor, but the rate has dropped a lot.
I'm just wondering, as the models evolve, and as they all become more capable, has that assumption of switching models will dramatically change the outcome does that still hold um actually we have seen that assumption break and like you know it's been now a few months that we have seen that assumption break in all honesty uh and what that means is like you know by the way like you know hallucination is a term like I think if there was a way to build a heat map of how frequently that term is being used, I think now over the last few months, it has become quite cold, that heat map, right?
We don't hear people talk about hallucinations as much when they are talking about the model performance.
Same thing.
Earlier, we really needed to craft a prompt for each of the models.
And that was a big thing that people needed to solve for.
Now we are seeing that models are getting...
better, that you don't need to micromanage your prompts as much to work with these different models, right?
Now, of course, like when you go from a frontier model to a very specialized model, the specialized model expects, sometimes even if you have fine-tuned some of these models, and we work still with some of the fine-tuned models, that expects prompt to come in a certain way, and that really helps to be able to change the prompts.
But like so long as you're working in, like beyond a certain level of...
call it the number of parameters in the model, right?
Like, you know, once you're working with a threshold, a model is larger than a certain threshold, you actually don't need to worry about switching prompts anymore to get similar kind of outputs now.
And that, of course, has downstream implications now in terms of implementation of the harness that you just, if you, you're able to make choices more independently now, right?
But at the same time, we, in our gateway product as well, we keep this flexibility that if people are switching model, they have a scope to be able to change the prompt as well.
And it's especially useful when you're using fine tuning and stuff, right?
So we actually leverage that capability also in our harness as well.
One of the other factors of model selection that I've experienced is that you might need at least a certain threshold of model to be able to effectively drive the harness.
Where a while ago, I had a conversation with somebody who had the analogy of the LLM is the engine and everything else that you put around it is the car.
So you can have all the horsepower you want, but if there's no car, you're not going anywhere.
And so bringing that a little further, it sort of feels like you have sort of, as you're going on the amusement park rides, you must have this many parameters to drive this harness.
And I'm wondering what you're seeing as some of the cutoff threshold where if you get below a certain size model, it's not going to be able to effectively use the harness because there is too much in the context window.
So, and like, you know, the most interesting thing that we are seeing with this is people, the users of the harness, including ourselves, right?
We have built out a lot of systems that leverage our own harness and people actually don't necessarily need to go.
below a certain threshold anymore to make things work for them, right?
Effectively, it's not like people are thinking about, should I use a 3 billion parameter open source model to run my entire harness versus should I use Fable 5, for example, right?
It's not like people are typically operating in those extremes.
People are thinking like, is...
Like, am I doing a Fable or a GLM?
Am I thinking Haiku, right?
So people are still operating in that domain.
And what's nice about this is the...
overall token cost has reduced by such a large margin over that period that intelligence per token, as I think about this, right?
Intelligence per dollar that you're spending has gone up so much that that's the ballpark in which we are talking about.
So now you are still able to like even if you choose a base model, something like a Haiku and above, basically, right?
The hardest works.
And then for extremely complex tasks, you go to like your GLMs and the Opus and the Fable equivalent, basically.
And otherwise, you're able to do a bunch of tasks using Haiku and Type models.
And as you have been...
working on TrueForge and TrueFoundry and working with your customers, what are some of the most interesting or innovative or unexpected ways that you've seen your harness used?
So first of all, we noticed that people are, like we saw through our product itself directly, that we see queries in languages that we absolutely don't understand.
We were under the assumption that most people are using the harness in English with the most common user group that we work with.
We are seeing queries coming in completely different languages that we don't understand.
That was just a fun thing that we noticed in terms of the usage of the platform.
Outside of this, what we've seen with our harness usage, people are doing a couple of things that we just did not imagine will be a use case because it's supposed to be such a deterministic use case.
People have actually built out their own pricing systems, like how they price their product overall based on some data that they capture using our gateway.
and how they build out the end pricing dashboard for their end users, right?
Using our harness.
And this entire loop came together for our end users, and they were able to build out this entire thing within a few hours that their understanding was it would have taken them weeks or months to build this entire system.
That kind of a usage, we were not expected people to leverage the harness for, right?
So like, you know, basically...
The input comes from the gateway, the implementation happens through the harness, and that combination became very powerful.
So what this has opened up for our users and also just in our mind in terms of what we were expecting versus what we were seeing is people leverage gateway to build out pretty much all of their agent capabilities.
All of the agent capabilities by default get routed through the gateway.
Now, any derivative component that they need to build from here, that's where now the harness is starting to get used very strongly.
So I think that's been an insight that we have learned through the usage of the product.
And in your experience of working in this space and building these systems, what are the most interesting or unexpected or challenging lessons that you've learned personally?
So one of the very interesting things that I've learned building these products over the last...
two, three, four years is when you work with different personas with their own priorities, right?
They are looking at the same solution from completely different lens, right?
And so turns out that if you're building a system that caters to people with different goals that they're optimizing for, and this is the only system through which all of those goals get a common ground, right, to be solved for, right, then the only way that you can think about building a product like that is by having a good mix of opinion from all of these different varying voices in the room.
And I'll give an example of why that has been such an important learning experience for us.
In the very beginning, I talked about different pillars that this Gateway and Harness product is solving for.
It's solving for reliability that your platform teams, your development teams care about.
The fact that you built an application, it should be always up, always running, operate at low latency.
Then there's a security aspect to it that your CISO org cares a lot about.
The four walls within my organization should always be secure.
Then there's a CFO aspect to it that they really care about being able to do like this cost optimization.
Like overall, they should have an ROI and that entire story should be clear.
Which agents are we building?
How do we do the reporting and stuff like that?
And then there's this compliance aspect to it that if tomorrow I have to be able to answer why my agent took a certain decision, how do I do that?
And it turns out that...
the interest of the CISO, the CFO, the compliance officer, the CTO, all of these things are now being answered by this common layer, right?
And the most interesting way that you can solve for this entire problem statement is by hearing the voice of all of these personas, right?
And where we have seen some of the...
solutions within the industry where they have tried to create a completely specialized view for the CFO in terms of the FinOps layer.
And then they have basically done so much instrumentation that it doesn't get adopted by the platform engineering or the actual development layer, right?
So yes, you have built this amazing, beautiful FinOps layer, but because there's no usage coming from the left side.
There's nothing to see on the right side, basically, right?
And so on and so forth.
We have seen multiple combinations of these break in the industry.
We have seen multiple personas pull our product in different directions.
And the way we are able to maintain this balance is by listening to these point of views.
And I guess that's been a personal lesson that you really want to understand that, yes, who is your main buyer persona, right?
But who are the other extremely important stakeholders in the room?
And you want to make sure that you hear that holistic voice.
That's been a major learning for me.
And as people are evaluating different agent harnesses, what are the cases where you would say that TrueForge is the wrong choice?
Overall, number one.
At the moment, like, no, this is all a point in time answer, right?
So as of whatever, September 4th, 2026, right?
This answer likely is valid today.
But today we are not specializing ourselves for a coding harness, right?
So like if somebody wanted to build out, use this as a coding harness, probably I would find out other open source or closed source harnesses that serve that use case better.
In some other cases, if you really cared about being able to absent, like basically if you have a lot of visibility that roughly this is how my system should behave and these are the three lanes in which my agent should go on and there should not be any other fourth lane that they need to explore because I care about like whatever 10 millisecond latency implications, et cetera, et cetera, right?
If such is your use case, then you are actually better off being able to like...
more be in control of the code that you're writing and not let it run through the hardest layer like TrueForge, I would choose a more deterministic framework for building out your agents.
I think those are a couple of situations where I would not pick TrueForge at the moment.
If you're building anything that is more general purpose, that you care about the evolution.
of the agents over a period of time.
Like, you know, you care about being able to build it quickly, you care about being able to build it efficiently, and you care about being in control of the type of models that you're able to use.
You care about being able to control the cost.
We feel like TrueForge is a great choice.
And as you continue to build and iterate on these systems and continue to explore the current state of the ecosystem, what are some of the things you have planned for the near to medium term or any particular projects or problem areas that you're excited to explore?
Absolutely.
Absolutely.
This is the thing that we have been talking a lot about and a lot of our own roadmap is shaped directly by our customers as well, right?
The current users of the platform as well.
So one of the things that we have been very excited about is going much deeper into this entire agent identity area.
And how do you manage cross-app access?
How do you manage this?
agent chaining, permission controls, where does, and I'll give an interesting problem statement here.
Like I had mentioned briefly about how do you do least privilege across agent chaining, right?
But when you start doing least privilege across agent chaining, one of the problems that happen is now your configuration management of which agents have access to which MCPs and which tools and which users have access to which MCPs and which agents, that becomes a very complex problem to solve.
So, then how do you take this into a direction where you're not compromising for the UX, right?
Where you understand the intent as well.
It's not just about, you know, theoretically what is possible, you understand the intent when you implement that without compromising the user experience while ensuring the safety and security of the system.
Like that's the direction that we are taking the product in.
And then, of course, you know, this...
constant work happening around cost optimization, sandbox security.
So some of those areas we are continuing to work on and that's part of our immediate roadmap as well.
Are there any other aspects of the work that you're doing at True Foundry or on True Forge or the overall space of agentic harnesses that we didn't discuss yet that you would like to cover before we close out the show?
There's one thing that maybe I'll call out here is overall.
One of the learnings that we have had in this space is people have been trying to address the problems around the control plane for enterprise AI or how you bring together this harness using smaller targeted point solutions.
The goal has been for a lot of folks has been that, can I do my...
model routing layer?
Can I do my MCP registry layer?
Can I build out my A to A optimal agent to agent communication layer?
People have been thinking a lot about that.
One of the learnings that we have had working with the organizations that we work with is the only meaningful way that you build this overall control plane is if you bring these systems together.
Like if you bring all of your tokens together in one place is how you only meaningfully solve this problem, right?
So it turns out that a model proxy or a separate MCB proxy or an agent registry in isolation or a guardrail solution overall is just not able to solve the problem that organizations are facing because The true problems don't happen within these point solutions.
The true problems happen at the interface of these point solutions.
And to meaningfully solve for this entire thing, you need to bring these components together and do it in a very AI-native way.
That's been one of the learnings that I wanted to share with the folks as well.
So whoever is thinking about the architecture of...
how they build and scale and manage all of their agents, maybe this is one consideration to be made here.
All right.
Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes.
And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling, technology, or human training that's available for AI systems today.
This is such a deep topic, right?
It's very difficult to give a quick answer to this.
But the way I'm seeing this is we need a holistic shift to solve for this entire problem.
I literally talk to teenagers, college kids, and folks are asking, what should we be thinking about?
What's the next thing that we should be doing?
What's the discipline that I should go study in?
Should I still learn coding?
Should I not learn coding?
How do I upskill myself in this entire world of AI, right?
And it's such a deep question that's there in everyone's mind that I feel like a tactical solution around how you learn AI or how you learn this latest new tool or any framework out there is just not an answer to the question.
I think the only way to meaningfully solve this is what are some of the more fundamental layers where we expect human beings to contribute and add value.
And one of the fundamental concepts, like aspects in which we expect AI to assist us, we need to think about that and that needs to get inculcated into our education system more fundamentally as opposed to solving this problem superficially.
So I don't have a great answer to this question, but it's a problem that I resonate with very deeply through many of the latest conversations.
Absolutely.
Yeah, it's definitely, A big thorny issue that the entire world is having to tackle right now, even just in my own small ecosystem of working in an engineering team, it requires a lot of reimagining how you work every day because you can't human review everything the agent ships because otherwise it'll actually never get to production.
So it requires rethinking and reimagining entire processes and entire workflows.
It'll be an interesting few years to come.
Absolutely.
Absolutely.
Well, Tobias, I really enjoyed the conversation.
Thank you so much for having me on the show here.
Yeah, thank you very much for taking the time today to join me and share the work that you and your team are doing on True Foundry and True Forge and sharing some of your lessons learned on harness engineering and AI gateways.
It's definitely a very interesting space and very relevant topics.
I appreciate all the time and effort you're putting into that.
And I hope you enjoy the rest of your day.
Thank you so much.
Take care.
Thank you for listening.
Don't forget to check out our other shows.
The Data Engineering Podcast covers the latest on modern data management, and podcast.init covers the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
And if you've learned something or tried out a project from the show, then tell us about it.
Email hosts at AIengineeringpodcast.com with your story.