# Uber's AI-Native Developer Infrastructure Strategy

**Podcast:** The Pragmatic Engineer Podcast
**Published:** 2026-10-14

## Transcript

What kind of internal tools do software engineers at Uber use to build software today?
We get an exclusive insider and visuals-heavy look from one of Uber's longest-tenured engineers, Udaykira Mendicetti.
Today, Uday leads much of the work on agentic AI at Uber.
In this conversation, we get into internal developer systems Uber has built and used pre-AI like UOwn, M3DB, Michelangelo, Cadence, Morpheus, and others.
AI foundational systems, a company built like its MCP Gateway, AI Guard, and AIFX.
Recent systems used by nearly all engineers like Minion, CodeInbox, Slate, Shepard, Wisdom and others.
If you want to understand what AI native development looks inside of a modern tech company with thousands of engineers like Uber and why it's much easier to retrofit AI on top of battle-tested developer Infra, this episode is for you.
This episode is presented by TurboPuffer, Vector and Full-Tex Search built on object storage.
It's fast, cheap and extremely scalable.
Today's episode is about the close to two dozen internal systems that Uber has built to help engineers ship with AI agents.
One of the takeaways will be that production-ready code is far more complicated than just using an AI coding harness.
The harness is the easy part.
What's hard is all the things around them.
This is where I need to mention our season sponsor, WorkOS.
One piece that is always a hard problem with agents is permissions.
Do you want an agent to have standing access to everything it might touch?
Or do you just approve every single prompt until you just stop reading the prompts?
WorkOS built Airlock for this.
Airlock is an authorization layer that sits between your agents and the tools they call, and it judges every single request against the agent's intent and your rules.
You write the rules in plain English, agents never see your credentials, and every verdict is logged.
It works with coding agents like Cloud Code and Codex.
WorkOS Airlock is how agents do real work without over-permissioning.
Check it out at workos.com slash airlock.
Uday, welcome to the podcast.
Thank you so much for having me.
I've been following your work.
It's amazing to see the kind of impact you've had and all of the great content you publish for all of the engineers.
So I'm super honored to be here and talk to you today about what you're doing.
Well, we worked together well before the pragmatic engineer was born back at Uber.
Not together together, but I knew of your work as well.
When did you join Uber?
Exactly.
Yeah.
So I joined Uber in May 2015.
In fact, like the two engineers who referred me, they're still here.
So I joined in our real-time API team.
It was during the period where we were trying to break our monoliths into microservices.
And I was working on our API gateway for all of our mobile apps.
And I worked on changing our mobile protocol.
Since then, I worked on many different business platforms from our Fulfillment, Uber for Business, on our earner onboarding, marketplace pricing.
And the last one year has been all agentic AI, both for our users, customer support, end users, and so on.
So, yeah, it's been like a roller coaster last one year on working on this.
Yeah.
And for those people who didn't work there with us, can we recall a little bit what real time API or RT APIs we used to call it inside, what it was and why it was a big deal and why it had so much load?
So when I joined, we had just two services at Uber.
It was called Dispatch written in Node.js and API written in Python.
dispatch just to handle all of the rider and driver traffic at that point in time and the protocol was pretty simple at that point it was like a four second ping protocol we used to call that would give you the entire state of the world and when the trip is completed we would hand off to api it would handle all of the post trip processing payments receipts and everything rtapi was born at that point in time to abstract all of the internal services and mobile and also changing the protocol from simple four second protocol to back like where we have both rest apis and we also have server to client communication we used to call it push protocol so we would because so much of our app is very dynamic everything keeps changing every few seconds so we needed a way to efficiently update the state of the app across both riders and drivers as and when we get new updates as in when things change in physical world so it used to be at that point like one of the highest qps services And over the years, we kind of, because all of our stack was just Node.js and Python, so that was the natural choice at that point.
But in the subsequent years, we have converted that into Go, and it's still one of our highest QPS services at Uber.
And I think for people listening, they're going to be surprised when they hear like, oh, there was only two services in 2015 or around that time at Uber, because of course, Uber was known for the microservices boom.
And Tuan was on this podcast, he talked a little bit about this, but would you mind just recalling?
how those microservices came to be and from your perspective, you know, like what it meant to suddenly have so many services.
Yeah, I mean, I think there's like so many blogs and like so many people talk about that.
So I think like when I joined, right, like we were, as I said, like we had just two services and there and you also think going back in those days, the kind of hyper growth phase where we were in.
We were constantly expanding to new countries.
We were constantly expanding our product portfolio across mobility.
And later we started Uber Eats and that also was expanding.
We needed a mechanism for teams to really build all kinds of functionality very quickly to match up to the scale at which the business was growing.
So we have invested a lot in all kinds of service infrastructure, making it easy for teams to build services, metrics, tracing, monitoring, observability, deployment, the whole bunch of stuff around that.
And there's not much off-the-shelf software.
for these things at that point at the scale and rate of change that we had and there's a lot of stuff that we had to build in those days internally and that created a boom cycle of a lot of we went from like two services to order of thousands of services then around 2018-2019 was when we standardized on our languages we picked go and java as our primary languages we modernized a lot of our tech stack We spent a lot of time in thinking about the layers of our architecture.
How do we platformize?
And over the years, in fact, when I worked on one of our fulfillment platform, we actually reduced some of the services to a few services.
Even now, we are kind of in the similar ballpark of number of services.
Even though our business is 10 times larger, we are doing many more products.
And this whole bunch of platformization and years of investment that has gone into it, like we are still running one single.
global tech stack across all the countries we operate in.
There's many, many platforms that serve all of the mobility verticals and all of the delivery verticals using one shared building block.
And I think that kind of allows us to build more mobility verticals, more delivery verticals, because we have platformized a lot over the years.
Would you mind telling us when you joined, how big was Uber's engineering team and how did it go over time?
Because I don't think...
people from the outside appreciate just how big that growth was?
So when I joined, I was around the first 600 engineers.
And in fact, the whole of marketplace team at that point used to be like maybe 50, 60 engineers.
And over the next few years, as we've scaled our business, as we scaled our traffic, we grew to a few thousands of engineers.
And around 2019, 2020, we started also expanding across different sites.
Now we have 12 global tech sites across the world in almost every continent we operate in.
In fact, and that helps us build like products for those hyperlocal markets.
And some of those products are then rolled out globally across all the markets where we could roll out that kind of functionality.
So I think that gives us a lot of ability to tap into talent globally and also build nuanced local products.
And that also helps us build one global tech stack.
And speaking of building products, tech stacks, you still work on the platform team, right?
Yeah.
So at least until my first 10 years, I worked on different business platform teams.
And in the last one year, I'm more...
spending my time on a developer platform, on a agentic AI for engineers, agentic AI for the rest of the company.
And in the process, I've also worked on some user-facing AI initiatives as well.
Can you take us back to 2015 on how Uber's internal tooling changed?
Because again, I was there and I was really surprised to see just how much custom tooling we had.
We talked about RT API, but there was things like Morpheus, Edge Gateway and other things.
Would you mind talking through like...
Back in 2015, 2016, if an engineer joined Uber, what kind of tools did they use and how did those evolve?
I'm talking, obviously, the main ones that most engineers back under mobile engineers touched at Uber.
Yeah, absolutely.
I think in those days, given the way in which we were scaling, there's not many off-the-shelf products that we could tap into.
So I think we talked about RTPI and Edge Gateway.
Some of the other stuff that we had to build was a system called YouOwn that will enable anyone in the company to identify who owns a specific asset.
Like it was like services, right?
And you could look at during on-call, if there was an outage, you could figure out which team owns it, who's on call for that kind of stuff, right?
Yeah, absolutely.
And as the engineering size grew, we also had to build like end-to-end like our engineering design docs, product design docs, getting all of those into one single place.
we have to work on getting all of our incident manager into one single place we are like really obsessed about how we handle our our incidents and the learnings from that so we will build a lot of really good tooling around that i think there was some of the more complex ones was our metrics like even back in 2015 2016 we used to probably have around like 500 million metrics emitted per second with with the cardinality and there's not there's no off-the-shelf uh metric system that we could use and that gave uh rise to m3db which was open source later and like there's even companies formed around that um and um that enabled us to build like the kind of uh like a handle kind of matrix volume that we had we also had to build our own end-to-end machine learning platform because we used to have like hundreds of ml models and if you take the whole life cycle from creating the features, building the models, monitoring, serving.
This is Michelangelo, right?
Michelangelo.
And like we recently, just a couple of months ago, we open sourced our deep learning and part of the Michelangelo infrastructure.
So there's a lot of like really good, solid foundational building blocks there.
The other interesting one is also Cadence, which is if you think about some of the...
workflows, right?
Like they span across multiple days, whether it's a driver onboarding, whether it could be some CRM campaign.
You could see a lot of engineers doing their own resiliency stuff, their own state management, and that gave rise to Cadence, which is a durable workflow engine.
And later even that got open sourced.
I think not many people will know Cadence, but a lot of people will know Temporal, right?
Which was, as I understand, either inspired by Cadence or...
a lot of the ideas came from there.
I'm not even sure it was the open source version of Cadence, but it had a lot of like it's kind of the external version of whatever Cadence is for Uber or used to be.
Yeah.
And I think those folks like this, all, you know, there's a real need for something like this.
And they were able to go and like build like a scale it even further beyond Uber.
But I think that the pattern that I'm trying to say, like there's a whole bunch of things that we had to like invent in those days for the scale that we had.
And over the years, then we are like now, as the industry also mature, like now we have few few products that we build.
So somewhere we have we think there's some unique need we buy.
And if we build anything, if we think it's interesting for the rest of the industry, then we try to open source wherever possible.
Yeah, I think two more tools that I think is worth mentioning, because I just remember them as one is Morpheus.
It's basically launched directly for Uber, but it was way more integrated into everywhere, right?
Like you could.
You could go in there and run experiments.
You could see the results immediately.
It was almost like feature flagging and experimentation, but integrated into mobile feature flags.
And it was like, I remember inside Uber, like all of these integrations, it was just integrated into a bunch of tools, all of them, right?
Yeah.
And since we were also expanding to so many countries, like you need a really good configuration platform, whether it's for like configuring our backend services or also having feature flag system and also experimentation.
And Uber is a very data-driven company and everything goes through a feature flag, everything goes through an XP.
Experiment.
And since we have also, it's not, and since we have two-sided marketplace and three-sided marketplace, some of the experiments are also not really complex.
It's not just simple A-B, we also have switchbacks.
And like we have to build a really deep experimentation that's baked into all of our tools and it can automatically generate analysis and like what are the p-values, what metrics to use.
And that also led to like working a lot on our metrics foundations, data foundations, so that whatever analysis that we generate from this metrics are trustworthy and we can make business decisions based on that.
Yeah.
And I remember I was always wondering why do we have our own feature flag system?
But then I was thinking like, hmm.
Like, show me a feature flagging solution from a vendor that supports cities.
In fact, cities are first class citizens.
Because inside Uber, like, I think cities were so important.
And like, a lot of our tools had cities as, for example, for cardinality, like every city had its own metrics.
And then you start to realize like, oh, very few things treat, like, Uber had a bunch of interesting constraints, right?
Like that no one else had at this scale.
Yeah, absolutely.
Yeah.
Yeah, I think the city one is an interesting one.
It's like so deeply baked into like our infrastructure.
Like, for example, some of our like critical platform services, we also do sharding by city.
Oh, sharding by city, yeah.
Because then we can also like observe across different tiers of cities.
Because like we need to make sure like even now, like across all of the regions where we serve our traffic, we shard.
a city to a specific region we can also at any point in time fail over that city to across regions and even and we also have like a you know logical deployment groups where a group of cities are routed to a group of containers and this is for few services we don't do it for all the services we do it for some of the services that are where we need that level of that level of sharding and that level of reliability One other system I wanted to ask you about that I remember using, and it was being developed in 2017, I believe, which is DevPod.
And it felt really ahead of the industry at the time.
How did that come along and what did it turn into?
Yeah, so I think the backstory for DevPod is because we already talked about the growth of microservices.
Initially, we had micro reposts where every service was its own repository.
And we didn't have a standard build system at that point.
Each service had their own, like...
we had at least two three build systems two three testing systems and it was very hard for mess even for a central team when we wanted to do library upgrades go across all the services and do upgrades so around 2017-18 we we we then changed the strategy to getting to mono repos even though they are micro services but all of the services were deployed from the mono repos we had one lang one mono repo per language so we had a mono repo for go one for java one for mobile ios one for android and so on then once the scale of that repository was pretty big enough then like it was pretty hard to run the build and like getting all of those systems set up on your local laptop also then you have to like go through all kinds of setup issues um so we took on this thing of like devpod which is like cloud developer boxes and you can have any number of them uh we had one dev pod per mono repo so if i'm working primarily in java monorepo i create the java version of the flavor of the dev pod and it has all the build system cached um i can have faster builds it's working on cloud machines as a beefier than even the laptops so then you could have like we could have really good slas on your build time on your test time um and we don't have to worry about the setup issues because we can uh we can install the right tools in place And now the interesting part is as we got to the AI era, like even the agents need the similar workspaces and cloud boxes.
And we are able to kind of latch on to that.
And we also created like now a mega dev pod with all of the repositories, because then I can have my agents create PRs across the repositories.
It has all the tools it has to work in any repository to look at context across any repository.
This also helped us to create the agents MD, Cloud MD at the top report, at the top folder because we have standardized build, standardized testing.
Once we set up the top level files, most teams will get like really good value out of the bat.
They don't have to like really go into per module setup.
And like on the dev pod, all of the AI tools are installed by default.
So you can get a new dev pod.
You don't have to go through even installing like Cloud Code, Codex, Open Code.
They're all there in the dev pod.
So this made it really easy even for non-engineers.
in the company to create a dev pod and start playing with cloud code, start playing with codex out of the bat.
Yeah, so I guess like having cloud development environments just like really early helped a lot.
Yeah, like I think all of those years of platformization bets also helped a lot as we expanded on our agentic journey because you could use all the things that we built making the life easier for engineers and it makes it easy for our agents as well.
Yeah, I wanted to talk a little bit about before we got into what's on today, because I think a lot of things are connected, right?
So like there was so many things being built for platforms, for tools, often custom for Uber.
But having talked enough about that, what does it look like today?
What does development look like today?
What kind of tools do you use?
And maybe you could give us a little bit of like a primer of like the foundations that you put in place specifically for AI and what you built and why.
First thing is, especially in the ai world one thing we realized very early on was because the landscape is changing pretty quickly harnesses are changing pretty quickly models are changing pretty quickly we wanted to invest in the right set of building blocks that are hardness and model agnostic so that no matter any of these things change we can easily port across different models different harnesses so few things that we invested like so we started working on our model gateway from the initial introduction of ChatGPT like late 2013, early 2014.
So we had a model gateway back then which would expose an OpenAI Compatibility API or Anthropic Compatibility API.
Sorry, so like this is 2023-24, right?
Yeah, so 2023-20, early 2024 when ChatGPT was first launched and GPT models were first launched and we quickly noticed, you know, we wanted to make it easy for any teams in the company who want to like leverage.
and build agents with these models.
And so we had a model gateway that would abstract all of the complexity behind where we had all of the security and policy guards in place.
We had all of the visibility in place.
So you could switch across different models or different vendors for these models.
And you could just have your the ID for the organization or the project.
And then you can you can use any of the models exposed.
So this will use expose it as like a model selector and it piped into the model gateway and it routed to the right, you know, may that be chat GPT at a time or Opus at the time, that kind of, those models.
Yeah, and it was built on Michelangelo itself.
So we create a Michelangelo Gen AI.
Oh, Michelangelo, but that's ML platform, right?
Yeah, and then now there's a Gen AI set of foundations on Michelangelo platform.
So then you go to create a Gen AI project or a Gen TKI project on Michelangelo.
Then you can set up.
The project, you can select the model that you want.
Then you'll get a unique project UID that then you can use for your agents.
So then you have all of the same, like we're able to use the same Michelangelo infrastructure, then just start building on AI-specific evals and stuff on the same foundations.
Looking back, I think that was pretty early investment that gave a lot of ROI later.
What is Model Gateway?
can show like a quick primer into like you know what we had built and this was late 2023 when the first set of llms were launched and we quickly realized that we had a whole bunch of internal teams wanting to use it for different use cases and we wanted to make it really easy for them and this was built on top of the michelangelo platform and one more component inside that they can set up there And Michelangelo, just to recap, that's the ML platform that was built from like 2015, 2016.
So it was built a long time before LLMs.
Exactly.
And now we have like a Gen AI or Agentic AI set of components also inside Michelangelo.
And we're also building a lot more right now that those are not in the open source version yet.
But we are actively building a lot of stuff for Agentic AI world.
But my model gateway was the first such one and the initial version we didn't have all of these middle version components and we built some of these over the last one two years.
But what the experience for any team in the company is, you know, they create a client, they give their project ID which they initialized in Michelangelo and then the caller token and that's all.
And we expose one OpenAI compatible API or Anthropic compatible API and all of the models across all the vendors.
open-weight or closed-weight, they're all behind this gateway.
And the first thing we added there was what we call AI guard.
What is AI guard?
Essentially, we wanted to make sure we are not sending any sensitive content back to the user.
So this was one entry point where we could add this middleware that we can enforce consistently across all the projects.
um and security team had ability to like you know then they could iterate on all of the ai guard specific models whether it's on content with some safety policy whether it's on like whole bunch of comms policy we're like five different set of fine-tuned models that they built uh there which would which runs on every request and every response uh with strict latency guard rails this helped a lot to make sure that all of our projects have same standard baselines and then because Every project is tracked here.
We could have like per user per caller attribution.
We could have like then as as we progressed later this year, we were able to add spent years for projects spent years for users.
And since all the requests go through one place, we could also add like audit logging, session traces and a whole bunch of infrastructure around this.
Every model request at Uber goes through this.
So there is no model request that doesn't go through this.
So that gives us like one single entry point.
for across all the models.
And I assume like you can just try billing as well, like costs and all of those things.
Exactly.
Like track billing, cache iterate and all kinds of things.
And now, you know, as we progress into like there's a lot of discussion of like, you know, model routing and all of that.
I think this could be like a good entry point that we can leverage because all of our models go through this.
I think on the same journey of like being able to make it easy for engineers to switch across models.
The next thing was.
how do we make it easy for engineers to switch across harnesses like if i'm using cloud code then if i switch to codex then i need to like carry forward all of my settings all the skills all the tools uh all the plugins um so we internally wrap all of the harnesses with a wrapper called ai fx ai foundations and this is pre-installed on devpods or you can install on your local machine and We have a concept of personas.
So then that personas comes with a whole bunch of default plugins and skills so that you don't even have to install.
Like any engineer, they get a whole bunch of default skills that they can use out of the bat.
Since we wrap everything, all of the settings are managed.
So even if you're using any of the harnesses on your local or on DevPod and or any of your DevPod, the settings are synced.
So then centrally we can manage.
uh one set of settings that we can propagate across all the users and any user specific settings they are synced across all of their environments and again this makes it really easy because now we have one entry point for switching the harnesses switching the tools and then we also made it like there's a status line where we can give tips where we can have tracking of your spend across harnesses within that harness in that session so All of those additional things that we could add without any explicit setup from the engineers.
Nice.
And then speaking of tracking spend, how do you approach that right now?
Like as an engineer, I do, or if I just join as a, you know, like a senior engineer somewhere, like, do I need to be aware of spend, of cost?
What's the thinking there?
See, we want to make sure engineers have access to any of the tools that are available.
And we have this notion of spend tiers.
so by default you get certain certain tier and then then we have many levels of tiers once you once you are close to that tier then then you can raise a request to go to the to the next year And you just need a simple manager approval.
So then you can discuss with the manager the ROI for that project.
Like, how are you spending?
And it is a conversation around responsible use of this, right?
But we don't want to restrict engineers.
Like, even if they're working on, like, if this is going to the right project use, they should be able to spend whatever they need.
Because ultimately, we want to maximize the gain from this.
But you also want to make sure there's a responsible use.
So we try to do the minimum amount of paperwork needed to cross the tiers and they will see across any of our tools how much they are spending within that hardness and across all the hardnesses and we also have different tiers for different managed agents.
So then for all of the interactive tiers, interactive hardnesses, we have some spent tier for some of our background coding agents, we have a separate.
independent tier so we make sure like they can easily manage across all of these yeah but but i guess it just makes sense even if i do my side projects i will be aware of my own spend on my whatever limit i i have and i guess i guess do i sense it correctly that the thinking is just don't use tokens on stupid stuff that is that makes no sense or or accidentally you know you might be just burning and you don't know so like just just give that feedback like these things are not free Exactly.
Like I think just having that feedback and we also have like a plugin called cost check that will run across all of your sessions and give you personalized tips based on your usage to say, hey, like this is the kind of thing this much you're spending on this project and this session.
These are the tips that you can use to further improve your cost profile.
And I think all of this creates a good conversation.
between managers and engineers, but also engineers will also see good feedback into how they are efficiently spending across.
And that helped us a lot with all of the real-time coaching with context health, model sizing, so that engineers are also aware of working in this new age.
We also don't want inefficient spend.
And what other foundations that you built are related to AI?
MCB Gateway.
oh yeah i think the the other um backstory was around like i think early 2025 um we started like as cursor kind of picked up like we enabled cursor and mcp was um was any was a spec that was like created around that time and we quickly realized that hey the power of we can really increase the power of ai by exposing all of our internal tools in an easy way So then we could potentially leverage this beyond just coding.
We could leverage it for all kinds of software engineering tasks.
So around April of 2025, we started working on like MCP gateway, like, you know, similar to model gateway.
This is one gateway for all of the tools.
So what we did was because we already had good IDL registry across all of our services, whether they're using Thrift or Proto.
Since we have the ideal specs, we auto-generate the MCP specs for any of these services.
And just to add, I guess, we also don't know, like, Uber services either emit thrift or product buffers as strong contracts, right?
Yeah.
And, like, we initially, we had just thrift and later we added support for protobuf as well.
But all of the new services in the last few years, they're all protobuf.
The ones that were created, like...
a few years back they're all in thrift and as they go through rewrites usually like team switch to proto but whether they're in thrift or proto like we we would have we since we have all of the ideals in the in one single registry we could use that registry to generate a default mcp spec for them so they could anyone in the company they can go to this tool mcp gateway they can see all of the mcp servers that we auto generated And if you go to any of that, you could see all of the tools that were generated from that existing API.
So these might be the APIs from those services that are projected as MCP tools.
And we auto-generate the request pack, response pack, the descriptions.
Later, we also, based on the API descriptions, you generate LLM-optimized descriptions for the APIs and fields so that we can, like we don't, like we saw some tools with like, 3000, 4000 tokens so then we can optimize it for model consumption and it is just one single config and security policy change and then they can enable it they don't have to do any code changes if they want any custom MCP they can always lose we have like a framework for creating a custom MCP and proxy their custom MCP also through MCP gateway the other interesting thing we did was even all of our vendor tools whether it's Google, Slack, all of them are also exposed through MCP Gateway.
This reduced a lot of burden on configuration management since we always spoke about AI effects, right?
So I could install any of the tools in the same manner whether it's internal MCP tools or external MCP tools.
In the last few months, we also projected all of these MCPs as CLIs.
So you could execute any MCP as a CLI.
So then if some agentic hardness wants to use it as CLI, they can use it as CLI.
If they want to use traditional MCP, they can use MCP.
And do I understand that this all comes because the APIs to start with were already either generated or written in, like a lot of them were generated that you already had a platform for the APIs.
So adding MCP was somewhat, I mean, trivial.
I mean, of course, it's work, but you know what I mean?
And now generating CLI is also somewhat trivial because like you had a foundation before AI, right?
Yeah.
And then so when you have it, that's what I'm starting to see with Uber.
You built a bunch of stuff before 2022 and you could just kind of add AI infra pretty easily on it.
Yeah.
And the same team that works on our API gateway also worked on MCP gateway.
So they're already in the business of building API gateway.
So they know how to do any kind of protocol transformations, any kind of API gateways.
So it was like we were able to pretty quickly build this.
so i think this was probably like uh especially once we got our google workspace and all of our slack tools also into this it kind of like created all kinds of like amazing agentive workflows people could automate anything right at that point like from slack to google jira linear like any any tool vendor tool or any internal service uh is we are at a cli call away and then the other set of foundations as we like um started thinking about the space was the I think all of these go in this camp of like vendor neutral hardness neutral building blocks that we think like you know as like you would have to do this irrespective of what happens because you would need agents need some way to access your tools then the other thing was around skills like you know we jumped onto this earlier this year since We already have this notion of monorepos.
We created a skill monorepo in some sense that has all of the core skills that central platform teams build and all of the team-specific skills individual teams can build.
And are these skills, these are just like the skills folders, like the plugins that you can use for cloud code, etc.?
Exactly.
So there's plugins and skills that you can use across all the harnesses.
Then we had one single quality gates that can run across all of them.
and we made it portable across all of the harnesses.
So you install with the same AIFX tooling that I mentioned and like if you do plugin add that skill, it will update the configurations across all of the harnesses whether you are at that point using that harness or not.
So that even if you switch across harnesses, you have these plugins auto-installed.
you don't have to worry about like reinstalling them and reconfiguring them and so on and since all of the skills are using the tools from the MCP Gateway we also have one standard mechanism for all these skills to access any of the tools now we are working on like you know some tools where engineers can go select their skill it will tell them all kinds of tips to improve their skills the next thing that in in-flight that you're working is how do we then collect the traces from the hardnesses and generate improvement patches for the skills so that we can continuously improve the skills based on learnings because there's a lot of discussion around the skill quality, skill evals and how do we improve that continuously like when you first build a skill they might not be perfect and you need some mechanism to continuously improve that.
I think both the MCP and skills this gave us like lot of like benefits and like if I look at like in the last few weeks there's around probably 30,000 skill executions per day.
This is kind of interesting for me to see because if you told someone like oh we're building a skills repo like what most people would think inside a startup right I'm not talking about Uber but like oh you just do a git just a repo and you have a bunch of skills there just a bunch of skill folders and and that's kind of it but it's pretty clear that it's more complex right is it more complex because of how many people, engineers you need to support.
Is it because you're also running evals, because you keep cleansing these skills or tracking if they're used or not used, those kind of things?
Yeah, absolutely.
I think it's like now with any of these latest models, it's very easy to create a V0 version of the skill.
It's very hard to make it really good across all kinds of use cases.
And that's why you want to really make sure we have one...
one place we can like update all of these skills in fact when we exposed the CLI projection for all of our MCPs just couple of engineers using AI they were they updated all of the skills in this repository to use the CLI pattern because it's more token efficient because if every skill says I want MCP ABC MCP DEF then if I install these two plugins I'm gonna have like six MCPs across these two always in my context and it is so much context gloat right and like then we replace all the skills with the CLI pattern so that that had a like lot of impact on our like token efficiency as well and the other thing is because we auto install this skill marketplace for everyone it reduces one more step of configuration and so then like this the discoverability is easy the setup and configuration and management is easy and we could build a whole bunch of like eval pipelines around that, discoverability pipelines around that.
We still have teams that have like their own repositories but majority of the skills go through this.
My favorite is like context graph.
What is context graph?
The backstory is back in December like one of the engineers built an AI assistant that you can go to like we call it Cortana like you can go to web or you can go to slack and ask any question from any code base so it would do it and it would like search across all of our repositories and answer questions and then we go and then we saw people asking all kinds of questions like it could be research it could be like what is how to run this query like what is the data source for this who owns this like can you do RCA for this whole bunch of software engineering tasks and when we're looking at some of these traces Like it was clearly obvious that, you know, we could optimize the number of turns it takes for the models to get grounding because we have so many systems like historically, all of these are like siloed systems, right?
Like engineer has to go into like the task management platform, engineer has to go to our like on-call dashboard, has to go to like our UON dashboard that we talked about.
Then we're like, okay, how do we plug all of that into one mental map for the agents in just in the last few months?
and there's a cool visualization uh wow it's built like they call it uber context graph like we have uh 100 plus node types and like 130 plus edge types and this is live data as of like say like like three days ago if you click on any person node like you know you know what assets they manage what design docs they wrote what features they are working on what on-call events they like worked on what are the configuration changes what incidents they managed what jira events and If you take any node, it has a relationship to all other nodes.
And what this allows us is now the same assistant I talked about.
It has the context graph as the default grounding.
So if you ask any question, it will first use this to like navigate.
OK, like what is this person asking about?
Once you get grounded into all like this is about this domain, this set of services, then it can go deep and read other documents, read other things.
So it reduces your agent traces.
and the amount of turns it needs to give you a good answer and we already see like in a whole bunch of categories of questions the quality goes up the cost goes down the latency goes down and we have just invested in early in harnessing the full potential of this and like we are super excited about like all all of the things that we can build on top of this going forward And then the nodes, are they people?
Are they tools?
This is the schema visualization.
So every person in the company will have like, you know, like there's a node for me, a node for like every employee.
And then it will have relationships to all of the instances, like every PR that I generate, there will be a node for that.
And there's a link saying, okay, Uday created this PR.
Uday manages this asset.
I work on this product feature.
I contributed to this design doc.
So then the nodes are both people, but they're also assets.
Yeah, exactly.
Like one simple example is, let's say if you ask any question like how many mobility trips around the world are with cash payment type?
Like you first need to understand what tables exist, what columns exist.
Are there other people who wrote these queries?
Are they from science team?
Are they from mobility science?
Like now all of this data is interconnected.
Like you can go and check.
Oh, are there other queries that were done in the last 30 days around this and what kind of tables exist?
Go and explore those columns because again, all of this is interlinked.
Then you can quickly get to like the blessed queries that we have.
and answer these kind of even complex questions pretty quickly.
Otherwise, you would have to go through many, many docs, many, many tool calls.
This makes it really efficient.
It's much harder for us to manually write all of these cipher queries, but agents are really good at navigating the graph, like how they're really good at navigating file system with Bash.
This seems very futuristic to me, but it's pretty cool.
And I assume it's a ton of data to store just because Uber has like 10,000 plus people working there easily.
Yeah.
The good part is, you know, similar to how our API Edge Gateway team worked on MCP Gateway effort.
Our graph platform team that manages graph or our production graph with billions of nodes, they also helped build this graph.
So this is in compared to our production graph, this is much smaller.
So this is like, so it's not like a scale challenge for them anymore.
Yeah.
And we also talked, I already talked about like the AI assistant Cortana, which which kind of plugs.
uh some of all of the context graph tools skills and it's omnipresent in slack web cli so anyone in the company like even someone in ops in some country they can ask any question about oh how is this built where is this located who works on this what was the history behind this and like it is able to take all of this data all of our google documents and then and then give them like a really good answer and it breaks down kind of the information silos across orgs right like they're able to like self-serve ask anything execute any skill anyone has built and i guess it must have like access to the data sources so for at least for the right right people so it can go into like things like logs and and yeah monitoring those kind of things yeah it has logs monitoring you can ask okay how what is the health of this service it can go look at all of its dependencies what kind of storage systems it uses run all the queries and give you like a 360 degree health report for that asset.
And we carry forward all of the existing user permissions.
So like, so then if I'm only able to access the stuff that I had access to, otherwise that data is not accessible.
Yeah, because inside Uber, I remember it used to be called break glass when you added a permission for a system you didn't have access to.
You go through an approval flow, which goes through your manager or director or whatnot.
But I know Uber was really big.
At my time, they were really big on this already.
Yeah, absolutely.
So I think those are like some of the foundations.
We have some more in the workings, like we're thinking about some more even more futuristic use cases.
But you know, some of these building blocks really helped us in our agentic journey and helping it take it from just pure using it for just coding to all kinds of engineering tasks.
Would I just mention how Uber wrapped up AI foundations work but already more foundations work is to come.
Our presenting sponsor, TurboPover, knows this all too well.
They are rebuilding their storage architecture from first principles.
They are doing this to make more types of storage faster, cheaper, and more reliable at scale.
The new system called TurboPover v3 is not yet in prod, but it already passes CI.
Still, it's not nearly as performant as the current production version, v2, and so the TPub team is now perf-maxing query plans until they are as fast or faster than v2.
They're doing this in public, documenting how they increase performance all the way up to the v3 production launch.
For example, we can see how the team increased full-text search performance 9x by batching reads.
A lot of this is basic foundational work right now, but they will be sharing some very cool systems work over the next few weeks.
But hang on, why is the team doing a full rewrite?
It's because search is changing, and so is TurboBuffer.
T5 started as a Vector database, but it is no longer VectorDB.
It has evolved into a high-performance database for search.
The V3 discards the Vector Primary Index, and all indexes will share the same columnar storage system.
Every query will run on vectorized query engines, which is something you don't usually see in search databases, but which will result in both very fast search and just generally a very fast database.
The TurboPuffer team is keeping a work log of their journey, and you can follow along at turbopuffer.com slash v3.
It's a blast to see how it's being built.
I also want to mention our season sponsor Linear, the place of coordinated work between humans as well as agents.
One of the biggest changes for us software engineers is the sheer amount of code that we need to review.
Uber has built custom tools to deal with this, but so has Linear.
It's called LinearDiffs and it lets engineers review PRs right inside of Linear.
One thing I really like about linear diffs is how a diff sits next to an issue inside a project and so it's super simple to see all of the context in one place and know why the change has happened without needing to jump through different tools.
For large PRs, guided reviews walk you through the core change with an explanation of what has changed.
Guided reviews also group the glue code and the supporting edits separately.
These small but important changes make a large PR readable in one sitting.
And Linear also supports stack PRs.
If you view stack PRs, you'll know that they're a great way to break up complex work.
And if you've not yet used them, you'll probably love them when you try them out.
How we do code reviews keeps changing as AI agents take on more of the work.
These neat changes coming from Linear help us engineers understand code reviews more and expect Linear to evolve diffs to keep up with how we work more with AI agents.
To learn more and to get six months off of Linear, head to linear.app.pragmatic.
With this, let's get back to Uber and how building a feature looks like today in terms of tools used by devs at the company.
ERD and ERD, I guess for those who don't know, product requirements documents and ERD was engineering requirements documents, right?
Like it's like the product spec and engineering plan for those not familiar inside Uber.
Yeah, so I think we went through some of the AI foundations and like now if we think about the life cycle of a feature, then how can we...
truly uplevel each of those life cycle with the right set of AI foundations and also rethinking those tools for the AI era and some we are more farther ahead some we are early in the discussion like I think even the notion of a combined plan that agents and engineers can iteratively work together there's like a lot of cool ideas but this is kind of like you know um that we also have like all kinds of agentic skills for working with any of your design documents and as i was saying before like whether it's a product design doc engineering design doc launch reviews xp reviews production readiness reviews all of those things are in one place i'm like filtering some really old projects uh so that we can like uh look through them but uh like you'll know exactly like um who is working on them uh track the status of that um and all of this data also goes into the graph as well so that we know like full history of what are the source of truth design docs approval status who works on them and each of these also have like the jira link or the linear link so then we can trace from from this project how was execution translated to this gives like a source of truth system for both humans and agents to know the specs for every project.
Yeah, I guess planning is still really important, especially for something that is not just a trivial, you know, like one prompt or one shot project.
Yeah, exactly.
And because we need to, like, there's so many with the kind of product heterogeneity and country heterogeneity, like, we need to, like, really think about all kinds of corner cases more than, like, you know, the happy path.
You have to think about all kinds of corner cases and like how they should exist.
And since we also operate in physical world, the number of corner cases are like even more, right?
I would really think about, oh, what happens if this goes on?
What happens if this goes on?
How do we handle a support case for this?
How do we handle?
Yeah, I guess just needless to add, but I'll just add it because it might not be as obvious, but Uber operates in so many countries and cities and even between two cities, the reality might be different or might be different expectations or people are used to different things and so on.
yeah absolutely i think and that's why it really becomes important to think about the right plan up front and this will also make it easy for both the engineers or agents to take that plan and decompose and execute on that like once we have the plan then the next thing is how do you convert that into decompose that and execute now we're kind of going to what we call minion internally so this is our background coding agent so around 15% of all of the production PRs go through go through minion right now essentially you know this whole bunch of PRs that some PRs engineers want to do interactively but we also want to make it really easy once you have for the task that I have really good spec or I want to do minimal steering I can hand off that to to a minion and we had again we had we did multiple iterations this is the latest iteration we call it minion v2 but like you can use it from web you can use it from slack you can do a plan mode and then let it execute behind the scenes or you give one you can give the prompt once and let it execute behind the scenes if it's like a simple simple prompt and it has access to this is using the mega dev pod so then it has access to all of the repositories all of the code base that we have it can work across repositories and and generate One PR or multiple PRs across repositories based on the plan and spec that you have when we initially started Mostly maintenance kind of tasks used to be offloaded to Minion as we progress on this journey as we are adding more and more tools into the Minion environment where Minion can Self validate its work in web or mobile or backend API's our goal is we can We can increase the kind of tasks people can do with Minion over time.
Increase the percentage of tasks over time.
I think we can also see how it's integrated in Slack.
You can just mention, do you mention at Minion or what do you say?
You can do at Minion and then it will show you the status.
And you can also, you don't need to go to the web, you can steer it on Slack itself.
maybe you have like a slack thread with the with the em with the pm you can iterate on the design if it's a simple thing you are just iterating and get alignment there and then say hey adminion follow this thread and can you can you create a pr and then it will take the slack thread context and then create a pr for you so it's been really useful even in our support channels i would see like someone reports a bug the on call ask Minion, hey, can you see what is the issue?
And then Minion will come back with, oh, this is the root cause.
Oh, this looks simple enough.
Let's just fix it right there.
And then we would fix it.
There's no loop of adding it to backlog, doing it in the next thing.
You're able to fix it right there.
So we see more and more cases of that as Minion gets powerful as well.
So I think we kind of went through the build step.
We also before talked about DevBot.
It's improved since, hasn't it?
Yeah, yeah.
So...
like this is showing all of my devpods um since i'm working on context graph like you see you see that uh but uh yeah as you if you if you want to create a new dev pod you'll see like which flavor you want uh for for minion there is a devpod flavor that has all of the repositories but um and optimized for minion usage but for interactive like it will create a devpod and within a matter of seconds um you will have that running.
That's impressive.
The second I'm sure you do some clever stuff like having things warm or pre-built so you can copy those kind of things.
Yeah, it's like it's running on all of this is because this is developer stuff.
We have like a separate multiple clouds that we are using for this and it's like Kubernetes cluster and like everything is warmed up.
So then when you ask it, like we are able to just clone from the pre-built image.
and it has build caches everything repositories everything already cloned like it's ready for you to build uh so i'm since we use basil across all of our repositories for build system so i'm i was able to go into a repository like run basil for that and then just like we went we went through this in the order of this like you know seconds where any engineer like if they if i want to make a code change here like within order of seconds i create devpod It's one single way to build any module and then anyone knows this command.
And now agents also knows the same way, right?
In the CloudMD, AgentsMD, they know how to build, test any module in any repository and they're able to use the exact same commands.
Yeah, so I think that covers kind of like, you know, what the kind of stuff that you have to do in cogeneration.
So you can do interactive hardness, you can do like through minion.
The next thing is, okay, like you have the PR.
What do you do after that?
Then...
We will go through self-healing CI first.
One thing that we realized is because our PR volume has gone up year over year.
Oh yeah, it must have explored.
Yeah, our P50 cycle time, P90 cycle time, both for inner loop where you have the PR and you want to get it to a green state and create the remote PR.
Or once you have your PR created, getting the human review, getting all the CI.
things to pass i think review has become a big bottleneck uh same as like any other company um and we see year over year the delta and we want to like really we're like really focused on like breaking down every aspect of this and like how can we go really deep and optimizing it for the ai era so first thing that we did was we took a whole bunch of like simple ci jobs that used to run once you create the pr and we moved it to inner loop and we kept all of this behind one single command like AI effects validate.
So then when agent creates a PR, it can run the linter, the coverage rules, the build rules, the template rules, like all of these things so that the probability of this PR is in a good condition like goes up.
Then we hooked on ability to do visual validation where I can spin up emulator and test it.
I can spin up like a web and test it I can spin up like a staging environment where I can deploy this change run the apis and test it and now minion or even my interactive harnesses they know these agents know how to like have the skills for this so then they can go through this verification loop and let's say you create the PR and still something fails then we we have like a self-healing ci you can apply to all the PRs or per PR So this is like an uber specific line that we add to all GitHub PRs that you can link to the issue that you're working on.
You can see which checks are failing.
You can have a whole bunch of automations, auto request review, auto merge, auto fix.
Essentially everything that we can do to reduce the P50, P90 cycle times.
And once you do all of this, then we have this thing called code inbox.
This is showing a mock version of that.
where any engineer can go, they can see all the PRs that are waiting for them, all the PRs that they created that they're waiting for reviews.
So if an engineer sent this for review, I go to this page, I can see all the things that we did so far.
So it makes it really easy for me to get, oh, the AI code review passed.
These are the things it checked.
This is the sanity test screenshot.
So I know, okay, I know.
You can just quickly take a look.
You have a video as well.
yeah I can see the the bill checks and all of that and we're also adding risk score blast ideas of this PR and so on so all of this will give engineers really good signals to figure out okay like is it good to land and which parts of the PR I should focus on and yeah I think we're still like we're aggressively like working on this this whole bunch of ideas that we have but we want to make sure like you know as we are ready for the influx of PRs.
We in fact had to like add internal code red for our CI system because year over year the number of CI jobs also doubled or more than doubled.
So we had to like also like beef up a whole bunch of foundations in this space so that we can we can be ready for the influx of new PRs.
Yeah and I guess like you're also seeing that There's no way engineers can review all PRs with the same rigor as we did in the past.
So I guess you're building tools to signal which ones to pay more attention to, which ones might be low risk in so many ways, maybe in impact, maybe in blast radius.
What is your take on what will happen with code review over time?
Do you think there will always be human code review or for some things it might go away?
Yeah, absolutely.
see i think uh one is for us like we we want to err always on the safer side because the cost of an incident for uber is like really high um so even with all of this tooling we like right now we still require an engineer to review every pr that that goes that goes in once the more confidence we get in our risk scoring and we can also back test it with years of data right we have all of this data and the more confidence we have in our AI code reviews, then maybe for some class of low-risk PRs, we could enable, we call it green lane, like auto-landing.
But we want to be really, really sure about all of the dimensions that we want before we go there.
But right now, the main priority is how do we make it really easy for engineers to get all the signals they need to review faster.
If we do a good job there, then I'm sure we can increase the percentage of Autola and PRs over time.
But that's kind of P1, P0 is first getting all the signals in the right place.
Nice.
Yeah, but Code Inbox is nice.
I like it.
This is a thing that we should touch upon is maybe how I can do sanity testing.
And sanity testing, just for those who don't know the internal lingo, that's the manual testing, right?
Yeah, manual testing.
Yeah, I want to do, let's say I'm working on some new feature.
It touches.
two or three services, how do I know if this feature is working end to end?
So then typical workflow for engineers is like pre-agent equaled, we call it Slate.
Essentially, you can create your own instance that deploys the branch that you're working on and you can route your test account UID to the instance that you created.
So when you log in you are executing that branch.
Exactly, that branch and rest of the upstream and downstream goes to the other staging setups.
So that way you can test end-to-end without worrying about other parts of the infrastructure and this will carry forward across the service graph.
So you can say from your mobile app, this is the test account I want to test.
it will go through other services and when it touches the service under test then it will go to your slate instance if you're testing two or three then it will go to the service under test across all two or three so you can do even multi-service tests with this now that we had this we we had to just teach agents how to deploy slate and we also give it all of the past apis that that you can use Oh, nice.
Any of these services.
Basically, this was a thing where I, as an engineer, wanted to sanity test, meaning I want to manual test up my changes.
I had a way to do this in a pretty cool way.
And now we can just give it to agents and they can actually test their own changes end to end in a safe way.
Yeah.
So here we are showing a web version of that.
And you can see, in fact, like this is like the CLI pattern we went through.
It's using Slate MCP, create new slate environment.
and it's saying this is the environment name and then once it creates the environment it's saying deploy service to this environment and giving the service name this is a test service and now it had like it it has its own environment this is like the slate ui and it has like you know and if you see here you can pin uh test account uids um to that particular instance uh and you will see all the services that are in that particular environment you can see the gitsha And now, like the agent internally can like use all of these skills and tools.
So if you see like what it did was it did A-B testing of that web thing.
It ran agent browser in that slate environment with what is on main and what is on my branch.
And it created like a report for me to review the full set of changes that happened.
so then like you know after this we can go to the like a report that it generates like see generated a report that shows before and after and showing me like all the there's no regressions and this is very easy for like you know engineers to validate and we can attach some of the screenshots and videos to to the pr and the same thing we have for uh ios android web and if you want to do back-end apis then you can also test like back-end apis I just want to add like, you know, like one of the things that again, you might not appreciate if you're not outside of Uber, but Slate is so important because Uber literally is still today still has thousands of microservices or maybe not micro, but so to run a change on just one service when you want to do end to end, that's just difficult, right?
Like this is why Slate took years to build to start with.
Yeah.
And you also have to think about like the routing of the pinned accounts, right?
And when you pin the accounts, like this is the test account that I want you to route to my branch that is just deployed in a staging environment.
That's not trivial at all.
So like, in fact, how this works is one of the things that we built back in maybe 2017, 18, like it's called Jaeger, which is a distributed tracing infrastructure.
It's open source now.
So and all of our services have Jaeger spans.
essentially like whenever wherever the call originates there's a parent span and at every hop there's a child span and there is a you can add any baggage on that trace which can be propagated across the call stack so when you pin from this late web we are essentially adding saying that like for these are the services under test and this is the account that i want to route so as it is going through our service mesh service mesh knows this baggage And then it can say, oh, I want to route this request, not to this environment, but to the slate environment.
So because we had tracing built out and because it's like propagated across the stack, now you could actually test and we can reliably route these kind of test requests to the right environments so that it makes it really easy to test.
This thing took like years to add, to make sure that all existing services were retrofitted with this and all new services were built with this in mind.
So like, I'm starting to sense a trend where like, uber build stuff over years pre-ai just infrastructure that is easy easy to test for like you know humans or or easy to automate and now with ai it seems like oh you just built this like the last like year but actually you built on top of foundation that was built for years right yeah i'm sure i think if we did not have like system like slate it could have been really hard for agents to self-validate like then like you cannot automate this thing and like yeah as you said like this kind of stuff really helps and I'll also like talk about like one more like pretty interesting thing which is so uber specific and it's very very yeah and this is like very very hard to get right and like we had like even many versions of build like this in the past but I think this is probably our best version and we can like really scale it to all the teams but essentially The idea is, because we have, even if you look at our co-rider app, we have hundreds of screens.
And the rider app is what most people think is Uber app, right?
Yeah, Uber app, exactly.
Yeah, we have different business, different product lines, different cities, different, like a whole bunch of dimensions.
Like if I want to test any new thing, how do I know every combination of these has no regression?
If you were to like create all permutations across this, you would have to maintain thousands and thousands of tests across the stack.
And that alone is like again, really hard problem in itself.
So for some of our most complex platforms, like we call it Capture Replay, like essentially you can capture real production traffic with just one middleware, because now we have standardized language frameworks across all the services.
It's very easy to hook this into.
And it will capture not just the inbound and outbound at that service layer but also any of the downstream APIs that service calls those requests and those responses are also captured so in some sense you have like full request response and all of your outbound request response is captured now with all of these captured requests they go through some pipeline and then you can build simple rules that says oh if you see this kind of field in the request this is the value for that dimension.
So then you can capture all kinds of dimensions that that platform team cares about.
And once you have all the requests and all the dimensions, you can in some sense generate the full set of live dimensions and live combinations that that platform is currently supporting, which we call it match guard that I can run pre-deploy in our CI CD that shows for all of the PRs that are going in this deploy, let me validate against the full combinatorial set of combinations this platform supports and show the data before and after.
So then you get really high confidence if am I breaking maybe cash trips only in Brazil with this configuration, which would have been really hard for engineers to manually test or agents also to manually test.
So step one is you capture what you want to capture and then you have the system that will...
rerun it, check that there's regressions?
Yeah, so like then I'm going to one of the domains that has this enabled so you can see all the sessions that it captures.
I can go to any of the sessions.
Then you can see all of the metadata and tags.
So essentially it's saying, oh, this request is tagged as marketplace type eats.
It's tagged as line of business eats.
It is in this kind of terminal state.
It knows.
uh captured all the requests you can also see all of the outbounds that it used if we click on that it will also show you the um the request and response for the outbounds and if you see at the top there is slate workflow uid or search environment name um so then let's say if someone if i'm working on a bug that's only affecting for some reason san francisco uber reach trips um i i fix that bug then i can create a slate environment for that bug um for my code changes, I can go here and then say replay just this request on my Slate environment and show me the delta here.
Show me the difference and you're expecting a fix or some change.
Yeah, I'm expecting the fix because I know the before request, after request, what I'm expecting.
And then I can do it for a specific dimension or I can use it.
And then we also generate what we mentioned like match guard, which is the full set of dimensions possible for that.
So then you can create like a, in some sense, a testing automated testing suite for that team.
And also like more easy way to manually test one specific combination that you are focused on.
When I was there, we didn't have this, but we have something called Deputy and Cerberus, but those were, I guess, very early versions.
Yeah, so in fact, what Deputy and Cerberus allowed was ability to route any request from production to your testing instance and Cerberus allowed from your local testing instance back to the staging servers.
But now that we have Slate, we don't necessarily have to use Deputy and Cerberus because you can deploy a Slate environment and we don't need to use any of them.
I just want to stress because I think it's hard to see from the outside, but that's really hard to build this because every single, again, you have thousands of services.
All of them have to support these things from routing, for example, to capturing production, to replaying, right?
Like this was a lot of work, I assume.
Yeah, and now I think the same, like similar to how we exposed like Slate to agents, we can expose all of the Capri functionality to agents so agents can...
check, oh, I'm trying to fix this.
Let me get all of the sessions of this type and then see what request looks like.
Can I compose a request of that type on my Slate instance and see, did it fix?
Oh, did it not fix?
Let me go in a loop until I fix this.
Yeah, yeah.
And now the agent can, even though it cannot see, but it can see the delta and it can tell that nothing has changed, for example.
So it can get feedback loops.
Exactly, exactly.
That's awesome.
that that helps a lot now like the same it can agent can also do capture replay on slate uh with any of these we already talked about like the edge gateway edge gateway that's just the kind of uber's internal api right like we just we it used to be rt api real-time api but now it's called edge it's uh it's a go verse is it rewritten in go if i remember right yeah i said it in go and now it supports a whole bunch of endpoint types so there's a sectional api where there's grpc webhooks graphql so like again all apis go through one single entry point um where we can build a whole bunch of tooling around that on monitoring on alerting um and like even there is like a um like with this all the reliability features the edge also has its own capture replay similar to the capri it's using capri behind the scenes and like if you want to look at reliability of anything it can like we look at the entire service graph for that service and tell you what is the potential unhealthy node that we need to look at the with the ager tracing we know exactly the service dependencies we've also automatically tagged which is a required dependency which is an optional dependency so that metadata is also there so as we are even only looking at alerts and diagnosing alerts all of this rich context really helps both engineers to debug faster and also agents to debug faster.
Yeah, we already talked about experiments.
The last thing we will cover is like what we call Shepard and Fox.
After we moved to Monorepo, like then we moved to Monorepo because upgrading libraries was very hard, right?
Yeah, on all of the different repos.
Yeah, exactly.
Then you move to Monorepo, but then now you need to someone has to go and upgrade this.
library usage across all of the modules in that repository.
So how do we automate that?
So we first took a stab at like using open rewrite policies saying, if it's a simple like, you know, just the signature change, you can write that in open write policy and then Shepard knows what are the unique modules owned by different teams and then it automatically creates PR for each of those modules.
When AI came into picture, then we could add like a AI based Shepard where okay if I want to do fixed context propagation across all the services so that tracing works better I can create set of prompts and skills that then agent can run across every module and fix all the issues in every service faster so this is kind of a large scale changes whether you want deterministic changes or whether you want like agentic changes across the modules that gave us like we were able to do a lot of migrations over the years through this and we call each one as a campaign and if you see JUnit 4 to JUnit 5 there will be a YAML file with the policy and you'll see a whole bunch of PRs created for each module and you can a campaign owner can track it the status of that and and go through that so it makes it really easy to manage these migrations Few months ago, then we took the same platform and added like one more feature.
We also want to make it easy.
We want to continuously improve the code base, not one time migrations.
Let's say I want to like code maintenance.
Yeah, code maintenance.
So we call it Fox.
Essentially, you want to create self improving code bases since we already have a whole bunch of agent skills.
Essentially, any service at Uber, they can go here and roll their service into different skills.
They can say, I want dead code cleanup or performance analysis or noisy logs.
There's like a whole bunch of these skills that you can onboard to.
Oh, smart.
And what Fox does is every Monday or essentially every Sunday, it will, when we don't have that much CI load, it will generate a new PR for this service using dead code cleanup.
Like, oh, can I...
clean up any code?
Can I improve the performance of this?
Can I fix any noisy logs?
So it's kind of automated then code base improvement.
Then it's showing you how you can select.
I like it because I heard people do this thing where they would set an AI loop, let's say in cloud code or codex saying, look for ways to improve my code base and they would execute it every night or once a week.
But it sounds like this is just a lot more yeah like uh optimize and i i love that you can enroll your code base because everyone knows like a code base is owned by an engineer or team in aduber right yeah exactly and and uh like you can see like i mean because you're rapidly scaling we have we have still in the opt-in phase uh once we improve some of the skills we'll move some of them to opt-out phase uh but I mean anytime you obtain a skill for a service then every week in some sense you are getting a PR created like a managed agentic loop so then you don't have to like you are continuously improving this in some sense the more toil we reduce the more amount of time engineers can spend on the true feature work so like constantly thinking of okay how do I eliminate more toil from engineers and like and automate all of these things and So we constantly look at all kinds of PRs with maintenance category PRs.
Do we need to create a new skill?
Do we need to improve that skill so that that class of PRs can be onboarded to this automated managed loop?
This is nice.
I like it.
And then finally, I heard about wisdom.
What is wisdom?
Oh, yeah.
So we use wisdom because all of our employees dog food are products.
so if you're using a rider app or earner app and if you see any bug then you can from the app report that bug we now have all kinds of ai stuff where like it will automatically route it to the right team based on the stuff and then when you report a bug because it's from the employee from the employee like beta apps we can capture a whole bunch of logs and telemetry around that that makes it really easy for the team owning that feature to fix it so Like in this case, it's like an internal feature.
It's essentially like if you want to test any feature in a different country, you can mock the location on the device.
So if someone reported that issue, the first thing is, okay, now how do you reproduce that?
Before it used to be a huge problem to reproduce.
So now we know the set of screens the user has gone through because we have that in the logs and then we can give the engineers like this is the reproduction steps and this is the expected behavior and this is the observed behavior and so that we can boil down to the issue quickly and all of the telemetry that we collected from the app you can see all the logs in one place so like whether it's analytics network requests all kinds of feature flags So that also gives a lot of context.
And now that we have all of this context, we built a managed agent called debug assist that looks all of this looks at the code base and it will try to pinpoint what is the root cause and like what is the impact evidence timeline around that.
Now, once I think, OK, this is the right root cause, I can hand off this to Minion that we talked about.
Yeah, coding agent.
This root cause looks good.
like why don't you create a PR for this minion will take all of that context that we have it creates a PR it will attach that PR to the Jira task the Jira task also has the root cause analysis so anyone who's working they know the root cause analysis if someone updated the root cause on the comments or like create a new PR again all of that is data that we can then figure out okay how do we improve the debug assist how do we improve minion we didn't get the right RCA or we didn't get the right PR from the RCA like both will be able to continuously improve and then so do I understand correctly that you can capture and reproduce bugs so much easier right yeah and we also added ability to because we for the employee reported bugs we have all the logs I can take my test device and I can ask it to simulate what that employee has experienced.
So then once I give the bug UID, we can essentially serve the same network responses that we served to the person who filed so that I can experience the same issue in my test device app too.
So that way, then once you fix it, then I can see, okay, this actually got fixed.
And then it makes it really easy to reproduce.
I love the recording of the thing because that's, I mean, so important, right?
Like it's hard enough to do even on the web.
You need to build like custom tools for yourself if you want to do a website.
But it's awesome that you did it.
Yeah, I think this helps a lot like then in reproducibility and like getting to the right root cause.
So you built all of these systems and especially last year, it seems so many brand new systems that are just doing things like, you know, they're maintaining.
the cold quality agents are are validating themselves how do you think the job of software engineers inside of uber is changing and what are you learning from all of these new tools it seems like you're learning yourself you know like in some sense like if you're in the agentic ai world like you're constantly trying to find the next bottleneck um and trying to like remove that and figuring out how you can like both using ai and both using like what's it systems engineering like reduce that bottleneck because ultimately our goal is to maximize the number of features that we can ship across all of our products and that will require us to look at the entire life cycle how do we rewire that for the new world and what kind of tools that we need to build and i think this is going to be like an ongoing journey right and there's a lot of discussion of you know this concept called capability overhang where i feel like even right now like there's so many things that we could you leverage the latest models for but we don't have the sufficient orchestrating capabilities or like the harnesses or the tools around that to be to leverage those models in the right places in the right way i feel like the more we work on this the more we can unlock um the more features we can ship the more productivity that we can unlock and like we are trying to measure a whole bunch of this and like constantly think about like what is the next set of foundations that we need to build to make it easy for engineers to work and to make it easy for agents to work.
You mentioned productivity.
How do you think about engineering productivity?
Because when I was at Uber, this was again, in like up to 2020, us managers did look at things.
It wasn't the main thing, but we did look at things like diffs per engineer just to show, you know, if someone didn't contribute for like two weeks, it was like, what are you doing here?
Or if someone did a lot of code reviews, that's actually what we wanted to notice.
But it sounds like a lot of those things, the code output, the code artifacts are less relevant potentially.
What is a productive engineer in your mind right now inside of Uber or inside of your team?
Yeah, so I think before the AI world, we were measuring the standard stuff like code throughput and stuff like that.
Those are the easy ones.
we would we also have like every few months we do the developer experience surveys we look at nps you know like sentiment across all kinds of dimensions we look at our code quality dimensions what's our incidence per def what's our code coverage and all of these metrics right um i think now once you go to like yeah i feel like we can leverage like the new set of capabilities to get more nuanced into this discussion like Because now we are internally talking about not necessarily just diff velocity but feature velocity.
Like how much of all of this additional throughput is going towards unlocking new features and how can we even change our organizational processes so that we can reduce the P50 feature time.
Because like you can create 100 diffs but if they are not all feature diffs then like maybe it's not necessarily increasing the value that we can provide to the users, right?
So like one for example, one of the things we did is tagging all of our PRs to different categories like on complexity level and is it a feature PR, bug fix, maintenance, testing, documentation on different categories and we could see year over year how is that mix changing.
Luckily like at least like right now I can see our percentage of feature PRs is going up, our percentage of maintenance PRs is going down.
So that gives us good indication that Okay, we're in the right direction.
Okay, we see the code volume go up and the percentage of PRs that for features going up.
Then we are investing a lot because before even though we had the U-Plan tool, sometimes you might not have the right tracing from this product design dog to this engine design dog to this particular task.
So it becomes a bit harder to measure what is the cycle time into end reliably across all the features.
So now we're using AI to like automatically cluster these things and like tag any missing things so that we can reliably get what is the p50 feature velocity for different complexity of features for every half over the last two years three years and then we can go into okay is the bottleneck in the prd phase is the bottleneck in the launch phase is the bottleneck in the rollout phase what is going on there and what kind of things we need to invest to accelerate uh the number of features that we can ship per half And how is your mental model changing of software engineering compared to pre-AI and pre-T's tools?
Is it still the same job?
Have things changed?
I mean, at least in my total tenure as in engineering, there's never been this much change to how I work.
Like, for example, like, you know, we say like all the engineers at Uber are hands-on, like even distinguished engineers at Uber are hands-on, they like, they create PRs.
and i used to think i was hands-on engineer but my hands-on definition year over year has completely changed i'm like even personally i'm doing like way more prs than what i was doing last year like it's more easy to do uh all of these things all of these things right and i'm able to like explore more number of ideas than what i could possibly do in the past but the job of the engineering is you know in some sense how do you build the right context for the agents to produce reliable output.
Like in some sense, like you need to get really good at context engineering.
Before I was operating with some context to do some tasks.
So I'm always thinking about, okay, if I want to offload some of this work to an agent, what kind of context it would need and how do I set up the environment with that right context so that it can produce that work.
Then you have to become really good at like, you know, how do you give the right specs, right constraints for the agents apart from building the right context.
then multitasking has become really like a thing before if you're like it would take so much time to create a PR so like at least laser focused on that but now you're constantly switching between so many things like so you need to become really good at multitasking across all the sessions figuring how do you scope the how do you scope the work how do you track the work how do you verify the work really quickly because if you're not able to verify the work and like pick any code smells from the agent then um then that's hard then you're shipping like some like some recreations into production right so um like the context engineering ability to multitask ability to like review and like verify the outputs these things have become like much more important uh now because you can because you're able to paralyze a lot more than before one thing i'm picking up is maybe a new primitive is we used to like build software thinking how do I build this software and how do I verify it?
Whereas now the question that you said is often how do I set up an agent to be able to build this and to be able to verify it?
So it's kind of like almost a meta.
Exactly.
Sounds like there's more of that, right?
Yeah, I'll give you an interesting example.
I was working with some of the engineers that work on our customer support team.
We are going through the journey of...
Having AI agents handle some of our customer support policies that process would go from oh You have some a run book that human agents would follow So then from that run book you are going through a whole bunch of steps to create a policy that AI agent can follow Then you have to think about okay.
There is some dynamic elements here then which API's I need to call call to get that data which attributes I need to use Then I need to configure my system, change bunch of configurations.
Then I need to do manual evals before launching.
Then I need to do continuous evals post-launch.
And then in the last few months, we were able to take down all of the steps.
How do I now take all of the context and all of the principles that I want to use for the policy conversion and teach agents to be able to do that so that I have a really good V0 policy that an agent can create.
then you automate the all of the configuration setup manual testing finding the right apis finding the right properties so that we can hook the right dynamic elements into this again like that used to take weeks now it's like order of days for any polls then instead of doing like manual evals then how can you like use all of the past context to create synthetic evals on top of that so that you can like you can accelerate you can deploy this and then verify this quickly Like essentially what we're discussing, right, which is now you're not, instead of doing that work, okay, how can you build the right tools and harnesses on top of this so that you can automate the workflow?
Obviously, you need the right guardrails and human verification steps in between, but you're designing a system that can do these things instead of like relying on so much manual work across all of these steps.
And it sounds like you're learning a lot by, because these are all new concepts, which you're now using, you're learning about it, you're doing limitations.
I see what you mean by more learning, more PRs.
And as closing, what is a tool that you really like using?
This can be a digital tool, this can be a non-digital tool, something you find helpful.
Yeah, the last few days I've been hooked to some of the personal agents like Instinct, Mios, GrokBots of the world.
I'm playing around with all of them.
It's helping automate a whole bunch of manual work that I do in my personal life.
In some sense, this follow the same process, give it all of the context it needs to do that task for me, give me all of the inputs I need.
Then I can have human in the loop verification step and do that task.
But it's helping a lot to automate all of the manual personal work.
Awesome.
Uday, thank you so much for walking through all of the internals that are being used at Uber.
I found it super interesting, especially that I haven't seen this for quite a while.
I hope you enjoyed this look into Uber's tooling as much as I did.
It's now been six years since I left Uber, so a lot of the tooling that Ulay showed me was also new to me.
Uber was known for having about 5,000 services with about 2,000 engineers 10 years ago.
But since then, the company started to consolidate services into well-sized services, and today it has fewer microservices than they did a decade ago.
Today, it's in the ballpark of 4,500 to 5,000, even though these services are not as micro, and they're much more well-sized instead.
One piece of software that I really liked is Capri.
the event capture and replay software.
Back in my time at Uber between 2016 and 2020, we had two tools called Deputy and Cerebrus that could do some level of production data replay, but it wasn't all that easy to use and it was just really hard to capture real base to start with.
With Capri, it's far more trivial to capture sessions than replay them, and this works really well to reproduce bugs and also to test that a fix works for a certain bug fix.
Building a system at Capri is especially complex when you have closer to 4,500 or 5,000 services running like Uber has.
Another thing that was interesting is how we're seeing where CodeReview is headed.
It's slowly dying everywhere.
But looking at the tooling Uber built around CodeReview with CodeInbox, Every PR still gets merged after a human engineer decides that it's ready to do so.
But Uber is building a lot of automated checks.
Many of them are with AI to give more signal about the riskiness of the change and the quality that AI agents make of it.
And so in the end, it's a lot easier for engineers to decide which PRs they should spend time looking at it in depth versus ones that are safe to approve with the symbol LGTM.
AKA you might decide you're not going to read the code.
Finally, I had the sense that Uber investing 10 plus years in developer productivity is really, really paying off with their AI integrations.
For example, they built cloud development environments called DevPods back in 2017.
Uber got to reuse almost all of this infra in building out their cloud agents called Minions.
The same is true for existing systems to validate software, to safely rule them out, and now agents can use these as tools or as feedback loops.
So it feels like companies that are getting some of the most benefit from AI are the ones that have had developer platform teams pre-AI as well, at least in the case of Uber.
Do check out the show notes for more of the Pragmatic Engineering deep dives on Uber's engineering culture.
Uber is probably one of the companies I have covered in the most detail so far.
If you've enjoyed this podcast, please do give it a rating and tell your colleagues and friends about it.
Thanks, and I'll see you in the next one.