# Scaling Slack and GitHub: Architecture, Load Testing, and AI

**Podcast:** The Pragmatic Engineer Podcast
**Published:** 2026-11-05

## Transcript

How do you scale a product that millions of people use daily when the architecture underneath it is starting to break?
Today's guest did this both at Slack and GitHub.
Modlam is a principal engineer and the author of the book Refactoring at Scale.
She spent eight years at Slack, worked on database scaling at GitHub, and improved time-to-first token at Cursor.
Today we cover...
Low testing with 500,000 WebSocket connections, a job where using headless Chrome instances were too resource-intensive and too expensive.
Why GitHub had only 16 database charts for all pull requests and issues, and why scatter queries were a massive headache inside a large Rails monolith like GitHub's.
Why platform and product engineers should rotate between teams, and more.
If you want to understand what scaling up looks like inside of products you probably use every day, this episode is for you.
This episode is presented by TurboBuffer, Vector and Full-Text Search built on object storage.
It's fast, cheap, and extremely scalable.
This episode is brought to you by Antisysys, the platform to release with confidence.
In today's episode we'll talk about rewriting master production systems as they scale up, which is a kind of engineering challenge that's rebuilding a 747 mid-air.
But why not just use CodeReview when using AI coding agents?
It's because CodeReview is an increasingly naive and ineffective approach to do so, which is why Antisysys goes well beyond this practice.
Antisysys runs your whole system in a hostile simulation.
This simulation includes both targeted testing and fast testing.
By running this simulation, it finds every bug before your users do.
And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it much easier to fix issues.
The first thing I thought when I heard about Antisys is that automated bug discovery and fully deterministic testing sounds like science fiction, but actually, it's hardcore engineering under the hood.
Jane Street, Supa-based Rapid Railway, and Datadog ship agent-written code with full confidence because they know it's been verified by Antisys.
And to give a sense of how Antisysys helps with large production systems, Datadog uses Antisysys to verify their new architecture when rewriting the Datadog event platform, a system that collects more than 100 trillion events per day.
Today, Antisysys is one of many correctness strategies being used as part of Datadog's harness for their coding agents.
To see more case studies and details, head to antisysys.com slash pragmatic.
Mode, welcome to the podcast.
Thank you for having me.
It is so nice to have you here.
This is a bucket list item, very much so.
Well, this is the first in the podcast.
Usually I ask, start with a question.
I'm not going to start with a question.
I will show you a photo and I will ask, what am I seeing here?
I'm seeing a baby who looks like, how many months?
Maybe about a year old.
Maybe a little before a year old, yeah.
sitting in front of a computer.
We're also putting this on the screen if you're watching it on video.
Same haircut.
So my question is, were you born with computers?
Very nearly.
There are pictures of me much younger than that, actually, maybe like a couple months old.
I truly like I'm barely keeping my head up where my dad has me sitting in his lap, like resting against him.
And he's I don't know what he's doing.
He's probably like.
grading some kind of assignment or um because he was teaching he taught computer science for 17 years before we moved to the U.S.
and uh yeah there's a bunch of pictures of me like holding ASCII tables um which now in hindsight is hilarious um but I ended up doing this sort of by accident um you'd think that all of the variables and the environment would have led to like a very logical sort of linear career trajectory.
And that is not at all.
So anyway, he had to think about that like eons before.
But I thought it sounded really boring.
I was like, no, no, no.
And I really liked my bio classes.
I thought I was going to become a genetics researcher.
That was my plan.
And my parents always told me to chill the heck out because I had plans.
I had when I started university.
I started in microbiology.
I signed up for molecular biology, organic chemistry, all the things.
I was already I'd already accrued enough AP credits for all the Americans listening that I was technically like a second year college student rather than a first year.
Oh, yeah.
Oh, wow.
Rather than a first year college student.
You want to speed run college.
Oh, yeah.
I was ready.
Skip the fun years.
Yeah.
Get straight to work.
Which I kept doing that.
I had like a 19 credit semester at some point.
I was like, why did I?
do that.
That was such a poor decision.
But I mean, I had fun doing it.
So I guess, you know, my brand of fun is, I guess, different from other people's brand of fun.
And yeah, I was going to do that.
And then like three weeks in, there were a couple of things that really bothered me.
One, it's not because you're good at something that you like doing it.
And I think I conflated the two for a really long time.
I mean, to be fair, when you're 18, like who, like, what do you know?
But I conflated being really good at memorization with liking memorization.
And I don't know if you've ever taken a molecular biology class, but it is a significant amount of memorization.
And you're learning the Krebs cycle.
So I was going to switch into organic chemistry, into chemical engineering.
And to do that, that's a different faculty at McGill.
I mean, it's a very big school.
There's faculty of science.
I was in the faculty of science.
There's faculty of engineering.
And I was looking at all of the requirements for first-year engineering students.
And there was a programming class.
There was an intro to computer science.
So I swapped out one of my bio courses the next semester with the intro to computer science.
And I told my mom I swapped it out for a computer science class.
Please do not tell dad.
Please do not tell him that I swapped it out for a computer science.
Like, I don't.
I'm scared I'm not going to like it.
And then he's going to be sad because he loves this so much.
This is like true love for him.
And she was like, sure.
OK, I'm pretty sure she told him like.
two minutes later.
I love her.
But like, man, I think she knew that it was not going to be, you know, a wreck.
And sure enough, eventually, like I tell them I'm taking this class and three weeks and I'm like, oh, I really like this.
Complete opposite from the molecular biology class where I was like, I really liked it.
It was like, oh, with the power, I'm sure you've had this feeling.
It's like, I only have a computer.
and I type into it and then it does things.
I feel like a god.
And then your first job, you did internships, but your first job, which followed the internship, was at Rent the Runway.
It was, yeah.
It's a pretty well-known company.
Thanks to anyone who's entering management will probably know of the book, The Manager's Path by Camille Fournier, and she was at Rent the Runway.
She was.
Yeah.
And actually, that was a big part of why I wanted to go back after my internship there was because I you didn't see very many women in those CTO type positions at the time.
Also, like, I mean, I my mom very nearly became a fashion designer.
And so we grew up watching Project Runway from the very beginning.
That was like the one thing I was allowed to like stay up late to watch was Project Runway.
And.
So it combined two things that I loved.
It combined, you know, programming, computer science.
I was mostly doing front end stuff at the time and fashion.
I could just like rent stuff for dirt cheap and like walk around in a Manic L'Eau-Livier like ball gown if I wanted to.
It was it was so cool.
I think there were so many women on the team in general, I think partly because of the space that the company was in.
But I also just for folks who've had the opportunity to be in a room with Camille, she just commands this presence.
And I had so much respect for that.
And I mean, even if, you know, I was many levels removed from her, I was like, I want to learn from this woman.
Like, I really admire this.
this presence that she has and this confidence.
And so it was an easy sell.
It was a very easy sell to go back.
But then she left a couple of months after I came back.
But she was still instrumental in helping me land a job at Slack a year later.
This is Slack in 2016.
So at the time, I vividly remember because in 2016, I joined Uber.
We still had HipChat, which was Atlassian's Slack offering or like Slack competitor.
It was used by a lot of companies.
And then they suffered two terrible and undefensible data leaks.
Uber almost immediately migrated off to a product.
And I remember that they evaluated Slack and Slack would not scale.
I thought this was later in 2019, Uber would move to Slack.
But I wrote a thread about this on Twitter.
And I said, like, well, we built something called uChat, which was built on top of Mattermost.
Yes, yes, yes, yes.
And I was thinking, like, did our in-house team just want to build something?
And did they just, like, say that Slack didn't scale?
But one of the engineers who built uChat, because they were working hand-in-hand with Mattermost, they were actually expanding it, the open source offering.
He actually said that they sat down with Slack, and Slack said that they cannot support 50,000 or 40,000 people in one room.
at the time so there was no way so like just as at the stage in 2016 slack still maybe struggles to serve more than 10 000 people in one room if i remember 20 000 or 30 000 i think 30 000 was comcast 30 000 but they can do 40 000 so this is the state this is the state of slack this is early slack it was it was already very wildly popular people love the the onboarding teams was in infancy i'm not even sure if it was born at the time so yeah i don't think it was you you get to slack yeah yeah and what were the problems you had to work on that you got to work yes yeah um yeah i very much got the the performance itch from from at the runway it just wasn't i just had to work on it you know on my own time but i still enjoyed doing it and then when i showed up at slack um i my onboarding buddy was um just like very senior engineer who was working on performance issues specifically at the time.
And I burned through some of my starter stuff.
And I was like, what are you working on?
What are you working on over there, Moysha?
And he was like, oh, let me send you some tickets.
Like, do you want to work on performance?
I was like, yes, yes, I do.
So he sent me a couple things, like relatively easy stuff.
In startup land, when you...
need to get to the next place and just get that next customer or build that next feature that you think is going to capture.
Like you make a lot of short term decisions that, you know, probably inside your soul, you will revisit at some point.
And a lot of the performance work that we were doing at the time was just a lot of revisiting of shortcuts.
I mean, like well calculated shortcuts, but like shortcuts that we knew were going to like bite us in the butt eventually down the line.
So a lot of that work.
I was doing.
And this was before the like enterprise, the Slack enterprise product was launched.
But it started a couple months before GA.
IBM was already trialing the product.
They were sort of on beta there.
They are a large company.
Lots of people work there.
That's 100k plus people, if I recall.
Yeah.
So one of the distinctions actually was it was sort of fine to have.
Tens of thousands of folks on one in one channel at one time.
And we were sort of spreading them out across different teams within the same Slack organization.
Obviously, they could DM each other among some other things, but that sort of worked.
But it was IBM was sort of falling over on a regular basis.
And I think.
I was not part of these conversations, but I get the impression in hindsight that when we agreed to GA, then they agreed that they would start scaling and adding more users to Slack.
Like once we reach GA, you know, we can't hold the reins on you adding more people as much.
So we hit GA and they were started adding more people like pretty aggressively.
We started, we ended up having to spin an actual team up that was just the enterprise performance team.
I ended up being able to join that team because I just hadn't to be working on odd random performance things.
I was definitely the most junior person on the team.
But I love that.
I feel like I've spent so much of my career to date being the most junior person on the team.
I mean, less so in the last couple of years.
But that's how I've learned the most.
And it's really just pushed me out of my comfort zone over and over again.
And it's just like it's the most fun.
And can we do a bit of a Slack in the 2016-2017 Slack Architecture 101?
Now, this is, as we know, Slack has changed a lot.
This is not how it will look like today at all.
But when we think about Slack workspaces and channels, private groups, DMs, behind the scenes, how was it built up in terms of databases, in terms of what happened when you sent a...
a message to someone or to a channel in terms of fan outs or rights?
So the pattern before Enterprise and still with Enterprise was everything belonged to a company, a workspace, a team unit.
And all of the data also was sharded by team.
And so if you were on a really big team, well, we just would give you a bigger database.
And eventually...
You run out of the largest size of database that you can throw money at Amazon for.
But you're saying that there was a time where this charting worked where...
a large company might just have its own database instance.
And for a while, they would work.
They didn't fill it up.
Yeah.
And smaller companies which share, they would be multiple on the same multi-tenant on the same database instance.
But you as a customer, your data was not spread across multiple database charts.
It was just on this one.
It was also how we distributed messages.
So for the WebSocket connection, when you were connecting, logging into Slack, establishing that WebSocket connection.
You were establishing that WebSocket connection with a server that was associated with your company.
And so if everyone at your company was very busy, very busy chatting on Slack, you all were on that one same server.
And again, like if we had really, really busy customers, eventually we could just say like, oh, well, this is your instance of the chat server.
And that was mostly fine.
for a very long time.
Enterprise started complicating that in some special ways in that we, I mean, again, startups, you like sort of make decisions that are fine in the moment, but then sort of don't scale.
We sort of made the assumption that like inside really big companies, it was smaller companies.
And so like marketing at say IBM is its own company.
So you sign into the marketing workspace.
at IBM or the engineering works.
I mean, engineering actually is too big.
It's some subset of engineering.
It sounds like it's a little bit of wishful thinking that you could just turn this large problem into smaller problems, right?
And then you can just make your current architecture work.
But there still needed to be like an overarching body.
So we created an enterprise, which was really just another row in the team table that just said, is enterprise one?
And then all of the teams associated with that enterprise had an enterprise ID.
And you still within IBM would want to say, you know, say you were working on one of the engineering teams.
I was working on a different engineering team.
We want to collaborate.
I want to be able to DM you.
We're at the same darn company.
I should be able to send you a message.
We pretended that these teams within a company were their own companies.
But then we had to create this other entity that overarched everything where you could still cross teams.
And that got very complicated very fast.
But then the part that became really terrible was everyone across all of IBM, whether they were on the marketing team, the engineering team, whatever, were all querying the enterprise shard for all of their DMs and the big company-wide channels, all of that stuff, all at the same time.
And then within that data, when you're booting up Slack, you have your channels that show up in the left sidebar.
You'd think that it was just like, cool, get, you know, for user ID.
get all channel membership entries.
That was not the case.
And I think it came from a good place in that DMs seemed like a separate thing from a regular public channel, right?
Like they seem different.
And so that was a separate table.
And a public channel is a public channel.
And so that was a separate table.
But then private channels.
seemed different.
And I think like from a product perspective, you can totally see it.
And from a product security perspective, you can also totally see that.
You want more scrutiny when you are querying for private channels.
And so that was a separate table as well.
And then group DMs came about later and it seemed like they were just sort of like a private channel.
And so they were just glued into the private channel table.
So when you were making that query to populate your sidebar.
You were actually doing a big, big old join union.
So what happened once a team had the idea or figured out that these tables don't make sense, but now you're talking about a lot of customers, including the likes of IBM, just putting a bunch of load on it, they're using it.
How do you migrate from these three tables that are actively being used into...
One.
All the time.
Yeah, I think it came about from calculating mentions.
So calculating mentions, like to get the badge icon, was also a very complex query whereby you had to join across all these tables, very latency sensitive.
And we had gone through the process of doing as much as we possibly could at that time to ease the load.
And in one of my teammates.
doing some work on mentions, he was like, we got to join these tables together.
It will not, like, there's only so much we can do before we have to like really start digging the hole.
So I was like, sure, yeah, I will endeavor to do this very scary thing that no one else wants to do.
That seems totally fine.
I have so much appreciation for this in hindsight too, in that if you look at Blamework on GitHub, which is the PHP framework, the Cal Henderson, open source after writing Flickr, it will explain many things about what Slack was like early on.
Yes, I highly recommend it's a good I think it's still there.
It's still there.
It's a good read in terms of like, oh, this explains so many things.
So we had everything was super procedural, for lack of a better term, in terms of how code was organized.
We had sort of like API entry points and it was like.
chat.login or whatever.
Right.
And it was like, OK, this is the API and this is all the code for the API with some helper functions in there.
There were a wild amount of functions that were like many, many, many, many, many lights long.
But there was just like straight SQL queries like embedded inside the code.
Like an ORM did not exist in this world.
Whatever you want to call your data access layer.
We did not have one.
And there were actually.
There's a lot of opinions about whether we should even do anything to hide the SQL queries.
And yeah, now, I mean, I totally understand why they had that stance and why we did the super minimal approach that we ended up doing, because it is so easy to write a SQL query that will shoot you in the foot over and over and over again, unless you're forced to actually understand what it does in the context that it is being called.
But we had like 400 something call sites that called into any number of these channel tables.
And at Moise call sites, these are SQL instructions in the code.
Just like select star from Teams channels where user ID blank.
Yeah.
Where team ID blank, user ID blank.
And so you just copy paste the pattern that exists, right?
That's always the problem with migrations is like.
You have to make it so clear that like this is the old pattern.
Stop doing this because there is still while it's going on so many examples of it in the code.
So I had to write some.
I started off by going through doing some kind of regex approach where I was finding all the places that we were querying for this.
Thankfully, the table names were like pretty easy to find and identify.
Got very lucky there.
And then I just one for one took the call and just pasted it into another file.
Yep.
That is.
All I did.
You created a layer that did the same thing.
Yeah.
Just a class, basically.
Yeah.
I didn't optimize anything.
I literally just copy pasted.
And now I was calling into a function instead of straight up calling SQL.
And you had a minimal abstraction layer, if you will.
Yeah.
Which already was like a little bit spicy.
But I tried to make sure that like the function name was like real darn clear as to what it was doing.
And I got the spicier folks to be OK.
with this approach.
So I felt that felt like a win.
And so I migrated everything over.
Everything was feature flagged.
Basically, I had I don't know how many feature flags at some point, but I had so many distinct ones just to make sure that I was doing it as carefully as humanly possible.
Even then, you know, I still like definitely borked some things along the way.
I accidentally when I thought that we were basically there and ready to start flipping on to prod traffic.
I'd run all the backfills because obviously like I was moving everything to a new table.
I was doing we were doing double writes for a very long time.
To make sure the data will be up to date.
Yes.
So that's where we started.
We started doing double writes, making sure that that all looked sensible.
I was doing.
Did you also do shadowing?
Yes.
The shadowing meaning that you're.
you know, like you're doing whatever you're doing and you're doing new operation and comparing that as the same result, right?
Yeah, yeah.
We did a bunch of that as well.
But obviously we had to, you know, make sure all the data was there first.
So we like did the double writing, did all the backfills.
And then we were shadow reading and comparing if there were any differences, throwing away the new call, obviously the new results and keeping reading from the old ones.
I was about to flip.
all of Slack to reading from the new table.
Which at this point you confirmed works.
So that should have been a little risk change.
Yes.
Yeah.
All the data looked right.
I had a bunch of data warehouse dashboards that showed me how many differences existed between the two sets of tables within reason.
And then I was about to turn it on to all of Slack.
This was like Thursdays back when we had gather hour, which was like after work hours, everyone would sort of hang out if they wanted, have appetizers, whatever.
And I turned it on and then I realized, oh, wait, no, something's not quite right.
Oh, we didn't just do a gradual rollout because with these things, you don't really do a gradual rollout.
Internally, we were like not that big at the time.
Everyone was like gather hour.
You know, I was like, oh, this seems safe.
We've been shadow reading.
It's fine.
Yeah, I turned it on.
I was like, oh, there's like data isn't there's some data that's like not quite right.
And I had been in.
so used to essentially like running the script that like would delete all of the backing data and rerun the backfill or at least chunks of it because I had been seeing discrepancies.
And honestly, it was so much easier to just nuke the data re-backfill than it was to try to like pinpoint and like scalpels fix the call sites.
So I forgot that the flag was still on to everyone.
And then I promptly deleted.
Which flag was on?
The flag to have everyone read their channel memberships from the new table.
Okay.
Yeah.
I forgot that it was on for Slack.
Not everyone externally, like just us.
And I was like, oh, there's some discrepancies.
Oh, right.
But I fixed those.
Let me just nuke the table for our team and backfill, which like took 25 minutes to do.
Like it was not.
Oh, but did everyone lose their messages until then?
Oh, yeah.
People had completely blank.
sidebars for about half an hour inside of all of those inside all of the slack and slack always uses slack for everything for everything but everyone was at gather hours so like it was only there was this only one guy kevin who had been sort of like help helping me like coach through this who dm'd me and he was like what's going on with my sidebar i was like i'm so sorry oh my gosh let me flip the flag so i reflipped to read from the old data um while it was backfilling it finished backfilling and then i swapped it back So what did you learn at Slack about load testing?
And I'm asking because right now load is going up on all systems.
We see this.
AI agents generate more pull requests, more code.
They put more load on the systems.
I assume we will have the point where we generate more data here and there.
But at Slack, you have this interesting challenge where a customer with 500,000 users wanted to go onto the platform.
Slack was architecting for it and you had a plan of 18 months and then COVID happened and you had to bring it up forward.
But I'm more interested in that.
And as you built load testing, something called Koi Pond on this, but when you think of load testing, of checking how a service will be able to handle this type of load before it hits, what are the techniques that you've used and would still use today?
Tested production.
But how do you think it's a production something where you don't have the load just yet, right?
Yeah, no, I mean, I think there's a lot of different ways to do it.
The simplest way that I think a lot of people can manage to do it, it depends on what kind of usage you are worried about with your product.
And nailing down those usage patterns first and where you're really concerned it's going to fall over is the determining factor in how you figure out how to simulate.
load in slack specifically what was usage patterned there were a couple a couple key ways that it had historically fallen over thundering herds where i mean i think most people have some kind of thundering herd problem at some point where you say can we spell out with the thundering yeah just for those oh yes yes suddenly heard suddenly everyone shows up Suddenly, like that, that is a thundering herd.
A thing happens and then suddenly everyone shows up and does the same thing at the same time.
I guess a good example is if your service goes down, like you're, I don't know, you're running, let's say, imaginary, like Google search and somehow like all your machines die for a bug.
And then you have like a thousand machines.
Obviously, Google has more, but we can just imagine.
And then you start to bring up machine one, machine two, machine three.
Each of them gets hit by the complete traffic and they go down again, right?
Yes, yes.
This is a classic example.
Yeah, yeah.
The classic example in Slack is say CEO says, oh, great quarter, everyone.
You all get Friday off.
Posts this message in the big company channel that has like 100,000 active users in it.
That notification gets sent out.
You see that notification.
You are very motivated to plop a really good custom emoji on that notification.
You click that notification.
You immediately land in the channel.
In the channel, you're fetching channel history.
And then on top of that, you are adding an emoji reaction because you're really pumped about getting Friday off.
But so is everyone else also getting, putting custom emojis on that message because they're also pumped.
In the same five seconds.
Yes, in the same five seconds.
So the message that came in to tell you that you're getting Friday off came in over the WebSocket connection.
And it was fanned out to all 100,000 active users.
But then all 100,000 within a couple seconds, right, of those people show up to that channel, issue a channel read, and then are sending emoji reactions.
Every single emoji reaction fans out also over the WebSocket.
So one custom emoji turns into 100,000 custom emojis being sent to everyone who's connected.
And if you add 10, well, then you can do the math and it just like balloons and balloons and balloons.
It's like prime example for like pure explosion.
And as much as I think we we had done some work because we knew that emoji fan out was like actually really expensive to rate the importance priority order, the types of messages that get sent over the WebSocket.
And so like message like actual text messages were the most important.
And then emojis, if.
you were in a very high load situation, would just get dropped on the floor, which is the thing to do, obviously.
I'm sorry to say, but the next generation, I'm not sure if Gen Z or Gen Apple might disagree that emojis are more important than text messages.
I need to know if someone reacted to my message.
No, but I hear it makes sense of prioritizing, especially based on size, right?
Yes, absolutely.
And there's some amount of and this is where like I think product thinking is you can't go around thinking you're going to load test strictly from an engineering perspective.
You have to be putting your product hat on.
You absolutely have to, because that's where understanding that like messages with text are more important than emojis.
That's more of a product call than it is like an engineering call.
I'm coming to you and I'm saying like we have to drop something.
Let's reason together about what makes sense to drop.
And understand it.
And sometimes it's like not what you would expect.
And I think I mean, I this example I keep coming back to because I had multiple conversations with folks over at Slack about it.
Like we should just limit the total number of emojis that are allowed on a message, period.
So we currently limit or I think last I checked, Slack still currently limits.
You can only have 23 distinct custom emojis as a reaction on a message, but you can have.
as many like plus ones on those emojis as you would like.
Why?
The 10,000 and one emoji does not matter.
We should just cut it off at some point.
And I think like whoever, I mean, I think part of the problem is like no one really owned custom emojis.
It would have been the messaging team, I guess.
But like, it's one of those things that's just so ubiquitous.
Is that a word to the Slack experience?
And like kind of no one owns it.
So who would have made the call on like, what is the proper limit to put on this?
But Thundering Herds was definitely super top of mind as one of those issues.
I wanted to ask, are event floods and Thundering Herds the same thing?
No, they're different.
What are event floods?
So event floods are when your client is personally overwhelmed by the events coming over the WebSocket connection.
And so any I mean, that could be from that specific example.
That's sort of a thundering herd and maybe an event flood event in that you're getting hammered with emoji reactions after the fact as well.
But an event flood is just say, you know, this message is happening.
The CEO you get Friday off, but it's happening in 16 different channels at the same time that you personally happen to be in.
Not everyone else is in super, super busy channels.
But maybe you happen to be.
This happened to, there was this guy who was our like main Slack guy at IBM.
His name is Jeff Robke.
He eventually became a PM at Slack, which I think is like the most interesting thing ever because this guy was on the receiving end of so many event floods.
And I felt so bad for Jeff all the time.
I mean, I was in some channels.
I'm like, tell Jeff I'm sorry.
But I still don't fully get like what the event flood experience is.
Can you tell me like what?
Yeah.
So one of the things that would happen to him, right, is that before some of the re-architecture work whereby you didn't have a bunch of distinct WebSocket connections for each team that you were logged in, Jeff was an admin at IBM.
So he was in hundreds of different workspaces.
At one point, I think the maximum number of workspaces you could be logged into was around like 30 something.
And you're saying he was in this because IBM as an enterprise had this concept of like the marketing team, the finance team, the thing.
Maybe they had an external customer team.
And as the admin, he was in all of.
He was in all of them.
And they had so many teams.
And he was in all of them.
And so if you were getting a message in an enterprise level DM.
or a channel that is shared across all 30 of those workspaces that he was currently logged into, well, he would get that message 30 times.
There is no deduplication of that whatsoever.
So his client, his machine, his notifications would be both overwhelmed from maybe a little bit less in processing.
That's fine.
But like from a UI, from being unable to see what's going on, annoyance point of view?
Yeah.
I mean, also his client was just dying all the time.
Which was not good.
Yeah.
Yeah.
And interesting.
Like, this is the first time I've heard about event floods.
Yeah.
And they were very user specific, but they were particularly painful for power users.
This happened to power users all the time.
And power users were also some of the, like, biggest champions of Slack.
And so it's a double edged sword of like, I'm sorry you get the absolute worst experience of our product, but also you love it, I guess.
And you are trying to, like, make sure that people adopt it more.
It's really, really rough.
Mo just made the point that the heaviest users of Slack got the worst of the event floods.
Today, the heaviest users of Git happen to be your agents that often push code in parallel, which brings us to our seasoned sponsor, Entire.
A few weeks ago, I told you that Entire was building a Git forge.
Well, it's finally here.
You can now host private repos directly on Entire in the US, Europe, India, and Australia.
And Entire has just reinvented a pull request for the agentic era.
The problem with PRs is how agents write it faster than how any of us can review it.
So, Entire just replaced it with something called Trails.
A Trail collects everything about the work in one place, the intent, the code changes, and the agent sessions.
A Tail can be spread or cause multiple repos, so the change to your backend and your web app gets reviewed once.
With a Trail, you can configure customized evals like code review, security reviews, slot reviews, and others.
They look like a heads-up display in an airplane and score your work on every push.
And you can see how each agent from passes or fails your user to find gates and then decide when you merge based on these.
The point is that you control when you allow your agents to be allowed to merge code like merging only after human review or after all gates pass.
Oh and one more thing.
Entire ship continuous memory.
This turns your repo into your brain so your agents can search past coding sessions to find the context behind earlier changes.
Basically, answer the why behind the code changes.
Memories get shared across the team and across the repo.
If you want to see what an AI-native GitForge looks like, head to entire.io slash pragmatic and check out Trails.
I'd also like to mention our presenting sponsor, TurboPuffer, who were rebuilding their storage engine to make storage faster and cheaper at the kind of scale we're talking about in this episode.
The new system, called TurboPuffer v3, is not yet in prod, but it already passes CI.
And when the rewrite started in September, it was not nearly as performant as the current production version, v2.
The TPuff team has been...
perf maxing query plans until they are as fast or faster than v2.
Doing all of this in public all the way up to the v3 production launch.
And it's so cool to see their progress.
The new system went from being 10 to 100 times slower across most metrics to several times faster on things like cold vector search, cold full-text search, or counts.
The v3 system is still slower in areas like hot full-text search but only one to two times slower versus 100 times slower when the perfmaxing started.
And the Teephuf team is sharing exactly how they optimize things in their development log with things like vectorized query execution, group count aggregation, compressed column encodings, and many more.
But hang on, why is the team doing a full rewrite?
It's because search is changing and so is TurboBuffer.
Teebof started as a vector database, but it's no longer a vector DB.
It has evolved into a high performance database for search.
The V3 discards the vector primary index and all indexes will share the same columnar storage system.
Every query will run on vectorized query engines, which is something that you don't usually see in search databases, but which will both result in very fast search and just generally a very fast database.
The TurboPuffer team is keeping a work log of their journey and you can follow along at turbopuffer.com slash v3.
It's a blast to see how it's being built.
And with this, let's get back to mode and how Slack load tested for a customer expecting to have 500,000 users.
And then back to load testing.
So how did you simulate?
the load that you would be expecting to get.
In this case, you knew that a big customer would be coming with 500,000 or a million users.
And you didn't know if Slack could handle it, but you want to figure out before testing and production, before they all came in, did you build a custom load testing tool?
How did you go about it?
Yeah.
So we'd had a couple of like half conceived load testing options to date.
Mostly that tested just...
API throughput and making sure that we could withstand the underlying load primarily on the database.
And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they've last changed it.
And that makes a particular database request a SQL read on that channel shard now because everything started differently.
But it makes a particular request.
We wanted to know that we could handle that RPS.
The problem with that is and any of the load testing solutions before this particular moment in time was none of it had active WebSocket connections.
And the two particular cases that we just talked about, the WebSocket is the problem in some ways.
I mean, we've never really had latency issues with sending messages through the WebSocket like those stateful services have.
One of the co-founders was like the mastermind behind all of these and like the amount of time and energy he sank into making them as performant as humanly possible is commendable.
They were never really a bottleneck.
It was either how the client handled it or the re-downstream event caused by you seeing and receiving this message that was the further bottleneck.
And so we needed a system that had an active WebSocket connection in order to be able to adequately test the whole flow.
It's not just sending like, as though you're reading a channel.
It's not just an API.
Yeah, yeah.
So we really, and it's not just like flooding a database with like.
But when you say you need an active WebSocket connection to test for 500,000 load, you needed 500,000 active WebSocket connections.
We needed 500,000 active WebSocket connections.
Yes, we did.
Okay.
Yeah, yeah.
Which is not easy to do.
And so we had a solution before this customer came in.
which I'm 90% sure it's public knowledge now that it's Amazon.
But we had a solution that would spin up Slack as a headless Chrome instance in like a bunch of Kubernetes pods and try to run a tiny little JavaScript script and like click buttons, like pretend to be a Slack user.
Which like was sort of fine.
But I don't know if you've tried to wrangle more than like 50,000 headless Chrome instances in Kubernetes.
I don't think many of us have.
I don't think many people listening or viewing have.
It is not a good time.
I mean, the problem with the headless Chrome instance, right, is like just running Slack and like 200 Chrome tabs and whatever on your laptop is like kind of a time.
And now you're running like, you know, LLMs on top of it.
But like, it's an experience in really pushing your laptop to the limit.
And now you're running like per pod, like attempting to run like 10,000 headless Chrome instances of Slack, which is a very memory and CPU, mostly memory intensive application.
The amount of money we were spending in these EC2 instances.
The problem seems to be that per node, per VM node, depending on the size, you will be able to run, you know, you'll know, like, I don't know, like 10, 20, 50, 100.
I don't know.
You figure out how much, but then you pay for every one of those VMs, those nodes.
You have a number of them.
Yeah.
I mean, look, like Slack raised a bunch of VC money, so I guess.
Sure, sure.
Yeah.
We know where some of that went.
Yeah.
And also from customers, of course.
Yeah, yeah, yeah, yeah.
Of course.
That is wicked expensive.
It is.
So it was 37 cents.
per headless Chrome instance, per five-hour tranche of load testing.
Yeah, we're talking about $150,000 for that one test.
Yeah, yeah, yeah.
Which like, okay, that's like financially not sensible.
Like it just didn't work.
But I guess if you compare it, like of course, I'm playing devil's advocate.
$150,000 is great.
But if you have a customer coming up with 500,000 seats, If they would get a discount, they would pay $5 million per month.
Like you would think there is there is a point of investing to make sure that it works for them and they don't complain.
I don't know where that comes in.
But but yes, the $150,000 still feels like pretty excessive for that one test.
Oh, yeah.
Yeah.
I think for and for that one, I mean, that's only a five hour chunk of time.
And that's assuming that you're able.
Yeah, that's only five hours.
OK.
Yeah, that's getting a problem.
And that's assuming you're testing all of the feature tests that you wanted to run during that five hours.
Which was not the case.
Which was not the case because we were spending half the time trying to bring pods back up because they were crashing.
Okay.
Did you figure out a better way to do it in the end?
Yes.
Yeah, we did.
Lower fidelity, but definitely a better way.
We gave up on the whole headless Chrome instance situation, which was obviously like the highest fidelity.
Best option if you wanted to make sure that it was truly testing how Slack would work.
But instead, we spun up a bunch of little Go processes that would each establish a WebSocket connection and run through a series of API requests that made it look like they were actual clients.
And so they would still receive everything over the WebSocket, send everything over the WebSocket, as you would expect, and still have a connection to the regular API that...
the client was making requests to and also our edge cache.
We didn't talk about our edge cache at all, but there's a bunch of data that needs to be really snappy to load when you load up Slack.
Like when you go to switch channels, the channel list is in an edge cache closer to where you are because all of the main API requests still hit US East 1, which is great.
I mean, we are so boned if US East 1 goes down.
All those main requests still hit US East 1.
So we still...
had to also test the latency associated with making requests to those edge caches.
And so we also established that request pattern there.
It sounds to me that with load testing, all I've heard from you is like, you know, step one is understand how your product will be used, what will likely top it over, which because once you know you're not going to just randomly load test, you'll actually can test target it for that.
And then building something that simulates it.
good enough knowing the resource constraints.
So in this case, don't boot up a whole slack in a headless Chrome, but just mock, not mock-up, but build out that critical path that you care about.
And if you combine this to your domain knowledge and your expertise, knowing how the product works and knowing how, you know, you knew how it used to fall over, you can now be reasonably confident that if it falls over, it will not be this way.
It might be a different way, but...
I think it's nice that we're talking about the cost of testing, especially like we've talked about the cost of LLM, but like there is cost considerations and also repeatability, how lightweight it is, how much easier it is to control it and to...
even tweak it or it sounds like your solution you could you could later if you find a different failing those case you can just rewrite it to to that it's not a big deal yeah yeah it was really lightweight um it just did what it needed to do we created a a pretty i mean we evolved the system whereby you could create like routines to send to these like fake users to run through over time at first it was just like a very sort of silly like json blob of like call this thing then call this thing then call this thing I wrote a mini recorder tool where I could, via a slash command in Slack, say, slash start recording.
Go through a whole flow in Slack.
It would record every single API request I made across both to the normal web app backend and to our edge cache.
It would record every single outgoing and incoming.
WebSocket message.
And then I would sort of sort through it all to just the pieces that I wanted to simulate and then inject that into another tool that I wrote that like turned that into a script that I could then send to the simulated clients.
Because the other half of the battle was figuring out what are all of the APIs that get called when I do this one product flow, which is like not a small number.
And you wanted to be sure that you captured the entire thing.
So that's where the recorder tool came in pretty handy.
And we used that over and over and over again.
And I would, again, to go back to the product thing, because I'm going to keep hammering the product thing.
I think like the best product managers are mindful of like performance and reliability are important aspects of any given feature.
And the best PMs know that and care about that and optimize for that.
And I was trying.
The last like several years that I was there, I was always trying to push more tools into the hands of PMs and say like, hey, you want to add your use case to be automatically tested with KoiPond with our load testing system?
Here's the recorder tool.
Go record the flow that you want us to test.
Send us back the file.
We'll plop it into the system and we'll run it at the frequency that you want it to run at.
Because they knew.
you know, oh, files do this weird thing when we like pop open a canvas and it like does this thing.
And like, I don't know.
I don't use canvases the way that like most of our users probably use them.
You do.
You're the PM.
So like go record it and then bring it back to us and then we'll just shove it into the system we wrote.
Speaking of reliability, after Slack, you joined GitHub as a principal software engineer and this was 2024.
So this was at a time where GitHub Copilot was already.
very big.
In fact, Cursor and other competitors were already, as tools, were starting to get up there.
The load on GitHub services already started to increase and GitHub's reliability started to visibly become shakier, not as bad as it is these days.
What did you observe inside of GitHub?
Can you tell us the vibe when you joined?
If you can give us a bit of an understanding and appreciation of what GitHub has been built on, this massive Ruby monolith.
And when you joined him, what was it like?
It was a really interesting time, particularly because one of my friends from Run the Runway, actually, we're going way back, had been at GitHub for many years at that point.
And we'd sort of been keeping tabs on each other's careers.
And she told me, Moad, we are going to run out of space.
Like GitHub is still primarily on data centers and it's like increasingly less so over time.
They used to have their own data centers.
Now they're moving to Azure.
This is public information that is shared.
Yes.
Yeah.
The move to Azure is front and center, but still a significant, or at least at the time when we were talking, a still significant amount of load was going through the data centers that like they had whole teams managing the metal.
And she was like, we're going to run out.
of space, the space is running out and we either well, we both have to figure out how to move to cloud.
And we also have to figure out how to do less because we're doing a bunch of dumb stuff, as everyone does, that we like shouldn't be doing.
And we can make the system a lot more efficient with what we have to give bias more runway.
I was like, oh, this sounds like a very interesting problem.
Would love to work on this problem.
But I was also seven months pregnant at the time.
And so what that ended up being was I showed up.
I was seven months pregnant, which is actually kind of if you can swing it a really great time way to join a company because you are you sit there and you're like, cool, I will just drop a bunch of opinions.
Like, you know, nothing very strongly held, but I will just drop a bunch of opinions on people.
I will own nothing for the next two months and then I'm going to go have a baby and then I'll come back.
Let's see what happened.
Let's see what happened.
And in my first couple of weeks, I was sort of floating around some spaces and I, having done a bunch of data migrations at Slack and some of the stuff that we didn't talk about, but was moving from this like team sharded.
tenant based model to a more dynamically sharded model with Vitesse, where it turns out, no, a team was too big.
So you mentioned Vitesse.
What is Vitesse?
Vitesse is essentially a wrapper on top of MySQL that lets you shard more dynamically.
And so you have a...
So instead of just having MySQL run on one database, and when you outgrow it...
good luck.
You manually move the tables or copy them or whatever.
It's a nightmare.
It allows you as it grows and as it outgrows, it will, I don't know, predict or it will now have a way to more seamlessly move over to multiple machines.
You can reshard, redistribute your data a lot more seamlessly than you would be able to otherwise.
And you can choose different sharding keys more easily as well.
And so before, it's like everything was sharded by team.
There's still data that is sharded by team, where team is the logical unit that makes the most sense for fetching that data.
But actually, it turns out that the smallest, most logical unit was channel.
And so we ended up...
re-sharding a bunch of our data to be fetchable by channel ID rather than by team ID.
And then that allows for much better horizontal sharding structure as well.
And the idea being that there is no channel that will ever be big enough.
Maybe one day it will be.
The bots keep posting in the channels, maybe, that will have enough data associated with it that it can never run on a single ultra-large database, right?
It's a much better unit from a scaling perspective.
So Vitesse has had some of its nascency embedded in GitHub, actually.
And it was really interesting because when I was interviewing at GitHub, I spoke with a bunch of the principal engineers that I would later end up working with.
And they were sort of trying to figure out, OK, we have Vitesse, but I don't think we're actually using it to its potential to where we should be using it.
And so they were asking me a bunch of questions about like, are Vitesse set up and what we were doing?
caveating this with I did not ever operate the tests or deal with the internals.
I was more of a I'm a product platform person.
And so I just like find that, like, no, we should not be starting this thing by this thing anymore.
Let's migrate it.
And so that was my perspective on it, which was also mostly their perspective on it as well.
And I came in and I started poking at the Vitesse setup and I was like, what are we doing?
What are we doing here?
I was used to thousands of shards per sharding key.
So say four channels, we have a bajillion because there's a bajillion channels and you want to sort of spread load evenly throughout those.
And when I arrived, I was looking at the Vitesse setup and there was, I mean, I don't know if this is still the case, but there were 16 shards for all of pull requests and issues.
Across all of GitHub.
Across all.
of GitHub, only 16, which is not very many given the amount of data that we know is associated with that whole space.
And my first question was like, wait, why do we only have 16 charts?
Oh my gosh, this makes me deeply, deeply nervous.
And part of the answer was, we can't have more.
Because we already have so many scatter queries.
So those are queries that are coming in.
What are scatter queries?
So those are queries that are coming in where you do not have the primary key in them.
So say you are querying for a pull request or an issue and you don't have the pull request ID, which is how this data is being distributed.
Well, you have to find it.
So you have to go through, you have to find and match the criteria.
So say I'm querying for all pull requests.
authored by me.
Yeah.
I don't know that that's an actual scatterer, but this could be in this scenario.
Let's just assume.
And so because I don't have a pull request ID, well, I'm going to hit every single one of those 16 shards.
Wow.
This will be bad in so many levels.
Uh-huh.
Because, well, so many people are doing it also.
And I understand then why you don't want to have many shards because if you have more shards, you would need to hit all of them.
We're talking like, it's almost like a linear scan.
Yeah.
Yeah.
Oh, no, it's quite.
quite, quite inefficient.
And again, I don't know that this is an actual example of one of the...
This was hypothetical.
This is completely hypothetical, but there were still very many significant scatter queries that were hitting almost every single shard before it knew that it could complete the query and return the result.
Which, yes, I understand why that was a constraint then.
You don't want to add more shards.
But then again, you also cannot make the shards much bigger and they're under very heavy load.
And so I was ready to apply the playbook that I'd been using for the last several years of like, well, let's just burn down the scatters.
Let's just get rid of the scatter queries.
Don't have scatter queries.
Just don't have scatter queries.
That can't be that hard.
Well, it turns out that can be very hard when you're dealing with a Rails monolith and you've got ActiveRecord.
And I always sort of...
thought it was funny.
The number of there were a lot of ex Twitter folks at Slack early on and everyone was like, ha ha, active record.
And I was like, what are you like before they did the whole like Scala Java thing?
Right.
And I was like, OK, I guess like we're all, you know, joking about active record.
I did not understand until I came face to face with it.
And I was like, oh, I really don't like this.
This is really terrible.
So like the record, this is Rails' abstraction layer for the database, right?
So you define things in, I guess, code and it kind of maps nicely to database, except it might be harder to fine tune your queries, etc.
Yeah, I mean, it does a very good job at hiding the underlying data model from you.
which again, pros and cons.
I came from a world where like SQL was, you know, religiously written out by hand for performance reasons.
And then I was being thrown into this world where at one point we were trying to figure out, we were doing a bunch of work around domain extraction.
And so how do we rip out everything related to say pull requests from the Rails monolith and extract that into a separate Go subsystem?
take everything that deals with public requests, just like take, make the requests go through, say the monolith still, but then just like immediately send them over to the other system.
Really standard sort of migration procedure.
We started doing that.
We started with stars and I think that project was eventually abandoned, but I was working, I was pairing with someone who had like a ton of Rails experience.
And I was like, okay, you gotta like, because I, as someone who was so used to reading just code that was very sort of.
straightforward and didn't have so many levels of abstraction.
It's a little bit daunting when you're going into a world where now there's abstractions upon abstractions upon abstractions.
And I was working with her to try to rip out more of the STARS-related queries inside of the Rails Molleth.
We'd found sort of the root caller that created this query that would call into the STARS table.
And it was like a five line function.
And I was looking at it with her on Zoom.
And I said, wait, Kate, where where is the call?
Like, where where is the sequel being made?
Like, what is happening?
It's a five line function.
She's like, Mo, that's the million dollar question.
And I was like, what are we looking at?
Because I half expected to see.
I mean, at this point, I think so many people use Prisma, right?
And Prisma, you can sort of it's not, you know.
plain SQL written out as a string between quotation marks, but you can sort of see the structure of the SQL query being made.
You do not get any of that in this case.
It's just, it just looks like a regular function call.
And in that moment is when I realized this is why this has been impossible to date.
And there is because no one had, and why the problem has gotten so bad because no one sees the impact.
of just writing a new line of code.
And also fixing it now will probably, it's not as easy as just tweaking your abstraction layer.
You would probably have to change how code is being written, being called, starting to call more either functions that call secret, et cetera.
Like it's just, you took a technology that.
worked great when you were small, you were able to scale it as long as you and they figured out how to how to grow it as long as things are going linearly.
Yeah.
And I think one of the things that like blew my mind the most that like both gave me a lot of respect for how they've managed to date and also filled me with much dread was when you work inside of GitHub, obviously you have like the admin view of.
GitHub, and there's this bar at the top where you can load up a page on GitHub, and it'll tell you exactly which MySQL queries were run to populate the page, which clusters they hit, how fast they all were, totals around all of this stuff.
Just like the internal employee view.
Yeah, yeah, yeah, yeah.
And what's called the files disambiguated page, which is sort of like the repo homepage that you land on with the readme.
That page lasts.
Last I checked, you know, before I left.
Before you left like a year ago.
Actually less than that.
But yeah, yeah, yeah.
So some amount of time ago was it was over 100 MySQL queries to populate that page across at least three different MySQL clusters.
And the thing with ActiveRecord is that it only caches the output automatically if the query is exactly the same.
And so if, say, you're fetching user information, like full blanket, like select star from users where user ID equals blank, and you have that full user, but then later on you do select star from users where user ID equals blank and something else, it's not going to reach back into the results that it cached for that first query because they are not exact same queries.
Every single count, or at least when I last left.
Almost every single count you see on a GitHub page is live calculated.
None of those are cached.
There's a lot of reasons why that is.
Mostly product reasons, actually.
But that's one of the first things that I was like, but why?
And to go back to my sort of silly emoji example of the 10,000 in month emoji, no one cares.
Why does anyone care?
Right.
And if you have low counts of things, sure.
OK, maybe it matters that there's three open pull requests on like this very small repo that like no one really does anything with.
But once it gets to like 99, don't just don't show anything.
Just say 99 plus.
Don't bother counting it.
Why are we counting this?
Really interesting that so far we talked about you did interesting thing that Slack and GitHub, you also worked at Copilot Code Review.
Afterwards, you worked at Cursor again.
They really pushed.
how we're coding with AI tools.
But so far, we've not really talked about AI.
It just didn't come up naturally.
And I wanted to ask, firstly, what do you think AI so far, you obviously use these tools all the time, increasingly, intensively.
What parts of the job does AI not help you with?
And what parts of it has it helped and has it changed how you work?
Oh, gosh.
Yeah.
It has not helped me with performance optimizations basically at all.
At least the investigation part.
Maybe the implementation part is a little bit faster.
Once you know what you want to do.
But yeah, figuring out what I want to do.
So say I was looking at improving time to first token, time to first visible token.
At Cursor.
At Cursor a couple of months ago.
where and there's a bunch of factors that feed into how fast you start getting those tokens back.
There's a significant amount of time that goes into when you are typing your prompt and you hit submit.
There's a significant amount of time that goes into building all of the context associated with the request that you are about to make to send to the LLM.
And then there's some amount of data that we are able to cache, some amount that we are not able to cache, either for product reasons, privacy reasons, but also strictly because of load and, again, cost of these systems.
It is not cheap to say cache absolutely everything that you have ever queried cursor for in Redis for 24 hours.
That's a lot of money.
So we're not going to do that.
And then eventually sending that request over to inference proxy, which then sends the request over to the underlying LLMs and providers.
Given that there's so much that happens, but also back and forth, a lot of this process is not really as synchronous as the experience leads you to believe.
And in the regular flow, if you're lucky enough to be able to use HTTP2, which some poor folks out there are stuck on HTTP1.
You establish that gRPC streaming connection with the backend servers, and you're able to split that connection into like four or more concurrent streams and send and receive different bits of information across those streams.
It's very complicated.
And also AIs think that they're really good at instrumenting these systems.
They're not, turns out.
They think they're, it's like writing tests.
They think they're really good at writing unit tests.
But they're really good at just confirming what they just wrote, which is why some of the skills that I ended up crafting run through intensive series of mutation tests on the code to then force the LLM to realize, oh, I didn't actually test any of the edge cases.
Let me rewrite those tests.
Thank you.
So the LLMs right now, I find, are good at doing the really basic level of instrumentation.
but not great at figuring out, especially in an asynchronous context, what makes sense to instrument how.
I love traces.
I think they're amazing.
You can get so much really good debugging information from them, particularly from a performance perspective.
We had this problem at Cursor where we had like too many and too little traces.
And by that, I mean that we had traces with millions upon millions of spans, which are completely useless.
You cannot do anything meaningful with a trace that has that many spans.
And then we also did the very classic mistake that basically everyone does, which is we started sampling for cost reasons, but not in one place, which is, again, really easy to have an LLM do because it's like, oh, like, find me a place where we can.
Send fewer traces to save money.
And it's like, cool, I'll just like sample it right before I send it to Datadog or whatever.
No, no, no, don't do that.
Please don't do that.
Always do it in the same place.
The problem with duplication is just like so much worse when you're dealing with this.
And so one of the first things that I ended up having to do was like, honestly, manually instrument a lot of these flows and actually read through the code myself.
and understand, okay, this chunk of processing is synchronous.
This chunk of processing is asynchronous and put all of those together.
And one of the ways that I started doing that was I had it generate a like big sequence diagram of everything that goes into time to first token.
So like from the time you hit submit to when you receive that first visible token and it was...
huge.
And I went to Staples and I had them printed out on like big blueprint paper because I'm still a really analog person.
And then from that, I went through and I was reading the code and I was annotating in on my big piece of paper like, OK, this this is correct or this is incorrect.
This piece we could probably do something about.
this other piece no actually definitely needs to happen before this other piece one of the other things that i was finding that it was doing um really painfully wrong is that it was finding sort of the classic things that could what was doing what the lm was doing wrong oh yeah so what it was doing wrong was um finding well wrong wrong is kind of the wrong word um Naively, naively what it was doing was say you have a bunch of things, you're finding a bunch of tasks that don't depend on each other, right?
But they're latency sensitive.
What's the classic solution?
You paralyze them.
You paralyze them.
However, again, product hat, right?
What if basically no one is ever doing all four of those things?
And why are you doing all of them at the same time?
Like you're wasting...
resources latency, you're increasing load for no reason.
And so I remember, I think it was Opus that was very excited that it found this opportunity to paralyze a bunch of these tasks.
And I was like, oh, on first glance, sure, that seems like a good thing to do.
Except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck, regardless.
And the longest one was always so much longer.
Then the other three and the other three tasks were being run such a small 1% of users were doing any of the other three.
And so it ended up being a complete waste of time.
Like this was not the optimization that it was like selling to me as being like the panacea to improving time to first token.
And a lot of the things that were actually quite helpful were really silly things like, I mean, silly, like.
increasing the TTL on what we stored in Redis where we could within reason.
How do you use LLs?
Because so far, my sense is, talking through your career, you look at a problem, you understand it, you figure out how to do it, you fail sometimes or make mistakes, you learn from it, you learn from others.
including right now in the last few years where AI is increasingly everywhere, but as you're making systems more reliable, more performance, it sounds like you're still doing so much thinking, so much work.
I sense that you're using the LLM on the side to generate some ideas to say, no, this is wrong.
No, that was a good idea.
Okay, let me use it.
But unlike a lot of software engineers I talk to, even on this podcast.
Most of them are saying like, oh, so much of my work that I used to do, the LLM does it.
I now can do more things.
I can focus on higher.
I'm not getting that sense from you.
Yeah, I mean, I'm spending all day like chatting with an LLM in some way, shape or form.
I think like I still do a lot of the normal engineering stuff, which is like I'm on call.
Sometimes I just have to fix a silly bug.
But I do tend to I do a lot of.
sort of preliminary exploration, but then verification manually.
And I sense that a lot.
Like, it sounds like even when you get something from an LLM or whatever, you will verify, you will make sure that you understand that this is right.
It sounds like you don't have a judgment if it's going to be right or wrong, but you need to verify it, right?
Yeah, yeah.
Because I think a lot of the problems that I work on are more of the like product platform-y problems.
And the whole point is to solve a problem holistically.
And to create that foundation that you can build on top of.
Right.
And if you only built the foundation to cover like 80 percent of the house, like what's the point?
Because that other 20 percent that was not built on the foundation is going to be really shaky and impact the rest of the house that you did build.
on the foundation.
And if you already did that over here, well, like nothing's really stopping you from say like tacking on another 10% off past the foundation and then that spiraling.
Right.
And so I think, especially, especially now that like, I think AI is so good at just duplicating things in places because that seems fine.
And it doesn't have a sense of, it's not great at going, you know, but why, but why, but like.
20 levels removed, it leans more towards, I will just tack on, I'll just build an extra bathroom past the foundation.
That's fine, right?
Like you asked me to build an extra bathroom, that's where I'm doing it.
Rather than questioning the underlying fundamental assumptions, which are, should one build a bathroom not on top of a foundation?
And those are the questions that I'm asking myself all the time.
And I think part of it is because I'm coming from a product platform perspective.
I have a sense that the product platform or the platform or reliably performance, it will only be more important because, again, we'll have systems that have so much load, so much test.
What advice would you have for a software engineer?
who may or may not have much experience, but who is excited about the idea of performance reliability and would like to get closer to eventually work on platform teams.
What are paths that you've seen people make it onto platforms that you worked on?
How did they transition?
I'm such a strong advocate for more product folks, more interchange actually between product folks and product platform teams.
I've seen it so many times where product platform teams are spun up out of product engineers who care a lot about the fundamentals, right?
They care about reliability.
They care about performance already.
They're trying to improve that and inject that into the way that they're building product all the time.
And then I think the best product platform folks that I've worked with get to a point where they've seen a pattern enough times in the things that they are building as product engineers.
And they're like, I want to dig in and solve this problem across the board.
Like this specific problem, I understand really deeply from having implemented it two, three, four times.
I want to go solve it once and for all and be done and like have this thing that other people can also use, right?
They get annoyed with the toil and then they go and maybe embed with a product platform team or they like spin one up with some other folks and then they do that.
And then a lot of times those folks end up.
staying on product platform teams and continuing to really like the product platform work, but then they get further and further detached from what product development actually feels like.
And I think that's where you see sort of the chasm that ends up being created between product platform teams and product teams is just there's not enough cross-pollination.
between the teams.
And so one of the ways that I've seen that chasm merge back together is having like rotation cycles between those teams, sort of always having someone on the product platform team, having a core set of folks who are like maintaining the main systems and platforms that they've built, but sort of always having someone on rotation from a product team on the product platform team, bringing like, oh, well, I don't know what you all are doing here, but like on product, we're just like YOLO shipping all this AI slop all the time.
Like what you're reading your code, what?
It's a thing.
It's a thing.
On product code reviews are totally dying.
It's completely different, right?
But if you're not close to that, then you also don't know how to bolster your platform to be the best that it can be so that they can do that.
I see my role in this phase of the world right now as a product platform person to be like the guardrails need to be better than they have ever been before.
Because Product people need, I mean, need is a strong word, but should ideally be able to slop out whatever the heck they want.
And the platform should stop them or warn them or.
Yes.
So do I understand that your suggestion is like, look, if you were today are on a product team, you're building products, you're shipping to customers, to users, obviously understand what the business cares about, builds, you know, try to build quality stuff, but also.
Ask yourself, like, are you carrying out performance and reliability?
If not, first of all, why not?
And second of all, as you start caring about this, and if you are working with the platform team, start working with them, hang out with them, see how you can, these problems that you're having, can I help build a solution for that?
If you have the option, transfer onto this team, it's pretty useful.
And if you are on this team, try to have a way to work a little bit, like implement the feature with the team, understand why they're shipping so quickly and so seemingly carelessly, because this will both ways will just make like it seems like we need both, especially now that product teams are pushed to ship a lot faster and they are shipping faster, by the way.
And there's no new tools.
Some of them work.
Some of them don't.
And, you know, this is how we're going to be better engineers and how we will build better systems that allow us to like ship.
reliable and quality experiences, which I'm all for.
I hate seeing quality degradations on software.
Oh, no, no, I know.
But we can stop it or we can actually build better software, hopefully.
Yeah, yeah, I think so.
I think it is really similar to, for folks who are familiar with like the embedded SRE model, like there's a reason why that's really, really effective.
And I think that there's real parallels with.
product platform as well.
I kept talking to my team about like, I mean, we always had to fight against spaghetti code, right?
So easy.
But now we're at this point where like the spaghetti has like a life of its own and it like actively wants to spill out of the plate.
Whereas before people were like, okay, like I got to kind of keep it on the plate.
No, no, no.
The spaghetti like wants out and we have to almost like red team our own situation.
We do.
Well, Mo, thank you so much for this conversation.
Thank you so much for having me.
It was nice to get an insider look of what scaling up looks like inside of products we all know and use like Slack, GitHub, and Cursor.
Until now, I'll admit, I never thought much about custom emojis causing a kind of scale problem.
But Mo, the example of the CEO sending out a everyone gets a Friday off message to 100,000 users was a good visualization.
Hundreds of people added custom emoji and this causes an event storm of custom emoji binary sent all across the network and messages.
Slack decided to fix the problem with a very simple solution.
Limit each message to 23 custom emoji reactions.
I'm sure they could have done a more proper fix, but who will notice the 24th custom emoji not appearing anyway?
It was also fascinating to hear GitHub's limit of 16 shards and why they couldn't add more.
It was because of scatter queries, queries with no primary key that had to run on each and every shard before they could return.
More shards would have slowed these queries down even more.
The solution, of course, is to not have any queries without primary keys.
But when you work with the legacy system with hundreds or thousands of queries without primary keys, rewriting those queries takes a non-trivial amount of time, not to mention that even as you're rewriting, you might have new queries added with no primary keys.
Finally, it was interesting to hear how while Maud is clearly AI-pilled, I mean, having worked on GitHub and Cursor, she still verifies pretty much everything by hand.
Because when doing platform work, she says that you want to have a stable foundation.
She also found that LLMs tend to write tests that just confirm the code written.
Maud is now a fan of using mutation tests to have them cover edge cases.
Do check out the show notes below for related to pragmatic engineer deep dives on Cursor, GitHub reliability engineering, and performance topics.
If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube.
A special thank you if you also leave a rating on the show.
Thanks and see you in the next one.