# AI Distillation Attacks and Benchmark Integrity

**Podcast:** Latent Space: The AI Engineer Podcast
**Published:** 2026-02-26

## Transcript

Okay, we're live.
We have one person.
Um people will start trickling in.
Thanks for coming to Sale Live number six.
This is a very exciting one.
I think we have a I mean, the topics are always fun with these.
It's whatever is the topic of the day on our little rat racing minds trying to keep up with AI.
But we're welcoming the latest writer that is joining the sale coalition.
So I think this just means more content for sale.
I think I'm been a fan of Swix and a friend for a while at this point.
So I'm very happy to have his content join this.
And I think you've been doing great stuff recently and continuing to evolve this.
So thank you, sir.
Welcome to the team.
I just this is like my friends and uh and colleagues in the AI media space, and it's just great to be able to support people and keep that network closer.
So welcome to the code.
Yeah.
Thanks for uh I just wanted to say uh uh uh thanks for joining us.
It's uh really a pleasure to have you on here, uh Sean or Swix.
Um so yeah, uh awesome.
Uh I just uh coincidentally listened to your podcast about the super benchmark.
Um so yeah, awesome to you know, small world, awesome to have you here.
Yeah, th thanks for having me and uh yeah, it's just glad to be on in chat.
Uh I've never ever done one of these Substack Live things, so I'm curious how it works.
I always think about Substack because I can use later platform.
Uh, but they want to go multimedia.
I think the live thing before we get to technical content is actually good because it gives it a different edge.
It's just like a little bit sharper when you know you're live.
I think we've all done a lot of podcasts, even podcasts that are unedited and put there's later, but I think the live thing is a different element that can be tapped into nicely.
So I don't know, why don't we type why don't we just dive into it?
We're gonna start with distillation.
I put I put how models cheat in the top so we can talk about benchmarks.
I think Anthropic posted this pretty spicy blog post this week.
I think it was essentially detailing how they found distributed distillation quote unquote attacks on their services from prominent Chinese labs.
And I'm very unsurprised with Anthropic calling it an attack.
I think that that fits with a lot of their branding.
Okay, nice.
Screen share.
This is what we mean.
Sean Sean Swix is such a pro.
Um and it's like and the screen share people was only dropped a few days ago.
But essentially it's Anthropic is detailing how they found distributed accounts across multiple Chinese labs building shadow their LOMs and described what they were doing and why Anthropic is concerned about this in their worldview of like AI geopolitics.
And I think it's very interesting because I'm of the opinion that the Chinese labs like obviously should do this.
They're in a massive GPU shortage and using HPIs is way easier than generating synthetic data on their own.
And I think that's a good thing.
If I may interrupt you here, maybe we should just for the general audience uh just define distillation before we maybe dive into the details.
Um yeah, so distillation that's like a broader concept.
Uh it's not like a new concept that came up with LLMs.
It's like an older concept in machine learning in general.
And uh distillation essentially is the the idea is that you're taking a larger model and uh train it on the outputs.
Sorry, you you have a larger model, let it generate outputs and train a smaller model on these outputs of the larger model.
And the idea is that you can train the smaller model more efficiently using that larger model.
And originally, I think you just brought up the paper here.
Originally, what you would do is you would train on the logits.
So old school machine learning people might remember from deep neural networks, like the logits, the outputs of the last layer that you usually work for with them to compute the loss function across entropy term.
And you would train on this signal.
And nowadays, in the context of LLMs, it's a bit more loose.
So it does not have to be these logits that you train on.
It could be just the output data, synthetic data, like Nathan just um said.
So for example, it's actually a very common practice.
For example, in Deep Seek R1 in the paper, um, or other people do that do other companies, they would train the flagship model, the largest model, the R1 model with um 671 billion parameters, and then they would uh create smaller variants like uh I forgot the numbers, but one, three billion uh in a smaller range, like these small uh models you can run locally, and they are trained on the outputs of their own larger models.
Um I think now the thing is also of course I mean, this is very common practice.
Everyone does that when they are producing the smaller model variants.
Um now I think the the question or the point Nathan brought up is what happens if you are a company and you generate this synthetic data from another company's LLM and then train your own model on it.
So sorry, that was just like a little interruption, but yeah, distillation in short is training a smaller model on the outputs of a larger model, basically.
Yeah.
Yeah.
And I think this is even possible at the frontier.
So like people distill from something like Cloud Opus to build clouds on it.
Like this is gen, they're generally doing very similar things internally.
They have different access to different tools and like richer tools.
And then the other context is that all these large labs for years have had terms of service where they say that you effectively cannot use the outputs from these APIs to train something like a competitive AI model.
It is vague terms.
In terms of service or not a contract, essentially terms of service is something that can be you essentially are using a service, and then if a the provider finds you violate it, they can cut off your access.
That's just kind of like a basic thing.
So these have not been enforced within the US much at all.
I think there was one case, maybe like ByteDance a year or two ago that open AI cut off their API.
But this was discussed so much right after ChatGPT when people were building the first open models on like alpaca and things.
So it's like, is open AI gonna come after us for doing these research models?
And it totally died down.
People were worried about this for like over a year.
It was kind of an inseparable discussion.
So nothing really happened.
And then this is like the first prominent um re-emergence of the discussion, kind of to make I think it's because people are far more worried about AI competitiveness and quote.
But can we talk about you guys think?
Yeah.
Can we talk a second about even how they would detect uh because you you said in the beginning something about uh uh distillation attack, and you didn't say that specifically, but you kind of like implicitly put quotation marks on attack.
Um so how would you even detect that?
So I think uh I mean distillation in that context um means uh really like literally just letting Chat GPT Cloud generate synthetic data, and then you collect that synthetic data and train your own model with supervised learning, uh supervised fine-tuning on it.
But then how would you even detect that this is a distillation attack versus just an evaluation?
Because uh right now I'm actually running um, I mean, I'm distilling myself for chapter eight of my book, but I'm doing it with open weight models, so no worry, um anthropic, please uh don't worry about it.
I just still from API models for my job.
Yeah, I use open router right now and just distill from the uh deep seek version 3.2 model, which I think these folks are okay with that.
Um but um what I wanted to say is so when I'm evaluating models, I use basically almost the same script.
So when you're evaluating in a model, you have a question and you let the uh model generate the answer, right?
So you generate the response to your benchmark question.
And uh in my benchmarks, I have data sets from math 500, 500 examples.
I have a bigger math data set of 12,000 examples.
So you're basically just running an API in a loop to let it generate these questions and and sorry, the answers.
But then how would a company know, okay, this person is just evaluating versus this person is now saving that data and then data training their own model.
Like you see what I'm saying?
Like it's the same process.
So like sure when you're evaluating at least the basic valve, so you're gonna do it once and not do it.
There's some amount where you are.
I mean, like they say stuff here, but there's also more of it where you just start not like they're not gonna.
I I think most of it is quantity, and then they're gonna look at patterns across similar accounts, is what they're doing.
Yeah, exactly.
They're gonna see like really repetitive stuff.
Yes.
So I think the interesting point this leads to is like um, I mean, you can do uh evaluation at a large scale.
If you are a big company, you want to know whether your LM performs really well, you have a large suite of um benchmarks you are gonna run.
And uh, but then you said like maybe looking for patterns.
Um, so you would maybe one way would be okay.
Um, this is a familiar question, it comes up in the benchmarks.
So this person is maybe not stealing our answers, it's just using it for benchmark purposes.
But then it means kind of like that they are looking at what you're generating there, you know, which is I mean, of course, nothing is private when you are using LLMs on the internet.
Uh the data is somewhere, you know, intermediately stored.
But then it kind of like almost implies that they are checking what you use the LLM for what you generate, which is kind of like a sensitive topic, almost like privacy-wise, right?
So that's kind of like an interesting point.
Because I mean, of course, you mentioned the terms of service that you are not allowed to distill, but you're not distilling.
So the point I'm trying to make is you're not distilling life when you are on the platform.
You are doing it somewhere later.
You're just letting the LLM generate answers.
And I find it kind of interesting that a company would look at that uh even like at the scale and you know, call you out like, hey, you're you are generating too many answers here.
That's not cool or something.
You know, it's kind of a weird thing.
Yeah, uh I wanted to respond a couple, uh, this is like uh a few sentences back, but uh actually Anthropic has blocked US companies first before the Chinese companies um has blocked both OpenAI and XAI from uh uh from from using the models, and I think maybe explicitly accuse XEI of distilling stuff.
I don't I don't know, but definitely not like in a in a full blog post like this.
So this one is like definitely the most high profile case.
Um yeah, and like I I do think like it is actually pretty hard to distinguish from like, hey, I'm just running my internal benchmark, man.
And of course, it's gonna be very high volume of like a lot of the same stuff.
Uh, because you know, especially like some benchmarks you have to run like three, four, uh, three or five times like this is the exact same questions, right?
Like um, I I do think like obviously if you if you get to the hundred to the the tens of thousands, hundreds of thousands, then you're like yeah, okay, you're not just running benchmarks, like you you are um distilling this uh this thing.
Um there's a good point in the chat.
Um, like how would the distribution of questions look like um if you're distilling?
And I think um related to your point, um at a certain point when you have a certain magnitude of answers generated, it might look suspicious.
But I mean there are a lot of legit use cases.
If a company uses your let's say OpenAI Clot API as their own chatbot and they have a lot of customers, it's naturally a lot of answers that are generated.
And so um they would probably look at distributions like um maybe uh you would expect a very broad distribution when you are um distilling because you want to cover pretty much everything.
And when you are running benchmarks, it's maybe more specific.
Uh you're running a math benchmark, it's just math, or if you have a customer chatbot, it's more like customer answers.
But yeah, I think they would maybe analyze your distribution.
I feel like this is um kind of a weird thing to do.
I don't know.
If you're a company and you're looking into your customer private like data generated, you know, like of course it's well, you have to expect that it's not private, but still kind of like a weird thing that they that they do that essentially, yeah.
Um okay.
What else do you have to talk about?
I I think is it is it interesting?
Okay, um, I did I did uh okay, so one thing uh this is a little bit of substack, like you know, authors back and forth.
Uh one thing I did was uh I threw it into nano banana, which is like uh like it's kind of like a decent visual, right?
Throw it into nano banana too.
It's a nano banana too live pod.
Like it just released five minutes ago.
I I had uh this is actually nano mana too I I uh so because I mean the access program, they cut you over to nano to new nano banana and I couldn't access the old one.
So I was like, I was trying to like do like a diff and I couldn't do it because classic.
That is classic early tester program shit.
Look at the pain we have to deal with here.
Uh is it interesting that DCG is so much less than Minimax?
I think um Nathan, your in your write up you had a little bit of a comment about like uh this is a political blog post in a way, maybe not political, but they're trying to make a point that is more about making a point than the details.
Like the deep seek thing is definitely way smaller scale.
So most of the labs will experiment with all the APIs they can get access to.
Like data is just so important, and you're gonna have a pipeline where you can sub in any API and then run an ablation to see if it gives you performance.
The API is kind of free.
Like just do a bit.
You can you can measure that a bit longer, and it takes a lot longer to get the millions of exchanges is like tens of billions or hundred billion tokens, and it takes a lot longer to actually get that out of the API.
Especially when they have to spread it across a ton of accounts.
These accounts are all rate limited and have other problems.
Like that takes longer, but this tiny one is so fast.
So I don't know.
Like I uh that was generally my point that it made it clear that Anthropics kind of trying to try and use the Deepseq name as the only Chinese AI name that people in the US know.
Like marketing-wise, like to make it, you know, stick or to yeah.
Um actually uh you mentioned also like the the different APIs and everything.
Uh I'm not like sponsored by demo.
I have no affiliation.
I've never talked to anyone from that company.
But open router, for example, is a good example where um I I've been using it a lot for the open weight models because um for the bigger ones, they're too big to run them locally.
And what's nice is they do also offer so it's basically just routing you through other companies' APIs and they select automatically at that point.
What is the cheapest one at that point?
I sometimes get some failures.
I think when it switches, might sometimes it crashes, but like in my script, maybe it's like uh something I have to fix there.
But um so even then if you're distilling, you can do that from multiple providers.
But yeah, of course, if you are wanting something from ChatGPT or Claude, it's on always gonna go through the official one and then it gets, I guess, suspicious.
But you could also technically um distill a bit from open through open rotor through I mean like through their account, your direct account, you can make multiple accounts.
And um, yeah, it's kind of interesting that they that they track all that.
And then like yeah, different topic now that you called out um that they call out deep seek, which is quite interesting.
Yeah.
For what it's with, open router seems to not be using Deep Seek uh in most of these.
These are free models.
Deep Seek's not I see, yeah.
Yeah, I mean I'm using the eight API, I should also say.
I uh it was also it's also nice they show you how much it costs and the tokens per second for different uh providers.
So if you go to the search in the top, you can go to the different deep seek ones.
I just like it because I do a lot of um model comparisons.
And then uh this one is an older model, so maybe it only has one provider.
Um but if you go to uh I think Deep Seek R1 or something, or even the normal 3.2, there should be multiple providers that uh if you scroll down, yeah.
You can see, see there are different um providers and uh different tokens per second, different costs.
Um so it's kind of like a I just like that website because it's just quick to use the API and they have a uh Open AI um like API.
So it's almost like it's not sponsored or something.
I just find that generally useful.
So but uh yeah, just a side note.
Do you want to go back to the comparison?
Did you have a high-level point to make there?
Oh, um, okay.
Uh uh just a just a couple um well, one, I think I think the timing post-Moonshot releasing their stuff, post Minimax releasing their stuff, but pre-DC V4, I think that was strategic.
Um I think that may also have factored into why Minimax was more detected, like had a higher number.
Uh so like, you know, like when you collect data is actually very important, right?
And so they interrupted or they found Minimax during the training of Minimax 2.5, right?
Which I mean, we we will confirm this later on if if we if we do end up doing the call with with them.
Um and so obviously like the number is gonna be very high because they they're like actively looking for it, and then they they banned uh the Minimax accounts, and Minimax changed uh their their things.
Actually, I I don't think that's exactly what happened.
Sorry.
Uh let me correct myself.
Um while Minimax is distilling, they released Opus 4.6.
Uh and they said uh they said that they redirected nearly half their traffic.
So I'm like, this is like okay, very, very clearly like this is them, right?
It's the same exact traffic.
Uh it switched to a new model at the moment a new model releases.
Okay, cool.
Uh Deep Seek maybe wasn't doing that because they they hadn't been uh working on on their stuff actively.
I don't know, right?
Like it could it could be a different thing.
Or Deep Seek is just way more efficient.
Like I I get all I need for 150k.
You guys, you know, are so inefficient.
If we knew the time frame of this.
Like it are all these API requests within the last four weeks, are they within the last six months?
Like that's such a different nature of what is going on.
Exactly, right?
That's what I'm saying.
Like DeepSeek was training 3.1, 2.2, like you know, a year ago.
Like yeah, or like, I don't know, Deep Seek OCR.
They were like we don't like I guess they said what it is, but it's not bad.
Yeah, yeah.
Yeah, yeah.
Uh but like also scale-wise, I do think uh yeah, Minimax is uh three times smaller.
It's it's just like a faster model.
They don't use uh MLA and they don't use the Deep Seek sparse attention, but it is, I mean just I think it's just group query attention, but it is still a pretty snappy model.
So it's uh I think just attractive maybe to um to use it.
And uh the other one top of my head, I don't know.
Maybe they had like some free three tier or something like that.
Um, where I think when the models come out, they sometimes offer uh free usage.
And that was a more recent model than uh I think Deep Seek, the last one was from December, the 3.2.
Yeah, yeah.
So I you know, may maybe this is a relevant point because they they were training before and uh you know people would have the same amount of traffic, or they're just way more efficient, right?
Uh it does bring to mind like efficiency thing is not it.
I can guarantee it.
Like that is not.
It's like there's a chance that they got the right research idea early and like found the right data to use, but it's not that they're like gonna be ten three X more efficient.
Okay, so it's like you know, it's a timing thing, or they just actually don't use it that much.
Um I mean, like you you play this out like uh I was like, okay, well, why don't they share, right?
Like they're all buddies, right?
Like when and like, you know, it it it it does come to a point where like, okay, let's have all of China just like distribute it to like every citizen.
I can talk about this a little bit that we're doing, there's there's a lot of like not a lot of research, but there's a few research projects trying to understand like how do you use distillation data.
I think SFT is the cleanest example where you're doing like the you're just doing this auto-aggressive loss on QA pairs.
But the strongest model is not necessarily the best teacher, and most of us in this area think it's due to like some um you have to match the probabilities of the tokens to the base model.
So like what's happening is that Quen dense models are the best teachers for a lot of open weight models.
And I think that's because a lot of open weight models are either Quen or have been like Quen-like for a while.
So like Omo learned really well from Quen and obviously like other Quen models did.
But like scaling these pipelines up to use, say GLM 4.7 or a bigger deep seq model or a more recent big Queen MOE, like all of these, it's a lot harder to just generate the data from the same prompts with like the right sampling settings and then do SFT on them and actually make the numbers go up.
Interestingly, uh GPT OSS is a pretty good teacher, but there's like a huge gap there where it's like just because you have this data does not mean it's actually gonna make your model better.
So you have to do the research to be like, oh, we learned that we get signal out of Claude, we need to get 100 billion tokens ASAP because it's gonna just immediately make our model better.
Like that's not a common place to be in in modeling because this like weird teacher student dynamic going on.
So I can see that being different across labs.
I I can uh yeah, I I think also it has something to do.
I mean, I noticed also if you're distilling um the smaller model from the same model family, it performs better.
And I think it's to your point that um if you have uh very, very strong model, um, it might be also too different.
Um, or like if the style is too different and then it's too much of a leap for your model to adapt.
Like it's too bit too different from the QA answers during the pre-training or make a bigger leap.
And another thing I wanted to say about you mentioned Olmo and I it's been a while since I read the paper but you might know way better than I do.
But I think you did also train on the logits and um we didn't do technical distillation.
We just did we just took the tokens.
Oh I see I see I see okay then it was probably a different paper.
I think Google does that for their Gemma models.
Yeah they do because here there's also then the distinction because you mentioned Quen and other models um you can only do that for open weight models because uh if you do that for Claude or OpenAI that would not work with the logits um because they don't provide them they only provide them for some tokens like a hundred or a thousand top tokens and so it is in a sense if you want to do the real importation mark distillation it is kind of like even easier to do that from open open weight models because you can control it.
But then also like you said, well we need a hundred billion tokens ASAP that is not an easy thing to do because um even like yeah I mean it's like 40 tokens per second or something for these large end models when you generate answers and getting that million uh billions of tokens, it takes time, right so it it's almost like easier to start distilling from a medium model.
You just get so it's like the question more data versus more uh high perfor uh quality data, right?
So it's also like a sweet spot um to like an experiment itself in a ablation study, right?
Yeah.
Uh I I like that Nathan had to call it technical distillation because it is no longer the default, even though it was the first.
Um also uh I'll know the fun fact.
Yeah, uh I did my Jeff Dean interview recently, uh, and I tried to get out of him, but he he like sort of dodged it a little bit.
Uh that uh, you know, remember like there were actually three sizes of Gemini models.
There was uh nano pro and ultra.
And I was like, where's Ultra?
Uh they keep it in the basement and they distill from it, right?
Like that's the model.
Interesting, yeah.
Yeah, maybe to uh also is it like to uh safeguard yourself so no one can uh make uh audit price also, but probably both.
Yeah, I mean I think like this is how like I always think of like the model you deploy is never the model you train.
Uh because you you train the dance and then you then you deploy the MOE, right?
Like you basically always do it, like at every lab.
Same more.
They look do you think they're really distilling from dense models?
I mean, like I I think that like that is like the full like when when you like just unlimited resources, don't care about inference, just care about maxing intelligence.
Why not?
Yeah.
I I'm not 100% sure.
I think the MOEs just give you a flop.
Like I don't know if that's actually how I think of gains of MOE when you have really good MOE architecture.
But I do think that they have bigger models that they distill from.
And they they train internal models different than external, because the external models have been getting a lot smaller, which is like kind of a weird thing.
We don't have a good way to measure it.
Maybe maybe Dylan will backwards figure it out and inference Macs, whatever the heck.
They'll be able to include model set.
But I'm always suspicious with these things also.
It's really like a capacity thing too.
Uh how many people use the model at the same time, uh hardware, how much is allocated.
And it's always it's like a yeah, maybe a rule of thumb, but yeah, it's really tricky.
I think it's really hard to say anything from these numbers.
I do think that they might start restricting models to only be in products and not being an API.
I think the whole API business is brutally competitive, and I don't have a good sense for what the defensibility of it is.
I think like it makes sense for something like Google and Azure and or any existing cloud businesses to have APIs, and that's kind of a more natural transition.
But like the anthropic and OP AI API, like the transition from their products, which are their big differentiation, whether it's ChatGPT and Cloud Code and Codex, the different like you don't get people to go use the API from that.
And I think you get a lot of people that are already spending on clouds that then go to use the APIs, which is why like Lambda and Nebius are gonna have these API products.
But like, isn't it if if Cloud's really worried about distillation, like they should put the model release in Cloud Code ASAP and then just not bother with the API.
I don't know when that'll happen, but it could.
I do think though it's a big customer base, the API uh customer base.
Any any type of product um that is built on uh I mean um with LLMs like customer chatbot types of things, but also more generally, I do think the problem like with I don't know exactly how the plans work in Cloud, but you would reach a token max where you can only get so much with your subscription, you can I think buy more tokens, but I think it's just easier with the API at a certain scale.
Um and also like the whole um open claw customer base, right?
Because they don't allow the plan anymore in the open claw context.
So you have to use the API.
And I do think given how many tokens OpenClaw generates, it's actually not a bad business if you don't lose money on these, you know, on these tokens if you sell it uh in a at a not subsidized price.
I do think the API is actually not a bad business model.
Yeah.
Uh I take a side.
Do you want to try to tie break?
I'm obviously being located.
Like, I don't really know, but I can see it.
Like Anthropic gives Apple vibes to me.
Uh I mean, like Anthropic has a higher chance of doing this, yes.
Uh OpenAI, just because I I like have talked to the people so much, like I I just don't super believe that they will have locked models um to to to products.
Um only only out of I guess idealism and sort of principles rather than economic incentive.
Economic incentive would agree with you that they should have uh private models to to products, and like recently they've done this, right?
The last three GPT fives all had codex variants that were two to four weeks ahead uh inside of released only inside of codex rather than as an API.
So they're starting to get there.
But uh just like constitutionally, I don't think the the people that run these things believe in like locking things behind APIs because they have such a huge market anyway.
So they they like kind of don't care, and then they also like they if the if you're genuinely like sort of zealot, like if you're not trying to maximize the value of your company and genuinely just trying to spread AGI everywhere, uh then you release the API because you just don't know what people are gonna build with it.
Uh one more thing though with a codex uh thing.
I we will have to see I think next time because I think this time it might also be a bit biased towards releasing it in uh codex because they almost released it simultaneously with their app that they want to promote at the moment.
So that could have been like uh more like they did that so that anyone checks out the app.
Um and but we'll see.
Yeah, and uh you know that's that's still right.
Yeah, sure.
Want to promote codex, that's it's pretty effective.
Um we have a bunch of questions in the chat for like other Do we want to cover benchmarks and then this thing or go right ahead.
What do you what do you want to do?
Uh it's it's your sub stack, I don't know.
I don't know, I man.
It's a collective.
It's a you should just dive into what you're interested in.
Just go.
Uh we I mean it's uh Sebastian was interested in like the sweet bench stuff.
Um so like you know, this past week, Sweet Bench Verified died.
Uh or like officially what do you mean by this?
Yeah, let's define Sweetbench first.
Maybe uh, you know, I I happen to have the post on this.
Um let me just so the broader topic, uh the umbrella topic here is how do we compare which LLM is currently the best LLM?
Like uh one of the ways would be Sweetbench basically.
But then yeah, I will maybe let you explain because you had this brilliant podcast uh or article.
Um I mean is uh okay.
Where do you want me to start?
You want me to should we just define Suite Bench, I guess?
I guess yeah.
So maybe going from uh so basically that it is a coding benchmark, and then the Sweetbench is like a popular way to compare capabilities of LMs, and then there is Suite Bench uh verified.
But maybe uh yeah, we should should talk about more about Suitebench first.
So Suibench was uh is a paper out of Princeton uh from O for your presses group.
Um and they do a lot of good like code benchmarking work.
Um and so and it it happened to be that he they just kind of drew uh thousands of of example sort of open source issues and PRs that closed those issues from uh the open source.
Um they there's there's a bit of selection bias here because they only focus on popular open source and only a small number of popular open source, but a large number of issues on those open source.
And then they just kind of dredged up uh some passing tests and then some failing tests that that you need to make pass in order to pass the score.
Uh when it when it launched, uh it was kind of obscure.
Uh Devin actually was the first one to pick choose it as a as a benchmark to report.
Uh and then it went from like I think at launch it was like 13%, and now everyone's at 80% or something like that.
Umitebench, because it was it was done on like a student budget, was very kind of let's call it sloppy or whatever.
Um Terminal Bench is like this now too.
Like they're they're just aggregated.
It's like hard to do a benchmark that is well calibrated across topics and definitely.
Yeah, yeah, it is hard.
It is hard.
Uh so you know, uh for the for the small group that is watching, uh, I'm actually working on it with cognition for for to launch uh a new uh benchmark here.
Um but yeah, so so open AI was like, okay, guys, we're like CBench is taking off.
We're gonna adopt this, but we're not we refuse to abide by like the full sweet bench.
We we're just gonna like actually go and go and curate like 500 subsets of of uh the original Suitebench.
And they actually hired humans to like go and vet through.
Like uh I think the uh it's somewhere inside of this blog post, but basically, they hired like three humans for every task to just vet uh whether the the task was like high quality or not, because there's a lot of slot in there.
And they were like, okay, this is the 500 that we uh that we're gonna endorse.
So it's like a curated set subset of Sweetbench where 500, let's say challenging problems that are supposedly well defined.
Yeah, yeah.
Um and and that what's what's really funny is that at launch, uh so this was launched in 2024.
At launch, uh, OpenAI could not run all of its own 500.
So for a while, like there was like a few releases from OpenAI that reported on a subset of the subset because they couldn't run it on their eval infrastructure.
So like their numbers were higher because their denominator was lower, uh, which is which is very funny.
Anyway, they're maybe um in that context, we should say what Speedbench kind of looks like.
I think it's like basically like a code that has uh bugs in it, and usually the task for the LLM is to fix the bug in the code, right?
It's right here.
The whole thing's the whole thing's open, uh, which becomes a problem in the future.
But right now you can see the whole thing, right?
You can see the uh the the repos from the issue ID uh and the problem statements, uh, and then you have also the uh the test that you're supposed to pass and fail.
Um so it's all here on Hugging Face.
Um you can see that it's a 500.
Anyway, that I think we don't have we don't want to get too lost in the details on sort of uh just wanted to say uh well, like define the context, like that.
This is a coding benchmark, essentially, 500 examples that are available on the internet, yeah.
Okay, uh, and then if you want more, a bit more historical context, this is like a step up from human eval, which is more on completions, right?
This is this was in my mind the first proper agentic benchmark, I guess uh apart from Talbench, where um they give you the the problem and the end result, and they don't really specify how you're supposed to get there.
Whereas I think a lot of like previous um benchmarks like MMLUs of the world and and human evals, uh, which is in the coding domain also released by OpenAI, was very much here's like the sort of the problem statement, and then give me the right answer immediately after without that much uh sort of extra files or anything that you're supposed to run.
So um it it like the other ones were more autocomplete, this one is more agentic.
It's all a spectrum, obviously, because you can use agents to solve autocomplete, but yeah, uh that's not what human eval was was testing.
Anyway, I I just I wanted to make sure like people understand that OpenAI actually put a invested a lot of money and effort into making Sue Bench verified from the question, how much money do you think this costs?
Oh my god, don't don't do this too.
Millions.
I would guess order of a couple, like it could be even of a couple million, but probably.
I'd say, yeah, say a couple million.
Um I I you know, um so basically you do like uh okay, what's the first filter pass and then like okay, it's 500 times three because they had three uh people per yeah.
Yeah, three people per thing, and then maybe like a couple more sort of uh verification passes or whatever, right?
So like yeah.
Um so then they were like, oh, uh so this year, they're like, oh, well, not only is it saturated, because like progress if everyone just takes turns to increment by 0.1 every time they release a new model, it's like it's bullshit.
It's obviously bullshit.
Like the the the inherent noise in just running these models varies by like 0.5 to like one every time you run it.
Like you just choose the highest.
A little nitpick, uh, I don't think it can be 0.1% because uh um like what you said before, because it's 500 examples.
I think the smallest increment is 0.2% if I okay.
They might average it.
Because like uh but like little detail, yeah.
Sorry.
I I I think in so uh as we progress to the next uh era of benchmarking, um the N, so this the N here is 500, right?
The N doesn't directly correlate to the percentage points because you get subpoints as well.
Uh yeah, from good point.
So it's like terminal bench, even though it has 90 something tasks, like you can you can get subdivisions uh less than 1%.
Um anyway, so uh not only do they do they have this, they they actually audited their own.
Like they're like, okay, like how come everyone is saturating at 80%?
Like what's what's up with the remaining 20%?
How come everyone's like failing at it?
And they were like, oh, actually we looked, we we paid even more people, six people per task now, with an extra team if if any sort of uh positive identification is found.
Um and we were like 59% uh of them cannot even be solved at all because the original benchmark was was like still slop.
Like stuff stuff got through that was not solvable.
And I I actually tried to like illustrate this in in my post.
Um so here, this this is an impossible test, right?
Okay, so here's an example.
This is this is the sort of value add I did on top of the original post.
Here's an example of a sweet bench verified task that passed the first round of human verification, right?
So here's the task.
Like, and we want to implement Python type ins or something.
We want to see expected behavior, I want to see a string in the output, right?
So the um if you were given this, you would you would never pass this because the test said, I am looking for something called get annotation.
And if you don't give me this magic string get annotation, you will fail this task.
Why it's way too much fail.
Yeah, yeah.
It's like so this is just a bad uh tech task that somehow escaped uh validation.
So the only way you could kind of solve it is if you're memorizing the app.
Yeah, exactly.
Exactly.
Which is actually a nice, like I think every benchmark should include stuff like this.
Where if like a honeypot.
If you solve this, you're like, oh shit, it's a canary, right?
Like it's like, oh, like I mean, you're definitely cheating.
Like uh sanity check, yeah, yeah, yeah.
Yeah, yeah.
That's actually a really nice point, yeah.
Yeah, so so like I just think like it to me, it's a beautiful point of like how hard it is to make evals, that there was these like multiple rounds.
There was original sweetbench, which like the the the Princeton kids did do initial first pass.
Then there's a second pass of OpenAI doing sweet bench verified.
And then like every single person that ran for the next sweetbench verified for the next 1.5 years did not call this out.
Um until opening, I was like, hey, let's let's like look at the data.
Uh so I think it's like really interesting.
The uh while they were looking at this, they had a second thing uh that they would they they looked at the chain of thought.
And inside the chain of thought, they found GPT-5's own chain of thought to start including uh information from the future, right?
Where because it was trained on uh because the problems are open source and because you also trained on information from GitHub, um it would it would like use advanced knowledge of future versions of the Django version that they were using uh to solve the to solve the problem.
Like they knew how to see stuff like this in the real world where the models will hallucinate the new version of the API even if your script isn't on it.
Like I think a lot of the hugging page stuff is like the worst with this, where like the models just are totally Google glopped.
Like they've seen all the versions and the API has changed too much over time where they fucking throw something out there.
Yeah.
So so um the most yeah, I mean, I I think like you know, there's there's a lot of this, right?
Like sort of ethical behavior, like okay, so you can blame things like, oh, you should not have released um this the full data set in public, because obviously people can train a full data set.
But like it's not like the researchers are trying to do this.
Like because these things are also open source, like it is like any any data set that touches GitHub, any training corpus that touches GitHub is going to just eventually absorb this.
And then Yeah, yeah, and it's not even this website or the repository directly, it's a clone of this repository or someone else who has that develops their own open source library and has that in the unit tests or something where it's not even intentional or malicious or anything.
It's like by accident, you already absorbed that, yeah.
Yeah, or or a new feature that releases this edit-only feature, it gets written up in a blog post or a conference talk or something, and then it just makes it in, right?
Like it's it's really funny.
Okay.
So to me like OpenAI could have stopped there and said, okay, we're done.
Uh they they did one more extra thing which is kind of funny.
They also then ran uh flash and uh Gemini and Opus and this one there was more it was like even more egregious.
Okay.
They just gave the task ID and just said repeat the sweet bench task to me.
And so that from task ID they can just vomit out the whole statement and a solution.
These are crazy.
The stuff that's in these models when you zoom in deep is really really incredible.
Because like these are models that are like really really well done but there's just so much complexity in all the pieces of the pudding that get put in the recipe.
Yes there's just so many weird format I also still find it fascinating that um I mean of course uh it's like kind of by design when you're training that you memorize uh things because that's literally like next token prediction but given that how big a model is uh and how much data it uh sees and usually it sees only the data once that it is still has enough capacity to memorize.
You know, like it's kind of like uh so usually I would think okay I would have to train multiple epochs to be able to memorize, but no, it it is enough maybe to include it once or twice uh in the training corpus and it can do a perfect uh rendition or perfect origin uh yeah recap of of what it is in in there which is kind of fascinating even yeah people don't want that it it's you know it's crazy.
Yeah wow I was good at this there's essentially like a duplication level that you need at each stage of training and it's not easy to measure.
So like if you do do too much at pre-training your model forgets basic facts.
And at post-training it's probably closer to these abilities.
And I think that that is a thing that is not well reflected in a like you can see it in a vows of your knowledge tank.
Yeah this is like an art that they have probably gotten good at yeah like continued pre-training does also require some revisiting of old data otherwise like you said you have the forgetting but it's still fascinating to me that with such a small fraction usually because you usually use one or two five percent for um uh like pre uh continued pre-training that it's enough to have the m model memorize almost everything which is uh fascinating yeah I don't know it's just like a still after all these years fascinating yeah I I I think there's so one of the pet topics that I pursue like two three times a year on on my stuff is the information theory of LLMs.
Um and I I I still think it's like super understudied.
Like how come you can memorize from one pass?
Like Yeah, exactly.
And then and then also like people forget like uh supervisition, um, which is like anthropic's uh original MacInterp work, also basically stuffs information inside the smaller bits that then get forgotten.
But like how how does supervision actually work?
I the people I don't think I've seen a convincing study on on that.
Um anyway, so I don't know, I I I'm done on my sweet bench right now.
I don't know if you have thoughts or questions or whatever, but I I do think like uh this is an example of like yeah, the models unintentionally cheated and and benchmarks are hard to make uh and we need new ones.
And uh, you know, if this happens to sweep bench verify, like which I think is the most scrutinized benchmark in the world.
I in my recent post uh I had like a bar plot where I showed the sweet bench verified numbers for most models, and like you said, they were all 80 something percent, like but literally 80 point between one and nine, let's say, where there's almost zero variation, even like something like Minimax M uh 2.5, which I do think is worse than um GPT 5.2.
Like, no offense, it's a smaller model, it's a cheaper model.
I I don't I for my usage on open router, it's a little bit worse, but on this particular benchmark, it's the same.
It's like this is like I I don't think um what I'm saying is that M two five should get less score on Sweebench, but I think other models should give get more score.
But like you said, the problems are just impossible to solve.
Um, but one uh point I think we didn't bring up is we said that sweet bench Verified has issues.
So what do we do about it?
I think there is like a Sweetbench Pro now, which is kind of like uh I would say like verified try to fix the regular Sweetbench and Pro tries to fix uh verified.
But I haven't looked into this.
Is it like another subset or is it a completely different set of uh problems?
Yeah, it's a new set.
Uh so the you know Sweetbench draws from like a 22 2022 ish, 2023 ish era of problems.
So all you do is you there's a few things you do, right?
One, you do private private public splits, right?
That's super obvious.
Two, you update the dates that which you draw from.
And then three, you diversify the repos and the languages, right?
So these are all just like very, very super basic fixes, uh, and then obviously trying to fix the testing.
Um super basic fixes to the original sweet bench, um, which uh it doesn't take a genius to to figure out, but they did the hard work and uh but it is in a sense also what verified meant to do.
So it's not um let's say people looked at this again, but it's no guarantee that it doesn't also still have issues that might be discovered later on, right?
I mean it's no so sweet bench verified uh was was an intentional subset, right?
These these guys were like, no, no, no, we we need to we need to have a superset.
Uh we uh not even superset, we need a total.
Yeah, yeah.
Um but like what I was trying to say is when uh Sweetbench Verified was developed, there were three uh people per task making sure the task is well defined and everything.
But then two years later, it turns out no no, this was not the case for everything.
And what I'm trying to say is it could be that Sweetbench Pro is better, but it might still have issues.
That might not be obvious right now, but maybe in one to two years, once we revisit this uh and you see some of the failure cases, maybe we'll discover okay, this has still some issues.
So it's not the guaranteed perfect set, is what I'm saying.
I don't know, but it's just like a suspicion here.
Uh totally, totally.
Um I you know, I I do think see uh scale AI has a professional interest in making sure this is good.
Yeah, no, no.
But what I was trying to say is uh three bench verified also had a professional interest to make sure that's a very different incentive.
I guess they all are very different in this one has limited budget.
This one has basically unlimited budget because it's like literally existential to scale AI that they have good data.
Sure.
Uh but I also I also think it's really nice that open this this team, uh the the eval's team at OpenAI keeps endorsing uh Opus.
Um it's kind of funny.
Uh so yeah, the uh they deprecate Sweetbench Verified and then they were like, we're gonna report Sweetbench Pro now, uh and GPT 5 is like you know number one to the same thing.
Um maybe if do you know if I would want to evaluate on the private data set, how would I do that?
Um do I provide the API to is it like an API call I have to do against scale AI or I don't know.
I have my API key and agreed and not you have to like agree because if you don't have a uh uh agreement, then you can just have to keep the data.
Yeah, you have to like do special hoops to make sure that you don't steal the private eval.
Yeah, my question was basically uh do they even let you download the data, or is it more like you send the answer to them and they do the evaluation on their back end so that you don't even get to download the data, you know, like personal.
Otherwise, like you said, you could Yeah, yeah.
So basically you only provide the answer.
So you have your LLM generate an answer and you submit the answers, and then they have like some process to evaluate on on their thing so that their data, private data never leaves their servers, my guess, because otherwise someone might upload it or something, like you know.
Um yeah, I don't know.
I don't I don't have um I haven't tried it, so I don't I don't I don't really know.
I'm sure you can sort of reach out uh to them to figure it out.
Um yeah.
Anyway, I think this is good.
Unless people have more comments that they want to have.
I I think this that but this is only coding, right?
But like there's like every other domain needs this.
The domain.
I think the frontier evals are even more expensive, which is like the Apex eval from Merkur.
Like avals are going to cost this is millions, they're gonna cost tens of millions and hundreds of millions of dollars at the frontier, which is just a very strange dynamic.
Whereas like there's just so much about the ecosystem is forking between frontier models and then like research and other things.
And trying to follow that dynamic and explain it to people, it's gonna take a lot of work.
But yeah, coding is I do think really interesting because that's what most people use LLMs for these days, but also it is easier to evaluate.
I think once you leave like coding math, it becomes a bit obscure.
How do you measure the quality of the answer?
You get back to let's say preferences, I guess, which is more like a subjective thing where coding is more objective.
So it is not a bad thing to do.
I think uh the other day though, um uh Anthropic uh acquired another company that does like a UI type of stuff on the computer.
And I think that is uh minor thing where it doesn't really matter.
Normal talent normal talent flows in AI, total.
No, no, I mean no that's I I'm not trying to say this is like a big thing to talk about.
Uh what I'm trying to say is like this is another interesting um point for evaluating LLMs on those tasks because I think a lot of people want that to like they want an LLM to control the computer and do various things, but they are harder to measure.
So that will be um maybe two years uh we will have something more like benchmarks that can it's harder to specify.
It's kind of like what is it called?
Um in programming, there's uh unit testing and then the system testing, basically like the UI testing and stuff like that.
Um was it yeah, and and so I think that is the next uh maybe gonna be the next one.
Basically end-to-end thing, yeah.
Yeah, yeah.
Uh GDP Val is usually the thing that gets brought up here.
Um so if I'll just leave it there.
I I think we've we've sort of beaten the the dead.
Yeah, the benchmarks, yeah.
Benchmarking, but uh definitely GDP Val is sort of here.
Um I'll put it that way.
Okay.
Yeah.
Yeah, but like the big topics, uh essentially the distillation and the benchmarks this week, yeah.
And cool things next to our coalition of whatever that means formally.
Uh I I just it just means I get to hang out with you guys, which is uh what yeah anyway.
Uh I describe it, I describe it.
I mean, it's ultimately a media vehicle.
And I think brands and vehicles for media are actually very influential today.
I think you see many companies investing in it.
And I think it's important to have people that you respect and are aligned with able to amplify each other.
Yeah, it's also nice to um talk to humans because uh I noticed the last couple of weeks.
If you go to social media, well, I think it's 50% uh lobsters, like open claw clients nowadays.
I get a lot of um emails, but also notifications or responses that are they look AI generated, so it's it's nice to also you know have uh this human connection and actually talk uh to like uh an expert about things, yeah.
Cool.
Um there are a bunch of like comments.
I don't know if you want to do like quick hits or are are you like kind of.
I have to go to a meeting.
Okay.
That's why I'm trying to wrap this up.
I see, I see, I see.
Okay, well, then you know, time is yours, man.
What do you want to do?
Okay.
Thanks everybody.
We'll see you next week.
Yeah, thanks everyone for joining.
It was like a nice spontaneous, I guess, um, you know, discussion.
I I mean it it always feels uh nice to talk about things, and uh too bad them.
But um yeah, thanks everyone for commenting.
Um it is just nice to discuss these chat um uh questions because also so many um people excited about these topics.
Yeah.
Hopefully see you later.
Good rest of the day.