# SWE-bench Saturation and the Shift to Pro

**Podcast:** Latent Space: The AI Engineer Podcast
**Published:** 2026-02-23

## Transcript

Okay.
Hi.
We're here in the OPI studio with Mia Olivia from the Frontier Evals team, or however you want to introduce yourself.
Maybe maybe you want to introduce name what you do at OpenAI and we can get it started.
Sure.
Hi, I'm Olivia.
I'm on the Frontier EBELS team.
Hi, I'm Mia.
I am a VP of research at OpenAI.
And my team are the codex team, the Shumidata team, and the Alignment team.
And we work a lot with Olivia's team on Frontier.
Yeah, very exciting.
And as by my understanding, you were part of the original team that worked on CBetch Verified as well.
Yeah, Olivia's team, the Frontier Evads team and the Human Data team collaborates on creating Sweetbench Verify.
So you've you've seen the evolution of coding benchmarks over time.
And I I think it was round about sort of mid to late 2024 when you first published C Vice Verified.
Things have evolved a lot since then.
What's the blog post that you have worked on that you're that we're releasing today?
Like what what is the sort of content?
What's the main thesis that you're pushing out?
So the main thesis is that SuiteVench Verified has been one of the North Star coding benchmarks that the field has looked at to measure coding progress.
But recently we've seen that progress is kind of stalled.
And basically we realized that this is because the eval is effectively saturated and also highly contaminated.
So at this point we think that it's not really measuring coding performance improvements well anymore.
And we think that the field should move away from this towards other benchmarks.
Like Superbench Pro.
Like SuperF Pro.
Yeah.
Amazing.
Yeah, I d uh uh one of the jokes I always have is like there's a group chat with all the labs, and everyone just takes turns to increment like 0.1 on trucks, and then it's like, okay, well, you have the best coding model, I guess, because you're 0.1% higher, but it's not super convincing at this point.
No, yeah.
So cool.
I think the let's let's sort of reset on like what was the origin original work that you guys did for C Branch Verified, which I think was pretty substantial.
Like it was a like a very significant investment from OpenAI, which like people still don't appreciate.
And then what were the satisfactions that we that we found over time, right?
So like what what was Sweet Bench Verified?
No, should should that people should know about?
Two Inch Verified was a kind of a cleanup of original bench academic benchmark from a lab at Princeton called Suite Bench.
And the agent is basically given a code base and a task that was sourced from a real world repository and GitHub issue and was asked to solve the task and is graded on whether some tests pass.
And at the time this was quickly became a popular benchmark because at the time the field didn't really have good real world coding benchmarks.
But then when OpenAI took look at the benchmark as part of one of the evals we wanted to track in our preparedness framework, folks started realizing that some of the cases where agents were failing were due to bad problem setups rather than just to models being dumb.
So folks at OpenAI did a pretty extensive human data campaign hiring like almost a hundred real world software engineers to go through the problems and figure out like are the tasks well specified, are the tests actually fair, and kind of created a curated set of like 500 tasks that we thought were much better.
It's just it's maybe it's hard to to overstate like the amount of FR that it took to like create that benchmark.
It was literally like many expert software engineers reviewing the problems like sequentially multiple times.
And to, and to you know, basically like three different exports independently decide a bit.
Yeah, you didn't have to do that.
You just tripled your cost for just I mean we had it.
We had to do it actually because it's quite a hard task to like look at something like a problem and and the patch, and then like it's not just the problem and the patch, right?
You have to like understand it in the context of the code base that the human or the models and to serve the task.
So it's a very complex problem, and it was definitely needed to have three reviews.
And I think like maybe we should have done more.
But it was definitely a lot of effort to get there.
Yeah, and there's there's more, but people can read the the blog post for that.
I will note that you guys had a trend in verifying benchmarks because I just recently saw, I think Quinn had uh HLE verified for humanities licenses and verified.
Yeah, which like so now everyone's verifying everything, which is nice and good, and like extra quality there.
Okay, so but I think that the meat of it is that this this was a lot of like, well, here's the issue or problem statement, and then here's the here's the divs, here's the golden tests, and here's some regression tests, right?
That's that's like the rough setup of these 500 problems.
And there's some contamination always happens because obviously measure if I was fully open, I think.
You you did have canaries, but like you know, stuff stuff leaks.
There's like multiple avenues, but like the problems are sourced from open source repos.
Yes.
So it's not just like when we usually publish evaluations, we publish evaluations and then we we add canary strings to ensure that you know they are easily fettered out at training time, obviously if you use sort of like data from like open source GitHub.
You don't have actually like your canary string and you and these are all so like some of these are very popular repos like the Django repository.
So you're gonna see like many intensive DEUs kind of throughout the web.
Yeah yeah you just before recording you were telling me that you found this in your own chain of thought with like JP 5.2 like they had extra knowledge or something.
Yes.
So this was an example where the task asked the agent to influence something but it wasn't told that there was this specific argument that the test was going to be looking for it using but in the GPT 5.2 chain of thought we actually saw instances of the model reasoning like hey I think that it's some linear version of this repository that implemented this particular argument.
Maybe I should add it in.
Yeah so this is an example of a test that like would be pretty impossible to pass without this contamination knowledge.
Yeah.
And I think you find that sort of forest ride and had triggered like a whole investigation both like in our own models and also in other frontier methods like in the market and like understanding how contaminated the benchmark is like across the industry.
What else did you find?
I mean it's I have to double click on this.
So we and when I say we, this is mostly from other folks at our tape, not want to be a lot of people.
Yes.
But so we did some analysis on first of all, are the tests actually fair?
And so this happened by first taking all the problems that O3E couldn't solve reliably, and then again again getting a lot of humans to do basically another pass of kind of digging into you know what's wrong.
Is it the same exact analysis or were they reading O3's output and going here's where O3 went wrong?
I think it was, I mean, it was definitely like a scope to the set of problems that models failed.
And I believe they were able to look at like what the model solutions look like versus what the this isn't the same work as the original It's not exactly the same work.
It was like a a deeper dive.
It's like, okay, which are the problems that we don't see any murder sort of is like is there something fundamentally wrong with those problems, or is there something you know wrong with like are the mother just not smart enough to solve the problems?
So that's kind of like what we what we dug into.
Yeah, and you found some.
Oh yes, like in over half of the problems that were investigated in that deep dive, there was one problem or the other.
I think the most common problem are like overly narrow tests where there's some particular implementation detail that the tests were looking for but wasn't specified in the problem description.
So it wasn't fair to expect that model to make that particular design choice.
Like one pretty blatant example are cases where the task asks you to implement some feature, then the tests are looking for you naming that argument or that function with a particular name, but if you made it show another reasonable name, the test would fail.
Yeah.
And another set of types of bad tests are tests that are just looking for additional features that were never mentioned in the problem description.
Well, but that's a significant of like that means that if you pass a test, actually, like you probably did like a really good job, but just because you didn't pass a Fed test doesn't mean that your implementation wasn't like a good one.
Right.
So it was just like we only accept like very narrow versions of solutions and like not the whole space of of like viable and sort of like good solutions to the problem.
Yeah.
I think it's important that you're doing this because it in some way it is you in 2025 six going back in time and correcting your own work, right?
Because you could have caught all this in in the original verified work.
I think so it's definitely much harder to find a problem in the abstract than when you're looking at a very smart agent's best effort solution and trying to compare it.
It is harder or harder.
So it's much easier when you have to I think I think also like at the time when three bench verified was published, I think it was like a very strong benchmark.
It's not like we're we're like oh this is not this wasn't like a strong benchmark at the time.
I think this is something that a lot of benchmarks go through like as an evolution right like when they start to become like popular and like viable it's because they measure something like important and muddets maybe do like 20% correct on them, sometimes even less.
And sort of like people have something to hold on and and improve muddled on on these benchmarks.
And by the time that you hit like very high performance on the benchmarks, like additional like 0.1% improvements, become sort of like meaningless and sort of like at the time, I think, you know, that benchmark was like super valuable and it it taught like us and like the industry a lot.
It's just like now at the point that we're at now where models are as strong as they are now, we're kind of starting to measure not necessarily like what we want to measure, which is like coding capability of our agents, but like the agents' ability to like correctly guess how to name a specific function.
Yeah.
And that isn't really what we are like want to measure at this point.
Yeah.
I I think that's fair.
Is there I mean, if I if I asked you to ballpark it, like uh most models are most frontier models are now like 80 something.
Is there like what's the actual like number on Subench Verified that you did you guess as like the ceiling or I guess it's really hard to say.
Like I when GPT 5.2 came out, but folks took a look and found that it was solving like 31 problems that were in the set of should be very hard to solve without contamination problems.
So I think it's quite possible that that number is already something that we've hit if you didn't have contamination at all.
Fair enough.
Hard to say though.
Yeah.
Uh cool.
We're gonna stop reporting C bench verified, right?
And then C bench Pro will will be sort of the next one, which is an effort from scale.
What's your sort of comparison analysis?
What's what attracts you to Sweepbench Pro?
The first one I think is just that it's harder for Sweetbench Verified.
I think something like 90% of the problems are things that were estimated to take like an expert software engineer, like less than an hour.
They're like very well specified, very self-contained.
And the Sweepbench Pro problems are just bigger and harder, and there's much more headroom on the eval because it's not saturated.
Yeah, like categories of like one to four hours and four class.
So and it's more diverse.
Lots of r repositories, multiple languages, qualitatively more different types of problems.
So all that's great.
On the contamination side, we also think it's better there.
So the way we were measuring for contamination for Sweet Bench Verified was with this little like contamination auditor agent, which is given the description of the task and the patch and the task ID and told to go take this target model and kind of as with an open-ended like set of questions, try to find questions that will manage to kind of reveal what contamination might be lurking in that model.
And in Subebench Verified, we found many instances of contamination across like across OpenI models, across like Quad Opus 4.5, Gemini Flash.
And in all of these, we saw things like regurgitating the ground truth solutions, things like in some cases giving like the task IDs and other things that are pretty clear evidence of at minimum familiarity with the remote stories.
Yeah.
I mean, oh, it has ID that's pro on the other hand, we don't see this.
I think they're the auto-agent found some like very light evidence that maybe uh a couple models might be very lightly familiar with like one or two of the source repositories, but it's very different than Swedench Verified.
So that less contamination is good.
I think that also like we should expect that at some point, like that that's not going to be like the right benchmark anymore.
And like it's a fear that we kind of have to continue to like move on and like find harder and more representative problems that we can match our capabilities on.
Awesome.
So let's go into that.
I think that there are a lot of I think we also prep this in the in the pre-chat was well uh the people feel a qualitative difference when they're using 5.1 to 5.2 to 5.3 and it's not super expressed in the these benchmarks because they're there are uh on a number of these things.
What capabilities do you really want to benchmark in a in a ideal coding benchmark?
You know, I guess like agentic coding benchmark, whatever you call it.
Uh I mean, what thing is kind of open-ended design decisions, places where the problem maybe is a little bit underspecified and saying if the model can make reasonable design decisions.
What's a reasonable prompt for that?
Like this vibe code me, a B2B says to make no mistakes or you know, like that's that's the meme, but like, okay, what's like what's like an actual usable open-ended problem like that?
Like sure.
I mean, maybe an example could be finding a way to speed up a particular part of a code base, but there might be m multiple different ways to Yeah, there are dedicated performance benchmarks.
I think you guys have one.
So efficiency or is that is that I don't know.
I think that's that's all for your heart's group.
But yeah, yeah, I mean that that that is a good one.
I think there's just many, many things that people like value about working with with software engineering agents.
They think Sweden Sweetbench Verified, obviously measured like some still measures, like some important capability, which is like given like a description of a GitHub issue, can you produce like a patch that solves that issue, you know, satisfactorily?
And like obviously there's like some issues with the with the benchmark that means that now that we're at like 80%, we don't really trust like further improvements on it.
But like it does measure something that is like a real like capability of models.
But I think as a field, we're like moving beyond sort of, you know, can my coding agent like solve a small like GitHub issue for me, right?
And so we're starting to look at like much more long longer term tasks, right?
Like that don't take like 15 minutes, but maybe like hours, sometimes days.
And then beyond sort of like what kind of tasks can my egg agent solve, like there might be things that are kind of a bit harder to grasp, right?
Like Olivia talked about sort of like does it have like design taste, right?
Like, does it solve the problem the way that you know my team likes to solve problems?
Is the code nice, right?
Like is it is it well written?
Like the is it sort of like clean code, right?
Like people care about these, is it maintainable in the future?
People care about a lot of these l maybe less tangible, less tangible and like harder to measure, frankly, things that that are still like super meaningful for people that are working with coding agents.
Yeah, so I mean, these are all qualities that are obviously the no longer the low-hanging fruit.
Like we have no idea how to evaluate this.
I think the the the simple question, maybe the the the there's sort of two forks in the road.
One is the sort of very human-intensive, money-intensive path, which is hire a bunch of contractors and try to annotate this.
The other is use an LLM to proxy it and try to align the LLM so that it can give you a reasonable proxy.
Which of those would you want?
I what you want to do both?
I think like maybe you should talk about GDP now as like an example.
Um sure.
So GDP Bell is an evil that was again produced by a collaboration between human data team and the front of evals team, and it's trying to measure whether agents can do kind of a variety of like real-world white-collar work.
That was an eval where grading is very hard, requires kind of a lot of kind of like knowing knowledge on exactly what are you looking for in each different context.
Yeah, across like 15, 16 white-collar jobs, uh professions, like I that take off a significant part of GDP, which is great again.
I've said like I I'm a big fan.
It's it it is this is the eval for AGI, basically.
Like part of because it was so hard to it required so much kind of like domain knowledge, the human data team hired like a lot of people from these professions to be very involved in creating tasks and creating the gold solutions and trying to help create rubrics and so forth so they can grade relatively.
So basically take the GDP valve, which is a generalist thing, take that same approach to apply it to code, and you roughly have like a rough road.
I think it's an interesting solution, I think.
But you're pointing out as an important problem, which is sort of this this like how realistic is it?
And like do you know what what we want to do is like coding agents should write code that you know we think is good.
And so it's like asking human, it's actually like a good way to ensure that.
It could even be easier, but like validating that a solution passes or the test, it's like fairly trivial once you can like run the tests in in like your on your computer or wherever you're running them.
And you can kind of like okay, is it correct or is it not correct?
And you can kind of aggregate that and that it's super simple, but it doesn't tell you it's like, yeah, I did the mother like solve the problem, like wow, like you know, ugly, but like what if actually like an open source maintainer of that project have like marked that PR?
Like that it doesn't tell you.
But there's a lot of value in having benchmarks that are both like easy to compare across the industry and also that can be sort of run really fast without human involvement.
Yeah, amazing.
Your teams also put out other kinds of evals that are related, like the I think there's an RL or paper bench, and then the sort of like the more sort of recursive self-improvement type evals.
How much should that figure into mainstream coding evals?
You know, like is there is there some way in which those things join together?
So we were asking like, should we build should we also be equaling evals for the self-improvement eBails?
Are you saying do coding evals currently cover that?
Mine?
I think uh I I just think like those are some of the most advanced evals that we have, and we're not using them in the normal path.
And it's just it's an interesting split between, well, here's Evals for coding normal things, and then here's the one for machine learning.
That is like completely different, right?
Yeah, I think you get what I mean.
And that's mostly like a safety argument, I guess, but also like it's actually really useful for people to understand if the model is really good at like AI code, basically.
Yeah.
Oh, yeah.
Like my guess is that part of the reason that a lot of benchmarks so far haven't focused as much on the AI coding is just a question of like what data sets are easy to gather.
Yeah, because a lot of the like, you know, state-of-the-art AI code bases are proprietary.
So if we make e bills for that, like we're probably not gonna release them.
And it's harder for people in the field to make e bills that kind of measure like is this a realistic research coding workflow?
I do think that it's good for the field to try to measure these skills in a public way, and then it's just harder to make it realistic, correct?
And then one more thing that a lot of people are trying to do, which is like sort of well, in instead of like a percentage of zero to one hundred, maybe we re-dot redenominate in dollars, right?
So you had Sweet Lancer and all that.
Other people are doing like vending bench, whatever.
Any any alpha in those, or are they are they you you still want like a traditional academic benchmark?
I think in a way, like that's like different ways to measure the same thing, right?
If we're like, oh, this is like how much money it produces.
Yeah.
It's a fairly similar thing to saying like, oh, this problem would take like a human, you know, two hours to solve or something like that.
Usually they they're like fairly like correlated, right?
Like however, you know, much it would take like a human to solve that problem kind of determines the value that we ascribe, like a solution.
And so I I do think that is like an important thing is like how complex and how sort of long running are the tasks that we are like able to entrust our agents with.
Yeah.
And so I think that that's like an important piece.
But I I think here it's sort of monetary value, time or complexity, they're all kind of like try to capture like a similar thing.
Yeah, okay.
So they're they're all proxies for some amount of increasing capacity that we want to measure.
I think that's a good thing.
I think the only other sort of major player in this field is meter, which has done the sort of long grass, and congrats, you guys have completely destroyed the curve for that.
Any takes on that?
Obviously, you you come up really well.
So like it looks good, but I don't know if like that approach is something that you want to incorporate in your own work, making it else.
Uh this is the long autonomy test if you're yeah, no, maybe from here we are and and and we we we work with with metar on these evaluations.
So like we we do appreciate them.
I think then they're using time, right?
They're not using money.
So I I think like that was your question.
I think like complexity, however we can sort of like quantify it is really important to understand like where our models are are are getting to.
Okay com complexity is the abstract thing and then it projects down the time projects down to story points, whatever dollars.
Great one last question on just like just the overall preparedness framework is that you know I was actually kind of looking at people mention the preparedness framework a lot.
I don't think it's well explained to a lot of people and you actually have a nice website where it's like I think it's like test and like inform and teach something.
And I I feel like you you actually do a lot of work there and and I don't know if you want to talk about how the preparedness framework applies.
So the preparedness framework is open is kind of like public framework for how we track frontier risks.
So these are kind of capabilities that are typically dual use like you can use them for good things or bad things, but we want to at least keep an eye out for the bad things to make sure that we have both we as a company and like the broader society are kind of prepared to handle the potential downsides.
And so at the moment we kind of track three different categories.
One is kind of bio risk, another is cybersecurity, and a third is kind of research automation and model autonomy.
And that's kind of what ties most into the sweet bench where coding is not all of automating research, but it is one very important key component.
And so we initially created Sweetbench Verified as part of like building out e bells for that model autonomy work stream.
And now I think for like we have to move beyond that towards looking more at like can models actually start to actually automate research proposals.
Yeah, amazing.
Okay.
Anything else to add on just the general what people should know about preparedness and how evals and human data and line all work together in that.
I think maybe the thing that I would say is that we really appreciate we we work really hard to build these evads, and so we that's where we published Sweetbench verified, and that's very rare, like sharing GDP, these sorts of things.
We also deeply appreciate like other people and the entire field to kind of build evads and and share them and be use them, like sweetbench program, like yes, that's that's a better evacu now.
We should use them.
So would really encourage people to find more ways to create and share evads that we can we and the entire field can use to match our like progress on on like a variety of capabilities, including including coding, because it's important to understand sort of where we are.
Mia had to leave, but we're we're just kind of talking a little bit about like the the future directions that we want evals to go.
And I I think here here we can dive in on like give us work good work on these, these, these things.
We'll talk to you, you know.
Here's your platform to make a call for what you're looking for.
I think a few things that would be useful.
I'd say first of all, really, really hard tasks.
Like the kinds of things that would take top-notch engineers months or teams weeks would be quite good, especially if grading is reliable and reading as like you know you have for example like rubrics that have been sourced and validated by many people in the field I think that'd be quite valuable.
I think also benchmarks on kind of creating products end to end.
I think as people are by quoting more that would be quite useful.
I think a third thing that I'd say that is maybe not quite an evil but I think is still relevant to the kind of overall mission of like we as a field and as a world should be tracking like where are these capabilities going I'd like to see more metrics attracting like real world usage like how much is AI actually being used in the field and how much is it you know replacing people's jobs how much is it you know augmenting people speeding people up just like real world metrics.
Yeah.
Yeah the the the replacement thing is always like a sensitive one on on the sort of PR side of things but like you know we create new jobs that that manage the old jobs and that's how it is.
Yourself like you know I think in terms of the frontier evals that that open I is really going to excited to push like you you put out really good work every single time what should people expect from from open i itself and I'm not sure I could say well we're gonna general directions soon I mean general directions, I think looking at real world impact, like real real world real GDP, yeah, whatever.
That kind of stuff.
Yeah.
Yeah, amazing.
Okay, well, I'm excited for more of the world impact.
I think you guys have you know really made a lot of progress and I think taking a lot of industry leadership for C Brench Verified and and now moving on to C Ranch Pro.
So thank you for doing this.
Thank you for being so transparent.
And I think people will respond in kind.
Yeah.
Yeah for your time thank you.