T4E7 · R2E-Gym — Procedural Environments and Hybrid Verifiers

Transcript

MayaHere's a number that should bother you. Take a strong open model, give it a batch of patches for one bug, ask it which is correct — don't run anything, just judge. It tops out around forty-two percent. Now throw out the judge, run the tests instead — actual execution, pass or fail, same batch. Also around forty-two percent. Two completely different ways of telling right from wrong, and they hit the same ceiling.

LeoSame ceiling, opposite methods. That's either a coincidence or it's the whole paper.

MayaIt's the whole paper. Because here's the twist — they fail on different patches. Combine them, and you don't get forty-two. You get fifty-one.

LeoWait, what?

MayaThat's today. R2E-Gym. Two stories braided — where the training tasks come from, and how you tell a good patch from a bad one when you can't fully trust either method.

LeoOkay, but orient me first. Last time was Agent-RLVR — the coach who points at the handholds so a sparse reward can finally fire. Guidance manufacturing successes. Where does today turn off from that road?

MayaClean handoff. Agent-RLVR assumed you already had the wall — the executable tasks, the tests that say yes or no. Today's paper backs up and asks: where does the wall come from? And a second thing nobody's nailed — once the agent throws a dozen patches, who decides which one to ship?

LeoSo: build the gym, then pick the winner.

MayaBuild the gym, then pick the winner. Two levers from one paper, and I want to keep them separate the whole way.

LeoThen start with the gym. Every episode this topic we've hit the same wall — there aren't enough real, executable software tasks. SWE-smith forged bugs. SWE-RL skipped execution entirely. What's R2E-Gym's move?

MayaLet me name the problem first — the move only makes sense against it. To train an agent on a software task you need three things bolted together: a repository at some moment in time, a failing state you want fixed, and a test that objectively says "fixed." The expensive part isn't the code. It's that last piece — the oracle. The thing that grades.

LeoRight. And normally that oracle is a human-written test attached to a human-written issue.

MayaWhich is why the supply is thin — real issues with real tests, there are only so many. So R2E-Gym's pipeline — they call it SYNGEN, synthetic generation — stops starting from issues at all. Start from commits. Every merged commit is a change someone already made and validated. The history is full of free oracles, if you read it the right way.

LeoRead it the right way meaning what — how do you turn "someone changed these lines" into a gradeable task?

MayaThis is the part I love. And it's not SWE-smith's move — they broke working code on purpose. Here it's the opposite. You take a real commit, generate tests that capture what that commit does, then back-translate: reconstruct the problem statement and the before-state from the after-state. The commit becomes the answer key, and you manufacture the exam around it.

LeoHuh. Backwards from the solution.

MayaFrom the solution you already trust. The human change is the ground truth — you're not inventing a fix and hoping it's right, you're taking a fix the world already merged and generating the test and the prompt that would make an agent reproduce it.

LeoHang on, I want to pin the oracle, because that's where I get nervous. The test is generated. By a model? So now my grade depends on a machine-written test. What stops a sloppy test from passing a wrong patch?

MayaGood — exactly the right place to push. The defense is execution. A generated test earns its place only if it runs in the real environment and distinguishes the before-state from the after-state — fails on the broken version, passes on the committed fix. Can't tell those two apart, it's thrown out. The filter is brutal and automatic.

LeoSo the test has to do work before it's trusted. Fail on broken, pass on fixed, in a live container.

MayaThat's the gate. Out the other side: roughly eighty-seven hundred procedurally-built tasks — each a real repo, a real failing state, a verified executable test — without a human writing a single issue. Call it the landmark for the first half: the Commit Mine. You're not collecting bugs, you're mining the version history for oracles that already exist.

LeoThe Commit Mine. Okay. And the agent they train on this — base model, results, give me the spine of half one.

MayaA thirty-two-billion-parameter model, fine-tuned on those mined trajectories. On SWE-bench Verified — real held-out human issues, the honest exam — it lands at thirty-four point four percent, single attempt. From a procedurally generated gym with no human-written tasks in it.

LeoThirty-four from synthetic. Not bad.

MayaNot bad at all for the first lever. But hold that number, because the second lever is where it gets strange — and better.

LeoThe picking-the-winner half. The agent's deployed, it generates a bunch of candidate patches for one bug. Why is choosing among them even hard? Just run the tests.

MayaThat's the assumption I want to break. Which tests? On a fresh bug you don't have the held-out hidden test — that's what you're graded against, you never see it. You only have tests you can generate yourself. And the rub the paper measures: a verifier that ranks patches by your self-generated tests — pure execution — saturates around forty-two, forty-three percent. Can't climb past it no matter how many patches you throw.

LeoWhy not? Execution is supposed to be the honest signal. That's been the gospel this whole topic.

MayaBecause the self-generated tests are weak. The paper's word is distinguishability — low distinguishability. Lots of wrong patches pass the easy tests you could write. Execution tells the truth about the tests you have, and they aren't sharp enough to separate the contenders. Everybody clusters at "passes the easy stuff."

LeoAh. The ruler's too blunt.

MayaToo blunt. So the paper tries the other ruler. Don't run anything — show a model the candidate patch, the diff, the trajectory, and let it judge, the way a senior reviewer eyeballs a pull request and goes "mm, no, that's not it." Execution-free verification. Pure judgment.

LeoAnd let me guess. Also forty-two.

MayaAlso forty-two. Different ruler, same ceiling. And now we have a real fight, because the two rulers don't agree on which patches are good — they just happen to be wrong equally often. Which one do you trust?

LeoOkay, this is the argument worth having. Let me take the side I believe — execution. Give me a choice between a machine that runs the code and a machine that vibes on the diff, I take the one that runs the code every time. The whole topic has been "tests don't lie." A reviewer model judges on style, on what looks plausible. It can be confidently, fluently wrong. Execution at least touches reality.

MayaMy turn, and I'm taking the judge. Sharpest form. Execution only touches the reality of the tests you could write — and you just heard, those are blunt. The judge has seen thousands of real fixes.

LeoSo the judge knows things the test can't?

MayaIt knows a patch that adds a sketchy try/except to swallow the failing assertion is gaming the test, not fixing the bug — and execution will happily wave that through, because it passes! The reviewer catches what the test can't: did this address the issue, or just make the red light turn green?

LeoSure, but it also rejects the right patch because it's unusual. You said it two episodes ago — penalize the correct-but-different fix. The judge has a style bias; it rewards patches that look like the training fixes. A novel-but-correct solution reads as "weird" and gets dinged.

MayaGranted. That's real — the paper says it outright, the execution-free verifier leans on stylistic features and carries bias. So it over-trusts the familiar. But notice what kind of error that is. The judge's mistakes and the executor's mistakes —

Leo—aren't the same mistakes. The executor passes the test-gamer, the judge fails the weird-but-right one.

MayaDifferent failure modes. That's the thesis. The executor is fooled by patches that pass weak tests. The judge is fooled by patches that look wrong but aren't. So a patch that survives both is far likelier to be genuinely correct than one that survives either alone.

LeoHm. So you don't pick a winner between the rulers. You make them both vote.

MayaBoth vote. That's the hybrid verifier — the second landmark, the Two-Judge Bench. One judge runs the code, one reads the diff, you combine their scores — and because their blind spots don't overlap, the combination sees what neither sees alone.

LeoAnd the number. You teased fifty-one. From forty-two.

MayaSame thirty-two-billion model, best-of-n at test time, scored by the hybrid verifier: fifty-one percent on SWE-bench Verified. Each verifier alone caps in the low forties. Together, fifty-one. That gap is pure complementarity — no bigger model, no extra training, just two imperfect rulers that fail in different directions.

LeoThat's a huge gap for "just add a second opinion."

MayaAnd it puts an open thirty-two-billion model in the neighborhood of the proprietary systems people quote — the o-ones, Sonnet with tools. From a procedurally mined gym and a two-judge bench.

LeoLet me resolve it honestly, though — I don't want to pretend I lost. Did execution lose?

MayaNo — and that's the resolution, not a draw. Neither ruler won. The evidence forces: execution and judgment are complementary verifiers, not rivals. Execution owns the patches that fake the result — the test-gamer the reviewer might forgive. Judgment owns the patches that fake the form — the plausible patch that passes weak tests. You were both right about the other guy's blind spot. The fifty-one only exists because you stopped trying to crown one.

LeoComplementary, not rivals. I'll hold that. The honest version is: I was right that the judge has a style bias, you were right that execution is only as sharp as its tests — and the win was admitting both and adding them up.

MayaThat's the one.

LeoRun our bug through it. The thing we've carried all topic — real repo, a failing test, a few files, a hidden check the agent never sees. Both levers, where do they touch it?

MayaTraining side: before R2E-Gym, your bug-fix task existed only if a human filed it with a test. The Commit Mine says someone in this repo's history already fixed a bug like yours and the merge proves it worked — generate the test, back-translate the problem, verify it separates broken from fixed, and your style of task exists by the thousands.

LeoAnd test time?

MayaThe agent throws a dozen patches. The execution judge runs the self-generated tests — narrows it, but a few test-gamers slip through. The reading judge looks at the survivors: "that one swallowed the exception, not a fix" — and the patch that passes both is the one you ship. The hidden check you never see, approximated by two cheap rulers that disagree usefully.

LeoAnd the artifact. The topic trained me to ask — what's the reusable thing that outlasts this thirty-two-billion checkpoint?

MayaTwo things, one per lever. The Commit Mine pipeline — the recipe for turning any repo's history into verified executable tasks without humans. And the hybrid-verifier scheme — you don't need a perfect grader, you need two imperfect ones with non-overlapping blind spots. The checkpoint ages out. Both of those you keep.

LeoLimitations. "Two rulers that disagree usefully" sounds a little too clean.

MayaTwo that bite. First — the generated oracle. A test verified to separate broken from fixed for the commit you mined can still be narrower than a real human suite. The execution filter helps, but "passes my generated test" is not "is correct."

LeoThe blunt ruler problem, upstream.

MayaSame disease, earlier in the pipe. Second — the hybrid's magic is the complementarity, and that isn't guaranteed. It held because these two verifiers happened to fail differently. Scale the model, change the task distribution, and their blind spots might overlap — and the day they do, fifty-one collapses back toward forty-two. The paper measured the gap; it didn't prove it's durable.

LeoSo the same ghost as always. The result is real where they measured it, and the question is whether the conditions that made it real travel.

MayaThat's the surface to watch.

LeoGive me the spine. One breath.

MayaExecutable SWE tasks are scarce because the oracle is expensive, so R2E-Gym mines them from commit history — back-translate verified human fixes into tasks, keep only generated tests that separate broken from fixed — building eighty-seven hundred tasks with no human-written issues, and a thirty-two-billion model trained on them hits thirty-four percent on SWE-bench Verified.

MayaThen at test time it uses two graders with opposite blind spots — execution fooled by weak tests, judgment fooled by unusual fixes — and because they fail differently, combining them lifts best-of-n from the low forties to fifty-one.

LeoAnd the line I'm keeping is the one you opened with. Two rulers, same ceiling, different mistakes — and the whole win was refusing to pick between them. The blunt parts didn't overlap, so the sharp parts added up.

MayaThat's the one. A perfect grader is a fantasy. Two flawed graders that fail in different directions are a strategy.

LeoSo here's what I'm turning over, and I genuinely don't know the answer. The hybrid only worked because the two verifiers had blind spots in different places — and nobody designed that on purpose, it just happened to be true. So if you were building the next verifier, would you chase a single grader that's a little more accurate — or two graders that are deliberately wrong in opposite ways?