T4E5 · SWE-RL — Reinforcement Learning on Software Evolution

Transcript

MayaPicture a coach who grades a thousand attempts a day and never once watches the game. No scoreboard, no stopwatch — the coach holds the answer key, looks at what you wrote, and says, "Eighty percent of the way to what the pro did. Here's your score." Then sends you back out. That's the bet of today's paper. Train a model to fix software, with reinforcement learning, and never run a single line of the code it produces.

LeoWait. Stop. Every single episode this whole topic, the punchline was execution. Build the cage, run the tests, the pass-or-fail is the truth. You just told me the truth got thrown out.

MayaI told you the stopwatch got thrown out. The truth gets approximated — and that swap is the entire argument we're going to have.

LeoOkay. That's a real hook.

MayaQuick orient first. Last time, SWE-smith manufactured tasks — break working code, keep the breaks a real test confirms. Verified failure was the whole point. Notice what every paper so far has shared: somewhere there's a test that runs, and a green-or-red answer.

LeoRight. The gym had a floor you could stand on. Run it, it passes or it doesn't.

MayaToday's source pulls the floor out on purpose. It's called SWE-RL — reinforcement learning on software evolution — out of Meta, Yuxiang Wei and his collaborators. And the move is: stop building executable gyms one repo at a time. Use the thing that already exists at planetary scale — the entire public history of how software changed.

LeoUnpack "software evolution," because that sounds like a poster, not a dataset.

MayaIt's the least glamorous thing imaginable, which is why it's good. Every public codebase has a paper trail — an issue gets filed, somebody opens a pull request, they change these files in this way, it gets merged. Snapshots, changes, the issue text, the discussion. Millions and millions of them. That trail is the raw material.

LeoSo no Docker image. No test harness. No hidden checks.

MayaNone of it. And here's the shape of one training example. You hand the model the situation before the fix — the issue describing what's broken, the relevant code as it stood. You ask it to produce the patch. And then you grade that patch against the one a real human actually merged.

LeoGrade it how, though. That's the whole ballgame. If there's no test, "grade it" has to mean something concrete.

MayaIt means: how similar is your patch to the human's patch. Literally a text-overlap score. Line them up, measure how much of the change matches the change the human made, and that overlap — a number between zero and one — is the reward.

LeoHold on. That's not "did it work." That's "did it look like what the person wrote." Those are not the same thing and you know it.

MayaI absolutely know it. Say more — name the fear out loud.

LeoMy fear is mimicry. A patch can use the same variable names, touch the same lines, read like the human's fix — and still be wrong. And flip it: I could write a completely different patch that fixes the bug perfectly, and this thing punishes me because I didn't match the answer key. Correct, but low score. That's backwards.

MayaThat's the sharp version. Hold it — it's exactly the debate.

LeoBefore we fight, I want the mechanism nailed down. "Text overlap" is fuzzy. What actually computes the number?

MayaA sequence matcher — the same family of tool that powers a "diff" view when you compare two files. It walks both patches and finds the longest stretches that line up, then reports the fraction that matches. One means identical, zero means nothing in common. No model judging, no opinion — a deterministic ratio.

LeoOkay. Mechanical. I like that it's not another model grading.

MayaAnd there's a floor under it that matters more than it sounds. If the model's output isn't even a valid patch — garbled, won't parse, can't be applied — it doesn't get a low score. It gets a negative one. A penalty.

LeoWhy does that detail earn its keep?

MayaBecause it separates two failures. Producing a wrong-but-real fix is a near miss — partial credit, keep learning. Producing nonsense isn't a near miss, it's off the map, and you want the model to flee from that region hard. The negative reward is a wall, not a nudge.

LeoSo the signal isn't binary at all. It's a gradient. Cold, warmer, warm, hot.

MayaThat's the central move. Every executable gym gives you one bit — pass or fail — for an entire trajectory of decisions. SWE-RL gives you a smooth dial. "You're seventy percent of the way there" is a sentence the test harness can never say. And a smooth dial is something reinforcement learning can actually climb.

LeoClimb with what? You said reinforcement learning — which flavor?

MayaA group-based one. You sample a batch of candidate patches for the same problem, score them all against the human's fix, and then nudge the model toward the ones that scored above the group's average and away from the ones below. No separate critic model — the group is its own yardstick. Generate a spread, reward the better-than-typical, repeat a few million times.

LeoHuh. The crowd grades itself.

MayaThe crowd grades itself against the answer key. And because the answer key is just "the patch a human merged," and humans have merged tens of millions of patches, you have a reward you can compute on essentially all of GitHub. That's the scale story. No environment to build, so no environment to bottleneck you.

LeoAlright. Now I want the fight, because I've been sitting on it. You've got a reward that measures resemblance, and I claim resemblance is the wrong target. Take the other side. Make it strong.

MayaMake yours first. Sharpest form.

LeoSharpest form. Similarity reward teaches a style, not a skill. You are paying the model, over and over, to sound like the human who happened to fix that bug — their naming, their structure, their idioms. That's imitation with extra steps.

MayaAnd the cost of that?

LeoIt has a cruel edge: the genuinely creative fix, the one that solves it a smarter way the human didn't think of, scores low because it diverged. You are actively penalizing originality and rewarding conformity. A test doesn't care how you got to green. This reward cares about nothing else but how you got there.

MayaMy turn.

LeoGo.

MayaThe resemblance worry is real, and it's also overblown — start with what you're actually resembling. At scale, the only patches you can possibly resemble are the ones that worked. These are merged pull requests. The answer key isn't "a human's vibe," it's "a human's fix that shipped and closed the issue." Resembling that is not nothing.

LeoGranted, but—

MayaLet me land it — and this next part is the one that should rattle you. If all it learned was mimicry, it would get worse at everything outside this narrow task. Memorizing one repo's style should make you a worse general reasoner. Instead — they tested the trained model on things it was never trained on. Plain function-writing. Using unfamiliar libraries. Code reasoning puzzles. Mathematics. General language understanding. It got better at all of them.

LeoWait, really? Math? It never saw a math problem in training and math went up?

MayaMath went up. And the comparison that twists the knife: the boring baseline — just fine-tuning the model to copy patches directly, supervised — got worse on those outside tasks. Narrowed itself, like you'd predict from mimicry. The reinforcement-learning version, grading itself against patches, generalized outward.

LeoOkay. That's the fact I can't wave away. Pure mimicry doesn't make you better at arithmetic. Something more transferable got learned.

MayaThat's the concession I was hunting for. But you get one back, because I owe it. You're right that a different-but-correct patch gets under-credited. The similarity reward genuinely cannot tell "wrong" from "right-but-unconventional" — both just look far from the key. That's a real blind spot, not a rounding error.

LeoSo I'm not crazy.

MayaYou're not crazy, you're early. And here's where it actually resolves — it's not a draw. The soft, similarity reward buys you reach: every patch in public history becomes a training signal, no harness required, which is how you get the breadth that made math go up. Execution buys you certainty: a passing test is ground truth in a way resemblance never will be. They're not competing for the same job.

LeoSay the division of labor cleanly.

MayaSimilarity reward is how you train cheap and wide. Execution is how you verify — and how you'd test whether the wide training actually worked. Which is exactly what they did. Train on resemblance across millions of patches; then put the model on SWE-bench Verified, where the tests really run, and see if it holds up.

LeoAnd — does it hold up? Because I will not let resemblance off the hook until something executes and passes.

MayaIt holds. The trained model — a seventy-billion-parameter Llama, tuned with this reward — resolves about forty percent of SWE-bench Verified. Best reported, at the time, for any model under a hundred billion parameters. In the same neighborhood as the big proprietary systems people were paying for.

LeoForty percent. Trained without ever running the code.

MayaTrained without ever running the code, tested by running the code. That's the whole argument in one sentence. The resemblance signal was good enough to produce patches that pass real tests — which means it was tracking something about correctness, not just costume.

LeoLet me run our bug through it, the one we've followed all topic. Real repo, a failing test, a few files, a hidden check the agent never sees. Where does SWE-RL touch that agent?

MayaOn the training side, like the others — but the lesson it carries in is different. The SWE-smith agent practiced on tens of thousands of manufactured failures it could run. The SWE-RL model practiced on tens of millions of real human fixes it could never run — graded only by how close it came. So when it walks up to your dropped-bug ticket, it's not pattern-matching one repo's tests. It's drawing on the texture of how humans, everywhere, tend to repair this kind of break.

LeoAnd the artifact. The topic trained me to ask — what's the reusable thing that outlasts the model? Not the seventy-billion checkpoint, that ages out.

MayaThe reward recipe. "Score any attempted code change by its overlap with the change a human actually merged, and penalize anything that won't even apply." That's a few lines of logic that turns the entire public history of software into a training signal — no environments to stand up, ever. SWE-smith handed you a blueprint for building cages. This hands you a way to skip the cage entirely and grade against history instead.

LeoA grader you can point at all of GitHub.

MayaAnd that reframes the bottleneck one more time. The whole topic, scarcity was about executable tasks — the expensive, harness-bound kind. SWE-RL says: there's an ocean of non-executable signal sitting in plain sight, every merged pull request ever, and a soft reward is the net that finally fits it.

LeoLimitations, though. "Grade against history" can't be free.

MayaTwo that bite. The blind spot we fought over doesn't close — the reward still can't distinguish a clever-but-different correct fix from a wrong one, so you're leaving real solutions undervalued, and nobody knows how much capability that costs.

LeoThat's the one we argued. What's the second?

MayaThe ghost that haunts this whole topic, louder here than anywhere: you trained on basically all of public GitHub, then tested on a benchmark built from public GitHub. Police that overlap obsessively, or your forty percent is partly the model remembering the fix it saw in training, not reasoning to it.

LeoContamination. The bigger your training ocean, the more of the exam might be floating in it.

MayaExactly the surface to watch.

LeoSpine, one breath.

MayaEvery other gym in this topic runs the code to grade it. SWE-RL refuses to — it grades each attempted patch by how closely it matches the fix a real human merged, a soft overlap score instead of pass-or-fail, computed across millions of pull requests with no environment to build.

MayaThat dense reward trains cheap and wide, lifts the model even on tasks it never trained for — math, reasoning, language — and the resemblance-trained model still resolves about two in five real, executable benchmark issues.

LeoAnd the line I'm keeping is the one you opened with. The stopwatch got thrown out — but the answer key did the job.

MayaThat's the one. Execution tells you whether it worked. Resemblance, it turns out, can tell you which way is warmer — and a warm signal you can compute a billion times beats a perfect one you can only run a thousand.

LeoSo here's what I'm turning over, and I genuinely don't know the answer. A reward that measures "how close to the human's fix" can never reward the fix that's better than the human's — only the one that's closer. So when we train the next generation of coding models against human history, are we teaching them to reach the best work people have already done — or quietly capping them at it?