T4E6 · Agent-RLVR — Guidance and Environment Rewards

Transcript

MayaPicture a climbing wall where the only handholds are forty feet up. The rule is honest: touch the top, you get a point. Fall, nothing. Now put a beginner at the bottom. They jump, slip, nothing. Jump again, slip again, nothing. A thousand attempts, a thousand zeros. The rule never lied — and they never learned a thing. That wall is reinforcement learning on a hard software task. Today's paper is about the coach who finally walks over and starts pointing at where to put your hands.

LeoHang on, though. Last episode the whole fight was don't run the code, grade against history. Now we're back to "touch the top, get a point." Which is it?

MayaDifferent lever entirely. SWE-RL threw out execution and graded by resemblance. Today we keep execution — the test really runs, pass or fail is real — and we change something else. We change what happens after the fall.

LeoOkay. Go on.

MayaQuick orient. The thing we keep saying — "verifiable reward" — has a proper name, and it's the spine of this paper. Reinforcement Learning from Verifiable Rewards. R-L-V-R. Plain version: you only hand out reward when something objectively checkable says the answer is right. A unit test passes. No human opinion, no judge model — a machine says yes or no, and that yes is your only signal.

LeoWhich is exactly the clean, honest thing everybody wants. So where's the problem?

MayaThe problem is the sparseness. RLVR is a triumph on math and competitive programming, because there a decent model already gets it right often enough. It tastes reward, it climbs. But point the same machinery at a real software-engineering agent — multi-step, clone the repo, read ten files, edit, run the tests — and a base model almost never reaches the top of the wall on its own.

LeoDefine "almost never." Give me the number, because that's the whole premise.

MayaThe base model here is a seventy-two-billion-parameter Qwen, instruct-tuned. Put it on SWE-bench Verified cold, and it resolves a little under one in ten. Nine point four percent.

LeoOof.

MayaAnd now think about what that does to RLVR. To learn from reward, you need to occasionally taste reward. If ninety percent of your tasks return a flat zero, ninety percent of your training data is the model being told "no" with no information about why. The gradient is mostly silence.

LeoRight — that's the climbing wall. A thousand zeros teaches nothing. So the obvious fix is, what, make the tasks easier? Hand-pick the ones it can already do?

MayaThat's one road, and it's a dead end — you'd only ever train on what it already knows. The paper's move is the opposite. Don't lower the wall. Add a coach. This is Agent-RLVR — Jeff Da and his collaborators — and the coach has a name in the paper. They call it guidance.

LeoGuidance. That word does a lot of quiet work. I want it pinned before I trust it. Is "guidance" a polite word for "the answer"?

MayaGood — that's the exact fear to hold.

MayaNo. And the difference is the whole episode, so let me build it carefully. When the agent fails a task, the environment doesn't reveal the merged human fix. It assembles a set of informational cues about that specific failure and feeds them back in. Then the agent attempts the same task again, now carrying the cues.

LeoWhat's actually in the cues? Concretely.

MayaThink of three drawers the coach can open. The strategy drawer — a high-level plan: "the bug is probably in how the cache is invalidated; look at the second call path, not the first." The error drawer — the dynamic feedback from what just went wrong: the stack trace, the failing assertion, the test output the agent's own actions produced. And the interaction drawer — how the agent moved through the environment: where it wasted steps, where it edited the wrong file.

LeoSo it's hints, runtime errors, and process notes. Not the patch.

MayaNever the patch. The coach points at the handhold; you still have to reach it. And here's the part that makes it more than a hint system — the agent has to re-attempt and succeed on its own for the trajectory to count. The guidance gets it close enough to find the top of the wall. But the reward at the top is still the same brutal, honest unit test.

LeoAh. So the hint doesn't earn the point.

MayaThe hint earns the chance. The test still earns the point. That's the hinge of the whole design.

LeoLet me say it back. A task that returned a flat zero forever — guidance turns it into a task the agent can occasionally solve. So now that task produces a reward signal where before it produced silence.

MayaThat's it exactly. Guidance isn't really teaching the model the answer. It's manufacturing successes on tasks that were previously all-or-nothing zeros — and those manufactured successes are what RLVR has been starving for. You're not changing the reward. You're populating the part of the wall where reward was reachable but never reached.

LeoHuh. The coach builds the curriculum out of your own near-misses.

MayaLet me give the landmark, because there are really two coaching moves and they're easy to blur. Call them the Spotter and the Scout. The Spotter is the guidance loop we just walked — fail, get cues, re-attempt, and a unit test certifies the win. That's the training-time coach. It builds the data.

LeoAnd the Scout?

MayaThe Scout shows up at test time. Those same trajectories — all the attempts, failed and rescued — turn out to be exactly the fuel to train a reward model: a separate model that looks at a finished attempt and predicts "this one's probably right," without running anything.

LeoWait — why do I need that if I already have the unit test?

MayaBecause at deployment the agent generates several candidate patches, and you want to ship one. The Scout reads the candidates and calls which is most likely to pass. It's best-of-several, decided by a learned eye instead of a coin flip. The Spotter makes the model better. The Scout picks better from what the model produces.

LeoTwo coaches, two jobs.

MayaAnd both fall out of the same guidance data. That's the elegance. The failed attempts aren't waste — they're the negative examples the Scout needs, and the rescued attempts are the positives.

LeoNumbers. I need to see the wall actually get climbed.

MayaBase model, cold: nine point four percent on SWE-bench Verified. Run the Spotter — guidance-augmented RLVR — and it lands at twenty-two point four. Then add the Scout at test time, the learned reward model picking the best candidate, and it reaches twenty-seven point eight.

LeoFrom under ten to nearly twenty-eight.

MayaOn the same base model, same benchmark. Roughly tripled. And the reading the paper wants you to take: most of that lift came from making the sparse reward reachable, not from a bigger model or a softer metric.

LeoOkay. That's where I want to plant the flag, because I think there are two ways to read this result and they fight. Set them up and let's argue it, because I don't think it's settled.

MayaLay out your side first. Sharpest form.

LeoSharpest form. The real teacher here is the verifiable reward — the unit test. Guidance is scaffolding. Useful scaffolding, sure, but temporary. The test is what tells the truth, the test is what certifies every win, the test is the thing that generalizes. Strip the guidance away after a few rounds and the model would still be learning from the same honest signal it always was. Guidance is the ladder you kick away. Don't confuse the ladder for the building.

MayaMy turn.

LeoTake it.

MayaThe ladder is the result. A reward you can't reach is not a teacher — it's a closed door. For this model, on these tasks, the unit test alone delivered ninety percent silence. You cannot learn from a signal you never trigger. Guidance isn't decorating the reward; it's the only reason the reward ever fired. The honest test was always there in plain RLVR — and plain RLVR on this model would have crawled, because it had almost nothing to climb toward.

LeoBut that's a starting-line problem. Once the model's better, it reaches the top on its own, and then guidance is dead weight. You're describing a crutch for a beginner, not a principle.

MayaPartly — and I'll give you that the crutch gets less load-bearing as the model improves. But watch what you just conceded. You agreed the beginner can't learn without it. That means on any genuinely hard frontier — the tasks even a strong model fails — you're a beginner again. The sparse-reward problem doesn't go away; it just moves up the wall. Guidance isn't a fix for weak models. It's a fix for hard tasks, and there are always harder tasks.

LeoHm. That one lands. The frontier always has a region where reward is unreachable, so the coach always has a job. Fine. But I get a concession back.

MayaYou do. Name it.

LeoThe honesty of the final signal is still all the test's. Guidance got the model to the handhold, but only the unit test can say it actually held. If guidance ever started handing out partial credit for "good effort," the whole thing rots into the mimicry trap we spent last episode on. The scaffold is safe because the reward stayed strict.

MayaAgreed, fully — and that's the resolution, not a draw. It's a division of labor in time. Guidance owns reachability: it manufactures the successes so the gradient isn't silence. The verifiable reward owns truth: nothing counts as a win until a real test passes. Coach to get you to the wall, judge to say you topped it. Neither one alone gets you the climb.

LeoCoach for reach, judge for truth. I can hold both.

MayaAnd the thing that settles it in their favor is the same move every paper this topic respects — they tested where the test really runs. Twenty-two percent on SWE-bench Verified, real containers, real hidden checks. The guidance happened in training and then left. What got measured at the end was the model climbing on its own.

LeoRun our bug through it. The one we've carried all topic — real repo, a failing test, a few files, a hidden check the agent never sees. Where does Agent-RLVR touch that agent?

MayaOn the training side. Before this, our agent either solved your dropped-bug-style task or it didn't — and if it didn't, that task taught it nothing, pure zero. With the Spotter, the first time it fails, the environment hands back the stack trace, a pointer at the likely file, a note that it burned six steps in the wrong module. It tries again, lands the patch — and now your task, once dead weight, is a live training example.

LeoAnd the artifact. The topic trained me to ask — what's the reusable thing that outlasts the checkpoint? Not the seventy-two-billion weights.

MayaThe guidance-augmented trajectories. Every failed attempt, the cues that were injected, the rescued success that followed. That data does double duty: it's the training signal that lifted the agent, and it's the exact fuel the Scout learns from. SWE-RL handed you a reward recipe. This hands you a coaching transcript — worth keeping long after this particular model is retired.

LeoLimitations. "Add a coach" can't be free.

MayaTwo that bite, and I'd give them names. Call the first one the coaching cost. Somebody has to generate the guidance, and good cues aren't free. A bad pointer sends the agent confidently into the wrong file; the quality of the coaching caps the quality of the data, and the paper leans on a capable model to produce those cues.

LeoAnd the second?

MayaThe deeper one I'd call the coach-gone worry — the model gets good at solving tasks with a coach behind it. How much survives when the coach is gone for keeps, on a task nobody's near-miss ever covered? It climbs solo at test time, which is real — but the frontier where guidance can't reach is still the frontier where we don't know.

LeoSo the same ghost as always, wearing a new coat. We measured it inside the conditions we built for it.

MayaThat's the surface to watch.

LeoGive me the spine. One breath.

MayaVerifiable reward — the honest unit test — is too sparse to teach a coding agent that almost always fails, so the reward fires too rarely to learn from. Agent-RLVR keeps the strict test but adds guidance: on a failed attempt the environment feeds back strategy, errors, and interaction notes, the agent re-attempts and has to win on its own, and that manufactures successes where there was only silence.

MayaAnd the payoff doubles up — tripling resolve rate on a seventy-two-billion model, and the same logged attempts then train a reward model to pick the best patch at test time.

LeoAnd the line I'm keeping is the one you opened with. The rule never lied — but a rule you can never satisfy teaches nothing. The coach didn't change the rule. The coach got you to where the rule could finally say yes.

MayaThat's the one. The test tells you whether you made it. Guidance is the only reason you ever got close enough to find out.

LeoSo here's what I'm turning over, and I genuinely don't know the answer. If the only tasks a model can learn from are the ones a coach can get it close to — then the shape of what it learns is set by the shape of the coaching, not just the rewards. So when we coach the next generation of agents, are we teaching them to solve hard problems — or only the hard problems we already know how to hint at?