Subscribe
Transcript
MayaImagine a lab has a fixed pile of compute. They can spend it on a bigger model, more data, longer RL training, more sampled answers per prompt, or more attempts at test time. Which knob should they turn?
LeoThat sounds like the whole AI industry compressed into one uncomfortable budget meeting.
MayaExactly. And today's source, Cameron Wolfe's "RL Scaling Laws for LLMs," is about why that budget meeting used to feel cleaner in pretraining, and why it gets messier when reinforcement learning enters the picture.
LeoSo this is not just "make everything bigger."
MayaRight. It is "figure out what kind of bigger you mean."
LeoLet's start where the article starts: pretraining scaling laws.
MayaIn pretraining, the classic mental model is fairly clean. You train a model to predict the next token. You measure loss on a held-out validation set. You estimate compute with something like model parameters times training tokens. Then you fit a power law. More compute, lower loss, in a surprisingly predictable way.
LeoThat is the part listeners may remember from our Mastering Language Models episodes.
MayaYes. If this part feels fast, queue up Mastering Language Models T2E1, "Scaling Laws for Neural Language Models: Predicting Progress." That episode is the clean pretraining version: loss curves, log-log plots, and why small experiments can forecast bigger runs.
LeoAnd T2E2, the Chinchilla episode, is the compute-allocation sequel.
MayaExactly. Chinchilla asks: if compute is fixed, should you buy more model size or more tokens? The answer shifted the field toward training smaller-than-expected models on more data.
LeoSo pretraining scaling laws gave labs a resource ledger. Model size here. Data there. Compute as the budget. Loss as the scoreboard.
MayaThat is the clean story. The key word is "clean." Not perfect, but clean enough to guide billion-dollar decisions.
LeoWhat changes in RL?
MayaThe y-axis gets messier, and the x-axis gets messier. In pretraining, the y-axis is usually validation loss. It is stable, dense, and measured over a big held-out sample. In RL for reasoning models, the y-axis is often reward, accuracy, pass rate, or error rate on a domain like math or coding.
LeoThat already sounds less universal.
MayaIt is. A reward curve in math is not the same object as a reward curve in coding. A reward based on a verifier is not the same object as a human preference model. A benchmark with easy questions behaves differently from one with rare, hard solutions.
LeoAnd the x-axis?
MayaPretraining compute is often approximated as one forward and backward pass over tokens. RL compute includes at least two phases: sampling completions from the current policy, then updating the policy from those sampled outcomes. So the budget can go into more prompts per batch, more rollouts per prompt, or more training steps. And those choices are not equivalent.
LeoThat connects to our Train-to-Test Scaling episode.
MayaYes. Mastering Language Models T3E9 is a good companion here. It makes the point that a model is expensive not only when you train it. It is expensive when you ask it to think, sample, retry, vote, or repair. RL scaling laws are another version of that same lifecycle-cost argument.
LeoThe article spends time on GRPO. Why?
MayaBecause GRPO has become a common public recipe for reasoning-model RL, especially after DeepSeek-R1-style training reports. The plain-English version is this: for one prompt, sample a group of completions. Score them. Use the group's average behavior as a baseline. Then push the model toward completions that did better than the group.
LeoSo instead of asking, "Was this answer good in isolation?" it asks, "Was this answer better than its siblings?"
MayaExactly. That relative comparison lets GRPO avoid training a separate value model. It still borrows the PPO-style idea of cautious policy updates, but it estimates advantage from the group.
LeoFor listeners who want the prior pieces, this points back to the RLHF topic.
MayaRight. Mastering Language Models T5E0 gives the broader RLHF map: preference signals, reward, and behavior shaping. T5E1, the PPO episode, explains the clipped update intuition that GRPO inherits. You do not need those to follow this episode, but they make the optimizer part much less mysterious.
LeoWhat is the hidden cost of GRPO?
MayaSampling. To compare completions inside a group, you need multiple completions per prompt. That means inference work becomes a major part of RL training compute. And that is the bridge to scaling laws. Once sampling is the expensive piece, the question becomes: should I sample more rollouts per prompt, use more prompts per batch, or train for more steps?
LeoThe article describes RL scaling curves as sigmoids rather than the classic pretraining power law. What should listeners picture?
MayaPicture an S-curve. Early in training, reward barely moves. Then the model finds useful behavior and the curve rises quickly. Eventually it approaches a ceiling. The important distinction is between two things: the ceiling, and the speed of reaching it.
LeoSo one intervention might raise the final ceiling, while another only gets you to the same ceiling faster.
MayaExactly. Wolfe's article summarizes ScaleRL-style studies that separate asymptotic performance from compute efficiency. In practice, that distinction matters a lot. If a trick raises the ceiling, it changes what the model can become. If it only improves efficiency, it may still be valuable, but you might recover the same result by training longer.
LeoThat sounds like a product decision, not just a research detail.
MayaIt is. If you are compute-rich but calendar-poor, efficiency matters. If you are trying to break through a quality ceiling on hard tasks, asymptotic performance matters more.
LeoAnd small runs can mislead you.
MayaYes. The article's warning is that a recipe that wins early may not be the one that wins at high compute. That is why fitting early training curves is useful: you are not just asking, "Who is ahead at step 500?" You are asking, "What curve does this setup seem to be on?"
LeoWhat does the article say about model size?
MayaThe broad trend is that bigger models tend to learn more efficiently from RL. But the caveat is compute constraints. With a tight budget, a smaller model can sometimes train for more steps and beat a larger model that consumes the budget too quickly. At higher budgets, the larger model may cross over and win.
LeoThat rhymes with Chinchilla, but it is not the same equation.
MayaGood distinction. Chinchilla is about allocating pretraining compute between parameters and tokens. RL scaling is asking a different allocation question: model size, RL data, sampled rollouts, and update steps all interact.
LeoAnd data reuse shows up too.
MayaYes. One of the article's practical points is that RL can be surprisingly robust to reusing high-quality data, as long as the data remains diverse and difficulty-aware. But that is not a license to loop over junk. It means the total number of useful optimization steps can matter more than the number of unique examples, within limits.
LeoLet's spend time on the most actionable part: sampling compute.
MayaThis is where the IsoCompute idea comes in. The article summarizes a study that treats RL sampling compute as a budget allocated across three resources: number of prompts per batch, number of rollouts per prompt, and number of update steps.
LeoIn symbols, it is basically prompts times rollouts times steps.
MayaRight. And the striking result is that, as the compute budget grows, it often helps to allocate more compute to additional rollouts per prompt, up to a saturation point.
LeoWhy would more rollouts beat simply training longer?
MayaFor easier problems, more rollouts can sharpen behavior. If the model already sometimes knows the solution, repeated samples help push it toward consistently correct trajectories. For harder problems, more rollouts increase exploration. You are trying to discover rare successful trajectories that one or two samples might miss.
LeoThis is exactly where our Agentic Coding Capability episodes become relevant.
MayaYes. T2E3, "Leaderboard Literacy," talks about sampling budget and why first-attempt success is not the same claim as best-of-many success. If an RL run trains on many rollouts per prompt, and an evaluation also allows many attempts, you need to know which claim is being made.
LeoAnd T2E13, "LLM-as-a-Verifier," connects because reward is often some kind of verifier.
MayaRight. In coding, a verifier might be tests, an oracle, a model judge, or a hybrid. If the verifier is noisy or gameable, scaling rollouts can scale the exploit, not just the capability. That is why Agentic Coding T2E2, "Public Tests, Hidden Tests, and Oracle Tests," is another useful companion. RL on verifiable rewards only helps if the reward actually verifies the behavior you care about.
LeoWhat is the live disagreement underneath all this?
MayaOne side says RL scaling laws are finally making post-training scientific. We can fit curves, extrapolate from cheaper runs, compare recipes, and allocate compute more deliberately. The strongest version of that argument is practical: frontier reasoning models are spending more compute on RL, so even imperfect predictive laws are better than expensive guesswork.
LeoAnd the other side?
MayaThe skeptical side says RL scaling laws are too bespoke. Change the reward, data difficulty, optimizer variant, batch construction, regularization, context length, or verifier, and the curve may change. The strongest skeptical argument is that pretraining scaling laws became powerful because the setup was standardized enough. RL does not yet have that stability.
LeoSo the synthesis is: use the curves, but do not confuse them for universal laws.
MayaExactly. Treat them like local maps. Very useful inside the terrain they were fit on. Dangerous if you pretend they describe the whole planet.
LeoIf a team is actually running RL experiments, what should they take away?
MayaFirst, define the y-axis honestly. Are you measuring reward, accuracy, pass@k, average score, hidden-test success, or human preference? Those are different promises.
MayaSecond, separate interventions that raise the ceiling from interventions that improve efficiency. Do not celebrate a faster climb if the final plateau is unchanged and your bottleneck is quality.
MayaThird, make sampling compute explicit. Track prompts per batch, rollouts per prompt, and update steps. If rollouts dominate cost, do not hide that inside a vague "training compute" number.
MayaFourth, fit curves early, but validate them later. The point of a scaling law is not decoration. It is to decide which run deserves the expensive continuation.
MayaFifth, cross-check the reward. In math, the answer can be verifiable. In coding, tests and verifiers have blind spots. If you optimize against the blind spot, RL will find it.
LeoThat last one is the warning label.
MayaYes. Scaling makes good signals more powerful and bad signals more dangerous.
LeoSo the short version is: pretraining scaling laws made compute feel predictable. RL scaling laws are trying to do the same for post-training, but the object is messier.
MayaMessier y-axis, messier x-axis, more knobs, and more dependence on the exact setup. But still useful. The article's best contribution is not a single magic formula. It is a way to ask better allocation questions.
LeoWhich knob are we scaling?
MayaModel size, data, RL steps, rollouts, prompts, context, verifier quality, or inference attempts.
LeoAnd is the thing we are measuring actually the thing we want?
MayaThat is the listener question for this episode. In your own AI system, if someone says "we need to scale RL," what is the y-axis you would trust, and which compute knob would you check before approving the run?