RLAIF vs. RLHF - Scaling Reinforcement Learning with AI Feedback

Transcript

MayaA Reddit post is on the evaluator's screen. Someone bought a gaming computer without telling their mother, and now she wants it returned. The summaries disagree about which purchase mattered and whose money paid for it. Which summary should get rewarded?

LeoI'd want the original post beside me. A smooth summary can still change the story.

MayaThat's an evaluation example in Harrison Lee and colleagues' paper, RLAIF versus RLHF. Can a language model supply preference judgments to train another model, with results people like as much as training from human judgments?

LeoWe're reading the local paper's twenty twenty-four revision. These are PaLM two experiments, not evidence about today's frontier models.

MayaThink of an assistant grading drafts. The marking can be delegated; someone still has to decide what the rubric should reward. Technically, R-L-H-F means reinforcement learning from human feedback. R-L-A-I-F substitutes AI feedback in the preference-labeling step.

LeoOkay. What actually learns?

MayaStart with supervised fine-tuning, or S-F-T. A language model learns to imitate examples of useful responses. That supplies a starting policy. Policy just means the model's rule for choosing its next token, given the prompt and what it's already written.

LeoThen we need comparisons. A context, candidate responses, and a preference between them. Here, those pairs come from existing preference datasets. For the AI-feedback condition, the researchers replace the human preference labels with judgments from a prompted model.

MayaRight. The grading assistant's judgments become training data for a separate reward model. It sees the context and a candidate response, and produces a score. Training makes the preferred response score higher relative to the other one—

Leo—without telling the policy which word to copy. But how does the AI labeler express a preference?

MayaThe paper prompts it with the context, the candidates, and rating instructions. It reads the model's probabilities for the candidate-choice tokens and normalizes them into a soft preference distribution. So the training target can preserve uncertainty instead of recording only a winner.

LeoEven the order matters.

MayaAnnoyingly, yes. Lee and colleagues find position bias. They judge each pair again with the candidates reversed, then average the distributions after matching them back to the responses. That reduces the influence of where an answer appeared.

LeoAnd the instructions aren't neutral wallpaper. For summaries, their detailed rubric asks about coherence, accuracy, coverage, and overall quality. That's already a human choice about what a good answer means.

MayaThey also try asking for a rationale before extracting the preference. That generally helps alignment with human labels. Detailed instructions and example demonstrations have mixed effects across tasks, though. More elaborate prompting doesn't reliably mean better judgments.

LeoQuick check for you listening: if the labeler agrees with a human on a preference pair, have we shown that the trained assistant improved? No. We've measured agreement on that pair. We still need to train the policy and evaluate its outputs.

MayaAnd there's a model in between. The reward model learns to predict those soft preferences from its response scores. That's distilling the labeler's judgments into a model we can call throughout reinforcement learning. The original AI labeler isn't called for every policy update in this canonical setup.

LeoSo a high reward means “this scoring model favors the response.” It doesn't certify that every claim in the response is true.

MayaNow the policy generates a complete response. The reward model scores it. In the paper's REINFORCE setup, that reward arrives at the end; the training algorithm changes the probabilities of the tokens that produced the response. There's also a learned value estimate—

LeoMm. What's that buying us?

MayaA reference for how much reward was expected from the context so far. The update uses how much better or worse the outcome was than that estimate. Roughly, unexpectedly rewarding generations become more likely. The baseline makes this noisy learning signal easier to use.

LeoThe paper also penalizes drifting away from the starting policy, using K-L divergence. Otherwise, the policy can discover peculiar language that scores well without actually being useful.

MayaYes. That penalty limits the drift; it doesn't make the reward model infallible. We're still optimizing a learned approximation to preference. An error in the judgments can become an error in the reward, then a habit in the policy.

LeoTry another check: do we improve the assistant by training only the reward model? No. That changes the scorer. The separate reinforcement-learning step changes the assistant that will answer users.

MayaSo where does the evidence come from? Human evaluators compare generated responses on held-out test contexts. For summaries and helpful dialogue, they rank quality. Harmless dialogue gets a different evaluation: each response is rated harmful or harmless.

LeoThe paper's main results table gives AI-feedback training a seventy-one percent win rate against the supervised baseline for summarization. Human-feedback training gets seventy-three percent. Those are separate comparisons against the baseline, not a direct contest between AI and human feedback.

MayaOh, good catch. Otherwise I hear those figures and immediately imagine the wrong comparison.

LeoIn the direct summarization contest, the table reports fifty percent for RLAIF against RLHF. The authors don't find a statistically significant preference between them. That's evidence of comparable performance in this experiment, not proof the methods are identical under every condition.

MayaThere is a design wrinkle, too. The training appendix says the summarization reward models started from different checkpoints: the AI-feedback scorer from the summarization model, the human-feedback scorer from the base Extra-Small model. So this is a comparison of their implemented systems, with that qualification.

LeoThat matters. I can use the result without pretending the label source was the only difference.

MayaHelpful dialogue looks similar: in that same table, AI feedback wins sixty-three percent against the baseline, human feedback sixty-four percent. Again, the difference isn't statistically significant. The AI preferences mostly come from PaLM two Large; the policies and reward models use Extra-Small models.

LeoWait, the harmlessness result is much less close. Human evaluators rate eighty-eight percent of RLAIF responses harmless, compared with seventy-six percent for RLHF and sixty-four percent for the baseline. The paper reports a significant improvement there.

MayaBut keep the metric attached. That's harmlessness on this test distribution. It doesn't establish that the model is equally helpful, or that those percentages transfer to adversarial use in some other setting.

LeoHere's my objection. Both trained policies produce longer answers. People often reward detail. Are we paying for a better summary, or just a longer one?

MayaThat changes the apparent size of the gain. In the paper's length-control appendix, summarization wins against the baseline fall to fifty-nine percent for RLAIF and sixty-one percent for RLHF. Both still improve, and remain close, but the raw result overstates the length-adjusted advantage.

LeoThat's a pretty large correction. How did they make it?

MayaThey fit a statistical model relating response-length ratios to preferences, then estimate the outcome for equal lengths. The authors explicitly call that imperfect. It relies on the fitted relationship; it isn't a fresh randomized experiment holding length fixed.

LeoI'd accept “useful substitute on these tasks.” I'd still reject “AI feedback solves human alignment.” The evaluator can inherit biases, and matching crowd judgments doesn't settle whose preferences deserve priority.

MayaI agree about that boundary. The paper's impact statement warns that AI feedback can transfer and amplify the labeler's biases. It calls carefully trained human experts the gold standard for high-stakes domains such as medicine, law, and employment.

LeoAnd even here, “no human preference labeling” doesn't mean no humans. People write the rubric, supply the starting data, evaluate results—

Maya—and inspect checkpoints. The training appendix describes manual inspection when selecting the final policy. The intervention being tested is narrower: replace the preference labels used for reward training, then see what happens downstream.

LeoStill, the scaling argument is real. Their cost appendix estimates AI labeling at more than ten times cheaper than human labeling.

MayaWith historical service-price assumptions. They use a GPT-four pricing proxy for the AI estimate, not a measured invoice for these PaLM experiments. That estimate concerns labeling; it isn't an end-to-end training cost comparison or a current price quote.

LeoAll right, cheaper labels are still useful. And this surprised me: the labeler doesn't always need to be larger than the policy.

MayaOn summarization, the paper's same-size setup uses Extra-Small for both. People prefer that trained policy to the supervised baseline sixty-eight percent of the time. But the labeler is instruction-tuned, while the initial summarization policy was fine-tuned on Reddit summaries. Same size isn't the same checkpoint.

LeoSo I shouldn't call that a model teaching itself from exactly its own starting weights.

MayaRight. There is an exact-checkpoint result in helpful dialogue, but it uses their other method: direct RLAIF. And that returns us to a weakness of the separate reward model. As the policy changes, it generates responses unlike the ones the scorer learned from—

Leo—and the scorer can become stale. The paper proposes skipping that separate model and asking the general-purpose labeler to score the policy's new responses directly.

MayaYes. It prompts for a quality rating from one to ten, uses the probabilities over those ratings to calculate an average score, then rescales that score for reinforcement learning. It's still a learned judge providing a reward; we've removed the reward-distillation step.

LeoWhat's the cleanest comparison? Because swapping the labeler size and swapping the reward method together would muddy it.

MayaThe direct and same-size summarization conditions both use the Extra-Small labeler. The paper reports that humans prefer direct RLAIF to same-size canonical RLAIF sixty percent of the time in their head-to-head comparison.

LeoThat supports the method. Does it prove staleness caused the difference?

MayaNo. The authors hypothesize that avoiding staleness and avoiding distillation both contribute. The result doesn't isolate their contributions. And direct scoring means calling the judge during reinforcement learning, so it changes the computational trade-off too.

LeoNow the exact-checkpoint claim?

MayaIn helpful dialogue, the starting policy and direct-feedback provider are the same checkpoint. The resulting policy beats the supervised baseline sixty-six percent of the time, according to the paper. That is evidence of improvement from that starting checkpoint under their evaluation, not evidence of unlimited self-improvement.

LeoBack to the computer purchase. If the judge likes a plausible summary that changes whose money was spent, the whole procedure can reward the mistake.

MayaIt can. Their qualitative analysis finds plausible hallucinations in some human-feedback summaries, and less fluent or repetitive wording in some AI-feedback summaries. But their follow-up comparison doesn't establish statistically significant differences in those qualities. We shouldn't turn selected examples into a general winner.

LeoThen I'd keep human review for errors the rubric misses and for testing a new domain. That's my practical inference, not an experiment they ran.

MayaI'd also check whether the judge's disagreements cluster around something we care about. Averaging candidate order helps with position bias. It doesn't answer whether the rating instructions represent the people affected by the eventual response.

LeoI'm glad the encouraging part survives all that. On these historical tasks, replacing the training preference labels still produced policies people preferred to the starting models, and broadly matched human-feedback training.

MayaHere's how: judgments trained a scorer, the scorer rewarded generated responses, and reinforcement learning changed what the policy would generate. Direct RLAIF removed the separate scorer. Human judgment still defined the task and checked what the optimization produced.

LeoIf you delegated feedback for an assistant you were building, which judgment would you keep asking people to make, even after the automated labels became cheap?