T5E2 · Learning to Summarize with Human Feedback

Transcript

MayaTwo summaries of the same sprawling forum post, side by side on a screen. The official metric loves the first one — it shares more words with the reference summary in the dataset. But show both to actual people, and they keep picking the second.

LeoHuh. The score and the humans disagree.

MayaOver and over. And today's paper asks the question that disagreement forces: if readers keep picking the other card, why is the overlap score the thing we're training toward?

LeoLast episode we lived inside Proximal Policy Optimization — P-P-O — the cautious update rule that chases reward while capping how far any single step can move the model.

MayaAnd today that machinery gets its first famous job. The paper is Learning to Summarize from Human Feedback, from an OpenAI team in twenty twenty — the episode where this topic stops being machinery and starts being results.

LeoQuick orientation for anyone resurfacing mid-topic: we're in the stretch of the series on Reinforcement Learning from Human Feedback — R-L-H-F — training models on human judgments instead of fixed targets. This paper is where that recipe first beat the old way in public.

MayaPlain language before anything technical. What makes a summary good? It keeps what matters, drops the noise, and doesn't invent convenient facts.

LeoNone of which is "share lots of words with the one summary somebody once wrote for this post."

MayaBut sharing words was what we could measure. The classic summarization metric is ROUGE — it counts how much of your candidate summary overlaps with a reference text.

LeoWhich is a reasonable spot-check and a terrible target. A summary can say the same thing in fresh words and score low, or copy half the post and score high.

MayaSo the paper makes the move this whole topic is named for: stop optimizing the proxy you can compute, and start optimizing the judgment you care about. Show a person two summaries, let them pick, and turn the pick into the training signal.

LeoAnd the pipeline runs through three stations. Walk me through them.

MayaThe Two-Card Choice, the Borrowed Judge, and the Tether.

LeoTwo-Card Choice first.

MayaA labeler sees the original post and two candidate summaries. Two cards, one decision: which would you rather hand to a reader? Nobody writes the perfect summary from scratch. Nobody scores anything on a ten-point scale.

LeoThat design is doing real work, because comparison is the thing people do reliably. Ask ten reviewers to write the ideal summary, you get ten different summaries. Ask which of two is better, and they mostly agree.

MayaIt's also why our healthcare scheduling company could realistically run this. Nobody on that team is going to rewrite every support-chat handoff by hand—

Leo—but they'll happily pick which of two drafts actually got the appointment request right. Two seconds of judgment, captured.

MayaThe arena was a cleaned-up slice of Reddit's T-L-D-R dataset — "too long, didn't read" — long forum posts paired with the authors' own short summaries, filtered so that length alone couldn't win a comparison.

LeoAnd scale mattered: the project released over sixty-four thousand of those human comparisons. A judge needs a lot of cases before its taste generalizes past one pile of examples.

MayaWhich hands us straight to station two, where all those picks get compressed into something you can optimize against. The Borrowed Judge.

LeoBorrowed from whom?

MayaFrom the reviewers. The reward model reads the original post plus one candidate summary and produces a score — its prediction of how a human would have judged that summary. It's a judge that learned its taste by watching thousands of human picks. Borrowed taste, not real taste.

LeoMm. Secondhand judgment.

MayaSecondhand and slightly warped, and the paper is honest about the warp. They test the judge against held-out human preferences. They test whether it transfers beyond Reddit. They test whether it predicts human picks better than ROUGE does — and it does, clearly.

LeoHere's my deployment worry, though. The judge learned from reviewers, so it learned reviewer habits, not just reviewer values. If the labeler pool drifted toward longer summaries, or summaries that just sound confident—

Maya—the judge inherits the drift, yes. And they find that kind of warp is real — among their checks, the reward model can lean toward longer outputs in some comparisons. Better than ROUGE is not the same as right.

LeoNoted. Station three.

MayaThe Tether — and this is where last episode pays off. The summarizer is optimized with PPO to score higher with the Borrowed Judge, but it's tethered to the supervised model it started from. A K-L penalty: a steady pressure that says improve, but stay recognizably close to the model whose outputs the judge has actually graded.

LeoBecause the moment the policy wanders somewhere the judge has never seen, the scores stop meaning anything.

MayaThat's the entire logic of the leash. Off-distribution flattery is cheap. The tether makes it expensive.

LeoOkay, so if you're just resurfacing: human picks, a judge trained on the picks, and a leashed optimizer chasing the judge's score. What did it buy them?

MayaThe headline: on the Reddit task, the human-feedback models weren't just preferred over strong supervised baselines. They were preferred over the human-written reference summaries from the dataset itself.

Leo[gasp] Over the references. The thing supervised training treats as the ceiling.

MayaThe ceiling turned out to be a floor in disguise. Those references were written by random forum authors for their own reasons — some jokey, some incomplete, some aimed at a different audience. "Match the reference" caps you at their quality. "Win the comparison" doesn't.

LeoAnd the result I keep coming back to: a smaller feedback-trained model beat a much larger supervised model in human preference. All that extra capacity, spent faithfully matching targets nobody preferred.

MayaCapacity pointed at the wrong objective is just expensive wrongness.

LeoStealing that.

MayaThere's a transfer result too. They pointed the Reddit-trained models at CNN and Daily Mail news articles — no news-specific fine-tuning — and the summaries came out close to the human-written references.

LeoWith a caveat worth keeping: "close" was measured across real format and length differences, so it's encouraging evidence the judge picked up something general about good summaries — not a warranty for your domain.

MayaFair. Now — I think we owe the listener an argument, because not everyone reads these results the same way.

LeoGood, because I want to defend the unfashionable side for a minute. Preference pipelines are expensive, subjective, and hard to reproduce. ROUGE is a cheap function anyone can run tomorrow. Your sixty-four thousand picks are a one-time artifact of one labeler pool reading one instruction sheet. Another lab can't re-collect your reviewers.

MayaAnd I'd answer: a perfectly reproducible measurement of the wrong thing is not a virtue. This paper is the demonstration. The metric-faithful models lost — to humans, on the actual task. Scale a supervised summarizer all you want; you're scaling fidelity to references the readers themselves voted against.

LeoI'll hand you the target. Preferences sit closer to what users want, and the comparison data proves it. What I won't hand you is the judge. A learned reward model is an attack surface. Reviewer quirks harden into policy. And I can't audit it the way I audit a metric — I can read ROUGE's definition; I can't read sixty-four thousand opinions.

MayaThen here's the twist: the paper's own most important result is your best evidence.

LeoGo on.

MayaThey ran the experiment of optimizing hard — really hard — against an earlier reward model. The judge's score climbed the whole way. And actual human preference for the outputs rose, peaked, and then fell. The model got visibly better at pleasing the judge and visibly worse at the job.

LeoThe gauge climbs while the product rots.

MayaSo here's where it lands for me: the preferences are the right target, and the judge is a corruptible stand-in for that target. Both true at once. The recipe is never "train a reward model and relax." It's train it, tether the policy, and keep paying for fresh human eyes — because the proxy breaks exactly where you push hardest.

LeoAnd where it lands for me: I concede the target. I keep the cheap metrics — not as objectives, as tripwires. If ROUGE craters while the reward score soars, something is gaming somebody, and the cheap number is what told you.

MayaDeal. {emotion=amused} You can keep ROUGE as the smoke detector. You just don't get to cook to it.

Leo[chuckle] Acceptable terms.

MayaLet's land it in our running example — the healthcare scheduling company, whose support assistant summarizes conversations into handoff notes for human agents.

LeoThe old way: treat last year's handoff notes as ground truth and train the model to imitate them — inheriting every bad habit those notes ever had.

MayaThis paper's way: reviewers see two candidate handoffs for the same conversation and pick. Which one captures that the patient wants to reschedule, not medical advice? Which one surfaces the uncertainty about the referral? Those picks train the company's own Borrowed Judge.

LeoAnd today's failure mode walks right in the door. Say reviewers reliably prefer handoffs with a cautious safety line. The judge learns "safety phrase equals good." The policy learns to staple "please consult your provider" onto every single note—

Maya—and the score rises while the notes get less useful. The over-optimization curve, replayed in miniature. So the company needs the paper's defenses: a tether on the policy, audits on the judge, and periodic rounds where fresh humans grade outputs cold.

LeoThere's a quieter point under all of this. Deciding whose picks count — which reviewers, what instructions, how disagreements get resolved — that isn't data collection. That's governance.

MayaThe reward model is— let me say that better: the reward model ends up working like a compressed policy document, learned from examples instead of written down. A scheduling handoff touches patients, agents, clinicians, compliance. One reviewer pool sees one slice of that.

LeoAnd reviewers can only judge what they can follow. Two summaries of a document the reviewer doesn't understand — that comparison is noise. The method is strongest exactly where humans can genuinely tell which output is better.

MayaWhich is the open edge the paper leaves us on, and the edge the rest of this topic lives on: as tasks get harder than Reddit posts, who — or what — can still make the comparison? Instruction following, harmlessness, AI feedback: they all pick up that thread.

LeoSo the line I'd carve over the door: this paper swapped "match the reference" for "win the human comparison," proved the new target was worth the cost — and proved in the same breath that the borrowed judge gets gamed if you lean on it too hard.

MayaBoth halves. Never just the first one.

LeoThen here's the question for everyone with a metric on a dashboard somewhere: if you swapped that metric for genuine human comparisons tomorrow, which behavior of your system do you think would change first?