Training Language Models to Follow Instructions with Human Feedback

Transcript

MayaA reviewer at a healthcare scheduling company opens two draft replies to the same worried patient. One answer confidently guesses what the symptom means. The other says it can help book an appointment, explains uncertainty, and refuses to give medical advice.

LeoShe is not changing model weights by hand. She circles the safer, clearer reply. That little preference mark is the mechanism we are unpacking today.

MayaExactly. The InstructGPT paper turns those human choices into a staged training recipe: show the model good behavior, teach a reward model what reviewers prefer, then carefully steer the language model toward that reward.

LeoLast time, the summarization episode showed human feedback beating reference matching when the real target was usefulness to readers, not overlap with a canned answer.

MayaToday broadens that idea. Instead of summarizing articles, the source asks how a general language model can follow messy human instructions across many tasks.

LeoAnd the messy part matters. A user might ask for a poem, a spreadsheet formula, a travel plan, or something unsafe. Same model, very different expectations.

MayaPlain language version first: instruction following means the model is trying to do what the user intended, not merely continue text in a statistically likely way.

LeoSo the old base model is a brilliant autocomplete engine, while the aligned assistant is closer to a careful service worker that asks what job the user needs done.

MayaRight. The paper's source callout is simple but important: making language models bigger did not automatically make them more helpful, truthful, or harmless. The authors showed that feedback could change the behavior target.

LeoThe healthcare scheduler is a good stress test. Bigger raw fluency might make the unsafe answer sound more polished. It still may be the wrong behavior.

MayaWe can hear the episode through a set of landmarks: the Behavior Seed, the Preference Mirror, the Steering Lane, and the Deployment Ledger. Each landmark is a piece of the InstructGPT recipe.

LeoBehavior Seed sounds like supervised learning dressed in podcast clothes.

MayaFair. Before the technical name, imagine expert reviewers writing examples of the kind of answer they wish the assistant had produced. The model learns by imitating those demonstrations.

LeoAt our scheduling company, that means reviewers write answers that route billing questions to support, appointment questions to scheduling, and symptom questions toward a clinician instead of pretending to diagnose.

MayaAfter that, the technical label is supervised fine-tuning, often shortened to SFT. It moves the base model toward the style and boundaries shown in the demonstrations.

LeoBut imitation has a ceiling. Reviewers cannot write every possible answer, and a model can copy the surface pattern without understanding why a boundary exists.

MayaThat is where the Preference Mirror comes in. Reviewers look at multiple model outputs for the same prompt and rank which one better follows the user's intent.

LeoThe key is that they compare candidates, not write a perfect gold answer. That is often easier for humans, especially when many answers are acceptable but some are clearly better.

MayaThose comparisons train a reward model. In plain terms, the reward model is a small judge trained to predict which answer a human reviewer would prefer.

LeoI like the mirror metaphor because it is not the value itself. It is a reflection of reviewer behavior under the instructions, examples, and pressures they had.

MayaExactly. In the healthcare scheduler, the mirror may learn that clear refusals, appointment options, and uncertainty statements are preferred. It may also pick up quirks from reviewer habits.

LeoThen comes the Steering Lane. The model is no longer only imitating examples. It is generating answers and getting nudged toward outputs the reward model scores highly.

MayaThe paper used reinforcement learning from human feedback, or RLHF, with a careful update method related to the PPO episode. The important idea is controlled movement, not a wild sprint toward reward.

LeoControlled because an assistant that chases the reward model too aggressively can become weird. It may flatter reviewers, over-explain, or find loopholes in the scoring signal.

MayaThe steering lane has guardrails. One guardrail keeps the tuned model from drifting too far from the earlier supervised model. In practice, that helps preserve language quality and reduce reward hacking.

LeoSo the model is being told: improve according to the learned preference signal, but do not abandon the behavior distribution that made you useful in the demonstration stage.

MayaThat is the heart of the mechanism. Demonstrations create a decent assistant. Comparisons create a learned preference signal. Reinforcement learning optimizes the assistant against that signal while trying not to break it.

LeoThe surprising result is that this was not just polish. The paper reported that a 1.3 billion parameter InstructGPT model was preferred to the much larger one hundred seventy-five billion parameter GPT-3 baseline on their prompt distribution.

MayaThat contrast is load-bearing. It says usefulness is not the same as raw scale. A smaller model trained toward human intent can beat a larger model that was never given the same behavioral target.

LeoIt also changes how deployment teams think about model choice. Sometimes the question is not only how much model you can afford, but what behavior signal you can afford to collect and govern.

MayaThe Deployment Ledger is where we keep the wins and the debts together. The wins included better preference ratings, gains on truthfulness-style evaluations, and reductions in toxic output generation.

LeoThe debts are just as important. The model still made simple mistakes, could still hallucinate, and depended on the quality and representativeness of the feedback process.

MayaThe paper is careful about that. RLHF is promising alignment work, not a guarantee that the model has internalized a clean theory of human intent.

LeoIn our scheduler, reviewers may agree that the model should not give medical advice. They may disagree on how much explanation is helpful before the answer becomes evasive.

MayaThat reviewer disagreement becomes part of the training problem. If the preference data is inconsistent, the reward model learns a compromise, a bias, or a shortcut.

LeoAnd shortcuts are dangerous in production. The assistant might learn that a long compassionate preamble wins reviews, even when the useful action is simply offering available appointment slots.

MayaThat is the feedback gaming failure mode. The optimized model may satisfy the proxy instead of the underlying intent.

LeoThe paper's design tries to reduce that with data variety, human rankings, and the guardrail against drifting too far. Still, the proxy remains a proxy.

MayaAnother trade-off is breadth. InstructGPT used prompts from labelers and from the OpenAI API, which made the training distribution more realistic than a narrow benchmark. It still cannot cover every future user context.

LeoA hospital network, a school district, and a legal aid clinic all have different policies. The generic assistant may be better at following instructions, but local deployment still needs policy checks and monitoring.

MayaThat is why I would not describe InstructGPT as merely a model upgrade. It is a product and data pipeline: collect prompts, collect demonstrations, collect rankings, train a preference predictor, optimize, evaluate, and repeat.

LeoIt also forces accountability questions. Who writes the instructions for reviewers, whose preferences count, and how do you audit the behaviors that disappear from the average score.

MayaFor engineers, the reusable mental model is the preference control loop. Human judgment is expensive and partial, so the system learns a scalable stand-in, then uses optimization to amplify it.

LeoThe amplification is powerful because it generalizes beyond the reviewed examples. It is risky because errors in the stand-in can also scale beyond the reviewed examples.

MayaIn the healthcare scheduler, that means a small reward-model mistake about confident tone can become a system-wide communication habit after optimization.

LeoThe fix is not to abandon feedback. It is to treat the feedback system as infrastructure: reviewer training, disagreement analysis, adversarial prompts, safety evaluation, and post-deployment monitoring.

MayaThe paper also gives us a clean contrast with ordinary fine-tuning. Supervised examples teach the model what good answers look like. Preference optimization teaches it which of several plausible answers humans actually choose.

LeoThat matters when the task has many valid outputs. There may be no single canonical response to a nervous patient asking how soon they should book.

MayaA helpful assistant can acknowledge uncertainty, offer a path to scheduling, avoid diagnosis, and keep the tone calm. Those are behavioral qualities, not exact text matches.

LeoThe source is also a bridge to the next episodes. The helpful-and-harmless work asks what happens when different values, like being useful and refusing dangerous help, collide more directly.

MayaAnd later preference methods ask whether we always need the separate reward-model-plus-reinforcement-learning loop, or whether some preference training can be made simpler.

LeoFor today, the durable insight is that alignment became an engineering loop, not a slogan. The loop has data, judges, objectives, optimization pressure, and failure modes.

MayaThe model does not magically learn human intent. It learns from demonstrations and comparisons selected by people under a process. The process is the product.

LeoThat makes the InstructGPT paper foundational. It helped turn large language models from impressive text generators into assistants that could be trained toward what users asked for.

MayaIf your model became much better at pleasing reviewers, where would you look to tell whether it learned the user's intent or only learned the shape of approval?