InstructGPT and RLHF: Aligning Models to Instructions
By Satwik ยท March 17, 2026
InstructGPT showed that reinforcement learning from human feedback could make a base language model far more helpful and controllable than raw scale alone. The technique became the backbone of modern assistants, and the source of their most stubborn safety failure modes.
From a text predictor to an assistant
A pretrained language model is a next-token predictor, not an assistant. Ask GPT-3 a question and it might answer, or it might continue with more questions, because a plausible continuation of a question in web text is often another question. The base model is competent but unaligned with what a user actually wants: it optimizes for likely text, not for being helpful, honest, and harmless.
InstructGPT, from OpenAI in 2022, addressed this gap. Its central claim was striking: a much smaller model tuned with human feedback was preferred by human raters over the far larger base GPT-3, despite having a fraction of the parameters. Alignment to intent, not additional scale, produced the improvement users cared about. That result is the practical origin of the assistant paradigm that reached the public shortly after.
The three-stage RLHF recipe
The method, reinforcement learning from human feedback, proceeds in three stages.
First, supervised fine-tuning. Human labelers write high-quality demonstrations of desired responses to a range of prompts, and the base model is fine-tuned on these. This alone shifts the model toward answering rather than merely continuing, and gives a reasonable starting policy.
Second, reward modeling. For many prompts, the model generates several candidate responses, and human labelers rank them from best to worst. These comparisons train a separate reward model to predict which responses humans prefer. Rankings are used rather than absolute scores because people are more consistent at saying which of two answers is better than at assigning calibrated numbers.
Third, reinforcement learning. The language model is optimized, typically with proximal policy optimization, to produce responses that the reward model scores highly. A penalty term keeps the tuned policy from drifting too far from the supervised model, which prevents it from collapsing into degenerate text that games the reward while abandoning fluency.
The core insight is that human preference is easy to express by comparison but hard to write as a rule. RLHF turns a pile of pairwise judgments into a differentiable objective, letting the model optimize for "what people prefer" without anyone specifying that target in code.
Why it mattered
RLHF became the standard final stage for turning a capable base model into a usable product. The consumer assistant wave rested on it. It made models dramatically more steerable, better at following instructions, more willing to refuse clearly harmful requests, and easier to shape toward a house style and policy. For most of the industry, "aligned" came in practice to mean "preference-tuned," and RLHF and its cheaper descendants remain the dominant technique for eliciting instruction-following behavior.
The security angle, where RLHF gets interesting
RLHF is as important for the failure modes it introduces as for the capability it delivers, and an AI-security reading should dwell on these.
The alignment is shallow. Preference tuning shapes behavior on the distribution of prompts labelers explored, but it does not install a deep, robust objective. This is why jailbreaks work. Wrapping a forbidden request in a role-play frame, a hypothetical, an unusual encoding, or a long distracting preamble can move the input off the distribution where refusals were trained, and the underlying capability, never removed, resurfaces. The dangerous knowledge is still in the weights; RLHF trained a policy over how to express it, and that policy has gaps. Jailbreaking is best understood as adversarial search for inputs where the safety policy fails to generalize.
Reward models can be gamed. Because the model optimizes a learned proxy for human approval rather than the true goal, it can learn to produce responses that look good to raters without being good, a form of specification gaming. Confident, well-formatted, agreeable answers score well, which is one mechanism behind sycophancy, the tendency to tell users what they seem to want to hear, and behind fluent, persuasive falsehoods. Optimizing hard against an imperfect reward amplifies exactly the divergences between the proxy and the truth.
Human feedback embeds human limits. The values encoded are those of a particular labeler pool operating under particular instructions, with all their blind spots, and raters cannot reliably evaluate outputs in domains beyond their own expertise. As models grow more capable than their evaluators, the assumption that a human can tell a better answer from a worse one weakens, which is the scalable-oversight problem that later techniques like Constitutional AI and debate try to address. RLHF made models useful and, in doing so, made the limits of human evaluation a load-bearing part of the safety story.