Publications

Constitutional AI: Principles Over Labels

By Satwik · March 20, 2026

Constitutional AI replaced much of the human feedback in alignment with a written set of principles the model uses to critique and revise its own outputs. It was an early, concrete attempt at scalable oversight, and a case study in the promises and limits of self-supervision.

The problem it targets

Standard RLHF trains harmlessness by having human labelers judge whether responses are harmful, then optimizing against those judgments. This works, but it has two structural weaknesses. It requires a large volume of human labor spent reviewing disturbing content, which is slow, costly, and taxing on the people doing it. And the resulting values are implicit, buried in thousands of individual labels, so they are hard to inspect, hard to audit, and hard to change deliberately. If you want to know why the model refuses a given request, "the aggregate of what labelers happened to penalize" is not a satisfying or governable answer.

Constitutional AI, introduced by Anthropic in 2022, proposed an alternative: encode the desired values as an explicit, written set of principles, a "constitution," and have the model use those principles to supervise itself. The aim was to make harmlessness training more scalable, more transparent, and more directly steerable.

How it works

The method has two phases.

The first is supervised self-critique. The model is prompted with inputs designed to elicit harmful responses. It generates an initial answer, then is asked to critique that answer against a principle drawn from the constitution, for example a principle that responses should avoid assisting with dangerous or unethical activities, and then to revise the answer in light of the critique. This produces revised, safer responses, and the model is fine-tuned on the revisions. The model is, in effect, teaching itself to be harmless by repeatedly applying stated rules to its own drafts.

The second phase is reinforcement learning from AI feedback, RLAIF. As in RLHF, the model generates pairs of responses, but instead of a human choosing which is better, the model itself is asked which response better satisfies a constitutional principle. These AI-generated preference labels train a reward model, and the policy is then optimized against it with reinforcement learning, exactly as in RLHF but with the human preference labels for harmlessness replaced by model-generated ones grounded in the constitution.

The key move is that human effort shifts from labeling thousands of individual outputs to writing and refining a compact set of principles. Oversight becomes a matter of authoring rules rather than adjudicating cases, and the model does the per-example work of applying them.

Why it mattered

Constitutional AI was one of the first deployed, concrete instances of scalable oversight: using AI assistance to supervise AI, so that the human effort required does not grow linearly with the amount of behavior being shaped. That framing matters because it targets a problem that gets worse as models improve. When a model's outputs become too numerous, too long, or too specialized for humans to review case by case, direct human labeling stops scaling, and some form of AI-assisted supervision becomes necessary rather than merely convenient.

It also advanced transparency in governance. Because the values live in an explicit document, they can be read, debated, revised, and pointed to. Changing the model's stance on a category of behavior can, in principle, be done by editing a principle rather than by re-running a large labeling campaign. This makes the value-setting process more legible to outside review, which is itself a governance asset in a field where "why did the model do that" is often unanswerable.

The security angle and the open questions

Constitutional AI is best read as a promising direction with real, unresolved limits, and the security-minded view should sit with both.

The dependency it creates is subtle. RLAIF makes the model's own judgment a load-bearing part of its safety training, so the reliability of the whole scheme rests on the model being able to correctly evaluate responses against principles. Where the model's judgment is biased or mistaken, that error is now baked into the training signal and can be amplified by optimization, rather than caught by an independent human. Self-supervision inherits the supervisor's flaws, and here the supervisor and the supervised are the same system.

The principles must be interpreted, not merely obeyed. A constitution is written in natural language, and the model's reading of a principle can diverge from its authors' intent, especially at the edges where hard cases live. Coverage is a genuine problem: a compact set of rules cannot anticipate every situation, and adversarial prompts specifically probe the gaps between principles, so the same jailbreak dynamic that afflicts RLHF persists. The constitution shapes behavior on the cases it addresses and generalizes imperfectly to the cases it does not.

There is also a deeper question about legitimacy. Making values explicit is a clear improvement over leaving them implicit, but it also concentrates a consequential choice, whose principles, chosen how, with what accountability, into a document written by a small group. That is a governance question more than a technical one, and Constitutional AI's lasting contribution may be that it forces the question into the open where it can be argued about, rather than leaving it buried in a mass of labels. For scalable oversight to be trustworthy, both the competence of the AI supervisor and the legitimacy of the encoded values have to hold, and Constitutional AI made both requirements concrete.