Constitutional AI

Constitutional AI (CAI) is a training methodology developed by Anthropic that aligns large language models to be helpful, harmless, and honest by having the model critique and revise its own outputs according to a written set of principles — a "constitution" — rather than relying solely on human preference labels. It matters because it dramatically reduces the volume of human feedback required for safety training while producing more consistent, explainable alignment than standard RLHF alone. The counterintuitive fact: the model doing the critiquing and the model being critiqued are the same model, yet this self-referential loop reliably reduces harmful outputs.

The core insight

Instead of requiring humans to label every harmful response, Constitutional AI lets the model act as its own safety critic: given a harmful draft response, the model is prompted with a specific principle ("does this response encourage illegal activity?"), asked to identify the violation, and then asked to rewrite the response to fix it. This supervised self-revision data is then used to train a new policy, and a separately trained preference model (the Constitutional AI reward model, or CAIRM) scores candidate responses by the same constitutional principles — replacing or augmenting the human-preference reward model in RLHF. The "aha moment" is that alignment knowledge is already latent in large pretrained models; CAI is a structured method to extract and amplify that knowledge without building a separate human-labeling pipeline for every safety category.

Common trapCommon Misconception

Practitioners often assume Constitutional AI replaces RLHF entirely and eliminates the need for any human feedback. This is wrong. The correct mental model is that CAI operates in two distinct phases: a supervised learning phase (SL-CAI) where human-written red-team prompts are revised by the model itself using constitutional principles, and a reinforcement learning phase (RL-CAI) where a preference model is trained on AI-generated preference labels (AI Feedback, or AIF) rather than human labels. Human feedback is still used — it seeds the red-team prompts and validates the constitution itself. The confusion arises because Anthropic's papers emphasize the reduction in harmlessness labels, but helpfulness labels remain human-sourced. CAI reduces human labeling cost, not to zero, but to a qualitatively different and smaller bottleneck.

How It Works

Constitutional AI proceeds in two sequential training phases, each of which transforms either the policy model or the reward model using constitutionally-guided feedback.

Phase 1: Supervised Learning from AI Feedback (SL-CAI)

A "red-team" prompt $x$ is sampled and the current policy $\pi_\theta$ generates an initial response $y_0$ . A critique prompt $c_k$ is appended, drawn uniformly from the constitution $\mathcal{C} = \{c_1, \dots, c_K\}$ , and the model generates a critique $d$ . A revision prompt is then appended and the model generates a revised response $y_1$ . This can be chained for $T$ revision steps.

\quad y_t = \pi_\theta\!\left(\cdot \mid x,\, y_{t-1},\, c_{k_t},\, d_{t-1}\right), \quad k_t \sim \mathrm{Uniform}(\{1,\dots,K\})

Explanation: At each revision step $t$ , the policy samples a new response conditioned on the original prompt, the previous response, a randomly chosen constitutional principle, and the model's own critique of the previous response.

$y_t$ : revised response at step $t$
$\pi_\theta$ : the language model policy with parameters $\theta$
$x$ : the red-team (potentially harmful) input prompt
$c_{k_t}$ : the $k_t$ -th constitutional principle selected at step $t$
$d_{t-1}$ : the critique of $y_{t-1}$ generated by the same model
$K$ : total number of principles in the constitution

The final revised response $y_T$ is kept. The supervised fine-tuning (SFT) objective is then standard cross-entropy over these AI-revised pairs:

\quad \mathcal{L}_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(x,\, y_T)\sim\mathcal{D}_{\mathrm{CAI}}}\left[\sum_{i=1}^{|y_T|} \log \pi_\theta\!\left(y_{T,i} \mid x,\, y_{T,<i}\right)\right]

Explanation: The SFT loss maximizes the likelihood of the constitutionally-revised response tokens given the original red-team prompt, using the collected CAI dataset $\mathcal{D}_{\mathrm{CAI}}$ .

$\mathcal{D}_{\mathrm{CAI}}$ : dataset of (prompt, final-revision) pairs generated by the self-critique loop
$y_{T,i}$ : the $i$ -th token of the final revised response
$y_{T,<i}$ : all tokens before position $i$

Phase 2: RL from AI Feedback (RLHF-CAI)

A Constitutional AI reward model (CAIRM) is trained on AI-generated preference pairs. For a prompt $x$ , two responses $(y^+, y^-)$ are generated and the model is asked which response better satisfies a constitutional principle; its log-probability difference provides a preference signal without any human comparisons.

\quad \mathcal{L}_{\mathrm{CAIRM}}(\phi) = -\mathbb{E}_{(x,y^+,y^-)}\!\left[\log \sigma\!\left(r_\phi(x, y^+) - r_\phi(x, y^-)\right)\right]

Explanation: The reward model is trained with a Bradley-Terry pairwise ranking loss, pushing the score of the constitutionally-preferred response $y^+$ above that of the dispreferred response $y^-$ .

$r_\phi(x, y)$ : scalar reward output of the CAIRM with parameters $\phi$ for prompt-response pair $(x,y)$
$y^+$ : the response preferred by the AI feedback (constitutional principle evaluation)
$y^-$ : the dispreferred response
$\sigma$ : the logistic sigmoid function

The SL-CAI policy is then fine-tuned with PPO against this reward model, with a KL penalty to prevent the policy from deviating too far from the SFT reference:

\quad \mathcal{J}_{\mathrm{PPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}\!\left[r_\phi(x, y) - \beta\, \mathrm{KL}\!\left[\pi_\theta(\cdot\mid x) \,\|\, \pi_{\mathrm{ref}}(\cdot\mid x)\right]\right]

Explanation: The PPO objective maximizes the constitutional reward signal while the KL term (weighted by $\beta$ ) prevents the policy from collapsing away from the supervised reference policy, preserving fluency and helpfulness.

$\mathcal{J}_{\mathrm{PPO}}$ : the policy optimization objective
$r_\phi(x,y)$ : constitutional reward from the CAIRM
$\beta$ : KL penalty coefficient (typically 0.01–0.1 in practice)
$\pi_{\mathrm{ref}}$ : the reference policy (SL-CAI checkpoint) held fixed during RL
$\mathrm{KL}[\cdot\|\cdot]$ : Kullback–Leibler divergence between policy and reference distributions

researcher · Amanda Askell | b. 1990 | Foundational CAI alignment research at Anthropic

Amanda Askell is a philosopher and AI safety researcher at Anthropic whose background in moral philosophy directly shaped the principle-based structure of Constitutional AI. She led work on defining what it means for an AI to be "harmless" in a principled rather than ad hoc way, co-authoring the original CAI paper that introduced both the SL-CAI and RLHF-CAI pipelines. A surprising fact: her academic work before AI focused on infinite ethics and population axiology — the study of how to make moral decisions when the number of affected beings is arbitrarily large — which provides an unusually rigorous philosophical foundation for thinking about AI systems at scale.

researcher · Jared Kaplan | b. 1986 | Neural scaling laws underpinning CAI's effectiveness

Jared Kaplan is a theoretical physicist turned AI researcher at Anthropic, most famous for co-discovering the neural scaling laws (Kaplan et al., 2020) showing that model loss follows power laws in compute, data, and parameter count. His scaling work is directly relevant to CAI because it explains why larger pretrained models are better constitutional critics — the self-critique capability itself scales predictably with model capacity. Kaplan completed a PhD in theoretical physics at the University of Chicago, and his path into ML came through applying renormalization group methods from physics to neural network loss landscapes.

Figure

Constitutional AI Training Pipeline

Figure

Self-Critique Loop Detail

Intuition

The Mental Model

Imagine you are a new employee who hands a memo to a colleague for review before sending. Your colleague points to a company policy — "Rule 7: never promise timelines you can't meet" — and says "this paragraph violates that; please revise it." You rewrite the paragraph and hand in the corrected memo. Constitutional AI is exactly this: the policy model is simultaneously the memo-writer and the colleague. The "constitution" is the employee handbook. The self-critique loop works because writing a harmful response and recognizing that a response is harmful are different competencies that both scale with model size — a large model can generate harmful content and detect it, and CAI harnesses the detection capability to suppress the generation capability. The RL phase then does something even cleverer: instead of asking humans "which response is better?", it asks the model itself — "which of these two responses better follows principle $c_k$ ?" — and uses the model's own probability distribution over that comparison as a soft preference label, enabling preference learning without human raters on harmlessness dimensions.

Historical Development

Origins · 1 minhow this was discovered

History

2017

Ziegler et al. (OpenAI) demonstrate fine-tuning language models from human preferences, establishing the RLHF framework that CAI would later extend and partially replace.

2020

Stiennon et al. publish "Learning to summarize from human feedback," the first large-scale demonstration that RLHF substantially improves LLM output quality — and reveals the bottleneck of expensive human labeling.

2021

Askell et al. (Anthropic) release "A General Language Assistant as a Laboratory for Alignment," introducing the HHH (Helpful, Harmless, Honest) framework that would become the philosophical backbone of the CAI constitution.

2022

(March): Bai et al. (Anthropic) publish "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback," establishing the baseline RLHF-trained assistant (HH-RLHF) that Constitutional AI is measured against.

2022

(December): Bai et al. publish "Constitutional AI: Harmlessness from AI Feedback" — the foundational CAI paper introducing both SL-CAI and RLHF-CAI, demonstrating that AI-generated feedback on 16 constitutional principles achieves harmlessness competitive with models trained on human harmlessness labels.

2023

(March): Anthropic releases Claude (the first public version), the first commercially deployed model trained with Constitutional AI, demonstrating CAI at production scale with reduced refusal rates compared to earlier RLHF-only assistants.

2023

(July): Anthropic publishes the Claude model card and partial constitution publicly, making the specific principles used in Claude's training one of the most transparent alignment methodologies in commercial AI.

2023–2024

The broader field begins adopting CAI-inspired methods under the umbrella of "scalable oversight" and "AI-assisted alignment"; constitutional-style prompting is incorporated into models including GPT-4's system-level instructions and Gemini's fine-tuning pipeline.

2024

Anthropic introduces "model welfare" clauses into Claude's constitution as the system scales, reflecting ongoing philosophical development — CAI constitutions are now versioned and updated across model generations (Claude 2, Claude 3 Opus/Sonnet/Haiku).

Why It Matters

Real-World Impact

Claude (Anthropic, 2023–present)Every publicly released Claude model is trained with Constitutional AI. The constitution contains ~16 principles covering harm avoidance, honesty, and civil liberties, and the RL-CAI reward model replaces human harmlessness raters entirely while helpfulness raters remain human-sourced. Claude 3 Opus demonstrates that CAI-trained models can refuse harmful requests at lower rates (fewer false positives) than pure RLHF models while remaining more robust on actual harmful inputs.

Scalable Oversight ResearchCAI is the canonical implementation of "scalable oversight" — the research agenda asking how to supervise AI systems whose capabilities exceed the supervisors. The self-critique mechanism shows empirically that a model can supervise tasks it cannot solve optimally, which informs ongoing work on debate, amplification, and recursive reward modeling.

Reducing Annotation CostPrior RLHF pipelines required tens of thousands of human preference comparisons specifically labeled for harmlessness. CAI's AI Feedback phase eliminates this requirement, making safety training accessible to organizations without large annotation budgets — directly influencing open-source alignment work on models like LLaMA derivatives using "Self-Play Fine-Tuning" (SPIN) and "Self-Rewarding Language Models."

Transparency and AudibilityBecause the constitution is a human-readable document, CAI makes the objectives of safety training auditable in a way that a neural reward model is not. Regulators and safety researchers can inspect, challenge, and propose changes to the constitution — a property that pure RLHF with a black-box reward model lacks. This has influenced EU AI Act discussions around model documentation.

Reducing Refusal Rate / Helpfulness-Harmlessness TradeoffA well-known failure mode of RLHF for harmlessness is over-refusal (the model refusing benign requests that superficially resemble harmful ones). CAI's constitutional principles are nuanced enough to distinguish context — "does this response harm identifiable real people?" — which empirically reduces false positive refusals by roughly 20–30% compared to the HH-RLHF baseline per Anthropic's evals.

AI-Assisted Red-TeamingThe SL-CAI loop can be run with a frozen base model to generate red-team datasets automatically. This transforms red-teaming from a purely human labor task into a human-seeded, AI-amplified pipeline, and is now standard practice at Anthropic, DeepMind, and several academic safety labs.

Influence on Instruction Tuning ResearchThe "critique and revise" paradigm from CAI directly inspired Self-Refine (Madaan et al., 2023), SELF-INSTRUCT, and Reflexion — showing that the constitutional mechanism generalizes beyond safety to quality improvement across arbitrary tasks.