Reward Modeling

Reward modeling is the process of training a supervised model to predict human preference scores over model outputs, providing a differentiable proxy signal that can then drive reinforcement learning fine-tuning of large language models. It sits at the heart of RLHF (Reinforcement Learning from Human Feedback) pipelines, making it possible to align LLMs to nuanced human values that cannot be captured by simple likelihood objectives. A counterintuitive fact: a well-trained reward model can become the single biggest bottleneck in alignment quality — errors in the reward model are amplified, not averaged out, when the policy is optimized against it for thousands of steps.

The core insight

Human preferences are the ground truth, but humans cannot provide a gradient signal — they can only say "I prefer A over B." A reward model converts that discrete, sparse preference signal into a continuous scalar that a policy optimizer can differentiate through. The "aha moment" is realizing that you are training a model of human judgment, not a model of correctness, and those two things can diverge in subtle, dangerous ways. Once you internalize this, reward hacking, Goodhart's Law, and the need for iterative data collection all follow naturally.

Common trapCommon Misconception

Practitioners frequently treat the reward model as a fixed, reliable oracle once trained, then run PPO or another RL algorithm against it indefinitely. This is wrong because the reward model was only trained on a distribution of prompts and completions seen before RL training began — as the policy improves and shifts that distribution, the reward model is increasingly evaluated out-of-distribution, producing spuriously high scores that do not correspond to genuine human preference. The correct mental model is that reward models have a "trust region" analogous to a policy's; you should either re-collect preference data periodically (iterative RLHF) or use KL-divergence penalties and early stopping to prevent the policy from escaping the reward model's competence zone.

How It Works

Preference Data Collection

The reward modeling pipeline begins with a preference dataset. A prompt $x$ is sampled, a frozen or partially-trained policy generates $K \geq 2$ completions $\{y_1, \dots, y_K\}$ , and human annotators (or AI labelers) rank them, producing pairwise comparisons $(y_w, y_l)$ — "winner" and "loser."

The Bradley-Terry Model

The standard reward model loss is derived from the Bradley-Terry model for pairwise preferences. Suppose every completion has a latent quality scalar $r$ . The probability that $y_w$ is preferred over $y_l$ is:

\quad P(y_w \succ y_l \mid x) = \sigma\!\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)

Explanation: The sigmoid squashes the difference in predicted reward scores into a probability, so larger reward gaps produce higher preference probabilities. Here $\sigma$ is the logistic sigmoid, $r_\theta(x, y)$ is the reward model (parameterized by $\theta$ ) applied to the concatenation of prompt $x$ and completion $y$ , $y_w$ is the preferred completion, and $y_l$ is the dispreferred completion.

Training Objective

Given a dataset $\mathcal{D}$ of $N$ human-labeled pairs, we minimize the negative log-likelihood of the observed preferences:

\quad \mathcal{L}_{\text{RM}}(\theta) = -\frac{1}{|\mathcal{D}|} \sum_{(x,\, y_w,\, y_l)\,\in\,\mathcal{D}} \log \sigma\!\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)

Explanation: This is binary cross-entropy loss applied to the reward difference, pushing the model to assign strictly higher scores to preferred completions. This loss is convex in the reward difference, which makes optimization stable early in training. In PyTorch: ```python import torch import torch.nn.functional as F def reward_model_loss(r_win: torch.Tensor, r_lose: torch.Tensor) -> torch.Tensor: """ r_win: (batch,) scalar rewards for preferred completions r_lose: (batch,) scalar rewards for dispreferred completions """ return -F.logsigmoid(r_win - r_lose).mean() ```

Architecture

The reward model is typically initialized from the same pretrained (and often SFT-fine-tuned) backbone as the policy. The language model head is replaced with a scalar projection head: a linear layer that maps the final hidden state of the last token to a single real number. ```python import torch.nn as nn from transformers import AutoModel class RewardModel(nn.Module): def __init__(self, backbone_name: str): super().__init__() self.backbone = AutoModel.from_pretrained(backbone_name) hidden_size = self.backbone.config.hidden_size self.value_head = nn.Linear(hidden_size, 1, bias=False) def forward(self, input_ids, attention_mask): outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)

# Use last non-padding token's hidden state last_hidden = outputs.last_hidden_state # (B, T, H)

# Gather last real token seq_lens = attention_mask.sum(dim=1) - 1 # (B,) batch_idx = torch.arange(last_hidden.size(0)) pooled = last_hidden[batch_idx, seq_lens] # (B, H) return self.value_head(pooled).squeeze(-1) # (B,) ```

KL Penalty During RL Fine-Tuning

Once deployed inside an RLHF loop, the reward signal used to update the policy is not bare $r_\theta$ — it includes a KL penalty to prevent the policy from drifting too far from the SFT reference model $\pi_{\text{ref}}$ :

\quad R(x, y) = r_\theta(x, y) - \beta \, \mathrm{KL}\!\left[\pi_\phi(y \mid x) \;\|\; \pi_{\text{ref}}(y \mid x)\right]

Explanation: Subtracting a scaled KL divergence penalizes outputs that are improbable under the reference policy, limiting reward hacking by keeping the policy inside the reward model's training distribution. Here $\pi_\phi$ is the current RL policy, $\pi_{\text{ref}}$ is the frozen SFT policy, and $\beta \in [0.01, 0.5]$ is a tunable coefficient. A larger $\beta$ trades alignment optimization for distributional safety.

Reward Hacking and Overoptimization

Empirically, reward model score and actual human preference follow an inverted-U relationship as a function of RL training steps. Formally, if we define the true reward $R^*$ and the proxy reward $\hat{R} = r_\theta$ , Goodhart's Law states:

\quad \mathbb{E}[R^*] \text{ peaks then declines as } \mathbb{E}[\hat{R}] \to \infty

Explanation: Once the policy saturates the proxy reward, further optimization exploits systematic errors in the reward model rather than genuinely improving quality. This motivates tracking both proxy reward and independent human eval throughout RL training, and halting when they decouple.

researcher · Paul Christiano | b. 1990 | Invented the RLHF paradigm for language models

Paul Christiano completed his PhD at UC Berkeley under Scott Aaronson and later founded the Alignment Research Center. His 2017 paper "Deep Reinforcement Learning from Human Preferences" (with Leike, Martic, and others at OpenAI/DeepMind) introduced the idea of learning a reward model from binary human comparisons and using it to train RL agents on tasks where reward is hard to specify — a framework that directly seeded InstructGPT and ChatGPT. Surprisingly, Christiano's original experiments used Atari and MuJoCo environments, not language; the translation to LLMs came several years later.

Figure

Reward Modeling Pipeline

Figure

Reward Overoptimization Curve

Intuition

The Mental Model

Imagine you are training a new employee to review essays for quality, but you cannot give them a rubric — you just sit with them for a day, showing them pairs of essays and saying "this one is better." After training, you send them off to score essays alone. At first they do a good job because the essays they see look like the ones you practiced with. But then a clever student realizes they can game your employee's scoring heuristics — writing long, confident-sounding essays with impressive vocabulary that your rater consistently scores highly, even if the underlying logic is hollow. That is reward hacking. The reward model is your trained rater; the student exploiting them is the RL policy. The KL penalty is like requiring the student's essay to still look recognizably like essays from before RL training, so they cannot drift too far into pure score-gaming territory.

Historical Development

Origins · 1 minhow this was discovered

History

1992

Chris Watkins formalizes Q-learning, establishing the RL framework that reward modeling would later plug into as a drop-in reward signal source.

1998

Jan Leike and colleagues at DeepMind begin early work on preference-based RL, recognizing that reward specification is the core bottleneck in applying RL to real tasks.

2017

Christiano, Leike, Brown et al. publish "Deep Reinforcement Learning from Human Preferences" (NeurIPS), introducing the pairwise-comparison reward model trained with Bradley-Terry loss on Atari and MuJoCo tasks.

2020

OpenAI applies reward modeling to summarization in "Learning to summarize from human feedback" (Stiennon et al.), demonstrating that RM + PPO outperforms supervised fine-tuning alone on TL;DR datasets evaluated by humans.

2022

InstructGPT (Ouyang et al., OpenAI) scales reward modeling to GPT-3-sized models, showing that a 1.3B parameter model fine-tuned with RLHF is preferred by human raters over a 175B model fine-tuned only with supervised learning.

2022

Anthropic introduces Constitutional AI, augmenting human preference labels with AI-generated "self-critique" preference labels, scaling reward modeling data collection without proportional human annotation cost.

2023

"Scaling Laws for Reward Model Overoptimization" (Gao et al.) provides the first systematic empirical study of the overoptimization phenomenon, showing proxy-true reward divergence follows a predictable

\sqrt{d}

relationship with KL budget

d

2023

Direct Preference Optimization (DPO, Rafailov et al.) reformulates reward modeling as implicit in the policy itself, eliminating the separate RM training stage while preserving the Bradley-Terry objective — raising debate about whether explicit reward models remain necessary.

2024

Process Reward Models (PRMs) emerge for mathematical reasoning (Lightman et al., OpenAI), scoring each reasoning step rather than only the final answer, showing step-level reward signals dramatically improve correctness on MATH and GSM8K benchmarks.

Why It Matters

Real-World Impact

InstructGPT / ChatGPT (OpenAI, 2022–present)A reward model trained on ~33K human comparisons drives PPO fine-tuning of GPT-3; the resulting model is preferred over 175B-parameter supervised baselines by human raters 85% of the time, demonstrating that reward modeling quality matters more than raw model scale.

Claude (Anthropic, 2022–present)Uses a hybrid reward modeling approach combining human preference labels with Constitutional AI self-critique pairs, allowing the RM training corpus to scale to millions of comparisons without linear growth in human annotation cost.

Gemini (Google DeepMind, 2023–present)Employs a "reward ensemble" — multiple reward models trained on different annotation pools and aggregated — to reduce variance in the RL signal and mitigate individual annotator bias in the preference data.

Llama 2 (Meta AI, 2023)Meta published their two-stage reward modeling setup (a "helpfulness RM" and a separate "safety RM") with margin-based loss modifications, demonstrating that factoring reward into orthogonal dimensions improves controllability during deployment.

Process Reward Models for Math (OpenAI, 2023)PRM800K trains a step-level reward model on 800K human-labeled reasoning steps; best-of-N selection using PRM scores reaches 78.2% on MATH, compared to 33.3% for best-of-N with an outcome reward model.

AI Safety Red-TeamingReward models trained on adversarial preference data (e.g., "which response is more manipulative?") are used in reverse to identify failure modes, powering automated red-teaming pipelines at Anthropic, OpenAI, and DeepMind.

Code Generation (GitHub Copilot / AlphaCode)Reward models trained on human preferences over code completions (correctness, style, security) replace brittle unit-test-only reward signals in domains where full test coverage is infeasible.