The Reward Model

A reward model is a learned function $r_\phi(x, y)$, parameterized by weights $\phi$, that maps a prompt $x$ and a candidate completion $y$ to a real-valued score representing how much a human would prefer that completion. The output is a single scalar, not a probability distribution over tokens — the reward model is not trying to generate text, only to evaluate it. This distinguishes it sharply from the base language model, whose job is next-token prediction. The key design decision is that reward models are trained on relative judgments rather than absolute ratings. Asking an annotator "rate this response from 1 to 10" produces noisy, poorly calibrated labels: different annotators use different internal scales, and the same annotator is inconsistent across sessions. Asking "which of these two responses is better" is a far easier and more reliable task — humans are much better at pairwise comparison than at absolute scoring. This is the same insight behind the classical Bradley-Terry model of paired comparisons, originally developed for ranking chess players and sports teams from win/loss records, and it gives the reward model its precise probabilistic training objective. Formally, given a prompt $x$ and two completions $y_w$ (the one a human preferred, "winner") and $y_l$ (the "loser"), the Bradley-Terry model assumes

where $\sigma$ is the logistic sigmoid. This says the probability that a human prefers $y_w$ over $y_l$ depends only on the difference in their underlying reward scores — larger gaps mean more confident preferences, and a reward model that assigns equal scores predicts a coin-flip preference. Training the reward model means finding $\phi$ that makes this probability match the observed human choices as closely as possible, which is done by maximum likelihood, equivalently by minimizing a logistic (cross-entropy) loss over preference pairs.

Step 1 — Collect comparison data. Sample a set of prompts $x$ from a distribution representative of real usage. For each prompt, generate $k$ completions (commonly $k = 4$ to $9$) from one or more instruction-tuned policy checkpoints, using varied sampling temperature to get diverse candidates.

Step 2 — Human ranking. Show annotators the prompt and the $k$ completions. Annotators produce a ranking (or a sequence of pairwise choices) reflecting overall quality — typically weighing helpfulness, honesty, and harmlessness. A ranking of $k$ items is decomposed into $\binom{k}{2}$ pairwise comparisons for training, though in practice only a subset of pairs, or a suitably weighted subset, is used to control redundancy.

Step 3 — Initialize the reward model. Start from an instruction-tuned language model (same tokenizer, same transformer backbone). Strip the token-prediction head (the final linear layer that outputs a distribution over the vocabulary) and attach a new linear head that maps the final hidden state at the last token to a single scalar. This reuses the rich language understanding already learned during pretraining and instruction tuning, rather than learning to read English from scratch.

Step 4 — Train with the pairwise loss. For every $(x, y_w, y_l)$ triple, compute $r_\phi(x, y_w)$ and $r_\phi(x, y_l)$ by running both sequences through the network, and minimize the negative log-likelihood of the observed preference under the Bradley-Terry model.

Step 5 — Freeze and deploy. Once trained, the reward model's weights are frozen and it is used purely for inference: given any prompt-completion pair, it emits a scalar reward. This scalar becomes the reward signal in an RL loop (foreshadowing Reinforcement Learning from Human Feedback), or it can be used directly for best-of-$n$ re-ranking of sampled completions. ```

# Training a reward model from pairwise human preferences Input: base_model # instruction-tuned transformer, weights theta_0 D # dataset of (prompt x, completion_w, completion_l)

# where w was preferred to l by a human annotator num_epochs, lr, batch_size Initialize: reward_model = base_model with LM head replaced by a linear layer: hidden_state -> scalar phi = reward_model.parameters() # includes reused backbone weights for epoch in 1..num_epochs: for batch in minibatches(D, batch_size): loss_sum = 0 for (x, y_w, y_l) in batch: seq_w = concat(x, y_w) seq_l = concat(x, y_l) h_w = reward_model.encode(seq_w) # final hidden state h_l = reward_model.encode(seq_l) # at last token r_w = linear_head(h_w) # scalar reward r_l = linear_head(h_l) # scalar reward

# Bradley-Terry pairwise logistic loss loss = -log_sigmoid(r_w - r_l) loss_sum += loss loss_sum = loss_sum / len(batch) gradients = backprop(loss_sum, wrt=phi) phi = phi - lr gradients # or Adam / AdamW update return reward_model # frozen after training; used to score (x, y) pairs ``` Two implementation details matter in practice. First, when multiple completions are ranked (not just pairs), all $\binom{k}{2}$ comparisons from a single prompt are usually normalized so that a batch is not dominated by prompts with many completions — a common approach is to compute the loss as a sum over all pairs from one ranking, divided by $\binom{k}{2}$, so that each prompt contributes comparably to the gradient regardless of $k$. Second, the reward model's raw scale is arbitrary: only score differences* matter, since $\sigma(r_w - r_l)$ is invariant to adding any constant to both. In practice the mean reward on a validation set is often normalized to zero to keep the scores centered before the RL stage, since RL algorithms are sensitive to the absolute scale and offset of the reward signal.

Time complexity. Each forward pass through the transformer backbone on a sequence of length $L$ costs $O(L^2 \cdot d + L \cdot d^2)$ for self-attention and feed-forward layers respectively, where $d$ is the hidden dimension — the same cost as a standard transformer forward pass, since the reward model reuses that architecture unchanged except for the final head. Training over a dataset of $N$ preference pairs for $E$ epochs costs $O(N \cdot E \cdot (L^2 d + L d^2))$ total, dominated by two forward-backward passes per pair (one for $y_w$, one for $y_l$). This is comparable in cost to one epoch of ordinary supervised fine-tuning over $2N$ sequences, so reward model training is not asymptotically more expensive than instruction tuning itself — its cost is dominated by data collection (human annotation), not compute.

Space complexity. Memory is $O(P)$ for model parameters plus $O(B \cdot L \cdot d)$ for activations in a batch of size $B$, identical in form to standard transformer training; the only structural difference is that the LM head, which is $O(d \cdot |V|)$ for vocabulary size $|V|$, is replaced by a $O(d)$ scalar head, making the reward model marginally smaller than the base language model.

Correctness argument. The claim to justify is that minimizing the pairwise logistic loss recovers a scoring function consistent with the underlying human preference structure, under the Bradley-Terry assumption. Consider the population-level objective: if preferences are generated by some latent scalar function $r^*(x,y)$ satisfying $P(y_w \succ y_l \mid x) = \sigma(r^*(x,y_w) - r^*(x,y_l))$, then the expected loss

is minimized in expectation precisely when $r_\phi$ induces the same pairwise probabilities as $r^*$ — this is the standard maximum-likelihood argument for logistic regression, here applied to the difference of two learned scores rather than to a single input. Formally, for a fixed pair of scores $(a, b) = (r_\phi(x,y_w), r_\phi(x,y_l))$ and true probability $p^* = \sigma(r^*(x,y_w) - r^*(x,y_l))$, the expected loss $-p^*\log\sigma(a-b) - (1-p^*)\log\sigma(b-a)$ is a strictly convex function of $a - b$ with a unique minimum at $a - b = r^*(x,y_w) - r^*(x,y_l)$ — this follows because the cross-entropy loss of a logistic model is minimized exactly at the true log-odds, a classical fact from logistic regression. Since this holds for every pair, at the population optimum $r_\phi$ and $r^*$ induce identical pairwise preference probabilities everywhere, and $r_\phi$ recovers $r^*$ up to an additive constant per prompt (the invariance noted earlier, since only differences enter the loss).

Honest caveats. This correctness argument has real gaps in practice. Human preferences are not a perfect Bradley-Terry process: annotators disagree with each other, are inconsistent with themselves, and the "reward" they implicitly optimize for is not a single well-defined scalar (it conflates helpfulness, safety, verbosity bias, and stylistic preference). The reward model is trained on a finite sample, so it generalizes only as well as the training distribution covers the space of completions the eventual RL policy will produce — completions far outside this distribution get unreliable scores. This mismatch is the root cause of reward hacking (discussed below), where a downstream policy finds completions that score highly under $r_\phi$ without actually being preferred by real humans.

Consider the prompt $x =$ "Explain why the sky is blue." Suppose the base policy sampled three completions, and a human annotator ranked them best-to-worst as $y_1 \succ y_2 \succ y_3$:

• $y_1$: "Sunlight scatters off air molecules; blue light scatters more due to its shorter wavelength (Rayleigh scattering)." — accurate, concise.
• $y_2$: "The sky is blue because of how light interacts with the atmosphere." — vague but not wrong.
• $y_3$: "The sky is blue because water reflects the ocean's color upward." — fluent but factually false.

This ranking decomposes into $\binom{3}{2}=3$ pairwise comparisons: $(y_1, y_2)$, $(y_1, y_3)$, $(y_2, y_3)$, each with the first element preferred. Suppose at some point during training the reward model produces these raw scores:

Compute the loss contribution from each pair using $\ell = -\log \sigma(r_w - r_l)$:

Averaging over the three pairs, this ranking contributes a mean loss of $(0.1674+0.0328+0.1674)/3 \approx 0.1225$. Gradient descent on this loss pushes $r_\phi(x, y_1)$ up and $r_\phi(x, y_3)$ down (and $y_2$'s score adjusts based on both comparisons it appears in), sharpening the score gaps until the predicted preference probabilities approach 1 for confidently-ranked pairs. If the model instead had assigned $y_3$ the highest score (a clear failure to reject the factually wrong answer), the loss for pair $(y_1, y_3)$ would spike — e.g. if $r_\phi(x,y_3) = 3.0$ while $r_\phi(x,y_1)=2.10$, then $r_w - r_l = -0.90$, $\sigma(-0.90) \approx 0.289$, giving a loss of $-\log(0.289) \approx 1.24$, roughly ten times larger than the correct-ranking case — illustrating how the loss surface directly penalizes score inversions on human-labeled pairs.

Reward models are the linchpin of the Reinforcement Learning from Human Feedback pipeline used to align production chat assistants: after supervised Instruction Tuning teaches a model to follow instructions, the reward model supplies the objective that a policy-gradient algorithm like PPO optimizes to make outputs more genuinely helpful and less harmful. Beyond RLHF, reward models power best-of-$n$ sampling, where multiple completions are generated for a single prompt and the reward model simply picks the highest-scoring one — a cheap inference-time alternative to full RL fine-tuning. They also underlie automated evaluation harnesses that compare model checkpoints during development without requiring a human to re-judge every candidate response, and they are central to red-teaming and safety pipelines that score outputs for harmfulness. Techniques like Direct Preference Optimization reformulate the same Bradley-Terry preference data to skip the explicit reward model and RL loop entirely, but they still rest on the identical probabilistic foundation this article develops — a testament to how central the reward-modeling idea has become to the entire alignment stack.

Ask the tutor: How does the KL-divergence penalty in RLHF prevent a policy from exploiting reward model weaknesses?