RLHF and Reward Learning Connections

Reinforcement Learning from Human Feedback formalizes a setting where the reward function is unknown and unspecifiable, but comparisons between two outcomes are cheap for a human to make. Instead of asking a human to assign an absolute numeric reward to an output — a task humans are notoriously bad at doing consistently — RLHF asks a human (or a panel of annotators) to state a preference: "output A is better than output B." These pairwise preferences are then used to fit a reward model $r_\phi(x, y)$, a neural network that takes an input $x$ (a prompt) and a candidate output $y$ (a completion) and predicts a scalar score consistent with the observed preference ordering. The reward model is trained under the Bradley-Terry model of pairwise comparisons, which assumes the probability that a human prefers $y_w$ (the "winner") over $y_l$ (the "loser") given the same prompt $x$ is a logistic function of the reward difference:

where $\sigma$ is the sigmoid function. This converts preference learning into a standard binary classification problem via maximum likelihood, and it is the same statistical model historically used to rank chess players (Bradley-Terry-Luce) and, before that, to model paired-comparison psychophysics data. Once the reward model is trained, it is frozen and used as a proxy environment reward for policy optimization. The policy — typically the same language model that was supervised-fine-tuned on demonstrations — is optimized to maximize expected reward under $r_\phi$, using a policy-gradient algorithm such as Proximal Policy Optimization (PPO). Because $r_\phi$ is only an approximation of true human preference, and because policies optimized aggressively against a learned proxy tend to find degenerate high-scoring regions the proxy was never trained on (a failure called reward hacking or Goodhart's law in action), the RL objective adds a penalty term: the KL divergence between the current policy $\pi_\theta$ and a fixed reference policy $\pi_{\text{ref}}$ (usually the supervised fine-tuned model before RL). The full objective per prompt $x$ is

The coefficient $\beta$ controls the tradeoff between reward maximization and staying close to the reference distribution: too small and the policy overfits to the reward model's blind spots; too large and the policy barely moves from its starting point. This structure is precisely why RLHF is best understood as a constrained inverse reinforcement learning problem: the reward is inferred from behavioral preference signal (as in IRL), but the "expert" is not a demonstrator producing trajectories — it is a comparison oracle producing rankings — and the resulting reward is used for standard forward RL with an explicit anchor to prevent divergence from plausible behavior.

The RLHF pipeline has three distinct stages, each of which can be studied and debugged independently.

Stage 1 — Supervised Fine-Tuning (SFT). Start from a pretrained base model and fine-tune it on a curated dataset of high-quality demonstrations (prompt, ideal-response pairs), using standard supervised cross-entropy loss. This gives an initial policy $\pi_{\text{SFT}}$ that is already reasonably well-behaved, dramatically narrowing the search space for later RL. This step is imitation learning in the sense of Imitation and Inverse Reinforcement Learning — behavioral cloning on expert demonstrations.

Stage 2 — Reward Model Training. Sample multiple completions from $\pi_{\text{SFT}}$ (or an earlier RLHF checkpoint) for a set of prompts. Have human annotators rank or pairwise-compare these completions. Fit $r_\phi$ by maximizing the log-likelihood of the observed preferences under the Bradley-Terry model.

Stage 3 — RL Fine-Tuning. Optimize $\pi_\theta$ (initialized from $\pi_{\text{SFT}}$) against the frozen reward model $r_\phi$, using PPO (or another policy-gradient method), with a per-token or per-sequence KL penalty against $\pi_{\text{ref}} = \pi_{\text{SFT}}$. ```pseudocode

# ---------- Stage 1: Supervised Fine-Tuning ---------- function SFT(base_model, demonstration_data): pi_sft = base_model for (prompt, target_response) in demonstration_data: loss = CrossEntropy(pi_sft(prompt), target_response) pi_sft.parameters -= lr * gradient(loss, pi_sft.parameters) return pi_sft

# ---------- Stage 2: Reward Model Training ---------- function TRAIN_REWARD_MODEL(pi_sft, prompts, num_samples_k): comparisons = [] for x in prompts: candidates = [ pi_sft.sample(x) for i in 1..num_samples_k ] ranking = HUMAN_RANK(x, candidates) # human orders candidates for (y_w, y_l) in ALL_PAIRS_CONSISTENT_WITH(ranking): comparisons.append((x, y_w, y_l)) initialize r_phi (usually pi_sft backbone + scalar head) for batch in MINIBATCHES(comparisons): loss = 0 for (x, y_w, y_l) in batch: score_w = r_phi(x, y_w) score_l = r_phi(x, y_l) loss += -log( sigmoid(score_w - score_l) ) # Bradley-Terry NLL r_phi.parameters -= lr * gradient(loss / |batch|, r_phi.parameters) return r_phi

# ---------- Stage 3: RL Fine-Tuning (PPO against r_phi) ---------- function RLHF_PPO(pi_sft, r_phi, prompts, beta, epochs): pi_theta = COPY(pi_sft) # trainable policy pi_ref = FREEZE(pi_sft) # fixed anchor for KL penalty value_fn = initialize_value_network() for iteration in 1..epochs: batch = [] for x in SAMPLE(prompts): y = pi_theta.sample(x) # rollout, token by token r_env = r_phi(x, y) # scalar reward at end of sequence kl = SUM_OVER_TOKENS( log(pi_theta(y_t | x, y_

• log(pi_ref(y_t | x, y_

r_total = r_env - beta * kl # KL-regularized reward batch.append( (x, y, r_total) )

# standard PPO update using r_total as the trajectory reward advantages = COMPUTE_GAE(batch, value_fn) for epoch in 1..ppo_inner_epochs: ratio = pi_theta.prob(actions) / pi_theta_old.prob(actions) clipped = CLIP(ratio, 1 - eps, 1 + eps) advantages policy_loss = -MEAN( MIN(ratio advantages, clipped) ) value_loss = MSE(value_fn(states), returns) pi_theta.parameters -= lr gradient(policy_loss, pi_theta.parameters) value_fn.parameters -= lr gradient(value_loss, value_fn.parameters) pi_theta_old = COPY(pi_theta) return pi_theta ``` The reward model architecture is usually the same transformer backbone as the policy, with the final unembedding layer replaced by a single scalar head that reads the representation at the last token of the sequence. This scalar head is the only newly initialized component; the backbone is initialized from the SFT checkpoint so that the reward model starts with strong language understanding.

Reward model training complexity. If there are $N$ prompts each with $k$ sampled completions ranked by humans, a full ranking of $k$ items induces up to $\binom{k}{2} = O(k^2)$ pairwise comparisons; in practice implementations use only the $k-1$ adjacent comparisons from the ranking or a sampled subset to keep training cost linear in $k$. Each forward/backward pass costs the same as one transformer forward pass, $O(L \cdot d^2 + L^2 \cdot d)$ for sequence length $L$ and hidden dimension $d$ (attention plus feedforward costs), so total reward-model training cost is $O(N \cdot k \cdot (L d^2 + L^2 d))$ per epoch, identical in order to ordinary supervised fine-tuning at the same data scale. Space cost is one model checkpoint plus optimizer state, same as any transformer fine-tune — no combinatorial blowup, because comparisons are trained as independent binary-classification examples rather than requiring joint ranking inference.

RL fine-tuning complexity. Each PPO iteration requires: (1) a rollout phase, sampling $B$ sequences of length up to $L$ from $\pi_\theta$, costing $O(B \cdot L)$ sequential decoding steps each $O(d^2)$ (this autoregressive sampling dominates wall-clock time in practice), (2) a reward-model forward pass per sample, $O(B \cdot (Ld^2 + L^2d))$, (3) a KL computation requiring a forward pass of $\pi_{\text{ref}}$ on the same sequences, same order, and (4) several epochs of PPO gradient updates over the collected batch, each $O(B \cdot (Ld^2+L^2d))$. So the RL stage requires four models resident simultaneously — policy, reference policy, reward model, and value function — which is the dominant memory cost relative to plain fine-tuning, typically 3–4× the memory footprint of the base model alone (a major practical driver toward memory-efficient variants and DPO-style methods that eliminate the explicit reward model and PPO loop).

Correctness argument — why the Bradley-Terry loss recovers a valid reward ordering. The maximum-likelihood estimator for the Bradley-Terry model is consistent: if preferences are genuinely generated by some latent reward function $r^*$ under the logistic-comparison assumption, then as the number of independent comparisons grows, $r_\phi \to r^*$ up to a global additive constant and positive scaling, because the Bradley-Terry log-likelihood is a strictly concave function of the reward differences and its unique maximizer (given enough connected comparisons spanning the completion space) coincides with the true parameter under standard M-estimation theory. The additive-constant/scale ambiguity is harmless for RL: policy gradients under $r_\phi$ and under $\alpha r_\phi + c$ for $\alpha>0$ define the same ordering of expected returns and hence (ignoring the interaction with the KL term's fixed weighting) similar optimal policies, though in practice the scale interacts with $\beta$ and must be tuned jointly.

Correctness argument — why the KL penalty is necessary, not optional. Without regularization, the RL objective is simply $\max_\theta \mathbb{E}_{\pi_\theta}[r_\phi(x,y)]$. Because $r_\phi$ is trained only on the support of completions the SFT/annotated policy actually produced, it is unconstrained (and potentially arbitrarily miscalibrated) outside that support. An unconstrained policy-gradient optimizer will drift toward regions of output space that maximize $r_\phi$'s score even if those regions are degenerate, repetitive, or nonsensical to a human — this is reward hacking, a direct instance of Goodhart's law ("when a measure becomes a target, it ceases to be a good measure"). The KL term $\beta \cdot \mathrm{KL}(\pi_\theta \| \pi_{\text{ref}})$ upper-bounds how far the induced output distribution can move from the well-understood reference distribution; formally, the regularized objective has the closed-form optimal solution

which is a tilted (reweighted) version of the reference policy rather than an unconstrained maximizer — this closed form is exactly the bridge exploited by Direct Preference Optimization, which substitutes it back into the Bradley-Terry loss to eliminate the RL loop analytically.

Honest tradeoffs. PPO-based RLHF is sample-inefficient (many rollouts per gradient step), sensitive to $\beta$ and learning-rate schedules, and requires nontrivial infrastructure (four models in memory, reward-model reward hacking monitoring, KL-divergence tracking). Its benefit is that it directly optimizes the thing you actually want (expected discounted reward under sequential decisions), which matters when responses are multi-turn or when the reward is genuinely sequential rather than a single terminal judgment. Simpler alternatives like DPO trade this generality for stability and much lower engineering cost, at the price of being provably equivalent only under the same Bradley-Terry assumptions and losing some flexibility for multi-step or non-bandit reward structures.

Consider a toy setting: prompt $x =$ "Summarize: The cat sat on the mat." Suppose the SFT policy generates two candidate completions:

• $y_1$ = "A cat sat on a mat." (accurate, concise)
• $y_2$ = "The cat, feline creature of legend, took its throne upon the mat, as cats do." (verbose, off-tone)

A human annotator prefers $y_1$. Suppose the reward model, before this example is used in training, currently scores $r_\phi(x, y_1) = 0.20$ and $r_\phi(x, y_2) = 0.35$ (miscalibrated — it currently prefers the wrong one).

Reward model gradient step. The Bradley-Terry loss for this single pair is:

Numerically, $\sigma(-0.15) \approx 0.4626$, so $\mathcal{L} \approx -\log(0.4626) \approx 0.7712$. The gradient with respect to $r_\phi(x,y_1)$ is $-(1 - \sigma(r_{y_1}-r_{y_2})) = -(1-0.4626) = -0.5374$, pushing $r_\phi(x,y_1)$ up, and symmetrically pushing $r_\phi(x,y_2)$ down. After many such updates across the dataset, the reward model learns to assign higher scores to concise, accurate summaries.

RL step with the trained reward model. Suppose after reward-model training, $r_\phi(x,y_1) = 0.9$ and $r_\phi(x,y_2) = -0.4$. Suppose the policy $\pi_\theta$, currently close to $\pi_{\text{ref}}$, assigns probability $0.55$ to sequences resembling $y_1$-style completions and $0.10$ to $y_2$-style ones, with $\pi_{\text{ref}}$ assigning the same probabilities initially (KL $\approx 0$). Take $\beta = 0.1$.

Initially the KL term is zero since $\pi_\theta = \pi_{\text{ref}}$, so the policy gradient pushes probability mass toward $y_1$-like completions (positive advantage) and away from $y_2$-like ones (negative advantage). After one PPO update suppose $\pi_\theta(y_1\text{-style})$ rises to $0.68$ while $\pi_{\text{ref}}$ stays at $0.55$. Now:

The KL penalty shaves a small amount off the effective reward as the policy drifts from the reference, which slows further movement once the marginal gain from increasing $r_\phi$ is outweighed by the KL cost — this is exactly the tilted-distribution equilibrium $\pi_\theta^* \propto \pi_{\text{ref}} \cdot \exp(r_\phi/\beta)$ described above, being approached iteratively.

RLHF's mathematical backbone predates deep learning by nearly a century: the Bradley-Terry model for paired comparisons was published in the 1950s for ranking objects from pairwise judgments, itself building on Thurstone's 1920s work on comparative judgment in psychology. The RL formalism that RLHF fine-tunes into came from the reinforcement learning tradition; the marriage of preference-based reward learning with deep policy gradients is a 2010s-2020s development.

RLHF is the mechanism behind the shift from raw pretrained language models to deployed conversational assistants: the supervised-fine-tuning-plus-preference-optimization recipe (InstructGPT and its successors) is the standard pathway used across major chat-oriented deployments, and reward models built this way are also used as automatic evaluators ("reward-model-as-judge") to filter and rank generated candidates at inference time. Beyond text, the same three-stage recipe applies to code-generation assistants (preferring correct, idiomatic, and safe code), to summarization systems (trading off faithfulness and brevity), and to robotics and simulated agents where a human's qualitative preference between two trajectories substitutes for an unavailable reward function — directly inheriting the reward-inference perspective from Imitation and Inverse Reinforcement Learning. Constitutional and rule-based variants replace or supplement human preference labels with model-generated critiques, but still rely on the same Bradley-Terry reward-modeling and KL-regularized policy-optimization core. Because the RL stage is exactly an instance of Deep Policy Gradient Methods applied to a learned reward, any advance in PPO stability, sample efficiency, or variance reduction propagates directly into better, cheaper RLHF pipelines, which is why the technique remains an active area connecting classical preference statistics, inverse reinforcement learning theory, and large-scale deep RL engineering.

Ask the tutor: How does the choice of the KL-penalty coefficient β interact with reward model miscalibration, and how would you tune it in practice?