Reinforcement Learning from Human Feedback

Formally, RLHF treats text generation as a sequential decision process. Given a prompt $x$, the language model $\pi_\theta$ is a policy: at each step it observes the prompt and tokens generated so far (the state) and samples the next token (the action) from a distribution over the vocabulary. Once the model emits an end-of-sequence token, the full response $y$ is scored by a fixed, pretrained reward model $r_\phi(x, y)$, itself trained on human preference comparisons (see The Reward Model). This scalar becomes the episode's reward, and reinforcement learning is used to adjust $\theta$ so that $\pi_\theta$ places more probability mass on high-reward completions. Two design choices make this different from generic RL. First, the reward is sparse and terminal — it arrives only once, after the whole sequence is generated, not per token, since human judgments of "good response" naturally apply to the full text rather than to individual words. Second, unconstrained reward maximization is dangerous: a policy can find degenerate text that fools the reward model (reward hacking) while being incoherent or repetitive to a human reader. RLHF addresses this by adding a KL-divergence penalty between the trained policy $\pi_\theta$ and a frozen reference policy $\pi_{\text{ref}}$ (usually the instruction-tuned model before RL). The per-token reward actually optimized is

where $\beta$ controls how tightly the policy is tethered to the reference. This penalty is what keeps RLHF from collapsing into gibberish that exploits reward-model blind spots; it is the load-bearing regularizer of the whole method.

The RLHF loop has four ingredients, three of which are frozen during RL: a frozen reference policy $\pi_{\text{ref}}$ (the instruction-tuned model), a frozen reward model $r_\phi$, a trainable policy $\pi_\theta$ (initialized from $\pi_{\text{ref}}$), and often a trainable value function $V_\psi$ used to reduce variance in the policy-gradient estimate. Training proceeds in repeated rounds: sample prompts, generate responses with the current policy, score them, compute advantages, and take a clipped policy-gradient step. This is Proximal Policy Optimization (PPO) specialized to autoregressive text generation.

Step 1 — Rollout. Sample a batch of prompts $x_1, \dots, x_B$ from a prompt distribution (often a mixture of instruction-tuning prompts and user-style queries). For each prompt, generate a response $y_i \sim \pi_\theta(\cdot \mid x_i)$ token by token, recording the log-probability the policy assigned to each chosen token.

Step 2 — Scoring. Pass each completed $(x_i, y_i)$ through the reward model to get a scalar $r_\phi(x_i, y_i)$. Compute the per-token KL penalty between $\pi_\theta$ and $\pi_{\text{ref}}$ at every generation step, and subtract $\beta$ times it from the reward, typically added only at the final token (or distributed across tokens).

Step 3 — Advantage estimation. Use the value function $V_\psi$ to estimate, at each token position $t$, how much better the actual outcome was than expected. Generalized Advantage Estimation (GAE) computes

where $\gamma$ is a discount factor (often close to 1) and $\lambda$ trades off bias and variance.

Step 4 — PPO update. Update $\theta$ by maximizing the clipped surrogate objective, and update $\psi$ by regressing $V_\psi$ toward observed returns. Clipping is what makes PPO "proximal": it prevents any single update from moving the policy so far that the advantage estimates (computed under the old policy) become invalid. ```

# RLHF training loop with PPO, applied to token-level generation Input: pi_theta : trainable policy, initialized from pi_ref pi_ref : frozen reference policy (instruction-tuned model) r_phi : frozen reward model V_psi : trainable value function prompts : distribution of prompts x beta : KL penalty coefficient epsilon : PPO clip range K : number of PPO epochs per batch of rollouts for iteration = 1, 2, ... do

# --- Step 1: Rollout --- batch = [] for i = 1 to B: x = sample(prompts) y, logp_old = generate(pi_theta, x) # token-by-token sampling batch.append((x, y, logp_old))

# --- Step 2: Scoring with KL penalty --- for (x, y, logp_old) in batch: reward_terminal = r_phi(x, y) for t = 1 to len(y): kl_t = logp_old[t] - logprob(pi_ref, x, y[:t]) r[t] = -beta * kl_t # per-token penalty r[len(y)] += reward_terminal # reward only at end

# --- Step 3: Advantage estimation (GAE) --- for (x, y, r) in batch: V = [V_psi(x, y[:t]) for t in 1..len(y)+1] delta[t] = r[t] + gamma V[t+1] - V[t] A[t] = sum_{k>=0} (gamma lambda)^k * delta[t+k] R[t] = A[t] + V[t] # target return for value fn

# --- Step 4: PPO clipped update --- for epoch = 1 to K: for (x, y, logp_old, A, R) in batch: logp_new = logprob(pi_theta, x, y) ratio = exp(logp_new - logp_old) # importance ratio per token unclipped = ratio A clipped = clip(ratio, 1 - epsilon, 1 + epsilon) A L_policy = -mean(min(unclipped, clipped)) L_value = mean((V_psi(x, y) - R)^2) L_entropy = -mean(entropy(pi_theta(x, y))) # optional bonus loss = L_policy + c1 L_value - c2 L_entropy update(theta, psi) via gradient descent on loss return pi_theta ``` The clipping term is the algorithmic heart of PPO. If the importance ratio $\rho_t = \pi_\theta(a_t\mid s_t) / \pi_{\theta_{\text{old}}}(a_t\mid s_t)$ moves far from 1 in a direction that would increase the surrogate loss, clipping caps its contribution, so the optimizer cannot be rewarded for straying arbitrarily far from the policy that generated the data. This is what allows multiple gradient epochs ($K > 1$) over the same rollout batch without the update becoming an invalid extrapolation.

Time and space complexity. Each PPO iteration requires: (1) a full autoregressive rollout from $\pi_\theta$, costing $O(B \cdot T \cdot L)$ where $B$ is batch size, $T$ is response length, and $L$ is the per-token forward-pass cost (roughly linear in model parameter count for a transformer decoding step, ignoring KV-cache reuse which makes decoding itself $O(T)$ sequential steps each $O(1)$ amortized model calls); (2) one reward-model forward pass per response, $O(B \cdot T \cdot L_r)$; (3) $K$ epochs of gradient updates over the batch, each requiring a forward and backward pass through $\pi_\theta$ and $V_\psi$, costing $O(K \cdot B \cdot T \cdot L)$. In practice, four full model copies must reside in memory or be swapped: the policy, the reference policy, the reward model, and the value function (often the value function shares weights with the policy via a separate head, reducing this to three). This is why RLHF is markedly more expensive than supervised instruction tuning — it multiplies the cost of standard fine-tuning by the rollout generation cost and the number of concurrently loaded models.

Why the algorithm is correct (policy-gradient argument). The RL objective is to maximize expected total reward,

The policy-gradient theorem gives

which is unbiased regardless of how $A_t$ is estimated, provided $A_t$ does not depend on $a_t$ in a way that reintroduces bias (a standard baseline-subtraction argument: subtracting any function of the state alone from the reward does not change the expected gradient, because $\mathbb{E}_{a_t}[\nabla_\theta \log \pi_\theta(a_t\mid s_t)] = 0$). This is why using the value function $V_\psi(s_t)$ as a baseline reduces variance without introducing bias, and why GAE — an exponentially weighted average of $k$-step advantage estimates — trades a controllable amount of bias for a large variance reduction, governed by $\lambda$. PPO's clipped surrogate is a correctness-preserving approximation to Trust Region Policy Optimization (TRPO): TRPO proves that maximizing a surrogate objective subject to a KL constraint on policy change guarantees monotonic improvement in expected reward, up to a bound involving the maximum KL divergence between old and new policy. PPO replaces the hard constraint with a soft clipping penalty that empirically achieves similar stability at much lower computational cost (no second-order trust-region computation), at the price of losing TRPO's formal monotonic-improvement guarantee — PPO's correctness is thus empirical and heuristic rather than proven, which is an honest caveat of the method.

Why the KL penalty matters for correctness of the overall objective, not just stability. The reward model $r_\phi$ is only accurate in the region of output space it was trained on — human preference labels were collected on responses from earlier policies (SFT / earlier RLHF checkpoints), not on arbitrary token sequences. Outside this region, $r_\phi$ can be arbitrarily wrong (this is the mechanism behind reward hacking: policy gradient will happily climb toward inputs that trick a fixed scoring function, exactly as adversarial examples exploit fixed classifiers). The KL penalty keeps $\pi_\theta$'s output distribution within a neighborhood of $\pi_{\text{ref}}$'s distribution, which is by construction the distribution the reward model was calibrated on, so the reward signal remains trustworthy in expectation. This is a distributional-shift argument, not merely a fluency heuristic.

Tradeoffs. Larger $\beta$ gives a safer, more conservative policy that stays close to instruction-tuned behavior but captures less of the reward model's signal; smaller $\beta$ extracts more preference-aligned behavior but risks reward hacking and degenerate text. Choosing $\beta$, the clip range $\epsilon$, GAE's $\lambda$, and learning rates is empirically delicate — PPO-based RLHF is notorious for being sensitive to these hyperparameters, and small implementation bugs (wrong reward normalization, KL computed against the wrong reference distribution, value-function collapse) silently degrade quality without crashing training.

Consider a toy instruction-tuned policy answering the prompt: "Write one sentence recommending a good first programming language." Suppose the reward model has learned to prefer concise, concrete, non-hedging answers (higher reward for specificity, lower for filler).

Rollout 3 illustrates reward hacking directly: some degenerate repetition happens to score very high under an imperfectly generalizing reward model, perhaps because the training data for $r_\phi$ under-represented such repetitive strings. Without the KL penalty, PPO's gradient would push hard toward more text like Rollout 3, since it has the single highest raw reward (0.95). With the KL penalty active, the enormous divergence from $\pi_{\text{ref}}$ (which would never produce such repetition) incurs a penalty of $0.1 \times 3.40 = 0.34$, dropping the total reward for Rollout 3 below Rollout 1's. The advantage estimates computed from these totals therefore favor Rollout 1, and the clipped PPO update nudges $\pi_\theta$'s probability mass toward concise, concrete answers like Rollout 1 while pulling it away from both hedging (Rollout 2) and degenerate repetition (Rollout 3). Over many iterations and prompts, this shifts the policy's overall output distribution toward the region of response-space that both scores well under $r_\phi$ and stays plausible under $\pi_{\text{ref}}$ — exactly the intended behavior.

RLHF is the training stage most directly responsible for making large language models behave like cooperative assistants rather than raw text predictors. Production conversational systems use RLHF (or closely related preference-optimization methods) to shape refusal behavior on harmful requests, calibrate hedging and uncertainty expression, enforce formatting and instruction-following fidelity, and reduce toxic or biased outputs relative to the pretraining distribution. Code-generation assistants use analogous pipelines with reward signals derived from unit-test pass rates or human coding-quality judgments. Because the entire method rests on optimizing against a proxy that only approximates true human intent, RLHF is the direct technical entry point into the broader questions studied under Alignment & the Alignment Problem — including specification gaming, goal misgeneralization, and the difficulty of scaling human oversight — and into the practical discipline of Safety & Red-Teaming, which stress-tests RLHF-trained models for exactly the reward-hacking and distributional-shift failures this article's optimization argument predicts.

Ask the tutor: How would the RLHF objective and update rule change if the reward model produced per-token rewards instead of a single terminal score?