Scalable Oversight

Formal Setup

Let $\mathcal{X}$ be a task space, $\mathcal{Y}$ an output space, and $r^* : \mathcal{X} \times \mathcal{Y} \to [0,1]$ the ground-truth reward function that a fully-informed, unlimited-capacity evaluator would assign. In practice, a human evaluator $H$ with bounded cognitive capacity $\kappa$ can only directly compute a proxy reward $\hat{r}_H : \mathcal{X} \times \mathcal{Y} \to [0,1]$. The fundamental scalable oversight objective is to design a protocol $\Pi$ such that the elicited reward $r_\Pi$ satisfies:

Explanation: The expected absolute error between the protocol-elicited reward and the ground-truth reward should be small across the task distribution, even when $H$ alone cannot compute $r^*$ directly. Here $\varepsilon$ is the alignment budget tolerated by the deployment context; $\mathcal{D}$ is the distribution over task–output pairs; $\Pi$ is an interaction protocol that may involve AI assistance, debate, decomposition trees, or multiple rounds of critique.

Debate as an Oversight Protocol

In the debate protocol (Irving et al., 2018), two AI agents $A_1, A_2$ argue opposing positions about the correctness of some claim $y$, and a human judge $H$ renders a verdict. The key claim is that if $H$ can reliably distinguish locally valid from locally invalid argument steps, then an honest agent can always win against a deceptive one in a sufficiently long debate. Define the debate value:

Explanation: The debate value is the expected human verdict when the first agent plays its best-response strategy against an adversarially optimal second agent — honest equilibria are stable when local step verification is easier than global claim verification. This is a two-player zero-sum game; if $H$ can verify single argument steps with error $\delta < \frac{1}{2}$, then under certain assumptions about the structure of proofs, the honest agent achieves $V_\text{debate}(x, y) > \frac{1}{2}$ for correct $y$ even for claims far beyond $H$'s direct comprehension.

Iterated Amplification

Christiano et al. (2018) propose Iterated Amplification (IDA), which constructs a sequence of increasingly capable oversight functions. Let $H$ denote the base human and $M_t$ the model at training step $t$. Define the amplified human $\text{Amp}(H, M_t)$ as a human who can, for any sub-question generated during evaluation, query $M_t$ for help. The amplification operator is:

Explanation: The amplified human solves task $x$ by consulting the current model on up to $k$ sub-tasks $x_1, \ldots, x_k$, then integrating those answers with their own judgment — recursively bootstrapping oversight capacity. The IDA training loop distills $\text{Amp}(H, M_t)$ into $M_{t+1}$:

Explanation: Each new model is trained to imitate the amplified human's judgments, which themselves depend on the previous model — provided the distillation loss is small at each step, capability and alignment co-evolve without either outrunning the other. Convergence of IDA requires that the distillation gap $\|M_{t+1} - \text{Amp}(H, M_t)\|$ shrinks at least as fast as the capability gap grows, a condition that remains an open theoretical question but is empirically tractable in current systems.

Recursive Reward Modeling

Leike et al. (2018) generalize the idea via Recursive Reward Modeling (RRM). A reward model $R_\theta$ is trained not on direct human labels but on labels produced by a human assisted by a previously trained $R_{\theta'}$:

Explanation: This is the standard Bradley–Terry preference loss, but the preference labels $(y \succ y')$ are elicited from a human evaluator whose judgments are augmented by querying an existing reward model, creating a recursive chain of increasingly reliable comparisons.