Debate as an Alignment Technique

Formal Setup

Let $\mathcal{Q}$ be a space of questions and $\mathcal{A}$ a space of answers. Two debater agents $D_1, D_2$ with policies $\pi_1, \pi_2 : \mathcal{Q} \times \mathcal{H} \to \mathcal{M}$ map a question and debate history $\mathcal{H}$ to messages in message space $\mathcal{M}$. A human judge $J : \mathcal{Q} \times \mathcal{H} \to [0,1]$ outputs the probability that $D_1$'s claimed answer is correct given the full debate transcript. The debaters play a zero-sum game; $D_1$ receives reward $r = J(q, h)$ and $D_2$ receives $1 - r$.

Explanation: The value of debate on question $q$ is the reward that the first debater can guarantee against the worst-case opponent, where $\tau$ denotes the transcript generated by both policies. Here $V^*(q)$ is the minimax value; if honest play achieves $V^*(q) = \mathbb{1}[\text{true answer}]$ then debate is said to be truth-tracking.

The Honesty Theorem

Irving et al. (2018) establish conditions under which truthful play is a Nash equilibrium. Let $\delta(a)$ be the indicator that answer $a$ is correct, and assume the judge is a polynomial-time verifier: they can check individual argument steps but cannot independently solve $\mathcal{NP}$-hard subproblems.

Explanation: The judge is more likely to side with the first debater when they hold the correct answer, even against an optimal adversary $\pi_2^*$, which makes honesty a best response. The proof proceeds by induction on argument depth $d$. At depth $d=0$ (leaf claims), the judge directly evaluates verifiable facts; correctness dominates. At depth $d > 0$, the honest agent can always expose one false sub-claim in the opponent's argument tree, reducing the problem to depth $d-1$ where the inductive hypothesis applies.

Computational Complexity View

The complexity-theoretic framing connects debate to the class PSPACE and interactive proof systems. Let $\mathsf{IP}[k]$ denote problems solvable by $k$-message interactive proofs.

Explanation: Every language decidable in polynomial space has an interactive proof with a polynomial-time verifier—the theoretical bedrock that makes debate's scalability plausible. Debate can be viewed as a two-prover interactive proof: $D_1$ plays the honest prover, $D_2$ plays the adversarial prover, and $J$ is the verifier. The class of languages decidable by two competing provers with a polynomial-time verifier is exactly $\mathsf{PSPACE}$, matching the class of problems solvable by interactive proofs. This provides a complexity-theoretic upper bound on the power of debate.

Training Objective

In practice, debater policies are trained via Reinforcement Learning from Human Feedback. The reward model $R_\phi$ parameterized by $\phi$ is trained on human judgments:

Explanation: The reward model is trained via binary cross-entropy so that it assigns higher scores to transcripts $h$ where the human label $y=1$ (first debater wins), with $\sigma$ denoting the logistic function. Debater policies are then optimized against $R_\phi$ with a KL-divergence penalty from a reference policy $\pi_\text{ref}$ to prevent reward hacking:

Explanation: The debater maximizes expected judge reward while staying close to a reference distribution, with $\beta > 0$ controlling the strength of the regularization that prevents policy collapse.

Scalable Oversight · Iterated Amplification · Reinforcement Learning from Human Feedback · Constitutional AI · Reward Modeling · Interactive Proof Systems and Complexity Theory · AI Red-Teaming and Adversarial Evaluation