AI Evaluations and Red-Teaming

AI Evaluations and Red-Teaming are systematic methodologies for probing the capabilities, limitations, and failure modes of AI systems—particularly large language models—through structured testing, adversarial prompting, and quantitative benchmarking. They matter because deployed models routinely exhibit dangerous or unintended behaviors that standard training metrics entirely miss, making post-hoc safety assessment a critical last line of defense before public release. The counterintuitive fact: red-teaming often finds that models are simultaneously more capable than benchmarks suggest (via jailbreaks revealing hidden competencies) and less reliable (via prompt sensitivity exposing brittle reasoning).

The core insight

Evaluation is not the same as benchmarking. Benchmarks measure average-case performance on a fixed distribution; evaluations and red-teaming are adversarial processes designed to find the worst-case distribution—the prompts, contexts, and edge cases where a model fails catastrophically or behaves in ways misaligned with human intent. The "aha moment" is recognizing that a model scoring 90% on MMLU can still produce bioweapon synthesis routes, sycophantically agree with false premises, or be trivially jailbroken with a fictional framing—none of which any standard benchmark would catch. Good evaluation is threat modeling applied to AI.

How It Works

The Evaluation Taxonomy

AI evaluation decomposes into three complementary regimes: capability evaluation (what can the model do?), alignment evaluation (does it do what users and developers intend?), and robustness evaluation (does it behave consistently under distribution shift or adversarial pressure?). Red-teaming specifically targets the intersection of capability and misalignment—behaviors the model can perform but shouldn't.

Formal Setup

Let a language model be parameterized as a conditional distribution pθ(y∣x)p_\theta(y \mid x) where xx is a prompt and yy is a completion. Define a harm scoring function H:X×Y→[0,1]H: \mathcal{X} \times \mathcal{Y} \to [0,1] that maps prompt-completion pairs to a harm score (often itself a trained classifier). The red-teaming objective is:

1
x∗=arg⁡max⁡x∈X  Ey∼pθ(⋅∣x)[H(x,y)]\quad x^* = \arg\max_{x \in \mathcal{X}} \; \mathbb{E}_{y \sim p_\theta(\cdot \mid x)}\bigl[H(x, y)\bigr]

Explanation: Find the prompt x∗x^* that maximizes expected harm of the model's output—this is the adversarial prompt search problem at the heart of red-teaming. Here X\mathcal{X} is the space of all possible prompts (constrained in practice by semantic coherence), pθp_\theta is the model being evaluated, and HH is the harm classifier. Because X\mathcal{X} is astronomically large and HH is expensive to evaluate, this optimization is almost never solved exactly—it drives the need for human red-teamers and automated approximations.

Automated Red-Teaming via a Separate Attack Model

A tractable approach trains a separate attacker model pϕ(x)p_\phi(x) to generate prompts that maximize harm, optimized via reinforcement learning against the target model:

2
L(ϕ)=−Ex∼pϕ[Ey∼pθ(⋅∣x)[H(x,y)]−λ⋅KL(pϕ(x) ∥ pref(x))]\quad \mathcal{L}(\phi) = -\mathbb{E}_{x \sim p_\phi}\Bigl[\mathbb{E}_{y \sim p_\theta(\cdot \mid x)}[H(x,y)] - \lambda \cdot \mathrm{KL}\bigl(p_\phi(x) \,\|\, p_{\text{ref}}(x)\bigr)\Bigr]

Explanation: The attacker model is trained to maximize harm while staying close to a reference distribution (via KL regularization), preventing it from degenerating into incoherent token soup. Here ϕ\phi are the attacker's parameters, λ>0\lambda > 0 controls the diversity-harm tradeoff, and prefp_{\text{ref}} is typically a pretrained language model acting as a fluency prior. This is exactly the RLHF objective in reverse—instead of maximizing human preference, it maximizes harm. The KL term ensures generated prompts remain semantically meaningful.

Measuring Evaluation Quality: Attack Success Rate and Coverage

The primary metric is Attack Success Rate (ASR):

3
ASR=1∣T∣∑x∈T1[Ey∼pθ(⋅∣x)[H(x,y)]>τ]\quad \text{ASR} = \frac{1}{|T|} \sum_{x \in T} \mathbf{1}\Bigl[\mathbb{E}_{y \sim p_\theta(\cdot \mid x)}[H(x,y)] > \tau\Bigr]

Explanation: The fraction of adversarial test prompts TT for which the model's expected output harm exceeds threshold τ\tau—higher ASR means the model is more vulnerable. But ASR alone is insufficient because it doesn't measure coverage—whether the red-team found diverse failure modes or just one exploit repeated many ways. A semantic diversity score over the attacking prompts is therefore often computed alongside ASR.

Evaluating Evaluators: The Meta-Problem

A subtler issue is evaluating the harm classifier HH itself. If HH is miscalibrated, all downstream metrics are corrupted. A standard calibration check uses expected calibration error:

4
ECE=∑b=1B∣Mb∣n∣acc(Mb)−conf(Mb)∣\quad \text{ECE} = \sum_{b=1}^{B} \frac{|M_b|}{n} \bigl|\text{acc}(M_b) - \text{conf}(M_b)\bigr|

Explanation: The ECE partitions predictions into BB confidence bins MbM_b and penalizes gaps between mean predicted confidence and actual accuracy in each bin, quantifying how well HH's scores correspond to true harm probabilities. Here nn is the total number of evaluated examples, ∣Mb∣|M_b| is the number of samples in bin bb, acc(Mb)\text{acc}(M_b) is empirical accuracy within the bin, and conf(Mb)\text{conf}(M_b) is mean predicted confidence. A well-calibrated harm classifier is a prerequisite for trustworthy automated evaluation pipelines.

researcher · Paul Röttger | b. 1993 | HarmBench and Behavioral Evaluation Frameworks

Paul Röttger is a computational social scientist and NLP researcher whose work bridges sociolinguistics and AI safety evaluation. He created the ToxiGen and HatEval datasets and co-developed systematic frameworks for evaluating LLM safety behaviors, particularly around hate speech and harmful content detection. Surprisingly, his background is in linguistics rather than computer science—he holds a DPhil from Oxford in computational linguistics—making him an outlier in a field dominated by ML engineers.

researcher · Ethan Perez | b. ~1995 | Automated Red-Teaming with Language Models

Ethan Perez is an AI safety researcher at Anthropic who, while at New York University and later Anthropic, pioneered using language models themselves as automated red-teamers to generate adversarial test cases at scale. His 2022 paper "Red Teaming Language Models with Language Models" established the formal framework now widely used for scalable automated evaluation. In a striking irony, he used GPT-2-class models to find failure modes in much larger models—demonstrating that attack capability doesn't require matching the target's scale.

Figure
AI Evaluation and Red-Teaming Pipeline
AI Evaluation & Red-Teaming Pipeline Human Red-Teamers adversarial prompts Attacker LM p_φ(x) — RL-trained to maximise harm Benchmark Suite MMLU · HarmBench · WMDP Target LLM p_θ(y | x) GPT-4 / Claude / Gemini or custom model under test Harm Classifier H(x, y) → [0,1] calibrated · audited Evaluation Metrics ASR · ECE · Coverage Semantic Diversity Score Capability Elicitation Rate RL reward signal
Figure
Jailbreak Attack Taxonomy
Jailbreak Attack Taxonomy Attack Vectors on target LLM Prompt Injection override system prompt via user turn Role-play / Fiction DAN, "pretend you are an evil AI" framing Many-Shot / Few-Shot examples normalise harmful completions GCG / Token Opt. gradient-based suffix optimisation attack Fine-tuning Attack few harmful examples remove safety training ─── corresponding defences ─── Prompt shields / spotlighting Persona refusal training Context-window limits Perplexity filtering Fine-tune auditing

Intuition

The Mental Model

Think of deploying an LLM like launching a passenger aircraft. Benchmarks are the flight simulator tests—they measure performance on anticipated scenarios under controlled conditions. Red-teaming is the adversarial testing the FAA mandates before any aircraft flies commercially: stress-testing the airframe beyond normal operating loads, introducing simulated bird strikes, testing what happens when two warning systems fail simultaneously. No airline executive would accept "we scored 97% on our simulation suite" as sufficient evidence of safety, because simulations only cover scenarios the engineers imagined. Red-teamers are the engineers who try to imagine the scenarios that weren't imagined—and then try to cause them. The crucial insight is that the model's training distribution and the real-world deployment distribution are never identical, and the gap between them is exactly where dangerous behaviors hide.

Historical Development

Origins · 1 minhow this was discovered
History
2016
OpenAI publishes "Concrete Problems in AI Safety" (Amodei et al.), formally introducing reward hacking, safe exploration, and distributional shift as core safety evaluation challenges—the intellectual foundation for modern red-teaming.
2019
Anthropic researchers (then at OpenAI) demonstrate that GPT-2 can be prompted into generating toxic content through simple role-play framing, motivating investment in structured adversarial testing before public model releases.
2021
Perez et al. publish "True Few-Shot Learning with Language Models," incidentally revealing that benchmark performance is highly sensitive to prompt format—sparking awareness that evaluation methodology itself is a research problem.
2022
Perez et al. publish "Red Teaming Language Models with Language Models" at Anthropic, establishing the first formal automated red-teaming framework using a separate attacker LM trained via reinforcement learning.
2022
Zou et al. develop GCG (Greedy Coordinate Gradient) attacks, showing that token-level gradient optimization can produce universal adversarial suffixes that jailbreak multiple aligned LLMs, including ChatGPT—demonstrating white-box attacks transfer to black-box targets.
2023
OpenAI's GPT-4 technical report includes the first major public disclosure of pre-deployment red-teaming outcomes, including contractor-assisted evaluations for CBRN (Chemical, Biological, Radiological, Nuclear) uplift risk.
2023
Anthropic releases the HarmBench framework and the WMDP (Weapons of Mass Destruction Proxy) benchmark—standardizing evaluation across labs and enabling reproducible safety comparisons.
2024
The UK AI Safety Institute (now AISI) and US AISI conduct frontier model evaluations for biological and cybersecurity capabilities under the Bletchley Declaration framework—the first government-run red-teaming exercises on pre-release frontier models.
2024
Anil et al. demonstrate many-shot jailbreaking: exploiting long-context windows by including hundreds of in-context examples of harmful Q&A, bypassing safety training with no gradient access—redefining the attack surface for long-context models.

Why It Matters

Real-World Impact
GPT-4 (OpenAI, 2023)OpenAI ran a six-month red-team exercise with 50+ external domain experts covering bioweapons, cyberoffense, and CBRN topics before release; red-teamers found that without mitigations the model could provide meaningful uplift to non-expert users seeking to synthesize dangerous chemicals, directly motivating refusal training on specific hazardous chemistry prompts.
Claude (Anthropic)Anthropic's Constitutional AI pipeline uses automated red-teaming (a red-team LM generating harmful prompts) to generate adversarial data, which then trains the model's critic in a self-critique loop—making red-teaming a training ingredient rather than only a post-hoc test.
Gemini (Google DeepMind, 2023)Google's Responsible Generative AI Toolkit includes a structured red-teaming framework for evaluating multimodal models, specifically targeting cross-modal prompt injection where a harmful text instruction is embedded in an image to bypass text-only safety filters.
WMDP BenchmarkHaimes et al. (2024) developed the Weapons of Mass Destruction Proxy benchmark as a capability unlearning evaluation—a set of multiple-choice questions about bio, cyber, and chemical weapons that should not be answerable by a safely aligned model, enabling quantitative measurement of dangerous knowledge retention after safety fine-tuning.
Microsoft Copilot / Azure AIMicrosoft's AI Red Team (AIRT) runs continuous adversarial testing of production AI products, maintaining a living taxonomy of attacks ("prompt injection," "jailbreak," "indirect injection via documents") and has published results showing that indirect prompt injection via retrieved documents is substantially harder to defend than direct user-turn attacks.
HarmBench (Zou et al., 2024)This standardized evaluation framework defines 18 attack methods and 7 harm categories, tested across 33 models—the first systematic comparison of red-teaming attack efficacy across the field, revealing that no single attack dominates across all models and harm categories.
UK AI Safety Institute Frontier EvaluationsAISI runs structured capability evaluations on pre-release frontier models from major labs, specifically testing for autonomous replication, cyberoffense capability, and deceptive alignment behaviors—operationalizing government-level AI safety evaluation for the first time.

Prerequisites

What This Leads To

See Also

Reinforcement Learning from Human Feedback · Constitutional AI · Adversarial Machine Learning · Scalable Oversight · AI Alignment and Safety · Mechanistic Interpretability · Benchmark Design and Evaluation Methodology

Before this

Where this leads