Safety & Red-Teaming

A jailbreak is any input (or sequence of inputs) that causes a model to violate its intended safety policy — producing content its developers explicitly tried to prevent, such as instructions for violence, extraction of training data, or generation of disallowed content categories. Red-teaming is the practice of proactively searching for such inputs, using either human experts (manual red-teaming) or automated search procedures (automated red-teaming), in order to measure and reduce the attack success rate (ASR): the fraction of adversarial attempts, drawn from some attack distribution, that succeed against a fixed defended model. The key conceptual move is treating safety not as a property to prove but as a security game between two policies: an attacker policy $\pi_{\text{atk}}$ that proposes prompts $x$, and a defended target model $\pi_\theta$ that must refuse harmful requests while remaining helpful for benign ones. Formally, given a harm classifier $H(x, y) \in \{0,1\}$ that labels whether output $y$ to prompt $x$ constitutes a harmful response, the attacker's objective is

The defender's objective is the complementary minimax: minimize the attacker's best achievable success rate over a class of attacks $\mathcal{A}$, while keeping helpfulness (measured on a benign benchmark) above a floor. This framing makes explicit why safety cannot be a single scalar pass/fail: it is a Pareto tradeoff between refusal rate on harmful prompts and compliance rate on benign prompts, and red-teaming exists to trace out and push that frontier, much as adversarial robustness research does for image classifiers, connecting directly to the evaluation methodology in Evaluation of LLMs. It is useful to distinguish three overlapping attack families. Static single-turn attacks append or rewrite the harmful request within one prompt (role-play framing, fictional wrapping, encoding tricks, "grandma exploit" emotional framing). Optimization-based attacks treat the model as a differentiable or query-able black box and search over a suffix or rewrite to maximize a proxy for compliance (e.g., token-level gradient search, genetic algorithms, or an attacker LLM fine-tuned to produce jailbreaks). Multi-turn and agentic attacks exploit conversational state or tool use — building up context across turns, or exploiting an agent's ability to call tools and browse the web, as covered in LLM Agents, where the "prompt" is effectively a whole session or an untrusted document the agent reads.

Automated red-teaming pipelines share a common algorithmic skeleton regardless of the specific attack family: propose a candidate attack, score it against the target model, and update the proposal distribution toward higher-scoring attacks, subject to a query budget. Below is a generic pseudocode template that specializes to gradient-based (e.g., token-suffix optimization), genetic/search-based, and LLM-red-teamer approaches by swapping the `Propose` and `Update` subroutines. ``` ALGORITHM: GenericRedTeamLoop INPUT: target_model M # black-box or white-box victim harm_goal g # e.g. "explain how to make X" scorer S(x, y) # harm/compliance score, higher = more successful attack budget B # max queries to M population_size K attack_template T # e.g. "{prefix} {g} {suffix}" INITIALIZE: population = [ SampleInitialCandidate(T, g) for _ in range(K) ] queries_used = 0 best_x, best_score = None, -inf WHILE queries_used < B: scored = [] FOR x in population: y = M.generate(x) # 1 query s = S(x, y) # harm scorer, e.g. keyword match or judge model queries_used += 1 scored.append( (x, y, s) ) IF s > best_score: best_x, best_score = x, s IF best_score >= SUCCESS_THRESHOLD: BREAK # early stop: attack succeeded

# --- selection + variation (this is where methods diverge) --- parents = SelectTopK(scored, k = K // 2) # elitism children = [] FOR i in range(K - len(parents)): p1, p2 = SampleTwo(parents) child = Recombine(p1.x, p2.x) # crossover on suffix tokens child = Mutate(child, rate = MUTATION_RATE) # token/word-level perturbation children.append(child) population = [p.x for p in parents] + children RETURN best_x, best_score, queries_used ``` Three concrete instantiations of `Propose`/`Update`/scorer make the abstraction precise: 1. Gradient-guided token search (white-box). `Mutate` replaces one suffix token at a time by computing the gradient of the negative log-likelihood of a target compliance string (e.g., "Sure, here is") with respect to the one-hot token embedding, then evaluating a shortlist of top-$k$ candidate token substitutions by actual forward passes (this is the greedy coordinate-gradient pattern). The scorer $S$ is the model's log-probability of the compliance prefix, a cheap and differentiable proxy for jailbreak success. 2. Genetic/black-box search. `Recombine` and `Mutate` operate on natural-language templates (swap synonyms, insert role-play framings, change punctuation/encoding), and $S$ is either a keyword-based refusal detector or a separate judge LLM that scores whether the output actually contains harmful content — needed because log-probability proxies are unavailable in a pure black-box setting. 3. LLM-as-red-teamer. The attacker itself is an LLM fine-tuned or prompted (e.g., via few-shot exemplars of successful jailbreaks, or via RL against the judge's score) to generate the next candidate prompt given the history of `(candidate, score)` pairs — effectively an in-context version of the `Update` step, replacing explicit mutation operators with a learned proposal distribution. The defensive side follows a mirrored loop: given a corpus of discovered jailbreaks, a defender fine-tunes the target model (or trains an input/output classifier) to refuse them, then reruns the attack loop against the patched model to check whether ASR has genuinely dropped or whether the attacker has simply found a new region of prompt space. This iterated attack-defend cycle is the operational definition of "red-teaming as a process."

Query/time complexity. The `GenericRedTeamLoop` performs at most $B$ queries to the target model, each costing one forward generation pass, so wall-clock time is $O(B \cdot T_{\text{gen}})$ where $T_{\text{gen}}$ is the cost of generating a response (itself $O(L \cdot d)$ per token for a transformer with hidden size $d$ and $L$ context length, or more precisely quadratic-in-context attention cost depending on architecture — this is inherited from the target model's own inference cost). The population-based (genetic/black-box) variant additionally costs $O(K)$ per generation for selection and mutation, negligible next to $T_{\text{gen}}$. The gradient-guided variant is more expensive per step: computing the gradient over the vocabulary for each of the $\ell$ suffix positions and evaluating the top-$k$ candidate substitutions by forward pass costs roughly $O(\ell \cdot k \cdot T_{\text{gen}})$ per outer iteration, which is why white-box attacks are typically run for far fewer outer iterations than black-box genetic search is run generations.

Sample/space complexity of the search. The fundamental reason red-teaming cannot be exhaustive is combinatorial: the space of natural-language prompts of length $\ell$ over a vocabulary of size $|V|$ is $|V|^\ell$, astronomically larger than any feasible query budget $B$. Both gradient-guided and genetic methods are local search procedures — greedy coordinate descent or evolutionary hill-climbing — that only ever explore a tiny, structured neighborhood of this space; they succeed empirically because compliance-inducing suffixes turn out to form connected, discoverable "basins" reachable by small greedy steps, not because the search is complete. This is the same phenomenon documented in the adversarial-examples literature for vision models: local perturbation search reliably finds adversarial points even though it explores a vanishing fraction of the space.

Why no defense can be a complete proof. A useful correctness framing: a defense is sound against an attack class $\mathcal{A}$ if ASR $= 0$ for every $x \in \mathcal{A}$, and complete if it also preserves helpfulness on all benign inputs. In principle one could imagine certifying soundness against a fixed, enumerable attack class (e.g., all suffixes up to length $\ell$ from a small vocabulary, via exhaustive or randomized-smoothing certification, analogous to certified robustness in vision). In practice, natural language jailbreaks are drawn from an attack class that is neither fixed nor enumerable — new attack strategies (new encodings, new personas, new languages, new multi-turn structures) are discovered continuously, so any certificate proven against today's known $\mathcal{A}$ says nothing about tomorrow's $\mathcal{A}'$. This is why safety claims in practice are stated as empirical ASR under a stated attacker budget and attack suite, not as formal guarantees — an honest and load-bearing distinction, not a rhetorical hedge.

Why iterated attack-defend converges to some improvement, but not to zero. Each round of the loop (attack, harvest failures, safety-fine-tune, re-attack) is a form of adversarial training: the defender's policy $\pi_\theta$ is updated to increase refusal probability on the discovered failure set $F_i$, i.e. minimizing $\mathbb{E}_{x \in F_i} [-\log \pi_\theta(\text{refuse} \mid x)]$ alongside the original helpfulness objective. This provably reduces ASR on $F_i$ (it is direct supervised fine-tuning on exactly those examples) but only empirically reduces ASR on the next round's attacker, because the attacker adapts to the new $\pi_\theta$ and searches for a disjoint failure region $F_{i+1}$. The process is analogous to fictitious play in a two-player zero-sum game: each side best-responds to a snapshot of the other, and the sequence of ASR values across rounds typically decreases but is not monotonic and has no guaranteed fixed point at zero — the game only terminates in practice when a budget or a diminishing-returns criterion is hit, not because either policy has become truly dominant.

Consider a concrete adversarial goal: get a deployed chat model to produce step-by-step instructions for synthesizing a controlled substance, call it goal $g$. The table traces a genetic/black-box red-team run.

Two things this trace demonstrates concretely: (1) compositional/multi-turn attacks succeed where single-turn attacks fail, because the safety fine-tuning that produced strong single-turn refusals was trained overwhelmingly on single-turn harmful requests, so the conversation-level harmful intent was outside the training distribution the refusal behavior generalizes from; (2) defense against a harvested attack transcript generalizes imperfectly — Round 4's fine-tune closes the exact discovered gap, but Round 5 finds a semantically adjacent rephrasing that partially reopens it, illustrating the fictitious-play dynamic argued in the complexity section: each round is a local patch, not a global fix.

Red-teaming as a term originates in Cold War-era military wargaming, where a designated "red team" role-played the adversary to stress-test defense plans; it was adopted by computer security in the form of penetration testing and later carried directly into machine learning safety as adversarial example research matured into adversarial robustness as a subfield. The specific application to LLM jailbreaks emerged rapidly alongside the deployment of instruction-tuned chat models, once it became clear that RLHF-based refusal training was a learned behavior with exploitable edges rather than a hard constraint.

Every major deployed chat assistant runs a red-teaming program before and after release, since providers explicitly commission internal and external red-teams to probe for weapons-related uplift, CSAM, biosecurity risks, and fraud/abuse before shipping a model, then continue bug-bounty-style programs post-launch. Content moderation pipelines for consumer chat products layer input classifiers (detecting harmful intent before generation), output classifiers (detecting harmful content after generation), and the base model's own refusal training as defense-in-depth, precisely because no single layer is complete on its own. Enterprise deployments of retrieval-augmented and agentic systems — where a model reads untrusted web pages or emails and can take actions — treat prompt injection from those documents as a first-class red-team target, since the attack surface there is any text the system ever ingests, not just user-typed prompts, directly extending the concerns raised in LLM Agents. Regulatory and evaluation frameworks increasingly require documented red-team results (attack suites used, ASR achieved, mitigations applied) as a condition of release, turning what began as an internal engineering practice into an auditable component of model governance, and tying red-teaming tightly to the broader measurement discipline of Evaluation of LLMs and to the normative goals set out in Alignment & the Alignment Problem.

Ask the tutor: How would you design a red-teaming evaluation specifically for an LLM agent that can browse the web and execute code, beyond the text-only attacks described here?