Mechanistic Interpretability

Mechanistic interpretability is the scientific program of reverse-engineering neural networks into human-understandable algorithms—identifying the specific circuits, features, and computations that implement model behavior, rather than treating the network as a black box characterized only by input-output statistics. It matters because as frontier models are deployed in high-stakes domains, behavioral evaluations alone cannot detect deceptive alignment, hidden capabilities, or subtle failure modes that only manifest under distribution shift. The remarkable fact: a handful of attention heads in GPT-2 Small implement an identifiable, transferable "indirect object identification" circuit that can be surgically ablated with predictable behavioral consequences—demonstrating that large language models contain discrete, interpretable subroutines rather than diffuse distributed computation.

The core insight

A trained neural network is not a learned statistical table—it is a compiled program written in weight space, and mechanistic interpretability is the discipline of decompiling it. The central hypothesis, formalized as the Linear Representation Hypothesis, is that neural networks represent concepts as directions in activation space and compose them via approximately linear superposition; this means that features, circuits, and algorithms are in principle recoverable using linear algebra over the residual stream. The "aha moment" is that what looks like inscrutable high-dimensional computation is, in many cases, implemented by a small number of attention heads and MLP neurons executing identifiable, nameable operations—induction, copying, inhibition, lookup—that compose into a recognizable algorithm. If this hypothesis holds at scale, interpretability becomes an engineering discipline rather than a philosophical one.

Common trapCommon Misconception

The most common mistake is conflating mechanistic interpretability with feature attribution / saliency methods (SHAP, integrated gradients, attention visualization). Practitioners assume that identifying which input tokens a model attends to is equivalent to understanding how the model computes its answer—but these are categorically different questions. Attribution methods answer "what inputs matter?" while mechanistic interpretability answers "what algorithm runs on those inputs?" A saliency map showing that a model attends to the subject noun says nothing about whether the model is doing syntactic agreement, semantic lookup, or shallow positional heuristics. The correct mental model: attribution is behavioral epidemiology (correlation between inputs and outputs), while mechanistic interpretability is molecular biology (the causal mechanism itself). Confusing the two leads to false confidence—a model can produce correct saliency maps for entirely wrong mechanistic reasons, a phenomenon directly relevant to alignment failures.

How It Works

The Residual Stream Framework

The transformer architecture is most naturally analyzed through the residual stream lens (Elhage et al., 2021). Each layer reads from a shared vector and writes additive updates back to it, enabling clean attribution of computational contributions.

\quad x^{(\ell+1)} = x^{(\ell)} + \text{Attn}^{(\ell)}(x^{(\ell)}) + \text{MLP}^{(\ell)}(x^{(\ell)})

Explanation: Each layer's output is a residual correction to the stream, meaning any layer's contribution can be isolated and analyzed independently via direct path decomposition. Here $x^{(\ell)} \in \mathbb{R}^d$ is the residual stream at layer $\ell$ , $\text{Attn}^{(\ell)}$ is the multi-head attention block, and $\text{MLP}^{(\ell)}$ is the feedforward sublayer. The additive structure means the final logits decompose as a sum of contributions from every component, amenable to logit lens analysis.

Attention Head Decomposition

Each attention head $h$ in layer $\ell$ computes an output that can be written as a rank-factored linear map composed with a softmax routing pattern:

\quad \text{Attn}_h(x) = \sum_{s} \alpha_{hs} \cdot (x_s W_V^h) W_O^h = \sum_s \alpha_{hs} \cdot x_s (W_V^h W_O^h)

Explanation: Each head moves information from source positions to destination positions, scaled by attention weight $\alpha_{hs}$ , via a fixed OV (output-value) circuit that can be analyzed as a standalone linear map. Here $W_V^h \in \mathbb{R}^{d \times d_h}$ and $W_O^h \in \mathbb{R}^{d_h \times d}$ are value and output projection matrices; $\alpha_{hs}$ is the attention score from position $s$ to destination; and $W_V^h W_O^h \in \mathbb{R}^{d \times d}$ is the OV matrix whose eigenstructure encodes what the head copies. Separately, the QK circuit $W_Q^{h\top} W_K^h$ encodes where it attends. This factorization enables independent circuit analysis.

Superposition and the Polysemanticity Problem

A key mathematical obstacle is superposition: networks represent more features than they have dimensions by encoding features as nearly-orthogonal directions in a compressed space. Elhage et al. (2022) formalize this via a sparse coding objective. Let $x \in \mathbb{R}^n$ be a data point with $m \gg n$ latent features $f_i$ with sparsity $s_i$ . The network learns a matrix $W \in \mathbb{R}^{n \times m}$ such that:

\quad \mathcal{L}(W) = \sum_i (1 - s_i)\left\| W W^\top e_i - e_i \right\|^2 + \lambda \sum_i s_i \left\| W^\top e_i \right\|_1

Explanation: The network trades off faithful reconstruction of frequent features against interference between all features, producing a geometry where features pack into superposition rather than occupying dedicated dimensions. The consequence is polysemanticity: individual neurons respond to multiple semantically unrelated features because they are simultaneously encoding components of many superposed feature directions, making naive neuron-level interpretation misleading.

Sparse Autoencoders for Feature Recovery

The current dominant approach to disentangling superposed representations is training a sparse autoencoder (SAE) on the residual stream (Cunningham et al., 2023; Bricken et al., 2023). Given activations $x \in \mathbb{R}^d$ , the SAE learns encoder $E \in \mathbb{R}^{m \times d}$ and decoder $D \in \mathbb{R}^{d \times m}$ with $m \gg d$ :

\quad \hat{x} = D \cdot \text{ReLU}(Ex + b), \quad \mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|^2 + \lambda \|\text{ReLU}(Ex + b)\|_1

Explanation: The $\ell_1$ sparsity penalty forces the hidden code to use only a few active features at once, recovering the sparse ground-truth features that the network has superposed into a lower-dimensional activation space. Here $b \in \mathbb{R}^m$ is a bias, $m$ is the dictionary size (typically $4\times$ – $64\times$ the activation dimension $d$ ), and $\lambda$ controls sparsity pressure. The decoder columns $D_{:,k}$ are interpreted as feature directions; the $k$ -th hidden unit activates when that feature is present in $x$ .

Causal Intervention and Circuit Analysis

Identifying a circuit requires not just correlation but causal proof via activation patching (Meng et al., 2022). The patch effect of replacing component $c$ 's activation with a counterfactual value $a^*$ on output logit $\ell$ is:

\quad \Delta_c = \mathbb{E}_{x, x^*}\left[\ell\!\left(x \;\big|\; \text{do}(a_c \leftarrow a_c^{x^*})\right) - \ell(x)\right]

Explanation: A large $|\Delta_c|$ implies that component $c$ causally mediates the behavior of interest, not merely correlates with it, providing the empirical basis for circuit identification. Here $x$ is the clean input, $x^*$ is a corrupted or counterfactual input, $a_c^{x^*}$ is the activation of component $c$ on input $x^*$ , and $\text{do}(\cdot)$ denotes Pearl's do-calculus intervention. Systematic patching across all components yields a causal attribution graph identifying the minimal circuit.

researcher · Chris Olah | b. 1990 | Founder of the circuits research program

Chris Olah trained as a computer scientist and joined Google Brain, later Anthropic, becoming the primary architect of mechanistic interpretability as a formal discipline. His 2020 "Zoom In" essay and subsequent circuits papers established the core methodology of identifying individual features in neurons, composing them into circuits, and verifying claims via visualization and ablation on CNNs before scaling to transformers. Unusually for a leading AI researcher, Olah has no PhD and is largely self-taught—his influence stems entirely from the explanatory clarity and empirical rigor of his published work.

researcher · Neel Nanda | b. 1998 | TransformerLens and scalable circuit analysis

Neel Nanda completed his undergraduate degree at Cambridge and joined Anthropic then DeepMind, becoming the most prolific empirical practitioner of transformer mechanistic interpretability. He developed TransformerLens, the standard open-source library for activation patching and attention head analysis, and produced the definitive mechanistic account of how a two-layer transformer learns modular arithmetic ("grokking"), revealing that generalization corresponds to a phase transition in the weight structure implementing discrete Fourier features. Nanda's public daily research logs and tutorials have trained the majority of the current generation of interpretability researchers.

researcher · David Bau | b. 1974 | Causal tracing and knowledge localization in language models

David Bau is a professor at UMass Amherst whose background spans computer graphics and program analysis, giving him an unusual perspective on networks as editable computational systems. His ROME paper (2022, with Meng et al.) introduced causal tracing—systematic activation patching to localize where factual associations are stored in GPT models—and demonstrated that mid-layer MLP modules in specific positions function as key-value stores for world knowledge. A striking result from this work: single factual associations can be edited with sub-1% collateral damage via rank-one weight updates, demonstrating that transformer weights are more modularly organized than previously assumed.

Figure

Mechanistic Interpretability Pipeline

Figure

Superposition Geometry

Intuition

The Mental Model

Imagine you have recovered an ancient device and want to understand how it works. You could characterize it behaviorally—press button A, light B turns on—and build a comprehensive input-output table. That is what behavioral evaluations of neural networks do. Mechanistic interpretability is the decision to open the device and trace the wiring: not just what outputs follow from what inputs, but why—which transistors fire, which signals propagate, what intermediate states exist. The complication is that the device was assembled by an optimizer with no obligation to use human-readable components. So the circuit diagram you find may look like a pentagonal arrangement of directions in a 2D space rather than clean labeled wires—the optimizer discovered that packing five overlapping signals into two wires is more efficient than using five separate wires, because the signals are rarely all active simultaneously. Sparse autoencoders are the tool you use to unpack those overlapping signals back into separate, labeled wires before you can trace the circuit. Once unpacked, you discover that specific groups of attention heads are doing specific named operations—like the heads in GPT-2 that exclusively implement "copy the previous token" or the heads that implement "skip back to the beginning of the current subject"—and these compose into a recognizable subroutine, not unlike a for-loop with a hash-table lookup.

Historical Development

Origins · 1 minhow this was discovered

History

2013

Zeiler & Fergus publish "Visualizing and Understanding Convolutional Networks," introducing the deconvnet for visualizing CNN feature detectors—the first systematic attempt to characterize internal representations rather than just outputs.

2015

Yosinski et al. demonstrate that CNN neurons are interpretable and transferable across networks, establishing that learned features are not arbitrary artifacts but robust representations with consistent semantics.

2020

Olah, Cammarata et al. publish "Zoom In: An Introduction to Circuits" in Distill, establishing the circuits framework—the claim that features, circuits, and universality are the three pillars of a mechanistic science of neural networks.

2021

Elhage, Nanda et al. (Anthropic) publish "A Mathematical Framework for Transformer Circuits," introducing the residual stream formalism, OV/QK decomposition, and identification of induction heads as the first mechanistically understood transformer circuit.

2022

Meng, Bau et al. publish "Locating and Editing Factual Associations in GPT" (ROME), demonstrating via causal tracing that factual knowledge is localized to specific MLP layers and can be surgically edited via rank-one updates.

2022

Wang et al. (Redwood Research) publish "Interpretability in the Wild," completing a full mechanistic account of the IOI (indirect object identification) circuit in GPT-2 Small across 26 attention heads—the most complete circuit analysis of a complex behavior to date.

2023

Bricken et al. (Anthropic) and Cunningham et al. independently demonstrate that sparse autoencoders trained on MLP activations recover monosemantic, interpretable features, resolving the polysemanticity bottleneck at small scale.

2024

Anthropic publishes "Scaling and Evaluating Sparse Autoencoders" (Gao et al.) and the "Mapping the Mind of a Large Language Model" interpretability analysis of Claude, identifying millions of features including abstract concepts like "the Inner Monologue" and evidence of safety-relevant internal structures.

2024

Lindsey et al. (Anthropic) demonstrate cross-layer transcoding SAEs that reconstruct MLP sublayers as sparse linear combinations of interpretable features, enabling circuit-level analysis without activation patching—a major scalability advance.

Why It Matters

Real-World Impact

Anthropic's Claude (internal alignment monitoring)Anthropic uses SAE-derived feature analysis to probe whether Claude's internal representations contain safety-relevant features (e.g., deceptive intent, topic-specific refusal states) that behavioral evaluations might miss; this is now part of the model evaluation pipeline for frontier deployments.

GPT-2 IOI Circuit (Wang et al., 2022)The indirect object identification circuit—where the model completes "John gave Mary the book; Mary gave [John]"—was fully reverse-engineered into 26 attention heads with named roles (duplicate token heads, S-inhibition heads, name mover heads), demonstrating that complex linguistic behavior is implemented by identifiable subroutines susceptible to surgical ablation.

ROME / MEMIT knowledge editing (Meng et al., 2022; Mitchell et al., 2022)Causal tracing in GPT-J and GPT-NeoX localized factual associations to specific MLP layers, enabling rank-one weight edits that update single facts with <1% collateral damage—a practically deployed technique for correcting factual errors in production LLMs without full retraining.

Induction head discovery (Olah et al., 2021)Induction heads—pairs of attention heads that implement in-context few-shot learning by pattern-matching and completing sequences—were found to be a universal circuit present in every transformer studied, explaining a large fraction of in-context learning capability and suggesting that removing them predictably degrades few-shot performance by ~50%.

Toy models of superposition (Elhage et al., 2022)The formal analysis of superposition in two-layer MLPs trained on synthetic data revealed the geometric structure of feature interference, directly motivating the SAE approach and enabling quantitative predictions about when polysemanticity is expected that have since been empirically validated in large models.

Grokking mechanistic account (Nanda et al., 2023)Nanda et al. fully reverse-engineered a small transformer learning modular arithmetic, identifying the exact algorithm (Fourier multiplication via specific frequency components) and demonstrating that generalization corresponds to a discrete phase transition—a mechanistic proof that neural networks can implement exact algorithmic reasoning that generalizes perfectly.

Automated interpretability (Bills et al., 2023; OpenAI)OpenAI deployed GPT-4 to automatically generate and score natural language descriptions of GPT-2 neuron behavior, achieving correlations of ~