Circuits in Neural Networks

Circuits in neural networks are sparse, compositional subgraphs of neurons and weights that implement identifiable computational primitives — ranging from simple edge detectors to complex multitoken reasoning patterns — discoverable through mechanistic interpretability methods such as activation patching, causal tracing, and sparse probing. Understanding circuits matters because it provides the first principled ladder from raw parameters to human-interpretable behavior, enabling AI safety researchers to verify whether a model's internal computation is aligned with its stated objectives. Remarkably, the same circuits — curve detectors, high-low frequency detectors, even multimodal neurons — have been found to form independently and near-identically across entirely different model families trained on different datasets, a phenomenon called universal circuits that suggests deep geometric constraints on learned representations.

The core insight

A neural network is not a monolithic black box but a modular collection of algorithms: specific subsets of weights form circuits that reliably mediate specific behaviors, and these circuits can be reverse-engineered the same way a hardware engineer reads a PCB. The "aha moment" is that gradient descent, despite having access to no inductive bias toward modularity, nevertheless discovers near-modular structure because modular solutions generalize better under natural data distributions. This means interpretability is not merely aspirational — it is a scientific program with falsifiable hypotheses: claim that circuit C computes function f, then verify by ablating C and confirming the model can no longer compute f while leaving unrelated behaviors intact. The completeness criterion (the circuit accounts for the model's full performance on the task) and the minimality criterion (no strict subgraph suffices) together make circuit claims empirically precise.

Common trapCommon Misconception

The most common mistake is conflating correlation-based feature attribution (e.g., gradient saliency, SHAP values) with circuit-level causal claims. Practitioners assume that high gradient magnitude at neuron v means v is "part of the circuit" for a behavior — but gradient tells you the counterfactual sensitivity of the loss at the current input, not the causal role of the neuron across the full distribution of inputs where the behavior manifests. The correct mental model is interventionist: a neuron or attention head belongs to a circuit for behavior b if and only if (i) patching its activation from a "clean" run into a "corrupted" run restores b (path patching / activation patching), and (ii) ablating it in isolation degrades b while ablating it alongside its downstream consumers does not further degrade it (redundancy-aware minimality). Gradient magnitudes are a useful hypothesis generator but never a circuit validator.

How It Works

Formal Circuit Definition

Let a transformer (or MLP) be a computational graph $G = (V, E)$ where nodes $v \in V$ are residual-stream positions or nonlinear units and directed edges $e = (u, v) \in E$ carry weight matrices. A circuit is a subgraph $C = (V_C, E_C) \subseteq G$ together with an associated task distribution $\mathcal{D}$ and behavior function $b : \mathcal{X} \to \mathcal{Y}$ .

\quad \mathcal{L}_{\text{faithfulness}}(C) = \mathbb{E}_{x \sim \mathcal{D}}\bigl[\ell\bigl(M_C(x),\, M(x)\bigr)\bigr]

Explanation: The faithfulness loss measures how well the isolated circuit $C$ , run with all non-circuit nodes mean-ablated, reproduces the full model's output distribution on the task. Here $M_C$ denotes the model with all nodes outside $C$ replaced by their mean activation over $\mathcal{D}$ , and $\ell$ is KL divergence or cross-entropy; a circuit is faithful when this quantity is near zero.

Activation Patching and Causal Tracing

Given a clean input $x^*$ and a corrupted input $\tilde{x}$ (e.g., a subject-swapped factual prompt), the activation patch effect of node $v$ at layer $\ell$ is:

\quad \Delta_{v}^{(\ell)} = \mathbb{E}\bigl[\text{logit}_{b}\bigl(M^{v \leftarrow h_v^*}(\tilde{x})\bigr) - \text{logit}_{b}\bigl(M(\tilde{x})\bigr)\bigr]

Explanation: This measures how much restoring node $v$ 's clean activation into the corrupted forward pass recovers the correct behavior logit, isolating $v$ 's causal contribution. The operator $M^{v \leftarrow h_v^*}$ denotes surgical replacement of activation $h_v$ during the forward pass on $\tilde{x}$ with the value $h_v^*$ computed on $x^*$ . Meng et al. (2022) used this in ROME/MEMIT to localize factual association storage to mid-layer MLP modules in GPT-2 and GPT-J.

Path Patching

Path patching (Wang et al., 2022) refines activation patching by measuring the effect of restoring $v$ 's activation only along a specific downstream path $p = v \to u_1 \to \cdots \to u_k$ , freezing all other incoming edges to $u_1$ :

\quad \Delta_{v \to p} = \mathbb{E}\bigl[\text{logit}_{b}\bigl(M^{(v,p) \leftarrow h_v^*}(\tilde{x})\bigr) - \text{logit}_{b}\bigl(M(\tilde{x})\bigr)\bigr]

Explanation: Path patching decomposes the total effect of a node into the contributions flowing through each downstream route, enabling edge-level rather than node-level circuit decomposition. This resolves the ambiguity that a node can have high $\Delta_v$ because it feeds many irrelevant heads that happen to compose with the relevant one.

Logit Lens and Direct Logit Attribution

In a transformer with residual stream $\mathbf{x}^{(\ell)} = \mathbf{x}^{(0)} + \sum_{i=1}^{\ell} \mathbf{c}_i$ (where $\mathbf{c}_i$ is the contribution of the $i$ -th component — attention head or MLP layer), the direct logit attribution of component $i$ to token $t$ is:

\quad \text{DLA}_i(t) = \mathbf{e}_t^\top W_U \mathbf{c}_i

Explanation: This decomposes the final logit for token $t$ into a sum of signed scalar contributions from each attention head and MLP layer, exploiting the linearity of the residual stream. Here $W_U \in \mathbb{R}^{d_{\text{vocab}} \times d_{\text{model}}}$ is the unembedding matrix and $\mathbf{e}_t$ is the one-hot token indicator. Elhage et al. (2021) showed that this decomposition is exact and admits clean circuit-level bookkeeping in the "Mathematical Framework for Transformer Circuits."

Superposition and Polysemanticity

Neurons are rarely monosemantic. The superposition hypothesis (Elhage et al., 2022) posits that a model with $d$ neurons represents $n \gg d$ near-orthogonal features by encoding them as interference patterns, tolerating cross-talk proportional to feature frequency and importance. The capacity of a layer under superposition scales as:

\quad n^* \approx d \cdot \frac{1}{1 - S} \cdot \frac{1}{\epsilon^2}

Explanation: The maximum number of features representable with at most $\epsilon$ interference noise scales inversely with feature sparsity $S$ (fraction of inputs on which each feature is active) and inversely with tolerated noise squared. This has profound consequences for circuit analysis: individual neurons are poor primitives; directions in activation space (or sparse dictionary atoms learned by sparse autoencoders) are the correct unit of circuit decomposition.

researcher · Chris Olah | b. 1991 | Mechanistic Interpretability and Circuit Discovery

Chris Olah trained as a self-taught researcher and joined Google Brain before co-founding Anthropic, where he leads mechanistic interpretability research. His 2020 "Zoom In" paper with colleagues at OpenAI introduced the circuits conjecture — that neural networks learn a universal library of reusable, reverse-engineerable circuits — by demonstrating specific curve detectors and high-low frequency detectors shared across InceptionV1, AlexNet, and VGG. Remarkably, Olah's most influential early work, the 2014 blog post "Neural Networks, Manifolds, and Topology," was written before he held any formal research position and circulated purely on his personal blog, yet it shaped a generation of representation learning researchers.

researcher · Neel Nanda | b. 1998 | Grokking Circuits and Automated Interpretability

Neel Nanda completed his undergraduate degree at Cambridge and joined DeepMind's interpretability team after a brief stint at Anthropic, publishing at an unusually high rate for a junior researcher. His 2022 paper "Progress Measures for Grokking via Mechanistic Interpretability" reverse-engineered the exact algorithm (modular arithmetic via Fourier components) that a one-layer transformer discovers during the grokking phase transition, providing the first complete mechanistic account of a generalization phenomenon. He also released TransformerLens, the open-source library that has become the de facto standard toolkit for mechanistic interpretability experiments, dramatically lowering the barrier to circuit research.

Figure

Transformer Circuit: Induction Head Composition

Figure

Sparse Autoencoder Feature Decomposition

Intuition

The Mental Model

Think of a mature city's electrical grid. The grid as a whole is vastly complex, but it is not random — it is made of identifiable, reusable components: substations, transformers, breaker boxes, and residential circuits. An electrical engineer does not need to simulate every electron to understand why turning off breaker 7 kills the kitchen lights; she traces the specific copper path from the panel to the outlet. Neural network circuits work the same way. Gradient descent, over millions of gradient steps on natural data, finds that the most weight-efficient solution is to reuse primitive computational motifs — edge detectors feeding curve detectors feeding shape detectors — stacked compositionally. The reason the same circuits appear in ResNet-50 trained by Google and VGG trained by Oxford is the same reason that every city's electrical grid uses the same three-phase AC standard: the problem has a unique near-optimal modular solution. Superposition is like running multiple circuits through the same physical wire using frequency-division multiplexing: interference is low as long as signals are sparse, but tracing individual circuits requires the equivalent of a spectrum analyzer — which is exactly what a sparse autoencoder is.

Historical Development

Origins · 1 minhow this was discovered

History

1986

Rumelhart, Hinton & Williams demonstrate that backpropagation learns internal representations, prompting early speculation that hidden layers encode features — but no systematic methodology for extracting them.

2013

Zeiler & Fergus ("Visualizing and Understanding Convolutional Networks") use deconvnet activations to show that AlexNet learns oriented Gabor-like filters in layer 1 and texture detectors in layer 2 — the first empirical circuit-adjacent analysis in modern deep learning.

2015

Szegedy et al. identify adversarial examples, implicitly raising the question of whether networks' internal computations match human-intelligible concepts — a precursor to the alignment motivation for circuits.

2020

Olah et al. ("Zoom In: An Introduction to Circuits") formally introduce the circuits framework, demonstrating curve detector circuits shared across four vision architectures and coining the universality conjecture.

2021

Elhage et al. (Anthropic) publish "A Mathematical Framework for Transformer Circuits," introducing the residual stream, QK/OV decomposition, and the concept of induction heads as the first fully mechanistic circuit analysis of a transformer.

2022

Wang et al. ("Interpretability in the Wild") reverse-engineer the indirect object identification (IOI) circuit in GPT-2 Medium — 26 attention heads mediating a syntactic rule — providing the first complete transformer circuit in a real language model.

2022

Meng et al. ("ROME: Locating and Editing Factual Associations in GPT") use causal tracing to show that factual associations are stored in specific MLP layers, enabling surgical model editing — circuits applied to safety-relevant editing.

2023

Bricken et al. (Anthropic) train sparse autoencoders on MLP activations of a one-layer transformer and recover thousands of interpretable monosemantic features, operationalizing superposition theory as a scalable tool.

2024

Lindsey et al. (Anthropic, "Scaling and Evaluating Sparse Autoencoders") train SAEs on Claude Sonnet's residual stream at scale, recovering features interpretable enough to steer model behavior, and McDougall et al. demonstrate automated circuit discovery via attribution patching at GPT-4 scale.

Why It Matters

Real-World Impact

Anthropic's Claude Safety ResearchMechanistic interpretability of circuits in Claude models is used to identify "deceptive alignment" precursors — checking whether the model's internal reasoning circuit matches its stated chain-of-thought, a direct alignment verification technique motivated by the "sycophancy circuit" hypothesis.

GPT-2 Indirect Object Identification (IOI)Wang et al. (2022) fully mapped a 26-head circuit in GPT-2 Medium that implements grammatical binding of indirect objects (e.g., "John gave Mary the book" → Mary is recipient). This circuit was used to test whether model editing via weight updates disrupts only the target behavior — a prerequisite for surgical RLHF fine-tuning.

ROME and MEMIT Model EditingLocating the factual recall circuit (mid-layer MLP modules acting as key-value stores) enabled Meng et al. to edit specific factual beliefs in GPT-J-6B without retraining, updating thousands of facts while preserving unrelated capabilities — directly relevant to correcting harmful beliefs in deployed models.

Sparse Autoencoder-Based Steering VectorsBricken et al.'s SAE features on Claude 3 Sonnet have been used to identify and suppress the "Assistant" token's role in syc