Neural Tangent Kernel Theory

The Neural Tangent Kernel (NTK) is a kernel function that describes the training dynamics of infinitely wide neural networks under gradient descent, showing that such networks evolve as linear models in function space governed by a fixed kernel determined at initialization. It matters because it provides the first rigorous theoretical framework explaining why overparameterized networks trained with gradient descent can generalize well, directly linking deep learning to classical kernel methods. Remarkably, the NTK predicts that an infinitely wide neural network never escapes its linear approximation during training — meaning all the expressive power attributed to "nonlinear feature learning" is, in this limit, an illusion.

The core insight

When a neural network is extremely wide, the parameters move so little during gradient descent that the network behaves exactly like kernel regression with a specific kernel — the NTK — that is determined entirely by the network's architecture and initialization distribution, not by the data. This means training dynamics become analytically tractable: loss decreases exponentially, and the trained function is the minimum-norm interpolant in the NTK's reproducing kernel Hilbert space. The "aha moment" is that infinite width buys you linearity — the network's Jacobian freezes at its initial value, and you're really just doing lazy training in a high-dimensional tangent space. This resolves the apparent paradox of overparameterized models that interpolate training data yet still generalize.

Common trapCommon Misconception

The most common mistake is assuming NTK theory applies to the finite-width networks actually used in practice. Practitioners read that "neural networks are kernel methods" and conclude that NTK explains the success of ResNet-50 or GPT-4 — it does not. The NTK regime requires width tending to infinity before any other limit is taken, and empirical work (e.g., Chizat et al., 2019; Fort et al., 2020) demonstrates that practical networks operate in the "rich" or "feature-learning" regime where the kernel changes substantially during training. The correct mental model is that NTK is a baseline and a proof-of-concept for trainability and generalization theory, not a description of what real networks do — finite-width corrections, mean-field theory, and the maximal update parameterization (muP) are needed to capture feature learning.

How It Works

The NTK Definition

Consider a neural network $f_\theta : \mathbb{R}^d \to \mathbb{R}$ parameterized by $\theta \in \mathbb{R}^P$ . During gradient flow on a loss $\mathcal{L}(\theta)$ , the parameter trajectory satisfies $\dot{\theta} = -\nabla_\theta \mathcal{L}$ . The function output evolves as:

\quad \dot{f}_\theta(x) = -\sum_{\mu=1}^{n} \Theta_\theta(x, x_\mu)\, \frac{\partial \mathcal{L}}{\partial f_\theta(x_\mu)}

Explanation: The rate of change of the network's prediction at any input $x$ is a weighted sum of loss gradients at training points, mediated by the NTK — predicting each point is influenced by how "similar" it is to training data under the kernel. Here, the Neural Tangent Kernel is defined as the inner product of Jacobians:

\quad \Theta_\theta(x, x') = \left\langle \frac{\partial f_\theta(x)}{\partial \theta},\, \frac{\partial f_\theta(x')}{\partial \theta} \right\rangle = \sum_{p=1}^{P} \frac{\partial f_\theta(x)}{\partial \theta_p}\frac{\partial f_\theta(x')}{\partial \theta_p}

Explanation: The NTK measures how correlated the network's responses to inputs $x$ and $x'$ are through the shared effect of each parameter — two inputs with similar gradient directions are "close" in NTK space.

Infinite-Width Limit and Convergence

Jacot, Gabriel, and Hongler (2018) proved that as width $n \to \infty$ (under NTK parameterization, where layer $l$ weights are initialized as $W^{(l)} \sim \mathcal{N}(0, 1)$ and scaled by $1/\sqrt{n_l}$ ), the kernel converges almost surely to a deterministic limiting kernel:

\quad \Theta_\theta^{(n)}(x, x') \xrightarrow{n \to \infty} \Theta^*(x, x') \quad \text{a.s.}

Explanation: The randomness from initialization washes out as width grows, leaving a fixed deterministic kernel that depends only on architecture, depth, and activation function — not on the specific random seed. The NTK admits a recursive formula for an $L$ -layer fully-connected network. Defining $\Sigma^{(0)}(x,x') = x^\top x' / d$ and $\dot{\Sigma}^{(h)}$ as the arc-cosine kernel derivative:

\quad \Theta^*(x,x') = \sum_{h=1}^{L} \left(\prod_{h'=h+1}^{L} \dot{\Sigma}^{(h')}(x,x')\right) \Sigma^{(h-1)}(x,x')

Explanation: The NTK decomposes layer-by-layer, summing contributions from each layer's parameters weighted by the product of "gradient propagation factors" through all subsequent layers — formalizing how depth affects the kernel.

Training Dynamics as Kernel Regression

Under gradient flow with MSE loss $\mathcal{L} = \frac{1}{2}\|f_\theta(X) - y\|^2$ , and assuming the NTK remains constant (which holds exactly at infinite width), the residual vector $u(t) = f_{\theta(t)}(X) - y$ evolves as:

\quad u(t) = e^{-\Theta^* t}\, u(0)

Explanation: The training error decays exponentially at a rate determined by the eigenvalues of the NTK Gram matrix $\Theta^*(X,X) \in \mathbb{R}^{n \times n}$ , so convergence speed is controlled entirely by the kernel's spectral structure on the training data. The trained predictor converges to $f^* = \Theta^*(x, X)[\Theta^*(X,X)]^{-1} y$ , which is exactly the kernel regression solution — the minimum RKHS-norm interpolant, connecting NTK theory directly to classical Reproducing Kernel Hilbert Spaces.

researcher · Arthur Jacot | b. 1991 | Originator of the Neural Tangent Kernel formalism

Jacot completed his PhD at EPFL under Clément Hongler, where he developed the NTK framework as part of a broader study of infinite-width limits of neural networks. His 2018 NeurIPS paper "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" (co-authored with Franck Gabriel and Clément Hongler) introduced the NTK definition, proved its deterministic convergence, and showed that infinite-width networks trained by gradient descent converge to kernel regression — establishing NTK as the foundation of modern overparameterization theory. Surprisingly, Jacot originally approached the problem from a mathematical physics background studying random matrices and Gaussian processes, not from an ML engineering perspective.

researcher · Jaehoon Lee | b. 1988 | Empirical validation and GP correspondence

Lee, a researcher at Google Brain, provided crucial empirical and theoretical bridges between NTK theory and practice, notably in "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent" (2019, NeurIPS) co-authored with Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Sohl-Dickstein, and Pennington. His work demonstrated that the NTK linearization accurately predicts the training trajectories of very wide finite networks and showed that the infinite-width neural network prior corresponds exactly to a Gaussian process with a specific covariance kernel. Lee's group also developed the `neural-tangents` JAX library, making NTK computations accessible to the research community.

Figure

NTK Training Dynamics and Kernel Structure

Figure

NTK Parameterization vs. Standard Parameterization

Intuition

The Mental Model

Imagine you're navigating a hilly landscape wearing giant stilts that are so tall, each step you take moves you only a millimeter in the actual landscape. The contours of the landscape (the loss surface) look curved and complex from above, but because your steps are infinitesimally tiny relative to the landscape's scale, you're effectively moving in a straight line through the surrounding space. The landscape's local slope at your starting point completely determines where you end up — you never "discover" the distant valleys or ridges. This is exactly what NTK theory says about infinitely wide networks: the network is so large that gradient descent takes infinitesimally small steps in parameter space, meaning the function space trajectory is perfectly linear and entirely determined by the initial Jacobian. The kernel is the "shape of the slope" at your starting point, encoded as a similarity function between training examples. More intuitively: an infinitely wide network's neurons are so numerous that they barely need to change individually — the collective tiny nudges produce the right function, just like how a massive committee of slightly-biased voters can produce a precise aggregate outcome even though no individual changes their opinion much.

Historical Development

Origins · 1 minhow this was discovered

History

1986

Rumelhart, Hinton, Williams demonstrate backpropagation, enabling gradient-based training of multilayer networks — the foundational algorithm NTK theory later analyzes.

1990

Neal (Robert Neal's PhD thesis, 1994; conceptual work earlier) establishes that single-hidden-layer networks with random weights converge to Gaussian processes as width → ∞, foreshadowing the NTK's deterministic limit.

1995

Bartlett and others develop generalization bounds for neural networks via VC dimension, motivating the need for a trainability theory independent of VC analysis.

2017

Daniely, Frostig, and Singer (COLT 2016/2017) introduce the concept of "conjugate kernel" and prove that SGD on wide networks converges, a precursor to the NTK formalism.

2018

Jacot, Gabriel, Hongler publish "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" at NeurIPS 2018, formally introducing the NTK, proving its deterministic infinite-width limit, and connecting gradient descent training of neural networks to kernel regression.

2019

Lee et al. (Google Brain) publish "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent" (NeurIPS 2019), providing empirical validation of NTK predictions and proving the GP-NTK correspondence for deep networks.

2019

Chizat, Oyallon, and Bach publish "On Lazy Training in Differentiable Programming," formalizing the "lazy training" concept and showing that NTK regime is an artifact of large output scaling — clarifying when NTK does and does not apply.

2020

Yang (Greg Yang, Microsoft Research) develops the Tensor Programs framework (arXiv 2020–2022), generalizing NTK to any architecture including Transformers, attention, and layer norm, and introducing the Maximal Update Parameterization (μP) as the correct feature-learning alternative.

2022

Fort, Dziugaite, Paul, Kharaghani, Roy, Ganguli publish large-scale empirical studies showing that practical networks (ResNets, ViTs) operate firmly outside the NTK regime, with kernel alignment evolving substantially — motivating mean-field theory and beyond-NTK approaches as