Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are special scalar-vector pairs associated with a square matrix A, where multiplying the matrix by the vector produces the same result as scaling the vector by the scalar: Av = λv. They are foundational to modern AI/ML — powering dimensionality reduction (PCA), graph neural networks, spectral clustering, and understanding why deep networks train or fail to train at all. The remarkable fact: a symmetric matrix's eigenvectors are always mutually orthogonal, meaning nature "chooses" maximally independent directions to encode its transformations.

The core insight

A matrix encodes a linear transformation — rotations, stretches, shears — but hidden inside every such transformation are special directions that only get scaled, never rotated. These are the eigenvectors, and how much they get scaled is the eigenvalue. The "aha moment" is realizing that decomposing a matrix into its eigenvectors and eigenvalues is equivalent to finding the natural coordinate system in which the transformation becomes trivially simple: just multiply each axis by a scalar. Almost everything hard about a matrix becomes easy once you find that coordinate system.

How It Works

The Defining Equation

The entire theory starts from one deceptively simple equation. For a square matrix A ∈ ℝⁿˣⁿ, a nonzero vector v ∈ ℝⁿ is an eigenvector if:

1
Av=λv\quad \mathbf{A}\mathbf{v} = \lambda \mathbf{v}

Explanation: Multiplying the matrix <strong>A</strong> by the vector <strong>v</strong> is identical to stretching (or flipping) <strong>v</strong> by the scalar λ — the vector's direction is preserved, only its magnitude changes. Here, A is any square matrix, v is the eigenvector (a nonzero column vector), and λ (lambda) is the corresponding eigenvalue — a real or complex scalar. Note that v = 0 is explicitly excluded because it satisfies the equation trivially for any λ.

Finding Eigenvalues: The Characteristic Equation

Rearranging [1]:

2
(A−λI)v=0\quad (\mathbf{A} - \lambda \mathbf{I})\mathbf{v} = \mathbf{0}

Explanation: This says the matrix (<strong>A</strong> − λ<strong>I</strong>) maps the nonzero vector <strong>v</strong> to zero, which is only possible if that matrix is singular (non-invertible). For a non-trivial solution (v ≠ 0) to exist, (A − λI) must be singular — it must "collapse" some direction to zero. A matrix is singular if and only if its determinant is zero, giving the characteristic equation:

3
det⁡(A−λI)=0\quad \det(\mathbf{A} - \lambda \mathbf{I}) = 0

Explanation: Expanding this determinant produces a degree-n polynomial in λ (the characteristic polynomial), whose roots are exactly the n eigenvalues of <strong>A</strong>. For a 2×2 matrix, this yields a quadratic; for a 3×3, a cubic — and so on. Once eigenvalues are found, each eigenvector is obtained by solving the linear system (A − λᵢI)v = 0 via Gaussian elimination.

Eigendecomposition

When A has n linearly independent eigenvectors (it is diagonalizable), we can write the full eigendecomposition:

4
A=QΛQ−1\quad \mathbf{A} = \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^{-1}

Explanation: <strong>A</strong> is factored into a matrix of eigenvectors <strong>Q</strong>, a diagonal matrix <strong>Λ</strong> of eigenvalues, and the inverse of <strong>Q</strong> — meaning the transformation is a change of basis, a pure scaling, and then a change back. Here Q = [v₁ | v₂ | … | vₙ] (columns are eigenvectors), and Λ = diag(λ₁, λ₂, …, λₙ). For symmetric matrices (A = Aᵀ), the Spectral Theorem guarantees eigenvectors are orthonormal, so Q⁻¹ = Qᵀ, giving the cleaner form:

5
A=QΛQ⊤\quad \mathbf{A} = \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^{\top}

Explanation: For symmetric matrices, the change-of-basis matrix is also an orthogonal rotation matrix — the eigenvectors form a perfectly perpendicular coordinate system aligned with the matrix's natural "stretch directions."

PyTorch Implementation

```python import torch

# Define a symmetric matrix (e.g., a covariance matrix) A = torch.tensor([[4.0, 2.0], [2.0, 3.0]])

# Compute eigenvalues and eigenvectors

# torch.linalg.eigh is for symmetric/Hermitian matrices (guaranteed real eigenvalues) eigenvalues, eigenvectors = torch.linalg.eigh(A) print("Eigenvalues:", eigenvalues)

# tensor([1.5858, 5.4142]) print("Eigenvectors (columns):\n", eigenvectors)

# Each column is a unit eigenvector

# Verify: A @ v ≈ λ v v0 = eigenvectors[:, 0] lam0 = eigenvalues[0] print("Av:", A @ v0) print("λv:", lam0 v0)

# Should be equal (up to floating point)

# Reconstruct A from eigendecomposition Q = eigenvectors Lambda = torch.diag(eigenvalues) A_reconstructed = Q @ Lambda @ Q.T print("Reconstruction error:", torch.norm(A - A_reconstructed).item())

# ≈ 0.0

# For general (non-symmetric) matrices — may return complex eigenvalues B = torch.tensor([[0.0, -1.0], [1.0, 0.0]]) # 90° rotation matrix evals, evecs = torch.linalg.eig(B) print("Complex eigenvalues of rotation:", evals)

# tensor([0.+1.j, 0.-1.j]) — pure imaginary, as expected ```

researcher · David Hilbert | 1862 – 1943 | Spectral Theory of Operators

Hilbert was a German mathematician who dominated early 20th-century mathematics, reformulating geometry, logic, and analysis. He extended eigenvalue theory from finite matrices to infinite-dimensional function spaces, creating spectral theory — the foundation of quantum mechanics and modern functional analysis. Surprisingly, Hilbert himself declared that physics would never find a use for his spectral theory within his lifetime; Max Born had to inform him that Heisenberg's matrix mechanics was precisely that application, just six years later.

Figure
Geometric Interpretation of Eigenvectors
Geometric Action of a Matrix on Vectors Input Space u v₁ v₂ → A· Output Space Au ↙ rotated λ₁v₁ scaled ×λ₁ λ₂v₂ scaled ×λ₂ eigenvectors: unchanged direction generic vectors: direction changes
Figure
PCA via Eigendecomposition of Covariance Matrix
PCA Pipeline via Eigendecomposition Raw Data X ∈ ℝⁿˣᵈ Covariance Matrix Σ = XᵀX/(n-1) Eigendecomp Σ = QΛQᵀ λ₁ ≥ λ₂ ≥ … ≥ λd Top-k Eigenvecs Principal Components Qₖ ∈ ℝᵈˣᵏ Variance explained by component i: λᵢ / Σⱼλⱼ × 100% Project data: Z = X · Qₖ ∈ ℝⁿˣᵏ Eigenvectors = directions of max variance · Eigenvalues = amount of variance captured

Intuition

The Mental Model

Imagine you have a lump of dough and you press it with your palms — stretching it in one direction and squishing it in another. No matter how you oriented your hands, the dough has two special directions: one it stretches along (large eigenvalue), one it compresses along (small eigenvalue). Any other direction the dough moves in is just a combination of these two. The eigenvectors are those "principal stretch directions" baked into the material itself, and the eigenvalues tell you how violently it stretches or contracts in each. Now replace "dough" with your data covariance matrix: PCA is simply finding those natural stretch directions in your data cloud. A direction with a large eigenvalue is one your data varies along a lot; a small eigenvalue means the data is nearly flat in that direction — and you can safely discard it without losing much information. For neural networks, the same logic applies to the loss landscape's curvature: large eigenvalues of the Hessian mean the loss is very steep in that direction (dangerous for learning rate), while near-zero eigenvalues mean flat directions (slow to optimize).

Historical Development

Origins · 1 minhow this was discovered
History
1748
Leonhard Euler implicitly uses the concept while solving systems of differential equations with constant coefficients, recognizing special "exponential modes."
1801
Carl Friedrich Gauss applies what are essentially eigenvector methods in his work on quadratic forms and celestial orbit determination in Theoria Motus.
1855
Augustin-Louis Cauchy formalizes the characteristic polynomial for real symmetric matrices, proving that a symmetric matrix always has real eigenvalues.
1904
David Hilbert extends spectral theory to infinite-dimensional spaces (integral operators), laying the groundwork for quantum mechanics and functional analysis.
1926
Werner Heisenberg, Max Born, and Erwin Schrödinger independently show that quantum states are eigenvectors of energy operators (Hamiltonians), making eigenvalues the observed energy levels of atoms.
1936
Harold Hotelling formalizes Principal Component Analysis as an eigendecomposition problem in statistics, directly linking eigenvalues to explained variance.
1965
Gene Golub and William Kahan publish the SVD algorithm for numerical computation, making large-scale eigendecompositions practically tractable and founding numerical linear algebra.
1998
Bernhard Schölkopf, Alex Smola, and Klaus-Robert Müller introduce Kernel PCA, extending eigendecomposition to nonlinear feature spaces via the kernel trick.
2003–2006
Mikhail Belkin, Partha Niyogi, and collaborators develop Laplacian Eigenmaps and Spectral Graph Theory, directly enabling modern graph neural network theory and spectral clustering.

Why It Matters

Real-World Impact
PCA in Data Preprocessing (scikit-learn, everywhere)PCA computes the eigendecomposition of the data's covariance matrix; the top-k eigenvectors become the new feature axes. Used ubiquitously to reduce input dimensionality from thousands of features to dozens while preserving the most variance — cutting training time and reducing overfitting.
Google PageRankThe PageRank vector is the dominant eigenvector (eigenvalue = 1) of the web's link transition matrix. Google's original search ranking was literally an eigenvalue problem on a ~30-billion-node matrix, solved iteratively via the power method.
Spectral Graph Neural Networks (GCN, ChebNet)Graph Convolutional Networks define convolution in the spectral domain using the eigenvectors of the graph Laplacian. The eigenvalues measure graph frequency — low eigenvalues correspond to smooth signals across the graph, enabling spectral filtering of node features in molecule property prediction and social network analysis.
Hessian Analysis in Neural Network OptimizationResearchers at OpenAI and DeepMind use eigenvalue spectra of the Hessian matrix of the loss to diagnose training instability. A large maximum eigenvalue (spectral radius) imposes an upper bound on safe learning rates: `lr < 2 / λ_max`. Tools like PyHessian compute this for production models.
Diffusion Models — Noise Schedule DesignThe covariance matrix of the noise process in Denoising Diffusion Probabilistic Models is analyzed via eigenvalues to ensure the diffusion forward process correctly destroys signal at calibrated rates across data dimensions, preventing anisotropic artifacts.
Attention Head Analysis in TransformersResearch on BERT and GPT-family models uses eigendecomposition of the attention weight matrices to identify redundant attention heads — heads whose attention maps are nearly identical correspond to similar leading eigenvectors, enabling structured pruning without accuracy loss.
Reinforcement Learning — Value Function ApproximationSuccessor representation methods in RL use the eigenvectors of the state transition matrix as basis functions for value function approximation, provably minimizing the number of basis functions needed for a given approximation error.

Prerequisites

What This Leads To

See Also

Singular Value Decomposition (SVD) · Principal Component Analysis (PCA) · Matrix Factorization · Spectral Clustering · Linear Algebra for Machine Learning

Before this