Invariant Subspaces

Invariant subspaces are not a museum piece — they are the backbone of how linear operators are analyzed across mathematics and its applications.

• Diagonalization and triangularization: a basis of eigenvectors is precisely a basis built from a chain of $1$-dimensional invariant subspaces, and Schur's triangularization theorem builds an entire increasing chain (a "flag") of invariant subspaces to put any complex matrix into upper-triangular form.
• Markov chains and stochastic processes: the invariant subspace (in fact, invariant vector) fixed by a transition matrix is exactly the stationary distribution of the chain — a $1$-dimensional invariant subspace for eigenvalue $1$.
• Quantum mechanics: observables are modeled by self-adjoint operators on Hilbert spaces, and invariant subspaces correspond to conserved quantities and superselection sectors; the eigenspaces of the Hamiltonian are invariant subspaces carrying definite energy.
• Control theory: the controllable and unobservable subspaces of a linear dynamical system are invariant subspaces of the system matrix, and their structure determines whether a system can be steered or must be decomposed into controllable/uncontrollable parts.
• Numerical linear algebra: algorithms like the QR algorithm and Krylov subspace methods iteratively build up approximate invariant subspaces to compute eigenvalues of very large matrices without full diagonalization.
• Representation theory: an invariant subspace under a group's linear action is called a subrepresentation, and Schur's Lemma (irreducibility precisely means no nontrivial invariant subspaces) is the representation-theoretic descendant of this article's core idea.
• Cryptography and coding theory: linear feedback shift registers and certain stream ciphers are analyzed via invariant subspaces of the underlying linear recurrence operator, controlling period length and structure of the output sequence.

Ask the tutor: How do chains of invariant subspaces get combined to prove every complex matrix can be triangularized?

Think of $T$ as a machine that takes vectors in $V$ and moves them somewhere else in $V$. A subspace $U$ is invariant if the machine, when fed only vectors from $U$, never spits out anything that leaves $U$ — $T$ takes $U$ and folds it back into itself, possibly stretching, rotating, or collapsing vectors within $U$, but never launching them into the rest of $V$. The cleanest invariant subspaces come from eigenvectors. If $Tv = \lambda v$ for some scalar $\lambda$ and nonzero $v$, then the line (or one-dimensional subspace) $U = \operatorname{span}(v)$ is invariant: any vector in $U$ has the form $cv$, and $T(cv) = c\lambda v \in U$. Geometrically, $T$ acts on this line by pure scaling — it cannot rotate the line into some other direction, because it is forced to send $U$ back into $U$. A useful mental picture in $\mathbb{R}^2$: take a linear map that stretches along one axis and compresses along another, fixing both axes as lines. Each axis is a $1$-dimensional invariant subspace, and the whole map is diagonal with respect to this pair of invariant lines — this is exactly the geometric content of diagonalizability. Contrast this with a rotation by $90°$ in $\mathbb{R}^2$: no line through the origin is sent back into itself (every line gets rotated to a different line), so this operator, viewed as acting on $\mathbb{R}^2$, has no nontrivial invariant subspaces at all. This single example is the cleanest illustration of why real and complex scalars behave so differently for this theory.

Example 1 (a genuinely asymmetric $3\times 3$ matrix and an invariant plane). Let $T : \mathbb{R}^3 \to \mathbb{R}^3$ be given in the standard basis by the matrix

Consider $U = \operatorname{span}(e_1, e_3)$, the $x z$-plane, where $e_1 = (1,0,0)^T$, $e_3 = (0,0,1)^T$. We check invariance directly by computing $T$ on a basis of $U$:

Both basis vectors of $U$ map back into $U$, and since a general element of $U$ is $c_1 e_1 + c_3 e_3$, linearity gives

So $U$ is indeed $T$-invariant — note that this hinges entirely on the fact that the $(2,1)$ and $(2,3)$ entries of $A$ (the coefficients that would send $e_2$-multiples into the plane, or that would push $e_1, e_3$ into the $e_2$ direction) happen to vanish. In matrix language: relative to the ordered basis $(e_1, e_3, e_2)$, the matrix of $T$ becomes block upper triangular,

with the top-left $2\times 2$ block acting on the invariant plane $U$. This is the mechanism by which invariant subspaces produce simplified (block-triangular) matrix representations — precisely the first step toward Triangularization of Operators. We can also verify $U$ is invariant via eigenvectors, cross-checking Example 1 against the eigenvector theorem. The restriction $T|_U$ has matrix $\begin{pmatrix} 2 & 0 \\ 1 & 3\end{pmatrix}$ in the basis $(e_1, e_3)$ (reading off the coefficients computed above: $T e_1 = 2e_1 + 1\cdot e_3$, $Te_3 = 0 \cdot e_1 + 3 e_3$). Its eigenvalues solve $(2-\lambda)(3-\lambda) = 0$, giving $\lambda = 2, 3$ — both of which are also eigenvalues of $A$ itself, since an eigenvector of $T|_U$, sitting inside $U \subseteq V$, is automatically an eigenvector of $T$ on $V$. Direct check: solving $(A - 2I)v = 0$,

row-reducing gives $v_2 = 0$ and $v_1 + v_3 = 0$, so $v = (1, 0, -1)^T$ up to scaling — and indeed this lies in $U = \operatorname{span}(e_1,e_3)$. This matches the block-triangular structure exactly.

Example 2 (a rotation-plus-scaling in $\mathbb{R}^3$, invariant subspaces found via complexification-style reasoning kept entirely real). Let $T: \mathbb{R}^3 \to \mathbb{R}^3$ have matrix

Here the top-left $2\times 2$ block $\begin{pmatrix}0 & -2\\ 1 & 0\end{pmatrix}$ acts on $\operatorname{span}(e_1,e_2)$ as a rotation-like map with no real eigenvalues (its characteristic polynomial is $\lambda^2 + 2$, whose roots $\pm i\sqrt{2}$ are not real), while $e_3$ is a genuine eigenvector with eigenvalue $5$. First, $U_1 = \operatorname{span}(e_1, e_2)$ is $T$-invariant: $Be_1 = (0,1,0)^T = e_2 \in U_1$ and $Be_2 = (-2,0,0)^T = -2e_1 \in U_1$. But — and this is the point of this example — $U_1$ contains no $1$-dimensional invariant subspace of its own. Indeed, suppose $\operatorname{span}(v) \subseteq U_1$ were $T$-invariant with $v = (a,b,0)^T \neq 0$. By the eigenvector correspondence theorem, $v$ would need to be a real eigenvector of $T|_{U_1}$, i.e. satisfy $\begin{pmatrix}0&-2\\1&0\end{pmatrix}\begin{pmatrix}a\\b\end{pmatrix} = \lambda \begin{pmatrix}a\\b\end{pmatrix}$ for real $\lambda$. This forces $\lambda^2 = -2$ (compute: $-2b = \lambda a$, $a = \lambda b$, so $-2b = \lambda^2 b$; if $b \ne 0$ then $\lambda^2 = -2$, impossible for real $\lambda$; if $b = 0$ then $a = \lambda b = 0$, contradicting $v \neq 0$). So no real $\lambda$ works, and $U_1$ has no nontrivial invariant lines — this is a genuine real-linear-algebra phenomenon (compare with the rotation example in the Intuition section). Second, $U_2 = \operatorname{span}(e_3)$ is $T$-invariant since $Be_3 = (0,0,5)^T = 5e_3$, matching the eigenvector correspondence exactly with eigenvalue $\lambda = 5$. So $T$ on $\mathbb{R}^3$ has exactly the nontrivial invariant subspaces $U_1$ (2-dimensional) and $U_2$ (1-dimensional), plus any subspace built from unions — but note $U_1 + U_2 = \mathbb{R}^3$ is the whole space (trivial) and $U_1 \cap U_2 = \{0\}$ (trivial); there is no further nontrivial invariant subspace strictly between dimensions $1$ and $2$ other than $U_1$ itself. This illustrates that the number and dimension of invariant subspaces genuinely depends on the field: had we instead viewed $B$ as acting on $\mathbb{C}^3$, the $2\times 2$ block would contribute two new $1$-dimensional complex invariant subspaces (spanned by complex eigenvectors for $\lambda = \pm i\sqrt 2$), matching the guarantee of the Existence Theorem over $\mathbb{C}$.

The modern finite-dimensional treatment of invariant subspaces, phrased purely in terms of vector spaces and linear operators rather than matrices, crystallized through the abstract linear algebra tradition of the early-to-mid 20th century, building on ideas of eigenvalue theory going back to Cauchy and Jacobi in the 19th century, and formalized in the operator-theoretic language associated with Hilbert and later von Neumann.