The Singular Value Decomposition

Every linear map distorts space in three ways. It can stretch, rotate, and possibly collapse dimensions. How can we measure this distortion in the cleanest possible way? The Singular Value Decomposition (SVD) answers this question. It finds two orthonormal bases — one for the domain, one for the codomain. Between these bases, the map acts as pure diagonal scaling. No other decomposition applies to every linear map between inner product spaces. This holds regardless of whether the map is square, invertible, or normal. The SVD matters because it isolates exactly the numerical invariants of a linear map. These invariants survive under orthonormal change of basis: the singular values. These numbers quantify how much the map stretches vectors. They measure stretching in the "best-fitting" orthogonal directions. They also generalize eigenvalues to maps that are not even square. Eigenvalues can be complex, negative, or simply nonexistent for a satisfactory basis. Singular values, by contrast, are always real and nonnegative. An orthonormal eigenbasis-like structure always exists for them. The construction proceeds through the positive operator $T^*T$ . Its eigenvalues are nonnegative real numbers. Its square roots are precisely the singular values of $T$ . This ties the SVD tightly to the theory of Positive Operators and Square Roots. It also connects closely to The Adjoint of a Linear Map. Both are prerequisites for the argument given here. The output is a clean geometric picture. $T$ sends an orthonormal basis of eigenvectors of $T^*T$ to orthogonal vectors. The lengths of these vectors are the singular values. This article builds the SVD from first principles. It defines singular values via $T^*T$ . It proves the existence of the full decomposition with matching orthonormal bases. It shows how to read the decomposition as a diagonal matrix representation. It closes by connecting the SVD to two of its major descendants. These are the Polar Decomposition and Least Squares Approximation.

Key takeaways

Singular values of $T: V \to W$ are the nonnegative square roots of the eigenvalues of the positive operator $T^*T$ on $V$ .
The SVD produces orthonormal bases $e_1,\dots,e_n$ of $V$ and $f_1,\dots,f_n$ of $\operatorname{range} T$ (extendable to all of $W$ ) with $Te_j = s_j f_j$ .
Every linear map $T:V\to W$ between finite-dimensional inner product spaces admits such a decomposition — no invertibility or squareness required.
In matrix language, $A = U\Sigma V^*$ with $U,V$ unitary/orthogonal and $\Sigma$ diagonal with nonnegative entries.
The number of nonzero singular values equals $\operatorname{rank} T$ ; the largest singular value equals the operator norm $\|T\|$ .
Key idea: diagonalize the positive operator $T^*T$ , take square roots of its eigenvalues, and transport the eigenbasis forward through $T$ to get an orthogonal target basis.

Where this lives today

The SVD underlies principal component analysis in statistics and machine learning, where the singular vectors of a data matrix reveal the directions of maximal variance. It gives the theoretically optimal low-rank matrix approximation (the Eckart–Young theorem), used in image compression, recommender systems, and model reduction. In numerical linear algebra, the SVD provides the most numerically stable way to compute the rank, condition number, and pseudoinverse of a matrix, and underlies the Least Squares Approximation solution to overdetermined linear systems via the Moore–Penrose pseudoinverse. In signal processing, it powers noise reduction and system identification. In quantum information theory, the analogous "Schmidt decomposition" of a bipartite quantum state is precisely an SVD of the state's coefficient matrix, and it quantifies entanglement. In cryptography and coding theory, spectral norms bounded via the largest singular value control error propagation and security parameters of certain lattice-based schemes. The SVD is also the computational engine behind the Polar Decomposition, writing any operator as an isometry composed with a positive operator, which in turn is used in continuum mechanics to separate rotation from stretching in deformation gradients.

Ask the tutor: How does truncating the SVD to its largest $k$ singular values give the provably best rank- $k$ approximation of a matrix?

V

Intuition

Think of $T$ as a machine that takes an input vector and produces an output vector, possibly changing its length, direction, and even the ambient dimension. The SVD says: however complicated $T$ looks in arbitrary coordinates, there is one specific orthonormal basis of the input space that $T$ treats as "principal axes." Feed $T$ a unit vector along one of these axes, and out comes a vector along a corresponding orthogonal axis in the output space, scaled by a factor $s_j \geq 0$ — the singular value. Geometrically, $T$ maps the unit sphere of $V$ to an ellipsoid in $W$ (possibly degenerate, if some $s_j = 0$ ). The singular values are exactly the semi-axis lengths of that ellipsoid, and the vectors $e_j$ are the directions in the domain that map onto those semi-axes. This is why the SVD works for every linear map: the image of a sphere under any linear map is always an ellipsoid (possibly flattened into a lower-dimensional one), and every ellipsoid has well-defined principal axes.

Figure

The unit ball in

V

maps to an ellipsoid in

W

with semi-axes

s_1, s_2

The key algebraic fact making this picture rigorous is that $T^*T$ is positive, hence orthogonally diagonalizable with nonnegative eigenvalues — a fact borrowed directly from the theory of positive operators. Taking square roots of those eigenvalues gives lengths, not just abstract scalars, and the adjoint relationship $T^*T$ (rather than $TT^*$ or some asymmetric combination) is exactly what forces the images $Te_j$ to be mutually orthogonal, as the proof below verifies.

V,W

T^*T \in \mathcal L(V)

Worked examples

Example 1: A non-square, asymmetric $3\times 2$ real matrix. Let $T: \mathbf R^2 \to \mathbf R^3$ be given, in standard bases, by the matrix

A = \begin{pmatrix} 2 & 0 \\ 0 & 0 \\ 1 & 1 \end{pmatrix}.

We compute $A^*A = A^TA$ :

A^TA = \begin{pmatrix} 2 & 0 & 1 \\ 0 & 0 & 1\end{pmatrix}\begin{pmatrix} 2 & 0 \\ 0 & 0 \\ 1 & 1\end{pmatrix} = \begin{pmatrix} 5 & 1 \\ 1 & 1 \end{pmatrix}.

This is a symmetric $2\times 2$ real matrix, hence self-adjoint on $\mathbf R^2$ with the standard inner product; it is positive since it equals $A^TA$ . Its characteristic polynomial is

\det(A^TA - \lambda I) = (5-\lambda)(1-\lambda) - 1 = \lambda^2 - 6\lambda + 4.

The eigenvalues are $\lambda = \dfrac{6 \pm \sqrt{36-16}}{2} = 3 \pm \sqrt5$ . Both are positive (since $\sqrt5 < 3$ ), consistent with positivity of $A^TA$ . The singular values of $T$ are

s_1 = \sqrt{3+\sqrt5}, \qquad s_2 = \sqrt{3-\sqrt5}.

Numerically, $\sqrt5 \approx 2.2361$ , so $s_1 \approx \sqrt{5.2361}\approx 2.2882$ and $s_2 \approx \sqrt{0.7639}\approx 0.8740$ . Eigenvectors of $A^TA$ . For $\lambda_1 = 3+\sqrt5$ : solve $(5-\lambda_1)x_1 + x_2 = 0$ , i.e. $x_2 = (\lambda_1-5)x_1 = (\sqrt5 - 2)x_1$ . Take $x_1 = 1$ , $x_2 = \sqrt5-2 \approx 0.2361$ . Normalizing: $\|(1,\sqrt5-2)\|^2 = 1 + (\sqrt5-2)^2 = 1 + 9 - 4\sqrt5 = 10-4\sqrt5 \approx 1.0557$ , so norm $\approx 1.0275$ . Thus

e_1 \approx (0.9732,\ 0.2298).

For $\lambda_2 = 3-\sqrt5$ : $x_2 = (\lambda_2-5)x_1 = (-\sqrt5-2)x_1$ ; take $x_1=1$ , $x_2 = -\sqrt5-2\approx -4.2361$ , normalize by $\sqrt{1+(\sqrt5+2)^2} = \sqrt{1+9+4\sqrt5} = \sqrt{10+4\sqrt5}\approx\sqrt{18.944}\approx 4.3524$ :

e_2 \approx (0.2298,\ -0.9732).

As a check, $e_1 \cdot e_2 \approx 0.9732(0.2298) + 0.2298(-0.9732) = 0$ , confirming orthogonality. Computing $f_j = \frac{1}{s_j}Ae_j$ .

Ae_1 \approx \begin{pmatrix}2(0.9732)\\0\\0.9732+0.2298\end{pmatrix} = \begin{pmatrix}1.9464\\0\\1.2030\end{pmatrix},\qquad \|Ae_1\| \approx \sqrt{3.7885+1.4472}=\sqrt{5.2357}\approx 2.2882 = s_1.\ \checkmark

f_1 \approx \frac{1}{2.2882}(1.9464,\,0,\,1.2030) \approx (0.8507,\ 0,\ 0.5258).

Similarly,

Ae_2 \approx \begin{pmatrix}2(0.2298)\\0\\0.2298-0.9732\end{pmatrix} = \begin{pmatrix}0.4596\\0\\-0.7434\end{pmatrix}, \qquad \|Ae_2\|\approx\sqrt{0.2113+0.5526}=\sqrt{0.7639}\approx0.8740 = s_2.\ \checkmark

f_2 \approx \frac{1}{0.8740}(0.4596,\,0,\,-0.7434) \approx (0.5259,\ 0,\ -0.8507).

Check orthogonality: $f_1\cdot f_2 \approx 0.8507(0.5259) + 0 + 0.5258(-0.8507) \approx 0.4475 - 0.4474 \approx 0$ . Good. Since $\operatorname{rank} A = 2$ (both singular values are nonzero, the columns of $A$ are visibly independent), we extend $f_1,f_2$ to an orthonormal basis of $\mathbf R^3$ by adjoining any unit vector $f_3$ orthogonal to both, e.g. via the cross product $f_1 \times f_2$ , normalized. This exhibits the full diagonal representation

Ae_1 = s_1f_1,\qquad Ae_2 = s_2f_2,

with $T$ represented as $\operatorname{diag}(s_1,s_2)$ (padded with a zero row) in the bases $(e_1,e_2)$ and $(f_1,f_2,f_3)$ .

Example 2: A singular $2\times 2$ complex matrix (rank deficient, illustrating a zero singular value). Let $T:\mathbf C^2\to\mathbf C^2$ be given by

A = \begin{pmatrix} 1 & i \\ 2 & 2i\end{pmatrix}.

Note the second row is $2$ times the first, so $\operatorname{rank} A = 1$ ; we expect exactly one nonzero singular value. Compute $A^* = \overline{A}^T = \begin{pmatrix}1 & 2\\ -i & -2i\end{pmatrix}$ , so

A^*A = \begin{pmatrix}1 & 2\\ -i & -2i\end{pmatrix}\begin{pmatrix}1 & i\\ 2 & 2i\end{pmatrix} = \begin{pmatrix}1+4 & i+4i\\ -i-4i & 1+4\end{pmatrix} = \begin{pmatrix}5 & 5i\\ -5i & 5\end{pmatrix}.

Check self-adjointness: $(A^*A)^* = \begin{pmatrix}5 & \overline{-5i}\\ \overline{5i} & 5\end{pmatrix} = \begin{pmatrix}5 & 5i\\-5i&5\end{pmatrix} = A^*A$ . Good, and it should be positive since it equals $A^*A$ . Eigenvalues: $\det(A^*A - \lambda I) = (5-\lambda)^2 - (5i)(-5i) = (5-\lambda)^2 - 25 = 0$ , so $(5-\lambda)^2 = 25$ , giving $5-\lambda = \pm5$ , i.e. $\lambda = 0$ or $\lambda = 10$ . Both nonnegative, as required. Singular values:

s_1 = \sqrt{10}, \qquad s_2 = 0.

Eigenvector for $\lambda_1=10$ : solve $(5-10)x_1 + 5ix_2 = 0 \Rightarrow -5x_1 + 5ix_2 = 0 \Rightarrow x_1 = ix_2$ . Take $x_2=1$ , $x_1 = i$ : vector $(i,1)$ , norm $\sqrt{|i|^2+1^2}=\sqrt2$ . So

e_1 = \frac{1}{\sqrt2}(i,1).

Eigenvector for $\lambda_2=0$ : solve $5x_1+5ix_2=0 \Rightarrow x_1 = -ix_2$ . Take $x_2=1,x_1=-i$ : vector $(-i,1)$ , norm $\sqrt2$ . So

e_2 = \frac{1}{\sqrt2}(-i,1).

Check orthonormality: $\langle e_1,e_2\rangle = \frac{1}{2}\left(i\overline{(-i)} + 1\cdot\overline{1}\right) = \frac12(i\cdot i + 1) = \frac12(-1+1)=0$ . Good (using the convention $\langle u,v\rangle = \sum u_k\overline{v_k}$ ). Compute $f_1 = \frac{1}{s_1}Ae_1$ :

Ae_1 = \frac{1}{\sqrt2}\begin{pmatrix}1&i\\2&2i\end{pmatrix}\begin{pmatrix}i\\1\end{pmatrix} = \frac{1}{\sqrt2}\begin{pmatrix}i\cdot1 + i\cdot1\\ 2i+2i\end{pmatrix}=\frac{1}{\sqrt2}\begin{pmatrix}2i\\4i\end{pmatrix} = \frac{1}{\sqrt2}(2i,4i).

Norm: $\left\|\frac{1}{\sqrt2}(2i,4i)\right\| = \frac{1}{\sqrt2}\sqrt{4+16} = \frac{\sqrt{20}}{\sqrt2} = \sqrt{10} = s_1.\ \checkmark$ So

f_1 = \frac{1}{\sqrt{10}}\cdot\frac{1}{\sqrt2}(2i,4i) = \frac{1}{\sqrt{20}}(2i,4i) = \frac{1}{\sqrt5}(i,2i).

Since $s_2 = 0$ , $Ae_2 = 0$ automatically (check: $Ae_2 = \frac{1}{\sqrt2}\begin{pmatrix}1&i\\2&2i\end{pmatrix}\begin{pmatrix}-i\\1\end{pmatrix} = \frac1{\sqrt2}\begin{pmatrix}-i\cdot1+i\\-2i+2i\end{pmatrix}=\frac1{\sqrt2}(0,0)=0$ , confirming), and we choose $f_2$ to be any unit vector orthogonal to $f_1$ in $\mathbf C^2$ , e.g.

f_2 = \frac{1}{\sqrt5}(2,-1),

since $\langle f_1,f_2\rangle = \frac{1}{\sqrt5}\cdot\frac{1}{\sqrt5}(i\cdot 2 + 2i\cdot(-1)) = \frac15(2i-2i)=0$ . This gives the SVD $Ae_1 = \sqrt{10}\,f_1$ , $Ae_2 = 0$ , matching $\operatorname{rank} A = 1$ .

T^*T

Scope, edge cases & common traps

Situation	What happens
$T$ is invertible (square, $\ker T = 0$ )	All $s_j > 0$ ; SVD coincides with writing $T$ through its polar decomposition $T = S\sqrt{T^*T}$ times an isometry.
$T$ is not injective	Some $s_j = 0$ ; corresponding $e_j \in \ker T$ , since $T^Te_j=0 \Rightarrow \\|Te_j\\|^2 = \langle T^Te_j,e_j\rangle = 0$ .
$\dim V \ne \dim W$	SVD still exists; the diagonal matrix is rectangular ( $m\times n$ ), padded with zero rows or the $f_j$ list is simply shorter than a full basis of $W$ .
$T$ already self-adjoint/positive	Singular values equal $\|\lambda_i\|$ where $\lambda_i$ are eigenvalues of $T$ ; $e_j, f_j$ coincide up to sign when $\lambda_j \geq 0$ .
Repeated singular value $s_j$	The eigenspace of $T^*T$ for $\lambda_j = s_j^2$ has dimension $>1$ ; the choice of orthonormal basis within that eigenspace (and correspondingly of $f_j$ 's) is not unique, but the decomposition still holds for any such choice.
Choosing $f_j$ when $s_j = 0$	These vectors are not determined by $T$ — any orthonormal completion works, since $Te_j = 0$ regardless of $f_j$ .

Common trapThe classic error: forgetting that singular vectors are not unique, and conflating

T^*T

with

TT^*

A common mistake is to assume the pair $(e_j, f_j)$ in an SVD is the unique such pair, and then to "prove" false uniqueness theorems by fixing an arbitrary such pair. In fact whenever a singular value is repeated, or is zero, there is real freedom in choosing eigenvectors and completions. Any correct statement about singular vectors must either restrict to simple (multiplicity-one, nonzero) singular values, or be phrased as "there exists a choice such that…", never "the unique $e_j$ with…". A second frequent error is silently swapping $T^*T$ (an operator on $V$ ) for $TT^*$ (an operator on $W$ ): both give the same nonzero eigenvalues (hence the same nonzero singular values), but their eigenvectors live in different spaces and play the roles of $e_j$ versus $f_j$ respectively — mixing them up produces vectors that fail the defining relation $Te_j = s_jf_j$ .

The mathematician behind it

Origins · 1 minEugenio Beltrami and how this was discovered

Eugenio Beltrami

1835–1900 · Italian differential geometer working on quadratic forms and bilinear forms

Beltrami is credited with an early form of the singular value decomposition for real square matrices, published in 1873, arising from his work on bilinear forms and their canonical reduction under orthogonal substitutions. Working independently, Camille Jordan (1874) and James Joseph Sylvester (1889) developed related canonical forms; the theory was later extended to complex matrices and to general linear operators between inner product spaces, with the operator-theoretic viewpoint (via $T^*T$ and the spectral theorem for positive operators) becoming standard in twentieth-century functional analysis and numerical linear algebra.

History

- 1873: Eugenio Beltrami derives a diagonalization of a bilinear form via orthogonal substitutions on both sides, an early real SVD.

- 1874–1889: Camille Jordan and James J. Sylvester independently develop related canonical reductions of bilinear/quadratic forms.

- 1907: Erhard Schmidt extends related ideas to integral operators, connecting the decomposition to what are now called Schmidt pairs in operator theory.

- 1936: Carl Eckart and Gale Young establish the low-rank approximation property of the SVD (best rank-

k

approximation in operator norm and Frobenius norm), foundational for statistics and numerical analysis.

- 1965: Gene Golub and William Kahan develop a numerically stable algorithm for computing the SVD, making it a practical computational tool.

Does every linear map have a singular value decomposition, even if it is not square or not invertible?

Yes. The existence theorem places no restriction on $\dim V$ versus $\dim W$ , nor does it require $T$ to be injective or surjective. The construction only uses that $T^*T$ is a positive operator on $V$ , which holds for every $T \in \mathcal L(V,W)$ with $V, W$ finite-dimensional inner product spaces. When $T$ is not injective, some singular values equal zero, and the corresponding eigenvectors of $T^*T$ span $\ker T$ . When $T$ is not surjective, the orthonormal list $f_1,\dots,f_n$ simply does not span all of $W$ , and must be extended arbitrarily to a full basis if a genuinely square diagonal matrix representation of $W$ 's basis is desired.

How does the SVD relate to eigenvalues of

T

when

T

is a square matrix?

In general, singular values and eigenvalues are different numbers computed from different operators: eigenvalues come from $T$ , singular values from $T^*T$ . They coincide (up to absolute value) exactly when $T$ is normal, i.e. $TT^*=T^*T$ , because normal operators admit an orthonormal eigenbasis with possibly complex eigenvalues $\lambda_j$ , and one can show the singular values equal $|\lambda_j|$ . For a general (non-normal) square matrix, the eigenvalues can be wildly different from the singular values; a classic example is a nilpotent matrix with all eigenvalues zero but nonzero singular values, since it moves some vectors a positive distance even though it has no long-term "stretching direction" fixed by an eigenvector.

Why are singular values always real and nonnegative, unlike eigenvalues?

Singular values are defined as square roots of eigenvalues of $T^*T$ , and $T^*T$ is always a positive operator, regardless of what $T$ looks like. Positivity means $\langle T^*Tv,v\rangle = \|Tv\|^2 \ge 0$ for every $v$ , which forces all eigenvalues of $T^*T$ to be real and nonnegative — this is a general fact about positive (in particular self-adjoint) operators proved via the spectral theorem. Taking square roots of nonnegative reals always yields nonnegative reals. This is precisely the mechanism that lets singular values exist and behave well even for maps like nilpotent operators or non-square matrices, where genuine eigenvalues may be complex, absent, or degenerate.

What is the connection between the SVD and the operator norm?

The largest singular value $s_1$ equals the operator norm $\|T\| = \max_{\|v\|=1}\|Tv\|$ , as proved directly from the Pythagorean identity applied to the orthogonal expansion $Tv = \sum_j s_j\langle v,e_j\rangle f_j$ . More generally, the $k$ -th singular value $s_k$ equals the operator norm of $T$ restricted to the orthogonal complement of the span of $e_1,\dots,e_{k-1}$ , giving a full hierarchy of "best-fit" norms. This is the theoretical basis for the Eckart–Young theorem: truncating the SVD to its top $k$ terms gives the best rank- $k$ approximation of $T$ in operator norm (and in Frobenius norm), a fact central to low-rank data compression.

Is the SVD unique?

The singular values themselves are unique, since they are determined as the (ordered) eigenvalues of $T^*T$ , and eigenvalues with multiplicity are basis-independent invariants of an operator. The orthonormal bases $(e_j)$ and $(f_j)$ , however, are generally not unique. Whenever a singular value has multiplicity greater than one, any orthonormal basis of the corresponding eigenspace of $T^*T$ works equally well, and the corresponding $f_j$ 's change correspondingly. Whenever a singular value is zero, the associated $f_j$ can be chosen arbitrarily (subject to orthonormality with the rest), since it plays no role in the formula for $Tv$ .

How does the SVD relate to the matrix factorization

A = U\Sigma V^*

often seen in linear algebra courses?

This is the same object phrased in matrix language. If $T:V\to W$ has matrix $A$ in some initial orthonormal bases, and $(e_j)$ , $(f_j)$ are as in the SVD, let $V_0$ be the (unitary or orthogonal) change-of-basis matrix whose columns are the coordinate vectors of $e_j$ , and $U$ the matrix whose columns are the coordinate vectors of $f_j$ (extended to a full basis of $W$ if needed). Then the diagonal representation of $T$ translates exactly into $A = U\Sigma V_0^*$ , where $\Sigma$ is the diagonal matrix of singular values. The operator-theoretic proof given here is basis-free, but choosing any orthonormal bases to write matrices recovers the familiar factorization.

Why is the SVD sometimes described as more fundamental than eigendecomposition?

Eigendecomposition (diagonalization by a single similarity transformation) requires $T$ to be a square operator with enough eigenvectors to form a basis — a strong and not always satisfied condition, even among self-adjoint-like operators over $\mathbf R$ if the operator isn't self-adjoint. Even when eigenvectors exist, the change-of-basis matrix need not be orthogonal, so geometric distances and angles are not preserved by the eigenbasis. The SVD, in contrast, exists for every linear map between finite-dimensional inner product spaces, uses genuinely orthonormal bases on both sides, and always yields real nonnegative diagonal entries — properties that make it the natural generalization of diagonalization to arbitrary linear maps.