Hilbert Spaces and Inner Products

A Hilbert space is a complete inner product space — a vector space equipped with a notion of length and angle (the inner product) in which every Cauchy sequence converges. Key result: such spaces possess orthonormal bases and the geometry of projection (the Projection Theorem), generalizing finite-dimensional Euclidean geometry to infinite dimensions. Applies in quantum mechanics, Fourier analysis, signal processing, and PDE theory. Named for David Hilbert (1862–1943); abstracted by von Neumann (1929).

A Hilbert space $H$ is a real or complex vector space carrying an inner product $⟨ \cdot, \cdot ⟩$ that induces a norm $∥ x ∥ = ⟨ x, x ⟩$ under which $H$ is complete (a Banach space whose norm comes from an inner product). The inner product encodes both length and angle, so concepts like orthogonality, projection, and the Pythagorean theorem extend verbatim to infinite dimensions. Hilbert spaces matter because they are the natural setting where linear algebra and analysis fuse: one can do calculus and geometry simultaneously, which is exactly what physics (quantum states) and analysis (Fourier series, $L^{2}$ functions) demand. The surprising fact is that any separable infinite-dimensional Hilbert space is isometrically isomorphic to the single concrete space $ℓ^{2}$ of square-summable sequences — there is essentially only one such space up to relabeling.

The Core Insight

The inner product turns an abstract vector space into a geometric object: you can measure how much one vector points along another. Completeness then guarantees that limits exist where geometry says they should. Together these give the single most powerful tool — orthogonal projection — letting you decompose any vector into a part inside a subspace and a part perpendicular to it, even when the subspace is infinite-dimensional.

Formal Statement

Inner Product

Let $H$ be a vector space over the field $F$ (either $R$ or $C$ ). An inner product is a map $⟨ \cdot, \cdot ⟩ : H \times H \to F$ satisfying, for all $x, y, z \in H$ and $α \in F$ :

Conjugate symmetry: $⟨ x, y ⟩ = \overline{⟨ y, x ⟩}$
Linearity in the first argument: $⟨ α x + z, y ⟩ = α ⟨ x, y ⟩ + ⟨ z, y ⟩$
Positive-definiteness: $⟨ x, x ⟩ \geq 0$ , with equality iff $x = 0$

Here $\overline{(\cdot)}$ denotes complex conjugation (trivial in the real case). The induced norm is $∥ x ∥ = ⟨ x, x ⟩$ , and the distance is $d (x, y) = ∥ x - y ∥$ .

∥ x ∥ = ⟨ x, x ⟩, d (x, y) = ∥ x - y ∥

⟳positive-definiteness guarantees

⟨ x, x ⟩ \geq 0

, so the square root is well-defined and real.

Convention on linearity

Mathematicians often take linearity in the first argument (physicists, via Dirac notation, take it in the second). The two differ only by which slot carries the conjugate. Throughout this article linearity is in the first slot, so $⟨ x, α y ⟩ = \overline{α} ⟨ x, y ⟩$ (conjugate-linearity in the second slot).

Hilbert Space

A Hilbert space is an inner product space $(H, ⟨ \cdot, \cdot ⟩)$ that is complete with respect to the metric $d (x, y) = ∥ x - y ∥$ : every Cauchy sequence $(x_{n})$ in $H$ converges to a limit in $H$ . An incomplete inner product space is called a pre-Hilbert space; its completion is a Hilbert space.

Cauchy–Schwarz Inequality

For all $x, y$ in an inner product space,

∣ ⟨ x, y ⟩ ∣ \leq ∥ x ∥ ∥ y ∥,

with equality if and only if

x

and

y

are linearly dependent.

⟳minimize

∥ x - t y ∥^{2} \geq 0

over the scalar

t

; the optimal

t

yields the inequality.

Parallelogram Law characterizes inner-product norms

A norm $∥ \cdot ∥$ on a vector space arises from an inner product if and only if it satisfies

∥ x + y ∥^{2} + ∥ x - y ∥^{2} = 2∥ x ∥^{2} + 2∥ y ∥^{2} .

When it does, the inner product is recovered by the polarization identity (complex case):

⟨ x, y ⟩ = \frac{1}{4} k = 0 \sum 3 i^{k} ∥ x + i^{k} y ∥^{2} .

⟳expand each

∥ x \pm y ∥^{2} = ⟨ x \pm y, x \pm y ⟩

and add; polarization comes from forming the same combination of the four norm expansions.

Projection Theorem

Let $M$ be a closed subspace of a Hilbert space $H$ . Then every $x \in H$ has a unique decomposition

x = m + n, m \in M, n \in M^{⊥},

where

M^{⊥} = {y \in H : ⟨ y, m ⟩ = 0 \forall m \in M}

. The vector

m = P_{M} x

is the unique closest point of

M

x

, and

H = M \oplus M^{⊥}

⟳minimize

∥ x - m ∥

over

m \in M

; completeness plus the parallelogram law makes the minimizing sequence Cauchy, hence convergent, giving the orthogonal component.

Why It Holds / Derivation

Projection Theorem

Step 1 — Set up the distance. Let

δ = in f_{m \in M} ∥ x - m ∥

. Choose a sequence

m_{k} \in M

with

∥ x - m_{k} ∥ \to δ

. Step 2 — Show the sequence is Cauchy. Apply the parallelogram law [3] to the vectors

x - m_{j}

and

x - m_{k}

∥ m_{j} - m_{k} ∥^{2} = 2∥ x - m_{j} ∥^{2} + 2∥ x - m_{k} ∥^{2} - 4 x - \frac{m _{j} + m _{k}}{2}^{2} .

⟳substitute

u = x - m_{j}

v = x - m_{k}

into [3]; note

u - v = m_{k} - m_{j}

and

u + v = 2 x - (m_{j} + m_{k})

Since $\frac{m _{j} + m _{k}}{2} \in M$ (subspace), the last term is $\geq δ^{2}$ . As $j, k \to \infty$ the first two terms $\to 2 δ^{2} + 2 δ^{2}$ , so $∥ m_{j} - m_{k} ∥^{2} \to 0$ . Hence $(m_{k})$ is Cauchy.

Step 3 — Use completeness.

H

complete and

M

closed imply

m_{k} \to m \in M

with

∥ x - m ∥ = δ

. Step 4 — Orthogonality. Set

n = x - m

. For any

w \in M

and scalar

t

∥ n - tw ∥^{2} \geq δ^{2} = ∥ n ∥^{2}

. Expanding:

- 2 Re (\overset{ˉ}{t} ⟨ n, w ⟩) + ∣ t ∣^{2} ∥ w ∥^{2} \geq 0 \forall t .

⟳

∥ n - tw ∥^{2} = ∥ n ∥^{2} - 2 Re (\overset{ˉ}{t} ⟨ n, w ⟩) + ∣ t ∣^{2} ∥ w ∥^{2}

; subtract

∥ n ∥^{2}

Choosing $t = s ⟨ n, w ⟩$ for small real $s > 0$ forces $⟨ n, w ⟩ = 0$ . Thus $n \in M^{⊥}$ .

Step 5 — Uniqueness. If

x = m_{1} + n_{1} = m_{2} + n_{2}

with

m_{i} \in M

n_{i} \in M^{⊥}

, then

m_{1} - m_{2} = n_{2} - n_{1} \in M \cap M^{⊥}

. But any

v \in M \cap M^{⊥}

satisfies

⟨ v, v ⟩ = 0

, so

v = 0

■

Riesz Representation Theorem

Every continuous linear functional $f : H \to F$ on a Hilbert space is given by inner product against a unique vector $y_{f} \in H$ :

f (x) = ⟨ x, y_{f} ⟩ \forall x \in H, ∥ f ∥ = ∥ y_{f} ∥.

⟳if

f = 0

take

y_{f} = 0

; otherwise

ker f

is closed with one-dimensional complement spanned by some

z \in (ker f)^{⊥}

, and

y_{f} = \frac{f ( z )}{∥ z ∥ ^{2}} z

John von Neumann | 1903–1957

Hilbert spaces are named for Hilbert, but their axiomatic definition is due to John von Neumann, who in his 1929–1932 work on the mathematical foundations of quantum mechanics isolated exactly the properties needed: a complex inner product space that is complete and (in his treatment) separable. He recognized that the states of a quantum system are unit vectors in such a space and that observables are self-adjoint operators on it. This unified Heisenberg's matrix mechanics ( $ℓ^{2}$ ) and Schrödinger's wave mechanics ( $L^{2}$ ) by showing both are the same Hilbert space in disguise — a striking confirmation of Riesz–Fischer Theorem in physics.

Building Intuition

Think of a Hilbert space as infinite-dimensional Euclidean space that never has holes. The inner product gives you a protractor and ruler: $⟨ x, y ⟩ = 0$ means perpendicular, $⟨ x, y ⟩ / (∥ x ∥∥ y ∥)$ is the cosine of the angle. The norm is straight-line length. Completeness is the promise that if a sequence of vectors looks like it should converge (terms bunch arbitrarily close together), then it genuinely converges to a vector still inside the space — no point is missing.

The single most useful mental picture is the drop-a-perpendicular operation: to approximate a function by, say, a trigonometric polynomial, you orthogonally project it onto the subspace of such polynomials. The Fourier coefficients are literally the components of that projection. "Best approximation" is geometric, not just analytic.

Orthogonal projection of x onto a closed subspace M: the residual n = x − P_M x is perpendicular to M

Worked Examples

:::example[Fourier coefficient as a projection in $L^{2} [- π, π]$ ] Given: Work in $H = L^{2} [- π, π]$ with $⟨ f, g ⟩ = \int_{- π}^{π} f (t) \overline{g (t)} d t$ . The functions $e_{n} (t) = \frac{1}{2 π} e^{in t}$ , $n \in Z$ , form an orthonormal set. Find the best $L^{2}$ approximation of $f (t) = t$ by the single mode $e_{1}$ , and its coefficient.

Step 1 — Orthonormality check.

⟨ e_{n}, e_{m} ⟩ = \frac{1}{2 π} \int_{- π}^{π} e^{i (n - m) t} d t = δ_{nm}

. ✓ Step 2 — Coefficient = projection onto $e_{1}$ .

c_{1} = ⟨ f, e_{1} ⟩ = \frac{1}{2 π} \int_{- π}^{π} t e^{- i t} d t .

Step 3 — Evaluate the integral. Integrate by parts:

\int t e^{- i t} d t = (i t + 1) e^{- i t}

. Evaluate from

- π

π

\int_{- π}^{π} t e^{- i t} d t = (iπ + 1) e^{- iπ} - (- iπ + 1) e^{iπ} = - (iπ + 1) - (- (- iπ + 1)) = - 2 π i .

c_{1} = \frac{- 2 π i}{2 π} = - i 2 π

. Step 4 — Best approximation. The projection is

c_{1} e_{1} = - i 2 π \cdot \frac{1}{2 π} e^{i t} = - i e^{i t}

. Answer:

c_{1} = - i 2 π, P_{e_{1}} f = - i e^{i t}

(In real-form,

- i e^{i t} = sin t - i cos t

; the real Fourier sine coefficient

b_{1} = 2

matches

t \approx 2 sin t

Closest point in $\ell^2$ to a finite-dimensional subspace

Given: In

ℓ^{2} = {(x_{1}, x_{2}, \dots) : \sum ∣ x_{k} ∣^{2} < \infty}

with

⟨ x, y ⟩ = \sum_{k} x_{k} \overset{y}{ˉ}_{k}

, let

x = (1, 2, \frac{1}{3}, \frac{1}{4}, \frac{1}{5}, \dots)

where

x_{k} = 1/ k

for

k \geq 3

, and

x_{1} = 1

x_{2} = 2

. Let

M = span {e_{1}, e_{2}}

(first two coordinate vectors). Find

P_{M} x

and the distance

∥ x - P_{M} x ∥

. Step 1 — Verify $x \in ℓ^{2}$ .

∥ x ∥^{2} = 1 + 4 + \sum_{k \geq 3} \frac{1}{k ^{2}}

. Since

\sum_{k \geq 3} \frac{1}{k ^{2}} = \frac{π ^{2}}{6} - 1 - \frac{1}{4} = \frac{π ^{2}}{6} - 1.25 \approx 0.3949

, finite. ✓ Step 2 — Project. Since

e_{1}, e_{2}

orthonormal:

P_{M} x = ⟨ x, e_{1} ⟩ e_{1} + ⟨ x, e_{2} ⟩ e_{2} = 1 \cdot e_{1} + 2 \cdot e_{2} = (1, 2, 0, 0, \dots)

. Step 3 — Residual.

x - P_{M} x = (0, 0, \frac{1}{3}, \frac{1}{4}, \dots)

, the "tail." Step 4 — Distance.

∥ x - P_{M} x ∥^{2} = k \geq 3 \sum \frac{1}{k ^{2}} = \frac{π ^{2}}{6} - 1 - \frac{1}{4} \approx 0.3949.

Answer:

P_{M} x = (1, 2, 0, 0, \dots), dist = \frac{π ^{2}}{6} - 1.25 \approx 0.6284

Common Misconceptions

Every norm is NOT an inner-product norm

Not all Banach spaces are Hilbert spaces. A norm comes from an inner product only if it satisfies the parallelogram law [3]. The spaces $ℓ^{p}$ and $L^{p}$ for $p \neq = 2$ are complete normed spaces but their norms violate the parallelogram law — they are Banach but not Hilbert. The case $p = 2$ is special precisely because it carries an inner product.

$\ell^1$ is not a Hilbert space

Take $x = e_{1} = (1, 0, 0, \dots)$ and $y = e_{2} = (0, 1, 0, \dots)$ in $ℓ^{1}$ with $∥ z ∥_{1} = \sum ∣ z_{k} ∣$ . Then $∥ x ∥_{1} = ∥ y ∥_{1} = 1$ , and $∥ x + y ∥_{1} = 2$ , $∥ x - y ∥_{1} = 2$ . The parallelogram law would require

∥ x + y ∥^{2} + ∥ x - y ∥^{2} = 4 + 4 = 8 = ? 2∥ x ∥^{2} + 2∥ y ∥^{2} = 2 + 2 = 4.

Since

8 \neq = 4

, no inner product induces the

ℓ^{1}

norm.

⟳just compute the four

ℓ^{1}

norms of

e_{1} \pm e_{2}

and substitute into [3].

Closedness is essential in the Projection Theorem

The decomposition $H = M \oplus M^{⊥}$ requires $M$ to be closed. For a non-closed subspace the infimum distance need not be attained, and $M \oplus M^{⊥}$ may fail to be all of $H$ . Example: the space of finitely-supported sequences in $ℓ^{2}$ is dense but not closed; its orthogonal complement is ${0}$ , yet it is not all of $ℓ^{2}$ , so $M \oplus M^{⊥} \neq = H$ .

Infinite-dimensional unit ball is not compact

In finite dimensions the closed unit ball is compact (Heine–Borel). In any infinite-dimensional Hilbert space it is not: the orthonormal sequence $(e_{n})$ has $∥ e_{n} ∥ = 1$ for all $n$ , yet $∥ e_{n} - e_{m} ∥^{2} = 2$ for $n \neq = m$ , so no subsequence is Cauchy and none converges. The geometry is genuinely "bigger" than intuition from $R^{3}$ .

Connections

Hilbert spaces sit at the crossroads of several theories. They are the special Banach spaces (Banach Spaces) whose norm satisfies the parallelogram law, so all of functional-analytic machinery applies, plus extra geometry. The Spectral Theorem for self-adjoint and normal operators is the infinite-dimensional generalization of diagonalizing symmetric matrices, and it lives most naturally on Hilbert spaces. Fourier Series and the Fourier Transform are concrete instances of expanding a vector in an orthonormal basis of $L^{2}$ . The Riesz Representation Theorem makes $H$ self-dual ( $H ≅ H^{*}$ ), a property failing for general Banach spaces. Finally, the abstract definition is the mathematical backbone of Quantum Mechanics, where states are rays in a complex Hilbert space.

Real-World Applications

Quantum mechanics. A system's state is a unit vector $ψ$ in a complex Hilbert space; observables are self-adjoint operators, and $⟨ ψ, A ψ ⟩$ is the expected measurement value. Probabilities are squared projection magnitudes (Born rule).
Signal processing & JPEG/MP3. Signals are vectors in $L^{2}$ ; decomposing them into orthonormal bases (Fourier, wavelet, DCT) is projection, and discarding small coefficients gives optimal lossy compression in the $L^{2}$ sense.
Least-squares & regression. Fitting data by minimizing squared error is orthogonal projection of the data vector onto the column space of the design matrix; the normal equations are exactly $⟨ residual, columns ⟩ = 0$ .
Quantum computing. Qubits live in $C^{2}$ , $n$ qubits in $(C^{2})^{\otimes n} = C^{2^{n}}$ ; gates are unitary operators preserving the inner product.
Partial differential equations. Weak/variational solutions (finite element method) are found via the Lax–Milgram theorem and projections in Sobolev spaces $H^{k}$ , which are Hilbert spaces.
Machine learning / kernel methods. Support vector machines and kernel regression operate in a Reproducing Kernel Hilbert Space (RKHS), where the "kernel trick" computes inner products of high-dimensional feature vectors implicitly.
Communications & MIMO. Matched filters and orthogonal frequency-division multiplexing (OFDM) rely on orthonormal carrier signals in $L^{2}$ to separate channels without interference.

Historical Development

1888–1900 — Inner products in mechanics/geometry. Quadratic forms and Euclidean dot products are well understood; Gram and Schmidt formalize orthogonalization.
1904–1910 — Hilbert's work on integral equations. David Hilbert studies infinite systems of equations and quadratic forms in infinitely many variables, implicitly using $ℓ^{2}$ .
1907 — Riesz–Fischer Theorem. Frigyes Riesz and Ernst Fischer prove $L^{2}$ and $ℓ^{2}$ are isometric via Fourier coefficients — the cornerstone linking function spaces to sequence spaces.
1907 — Riesz Representation Theorem. Riesz characterizes continuous linear functionals on $L^{2}$ , establishing self-duality.
1918 — Banach spaces emerge. Stefan Banach and others develop the general theory of complete normed spaces, of which Hilbert spaces are the geometrically richest example.
1927–1929 — Schmidt and the geometric language. Erhard Schmidt formulates the geometry of $ℓ^{2}$ (orthogonality, projection) in vector terms.
1929–1932 — von Neumann's axioms. John von Neumann gives the modern abstract definition of a (separable) Hilbert space and applies it in Mathematical Foundations of Quantum Mechanics (1932), unifying matrix and wave mechanics.
1930s — Spectral theory matures. von Neumann and M. H. Stone establish the spectral theorem for unbounded self-adjoint operators, essential for quantum dynamics.
1950 — RKHS theory. Nachman Aronszajn systematizes reproducing kernel Hilbert spaces, later central to statistics and machine learning.

Subtleties & Limits

Non-separability, bases, and structural questions

Most applications use separable Hilbert spaces (those with a countable orthonormal basis), all isomorphic to $ℓ^{2}$ . Non-separable Hilbert spaces (e.g. almost-periodic functions, certain spaces in quantum field theory) require an uncountable orthonormal basis and a transfinite version of Bessel/Parseval; their analysis is subtler and their use in physics (e.g. the "fermionic Fock space" subtleties, or representations of canonical commutation relations) remains an active interface. A deeper structural issue: while finite-dimensional inner product spaces are completely classified by dimension, and separable infinite-dimensional ones are all $≅ ℓ^{2}$ , the operator theory on them is vastly richer — the invariant subspace problem (does every bounded operator on a separable Hilbert space have a nontrivial closed invariant subspace?) is still open for Hilbert spaces, despite being resolved (negatively) on some Banach spaces. This shows the geometry is clean but the dynamics of operators are not.

Hilbert Spaces and Inner Products

The Core Insight

Formal Statement

Inner Product

Convention on linearity

Hilbert Space

Cauchy–Schwarz Inequality

Parallelogram Law characterizes inner-product norms

Projection Theorem

Why It Holds / Derivation

Projection Theorem

Riesz Representation Theorem

John von Neumann | 1903–1957

Building Intuition

Orthogonal projection of x onto a closed subspace M: the residual n = x − P_M x is perpendicular to M

Worked Examples

Closest point in $\ell^2$ to a finite-dimensional subspace

Common Misconceptions

Every norm is NOT an inner-product norm

$\ell^1$ is not a Hilbert space

Closedness is essential in the Projection Theorem

Infinite-dimensional unit ball is not compact

Connections

Real-World Applications

Historical Development

Subtleties & Limits

Non-separability, bases, and structural questions

See Also

Before this

Where this leads