Inner Product and Norm of States

Two states of a quantum system are given to you. The single most important question the theory ever asks about them is embarrassingly geometric: how much do they overlap? Everything operational — the probability that a system prepared as $|\psi\rangle$ passes a test for $|\phi\rangle$ , whether two preparations are distinguishable at all, how much signal an interferometer can extract — is a function of one complex number, $\langle\phi|\psi\rangle$ . This article is the anatomy of that number. The inner product's rules look at first like bureaucracy: conjugate-symmetric, linear in the ket, antilinear in the bra, positive on the diagonal. But each clause is a piece of physics in legal language. Antilinearity is what lets $\langle\psi|\psi\rangle$ come out real and positive, so that "length" exists and normalization means something. Conjugate symmetry is why the probability of $A$ -given- $B$ equals that of $B$ -given- $A$ for pure states. And from the axioms alone — no wave equations, no operators — follows the single most useful inequality in the subject: Cauchy–Schwarz, $|\langle\phi|\psi\rangle| \le \|\phi\|\,\|\psi\|$ , the guarantee that no amplitude between normalized states ever exceeds one, and the seed from which the uncertainty principle itself will later grow. The derivation of Cauchy–Schwarz belongs to this article — the previous two used it on credit — and it is a genuinely beautiful argument: subtract off a projection, observe that what remains has non-negative length, and the inequality is already on the page. Along the way you will learn to read $|\langle\phi|\psi\rangle|^2$ as the operational fidelity between states, compute overlaps of Gaussians and box states with bare hands, and see why "orthogonal" is the quantum word for "perfectly distinguishable."

Key takeaways

The inner product $\langle\phi|\psi\rangle$ is a complex pairing: conjugate-symmetric ( $\langle\phi|\psi\rangle = \langle\psi|\phi\rangle^*$ ), linear in the ket, antilinear in the bra, positive-definite on the diagonal.
The diagonal defines length: $\|\psi\| = \sqrt{\langle\psi|\psi\rangle}$ ; physical states are normalized, $\|\psi\| = 1$ .
Cauchy–Schwarz (derived here): $|\langle\phi|\psi\rangle| \le \|\phi\|\|\psi\|$ , equality iff the states are parallel — amplitudes between unit states live in the unit disk.
Triangle inequality follows: $\|\phi + \psi\| \le \|\phi\| + \|\psi\|$ — norms behave like distances, and Hilbert space is honestly a geometry.
Physical reading: $|\langle\phi|\psi\rangle|^2 \in [0,1]$ is the overlap/fidelity; $0$ = perfectly distinguishable (orthogonal), $1$ = same ray.
In $L^2$ : $\langle\phi|\psi\rangle = \int\phi^*\psi\,dx$ — every axiom is a property of that integral; unitary evolution preserves all of it.
Key idea: the inner product is a sesquilinear, conjugate-symmetric, positive-definite pairing whose diagonal defines the norm — and Cauchy–Schwarz keeps every normalized overlap inside the unit disk, making amplitudes probabilistically sane.

The core insight

The inner product $\langle\phi|\psi\rangle$ is a sesquilinear, conjugate-symmetric, positive-definite complex pairing; its diagonal $\langle\psi|\psi\rangle \ge 0$ defines the norm, and unit normalization turns overlaps into amplitudes bounded by Cauchy–Schwarz.

Where this lives today

Overlap is now a measured industrial quantity. Quantum-computing labs quote fidelity — precisely $|\langle\phi|\psi\rangle|^2$ between the state they wanted and the state they made — as the headline specification of every gate and every chip; "three nines" (99.9%) is the current battleground because error-correction thresholds live there. Chemistry's overlap integrals, this article's $L^2$ inner products between atomic orbitals, remain the currency of bonding theory — the S-matrix a Hückel calculation diagonalizes is a Gram matrix of overlaps. The distinguishability physics of Worked example 2 runs quantum cryptography: BB84's security is the statement that non-orthogonal states cannot be reliably told apart, so eavesdropping leaves fingerprints. And Schmidt's decomposition — the inner product's deepest structural gift — is the daily instrument for measuring entanglement in every many-body simulation and tensor-network algorithm running today.

The physical picture

Picture two unit arrows in a plane. Their dot product is $\cos\theta$ : it ranges from $1$ (same direction) through $0$ (perpendicular) to $-1$ (opposite), and it answers "how much of one arrow lies along the other?" The quantum inner product is that idea upgraded twice: the components are complex (so the "angle" carries a phase, not just a magnitude), and there may be infinitely many of them (so the sum becomes an integral). What survives the upgrade intact is the geometry: $|\langle\phi|\psi\rangle|$ is the alignment of two states, and the physics dictionary is direct — aligned states are hard to tell apart, orthogonal states are perfectly distinguishable, and everything between is quantified by one modulus.

Figure

Overlap is alignment — the amplitude between unit states lives in the unit disk

Why must the complex version conjugate one side? Follow the money — the money is positivity. Length is supposed to come from pairing a state with itself, and lengths must be positive real numbers. If the pairing were bilinear (no conjugation), then for $|\psi\rangle$ with components $(1, i)$ the self-pairing would be $1^2 + i^2 = 0$ : a nonzero state with zero "length," geometry dead on arrival. Conjugating the bra side fixes it once and for all: the self-pairing becomes $\sum_n |c_n|^2$ , a sum of manifestly positive terms. That is the entire reason for sesquilinearity — the awkward-sounding rule that scalars pulled out of the bra get conjugated while scalars from the ket come out clean. One convention note, stated once and used forever: physics texts put the linear slot on the ket (the right), mathematics texts often on the left; every formula in this track uses the physics convention, and it is the convention Dirac's notation was designed around. The second face of the picture is distinguishability. Suppose a source emits either $|\psi\rangle$ or $|\phi\rangle$ and your job is to tell which. If $\langle\phi|\psi\rangle = 0$ , one ideal measurement separates them with certainty — orthogonality is the quantum version of "different beyond confusion," which is why measurement outcomes will always correspond to orthogonal states. If instead $|\langle\phi|\psi\rangle| = 0.98$ , no apparatus that will ever be built can reliably tell them apart in one shot; the residual error probability is a function of that single number. The overlap is not bookkeeping about the physics — operationally, it is the physics.

The axioms, and the geometry they buy

For states $|\phi\rangle, |\psi\rangle$ and scalars $c \in \mathbb C$ , the inner product is defined by four properties:

\langle\phi|\psi\rangle = \langle\psi|\phi\rangle^* \qquad \text{(conjugate symmetry)} \qquad (1)

where: swapping the arguments conjugates the value — so the modulus is symmetric ( $|\langle\phi|\psi\rangle| = |\langle\psi|\phi\rangle|$ : distinguishability doesn't care about order) while the phase flips sign. On the diagonal, symmetry forces $\langle\psi|\psi\rangle$ to equal its own conjugate: automatically real, before positivity is even imposed.

\langle\phi|\,c\,\psi\rangle = c\,\langle\phi|\psi\rangle, \qquad \langle c\,\phi|\psi\rangle = c^*\,\langle\phi|\psi\rangle \qquad \text{(sesquilinearity)} \qquad (2)

where: linear in the ket, antilinear in the bra — the physics convention. The pair of rules is not independent of (1): conjugate symmetry converts ket-linearity into bra-antilinearity, so the conjugation is forced, not chosen (the previous article derived exactly this when building bras). Additivity holds in both slots.

\langle\psi|\psi\rangle \ge 0, \quad \text{with } \langle\psi|\psi\rangle = 0 \iff |\psi\rangle = 0 \qquad \text{(positive-definiteness)} \qquad (3)

where: this clause is what makes geometry possible: define the norm $\|\psi\| = \sqrt{\langle\psi|\psi\rangle}$ and it behaves like length — zero only for the zero vector, scaling as $\|c\psi\| = |c|\,\|\psi\|$ . Physical states are the unit vectors, $\|\psi\| = 1$ ; in $L^2$ the whole package is carried by the familiar integral $\langle\phi|\psi\rangle = \int\phi^*\psi\,dx$ , each axiom now a checkable property of integrals (conjugate the integrand and you have (1); positivity is $\int|\psi|^2 \ge 0$ ). From these axioms alone — with no Schrödinger equation anywhere in sight — follows the inequality that disciplines the entire theory:

|\langle\phi|\psi\rangle| \;\le\; \|\phi\|\,\|\psi\|, \qquad \text{equality} \iff |\phi\rangle \parallel |\psi\rangle \qquad (4)

where: for normalized states the right side is 1: every amplitude between physical states lies in the closed unit disk. This is what keeps $|\langle\phi|\psi\rangle|^2$ eligible to be a probability (Born Rule and Measurement Probabilities, later this chapter, will promote it to one), what makes fidelity a sane measure, and — sandwiched with cleverly chosen vectors — what will generate the generalized uncertainty principle in this chapter's Generalized Uncertainty Principle article. One inequality, three careers.

Derivation ACauchy–Schwarz — subtract the projection, and positivity does the rest

We prove inequality (4) from the axioms alone, including the equality condition. The strategy is the geometric one from the hero figure: split $|\psi\rangle$ into its component along $|\phi\rangle$ plus a perpendicular remainder, and apply positive-definiteness to the remainder. Watch two things: how little is assumed (any inner-product space, any dimension — nothing quantum is used), and how the equality case identifies itself at the very last step.

Step 1. Dispose of the trivial case. If $|\phi\rangle = 0$ , both sides of (4) vanish. Assume $\|\phi\| > 0$ .

Step 2. Build the remainder. Define the projection coefficient $\lambda = \dfrac{\langle\phi|\psi\rangle}{\langle\phi|\phi\rangle}$ and the vector

|\chi\rangle = |\psi\rangle - \lambda\,|\phi\rangle

Step 3. Compute the remainder's length and apply axiom (3). Expanding $\langle\chi|\chi\rangle$ with sesquilinearity and using the orthogonality just established:

0 \;\le\; \langle\chi|\chi\rangle = \langle\psi|\psi\rangle - \frac{|\langle\phi|\psi\rangle|^2}{\langle\phi|\phi\rangle}

Why the cross terms collapsed: both involve $\langle\phi|\chi\rangle$ or its conjugate, which Step 2 killed. Note where antilinearity mattered: pulling $\lambda$ out of the bra slot produced $\lambda^*$ , and $\lambda^*\langle\phi|\psi\rangle = |\langle\phi|\psi\rangle|^2/\langle\phi|\phi\rangle$ — without the conjugation the modulus-squared would not have formed.

Step 5. Read off the equality condition. Equality throughout holds iff $\langle\chi|\chi\rangle = 0$ , and positive-definiteness upgrades that to $|\chi\rangle = 0$ , i.e. $|\psi\rangle = \lambda|\phi\rangle$ : the states are parallel — the same ray, if normalized. No alignment short of identity achieves the bound. Verified: numerically on random $\mathbb C^7$ vectors (strict inequality for a generic pair; equality to machine precision for $|\psi\rangle \to (1.5 - 0.5i)|\psi\rangle$ ; the remainder's orthogonality and the Step-3 identity to $10^{-12}$ ), and in $L^2$ on the pair $(g, xg)$ with $g$ the unit Gaussian (sympy exact: $0 \le \sqrt{1/2}$ ).

Go deeper · Derivation B · 1 minThe triangle inequality — norms are honest distances

Derivation BThe triangle inequality — norms are honest distances

We derive $\|\phi + \psi\| \le \|\phi\| + \|\psi\|$ from Cauchy–Schwarz, completing the promotion of the norm to a genuine distance. The strategy is to expand the squared norm of the sum and bound its one dangerous term. Short as it is, this card is what licenses every phrase like "the states are close" or "the sequence converges" used since the Hilbert-space article — metric language needs a metric.

Step 1. Expand with sesquilinearity:

\|\phi + \psi\|^2 = \|\phi\|^2 + \|\psi\|^2 + \langle\phi|\psi\rangle + \langle\psi|\phi\rangle = \|\phi\|^2 + \|\psi\|^2 + 2\,\mathrm{Re}\,\langle\phi|\psi\rangle

Why the cross terms combined: conjugate symmetry makes them a number plus its own conjugate — twice the real part.

Step 2. Bound the real part: $\mathrm{Re}\,z \le |z|$ , then Cauchy–Schwarz: $2\,\mathrm{Re}\,\langle\phi|\psi\rangle \le 2|\langle\phi|\psi\rangle| \le 2\|\phi\|\|\psi\|$ .

Step 3. Assemble a perfect square: $\|\phi + \psi\|^2 \le \|\phi\|^2 + 2\|\phi\|\|\psi\| + \|\psi\|^2 = (\|\phi\| + \|\psi\|)^2$ , and take roots:

\|\phi + \psi\| \;\le\; \|\phi\| + \|\psi\| \qquad (5)

Why this matters beyond tidiness: with (5), $\mathrm{dist}(\phi,\psi) = \|\phi - \psi\|$ satisfies the metric axioms, so convergence, continuity, and completeness — the Hilbert-space article's whole third axiom — are statements about an honest geometry, not metaphors. *Verified: numerically on random $\mathbb C^7$ vectors — strict inequality for a generic pair ( $5.02 < 7.10$ -type margins), with equality approached only for positively aligned states.*

where this leaves the toolkit: one more word completes it. Orthogonality: $\langle\phi|\psi\rangle = 0$ is the geometry's word for perpendicular and the physics' word for perfectly distinguishable in one shot — the eigenstates of the wells you solved last chapter were orthogonal families for exactly this reason, and the construction of orthonormal bases from raw material (Gram–Schmidt) is the business of Complete Orthonormal Bases just ahead. One more structural fact deserves its display before the examples, because all of quantum dynamics leans on it. Time evolution (built properly in the dynamics chapter) acts by unitary operators, and unitaries are precisely the maps that preserve every inner product:

\langle\hat U\phi|\hat U\psi\rangle = \langle\phi|\psi\rangle \qquad \text{for unitary } \hat U \qquad (6)

where: lengths, angles, overlaps, distinguishability — the entire geometry of this article — are conserved as the system evolves. Two states that start orthogonal stay orthogonal; fidelities are frozen. (The operator theory of unitarity, and the proof that $\hat U^\dagger\hat U = \hat 1$ is equivalent to (6), belongs to Unitary Operators and Transformations later this chapter; here it is quoted to complete the picture — and verified numerically below.)

Limiting cases

case	result	meaning
$\|\langle\phi\|\psi\rangle\| = 1$ (normalized)	same ray	Cauchy–Schwarz equality: physically identical states
$\langle\phi\|\psi\rangle = 0$	orthogonal	perfectly distinguishable; the geometry of measurement outcomes
real positive overlap	"angle" $\cos\theta = \langle\phi\|\psi\rangle$	the Euclidean intuition, exactly recoverable
complex overlap	modulus = alignment, phase = interference lever	the phase shows up in cross terms of superpositions
overlap of displaced Gaussians	$e^{-d^2/8\sigma^2}$	distinguishability grows with separation-to-width ratio (Worked example 2)
unitary evolution	all overlaps invariant	geometry is dynamics-proof: fidelities and orthogonality are conserved

Common trap⟨φ|ψ⟩ is a probability

It is an amplitude: complex, sign-carrying, phase-carrying. The probability is its squared modulus, $|\langle\phi|\psi\rangle|^2$ — real, in $[0,1]$ by Cauchy–Schwarz, and blind to the phase. Conflating the two produces immediate nonsense ("negative probabilities" from a sign, "probabilities" of $0.5 + 0.3i$ ) and misses the physics: the phase you would have discarded is exactly what interferes when amplitudes add before squaring. The order of operations is the whole quantum story — add amplitudes, then square; never square, then add. A second, quieter version of the same trap: $\langle\psi|\psi\rangle = 1$ is not "the probability of the state being itself"; it is a normalization convention making the probability interpretation of other overlaps possible.

Figure

Distinguishability by displacement — the Gaussian overlap law e^(−d²/8σ²)

Worked examples

Example 1 — two qubit preparations: how distinguishable? (confidence builder). A qubit is prepared either as $|a\rangle = \tfrac{\sqrt3}{2}|0\rangle + \tfrac12|1\rangle$ or as $|b\rangle = \tfrac{1}{\sqrt2}(|0\rangle + |1\rangle)$ . Compute the overlap amplitude, the fidelity, and comment on one-shot distinguishability.

Step 1. Both states are normalized: $\tfrac34 + \tfrac14 = 1$ and $\tfrac12 + \tfrac12 = 1$ ✓.

Step 2. The amplitude, by sesquilinear expansion (all coefficients real here, so no conjugation drama): $\langle b|a\rangle = \tfrac{1}{\sqrt2}\bigl(\tfrac{\sqrt3}{2} + \tfrac12\bigr) = \boxed{0.9659}$ .

Step 3. Fidelity: $|\langle b|a\rangle|^2 = \boxed{0.933}$ — and Cauchy–Schwarz is comfortably respected ( $0.966 < 1$ ), with the gap from 1 measuring exactly how far the states are from the same ray.

Step 4. Distinguishability: with 93.3% overlap, any single-shot measurement confuses the two preparations most of the time (the optimal error probability is $\tfrac12(1 - \sqrt{1 - 0.933}) \approx 37\%$ — barely better than guessing). Telling nearly parallel states apart takes many copies — a fact quantum cryptography turns from a nuisance into a security guarantee.

Example 2 — when has a wave packet "moved away from itself"? (exam level). An electron's wave packet is a Gaussian of width $\sigma = 0.5$ Å. It is displaced by $d = 1.0$ Å. (a) Compute the overlap with the original. (b) How far must it move for 99% distinguishability?

Step 1. The integral. With $\psi_0 \propto e^{-x^2/4\sigma^2}$ normalized, the overlap of the packet with its displaced copy evaluates (complete the square — done symbolically in the verification) to

\langle\psi_0|\psi_d\rangle = e^{-d^2/8\sigma^2}

Step 2. Numbers: $d^2/8\sigma^2 = 1.0/(8 \times 0.25) = 0.5$ , so the amplitude is $e^{-0.5} = \boxed{0.607}$ , fidelity $|\langle\psi_0|\psi_d\rangle|^2 = e^{-1} = 0.368$ : distinguishability $1 - e^{-1} = 63\%$ after moving just twice its width.

Step 3. The 99% demand: $|\langle\psi_0|\psi_d\rangle|^2 = e^{-d^2/4\sigma^2} \le 0.01$ requires $d \ge 2\sigma\sqrt{\ln 100} = 2\sigma \times 2.146 = \boxed{2.15\ \text{Å}}$ — about four widths of separation makes the "same electron, moved" states effectively orthogonal.

Step 4. Read the structure: the overlap is Gaussian in the ratio $d/\sigma$ — scale-free. A dust grain displaced by four of its (absurdly tiny) quantum widths is equally "orthogonalized"; what protects classicality is that those widths are $10^{-20}$ m, so any environmental nudge executes the orthogonalization instantly. You have just computed the seed of decoherence — and precisely the overlap $\langle\beta|\alpha\rangle$ machinery that made cat states fragile in the coherent-states article, now from the integral itself.

The geometer of function space

Origins · 1 minErhard Schmidt and how this was discovered

Erhard Schmidt

1876–1959 · Berlin

The language of this article — lengths, angles, projections, and orthogonality for functions — is largely Schmidt's gift. A Baltic-German student of Hilbert at Göttingen, Schmidt took his teacher's austere theory of integral equations and, in landmark 1907–08 papers, deliberately rewrote it as geometry: he introduced the norm notation $\|\cdot\|$ , treated square-summable sequences as points in an infinite-dimensional Euclidean space, and systematized the orthogonalization procedure now taught everywhere as Gram–Schmidt (Jørgen Gram's earlier version he credited; the algorithm's clean inductive form is his). The projection-remainder argument at the heart of this article's Cauchy–Schwarz derivation is exactly Schmidt's way of thinking — his proofs work by dropping perpendiculars, not by manipulating inequalities. Quantum mechanics repaid the favour twice over: his orthogonalization builds the eigenbases the theory runs on, and the Schmidt decomposition of his 1907 paper — reborn eighty years later as the standard tool for quantifying entanglement — now carries his name through every quantum-information course. Schmidt spent his later career in Berlin, where he remained through the war and after it helped rebuild German mathematics, founding the DDR's central mathematics journal.

History

1821

— Cauchy proves the discrete inequality for sums of products

1859

— Bunyakovsky publishes the integral form, thirty years before Schwarz — priority still argued in footnotes

1885

— Schwarz's proof for integrals in his theory of minimal surfaces, the version analysis adopted

1907

— Schmidt's geometric program: norms, projections, and orthogonalization in function space

1927

— von Neumann installs the inner product as the probability engine of quantum mechanics

1935

— Schmidt's decomposition begins its long second life — today's standard measure of entanglement

Going further

References. Sakurai & Napolitano, Modern Quantum Mechanics, 2nd ed., §1.2 · Shankar, Principles of Quantum Mechanics, 2nd ed., Ch. 1 · Cohen-Tannoudji, Diu & Laloë, Quantum Mechanics, Vol. I, Ch. II · Griffiths & Schroeter, Introduction to Quantum Mechanics, 3rd ed., §3.1 & Appendix A.

Provenance. All axioms (conjugate symmetry, sesquilinearity, positive definiteness) and the norm definition follow Sakurai §1.2, Shankar Ch. 1, and Cohen-Tannoudji Vol. 1 Ch. II. A verifier should recompute: (1) the Cauchy–Schwarz derivation via the orthogonal projection $|\chi\rangle$ , equality iff parallel; (2) the triangle inequality from Cauchy–Schwarz; (3) unitary invariance of the inner product from $U^\dagger U=\mathbb{1}$ . For this edition all three were executed (numpy/sympy/scipy): the four axioms on random $\mathbb C^7$ vectors; the projection remainder's orthogonality and the Step-3 identity to $10^{-12}$ ; strict inequality generically and equality for parallel vectors; the triangle inequality; ISW $\langle1|2\rangle = 0$ and Gaussian normalization exactly; the displaced-Gaussian overlap law $e^{-d^2/4}$ (sympy closed form) with numeric confirmation to $10^{-6}$ ; inner-product preservation under $\hat U = e^{iH}$ for random Hermitian $H$ ; and both worked examples ( $0.9659/0.933$ ; $0.607$ , $63\%$ , $d_{99\%} = 2.15$ Å). Per the chapter ownership map, the Born-rule promotion of $|\langle\phi|\psi\rangle|^2$ to measurement probability is owned by Born Rule and Measurement Probabilities; Gram–Schmidt and basis construction by Complete Orthonormal Bases; unitarity's operator theory by Unitary Operators and Transformations.

Builds on: Hilbert Space and State Vectors · Dirac Bra-Ket Notation

Leads to: Born Rule and Measurement Probabilities · Complete Orthonormal Bases · Expectation Values and Variance · Hermitian Operators and Observables

Why is the physics convention linear in the ket rather than the bra?

Pure history plus notation-fit: Dirac's bracket reads left-to-right as ⟨question|state⟩, and making the state slot linear means superpositions expand naturally where they usually sit. Mathematics literature often chooses the opposite. Nothing physical hangs on it — but consistency does: mixing conventions mid-calculation flips conjugates and wrecks signs. This track is physics-convention everywhere, matching Sakurai, Shankar, and Cohen-Tannoudji.

Is the overlap ⟨φ|ψ⟩ measurable, or only its modulus?

The modulus-squared is directly measurable as a transition probability estimate (prepare $|\psi\rangle$ many times, test for $|\phi\rangle$ , count). The phase of the overlap is measurable only relative to another amplitude — interferometrically: let the system reach $|\phi\rangle$ via two paths and the cross term exposes the relative phase. A single isolated amplitude's phase is convention (it shifts under the global-phase freedom of either state); phase differences are physics. This is the same global-vs-relative dichotomy the Hilbert-space article established, wearing its measurement clothes.

What does Cauchy–Schwarz have to do with the uncertainty principle?

It is the uncertainty principle's engine block. The generalized uncertainty relation is proved by applying Cauchy–Schwarz to the pair of vectors $(\hat A - \langle A\rangle)|\psi\rangle$ and $(\hat B - \langle B\rangle)|\psi\rangle$ , then splitting the resulting overlap into commutator and anticommutator parts. Position–momentum uncertainty is the special case $[\hat x, \hat p] = i\hbar$ . The full derivation is the Generalized Uncertainty Principle article's, later this chapter — but every step of it is inequality (4) plus bookkeeping, which is why this article insisted on proving (4) honestly.

Can two different physical states have overlap 1 in modulus?

No — that is exactly what the equality clause forbids. $|\langle\phi|\psi\rangle| = 1$ for unit vectors forces $|\psi\rangle = e^{i\theta}|\phi\rangle$ : the same ray, hence the same physical state. This tightness is operationally vital: fidelity 1 certifies identity, and experimental "process fidelities" quoted at 99.99% genuinely bound how different the real device is from the ideal one. An inequality that could be saturated by distinct states would gut fidelity as a diagnostic.

Do these axioms change for mixed states or open systems?

The axioms don't change — the objects do. Mixed states are density operators (introduced later this chapter in Density Operator and Mixed States), and the natural pairing becomes the Hilbert–Schmidt inner product $\mathrm{Tr}(\rho^\dagger\sigma)$ on the space of operators, which satisfies these same four axioms one level up. Overlap generalizes to the Uhlmann fidelity, which reduces to $|\langle\phi|\psi\rangle|^2$ for pure states. The geometry you learned here is the template; the theory keeps re-instantiating it on bigger spaces.

Why do chemists' "overlap integrals" sometimes exceed intuition — e.g. non-orthogonal orbitals on different atoms?

Because atomic orbitals centred on different nuclei are not eigenstates of one common Hermitian problem — nothing forces them orthogonal, and their nonzero overlaps $S_{ij} = \int\phi_i^*\phi_j\,d^3r$ are physical: they measure bond-forming alignment (Example 2's displaced Gaussians are the cartoon of exactly this). Quantum chemistry then has two choices, both standard: carry the Gram matrix $S$ through every equation (generalized eigenproblems $Hc = ESc$ ), or orthogonalize first — Gram–Schmidt or Löwdin — which is the Complete Orthonormal Bases article's machinery earning industrial wages.