Constrained Variation and Lagrange Multipliers

The augmented-functional idea has quietly become a load-bearing beam of computational science. In machine learning, the support-vector machine is literally a constrained variational problem: maximize a margin subject to classification inequalities, solved through its Lagrangian dual, where the nonzero multipliers pick out the "support vectors," and its Karush–Kuhn–Tucker conditions are the inequality-constraint generalization of equation (1). In optimal control and spacecraft trajectory design, Pontryagin's maximum principle introduces multiplier functions called costates, and a modern low-thrust interplanetary transfer is found by numerically solving exactly the pointwise-multiplier system of equation (5). In electronic-structure theory, density-functional calculations minimize an energy functional at fixed particle number, and the enforcing multiplier is the chemical potential $\mu$ that sets whether electrons flow between two materials in contact. An open frontier is enforcing hard physical constraints — conservation laws, positivity, incompressibility — inside neural-network surrogates for physics, where naïvely learned models drift off the constraint surface exactly as a bare $\delta J=0$ would.

Imagine every admissible function as a point, and the whole space of functions as a landscape whose height is $J$. Unconstrained, you roll to the bottom straight downhill, along $-\nabla J$. But a constraint carves a curved surface out of that landscape and forbids you to leave it — you can only slide along it. You stop not where the ground is flat, but where it is flat along the surface: the downhill pull of $J$ has been used up in directions you are not allowed to take, and whatever remains of $\nabla J$ points straight out of the surface, parallel to its gradient $\nabla g$. The multiplier $\lambda$ is nothing but the ratio between those two parallel vectors.

Why the two constraint types cost different things follows from the same picture. An integral constraint fixes one global number — a length, an area, a normalization — so it removes exactly one degree of freedom, and one constant $\lambda$ restores it. A pointwise constraint is far more demanding: holding separately at every $x$, it is really an infinite family of constraints stacked along the interval, each point needing its own multiplier, which assemble into a field $\lambda(x)$. Operationally the multiplier lets you pretend the variations are free: you overcount by one degree of freedom, then spend the multiplier to annihilate the correlated piece of $\delta y$, and the fundamental lemma applies to what is left. The payoff is that this forced-in multiplier turns out to be the quantity you most wanted to know — as the isoperimetric problem shows in one hard number.

The starting point is finite-dimensional, and it is worth stating cleanly because the whole method is its infinite-dimensional echo. To extremize $f$ on the surface $g_a=0$, the gradient of $f$ cannot have any component tangent to the surface, so it must be a combination of the constraint normals.

where: $f$ — the function being extremized; $g_a$ — the constraint functions, $a=1,\dots,m$; $\lambda_a$ — the undetermined multipliers; $\mathbf{x}$ — a point in $\mathbb{R}^n$. Read it as: at the optimum the objective pushes purely along the constraints' normals, and the $\lambda_a$ record how hard. The variational problem replaces $f$ by a functional and $\mathbf x$ by a function. The isoperimetric setup fixes one integral quantity while extremizing another.

where: $J$ — the functional to extremize; $K$ — the integral constraint functional; $\ell$ — its fixed prescribed value; $F,G$ — the two integrands; $y(x)$ — the unknown function with fixed endpoints. Read it as: one number, $\ell$, is frozen while the whole shape of $y$ is otherwise free. Adjoining $K$ with a single constant $\lambda$ produces an ordinary Euler–Lagrange equation for the augmented integrand — the central result for integral constraints.

where: $\Phi$ — the augmented integrand $F-\lambda G$; $\lambda$ — the single constant multiplier. Read it as: solve the ordinary Euler–Lagrange equation, but for $F-\lambda G$ instead of $F$; the constant $\lambda$ rides along as an unknown. A pointwise constraint is a different animal. Requiring $g=0$ at every $x$ demands a multiplier that can vary with $x$.

where: $g$ — the constraint required at every $x$; $\lambda(x)$ — the multiplier function; $y_i$ — several dependent functions. Read it as: enforced pointwise, the constraint's price is a function, not a number. Varying the several $y_i$ then gives an augmented Euler–Lagrange system whose extra term is a generalized constraint force.

where: $i$ — runs over the dependent functions; $\lambda(x)\,\partial g/\partial y_i$ — the generalized force that keeps the solution on the constraint. Read it as: the ordinary equations of motion plus a force normal to the constraint, of strength $\lambda(x)$. The same machinery converts a ratio of integrals into an eigenvalue problem. Extremizing the numerator at fixed denominator turns the Rayleigh quotient into a Sturm–Liouville equation with the multiplier as eigenvalue.

where: $p,q,w$ — the Sturm–Liouville coefficient functions with weight $w>0$; $\lambda$ — the multiplier, now identified as the eigenvalue; $\int w\,y^2$ — the normalization that is being held fixed. Read it as: normalization is the constraint, and its multiplier is the energy. Every case above is one line of a single master recipe: vary the augmented functional freely, then let the constraint pin the multiplier.

where: $J-\lambda K$ — the augmented functional; $K[y]=\ell$ — the constraint that closes the system. Read it as: the multiplier is the one extra unknown that matches the one extra equation, the constraint itself.

The augmented machinery collapses onto familiar unconstrained and finite-dimensional results in each limit.

1 · Maximum-entropy dice: the multiplier is an inverse temperature

A six-sided die is loaded so that its long-run mean is $\langle i\rangle=4.5$ instead of the unbiased $3.5$. Of all distributions with that mean, which is the least presumptuous — the one of maximum entropy — and what does the constraint's Lagrange multiplier turn out to be?

Step 1. Set up the constrained extremum. Maximize the Gibbs entropy $S=-\sum_{i=1}^6 p_i\ln p_i$ subject to two constraints: normalization $\sum_i p_i=1$ and the mean $\sum_i i\,p_i=4.5$. Two constraints means two multipliers, $\alpha$ for normalization and $\beta$ for the mean.

Step 2. Extremize the augmented function. Setting $\partial/\partial p_i\big[-\sum p_i\ln p_i-\alpha\sum p_i-\beta\sum i\,p_i\big]=0$ gives $-\ln p_i-1-\alpha-\beta i=0$, hence $p_i=e^{-\beta i}/Z$ with $Z=\sum_{i=1}^6 e^{-\beta i}$ absorbing the normalization multiplier. This is the Boltzmann form — an exponential in the constrained quantity, with $\beta$ the multiplier on it.

Step 3. Fix $\beta$ by re-imposing the constraint. Requiring $\langle i\rangle=-\,\mathrm{d}\ln Z/\mathrm{d}\beta=4.5$ is one transcendental equation for $\beta$; solving it numerically gives $\boldsymbol{\beta=-0.371}$ (equivalently $e^{-\beta}=1.449$). The multiplier is negative because we forced the mean above the unbiased value, tilting weight toward the high faces — a formal negative temperature.

Step 4. Read off the distribution and check it. The probabilities are $p_1=0.054,\ p_2=0.079,\ p_3=0.114,\ p_4=0.165,\ p_5=0.240,\ p_6=0.347$; they sum to $1$, give $\langle i\rangle=4.5$ by construction, and yield entropy $S=1.614$ nats — below the unconstrained maximum $\ln 6=1.792$ nats, as it must be once a constraint bites. The multiplier β is exactly the inverse temperature of statistical mechanics: had the constraint been energy, $\beta=1/k_BT$ and $p_i\propto e^{-\beta E_i}$ would be the Boltzmann distribution itself.

2 · Rayleigh–Ritz: the multiplier is an energy eigenvalue

Estimate the ground-state energy of a particle of mass $m$ in an infinite square well of width $L$ (potential zero on $[0,L]$, infinite outside) by extremizing $\langle H\rangle$ subject to normalization, using the simple trial function $\psi(x)=x(L-x)$.

Step 1. Frame it as a constrained variation. Minimize $\langle H\rangle=\int_0^L\psi^*\hat H\psi\,\mathrm{d}x$ subject to $\int_0^L|\psi|^2\,\mathrm{d}x=1$. Equation (6) tells us the multiplier on the normalization is the energy eigenvalue: the stationary value of the Rayleigh quotient is an upper bound on the true ground-state energy.

Step 2. Form the Rayleigh quotient with the trial function. With $\hat H=-(\hbar^2/2m)\,\mathrm{d}^2/\mathrm{d}x^2$, integrate by parts to get $\langle H\rangle=\dfrac{\hbar^2}{2m}\dfrac{\int_0^L\psi'^2\,\mathrm{d}x}{\int_0^L\psi^2\,\mathrm{d}x}$. The trial $\psi=x(L-x)$ vanishes at both walls, so it is admissible.

Step 3. Evaluate the two integrals. With $\psi'=L-2x$, the numerator is $\int_0^L(L-2x)^2\,\mathrm{d}x=L^3/3$, and the denominator is $\int_0^L x^2(L-x)^2\,\mathrm{d}x=L^5/30$. Their ratio is $10/L^2$, so $\langle H\rangle=\dfrac{\hbar^2}{2m}\cdot\dfrac{10}{L^2}=\dfrac{5\hbar^2}{mL^2}$.

Step 4. Compare with the exact answer. The true ground state is $E_1=\dfrac{\pi^2\hbar^2}{2mL^2}=\dfrac{4.9348\,\hbar^2}{mL^2}$. The estimate $\mathbf{5.000\,\hbar^2/mL^2}$ sits just 1.32 % above it — and above, never below, because the multiplier (the stationary Rayleigh quotient) is a rigorous upper bound on the lowest eigenvalue. A crude parabola pins the ground-state energy to better than 2 % in one line — the whole engine of the Rayleigh–Ritz method.

Constrained variation is where geometry, mechanics, and probability meet, and one name presides over all three: the man who turned his predecessors' ad-hoc constraint tricks into a single systematic calculus.

Texts. Stone & Goldbart, Mathematics for Physics, ch. 1 (Calculus of Variations) · Arfken & Weber, Mathematical Methods for Physicists, 7th ed., ch. 22 · Riley, Hobson & Bence, Mathematical Methods for Physics and Engineering, 3rd ed., ch. 22 · Gelfand & Fomin, Calculus of Variations, ch. 1–2 (the cleanest proof of the isoperimetric rule) · Lanczos, The Variational Principles of Mechanics, ch. II (multipliers as constraint forces) · Hassani, Mathematical Physics, ch. on variational methods. The multiplier theorem (Derivation 1) uses a two-parameter argument; specializing it recovers $\nabla f=\lambda\nabla g$ of equation (1). The catenary is $y=c\cosh(x/c)$ with $c=T_0/\rho g$. The isoperimetric figure uses $r=P/2\pi=0.637$ m, $A=P^2/4\pi=1.273$ m² for $P=4$ m, saturating $4\pi A\le P^2$. Worked example 1 (maximum-entropy dice, mean $4.5$) gives $\beta=-0.371$, $p=(0.054,0.079,0.114,0.165,0.240,0.347)$, $S=1.614$ nats $<\ln 6$; the identification $\beta=1/k_BT$ is textbook MaxEnt. Worked example 2 (Rayleigh–Ritz, $\psi=x(L-x)$) gives $\langle H\rangle=5\hbar^2/mL^2$ versus exact $\pi^2\hbar^2/2mL^2=4.9348\,\hbar^2/mL^2$, a 1.32 % upper bound.

Prerequisites. First Variation and Stationarity · Euler–Lagrange Equation · Fundamental Lemma of the Calculus of Variations · Functionals and Function Spaces. Leads to. Isoperimetric Problems · Geodesics as Variational Problems · Lagrangian Mechanics · Hamilton's Principle of Stationary Action.

Ask the tutor: Ask me to work any constrained problem end to end — pick the catenary, the isoperimetric circle, a geodesic, or a Rayleigh–Ritz eigenvalue, and we will build the augmented functional, extremize, and pin down exactly what its multiplier means.