Constraints and Lagrange Multipliers

Constraints and Lagrange Multipliers is the method for finding extrema (maxima/minima) of a function

f

subject to one or more constraints

g = 0

. Key result: at a constrained extremum, the gradient of the objective is a linear combination of the constraint gradients,

\nabla f = \sum_{i} λ_{i} \nabla g_{i}

. Applies to optimization in economics, physics, engineering, machine learning, and any problem where variables are not free but tied together. Who/when: Joseph-Louis Lagrange, 1788, in Mécanique analytique.

A constrained optimization problem asks for the extreme values of a smooth function $f (x)$ when $x$ is restricted to a surface (or curve) defined by equations $g_{i} (x) = 0$ . The method of Lagrange multipliers transforms this into an unconstrained problem by introducing auxiliary scalars $λ_{i}$ (the multipliers) and solving a system in which the objective's gradient is balanced against the constraint gradients. It matters because almost every real optimization — minimizing cost subject to a budget, finding the shape of a hanging chain, maximizing entropy subject to fixed energy — is constrained, not free. The surprising fact: the multiplier $λ$ is not mere bookkeeping; it equals the sensitivity of the optimal value to a relaxation of the constraint (the "shadow price"), giving it deep physical and economic meaning.

The Core Insight

At a constrained extremum, you cannot improve $f$ by moving along the constraint surface. This means the directional derivative of $f$ vanishes in every direction tangent to the surface. Equivalently, $\nabla f$ has no component along the surface — it must point entirely perpendicular to it. Since the constraint gradients $\nabla g_{i}$ span exactly that perpendicular (normal) space, $\nabla f$ must be a linear combination of them. The coefficients are the Lagrange multipliers.

Formal Statement

Constrained Optimization Problem

Given a smooth objective $f : R^{n} \to R$ and constraint functions $g_{1}, \dots, g_{m} : R^{n} \to R$ (with $m < n$ ), the problem is to find the extrema of $f (x)$ subject to

g_{i} (x) = 0, i = 1, 2, \dots, m .

⟳Each equation carves out an

(n - 1)

-dimensional hypersurface; their intersection is the

(n - m)

-dimensional feasible set on which

f

is optimized.

Here $x = (x_{1}, \dots, x_{n})$ , $f$ is the objective function, each $g_{i}$ is a constraint, and the feasible set is $S = {x : g_{i} (x) = 0 \forall i}$ .

The Lagrangian

The Lagrangian is the auxiliary function of $n + m$ variables

L (x, λ) = f (x) - i = 1 \sum m λ_{i} g_{i} (x),

⟳Subtracting

λ_{i} g_{i}

encodes the constraints; differentiating

L

λ_{i}

recovers

g_{i} = 0

, and in

x

recovers the gradient condition.

where the scalars $λ_{1}, \dots, λ_{m}$ are the Lagrange multipliers. The sign convention ( $-$ vs $+$ ) is arbitrary and only flips the sign of $λ_{i}$ .

Method of Lagrange Multipliers

Let $f$ and $g_{1}, \dots, g_{m}$ be continuously differentiable near a point $x^{*}$ that is a local extremum of $f$ on $S$ . Suppose the constraint qualification holds: the gradients $\nabla g_{1} (x^{*}), \dots, \nabla g_{m} (x^{*})$ are linearly independent. Then there exist unique multipliers $λ_{1}^{*}, \dots, λ_{m}^{*}$ such that

\nabla f (x^{*}) = i = 1 \sum m λ_{i}^{*} \nabla g_{i} (x^{*}) .

⟳

\nabla f

must be orthogonal to the tangent space

T_{x^{*}} S

; this orthogonal complement is spanned by the

\nabla g_{i}

Equivalently, $(x^{*}, λ^{*})$ is a stationary point of the Lagrangian: $\nabla_{x} L = 0$ and $\nabla_{λ} L = 0$ , giving the system

\frac{\partial L}{\partial x _{j}} = 0 (j = 1, \dots, n), \frac{\partial L}{\partial λ _{i}} = - g_{i} = 0 (i = 1, \dots, m) .

⟳The

x_{j}

-equations give [3] component-wise; the

λ_{i}

-equations restore the original constraints [1].

Why It Holds

Gradient Orthogonality at a Constrained Extremum

We prove the single-constraint case ( $m = 1$ ); the general case follows by the same argument applied to the tangent space.

Step 1 — Parametrize the constraint surface. Near

x^{*}

, let

r (t)

be any smooth curve lying in

S

with

r (0) = x^{*}

. Since

r (t) \in S

, we have

g (r (t)) = 0

for all

t

. Step 2 — Differentiate the constraint. Apply the chain rule at

t = 0

\frac{d}{d t} g (r (t))_{0} = \nabla g (x^{*}) \cdot r^{'} (0) = 0.

⟳

g \circ r \equiv 0

, so its derivative is zero; chain rule turns this into a dot product. Hence

\nabla g ⊥

every tangent vector.

So $\nabla g (x^{*})$ is orthogonal to every tangent direction $r^{'} (0)$ — i.e. orthogonal to the tangent space $T_{x^{*}} S$ .

Step 3 — Use that $x^{*}$ is an extremum. The function

h (t) = f (r (t))

has a local extremum at

t = 0

, so

h^{'} (0) = 0

\frac{d}{d t} f (r (t))_{0} = \nabla f (x^{*}) \cdot r^{'} (0) = 0.

⟳A differentiable function at an interior extremum has zero derivative; chain rule converts to the gradient dotted with the tangent.

So $\nabla f (x^{*})$ is also orthogonal to every tangent direction.

Step 4 — Conclude collinearity. Both

\nabla f

and

\nabla g

lie in the one-dimensional orthogonal complement of

T_{x^{*}} S

(since the surface is

(n - 1)

-dimensional, its normal space is one-dimensional). Two vectors in a 1-D space are parallel: there exists a scalar

λ^{*}

with

\nabla f (x^{*}) = λ^{*} \nabla g (x^{*}),

⟳

\nabla g \neq = 0

by the constraint qualification spans the normal line;

\nabla f

lies on that line, so it is a scalar multiple.

□

provided $\nabla g (x^{*}) \neq = 0$ , which is exactly the constraint qualification.

Joseph-Louis Lagrange | 1736–1813

Born in Turin as Giuseppe Lodovico Lagrangia, Lagrange reformulated all of mechanics in Mécanique analytique (1788) with the boast that the book contained not a single diagram — everything followed from the calculus of variations and his multipliers. He introduced the multiplier technique precisely to handle constraints in mechanical systems (rigid rods, surfaces of contact) without solving for the constraint forces explicitly. The multipliers, it turned out, are the constraint forces. Lagrange succeeded Euler at the Berlin Academy and later became a founding professor at the École Polytechnique in Paris.

Building Intuition

Picture standing on a hillside ( $f$ = altitude) but confined to a winding path ( $g = 0$ ). To find the highest point of your path, walk along it: whenever the path still climbs or descends, you are not at the top. You stop where the path is momentarily level — where the slope of the terrain has no component along the path. At that instant the steepest-ascent direction $\nabla f$ points straight across the path, perpendicular to it. Since $\nabla g$ also points perpendicular to the path (it's the normal to the constraint curve), the two arrows line up. The multiplier $λ$ is just the ratio of their lengths.

A second picture: tangent level curves. The constrained extremum occurs where a level curve of $f$ is tangent to the constraint curve. If they crossed transversally, you could slide along the constraint into a higher level curve. Tangency forces the gradients (both normal to their respective curves) to be parallel.

At a constrained maximum, the level curve of $f$ kisses the constraint curve; the gradients align.

Worked Examples

Rectangle of Maximum Area Inscribed in a Circle

Given: Maximize the area

f (x, y) = 4 x y

of a rectangle with corners

(\pm x, \pm y)

inscribed in the circle of radius

5

, so the constraint is

g (x, y) = x^{2} + y^{2} - 25 = 0

, with

x, y > 0

. Step 1 — Gradients.

\nabla f = (4 y, 4 x)

, and

\nabla g = (2 x, 2 y)

. Step 2 — Lagrange equations

\nabla f = λ \nabla g

4 y = λ (2 x), 4 x = λ (2 y) .

Step 3 — Eliminate $λ$ . From the first,

λ = 2 y / x

; from the second,

λ = 2 x / y

. Setting equal:

2 y / x = 2 x / y \Rightarrow y^{2} = x^{2} \Rightarrow y = x

(positive root). Step 4 — Apply the constraint.

x^{2} + x^{2} = 25 \Rightarrow x^{2} = 12.5 \Rightarrow x = y = 12.5 = \frac{5}{2} \approx 3.536

. Step 5 — Evaluate. Area

= 4 x y = 4 \cdot 12.5 = 50

. (It's a square — as expected.)

Max area = 50 square units, x = y = \frac{5}{2}, λ = 2.

Closest Point on a Plane to the Origin

Given: Find the point on the plane

2 x + 3 y + z = 14

nearest the origin. Minimize squared distance

f (x, y, z) = x^{2} + y^{2} + z^{2}

subject to

g = 2 x + 3 y + z - 14 = 0

. Step 1 — Gradients.

\nabla f = (2 x, 2 y, 2 z)

\nabla g = (2, 3, 1)

. Step 2 — Lagrange equations

\nabla f = λ \nabla g

2 x = 2 λ, 2 y = 3 λ, 2 z = λ \Rightarrow x = λ, y = \frac{3}{2} λ, z = \frac{1}{2} λ .

Step 3 — Substitute into the constraint.

2 (λ) + 3 (\frac{3}{2} λ) + \frac{1}{2} λ = 14 \Rightarrow 2 λ + \frac{9}{2} λ + \frac{1}{2} λ = 14 \Rightarrow 7 λ = 14 \Rightarrow λ = 2.

Step 4 — Recover the point.

x = 2, y = 3, z = 1

. Step 5 — Distance.

∥ (2, 3, 1) ∥ = 4 + 9 + 1 = 14 \approx 3.742

Nearest point = (2, 3, 1), distance = 14 units .

(Note: the answer is

14/∥\nabla g ∥ = 14/ 14 = 14

— consistent with the point-to-plane formula.)

Common Misconceptions

The method finds critical points, not maxima. Equation [3] is a necessary condition for an interior extremum, not sufficient. Solving the Lagrange system gives candidates; you must then compare

f

-values among them (and check boundaries/non-smooth points) or use a bordered Hessian second-order test to classify each as max, min, or saddle. On a compact feasible set, the extreme value theorem guarantees the global max and min occur among the critical points, so comparison alone suffices. The sign and size of $λ$ are meaningful.

λ

is not arbitrary slack. By the envelope theorem,

λ^{*} = \partial f^{*} / \partial c

where the constraint is

g = c

: it is the rate at which the optimal value improves as the constraint is loosened — the shadow price. Failure when the constraint qualification breaks. Minimize

f (x, y) = x

subject to

g (x, y) = y^{2} - x^{3} = 0

(a cusp curve). The minimum of

x

on the curve is at the origin

(0, 0)

, since

x = y^{2/3} \geq 0

. But there

\nabla g = (- 3 x^{2}, 2 y) = (0, 0)

— the constraint gradient vanishes. The Lagrange equation

\nabla f = λ \nabla g

becomes

(1, 0) = λ (0, 0)

, which has no solution for any

λ

. The genuine constrained minimum is invisible to the method because the constraint qualification (nonvanishing, linearly independent

\nabla g_{i}

) fails at the cusp. Lesson: always check that the constraint gradients are independent at candidate points, and inspect singular points of the feasible set separately.

Connections

The method is the equality-constrained special case of the Karush-Kuhn-Tucker Conditions, which add inequality constraints $h_{j} (x) \leq 0$ with sign-restricted multipliers and complementary slackness. It is the finite-dimensional shadow of the Euler-Lagrange Equation in the Calculus of Variations, where one extremizes a functional subject to integral or pointwise constraints (here $λ$ becomes a function). Geometrically it rests on Gradient and Directional Derivative (the gradient is normal to level sets) and the Tangent Space of a manifold. In mechanics, the multipliers reappear as constraint forces in Lagrangian Mechanics, and in convex optimization they generate the Lagrangian Duality that underpins linear and quadratic programming.

Real-World Applications

Economics — Utility maximization. A consumer maximizes utility $U (x, y)$ subject to the budget line $p_{x} x + p_{y} y = I$ . The multiplier $λ$ is the marginal utility of income (extra happiness per extra dollar).
Thermodynamics — Maximum entropy. Deriving the Boltzmann distribution: maximize entropy $S = - \sum p_{i} ln p_{i}$ subject to $\sum p_{i} = 1$ and $\sum p_{i} E_{i} = ⟨ E ⟩$ . The multiplier on the energy constraint is $β = 1/ k_{B} T$ — inverse temperature.
Machine learning — SVMs and regularization. Support vector machines solve a constrained quadratic program; the multipliers identify the support vectors. Penalized regression (ridge/LASSO) is dual to a constrained least-squares problem.
Engineering — Structural design. Minimizing material/weight subject to stress or deflection constraints; multipliers give sensitivity of optimal weight to each constraint (used in topology optimization).
Physics — The catenary. A hanging chain minimizes potential energy subject to fixed length; the multiplier sets the curve's tension scale, yielding $y = a cosh (x / a)$ .
Operations research — Resource allocation. Maximizing output subject to limited labor and capital; multipliers are shadow prices guiding where to invest the marginal resource.
Statistics — Method of maximum likelihood with constraints, and deriving estimators (e.g. the constrained MLE for multinomial probabilities) via the normalization constraint $\sum p_{i} = 1$ .

Historical Development

1755 — At age 19, Lagrange writes to Euler outlining the analytic ( $δ$ ) calculus of variations, the seedbed of the multiplier idea.
1760s — Euler and Lagrange develop variational methods; constraints handled ad hoc.
1788 — Lagrange publishes Mécanique analytique, systematically introducing multipliers to enforce mechanical constraints, identifying them with constraint forces.
1797 — Lagrange's Théorie des fonctions analytiques refines the analytic foundations of the differential calculus he uses.
1813 — Lagrange dies in Paris; his methods are by now standard in continental mechanics.
1870s–1900s — The geometric reformulation (gradients normal to level surfaces, tangent spaces) is sharpened with the rise of vector analysis (Gibbs, Heaviside).
1939 — William Karush, in his master's thesis, derives optimality conditions with inequality constraints — generalizing Lagrange.
1951 — Harold Kuhn and Albert Tucker independently rediscover and popularize these conditions; the modern KKT framework is born.
1950s–60s — Linear/nonlinear programming and duality theory (Dantzig, von Neumann, Kuhn–Tucker) place multipliers at the center of optimization.

Subtleties & Limits

When does the method guarantee a global optimum? For general nonconvex

f

or nonconvex feasible sets, the Lagrange conditions only locate stationary points, and there may be exponentially many; certifying global optimality is NP-hard in general. Convexity rescues us: if

f

is convex and the feasible set convex, any KKT point is a global minimizer, and strong duality holds under a constraint qualification (e.g. Slater's condition). Open and active areas include: tight duality gaps for nonconvex problems (e.g. when does the semidefinite relaxation match the true optimum?), multiplier theory on non-smooth or stratified feasible sets (where constraint qualification fails on a measure-zero singular set, as in the cusp counterexample), and robust/distributionally-robust optimization where the multipliers themselves become uncertain objects. In infinite dimensions, the existence and regularity of multipliers (Lagrange multiplier rules in Banach spaces) require subtle hypotheses and remain a fertile research domain.