Critical Points and Extrema

Critical point analysis via the vanishing gradient is the entry point for optimization in every quantitative discipline. In machine learning, training a model by minimizing a loss function $L(\mathbf w)$ over weights $\mathbf w$ relies on the same first-order condition: gradient descent moves iteratively toward points where $\nabla L \approx \mathbf 0$, and understanding saddle points (which are ubiquitous in high-dimensional loss landscapes) is central to why plain gradient descent can stall. In economics, utility- and profit-maximization problems set the gradient of a utility or profit function to zero to find optimal consumption or production bundles, often then layering budget constraints via Lagrange Multipliers. In physics, equilibrium configurations of a system (e.g. the shape of a hanging chain, or the resting position of a mechanical system) are critical points of a potential energy function, with stability determined by the second-order behavior mentioned above via The Hessian Matrix. In engineering design, minimizing material stress, cost functions, or drag subject to structural constraints again begins by finding the critical points of a differentiable objective. In statistics, maximum-likelihood estimation finds parameter values maximizing a likelihood function, again by solving $\nabla_{\theta} \ell(\theta) = \mathbf 0$ for the log-likelihood $\ell$. And in numerical analysis and scientific computing, root-finding methods (Newton's method in several variables) are frequently deployed precisely to solve the critical-point equation $\nabla f = \mathbf 0$ when it cannot be solved in closed form, as illustrated in the boundary calculation of Example 2.

Ask the tutor: How does the second-derivative (Hessian) test decide whether a critical point is a max, min, or saddle?

In one variable, a smooth curve $y = f(x)$ has a horizontal tangent line at a local max or min — the derivative vanishes because the curve momentarily "levels off." In several variables, the graph of $f$ is a surface (or hypersurface), and the analogous statement is that the tangent plane is horizontal: every directional derivative at $\mathbf{a}$ vanishes, which forces the entire gradient vector to vanish, since the gradient encodes all directional information at once. Picture a mountain range. At a summit, walking in any direction takes you downhill (or keeps you level, right at the peak) — there is no direction of ascent. At a mountain pass (a saddle point between two peaks), the gradient is also zero, yet walking along the ridge keeps you level or takes you up, while walking across the ridge takes you down. The gradient vanishing only tells you that to first order, no direction is preferred; it cannot by itself distinguish a summit from a pass from a basin. That distinction needs second-order information — the subject of The Hessian Matrix. The other crucial geometric fact: this entire argument is a purely interior phenomenon. If you are standing at the edge of a plateau that has been cut off by a fence (the boundary of the domain), the ground might slope steeply upward just beyond the fence — but you cannot walk there. The function can be increasing at the boundary point in the "forbidden" direction, so the gradient there need not vanish at all, yet the point can still be a genuine (constrained) local max or min of $f$ restricted to the domain.

Example 1: A saddle disguised in cross-terms. Let $f(x,y) = x^3 - 3xy^2 + 2y$ on $\mathbb{R}^2$. Find all critical points and determine, using only elementary reasoning (not yet the Hessian test), which ones cannot be extrema. Solution. Compute the gradient:

Setting both to zero:

Case $y = x$: then $x^2 = 1/3$, so $x = \pm 1/\sqrt3$, giving points $\left(\tfrac{1}{\sqrt3}, \tfrac{1}{\sqrt3}\right)$ and $\left(-\tfrac{1}{\sqrt3}, -\tfrac{1}{\sqrt3}\right)$. Case $y = -x$: then $xy = -x^2 = 1/3$ forces $x^2 = -1/3$, impossible for real $x$. So this case contributes no real solutions. Thus $f$ has exactly two critical points:

Now examine $P_1$ directly. Along the line $y = x$ near $P_1$, write $x = \tfrac{1}{\sqrt3} + t$, $y = \tfrac{1}{\sqrt3} + t$:

Let $h(x) = -2x^3 + 2x$. Then $h'(x) = -6x^2 + 2$, and at $x_0 = 1/\sqrt3$, $h'(x_0) = -6(1/3) + 2 = 0$ — consistent with $P_1$ being critical along this slice too. But $h''(x) = -12x$, so $h''(x_0) = -12/\sqrt3 < 0$: along the line $y=x$, $f$ has a strict local max at $P_1$. Now look along the line $x = \tfrac{1}{\sqrt3}$ fixed, varying $y$ near $y_0 = 1/\sqrt3$:

Then $k'(y) = -2\sqrt3 y + 2$, and $k'(y_0) = -2\sqrt3\cdot \tfrac{1}{\sqrt3} + 2 = -2+2=0$, again consistent. But $k''(y) = -2\sqrt3 < 0$ as well — so along this line it is also a local max in $y$ alone... yet this single-variable check is not enough to conclude a genuine 2D local max, since we must test all directions, not just two. Test the direction $y = x + s$ for small deviations transverse to the diagonal, i.e. approach $P_1$ along $x = \tfrac{1}{\sqrt3}+t,\ y = \tfrac{1}{\sqrt3} - t$ (a direction not equal to $y=x$). Compute $f$ along this path to second order in $t$. With $x = a+t,\ y=a-t$ where $a = 1/\sqrt3$:

Expand: $(a+t)^3 = a^3+3a^2t+3at^2+t^3$, and $(a-t)^2 = a^2 - 2at+t^2$, so

So

At $a = 1/\sqrt3$: $6a^2 = 2$, so the linear-in-$t$ coefficient is $6a^2t - 2t = 2t-2t=0$ ✓ (confirms critical point). The quadratic term is $6at^2 = 6\cdot\tfrac1{\sqrt3} t^2 = 2\sqrt3\, t^2 > 0$ for $t \ne 0$. So along this transverse direction, $f$ increases near $P_1$ (to second order), while along the diagonal $y=x$ direction we found $f$ decreases (a max, $h''<0$). Since $f$ increases in one direction and decreases in another arbitrarily close to $P_1$, $P_1$ is not a local extremum at all — it is a saddle point. By an identical computation (or by the symmetry $f(-x,-y) = -f(x,y)$, which sends critical points to critical points with the roles of increase/decrease swapped), $P_2$ is also a saddle point. This example is exactly why Fermat's theorem is only a necessary condition: both critical points exist as the theorem guarantees, but neither is an extremum. Resolving this efficiently — without hand-testing several directions — is precisely the motivation for The Hessian Matrix.

Example 2: Interior vs. boundary on a closed disk. Let $f(x,y) = x^2 + 2y^2 - 4x - 4y$ on the closed disk $D = \{(x,y) : x^2+y^2 \le 9\}$. Find the global maximum and minimum of $f$ on $D$. Solution. $D$ is closed and bounded (compact), and $f$ is continuous (indeed a polynomial), so by the Extreme Value Theorem, $f$ attains a global max and global min somewhere on $D$. Any such extremum occurs either at an interior point of $D$ (the open disk $x^2+y^2<9$) or on the boundary circle $x^2+y^2=9$. Interior candidates. On the open disk, $f$ is differentiable everywhere, so by the theorem just proved, any interior extremum must be a critical point. Compute:

Setting this to zero: $x = 2$, $y=1$. Check this lies in the open disk: $2^2+1^2 = 5 < 9$. Yes. So $(2,1)$ is the unique interior critical point, with

Boundary candidates. On the boundary circle, parametrize $x = 3\cos\theta,\ y = 3\sin\theta$, $\theta \in [0,2\pi)$. Then

(Using $9\cos^2\theta = 9-9\sin^2\theta$.) This reduces the search to a one-variable problem on the boundary — the general strategy for constrained extrema, later formalized via Lagrange Multipliers. Differentiate:

Rather than solve this transcendental equation symbolically, note $f'(\theta)=0$ defines the boundary's own critical points; a numerical solve (Newton's method from a few seed points) locates them near $\theta \approx 0.4636$ rad and $\theta \approx 3.605$ rad (verify: at $\theta=0.4636$, $\cos\theta\approx0.8944,\ \sin\theta\approx0.4472$ — this is the direction of the point $(3\cdot0.8944, 3\cdot0.4472) = (2.683, 1.342)$, i.e. the boundary point in the exact direction from the origin through $(2,1)$ extended to radius $3$, since $(2,1)$ has direction angle $\arctan(1/2)\approx 0.4636$). Indeed this makes geometric sense: $f$'s interior critical point $(2,1)$ is the unconstrained minimum of the paraboloid-like function $f$, and the boundary point diametrically farthest from it in that radial direction, $\theta\approx 0.4636+\pi \approx 3.605$, is a natural candidate for the boundary max, while the near point $\theta\approx0.4636$ is a candidate for the boundary min. Evaluate $f$ at these two boundary points:

• Near point $(x,y) \approx (2.683, 1.342)$: $f \approx (2.683)^2 + 2(1.342)^2 - 4(2.683) - 4(1.342) \approx 7.20 + 3.60 - 10.73 - 5.37 \approx -5.30$.
• Far point $(x,y) \approx (-2.683,-1.342)$: $f \approx 7.20+3.60+10.73+5.37 \approx 26.90$.

We also must check the endpoints of the $\theta$-range are not separately extremal — but since $\theta$ ranges over a full circle (a closed loop with no boundary of its own), the only boundary-of-$D$ candidates are these two critical $\theta$'s (a continuous periodic function on a circle attains its extrema only at its own critical points, by the same Fermat argument applied to the $1$-variable function $f(\theta)$ on $\mathbb R/2\pi\mathbb Z$). Comparing all candidates: $f(2,1) = -6$ (interior), $f \approx -5.30$ (boundary near point), $f\approx 26.90$ (boundary far point). The global minimum on $D$ is $-6$ at the interior point $(2,1)$, and the global maximum is approximately $26.90$ at the boundary point near $(-2.683,-1.342)$. This example shows both halves of the theorem's scope at once: the interior extremum was correctly found via $\nabla f = 0$, but the true global maximum lived on the boundary, where the gradient condition simply does not apply and a separate one-dimensional analysis was required.