The gradient points uphill. Climb to peaks, tell maxima from saddles, and find the best point on a constraint with Lagrange multipliers
The gradient is a compass that always points uphill. Follow it and you climb; walk against it and you descend, which is how machines learn. Where it vanishes you are at a peak, a pit or a pass, and the second derivatives tell you which. And when you are confined to a path, the best point on it is where the gradient of the landscape lines up with the gradient of the path.
Click the map to drop hikers who follow the gradient. Change the step size to see them creep, zig-zag or fly off, and switch on the basins to see which starting points lead to which summit.
Each hiker takes the fixed-step update (x, y) ← (x, y) ± η∇f(x, y), three steps per frame, using the exact gradient. The arrows show the direction of ±∇f only, not its size. Basins colour each starting cell of a 56 × 56 grid by the point its hiker settles on with the current η and direction (grey: it left the map or had not settled after 700 steps); hollow rings mark those points. For Rosenbrock and Himmelblau the contour levels are spaced geometrically and the 3D height is log(1 + f − min).
Since D_u f = ∇f · u = |∇f| cos θ, the direction of fastest increase is u = ∇f/|∇f|, and the fastest decrease is the opposite way. Gradient ascent repeats the step
(x, y) ← (x, y) + η ∇f(x, y)
with a fixed step size η (descent uses −η∇f). Augustin-Louis Cauchy proposed this method of steepest descent in 1847, for solving systems of equations. Today, with millions of variables instead of two, it is how neural networks are trained.
The step size matters. Near a minimum f looks like a bowl with curvatures λ₁ ≥ λ₂ > 0 (the eigenvalues of the Hessian). Along each principal direction the error is multiplied by 1 − ηλ at every step, so descent converges only if η < 2/λ₁, and progress along the shallow direction is governed by the factor 1 − ηλ₂. When λ₁/λ₂ is large (a long narrow valley) any η small enough to be stable across the valley crawls along it, and a slightly larger one makes the path zig-zag from wall to wall. Rosenbrock’s banana function (1960) was designed to show exactly this.
Each summit drains a basin of starting points. Himmelblau’s function has four minima, and the basin picture shows the plane carved into four regions whose boundaries run through the saddle points and the local maximum between them.
Every critical point is found and marked by type. Click one to overlay its quadratic approximation, then morph the surface into it. Try the monkey saddle and x⁴ + y⁴, where the test has nothing to say.
Critical points come from Newton’s method on ∇f = 0 started at a 16 × 16 grid of seeds, with duplicates merged, so a point can be missed if no seed is drawn to it. The Hessian entries are exact (symbolic second derivatives). The violet patch is (1 − t)f + tQ on the dashed window, where Q = f(a, b) + ½(f_xx Δx² + 2f_xy ΔxΔy + f_yy Δy²); at a non-degenerate point max |f − Q| shrinks like w³ as the window shrinks. Heights in 3D are rescaled to fit the scene.
At a critical point ∇f = 0, so the tangent plane is horizontal and the first interesting term of Taylor’s formula is quadratic:
f(a + h, b + k) ≈ f(a, b) + ½(f_xx h² + 2f_xy hk + f_yy k²)
The matrix of second partials is the Hessian, named after Otto Hesse. Completing the square in the quadratic form gives the test. Let D = f_xx f_yy − f_xy². If D > 0 and f_xx > 0 the quadric is a bowl and f has a local minimum; if D > 0 and f_xx < 0 it is a dome and f has a local maximum; if D < 0 it curves up one way and down the other, a saddle. In terms of eigenvalues, D is their product: same signs give an extremum, opposite signs a saddle.
If D = 0 the quadric is flat in some direction and the cubic or quartic terms take over. The test is then silent, and genuinely so: x⁴ + y⁴ has a minimum at the origin, −x⁴ − y⁴ a maximum, and the monkey saddle x³ − 3xy² a saddle with three valleys for a monkey’s legs and tail, yet all three have every second partial equal to zero there.
To find the critical points at all, the demo runs Newton’s method on the system f_x = 0, f_y = 0 from a grid of starting points. Each Newton step solves a linear system with the Hessian as its matrix, the same matrix the test then inspects.
Slide the point along the constraint curve and watch the two gradient arrows. Where f is largest or smallest on the curve, they line up and the dashed level curve just touches the constraint.
Arrows show the directions of ∇f (pink) and ∇g (green), not their lengths; λ is computed from the true gradients. The constraint is traced by marching squares and each point is pulled onto g = c by Newton steps; if g = c has several pieces in the window, the longest is used. Constrained extrema are local maxima and minima of f along the curve (end points of an open curve are not counted). “Fence by a river”: fencing on three sides with 4 units (x + 2y = 4) encloses the most area xy at x = 2, y = 1.
To maximise f(x, y) subject to g(x, y) = c, move along the constraint curve. The rate of change of f along it is ∇f · T, where T is the unit tangent. At a constrained maximum or minimum this must be zero, so ∇f is perpendicular to the curve. But ∇g is perpendicular to the curve too, because the curve is a level set of g. Two vectors perpendicular to the same line in the plane are parallel:
∇f = λ ∇g, g(x, y) = c
Three equations for x, y and λ. Joseph-Louis Lagrange used the method in his Mécanique analytique of 1788. For f = xy on x² + y² = 1 they give y = 2λx, x = 2λy, so x² = y² = ½: maxima ½ at (±1/√2, ±1/√2) with matching signs, minima −½ where the signs differ.
The multiplier is more than a by-product. If M(c) is the best value of f on g = c, then dM/dc = λ: it measures how much the optimum improves per unit of extra constraint. In the fence problem, λ = 1 says one more unit of fencing buys about one more unit of area. Economists call λ a shadow price.
Spot a mistake or have an idea?