First PrinciplesStart anywhere. Prove it, then move on.

Rung 42

Lagrange multipliers

Optimising when you are not free to go anywhere.

Best attempted after 33. The gradient. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Maximise a function subject to a constraint using Lagrange multipliers, by hand. Verify by sampling many points along the constraint curve and confirming none beats your answer. Then show, at the solution, that the two gradients are parallel — and say what the multiplier's value means.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

Rung 41 walked downhill wherever it liked. Real problems rarely allow that. Find the best allocation given a fixed budget. Find the highest point on this path. The constraint is not an obstacle to the problem; it is part of it.

The insight is geometric and, once seen, hard to unsee. Walk along the constraint curve. As long as the function is still increasing in the direction you are walking, keep going. You stop where the function stops changing along the curve — which happens exactly when its gradient is perpendicular to your direction of travel, that is, parallel to the constraint's gradient.

∇f = λ∇g. The multiplier λ is just the ratio of their lengths.

Why this is on the ladder

Because constrained optimisation is everywhere above it. Rung 43's entropy results are derived by maximising subject to probabilities summing to one. Regularisation is a constrained problem in disguise — limiting how large weights may grow while minimising error.

The multiplier is also meaningful in itself: it is the rate at which the best achievable value improves as the constraint is relaxed. The shadow price of the budget.

Do this

Maximise f(x, y) = xy subject to x + y = 10.

By hand: set ∇f = λ∇g with g = x + y. That gives y = λ and x = λ, so x = y, and the constraint makes both 5. The maximum is 25.

Verify by brute force. Sample a thousand points along x + y = 10 and evaluate xy at each. Nothing beats 25, and the sampled maximum sits at (5, 5).

Then look at the gradients at the solution. ∇f = (y, x) = (5, 5) and ∇g = (1, 1). Parallel, with λ = 5.

Now check what λ means. Change the constraint to x + y = 11 and re-solve: the new maximum is 30.25, an improvement of 5.25. Relaxing by one unit bought roughly λ. Try 10.1 and watch the agreement tighten, because it is a derivative.

Where people get stuck

Solving for λ as though it were the goal. It is scaffolding for finding x and y — and then, separately, it turns out to mean something. Both facts are true and students usually learn only the first.

The other snag is forgetting the constraint equation itself. There are three unknowns and you need three equations: two from the gradient condition and the constraint. Dropping it leaves an underdetermined system, which rung 30 taught you to recognise.

Reading