First PrinciplesStart anywhere. Prove it, then move on.

Rung 41

Convexity and gradient descent

Rolling downhill, and when you can trust where you land.

Best attempted after 33. The gradient. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Implement gradient descent from scratch on a convex function and show it reaches the minimum. Then run it on a non-convex one from several starting points and show it lands in different places. Then add momentum and compare the number of steps to a chosen tolerance. Finally, raise the learning rate until it diverges, and report the value where that begins.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

Rung 33 gave you the arrow pointing uphill. Turn around and take a small step. Repeat. That is gradient descent, and it is how essentially every model is trained.

The interesting question is not how to do it but when it works. A convex function has a single valley — any local minimum is the global one, so arriving anywhere flat means arriving at the answer. Non-convex functions have many valleys, and where you finish depends on where you began.

Neural networks are emphatically non-convex.

Why this is on the ladder

Because this is the training loop. Rung 45 differs only in scale: the function is the loss, the variables are the weights, and the gradient comes from backpropagation instead of by hand. Everything about the loop's behaviour — learning rates, momentum, why two runs disagree — is visible here in two dimensions, where you can watch it.

Do this

Start convex: f(x) = x². Implement the loop — compute the gradient, step against it scaled by a learning rate, repeat. From any start it converges to zero.

Now vary the learning rate. Small: correct but slow. Larger: faster. Larger still: it overshoots, oscillates, and diverges to infinity. Find the value where that begins and report it. For this function the threshold is sharp and computable, and hitting it deliberately is worth more than being warned about it.

Now non-convex: something with two valleys of different depths. Run from several starting points. Some runs find the deep valley, some settle in the shallow one. Same code, same function, different answers — and this is normal, not a bug. It is why "train it again and see" is a real technique.

Then add momentum: carry a fraction of the previous step into the current one. Compare step counts to reach a tolerance on a stretched, narrow function, where plain descent zig-zags across the valley and momentum damps the oscillation.

Where people get stuck

Blaming the code when two runs disagree. On a non-convex surface disagreement is the expected behaviour, and the gate makes you produce it deliberately so it is not alarming later.

The other trap is tuning the learning rate by feel without ever finding the divergence threshold. Knowing where the cliff is tells you far more than knowing one value that happened to work.

Reading