First PrinciplesStart anywhere. Prove it, then move on.

Rung 33

The gradient

The arrow that points straight uphill — and therefore, straight downhill.

Best attempted after 32. Partial derivatives. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Compute a gradient by hand and verify every component numerically. Then prove to yourself that it points the steepest way: sample a few hundred random unit directions at the same point, measure the rate of change along each, and confirm none beats the gradient's own direction. Report the best random direction's rate as a fraction of the gradient's.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

Rung 32 left a gap. The partials tell you the slope along each axis, but you can walk in any direction, and most directions are not axes. So: of all the ways you could step from here, which one climbs fastest?

Collect the partials into a vector. That vector is the gradient, and it answers the question exactly. Its direction is the steepest ascent. Its length is how steep that is.

That this works is not obvious, and it is worth being surprised by. Two numbers describing two particular directions somehow determine the best of infinitely many.

Why this is on the ladder

Because this is the rung that AI training actually runs on. You have an error — one number saying how wrong the model is. You have millions of parameters. The gradient of the error with respect to those parameters is an arrow in a million-dimensional space pointing the direction of fastest increase in error.

So you step the other way. That is gradient descent, that is rung 41, and that is how every model you have ever used was trained. Not a metaphor for it — the mechanism.

Why it points uphill

Worth seeing rather than accepting. The rate of change in a direction u (a unit vector) is the dot product ∇f · u. Rung 28 established that a dot product is largest when the two vectors point the same way. So the rate is maximised exactly when u points along ∇f.

That is the entire proof, and it needs only the dot product. Everything about training rests on it.

Do this

Take f(x, y) = x² + 3y² at the point (1, 1).

By hand: ∂f/∂x = 2x = 2, ∂f/∂y = 6y = 6. So ∇f = (2, 6). Verify each component numerically, as at rung 32.

Now the gate, which is where the belief becomes knowledge. Generate a few hundred random unit vectors. For each, step a tiny distance h from (1, 1) in that direction, measure (f(new) − f(old)) / h, and keep the best. Then do the same along ∇f normalised to length 1.

No random direction beats it. Many come close — those are the ones that happened to point nearly the same way — and the near-misses are as instructive as the result. Report the best random rate as a fraction of the gradient's.

Then step against the gradient a few times, recomputing as you go, and watch f fall. You have just run gradient descent by hand, before meeting the name.

Where people get stuck

Treating the gradient as a slope — a single number. It is a vector: a direction and a magnitude. "The gradient is steep here" means the arrow is long.

The other error is expecting the gradient to point at the minimum. It does not. It points the steepest way from where you are standing, which is local information only. On a curved surface, following it faithfully still traces a curved path, and this is exactly why training takes many small steps instead of one large confident one.

Reading