First PrinciplesStart anywhere. Prove it, then move on.

Rung 22

The chain rule

The one rule every neural network is built out of.

Best attempted after 21. Rules of differentiation. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Differentiate a three-deep composition by hand, then verify it numerically at a specific point — the two must agree to at least four decimal places. Then take a chain of five simple functions, compute the derivative as a product of five factors, and say what happens to that product when each factor is around 0.1.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

This is the most important rung on the ladder. Everything from rung 44 onward is this rule applied at scale, and a shaky understanding here produces failures much higher up whose cause is very hard to find.

The rule itself is short. If a change in x moves u at some rate, and a change in u moves y at some rate, then the rate at which x moves y is the two rates multiplied.

Why this is on the ladder

Because a neural network is a composition. Input goes through a layer, that output goes through the next layer, and so on to a number measuring the error. Asking "how does a weight in the first layer affect the final error?" is asking for a derivative through the whole composition — which is a product of one factor per layer.

That is backpropagation. Not an analogy for it: it is it. Rung 45 writes this out in code, and the code will make sense in proportion to how solid this rung is.

Do this

Take y = sin(3x² + 1). Three layers: square-and-scale, add one, take sine. Peel them apart, differentiate each, multiply. Write every factor down separately before combining — the discipline of naming the layers is what scales.

Then verify at x = 0.7: compute your symbolic answer, and compute the numerical slope with a small h. Four decimal places of agreement, or you have an error to find.

Now the part that matters most. Take five functions chained together, each with derivative about 0.1 at the point of interest. The overall derivative is their product: about 0.00001. Five layers, and the signal has nearly vanished.

That is the vanishing gradient, and you have just derived the central difficulty in training deep networks from a rule you can hold in one hand. When rung 45's network trains slowly, this is the first thing to suspect.

Where people get stuck

Forgetting the inner derivative — writing cos(3x² + 1) and stopping, without the 6x. The numerical check catches this instantly, which is why it is in the gate rather than offered as advice.

The other difficulty is identifying the layers at all. Read the expression from the outside in, and ask at each step: what is the last thing done to produce this? Practise on expressions you have no intention of differentiating; naming the structure is the skill.

Reading