Backpropagation from scratch

Backpropagation is rung 22's chain rule, applied to a computation with a few thousand intermediate quantities instead of three, and organised so that no derivative is ever computed twice. That is the whole algorithm. It was not obvious historically, and it is worth being precise about why it is not obvious: the chain rule tells you the derivative of a composition, but it does not tell you in which order to evaluate the product, and the order is where all the savings are.

Rung 35's DAG is the object you walk. The forward pass moves along the edges computing values; backpropagation walks the same edges in reverse, carrying one number per node — the derivative of the loss with respect to that node. Each node needs only its local derivative and the number handed back to it from downstream. Nothing has global knowledge.

Rung 33 supplies the reason to want any of this. The gradient is the direction of steepest increase, so its negative is the direction to step. Backpropagation is simply the cheapest known way to get one.

Why this is on the ladder

Because a framework will compute these gradients for you and give you no way to tell a subtly wrong model from a correct one. Every practitioner who can debug training rather than reroll it has, at some point, done this by hand. This rung is that point.

It is also the last rung where the whole computation still fits in your head. Take advantage of that.

Do this

Work out the derivatives on paper first, for the network from rung 44: input, linear layer, ReLU, linear layer, softmax, cross-entropy loss.

Start at the loss and go backward. The first result is the one worth deriving slowly, because it is famously tidy: differentiate rung 43's cross-entropy with respect to the softmax inputs — not its outputs — and the whole mess collapses to probabilities − labels. Do not take that on trust. Derive it, including the awkward case where the class you are differentiating is not the correct class, and watch the terms cancel.

From there each step is mechanical. For Z = A @ W + b, given the gradient dZ flowing back, the gradients are dW = A.T @ dZ, db = dZ.sum(axis=0), and dA = dZ @ W.T. Derive each one rather than copying it, and check the shapes: dW must match W, and if it does not, the transpose you need is the one you left out.

ReLU's derivative is 1 where its input was positive and 0 elsewhere, so backward means multiplying by that mask. Keep the mask from the forward pass rather than recomputing it from the output.

Then verify before you train. For each parameter, perturb it by a small h, recompute the loss twice, and compare (L(θ+h) − L(θ−h)) / 2h against your analytic gradient. Use the relative error, not the absolute one. Pick h around 1e-5, and remember rung 20: too small is as bad as too large. Any gradient that fails this check is a bug you would otherwise spend a week blaming on the learning rate.

Only now, train. Plain gradient descent on minibatches of 64, a learning rate around 0.1, and the initialisation scaled by roughly 1/sqrt(fan_in) will clear 90 percent on MNIST within a few passes over the data. Hold out a test split and score against that, not against what you trained on. Load the digits from OpenML dataset 554 or any local copy; the loader is not the exercise.

Where people get stuck

Forgetting to average over the batch. If your loss sums over 64 examples but your learning rate was chosen for a mean, your effective step is 64 times too large and the loss goes to nan within a few iterations. Decide once, and be consistent between loss and gradient.

Chasing accuracy before checking gradients. A network with one wrong sign still trains — badly, to about 60 percent — and looks like a tuning problem. The numerical check takes ten minutes and settles it.

Saturated sigmoids, if you used them. Where the curve is flat its derivative is near zero, the gradient arriving from downstream gets multiplied by that, and learning stops. This is the vanishing gradient, and seeing it once in your own code explains why ReLU won.