First PrinciplesStart anywhere. Prove it, then move on.

Rung 44

Forward pass from scratch

A network is a stack of matrix multiplies with a nonlinearity between them.

Best attempted after 29. Matrices as transformations, 35. Graphs, DAGs, tensors and 43. Entropy, cross-entropy, KL divergence. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Write a two-layer network's forward pass in NumPy with no framework: weights, biases, one hidden nonlinearity, softmax at the output. Feed it a batch of at least 32 inputs at once and print the shape of every intermediate. Then show that removing the nonlinearity collapses the whole network to a single matrix, by multiplying the two weight matrices together and checking the outputs agree to floating-point tolerance.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

Everything you have built since rung 28 arrives here. A neural network's forward pass is a matrix multiply, an addition, and a function applied elementwise — repeated. There is no fourth idea. The mystique is entirely in the scale and the training; the computation itself is rung 29's linear map with a kink in it.

The kink is the point. Rung 29 showed that composing linear transformations gives you another linear transformation, so a hundred stacked matrix multiplies can do no more than one. Insert a nonlinear function between them and that collapse stops being possible. That single fact is why activations exist, and it is what your gate measures.

Write it yourself once and the shapes stop being mysterious. Write it only in a framework and you will spend the rest of your time guessing which axis is which.

Why this is on the ladder

Because rung 45 asks you to differentiate this by hand, and you cannot differentiate a computation you cannot write down. The forward pass is also the thing you will debug for the rest of your working life: a wrong answer is nearly always a wrong shape, and a wrong shape is nearly always a silent broadcast rather than an error.

Do this

Start with one layer. Take an input batch X of shape (N, d_in) and weights W of shape (d_in, d_out). Compute X @ W + b, where b has shape (d_out,) and broadcasting adds it to every row. Confirm you understand why the batch dimension comes first and why it never appears in a weight matrix.

Now the activations. Implement three yourself:

  • ReLU: max(0, z). One line, no exponentials, and it is what almost everything uses.
  • Sigmoid: 1 / (1 + exp(−z)). Squashes to (0, 1). Plot it and find where its slope goes flat — you will meet that flatness again as a training failure.
  • Softmax: exponentiate, then divide by the sum along the class axis. It turns a row of arbitrary numbers into a distribution, which is exactly what rung 43's cross-entropy expects to be handed.

Softmax needs one defensive trick. Subtract the row maximum before exponentiating. The result is mathematically identical — the constant cancels top and bottom — but exp(1000) overflows to infinity and exp(0) does not. Try it both ways on inputs around 800 and watch the naive version produce nan. This is rung 20's floating-point floor showing up in a new place.

Then stack two layers with ReLU between them and softmax at the end, and print every intermediate shape. Make the printing a habit, not a one-off: a shape line after each step is the cheapest debugging tool in this subject.

Finally, the collapse experiment. Replace ReLU with the identity, run the network, then compute W1 @ W2 and run the single resulting matrix on the same input. The outputs match. Depth without nonlinearity buys nothing at all.

Where people get stuck

Silent broadcasting. Adding a bias of shape (N,) instead of (d_out,) does not raise an error when N happens to equal d_out — it transposes your intent and returns plausible nonsense. Assert your shapes rather than trusting them.

Softmax along the wrong axis is the other classic. Sum your output rows and confirm each is 1. If your columns sum to 1 instead, you have normalised across the batch, which quietly makes every prediction depend on the other examples in it.

And confusing "the network" with "the trained network". Nothing here learns yet. The weights are random, the answers are wrong, and that is correct for this rung.

Reading