Attention from scratch

Attention answers a question the network from rung 44 cannot even ask: given this position in a sequence, which other positions matter? A fixed weight matrix has to decide that in advance, identically for every input. Attention decides it per input, from the input itself.

The mechanism is rung 28's dot product doing the work. Each position produces three vectors — a query, a key, a value. To decide how much position i should draw from position j, take the dot product of i's query with j's key. Large when they point the same way, near zero when they are orthogonal. Rung 43's softmax turns that row of scores into weights that sum to 1, and the output at i is the weighted average of every value vector.

That is it. softmax(Q @ K.T / sqrt(d_k)) @ V, and every symbol in it is something you have already built.

Why this is on the ladder

Because it is the last structural idea in the architecture that currently dominates the field, and because it is genuinely simple once the linear algebra is in place. Almost everyone meets it as a diagram and a library call, which leaves the scaling factor and the shapes as folklore. You are in a position to derive them.

Do this

Build it on a sequence you can predict. Make three or four positions with hand-chosen query and key vectors — one pair deliberately aligned, one pair orthogonal — and work out on paper which attention weights should be large. Then run your implementation and check it agrees. If you cannot predict the output, you are testing nothing.

Then the scaling factor, which is the part worth your attention. Take random query and key vectors with independent components of unit variance. Their dot product has variance equal to d_k, so as the dimension grows the scores spread out, and rung 43's softmax on widely spread inputs returns something almost one-hot. Measure it: compute the attention weights at d_k of 4, 64 and 1024, unscaled, and watch the maximum weight climb toward 1. Then divide by sqrt(d_k) and watch the distribution stay sensible at every dimension. The gradient through a saturated softmax is nearly zero, which is why an unscaled model trains badly rather than merely differently.

Add causal masking next. Set the scores above the diagonal to negative infinity before the softmax, so a position cannot draw from positions after it. Confirm by changing a later token and checking that earlier outputs do not move.

Multiple heads are less than they sound. Split the model dimension into h chunks, run the same computation independently on each, concatenate the results, and pass them through one more linear layer. Different heads can attend to different things because they see different projections.

Then assemble the block: attention, residual add, layer normalisation, a feed-forward layer that expands and contracts, another residual add and normalisation. Check that input and output shapes match — that is what makes the block stackable, and it is the whole reason the architecture scales.

Where people get stuck

Transposing K incorrectly and getting a matrix that is still square, still runs, and computes the wrong thing. Attention scores are (seq, seq). Verify each row sums to 1 after the softmax.

Masking after the softmax instead of before. Zeroing weights afterwards leaves the remaining ones no longer summing to 1, and the leak is small enough to look like noise.

Expecting attention to be interpretable. The weights show what was read, not why. A head with a tidy diagonal pattern is a pleasing picture and not an explanation — treat it as rung 24's lesson again, that a computation can be evidence and never proof.