First PrinciplesStart anywhere. Prove it, then move on.

Rung 34

Jacobians and vector fields

Derivatives when both the input and the output have many dimensions.

Best attempted after 33. The gradient. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Compute the Jacobian of a map from two dimensions to two by hand, and verify every entry numerically. Then take a small square near a point, push it through the map, measure the area of the image, and show the ratio matches the determinant of the Jacobian there.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

The gradient handled many inputs and one output. Now allow many outputs too.

Each output has its own gradient. Stack them as rows and you have the Jacobian — a matrix holding every partial of every output with respect to every input. If that sounds like bookkeeping, it is; the content is what the matrix means.

Near any point, a smooth map is approximately a linear transformation, and the Jacobian is that transformation. Rung 29 told you a matrix is something that happens to space. The Jacobian is what this map does to space, locally.

Why this is on the ladder

Because a layer of a network takes many numbers in and gives many numbers out, and the derivative of such a thing is a Jacobian. Rung 45's backpropagation is the chain rule from rung 22, with the multiplications being Jacobians rather than single numbers.

Practical implementations rarely build these matrices — they are enormous, and what is actually needed is a Jacobian times a vector. But knowing what the object is turns that optimisation from mysterious to obvious.

Do this

Take the map (x, y) → (x² − y, 2xy).

Compute all four partials and arrange them as a 2×2 matrix. Verify each entry numerically by nudging one input and watching one output.

Then the geometric check. At (1, 1), take a small square of side s — say 0.001. Push its four corners through the map. The image is very nearly a parallelogram; compute its area. Divide by s². That ratio should equal the determinant of the Jacobian at (1, 1).

That is rung 29's determinant-as-area-scaling, arriving in a setting where the transformation differs from point to point. Try it at a second point and watch the scaling change.

Where people get stuck

Row and column order. Whether outputs index rows or columns is a convention, and conventions differ between texts. Fix yours, write it in a comment, and check the shape before every multiply — at rung 45 a transposed Jacobian produces code that runs and trains nothing.

The other snag is expecting the area check to be exact. It is a limit, so it holds as the square shrinks. Try s = 0.1 and s = 0.001 and watch the agreement improve, exactly as rung 20's numerical derivative did.

Reading