First PrinciplesStart anywhere. Prove it, then move on.

Rung 43

Entropy, cross-entropy, KL divergence

Measuring surprise, and the price of being wrong about it.

Best attempted after 39. Expectation, variance, covariance. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Compute the entropy of several distributions and show it is largest when the distribution is uniform. Then show numerically that cross-entropy is never below entropy, with equality only when the two distributions match. Then confirm that minimising cross-entropy against one-hot labels is exactly maximising the log probability of the correct answer — and say why that makes it the right loss for a classifier.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

This rung answers a question that sounds unanswerable: how much information is in a message? Shannon's answer is that information is surprise, and surprise can be measured.

An event you were certain of tells you nothing when it happens. An event you thought impossible tells you a great deal. The right measure of the surprise in an event of probability p is −log(p) — zero when p = 1, growing without bound as p approaches zero. Rung 9's logarithm is doing the work.

Entropy is the average surprise of a distribution: H(p) = −Σ p·log(p).

Why this is on the ladder

Because cross-entropy is the loss function used to train essentially every classifier in existence, including every language model. Rung 45 will use it, and using it without knowing what it measures means being unable to tell an unlucky run from a broken one.

The three quantities

Entropy H(p): the average surprise if the world really is p. The irreducible uncertainty.

Cross-entropy H(p, q) = −Σ p·log(q): your average surprise when the world is p but you believed q. You are surprised according to your own beliefs, but those surprises occur at the world's frequencies.

KL divergence D(p‖q) = H(p, q) − H(p): the excess — how much extra surprise your wrong belief costs you. It is never negative, and it is zero only when q equals p. That last fact is what makes minimising cross-entropy a sensible thing to do: since H(p) is fixed by the world, pushing cross-entropy down pushes your beliefs toward the truth and nowhere else.

Do this

Compute entropy for a fair coin (1 bit), a biased coin at 0.9 (about 0.47 bits), and a certain outcome (0 bits). Then a fair six-sided die, and a loaded one. In every pair, the uniform case is larger — maximum uncertainty means maximum average surprise.

Now fix a true p and vary a guess q, computing H(p, q) at each step. Plot it. The minimum sits exactly at q = p, and the value there is H(p). Nowhere does the curve dip below. Compute D(p‖q) alongside and confirm it is never negative.

Then the classifier case. With one-hot labels, p is 1 on the correct class and 0 elsewhere, so the sum collapses to a single term: −log(q_correct). Cross-entropy loss is just the negative log of the probability you assigned to the right answer. Minimising it maximises that probability, and the log means near-zero confidence in the truth is punished savagely while the difference between 0.8 and 0.9 barely registers.

Verify that asymmetry numerically. It is why a confidently wrong model produces an enormous loss, and why a single mislabelled example can dominate a batch.

Where people get stuck

Expecting KL divergence to be a distance. It is not symmetric — D(p‖q) and D(q‖p) genuinely differ — so it fails the definition. Compute both for a pair of distributions and watch them disagree. Which order you use is a modelling decision with real consequences, not a convention.

The other snag is log base confusion. Base 2 gives bits, natural log gives nats, and both are correct. Everything scales by a constant factor, so conclusions do not change — but a number that disagrees with a reference by a factor of about 0.693 is this and nothing more.

Reading