First PrinciplesStart anywhere. Prove it, then move on.

Rung 38

Continuous distributions and PDFs

Probability when the outcomes cannot be counted.

Best attempted after 23. Integration and the fundamental theorem. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Verify numerically that a probability density integrates to one. Then compute the probability of an interval two ways — by integrating the density, and by sampling many random draws and counting — and show the two agree. State how the sampling error shrinks as you increase the number of draws.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

Rung 16 counted outcomes: favourable over possible. That breaks the moment the outcomes are continuous. Ask for the probability that a value is exactly 1.5 and the answer is zero — not because it cannot happen, but because there are uncountably many alternatives.

So the question changes. Instead of the probability of a point, ask for the probability of an interval, and get it as the area under a curve. The curve is a probability density, and rung 23's integration is how you read it.

Why this is on the ladder

Because model outputs are continuous. A network's confidence, a predicted value, a noise level at rung 47 — all continuous quantities described by densities. Every loss averaged over a distribution is an integral, and rung 43's entropy is an integral of a density against its own logarithm.

Do this

Take the standard normal density. First confirm it is a density at all: integrate it numerically over a wide range with rung 23's rectangle sum and get 1 to several decimals. A density that does not integrate to one is not a density.

Then compute the probability of landing between −1 and 1 by integrating: about 0.68.

Now get the same number a completely different way. Draw a hundred thousand random samples, count how many fall in the interval, divide. The two answers should agree to two or three decimals.

Then study the error. Repeat with 100, 10,000 and 1,000,000 samples, several times each. The error falls roughly as one over the square root of the sample count — which means a hundredfold increase in work buys one decimal place. That is a genuinely poor rate, and knowing it is why "just sample more" stops being a plan.

Where people get stuck

Reading a density value as a probability. A density can exceed one — a narrow distribution has a tall peak — because it is probability per unit, and only area is probability. This is the single most common misreading on this rung.

The other snag is trusting one Monte Carlo run. Any single estimate is a draw from its own distribution; repeat it several times and watch the answers scatter before believing any one of them.

Reading