First PrinciplesStart anywhere. Prove it, then move on.

Rung 39

Expectation, variance, covariance

The centre, the spread, and whether two quantities move together.

Best attempted after 38. Continuous distributions and PDFs. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Compute the mean and variance of a distribution analytically and from a large sample, and show they agree. Then construct two correlated variables and show the variance of their sum is not the sum of their variances — quantify the gap and identify it as the covariance term.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

Three summaries do most of the work. Expectation is the long-run average — the balance point of the distribution. Variance is the average squared distance from that centre, so it measures spread. Covariance extends the idea to two variables and asks whether they rise and fall together.

Squared distance rather than plain distance, because plain distances above and below the mean cancel exactly. Squaring also makes the algebra behave, at the cost of units, which is why the square root gets taken back at the end and called the standard deviation.

Why this is on the ladder

Because every loss is an expectation. Mean squared error is precisely the average squared distance between prediction and truth — a variance in disguise. And covariance is what a correlation is built from, which matters directly at rung 37, where PCA finds the eigenvectors of a covariance matrix.

Do this

Take a distribution whose mean and variance you can compute by hand — a uniform on [0, 1] gives mean 1/2 and variance 1/12. Do the integrals.

Then sample heavily and compute the sample mean and variance. They should converge on your analytic values, with the same square-root error behaviour rung 38 established.

Now the part that carries the rung. Generate X at random. Set Y = X plus a little independent noise, so the two are strongly correlated. Compute Var(X) + Var(Y), then compute Var(X + Y) directly. They do not match, and the gap is 2·Cov(X, Y). Verify that numerically.

Then make Y independent of X and watch the covariance term vanish and the variances add. "Variance adds" is only true for independent quantities, and having seen it fail once, you will not misapply it.

Where people get stuck

Assuming independence because two things look unrelated. The gate exists to make that assumption visible and testable rather than tacit.

The other classic is reading zero covariance as independence. It means no linear relationship. Points on a circle have essentially zero covariance and are about as dependent as two variables can be — worth constructing, because it is a memorable correction to an intuition almost everyone has.

Reading