Diffusion

Generating a sample from a complicated distribution is hard. Destroying one is trivial: add a little noise, repeat, and eventually you have noise and nothing else. Diffusion's idea is that if the destruction is done in small enough steps, each step can be undone approximately — and a chain of approximate undoings, run from pure noise, produces a new sample.

The forward process is not learned. It is fixed, and it has a closed form: at step t the noised sample is sqrt(ᾱ_t)·x₀ + sqrt(1 − ᾱ_t)·ε, where ε is standard normal and ᾱ_t shrinks toward zero. That formula means you can jump straight to any step without simulating the ones before it, which is what makes training affordable.

The learned part is one network with one job: given a noised sample and the step number, predict the noise that was added. Not the clean sample, not the next step — the noise. Everything else is arithmetic around that prediction.

Why this is on the ladder

Because it is the clearest example in current practice of a hard problem solved by choosing a better decomposition rather than a better optimiser. And because the whole thing rests on rung 38's normal distribution and rung 39's variance arithmetic, so you are equipped to check every claim rather than accept it.

Keep it to one dimension. The images are what made diffusion famous, and they are also what makes it uncompletable in an afternoon. One dimension keeps every intermediate plottable, which is the only reason you will notice when it is wrong.

Do this

Fix a schedule first. A few hundred steps with β_t rising linearly from about 1e-4 to 0.02 is the standard starting point. Compute α_t = 1 − β_t and the cumulative product ᾱ_t. Plot ᾱ_t against t and confirm it decays smoothly to near zero — that curve is the entire noising schedule and it is worth looking at before you use it.

Now verify the closed form rather than trusting it. Simulate the step-by-step noising for a thousand samples and, separately, jump directly to step t with the formula. The two histograms should agree. This is rung 39's variance addition doing the work, and checking it here saves you from debugging it later through a network.

Train the predictor. Sample a clean point, sample a step t uniformly, sample noise, form the noised point, and ask the network to predict the noise from the noised point and t. The loss is plain squared error. Feed t in as a feature — a couple of sinusoidal components of t works, and so does t/T for a problem this small. A network with two hidden layers is enough.

Then sample backwards. Start from standard normal noise and step down from T to 1, at each step subtracting the scaled noise prediction and adding a smaller amount of fresh noise. Plot the histogram of your results against the true distribution. Both modes should be there, at roughly the right heights.

Finally, halve the number of reverse steps and look again. Modes blur, proportions drift. That degradation is the honest cost of the approximation, and knowing its shape is worth more than a working sampler.

Where people get stuck

Mixing up α_t and ᾱ_t. The single-step coefficient and the cumulative one differ by a product over hundreds of terms. Name them differently in your code.

Adding noise on the final reverse step. At t = 1 there is nothing left to randomise, and adding noise there leaves a visible fuzz on the output distribution that no amount of training will remove.

Concluding the model has failed when one mode is thin. With few samples that is sampling variance, not a broken model. Draw ten thousand and compare against the same count drawn from the truth before you go looking for a bug.