First PrinciplesStart anywhere. Prove it, then move on.

Rung 48

Reinforcement learning

Learning from consequences instead of answers.

Best attempted after 40. Bayes' theorem, 41. Convexity and gradient descent and 45. Backpropagation from scratch. Nothing stops you trying this now — the gate will tell you if you were right.

The gate

Build a gridworld with a goal, a pit and walls. Solve it twice: once with tabular Q-learning, once with a policy gradient using the network from rung 45. For each method, take one update from a real run, write out every number in it by hand, and say why the value moved in the direction it did. Then set the discount factor to zero and to one, and explain the two failures you observe.

Nobody checks this but you. Do it honestly and the rungs above hold; do it loosely and they will not, somewhere further up where the cause is much harder to find.

Every rung since 44 has assumed someone hands you the right answer for each input. Reinforcement learning removes that assumption. You get a number after acting, sometimes long after, and it does not tell you what you should have done — only how things went.

The formal object is a Markov decision process: states, actions, a transition rule, a reward, and a discount factor γ. The Markov part is a strong claim — that the current state contains everything relevant to the future, so history can be discarded. It is usually false and often good enough, and knowing which of those applies to your problem is most of the modelling work.

Two quantities carry the load. The value of a state is the discounted reward you expect from it under your policy; the Q-value of a state-action pair is the same thing having committed to one action first. Both are defined recursively — a value in terms of the values that follow it — which is rung 25's recursion appearing somewhere unexpected.

Why this is on the ladder

Because it is the last rung, and because it is the one where the honest failure modes are hardest to see. A supervised model that is wrong has a loss that says so. An agent that is wrong has a reward curve that wobbles, and telling a bad algorithm from an unlucky seed from a misspecified reward takes understanding rather than tooling. That is why the gate makes you narrate the arithmetic.

Do this

Build the world first, and keep it small. A grid of about five by five, a goal worth +1, a pit worth −1, a couple of walls, and a small negative reward on every step so that dithering costs something. Fewer than thirty states means you can print the entire value table and look at it.

Tabular Q-learning. Hold a table of Q[state][action], initialised to zero. Act mostly greedily but pick at random with probability ε, and after each transition apply

Q[s][a] ← Q[s][a] + α·( r + γ·max_a' Q[s'][a'] − Q[s][a] )

The bracketed term is the temporal-difference error: the difference between what you now believe and what you believed a moment ago. Run it, print the table every so often, and watch value spread backwards from the goal one cell per episode at first. That propagation is the algorithm's whole character.

Take one update from that run and write it out — the old value, the reward, the discounted best next value, the learning rate, the result. If you cannot say why it moved in that direction, the table converging means nothing.

Policy gradients. Now parameterise the policy directly: a network from rung 45 mapping state to a distribution over actions, via rung 44's softmax. Run a full episode, compute the discounted return from each step onward, and increase the log probability of each action taken in proportion to that return. Actions followed by good outcomes become more likely; actions followed by bad ones become less so. Subtract the mean return across the batch as a baseline and watch the variance of your updates drop sharply — same expected gradient, far less noise.

Then the two γ experiments. At γ = 0 the agent optimises only immediate reward and never learns to walk toward a goal it cannot reach this step. At γ = 1 in a world with no guaranteed termination the returns do not converge, and values grow without bound. Both failures are informative, and neither looks like a bug in the code.

Where people get stuck

Rewarding the wrong thing. An agent optimises exactly what you wrote down, which is rarely exactly what you meant. Add a small reward for moving toward the goal and you will likely produce something that paces back and forth collecting it forever. Do this deliberately once — it is the cheapest lesson in the subject.

Never exploring. With ε at zero the agent commits to whatever looked good first and never discovers better. Rung 40 is relevant here: a couple of successes is weak evidence about an action's true value, and acting as though it were strong is how an agent talks itself into a local optimum.

Comparing runs from single seeds. Reinforcement learning is noisy enough that one run tells you almost nothing. Run five and report the spread, not the best.

That is the ladder. Forty-eight rungs from instructions a machine could follow to writing a transformer with nothing underneath it but arithmetic you can derive. Nothing here was tracked or graded, so what you have at the end is whatever you actually built — which was the point.

Reading