// the one-minute version
Landmark applications succeed because a learning rule fits a carefully constructed problem. TD-Gammon combined self-play, stochastic games, nonlinear values, and TD. Samuel’s checkers combined learned evaluation with search. Watson used value-like reasoning for risk-sensitive wagering. Memory control learned which item to evict. Atari coupled convolutional Q-learning with replay and target networks. AlphaGo combined policy/value learning with MCTS; AlphaGo Zero replaced human data with self-play and known rules. Web personalization faces counterfactual evaluation, while thermal soaring learns online in the physical world. The transferable lesson is to decompose state, action, reward, data, model, planning, constraints, and evaluation.
Asha asks which famous algorithm to copy into the warehouse. That is the wrong first question. Backgammon supplies millions of cheap self-play games; a glider receives scarce, risky real flights; a website sees logged feedback distorted by its previous recommendations. Chapter 16 is a set of engineering arguments about matching methods to interaction.
01 Read every application as a designed learning system
Ask eight questions: What is the state? What actions are available? What does reward measure? Where does experience come from? Is there a model or simulator? What function is approximated? Is planning used at decision time? How is performance and safety established? The update equation becomes meaningful only after these are answered.
The surrounding loop—not an isolated learner—determines what the result means.
02 TD-Gammon: stochastic self-play teaches a value function
State is the backgammon board, action is a legal move after dice, reward arrives mainly at game end, and a neural network estimates winning prospects. Games generated by self-play supply transitions. TD updates make each board prediction agree with the next, propagating final outcomes backward.
Why did this fit? Dice force exploration of diverse positions; rules give a perfect transition generator; self-play provides an adaptive opponent and unlimited labelled outcomes. The network’s representation and enormous experience mattered alongside TD. Later search could improve decision quality further, showing learning and planning are complements.
03 Samuel’s checkers: evaluation learning inside search
Samuel’s program used a parameterized board evaluation and lookahead. Search converted the evaluator into action; game results and comparisons improved evaluation. This early work already contained themes now called self-play, feature learning, bootstrapping, and search-control interaction.
The lesson is architectural: if an exact model is available, learning need not replace search. A learned evaluator makes limited search useful; search supplies stronger targets and decisions than either component alone.
04 Watson’s wagering: action value under asymmetric risk
In Jeopardy!, deciding whether to answer and how much to wager depends on confidence, score, remaining clues, opponent scores, and game rules. Reward is probability of winning—not simply immediate dollars. A locally high expected score can lower match-winning probability.
Simulation and dynamic decision analysis estimate consequences. The case shows that “reward” must express the real terminal objective and that calibrated uncertainty is part of state. The learned component lives inside hard rules and strategic models.
05 Optimizing memory control
A cache or memory system repeatedly chooses what to retain or evict. State summarizes contents and access history; action selects an item; reward reflects future hits, latency, or cost. The consequence is delayed: an eviction is judged only if the item is requested later.
RL can learn patterns that fixed recency heuristics miss, but action spaces and nonstationary workloads are difficult. A strong baseline such as LRU is essential, as are workload-shift tests, bounded overhead, and fail-safe fallback.
06 Atari: representation learning meets the deadly triad
Input is a stack of image frames so motion is partially observable; actions are joystick commands; reward comes from the game; a convolutional network approximates Q. Q-learning reuses transitions from replay and bootstraps from a slowly updated target network.
Replay improves data reuse and reduces sequential correlation; a target network slows target drift. Both are practical responses to instability, not removal of the deadly triad. Scores reveal competence under a defined emulator interface, not general vision or reasoning. Frame preprocessing, action repeat, reward clipping, training budget, and evaluation protocol materially shape results.
07 Go: policy, value, search, and self-play form one machine
Go has exact rules but an enormous tree. AlphaGo used a policy network to propose promising moves, a value network to estimate positions, and Monte Carlo tree search to allocate computation. Supervised human games initialized useful policy behavior; reinforcement learning through self-play improved it.
AlphaGo Zero began from rules and self-play rather than human games. Search produced improved move distributions; the network learned both policy and value targets from self-play; the new network guided stronger future search. This is generalized policy iteration at scale.
08 Personalized web services: logged feedback is counterfactual
State may include session context, action is content or layout, and reward is engagement or a longer-term outcome. But logs show feedback only for displayed actions. The serving policy determines data, making policy improvement off-policy and susceptible to confounding and feedback loops.
Randomized exploration, propensity logging, importance weighting, doubly robust evaluation, conservative rollout, guardrail metrics, and A/B tests become core system components. Optimizing clicks can harm satisfaction or diversity; reward design and societal effects cannot be delegated to the optimizer.
09 Thermal soaring: learning with scarce physical interaction
An autonomous glider must find rising air, circle to gain altitude, and leave when useful lift fades. State is noisy flight and atmospheric information; actions control flight; reward relates to energy or altitude. The environment changes and failures cost real hardware.
Online learning must respect flight envelopes, use strong prior control, and learn from few trajectories. This is the opposite of unlimited game self-play: sample efficiency, safety constraints, system identification, and fallback behavior dominate.
10 Questions a master’s student should be able to answer
Why was self-play especially suitable for board games?
Rules provide exact simulation, outcomes are unambiguous, experience is cheap, and the opponent improves with the agent. Physical and social domains rarely offer all four.
What did search add beyond learned networks in Go?
It focused computation on the current position, combined many possible continuations, and returned an improved move distribution. Networks amortized past search; search refined the present decision.
Why are web logs not ordinary supervised labels?
They reveal outcomes only for actions selected by the logging policy. Unseen alternatives are counterfactual, and user population and behavior can change in response to the policy.
What is the right baseline for an RL application?
The strongest simple operational method under the same data, compute, latency, and safety constraints—not merely a random policy or an outdated paper result.
How should I present a case study?
Specify formulation, data generation, representation, learning target, planning, constraints, baselines, evaluation distribution, failure modes, and what evidence does not establish.
11 Chapter summary and exam-ready map
- TD-Gammon: stochastic self-play plus nonlinear TD value learning.
- Samuel: evaluation learning and search were complementary from the field’s early history.
- Watson/memory: real objectives, uncertainty, delayed effect, and strong operational baselines matter.
- Atari: pixel representation, replay, target networks, and precise protocol surround Q-learning.
- Go: policy, value, MCTS, and self-play form a mutually improving loop.
- Personalization: off-policy counterfactual evaluation and reward externalities dominate safe deployment.
- Soaring: scarce real interaction reverses the priorities of simulator-rich applications.
decompose
transfer
12 Source trail and scope
These are independent companion notes, not a replacement for the textbook. The chapter organization follows Sutton and Barto's second edition; explanations and examples here are original. Use the authors' book page, the MIT Press edition page, and the official table of contents for the primary source and exact section sequence.