An easy-to-read, detailed companion to Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd edition. Each chapter builds intuition first, then explains the equation, works through an example, warns about the common mistake, answers practical questions, and ends with a compact study card.
The reinforcement-learning problem, its four main elements, delayed consequences, and why trial-and-error learning is different from supervised learning.
chapter 2 · Part I · Tabular MethodsThe exploration–exploitation dilemma in its cleanest form, including ε-greedy action selection, optimistic values, UCB, and gradient bandits.
chapter 3 · Part I · Tabular MethodsThe mathematical language of sequential decisions: states, actions, rewards, returns, policies, value functions, and Bellman equations.
chapter 4 · Part I · Tabular MethodsHow a known environment model enables policy evaluation, policy improvement, policy iteration, value iteration, and generalized policy iteration.
chapter 5 · Part I · Tabular MethodsLearning values and policies from complete sampled episodes, with on-policy control and off-policy importance sampling.
chapter 6 · Part I · Tabular MethodsLearning before an episode ends by bootstrapping, then extending TD prediction to Sarsa, Q-learning, Expected Sarsa, and Double Q-learning.
chapter 7 · Part I · Tabular MethodsThe continuum between one-step TD and Monte Carlo, including n-step Sarsa, off-policy correction, Tree Backup, and Q(σ).
chapter 8 · Part I · Tabular MethodsHow learned models turn experience into simulated experience through Dyna, prioritized sweeping, rollout planning, and Monte Carlo tree search.
chapter 9 · Part II · ApproximationReplacing value tables with parameterized functions, defining a prediction objective, and learning with linear features, tile coding, and neural networks.
chapter 10 · Part II · ApproximationExtending approximate value functions to action selection, episodic semi-gradient Sarsa, and average-reward control for continuing tasks.
chapter 11 · Part II · ApproximationWhy bootstrapping, function approximation, and off-policy learning can diverge—and how Gradient-TD and emphatic methods restore stability.
chapter 12 · Part II · ApproximationA memory of recently active features that efficiently blends many n-step returns through TD(λ), Sarsa(λ), and true-online variants.
chapter 13 · Part II · ApproximationDirectly parameterizing the policy, deriving the policy-gradient idea, reducing variance with baselines, and combining actors with critics.
chapter 14 · Part III · Looking DeeperConnections between computational RL and animal learning: classical and instrumental conditioning, prediction error, habits, and cognitive maps.
chapter 15 · Part III · Looking DeeperHow reward-prediction errors relate to dopamine signals, what supports the correspondence, and where computational and biological claims must be separated.
chapter 16 · Part III · Looking DeeperWhat landmark applications reveal about representation, simulation, search, engineering, evaluation, and the limits of algorithm-only explanations.
chapter 17 · Part III · Looking DeeperOpen directions in prediction, temporal abstraction, state construction, reward design, continual learning, safety, and the societal effects of capable agents.