The textbook, the canonical lecture courses, and the landmark papers worth keeping for Deep Reinforcement Learning — the mid-sem foundations and the deep-RL milestones beyond them.
Reading map by lecture
If you only have time for the mid-sem, read these in order. Every lecture maps to a chapter of the textbook (T1):
| Lecture | Topic | Read |
|---|---|---|
| CS1 | Intro & elements | S&B Ch 1 · notes |
| CS2–3 | Multi-armed bandits | S&B Ch 2 · notes |
| CS3–5 | MDPs & Bellman | S&B Ch 3 · notes |
| CS4–6 | Dynamic programming | S&B Ch 4 · notes |
| CS6–7 | Monte Carlo | S&B Ch 5 · notes |
Textbooks
- Sutton & Barto — Reinforcement Learning: An Introduction (2nd ed., MIT Press, 2018) · the course textbook (T1); free official PDF. Ch 1–5 = the entire mid-sem. The single most important resource — read it.
- Sutton & Barto — 1st edition (1998) · older but still-clear treatment of the tabular methods.
- Bertsekas — Dynamic Programming and Optimal Control · the rigorous reference for the DP foundations behind Chapter 4.
- Powell — Approximate Dynamic Programming · DP at scale, the bridge from exact DP to function approximation.
Lecture courses
- David Silver — UCL Reinforcement Learning course · the classic 10-lecture series; tracks Sutton & Barto almost lecture-for-lecture. The canonical companion to the mid-sem.
- DeepMind × UCL — Reinforcement Learning lecture series (Hado van Hasselt) · a modern, deeper re-recording of the UCL course.
- Berkeley CS285 — Deep Reinforcement Learning · Sergey Levine's graduate deep-RL course; where to go after the tabular mid-sem.
- OpenAI — Spinning Up in Deep RL · practical, code-first intro to policy-gradient & actor-critic methods, with clean implementations.
- Hugging Face — Deep RL Course · hands-on, free, train agents in simulators; great for building intuition by doing.
Landmark papers
Beyond the mid-sem, these are the milestones the lectures gesture at — the value-based, policy-based, and model-based lineages of deep RL:
- Watkins & Dayan (1992) — Q-learning · the off-policy TD control algorithm; the direct sequel to the mid-sem.
- Williams (1992) — REINFORCE · the original policy-gradient algorithm; the gradient-bandit idea generalized.
- Tesauro (1995) — TD-Gammon · TD learning + a neural net reaches expert backgammon; an early "deep RL" triumph.
- Mnih et al. (2013) — Playing Atari with Deep RL (DQN) · deep Q-networks; the spark of modern deep RL. Cited in the MDP lecture.
- Mnih et al. (2015) — Human-level control through deep RL · the Nature DQN paper (experience replay + target network).
- Mnih et al. (2016) — Asynchronous Methods for Deep RL (A3C) · asynchronous advantage actor-critic.
- Lillicrap et al. (2016) — DDPG · continuous-control actor-critic.
- Schulman et al. (2017) — Proximal Policy Optimization (PPO) · the workhorse policy-gradient algorithm in use today.
- Silver et al. (2016) — Mastering the game of Go (AlphaGo) · MCTS + deep RL beats a Go champion.
- Silver et al. (2017) — AlphaGo Zero / AlphaZero · superhuman play from self-play alone, no human data.
- Schrittwieser et al. (2020) — MuZero · planning with a learned model — model-based RL come full circle.
- Ault et al. (2020) — Interpretable Traffic Signal Control Policy · the traffic-signal MDP example from the slides.
Rest of this vault
- Slides Explained · CS1–CS7, plain language
- Book Explained · Sutton & Barto Ch 1–5
- Cheatsheet · one-card recap
- Formula Sheet · every equation
- Question Bank · test yourself