// the one-minute version
The frontier is larger than a better optimizer. General value functions turn value learning into a language for asking many predictive questions under many policies; auxiliary tasks can build useful representations continually. Options package initiation, policy, and termination into temporally extended actions. Both depend on constructing state from partial observations and deciding which predictions or skills are worth learning. Reward design remains the deepest specification problem: a scalar proxy is not human intent. Continual learning, transfer, planning, exploration, stability, multi-agent effects, safety, reproducibility, and social governance remain open precisely because capable agents change the worlds that generate their data.
Asha’s robot now delivers parcels reliably, but a small layout change exposes the limits of the entire pipeline. The controller knows one task reward, one action timescale, and a state designed for yesterday’s aisles. It cannot answer simple questions—“Will this route overheat the battery?” “Can I reach a charger within two minutes?”—without retraining a task. The last chapter asks what a generally useful learning system should know before the next reward arrives.
01 From one objective to a question-answering machine
Ordinary value predicts discounted reward under a policy. Replace “reward” with an arbitrary signal called a cumulant, allow a question-specific continuation γ, and specify a policy. The expected discounted cumulant is a general value function (GVF).
Battery drain, collision probability, travel time, temperature, human presence, and task success can all become learned predictions. GVFs ground knowledge in forecasts testable against future experience rather than free-floating symbols.
02 General value functions make prediction plural
Each GVF has a policy, cumulant, continuation, and sometimes interest. Off-policy methods let one behavior stream train predictions about many target policies. A question can be short-lived through γ=0 at an event, or continuing with γ near one. Probability of an event is expressible with a suitable cumulant and termination.
The challenge shifts from answering questions to discovering good questions. Predictions are useful when they improve control, compress dynamics, expose hazards, or support human understanding. Thousands of arbitrary forecasts can consume capacity and interfere. Question selection is representation learning at the level of semantics.
03 Auxiliary tasks build reusable representation
A primary reward is sparse and narrow. Auxiliary predictions supply dense learning signals from every observation. Their internal features may become useful for the main task, transfer, and rapid adaptation. This is continual supervised structure generated from interaction itself.
But auxiliary loss weight matters. A perfectly predictable yet irrelevant signal can dominate features; conflicting tasks can cause negative transfer. Evaluate whether an auxiliary task improves sample efficiency, robustness, or transfer—not only whether its own error falls.
State, prediction, and temporally extended action can improve one another in a continual loop.
04 Options let decisions span meaningful time
An option is a triple: initiation set I, internal policy πo, and termination condition βo. “Go charge” may start when a charger is reachable, execute navigation actions, and end on arrival or failure. At the higher level it behaves like one action whose duration is variable.
Options support semi-Markov decision processes and backups over entire executions. They can accelerate planning because a search tree branches over meaningful behaviors rather than motor ticks. The option-critic direction learns internal policies and termination, but option discovery remains difficult: useful skills should be reusable, controllable, and neither terminate immediately nor monopolize behavior.
05 Observation is not state
The Markov state contains everything from history needed to predict future consequences of action. A camera frame, sensor vector, or database row may not. Occluded velocity, previous commitments, hidden user intent, and changing regimes create partial observability.
History windows, recurrent networks, belief states, predictive-state representations, and GVF collections all attempt state construction. A representation is good relative to a set of questions and decisions; there is no task-independent guarantee that one compact state preserves everything.
06 Reward design is specification, not decoration
The reward defines optimality inside the formal problem. It does not automatically encode safety, fairness, human preference, or every side effect. If Asha rewards deliveries per hour, the robot may block walkways, wear batteries aggressively, or neglect rare urgent parcels. These are rational responses to an incomplete objective.
Reward shaping can speed learning, and potential-based shaping can preserve optimal policies under assumptions. Constraints, risk-sensitive criteria, human oversight, preference learning, and multi-objective evaluation address other gaps. None eliminates the need to audit who chose the goal and who bears its externalities.
07 Continual learning means the target never holds still
A lifelong agent faces nonstationary dynamics, new tasks, limited memory, and no clean train/test boundary. It must learn without catastrophic forgetting, detect change without overreacting to noise, reuse old knowledge without negative transfer, and keep exploring without unsafe behavior.
Replay, modular representations, meta-learning, uncertainty, and adaptive step sizes help, but stability–plasticity remains fundamental: parameters must preserve useful knowledge and still move when the world changes.
08 Remaining technical frontiers
Exploration must seek information across long horizons. Planning needs models whose errors do not compound. Offline RL must avoid unsupported actions. Multi-agent learning faces strategic nonstationarity. Representation must preserve causal, controllable structure. Theory still trails practice for nonlinear off-policy bootstrapping.
Evaluation needs distributions, uncertainty, adversarial conditions, ablations, compute accounting, reproducible protocols, and real-world monitoring. A mean benchmark score is not a safety case.
09 Social effects belong inside the system boundary
Deployed agents allocate attention, prices, opportunities, resources, and risk. Their behavior changes users and institutions, invalidating the stationary-environment fiction. Feedback loops can amplify popularity, exclusion, manipulation, or arms races between agents.
Governance questions—consent, contestability, accountability, access, monitoring, and shutdown—are not downstream public relations. They determine allowed actions, observations, objectives, and deployment policies, so they are part of the sequential decision problem.
10 Questions a master’s student should be able to answer
How is a GVF different from the task value function?
It can predict any specified cumulant, continuation, and policy, not only the environment’s task reward under the control policy. Many GVFs can be learned concurrently as knowledge.
What makes an option more than a macro?
It has a policy conditioned on intermediate states and a stochastic termination rule, so it can react during execution rather than replaying a fixed action string.
Can a recurrent network solve the state problem automatically?
It can learn history-dependent features, but only from training signals and distributions provided. It may forget rare information, encode spurious correlations, or fail under regime change.
Why not fix reward hacking by adding more penalties?
Each penalty is another proxy and can introduce interactions or new loopholes. Constraints and oversight help, but the deeper need is iterative specification, adversarial testing, and monitoring real outcomes.
What is a credible frontier research project?
State an unresolved assumption, build a minimal test plus realistic stress case, compare strong baselines, measure uncertainty and resource cost, release protocol, and separate algorithmic gain from representation or data advantage.
11 Whole-book synthesis
- Prediction: value functions, TD errors, traces, and approximation turn experience into forecasts.
- Control: generalized policy iteration couples evaluation with improvement, from tables to actor–critic.
- Planning: models and search reuse imagined experience while remaining vulnerable to model error.
- Knowledge: GVFs broaden reward prediction into testable predictive questions.
- Temporal abstraction: options make policies and planning operate across multiple timescales.
- State: useful Markov representations must often be constructed from history.
- Objectives: optimized reward is only as good as the specification and governance surrounding it.
- Scientific stance: algorithms are hypotheses tested by distributions, interventions, diagnostics, and reproducible evidence.
design
red team
research
12 Source trail and scope
These are independent companion notes, not a replacement for the textbook. The chapter organization follows Sutton and Barto's second edition; explanations and examples here are original. Use the authors' book page, the MIT Press edition page, and the official table of contents for the primary source and exact section sequence.