// the one-minute version
Psychology asks the same two questions as RL: how are future events predicted, and how do consequences change action? Error-correction models explain why a surprising outcome teaches more than a predicted one and why an established cue can block
Asha notices her robot slowing before a blind corner even when no obstacle is visible. A warning light had repeatedly preceded congestion; eventually the light itself changed behavior. Then she adds an audible tone beside the already reliable light, but the robot’s learned predictor barely assigns the tone importance. More exposure is not always more learning. What matters is surprise.
01 Prediction and control organize animal learning
Classical conditioning studies how one event predicts another: a cue comes before food, shock, or another significant outcome. Instrumental conditioning studies how an action’s consequences change later action. In RL language these align roughly with prediction and control, although biological experiments include motivational, perceptual, and innate response systems beyond a simple MDP.
02 Classical conditioning is about informative relationships
A conditioned stimulus (CS), such as a tone, initially evokes little relevant response. An unconditioned stimulus (US), such as food, naturally evokes one. Repeated predictive pairing makes the CS elicit a conditioned response. Timing, contingency, context, and prior knowledge matter; mere co-occurrence is insufficient.
Learning may concern the sensory identity, timing, or value of an outcome. A scalar reward-prediction model captures only part of this richness, but it provides a disciplined account of trial-by-trial changes.
03 Rescorla–Wagner: shared surprise explains blocking
In a trial-level error-correction model, all present cues predict the outcome together. Cue i changes by a learning-rate term times its salience times the common error:
First train light→food until the light predicts food. Then present light+tone→food. Because ΣV is already near λ, error is near zero and the tone gains little association: blocking. If the outcome is unexpectedly larger, error reappears and both cues can learn. This refutes theories in which every paired cue gains strength automatically.
04 TD puts time inside the prediction
Rescorla–Wagner treats a trial largely as one unit. TD divides it into moments. With δt=Rt+1+γV(St+1)−V(St), prediction can move backward from the outcome to earlier cues.
The outcome still matters, but once fully predicted it produces little teaching signal; omission produces a negative error.
Eligibility traces allow cues separated from outcomes to receive credit. Feature representations let multiple stimuli share and compete for prediction. TD simulations reproduce acquisition curves, second-order conditioning, and aspects of blocking, while also exposing phenomena that demand richer representations or attention mechanisms.
05 Higher-order and contextual conditioning stretch the simple model
Once a light predicts food, a tone paired with the light can acquire predictive power even without direct food pairing: higher-order conditioning. Context can act as another cue, occasion setter, or part of state. Extinction—presenting the cue without the outcome—is usually new inhibitory or contextual learning, not simple erasure, as spontaneous recovery and renewal demonstrate.
For an RL model, the state representation decides whether context, elapsed time, and cue configuration are distinguishable. Apparent algorithm failure may be a state-construction failure.
06 Instrumental conditioning introduces agency
In instrumental tasks, a response changes outcome probability. Reinforcement tends to increase a response; punishment tends to reduce it. But “reinforcer” is defined through its effect and contingencies, not simply pleasure. The action–outcome relationship, motivational state, and alternatives matter.
RL control supplies states, actions, rewards, and policies. Exploration explains how a behavior can be discovered; action values explain competition; temporal credit handles delay. Yet animals bring structured motor repertoires and biological priors, so a flat action list is an abstraction.
07 Delayed reinforcement and eligibility
A consequence can strengthen earlier behavior despite intervening time. Traces provide a computational bridge: recent state-action features remain eligible when a reward-prediction error arrives. Longer delay normally weakens learning because eligibility decays and competing events intervene.
Bridging cues can help because they create intermediate predictors. Shaping helps by reinforcing successive approximations, effectively building a denser sequence of informative errors.
08 Cognitive maps and latent learning
Animals can learn relationships without immediate reward and later use them flexibly. Detour behavior and latent-learning experiments motivated cognitive maps: internal knowledge of how places or events relate. In RL terms, a learned transition model can support new planning when rewards change, without relearning every cached action value.
Model-based does not require a literal geometric map; it means using predicted consequences to evaluate actions. Replay and planning can integrate direct experience with simulated trajectories, echoing Dyna.
09 Habitual and goal-directed control
Outcome devaluation is a key test. If food becomes undesirable, goal-directed action should decline because planning uses the new outcome value. A cached model-free action value may persist until new experience updates it, resembling habit. Contingency degradation tests whether action depends on its causal relation to outcome.
Behavior often reflects arbitration: planning is flexible but computationally expensive; habits are fast but can be stale. Training amount, stress, uncertainty, and available time can shift the balance.
10 Questions a master’s student should be able to answer
Why does blocking favor error-correction theories?
The new cue co-occurs with the outcome but learns little because the old cue already predicts it. A shared prediction error explains this; pure pairing count does not.
What does TD add beyond Rescorla–Wagner?
TD represents within-trial time and bootstraps between successive predictions, explaining how predictive credit and error migrate through cue sequences.
Is extinction unlearning?
Often not purely. Renewal, reinstatement, and spontaneous recovery suggest that extinction adds context-sensitive knowledge competing with the original association.
How does outcome devaluation distinguish control systems?
Immediate sensitivity without re-experiencing the action suggests access to an outcome model; persistent responding suggests cached control, though interpretation depends on representation and test conditions.
What makes a strong computational psychology claim?
Specify task, state representation, learning rule, observation model, and fitted parameters; compare alternatives; predict trial-by-trial behavior and a held-out intervention rather than only retelling known results.
11 Chapter summary and exam-ready map
- Two problems: classical conditioning concerns prediction; instrumental conditioning adds action and control.
- Surprise: shared error explains cue competition and blocking.
- Time: TD moves prediction and teaching signals from outcome toward earlier cues.
- Representation: context, timing, configurations, and outcome identity determine what a model can learn.
- Delay: traces and bridging predictors connect remote consequences to behavior.
- Cognitive maps: learned models permit flexible revaluation and planning.
- Dual control: cached and planned evaluation illuminate habits and goal-directed action without exhausting their biology.
explain
critique
12 Source trail and scope
These are independent companion notes, not a replacement for the textbook. The chapter organization follows Sutton and Barto's second edition; explanations and examples here are original. Use the authors' book page, the MIT Press edition page, and the official table of contents for the primary source and exact section sequence.