// the one-minute version
Reward, reinforcement, value, and reward-prediction error are different variables. A major hypothesis links phasic activity in some midbrain dopamine neurons to the TD error δ=r+γV′−V: response to unexpected reward, transfer toward a predictive cue, scaling with better- or worse-than-expected outcomes, and a dip when expected reward is omitted. An actor–critic interpretation places critic-like predictions and actor-like action tendencies in interacting neural circuits. The match is influential but incomplete: dopamine cells are heterogeneous, state representation and timing require neural machinery, and movement, novelty, motivation, model-based inference, and addiction extend beyond a scalar textbook signal.
At first, a drop of juice surprises an animal and a brief burst appears in recordings from certain dopamine neurons. After a light reliably predicts the juice, the burst moves to the light; juice itself produces little phasic change. Omit the expected juice and activity dips at the expected moment. This three-part pattern is the chapter’s central scientific clue.
01 Separate four concepts before mapping anything to the brain
Reward is the evaluative signal in the computational problem. Reinforcement is a behavioral effect: a consequence increases future action. Value predicts accumulated future reward. Reward-prediction error reports a change in prediction, not how good the situation is by itself.
A fully predicted reward can have high value and zero error. A reward omission can produce negative error although received reward is merely zero. Confusing these signals destroys the hypothesis’s testable content.
02 The evidence pattern mirrors TD learning
Before learning, V is small, so reward makes δ positive. After a reliable cue, its onset raises predicted future reward and creates positive δ there. At reward delivery the prediction is already fulfilled, so δ is near zero. If delivery fails, the disappearance of expected value makes δ negative.
The temporal location and sign of the response matter more than a loose correlation with reward.
Responses can also scale with magnitude and probability, broadly matching signed surprise. A cue predicting a 50% chance establishes a different expectation from one predicting certainty. However, averaging across trials and cells can conceal richer subpopulation responses.
03 Timing forces a theory of neural state
To dip exactly when an omitted reward was expected, a learner must represent elapsed time or a sequence of internal states. A tapped-delay-line representation activates successive features after a cue; a spectral or distributed timing representation uses overlapping temporal fields. TD then propagates value across them.
This is a general lesson from function approximation: the update rule cannot predict distinctions absent from its state. Neural state may combine sensory cues, context, working memory, spatial location, uncertainty, and internal needs.
04 A neural actor–critic interpretation
In a computational actor–critic, the critic estimates value and emits δ; the actor changes action preferences using that error. One family of biological models associates critic-like functions with ventral striatal circuits and actor-like selection with dorsal striatal circuits, influenced by midbrain dopamine projections.
The mapping is many-to-many, not a wiring diagram copied from pseudocode. Brain regions participate in multiple loops, dopamine changes learning, vigor, and motivation, and synaptic plasticity depends on local eligibility signals as well as neuromodulation.
05 Three-factor learning rules resemble eligibility traces
A synapse may need presynaptic activity, postsynaptic activity, and a later modulatory signal before changing. The first two create a local eligibility trace; a dopamine-related error arriving later converts eligibility into plasticity. This resembles Δw∝δz.
The resemblance solves a timing problem: a global outcome signal need not identify every responsible synapse if local activity has marked candidates. Biological trace durations, receptor dynamics, and cell types are empirical matters rather than the single λ of an engineering model.
06 Hedonic impact, wanting, and learning can dissociate
Dopamine should not be reduced to pleasure. Reward consumption, subjective liking, motivational vigor, cue-triggered wanting, and learning can dissociate. Some neural systems relate more closely to hedonic impact; dopamine manipulations can strongly affect pursuit and learning without changing every pleasure response in parallel.
This also warns engineers: one scalar reward bundles distinctions biology may keep separate—need, salience, identity, and utility.
07 Collective reinforcement learning in neural circuits
A biological learner is a population, not one scalar register. Different neurons may encode distributional aspects of future reward, distinct timescales, actions, or features. A population can collectively approximate a value or error even if no single neuron matches the textbook signal perfectly.
Heterogeneity is not automatically noise; it can enlarge representation. But fitting a flexible RL model after observing neural data risks circularity. Strong tests predict responses under changes in probability, timing, magnitude, and state before seeing the result.
08 Model-based signals and replay
Brains also support prospective evaluation. Hippocampal spatial representations, sequential replay, and prefrontal/orbitofrontal signals have been discussed in relation to maps, latent states, and planned outcomes. Revaluation can change behavior before direct re-experience, challenging a purely cached-value account.
Model-free and model-based computations may cooperate: replay can train cached values, planning can use learned predictive representations, and a common error signal can update multiple systems.
09 Addiction as a stress test for the framework
Some theories treat drug effects as persistent positive prediction-error-like signals that fail to diminish with ordinary expectation, driving pathological value learning. The idea shows how a normally adaptive update could be captured by a consequence that directly alters its teaching machinery.
But addiction is multi-causal and involves pharmacology, plasticity, stress, habit, social context, and choice. A computational account is one explanatory level, not a complete clinical theory.
10 Questions a master’s student should be able to answer
How can a valuable reward produce no dopamine burst?
If it is fully predicted, reward and the loss of predicted future value cancel in δ. Value remains high while new information is near zero.
Why is omission evidence especially important?
A negative response at the precise expected time is predicted by a signed error account but not by a simple theory in which activity merely reports received reward or arousal.
Where might eligibility traces appear biologically?
Recently active synapses can enter temporary biochemical states; a later neuromodulatory signal can consolidate or reverse plasticity. The computational trace summarizes this three-factor possibility.
Does dopamine only encode model-free error?
No broad claim is safe. Responses can reflect inferred state, sensory features, movement, and model-based knowledge; populations are heterogeneous and task dependent.
What would falsify a proposed neural RL mapping?
Pre-register which model variable predicts timing, sign, and magnitude under controlled expectation changes, then show competing variables cannot explain the signal. A mapping that can relabel every result is not informative.
11 Chapter summary and exam-ready map
- Distinctions: reward, reinforcement, value, and prediction error are not synonyms.
- Core evidence: phasic responses shift to cues, scale with surprise, and dip at omitted expected reward.
- Representation: precise timing and context require internal states beyond a raw stimulus.
- Actor–critic: offers a systems-level hypothesis, not a literal one-region-per-variable map.
- Plasticity: local eligibility plus global error resembles three-factor synaptic learning.
- Beyond scalar error: population diversity, planning, replay, motivation, and movement enrich the account.
- Epistemic rule: computational correspondence must generate discriminating experiments.
sketch
critique
12 Source trail and scope
These are independent companion notes, not a replacement for the textbook. The chapter organization follows Sutton and Barto's second edition; explanations and examples here are original. Use the authors' book page, the MIT Press edition page, and the official table of contents for the primary source and exact section sequence.