Chapter 11 — Off-policy Methods with Approximation, explained.

Asha trains a safer routing policy from yesterday’s logs. The logs came from the old controller, but the values describe the new one. Nothing seems exotic: linear features, tiny steps, bounded rewards. Yet the weight norm doubles, then doubles again. The bug is not floating point. Chapter 11 explains how three individually useful ideas can create a positive feedback loop.

01 One trajectory, two policies, two distributions

The behavior policy b selects actions and determines which states appear. The target policy π defines the values we want. Per-step importance ratio ρ_t=π(A_t|S_t)/b(A_t|S_t) corrects the sampled action, provided b covers every action π may choose.

But approximation minimizes errors under some state weighting. The samples arrive under b’s stationary distribution, while the Bellman equation follows π’s transitions. Correcting actions does not magically turn b’s state distribution into π’s. That geometric mismatch is harmless in a table more often than in shared features.

02 Why ordinary semi-gradient TD can point uphill

The naive off-policy update is w←w+αρδx, with δ=R+γv̂(S′,w)−v̂(S,w). It differentiates the current prediction but not the bootstrapped target. On-policy, the expected linear update has favorable stability. Off-policy, the matrix governing expected weight motion need not be positive definite. A direction that looks locally corrective can increase future targets faster than current predictions catch them.

Monte Carlo with importance sampling avoids the bootstrap member of the triad and can converge, though variance may be enormous. Tabular off-policy TD avoids harmful parameter sharing. Remove any one corner and the classic divergence mechanism is broken; having all three does not guarantee failure, but permits it.

03 Baird’s counterexample is a proof, not a benchmark trick

Baird’s seven-state “star” uses zero rewards, linear features, one behavior policy, and another target policy. The true value is zero everywhere. Nevertheless, expected semi-gradient TD repeatedly pushes shared weights away from zero. Because the example can be analyzed without sampling noise, lowering α only slows divergence.

04 Bellman geometry: backup, then projection

Let T_π map any value function to its one-step Bellman target. With approximation, T_πv̂ usually lies outside the representable subspace. Projection Π maps it back to the closest representable function under a chosen state distribution. The approximate fixed point solves

On-policy weighting makes ΠT_π contract in a useful norm. Under behavior weighting this need not hold. Repeated backup-and-projection can enlarge error. The usual mean-squared Bellman error is also difficult because an unbiased gradient estimate generally needs two independent next-state samples—the double-sampling problem.

05 What objective can be optimized from one sample?

The mean-squared projected Bellman error (MSPBE) measures the distance between v̂ and the projection of its Bellman target. It matches the fixed-point geometry and can be optimized incrementally. Gradient-TD algorithms introduce an auxiliary vector h that estimates the correction hidden inside the projection.

TDC/GTD2 updates h on a faster timescale to track a feature-weighted TD error, while w follows an actual negative gradient of MSPBE. Importance ratios handle the target action probabilities. The exact equations matter in implementation, but the conceptual division is more important: h estimates what ordinary semi-gradient TD ignores, and w uses that estimate to follow a stable objective.

06 Expected updates reveal the algorithm beneath sampling

Writing the linear expected update as w←w+α(b−Aw) separates stochastic noise from mean dynamics. Stability depends on the eigenstructure of A, not on whether one run happens to look smooth. The chapter uses this view to classify algorithms and objectives.

This is also why weight-norm plots are essential. Return can look acceptable temporarily while parameters grow along a weakly observed direction. Monitor prediction error, MSPBE estimates, TD errors, importance ratios, and ‖w‖.

07 Emphatic TD changes who receives learning weight

Suppose an interest i(s) says where accurate prediction matters. If an interested state can lead under π to another state, that successor’s value helps the interested prediction and deserves follow-on weight. Emphatic TD accumulates this dependency in a follow-on trace F and forms emphasis M, combining current interest with inherited importance.

The update uses M_tρ_tδ_tx_t. Rather than forcing target visitation, it constructs a weighting that reflects target-policy bootstrapping and yields stable expected dynamics for linear prediction under assumptions. Variance can still be high; stability and statistical efficiency are different questions.

08 Control is harder than prediction

For prediction, π is fixed. In control, π changes as q̂ changes, so the target operator, state distribution, and importance ratios all move. Semi-gradient Q-learning often works spectacularly in practice, especially with stabilizers, but this chapter explains why broad convergence claims are difficult.

09 Practical stabilizers do not erase the theory

10 Questions a master’s student should be able to answer

11 Chapter summary and exam-ready map

• Mismatch: b supplies data; π defines targets; ratios correct actions but not the entire state geometry.
• Deadly triad: approximation, bootstrapping, and off-policy learning jointly permit divergence.
• Counterexample: Baird proves structural failure with linear features and zero rewards.
• Geometry: approximation seeks a projected Bellman fixed point, and off-policy projection may lose contraction.
• MSPBE/GTD: a defined projected objective plus an auxiliary estimator restores linear convergence guarantees.
• Emphasis: interest and follow-on dependencies construct stable target-relevant state weighting.
• Control: changing policies add another moving part; practical success must be demonstrated with stability diagnostics.

12 Source trail and scope

These are independent companion notes, not a replacement for the textbook. The chapter organization follows Sutton and Barto's second edition; explanations and examples here are original. Use the authors' book page, the MIT Press edition page, and the official table of contents for the primary source and exact section sequence.