Sutton & Barto Chapter 13 — Policy Gradient Methods Explained

// the one-minute version

Policy-gradient methods parameterize π(a|s,θ) directly and ascend expected performance. The likelihood-ratio identity turns an unknowable derivative of trajectory probability into sampled terms ∇logπ multiplied by return or action value. The policy-gradient theorem removes the need to differentiate the state distribution. REINFORCE is unbiased but noisy; subtracting a state baseline preserves expectation and turns return into advantage. An actor–critic replaces the completed return with a critic’s TD error for online, lower-variance learning. Softmax policies handle discrete actions; Gaussian policies produce continuous actions.

Asha’s robot must choose a steering angle, not one of five labelled buttons. A Q-function plus argmax would require solving a continuous optimization problem at every step. Worse, a deterministic winner can jump abruptly when two approximate values cross. She instead builds a policy that emits a distribution over steering directly. Now the learning question is beautifully sharp: how should its parameters move so successful behavior becomes more probable?

01 Why represent the policy itself?

A parameterized policy π(a|s,θ) changes smoothly with θ, naturally stays stochastic, and can generate continuous actions without maximizing a learned surface. Stochasticity is useful when the optimal behavior is genuinely mixed, observations alias different states, or exploration must remain part of behavior.

The objective J(θ) is expected return from the task’s start distribution in episodic problems, or average reward in continuing problems. Unlike supervised learning, there is no label saying which action was correct. The agent must infer a direction from sampled consequences.

02 The log-derivative trick creates a learnable signal

A trajectory’s probability is a product of policy probabilities and environment transitions. The environment dynamics do not depend on θ, but differentiating a long product is awkward. Use ∇p=p∇log p. The log of a product becomes a sum, yielding one score term per selected action.

score function∇_θlogπ(A_t|S_t,θ) says which parameter change would make the sampled action more likely.

Multiply that direction by later return. Good outcomes reinforce their sampled actions; poor outcomes suppress them. This is credit assignment without differentiating through unknown environment transitions.

03 The policy-gradient theorem removes a hidden derivative

Changing θ changes actions, which changes future states, so one might expect to differentiate the state-visitation distribution. The theorem shows those effects collapse into action values:

policy gradient∇J(θ) ∝ Σ_sμ(s)Σ_aq_π(s,a)∇π(a|s,θ) = E[q_π(S,A)∇logπ(A|S,θ)].

The proportionality depends on episodic versus continuing normalization, but the ascent direction is usable. Sample a visited state-action pair, estimate its q value, multiply by the score. The theorem is the bridge from a global performance objective to local online updates.

The critic evaluates; the actor improves. Their coupling is generalized policy iteration in parameter space.

04 REINFORCE: Monte Carlo policy gradient

Replace q_π(S_t,A_t) with the sampled return G_t:

REINFORCEθ←θ+αγ^tG_t∇logπ(A_t|S_t,θ).

The γ^t factor appears when the start-state objective weights later visited states differently; implementations must be explicit about the objective convention. REINFORCE is unbiased under on-policy sampling, but it waits for episode completion and returns vary because of both action choices and environment randomness.

05 Baselines reduce variance without changing the mean

Subtract any b(s) independent of the chosen action:

baseline identityE[b(S)∇logπ(A|S) | S]=b(S)Σ_aπ(a|S)∇logπ(a|S)=b(S)∇1=0.

Therefore G_t−b(S_t) has the same expected policy gradient. Choosing b≈v_π makes the multiplier an advantage estimate: was this action better or worse than normal for this state? A learned baseline has its own regression update and must not backpropagate into the actor through the sampled action unless the derivation supports it.

06 Actor–critic trades unbiased returns for online TD errors

A one-step critic estimates v_π. Its TD error δ=R+γv̂(S′,w)−v̂(S,w) is an estimate of action advantage. Update the critic with α_wδ∇v̂ and the actor with α_θδ∇logπ. The method learns every step and usually has lower variance, but bootstrap and critic approximation introduce bias.

Eligibility traces yield actor–critic with traces: one trace assigns credit to recent policy scores, another trains the critic. Continuing formulations replace return with average reward and use the same differential TD logic as Chapter 10.

07 Discrete actions: softmax preferences

Let h(s,a,θ) be a preference, not a value. A softmax policy is exp(h_a)/Σ_bexp(h_b). Its score gradient increases features for the selected action and decreases their policy-weighted expectation across actions. This preserves normalization automatically.

Temperature rescales preferences: high temperature spreads probability; low temperature approaches greedy choice and can starve alternatives of gradient. Use a numerically stable log-softmax by subtracting the maximum preference.

08 Continuous actions: parameterize a density

A Gaussian policy can output mean μ(s,θ) and positive standard deviation σ(s,θ), usually via log σ. Sample a torque a∼N(μ,σ²). The log-probability gradient moves μ toward successful samples and adjusts σ according to whether useful actions lie farther from or nearer the mean.

Exploration is learned, but unconstrained variance can collapse too early or explode. Bounded actuators require clipping or a squashing transform; then the density and log-probability must account for that transformation if an exact gradient is intended.

09 Asha’s steering experiment

walk it throughThe actor emits Gaussian steering with mean and log standard deviation. The critic predicts route return. A turn that avoids congestion yields positive δ, increasing its log-probability; an equally positive raw reward can produce negative δ if the critic expected even more. Learning follows surprise relative to context.

gotchasPolicy gradients find local stationary points, not guaranteed global optima. Report entropy or action variance, gradient norms, and seed distributions. Normalize advantages only with care and document it. A weak critic can confidently send the actor the wrong way; actor and critic learning rates must be tuned jointly.

10 Questions a master’s student should be able to answer

Why take the gradient of log probability?

It converts ∇π into π∇logπ, allowing an expectation over actions already sampled from π. It also converts trajectory probability products into sums of local score terms.

Can the baseline depend on action?

Not arbitrarily. A state-only baseline integrates to zero under the policy. An action-dependent control variate needs an additional correction; otherwise it can bias the gradient.

Is the critic optimizing the same objective as the actor?

Not directly. The critic minimizes a prediction objective so that its values or TD errors provide useful estimates for the actor’s performance gradient. Compatible approximation creates a tighter theoretical relationship.

Why keep a stochastic policy after training?

The task may require randomization, observations may be ambiguous, or continued exploration may matter. For deployment one may use the mean or mode, but that is a different policy and should be evaluated separately.

What is the fairest REINFORCE baseline experiment?

Use identical trajectories or seeds where possible, compare no baseline against a learned state-value baseline, and report gradient variance, learning speed, final return, and multiple-seed confidence intervals.

11 Chapter summary and exam-ready map

Direct policies: smooth stochastic and continuous control without an argmax.
Score function: ∇logπ is the local direction that raises sampled-action probability.
Theorem: expected q times score is the performance gradient despite changing state visitation.
REINFORCE: complete returns give an unbiased, high-variance estimator.
Baseline: a state-only subtraction preserves expectation and exposes advantage.
Actor–critic: TD error supplies online lower-variance actor updates at the cost of critic bias.
Parameterization: softmax and Gaussian policies make action constraints and exploration part of the model.

// master’s study sheetchapter 13

derive

likelihood ratioDerive ∇E[R]=E[R∇log p] and specialize trajectory scores to policy terms.

baselineProve a state-only baseline has zero expected score contribution.

implement

experimentCompare REINFORCE, REINFORCE with baseline, and one-step actor–critic across seeds; report entropy and gradient variance.

12 Source trail and scope

These are independent companion notes, not a replacement for the textbook. The chapter organization follows Sutton and Barto's second edition; explanations and examples here are original. Use the authors' book page, the MIT Press edition page, and the official table of contents for the primary source and exact section sequence.

Chapter 13 — Policy Gradient Methods, explained.