Chapter 1 — Introduction, explained.

At 2:13 a.m., Asha's warehouse robot freezes in front of a blocked aisle. She is a master's student building her first autonomous controller. Turning left looks slower now but avoids a dead end; turning right looks fast but may trap the robot behind traffic. Her supervisor asks a deceptively simple question: “What is the correct action?” Asha realizes she cannot label the right motor command for every possible warehouse situation. The robot has to act, observe what follows, and let success or failure change later choices. That is the problem this chapter derives from first principles: sequential decisions whose consequences must be discovered through experience.

01 What reinforcement learning actually is

Reinforcement learning is a computational way to study goal-directed learning and decision-making. The learner is an agent; everything it interacts with is the environment. The environment presents a situation, the agent chooses an action, and the environment responds. This repeats, so today's action changes not only today's reward but also the situations available tomorrow.

That last point separates RL from a one-shot prediction problem. A spam classifier predicts a label but does not normally change the next email by making that prediction. A delivery robot changes its location when it moves. A recommender changes what a user sees and therefore what future data will be collected. In RL, the learner is inside the data-generating loop.

02 The agent–environment loop

The interface is deliberately small. At time step t, the agent receives state S_t and selects action A_t. One step later, the environment returns reward R_t+1 and the next state S_t+1. The subscript on the reward is worth noticing: R_t+1 is the reward produced after taking A_t.

The state is the information the agent uses to choose. It may be a board position, sensor readings, account history, or a learned representation. The action may be discrete (“left” or “right”) or continuous (a steering angle). The reward is one number, even when the real objective combines speed, cost, safety, and quality. Designing these boundaries is part of the problem, not administrative setup.

For Asha, the robot is the agent and the warehouse—including people, shelves, traffic, and battery physics—is the environment. A useful state might contain position, heading, battery, load, nearby obstacles, and current job priority. Actions could be motion commands, waiting, rerouting, or charging. If she quietly omits battery health, the interface still runs, but the “state” is not sufficient: two observations that look identical can have very different safe actions. This is the first master's-level lesson—the formalism is compact because difficult modelling decisions have been pushed into the definitions.

03 How RL differs from other machine learning

Supervised learning learns from examples paired with correct targets. If an image is labelled “cat,” the learner can compare its prediction directly with that answer. In RL there is usually no label saying “A_t should have been left.” There is only a consequence, perhaps many steps later. The agent must work out which earlier decisions deserve credit or blame.

Unsupervised learning finds structure in unlabelled data—clusters, representations, or densities—but does not by itself define a goal-directed interaction problem. RL can use supervised and unsupervised components, yet its central question remains: which action should be taken to increase future reward?

Planning traditionally assumes a known model: the agent can reason about what each action will cause. RL does not require that knowledge up front. It may learn without a model, learn a model from experience, or combine learning and planning. This is why Sutton and Barto treat the line between trial-and-error learning and planning as a spectrum rather than two unrelated fields.

04 The four elements: policy, reward, value, model

A policy, written π, is the agent's behavior rule. A deterministic policy picks one action for a state. A stochastic policy gives probabilities, such as π(left|s)=0.7 and π(right|s)=0.3. A policy can be a table, a neural network, or an expensive search procedure. Whatever its implementation, it is the component that directly controls behavior.

A reward signal defines the objective through immediate numbers. It says what events are desirable now, not how to obtain them. A robot might receive +100 for a completed delivery, −1000 for a collision, and −1 per second. The agent's job is not to maximize the next reward in isolation; it is to maximize accumulated reward over time.

A value function predicts that accumulated future reward. State value asks, “How good is it to be here under this policy?” Action value asks, “How good is it to take this action here and then continue under the policy?” Values are harder to learn than rewards because they summarize consequences that have not happened yet—but values are what make farsighted decisions possible.

A model predicts how the environment responds: what next state and reward may follow an action. Models enable planning with imagined experience. Model-free agents learn policies or values directly from real transitions. Neither family is automatically superior; the useful choice depends on model accuracy, data cost, and available computation.

Asha now has four separate research questions. Her policy decides whether the robot turns or waits. Her reward encodes delivery progress, energy use, time, and safety. Her value function estimates whether a location is promising after considering the route ahead. Her model predicts congestion and battery change. If an experiment fails, this separation tells her which claim failed; without it, “the agent is bad” is not a diagnosis.

05 Reward is not value—and the difference is everything

Imagine a chess move that sacrifices a queen but forces checkmate. Its immediate appearance is bad; its long-term value is excellent. Conversely, grabbing a small reward can enter a state from which future rewards are impossible. Reward reports the immediate event. Value predicts the future stream created by that event and everything after it.

The future stream is called the return. Later chapters define its exact forms, but the idea is simple: add future rewards, often discounting distant ones by a factor γ. A compact episodic form is:

When γ is near zero, the agent is short-sighted. When γ is near one, distant rewards matter strongly. Discounting can express uncertainty or time preference and can keep infinite continuing sums finite. It does not fix a poorly chosen reward.

Asha sees the distinction immediately. A “move forward” action earns +2 for progress but enters the blocked aisle, producing a long delay. A “turn left” action earns no progress now but opens a reliable route to the destination. Immediate reward favors forward; return and value favor left. The algorithm has not become mysterious—it is simply optimizing the entire consequence chain rather than the next number.

06 Exploration versus exploitation

To obtain reward, the agent should choose actions already known to be good—exploitation. To discover whether something better exists, it must sometimes choose uncertain actions—exploration. These goals conflict because every exploratory action spends a real opportunity.

A restaurant recommendation system that always recommends the current winner may never learn that a new restaurant is better. A system that explores constantly annoys users with weak recommendations. RL methods make this tradeoff explicit. Chapter 2 removes state entirely and studies it through multi-armed bandits; later chapters handle exploration when actions also change future states.

In the warehouse, blind exploration is unacceptable: Asha cannot let a loaded robot try random high-speed turns near workers. Her exploration space must already respect hard safety constraints. Within that safe set, the agent may compare two routes or waiting strategies. This distinction—exploration chooses among allowed actions; it does not suspend engineering constraints—will matter throughout the series.

07 Tic-tac-toe: the whole subject in miniature

Consider an agent playing noughts and crosses. A state is a board position. An action is an empty square. A terminal reward can be +1 for a win, 0 for a draw, and −1 for a loss. The policy usually chooses a move leading to the highest-valued next position, but occasionally explores another legal move.

At first, the values are guesses. After each greedy move, the agent updates the previous state's value toward the value of the state that followed:

The bracket is a prediction difference; α is the step size. If the next position looks better than expected, the earlier position moves upward. If it looks worse, the estimate moves downward. Terminal states provide the grounded win/draw/loss values, and those values propagate backward through repeated games.

This tiny learner contains ideas that recur throughout the book: learning from experience, estimating value, backing up later information to earlier states, acting mostly greedily, deliberately exploring, and improving behavior without a teacher listing the correct move. It also shows why self-play is powerful: as the agent improves, its opponent improves too, automatically creating harder experience.

08 Scope, historical roots, and what RL does not solve

RL assumes that a scalar reward can meaningfully express the objective and that experience is available. Those assumptions can be expensive or unsafe in medicine, finance, infrastructure, and robotics. An online learner cannot casually explore a treatment, delete production data, or crash a vehicle. Simulators, logged data, human oversight, constraints, and offline evaluation become essential.

The framework also does not decide the correct state representation. If a thermostat sees temperature but not whether a window is open, the same observation can require different actions. Hidden information makes the problem partially observable. Memory or learned state must summarize relevant history.

Finally, RL is not synonymous with deep RL. Tables, linear functions, search, and hand-designed features are central to understanding the algorithms. Neural networks help scale representations; they do not remove delayed credit, exploration, distribution shift, or reward-design problems.

The chapter also places RL at the meeting point of three historical threads. Trial-and-error learning contributed the idea that actions are strengthened or weakened by consequences, from animal-learning theories to learning automata. Optimal control and dynamic programming contributed value functions, state-based decision processes, and Bellman's principle of optimality. Temporal-difference learning contributed the idea of updating one prediction from a later prediction before the final outcome is known. Modern RL unifies these threads: it learns through trial and error, reasons with values, and propagates predictive errors through time.

09 Questions learners actually ask

10 Key takeaways

• RL studies sequential decisions: actions change both immediate reward and the future situations the agent will face.
• The core interface is state → action → reward + next state, repeated as a closed feedback loop.
• A policy chooses, reward defines the immediate objective, value predicts long-term return, and a model predicts consequences.
• Reward and value are different: a low-reward step can have high value if it leads to excellent future outcomes.
• There is no correct-action label, so the learner must solve temporal credit assignment and balance exploration with exploitation.
• The tic-tac-toe update demonstrates value learning, bootstrapping, controlled step sizes, exploration, and self-play in one small example.
• Representation, safety, evaluation, and reward design remain real engineering problems; “use deep RL” is not a substitute for solving them.

11 Wrapping up and source trail

Chapter 1 gives the map for everything ahead. The field is not a bag of algorithms; it is one interaction problem viewed through policies, rewards, values, models, and the tension between present evidence and future consequence. Next comes the cleanest possible version of that tension: multi-armed bandits, where there is only one situation and the entire challenge is learning which action to choose.

These are independent companion notes written in original language, not a replacement for the textbook. Primary references: the authors' second-edition page, the author-hosted open-access draft, the MIT Press edition page, and the official contents.