RL question bank — questions only, no answers.

A large bank of exam-style questions pulled straight from the CS1–CS7 slides — grouped by lecture and tagged concept, derive, numerical, or scenario. Questions only, no answers. Work them cold; then check against the slides explained, formula sheet, and book explained.

CS1 — introduction & elements

1. Define reinforcement learning. Why is it called feedback-based (or reward-based) learning? concept
2. Is RL a type of neural network, or an alternative to one? Justify in two lines. concept
3. Distinguish evaluative from instructive feedback, and say which one RL uses. concept
4. Compare supervised, unsupervised, and reinforcement learning across: feedback, data, objective, type of problem, and one example algorithm each. concept
5. State three situations in which RL is the appropriate tool. concept
6. Draw the agent–environment interaction loop, labelling state, action and reward, and write the resulting trajectory of symbols. concept
7. Name and define the four sub-elements of an RL system beyond the agent and environment. concept
8. Explain the difference between a reward signal and a value function. Can a state with low immediate reward have high value? Can a high-reward state have low value? Give an intuition for each. concept
9. "The most important component of almost all RL algorithms is a method for efficiently estimating values." Explain why values, not rewards, are what we estimate and decide by. concept
10. What is a model of the environment, and what is it used for? Distinguish model-based from model-free methods. concept
11. List the four characteristics of RL (no supervision, sequential decisions, time, delayed feedback) and explain why delayed feedback creates the credit-assignment problem. concept
12. Using a child learning to ride a bicycle, identify the agent, an action, a state, and a reward. scenario
13. What is the exploration–exploitation dilemma, and why does it not arise in supervised learning? concept
14. In the tic-tac-toe example, what does each table entry represent, and how are the entries initialized? scenario
15. Write the tic-tac-toe value-update rule and identify the step size and the error term. Why is it called a temporal-difference method? derive

CS2–CS3 — multi-armed bandits

1. State the \(k\)-armed bandit problem and its objective. Why is it studied before full RL? concept
2. Define the value of an action \(q_*(a)\). If every \(q_*(a)\) were known, what would the optimal strategy be? concept
3. Write the sample-average estimate \(Q_t(a)\). Given pulls of arm \(a\) returning \(-1, -1, 5\), compute \(Q(a)\). Then arm \(b\) returns \(-0.2, -0.2\); compute \(Q(b)\). numerical
4. Derive the incremental update \(Q_{n+1}=Q_n+\tfrac1n(R_n-Q_n)\) from the definition of the average. What general form does it take? derive
5. Define greedy and \(\varepsilon\)-greedy action selection. State the exploration–exploitation trade-off in terms of immediate vs long-run reward. concept
6. Why does \(\varepsilon\)-greedy guarantee that every \(Q_t(a)\) converges to its \(q_*(a)\)? derive
7. Ex-1: In \(\varepsilon\)-greedy with two actions and \(\varepsilon=0.5\), what is the probability the greedy action is selected? Show the full conditioning argument. numerical
8. Repeat Ex-1 for \(k\) actions and general \(\varepsilon\): give \(P(\text{greedy})\) and \(P(\text{a specific non-greedy action})\). numerical
9. For a nonstationary bandit, why is a constant step size \(\alpha\) preferred over \(\tfrac1n\)? Show that constant \(\alpha\) yields an exponential recency-weighted average. derive
10. State the two conditions on a step-size sequence \(\{\alpha_n\}\) that guarantee convergence with probability 1. Does \(\alpha_n=\tfrac1n\) satisfy them? Does a constant \(\alpha\)? concept
11. Explain how optimistic initial values encourage exploration without using \(\varepsilon\). Why is this trick poorly suited to nonstationary problems? concept
12. Write the UCB action-selection rule and explain the role of the square-root term and the constant \(c\). derive
13. Describe the 10-armed testbed: how are the true values and the rewards generated, and how is a method's average behaviour measured? scenario
14. On the 10-armed testbed, contrast the long-run performance of greedy, \(\varepsilon=0.1\), and \(\varepsilon=0.01\). When does more exploration help? concept
15. What is a gradient-bandit algorithm? Write the soft-max selection probability. derive
16. Define associative search (contextual bandit) and explain how it bridges bandits to the full RL problem. concept

CS3–CS5 — MDPs, returns & value functions

1. List the components that define a finite MDP. concept
2. Write the dynamics function \(p(s',r\mid s,a)\) and state the constraint it must satisfy. State the Markov property in words. derive
3. From \(p(s',r\mid s,a)\), derive the state-transition probability \(p(s'\mid s,a)\) and the expected reward \(r(s,a)\). derive
4. For the noisy grid world, give the movement probabilities for action "North" and the per-step and terminal rewards. Why does the noise matter? scenario
5. For the recycling robot: list its states, the actions in each, and explain why \(r_{\text{search}} > r_{\text{wait}}\) and why a \(-3\) reward appears. Reconstruct the transition table. scenario
6. State the reward hypothesis. Give two examples (e.g. chess, vacuum cleaner) where rewarding a proxy instead of the goal causes the wrong behaviour. concept
7. Define the return \(G_t\) for an episodic task and for a continuing task. Why does a continuing task require discounting? concept
8. Write the discounted return and derive the recursion \(G_t=R_{t+1}+\gamma G_{t+1}\). derive
9. For a constant reward of \(+1\) every step with \(\gamma<1\), evaluate \(G_t\). What does \(\gamma=0\) and \(\gamma\to1\) mean behaviourally? numerical
10. Define a policy \(\pi(a\mid s)\), the state-value \(v_\pi(s)\), and the action-value \(q_\pi(s,a)\). Give the relation between \(v_\pi\) and \(q_\pi\). concept
11. Derive the Bellman expectation equation for \(v_\pi(s)\) from its definition, and explain each factor in words. derive
12. Write the Bellman optimality equation for \(v_*(s)\) and for \(q_*(s,a)\). What single change distinguishes them from the expectation equations? derive
13. Given \(v_*\), how do you read off an optimal policy? Given \(q_*\), how? Why is a one-step-greedy policy on \(v_*\) optimal in the long run? concept
14. State the three assumptions under which the Bellman optimality equation can be solved exactly, and explain why each typically fails in practice. concept

CS4–CS6 — dynamic programming

1. What does iterative policy evaluation compute, and what backup does it repeat? Write the update. derive
2. Why can policy evaluation be done "in place" with a single array, and why does that often converge faster? concept
3. State the policy improvement step and the policy improvement theorem precisely. derive
4. Describe policy iteration as a sequence of evaluations and improvements. Why must it terminate in finitely many iterations for a finite MDP? concept
5. Write the value-iteration update. How does it relate to policy evaluation and improvement, and how do you extract the optimal policy afterwards? derive
6. What is asynchronous DP, and what condition must it satisfy to still converge? concept
7. Define generalized policy iteration (GPI). In what sense do evaluation and improvement both compete and cooperate? concept
8. What does it mean for a method to "bootstrap"? Does DP bootstrap? concept
9. State two limitations of DP (model requirement, cost / curse of dimensionality) that motivate model-free methods. concept

CS6–CS7 — Monte Carlo methods

1. What is the core idea of Monte Carlo value estimation? What kind of task does it require, and why? concept
2. Distinguish first-visit from every-visit Monte Carlo prediction. Why does first-visit MC converge by the law of large numbers? derive
3. Why do MC control methods estimate action values \(q_\pi(s,a)\) rather than state values? concept
4. State the problem of maintaining exploration in MC and the two general ways to solve it. concept
5. What does "exploring starts" assume, and why is it often impractical for learning from real interaction? concept
6. Define an \(\varepsilon\)-soft policy. Give the selection probability of the greedy and of each non-greedy action under \(\varepsilon\)-greedy. numerical
7. Contrast on-policy and off-policy MC control. Define the target policy and the behaviour policy. concept
8. Write the importance-sampling ratio \(\rho_{t:T-1}\). Why do the environment's transition probabilities cancel? State the coverage assumption. derive
9. Contrast ordinary and weighted importance sampling on bias and variance. Which is preferred in practice? concept
10. Compare DP and MC on: model requirement, bootstrapping, when updates happen, and variance. Name one advantage MC has over DP. concept
11. The mid-sem ends with the open question "can we learn from experience and bootstrap?" Which family of methods answers it, and what do they combine? concept

After you've attempted these

• Slides Explained · full worked concepts
• Formula Sheet · check your equations
• Cheatsheet · one-card recap
• Book Explained · deep per-chapter answers in prose