Reinforcement Learning

How an agent learns to act from rewards, from bandits to PPO

About this course

Learning to act from consequences — bandits, Markov decision processes and dynamic programming, Monte Carlo and temporal-difference methods, function approximation and deep Q-networks, policy gradients, actor-critic methods, model-based planning and offline evaluation.

What you will be able to do

  • Balance exploration and exploitation in a bandit problem, and formulate a task as a Markov decision process
  • Solve a small MDP with the Bellman equations and dynamic programming
  • Implement Monte Carlo, TD, SARSA and Q-learning, and distinguish on-policy from off-policy control
  • Derive the policy gradient and apply REINFORCE, actor-critic and PPO updates
  • Evaluate an agent offline, identify reward-design flaws, and audit its behaviour

What this course covers

  • Function approximation and parameter sharing
  • Linear predictors: states, features, and weights
  • Gradient update from a fixed return target
  • Tabular learning as a special case of linear function approximation
  • Sample weighting and the best-fit compromise
  • Semi-gradient TD update vs. full residual gradient
  • Unstable replay recurrence and coverage failure
  • Practical evaluation discipline for function approximation
  • Monte Carlo prediction: returns from episodes
  • First-visit vs. every-visit MC prediction
  • Backward return calculation
  • Sample-average update and convergence caveats
  • Complete-return target vs. bootstrapping and termination semantics
  • Action value estimation and epsilon-greedy MC control
  • On-policy vs. off-policy distinction and importance sampling
  • MC methods in context: targets, limitations, and next steps
  • Dyna: integrating acting, learning, and planning
  • Models vs. value functions: distinct objects
  • Planning as value propagation through a model
  • Planning backup scheduling and ordering effects
  • Expected vs. sampled model backups and mean-state shortcut error
  • Model inaccuracy, stale models, and limits of planning
  • Information acquisition vs. computational propagation
  • Minimal Dyna agent design and audit discipline
  • The Agent–Environment Interaction Loop
  • States, Observations, and the Action Set
  • Reinforcement Learning vs. Deep Learning
  • Returns, Discounting, and the Break-Even Discount Factor
  • Value Functions and the Reward-vs.-Return Distinction
  • Exploration, Exploitation, and the ε-Greedy Rule
  • Credit Assignment and the Learning–Evaluation Distinction
  • Parameterized policy and sigmoid logit representation
  • Unbiasedness and its limits for sampled gradient steps
  • Score function gradient estimator
  • Baselines: bias-free variance reduction
  • Trajectory-level policy gradient and causal reward removal
  • Two-stage discounted policy gradient: worked derivation
  • REINFORCE implementation, on-policy data, and practical pitfalls
  • Stochastic Bandit Problem and Exploration–Exploitation Tension
  • Sample Average Estimation and Incremental Update Rule
  • Greedy Action Selection and Its Limitations
  • Epsilon-Greedy Exploration Policy
  • Confidence-Bonus (UCB-style) Action Selection
  • Posterior Sampling (Thompson Sampling) with Beta Priors
  • Pseudo-Regret, Evaluation Metrics, and Bandit Objectives
  • Temporal Difference Learning: Core Idea
  • TD vs Monte Carlo on a Deterministic Chain
  • TD Update Mechanics: Target, Error, and Common Pitfalls
  • Sampling vs Bootstrapping: TD's Two-Axis Position
  • Step Sizes, Convergence Conditions, and Implementation Checklist
  • N-Step TD Targets
  • Lambda Return: Weighted Mixture of N-Step Targets
  • Eligibility Traces and the Frozen-Table TD-Error Identity
  • Four-clock framework for deep Q-learning
  • Q-learning target semantics with a neural network
  • Experience replay: semantics, coverage, and limits
  • Mini-batch indexing, shapes, and loss conventions
  • Online network, target network, and update schedules
  • Huber loss as an alternative to squared error
  • Double DQN: separating selection from evaluation
  • Terminal mask semantics and truncation vs. termination
  • Evaluation, learning curves, and run accounting
  • Offline vs. online and off-policy distinctions
  • Identification problem from missing action coverage
  • Behavior cloning as a supervised imitation baseline
  • Value extrapolation and unsupported action estimates
  • Coverage as a relationship, not a count
  • Importance sampling for off-policy evaluation
  • Reward shaping via potential-based transformation
  • Evaluation protocol: population, selection, and uncertainty
  • Advantage Function: Definition and Properties
  • One-Step TD Error as Advantage Estimator
  • Critic Approximation Error and Actor Gradient Bias
  • Separate Actor and Critic Update Mechanics
  • Implementation Semantics: Frozen Snapshots, Normalization, and On-Policy Identity
  • Generalized Advantage Estimation (GAE)
  • Rollout Boundary Handling in GAE
  • Probability ratio as surrogate re-weighting
  • KL divergence as a policy distance measure
  • Local trust-region step in logit space
  • Clipped PPO surrogate objective mechanics
  • Clipping does not impose a hard KL constraint
  • PPO update loop implementation and bookkeeping
  • Continuous action log densities and transformation corrections
  • Three-Layer Experiment Architecture
  • Toy Task Definition and Exact Solution
  • Transition Record and Terminal Masking
  • Epsilon-Greedy Action Selection and Tie Rules
  • Tabular Q-Learning Update Mechanics
  • Reproducible Runs with Separate Random Streams
  • Learned Estimates vs. Exact Values and Policy Evaluation
  • Policy Comparison: Return, Failure Probability, and Variance
  • Experiment Design Principles and Ablations
  • Markov Decision Process Structure
  • Policies, Trajectories, and Stochastic Processes
  • Discounted Return and Partial-Sum Bookkeeping
  • State Values, Action Values, and the Bellman Expectation Equation
  • Solving the Battery MDP: Policy Evaluation
  • Policy Evaluation vs. Policy Optimisation
  • Finite-Horizon Values and True Termination vs. Truncation
  • Three continuation rules: sampled, expected, maximum
  • SARSA: on-policy sampled continuation
  • Q-learning: off-policy greedy continuation
  • Expected SARSA: policy-averaged continuation
  • Continuation policy and the route/direct example
  • Coverage, convergence, and action masks
  • Maximization bias and Double Q-learning
  • Evaluation protocol and closing synthesis
  • Dynamic programming overview and three-state planning model
  • Value tables and policies as distinct objects
  • Iterative policy evaluation and Bellman contraction
  • Policy improvement and stable tie-breaking
  • Value iteration: optimality backup and synchronous vs. in-place updates
  • Bellman residual and value-error bound
  • Policy quality vs. value accuracy, model error, and scalability

Syllabus

  1. Decisions, Bandits and MDPs

    1. L1 · Learning from Actions and Consequences
    2. L2 · Bandits and the Exploration Problem
    3. L3 · Markov Decision Processes and Returns
    4. L4 · Bellman Equations and Dynamic Programming

  2. Tabular Learning

    1. L5 · Monte Carlo Prediction and Control
    2. L6 · Temporal-Difference Learning
    3. L7 · SARSA, Q-Learning, and Off-Policy Control

  3. Function Approximation and Deep Q-Learning

    1. L8 · Function Approximation and Generalisation
    2. L9 · Deep Q-Networks and Stable Training

  4. Policy Optimisation

    1. L10 · Policy Gradients and REINFORCE
    2. L11 · Actor-Critic Methods and Advantage Estimation
    3. L12 · Trust Regions, PPO, and Continuous Control

  5. Models, Offline Learning and Evaluation

    1. L13 · Models, Planning, and Dyna
    2. L14 · Offline Learning, Reward Design, and Evaluation
    3. L15 · Building and Auditing a Reinforcement Learning Agent