The History of Reinforcement Learning

Reinforcement Learning (RL) is a paradigm of machine learning in which an agent learns to make decisions by interacting with an environment, receiving rewards or penalties based on its actions, and gradually improving its strategy over time. It matters profoundly in modern AI because it underlies breakthroughs from game-playing systems like AlphaGo to robotics and large language model alignment. The counterintuitive fact: RL's foundational ideas predate computers entirely — they trace back to 19th-century animal psychology experiments on trial-and-error learning.

The core insight

The central "aha moment" in reinforcement learning is that intelligence can emerge purely from feedback — an agent does not need to be told the right answer, only whether its outcome was good or bad. Like a child learning to walk by falling and trying again, an RL agent explores its environment, collects signals about what worked, and slowly builds a policy that maximizes long-term reward. The key shift from supervised learning is that the agent generates its own training data through interaction, meaning the quality of what it learns depends entirely on how it explores. This makes RL both incredibly powerful and notoriously difficult to train reliably.

How It Works

Reinforcement learning is built around a loop: an agent observes the current state of an environment, takes an action, receives a reward, and moves to a new state. The goal is to learn a policy — a mapping from states to actions — that maximizes total reward over time. The fundamental quantity an agent tries to maximize is called the return — the cumulative reward from a point in time forward.

1
Gt=rt+1+γ⋅rt+2+γ2⋅rt+3+⋯=∑k=0∞γk⋅rt+k+1\quad G_t = r_{t+1} + \gamma \cdot r_{t+2} + \gamma^2 \cdot r_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^k \cdot r_{t+k+1}

Explanation: The return at time t is the sum of all future rewards, but rewards further in the future are shrunk by the discount factor γ (gamma), making immediate rewards worth more than distant ones.

  • G_t: Total discounted return starting from timestep t
  • r_{t+k+1}: Reward received k steps in the future
  • γ (gamma): Discount factor, a number between 0 and 1 (e.g., 0.99); closer to 1 means the agent cares a lot about the future

The agent learns by estimating how good it is to be in a particular state — called the value function:

2
V(s)=E[Gt∣st=s]\quad V(s) = \mathbb{E}\left[G_t \mid s_t = s\right]

Explanation: The value of a state s is the expected (average) total future return the agent will accumulate starting from that state.

  • V(s): Value of state s — how much reward we expect to get from here
  • 𝔼[...]: Expected value (average over many possible futures)
  • s_t = s: Conditional on being in state s at time t

The simplest RL update rule is temporal difference (TD) learning, which updates value estimates using a one-step look-ahead:

3
V(s)←V(s)+α[r+γ⋅V(s′)−V(s)]\quad V(s) \leftarrow V(s) + \alpha \left[ r + \gamma \cdot V(s') - V(s) \right]

Explanation: The agent nudges its estimate of V(s) toward a better estimate formed by actually observing reward r and the value of the next state s'.

  • α (alpha): Learning rate — how big a step to take toward the new estimate
  • r: Reward just received
  • s': The next state arrived at after taking an action
  • r + γ·V(s') − V(s): The TD error — the difference between what we expected and what we got

Here is that same TD update in plain Python: ```python

# Simple TD(0) value update alpha = 0.1 # learning rate gamma = 0.99 # discount factor

# Agent is in state s, takes action, gets reward r, lands in state s_prime def td_update(V, s, r, s_prime, alpha=0.1, gamma=0.99): td_error = r + gamma V[s_prime] - V[s] V[s] = V[s] + alpha td_error return V

# Example: tiny grid world with 5 states (0-4), goal at state 4 import random V = {i: 0.0 for i in range(5)} # initialise all values to 0

# Simulate one episode of random exploration state = 0 for step in range(20): action = random.choice([-1, 1]) # move left or right next_state = max(0, min(4, state + action)) reward = 1.0 if next_state == 4 else 0.0 # reward only at goal V = td_update(V, state, reward, next_state) state = next_state print("Learned values:", V) ```

researcher · Richard S. Sutton | b. 1956 | Temporal Difference Learning and the Modern RL Framework

Richard Sutton is widely regarded as the father of modern reinforcement learning, having spent his career at UMass Amherst, AT&T Bell Labs, and the University of Alberta. His 1988 paper introducing TD(λ) — a family of temporal difference learning algorithms — provided the mathematical backbone that unified trial-and-error learning with dynamic programming, making RL tractable. Remarkably, Sutton co-authored the definitive textbook Reinforcement Learning: An Introduction with Andrew Barto, which has been freely available online since 2018 and has been downloaded millions of times.

researcher · Andrew G. Barto | b. 1948 | Actor-Critic Methods and Foundational RL Theory

Andrew Barto, professor emeritus at UMass Amherst, mentored Richard Sutton and together they built many of the theoretical pillars of RL throughout the 1980s. His 1983 paper with Sutton and Anderson introduced the actor-critic architecture — splitting the learning agent into a policy-learner (actor) and a value-estimator (critic) — a design that persists in cutting-edge systems like Proximal Policy Optimization today. Barto's early inspiration came from neuroscience: he was fascinated by how dopamine neurons in the brain fire in patterns eerily similar to TD error signals, long before neuroscientists confirmed this connection experimentally.

researcher · Christopher Watkins | b. 1962 | Q-Learning

Christopher Watkins developed Q-learning in his 1989 Cambridge PhD thesis, providing the first provably convergent, model-free algorithm for learning optimal action-value functions. His key insight was that an agent could learn the value of actions (not just states) without needing a model of the environment's dynamics, making RL applicable to problems where the rules are unknown. Q-learning became so foundational that decades later it powered DeepMind's Deep Q-Network (DQN), the first system to surpass human performance on a broad range of Atari games.

Figure
The Reinforcement Learning Timeline: Key Eras
1950s 1970s 1988 1992–98 2013–15 2016–Now Minsky's SNARC Neural stochastic machine Bellman's DP Value functions born Sutton TD(λ) 1988 TD learning unified Q-Learning + TD-Gammon First game superhuman RL DQN — DeepMind 2013 Deep RL revolution begins AlphaGo + RLHF/LLMs RL meets language models Reinforcement Learning: Key Historical Milestones
Figure
The RL Agent-Environment Loop
AGENT Policy π(a|s) Value V(s) ENVIRONMENT State transitions Reward function Action aₜ State sₜ₊₁ Reward rₜ₊₁ The RL Interaction Loop At each timestep t, the agent observes state, acts, receives reward

Intuition

The Mental Model

Imagine you are dropped into a completely dark room with a game controller you've never seen before. You press buttons at random — sometimes a pleasant chime plays (reward!), sometimes nothing happens, sometimes you hear a buzzer (penalty). Over thousands of button presses, you start to notice patterns: pressing the red button after a green light usually leads to the chime. You don't need anyone to explain the rules — you discover them through experience and feedback. That is reinforcement learning in a nutshell. The state is what you currently observe (the color of the light, your position). The action is what button you press. The reward is the chime or buzzer. The policy is the strategy your brain builds: "when I see green, press red." The discount factor is why you prefer getting the chime now rather than waiting — a reward today is worth more than the same reward ten minutes from now. The hardest part — and the part that tripped up researchers for decades — is the credit assignment problem: if you finally got a chime after 50 button presses, which of those presses actually caused it? This is why learning is slow, and why good reward shaping is more art than science.

Historical Development

Origins · 1 minhow this was discovered
History
1911
Edward Thorndike publishes his Law of Effect — behaviors followed by satisfying outcomes are more likely to recur. This is the psychological bedrock of all RL; the agent/environment loop is a formalization of this idea.
1950
Alan Turing's paper "Computing Machinery and Intelligence" suggests machines could learn through a reward-penalty scheme — one of the first explicit computational descriptions of what we now call reinforcement learning.
1951
Marvin Minsky builds the SNARC (Stochastic Neural Analogy Reinforcement Computer), a physical machine with 40 artificial neurons using reinforcing signals to learn how to navigate a maze — arguably the first hardware RL system.
1957–1966
Richard Bellman develops Dynamic Programming and introduces the Bellman equation, formalizing the concept of value functions and optimal policies. His work on Markov Decision Processes gives RL its mathematical language.
1983–1988
Barto, Sutton, and Anderson publish actor-critic methods (1983); Sutton's landmark 1988 paper introduces Temporal Difference (TD) learning and TD(λ), unifying DP with trial-and-error learning and making RL computationally tractable.
1989–1992
Christopher Watkins' PhD thesis introduces Q-learning (1989), proven to converge to optimal policies; Gerald Tesauro's TD-Gammon (1992) uses TD learning to reach near-grandmaster level at backgammon — the first RL system to stun the world.
1998
Sutton and Barto publish Reinforcement Learning: An Introduction, consolidating 15 years of research into a unified textbook that defines the field for a generation of researchers.
2013–2015
DeepMind publishes the Deep Q-Network (DQN) paper, combining Q-learning with deep convolutional neural networks to master 49 Atari games from raw pixels — launching the Deep RL era and triggering enormous investment in the field.
2016–2017
DeepMind's AlphaGo defeats world champion Lee Sedol at Go using a combination of deep RL and Monte Carlo Tree Search — a landmark moment widely covered in the 2017 documentary AlphaGo, bringing RL into mainstream culture.
2022–2024
RLHF (Reinforcement Learning from Human Feedback) becomes central to aligning large language models; ChatGPT, Claude, and Gemini all use RL-based fine-tuning to shape model behavior — making RL invisible infrastructure for billions of users.

Why It Matters

Real-World Impact
AlphaGo / AlphaZero (DeepMind, 2016–2017)Used policy gradient RL and self-play to master Go, Chess, and Shogi — games long thought too complex for AI. AlphaZero trained entirely through self-play with no human game records, discovering strategies human professionals had never seen.
ChatGPT and Claude (OpenAI / Anthropic, 2022–present)Both use RLHF — human raters score model outputs, and RL trains the model to produce responses humans prefer. Without this RL step, early GPT-4-class models were prone to harmful outputs and poor instruction-following.
Atari DQN (DeepMind, 2013)The first demonstration that a single RL algorithm could learn to play 49 different video games from raw screen pixels, using experience replay and target networks — core techniques still used in modern deep RL systems.
OpenAI Five (OpenAI, 2019)Defeated world-champion Dota 2 teams using Proximal Policy Optimization (PPO), training for the equivalent of 45,000 years of game experience in real-world months via massive parallelism.
Robotics at Boston Dynamics and Google DeepMindSim-to-real RL pipelines train robot locomotion policies in simulation for millions of steps, then transfer them to physical hardware — enabling robots to walk, run, and recover from pushes without hand-coded controllers.
Recommendation Systems (YouTube, Netflix, TikTok)Model user engagement sessions as sequential decision problems; RL-based rankers optimize for long-term watch time rather than just immediate click probability, fundamentally changing how content is surfaced at scale.
AlphaFold 3 / Drug DiscoveryRL-inspired techniques guide molecular generation processes, treating molecule construction as a sequential decision problem where the reward is predicted binding affinity or stability — accelerating drug discovery pipelines.

Prerequisites

What This Leads To

See Also

Markov Decision Processes · Q-Learning and Deep Q-Networks · Policy Gradient Methods · Reinforcement Learning from Human Feedback (RLHF) · AlphaGo and Game-Playing AI · Temporal Difference Learning · Exploration vs. Exploitation Tradeoff