Rewards and Returns

At each discrete time step $t = 0, 1, 2, \dots$, the environment sends the agent a reward $R_{t+1} \in \mathbb{R}$, a single real number, immediately after the agent takes action $A_t$ in state $S_t$ and the environment transitions to state $S_{t+1}$. (The subscript convention $R_{t+1}$, rather than $R_t$, is standard: it emphasizes that the reward is a consequence of the transition out of $S_t$, jointly determined by the state, the action, and the next state.) The reward is the only evaluative feedback the agent receives — there is no separate channel that says "that was a good idea" versus "that was efficient" versus "that was safe." Every objective the designer cares about must be folded into this one number. This design choice is elevated to a guiding principle called the reward hypothesis: > All of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (reward). The reward hypothesis is not a theorem — it is an empirical and design commitment. It says that instead of hand-crafting complex, multi-objective goal specifications, we can (and should) try to compress everything the designer cares about into a scalar reward function, and let the agent discover the behavior that maximizes its long-run accumulation. This is what makes reinforcement learning a general framework: chess (reward $+1$ for winning, $-1$ for losing, $0$ otherwise), robotics (reward for forward velocity minus energy cost), and dialogue systems (reward for user satisfaction) are all instances of the same mathematical object. Because the agent cares about the future, not just the present reward, we need a way to aggregate an entire trajectory of rewards into one number the agent can compare across actions. That aggregate is the return, denoted $G_t$, and it is formally the quantity the agent seeks to maximize in expectation. The rest of this article is devoted to defining $G_t$ correctly in the two settings you will encounter throughout RL: tasks that end, and tasks that don't.

Episodic tasks: the natural sum

An episodic task has a well-defined endpoint: a special terminal state, after which the interaction resets to a fresh, independent episode. Board games, single trials of a maze, one attempt at parking a car — these are episodic. Let the episode run for $T$ steps, ending at time $T$. The undiscounted return at time $t$ is simply the sum of all rewards received from $t+1$ up to the end of the episode:

Because $T$ is finite (with probability 1, in any reasonable episodic task), this sum is a finite sum of finite numbers, so it is always well-defined — no convergence worries.

Continuing tasks: the problem with infinite sums

A continuing task has no terminal state: a thermostat regulating temperature forever, a stock-trading agent, a server load balancer. Here $T = \infty$, and the naive analogue of the sum above,

is an infinite series. If the agent receives, say, a constant reward of $+1$ at every step forever, this sum diverges to $+\infty$. Two policies that both "succeed forever" would both score $+\infty$, and the agent would have no way to prefer one over the other — the objective becomes useless for comparison and optimization.

Discounting: the fix

The standard resolution is to introduce a discount factor $\gamma \in [0, 1]$ and define the discounted return:

The discount factor $\gamma$ determines the present value of future rewards: a reward received $k$ steps from now is worth only $\gamma^k$ of what it would be worth if received immediately.

• If $\gamma = 0$, the agent is myopic: it only cares about the very next reward, $G_t = R_{t+1}$.
• As $\gamma \to 1$, the agent becomes far-sighted, weighting distant rewards almost as heavily as immediate ones.
• For $0 \le \gamma < 1$ and rewards bounded by some constant $R_{\max}$, the series is a geometric series and is guaranteed to converge:

This single inequality is why discounting is not merely a "trick to make numbers smaller" — it is the mathematical condition that makes the return a well-defined, finite real number that the agent can meaningfully compare across different action choices, even in a task that never ends.

The recursive structure of the return

The definition above expands the return as an infinite sum, but there is an equivalent, and far more useful, recursive definition. Peel off the first term:

This identity,

says that "the return from now" equals "the immediate reward" plus "the discounted return from one step later." It holds for both episodic tasks (with the convention that $G_T = 0$ at the terminal step, since there is nothing left to accumulate) and continuing tasks (with $\gamma < 1$). This recursion is the seed from which the Bellman equation — the central recursive relationship satisfied by value functions — is built; every dynamic-programming and value-based algorithm you will encounter later exploits exactly this one-step unrolling.

A unified notation

To treat episodic and continuing tasks with one formula, it is common to allow $\gamma = 1$ for episodic tasks (finite sum, no convergence issue) and $\gamma < 1$ for continuing tasks, and to write the return uniformly as

where $T$ may be $\infty$. Some formulations unify the two cases further by introducing an absorbing terminal state that transitions to itself forever with reward $0$; then an episodic task can formally be treated as a continuing task whose tail contributes nothing to the sum, and a single $\gamma \in [0,1]$ works uniformly.

Algorithmic view: computing returns from a trajectory

Although the return is a definition, not an algorithm, computing it from a recorded trajectory (as is done constantly in Monte Carlo methods, described in later articles) is a simple backward pass that directly exploits the recursion above. ```

# Compute the return G_t for every time step t = 0, ..., T-1

# given a finite reward sequence R[1..T] and discount factor gamma. function COMPUTE_RETURNS(R[1..T], gamma): G = array of length T+1 G[T] = 0 # terminal boundary condition for t = T-1 downto 0: G[t] = R[t+1] + gamma * G[t+1] return G[0..T-1] ``` This runs in a single backward pass over the trajectory. It is correct precisely because it implements the recursion $G_t = R_{t+1} + \gamma G_{t+1}$ derived above, with the base case $G_T = 0$ encoding "no more rewards after the episode ends." For continuing tasks, one cannot literally sum to $\infty$; in practice the return is either (a) estimated by truncating the sum after enough terms that the discarded tail is provably negligible, or (b) replaced by bootstrapped estimates from a learned value function, a topic developed once value functions are introduced.

Space and time to compute returns from a trajectory. Given a recorded finite trajectory of length $T$, the backward-pass algorithm above visits each reward exactly once, doing $O(1)$ work per step (one multiply, one add), so it runs in $O(T)$ time and, if all returns are retained, $O(T)$ space (or $O(1)$ extra space if only the current $G_t$ is needed on the fly, streaming backward). This is optimal: any correct algorithm must read every one of the $T$ rewards at least once, so $\Omega(T)$ time is a lower bound, matching the $O(T)$ upper bound.

Why the backward recursion is correct. This follows from a simple inductive argument on the recursion $G_t = R_{t+1} + \gamma G_{t+1}$, with base case $G_T = 0$. Base case: at the terminal step $T$, there are no more rewards to accumulate, so $G_T = 0$ by definition — this matches the convention used to unify episodic and continuing notation. Inductive step: suppose $G_{t+1}$ has been correctly computed as $\sum_{k=0}^{T-t-2} \gamma^k R_{t+2+k}$. Then

which is exactly the closed-form definition of $G_t$. By induction, the recursion computes the correct return at every $t$, and computing it backward from $t = T-1$ down to $t=0$ correctly uses only already-finalized values.

Why discounting guarantees finiteness (the convergence argument). For a continuing task with $|R_t| \le R_{\max}$ for all $t$ (a mild boundedness assumption satisfied by essentially every practical reward function), the discounted return's absolute value is bounded by a geometric series:

The equality $\sum_{k=0}^\infty \gamma^k = \frac{1}{1-\gamma}$ holds whenever $|\gamma| < 1$, by the standard geometric series formula; the partial sums $\sum_{k=0}^{n}\gamma^k = \frac{1-\gamma^{n+1}}{1-\gamma}$ converge to $\frac{1}{1-\gamma}$ as $n \to \infty$ because $\gamma^{n+1} \to 0$. This is precisely why the constraint $\gamma < 1$ (not $\gamma \le 1$) is required for continuing tasks: it is the exact condition under which the infinite return is guaranteed to be a finite, well-defined real number regardless of how the reward sequence unfolds.

Tradeoffs in choosing $\gamma$. There is no universally "correct" discount factor — it encodes a modeling choice, not a derived quantity:

• Small $\gamma$ (e.g., $0.9$) makes the return depend mostly on near-term rewards. Estimation of returns from sampled data has lower variance (fewer terms contribute meaningfully), and value functions are easier to learn, but the resulting policy may be myopic and ignore consequences far in the future.
• Large $\gamma$ (e.g., $0.999$) makes the agent plan over long horizons — important for tasks with long-delayed consequences — but the effective horizon $\frac{1}{1-\gamma}$ grows, increasing the variance of return estimates and often slowing convergence of learning algorithms.
• $\gamma = 1$ is sometimes used for episodic tasks with a guaranteed, finite horizon, where the convergence issue never arises; using $\gamma = 1$ on a continuing task is unsafe unless one switches to a different objective (such as the average reward per step) precisely because the sum can diverge.

Consider a small episodic task: a robot navigating a $1 \times 5$ corridor toward a goal, receiving $-1$ reward per step until it reaches the goal, at which point the episode ends. Suppose one episode produces the reward sequence

so $T = 5$. Let's compute the return at every time step with two different discount factors, $\gamma = 1$ (undiscounted) and $\gamma = 0.9$, using the backward recursion $G_t = R_{t+1} + \gamma G_{t+1}$, $G_T = 0$.

Undiscounted ($\gamma = 1$):

Sanity check by direct summation: $G_0 = R_1+R_2+R_3+R_4+R_5 = -1-1-1-1+10 = 6$. Matches.

Discounted ($\gamma = 0.9$):

Direct-sum check: $G_0 = -1 + 0.9(-1) + 0.81(-1) + 0.729(-1) + 0.6561(10) = -1 - 0.9 - 0.81 - 0.729 + 6.561 = 3.122$. Matches, confirming both the recursion and the closed-form expansion agree. Notice the qualitative lesson: with $\gamma=1$, the $+10$ goal reward is worth exactly $10$ no matter how far away it is; with $\gamma = 0.9$, the same $+10$ reward, four steps away from $t=0$, is discounted down to $0.9^4 \times 10 = 6.561$ of its face value by the time it is "seen" from $t=0$. This is exactly why an agent with smaller $\gamma$ will prefer a shorter path to the goal even if a longer path eventually yields the same terminal reward — the discounting itself creates an implicit preference for speed, on top of whatever the raw reward function specifies. Now contrast with a continuing task: a server that gets reward $+1$ every single step forever if it stays online. With $\gamma = 1$, $G_t = 1+1+1+\cdots = \infty$ for every $t$ and every policy that keeps the server alive — utterly useless for comparing an occasionally-crashing policy against a perfectly reliable one, since both could plausibly diverge. With $\gamma = 0.99$, a policy that is online every step earns $G_t = \sum_{k=0}^\infty 0.99^k = \frac{1}{1-0.99} = 100$, while a policy that crashes and receives $0$ reward on 10% of steps earns strictly less than $100$ — a finite, comparable number that correctly ranks policies by long-run reliability.

The formalization of reward and return as the objective of adaptive behavior draws on ideas from optimal control, animal learning theory, and stochastic approximation, synthesized into the modern RL formalism largely through the work of Richard Sutton and Andrew Barto, who crystallized the reward hypothesis and the discounted-return formalism into the standard textbook framework used throughout the field.

The discounted return is not an abstract nicety — it is the literal training objective baked into essentially every reinforcement learning system deployed in practice. Deep Q-Networks and their successors, used in game-playing agents from Atari to StarCraft II, learn to estimate $G_t$ (via value functions built on the recursion above) directly from pixels. Policy-gradient methods such as REINFORCE, PPO, and actor-critic algorithms explicitly weight the gradient of the policy by sampled or estimated returns. In robotics, the choice of $\gamma$ directly determines whether a learned controller optimizes for quick, jerky task completion or smoother, longer-horizon stability. In recommendation systems and RLHF-tuned large language models, the "reward model" that scores an output, combined with a discount factor over a multi-turn conversation, defines the entire optimization target that shapes deployed model behavior. Financial trading agents, HVAC control systems, and data-center cooling optimizers are all, at bottom, engines for maximizing a discounted return under an environment with genuinely never-ending dynamics — the continuing-task setting this article makes precise. Every one of these systems inherits its correctness (and its failure modes, like reward hacking) directly from how carefully the reward and return were specified, which is why this foundational definition anchors the rest of the field, including Markov Decision Processes and Policies and Value Functions.

Ask the tutor: How does the discounted return connect to the Bellman equation and value functions in a full Markov Decision Process?