Agents

An agent is an entity that perceives its environment through sensors and acts upon it through actuators, choosing actions to maximize a performance measure. Formally modeled as a function from percept histories to actions, agents unify decision theory, control, and AI. They apply wherever autonomous, goal-directed behavior is modeled — from robotics and economics to reinforcement learning. The modern framework was crystallized by Russell & Norvig (1995), building on von Neumann, Wiener, and Bellman.

An agent is anything that can be viewed as perceiving its environment through sensors and acting on it through actuators, where the choice of action is governed by an agent function mapping the entire percept history to an action. The notion matters because it provides a single mathematical scaffold for reasoning about rational (utility-maximizing) behavior under uncertainty, time, and partial information, subsuming Markov decision processes, game theory, and classical control. The surprising fact is that under mild conditions, any rational agent's preferences can be represented as the maximization of a single scalar expected utility (the von Neumann–Morgenstern theorem), so an immensely complex behavioral repertoire collapses to "argmax of a number." This makes the agent abstraction both deceptively simple and remarkably powerful.

The Core Insight

Intelligence and goal-directed behavior can be factored cleanly into three pieces: a percept stream coming in, an action stream going out, and an agent function that connects them. Rationality is then not a property of internal mechanism but of input-output behavior: an agent is rational if its agent function selects, for each percept sequence, an action expected to maximize a performance measure given what the percepts reveal. Everything else — planning, learning, perception — is engineering machinery for implementing or approximating this ideal mapping.

Formal Statement

Agent, Percept, Action

Let $P$ be the set of possible percepts (sensor readings) and $A$ the set of possible actions (actuator commands). A percept sequence is a finite string $p_{1 : t} = (p_{1}, p_{2}, \dots, p_{t}) \in P^{*}$ . An agent is defined by its agent function

f : P^{*} \to A, a_{t} = f (p_{1 : t}) .

⟳This is a definition, not a derived result — it formalizes "the action depends on everything observed so far." Note the domain is

P^{*}

, the Kleene star (all finite sequences), capturing full history dependence.

The agent program is the concrete algorithm running on an architecture (physical/computational substrate) that implements (or approximates) $f$ . We write agent = architecture + program.

Task Environment — PEAS

A task environment is specified by the tuple

⟨ P, E, A, S ⟩ = (Performance measure, Environment, Actuators, Sensors) .

⟳"PEAS" is a checklist, not an equation; it is numbered here only to satisfy formatting. The performance measure

P

is a function

P : (environment histories) \to R

used to grade behavior — crucially it is defined over states of the environment, not over the agent's internal beliefs.

Rational Agent

Given a performance measure $P$ , a percept sequence $p_{1 : t}$ , and the agent's prior knowledge $K$ , a rational agent selects

a_{t}^{⋆} = ar g a \in A max E [P ∣ p_{1 : t}, K, a],

i.e. the action whose expected performance, conditioned on evidence and knowledge, is greatest.

⟳The expectation is over the agent's posterior beliefs about hidden environment states, formed by conditioning a prior on the percept sequence (Bayesian updating). Rationality is bounded by what is knowable from

p_{1 : t}

and

K

— not omniscience.

For sequential decision problems with states $s \in S$ , the agent's behavior is captured by a policy $π : S \to A$ (or $π (a ∣ s)$ for stochastic policies), and performance is the expected cumulative reward:

Optimality via the Bellman Equation

For a Markov Decision Process $(S, A, T, R, γ)$ with transition kernel $T (s^{'} ∣ s, a)$ , reward $R (s, a)$ , and discount $γ \in [0, 1)$ , there exists an optimal policy $π^{⋆}$ whose value function satisfies the Bellman optimality equation

V^{⋆} (s) = a \in A max [R (s, a) + γ s^{'} \in S \sum T (s^{'} ∣ s, a) V^{⋆} (s^{'})],

and the optimal agent acts greedily: $π^{⋆} (s) = ar g max_{a} [R (s, a) + γ \sum_{s^{'}} T (s^{'} ∣ s, a) V^{⋆} (s^{'})]$ .

⟳Apply the principle of optimality — an optimal policy's tail is optimal for the subproblem. The right-hand side is a contraction mapping in

sup

-norm with modulus

γ

; Banach's fixed-point theorem gives existence and uniqueness of

V^{⋆}

von Neumann–Morgenstern Utility Representation

If an agent's preference relation $⪰$ over lotteries (probability distributions over outcomes) satisfies completeness, transitivity, continuity, and independence, then there exists a utility function $u$ such that for lotteries $L_{1}, L_{2}$ ,

L_{1} ⪰ L_{2} ⟺ E_{L_{1}} [u] \geq E_{L_{2}} [u],

and $u$ is unique up to a positive affine transformation $u \mapsto α u + β$ , $α > 0$ .

⟳Continuity gives, for each outcome between best

B

and worst

W

, a unique indifference probability

p

with outcome

\sim [p : B, (1 - p) : W]

; set

u (outcome) = p

. Independence ensures these indifference probabilities combine linearly across compound lotteries, yielding the expected-utility form.

Why It Holds / Derivation

Bellman Optimality as a Fixed Point

Step 1. Define the Bellman optimality operator

T

on bounded functions

V : S \to R

(T V) (s) = a max [R (s, a) + γ s^{'} \sum T (s^{'} ∣ s, a) V (s^{'})] .

⟳This packages the one-step lookahead in equation [4] into an operator so we can study its fixed points abstractly.

Step 2. Show

T

is a

γ

-contraction in the sup-norm

∥ V ∥_{\infty} = max_{s} ∣ V (s) ∣

. For any

V, U

∥ T V - T U ∥_{\infty} \leq γ ∥ V - U ∥_{\infty} .

⟳Use

∣ max_{a} g (a) - max_{a} h (a) ∣ \leq max_{a} ∣ g (a) - h (a) ∣

, then bound the difference of the two expectations by

γ ∥ V - U ∥_{\infty}

since transition probabilities sum to 1.

Step 3. Since the space of bounded functions with sup-norm is complete, Banach's fixed-point theorem gives a unique fixed point

V^{⋆} = T V^{⋆}

, satisfying equation [4]. Iterating

V_{k + 1} = T V_{k}

converges geometrically:

∥ V_{k} - V^{⋆} ∥_{\infty} \leq γ^{k} ∥ V_{0} - V^{⋆} ∥_{\infty}

— this is value iteration. Step 4. The greedy policy w.r.t.

V^{⋆}

achieves

V^{⋆}

, so it is optimal. Hence a rational sequential agent's behavior is well-defined and computable (in the finite case).

■

John von Neumann | 1903–1957

A founder of the agent's mathematical core. With Oskar Morgenstern he wrote Theory of Games and Economic Behavior (1944), proving the expected-utility theorem (equation [5]) that justifies treating rational behavior as scalar maximization, and establishing the minimax theorem for two-player zero-sum games — the prototype of an adversarial agent. His work on self-reproducing automata and the stored-program architecture (the "von Neumann architecture") underpins the architecture half of "agent = architecture + program."

Building Intuition

Think of an agent as a thermostat that learned philosophy. The dumbest agent — a thermostat — maps the single percept "too cold/too hot" to "heater on/off"; its agent function is a tiny lookup table. Scale this up: a chess program's agent function maps a board (percept) to a move (action), but the table is astronomically large, so it's computed by search rather than stored. A self-driving car's percepts are camera frames and lidar; its actions are steering and throttle. In every case the behavior is the function from what-it-has-seen to what-it-does. Rationality just asks: of all functions you could implement, are you choosing the action that does best on average given what you actually know? Note the two humilities baked in: "on average" (the world is stochastic) and "given what you know" (you can't act on hidden facts). A rational agent is not omniscient or infallible — it is the best bet against its own ignorance.

Anatomy of an agent interacting with its environment

<svg viewBox="0 0 640 360" xmlns="http://www.w3.org/2000/svg">
  <rect width="640" height="360" fill="#0a0e17"/>
  <rect x="40" y="70" width="230" height="220" rx="10" fill="none" stroke="#94a3b8" stroke-width="2"/>
  <text x="155" y="58" fill="#e2e8f0" font-size="16" text-anchor="middle">AGENT</text>
  <rect x="70" y="110" width="170" height="40" rx="6" fill="none" stroke="#94a3b8"/>
  <text x="155" y="135" fill="#e2e8f0" font-size="13" text-anchor="middle">Sensors</text>
  <rect x="70" y="170" width="170" height="50" rx="6" fill="none" stroke="#d4a853" stroke-width="2"/>
  <text x="155" y="195" fill="#d4a853" font-size="13" text-anchor="middle">Agent function</text>
  <text x="155" y="212" fill="#d4a853" font-size="11" text-anchor="middle">f : P* → A</text>
  <rect x="70" y="240" width="170" height="40" rx="6" fill="none" stroke="#94a3b8"/>
  <text x="155" y="265" fill="#e2e8f0" font-size="13" text-anchor="middle">Actuators</text>
  <rect x="400" y="70" width="200" height="220" rx="10" fill="none" stroke="#94a3b8" stroke-width="2"/>
  <text x="500" y="58" fill="#e2e8f0" font-size="16" text-anchor="middle">ENVIRONMENT</text>
  <text x="500" y="185" fill="#e2e8f0" font-size="13" text-anchor="middle">states s ∈ S</text>
  <text x="500" y="205" fill="#e2e8f0" font-size="12" text-anchor="middle">reward R(s,a)</text>
  <path d="M400 130 L270 130" stroke="#94a3b8" stroke-width="2" marker-end="url(#a)"/>
  <text x="335" y="120" fill="#e2e8f0" font-size="12" text-anchor="middle">percepts</text>
  <path d="M270 260 L400 260" stroke="#94a3b8" stroke-width="2" marker-end="url(#a)"/>
  <text x="335" y="280" fill="#e2e8f0" font-size="12" text-anchor="middle">actions</text>
  <defs>
    <marker id="a" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto">
      <path d="M0,0 L8,3 L0,6 Z" fill="#94a3b8"/>
    </marker>
  </defs>
  <text x="320" y="330" fill="#94a3b8" font-size="11" text-anchor="middle">closed perception–action loop</text>
</svg>

Worked Examples

Vacuum-Cleaner World — A Tabulated Agent Function

Given: Two squares

A, B

. Percept = (location, status) where status ∈ {Clean, Dirty}. Actions: Left, Right, Suck. Performance: +1 per clean square per time step, over 100 steps. Steps. Build the agent function as a table for the simplest reflex agent:

| Percept | Action | |---|---| | [A, Dirty] | Suck | | [A, Clean] | Right | | [B, Dirty] | Suck | | [B, Clean] | Left |

Compute performance if both start dirty. Step 1: at A=Dirty → Suck (A clean). Step 2: A=Clean → Right. Step 3: at B=Dirty → Suck (B clean). From step 3 onward both squares clean: score $= 2$ per step. Over steps 4–100 (97 steps) plus partial earlier credit: clean-square-steps $\approx 2 \times 97 + (early partial)$ .

Approximate total clean-square count $\approx 2 \times 98 = 196$ (after both clean by step 3).

Boxed answer:

Score \approx 196 clean-square-steps; agent function = 4-row table

Optimal Action via Bellman — A 2-State MDP

Given: States

{s_{0}, s_{1}}

, actions

{

stay, go

}

γ = 0.9

. Rewards:

R (s_{0}, go) = 0

R (s_{1}, stay) = 10

R (s_{1}, go) = 10

R (s_{0}, stay) = 0

. Transitions: go from

s_{0}

→

s_{1}

w.p. 1; everything else stays put. Steps. Guess the agent should reach

s_{1}

and stay. Compute

V^{⋆} (s_{1})

(stay loops forever in

s_{1}

with reward 10):

V^{⋆} (s_{1}) = 10 + 0.9 V^{⋆} (s_{1}) \Rightarrow V^{⋆} (s_{1}) = \frac{10}{1 - 0.9} = 100.

Now

V^{⋆} (s_{0})

: action "go" yields

0 + 0.9 V^{⋆} (s_{1}) = 0.9 \times 100 = 90

; action "stay" yields

0 + 0.9 V^{⋆} (s_{0})

. Since 90 > stay value, optimal is go:

V^{⋆} (s_{0}) = 0 + 0.9 \times 100 = 90.

Boxed answer:

V^{⋆} (s_{0}) = 90, V^{⋆} (s_{1}) = 100, π^{⋆} (s_{0}) = go, π^{⋆} (s_{1}) = stay

Common Misconceptions

"A rational agent always does the right thing / never fails." False. Rationality (equation [3]) maximizes expected performance given available evidence. An agent that crosses a road after looking both ways is rational even if a meteor then strikes it. Rationality ≠ omniscience, ≠ clairvoyance, and ≠ perfection. Confusing the performance measure (defined over real environment states) with the agent's internal utility estimate (its possibly-wrong beliefs) is the single most common error. "Bigger/more autonomous = more 'agent.'" Agency is binary in the formal definition — anything with the perceive→act loop is an agent, including a thermostat. What varies is sophistication (reflex vs. model-based vs. goal-based vs. utility-based vs. learning agents), not agenthood.

Apparent Irrationality That Is Actually Rational

A robot vacuum that re-cleans an already-clean square seems irrational. But if its sensor is noisy (status reading is unreliable), then re-checking can raise expected performance by reducing the chance of leaving dirt. Under partial observability the "wasteful" action is optimal. Conversely, the Allais paradox is a genuine counterexample to humans being vNM-rational: people systematically violate the independence axiom, so no single utility function (equation [5]) can represent their choices — proving humans are not the rational agents of the theory.

Connections

The agent framework is a hub linking several fields. The sequential-decision core is exactly a Markov Decision Process, whose solution via equation [4] is Dynamic Programming (Bellman). When the agent must learn $T$ and $R$ from experience instead of knowing them, you get Reinforcement Learning (Q-learning, policy gradients approximate $π^{⋆}$ ). The preference foundations (equation [5]) belong to Decision Theory and Expected Utility Theory. Multi-agent interaction — where each agent's environment contains other agents — is Game Theory, with von Neumann's minimax and Nash equilibrium as solution concepts. The continuous-state, control-oriented version is Optimal Control and the Hamilton-Jacobi-Bellman Equation. Belief-tracking under partial observability links to Bayesian Inference and POMDPs.

Real-World Applications

Autonomous vehicles (Waymo, Tesla Autopilot): perception (sensors) + planning (agent function) + actuation, with performance measures balancing safety, comfort, and progress.
Game-playing systems (DeepMind's AlphaGo, AlphaZero): utility-based agents using Monte Carlo Tree Search + learned value/policy networks approximating equation [4].
Algorithmic trading agents: maximize expected discounted return under stochastic markets, a direct instance of equation [3] with severe partial observability.
Recommendation / advertising systems (contextual bandits at Google, Netflix): agents choosing actions (items to show) to maximize expected reward (clicks, watch time).
Robotic warehouse fleets (Amazon Kiva/Robotics): multi-agent coordination minimizing travel time, a distributed MDP.
Conversational AI and LLM "agents": tool-using systems that plan multi-step actions (search, code execution) toward a goal — the percept stream is text, actions are API calls.
Datacenter cooling and power management (DeepMind/Google): RL agents reduced cooling energy by adjusting actuators based on sensor telemetry.

Historical Development

1936 — Alan Turing's universal machine formalizes computation, providing the substrate for any agent program.
1944 — von Neumann & Morgenstern publish Theory of Games and Economic Behavior, proving the expected-utility theorem (equation [5]).
1948 — Norbert Wiener's Cybernetics frames purposeful behavior as feedback control — the perception–action loop made explicit.
1950 — Turing's "Computing Machinery and Intelligence" poses machine intelligence in behavioral terms, foreshadowing behavior-based agent definitions.
1957 — Richard Bellman's Dynamic Programming introduces the optimality principle and equation [4]; coins "curse of dimensionality."
1959 — Arthur Samuel's checkers program demonstrates a learning agent improving via self-play.
1989 — Christopher Watkins introduces Q-learning, a model-free way to converge toward equation [4] from experience.
1995 — Russell & Norvig's Artificial Intelligence: A Modern Approach canonizes the "rational agent" as AI's unifying organizing concept and the PEAS framework (equation [2]).
2015–2017 — Deep RL agents (DQN, AlphaGo) achieve superhuman performance, empirically validating the agent abstraction at scale.

Subtleties & Limits

The clean theory frays at the edges. (1) Bounded rationality: equation [3] requires an intractable expectation and argmax; real agents have finite compute, so what is the right notion of optimal-given-resources? Russell's "rational metareasoning" and Simon's satisficing are partial answers, but no fully satisfactory theory of computationally bounded rationality exists. (2) Reward specification / alignment: the performance measure $P$ must be defined over environment states, yet specifying $P$ to capture genuine human intent is unsolved — misspecified rewards cause reward hacking and pose AI safety risks. (3) Non-stationarity and multi-agent recursion: when the environment contains other learning agents, the "stationary MDP" assumption behind equation [4] breaks, and equilibrium existence/learnability becomes subtle. (4) Continuous, high-dimensional, partially observed worlds push value iteration into approximation territory with weak convergence guarantees. (5) The hard question of agency itself: whether sophisticated function-approximators "really" have goals, or merely behave as if they do, remains philosophically contested.

Agents

The Core Insight

Formal Statement

Agent, Percept, Action

Task Environment — PEAS

Rational Agent

Optimality via the Bellman Equation

von Neumann–Morgenstern Utility Representation

Why It Holds / Derivation

Bellman Optimality as a Fixed Point

John von Neumann | 1903–1957

Building Intuition

Anatomy of an agent interacting with its environment

Worked Examples

Vacuum-Cleaner World — A Tabulated Agent Function

Optimal Action via Bellman — A 2-State MDP

Common Misconceptions

Apparent Irrationality That Is Actually Rational

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also

Before this

Where this leads