Agentic Loops and Scaffolding

The Agentic Loop as a Markov Decision Process

An agentic loop can be formalized as a partially observable Markov decision process (POMDP) where the agent policy $\pi_\theta$ is parameterized by a language model. At each discrete timestep $t$, the agent maintains a belief state summarizing all information so far.

Explanation: The agent's effective state at step $t$ is the concatenation of the system instruction $I$, all prior memory chunks $m$, prior observations $o$, and prior actions $a$—this is the full prompt fed to the model.

Symbols:

• $s_t$ — the prompt context (belief state) at step $t$
• $I$ — the system prompt / task instruction
• $m_{1:t-1}$ — retrieved or summarized memory up to step $t-1$
• $o_{1:t-1}$ — tool-call observations (environment responses) up to step $t-1$
• $a_{1:t-1}$ — the agent's own prior actions (tool calls + reasoning traces)

The policy samples the next action from the model's conditional distribution:

Explanation: The model draws the next action token-by-token from a softmax distribution over the vocabulary, tempered by $\tau$, conditioned on the full accumulated context.

Symbols:

• $\pi_\theta$ — the language model policy with weights $\theta$
• $f_\theta(s_t, a)$ — the unnormalized log-probability (logit) the model assigns to action $a$ given context $s_t$
• $\tau$ — temperature controlling exploration vs. exploitation

After the scaffolding executes $a_t$ (e.g., calls a Python interpreter or a search API), the environment returns an observation:

Explanation: The environment function $\mathcal{E}$ deterministically or stochastically maps the agent's action to a structured observation string—the raw output of the tool.

Symbols:

• $\mathcal{E}$ — environment / tool executor (Python REPL, web API, database, file system, etc.)
• $o_t$ — the observation string returned and appended to context

The loop terminates when the policy emits a designated FINISH token or action, or when a scaffolding-enforced bound is reached:

Explanation: The actual episode length is whichever comes first—the model deciding it is done, or the scaffolding's hard maximum step limit $T_{\max}$, preventing infinite loops.

Symbols:

• $T$ — actual episode termination step
• $T_{\max}$ — scaffolding-enforced hard cap on loop iterations
• $\texttt{FINISH}$ — the special action token or function call signaling task completion

Context Budget and Memory Compression

Because $|s_t|$ grows linearly with $t$, practical scaffolds apply a compression operator to keep the prompt within the model's context window $C$:

Explanation: The scaffolding's context manager $\mathcal{K}$ trims or summarizes $s_t$ to fit within the model's window $C$—the design of $\mathcal{K}$ (summarize oldest, evict least-relevant, or RAG-retrieve) is one of the most consequential scaffolding decisions.

Symbols:

• $s_t'$ — the compressed/managed context actually sent to the model
• $\mathcal{K}$ — the context management operator (summarizer, sliding window, RAG retriever)
• $C$ — the model's maximum context length in tokens

---

PyTorch / Implementation Sketch

```python import json from typing import Callable def agentic_loop( model_call: Callable[[str], str], # π_θ: prompt → raw action string tools: dict[str, Callable], # environment ε system_prompt: str, task: str, max_steps: int = 20, # T_max context_manager: Callable = None, # K compression operator ) -> str: history = [] # stores (action, observation) pairs context = system_prompt + "\n\nTask: " + task for step in range(max_steps):

# Apply context compression if provided (eq [5]) prompt = context_manager(context, history) if context_manager else ( context + "\n" + format_history(history) )

# Sample action from policy (eq [2]) raw_action = model_call(prompt)

# Parse structured action (tool name + arguments) action = parse_action(raw_action) # e.g. {"tool": "python", "code": "..."} if action["tool"] == "FINISH": # termination check (eq [4]) return action.get("answer", raw_action)

# Execute tool — environment step (eq [3]) tool_fn = tools.get(action["tool"]) if tool_fn is None: observation = f"ERROR: unknown tool '{action['tool']}'" else: try: observation = str(tool_fn(**action.get("args", {}))) except Exception as e: observation = f"ERROR: {e}" # defensive scaffolding history.append({"action": raw_action, "observation": observation}) return "MAX_STEPS_REACHED" # hard cap — never infinite loop

def format_history(history): return "\n".join( f"Action: {h['action']}\nObservation: {h['observation']}" for h in history ) ``` This sketch deliberately keeps `context_manager` pluggable—swapping in a RAG retriever versus a sliding-window summarizer changes the agent's effective memory without touching the loop logic. ---

ReAct Framework · Chain-of-Thought Prompting · Retrieval-Augmented Generation · Tool Use and Function Calling · Multi-Agent Systems · AutoGPT and Open-Ended Agents · LangChain and LangGraph