Chain-of-Thought Reasoning

Chain-of-Thought (CoT) reasoning is a prompting technique that encourages large language models to produce explicit intermediate reasoning steps before arriving at a final answer, mimicking the way humans "think out loud" to solve complex problems. It matters enormously in modern AI because it dramatically improves LLM performance on multi-step tasks — arithmetic, logic, planning, and agentic decision-making — without any additional model training. The counterintuitive fact: simply adding the phrase "Let's think step by step" to a prompt can more than double a model's accuracy on math word problems.

The core insight

A language model predicts tokens one at a time, and each token it generates becomes part of the context for the next prediction. When you force a model to write out reasoning steps, each intermediate step conditions the next step, effectively letting the model "use the page as scratch paper." The "aha moment" is realizing that CoT is not a training trick or a new architecture — it is a prompting strategy that exploits the autoregressive nature of transformers to unlock reasoning that the model already has latent capability for, but cannot access in a single forward pass.

How It Works

At its core, Chain-of-Thought works by conditioning the model's next-token prediction on an explicitly generated reasoning trace. Let's build up the mechanism from first principles. A standard language model computes the probability of an answer $A$ given a question $Q$ :

\quad P(A \mid Q)

Explanation: The model maps a question directly to an answer in one shot, with no intermediate steps visible. With Chain-of-Thought, we instead decompose this into a sequence of reasoning steps $R_1, R_2, \ldots, R_k$ before producing the answer:

\quad P(A \mid Q) = \sum_{R} P(A \mid Q, R) \cdot P(R \mid Q)

Explanation: The answer is now computed by marginalizing over all possible reasoning traces $R$ ; in practice we sample one trace greedily or with low temperature. Because the model generates $R_1$ first, then $R_2$ conditioned on $R_1$ , and so on, each step narrows the token distribution toward logically consistent continuations:

\quad P(R_1, R_2, \ldots, R_k \mid Q) = \prod_{i=1}^{k} P(R_i \mid Q, R_1, \ldots, R_{i-1})

Explanation: This is just the chain rule of probability applied to the reasoning sequence — each step depends on everything that came before it, making earlier steps act as a working memory. Here is the most minimal Python illustration of the difference between standard prompting and CoT prompting using the OpenAI API: ```python from openai import OpenAI client = OpenAI() # uses OPENAI_API_KEY from environment question = "A farmer has 17 sheep. All but 9 run away. How many sheep remain?"

# ── Standard prompt ────────────────────────────────────────────────────────── standard_response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": question}], temperature=0, ) print("Standard:", standard_response.choices[0].message.content)

# → Often: "8" (wrong — subtracts instead of reading "all but 9")

# ── Chain-of-Thought prompt ─────────────────────────────────────────────────── cot_response = client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": question + "\n\nLet's think step by step." }], temperature=0, ) print("CoT:", cot_response.choices[0].message.content)

# → "The phrase 'all but 9' means 9 remain. Answer: 9" (correct) ``` There are three main flavours of CoT prompting in practice:

Variant	How it works	Best for
Zero-shot CoT	Append "Let's think step by step."	Quick wins, no examples needed
Few-shot CoT	Provide 3–8 worked examples in the prompt	Consistent format, domain tasks
Auto-CoT	Model generates its own examples	Large-scale pipelines

Here is a few-shot CoT prompt builder in Python, which is the approach most commonly used in agentic systems: ```python def build_cot_prompt(examples: list[dict], question: str) -> str: """ examples: list of {"question": str, "reasoning": str, "answer": str} """ prompt_parts = [] for ex in examples: prompt_parts.append( f"Q: {ex['question']}\n" f"Reasoning: {ex['reasoning']}\n" f"A: {ex['answer']}" ) prompt_parts.append(f"Q: {question}\nReasoning:") return "\n\n".join(prompt_parts)

examples = [ { "question": "Roger has 5 tennis balls. He buys 2 more cans of 3 balls each. How many tennis balls does he have?", "reasoning": "Roger starts with 5 balls. 2 cans × 3 balls = 6 new balls. 5 + 6 = 11.", "answer": "11", }, { "question": "A store had 23 apples. They sold 15 and received a shipment of 7. How many now?", "reasoning": "Start: 23. Sold: 23 - 15 = 8. Received: 8 + 7 = 15.", "answer": "15", }, ] prompt = build_cot_prompt( examples, question="There are 48 students in a class. Half go on a field trip. Then 6 return early. How many are still on the trip?" ) print(prompt) ```

researcher · Jason Wei | b. 1997 | Pioneered Chain-of-Thought Prompting at Google Brain

Jason Wei is a researcher at Google Brain (later OpenAI) who co-authored the landmark 2022 paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," which introduced the term and the systematic study of CoT. His paper demonstrated that CoT only emerges as a useful technique above a certain model scale threshold (roughly 100B parameters for the models of 2022), a finding that catalyzed enormous subsequent research into why scale unlocks reasoning. Surprisingly, Wei originally joined Google Brain as a part-time contractor while completing his undergraduate degree.

researcher · Takeshi Kojima | b. ~1995 | Discovered Zero-Shot Chain-of-Thought

Takeshi Kojima, a researcher at the University of Tokyo working with collaborators at Google, discovered in 2022 that simply appending "Let's think step by step" to a prompt — with no worked examples at all — yields dramatic accuracy improvements across benchmarks. This zero-shot variant was arguably more surprising than few-shot CoT because it required no human-crafted reasoning examples, suggesting the model had internalized reasoning templates from pretraining. His finding shifted the field's understanding of what pretraining on internet text implicitly teaches language models.

Figure

Chain-of-Thought vs Standard Prompting — Data Flow

Figure

CoT in an Agentic Loop

Intuition

The Mental Model

Imagine you are asked to calculate a restaurant tip in your head with a single blink — hard, error-prone. Now imagine you are given a napkin to write on. You jot: "bill is $47, 20% means I move the decimal:$ 4.70, double it: $9.40." The napkin did not make you smarter; it gave your existing arithmetic skill a workspace. Chain-of-Thought is that napkin for a language model. The model always had the knowledge of arithmetic, logic, or planning inside its weights, but a single forward pass (answering in one go) is like blinking — there is nowhere for intermediate thoughts to go. By generating text step by step, each sentence the model writes becomes a visible "thought on the napkin" that the model can build on for the next sentence. The reasoning quality improves not because the model learned anything new, but because the scratchpad lets partial results persist and compound.

Historical Development

Origins · 1 minhow this was discovered

History

2017

The Transformer architecture (Vaswani et al., "Attention Is All You Need") establishes the autoregressive, token-by-token generation paradigm that makes CoT possible — each generated token becomes future context.

2020

GPT-3 (Brown et al., OpenAI) demonstrates that large language models can perform few-shot in-context learning, hinting that the format of a prompt dramatically affects model behavior — a precursor insight to CoT.

2021

Nye et al. ("Show Your Work: Scratchpads for Intermediate Computation with Language Models") formally propose that LLMs be trained to produce intermediate computation steps, showing accuracy gains on symbolic tasks — the direct conceptual ancestor of CoT prompting.

2022

(January): Wei et al. (Google Brain) publish "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," coining the term and demonstrating state-of-the-art results on GSM8K math and commonsense benchmarks through few-shot CoT examples. They also discover the critical scale threshold — CoT helps almost exclusively in models above ~100B parameters.

2022

(May): Kojima et al. (U. Tokyo / Google) publish "Large Language Models are Zero-Shot Reasoners," showing that "Let's think step by step" alone — without any examples — achieves strong CoT performance, democratizing the technique.

2022

(October): Wang et al. introduce Self-Consistency, sampling multiple CoT reasoning paths and taking a majority vote over final answers, pushing GSM8K accuracy well above single-path CoT.

2023

OpenAI releases GPT-4, which exhibits CoT-like reasoning internally and is widely used with structured CoT prompts in production agentic systems (LangChain, AutoGPT, etc.), bringing the technique into mainstream software engineering.

2023

The ReAct framework (Yao et al.) interleaves CoT reasoning traces with tool-use actions (search, calculator, code execution), establishing the standard loop used in modern AI agents.

2024

OpenAI releases the o1 model series, which applies reinforcement learning to train models to produce long internal reasoning traces before answering — representing CoT "baked into" the model weights rather than elicited by prompting.

Why It Matters

Real-World Impact

OpenAI o1 / o3These models apply reinforcement learning over CoT traces, training the model to spend more "thinking tokens" on harder problems. CoT is no longer just a prompting trick here — it is the core mechanism by which the model allocates compute to reasoning, achieving human-expert-level scores on competition math (AIME) and coding (Codeforces).

LangChain AgentsLangChain's `AgentExecutor` uses a CoT-structured prompt (the ReAct format) to have an LLM alternate between writing a "Thought," choosing an "Action" (tool call), and observing the "Observation" (tool result). Every production LangChain agent loop is a direct implementation of Chain-of-Thought interleaved with tool use.

GitHub Copilot / CursorCode completion tools use CoT-style prompting internally when handling multi-step tasks like "refactor this function and update all its callers" — the model reasons through the dependency graph step by step before emitting edits.

Google's Gemini for Search (AI Overviews)Gemini uses CoT reasoning to decompose complex search queries into sub-questions, retrieve evidence for each sub-question, and synthesize a final answer — a RAG pipeline where CoT structures the retrieval planning.

Medical AI (Med-PaLM 2)Google's medical LLM uses few-shot CoT prompts with clinician-authored reasoning examples to achieve expert-level performance on US Medical Licensing Exam questions, where step-by-step diagnostic reasoning is critical to answer correctness and clinical safety.

AutoGPT / BabyAGI style agentsEarly autonomous agent frameworks (2023) use CoT prompting to have a model decompose a high-level goal into a task list, execute tasks, observe results, and re-plan — the entire planning loop is a structured CoT trace persisted in memory.

LLM-based SQL generation (Text-to-SQL)Production systems at companies like Snowflake and Databricks use CoT prompts that ask the model to first identify relevant tables, then reason about joins, then write the SQL — dramatically reducing hallucinated column names compared to direct generation.

Chain-of-Thought Reasoning

How It Works

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Before this