Chain-of-Thought Reasoning

A standard prompt-and-answer interaction asks a model to map an input x directly to an output y. For multi-step problems — arithmetic word problems, multi-hop factual questions, logical deduction — this direct mapping forces the model to perform an entire chain of latent computation within a single forward pass, compressed into whatever internal representation the final answer token can carry. Chain-of-thought reasoning instead asks the model to generate an intermediate sequence of natural-language reasoning steps z₁, z₂, …, zₖ between x and y, so the joint generation is x → z₁ → z₂ → … → zₖ → y. Each intermediate step is conditioned on all the previous ones, because generation is autoregressive: the model's own prior output becomes part of its input. The technique was popularized as a prompting strategy exploiting in-context learning: instead of fine-tuning the model, one includes a handful of demonstrations in the prompt showing a question, a worked chain of reasoning, and an answer. The model, conditioned on these demonstrations, imitates the pattern — question, reasoning, answer — on a new question. This is called few-shot CoT. A simpler variant, zero-shot CoT, needs no worked examples at all: appending a fixed trigger phrase such as "Let's think step by step" after the question is often enough to induce a reasoning trace, because the model has seen enormous amounts of step-by-step explanatory text during pretraining and the phrase cues that distributional mode. A third regime, increasingly dominant in modern systems, is trained-in chain-of-thought: models are fine-tuned or reinforcement-learned specifically to produce long, high-quality reasoning traces before answering, sometimes with the trace itself optimized against a verifier or reward signal rather than merely imitated from prompt demonstrations. In all three regimes the underlying mechanism is identical — the reasoning tokens are genuinely part of the sequence the model conditions on, so they change what the final answer token can "see."

The mechanics of eliciting chain-of-thought are simple to describe, but the effect they have on the model's underlying computation is the important part to internalize.

Step 1 — Prompt construction. For few-shot CoT, build a prompt containing several `(question, reasoning, answer)` triples, followed by the target question alone. For zero-shot CoT, take the target question and append a reasoning trigger.

Step 2 — Autoregressive decoding. The model generates tokens left to right. Because the reasoning steps are generated before the final answer, each subsequent token — including the answer — is conditioned on the entire reasoning trace, not just the original question.

Step 3 — Answer extraction. Once generation stops (at an end token or a fixed pattern like "The answer is"), a simple parser extracts the final answer from the generated text, discarding the reasoning trace for downstream use (though the trace is often kept for interpretability or debugging).

Step 4 (optional) — Self-consistency. Rather than decoding once greedily, sample the reasoning-and-answer generation multiple times with temperature or nucleus sampling, extract the final answer from each sample, and return the majority answer across samples. This trades extra inference compute for higher reliability, because different sampled reasoning paths that agree on an answer are strong evidence that answer is correct, while greedy decoding commits irrevocably to one path even if it goes astray early. ``` function CHAIN_OF_THOUGHT_PROMPT(question, exemplars, mode): if mode == "few-shot": prompt = "" for (q_i, reasoning_i, answer_i) in exemplars: prompt += "Q: " + q_i + "\n" prompt += "A: " + reasoning_i + " So the answer is " + answer_i + "\n\n" prompt += "Q: " + question + "\nA:" else if mode == "zero-shot": prompt = "Q: " + question + "\nA: Let's think step by step.\n" return prompt

function GENERATE(model, prompt, temperature): tokens = tokenize(prompt) generated = [] loop: logits = model.forward(tokens + generated) next_token = sample(logits[-1], temperature) # temperature = 0 -> greedy/argmax if next_token == END_OF_SEQUENCE: break generated.append(next_token) if length(generated) > MAX_LEN: break return detokenize(generated)

function EXTRACT_ANSWER(text):

# crude but typical: take content after last occurrence of an answer marker if "the answer is" in text.lower(): return text.lower().split("the answer is")[-1].strip(" .\n") return last_number_or_token(text)

function SELF_CONSISTENCY(model, question, exemplars, mode, num_samples, temperature): prompt = CHAIN_OF_THOUGHT_PROMPT(question, exemplars, mode) answer_counts = empty map from answer -> count for i in 1..num_samples: completion = GENERATE(model, prompt, temperature) a = EXTRACT_ANSWER(completion) answer_counts[a] += 1 return argmax over answer_counts # majority vote ``` The pseudocode has two edge cases worth noting explicitly. First, `MAX_LEN` guards against unbounded generation if the model never emits an end token — a real failure mode in practice for degenerate reasoning loops. Second, `EXTRACT_ANSWER` must handle the case where no explicit marker phrase appears; a production system falls back to heuristics (last number in the text, last quoted token) or simply re-prompts the model to restate its answer in a fixed format.

Inference cost. Let a direct-answer generation cost O(1) decoding steps beyond the prompt (a short answer) with a single forward pass of cost O(n·d² + n²·d) for sequence length n and hidden size d (the standard transformer self-attention plus feed-forward cost per layer, summed over layers). Chain-of-thought generation instead produces k additional reasoning tokens, so total decode cost grows to roughly O((n+k)·d² + (n+k)²·d), i.e. linear-to-quadratic in the extra reasoning length k. Self-consistency multiplies this by the number of sampled paths m, giving O(m·(n+k)·(d² + (n+k)·d)). This is the fundamental tradeoff: CoT and self-consistency buy accuracy with extra compute and extra latency, paid at inference time rather than training time.

Why extra generated tokens help — the expressivity argument. A transformer with a fixed number of layers L computes, in one forward pass, a function whose "circuit depth" is bounded by L regardless of input length. Problems that require more than L sequential composition steps — such as multi-digit multiplication, which needs a number of dependent carry-propagation steps growing with the number of digits — cannot in general be computed by such a bounded-depth circuit in a single pass, no matter how wide the layers are, unless the model has memorized shortcuts specific to the training distribution. This is analogous to the classical circuit-complexity fact that bounded-depth circuits (e.g., AC⁰) cannot compute certain functions (like parity) that unbounded-depth circuits can compute trivially. Chain-of-thought sidesteps the fixed-depth ceiling: each new generated token is produced by a fresh forward pass that can attend to all previously generated tokens, so a reasoning trace of length k effectively simulates a computation of depth proportional to k layers, chained together through the autoregressive loop. This is why CoT specifically helps on tasks whose intrinsic sequential-computation requirement exceeds the model's depth, and why it gives little benefit on tasks a single pass can already solve (simple lookups, short common-sense facts).

Correctness is not guaranteed — an important caveat. Chain-of-thought is a mechanism that increases the ceiling on achievable accuracy, not a proof procedure. Three distinct failure modes must be kept separate: 1. Reasoning error: the trace itself contains a wrong step (e.g., a mis-multiplication), propagating to a wrong final answer. More reasoning tokens create more surface area for such errors, which is why self-consistency (voting across independent samples) helps — errors in different sampled paths are less correlated than the signal toward the correct answer, so majority vote suppresses idiosyncratic mistakes if correct reasoning paths are more probable than any single wrong one under the model's distribution. Formally, if the model assigns probability p > 1/2 to reaching the correct final answer along some self-consistent majority of sampled reasoning paths (not literally that each path is correct with probability > 1/2, but that correct-answer paths outnumber any single incorrect-answer variant), then majority voting over m i.i.d. samples converges to the correct answer with probability approaching 1 as m grows, by a Chernoff-bound argument identical to that used for ensemble/bagging methods. 2. Unfaithful rationalization: the model produces a fluent, plausible-looking chain of reasoning that does not correspond to the actual computation that produced the answer — the trace is generated to look like good reasoning given the question and the answer distribution the model tends toward, not necessarily reflecting genuine intermediate belief states. This is a known and actively studied gap between CoT interpretability appeal and CoT faithfulness. 3. Prompt-format sensitivity: small changes in exemplar wording, ordering, or the trigger phrase can shift performance non-trivially, because the model is pattern-matching to the style of reasoning in-context, not executing a verified algorithm. None of this contradicts the core claim — CoT reliably raises accuracy on multi-step benchmarks in sufficiently capable models — but "reliably raises accuracy" is an empirical, statistical statement about aggregate performance, not a per-instance correctness guarantee.

Consider the arithmetic word problem: "A juggler has 16 balls. Half are golf balls, and half of the golf balls are blue. How many blue golf balls are there?" The correct answer is 4.

Now trace self-consistency on a harder variant with m = 5 samples at nonzero temperature:

Majority vote across the five samples: 4 appears 3 times, 2 once, 8 once → the self-consistency answer is 4, the correct one, even though 2 of 5 individual samples were wrong. This concretely illustrates the Chernoff-bound intuition above: as long as correct reasoning is the model's dominant mode rather than merely a possible mode, sampling and voting suppresses the minority error modes.

Chain-of-thought prompting emerged from empirical observations that large language models' latent capabilities could be unlocked purely through how a prompt is phrased, without any change to model weights — a discovery closely tied to the broader study of In-Context Learning and emergent abilities in scaled models. The technique's clean articulation — that inserting worked reasoning examples into a few-shot prompt substantially improves multi-step arithmetic, commonsense, and symbolic reasoning benchmarks in sufficiently large models — reframed prompting from a surface-level trick into a window onto how autoregressive computation can simulate multi-step algorithms.

Chain-of-thought is now a default ingredient in production LLM deployments wherever multi-step correctness matters more than latency. Coding assistants use it to plan an implementation before writing code, reducing logically inconsistent output. Math and STEM tutoring systems rely on step-by-step traces both for correctness and for pedagogical value — showing the student the reasoning, not just the answer. Retrieval-augmented question-answering pipelines use intermediate reasoning to decide what to search for next, chaining retrieval and reasoning steps together. This naturally extends toward LLM Agents, where a model does not just reason in text but interleaves reasoning steps with tool calls, API invocations, and environment observations — chain-of-thought is the connective tissue that lets an agent plan, act, observe a result, and re-plan, all within the same autoregressive stream. Reasoning-specialized models increasingly generate very long internal chains of thought before producing a final answer, treating extended sequential computation as a first-class capability rather than a prompting trick, which is the direct technological descendant of the ideas covered here.

Ask the tutor: How would you design an experiment to test whether a model's chain-of-thought trace is faithful to its actual computation, rather than a post-hoc rationalization?