OpenAI o1 and o3: Chain-of-Thought at Scale

OpenAI's o1 and o3 model families represent a paradigm shift in large language model deployment: rather than producing answers in a single forward pass, these models perform extended chain-of-thought reasoning internally before emitting a final response, trading compute at inference time for dramatically higher accuracy on hard reasoning tasks. This matters because it decouples "thinking budget" from model size — a smaller model that thinks longer can outperform a larger model that answers immediately. The counterintuitive fact is that o3, on the ARC-AGI benchmark, achieved ~88% accuracy compared to ~60% for GPT-4o, not by being larger, but by being allowed to search over its own reasoning trajectories for longer.

The core insight

The fundamental insight is that inference-time compute is a new scaling axis. Traditional scaling laws (Chinchilla, Kaplan et al.) describe how model quality improves with more parameters and training tokens — but o1/o3 demonstrate a complementary law: for a fixed model, allowing more sequential reasoning steps (or parallel search over reasoning paths) reliably improves performance on tasks with verifiable answers. The "aha moment" is realizing that a language model generating its own scratchpad tokens is performing a learned search through solution space, and that this search can be made arbitrarily deeper by simply allowing more tokens — without retraining.

How It Works

Inference-Time Compute and the Reasoning Policy

At the heart of o1/o3 is a reasoning policy π_θ that maps a question q to a distribution over (thinking trace, answer) pairs. Rather than directly predicting P(answer | question), the model learns:

1
P(a∣q)  =  ∑z∈ZPθ(a∣q,z) Pθ(z∣q)\quad P(a \mid q) \;=\; \sum_{z \in \mathcal{Z}} P_{\theta}(a \mid q, z)\, P_{\theta}(z \mid q)

Explanation: The final answer probability marginalizes over all possible latent reasoning traces z, so the model implicitly considers many thinking paths before committing to an answer. Here:

  • a = the final answer token sequence
  • q = the input question
  • z ∈ 𝒵 = a latent chain-of-thought trace (a sequence of "thinking" tokens)
  • P_θ(z | q) = the model's learned prior over reasoning traces
  • P_θ(a | q, z) = the likelihood of the answer given a specific reasoning trace

In practice, 𝒵 is too large to sum over exactly. Instead, o1/o3 use Monte Carlo sampling — drawing N reasoning traces, then either returning the best or aggregating via majority vote (a form of self-consistency):

2
a^  =  ∑i=1N1 ⁣[ai=a],ai∼Pθ(⋅∣q,zi),  zi∼Pθ(⋅∣q)\quad \hat{a} \;=\; \underset{a}{\arg\max} \sum_{i=1}^{N} \mathbf{1}\!\left[a_i = a\right], \quad a_i \sim P_{\theta}(\cdot \mid q, z_i),\; z_i \sim P_{\theta}(\cdot \mid q)

Explanation: The best-of-N answer is chosen by majority vote across N independently sampled reasoning traces, exploiting the fact that correct reasoning paths tend to converge on the same answer.

Reinforcement Learning from Outcome Supervision

The policy is trained using a process- or outcome-reward model (PRM / ORM). For a reasoning trace z = (s₁, s₂, …, s_T) composed of T reasoning steps, an outcome reward model scores the final answer:

3
LRL(θ)  =  − Ez∼πθ(⋅∣q) ⁣[R(a(z), a∗)]+β DKL ⁣[πθ(⋅∣q)  ∥  πref(⋅∣q)]\quad \mathcal{L}_{\mathrm{RL}}(\theta) \;=\; -\,\mathbb{E}_{z \sim \pi_\theta(\cdot|q)}\!\left[R(a(z),\, a^*)\right] + \beta\, D_{\mathrm{KL}}\!\left[\pi_\theta(\cdot|q) \;\|\; \pi_{\mathrm{ref}}(\cdot|q)\right]

Explanation: The RL objective maximizes expected reward for producing correct final answers while the KL-divergence penalty keeps the reasoning policy close to a reference model, preventing degenerate or incoherent traces. Here:

  • R(a(z), a*) = reward signal (1 if correct, 0 if wrong, for math/code tasks)
  • β = KL penalty coefficient, typically small (0.01–0.1)
  • π_ref = a frozen reference policy (the base SFT model)
  • This is essentially RLHF/RLAIF with a verifiable reward signal, similar in spirit to InstructGPT but with correctness replacing human preference

Process Reward Models

A process reward model (PRM) assigns credit to individual reasoning steps rather than only the final answer:

4
RPRM(z)  =  ∑t=1TγT−t rt(s1,…,st)\quad R_{\mathrm{PRM}}(z) \;=\; \sum_{t=1}^{T} \gamma^{T-t}\, r_t(s_1,\ldots,s_t)

Explanation: The PRM reward is a discounted sum of per-step rewards, allowing the model to receive learning signal for intermediate reasoning steps rather than waiting until the answer is complete. Where:

  • r_t = reward assigned to step t by the PRM (often trained from human annotations of step correctness)
  • γ = temporal discount factor (often 1.0 for short reasoning chains)
  • s_t = the t-th reasoning step token sequence

Scaling Law for Inference Compute

Empirically, OpenAI and independent researchers have found a log-linear relationship between the number of sampled reasoning tokens C and benchmark accuracy A:

5
A(C)  ≈  A0  +  α⋅log⁡ ⁣(CC0)\quad A(C) \;\approx\; A_0 \;+\; \alpha \cdot \log\!\left(\frac{C}{C_0}\right)

Explanation: Each doubling of inference-time compute (measured in reasoning tokens) yields a roughly constant additive gain in accuracy, analogous to the log-linear training scaling laws of Kaplan et al. Where:

  • A₀ = baseline accuracy at reference compute C₀
  • α = compute efficiency coefficient (task-dependent; higher for math/code)
  • C = total reasoning tokens generated (across all sampled traces)
  • This mirrors Chinchilla-style scaling but on the inference axis

---

PyTorch Sketch: Best-of-N Reasoning Sampling

```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from collections import Counter def best_of_n_reasoning( model: AutoModelForCausalLM, tokenizer: AutoTokenizer, question: str, n_samples: int = 16, max_think_tokens: int = 2048, device: str = "cuda", ) -> str: """ Approximate o1-style best-of-N inference. Model must be fine-tuned to produce ...... format. """ prompt = f"<|user|>{question}<|assistant|>" inputs = tokenizer(prompt, return_tensors="pt").to(device) answers = [] for _ in range(n_samples): with torch.no_grad(): output_ids = model.generate( **inputs, max_new_tokens=max_think_tokens, do_sample=True, temperature=0.8, # Diversity over reasoning traces top_p=0.95, pad_token_id=tokenizer.eos_token_id, ) decoded = tokenizer.decode(output_ids[0], skip_special_tokens=False)

# Parse answer from structured output if "" in decoded and "" in decoded: start = decoded.index("") + len("") end = decoded.index("") answers.append(decoded[start:end].strip()) if not answers: return "No answer extracted."

# Majority vote (self-consistency) vote_counts = Counter(answers) best_answer, votes = vote_counts.most_common(1)[0] print(f"Majority answer: '{best_answer}' ({votes}/{len(answers)} votes)") return best_answer ```

researcher · Ilya Sutskever | b. 1986 | Co-inventor of deep learning for sequence models and co-founder of OpenAI

Sutskever completed his PhD under Geoffrey Hinton at the University of Toronto, co-authoring the seminal AlexNet paper and the sequence-to-sequence (Seq2Seq) paper that underpins modern language models. His foundational work on training deep recurrent networks and on scaling transformer-based language models directly enabled OpenAI's research culture of treating compute as the primary lever — a philosophy that culminates in the inference-compute scaling embodied by o1/o3. Surprisingly, Sutskever co-authored a 2014 paper demonstrating that simply reversing the input sequence in Seq2Seq improved translation quality — an insight that foreshadowed the value of restructuring computation rather than just adding parameters.

researcher · Jason Wei | b. ~1997 | Lead author of Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Wei, a researcher at Google Brain (later Google DeepMind) before joining OpenAI, published the landmark 2022 NeurIPS paper demonstrating that prompting large models with step-by-step reasoning examples dramatically improved performance on multi-step arithmetic and commonsense tasks. His work established chain-of-thought as a first-class capability rather than a curiosity — directly laying the conceptual groundwork for training models to reason internally, as o1 does. Wei's paper showed the effect only emerged at ~100B parameters, a "phase transition" result that made chain-of-thought one of the most studied emergent behaviors in LLM research.

Figure
o1/o3 Inference-Time Reasoning Architecture
o1 / o3 — Inference-Time Reasoning Pipeline User Question q Reasoning Policy π_θ Sample N traces z_i ~ P_θ(z|q) Temperature T=0.8, top-p=0.95 (hidden from user) Trace z₁ ans: 42 Trace z₂ ans: 42 Trace z₃ ans: 17 Majority Vote / PRM Score Best answer: 42 (2/3 votes) Final Answer â Shown to user Training Loop (RL) SFT warm-start on CoT data π_ref ← π_SFT GRPO / PPO RL Reward: answer correctness + KL pen. vs π_ref Process Reward Model (PRM) Step-level credit assignment Updated π_θ → o1 / o3 Accuracy ≈ A₀ + α·log(C/C₀) — more tokens → better answers
Figure
Inference Compute Scaling Curve
Inference Compute Scaling (log-linear) Inference Tokens (log scale) Accuracy (%) 2⁸ 2¹⁰ 2¹² 2¹⁴ 2¹⁶ 2¹⁸ 40 55 70 80 88 GPT-4o o1 o3 ~88% ~78% o3 (ARC-AGI) o1 (ARC-AGI) GPT-4o (no CoT scaling)

Intuition

The Mental Model

Imagine you're taking a difficult math olympiad exam. You could try to answer each problem instantly from gut instinct — that's what standard LLMs do, producing the answer in one forward pass. Or you could use scratch paper: try an approach, notice it's going wrong, back up, try another angle, verify intermediate steps — and only write your final answer once you're confident. That's what o1/o3 do with their hidden reasoning tokens. Now extend the metaphor: o3 doesn't just use scratch paper — it's like having 16 students take the same exam simultaneously, each using their own scratch paper and their own approach, and then checking whether the majority agree. When most of them converge on "42," you're much more confident than if one person guessed "42" instantly. The more "students" (sampled traces) and the more "scratch paper" (reasoning tokens) you allow, the better the result — up to a log-linear scaling limit determined by the task's inherent difficulty. The key realization is that this is learned behavior, not just prompted: the model has been trained by RL to make its scratch-paper thinking genuinely useful — pruning dead ends, double-checking algebra, and structuring its approach — because those are the reasoning styles that historically led to correct final answers and earned reward.

Historical Development

Origins · 1 minhow this was discovered
History
2017: Vaswani et al. publish Attention Is All You Need, establishing the Transformer as the universal substrate on which all subsequent reasoning work is built.
2021: Cobbe et al. (OpenAI) introduce the GSM8K benchmark and show that training a verifier (ORM) to re-score generated solutions dramatically improves math accuracy — the first concrete evidence that scoring reasoning is easier than generating it.
2022: Wei et al. (Google Brain) publish Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (NeurIPS 2022), demonstrating that few-shot prompts with step-by-step examples unlock multi-step reasoning in LLMs ≥ 100B parameters.
2022: Wang et al. introduce self-consistency — sampling multiple chain-of-thought paths and majority-voting — showing consistent gains over single-path CoT, providing the sampling-and-aggregation blueprint used in o1.
2023: Lightman et al. (OpenAI) publish Let's Verify Step by Step, training a Process Reward Model on human-labeled per-step correctness judgments for MATH problems, demonstrating PRMs outperform ORMs at identifying which reasoning paths are actually correct.
2023: DeepMind's AlphaCode 2 and Google's Minerva demonstrate that repeated sampling and filtering using code execution or equation verification can push benchmark scores far beyond single-sample generation, validating inference-time compute scaling empirically.
2024 (September): OpenAI releases o1-preview and o1-mini, the first publicly deployed models explicitly framed around extended internal chain-of-thought; o1-preview achieves 83rd-percentile performance on Codeforces and top scores on AIME 2024.
2024 (December): OpenAI releases o1 (full version) and begins previewing o3; o3 scores ~88% on ARC-AGI (vs. ~60% for GPT-4o) and achieves gold-medal-equivalent performance on IMO 2024 problems, stunning the mathematics community.
2025: Independent scaling analyses (e.g., Snell et al., Scaling LLM Test-Time Compute Optimally) formally characterize the inference compute scaling law and the conditions under which PRMs vs. majority vote vs. beam search over reasoning is most efficient.

Why It Matters

Real-World Impact
OpenAI o1 (Production API)Used by developers for tasks requiring multi-step legal reasoning, financial modeling, and complex code generation; OpenAI reports o1 reduces coding errors by ~25% on internal evals compared to GPT-4o on hard algorithmic problems.
OpenAI o3 on ARC-AGIAchieved ~88% on the Abstraction and Reasoning Corpus, a benchmark explicitly designed to resist pattern memorization; this result is widely cited as evidence that inference-time search can approximate genuine novel problem-solving in ways pure scaling cannot.
Mathematical Research Assistanceo1/o3-class models are being used by research mathematicians at institutions including MIT and Cambridge to verify proof steps and explore combinatorial conjectures, with the extended reasoning budget allowing multi-page derivations to be checked end-to-end.
Competitive Programmingo1 achieved Codeforces ratings equivalent to a 1700+ ELO competitive programmer; o3 has reportedly solved problems from recent IOI rounds, surpassing the median human contestant — a milestone that required both world knowledge and sustained algorithmic reasoning.
Scientific Drug Discovery (BioTech)Startups including Recursive Pharmaceuticals are integrating o1-class reasoning for proposing novel synthesis pathways — tasks that require chaining hundreds of chemistry inference steps, which single-pass LLMs routinely fail.
Curriculum Learning for Smaller ModelsThe reasoning traces generated by o1/o3 are being used as training data to distill chain-of-thought capabilities into smaller, cheaper models (e.g., DeepSeek-R1-Distill series), democratizing extended reasoning beyond frontier-scale budgets.
Safety and Interpretability ResearchThe explicit scratchpad structure of o1-style models provides an unprecedented window into model reasoning, enabling mechanistic interpretability researchers to study where a model goes wrong in a multi-step argument — something that single-token-output models cannot offer.

Prerequisites

What This Leads To

leads

Before this