GPT-3 and In-Context Learning

GPT-3 (Generative Pre-trained Transformer 3) is a 175-billion-parameter autoregressive language model released by OpenAI in 2020 that demonstrated a striking new capability called in-context learning: the ability to perform tasks it was never explicitly trained on, simply by reading a few examples in the input prompt. It matters because it shifted how practitioners think about AI — from "train a model per task" to "prompt a single model for any task." The counterintuitive fact: GPT-3's weights are never updated during in-context learning; all the "learning" happens inside a single forward pass through the network.

The core insight

Before GPT-3, adapting a model to a new task meant fine-tuning — updating millions of weights with labeled data. GPT-3 showed that a large enough model, trained to predict the next word on vast text, implicitly learns how to learn from examples baked into the prompt itself. You hand it a few input-output pairs as plain text, and it figures out the pattern and applies it to a new input — no gradient updates, no retraining, no data pipeline. The "aha moment" is this: the prompt is the program, and the model is the interpreter.

How It Works

Autoregressive Language Modeling

GPT-3 is trained on one objective: predict the next token given all previous tokens. During pretraining on hundreds of billions of tokens of internet text, the model learns an enormous compressed representation of language, facts, reasoning patterns, and task formats.

1
P(tokent∣token1,token2,…,tokent−1)\quad P(\text{token}_t \mid \text{token}_1, \text{token}_2, \ldots, \text{token}_{t-1})

Explanation: The model assigns a probability to every possible next token, conditioned on the entire sequence of tokens that came before it.

  • token_t — the token we are predicting at position t
  • token₁ … token_{t-1} — all preceding tokens in the context window
  • P(·|·) — a conditional probability distribution produced by the softmax output layer

The Prompt as a Conditional Distribution

When you prepend examples to a query, you are simply concatenating them into the context. The model then predicts the next token of your query as if continuing a document it has seen many times in analogous form.

2
P(answer∣example1,…,examplek,query)\quad P(\text{answer} \mid \text{example}_1, \ldots, \text{example}_k, \text{query})

Explanation: The probability of the correct answer increases when relevant examples are present in the context, because the model has seen this prompt-completion pattern structure repeatedly during pretraining.

  • example₁ … example_k — the k few-shot demonstrations in the prompt
  • query — the new input you want the model to answer
  • answer — the token sequence the model generates greedily or by sampling

In-Context Learning Regimes

GPT-3 introduced three distinct prompting regimes:

RegimeExamples in PromptWeight Update?
Zero-shot0No
One-shot1No
Few-shot2–100+No

Here is a minimal Python snippet showing how these look in practice using the OpenAI API pattern: ```python import openai

# Zero-shot: just describe the task zero_shot_prompt = "Translate English to French.\nEnglish: cheese\nFrench:"

# Few-shot: show examples before the query few_shot_prompt = """Translate English to French. English: sea French: mer English: sky French: ciel English: cheese French:""" response = openai.chat.completions.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": few_shot_prompt}], max_tokens=5, temperature=0.0, ) print(response.choices[0].message.content) # → fromage ```

Scaling Laws and Emergence

GPT-3's in-context learning ability is closely tied to model scale. Researchers found a clean power-law relationship between model size, training compute, and loss:

3
L(N)≈(NcN)αN\quad L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}

Explanation: As the number of model parameters N grows, the test loss L follows a predictable power-law decrease — larger models reliably get better, all else equal.

  • L(N) — cross-entropy loss at parameter count N
  • N_c — a fitted constant representing a critical scale
  • αN — the scaling exponent (empirically ~0.076 for parameters alone)
  • This is the Chinchilla / Kaplan scaling law family

The crucial implication: certain capabilities, including reliable few-shot arithmetic and multi-step reasoning, appear discontinuously above a threshold size. Below ~7B parameters, few-shot performance on many benchmarks is near random; above it, performance jumps. This is called emergent behavior. ```python import numpy as np import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt

# Simulate a scaling curve params = np.logspace(8, 11, 100) # 100M to 100B parameters alpha = 0.076 Nc = 1e13 loss = (Nc / params) ** alpha plt.figure(figsize=(6, 3)) plt.loglog(params, loss) plt.xlabel("Parameters (N)") plt.ylabel("Loss L(N)") plt.title("Scaling Law: Loss vs. Model Size") plt.tight_layout() plt.savefig("scaling_law.png", dpi=120) ```

researcher · Tom B. Brown | b. 1990 | Lead author of the GPT-3 paper

Tom Brown was a research scientist at OpenAI who led the team that designed and trained GPT-3, co-authoring the landmark 2020 paper "Language Models are Few-Shot Learners" with over 30 collaborators. His central contribution was framing the in-context learning evaluation methodology — defining zero-shot, one-shot, and few-shot precisely and systematically benchmarking all three across dozens of NLP tasks. Surprisingly, Brown has noted in interviews that the team was genuinely uncertain whether 175B parameters would produce qualitatively different behavior, or just a quantitatively better version of GPT-2.

researcher · Alec Radford | b. 1990 | Architect of the GPT lineage

Alec Radford at OpenAI developed GPT-1 and GPT-2, establishing the pretrain-then-adapt paradigm that GPT-3 scaled dramatically. His 2019 GPT-2 paper "Language Models are Unsupervised Multitask Learners" was the conceptual seed of in-context learning: it showed that a language model trained only on next-token prediction implicitly learned to translate, summarize, and answer questions. Radford later became a key contributor to CLIP and Whisper, two multimodal systems that extend the same "scale up a simple objective" philosophy.

Figure
GPT-3 In-Context Learning Data Flow
GPT-3 In-Context Learning: Forward Pass Only PROMPT (Input) Example 1: Input: "Hello" → Hola Example 2: Input: "Bye" → Adiós Query (new): Input: "Cat" → ??? tokens TRANSFORMER 175B parameters (FROZEN — no update) Self-Attention ×96 Feed-Forward ×96 Softmax Output No backprop ✗ probs OUTPUT "Gato" (sampled from P(token|ctx))
Figure
Few-Shot vs Zero-Shot Accuracy vs Scale
Emergent Accuracy vs. Model Scale (Schematic) Model Parameters (log scale) → Accuracy % 1B 13B 70B 175B 20 40 60 80 emergence threshold Few-shot (k=32) Zero-shot

Intuition

The Mental Model

Imagine hiring an extremely well-read consultant who has read essentially all of the internet. You don't retrain them — they already know everything they know. Instead, you walk them into a meeting and hand them a sheet of paper that says: > "We're doing Spanish translations today. Here are two examples: Hello → Hola, Goodbye → Adiós. Now translate: Cat → ?" The consultant doesn't need a training course. They recognize the pattern immediately from the sheet and from years of reading bilingual documents. That sheet of paper is your prompt. The consultant's brain is the frozen GPT-3 weights. The meeting room conversation is a single forward pass. The more examples you give (few-shot), the less ambiguous the task becomes. The key insight is that GPT-3 saw millions of analogous documents during pretraining — instruction manuals, Q&A forums, code tutorials — and learned the structure of task-solving itself, not just facts. When you show it examples, you are essentially tuning a radio dial to the right frequency within that vast learned space, not installing new hardware.

Historical Development

Origins · 1 minhow this was discovered
History
2017: Vaswani et al. publish "Attention Is All You Need", introducing the Transformer architecture — the foundation GPT-3 is built on.
2018: OpenAI releases GPT-1 (Radford et al.) — demonstrates that pretraining on language modeling followed by fine-tuning beats task-specific architectures on NLP benchmarks.
2019: GPT-2 (Radford et al.) scales to 1.5B parameters and shows that, without any fine-tuning at all, a language model can perform rudimentary translation and QA — the first glimpse of zero-shot generalization.
2020: Brown et al. release "Language Models are Few-Shot Learners" — GPT-3 at 175B parameters, formally defining zero-shot, one-shot, and few-shot in-context learning and benchmarking them across 20+ NLP tasks.
2020: Kaplan et al. (OpenAI) publish "Scaling Laws for Neural Language Models" — establishes power-law relationships between model size, data, compute, and loss that explain why GPT-3's emergent abilities arise.
2021: OpenAI releases the GPT-3 API commercially, triggering an explosion of prompt engineering as a discipline and spawning the first wave of AI-native startups.
2022: Wei et al. (Google) publish "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" — shows that including reasoning steps in few-shot examples dramatically boosts GPT-3-class model performance on math and logic.
2022: OpenAI releases InstructGPT, applying RLHF on top of GPT-3 to align outputs with human intent — demonstrating that in-context learning alone is insufficient for reliable instruction-following.
2023: GPT-4 extends the paradigm with multimodal in-context learning (images + text), longer context windows (up to 128k tokens in later versions), and substantially improved emergent reasoning.

Why It Matters

Real-World Impact
GitHub CopilotUses GPT-3-class models with few-shot prompting to complete code; the "examples" are the code already written in the file — a natural in-context demonstration of the programmer's style and intent.
ChatGPT (OpenAI)Built on InstructGPT, itself a GPT-3 variant fine-tuned with RLHF; the entire conversation history functions as a growing few-shot context that conditions each new response.
Bing Chat / Microsoft CopilotInjects retrieved web documents into GPT-4's context window as implicit few-shot grounding, a technique called Retrieval-Augmented Generation (RAG) that exploits in-context learning to add up-to-date knowledge.
Medical NLP (GatorTron, Med-PaLM)Few-shot prompting of GPT-3-scale models lets researchers perform clinical entity extraction and question answering without labeled medical training data — critical when annotations are expensive and scarce.
Semantic Kernel / LangChain (Tool-Use Agents)These frameworks structure in-context examples to teach models to call APIs and tools; the examples in the prompt demonstrate the "use a tool, observe result, continue" loop.
DALL·E prompt engineeringUsers discovered that prefixing image prompts with style descriptors (e.g., "oil painting by Rembrandt") acts as a zero-shot style transfer instruction — exploiting GPT-3-family text understanding baked into the text encoder.
Automated customer support (Intercom, Zendesk AI)Few-shot prompts containing company-specific Q&A examples condition GPT-3-class models to answer in a brand voice without full fine-tuning, saving weeks of data labeling.

Prerequisites

What This Leads To

See Also

Transformer Architecture · RLHF and InstructGPT · Prompt Engineering · Scaling Laws · Chain-of-Thought Prompting

Where this leads