In-Context Learning

In-context learning is the ability of a pretrained autoregressive language model to perform a new task after conditioning on a prompt containing a task description and/or a small number of input-output demonstrations, with the model's parameters $\theta$ held completely fixed. Formally, given a sequence of demonstrations $(x_1, y_1), (x_2, y_2), \ldots, (x_k, y_k)$ and a new query $x_{k+1}$, the model computes

where $p_\theta$ is the same conditional distribution the model uses for ordinary next-token prediction. Nothing about the architecture or the training objective changes; the "learning" happens entirely inside a single forward pass, driven by the content of the context window. This is why the number of demonstrations $k$ is called the shot count: $k=0$ is zero-shot prompting, $k=1$ is one-shot, and $k>1$ is few-shot. It is crucial to separate ICL from three superficially similar ideas. Fine-tuning updates $\theta$ via gradient descent on task-specific data; ICL updates nothing. Meta-learning (e.g., MAML) explicitly trains a model, across many tasks, to be good at rapid adaptation via a few gradient steps at test time; ICL requires no such specialized meta-training objective — it emerges from vanilla next-token prediction on internet-scale text, though the diversity of that text plausibly makes pretraining resemble an implicit multi-task curriculum. Retrieval-augmented generation fetches external documents to append to the prompt; this can supply demonstrations for ICL but is a separate mechanism for sourcing content, not the learning mechanism itself. The prompt used for ICL typically has three components: an optional natural-language instruction, a set of demonstrations formatted consistently (e.g., `Input: ... \n Output: ...`), and the query. The consistency of formatting matters enormously, because the model is pattern-matching the structure of the demonstrations as much as their content — a fact that motivates the careful craft discussed under Prompting & Prompt Engineering.

Mechanically, ICL requires no special machinery beyond a standard Transformer decoder — the "algorithm" is just careful prompt construction followed by a single forward pass and greedy or sampled decoding. What differs from ordinary generation is how the prompt is built and how the output is extracted and scored.

Step 1 — Task framing. Choose a natural-language instruction (optional) and decide on the answer format (e.g., a single word, a label from a fixed set, a JSON object).

Step 2 — Demonstration selection. Choose $k$ example pairs $(x_i, y_i)$ that are representative of the mapping to be learned. In practice these are drawn from a labeled pool, sometimes selected to be similar to the query (nearest-neighbor retrieval) to improve accuracy.

Step 3 — Demonstration ordering and formatting. Concatenate the examples into a single string using a fixed template, then append the query with the output slot left blank.

Step 4 — Forward pass. Feed the full prompt through the Transformer. Every token — instruction, demonstration inputs, demonstration labels, and query — is embedded and processed by self-attention across all preceding tokens (causal masking permitting only left-to-right attention).

Step 5 — Decoding. Read off $p_\theta(y \mid \text{prompt})$ at the answer position(s), and decode (greedily, via sampling, or by scoring a fixed candidate set and picking the highest-probability candidate — useful for classification tasks with a known label set).

Step 6 (optional) — Calibration. Because raw next-token probabilities can be skewed by surface-form frequency (e.g., "positive" being a common word regardless of task), practitioners sometimes recalibrate by comparing to the model's output on a content-free query. ``` function IN_CONTEXT_LEARNING(instruction, demonstrations, query, model, label_set = None):

# demonstrations: list of (x_i, y_i) pairs

# Step 1-3: build the prompt string prompt = instruction + "\n" for (x_i, y_i) in demonstrations: prompt += format_input(x_i) + format_output(y_i) + "\n" prompt += format_input(query) # leave output slot open tokens = TOKENIZE(prompt) # length n if n > model.context_window: error "prompt exceeds context window" # hard constraint

# Step 4: single forward pass (no gradient computation, no weight update) hidden_states = model.FORWARD(tokens) # causal self-attention, O(n^2) naive logits = model.OUTPUT_HEAD(hidden_states[-1]) # distribution over vocabulary

# Step 5: decode if label_set is None: y_hat = GREEDY_OR_SAMPLE_DECODE(model, tokens) # free-form generation else:

# classification-style scoring: pick the label with highest sequence log-prob best_label, best_score = None, -infinity for label in label_set: score = LOG_PROB(model, tokens ++ TOKENIZE(label)) if score > best_score: best_label, best_score = label, score y_hat = best_label return y_hat ``` The pseudocode is deliberately unremarkable: that is the point. There is no learning-rate schedule, no loss function, no optimizer state. The entire "few-shot learner" is the frozen function `model.FORWARD` combined with a carefully chosen input string. All of the interesting behavior is a property of what happened during pretraining, not of any code written here.

Time complexity. Let $n$ be the total prompt length in tokens (instruction + demonstrations + query) and $L$ the number of Transformer layers with hidden dimension $d$. A single forward pass costs

— the $n^2 d$ term from self-attention (each of $n$ query positions attends to up to $n$ key positions, each attention operation costing $O(d)$), and the $n d^2$ term from the feed-forward and projection sublayers applied at each position. Adding one more demonstration of length $m$ tokens increases $n$ by $m$, so the marginal cost of the $k$-th shot grows the quadratic term roughly linearly in $n$ per added token, meaning total cost grows roughly quadratically in the number of shots if each shot has fixed length. In deployed systems, key-value caching lets the demonstration prefix be processed once and reused for multiple queries, amortizing this cost when many queries share the same few-shot prompt; generation of the answer itself, token by token, costs $O(n)$ per new token with caching (versus recomputing the full $O(n^2)$ attention from scratch without it). Contrast this with fine-tuning: gradient-based adaptation costs $O(E \cdot |D| \cdot C_{\text{step}})$ for $E$ epochs over a dataset of size $|D|$, paid once, after which inference is cheap and prompt-length-independent. ICL pays no training cost but re-pays an inference-time cost — proportional to $n^2$ per query — every single time the task is invoked, unless caching is used. This is the central efficiency tradeoff: ICL trades upfront training cost for recurring, prompt-length-dependent inference cost, and is advantageous when the task changes frequently or training data is too scarce to fine-tune on, but is disadvantageous at very high query volume for a fixed task.

Space complexity. The KV cache for a prompt of length $n$ requires $O(L \cdot n \cdot d)$ memory (two matrices, keys and values, per layer, per token). This is why long few-shot prompts with many demonstrations directly compete with the model's context-window budget and memory footprint — a real deployment constraint, not just an abstraction.

Correctness — why does it work at all? There is no single proof that ICL "is correct" in the sense of a sorting algorithm; instead there are two complementary explanations, each with supporting theoretical and empirical evidence. View 1: ICL as implicit Bayesian inference. Pretraining on a diverse text corpus can be modeled as training on a mixture of latent "concepts" or tasks $\tau$. The pretraining objective — next-token prediction — implicitly forces the model to learn

When a prompt contains several demonstrations consistent with a single latent task $\tau^*$, the posterior $p(\tau \mid \text{context})$ concentrates on $\tau^*$ as more demonstrations are added (each additional example is further evidence, in the Bayesian-updating sense), sharpening the model's implicit prediction. This view — supported by work formalizing pretraining as latent-variable model estimation and by controlled synthetic experiments (e.g., Hidden Markov Model-generated data with recoverable latent tasks) — correctly predicts that ICL accuracy should increase with number of demonstrations, that low-diversity demonstrations concentrate the posterior faster, and that models can perform ICL even on tasks with example distributions unlike anything in pretraining, as long as the format triggers task-identification. View 2: ICL as implicit gradient descent in attention. A separate line of theoretical work shows that a linear self-attention layer, given a prompt of demonstrations, can be constructed (and is empirically approximated by trained Transformers) to perform an update mathematically equivalent to one step of gradient descent on a linear regression loss fit to the in-context demonstrations. Concretely, for a linear attention layer with appropriately structured weight matrices, the layer's output on the query equals

for an implicit learning rate $\eta$ determined by the attention weights — i.e., the forward pass literally computes what one gradient step of an outer optimizer would compute, using the demonstrations as the training set for that single step. This is a constructive existence proof (such attention weights exist and are reachable by training) rather than a proof that all Transformers learn this exact mechanism, but empirically the predictions and even the intermediate activations of trained models match this gradient-descent construction closely on synthetic regression tasks. It aligns with the discovery of induction heads — attention heads that specifically implement a "copy the token that followed the last occurrence of the current token" operation, which is exactly the primitive needed to complete a pattern established by demonstrations. Neither theory is fully general, and both are honestly incomplete for the full range of real-world ICL behavior (arbitrary natural-language tasks, non-linear mappings, multi-step reasoning). What is well established empirically and theoretically is: (1) ICL performance improves with model scale in a manner tightly linked to Emergent Abilities & Scaling Laws, (2) ICL performance improves (with diminishing returns, and occasional degradation from long-context effects) with more demonstrations, and (3) ICL is genuinely doing conditional inference over the prompt, not simple memorized lookup, since it generalizes to label mappings never seen during pretraining (e.g., flipped or randomized labels in controlled studies still shift model behavior, showing sensitivity to the actual demonstration content, though performance often degrades relative to correct labels — evidence that both the input-output format and label correctness both matter, and their relative contribution is task-dependent).

Consider a sentiment-classification task the model was never explicitly fine-tuned for: classifying restaurant reviews as `POS` or `NEG`. We build a 3-shot prompt.

Prompt constructed: ``` Classify the sentiment of the review as POS or NEG. Review: The service was slow but the food made up for it. Sentiment: POS Review: Cold soup, rude waiter, and we waited 40 minutes. Sentiment: NEG Review: Portions were tiny and overpriced for what we got. Sentiment: NEG Review: Best pasta I've had all year, will absolutely return. Sentiment: ``` Trace through the pipeline:

Now perturb the experiment to test robustness, which is the standard way researchers probe why ICL is working rather than merely memorizing surface strings:

• Swap labels randomly (demonstration 1 relabeled NEG despite positive content): accuracy on new queries typically drops, but does not collapse to chance for large models — evidence the model partially relies on prior semantic knowledge of "sentiment," not purely on the mapping shown.
• Keep labels correct but change the label tokens to arbitrary symbols (`Sentiment: Foo` / `Sentiment: Bar`): large models can still learn the mapping correctly from the pattern alone, showing the mechanism generalizes beyond memorized label words — strong evidence for genuine in-context pattern induction, not lookup of familiar words.
• Increase shots from 3 to 20: accuracy improves with diminishing returns, consistent with the Bayesian posterior-concentration account, though very long prompts can sometimes hurt accuracy due to attention dilution or recency bias (the model over-weighting the last few demonstrations) — a documented practical failure mode.

physicist[Alec Radford|active researcher, OpenAI, 2010s–2020s|Co-developed GPT-2 and GPT-1, demonstrating that generative pretraining on raw text alone yields transferable task capability, laying the empirical groundwork that culminated in explicit few-shot in-context learning with GPT-3.]

In-context learning underlies nearly every practical interaction with a modern large language model API. Customer-support automation systems inject a handful of example resolved tickets into a prompt to adapt a single deployed model to a company's specific tone and policies, without any fine-tuning pipeline. Structured data extraction tools (turning unstructured text into JSON records) rely on few-shot examples in the prompt to specify the exact schema desired. Code assistants use in-context examples of a repository's coding style or API usage patterns to generate consistent completions. Evaluation benchmarks for new foundation models almost universally report both zero-shot and few-shot (in-context) scores, since the gap between them is itself a diagnostic of model quality and scale. The technique also underlies retrieval-augmented generation systems, where retrieved documents function as in-context "demonstrations" or evidence the model conditions on at inference time. Finally, ICL is the substrate on which more sophisticated prompting strategies are built — Prompting & Prompt Engineering studies how to construct better in-context examples and instructions, and Chain-of-Thought Reasoning studies a special case of in-context demonstration where the shown examples include intermediate reasoning steps rather than only final answers, dramatically improving performance on multi-step problems.

Ask the tutor: Why does adding more few-shot examples sometimes hurt accuracy instead of helping it?