Zero-Shot and Few-Shot Prompting

At the most basic level, prompting is just text concatenation. You build a string that contains some combination of a system instruction, optional examples, and the actual input you want the model to reason about. The model then predicts the most likely continuation of that string — and if your prompt is well-constructed, that continuation is exactly the output you wanted.

The Anatomy of a Prompt

A prompt has three logical zones: ``` [System / Task Instruction] [Few-shot examples — zero or more] [Live input to process] ``` In zero-shot mode the middle zone is empty. In few-shot mode it holds `k` complete input→output pairs.

Probability Framing

When an LLM generates an answer, it is computing a conditional probability over all possible output tokens given the prompt tokens. We can write this compactly:

Explanation: The model scores the full output sequence y by multiplying together the probability of each output token, conditioned on all previous tokens, the live input x, and the context C (your instruction plus examples).

Symbol guide:

• $y$ — the output sequence (e.g., the label "positive")
• $x$ — the live input (e.g., the sentence to classify)
• $C$ — the context, which is your instruction and any few-shot examples
• $y_t$ — the $t$-th token of the output
• $y_{<t}$ — all output tokens generated before position $t$

Adding few-shot examples changes $C$, which shifts the entire probability distribution toward outputs that match the demonstrated pattern — without ever touching the model weights.

In-Context Learning Score

For classification tasks, a useful way to think about how few-shot examples help is through the in-context learning objective. Given $k$ examples $\{(x_i, y_i)\}_{i=1}^{k}$, the model selects the label $\hat{y}$ that maximises:

Explanation: Among all possible labels in the label set Y, pick the one the model considers most probable given the live input and all k demonstration pairs.

Symbol guide:

• $\hat{y}$ — the predicted label
• $\mathcal{Y}$ — the set of valid output labels (e.g., {positive, negative})
• $k$ — the number of few-shot examples (0 for zero-shot)

A Minimal Python Example

Here is the full zero-shot → few-shot progression using OpenAI's API: ```python from openai import OpenAI client = OpenAI() # uses OPENAI_API_KEY from environment

# ── Zero-shot ────────────────────────────────────────────────────────────── zero_shot_prompt = """ Classify the sentiment of the following review as POSITIVE or NEGATIVE. Reply with a single word only. Review: "The battery died after two hours. Completely useless." Sentiment:""" resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": zero_shot_prompt}], max_tokens=5, ) print("Zero-shot:", resp.choices[0].message.content.strip())

# → "NEGATIVE"

# ── Few-shot (3 examples) ────────────────────────────────────────────────── few_shot_prompt = """ Classify the sentiment of a review as POSITIVE or NEGATIVE. Reply with a single word only. Review: "Absolutely love it, works perfectly." Sentiment: POSITIVE Review: "Stopped working after one week. Waste of money." Sentiment: NEGATIVE Review: "Decent quality for the price, very happy." Sentiment: POSITIVE Review: "The battery died after two hours. Completely useless." Sentiment:""" resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": few_shot_prompt}], max_tokens=5, ) print("Few-shot:", resp.choices[0].message.content.strip())

# → "NEGATIVE" ``` The only difference between the two calls is the text inside the prompt. The model is identical, the weights are identical — only the context $C$ changes.

How Many Examples Is "Few"?

The term "few-shot" is deliberately loose. In practice:

Beyond roughly 10 examples the gains per additional example diminish rapidly, and you start to risk hitting context window limits.

Chain-of-Thought Prompting · Retrieval-Augmented Generation · Instruction Tuning · Prompt Engineering · In-Context Learning