Instruction Tuning

Instruction tuning is a form of supervised fine-tuning (Supervised Learning). You start with a pretrained model and continue training it, but now every training example has a specific structure: a system prompt (optional), a user instruction, and a target response written by a human or a stronger model.

The Training Data Format Each example follows a template, for instance: ```

### Instruction: Summarize the following paragraph in one sentence.

### Input: The mitochondria is the powerhouse of the cell. It produces ATP through...

### Response: The mitochondria generates ATP, providing energy for cellular processes. ``` The model only receives a loss signal on the Response tokens — it is not penalized for how it processes the instruction or input. This is called a masked language modeling loss or causal LM loss with prompt masking.

The Loss Function The model is trained to minimize the standard cross-entropy loss over the response tokens:

Explanation: The model is penalized for each response token it gets wrong — lower loss means the model assigns higher probability to the correct next word in the target response. Where:

• $\mathcal{L}$ is the average loss over the response
• $T$ is the number of tokens in the target response
• $y_t$ is the $t$-th token of the response
• $x$ is the full instruction/input context
• $P_\theta$ is the probability the model (with parameters $\theta$) assigns to token $y_t$

The key difference from standard pretraining is that tokens in the instruction portion have their loss masked to zero — you only backpropagate through the response.

A Minimal Python Example Here is how you would construct this masked loss in code using Hugging Face: ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch tokenizer = AutoTokenizer.from_pretrained("gpt2") model = AutoModelForCausalLM.from_pretrained("gpt2") instruction = "### Instruction:\nTranslate to French.\n### Input:\nHello\n### Response:\n" response = "Bonjour"

# Tokenize full sequence full_text = instruction + response inputs = tokenizer(full_text, return_tensors="pt") input_ids = inputs["input_ids"]

# Build labels: mask the instruction tokens with -100 instruction_ids = tokenizer(instruction, return_tensors="pt")["input_ids"] n_instruction_tokens = instruction_ids.shape[1] labels = input_ids.clone() labels[:, :n_instruction_tokens] = -100 # ignore instruction in loss

# Forward pass — loss is only computed on response tokens outputs = model(**inputs, labels=labels) loss = outputs.loss print(f"Loss: {loss.item():.4f}") ``` `-100` is PyTorch's convention for "ignore this token in the loss calculation." This is the single most important implementation detail in instruction tuning.

Scaling with Multiple Tasks One powerful discovery is that mixing many different types of instructions (summarization, translation, question answering, coding, etc.) during fine-tuning causes the model to generalize to new task types at inference time. This is sometimes called zero-shot generalization via instruction tuning.

Explanation: The fine-tuning dataset is simply the union of K different task datasets, and diversity across task types is what drives generalization. Where $\mathcal{D}_k$ is the dataset for task type $k$ (e.g., summarization, translation) and $K$ is the number of distinct task types included.

Pretraining · RLHF · Few-Shot Learning · Fine-Tuning · Chain-of-Thought Prompting