MMLU and Standard AI Benchmarks

MMLU works by presenting a language model with a question and four labeled answer choices (A, B, C, D), then checking whether the model selects the correct one. The question pool spans 57 subjects organized into four broad categories: STEM, humanities, social sciences, and "other" (medicine, law, business, etc.). Each subject contains between 100 and 1000+ questions drawn from practice exams, textbooks, and standardized tests.

How Models Are Evaluated

There are two common evaluation modes:

Few-shot prompting: The model is shown a few example question-answer pairs before seeing the real question. This is the standard MMLU setup — typically 5-shot.

Zero-shot: The model sees only the question with no examples. Here is a minimal Python implementation of how MMLU evaluation works: ```python import anthropic # or openai, transformers, etc. def evaluate_mmlu_question(model_fn, question, choices, correct_answer): """ model_fn: callable that takes a prompt string, returns a string question: str choices: dict like {"A": "...", "B": "...", "C": "...", "D": "..."} correct_answer: str, one of "A", "B", "C", "D" """ prompt = f"""Question: {question} A) {choices['A']} B) {choices['B']} C) {choices['C']} D) {choices['D']} Answer with a single letter (A, B, C, or D):""" response = model_fn(prompt).strip().upper()

# Extract just the letter predicted = response[0] if response and response[0] in "ABCD" else None is_correct = (predicted == correct_answer) return { "predicted": predicted, "correct": correct_answer, "is_correct": is_correct }

def compute_accuracy(results): """Compute accuracy across a list of result dicts.""" correct = sum(1 for r in results if r["is_correct"]) return correct / len(results) if results else 0.0

# Example usage example_question = "What is the powerhouse of the cell?" example_choices = { "A": "Nucleus", "B": "Mitochondria", "C": "Ribosome", "D": "Golgi apparatus" }

# Simulate a perfect model for illustration results = [evaluate_mmlu_question( lambda p: "B", # dummy model always says B example_question, example_choices, "B" )] print(f"Accuracy: {compute_accuracy(results):.2%}") # 100.00% ```

The Accuracy Formula

Explanation: This tells us what fraction of questions the model got right — the single number most benchmark leaderboards report. For MMLU specifically, a random-guessing baseline on 4-choice questions gives:

Explanation: Any model scoring below 25% is doing worse than random guessing, meaning it has learned to avoid the right answer. A common refinement is subject-weighted accuracy, which prevents subjects with more questions from dominating the overall score:

Explanation: This averages the accuracy across all 57 subjects equally, so a model isn't rewarded for being great at one popular subject and ignoring rare ones; here $\mathcal{S}$ is the set of all subjects and $\text{Accuracy}(s)$ is the model's accuracy on subject $s$.

A Worked Example: Few-Shot Prompting

```python def build_few_shot_prompt(examples, question, choices): """ Build a 5-shot MMLU-style prompt. examples: list of dicts with keys: question, choices, answer """ prompt = "" for ex in examples: prompt += f"Question: {ex['question']}\n" for letter, text in ex['choices'].items(): prompt += f"{letter}) {text}\n" prompt += f"Answer: {ex['answer']}\n\n"

# Now add the real question prompt += f"Question: {question}\n" for letter, text in choices.items(): prompt += f"{letter}) {text}\n" prompt += "Answer:" return prompt

# Demonstrate examples = [ { "question": "Which planet is closest to the Sun?", "choices": {"A": "Venus", "B": "Mercury", "C": "Mars", "D": "Earth"}, "answer": "B" } ] real_q = "What gas do plants absorb from the atmosphere during photosynthesis?" real_choices = {"A": "Oxygen", "B": "Nitrogen", "C": "Carbon dioxide", "D": "Hydrogen"} prompt = build_few_shot_prompt(examples, real_q, real_choices) print(prompt) ```

Other Standard Benchmarks

MMLU is one of many. Here is a map of the standard benchmark landscape:

```python

# Practical example: detecting benchmark regression after fine-tuning

# Run this after any fine-tuning step to catch capability loss import json def benchmark_regression_check(baseline_scores, new_scores, threshold=0.02): """ Compare benchmark scores before and after fine-tuning. Raise an alert if any subject drops by more than threshold. baseline_scores: dict {subject: accuracy} new_scores: dict {subject: accuracy} threshold: float, e.g. 0.02 = allow up to 2% drop """ regressions = [] for subject, baseline_acc in baseline_scores.items(): new_acc = new_scores.get(subject, 0.0) drop = baseline_acc - new_acc if drop > threshold: regressions.append({ "subject": subject, "baseline": baseline_acc, "new": new_acc, "drop": drop }) if regressions: print("⚠️ REGRESSION DETECTED after fine-tuning:") for r in sorted(regressions, key=lambda x: -x["drop"]): print(f" {r['subject']}: {r['baseline']:.1%} → {r['new']:.1%} " f"(dropped {r['drop']:.1%})") else: print("✅ No significant regression detected.") return regressions

# Simulated example baseline = {"medicine": 0.82, "law": 0.79, "mathematics": 0.75, "history": 0.88} after_rlhf = {"medicine": 0.80, "law": 0.74, "mathematics": 0.75, "history": 0.87} benchmark_regression_check(baseline, after_rlhf, threshold=0.02)

# Output:

# ⚠️ REGRESSION DETECTED after fine-tuning:

# law: 79.0% → 74.0% (dropped 5.0%) ```

Reinforcement Learning from Human Feedback · Constitutional AI · Hallucination and Factuality in LLMs · AI Safety and Red-Teaming · Transformer Architecture