Evaluation of LLMs

Evaluating a large language model means estimating, from a finite set of test interactions, some underlying quantity — accuracy, helpfulness, harmlessness, factual reliability — that is really a property of the model's behavior over an effectively infinite space of possible prompts. Formally, if $f_\theta$ is a model and $\mathcal{D}$ is the true (unknown) distribution of deployment queries, we want an estimate of $\mathbb{E}_{x \sim \mathcal{D}}[\, \text{score}(f_\theta(x), x)\, ]$. Every benchmark is a finite sample $\{x_i\}_{i=1}^n$ drawn from some proxy distribution $\mathcal{D}'$, and every scoring function is a proxy for the quantity we actually care about. The gap between $\mathcal{D}'$ and $\mathcal{D}$, and between the proxy score and true usefulness, is where evaluation goes wrong. Three broad families of evaluation methodology exist. Automatic reference-based evaluation compares model output to a gold answer using exact match, F1 overlap, ROUGE/BLEU n-gram overlap, or code execution (pass@k on unit tests). This is cheap, reproducible, and deterministic, but only works when there is a well-defined correct answer — it fails for open-ended generation, creative writing, or multi-turn dialogue. Human evaluation asks people to rate or compare outputs directly, typically via pairwise comparison ("which response is better?") or Likert-scale absolute scoring (1–5 helpfulness, harmlessness). This captures nuance that automatic metrics miss but is expensive, slow, and subject to annotator disagreement and bias. LLM-as-judge evaluation uses a capable model (often GPT-4-class) to play the role of the human rater, either scoring a single response against a rubric or judging a pairwise comparison. This scales to millions of examples at low cost but inherits and amplifies specific biases from the judge model itself. A key conceptual distinction is capability evaluation versus alignment evaluation. Capability evaluation asks "can the model solve this task at all" (e.g., can it prove a theorem, pass a coding test, answer a trivia question) — closely tied to what Hallucination & Factuality measures when the task requires accurate world knowledge. Alignment evaluation asks "does the model behave the way we want it to" — is it helpful, honest, harmless, and does it refuse the right things at the right times. The two axes are not independent: a maximally capable model that is not aligned can produce highly fluent, confident, and wrong or harmful text, which is exactly why alignment evaluation increasingly leans on adversarial and red-teaming-style probes, foreshadowing the deeper treatment of Safety & Red-Teaming.

Automatic benchmark evaluation

The classical pipeline is: fix a test set with gold labels, run the model, score with a deterministic function, and aggregate. For multiple-choice benchmarks (e.g., MMLU-style), the score is exact-match accuracy over answer letters. For generative benchmarks with a reference (e.g., summarization), softer overlap metrics (ROUGE-L, BLEU) or embedding-based metrics (BERTScore) are used because exact string match is too strict for free-form text. For code, benchmarks like HumanEval use functional correctness: generate $k$ candidate completions, execute them against unit tests, and report `pass@k` — the probability that at least one of $k$ samples passes. ``` function PASS_AT_K(problem, model, n, k):

# n: total samples drawn, k: budget considered "allowed attempts" samples <- [] for i in 1..n: samples.append(model.generate(problem.prompt, temperature=0.8)) c <- count(s in samples such that EXECUTE(s, problem.tests) == PASS)

# unbiased estimator (avoids re-sampling k from n combinatorially) if n - c < k: return 1.0 else: return 1.0 - PRODUCT_{i=0}^{k-1} ( (n - c - i) / (n - i) ) ``` The unbiased estimator above is important: naively drawing $k$ samples and checking if any pass has high variance for small $k$. Instead, Chen et al.'s pass@k estimator draws a larger $n \gg k$, counts correct samples $c$, and computes the exact probability that a random $k$-subset of the $n$ samples contains at least one correct one, which is $1 - \binom{n-c}{k}/\binom{n}{k}$ — the product form above is an equivalent, numerically stable way to compute this without overflowing large binomial coefficients.

Human evaluation protocol

Human evaluation typically follows a pairwise comparison design because absolute Likert ratings suffer from calibration drift across annotators (one annotator's "4/5" is another's "3/5"). The standard pipeline: ``` function HUMAN_PAIRWISE_EVAL(model_A, model_B, prompts, annotators_per_item): results <- [] for prompt in prompts: response_A <- model_A.generate(prompt) response_B <- model_B.generate(prompt)

# randomize left/right position to cancel position bias if random_bit() == 1: (left, right) <- (response_A, response_B) else: (left, right) <- (response_B, response_A) votes <- [] for annotator in sample(annotators_per_item): vote <- annotator.compare(prompt, left, right) # "left" | "right" | "tie" votes.append(vote) results.append(votes) win_rate_A <- AGGREGATE(results, resolve_position(A, B)) kappa <- INTER_ANNOTATOR_AGREEMENT(results) return win_rate_A, confidence_interval(win_rate_A, n=len(prompts)), kappa ``` Two statistical safeguards are essential and are frequently skipped in informal evaluations: randomized position (models are shown left/right at random per item, since annotators exhibit a measurable bias toward whichever side they see first or second) and inter-annotator agreement (typically Cohen's kappa for two raters or Fleiss' kappa for more, computed as

where $p_o$ is observed agreement and $p_e$ is agreement expected by chance). A $\kappa$ below roughly 0.4 signals that the task itself is too subjective or the rubric too ambiguous for the resulting win-rate numbers to be trustworthy. Model skill from pairwise comparisons is typically aggregated with the Bradley–Terry model, which underlies public chat leaderboards. Each model $i$ is assigned a latent strength $\beta_i$, and the probability model $i$ beats model $j$ is

Given a set of pairwise outcomes (often from live human votes), maximum likelihood estimation recovers $\{\beta_i\}$, which are rescaled into an Elo-like rating. This handles the fact that not all models are compared against all others directly — Bradley–Terry propagates transitive information through the comparison graph.

LLM-as-judge evaluation

To scale beyond what human annotators can provide, a strong LLM is prompted to act as the judge, either scoring a single response on a rubric (absolute grading) or choosing between two candidate responses (pairwise grading). ``` function LLM_JUDGE_PAIRWISE(judge_model, prompt, response_A, response_B, rubric):

# Mitigate position bias: judge each ordering, keep only consistent verdicts verdict_1 <- judge_model.evaluate( instructions = rubric, question = prompt, answer_1 = response_A, answer_2 = response_B ) verdict_2 <- judge_model.evaluate( instructions = rubric, question = prompt, answer_1 = response_B, # swapped order answer_2 = response_A ) if verdict_1 == "1" and verdict_2 == "2": return "A wins" elif verdict_1 == "2" and verdict_2 == "1": return "B wins" else: return "inconsistent" # discard or mark as tie; do not silently pick one ``` Swapping the order and requiring consistency is the single most important debiasing step: LLM judges are empirically shown to favor whichever answer appears first (or second, depending on the judge) at a rate well above chance, and averaging over both orders — or discarding inconsistent verdicts — removes most of this artifact. A further correction addresses length bias (judges tend to prefer longer, more verbose answers independent of quality) and self-preference bias (a judge model rates outputs written in its own style more favorably, which is a serious concern when the same model family is used both to generate candidate answers and to judge them).

The evaluation pipeline as a whole

Computational cost. Automatic reference-based evaluation costs $O(n)$ model forward passes plus $O(n)$ cheap string/overlap computations, where $n$ is the number of test items — this is the fastest and cheapest method, dominated purely by inference cost. Human pairwise evaluation costs $O(n \cdot a)$ human-labor units where $a$ is annotators per item (typically 3–5 for reliable aggregation), and each unit is orders of magnitude more expensive (dollars, minutes) than a model forward pass, making large-scale human evaluation the bottleneck resource. LLM-as-judge evaluation costs $O(n)$ extra judge-model forward passes (or $O(n)$ with a constant factor of 2 if order-swapping is used for debiasing), which is far cheaper than human labor but more expensive than pure automatic scoring, and scales to large $n$ far more easily than human panels — this cost profile is precisely why LLM-as-judge has become the default for iterative model development, with human evaluation reserved for final validation.

Statistical correctness of aggregate scores. A benchmark accuracy is a sample mean $\hat{p} = \frac{1}{n}\sum_i \mathbb{1}[\text{correct}_i]$ estimating a true success probability $p$. By Hoeffding's inequality, for a benchmark of $n$ i.i.d. items,

which means that with only, say, $n=500$ items (common for many capability benchmarks), a 95% confidence interval half-width $\varepsilon$ satisfies $\varepsilon \approx \sqrt{\ln(2/0.05)/(2\cdot 500)} \approx 0.06$, i.e., roughly ±6 percentage points — a gap frequently larger than the score difference used to declare one model "better" in press releases. Correctness in benchmark reporting requires citing this uncertainty, not a bare point estimate, and increasing $n$ or reporting bootstrap confidence intervals when comparing close scores.

Why contamination breaks the argument. The i.i.d. sampling assumption underlying Hoeffding's bound requires that test items are drawn independently from the true target distribution and that the model has not seen them during training. If benchmark text (or close paraphrases) leaked into the pretraining or instruction-tuning corpus, the model's score on that item no longer reflects generalization — it reflects memorization, and the resulting $\hat p$ is a biased, upward-shifted estimator of true capability. Contamination is detectable via n-gram overlap between the test set and training corpus, via canary strings (unique tokens inserted into official test releases that should never appear in a well-curated training corpus), and via comparing performance on the public benchmark against a freshly constructed, unreleased "held-out" variant with the same task distribution but no possibility of memorization — a persistent score gap between the two is direct evidence of contamination.

Why Bradley–Terry is the right aggregation model for pairwise human/judge comparisons. The Bradley–Terry model is the maximum-entropy model consistent with the assumption that pairwise win probabilities depend only on a one-dimensional latent strength difference; maximizing the pairwise comparison log-likelihood

is a concave optimization problem (in $\beta$, since $\log\sigma$ is concave in its argument and the objective is a sum of such terms composed with a linear map), so gradient-based fitting converges to a global optimum and small perturbations in the observed match outcomes produce bounded perturbations in the fitted ratings — this is what makes leaderboard Elo-style scores stable and comparable across many models even though not every pair has been directly compared.

Honest tradeoffs. Automatic metrics are cheap and reproducible but blind to nuance the reference answer doesn't capture (a factually correct paraphrase scores zero on exact match). Human evaluation captures nuance but is slow, expensive, and only as reliable as its inter-annotator agreement — a low-κ study should not be trusted regardless of sample size. LLM-as-judge is fast and cheap at scale but is only as good as the correlation between judge verdicts and true human preference, which must itself be measured (typically by checking judge-human agreement on a held-out calibration set) before the judge is trusted for large-scale, unsupervised evaluation.

Consider comparing two chat models, Model A and Model B, on 200 prompts, using an LLM judge with position-swap debiasing. Suppose the true human-preferred win rate for A (established on a small, carefully controlled human study of 50 items) is 55%. We simulate what a naive (non-swapped) LLM-judge study would report versus a properly debiased one.

The naive study overstates Model A's advantage by 14 percentage points — enough to completely reverse a close product decision — because two biases stack in the same direction: position bias (whichever answer is shown first that day happened to be A in this batch) and verbosity bias (A's fine-tuning happened to produce longer answers). The swap-and-check protocol removes essentially all of the position-bias component by symmetry (each response is shown first exactly once, so any first-position advantage cancels across the two judgments), and it also surfaces the residual verbosity bias as a diagnosable quantity rather than hiding it: since 12% of verdicts flip depending on order, that inconsistency rate itself is a red flag suggesting the judge's decisions are unstable and a length-controlled rubric ("ignore length; reward per unit of substantive content") should be added, which — in practice — drives the residual error further toward zero. This example is deliberately non-symmetric (A is not simply a coin-flip against B) precisely because averaging effects are much easier to hide when the true answer is 50/50; here the systematic errors compound to nearly triple the apparent gap over the true one, illustrating why a bare LLM-judge win-rate, reported without swap-checking and without a calibration study against human labels, should not be trusted as a capability or alignment claim on its own.

Model providers use automatic benchmarks (MMLU-style knowledge tests, HumanEval-style code tests, GSM8K-style math tests) as the standard reporting suite in model release papers and cards, giving a comparable, cheap first-pass signal across releases. Public chat leaderboards built on crowd-sourced pairwise human comparisons, aggregated via Bradley–Terry into Elo-like ratings, have become a primary reference point for which conversational model "feels" best to real users, directly influencing product adoption decisions. Inside frontier labs, LLM-as-judge pipelines are embedded in the reinforcement-learning-from-AI-feedback loop itself — the same judge that evaluates a checkpoint's outputs can also generate the reward signal used to fine-tune the next checkpoint, tightly coupling evaluation methodology to Instruction Tuning and to the reward modeling stage of RLHF. Enterprise deployments of LLMs in regulated domains (healthcare, finance, legal) increasingly require factuality-focused evaluation suites tied to Hallucination & Factuality metrics before a model can be approved for production use, and safety teams use adversarially constructed evaluation sets — deliberately probing jailbreaks, harmful-content elicitation, and edge-case refusals — as the evaluation methodology underlying Safety & Red-Teaming, where the goal shifts from average-case accuracy to worst-case robustness.

Ask the tutor: How would you design a contamination-resistant benchmark refresh cycle for a model that is re-evaluated monthly?