Hallucination & Factuality

A hallucination is a generated statement that is fluent and often stated with high confidence, but that is false, fabricated, or unsupported by any available evidence. The term is borrowed from vision/perception but is somewhat misleading: the model is not "seeing things" — it is doing exactly what it was trained to do, sampling from a learned distribution over token sequences that maximizes likelihood given context, not one that maximizes correspondence to external reality. It is useful to distinguish two families. An intrinsic hallucination contradicts information explicitly present in the model's own input (e.g., a summarization model asserts a fact that contradicts the source document it was asked to summarize). An extrinsic hallucination adds information that is neither supported nor contradicted by the input — it may be true by coincidence, false, or simply unverifiable (e.g., a model asked to summarize a short news snippet adds a statistic that appears nowhere in any accessible source). A further useful split is between closed-domain hallucination, where a ground-truth context is given (as in RAG or summarization, and correctness is checked against that context), and open-domain hallucination, where the model draws on its parametric memory of world facts with no supplied context, and correctness must be checked against the world. Formally, an autoregressive language model defines a distribution over the next token $x_t$ given a prefix $x_{<t}$:

where $h_t$ is the model's hidden state after processing $x_{<t}$. Training minimizes cross-entropy against a corpus:

Nothing in this loss distinguishes "the corpus said something true" from "the corpus said something false" — the model is fit to the distribution of tokens seen, weighted by frequency, not to an oracle of truth. Facts that are rare, contested, post-training-cutoff, or simply absent from the corpus get no gradient signal pulling the model toward a correct answer; instead, the model's best strategy at inference time is to produce the most probable-looking continuation, which for named entities, dates, and citations tends to have the surface form of a fact even when its content is invented.

Understanding hallucination requires tracing where in the pipeline unfaithfulness gets introduced, and then tracing where a countermeasure can intercept it. Four distinct mechanisms compound:

1. Knowledge gaps at training time. If a fact appears rarely, inconsistently, or not at all in pretraining data, the model has no strong parametric representation of it. When queried, it interpolates from related patterns (similar names, similar formats) and produces a plausible-looking but wrong answer, because the softmax always outputs some high-probability token — it has no "abstain" option baked in by default.

2. Exposure bias in autoregressive decoding. During training the model conditions on ground-truth prefixes (teacher forcing). At inference it conditions on its own previously generated (possibly already wrong) tokens. One hallucinated token early in a citation can commit the model to continuing the fabrication, because subsequent tokens are chosen to be probable given the corrupted prefix, not corrected against it.

3. Sampling stochasticity. Decoding strategies like temperature sampling or nucleus sampling intentionally inject randomness for diversity. This raises the probability of selecting a lower-probability continuation that happens to be false, especially in the "long tail" of rare entities, exact numbers, or precise dates — precisely where factual precision matters most and where the model's distribution is flattest (least peaked, most uncertain).

4. Reward misspecification during alignment. In RLHF or DPO fine-tuning, human raters (or a reward model standing in for them) tend to rate confident, complete-sounding answers higher than hedged or abstaining ones, even when the confident answer is wrong. This creates a training signal that actively rewards fluent fabrication over honest uncertainty — a well-documented failure mode sometimes called sycophancy-adjacent reward hacking. The countermeasures intercept these mechanisms at different pipeline stages. The pseudocode below sketches a realistic factuality-checking pipeline used to detect hallucinations post-hoc (as in FActScore-style evaluation and in production RAG systems with citation verification): ``` function CHECK_FACTUALITY(response, retrieved_docs):

# Step 1: decompose the response into atomic, checkable claims claims = DECOMPOSE_INTO_ATOMIC_FACTS(response)

# e.g. "Marie Curie won two Nobel Prizes, in Physics (1903)

# and Chemistry (1911)" ->

# ["Marie Curie won a Nobel Prize in Physics in 1903",

# "Marie Curie won a Nobel Prize in Chemistry in 1911"] results = [] for claim in claims:

# Step 2: retrieve evidence for this specific claim evidence = RETRIEVE(claim, corpus_or_docs=retrieved_docs, k=5) if evidence is EMPTY: verdict = "UNVERIFIABLE" else:

# Step 3: use an entailment / NLI model to check support score = ENTAILMENT_SCORE(premise=evidence, hypothesis=claim) if score > SUPPORT_THRESHOLD: verdict = "SUPPORTED" elif score < CONTRADICT_THRESHOLD: verdict = "CONTRADICTED" else: verdict = "UNVERIFIABLE" results.append((claim, verdict, evidence))

# Step 4: aggregate into a factuality score supported = count(v == "SUPPORTED" for (_, v, _) in results) factuality_score = supported / max(len(claims), 1) return factuality_score, results

function RETRIEVAL_AUGMENTED_GENERATION_WITH_ABSTENTION(query, corpus): docs = RETRIEVE(query, corpus, k=8) if RELEVANCE_SCORE(docs, query) < MIN_RELEVANCE: return "I don't have reliable information to answer this." draft = GENERATE(prompt=CONSTRUCT_PROMPT(query, docs)) score, per_claim = CHECK_FACTUALITY(draft, docs) if score < ACCEPTANCE_THRESHOLD:

# attempt one grounded rewrite, forcing citation of doc spans draft = REGENERATE_WITH_FORCED_CITATIONS(query, docs, per_claim) score, per_claim = CHECK_FACTUALITY(draft, docs) if score < ACCEPTANCE_THRESHOLD: return ABSTAIN_OR_HEDGE(draft, per_claim) return ANNOTATE_WITH_CITATIONS(draft, per_claim) ``` Each stage targets a specific mechanism: `RETRIEVE` supplies grounding text so the model is not relying solely on possibly-absent parametric knowledge (attacking mechanism 1); `CHECK_FACTUALITY` catches drift introduced by exposure bias and sampling (mechanisms 2–3) after generation, rather than trying to prevent it during decoding; `RELEVANCE_SCORE` gating and `ABSTAIN_OR_HEDGE` directly counteract the reward-hacking incentive to always produce a confident-sounding answer (mechanism 4), by making abstention a legitimate, scoreable output rather than a failure.

Detection cost. Let a response contain $m$ atomic claims, and let retrieval over a corpus of size $N$ cost $O(\log N)$ per query using an approximate nearest-neighbor index (e.g., HNSW), or $O(N)$ for brute-force dense retrieval. Claim decomposition costs one additional LLM call of length proportional to the response, $O(L)$ tokens where $L$ is response length. Then the full `CHECK_FACTUALITY` pipeline costs:

where $C_{\text{NLI}}$ is the cost of one entailment-scoring call (typically a smaller cross-encoder, $O(\ell^2)$ in sequence length $\ell$ for a Transformer, or $O(\ell)$ for a distilled scorer). Since $m$ (atomic facts per response) is typically small (5–30) and each NLI call is cheap relative to generation, factuality checking is usually a small constant-factor overhead on top of generation cost, not an asymptotic blow-up — but it does multiply latency and API cost linearly in $m$, which matters at production scale.

Why retrieval reduces hallucination (the mechanism, formally). Without retrieval, the model must answer using only its parametric estimate $p_\theta(\text{answer} \mid \text{query})$, which is a marginal over everything seen in training — for rare facts this distribution is close to uniform over plausible-looking completions, i.e., high entropy, i.e., unreliable. With retrieval, the model instead computes $p_\theta(\text{answer} \mid \text{query}, \text{evidence})$, conditioning on evidence text that (if retrieval succeeds) sharply concentrates the token distribution around the supported answer — the conditioning reduces the conditional entropy of the next-token distribution when the evidence is relevant and correct. This is the same principle as Retrieval-Augmented Generation: retrieval does not change the model's parameters, it changes what conditional distribution is being sampled from at inference time. The correctness of this reduction is contingent, not guaranteed: it holds only if (a) retrieval returns relevant, correct evidence, and (b) the model is faithful to that evidence rather than overriding it with parametric priors (a known failure called context-parametric conflict, where a confidently wrong prior in the weights outvotes correct retrieved context).

Why detection (NLI-based checking) is a sound but incomplete filter. An entailment classifier trained to predict whether a premise supports a hypothesis provides a calibrated proxy for factual support, assuming its own training data is representative of the claim/evidence pairs encountered at test time. It is sound in the sense that high-confidence contradictions reliably flag real hallucinations (few false negatives for blatant fabrications). It is incomplete because: (1) it cannot verify claims for which no evidence was retrieved (labeled "unverifiable," which is not the same as "false" — a precision/recall gap the pipeline must live with); (2) it inherits any bias or blind spot of the NLI model itself; (3) atomic-fact decomposition is itself an LLM call and can silently drop or distort claims.

Honest tradeoffs. Every mitigation trades some quantity for factuality:

• Retrieval grounding trades latency and infrastructure complexity (index maintenance, retrieval quality) for accuracy, and still fails when retrieval is wrong or the corpus lacks the needed document.
• Abstention trades helpfulness (coverage — the fraction of queries answered at all) for precision (fraction of answered queries that are correct). A model that abstains on everything has zero hallucination and zero utility.
• Lower decoding temperature reduces variance-driven hallucination but also reduces diversity and can increase repetition or degenerate output.
• Self-consistency (sampling $k$ times and taking a majority-vote answer) improves precision on questions with a single stable answer but multiplies inference cost by $k$ and does not help when the model is systematically wrong (all $k$ samples share the same misconception), since majority voting over correlated errors converges to the wrong answer, not the right one.

Consider a RAG system asked: "When did Alan Turing publish 'Computing Machinery and Intelligence,' and what test did it introduce?" Suppose the retrieval corpus contains one relevant document (a Wikipedia-style summary) but the retriever, under time pressure, returns three passages, one of which is only tangentially related.

This trace illustrates the key dynamic: retrieval alone did not prevent hallucination (Doc A/B contained no "70% by 2014" claim, yet the model produced one anyway — a classic case of the parametric prior injecting plausible-sounding but unsupported detail into an otherwise grounded answer). Only the post-hoc verification-and-regeneration loop caught and removed it. Note also that claim (c) is not necessarily false — Turing's actual 1950 paper does contain a similar prediction — but the pipeline correctly labels it "unverifiable" given the retrieved evidence, which is the honest and reproducible verdict a system without a perfect oracle can give. This distinction (unverifiable ≠ false) is one of the most commonly mishandled subtleties in factuality tooling.

The study of factuality in generated text predates modern LLMs, growing out of work on abstractive summarization, natural language inference, and calibration in machine learning, before being reframed for autoregressive foundation models.

Production RAG systems in search engines, enterprise document assistants, and customer-support bots all depend on citation verification pipelines like the one above to avoid presenting fabricated content as sourced fact. Medical and legal LLM applications require explicit abstention thresholds, because an incorrect but confident answer in these domains carries real-world harm — several deployed systems refuse to answer, or route to a human, when factuality scores fall below a strict threshold. Search engines that summarize web results ("AI overviews") rely on the retrieval-relevance gate to avoid summarizing irrelevant or low-quality pages as if they were authoritative. Coding assistants face a specialized version of the problem — "API hallucination," where a model invents plausible-looking but nonexistent library functions — addressed by static-analysis-based post-hoc verification rather than NLI. Academic and scientific writing tools use citation-verification pipelines nearly identical to the pseudocode above to catch fabricated references before they reach a manuscript. All of these systems foreshadow the broader theme of Evaluation of LLMs (how do we benchmark factuality at scale) and Safety & Red-Teaming (how adversaries deliberately construct prompts that maximize hallucination rates, such as asking for citations on intentionally obscure or fictitious topics).

Ask the tutor: How would you design an evaluation benchmark that separately measures a model's hallucination rate on well-represented facts versus rare, long-tail facts?