Large Language Models

Large Language Models (LLMs) are neural networks—predominantly based on the Transformer architecture—trained on vast text corpora to model the probability distribution of token sequences, enabling them to generate and understand natural language. Key result: by minimizing next-token prediction loss over billions of tokens, a single model acquires broad linguistic, factual, and reasoning capabilities that scale predictably with model size, data, and compute. Applies to: text generation, translation, question answering, code synthesis, summarization. Who/when: rooted in n-gram models (1950s–1980s), revolutionized by the Transformer (Vaswani et al., 2017) and scaled by GPT/BERT-class models (2018–present).

A Large Language Model is a parameterized probability distribution $p_{θ} (x_{1}, \dots, x_{n})$ over sequences of discrete tokens, factorized autoregressively and realized by a deep neural network with billions of parameters $θ$ . It matters because a single training objective—predicting the next token—turns out to be a powerful surrogate for learning grammar, world knowledge, and rudimentary reasoning, all without task-specific supervision. The surprising fact is emergence: capabilities like multi-step arithmetic or in-context learning appear abruptly only beyond certain scale thresholds, absent in smaller models trained identically. LLMs are the first machine learning systems to exhibit broad, general-purpose competence from a uniform self-supervised objective. They have reshaped natural language processing from a collection of bespoke pipelines into a paradigm of one large pretrained model adapted to many tasks.

The Core Insight

The entire enterprise reduces to a deceptively simple idea: language modeling is compression of structured knowledge into a conditional probability. If you can reliably predict the next word given everything before it, you must have implicitly modeled syntax, semantics, facts, and inference patterns. The Transformer makes this tractable by replacing recurrence with self-attention, allowing every token to directly attend to every other token in parallel, and scale makes it powerful.

Formal Statement

Autoregressive Language Model

A language model defines a joint probability over a token sequence $x = (x_{1}, \dots, x_{n})$ , where each $x_{i} \in V$ (a finite vocabulary). By the chain rule of probability:

p_{θ} (x_{1}, \dots, x_{n}) = i = 1 \prod n p_{θ} (x_{i} ∣ x_{1}, \dots, x_{i - 1})

⟳Apply the chain rule of conditional probability iteratively; no independence assumption is made, only an ordering.

Here $θ$ denotes the model's learnable parameters, $V$ is the vocabulary (typically $∣ V ∣ \approx 30 k - 100 k$ subword tokens), and $p_{θ} (x_{i} ∣ x_{< i})$ is a categorical distribution produced by a softmax over $V$ .

Training Objective: Cross-Entropy Loss

The model is trained by minimizing the average negative log-likelihood (cross-entropy) over a corpus $D$ :

L (θ) = - \frac{1}{N} i = 1 \sum N lo g p_{θ} (x_{i} ∣ x_{< i})

⟳Take the negative logarithm of the likelihood [1] and average over

N

tokens; minimizing this maximizes the data likelihood.

where $N$ is the total number of training tokens. The exponentiated per-token loss defines perplexity:

PPL (θ) = exp (- \frac{1}{N} i = 1 \sum N lo g p_{θ} (x_{i} ∣ x_{< i}))

⟳Perplexity is

exp

of cross-entropy; it equals the effective average branching factor—the number of equally likely choices the model faces per token.

Self-Attention

The core computation. Given input embeddings packed as rows of $X \in R^{n \times d}$ , we compute queries, keys, and values via learned projections $W_{Q}, W_{K}, W_{V} \in R^{d \times d_{k}}$ :

Q = X W_{Q}, K = X W_{K}, V = X W_{V}

⟳These are linear maps; each row of

Q, K, V

corresponds to one token's query/key/value vector.

The attention output is a weighted average of values, weighted by query–key compatibility:

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

⟳

Q K^{⊤}

gives pairwise dot-product similarities

(n \times n)

; dividing by

d_{k}

controls variance growth; softmax normalizes each row to a probability distribution over tokens.

Universal Approximation of Sequence-to-Sequence Maps

Transformers with sufficient depth and width are universal approximators of continuous, permutation-equivariant sequence-to-sequence functions on compact domains (Yun et al., 2020). Adding positional encodings removes permutation equivariance, enabling approximation of arbitrary continuous sequence functions to any desired accuracy $ϵ$ .

The theorem guarantees representational capacity, not learnability. A function being approximable says nothing about whether gradient descent on finite data will find the right parameters. Most of an LLM's practical capability comes from data and optimization, not from existence proofs.

Why It Holds / Derivation

Why next-token prediction yields general capability — a sketch.

Factorization is exact. Equation [1] involves no approximation: any distribution over sequences factors via the chain rule. The model's only job is to approximate each conditional $p_{θ} (x_{i} ∣ x_{< i})$ .

Cross-entropy is a proper scoring rule. Minimizing [2] is equivalent to minimizing the KL divergence $D_{KL} (p_{data} ∥ p_{θ})$ up to a constant (the data entropy). Thus the global optimum is $p_{θ} = p_{data}$ .

L (θ) = H (p_{data}) + D_{KL} (p_{data} ∥ p_{θ})

⟳Expand cross-entropy

H (p, q) = - \sum p lo g q

H (p) + D_{KL} (p ∥ q)

; the first term is independent of

θ

Information bottleneck forces abstraction. To predict well across diverse contexts with finite parameters, the model cannot memorize every sequence; it must compress regularities—grammar, coreference, factual associations—into shared representations.

Attention enables long-range structure. Equation [5] lets any position aggregate information from all others in $O (1)$ sequential steps (vs. $O (n)$ for RNNs), so dependencies across long contexts are learnable without vanishing gradients.

Scale closes the gap. Empirically, increasing $θ$ , data, and compute drives $L$ smoothly toward $H (p_{data})$ , and the resulting better conditional models manifest as qualitatively new behaviors. $■$

Transformer decoder block and the autoregressive prediction loop

Building Intuition

Think of an LLM as an extraordinarily well-read autocomplete. Imagine someone who has read most of the internet and, when shown any partial text, can guess what comes next based on patterns they've absorbed. The "masked" attention enforces causality: when predicting token $i$ , the model may only look at tokens $1$ through $i - 1$ (the triangular mask in [5] sets future attention weights to $- \infty$ before softmax).

The key conceptual leap is that prediction requires understanding. To finish "The capital of France is ___" correctly, the model must encode a fact. To finish " $2 + 2 =$ ___" it must encode arithmetic. The richer and more varied the contexts it learns to predict, the broader the implicit knowledge it must store. Attention is the mechanism that lets the model dynamically decide which earlier words matter for the current prediction—like a reader who, encountering "it," instantly looks back to find the noun "it" refers to.

Claude Shannon | 1916–2001

Though not an LLM researcher, Shannon laid the foundation in his 1948 A Mathematical Theory of Communication and his 1951 paper Prediction and Entropy of Printed English. He framed language as a stochastic process, estimated the entropy of English at roughly 1 bit per character, and conducted experiments where humans guessed the next letter—the conceptual ancestor of next-token prediction. Every modern LLM minimizes precisely the cross-entropy Shannon defined, and perplexity [3] is the direct operationalization of his entropy of language.

Worked Examples

Computing Perplexity of a Tiny Model

Given: A model assigns the following probabilities to a 3-token sequence "the cat sat":

$p ("the") = 0.5$
$p ("cat" ∣ "the") = 0.1$
$p ("sat" ∣ "the cat") = 0.2$

Find: the sequence log-likelihood and perplexity. Steps:

Sum of log-probabilities (natural log):

lo g 0.5 + lo g 0.1 + lo g 0.2 = - 0.693 - 2.303 - 1.609 = - 4.605

Average negative log-likelihood (cross-entropy), $N = 3$ :

L = - \frac{1}{3} (- 4.605) = 1.535

nats/token

Perplexity via [3]:

PPL = e^{1.535} = 4.64

Answer:

PPL \approx 4.64 tokens

— the model is, on average, as uncertain as if choosing uniformly among ~4.6 options per token.

Attention Weight for Two Tokens

Given: Two tokens with query/key dimension

d_{k} = 4

. For predicting the current token, its query is

q = (1, 0, 1, 0)

. Two candidate keys:

k_{1} = (1, 0, 1, 0)

(a matching past token) and

k_{2} = (0, 1, 0, 1)

(an unrelated one). Find: the attention weights assigned to value 1 vs. value 2. Steps:

Scaled dot products ( $d_{k} = 2$ ):

s_{1} = \frac{q \cdot k _{1}}{2} = \frac{1 + 0 + 1 + 0}{2} = 1.0

s_{2} = \frac{q \cdot k _{2}}{2} = \frac{0 + 0 + 0 + 0}{2} = 0.0

Softmax over $(s_{1}, s_{2}) = (1.0, 0.0)$ :

α_{1} = \frac{e ^{1.0}}{e ^{1.0} + e ^{0.0}} = \frac{2.718}{3.718} = 0.731

α_{2} = \frac{1}{3.718} = 0.269

Answer:

α_{1} = 0.73, α_{2} = 0.27

— the model places 73% of its attention on the semantically matching token, illustrating how dot-product similarity routes information.

Common Misconceptions

"LLMs look up answers in a database." They do not. After training, the corpus is discarded; all "knowledge" is compressed into the weights

θ

as continuous parameters. The model regenerates text statistically, which is precisely why it can also hallucinate—produce fluent but false statements—since there is no factual lookup, only probability maximization. "Bigger context window = better memory." The context window is the number of tokens the model can attend to at once. Beyond it, the model has no access to earlier text. Moreover, attention is

O (n^{2})

in context length

n

, so doubling context quadruples attention cost—a hard practical limit. Reversal failure (the "reversal curse"). A model trained on "Tom Cruise's mother is Mary Lee Pfeiffer" often cannot answer "Who is Mary Lee Pfeiffer's son?" This exposes that LLMs learn directional token associations, not symmetric logical relations. It refutes the naive assumption that learning a fact implies learning its logically equivalent inverse—next-token prediction optimizes [2] in one direction only.

Connections

LLMs sit at the intersection of several fields. The objective [2] is grounded in Information Theory (cross-entropy, perplexity = Shannon entropy). The optimization relies on Stochastic Gradient Descent and Backpropagation to compute $\nabla_{θ} L$ . The architecture builds on Neural Networks and specifically the Transformer Architecture and its Attention Mechanism. The probabilistic framing connects to Maximum Likelihood Estimation and Markov Chains (n-gram models being the finite-order special case). Sampling generation strategies relate to Monte Carlo Methods.

Real-World Applications

ChatGPT / GPT-4 (OpenAI) — conversational assistants for drafting, tutoring, and coding, built on instruction-tuned and RLHF-aligned LLMs.
GitHub Copilot — code completion and synthesis trained on public repositories, accelerating software development.
Google Translate (neural) — modern machine translation uses Transformer encoder–decoder LLMs, replacing earlier statistical phrase-based systems.
AlphaCode / competitive programming — LLMs generating and ranking candidate solutions to algorithmic problems.
Bloomberg GPT — a domain-specialized LLM trained on financial text for sentiment, classification, and reporting.
Med-PaLM (Google) — medical question answering reaching expert-level accuracy on licensing-style exams.
Retrieval-Augmented Generation (RAG) — combining LLMs with vector search to ground answers in up-to-date documents, mitigating hallucination.

Historical Development

1948 — Claude Shannon formalizes language as a stochastic source; introduces entropy.
1980s–1990s — n-gram statistical language models dominate speech recognition and translation (IBM models).
2003 — Bengio et al. introduce the neural probabilistic language model, learning distributed word representations.
2013 — Mikolov et al. release word2vec, popularizing dense word embeddings.
2014–2015 — Sequence-to-sequence learning (Sutskever et al.) and the attention mechanism (Bahdanau et al.) for translation.
2017 — Vaswani et al. publish Attention Is All You Need, introducing the Transformer—dispensing with recurrence entirely.
2018 — BERT (Google, bidirectional) and GPT (OpenAI, autoregressive) launch the pretrain-then-fine-tune era.
2020 — GPT-3 (175B parameters) demonstrates in-context/few-shot learning; Kaplan et al. publish scaling laws.
2022–2023 — InstructGPT and ChatGPT add RLHF; GPT-4, PaLM, LLaMA push capability and accessibility; emergent abilities and alignment become central topics.

Subtleties & Limits

Scaling laws and their ceiling. Kaplan/Hoffmann scaling laws empirically predict loss as a power law in parameters

N

and data

D

L (N, D) \approx L_{\infty} + (\frac{N _{c}}{N})^{α_{N}} + (\frac{D _{c}}{D})^{α_{D}}

⟳Fit log-loss versus log-compute on observed runs;

α_{N}, α_{D} \approx 0.05 - 0.5

are empirical exponents, and

L_{\infty}

is the irreducible entropy floor.

Open questions remain profound: Why do these power laws hold? What is the irreducible loss $L_{\infty}$ for natural language, and how close can we get? Emergence is poorly understood—are abrupt capability jumps real phase transitions or artifacts of discontinuous metrics? Mechanistic interpretability—reverse-engineering what circuits inside $θ$ compute—is in its infancy. Alignment: ensuring outputs match human intent and values is unsolved, as is hallucination: there is no principled guarantee a probability-maximizing model states truths. Finally, whether next-token prediction can ever yield genuine reasoning versus sophisticated pattern interpolation is fiercely debated.