Scaling Laws for Neural Language Models

Scaling laws for neural language models are empirical relationships that describe how model performance (measured by cross-entropy loss) changes predictably as a function of model size (parameter count), dataset size, and compute budget. They matter because they allow practitioners to forecast training outcomes before committing to expensive runs, enabling rational resource allocation at the frontier of AI. The counterintuitive fact: performance follows smooth power laws across many orders of magnitude, meaning a model trained on 10× more compute is predictably better — there are no mysterious thresholds, just math.

The core insight

If you plot validation loss against model size, dataset tokens, or compute on a log-log scale, you get a straight line — a power law. This means that throwing more compute, data, or parameters at a language model yields returns that are consistent and foreseeable, not random. The "aha moment" is realizing that language modeling is not a chaotic optimization problem with unpredictable results: it is a systematic engineering discipline where you can extrapolate from small experiments to predict the behavior of models that haven't been built yet. This transforms AI research from alchemy into something closer to physics.

Common trapCommon Misconception

The most common mistake is conflating the Kaplan et al. (2020) scaling laws with the Chinchilla (2022) scaling laws and assuming they give the same prescription. Kaplan et al. found that, for a fixed compute budget, you should scale model size much faster than data — leading to the practice of training huge models on relatively few tokens. Hoffmann et al. (Chinchilla) corrected a flaw in this analysis (Kaplan held learning rate schedule fixed across runs, biasing results toward large models) and showed that model size and training tokens should scale equally. The correct mental model: Chinchilla-optimal training means roughly 20 tokens per parameter, not 1–5 as implied by Kaplan. GPT-3 (175B params, 300B tokens) is severely undertrained by Chinchilla standards; Llama-2 7B (trained on 2T tokens) is Chinchilla-overcomplete and much more efficient at inference time.

How It Works

The Loss-Compute Power Law

The central empirical finding is that validation cross-entropy loss $L$ falls as a power law in each of three variables independently when the other two are held non-limiting:

\quad L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}

Explanation: Loss decreases as a power law in parameter count $N$ , with exponent $\alpha_N \approx 0.076$ (Kaplan) — every 10× increase in parameters reduces loss by a fixed multiplicative factor. Here $N$ is the number of non-embedding parameters, $N_c$ is a constant fitted from data, and $\alpha_N$ is the scaling exponent. Similarly for dataset size $D$ (tokens):

\quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}

Explanation: Loss decreases as a power law in training tokens $D$ with exponent $\alpha_D \approx 0.095$ , slightly steeper than the parameter exponent, suggesting data is marginally more valuable than parameters per unit cost.

The Unified Compute Law

When both $N$ and $D$ are free but compute $C \approx 6ND$ (FLOPs, ignoring attention cost) is fixed, Kaplan et al. derived:

\quad L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}, \quad \alpha_C = \frac{\alpha_N \alpha_D}{\alpha_N + \alpha_D} \approx 0.050

Explanation: The compute-optimal loss exponent is the harmonic-mean combination of the individual exponents, capturing the tradeoff between spending FLOPs on more parameters versus more data.

The Chinchilla Optimal Allocation

Hoffmann et al. reformulated the problem by fitting a more general loss surface:

\quad \hat{L}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

Explanation: The irreducible entropy $E$ represents the true entropy of natural language; the $A/N^{\alpha}$ term captures model underfitting from finite capacity; and $B/D^{\beta}$ captures underfitting from finite data — minimizing over $N$ and $D$ subject to $C = 6ND$ gives the compute-optimal frontier. Minimizing $\hat{L}$ subject to the compute constraint $C = 6ND$ via Lagrange multipliers yields the compute-optimal model size:

\quad N^*(C) = G \cdot C^{a}, \quad D^*(C) = G^{-1} \cdot C^{b}

Explanation: Both optimal model size and optimal token count scale as power laws in compute with exponents $a = b \approx 0.50$ (Chinchilla), meaning you should double both simultaneously — the key departure from Kaplan's exponent of $a \approx 0.73$ . The fitted Chinchilla constants give the memorable rule: $D^* \approx 20 \cdot N^*$ tokens per parameter at the compute-optimal frontier.

researcher · Jared Kaplan | b. 1988 | Power-law scaling laws for language models

Kaplan is a theoretical physicist by training (PhD Johns Hopkins, cosmology), who joined OpenAI and applied physicist instincts — seeking universal laws — to neural language models. His 2020 paper "Scaling Laws for Neural Language Models" with McCandlish, Henighan, et al. established the foundational power-law framework that directly motivated the design of GPT-3. A surprising fact: Kaplan's physics background led the team to deliberately model loss curves the same way cosmologists model the CMB power spectrum — fitting smooth functional forms to noisy empirical data across orders of magnitude.

researcher · Jordan Hoffmann | b. 1993 | Chinchilla optimal compute allocation

Hoffmann is a research scientist at Google DeepMind who led the "Training Compute-Optimal Large Language Models" (Chinchilla) paper in 2022 with Sebastian Borgeaud and colleagues. By carefully controlling for learning rate schedule across all model sizes — a confound Kaplan's study did not address — Hoffmann's team showed the Pareto frontier lay at equal scaling of parameters and data, not heavily parameter-biased. The Chinchilla model (70B parameters, 1.4T tokens) outperformed Gopher (280B params) despite being 4× smaller, validating the theory and shifting the entire field's training recipes.

Figure

Scaling Laws: Loss vs. Compute Frontier

Figure

Parameter-Data Tradeoff at Fixed Compute

Intuition

The Mental Model

Imagine you have a fixed budget to build a reference library. You can spend it on buying more encyclopedias (parameters — more capacity to memorize patterns) or hiring more editors to read and process more source documents (training tokens — more diverse examples). Kaplan's original advice was "buy as many encyclopedias as possible, even if your editors barely skim each book." Chinchilla's correction said "actually, you need your editors to read each book thoroughly — a smaller library of well-read encyclopedias beats a vast library where each book was only glanced at." The power-law structure means the relationship between budget and quality is not arbitrary: it's as regular as compound interest. Spending 2× the budget gets you a fixed, predictable improvement in library quality — not sometimes better, not randomly better, but reliably and measurably better by the same multiplicative factor each time.

Historical Development

Origins · 1 minhow this was discovered

History

2001

Neural scaling behavior first noted informally by researchers studying perceptron generalization bounds, but without empirical validation at scale.

2017

Vaswani et al. introduce the Transformer architecture ("Attention Is All You Need"), creating the substrate on which scaling laws would be discovered — attention's compute scales cleanly with model size.

2019

Hestness et al. ("Deep Learning Scaling is Predictable, Empirically") publish the first systematic study of power-law scaling in NLP and vision, covering 6 orders of magnitude of compute.

2020

Kaplan, McCandlish, Henighan et al. (OpenAI) publish "Scaling Laws for Neural Language Models," establishing the canonical

L \sim N^{-\alpha}

framework and directly motivating GPT-3's design at 175B parameters.

2020

Brown et al. release GPT-3 (175B), a direct implementation of Kaplan scaling prescriptions — trained on 300B tokens, heavily parameter-biased relative to what Chinchilla would later recommend.

2022

Hoffmann, Borgeaud et al. (DeepMind) publish "Training Compute-Optimal Large Language Models" (Chinchilla paper), correcting Kaplan's learning-rate confound and establishing the equal-scaling law and 20 tokens/parameter rule.

2022

Wei et al. document "emergent abilities" — capabilities appearing discontinuously at specific scale thresholds — raising questions about whether power laws fully describe all aspects of scaling.

2022

Tay et al. show scaling laws depend on model architecture and tokenization, cautioning against universalizing constants across model families.

2023

Muennighoff et al. extend Chinchilla analysis to "over-trained" (inference-optimal) regimes, showing training beyond compute-optimal tokens is rational when inference cost dominates — motivating Llama-2 7B's 2T-token training.

2024

Bi et al. (DeepSeek) and Grattafiori et al. (Llama-3) publish internal scaling law fits, showing frontier labs now routinely run dozens of small "scaling law experiments" before committing to large training runs.

Why It Matters

Real-World Impact

GPT-3 (OpenAI, 2020)Designed using Kaplan scaling laws — 175B parameters chosen because the laws predicted it would cross task-relevant capability thresholds; training on 300B tokens reflected Kaplan's parameter-heavy prescription, later shown to be suboptimal.

Chinchilla 70B (DeepMind, 2022)Direct experimental validation of compute-optimal scaling — 70B parameters × 1.4T tokens outperformed Gopher (280B, 300B tokens) and GPT-3 on most benchmarks, demonstrating that the 20 tokens/param rule yields superior loss at identical compute cost.

Llama-2 7B (Meta, 2023)Trained on 2T tokens (286 tokens/param, far exceeding Chinchilla-optimal 140B tokens), deliberately exploiting the inference-optimal regime — scaling laws tell you training past optimal costs relatively little in loss but dramatically reduces inference-time parameter count, saving billions in serving costs.

GPT-4 (OpenAI, 2023)OpenAI used internal scaling law experiments at sub-trillion-FLOP scales to predict GPT-4's performance before training, reportedly achieving accurate capability forecasts — one of the first public demonstrations of reliable scaling-law extrapolation for practical capability planning.

Gemini Ultra (Google DeepMind, 2023)Scaling law analysis informed the decision to use mixture-of-experts (MoE) architecture variants, since active-parameter scaling laws differ from dense-parameter laws, requiring bespoke fitting.

Frontier Lab Resource AllocationEvery major AI lab (OpenAI, Anthropic, DeepMind, Meta FAIR) now runs "IsoFLOP" sweeps — training many small models at varied N, D ratios for a fixed compute budget — to fit private scaling law constants before committing to runs exceeding $10M in compute.

LLM-as-Judge and Benchmark SelectionScaling laws predict when a model crosses task thresholds, allowing researchers to select benchmarks that will discriminate between model generations and avoid "ceiling effects" (benchmarks where all large models score ~100%).

Scaling Laws for Neural Language Models

How It Works

The Loss-Compute Power Law

The Unified Compute Law

The Chinchilla Optimal Allocation

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also