Emergent Abilities & Scaling Laws

A scaling law is an empirical relationship, of power-law form, between a language model's test loss $L$ and the resources used to produce it: the number of trainable parameters $N$, the size of the training dataset $D$ (in tokens), and the total training compute $C$ (in floating-point operations). The canonical form, per parameter or per fixed pair with the other held effectively unconstrained, is

where $N_c, D_c, C_c$ are constants fit from data and the exponents $\alpha_N, \alpha_D, \alpha_C$ are small positive numbers (empirically around $0.05$–$0.4$ depending on the study and architecture family). The loss is cross-entropy on held-out text, and it decreases as any one resource grows while the others are not the bottleneck. Crucially the relationship is linear in log-log space: $\log L$ is approximately affine in $\log N$. This is what makes the law useful — a line fit to small-scale runs extrapolates to scales orders of magnitude larger.

Compute-optimal scaling asks a sharper question: given a fixed compute budget $C$, how should you split it between $N$ and $D$ to minimize loss, rather than simply asking what loss a given $N$ or $D$ achieves in isolation? Since compute for training a dense Transformer is well approximated by $C \approx 6ND$ (roughly, 2 FLOPs per parameter per forward pass, times 3 for forward + backward, times $D$ tokens seen), there is a genuine trade-off: doubling $N$ at fixed $C$ halves the usable $D$. The optimal split is itself a power law, $N_{\text{opt}} \propto C^{a}$ and $D_{\text{opt}} \propto C^{b}$ with $a + b \approx 1$, and the Chinchilla study found $a \approx b \approx 0.5$ — parameters and data should grow at roughly the same rate, contrary to earlier practice (e.g., GPT-3) which scaled parameters much faster than data.

Emergent abilities are a separate but related phenomenon. They refer to capabilities — such as multi-step arithmetic, unit conversion in word problems, or successful chain-of-thought reasoning — that are not present (near-random performance) in a family of models below some scale threshold, and then appear with a sharp, seemingly discontinuous rise above that threshold, on a specific benchmark's accuracy metric. This is qualitatively different from a smooth power-law improvement in loss; the same underlying smooth improvement in log-likelihood of the correct answer can look like an on/off switch when the evaluation metric is something binary like exact-match accuracy, because exact-match requires every token in a multi-step answer to be correct simultaneously.

Step 1: Fit the scaling law from small-scale runs. Train a family of models at varying $N$ (holding $D$ large enough not to bottleneck) and record final test loss. Similarly vary $D$ at large fixed $N$. Fit a line in log-log space by least squares.

Step 2: Determine the compute-optimal frontier (IsoFLOP method). For a range of fixed compute budgets $C_1, \dots, C_k$, train many models with different $(N, D)$ pairs satisfying $C \approx 6ND$, and record the loss for each. For each budget, the $(N, D)$ pair achieving minimum loss traces out the optimal allocation curve. Fit power laws $N_{\text{opt}}(C)$ and $D_{\text{opt}}(C)$ to these minima.

Step 3: Extrapolate to the target budget. Given a real budget $C^*$ (e.g., a planned production training run), plug into the fitted $N_{\text{opt}}(C^*)$ and $D_{\text{opt}}(C^*)$ to decide model size and dataset size before spending the money on the full run.

Step 4: Evaluate for emergence separately. Scaling laws describe loss. To check whether a task is emergent, evaluate the same model family on downstream benchmarks, plotting task accuracy (or another metric) against scale. Compare with a smoother metric (log-likelihood of the correct token sequence, or partial-credit scoring) on the same task — this reveals whether the jump is intrinsic or metric-induced. ```

# Pseudocode: fitting a scaling law and finding the compute-optimal frontier function FIT_POWER_LAW(x_values, y_values):

# y = A x^(-alpha) <=> log(y) = log(A) - alpha log(x) log_x = [log(x) for x in x_values] log_y = [log(y) for y in y_values] (slope, intercept) = LEAST_SQUARES_LINE(log_x, log_y) alpha = -slope A = exp(intercept) return (A, alpha) function ISO_FLOP_SEARCH(compute_budgets, candidate_ratios):

# candidate_ratios: list of N/D splits to try at each fixed C optimal_points = [] for C in compute_budgets: best_loss = INFINITY best_N, best_D = None, None for ratio in candidate_ratios:

# solve N, D from C = 6ND and N/D = ratio N = sqrt(C ratio / 6) D = sqrt(C / (6 ratio)) model = TRAIN_MODEL(params=N, tokens=D, budget=C) loss = EVALUATE(model, held_out_set) if loss < best_loss: best_loss = loss best_N, best_D = N, D optimal_points.append((C, best_N, best_D, best_loss)) return optimal_points function DERIVE_ALLOCATION_LAW(optimal_points): Cs = [p.C for p in optimal_points] Ns = [p.N for p in optimal_points] Ds = [p.D for p in optimal_points] (A_N, a) = FIT_POWER_LAW(Cs, Ns) # N_opt(C) = A_N C^a (A_D, b) = FIT_POWER_LAW(Cs, Ds) # D_opt(C) = A_D C^b return (A_N, a, A_D, b) function PLAN_TRAINING_RUN(target_compute, A_N, a, A_D, b): N_opt = A_N (target_compute ^ a) D_opt = A_D (target_compute ^ b) return (N_opt, D_opt) function CHECK_EMERGENCE(model_family, task, scales): hard_metric_curve = [] # e.g. exact-match accuracy soft_metric_curve = [] # e.g. avg log-likelihood of correct answer for scale in scales: model = model_family[scale] hard_metric_curve.append(EVAL_EXACT_MATCH(model, task)) soft_metric_curve.append(EVAL_LOG_LIKELIHOOD(model, task))

# emergence is "real" only if the smooth metric also shows

# a sharp change in slope, not just the hard metric return (hard_metric_curve, soft_metric_curve) ```

Fitting cost. Fitting a scaling law itself is cheap: it is ordinary least squares on $O(k)$ points, where $k$ is the number of small-scale training runs — $O(k)$ time and space, negligible next to training cost. The expensive part is generating the data points, since each point requires an actual training run. This is the entire economic argument for scaling laws: instead of training $M$ full-scale candidate models to pick the best configuration ($O(M)$ full-scale runs, potentially catastrophic in cost), you train $k \ll M$ small models and extrapolate, turning an $O(M \cdot C_{\text{large}})$ search into $O(k \cdot C_{\text{small}} + C_{\text{large}})$ for one confirmatory large run.

Compute-optimal derivation, sketched. Given the constraint $C = 6ND$ and an additive loss model motivated by empirical fits,

(an irreducible entropy term $L_\infty$ plus separate parameter- and data-limited error terms), minimize $L$ subject to $ND = C/6$ using a Lagrange multiplier. Substituting $D = C/(6N)$:

Differentiating with respect to $N$ and setting to zero:

Solving for $N$ gives $N_{\text{opt}} \propto C^{\beta/(\alpha+\beta)}$, and correspondingly $D_{\text{opt}} \propto C^{\alpha/(\alpha+\beta)}$ — a genuine power law in $C$, confirming the empirical form used in Step 2/3 above. When empirical fits give $\alpha \approx \beta$, the exponents both come out near $1/2$, which is the Chinchilla finding: split growth roughly evenly between model size and data size. This is a real optimization result, not just curve-fitting folklore: it follows from minimizing a convex (in $\log N$, for reasonable exponent ranges) loss surface under a linear compute constraint.

Honesty about the model's limits. The additive two-term loss form is an empirical approximation, not a law of nature; it can break down when data is repeated (multiple epochs), when architecture changes qualitatively (e.g., mixture-of-experts routing, which decouples parameter count from FLOPs — foreshadowing Mixture-of-Experts), or at extremes of scale where new bottlenecks (data quality, tokenizer effects) dominate. The power law is a strong empirical regularity across many studied ranges, not a proven theorem.

Emergent abilities: correctness of the "mirage" explanation. Consider a task scored by exact-match over an answer of length $\ell$ tokens, where each token's probability of being correct scales smoothly with $N$ as $p(N) = 1 - \epsilon(N)$ with $\epsilon(N) \to 0$ smoothly. The probability the entire sequence is correct is approximately

for small $\epsilon$. Because of the exponent $\ell$, even a smooth, gradual decrease in per-token error $\epsilon(N)$ produces a sharply nonlinear rise in $P_{\text{exact match}}(N)$: the metric stays near zero while $\ell \epsilon(N) \gg 1$, then rises quickly once $\ell \epsilon(N)$ drops below roughly $1$. This is a mathematically rigorous demonstration that discontinuous-looking metrics can arise from perfectly smooth underlying quantities — it does not prove all emergent abilities are mirages, but it proves the mirage explanation is a valid and sufficient mechanism for at least some observed cases, which is why choosing evaluation metrics carefully is essential to correctly characterizing what "emerges."

Suppose (illustrative numbers, in the style of published scaling studies) small-scale experiments give the fitted per-token-error model $\epsilon(N) = 20 \cdot N^{-0.075}$ where $N$ is parameters in billions, and a benchmark requires $\ell = 8$ correct tokens in a row for an exact-match to count as correct.

For this metric to leave the "essentially zero" region, $\ell \epsilon(N)$ must drop from the hundreds down to order 1 — that is, $\epsilon(N)$ must fall below about $0.125$. Solving $20 N^{-0.075} = 0.125$:

This corresponds to $N$ on the order of $10^{27}$ — absurdly large — which illustrates the actual mechanism: a benchmark can require combinatorially small per-token error before exact-match rises above zero at all, and then it rises over a comparatively narrow band of scale once that threshold is neared. Real published emergent-ability curves (e.g., for arithmetic or multi-step reasoning benchmarks) show exactly this signature — flat-at-zero for many orders of magnitude, then a rise within roughly one order of magnitude of scale, consistent with a smoothly improving $\epsilon(N)$ raised to a power $\ell$. Choosing a partial-credit metric (e.g., token-level accuracy or log-likelihood, rather than exact-match) on the same checkpoints, in published studies, reveals a smooth curve throughout — no "emergence" appears, only the same underlying signal viewed through a metric that does not exponentiate error into invisibility. A second, complementary trace: compute-optimal allocation. Suppose small-scale IsoFLOP sweeps yield fitted exponents $\alpha \approx 0.34$, $\beta \approx 0.28$ for the loss model above. Then

If a lab plans a $10\times$ increase in compute over its last well-fit run (which used $N=13$B parameters and $D=300$B tokens, roughly consistent with $C = 6ND \approx 2.3\times10^{22}$ FLOPs), the model predicts

Notice both roughly triple rather than one scaling ten-fold and the other staying flat — this is the core Chinchilla lesson: earlier practice that scaled parameters much faster than tokens (fixing dataset size while growing $N$ by $10\times$–$100\times$) was leaving loss improvements on the table that could have come from more data at a smaller model size, for the same compute cost.

Scaling laws are now load-bearing infrastructure for the economics of frontier AI labs. Before committing tens of millions of dollars to a training run, teams fit scaling laws on a ladder of smaller models to choose architecture hyperparameters, dataset size, and stopping points, essentially replacing expensive full-scale hyperparameter search with cheap extrapolation. The Chinchilla-style compute-optimal recipe directly influenced the design of later production models, which shifted toward much larger training datasets relative to parameter count compared to earlier scaling-heavy designs. Inference cost considerations push real deployed systems to deliberately deviate from pure training-compute optimality, training smaller models on more tokens than the "optimal" point suggests, because inference happens far more often than training and a smaller model is cheaper to serve at scale. The emergent-abilities debate shapes how capability evaluations and safety audits are designed: if a jump in a benchmark score could be a metric artifact, evaluators are pushed toward multiple complementary metrics (log-likelihood, calibration, partial credit) rather than trusting a single exact-match number, especially when trying to anticipate dangerous capabilities before they become fully reliable. Both threads — predictable scaling of loss and unpredictable-looking scaling of specific skills — motivate architectural responses covered in related articles: In-Context Learning emerges as a byproduct of scale and is itself studied through the same emergent-versus-smooth lens, while Mixture-of-Experts architectures attempt to decouple total parameter count from per-token compute cost, changing the very compute-accounting ($C \approx 6ND$) that the classical scaling laws assume, and requiring the laws to be refit for that architecture family.

Ask the tutor: If I only have compute to train three small model checkpoints, how should I choose their sizes to get a reliable scaling-law fit?