Scaling Laws & the Compute-Data-Parameters Tradeoff

A scaling law in this context is an empirical relationship of the form

where $L$ is the cross-entropy test loss (in nats or bits per token) of a Transformer language model, $x$ is a resource — parameter count $N$, dataset size in tokens $D$, or training compute $C$ — and $\alpha$ and $x_c$ are constants fit from data. Plotted on log-log axes, this relationship is a straight line: $\log L$ decreases linearly in $\log x$. The exponent $\alpha$ is the scaling exponent; larger $\alpha$ means faster returns to additional resource. The crucial move is to hold two of $\{N, D, C\}$ effectively unconstrained and vary the third, then fit separate power laws $L(N)$, $L(D)$, and $L(C)$. Because these laws hold over several orders of magnitude — often four to five decades of $N$ or $D$ — a curve fit on models with $10^6$ to $10^9$ parameters reliably extrapolates to models with $10^{11}$ parameters, well outside the fitted range. This is what makes scaling laws practically powerful: they convert an expensive extrapolation question ("how good will a 500B-parameter model be?") into a cheap curve-fitting exercise using dozens of small, fast training runs. The second half of the idea is compute-optimal allocation. Since $C$ is (approximately) a fixed product of $N$ and $D$, and both $L(N)$ and $L(D)$ independently push loss down, a fixed compute budget $C$ can be spent many ways: a huge model on little data, a tiny model on huge data, or something balanced in between. The scaling laws for $L(N)$ and $L(D)$ jointly determine an optimal point $N^*(C)$, $D^*(C)$ that minimizes loss for that budget — and this optimal split itself follows a power law in $C$. Two major empirical studies of this optimum — Kaplan et al. (2020) and Hoffmann et al. (2022, the "Chinchilla" paper) — reached different conclusions about the exponents, with major practical consequences, discussed below.

Step 1: Fix the compute-loss relationship. For a dense Transformer trained with the standard forward-plus-backward pass, the compute cost per training token is approximately $6N$ FLOPs: roughly $2N$ for the forward pass (one multiply-add per parameter, per token, dominated by matrix multiplications) and roughly $4N$ for the backward pass (twice the forward cost, since gradients must be computed with respect to both activations and weights). Training on $D$ tokens for one epoch (or $D$ total token-steps, possibly with repetition) therefore costs

This holds independent of the exact split between $N$ and $D$ — it only assumes a dense architecture where the forward pass cost per token scales linearly in $N$ (true for standard Transformers when the token count is large relative to attention's quadratic term, which is usually small next to the linear feed-forward and projection costs).

Step 2: Fit the marginal power laws. Train a family of models varying $N$ (holding $D$ effectively unlimited, i.e., large enough not to bottleneck), and separately vary $D$ (holding $N$ large enough not to bottleneck). Fit

using least squares on the log-log data. Empirically (Kaplan et al.), $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$ for GPT-2/GPT-3-style Transformers, meaning loss decreases slowly but steadily and predictably as either resource grows — a factor of 10 in $N$ buys roughly a $10^{0.076} \approx 1.19\times$ reduction in loss's excess over its floor, all else fixed.

Step 3: Combine into a joint law. A common joint form (Kaplan et al.) is

which reduces to the marginal laws in the appropriate limits and captures the fact that once one resource is abundant, the other becomes the bottleneck (diminishing returns saturate against whichever resource is scarcer).

Step 4: Solve the compute-optimal allocation problem. Given a compute budget $C$, minimize $L(N, D)$ subject to $C = 6ND$. This is a constrained optimization: substitute $D = C / 6N$ into $L(N,D)$ and minimize over $N$ alone (a one-dimensional calculus problem, solvable by setting $dL/dN = 0$). The solution takes the power-law form

(the exponents must sum to one because $N \cdot D \propto C$ and both $N^*$ and $D^*$ are themselves power laws in $C$). The value of $a$ (equivalently $b = 1-a$) is the crux of the entire debate: Kaplan et al. found $a \approx 0.73$ (model size should absorb most of a compute increase, data barely grows), while Hoffmann et al.'s Chinchilla analysis found $a \approx 0.5$ (parameters and data should scale in equal proportion).

Step 5: Use the law to plan a real run. Given a compute budget (often fixed by hardware-hours available), compute $N^*(C)$ and $D^*(C)$ from the fitted exponents, then choose an architecture (depth, width, number of heads) that hits the target $N^*$ and a training corpus of size $D^*$ tokens. This turns the “what should we train?” design question into arithmetic. ```

# Fitting and using a compute-optimal scaling law function FIT_SCALING_LAW(models):

# models: list of (N_i, D_i, C_i, loss_i) from completed small-scale training runs

# Step 1: fit marginal laws holding one resource near-unconstrained (alpha_N, N_c) <- LEAST_SQUARES_LOGLOG( x = [m.N for m in models if m.data_unconstrained], y = [m.loss for m in models if m.data_unconstrained]) (alpha_D, D_c) <- LEAST_SQUARES_LOGLOG( x = [m.D for m in models if m.model_unconstrained], y = [m.loss for m in models if m.model_unconstrained])

# Step 2: fit joint law L(N, D) on the full grid of runs (alpha_N, alpha_D, N_c, D_c) <- FIT_JOINT_LOSS(models, init=(alpha_N, alpha_D, N_c, D_c)) return (alpha_N, alpha_D, N_c, D_c) function COMPUTE_OPTIMAL_SPLIT(C, alpha_N, alpha_D, N_c, D_c):

# Minimize L(N, C/(6N)) over N == solve dL/dN = 0 analytically or numerically best_loss <- INFINITY best_N <- NONE for N in LOG_SPACE(low=1e6, high=1e13, num_points=10000): D <- C / (6 N) loss <- JOINT_LOSS(N, D, alpha_N, alpha_D, N_c, D_c) if loss < best_loss: best_loss <- loss best_N <- N N_star <- best_N D_star <- C / (6 N_star) return (N_star, D_star, best_loss) function PLAN_TRAINING_RUN(compute_budget_flops, alpha_N, alpha_D, N_c, D_c): (N_star, D_star, predicted_loss) <- COMPUTE_OPTIMAL_SPLIT( compute_budget_flops, alpha_N, alpha_D, N_c, D_c) architecture <- CHOOSE_ARCHITECTURE(target_params = N_star) corpus <- SELECT_CORPUS(target_tokens = D_star) return (architecture, corpus, predicted_loss) ``` The pseudocode has two honest edge cases worth flagging. `LEAST_SQUARES_LOGLOG` requires strictly positive losses above an irreducible floor (an "entropy floor" $L_0$ representing the true entropy of natural language); fits are usually done on $L - L_0$, and if $L_0$ is misestimated the whole exponent shifts. `COMPUTE_OPTIMAL_SPLIT`'s grid search is a stand-in for the closed-form solution obtainable by calculus — real implementations solve $dL/dN = 0$ analytically for whichever joint-loss functional form they've chosen, since a grid search over 10,000 points is not exact and can miss the true minimum between points.

Fitting cost. Fitting a single marginal power law $L(x) = (x_c/x)^\alpha$ is ordinary least squares in log-log space after the substitution $\log L = \alpha \log x_c - \alpha \log x$: this is a linear regression with two parameters, solvable exactly in $O(k)$ time for $k$ data points via the normal equations, or $O(k)$ per gradient step for iterative fitting. The joint law over $(N, D)$ has more parameters ($\alpha_N, \alpha_D, N_c, D_c$, sometimes an irreducible floor $L_0$) and is fit by nonlinear least squares (e.g., L-BFGS), typically converging in tens of iterations over a grid of a few hundred training runs — negligible compute compared to the runs themselves. The dominant cost of the whole scaling-law methodology is running the family of small models used to generate the fitting data, but these are chosen to be orders of magnitude cheaper than the target run, which is the entire point.

Why the exponent-sum identity holds. Given $C = 6ND$ and the two candidate solutions $N^*(C) \propto C^a$, $D^*(C) \propto C^b$, correctness of $a + b = 1$ follows directly from substitution:

For this to hold as an identity in $C$ (not just at one point), the exponents on both sides must match, forcing $a + b = 1$. This is not an empirical finding but an algebraic constraint — any correctly fit compute-optimal frontier must satisfy it, and it serves as a sanity check on fitted exponents (Chinchilla's $a \approx b \approx 0.5$ and Kaplan's $a \approx 0.73, b \approx 0.27$ both satisfy it).

Deriving the optimal split via Lagrange multipliers. To see why an interior optimum exists (rather than "always spend everything on $N$" or vice versa), consider the joint loss in the simplified additive-in-logs form

as a local approximation when neither resource dominates. Minimizing subject to $\log C = \log 6 + \log N + \log D$ via a Lagrange multiplier $\lambda$:

This naive additive approximation actually predicts $\alpha_N = \alpha_D$ at the optimum, which is not quite the empirical picture — the real joint loss function is not additive but has the "harmonic-sum" saturating shape from Step 3, where increasing $N$ alone with fixed $D$ eventually saturates (loss bottoms out at the $D$-dependent floor). Solving the true constrained optimization on the saturating form yields

which is why the ratio of the fitted marginal exponents $\alpha_N/\alpha_D$, not their absolute values, determines the compute-optimal split — a small change in how $\alpha_N$ and $\alpha_D$ are estimated (e.g., using a bigger, more diverse validation set, or correcting for learning-rate schedule artifacts) can swing $a$ substantially, which is exactly what happened between the Kaplan and Chinchilla studies.

Why Chinchilla revised Kaplan's exponent. The Chinchilla paper (Hoffmann et al., 2022) identified two methodological issues in the earlier fit: Kaplan et al. fit $L(N)$ using models trained with a learning-rate schedule tuned for a fixed, large number of steps regardless of $N$, which understates the achievable loss of smaller models (they were effectively undertrained relative to their optimal schedule), and it fit over a narrower range of $D$ at each $N$. Correcting the learning-rate schedule per model size and training over three independent methodologies (fixing $N$ and varying $D$; fixing training FLOPs and varying $N$; and fitting a parametric joint loss surface directly) all converged on $a \approx b \approx 0.5$, i.e., parameters and data should scale at the same rate as compute grows — a substantially more data-hungry prescription than Kaplan's.

Honest tradeoffs. Scaling laws are extrapolations, not proofs; they assume the power-law form continues to hold far outside the fitted range, an assumption that can break down (e.g., data quality bottlenecks, or the eventual exhaustion of high-quality text corpora, mean $D$ cannot grow indefinitely in practice regardless of what the law prescribes). They also describe training loss/perplexity, not downstream task accuracy directly — a compute-optimal model by the scaling-law criterion is not automatically the best model for every deployment (inference cost, latency, and fine-tuning behavior are separate constraints the law does not model, since a smaller model trained longer is often cheaper to serve even at equal training loss).

Suppose a lab has a fixed budget of $C = 3 \times 10^{23}$ FLOPs and wants to compare what the Kaplan-style law and the Chinchilla-style law each prescribe.

Step 1 — Kaplan-style split ($a \approx 0.73$, $b \approx 0.27$). Using the fitted proportionality constants from Kaplan et al. (approximately, reading off their published fits):

Plugging in $C = 3\times10^{23}$ gives a ratio $C / 6\times10^{18} \approx 5\times10^4$, and $(5\times10^4)^{0.73} \approx 3{,}000$, so $N^*_{\text{Kaplan}} \approx 1.3\times10^9 \times 3000 \approx 3.9 \times 10^{12}$ parameters — roughly a 4-trillion-parameter model. This is (deliberately, to illustrate the point) an extreme figure, larger than any model actually deployed, because Kaplan's fit was later shown to overweight parameter growth; real GPT-3-scale planning under this law targeted the hundreds-of-billions range with correspondingly modest data.

Step 2 — corresponding data budget under Kaplan. $D^*_{\text{Kaplan}} = C / (6N^*) = 3\times10^{23} / (6 \times 3.9\times10^{12}) \approx 1.3 \times 10^{10}$ tokens — about 13 billion tokens, strikingly small for a multi-trillion-parameter model (for comparison, GPT-3, at 175B parameters, was trained on roughly 300B tokens, already showing the tension: Kaplan's own prescription, taken to its logical extreme, implies models even more undertrained-relative-to-size than GPT-3 actually was).

Step 3 — Chinchilla-style split ($a = b = 0.5$). Hoffmann et al.'s fitted law gives approximately

(constants fit in Hoffmann et al.'s Table 3; units chosen so $C$ is in FLOPs). Plugging in $C = 3\times10^{23}$: $\sqrt{3\times10^{23}} \approx 5.5\times10^{11}$, giving $N^*_{\text{Chinchilla}} \approx 0.6 \times 5.5\times10^{11} \approx 3.3\times10^{11} \approx 330$ billion parameters and $D^*_{\text{Chinchilla}} \approx 0.3 \times 5.5\times10^{11} \approx 1.65\times10^{11} \approx 165$ billion tokens.

Step 4 — sanity check via the constraint. Verify $C = 6ND$ for the Chinchilla answer: $6 \times 3.3\times10^{11} \times 1.65\times10^{11} \approx 3.27\times10^{23}$, matching the budget to within rounding — the constraint holds, confirming the split is a valid point on the compute frontier (not merely a plausible-looking pair of numbers).

Step 5 — compare implied predictions. The actual Chinchilla model, trained by Hoffmann et al. at a smaller compute budget than GPT-3's successor Gopher, used 70 billion parameters and 1.4 trillion tokens (a compute-matched comparison to Gopher's 280 billion parameters trained on only 300 billion tokens) and outperformed the much larger, more compute-equal Gopher on nearly every downstream benchmark — the empirical confirmation that at fixed compute, Gopher-style large-and-undertrained was suboptimal relative to smaller-and-longer-trained.

The "tokens per parameter" row is the number practitioners actually remember: the Chinchilla result is often summarized as "train on roughly 20 tokens per parameter" (165B/330B $\approx 0.5$ in this toy example's units works out closer to 20 with the paper's exact fitted constants) — a rule of thumb directly derived from setting $a = b = 0.5$ in the frontier equations.

Scaling laws are now a standard planning tool at every organization training frontier language models. Compute budgets for models like GPT-4-class systems, PaLM, LLaMA, and their successors are allocated using compute-optimal (or deliberately inference-optimal, over-trained-beyond-Chinchilla) heuristics derived directly from this line of research; LLaMA's authors, for instance, explicitly chose to train smaller models on far more tokens than Chinchilla-optimal would suggest, because a slightly higher training loss was an acceptable price for a model cheap enough to serve at scale — a direct, documented application of the tradeoff analyzed here. Cloud providers and AI labs use scaling-law extrapolation to decide, months in advance, how many GPU-hours to reserve and how large a training corpus to curate, since the fitted curves let them predict the final loss (and, empirically, downstream benchmark performance) of a run before it starts. The same methodology — small, cheap proxy experiments extrapolated via power-law fits — has since been extended to vision Transformers, multimodal models, reinforcement-learning-from-human-feedback pipelines, and even to predicting the compute needed to reach a target level of "emergent" capability on reasoning benchmarks, making scaling-law analysis a load-bearing part of nearly every large-scale pretraining pipeline built on top of the autoregressive language modeling paradigm.

Ask the tutor: How would the compute-optimal split change if we account for the fixed extra cost of inference at deployment, not just training?