Neural Scaling Laws and the Kaplan Predictions
By Satwik ยท March 15, 2026
Kaplan and colleagues showed that language model loss falls as a smooth power law in model size, data, and compute, spanning many orders of magnitude. Turning capability into a forecastable quantity reshaped strategy and created new safety obligations.
The empirical discovery
In early 2020, a team at OpenAI published a study of how Transformer language model performance depends on scale. The finding was clean and surprising in its regularity: the cross-entropy loss on held-out text falls as a power law in each of three quantities, the number of model parameters, the size of the training dataset, and the amount of compute spent, when the other factors are not bottlenecks. Plotted on log-log axes, the loss traces a straight line across many orders of magnitude.
Two features made this more than a curiosity. First, the relationships were smooth and stable, not noisy trends but tight fits that held from small models up to the largest then studied. Second, the details of architecture, depth versus width, exact attention shape, mattered remarkably little compared with scale. Within wide ranges, how you spent parameters was less important than how many you had. Performance was, to first approximation, a function of scale rather than of clever design.
Why forecastability is the real product
A power law is a prediction engine. If loss follows a known functional form in compute, you can train a series of small, cheap models, fit the curve, and extrapolate the loss of a model far larger than any you have built, before committing the budget to build it. Kaplan's team also derived how, given a fixed compute budget, to allocate it between making the model bigger and training it longer, and their analysis suggested that most of the budget should go into model size, with training data scaled more modestly.
This converted frontier development from a gamble into planning. A lab could decide how much compute to raise, forecast the resulting capability, and justify enormous expenditures with an extrapolated curve rather than a hope. The scaling laws are the intellectual foundation under the entire subsequent buildout of large-scale training; they are why it was rational to spend more each year.
An important later correction
The Kaplan allocation, favoring parameters heavily over data, was influential and, as later work showed, not quite right. In 2022 the Chinchilla study from DeepMind re-examined the compute-optimal tradeoff with a more careful treatment of the learning rate schedule and found that data should scale roughly in proportion to parameters, meaning many models of the era were substantially undertrained for their size. The scaling-law framework survived the correction intact; what changed was the coefficient of optimal allocation, not the existence of a smooth law. This is a healthy example of a quantitative claim being refined by better experiments rather than overturned, and it is a caution against treating any single fitted exponent as settled.
Why it mattered, and the caveats
The scaling laws gave the field its dominant strategic thesis: capability is buyable with compute, and progress is, to a useful degree, predictable. That thesis drove capital, hardware demand, and the concentration of frontier work in a few well-funded labs.
Two caveats deserve emphasis. First, the laws describe pretraining loss, a smooth quantity, not downstream task performance, which can improve in sudden jumps as loss crosses thresholds that matter for a given task. A smooth loss curve can hide abrupt changes in what the model can actually do. Second, extrapolation always assumes the regime continues, and power laws must eventually bend as data or other resources become limiting.
The security angle
Predictable scaling has a double-edged safety implication that sits at the center of frontier governance.
On one side, forecastability is a safety asset. If you can predict roughly how capable the next model will be before training it, you can plan evaluations, red-teaming, and safeguards in advance rather than discovering dangerous capabilities after deployment. Scaling laws are a precondition for the kind of anticipatory governance that responsible scaling policies attempt.
On the other side, the laws are the quantitative form of the capability-overhang worry. They tell you that spending more compute reliably yields a more capable model, but they say nothing about which specific dangerous abilities appear at which scale, or whether such abilities emerge smoothly or suddenly relative to a benchmark. So a lab can know with confidence that its next model will be broadly stronger while remaining genuinely uncertain about whether it crosses a specific hazardous threshold. The predictability of aggregate loss coexists with unpredictability of specific behaviors. Managing that gap, confident about the average, uncertain about the tails, is the core difficulty that scaling laws bequeathed to AI security.