Scaling Laws for Neural Language Models

By Satwik · January 31, 2026

Kaplan et al. (2020) put empirical structure under the intuition that bigger models are better. They trained many language models across ranges of parameters, dataset size, and compute, and found that test loss falls as a smooth power law in each of these factors when the others are not bottlenecking. The relationships held across several orders of magnitude, which is what made the paper influential: loss became something you could forecast rather than discover after the fact.

A key practical takeaway was that, given a fixed compute budget, most of it should go into making the model larger rather than training a smaller model for longer. Architecture details mattered surprisingly little compared with scale. This gave labs a planning tool: pick a compute budget, read off the expected loss, and choose model and data sizes accordingly.

Why it mattered

Scaling laws turned model building into an engineering roadmap and directly motivated the run toward GPT-3-scale systems. They also framed a decade of investment decisions.

Reading angle

The 2020 recommendations were later corrected by Chinchilla (2022), which showed the Kaplan runs undertrained large models on too little data and that parameters and tokens should scale together. Read Kaplan as the foundational but not final word. For a security reader, the deeper point is that capabilities became predictable enough to anticipate before deployment. That predictability is an opportunity for governance: if you can forecast the loss of a planned run, you can reason about its likely capabilities, and the risks that come with them, before spending the compute.