Description
The paper empirically uncovers robust power-law relationships between a language model’s performance (measured via cross-entropy loss) and three core factors: model size, dataset size, and compute budget. Remarkably, these trends hold across more than seven orders of magnitude, while architectural variations like layer depth or width exert minimal impact. The authors further derive simple formulas that predict overfitting behavior and optimal compute allocation—revealing that larger models are significantly more sample-efficient, and that achieving compute-efficient training often means using very large models trained on modest data volumes and stopping well before full convergence.
