Scaling laws
The finding that models improve in a predictable way as they're trained bigger, on more data, with more computing power.
In 2020, researchers at OpenAI showed that a language model's error falls smoothly and predictably as you increase 3 things: the size of the model, the amount of data and the computing used to train it. In 2022, a DeepMind team refined the recipe: size and data should grow together, roughly 20 tokens of training text for every parameter.
That predictability changed the industry. If spending more reliably buys a better model, building bigger is a sound bet, hence the race for chips and power. More recently, labs found a second dial: letting a finished model think longer at answer time also improves results on hard problems.
For example
Scaling laws are why AI companies announce data centres costing tens of billions: the pattern so far says more computing buys a better model.
Put it to use
Where you'll meet scaling laws in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.