Pretraining
Also called pre-training, pretrained
The first and biggest stage of training, where a model learns to predict the next word across an enormous amount of text.
Pretraining is where a model gets most of what it knows. It reads a vast collection of text, far more than any person could, and practises one task billions of times: guess the next word, check, adjust. To get good at that, it has to pick up grammar, facts, styles of argument, even how code works.
What comes out isn't an assistant yet. A freshly pretrained model continues text rather than answering it, and it isn't good at following instructions. Post-training turns it into the chatbot you use. Pretraining also fixes the model's knowledge at a date, its cutoff.
For example
A model fresh from pretraining, given "What's the capital of France?", might reply with 3 more quiz questions, because that's how such lines often continue on the web.
Put it to use
Where you'll meet pretraining in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.