Synthetic data
Training material made by AI rather than collected from people: generated questions, worked answers, conversations or images.
High-quality human text is limited, and some skills, like long chains of maths or safe answers to risky questions, are rare in it. So labs increasingly make training data with AI: models write questions, worked answers, conversations and code, and other checks filter out the bad ones.
Done well, this works. Microsoft reported that synthetic data made up the bulk of the training data for its small Phi-4 model, and reported that it beat its larger teacher model on science and maths questions. Done carelessly, it copies a model's errors and narrows its range, and training only on generated data over many rounds leads to model collapse.
For example
To teach a model maths, a lab can have a strong model write thousands of practice problems with worked solutions, check the answers automatically, and keep only the correct ones.
Put it to use
Where you'll meet synthetic data in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.