Benchmark
Also called AI benchmark
A fixed test used to score and compare AI models, like an exam where every model gets the same questions.
A benchmark is a set of questions or tasks with known answers: maths problems, coding challenges, exam questions, real software bugs. Run a model on it and you get a score that can be compared with other models. Every launch comes with a chart of them.
Scores are useful and easy to over-read. Once a test is public, its questions can end up in the training data, which inflates scores. Popular tests get "saturated" when top models all score near the maximum, so labs build harder ones, such as Humanity's Last Exam, about 2,500 expert-written questions. And no test covers the exact work you'll give the model.
For example
A launch chart shows a new model beating its rivals on a coding benchmark by 4 points. Your spreadsheet formulas weren't on that test.
Put it to use
Where you'll meet benchmark in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.