Eval
Also called evals, evaluation
A repeatable test for an AI feature: fixed cases, a pass/fail check on each, and a score you track every time something changes.
An ordinary test compares output with an expected value. An AI feature's output varies from run to run, and many different answers can be right, so an eval checks what must be true of a good answer instead: the fact is there, the right tool was called, nothing was invented.
It has 3 parts: fixed cases, usually real inputs; a pass/fail check for each failure that matters; and a score you track across changes. A prompt edit or a model upgrade then has to do better on the same cases, rather than look better on 3 examples tried by hand.
For example
20 real support questions, each checked for the right order lookup and the current price, scored before and after every prompt edit.
Put it to use
Where you'll meet eval in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.