LLM-as-judge
Also called LLM-as-a-judge, LLM judge, model grader
Using a language model to grade another model's answers, for checks that take judgement rather than a rule code can apply.
Some qualities can't be checked with code: whether a reply answered the question the person actually asked, or whether its tone fits. LLM-as-judge hands that check to a model, which reads the answer, applies a rubric and returns a verdict, quickly and for thousands of answers.
Strong models agree with people's preferences much of the time, but they have measured biases, such as preferring longer answers. So a grader is an instrument to calibrate: compare its verdicts with labels from people who know the task before relying on it.
For example
A grader model reads each support reply and answers one question, "Did it give the customer the refund amount?", with pass or fail.
Put it to use
Where you'll meet LLM-as-judge in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.