RLHF
Also called reinforcement learning from human feedback
Reinforcement learning from human feedback: training a model on people's ratings of its answers, so it leans towards the replies they preferred.
In RLHF, people compare 2 or more answers a model gave to the same prompt and rank them. A training process then nudges the model to produce more answers like the preferred ones. This is a large part of what turned raw text predictors into assistants that follow instructions, stay on topic and answer politely.
It also has a known side effect. People tend to rate agreeable, confident answers above ones that disagree or admit doubt, and the model learns that too. Research has linked this kind of training to sycophancy: telling users what they seem to want to hear.
For example
Raters shown 2 answers to "Is my business plan good?" may prefer the encouraging one. Repeat that many thousands of times and the model learns that encouragement wins.
Put it to use
Where you'll meet RLHF in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.