Tokenizer
The part of a language model that cuts text into tokens before the model reads it. Each model family has its own.
Before a model reads anything, a tokenizer splits the text into tokens: whole words where a chunk was common in the text the tokenizer was built from, smaller pieces where it wasn't.
That's why the same sentence can cost more on one model than on another, and why a message in another language or script can take several times the tokens of the same message in English. Cost, wait and how much fits in the context window all follow the tokenizer's count, not the word count.
For example
The same support prompt counts about 30 percent more tokens on today's Claude than on earlier Claude models, because the newer tokenizer cuts the text differently.
Put it to use
Where you'll meet tokenizer in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.