Quantization
Also called quantisation, quantized, quantised
Storing a model's numbers with less precision, so it needs less memory and runs faster, at some cost in quality.
A model is billions of numbers, and by default each one is stored with a lot of precision, typically 16 bits. Quantization rounds them to fewer bits, 8, 4 or even 2. Like saving a photo as a smaller file, it throws away detail most people won't notice.
Halving the bits roughly halves the memory a model needs, and it runs faster too. That's what makes large models usable on an ordinary laptop and small ones usable on a phone. The cost is accuracy: round too hard and the model starts to slip, especially on tasks that need precision, like maths or exact facts.
For example
Apple's on-device model stores each of its roughly 3 billion parameters in just 2 bits, which is how a capable language model fits on a phone.
Put it to use
Where you'll meet quantization in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.