Mixture of experts

In an ordinary model, every parameter does work for every word. A mixture-of-experts model divides much of the network into many smaller blocks, the "experts", and adds a router that picks a few of them for each token. The rest sit idle for that step.

The trade is worth it. The model can hold far more learned knowledge in total, while each word costs only as much computing as a much smaller model. Several large open models use this design, including OpenAI's gpt-oss. The catch is memory: all the experts still have to be loaded, even though only a few work at once.