Mixture of Experts (MoE): the trick behind giant LLMs

In August, DeepSeek released the final version of DeepSeek-V4-Pro: 1.6 trillion parameters, with open weights anyone can download. Moonshot’s Kimi K3 reaches 2.8 trillion. Those are impossible-to-move model sizes. And yet, to write each word, V4-Pro uses only 49 billion: 3%.

It isn’t a one-off. Almost every large open model of 2026 works this way: DeepSeek, Kimi, Qwen, Llama 4, OpenAI’s gpt-oss and Google’s mid-size Gemma 4. They’re called Mixture of Experts, or MoE. Which makes the dumb question compulsory: if it only uses 3%, what does it want the other 97 for?

A quick note before going on: a language model doesn’t read words but tokens, which are pieces of words. I’ll say “word” here to keep things simple.

A traditional LLM, the kind engineers call dense, is like one huge cook who makes every dish. For every word it writes, the whole model works: all its parameters, every time.

An MoE is a kitchen with lots of cooks and a head chef. An order (the word) comes in, the chef looks at what it’s about and hands it to two or three cooks. The rest take a break.

The analogy gives us the two ideas of this post:

1. Each dish costs the work of two cooks, not the whole staff. That’s why you can hire many more without each dish getting more expensive. And more cooks means more knowledge stored in the model.
2. The kitchen needs room for everyone, working or not. That room is the computer’s memory. Cooks on a break don’t get paid, but they still take up space.

In technical terms: the cooks who work are the active parameters, and the whole staff is the total parameters.

An LLM is a stack of layers, and each layer has two parts. The first is attention: each word looks at the others to understand the context. The second is a network that transforms each word on its own and holds most of the parameters. An MoE only changes the second part: it splits it into many small networks, the experts, and adds a router, which plays the head chef.

The router scores every expert, keeps the best ones (2 of 8 in Mixtral, 8 of 256 in DeepSeek-V3, 16 of 896 in Kimi K3) and blends their answers with weights. The choice is made for every word and in every layer: in the same sentence, each word can go through different cooks, and through others in the next layer.

A warning, because the name is misleading: “expert” doesn’t mean one knows medicine and another knows poetry. When the Mixtral authors looked at which experts got biology texts, philosophy texts and scientific papers, they found no split by topic. What they really split is weirder and more fun: I tell that story in MoE myths.

Because learning gets cheaper. With the same compute budget, an MoE can have many more parameters, and more well-trained parameters make a better model. Google measured it in 2021 with the Switch Transformer: with the same resources it trained up to 7 times faster than its dense equivalents, and 4 times faster than the largest T5. In 2023, Mixtral 8x7B matched or beat Llama 2 70B, a dense 70-billion model, on every test in its paper, using only 13 billion per word.

The idea isn’t new: it shows up in a 1991 paper co-signed by, among others, Geoffrey Hinton. But until a few years ago it was hard to train without it going off the rails. Today it’s the norm:

What about closed models, like Claude, GPT or Gemini? Their companies don’t publish the architecture, so nobody outside knows. The exception is Gemini 1.5 Pro, which its technical report describes as an MoE.

Here’s the catch, and it’s the one that most confuses people who want to run these models at home:

• How much it computes depends on the active parameters. Computing little makes each word fast and cheap.
• How much room it takes depends on the total parameters. Every expert has to be loaded in the graphics card’s memory, because the router may ask for any of them on the next word.

The clearest example is gpt-oss-120b, OpenAI’s open model: it uses 5.1 billion parameters per word, but it has 117 billion. It computes like a small model and takes room like a big one. If it fits on an 80 GB server GPU, that’s because its experts ship compressed to about 4 bits per parameter, not because it’s an MoE.

Try the calculator: pick a model or move the sliders, and see where it fits.

Two pairs are worth comparing because they come from the same family: Qwen3-32B (dense) versus Qwen3-30B-A3B (MoE), and Gemma 4 31B versus Gemma 4 26B A4B. Each pair takes almost the same memory, but the MoE computes 8 to 10 times less per word. Google presents its MoE as almost as fast as a 4-billion-parameter model.

If you use ChatGPT, Claude or any API, there’s nothing to decide: you pay per word, and the MoE savings, if any, already reach you in the price. The question matters when you’re the one running the model.

• “It uses 5B per word, so it fits wherever a 5B model fits.” No: memory is set by the total parameters. gpt-oss-120b needs room for its 117 billion, plus the conversation’s memory on top.
• “Each expert knows one topic.” No: in Mixtral, biology, philosophy and scientific-paper texts were spread across experts almost identically. What they actually do is in MoE myths.
• “An MoE is always faster.” It computes less, yes. But with many users at once, each word asks for different experts, almost all of them have to be read, and the busiest expert sets the pace. Measure your own case, not the model card.

1. A dense model uses everything it has on every word; an MoE stores a lot and uses little.
2. Active → compute. Total → memory. Always ask for both numbers.
3. Little memory and few users: dense. Lots of memory, lots of traffic or an API in between: MoE.

If you’re not sure what a parameter or a neuron is, start with the lesson Logistic regression or neural network?.