Mixture of Experts, or MoE, is an architecture that splits a model's parameters into specialized sub-networks and activates only a few of them per token. A model can hold a trillion parameters on disk while only running a fraction of that number during inference. That gap between total size and active size is the entire point. It's why models like Mixtral and DeepSeek-V3 punch far above their apparent weight class without needing the compute budget of a small country.
Mixture of experts explained the simple way: instead of one giant neural network processing every input with every parameter, you build dozens of smaller networks called experts, plus a router that decides which two or four experts should handle each token. Most parameters sit idle for any given piece of text. Only the relevant experts fire. It's the difference between calling your entire company into a meeting versus pulling the two people who actually know the answer.
A standard transformer has a feed-forward block after every attention layer, and that block runs on every single token, no exceptions. MoE replaces that block with a bank of parallel feed-forward networks, each one a distinct expert with its own weights. A small routing network looks at each token and picks the top-k experts, usually one or two out of eight, or two out of sixty-four in bigger systems.
The experts aren't hand-labeled by topic. Nobody assigns "expert 3 handles grammar" or "expert 7 handles math." The specialization emerges during training as the router learns which experts produce lower loss for which kinds of tokens. Researchers who've probed trained MoE models find loose patterns, some experts lean toward punctuation, some toward specific languages, but the boundaries are fuzzy and inconsistent across layers.
Every transformer layer can have its own MoE block with its own router. A 64-layer model might make routing decisions 64 separate times per token, and the experts chosen in layer 12 have nothing to do with the experts chosen in layer 40. This is what makes MoE models hard to reason about from the outside. You can't point to a single expert and say "that's the coding one."
The router is just a small linear layer followed by a softmax. It takes the token's hidden state and outputs a probability score for every expert. The top-k experts get selected, their outputs get weighted by those probabilities, and the weighted sum replaces what a normal feed-forward layer would have produced.
This sounds clean until you hit the obvious failure mode: without correction, the router collapses onto a handful of favorite experts and ignores the rest. Early tokens that happen to get good gradients through certain experts make those experts better, which makes the router pick them more, which starves the other experts of training signal entirely. Left unchecked, you end up paying for sixty-four experts and effectively training eight.
Fixing this requires an auxiliary load-balancing loss added directly to training. It penalizes the model when routing is too uneven, nudging tokens toward underused experts even when they aren't the router's first choice. Get this wrong and you've built an expensive, mostly-decorative model. Get it right and utilization across experts stays roughly even, which is what actually delivers the promised efficiency.
Compare two models with similar total parameter counts. A dense 70-billion-parameter model activates all 70 billion parameters on every single token, every forward pass, no exceptions. A sparse MoE model with 140 billion total parameters but only 20 billion active per token costs less to run than the dense model while carrying twice the raw capacity.
This is the actual selling point, not a footnote. Training and serving cost scale with active parameters, not total parameters. Capacity for storing knowledge scales with total parameters. MoE decouples two things that used to move together, and that decoupling is worth more to a company running inference at scale than almost any other architectural trick released in the last five years.
| Model type | Total params | Active params per token | Inference cost driver |
|---|---|---|---|
| Dense transformer | 70B | 70B | Full model, every token |
| Sparse MoE | 140B | 20B | Only routed experts |
| Sparse MoE (Mixtral 8x7B style) | 47B | 13B | Two of eight experts |
Mixtral 8x7B is the cleanest public example. Eight experts, roughly 47 billion parameters total, but only two experts activate per token, bringing active compute down to around 13 billion. It ran faster than dense 13B models while matching or beating dense 70B models on several benchmarks. That's the trade MoE makes possible, and no amount of quantization or distillation gets you that same shape of result.
MoE isn't free efficiency. It moves the cost somewhere else, mostly into memory and communication. You still have to store every expert's weights in memory, even the ones sitting idle for a given token, because the next token might route somewhere completely different. A 140-billion-parameter MoE model needs the VRAM to hold 140 billion parameters, not 20 billion, even though it only computes with 20 billion at a time.
This is why MoE models feel deceptively heavy to deploy despite their fast inference. You get GPU-poor teams excited about "only 13B active parameters" and then watching their deployment fail because the full 47B still has to sit loaded across the cluster. Cheap compute, expensive memory. That trade-off doesn't show up in most casual explanations of the architecture.
Training adds another wrinkle: expert parallelism. Different experts often live on different GPUs, so a token routed to an expert on another device has to physically travel across the network mid-forward-pass. This all-to-all communication becomes a real bottleneck at scale, and it's a major reason why building MoE training infrastructure is harder than building dense-model infrastructure, even though the underlying math looks simpler on paper.
Mixtral popularized the pattern in the open-weight world, but MoE isn't new and isn't limited to one lab. Google's Switch Transformer and GLaM papers laid the groundwork years earlier, routing to as many as 2,048 experts in research settings. DeepSeek-V3 pushed the idea further with finer-grained experts and a shared-expert design that keeps some capacity always-on while routing the rest, improving stability over naive top-k routing.
There's also strong reason to believe GPT-4 uses a mixture-of-experts design, based on leaked details and the sheer economics of running a model that large at OpenAI's request volume. A dense model at that scale would be brutally expensive to serve; a sparse one makes far more business sense. Nobody outside OpenAI has confirmed the exact configuration, and I'd treat specific numbers floating around online with real skepticism, but the architectural logic points the same direction every time: if you're serving hundreds of millions of requests, sparse activation is the only way the unit economics work.
MoE earns its keep at scale and loses value below it. If you're training or serving a small model, the memory overhead of storing unused experts outweighs any compute savings, and you're better off with a dense model that's simpler to train, debug, and deploy. Most teams fine-tuning a 7B or 13B model for a specific task gain nothing from sparsity and take on real infrastructure complexity for no payoff.
Fine-tuning MoE models is also genuinely harder than fine-tuning dense ones. Routing behavior learned during pretraining can shift under fine-tuning in ways that are difficult to predict, sometimes collapsing expert diversity right back into the imbalance problem the load-balancing loss was supposed to solve. Teams doing serious MoE fine-tuning usually freeze the router or apply extra regularization specifically to keep routing stable, which is one more thing to get right on top of everything a normal fine-tuning job already demands.
The honest takeaway: MoE is a scaling tool for organizations running frontier-size models at commercial volume, not a general-purpose upgrade every model should adopt. If you're not already bottlenecked by dense-model compute costs at massive scale, sparsity is solving a problem you don't have yet.
Mixture of Experts is a model design that splits parameters into separate sub-networks called experts, and routes each piece of input to only a few of them instead of running everything through the whole model. This lets a model hold far more total parameters than it actually uses on any single input, cutting compute cost while keeping capacity high. Mixtral and DeepSeek-V3 are both public examples of this design.
A small routing network scores every expert for each token and picks the top one or two highest-scoring experts, usually called top-k routing. The scores come from a learned linear layer, not hand-written rules, so the router improves purely through training. A load-balancing loss added during training stops it from collapsing onto a small handful of favorite experts.
It makes compute cheaper per token because only a fraction of total parameters activate on any given forward pass. It does not make memory cheaper, since every expert's weights still have to be loaded even if most sit idle for a specific token. The trade is lower compute cost paired with higher memory requirements compared to a dense model of similar active size.
There is strong circumstantial evidence, including leaked reports and basic serving economics, that GPT-4 uses some form of sparse mixture-of-experts architecture. OpenAI has never officially confirmed the exact configuration, so specific numbers circulating online should be treated as unverified. The broader pattern holds regardless: any lab serving a frontier-scale model at massive volume has strong financial reasons to use sparse activation.
Usually not. MoE's advantages only show up at large scale, and small teams working with 7B to 13B models typically gain nothing from the added routing complexity. Fine-tuning MoE models is also harder because routing behavior can destabilize during training, often requiring frozen routers or extra regularization that a dense model simply does not need.