A recruiter says the model has 671 billion parameters, but only 37 billion of them fire on any given token. If you can say what that gap means for GPU memory and for latency, you’ve cleared the first bar of a mixture-of-experts question. A surprising number of candidates can’t, and the interviewer finds out in about ninety seconds.
Mixture of experts, MoE for short, is now the default shape of a frontier model. DeepSeek-V3, Qwen3, Llama 4, and most serious open-weight releases in 2026 ship as MoE, so the questions have moved from “what is it” to “prove you understand the tradeoffs well enough to serve one.”
What a token does inside one MoE layer
A standard transformer block has an attention sublayer and a feedforward sublayer, the FFN. In a dense model, every token passes through the same FFN. MoE replaces that single FFN with many of them, called experts, plus a small router that decides which experts each token should visit.
The router is usually one linear layer. It takes the token’s hidden vector, produces a logit per expert, and picks the top-k highest, often the top 2 or top 8. The token flows only through those k experts. Their outputs get combined, weighted by the router’s renormalized scores, and that sum leaves the layer. Every other expert sits idle for this token. Swap experts on the next token, and a different subnetwork does the work.
Routing to a single expert (top-1, the way the Switch Transformer did it) is the cheapest option, but most models use top-2 or more. Two reasons: the router gets a gradient signal it can compare across the chosen experts, and a token isn’t held hostage to one bad routing decision. That is the whole trick. The model holds the parameters of a very large network, but any single token only pays the compute of a small one. Capacity and compute, welded together in a dense model, come apart.
Active versus total, and why VRAM doesn’t get the discount
This is the distinction interviewers probe hardest, because it separates people who read a blog post from people who have served one of these models. Active parameters set the FLOPs per token, so they govern latency and throughput. Total parameters set how much has to sit in memory, because the router might send the next token to any expert, so all of them have to be resident.
Mixtral 8x7B made this concrete for a lot of people. It runs at roughly the speed of a 13B dense model but occupies the memory of a 47B one. You get the small model’s latency and the big model’s knowledge, and you pay for it in RAM. Tell an interviewer that MoE “saves memory” and you’ve failed the question. It saves compute. The memory bill, if anything, goes up.
| Mixture-of-experts flagship models: total vs active parameters and routing (2023–2025) | ||||
|---|---|---|---|---|
| Model (release) | Total parameters | Active parameters per token | Experts per MoE layer (routed + shared) | Experts activated per token |
| Mixtral 8x7B (2023) | 46.7B | 12.9B | 8 routed, 0 shared | 2 routed |
| DeepSeek-V3 (2024) | 671B | 37B | 256 routed + 1 shared | 8 routed + 1 shared |
| Qwen3-235B-A22B (2025) | 235B | 22B | 128 routed, 0 shared | 8 routed |
| Llama 4 Maverick (2025) | ~400B | ~17B | 128 routed + 1 shared | 1 routed + 1 shared |
One more consequence people miss: the compute savings partly wash out at high batch sizes. Route a single token and you touch 8 experts. Route a batch of thousands, and collectively they hit nearly every expert, so you can’t skip loading any of them, and the sparse model starts to look dense from the GPU’s point of view. MoE buys the most when each expert gets a healthy but not saturating share of the batch.
Routing collapse, and the tricks that stop it
Left alone, routers cheat. Early in training a few experts get slightly better, the router notices, sends them more tokens, they improve faster, and within a few thousand steps the model is funneling almost everything through a handful of experts while the rest starve. That failure is called routing collapse, and handling it is most of what makes MoE training hard.
The classic fix, from the Switch Transformer and GShard work, is an auxiliary load-balancing loss: add a penalty that grows when token assignments are lopsided, pushing the optimizer to spread load. It works, but it fights the real objective, and tuning its weight is finicky. Too strong and you hurt model quality to satisfy a balance target nobody actually cares about.
DeepSeek’s answer, and a reason V3 gets cited so often, is auxiliary-loss-free balancing. Instead of a loss term, they add a per-expert bias to the routing logits and nudge it up or down between steps based on how loaded each expert has been. Overworked experts get a lower bias and shed tokens; idle ones get raised and pull more. The language-modeling loss stays clean. A second lever is expert capacity: cap how many tokens any expert will take per batch, and let overflow tokens skip the layer through the residual connection rather than wait.
Stability is the other tax. Router logits can grow without bound, and once they are large the softmax saturates and gradients back to the router die, so training stalls. The common guard is a router z-loss that penalizes large logits, plus running the router itself in fp32 even when the rest of the model is bf16, since a noisy top-k decision in low precision flips which experts a token sees and adds variance the optimizer then has to fight.
Two design choices show up in most current models. Shared experts, which every token always visits, soak up the common knowledge so the routed experts are free to specialize instead of each relearning the basics. And fine-grained experts, splitting the FFN into many small experts rather than a few fat ones, which gives the router more combinations to assign and tends to improve quality at the same active-parameter budget. DeepSeek-V3’s 256 routed experts plus one shared expert is the canonical example of both.
Serving MoE and the all-to-all tax
At inference the experts usually don’t fit on one GPU, so you shard them across devices, a scheme called expert parallelism. Now routing has a network cost. A token computed on GPU 0 might be routed to an expert living on GPU 3, so every MoE layer runs an all-to-all: each device ships its tokens to whichever device owns their chosen expert, the experts run, and a second all-to-all sends the results home. That communication, not the matrix multiplies, is often the bottleneck, and it is why MoE serving frameworks work so hard to overlap the all-to-all with compute.
Load imbalance bites again here, in a different way. If real traffic favors a few experts, the GPUs holding them become stragglers while the rest idle, and the slowest device sets the latency for the whole batch. Production stacks fight this by replicating hot experts and, sometimes, reshuffling which experts live on which GPU based on observed traffic.
What the questions sound like, and what a strong answer covers
A senior loop rarely asks “explain MoE” and stops. It probes whether you can reason about the tradeoffs under pressure. The phrasings I’ve seen recently:
- “The model card says 671B total, 37B active. What does each of those numbers predict about serving cost?”
- “Your router is sending 80% of tokens to three of thirty-two experts. What went wrong, and what would you change?”
- “Walk one token through a single MoE layer, from hidden state to output.”
- “You have a fixed compute budget per token. Argue for or against MoE versus a dense model of the same active size.”
- “Where does expert parallelism add latency that tensor parallelism doesn’t?”
That last one is a good filter. A dense model sharded with tensor parallelism does an all-reduce within a layer, whose cost is predictable. MoE adds an all-to-all whose cost depends on where tokens happen to route, which is data-dependent and bursty. Candidates who have only read about MoE talk about experts specializing on syntax versus math, which they mostly don’t do in any clean, interpretable way. Candidates who have served one talk about all-to-all bandwidth, capacity factors, and stragglers.
Say the limit out loud, because interviewers like hearing it. MoE wins when you are memory-rich and want more capacity without more compute, which describes a datacenter with plenty of VRAM. On a single consumer GPU where memory is the binding constraint, a dense model of the same footprint usually beats an MoE of the same active size, because you paid for all that VRAM and are only using a slice of it per token. Knowing which regime you are in is the difference between a model that flies and one that quietly wastes half your hardware.
