- Published on
Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
- Authors
- Name
- DeepHorus
Dense and Mixture-of-Experts (MoE) models organize parameters differently, affecting throughput, memory, and serving complexity. A dense model activates every parameter for every token; an MoE model stores multiple expert networks and routes each token through a selected subset. NVIDIA's Nemotron 3.5 Lightning shows the trade-off: a 30B-parameter model activates only 3B parameters per token while using the larger model's capacity.
In a dense model, all parameters participate in every forward pass. A 27B dense model fires all 27B parameters per token through one shared feed-forward network (FFN) per decoder layer. An MoE model replaces that single FFN with multiple FFN blocks, or experts, often 8, 64, or 128.
A learned router assigns each token to the top k scoring experts and skips the rest for that layer. Most modern MoEs, such as Mistral Small 4, also run one shared expert that every token uses. Routing is layer-specific: tokens are rerouted at each decoder layer based on their evolving representation.
These experts do not necessarily specialize in subjects like medicine or law. Their specialization often lies in syntax and token-type patterns, such as punctuation or numbers, though this varies by architecture and training. Tokens still pass through full attention, and "3B active parameters" includes attention and embedding weights plus selected FFN weights.
MoE models are often faster for token throughput because they activate only a subset of feed-forward parameters. However, they can demand more memory and more complex serving infrastructure. Variants add nuance: NVIDIA's model card for Nemotron 3.5 Lightning specifies a Mamba-2 + MoE + Attention hybrid.
Mamba-2 layers replace attention in most layers and carry a constant-size recurrent state instead of a growing KV cache, changing long-context memory beyond sparsity alone. Neither architecture is universally better. Dense models favor simpler deployment, while MoE models can deliver greater capacity and throughput when memory and serving complexity are manageable.
