Back to series
· 9 min read

Dense vs. MoE Models: An Architectural Comparison from Parameter Activation to Computational Efficiency


1. Dense vs. MoE: Two Different Model Architectures

Dense models and MoE models are two important network architecture designs in large language models.

Their core difference lies in whether all parameters in the model participate in every forward computation.

1. What Is a Dense Model?

Dense models are usually called dense models.

In a Dense Transformer, each layer typically contains:

  • Self-Attention: handles information exchange between tokens.
  • FFN: performs a nonlinear transformation on each token's representation.
  • Residual connections and normalization: help the model train stably.

When a token passes through a Dense Transformer Block, it usually goes through the full Attention and FFN computation.

Suppose the model has 70B parameters; then under a standard Dense architecture, the vast majority of model parameters participate in the forward computation for every token.

Here, "fully participate" refers to the model's dense parameter path; it does not mean that every parameter has the same degree of influence on every input.

2. What Is a MoE Model?

MoE (Mixture of Experts) replaces some FFNs with multiple independent expert networks.

After each token passes through the router network (Router), only a subset of the experts is selected to participate in the computation.

For example:

  • Total number of experts: 8.
  • Activated per token: 2 experts.
  • Each token passes through only 25% of the experts.

Therefore, MoE can have a large total parameter count while activating only some expert parameters each time.

3. Computation Flow of the Two Architectures

Dense model:

flowchart TB
    A["Input Token"]
    B["Embedding"]
    C["Self-Attention"]
    D["Full FFN"]
    E["Output Hidden State"]

    A --> B
    B --> C
    C --> D
    D --> E

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#E5F0E9,stroke:#83A995,color:#315D49

MoE model:

flowchart TB
    A["Input Token"]
    B["Embedding"]
    C["Self-Attention"]
    D["Router"]
    E["Expert 1"]
    F["Expert 2"]
    G["Expert 3"]
    H["Expert N"]
    I["Expert Output Aggregation"]
    J["Output Hidden State"]

    A --> B
    B --> C
    C --> D
    D --> E
    D --> F
    D -.-> G
    D -.-> H
    E --> I
    F --> I
    I --> J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#E5F0E9,stroke:#83A995,color:#315D49
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

In the figure, solid lines indicate the selected experts, and dashed lines indicate experts that are not activated.

Note that MoE usually replaces only some of the FFN layers in the Transformer; it does not turn the entire Transformer into a set of unrelated expert networks.


2. Core Differences Between Dense and MoE

Let us first compare the two architectures as a whole.

Comparison dimension Dense model MoE model
Chinese name Dense model Mixture of Experts model
Network structure Single dense computation path Multiple sparse expert computation paths
FFN structure One complete FFN Multiple expert FFNs
Parameter activation Usually all activated Only some experts activated
Routing mechanism No expert routing needed Requires a Router
Total parameter count Parameter scale is tightly coupled with compute scale Total parameters can be expanded while keeping the activated scale under control
Compute characteristics Dense matrix computation Sparse expert computation
Load balancing No expert assignment involved Load balancing required
Communication overhead Conventional model-parallel communication May add All-to-All communication between experts
Memory requirements Depends on model parameters and training state Must manage all expert parameters and training states
Engineering complexity Relatively simple Relatively complex
Inference performance Regular computation path Affected by routing, communication, and expert load

The most important thing is to distinguish three concepts:

  • Total model parameter count: the number of all trainable parameters in the model.
  • Activated parameter count: the number of parameters that actually take part in the computation during a single forward pass.
  • Compute (FLOPs): the number of floating-point operations required to perform one forward or backward computation.

These three metrics are not equivalent.

This is especially true for MoE: you cannot simply use the activated parameter count in place of the total parameter count, nor can you directly use the ratio of activated parameters to estimate actual inference speed.


3. The difference in parameter count: why can MoE have more parameters?

One of the most important differences between Dense and MoE is the relationship between the total parameter count and the activated parameter count.

1. Parameter count of a Dense model

Suppose a Dense FFN contains two linear transformations:

FFN(x)=W_2\sigma(W_1x)

Where:

  • The input dimension is d.
  • The intermediate layer dimension is d_{ff}.
  • W_1\in\mathbb{R}^{d_{ff}\times d}.
  • W_2\in\mathbb{R}^{d\times d_{ff}}.

Ignoring bias parameters, the FFN parameter count is:

P_{FFN}=2dd_{ff}

Every token passing through this FFN performs both sets of linear transformations.

Therefore, all weights of this FFN take part in the computation.

2. Parameter count of an MoE model

Suppose the MoE contains N experts with identical structure, each of which is an FFN.

Then:

P_{MoE}=N\cdot 2dd_{ff}+P_{Router}

The Router's parameter count is usually far smaller than the total expert parameter count.

If each token selects K experts, then the activated parameter count of the expert portion is:

P_{Active}=K\cdot 2dd_{ff}

The ratio of the two is:

\frac{P_{Active}}{P_{Experts}}=\frac{K}{N}

Assume:

N=8,\quad K=2

Then:

\frac{P_{Active}}{P_{Experts}}=25\%

This means each token activates only about 25% of the parameters in the expert portion.

3. A concrete example

Suppose:

  • Hidden dimension: 4096.
  • FFN intermediate dimension: 14336.
  • Number of experts: 8.
  • Each token activates 2 experts.

Parameter count of a single expert:

P_{Expert}=2\times4096\times14336
P_{Expert}\approx117.4M

Total parameter count of the 8 experts:

P_{Experts}=8\times117.4M
P_{Experts}\approx939.5M

Expert parameter count activated per token:

P_{Active}=2\times117.4M
P_{Active}\approx234.9M

This counts only the weights of the expert FFNs and does not include the Router, Attention, Embedding, output layer, biases, and other parameters.

Parameter scale comparison

Total expert parameters 939.5M {#each Array.from({length:8},(_,i)=>i) as i}<box flex="1" background={i<8?"#83A995":"surface-tertiary"}/>{/each} Expert parameters activated per Token 234.9M 2 / 8 experts participate in the computation

This example illustrates the key advantage of MoE:

Expand model capacity by increasing the number of experts, without requiring every Token to compute all experts.

But note that a complete model also includes shared parameters and other computational modules, so the ratios above cannot be taken directly as the activated-parameter ratio of the whole model.


IV. The Difference in Compute: Fewer Activated Parameters Does Not Mean Compute Drops Proportionally

To understand Dense and MoE, it is essential to further distinguish parameter count from compute.

1. Compute in a Dense Model

For a standard Dense FFN, the main computation comes from two matrix multiplications.

The main compute for a single Token is approximately:

FLOPs_{Dense}\approx4dd_{ff}

Here, one multiplication and one addition are counted as two FLOPs.

Actual compute is also affected by the activation function, biases, gating structure, and the specific FFN implementation.

2. Compute in an MoE Model

Suppose there are N experts and each Token activates K experts.

Then the main compute of the expert FFNs is approximately:

FLOPs_{MoE}\approx4Kdd_{ff}

Therefore, relative to all FFN parameters across N experts, the per-Token expert compute ratio is approximately:

\frac{FLOPs_{MoE}}{FLOPs_{AllExperts}} \approx\frac{K}{N}

But this only compares the expert FFN portion.

MoE also needs to perform additional operations:

  • Router linear projection.
  • Softmax and Top-K selection.
  • Token dispatch.
  • Expert output aggregation.
  • Cross-device communication.

So the actual computational cost cannot be judged by expert FFN FLOPs alone.

3. Comparing the Computational Cost of Dense and MoE

Component Dense MoE
Attention Normal computation Normal computation
FFN Full FFN Selected expert FFNs
Router None Extra computation
Top-K selection None Extra computation
Token dispatch No expert dispatch May be required
Output aggregation No expert aggregation Required
Cross-device communication Regular communication May add All-to-All

Pay particular attention to the following:

If you are comparing a single Dense FFN with an MoE layer containing multiple complete FFN experts, the two may inherently have different parameter scales.

Therefore, one cannot simply say that MoE always requires less compute than Dense.

A more accurate way to compare is:

  • Compare activated compute under the same total model parameter count.
  • Compare model capacity under the same activated parameter count.
  • Compare actual throughput and latency under the same hardware resources.
  • Compare final model quality under the same training budget.

V. The Difference in Memory: Why Doesn't MoE Necessarily Save Memory?

This is where Dense and MoE are most easily misunderstood.

Having fewer activated parameters does not mean the model only needs to store the activated parameters.

1. Weight Memory During Inference

A Dense model typically needs to store all model parameters.

An MoE model must also be able to access all expert parameters, because different Tokens may be routed to different experts.

Suppose:

  • Total parameter count of the Dense model: 70B.
  • Total parameter count of the MoE model: 200B.
  • Activated parameter count per Token in MoE: 40B.

Then, in a deployment mode where all experts reside on devices:

  • Dense needs to store weights for approximately 70B parameters.
  • MoE needs to store weights for approximately 200B parameters.

Although MoE activates only 40B parameters per Token, weight storage still needs to cover all experts.

In actual deployment, expert parallelism can distribute weights across multiple GPUs, but this does not magically reduce the model parameters that the entire cluster needs to store.

2. GPU memory during training

Training GPU memory includes not only model parameters but also:

M_{Total}=M_{Param}+M_{Grad}+M_{Optimizer}+M_{Activation}+M_{Other}

Where:

  • M_{Param}: model parameters.
  • M_{Grad}: gradients.
  • M_{Optimizer}: optimizer states.
  • M_{Activation}: activations saved during forward computation.
  • M_{Other}: communication buffers, temporary tensors, etc.

Because MoE has more expert parameters, training also needs to account for the gradients and optimizer states corresponding to all expert parameters.

Of course, techniques such as parameter sharding, mixed precision, optimizer state offloading, and activation checkpointing can change the actual GPU memory requirement per card.

3. GPU memory comparison

Dimension Dense MoE
Weight storage Full Dense parameters Usually needs to manage all expert parameters
Activations Related to the dense computation path Related to the actually activated experts and routing
Gradients Covers model training parameters Covers all expert parameters involved in training
Optimizer states Related to model parameter scale Related to the scale of all expert parameters
Communication cache Regular communication May increase expert dispatch cache
Distributed deployment Regular model partitioning Often combined with expert parallelism

Conclusion:

The main advantage of MoE is controlling the scale of expert activation in a single computation, rather than inherently reducing the parameter storage requirement of the entire model.


VI. Difference in inference speed: Is Dense necessarily slower than MoE?

Inference speed cannot be judged based only on parameter count or activated parameter count.

Actual inference performance depends on multiple factors such as computation, memory access, communication, and scheduling.

1. Characteristics of Dense inference

The computation path of a Dense model is relatively regular:

  • Each layer performs Attention.
  • Each layer executes the full FFN.
  • The matrix computation structure is relatively fixed.
  • It is easy for the GPU to form regular batched matrix operations.

This structure is conducive to mature GPU operator optimization and computation scheduling.

2. Characteristics of MoE inference

MoE needs to perform routing first, then assign Tokens to different experts.

If experts are distributed across multiple GPUs, cross-device communication is also required.

Therefore, the inference process of MoE is more complex.

flowchart TB
    A["Receive input Tokens"]
    B["Router computes routing"]
    C["Select Top-K experts"]
    D["Regroup Tokens by expert"]
    E["Dispatch across GPUs"]
    F["Experts perform computation"]
    G["Expert outputs returned"]
    H["Restore Token order and aggregate"]
    I["Proceed to next layer"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

These extra steps may offset some of the advantages brought by sparse computation.

3. Difference between Prefill and Decode

Large language model inference is usually divided into two stages.

Prefill stage

The model processes a large number of Tokens in the Prompt at once.

For example, an input contains 4096 tokens, and the Router needs to select experts for these tokens.

In this case, a relatively large expert computation batch can usually be formed, which benefits matrix computation and device utilization.

Decode Phase

The model generates tokens step by step.

At each step, it usually generates only one token or a small number of tokens.

This can lead to:

  • Experts receive fewer tokens.
  • The expert matrix computation batch is smaller.
  • GPU compute utilization decreases.
  • Communication and scheduling overhead takes up a larger share.

Therefore, the performance of the same MoE model can differ significantly between the Prefill and Decode phases.

4. What Does Inference Speed Depend On?

Factor Dense MoE
Matrix computation efficiency Usually more regular Depends on expert batch size
GPU utilization Relatively easy to optimize Affected by routing distribution
Communication overhead Conventional parallel communication May have higher expert communication overhead
Memory access Relatively contiguous Expert weight access may be scattered
Token scheduling Relatively simple Requires routing and regrouping
Decode latency The computation path is relatively stable May be affected by communication and small batches

Therefore:

  • Dense is not necessarily slower than MoE.
  • MoE is not necessarily faster than Dense.
  • MoE having fewer activated parameters does not mean that its actual latency is necessarily lower.

Benchmarking must be performed with the specific hardware, model configuration, batch size, and inference framework.


7. Differences in Training Difficulty: Why Is MoE More Complex?

The training objective of a Dense model is usually to directly optimize the model's prediction loss.

In addition to optimizing the model itself, MoE also needs to optimize expert routing and expert load distribution.

1. Dense Model Training

Take an autoregressive language model as an example:

L_{Dense}=-\sum_{t=1}^{T}\log P(x_t|x_{<t})

Training process:

flowchart TB
    A["Input Token"]
    B["Dense Transformer"]
    C["Predict Next Token"]
    D["Compute Cross Entropy Loss"]
    E["Backpropagation"]
    F["Optimizer Updates Parameters"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49

2. MoE Model Training

MoE likewise needs to optimize the language model prediction loss, but it also needs to handle:

  • Router parameter training.
  • Expert parameter training.
  • Expert load balancing.
  • Expert capacity constraints.
  • Token dispatch and aggregation.
  • Cross-device gradient synchronization.

A common training objective can be written as:

L_{Total}=L_{LM}+\lambda L_{aux}

Where:

  • L_{LM}: the main language modeling task loss.
  • L_{aux}: the auxiliary load-balancing loss.
  • \lambda: the auxiliary loss weight.

Different MoE architectures may adopt different load-balancing mechanisms, and they do not necessarily all use the traditional Auxiliary Loss.