Back to series
· 12 min read

What Is the MoE Mixture-of-Experts Model? Architecture Principles from Sparse Activation to Expert Routing


1. What Is an MoE (Mixture of Experts) Model?

MoE (Mixture of Experts) is a model architecture composed of multiple expert subnetworks and a routing network.

Its core idea is:

Instead of letting all parameters participate in every computation, it dynamically selects a subset of experts to perform the computation based on the current input.

In a traditional Dense Transformer, when each Token passes through a Transformer layer, it usually goes through the complete feed-forward neural network (FFN).

MoE replaces the FFN with multiple expert networks and uses a routing mechanism to select a subset of them.

For example, an MoE layer contains 8 experts, and each Token activates only 2 of them:

  • Total number of experts: 8.
  • Activated per Token: 2.
  • Activation ratio: 25%.
  • Experts that are not selected do not participate in the expert computation for the current Token.

This means the model can have a larger total parameter count, but each forward pass uses only a subset of those parameters.

Note that MoE does not have the model randomly select a few neural networks; instead, it uses a trainable routing network to dynamically assign computation tasks based on the Token's representation.

Dense Models and MoE Models

Dense Transformer

flowchart TB
    A["Input Token"]
    B["Attention"]
    C["Full FFN"]
    D["Output Representation"]

    A --> B
    B --> C
    C --> D

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#E5F0E9,stroke:#83A995,color:#315D49

MoE Transformer

flowchart TB
    A["Input Token"]
    B["Attention"]
    C["Router Network"]
    D["Expert 1"]
    E["Expert 2"]
    F["Expert 3"]
    G["Expert N"]
    H["Weighted Aggregation"]
    I["Output Representation"]

    A --> B
    B --> C
    C --> D
    C --> E
    C -.-> F
    C -.-> G
    D --> H
    E --> H
    H --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#E5F0E9,stroke:#83A995,color:#315D49
    style E fill:#E5F0E9,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

In the figure, solid lines indicate that the selected experts participate in the computation, while dashed lines indicate experts that are not activated.

The key to MoE is not simply increasing the number of experts, but establishing an effective dynamic routing mechanism so that different Tokens can choose suitable computation paths.


2. Why Do We Need MoE?

The capabilities of large language models are closely related to model capacity, training data, training methods, and computational resources.

As model scale continues to expand, Dense models face a practical problem:

The more model parameters there are, the more parameters each forward pass and backpropagation typically needs to process.

For example, suppose a Dense model has 100B parameters; then when each Token passes through the model, the vast majority of model parameters will participate in the computation.

If we want the model to have greater parameter capacity, the Dense architecture usually has to bear a correspondingly higher computational cost.

MoE offers another way to scale:

  • Add expert networks to expand the model's total parameter scale.
  • Use a routing network to select a small number of experts.
  • Let only the selected experts participate in the expert computation for the current Token.
  • Control the amount of computation per pass through sparse activation.

Key Differences Between Dense and MoE

Comparison Dimension Dense Model MoE Model
Network Structure Single FFN path Multiple expert FFNs
Expert Selection Not involved Dynamic selection by Router
Parameter Activation Usually all activated Only some experts activated
Total Parameter Count More closely tied to compute scale Can expand total parameters while controlling activated parameters
Computational Characteristics Dense computation Sparse computation
Engineering Complexity Relatively simple Routing, communication, and load balancing are more complex
GPU Memory Requirements Depends on model parameters and training state Needs to accommodate all expert parameters and related state

Two concepts need to be distinguished here:

  • Total Parameters: The sum of all trainable parameters in the model.
  • Activated Parameters: The scale of parameters actually involved in computation during one forward pass.

MoE can have a very large total parameter count while the number of activated parameters per pass is relatively small.

However, fewer activated parameters do not mean the model's overall memory usage is also low. During inference, it is usually still necessary to store all expert weights, or manage the weights through distributed expert deployment, on-demand loading, and other methods.


III. Core Components of MoE

A typical Transformer MoE layer mainly consists of three parts:

  1. Router: routing network.
  2. Experts: expert networks.
  3. Aggregation: aggregation of expert outputs.

1. Router: Routing Network

The Router is responsible for determining which experts the current Token should be assigned to.

It is usually a trainable linear projection layer that computes routing scores for each expert based on the hidden state of the input Token.

Assume:

  • The input hidden state dimension is d.
  • The number of experts is N.
  • The Router weight matrix is W_r.

Input hidden state:

h\in\mathbb{R}^{d}

Routing weight matrix:

W_r\in\mathbb{R}^{N\times d}

The Router first computes the expert logits:

z=W_rh

where:

z\in\mathbb{R}^{N}

Then, Softmax is used to obtain the routing probability for each expert:

p_i= \frac{\exp(z_i)} {\sum_{j=1}^{N}\exp(z_j)}

where:

  • z_i: the routing logit of the i-th expert.
  • p_i: the probability that the Token is assigned to the i-th expert.
  • N: total number of experts.

For example, after a Token passes through the Router, we obtain:

Expert Routing Probability
Expert 1 0.08
Expert 2 0.35
Expert 3 0.12
Expert 4 0.28
Expert 5 0.05
Expert 6 0.04
Expert 7 0.05
Expert 8 0.03

If the Top-2 routing mechanism is used, the model selects the two experts with the highest probabilities:

  • Expert 2: 0.35.
  • Expert 4: 0.28.

These two experts participate in the computation for the current Token.

Note that the probabilities output by the Router are assignment weights computed by the routing network; they do not represent the probability that an expert's answer is correct, nor do they imply that an expert has fixed, human-interpretable skills.

2. Experts: Expert Networks

An Expert is a subnetwork that actually performs computation.

In Transformers, the most common MoE implementation replaces the original FFN with multiple independent FFN experts.

Each expert usually has its own parameters.

For example:

E_1(h),E_2(h),\ldots,E_N(h)

Each expert can adopt a standard feedforward neural network structure.

Typical FFN:

FFN(x)=W_2\sigma(W_1x+b_1)+b_2

Where:

  • W_1: up-projection matrix.
  • W_2: down-projection matrix.
  • \sigma: activation function, such as GELU or SiLU.

In MoE, each expert has its own independent FFN parameters:

E_i(x)=W_{2,i}\sigma(W_{1,i}x+b_{1,i})+b_{2,i}

The parameters of different experts are trained independently, but usually under the same model training objective.

It should be emphasized:

Experts are not function modules that are manually designated in advance.

For example, one cannot simply assume:

  • Expert 1 is specifically responsible for mathematics.
  • Expert 2 is specifically responsible for code.
  • Expert 3 is specifically responsible for Chinese.

During training, the Router and the Experts are optimized jointly and may gradually form different patterns of division of labor, but such division is not necessarily stable, independent, or easy to interpret.

3. Aggregation: Aggregating Expert Outputs

After the Router selects multiple experts, each expert computes its output separately, and the model needs to combine these results.

Suppose the Top-2 experts are selected:

E_2(h),E_4(h)

With corresponding routing weights:

g_2,g_4

Then the output can be expressed as:

y=g_2E_2(h)+g_4E_4(h)

Where:

  • E_2(h): the output of the second expert.
  • E_4(h): the output of the fourth expert.
  • g_2,g_4: the routing weights of the corresponding experts.

Different MoE implementations may differ in how they normalize, scale, and combine routing weights.

For example, some implementations renormalize the selected Top-K weights:

g_i= \frac{p_i} {\sum_{j\in TopK}p_j}

This makes the weights of the selected experts sum to 1.

Other implementations keep the original Softmax probabilities, or use some other way of handling routing weights.

Therefore, when understanding MoE, one should not treat any single weight normalization scheme as a fixed rule for all architectures.


4. The Top-K Routing Mechanism: How Does MoE Select Experts?

Top-K Routing is a common MoE routing approach today.

Its core logic is:

Based on the expert probabilities computed by the Router, select the K experts with the highest scores.

Suppose:

N=8,\quad K=2

This means the model has 8 experts, and each token selects 2 experts.

The routing process:

flowchart TB
    A["Token hidden state h"]
    B["Router linear projection"]
    C["Compute expert logits"]
    D["Softmax to obtain routing probabilities"]
    E["Top-K select experts"]
    F["Expert networks compute in parallel"]
    G["Aggregate by routing weight"]
    H["Output result"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

Top-1 vs Top-2

Top-1 Routing

Each token selects only one expert:

y=E_{i^*}(h)

Where:

i^*=\arg\max_i p_i

Advantages:

  • Fewer activated experts.
  • Lower expert computation overhead.
  • Relatively simple routing path.

Disadvantages:

  • Only one expert participates in the computation for each token.
  • When the expert selection is wrong, there is no opportunity for other experts to participate as well.
  • Load balancing and routing stability still need to be handled.

Top-2 Routing

Each token selects two experts:

y=g_1E_{i_1}(h)+g_2E_{i_2}(h)

Where i_1,i_2 are the two experts with the highest routing probabilities.

Advantages:

  • Multiple experts process tokens together.
  • It can combine representations from different experts.
  • In some architectures, it helps improve the model’s expressive capacity.

The cost is increased expert computation and communication.

A larger K in Top-K usually means more experts participate in computation, but it does not necessarily mean better model performance.

Routing Choices Are Not Expert Capability Scores

A common point of confusion is:

The Router probability reflects the routing preference learned by the current model, rather than an objective ranking of experts’ capabilities on a given task.

For example, the fact that Expert 2 has the highest routing probability does not mean that it is stronger than Expert 4 on all tasks.

Routing decisions depend on the hidden state of the current token and on the routing parameters formed during model training.


V. Where Does MoE Fit in a Transformer?

MoE usually does not completely replace the Transformer; instead, it replaces some of the FFN sublayers in the Transformer.

A typical Transformer Block contains:

  1. Self-Attention.
  2. FFN.
  3. Residual connections.
  4. Normalization structures such as LayerNorm or RMSNorm.

MoE architectures usually keep Attention and replace the FFN with an MoE layer.

Dense Transformer Block

flowchart TB
    A["Input Hidden State"]
    B["Normalization"]
    C["Self-Attention"]
    D["Residual Connection"]
    E["Normalization"]
    F["Dense FFN"]
    G["Residual Connection"]
    H["Output Hidden State"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

MoE Transformer Block

flowchart TB
    A["Input Hidden State"]
    B["Normalization"]
    C["Self-Attention"]
    D["Residual Connection"]
    E["Normalization"]
    F["MoE Router"]
    G["Expert 1"]
    H["Expert 2"]
    I["Expert N"]
    J["Expert Output Aggregation"]
    K["Residual Connection"]
    L["Output Hidden State"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    F --> H
    F -.-> I
    G --> J
    H --> J
    J --> K
    K --> L

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F

There is an important detail here:

Not all Transformer layers must use MoE.

Real models can adopt a hybrid design:

  • Some layers use Dense FFN.
  • Some layers use MoE FFN.
  • Attention layers still use the standard attention mechanism.
  • Different MoE layers can have different numbers of experts and different Top-K configurations.

This design allows trade-offs among model capacity, computational cost, and training stability.


6. How Should We Understand MoE's Parameter Count and Compute?

This is one of the most important parts of understanding MoE.

The value of MoE lies in this: the total parameter scale and the per-activation parameter scale can be separated.

1. Dense FFN Parameter Count

Assume:

  • The hidden dimension is d.
  • The FFN intermediate dimension is d_{ff}.
  • Two linear projections are used.

Ignoring bias parameters, the FFN parameter count is:

P_{FFN}=d\cdot d_{ff}+d_{ff}\cdot d

Therefore:

P_{FFN}=2dd_{ff}

2. MoE Expert Parameter Count

Assume a MoE layer contains N experts with identical structure, each of which is the FFN described above.

Then the total expert parameter count is:

P_{Experts}=N\cdot 2dd_{ff}

If there is also a Router, its parameter count is approximately:

P_{Router}=Nd

Ignoring biases and other structural parameters, the parameter count of a MoE layer is approximately:

P_{MoE}=2Ndd_{ff}+Nd

It can be seen that as the number of experts increases, the total parameter count of MoE also increases.

3. MoE Activated Parameter Count

If each Token selects only K experts, then the number of expert parameters activated by a single Token is approximately:

P_{Active}=2Kdd_{ff}

The Router still needs to compute routing scores for all experts. Its compute is usually much smaller than the compute of the expert FFNs, but it cannot be regarded as zero.

Assume:

N=8,\quad K=2

Then:

\frac{P_{Active}}{P_{Experts}}=\frac{2}{8}=25\%

In other words, for the expert portion, each Token activates only about one quarter of the expert parameters.

This does not mean that the activated parameter count of the entire Transformer is exactly 25% of the total parameter count, because Attention, Embedding, normalization, the output layer, and other structures still need to be computed.

4. Parameter Count Example

Assume:

  • Hidden dimension: 4096.
  • FFN intermediate dimension: 14336.
  • Number of experts: 8.
  • Each Token activates 2 experts.

Parameter count of a single expert:

2\times4096\times14336 =117,440,512

This is approximately 117.4M parameters.

Total parameter count of 8 experts:

8\times117.4M\approx939.5M

Number of expert parameters activated by each Token:

2\times117.4M\approx234.9M

The above calculation only counts the MoE expert FFN weights and does not include the Router, Attention, Embedding, output layer, biases, and other parameters.

This example shows:

MoE can expand model capacity by increasing the number of experts, while letting each Token use only some of the experts.

However, it cannot be directly inferred from this that the actual inference latency of MoE is one quarter that of a Dense model. Expert dispatch, communication, load imbalance, memory access, and scheduling overhead all affect real performance.


7. The Key Difficulty of MoE: Load Balancing

MoE appears to only need to select a few experts, but in actual training there is a very important problem:

The number of Tokens received by different experts may be severely imbalanced.

For example, a MoE layer has 8 experts, and each Token selects 2 experts.

Ideally, each expert would be allocated a relatively balanced number of Tokens.

But in actual training, the Router may gradually favor a small number of experts:

Expert Number of Tokens Received
Expert 1 800
Expert 2 720
Expert 3 30
Expert 4 20
Expert 5 15
Expert 6 10
Expert 7 5
Expert 8 0

This creates several problems:

  • Popular experts are overloaded and become a compute bottleneck.
  • Unpopular experts lack training samples and are difficult to train sufficiently.
  • The amount of computation on different devices is imbalanced.
  • Inter-expert communication and scheduling efficiency decrease.
  • Model capacity is not fully utilized.

This phenomenon is usually called Expert Load Imbalance.

1. Why Does Load Imbalance Occur?

The Router and Experts are trained jointly.

When certain experts are selected more frequently, they receive more training samples and their parameters are updated more fully.

This may further reinforce the Router's preference for these experts.

This creates a positive feedback loop:

flowchart TB
    A["Router favors a few experts"]
    B["A few experts receive more tokens"]
    C["Popular experts receive more training"]
    D["Training opportunities are uneven among experts"]
    E["Routing allocation becomes further concentrated"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> A

    style A fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

This is not an inevitable outcome of all MoE training, but without appropriate load control mechanisms, routing concentration and reduced expert utilization may occur.

2. Auxiliary Load Balancing Loss

A classic method is to add an auxiliary load balancing loss.

Through an additional training objective, it encourages the Router to distribute tokens relatively evenly among different experts.

A common form is:

L_{aux}=\alpha N\sum_{i=1}^{N}f_iP_i

Where:

  • N: number of experts.
  • f_i: proportion of tokens actually assigned to expert i.
  • P_i: average routing probability of the Router for expert i.
  • \alpha: weight of the auxiliary loss.

Where:

f_i=\frac{1}{TK}\sum_{t=1}^{T} \mathbf{1}[i\in TopK(t)]

Here:

  • T: number of tokens in the current statistical scope.
  • K: number of experts selected per token.
  • \mathbf{1}: indicator function indicating whether an expert is selected by the current token.

Average routing probability:

P_i=\frac{1}{T}\sum_{t=1}^{T}p_{t,i}

When token allocation is relatively balanced, the distributions of f_i and P_i are usually not overly concentrated.

Note that this is a classic form of auxiliary loss; not all MoE models use exactly the same formula.

For example, some architectures use other balancing objectives, routing constraints, or load control methods that do not rely on traditional auxiliary losses.

3. Expert Capacity

In addition to auxiliary loss, we also need to consider how many tokens each expert can receive.

Suppose a batch contains T tokens, each token selects K experts, and the number of experts is N.

The average number of tokens assigned to each expert is approximately:

C_{avg}=\frac{TK}{N}

Real systems usually set an expert capacity, allowing each expert to receive up to a certain number of tokens.

For example: