I. What Is FFN?
FFN (Feed-Forward Network) is a neural network structure in which information propagates in a single direction.
In a Transformer, FFN usually refers to the Position-wise Feed-Forward Network, which acts independently on each Token representation in the sequence and applies a nonlinear feature transformation to it.
The FFN in a Transformer typically consists of two linear transformations and a nonlinear activation function:
Where:
- x: the input feature vector.
- W_1: the weight matrix of the first linear transformation.
- b_1: the bias of the first layer.
- \sigma: the nonlinear activation function.
- W_2: the weight matrix of the second linear transformation.
- b_2: the bias of the second layer.
The core role of the FFN is: to map input features into a higher-dimensional hidden space, extract and combine features through nonlinear transformation, and then map them back to the original model dimension.
Two concepts need to be distinguished:
- Feed-Forward Neural Network: a broad network structure in which information propagates unidirectionally from the input layer to the output layer.
- Transformer FFN: the specific position-wise feed-forward network used inside a Transformer block, typically adopting two linear layers and a nonlinear activation function.
II. Where FFN Sits in a Transformer
A Transformer does not rely on Attention alone to complete all computation.
Attention is mainly responsible for information exchange between different Tokens, whereas the FFN is mainly responsible for independently transforming the features of each Token.
In a typical Transformer Encoder Layer, the FFN is located after the Self-Attention sublayer.
The overall structure can be represented as:
flowchart TB
A["Input Sequence Representation"] --> B["Multi-Head Self-Attention"]
B --> C["Residual Connection and LayerNorm"]
C --> D["FFN Feed-Forward Network"]
D --> E["Residual Connection and LayerNorm"]
E --> F["Output Sequence Representation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
What is shown here is the structural order of the Post-LN form. Actual models may also use Pre-LN, placing LayerNorm before the sublayer computation.
The division of labor between FFN and Attention:
| Component | Main Role | Scope of Information Interaction |
|---|---|---|
| Self-Attention | Establishes information associations between different Tokens | Between different Tokens |
| FFN | Performs nonlinear transformation and combination of features | Within a single Token |
| LayerNorm | Normalizes the feature distribution | The feature dimension of a single Token |
| Residual Connection | Preserves input information and improves the optimization process | Between sublayer input and output |
Attention and FFN are not substitutes for each other; rather, they take on different computational responsibilities.
III. The Basic Structure of FFN
Taking a typical Transformer FFN as an example:
- Model dimension: d_{model}=768
- FFN hidden dimension: d_{ff}=3072
- First layer: 768 dimensions mapped to 3072 dimensions.
- Activation function: GELU.
- Second layer: 3072 dimensions mapped back to 768 dimensions.
The structure is as follows:
flowchart TB
A["Input features x<br/>768 dimensions"] --> B["Linear transformation<br/>768 → 3072"]
B --> C["GELU activation function"]
C --> D["Hidden features h<br/>3072 dimensions"]
D --> E["Linear transformation<br/>3072 → 768"]
E --> F["Output features y<br/>768 dimensions"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
3.1 First Linear Transformation: Dimension Expansion
The first layer maps input features from the model dimension to a larger hidden dimension:
Suppose:
Then:
Finally:
The main role of this step is to expand the feature representation space, enabling the network to combine and transform input features in a higher-dimensional space.
Increasing the dimension does not mean simply increasing the amount of information; rather, it provides the model with a larger learnable representation space.
3.2 Non-linear Activation: Introducing Expressive Power
After completing the first linear transformation, nonlinearity needs to be introduced through an activation function:
Common activation functions include:
- ReLU
- GELU
- SiLU / Swish
- The SiLU gated branch used in SwiGLU
Why is an activation function needed?
If there is no nonlinear activation between the two linear transformations, then:
can be combined into a single linear transformation:
Even if more linear layers are added, as long as there is no nonlinear operation in between, the whole is still equivalent to a single linear transformation.
Therefore, the nonlinear activation function is key to enabling the FFN to acquire complex feature representation capabilities.
3.3 Second Linear Transformation: Dimension Reduction
The second layer maps the hidden features back to the model dimension:
where:
Finally:
The purpose of dimension reduction is mainly to keep dimensions consistent within the Transformer module, so that the FFN output can be added to the input through the residual connection.
Note: Dimension reduction is not simply information compression. The second layer recombines features in the hidden space according to the trained parameters, generating representations suitable for subsequent network processing.
IV. Mathematical Computation Process of FFN
Combining the above process yields the standard FFN formula:
Here, row vector notation is used.
If column vector notation is used, it is written as:
The two forms differ only in matrix multiplication convention and express the same computation process.
4.1 Computation for a Single Token
Suppose the input vector of a token is:
First linear transformation:
resulting in:
After GELU:
Second linear transformation:
resulting in:
The entire process can be understood as:
- Input features enter a high-dimensional hidden space.
- The neural network learns the combinatorial relationships among different features.
- The activation function introduces nonlinearity.
- The second layer recombines the hidden features.
- A new token representation is output.
4.2 Computation for Batched Tokens
In practice, a Transformer processes not a single token but multiple tokens in a sequence.
Suppose the input tensor is:
where:
- B: Batch size.
- N: Sequence length.
- d_{model}: Model hidden dimension.
For example:
The FFN performs the same computation separately for each token:
Where:
Output:
As you can see, the input and output tensor shapes are exactly identical.
But this does not mean the FFN leaves the features unchanged. What it changes is the feature representation inside each token vector, not the number of tokens or the model dimension.
V. Why Does the FFN Compute Position-wise?
An important characteristic of the Transformer FFN is that it is position-wise, that is, it computes at each position separately.
For a sequence:
The FFN computes as follows:
All tokens use the same set of FFN parameters, but each completes its forward computation independently.
5.1 Parameter Sharing
Suppose the sequence contains 128 tokens.
The FFN does not create a separate set of weight matrices for each token; instead, all positions share:
This keeps the number of model parameters under control while ensuring that the feature transformations at different positions follow the same rules.
5.2 No Direct Information Exchange Between Positions
Inside the FFN:
When computing y_i, it does not directly read x_j from other positions.
Therefore:
The distinction here is very important:
- Attention: different positions can exchange information.
- FFN: each position performs its feature transformation independently.
However, one cannot conclude from this that the FFN knows nothing about context.
Because the FFN's input x_i has usually already been processed by Attention and contains context information from other tokens.
In other words, although the FFN does not directly interact across positions, the features it processes may already have fused information from the entire context.
5.3 Engineering Advantages of Position-wise Computation
A position-wise FFN has several clear advantages:
- Parameter sharing: different tokens do not need their own FFN parameters.
- Parallel computation: computations at different positions can be executed simultaneously.
- Batched matrix multiplication: the computations for multiple tokens can be organized into efficient matrix operations.
- Simple structure: easy to optimize for GPUs and to fuse operators.
VI. Why Does the FFN Usually Use a 4× Dimension Expansion?
Classic Transformers often use:
For example:
| Model dimension d_{model} | FFN hidden dimension d_{ff} |
|---|---|
| 256 | 1024 |
| 512 | 2048 |
| 768 | 3072 |
| 1024 | 4096 |
| 2048 | 8192 |
This 4× ratio comes from the classic Transformer architecture design; it is a common configuration, not a constraint that must be satisfied in theory.
6.1 Why Is a Larger Hidden Dimension Needed?
The model dimension carries the contextual representation of tokens, while the FFN hidden dimension provides a larger intermediate feature space.
From a computational perspective:
A larger hidden dimension means:
- More learnable combinations of intermediate features.
- A larger space for nonlinear transformation.
- Higher FFN parameter capacity.
- A larger matrix multiplication workload.
But a larger hidden dimension is not always better.
Increasing the hidden dimension brings more parameters, more computation, and higher memory requirements, and may also raise training costs.
Real models usually need to balance model capacity, compute budget, and inference latency.
6.2 Why Does the FFN Have So Many Parameters?
Ignoring the bias terms, the parameter count of a standard two-layer FFN is:
Layer 1:
Layer 2:
Total parameter count:
When:
Then:
If:
Then:
About 4.72 million weight parameters.
Adding the biases of the two layers:
The total number of FFN parameters is about 4.722 million.
This also explains why, in many Transformer models, the FFN is an important component of the parameter count.
7. The Essential Difference Between FFN and Attention
Attention and FFN are among the most important computational modules in Transformer.
They solve different problems.
7.1 Attention: Information Exchange Between Different Tokens
Self-Attention computes attention weights from Query and Key, and then performs weighted aggregation over Value:
Its main role is to let a Token selectively aggregate information from other Tokens based on context.
For example:
Sentence:
Xiaoming put the apple in the refrigerator because it was already ripe.
The model needs to understand from context which object "it" refers to.
Attention can help the model integrate contextual information through association weights between different Tokens.
7.2 FFN: Feature Transformation Within a Single Token
The FFN is not responsible for directly computing attention relationships between Tokens.
It receives the features output by Attention and applies a nonlinear mapping to each position:
Its main roles are:
- Combine features from different dimensions.
- Extract more complex nonlinear patterns.
- Enhance the representational capability of Tokens.
- Convert contextual information into new feature representations.
7.3 How Do the Two Work Together?
flowchart TB
A["Input Token Representation"] --> B["Self-Attention"]
B --> C["Cross-Token Information Interaction"]
C --> D["Context-Enhanced Feature Representation"]
D --> E["FFN"]
E --> F["Per-Token Nonlinear Feature Transformation"]
F --> G["Updated Token Representation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
It can be summarized in one sentence:
Attention is responsible for passing and aggregating information between Tokens, while the FFN is responsible for applying nonlinear processing to the aggregated features.
Together, the two form the core computational mechanism of Transformer.
8. Common Activation Functions in the FFN
The activation function determines how the FFN introduces nonlinearity, and different activation functions affect the model's gradient propagation, computational efficiency, and representational capacity.
8.1 ReLU
ReLU (Rectified Linear Unit) is defined as:
Characteristics:
- Simple computation.
- The gradient is 1 in the positive region.
- The output is 0 in the negative region.
- Neurons may remain in the negative region for a long time with zero gradient, i.e., the dying ReLU problem.
ReLU was widely used in early neural networks and some Transformer variants.
8.2 GELU
GELU (Gaussian Error Linear Unit) is defined as:
where:
\Phi(x) is the cumulative distribution function of the standard normal distribution.
GELU can be understood as a smooth, gated activation: instead of being hard-clipped at zero as in ReLU, the input value is smoothly adjusted according to its magnitude.
A common approximation formula:
Characteristics:
- The function is smooth.
- It still retains some output in the negative region.
- The gradient changes continuously.
- Used in some classic implementations of models such as BERT and GPT.
8.3 SiLU / Swish
SiLU is also called Swish:
where:
Characteristics:
- Smooth and continuous.
- Allows some negative values to pass through.
- Has gating-style modulation characteristics.
- Commonly used in modern neural network architectures.
8.4 Comparison of Activation Functions
| Activation Function | Mathematical Expression | Main Characteristics |
|---|---|---|
| ReLU | \max(0,x) | Simple and efficient; output is zero on the negative half-axis |
| GELU | x\Phi(x) | Smooth nonlinearity; a classic choice for Transformers |
| SiLU | x\sigma(x) | Smooth gating; commonly used in modern architectures |
| SwiGLU | Swish(xW)\odot xV | Introduces a gating branch, enhancing feature selection capability |
It should be noted that the choice of activation function depends on model architecture, training configuration, and hardware implementation; there is no absolutely optimal choice that applies to all models.
9. FFN Variants in Modern Transformers
The classic FFN uses two linear transformations and one activation function.
With the development of large language models, researchers began using gated structures to enhance the expressive power of FFN.
Representative examples include GLU, GEGLU, and SwiGLU.
9.1 GLU: Gated Linear Unit
GLU (Gated Linear Unit) generates different feature branches through two linear projections:
where:
- One branch provides feature content.
- The other branch generates the gating signal.
- \odot denotes element-wise multiplication.
The gating signal can adjust how much different features in the other branch pass through.
Compared with a standard FFN, GLU does more than apply a fixed-form nonlinear transformation to the input; it also adds a learnable feature modulation mechanism.
9.2 GEGLU
GEGLU (GELU-Gated Linear Unit) replaces the gating activation function in GLU with GELU: