1. Why Does Transformer Need Residual Connections and LayerNorm?
A Transformer is usually composed of multiple stacked Encoder or Decoder Layers. As network depth increases, the model faces two important issues:
- Gradient propagation in deep networks: When gradients are backpropagated through a multi-layer network, they may gradually become smaller or larger, making training difficult.
- Numerical stability of intermediate representations: The feature distributions of different layers may change, affecting subsequent computation and optimization.
Residual Connections and LayerNorm (Layer Normalization) address these problems from different perspectives.
| Mechanism | Core Role | Main Problem Solved |
|---|---|---|
| Residual Connection | Adds the sublayer input directly to the sublayer output | Improves gradient propagation and preserves original information |
| LayerNorm | Normalizes the feature dimension of a single sample | Stabilizes feature value distribution and improves optimization |
| Combination of both | Maintains information and numerical stability in deep networks | Improves trainability of deep Transformers |
Note: Residual connections and LayerNorm are not the same operation.
- Residual connections change the information flow path of the network.
- LayerNorm changes the numerical distribution of features.
- The two can be combined, but the order of combination affects the model's training behavior.
2. What Is a Residual Connection?
2.1 Basic Concept
A Residual Connection, also called a Skip Connection, is a structure that passes the network input directly to subsequent layers and adds it to the output of the current sublayer.
The computation form of an ordinary neural network:
After adding a residual connection:
Where:
- x: the sublayer input.
- F(x): the transformation performed by the sublayer.
- y: the output after the residual connection.
Here, F can be Attention, FFN, or another network module.
The key to a residual connection is that the network no longer relies solely on the sublayer to learn the complete input-to-output mapping; instead, it learns the change relative to the input.
Therefore:
This is also the origin of the name Residual.
2.2 Computation Process of a Residual Connection
flowchart TB
A["Input x"] --> B["Sublayer F(x)"]
B --> C["Sublayer output"]
A --> D["Skip connection"]
C --> E["Element-wise addition"]
D --> E
E --> F["Output y = x + F(x)"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
A residual connection contains two paths:
- Main path: the input passes through the network sublayer to produce transformed features.
- Skip path: the input bypasses the sublayer and is passed directly to the addition operation.
The two paths are finally fused by element-wise addition.
2.3 Why Can Residual Connections Improve Gradient Propagation?
This is the most important mathematical principle for understanding residual connections.
Assume:
By the chain rule:
As can be seen, the gradient consists of two parts:
A standard network:
Its gradient is:
If the gradients of a multi-layer network keep shrinking, the gradients may gradually decay.
A residual network adds an identity mapping path, allowing gradients to propagate directly through the skip connection.
For a multi-layer residual structure:
The gradient from layer l to layer L can be expressed as:
Where:
- I: the identity matrix.
- J_{F_i}: the Jacobian matrix of the transformation function of layer i.
The identity term provides an additional propagation path for gradients.
However, this must be understood accurately: residual connections do not guarantee that gradients will never vanish or explode. They improve the conditions for gradient propagation and make deep networks easier to optimize, but training stability is still affected by factors such as network architecture, initialization, normalization, and learning rate.
2.4 Why do residual connections make networks easier to learn?
Suppose a network can already produce a good input representation, but a newly added network layer temporarily cannot learn an effective transformation.
Without a residual connection, that layer needs to directly learn an appropriate mapping:
If a residual connection is added, the network can first make:
Then:
In other words, even if the current sublayer has not learned effective new features, it can approximately preserve the input representation.
Subsequent training only needs to gradually learn valuable feature changes.
This reduces the difficulty of optimizing deep networks.
3. Applications of Residual Connections in Transformer
Transformer does not establish only one residual connection between the input and output of the entire Layer; instead, it usually establishes a separate residual connection around each main sublayer.
Taking a Transformer Encoder Layer as an example, it contains two main sublayers:
- Multi-Head Self-Attention.
- Feed-Forward Network.
Each sublayer is equipped with a residual connection.
3.1 Residual structure in the Encoder
Using the Post-LN form as an example:
The structure is as follows:
flowchart TB
A["Input X"] --> B["Multi-Head Self-Attention"]
B --> C["Attention output"]
A --> D["Residual path"]
C --> E["Element-wise addition"]
D --> E
E --> F["LayerNorm"]
F --> G["Intermediate representation H1"]
G --> H["Feed-Forward Network"]
H --> I["FFN output"]
G --> J["Residual path"]
I --> K["Element-wise addition"]
J --> K
K --> L["LayerNorm"]
L --> M["Encoder Layer output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
As can be seen:
- The Attention sublayer has its own residual connection.
- The FFN sublayer also has its own residual connection.
- LayerNorm is applied after each addition.
This means each sublayer can learn new feature transformations while preserving the original information.
3.2 Why do residual connections require consistent dimensions?
The basic operation of a residual connection is element-wise addition:
Therefore, it requires:
For example:
Then the sublayer output must also satisfy:
The Attention output in Transformer usually passes through an output projection to restore it to the model dimension.
FFN, on the other hand, first increases the dimension and then maps back to the model dimension through a second linear transformation.
That is how it can be added to the original input.
If the two tensors have different dimensions, standard element-wise residual addition cannot be performed directly.
In other network architectures, the dimensions can be adjusted through means such as linear projection, but that is no longer a simple identity residual path.
IV. What Is LayerNorm?
4.1 Basic Concepts
LayerNorm (Layer Normalization) is a normalization technique used to standardize the features in a neural network.
In a Transformer, LayerNorm typically computes the mean and variance along the feature dimension of each token.
Suppose the feature vector of a token is:
First, compute the mean:
Then compute the variance:
Next, perform standardization:
Finally, apply a transformation using learnable scaling and shifting parameters:
Where:
- \mu: the mean of the current feature vector.
- \sigma^2: the variance of the current feature vector.
- \epsilon: a small constant that prevents division by zero.
- \gamma: a learnable scaling parameter.
- \beta: a learnable shifting parameter.
Therefore, the complete LayerNorm can be expressed as:
where \odot denotes element-wise multiplication.
4.2 The LayerNorm Computation Process
flowchart TB
A["Input feature vector x"] --> B["Compute feature mean μ"]
A --> C["Compute feature variance σ²"]
B --> D["Subtract the mean"]
A --> D
D --> E["Divide by the standard deviation"]
C --> E
E --> F["Multiply by learnable parameter γ"]
F --> G["Add learnable parameter β"]
G --> H["Output normalized features"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
LayerNorm does not simply squeeze all values into some fixed range; instead, it first standardizes them and then uses learnable parameters to restore the scale and shift of expression that the model needs.
4.3 Why Does LayerNorm Need Learnable Parameters?
If only standardization were performed:
the model's output would be constrained by the standardization operation.
However, different network layers and different feature dimensions may require different numerical scales and shifts.
Therefore, LayerNorm introduces:
Where:
- \gamma controls the degree of scaling for each feature dimension.
- \beta controls the degree of shifting for each feature dimension.
Both of these parameters can be learned through backpropagation.
For LayerNorm with feature dimension d:
- \gamma has d learnable parameters.
- \beta has d learnable parameters.
Therefore, a standard LayerNorm typically contains:
learnable parameters.
Note that the mean and variance are statistics computed from the current input, not parameters learned directly through training like \gamma and \beta.
V. Which Dimensions Does LayerNorm Actually Normalize?
This is one of the most confusing points when understanding LayerNorm.
Suppose the Transformer input is:
where:
- B: batch size.
- N: sequence length.
- d: feature dimension.
In a standard Transformer, LayerNorm is usually set up as:
nn.LayerNorm(d_model)
It normalizes only along the last feature dimension.
5.1 Each Token Is Computed Independently
Suppose:
This means:
- 2 samples.
- Each sample has 3 tokens.
- Each token has 4 features.
LayerNorm computes the mean and variance separately over the 4 features of each token.
In total there will be:
groups of independent statistics.
Features from different tokens are not mixed together in the computation, nor is data from different samples mixed together.
5.2 Illustration of the Dimensions LayerNorm Computes Over
flowchart TB
A["Input tensor<br/>B × N × d"] --> B["Locate by batch and token"]
B --> C["Extract the d-dimensional vector of a single token"]
C --> D["Compute the mean and variance of that vector"]
D --> E["Perform normalization"]
E --> F["Apply γ and β"]
F --> G["Obtain the output for that token"]
G --> H["All tokens form the output tensor"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
This means that even when different samples have different sequence lengths, LayerNorm does not need to rely on statistics from the entire batch, as long as the input tensor is organized correctly.
This is also one of the reasons LayerNorm is particularly well suited to NLP and Transformers.
VI. The Difference Between LayerNorm and BatchNorm
BatchNorm (Batch Normalization) and LayerNorm are both normalization methods, but they compute their statistics over different dimensions.
6.1 BatchNorm
BatchNorm typically computes the mean and variance of a feature along the batch dimension.
Taking a common 2D input as an example:
For each feature channel c, BatchNorm computes:
That is, the same feature dimension across different samples participates in the statistics together.
In typical convolutional image networks, BatchNorm usually computes statistics for each channel across the batch and spatial positions.
6.2 LayerNorm
LayerNorm, by contrast, computes statistics over the feature dimensions within each individual sample.
For:
each token is computed separately:
Therefore:
- BatchNorm relies on statistics along the batch dimension.
- LayerNorm does not rely on statistics from other samples.
6.3 Comparison of the Two
| Comparison Dimension | BatchNorm | LayerNorm |
|---|---|---|
| Statistical dimension | Usually across the batch dimension | Feature dimension |
| Depends on Batch Size | Yes, statistics are affected by the batch | No |
| Requires running statistics | Classic BatchNorm requires them | No |
| Training vs. inference | Usage is usually different | Computation is basically the same |
| Common applications | CNNs, vision networks | Transformers, NLP |
| Parameters | Learnable scale and shift | Learnable scale and shift |
Note:
BatchNorm uses the statistics of the current batch during training and maintains running mean and running variance for inference; LayerNorm, by contrast, computes statistics from the current input's own features during each forward computation.
Therefore, LayerNorm is better suited to scenarios such as Transformers, where sequence length and batch size may vary considerably.
7. Pre-LN and Post-LN: Where Should LayerNorm Be Placed?
This is a very important distinction in Transformer architecture design.
There is not only one fixed way to place LayerNorm.
According to LayerNorm's position relative to the residual connection, it can be divided into:
- Post-LN: Perform the sublayer transformation and residual addition first, then apply LayerNorm.
- Pre-LN: Apply LayerNorm first, then feed the result into the sublayer, and finally perform the residual addition.
The two structures differ significantly in forward computation and gradient propagation.
7.1 Post-LN: Residual Addition First, Then Normalization
Post-LN is the typical structure used in the original Transformer.
Its formula is:
Computation process:
- The input X enters the sublayer.
- The sublayer computes F(X).
- Add the sublayer output to the original input.
- Apply LayerNorm to the addition result.
- Obtain the output Y.
Structure:
flowchart TB
A["Input X"] --> B["Sublayer F(X)"]
B --> C["Sublayer output"]
A --> D["Residual path"]
C --> E["Element-wise addition"]
D --> E
E --> F["LayerNorm"]
F --> G["Output Y"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
Characteristics of Post-LN:
- The result after residual addition passes through LayerNorm.
- Normalization acts directly on the fused result of the sublayer output and the input.
- In deep networks, gradient propagation is affected by multiple LayerNorms.
- Under some training configurations, deep Post-LN Transformers are harder to optimize.
7.2 Pre-LN: Normalize First, Then Execute the Sublayer
The formula for Pre-LN is:
Computation process:
- The input X enters LayerNorm.
- The normalized result enters the sublayer.
- The sublayer output is added to the original input.
- Obtain the output Y.
Structure:
flowchart TB
A["Input X"] --> B["LayerNorm"]
B --> C["Sublayer F(LayerNorm(X))"]
C --> D["Sublayer output"]
A --> E["Residual path"]
D --> F["Element-wise addition"]
E --> F
F --> G["Output Y"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#E5F0E9,stroke:#83A995,color:#315D49