I. Overview of the Transformer Architecture
Transformer is a deep learning architecture centered on Attention. Through multiple network layers, it performs feature extraction, contextual information fusion, and task output on the input sequence.
The original Transformer uses an Encoder-Decoder architecture and mainly consists of two parts:
- Encoder: Converts the input sequence into hidden representations containing contextual information.
- Decoder: Gradually generates the target sequence based on the existing context and the Encoder output.
Overall structure:
flowchart TB
A["Input sequence"] --> B["Token Embedding"]
B --> C["Positional Encoding"]
C --> D["Encoder"]
D --> E["Encoder output"]
F["Right-shifted target sequence input"] --> G["Target Embedding"]
G --> H["Positional Encoding"]
H --> I["Decoder"]
E --> I
I --> J["Linear output projection"]
J --> K["Softmax"]
K --> L["Target Token probability distribution"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#E5F0E9,stroke:#83A995,color:#315D49
Note that during training, the Decoder of the original Transformer receives the right-shifted target sequence and uses Masked Self-Attention to avoid accessing future Tokens; during inference, it generates subsequent Tokens step by step based on the content already generated.
Modern large language models usually adopt a Decoder-only architecture and do not necessarily retain the complete Encoder-Decoder structure of the original Transformer.
II. Core Modules of Transformer
From the perspective of network structure, a typical Transformer Layer consists of attention modules, feed-forward networks, residual connections, and normalization modules.
2.1 List of Core Modules
| Module | Main Function |
|---|---|
| Token Embedding | Maps Token IDs to continuous vectors |
| Positional Encoding | Provides positional information to the model |
| Multi-Head Attention | Establishes information associations between Tokens |
| Cross-Attention | Enables the Decoder to attend to the Encoder output |
| Feed Forward Network | Applies nonlinear transformations to features |
| Residual Connection | Provides a direct propagation path for information and gradients |
| Layer Normalization | Stabilizes feature distributions in the network |
| Mask | Restricts the positions that attention can access |
| Linear Projection | Maps hidden states to the target output space |
| Softmax | Converts Logits into a probability distribution |
2.2 Relationships Between Modules
The core computation of Transformer can be summarized as:
flowchart TB
A["Token Embedding"] --> B["Add positional information"]
B --> C["Attention information interaction"]
C --> D["Residual connection and normalization"]
D --> E["FFN nonlinear transformation"]
E --> F["Residual connection and normalization"]
F --> G["Output hidden representation"]
G --> H["Proceed to next Transformer layer"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
The calculation order shown here is a common Post-LN style. Modern models also widely use the Pre-LN structure, and the exact arrangement varies.
III. Input Layer: Embedding and Positional Encoding
Transformer does not directly process natural language text; it processes numerically represented Tokens.
3.1 Tokenizer: Text Segmentation
First, the Tokenizer splits the input text into Tokens and converts them into Token IDs.
For example:
Input:
I like artificial intelligence
Token:
["I", "like", "artificial intelligence"]
Token ID:
[101, 205, 308]
The Tokens and Token IDs here are illustrative only; the actual segmentation result depends on the Tokenizer's vocabulary and algorithm.
The Tokenizer is responsible for discretizing text. It is not part of the Transformer Layer itself, but it is an important component of the language model's input processing.
3.2 Token Embedding
Token IDs are discrete integers and cannot be used directly as semantic features of the Transformer.
Therefore, an Embedding matrix is needed to map Token IDs to continuous vectors.
Let:
- The vocabulary size be V.
- The hidden dimension be d_{model}.
Embedding matrix:
For Token ID i:
If the input sequence length is n, then the entire input representation is:
For example:
Then:
This represents 128 Tokens, with each Token corresponding to a 768-dimensional vector.
3.3 Positional Encoding
Standard Self-Attention itself does not contain sequence order information, so positional encoding is needed to provide position information.
The original Transformer uses sinusoidal positional encoding:
The input representation is:
Modern Transformers may also use:
- Learnable absolute positional encoding.
- Relative positional encoding.
- RoPE.
- ALiBi.
Different positional encoding methods have different injection positions and computation mechanisms. For example, RoPE usually operates on the Query and Key in Attention, rather than simply being added to the Token Embedding.
IV. Encoder: Encoder Architecture
The main responsibility of the Encoder is to understand the input sequence and convert it into context-dependent hidden representations.
The Encoder of the original Transformer is formed by stacking N Encoder Layers of identical structure.
Each Encoder Layer contains two main sublayers:
- Multi-Head Self-Attention.
- Feed Forward Network.
Each sublayer also uses residual connections and Layer Normalization.
4.1 Encoder Layer Structure
flowchart TB
A["Input X"] --> B["Multi-Head Self-Attention"]
B --> C["Residual Addition"]
A --> C
C --> D["Layer Normalization"]
D --> E["Feed Forward Network"]
E --> F["Residual Addition"]
D --> F
F --> G["Layer Normalization"]
G --> H["Encoder Layer Output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
Here, the Post-LN representation is used, corresponding to:
Where:
- MHA: multi-head self-attention.
- FFN: feed-forward neural network.
- LN: Layer Normalization.
4.2 Encoder Self-Attention
The Encoder uses bidirectional Self-Attention.
Each token can attend to other valid tokens in the input sequence, without being limited by causal direction.
For example:
Input:
I like learning Transformer
Processing:
"learning" can attend to "I"
"learning" can attend to "like"
"learning" can attend to "Transformer"
In this way, each position in the Encoder can fuse contextual information from the entire input sequence.
Bidirectional here does not mean the model necessarily understands all semantics; rather, it means the attention computation is not restricted by a left-to-right causal mask.
4.3 Encoder Output
After N Encoder layers, we obtain:
Where:
- n: input sequence length.
- d_{model}: hidden dimension.
The output at each position incorporates contextual information from the input sequence.
These hidden representations can be used for:
- Text classification.
- Named entity recognition.
- Semantic feature extraction.
- Target sequence generation in machine translation.
- Other downstream tasks.
5. Decoder: Decoder Architecture
The Decoder in the original Transformer is responsible for generating the target sequence.
Its main differences from the Encoder are:
- Uses Masked Self-Attention.
- Adds Encoder-Decoder Cross-Attention.
- Predicts target tokens in an autoregressive manner.
The original Decoder is stacked from N Decoder Layers.
5.1 Decoder Layer Structure
An original Transformer Decoder Layer contains three main sublayers:
- Masked Multi-Head Self-Attention.
- Encoder-Decoder Cross-Attention.
- Feed Forward Network.
The structure is as follows:
flowchart TB
A["Decoder Input"] --> B["Masked Multi-Head Self-Attention"]
B --> C["Residual Add & LayerNorm"]
A --> C
C --> D["Cross-Attention"]
E["Encoder Output"] --> D
D --> F["Residual Add & LayerNorm"]
C --> F
F --> G["Feed Forward Network"]
G --> H["Residual Add & LayerNorm"]
F --> H
H --> I["Decoder Layer Output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
5.2 Masked Self-Attention
The Decoder generates autoregressively.
When predicting the Token at the current position, it can only access the current position and the Tokens before it.
For example:
Target sequence:
I like artificial intelligence
Prediction position 1: can only access "I"
Prediction position 2: can access "I", "like"
Prediction position 3: can access "I", "like", "artificial intelligence"
A causal mask (Causal Mask) is used to mask out future positions:
Here, the Mask value corresponding to future positions is negative infinity, which makes their Softmax weight zero.
5.3 Cross-Attention
Cross-Attention is used to establish the information link between the Decoder and the Encoder.
Its key difference from Self-Attention lies in where Q, K, and V come from.
In the original Transformer:
- Query comes from the Decoder's current hidden state.
- Key comes from the Encoder output.
- Value comes from the Encoder output.
Let:
Then:
Cross-Attention:
Where:
- m: the current target sequence length.
- n: the source sequence length.
The shape of the attention matrix is:
This means that each position in the Decoder can, based on its own state, extract the relevant information from all input positions of the Encoder.
5.4 The Role of Cross-Attention
Take machine translation as an example:
Source language:
I love artificial intelligence.
Target language:
我喜欢人工智能。
The Encoder encodes the source language into a contextual representation.
When generating the target language, the Decoder accesses the Encoder output through Cross-Attention, thereby using the source-language information to generate the target text.
Cross-Attention does not simply concatenate the source-language vectors onto the target-language vectors; instead, it performs dynamic information aggregation via Query, Key, and Value.
6. Differences Between Encoder and Decoder
Both the Encoder and the Decoder use the Transformer Layer, but their attention mechanisms and task objectives differ.
| Comparison Dimension | Encoder | Decoder |
|---|---|---|
| Primary task | Input understanding and feature encoding | Autoregressive generation of the target sequence |
| Self-Attention | Bidirectional attention | Causal attention |
| Causal Mask | Usually not used | Used |
| Cross-Attention | Not present in the original architecture | Used in the original architecture |
| Input | Source sequence | Right-shifted input of the target sequence |
| Output | Contextual hidden representations | Hidden representations of the target sequence |
| Typical applications | Text understanding, feature extraction | Text generation, sequence generation |
Note that Encoder and Decoder here refer to the structures in the original Encoder-Decoder Transformer.
Modern Decoder-only models typically have no Cross-Attention and do not need a separate Encoder.
7. Attention Mechanisms in the Transformer
Attention is the core computational module of the Transformer.
7.1 Scaled Dot-Product Attention
Given:
Compute:
The complete formula:
If a Mask is present, the mask must be applied before Softmax.
7.2 Multi-Head Attention
Multiple Attention Heads compute independently in different representation subspaces:
After concatenation, an output projection is applied:
Multi-head attention improves the model's ability to learn different patterns of association.
7.3 MHA, MQA and GQA
To reduce KV Cache footprint, modern large language models have developed different attention structures.
| Type | Query Head | Key/Value Head | Key characteristics |
|---|---|---|---|
| MHA | h | h | Each Query Head uses its own K/V |
| MQA | h | 1 | All Query Heads share a single set of K/V |
| GQA | h | g | Multiple Query Heads share K/V in groups |
Where:
GQA strikes a trade-off between KV Cache footprint and the representational capacity of attention.
8. FFN: The Computation Mechanism of the Feed-Forward Network
Attention is responsible for information exchange between different Tokens, while the FFN is responsible for applying a non-linear transformation to each Token's representation.
8.1 Original FFN Structure
The original Transformer uses two linear transformations and the ReLU activation function:
Typically, the hidden dimension is first expanded and then mapped back to the original dimension.
For example:
Then:
8.2 Modern FFN Structure
Modern large language models may adopt activation structures such as GELU and SwiGLU.
For example:
Where:
- W_g: gating branch parameters.
- W_v: value branch parameters.
- W_o: output projection parameters.
- \odot: element-wise multiplication.
The gating mechanism enables the model to dynamically control the transmission of feature information.
8.3 Collaboration Between Attention and FFN
They can be understood as computations at two different levels:
- Attention: exchanges and aggregates information across sequence positions.
- FFN: performs nonlinear feature transformations on the fused representation at each position.
The two are stacked alternately, enabling the model to build increasingly complex contextual representations layer by layer.
9. Residual Connections and Layer Normalization
Transformers usually contain dozens or even hundreds of layers.
Without reasonable information propagation and normalization mechanisms, deep networks may face training instability.
9.1 Residual Connection
The basic formula for a residual connection:
Where:
- x: sublayer input.
- F(x): output of Attention or FFN.
- y: residual output.
It provides a direct path for information and gradients to propagate.
9.2 Layer Normalization
LayerNorm normalizes the feature dimension of a single Token.
Where:
- \gamma: learnable scaling parameter.
- \beta: learnable bias parameter.
- \epsilon: numerical stability term.
9.3 Post-LN and Pre-LN
Post-LN:
Pre-LN:
The main difference between the two structures lies in the position of LayerNorm.
| Structure | Normalization Position | Characteristics |
|---|---|---|
| Post-LN | After residual addition | Used by the original Transformer |
| Pre-LN | Before sublayer computation | Widely used in modern deep models |
Pre-LN generally offers more stable deep training characteristics, but the specific design still depends on the model architecture and training configuration.
10. Tensor Dimensions: Understanding Data Flow in Transformer
To understand the Transformer architecture, one must master the tensor dimension changes across modules.
Assume:
- Batch Size: B=32
- Sequence Length: N=128
- Hidden Dimension: d_{model}=768
- Number of Attention Heads: h=12
- Per-Head Dimension: d_k=d_v=64
10.1 Input Tensor
Embedding output:
Substituting:
10.2 Q, K, V Projections
After linear projections:
Then split into multiple attention heads:
Dimension meanings:
| Dimension | Meaning |
|---|---|
| 32 | Batch Size |
| 12 | Number of Attention Heads |
| 128 | Sequence Length |
| 64 | Head Dimension |
10.3 Attention Score Matrix
Compute:
This yields:
Each Batch and each Head has a 128\times128 attention score matrix.
10.4 Attention Output
Multiply the attention weights by V:
This yields:
Concatenate multiple heads:
Then apply an output projection, keeping the hidden dimension unchanged.
10.5 FFN Dimension Changes
Assume:
Then:
The input and output of a Transformer Layer usually maintain the same hidden dimension to enable residual connections.
10.6 Summary of Dimension Changes
| Computation Stage | Tensor Shape |
|---|---|
| Embedding output | [32, 128, 768] |
| Q/K/V projection | [32, 128, 768] |
| Multi-head split | [32, 12, 128, 64] |
| Attention Score | [32, 12, 128, 128] |
| Attention output | [32, 12, 128, 64] |
| Head concatenation | [32, 128, 768] |
| FFN first layer | [32, 128, 3072] |
| FFN second layer | [32, 128, 768] |
These dimensions are a typical example of standard MHA. GQA, MQA, different hidden dimensions, and different attention implementations will vary accordingly.
11. Original Transformer and Modern LLM Architectures
The original Transformer uses an Encoder-Decoder structure, but most modern large language models adopt a Decoder-only architecture.