I. What Is a Transformer?
A Transformer is a deep neural network architecture based on the attention mechanism, proposed by a Google research team in the 2017 paper “Attention Is All You Need.”
It was initially used for machine translation tasks, then gradually became the mainstream architecture in natural language processing and expanded into fields such as computer vision, speech recognition, multimodal learning, and generative artificial intelligence.
The most important design principle of the Transformer is:
Directly establish information associations between different positions in a sequence through the attention mechanism, rather than relying on recurrent structures to process the sequence step by step.
Before Transformers emerged, sequence modeling mainly relied on recurrent neural networks such as RNN, LSTM, and GRU.
These models process input sequentially by time step:
Token 1 → Token 2 → Token 3 → Token 4
The computation at the current moment depends on the hidden state from the previous moment, so there is a certain serial computation limitation.
The Transformer, by contrast, uses Self-Attention to let tokens in a sequence directly compute their associations with one another.
Input Token Sequence
↓
Embedding
↓
Positional Encoding
↓
Self-Attention
↓
Feedforward Neural Network
↓
Multi-layer Transformer
↓
Output Representation
The main features of the Transformer are:
- Parallel computation: Multiple tokens in the same layer can be computed in parallel.
- Global dependency modeling: A token can directly attend to other tokens in the sequence.
- Contextual representation: Each token’s representation can integrate information from other positions.
- Modular structure: Attention, FFN, residual connections, and normalization can be combined into multi-layer networks.
- Scalability: Large models can be built by increasing the number of layers, hidden dimensions, and parameter scale.
II. Why Do We Need Transformers?
To understand the Transformer, we first need to understand the problems faced by traditional sequence models.
2.1 The Computation Mechanism of RNN
A Recurrent Neural Network (RNN) passes historical information through hidden states.
For an input sequence:
RNN performs the following at each time step:
where:
- x_t: input at the current time step.
- h_t: current hidden state.
- h_{t-1}: hidden state at the previous time step.
- W_x: input weight matrix.
- W_h: recurrent weight matrix.
- f: nonlinear activation function.
Its computational dependencies are:
flowchart LR
A["Input x₁"] --> B["RNN Cell"]
B --> C["Hidden State h₁"]
C --> D["RNN Cell"]
E["Input x₂"] --> D
D --> F["Hidden State h₂"]
F --> G["RNN Cell"]
H["Input x₃"] --> G
G --> I["Hidden State h₃"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
The main problems with RNNs are:
- Serial computation: The current time step must wait for the previous time step to complete, making it difficult to fully utilize GPU parallelism.
- Long-distance dependencies: Information from earlier positions must pass through multiple state transitions before it can affect later positions.
- Gradient problems: Training on long sequences is prone to vanishing or exploding gradients.
- Information compression: Historical information must be continuously integrated into a fixed-dimensional hidden state, which may cause information loss.
LSTM and GRU improve long-term dependency modeling through gating mechanisms, but they still retain a recurrent computation structure.
2.2 Transformer’s Solution Approach
Transformer no longer passes information step by step through recurrent hidden states; instead, it uses attention mechanisms to directly compute the relationships between Tokens.
For example:
I deposit money at the bank
When the model processes “bank,” it can attend to “deposit money” through Attention, thereby obtaining a semantic representation that better fits the context.
This mechanism allows different positions in the sequence to establish direct connections.
| Comparison Dimension | RNN / LSTM | Transformer |
|---|---|---|
| Core Mechanism | Recurrent state propagation | Attention mechanism |
| Sequence Processing | Has time-step dependency | Can be processed in parallel during training |
| Long-Range Dependency | Requires step-by-step propagation | Can directly establish associations |
| Parallelization Capability | Limited by recurrent structure | Tokens in the same layer can be processed in parallel |
| Computational Complexity | Typical recurrent computation takes about O(n) time steps | Full attention computation is about O(n^2d) |
| Typical Applications | Time series, speech, early NLP | LLMs, machine translation, multimodal models |
Note that Transformer is not more efficient than RNN in all scenarios. The computation cost of standard Self-Attention and the storage required for the attention matrix grow quadratically with sequence length, so long-context scenarios require additional optimization.
3. Overall Architecture of Transformer
The original Transformer uses an Encoder-Decoder architecture.
It consists of two parts overall:
- Encoder: Responsible for understanding the input sequence and generating contextual representations.
- Decoder: Generates the target sequence based on the existing context.
3.1 Encoder
The Encoder is formed by stacking multiple structurally identical Encoder Layers.
Each layer mainly contains:
- Multi-Head Self-Attention.
- Feed Forward Network.
- Residual Connection.
- Layer Normalization.
Its structure can be abstracted as:
flowchart TB
A["Input Representation"] --> B["Multi-Head Self-Attention"]
B --> C["Residual Connection and Layer Normalization"]
C --> D["Feed Forward Network"]
D --> E["Residual Connection and Layer Normalization"]
E --> F["Output Representation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
After passing through multiple Encoder layers, each Token can integrate contextual information from other Tokens in the input sequence.
3.2 Decoder
The Decoder of the original Transformer is likewise formed by stacking multiple Decoder Layers.
Each layer contains:
- Masked Multi-Head Self-Attention.
- Encoder-Decoder Cross-Attention.
- Feed Forward Network.
- Residual Connection.
- Layer Normalization.
Its structure is as follows:
flowchart TB
A["Decoder Input"] --> B["Masked Multi-Head Self-Attention"]
B --> C["Residual Connection and Layer Normalization"]
C --> D["Encoder-Decoder Cross-Attention"]
D --> E["Residual Connection and Layer Normalization"]
E --> F["Feed Forward Network"]
F --> G["Residual Connection and Layer Normalization"]
G --> H["Decoder Output"]
I["Encoder Output"] --> D
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Where:
- Masked Self-Attention: restricts the current position to only access the current position and positions before it.
- Cross-Attention: allows the Decoder to attend to the information output by the Encoder.
- FFN: applies a non-linear transformation to the representation at each position.
3.3 The Original Transformer and Modern Large Language Models
Modern large language models do not necessarily adopt the complete Encoder-Decoder structure.
Based on their architecture, they can be divided into three categories:
| Architecture | Main Characteristics | Typical Models |
|---|---|---|
| Encoder-only | Bidirectional context modeling, suitable for understanding tasks | BERT |
| Decoder-only | Causal attention, suitable for autoregressive generation | GPT, Llama, Qwen |
| Encoder-Decoder | Encodes the input, then generates the output through the Decoder | T5, the original Transformer |
Encoder-only
It can utilize context information from both before and after the current position at the same time, and is commonly used for:
- Text classification.
- Named entity recognition.
- Semantic understanding.
- Feature extraction.
Decoder-only
It restricts the attention scope through a causal mask and predicts the next Token in left-to-right order, and is commonly used for:
- Dialogue generation.
- Code generation.
- Text continuation.
- General-purpose large language models.
Encoder-Decoder
It first encodes the input, then generates the target sequence based on the encoding result, and is commonly used for:
- Machine translation.
- Text summarization.
- Text transformation.
- Sequence-to-sequence tasks.
IV. Self-Attention: The Core Mechanism of the Transformer
Self-Attention is the most core component of the Transformer.
Its role is:
To dynamically aggregate information from other Tokens for each Token according to the degree of association between different Tokens in the sequence.
4.1 Why Do We Need Attention?
Consider the following sentence:
Xiaoming put the book on the table because it is relatively stable.
What "it" refers to here needs to be determined in combination with the context.
When the model processes "it", it needs to consider the words that appeared earlier and their semantic relationships.
Self-Attention allows the current position to directly compute association weights with other positions.
Unlike RNNs, which rely on step-by-step propagation, Attention can directly establish information interaction between distant positions.
4.2 Query, Key, Value
The core of Self-Attention is three vectors:
- Query (Q): the query vector, representing what information the current position wants to find.
- Key (K): the key vector, representing what information the current position can provide.
- Value (V): the value vector, representing the content of the information actually provided by the current position.
For the input matrix:
A linear transformation is performed through three learnable weight matrices:
Where:
Therefore:
Here, n represents the sequence length.
Note that Q, K, and V are not manually defined semantic labels, but different linear projections learned through training.
4.3 The Attention Computation Process
First, compute the dot product between Query and Key:
This gives an n\times n matrix.
Each element in the matrix represents the matching score between a Query and a Key.
Then scale it:
Then apply Softmax to convert it into normalized weights:
Finally, use the attention weights to compute a weighted sum of Value:
Complete formula:
Where:
- QK^T: computes the relevance scores between Tokens.
- \sqrt{d_k}: scaling factor.
- Softmax: converts scores into a weight distribution.
- V: provides the information actually aggregated.
4.4 Why divide by \sqrt{d_k}?
This is a very important design in Scaled Dot-Product Attention.
Assume that each dimension of Q and K is an independent random variable with mean 0 and variance 1.
The dot product:
Under the above assumptions, its variance is approximately:
As the dimension increases, the fluctuation in the dot product values also increases.
If larger dot products are fed directly into Softmax, the distribution may become too sharp.
For example:
and:
By comparison, the latter usually produces a more concentrated probability distribution.
A too sharp Softmax may make gradients smaller and affect training stability.
Therefore, we use:
to scale the dot product so that its numerical scale is more stable.
Note that this is a scale-control method based on statistical assumptions; it does not strictly guarantee constant variance for all input distributions.
4.5 Complete Attention Computation Flow
flowchart TB
A["Input matrix X"] --> B["Linear projection"]
B --> C["Generate Q, K, V"]
C --> D["Compute the dot product of Q and K"]
D --> E["Divide by √dₖ to scale"]
E --> F["Compute attention weights with Softmax"]
F --> G["Multiply by V"]
G --> H["Obtain the Attention output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
4.6 A Simple Numerical Example
Suppose there are three Tokens:
I like AI
After linear projection, we obtain Q, K, and V.
To simplify the calculation, assume:
Then:
Since:
After scaling:
Apply Softmax to each row of the matrix to obtain the attention weight matrix:
Each row represents the attention distribution of one Query over all Keys.
Finally:
This means the output of each Token is obtained by aggregating the Values of multiple Tokens according to their weights.
V. Multi-Head Attention
A single Attention can only compute associations based on one set of Q, K, V.
But language contains many kinds of relationships:
- Syntactic relationships.
- Semantic relationships.
- Coreference relationships.
- Positional relationships.
- Long-distance dependencies.
- Domain associations.
Multi-Head Attention uses multiple independent attention heads to let the model learn different informational relationships in different representation subspaces.
5.1 Computation of Multi-Head Attention
For the i-th attention head:
The outputs of multiple attention heads are concatenated:
Then passed through one more linear projection:
Where:
- h: the number of attention heads.
- W_i^Q: the Query projection matrix of the i-th head.
- W_i^K: the Key projection matrix of the i-th head.
- W_i^V: the Value projection matrix of the i-th head.
- W^O: the output projection matrix.
5.2 Structure of Multi-Head Attention
flowchart TB
A["Input X"] --> B["Linear Projection"]
B --> C["Attention Head 1"]
B --> D["Attention Head 2"]
B --> E["Attention Head 3"]
B --> F["Attention Head h"]
C --> G["Concat"]
D --> G
E --> G
F --> G
G --> H["Output Linear Projection"]
H --> I["Multi-Head Attention Output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
Suppose:
If all heads have the same dimension, each head typically has:
Each attention head computes attention in a 64-dimensional subspace, and the results are finally concatenated back into 768 dimensions.
5.3 Why Are Multiple Heads Needed?
Different heads can learn different attention patterns.
For example, some heads may focus more on:
- Adjacent Tokens.
- Distant Tokens.
- Subject–predicate relationships.
- Relationships between pronouns and entities.
- Specific structures within a sentence.
But one should not simply assume that every head corresponds to a well-defined linguistic function.
In actual training, the functions of multiple heads may overlap, and some heads may be underutilized.
The core value of Multi-Head Attention is to increase the diversity of attention representations.
5.4 MHA, MQA, and GQA
During inference in large language models, Keys and Values usually need to be cached; this is the KV Cache.
As the number of model layers and the context length increase, the KV Cache consumes a large amount of GPU memory.
Therefore, different attention structures have emerged:
| Structure | Query Heads | Key/Value Heads | Characteristics |
|---|---|---|---|
| MHA | Multiple | Same as the number of Query Heads | Flexible expressive capacity, larger KV Cache |
| MQA | Multiple | 1 | Multiple Query Heads share a single set of K/V |
| GQA | Multiple | Fewer than the Query Heads | A compromise between expressive capacity and cache cost |
- MHA: Multi-Head Attention.
- MQA: Multi-Query Attention.
- GQA: Grouped-Query Attention.
GQA groups multiple Query Heads, with each group sharing one set of Keys and Values.
These structures are mainly used to improve the inference efficiency of large models, especially to reduce the GPU memory footprint and memory access pressure of the KV Cache.
6. Positional Encoding: How Does a Transformer Understand Order?
The Self-Attention of a Transformer does not inherently contain the sequential order of Tokens.
If only Token Embeddings are fed into Attention, the model cannot reliably identify the original sequence order from the input set alone.
Therefore, positional information needs to be introduced.
6.1 Why Is Positional Encoding Needed?
Consider:
cat chases dog
dog chases cat
The two sentences contain the same Tokens, but the order differs, and so does the meaning.
Without positional information, standard Self-Attention is permutation-equivariant with respect to the arrangement of input Tokens: if the input positions are rearranged, the output is rearranged accordingly, and the model does not automatically know the meaning of the original positions.
Therefore, positional information must be provided to the model.
6.2 Sinusoidal Positional Encoding
The original Transformer constructs positional encodings using sine and cosine functions.
For position pos and dimension i:
Where:
- pos: the position of the Token in the sequence.
- i: the vector dimension index.
- d_{model}: the model hidden dimension.
Different dimensions have different frequencies, allowing the positional encoding to express positional variation at different scales.
The original Transformer adds the positional encoding to the Token Embedding:
6.3 Learnable Positional Encoding
Another approach is to make the position vectors directly learnable parameters.
Assuming a maximum sequence length of L and a hidden dimension of d, the positional encoding matrix is:
The model learns the vector corresponding to each position through training.
Advantages:
- Simple to implement.
- Position representations can be learned according to the task.
Limitations:
- Usually constrained by the maximum position length seen during training.
- Beyond the trained position range, good generalization cannot be directly guaranteed.
6.4 Relative Positional Encoding
Relative positional encoding focuses on the relative distance between two Tokens, rather than merely their respective absolute positions.
For example:
Token A and Token B are 2 positions apart
Token A and Token C are 5 positions apart
The model can use this relative relationship to establish attention associations.
Relative positional encoding and its variants are very important in modern Transformers.
6.5 RoPE: Rotary Positional Embedding
RoPE (Rotary Positional Embedding) is a positional encoding method widely used in modern Decoder-only large language models.
Its core idea is:
By applying a position-dependent rotation transformation to the Query and Key vectors, positional information is incorporated into the attention computation.
For a two-dimensional vector:
At position m, the rotation transformation can be expressed as:
Then:
RoPE groups different dimensions for rotation and applies it to both Query and Key.
In this way, the dot product between two positions can reflect relative positional information.
The advantages of RoPE include:
- Integrate position information into the Attention computation.
- Can naturally express relative positional relationships.
- Suitable for autoregressive language models.
- Can be combined with methods such as position interpolation to extend context length.
However, RoPE does not mean that a model can extend its context indefinitely. Long-context capability is also affected by factors such as training data, attention mechanisms, position frequency configuration, and inference strategies.
6.6 Comparison of Mainstream Position Encoding Methods
| Type | Core Idea | Representative Characteristics |
|---|---|---|
| Sinusoidal | Uses sine and cosine functions | Used by the original Transformer |
| Learned Absolute | Learns absolute position vectors | Parametric and simple to implement |
| Relative Position | Models relative distances between positions | Focuses on relative relationships |
| RoPE | Applies position-dependent rotations to Q and K | Widely used in modern LLMs |
| ALiBi | Adds position-dependent bias to attention scores | Models positions via linear distance bias |
7. Feed Forward Network: Feedforward Neural Network
Attention is responsible for information exchange between tokens.
But Attention alone is not enough; the Transformer also needs to apply further nonlinear transformations to the representation at each position.
This is the Feed Forward Network (FFN).
7.1 Basic Structure of FFN
The original Transformer uses two linear transformations, with a ReLU activation function in between:
Where:
- W_1: Weight matrix of the first layer.
- b_1: Bias of the first layer.
- W_2: Weight matrix of the second layer.
- b_2: Bias of the second layer.
Usually, the hidden dimension is first expanded and then mapped back to the original dimension.
For example:
The computation process is:
7.2 Differences Between Attention and FFN
| Aspect | Attention | FFN |
|---|---|---|
| Main role | Information exchange between tokens | Applies nonlinear transformation to a single position |
| Input relationship | Multiple positions in the same sequence | Computed independently at each position |
| Core computation | Q, K, V, Softmax | Linear layers and activation functions |
| Information scope | Can aggregate across tokens | Usually does not directly interact across tokens |
| Role | Context information fusion | Feature transformation and enhanced representational capacity |
Note that in a standard Transformer, the FFN in the same layer is usually applied independently to each token position, but its parameters are shared across all positions.
7.3 Changes in Modern FFN
Modern large language models do not necessarily continue to use the original ReLU FFN.
Common structures include:
- GELU.
- SwiGLU.
- GEGLU.