1. Why Are There Different Transformer Architectures?
Transformer was first proposed by Google in 2017 in the paper “Attention Is All You Need.”
It was not originally designed specifically for large language models, but was built as a sequence-to-sequence (Seq2Seq) model for machine translation tasks.
For example, machine translation needs to perform:
Input: I love artificial intelligence.
Output: 我喜欢人工智能。
The model needs to first understand the source language and then generate the target language.
The original Transformer therefore adopted an Encoder-Decoder architecture:
- Encoder: reads the input sequence and extracts contextual representations.
- Decoder: gradually generates the output sequence based on what has already been generated and the information provided by the Encoder.
With the development of deep learning, researchers found that different parts of the Transformer can be used independently, forming three main architectures:
- Encoder-only: keeps only the Encoder.
- Decoder-only: keeps only the causal self-attention structure in the Decoder.
- Encoder-Decoder: keeps both the Encoder and Decoder.
Although all are based on the Transformer, they differ significantly in attention mechanisms, information flow, training objectives, and application scenarios.
2. Overall Structure of Transformer
First, understand the basic components of the original Transformer.
flowchart TB
A["Input Token Sequence"] --> B["Token Embedding"]
B --> C["Positional Encoding"]
C --> D["Encoder"]
D --> E["Contextual Representation"]
F["Target Sequence Tokens (Shifted Right)"] --> G["Token Embedding"]
G --> H["Positional Encoding"]
H --> I["Decoder"]
E --> J["Cross-Attention"]
I --> J
J --> K["Linear Projection"]
K --> L["Softmax"]
L --> M["Output Token Probabilities"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
The core of the original Transformer consists of the following components:
| Component | Main Function |
|---|---|
| Embedding | Converts Token IDs into continuous vectors |
| Positional Encoding | Injects positional information |
| Multi-Head Attention | Models relationships between Tokens |
| Feed Forward Network | Applies nonlinear feature transformation to each position |
| Residual Connection | Provides residual paths and improves deep network training |
| LayerNorm | Stabilizes feature distributions in the network |
| Linear + Softmax | Converts hidden states into a probability distribution over the vocabulary |
The core difference between the Encoder and Decoder is not whether Attention is used, but what type of Attention is used and which input information is allowed to be accessed.
3. Encoder: The Encoder Architecture
The Encoder-only architecture keeps only the Encoder part of the Transformer.
Its core task is:
Read the full input sequence, establish contextual relationships between Tokens, and generate hidden representations that contain contextual information.
Typical representatives include:
- BERT
- RoBERTa
- ALBERT
- DeBERTa
3.1 Overall Structure of the Encoder
A standard Transformer Encoder Layer usually consists of two main sublayers:
- Multi-Head Self-Attention.
- Feed Forward Network.
Each sublayer is also paired with a residual connection and LayerNorm.
flowchart TB
A["Input Hidden States X"] --> B["Multi-Head Self-Attention"]
B --> C["Residual Connection and LayerNorm"]
A --> C
C --> D["Feed Forward Network"]
D --> E["Residual Connection and LayerNorm"]
C --> E
E --> F["Output Hidden States"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
Note: The figure above shows a conceptual structure. The original Transformer uses Post-LN, meaning that after the sublayer computation, residual addition and LayerNorm are performed. Modern Encoders may also use different variants such as Pre-LN.
3.2 Encoder Uses Bidirectional Self-Attention
This is one of the most important characteristics of the Encoder.
Suppose the input sequence is:
I like learning artificial intelligence
Corresponding positions:
Token: I like learning artificial intelligence
Position: 0 1 2 3
In standard bidirectional Self-Attention, each Token can attend to all other valid Tokens in the sequence.
For example:
- “I” can attend to “like,” “learning,” and “artificial intelligence.”
- “like” can attend to “I,” “learning,” and “artificial intelligence.”
- “artificial intelligence” can also attend to all preceding Tokens.
For position i, the attention score can be expressed as:
After Softmax, the attention weights are obtained:
Here there is no causal constraint, so the model can use information both before and after the current position.
Therefore, the Encoder can build a representation containing the full context for each Token.
3.3 Why Does the Encoder Not Need a Causal Mask?
Decoder-only models need to avoid seeing future Tokens, whereas the Encoder usually does not need this restriction.
For example:
Today's weather is very good
When the Encoder understands “weather,” it can simultaneously use:
- “today’s” on the left.
- “very good” on the right.
Information on the right helps in understanding the full context in which the current Token appears.
If a Causal Mask is forcibly used, the Encoder cannot use the following contextual information and loses the advantage of bidirectional modeling.
However, the Encoder may still use a Padding Mask to prevent the model from attending to invalid Tokens produced when padding sequences.
3.4 The Encoder’s Training Objective: Masked Language Modeling
Taking BERT as an example, it uses Masked Language Modeling (MLM) as one of its core pretraining tasks.
The basic idea is:
Randomly select some Tokens in the input, mask them, and then have the model predict the original Tokens based on context.
For example:
Original text:
I like learning artificial intelligence
After masking:
I like [MASK] artificial intelligence
Model prediction:
learning
The training objective can be expressed as:
where:
- M: the set of selected Mask positions.
- x_i: the original Token at the masked position.
- X_{\backslash M}: the input containing visible context.
The model needs to use the context on both sides of the masked Token to recover the original content.
Note that the masking strategy in BERT's original training scheme does not mean that all selected positions are directly replaced with [MASK]; it also includes random replacement and keeping the original Token unchanged.
3.5 What Is the Encoder’s Output?
The Encoder outputs a set of contextual hidden vectors:
Suppose:
Then:
Where:
- B: Batch Size.
- N: Input sequence length.
- d: Hidden dimension.
Each position obtains a contextual representation.
These representations can be used for downstream tasks, such as:
| Task | How the Encoder is used |
|---|---|
| Text classification | Use [CLS] or the pooled representation |
| Sentiment analysis | Classify based on contextual representations |
| Named entity recognition | Classify each token |
| Semantic matching | Compare the vector representations of two texts |
| Information extraction | Identify entities, relations, and text boundaries |
| Retrieval | Encode text into vectors for similarity search |
The advantage of the Encoder lies in bidirectional context understanding, making it suitable for tasks that need to fully utilize the complete input content.
4. Decoder: Decoder Architecture
The Decoder is the part of the original Transformer responsible for generating the target sequence.
Its core task is:
Generate the target sequence step by step based on the tokens already generated and the input representations provided by the Encoder.
The biggest difference between the Decoder and the Encoder is that it must follow the autoregressive generation constraint and cannot access future tokens in advance.
4.1 Structure of the Original Transformer Decoder
The original Transformer Decoder Layer contains three main parts:
- Masked Multi-Head Self-Attention.
- Encoder-Decoder Attention, also called Cross-Attention.
- Feed Forward Network.
The structure is as follows:
flowchart TB
A["Decoder Input"] --> B["Masked Self-Attention"]
B --> C["Residual Connection and LayerNorm"]
A --> C
C --> D["Cross-Attention"]
E["Encoder Output"] --> D
D --> F["Residual Connection and LayerNorm"]
C --> F
F --> G["Feed Forward Network"]
G --> H["Residual Connection and LayerNorm"]
F --> H
H --> I["Decoder Output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
Compared with the Encoder, the original Decoder has an additional Cross-Attention sublayer.
It allows the Decoder to access the Encoder's output representations during generation.
4.2 Masked Self-Attention
The Decoder must follow the autoregressive generation mechanism.
Suppose the target sequence is:
I like artificial intelligence
During generation:
- When predicting “I”, it cannot see “like” and “artificial intelligence”.
- When predicting “like”, it can see “I” but cannot see “artificial intelligence”.
- When predicting “artificial intelligence”, it can see “I” and “like”.
Therefore, a Causal Mask is needed.
It restricts the attention matrix to prevent the current position from accessing future positions.
For a sequence length of N, the Causal Mask can be expressed as:
After adding the Mask:
The attention scores for future positions become negative infinity, and after Softmax the corresponding weights become zero.
For example, a causal attention matrix of length 4:
Allowed attention positions:
1 0 0 0
1 1 0 0
1 1 1 0
1 1 1 1
Where:
- 1 means attention is allowed.
- 0 means attention is forbidden.
The corresponding visibility relationships:
flowchart TB
A["Position 0: can only see position 0"]
B["Position 1: can see positions 0, 1"]
C["Position 2: can see positions 0, 1, 2"]
D["Position 3: can see positions 0, 1, 2, 3"]
A --> B
B --> C
C --> D
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#E5F0E9,stroke:#83A995,color:#315D49
Note that the figure above shows the range of information each position is allowed to access; it does not mean that the entire history sequence must actually be recomputed step by step during generation.
4.3 Cross-Attention: How Does the Decoder Read Information from the Encoder?
Cross-Attention is an important component of the original Encoder-Decoder architecture.
It allows the Decoder, when generating a target Token, to extract relevant information from the Encoder's output.
Unlike Self-Attention, in Cross-Attention the Q, K, and V come from different sequence representations.
Specifically:
Where:
- H_D: the Decoder's current hidden state.
- H_E: the contextual representation output by the Encoder.
- Q: comes from the Decoder.
- K, V: come from the Encoder.
Attention computation:
Its meaning is:
The Decoder forms a Query based on its current generation state, then extracts from the Key and Value provided by the Encoder the information most relevant to the current generation.
For example, in a translation task:
Source language:
I love artificial intelligence.
Target language:
我 喜欢 人工智能
When the Decoder generates “人工智能”, it can use Cross-Attention to focus on reading the relevant content in the source language.
Cross-Attention does not simply concatenate the Encoder output onto the Decoder input; instead, it dynamically selects and aggregates input information through the attention mechanism.
4.4 The Decoder's Training Objective: Autoregressive Language Modeling
The original Transformer Decoder predicts the target sequence in an autoregressive manner.
Given the target sequence:
The model learns:
Where:
- X: the Encoder input.
- y_{<t}: the target Tokens already generated before the current Token.
- y_t: the Token that needs to be predicted at present.
During training, Teacher Forcing is usually adopted:
The target sequence is shifted right by one position and used as the Decoder input.
For example:
Target sequence:
<BOS> 我 喜欢 人工智能 <EOS>
Decoder input:
<BOS> 我 喜欢 人工智能
Training labels:
我 喜欢 人工智能 <EOS>
The model predicts the next Token at each position.
The training loss typically uses cross-entropy:
Because Causal Mask is used, each position can only make use of the preceding target Tokens and cannot see future labels.
4.5 The Decoder's Output
The Decoder's hidden state is passed through a linear projection and mapped to the vocabulary dimension:
Where:
- H: the Decoder hidden state.
- W_{vocab}: the output projection matrix.
- Logits: the unnormalized score corresponding to each Token in the vocabulary.
Softmax is then applied to obtain a probability distribution:
The model can select the next Token based on the probability distribution.
Common selection strategies include:
- Greedy Search.
- Beam Search.
- Top-K Sampling.
- Top-P Sampling.
- Temperature Sampling.
V. Decoder-only: Decoder-Only Architecture
Decoder-only is a very important architecture in modern large language models.
Typical examples include:
- GPT series (OpenAI).
- Llama series (Meta).
- Qwen series (Alibaba).
- DeepSeek series (DeepSeek).
- Mistral series (Mistral AI).
Its core characteristics are:
It uses only Decoder-style causal Self-Attention, does not use a separate Encoder, and does not require the Encoder-Decoder Cross-Attention found in the original Transformer.
It should be noted in particular:
The Decoder-only here does not simply retain the entire structure of the original Transformer Decoder.
The original Decoder contains two attention sublayers: Masked Self-Attention and Cross-Attention.
Decoder-only typically retains causal Self-Attention and removes the Cross-Attention that depends on the output of a separate Encoder.
5.1 Overall Structure of Decoder-only
flowchart TB
A["Input Token Sequence"] --> B["Token Embedding"]
B --> C["Positional Encoding or RoPE"]
C --> D["Decoder-only Block 1"]
D --> E["Decoder-only Block 2"]
E --> F["Decoder-only Block N"]
F --> G["Final LayerNorm"]
G --> H["Linear Output Projection"]
H --> I["Logits"]
I --> J["Softmax"]
J --> K["Next Token Probability Distribution"]
K --> L["Sample or Select Token"]
L --> M["Append to Input Sequence"]
M --> N{"Stop Condition Reached?"}
N -->|"No"| C
N -->|"Yes"| O["Generation Complete"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style O fill:#E5F0E9,stroke:#83A995,color:#315D49
Each Decoder-only Block typically includes:
- Causal Self-Attention.
- Feed Forward Network.
- Residual Connection.
- LayerNorm or RMSNorm.
Some modern models also adopt:
- RoPE positional encoding.
- GQA or MQA attention structure.
- SwiGLU feed-forward network.
- RMSNorm.
- Efficient attention implementations such as FlashAttention.
These are structural designs of specific models, and are not components that all Decoder-only models must have.
5.2 Why Doesn't Decoder-only Need Cross-Attention?
This is the key to understanding Decoder-only.
The original Encoder-Decoder architecture needs to process the source sequence and the target sequence separately:
- The Encoder encodes the source sequence.
- The Decoder generates new Tokens based on the target historical state.
- Cross-Attention enables the Decoder to read the Encoder's representations.
However, Decoder-only uses a unified Token sequence as input.
For example:
User input:
What is artificial intelligence?
Model output:
Artificial intelligence is the technical field that studies how to make computers exhibit intelligent behavior.
The model places the input and the generated content in the same sequence.
It does not need a separate Encoder to encode the user input, nor does it need Cross-Attention to connect two independent networks.
All information is processed through the same causal Self-Attention network.
5.3 The Training Objective of Decoder-only
The core pre-training objective of Decoder-only is usually Causal Language Modeling, also known as autoregressive language modeling.
The model predicts the next Token based on the preceding Tokens:
Where:
- X: the complete Token sequence.
- x_{<t}: all Tokens before the current position.
- x_t: the Token to be predicted at the current position.
Training objective:
For example:
Input:
The development of artificial intelligence cannot be separated from
Prediction target:
data, algorithms, computing power……
During training, the model does not compute the prediction loss only at the last position of the sequence.
Typically, the next-Token prediction loss is computed simultaneously at multiple valid positions.
Because a Causal Mask is used, each position can only see itself and the preceding Tokens.
This allows the training process to compute predictions for multiple positions in parallel while preserving the autoregressive constraint.
5.4 How Does Decoder-only Accomplish Understanding Tasks?
A common question:
The Encoder can understand text bidirectionally, while Decoder-only can only see the left context, so why can it also complete text understanding tasks?
The key is: understanding and generation are not entirely independent capabilities.
In the process of predicting the next Token, Decoder-only needs to learn:
- Semantic relationships between words.
- Syntactic structure.
- Contextual dependencies.
- Relationships between entities and concepts.
- Logical and knowledge associations in the text.
These capabilities can all be gradually learned through the autoregressive language modeling objective.
For example:
Input:
The capital of China is
The model needs to predict:
Beijing
To correctly predict the next Token, the model needs to understand the meaning expressed by the preceding context.
Furthermore, through training methods such as instruction tuning, supervised fine-tuning, and preference optimization, the model can learn to:
- Answer questions.
- Summarize text.
- Translate languages.
- Generate code.
- Analyze documents.
- Execute multi-step tasks.
Therefore, Decoder-only is not limited to generating text; rather, through a unified autoregressive modeling approach, it acquires both contextual understanding and content generation capabilities.
Six: The Core Differences Among the Three Architectures
To understand the three architectures, the most important thing is to distinguish attention scope, information sources, and training objectives.