Back to series
· 9 min read

The Overall Architecture of Transformer: From Encoder and Decoder to the Complete Computation Flow


I. Overview of the Transformer Architecture

Transformer is a deep learning architecture centered on Attention. Through multiple network layers, it performs feature extraction, contextual information fusion, and task output on the input sequence.

The original Transformer uses an Encoder-Decoder architecture and mainly consists of two parts:

  • Encoder: Converts the input sequence into hidden representations containing contextual information.
  • Decoder: Gradually generates the target sequence based on the existing context and the Encoder output.

Overall structure:

flowchart TB
    A["Input sequence"] --> B["Token Embedding"]
    B --> C["Positional Encoding"]
    C --> D["Encoder"]
    D --> E["Encoder output"]
    F["Right-shifted target sequence input"] --> G["Target Embedding"]
    G --> H["Positional Encoding"]
    H --> I["Decoder"]
    E --> I
    I --> J["Linear output projection"]
    J --> K["Softmax"]
    K --> L["Target Token probability distribution"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#E5F0E9,stroke:#83A995,color:#315D49

Note that during training, the Decoder of the original Transformer receives the right-shifted target sequence and uses Masked Self-Attention to avoid accessing future Tokens; during inference, it generates subsequent Tokens step by step based on the content already generated.

Modern large language models usually adopt a Decoder-only architecture and do not necessarily retain the complete Encoder-Decoder structure of the original Transformer.

II. Core Modules of Transformer

From the perspective of network structure, a typical Transformer Layer consists of attention modules, feed-forward networks, residual connections, and normalization modules.

2.1 List of Core Modules

Module Main Function
Token Embedding Maps Token IDs to continuous vectors
Positional Encoding Provides positional information to the model
Multi-Head Attention Establishes information associations between Tokens
Cross-Attention Enables the Decoder to attend to the Encoder output
Feed Forward Network Applies nonlinear transformations to features
Residual Connection Provides a direct propagation path for information and gradients
Layer Normalization Stabilizes feature distributions in the network
Mask Restricts the positions that attention can access
Linear Projection Maps hidden states to the target output space
Softmax Converts Logits into a probability distribution

2.2 Relationships Between Modules

The core computation of Transformer can be summarized as:

flowchart TB
    A["Token Embedding"] --> B["Add positional information"]
    B --> C["Attention information interaction"]
    C --> D["Residual connection and normalization"]
    D --> E["FFN nonlinear transformation"]
    E --> F["Residual connection and normalization"]
    F --> G["Output hidden representation"]
    G --> H["Proceed to next Transformer layer"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

The calculation order shown here is a common Post-LN style. Modern models also widely use the Pre-LN structure, and the exact arrangement varies.

III. Input Layer: Embedding and Positional Encoding

Transformer does not directly process natural language text; it processes numerically represented Tokens.

3.1 Tokenizer: Text Segmentation

First, the Tokenizer splits the input text into Tokens and converts them into Token IDs.

For example:

Input:
I like artificial intelligence

Token:
["I", "like", "artificial intelligence"]

Token ID:
[101, 205, 308]

The Tokens and Token IDs here are illustrative only; the actual segmentation result depends on the Tokenizer's vocabulary and algorithm.

The Tokenizer is responsible for discretizing text. It is not part of the Transformer Layer itself, but it is an important component of the language model's input processing.

3.2 Token Embedding

Token IDs are discrete integers and cannot be used directly as semantic features of the Transformer.

Therefore, an Embedding matrix is needed to map Token IDs to continuous vectors.

Let:

  • The vocabulary size be V.
  • The hidden dimension be d_{model}.

Embedding matrix:

E\in\mathbb{R}^{V\times d_{model}}

For Token ID i:

\mathbf{x}_i=E[i]

If the input sequence length is n, then the entire input representation is:

X\in\mathbb{R}^{n\times d_{model}}

For example:

n=128,\quad d_{model}=768

Then:

X\in\mathbb{R}^{128\times768}

This represents 128 Tokens, with each Token corresponding to a 768-dimensional vector.

3.3 Positional Encoding

Standard Self-Attention itself does not contain sequence order information, so positional encoding is needed to provide position information.

The original Transformer uses sinusoidal positional encoding:

PE(pos,2i)= \sin\left( \frac{pos}{10000^{2i/d_{model}}} \right)
PE(pos,2i+1)= \cos\left( \frac{pos}{10000^{2i/d_{model}}} \right)

The input representation is:

X=Embedding(Token)+PE(position)

Modern Transformers may also use:

  • Learnable absolute positional encoding.
  • Relative positional encoding.
  • RoPE.
  • ALiBi.

Different positional encoding methods have different injection positions and computation mechanisms. For example, RoPE usually operates on the Query and Key in Attention, rather than simply being added to the Token Embedding.

IV. Encoder: Encoder Architecture

The main responsibility of the Encoder is to understand the input sequence and convert it into context-dependent hidden representations.

The Encoder of the original Transformer is formed by stacking N Encoder Layers of identical structure.

Encoder(X)=Layer_N(\cdots Layer_2(Layer_1(X)))

Each Encoder Layer contains two main sublayers:

  1. Multi-Head Self-Attention.
  2. Feed Forward Network.

Each sublayer also uses residual connections and Layer Normalization.

4.1 Encoder Layer Structure

flowchart TB
    A["Input X"] --> B["Multi-Head Self-Attention"]
    B --> C["Residual Addition"]
    A --> C
    C --> D["Layer Normalization"]
    D --> E["Feed Forward Network"]
    E --> F["Residual Addition"]
    D --> F
    F --> G["Layer Normalization"]
    G --> H["Encoder Layer Output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

Here, the Post-LN representation is used, corresponding to:

Y=LN(X+MHA(X))
Z=LN(Y+FFN(Y))

Where:

  • MHA: multi-head self-attention.
  • FFN: feed-forward neural network.
  • LN: Layer Normalization.

4.2 Encoder Self-Attention

The Encoder uses bidirectional Self-Attention.

Each token can attend to other valid tokens in the input sequence, without being limited by causal direction.

For example:

Input:
I like learning Transformer

Processing:
"learning" can attend to "I"
"learning" can attend to "like"
"learning" can attend to "Transformer"

In this way, each position in the Encoder can fuse contextual information from the entire input sequence.

Bidirectional here does not mean the model necessarily understands all semantics; rather, it means the attention computation is not restricted by a left-to-right causal mask.

4.3 Encoder Output

After N Encoder layers, we obtain:

H_{enc}\in\mathbb{R}^{n\times d_{model}}

Where:

  • n: input sequence length.
  • d_{model}: hidden dimension.

The output at each position incorporates contextual information from the input sequence.

These hidden representations can be used for:

  • Text classification.
  • Named entity recognition.
  • Semantic feature extraction.
  • Target sequence generation in machine translation.
  • Other downstream tasks.

5. Decoder: Decoder Architecture

The Decoder in the original Transformer is responsible for generating the target sequence.

Its main differences from the Encoder are:

  • Uses Masked Self-Attention.
  • Adds Encoder-Decoder Cross-Attention.
  • Predicts target tokens in an autoregressive manner.

The original Decoder is stacked from N Decoder Layers.

5.1 Decoder Layer Structure

An original Transformer Decoder Layer contains three main sublayers:

  1. Masked Multi-Head Self-Attention.
  2. Encoder-Decoder Cross-Attention.
  3. Feed Forward Network.

The structure is as follows:

flowchart TB
    A["Decoder Input"] --> B["Masked Multi-Head Self-Attention"]
    B --> C["Residual Add & LayerNorm"]
    A --> C
    C --> D["Cross-Attention"]
    E["Encoder Output"] --> D
    D --> F["Residual Add & LayerNorm"]
    C --> F
    F --> G["Feed Forward Network"]
    G --> H["Residual Add & LayerNorm"]
    F --> H
    H --> I["Decoder Layer Output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

5.2 Masked Self-Attention

The Decoder generates autoregressively.

When predicting the Token at the current position, it can only access the current position and the Tokens before it.

For example:

Target sequence:
I like artificial intelligence

Prediction position 1: can only access "I"
Prediction position 2: can access "I", "like"
Prediction position 3: can access "I", "like", "artificial intelligence"

A causal mask (Causal Mask) is used to mask out future positions:

Attention(Q,K,V) = Softmax\left( \frac{QK^T}{\sqrt{d_k}}+M \right)V

Here, the Mask value corresponding to future positions is negative infinity, which makes their Softmax weight zero.

5.3 Cross-Attention

Cross-Attention is used to establish the information link between the Decoder and the Encoder.

Its key difference from Self-Attention lies in where Q, K, and V come from.

In the original Transformer:

  • Query comes from the Decoder's current hidden state.
  • Key comes from the Encoder output.
  • Value comes from the Encoder output.

Let:

H_{dec}\in\mathbb{R}^{m\times d_{model}}
H_{enc}\in\mathbb{R}^{n\times d_{model}}

Then:

Q=H_{dec}W_Q
K=H_{enc}W_K
V=H_{enc}W_V

Cross-Attention:

CrossAttention(Q,K,V) = Softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V

Where:

  • m: the current target sequence length.
  • n: the source sequence length.

The shape of the attention matrix is:

m\times n

This means that each position in the Decoder can, based on its own state, extract the relevant information from all input positions of the Encoder.

5.4 The Role of Cross-Attention

Take machine translation as an example:

Source language:
I love artificial intelligence.

Target language:
我喜欢人工智能。

The Encoder encodes the source language into a contextual representation.

When generating the target language, the Decoder accesses the Encoder output through Cross-Attention, thereby using the source-language information to generate the target text.

Cross-Attention does not simply concatenate the source-language vectors onto the target-language vectors; instead, it performs dynamic information aggregation via Query, Key, and Value.

6. Differences Between Encoder and Decoder

Both the Encoder and the Decoder use the Transformer Layer, but their attention mechanisms and task objectives differ.

Comparison Dimension Encoder Decoder
Primary task Input understanding and feature encoding Autoregressive generation of the target sequence
Self-Attention Bidirectional attention Causal attention
Causal Mask Usually not used Used
Cross-Attention Not present in the original architecture Used in the original architecture
Input Source sequence Right-shifted input of the target sequence
Output Contextual hidden representations Hidden representations of the target sequence
Typical applications Text understanding, feature extraction Text generation, sequence generation

Note that Encoder and Decoder here refer to the structures in the original Encoder-Decoder Transformer.

Modern Decoder-only models typically have no Cross-Attention and do not need a separate Encoder.

7. Attention Mechanisms in the Transformer

Attention is the core computational module of the Transformer.

7.1 Scaled Dot-Product Attention

Given:

Q\in\mathbb{R}^{n_q\times d_k}
K\in\mathbb{R}^{n_k\times d_k}
V\in\mathbb{R}^{n_k\times d_v}

Compute:

S=\frac{QK^T}{\sqrt{d_k}}
A=Softmax(S)
Output=AV

The complete formula:

Attention(Q,K,V) = Softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V

If a Mask is present, the mask must be applied before Softmax.

7.2 Multi-Head Attention

Multiple Attention Heads compute independently in different representation subspaces:

head_i=Attention(QW_i^Q,KW_i^K,VW_i^V)

After concatenation, an output projection is applied:

MHA(Q,K,V) = Concat(head_1,\ldots,head_h)W^O

Multi-head attention improves the model's ability to learn different patterns of association.

7.3 MHA, MQA and GQA

To reduce KV Cache footprint, modern large language models have developed different attention structures.

Type Query Head Key/Value Head Key characteristics
MHA h h Each Query Head uses its own K/V
MQA h 1 All Query Heads share a single set of K/V
GQA h g Multiple Query Heads share K/V in groups

Where:

1\leq g\leq h

GQA strikes a trade-off between KV Cache footprint and the representational capacity of attention.

8. FFN: The Computation Mechanism of the Feed-Forward Network

Attention is responsible for information exchange between different Tokens, while the FFN is responsible for applying a non-linear transformation to each Token's representation.

8.1 Original FFN Structure

The original Transformer uses two linear transformations and the ReLU activation function:

FFN(x)= ReLU(xW_1+b_1)W_2+b_2

Typically, the hidden dimension is first expanded and then mapped back to the original dimension.

For example:

d_{model}=512
d_{ff}=2048

Then:

512\rightarrow2048\rightarrow512

8.2 Modern FFN Structure

Modern large language models may adopt activation structures such as GELU and SwiGLU.

For example:

SwiGLU(x)= (SiLU(xW_g)\odot xW_v)W_o

Where:

  • W_g: gating branch parameters.
  • W_v: value branch parameters.
  • W_o: output projection parameters.
  • \odot: element-wise multiplication.

The gating mechanism enables the model to dynamically control the transmission of feature information.

8.3 Collaboration Between Attention and FFN

They can be understood as computations at two different levels:

  • Attention: exchanges and aggregates information across sequence positions.
  • FFN: performs nonlinear feature transformations on the fused representation at each position.

The two are stacked alternately, enabling the model to build increasingly complex contextual representations layer by layer.

9. Residual Connections and Layer Normalization

Transformers usually contain dozens or even hundreds of layers.

Without reasonable information propagation and normalization mechanisms, deep networks may face training instability.

9.1 Residual Connection

The basic formula for a residual connection:

y=x+F(x)

Where:

  • x: sublayer input.
  • F(x): output of Attention or FFN.
  • y: residual output.

It provides a direct path for information and gradients to propagate.

9.2 Layer Normalization

LayerNorm normalizes the feature dimension of a single Token.

\mu=\frac{1}{d}\sum_{i=1}^{d}x_i
\sigma^2=\frac{1}{d}\sum_{i=1}^{d}(x_i-\mu)^2
LN(x)= \gamma\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta

Where:

  • \gamma: learnable scaling parameter.
  • \beta: learnable bias parameter.
  • \epsilon: numerical stability term.

9.3 Post-LN and Pre-LN

Post-LN:

y=LN(x+F(x))

Pre-LN:

y=x+F(LN(x))

The main difference between the two structures lies in the position of LayerNorm.

Structure Normalization Position Characteristics
Post-LN After residual addition Used by the original Transformer
Pre-LN Before sublayer computation Widely used in modern deep models

Pre-LN generally offers more stable deep training characteristics, but the specific design still depends on the model architecture and training configuration.

10. Tensor Dimensions: Understanding Data Flow in Transformer

To understand the Transformer architecture, one must master the tensor dimension changes across modules.

Assume:

  • Batch Size: B=32
  • Sequence Length: N=128
  • Hidden Dimension: d_{model}=768
  • Number of Attention Heads: h=12
  • Per-Head Dimension: d_k=d_v=64

10.1 Input Tensor

Embedding output:

X\in\mathbb{R}^{B\times N\times d_{model}}

Substituting:

X\in\mathbb{R}^{32\times128\times768}

10.2 Q, K, V Projections

After linear projections:

Q,K,V\in \mathbb{R}^{32\times128\times768}

Then split into multiple attention heads:

Q,K,V\in \mathbb{R}^{32\times12\times128\times64}

Dimension meanings:

Dimension Meaning
32 Batch Size
12 Number of Attention Heads
128 Sequence Length
64 Head Dimension

10.3 Attention Score Matrix

Compute:

QK^T

This yields:

A\in \mathbb{R}^{32\times12\times128\times128}

Each Batch and each Head has a 128\times128 attention score matrix.

10.4 Attention Output

Multiply the attention weights by V:

Output=AV

This yields:

Output\in \mathbb{R}^{32\times12\times128\times64}

Concatenate multiple heads:

Output\in \mathbb{R}^{32\times128\times768}

Then apply an output projection, keeping the hidden dimension unchanged.

10.5 FFN Dimension Changes

Assume:

d_{ff}=3072

Then:

[32,128,768] \rightarrow [32,128,3072] \rightarrow [32,128,768]

The input and output of a Transformer Layer usually maintain the same hidden dimension to enable residual connections.

10.6 Summary of Dimension Changes

Computation Stage Tensor Shape
Embedding output [32, 128, 768]
Q/K/V projection [32, 128, 768]
Multi-head split [32, 12, 128, 64]
Attention Score [32, 12, 128, 128]
Attention output [32, 12, 128, 64]
Head concatenation [32, 128, 768]
FFN first layer [32, 128, 3072]
FFN second layer [32, 128, 768]

These dimensions are a typical example of standard MHA. GQA, MQA, different hidden dimensions, and different attention implementations will vary accordingly.

11. Original Transformer and Modern LLM Architectures

The original Transformer uses an Encoder-Decoder structure, but most modern large language models adopt a Decoder-only architecture.