Back to series
· 11 min read

What Is a Transformer? From Self-Attention to the Core Architecture of Modern Large Language Models


I. What Is a Transformer?

A Transformer is a deep neural network architecture based on the attention mechanism, proposed by a Google research team in the 2017 paper “Attention Is All You Need.”

It was initially used for machine translation tasks, then gradually became the mainstream architecture in natural language processing and expanded into fields such as computer vision, speech recognition, multimodal learning, and generative artificial intelligence.

The most important design principle of the Transformer is:

Directly establish information associations between different positions in a sequence through the attention mechanism, rather than relying on recurrent structures to process the sequence step by step.

Before Transformers emerged, sequence modeling mainly relied on recurrent neural networks such as RNN, LSTM, and GRU.

These models process input sequentially by time step:

Token 1 → Token 2 → Token 3 → Token 4

The computation at the current moment depends on the hidden state from the previous moment, so there is a certain serial computation limitation.

The Transformer, by contrast, uses Self-Attention to let tokens in a sequence directly compute their associations with one another.

Input Token Sequence
      ↓
Embedding
      ↓
Positional Encoding
      ↓
Self-Attention
      ↓
Feedforward Neural Network
      ↓
Multi-layer Transformer
      ↓
Output Representation

The main features of the Transformer are:

  • Parallel computation: Multiple tokens in the same layer can be computed in parallel.
  • Global dependency modeling: A token can directly attend to other tokens in the sequence.
  • Contextual representation: Each token’s representation can integrate information from other positions.
  • Modular structure: Attention, FFN, residual connections, and normalization can be combined into multi-layer networks.
  • Scalability: Large models can be built by increasing the number of layers, hidden dimensions, and parameter scale.

II. Why Do We Need Transformers?

To understand the Transformer, we first need to understand the problems faced by traditional sequence models.

2.1 The Computation Mechanism of RNN

A Recurrent Neural Network (RNN) passes historical information through hidden states.

For an input sequence:

X=(x_1,x_2,\ldots,x_T)

RNN performs the following at each time step:

h_t=f(W_xx_t+W_hh_{t-1}+b)

where:

  • x_t: input at the current time step.
  • h_t: current hidden state.
  • h_{t-1}: hidden state at the previous time step.
  • W_x: input weight matrix.
  • W_h: recurrent weight matrix.
  • f: nonlinear activation function.

Its computational dependencies are:

flowchart LR
    A["Input x₁"] --> B["RNN Cell"]
    B --> C["Hidden State h₁"]
    C --> D["RNN Cell"]
    E["Input x₂"] --> D
    D --> F["Hidden State h₂"]
    F --> G["RNN Cell"]
    H["Input x₃"] --> G
    G --> I["Hidden State h₃"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

The main problems with RNNs are:

  1. Serial computation: The current time step must wait for the previous time step to complete, making it difficult to fully utilize GPU parallelism.
  2. Long-distance dependencies: Information from earlier positions must pass through multiple state transitions before it can affect later positions.
  3. Gradient problems: Training on long sequences is prone to vanishing or exploding gradients.
  4. Information compression: Historical information must be continuously integrated into a fixed-dimensional hidden state, which may cause information loss.

LSTM and GRU improve long-term dependency modeling through gating mechanisms, but they still retain a recurrent computation structure.

2.2 Transformer’s Solution Approach

Transformer no longer passes information step by step through recurrent hidden states; instead, it uses attention mechanisms to directly compute the relationships between Tokens.

For example:

I deposit money at the bank

When the model processes “bank,” it can attend to “deposit money” through Attention, thereby obtaining a semantic representation that better fits the context.

This mechanism allows different positions in the sequence to establish direct connections.

Comparison Dimension RNN / LSTM Transformer
Core Mechanism Recurrent state propagation Attention mechanism
Sequence Processing Has time-step dependency Can be processed in parallel during training
Long-Range Dependency Requires step-by-step propagation Can directly establish associations
Parallelization Capability Limited by recurrent structure Tokens in the same layer can be processed in parallel
Computational Complexity Typical recurrent computation takes about O(n) time steps Full attention computation is about O(n^2d)
Typical Applications Time series, speech, early NLP LLMs, machine translation, multimodal models

Note that Transformer is not more efficient than RNN in all scenarios. The computation cost of standard Self-Attention and the storage required for the attention matrix grow quadratically with sequence length, so long-context scenarios require additional optimization.

3. Overall Architecture of Transformer

The original Transformer uses an Encoder-Decoder architecture.

It consists of two parts overall:

  • Encoder: Responsible for understanding the input sequence and generating contextual representations.
  • Decoder: Generates the target sequence based on the existing context.

3.1 Encoder

The Encoder is formed by stacking multiple structurally identical Encoder Layers.

Each layer mainly contains:

  1. Multi-Head Self-Attention.
  2. Feed Forward Network.
  3. Residual Connection.
  4. Layer Normalization.

Its structure can be abstracted as:

flowchart TB
    A["Input Representation"] --> B["Multi-Head Self-Attention"]
    B --> C["Residual Connection and Layer Normalization"]
    C --> D["Feed Forward Network"]
    D --> E["Residual Connection and Layer Normalization"]
    E --> F["Output Representation"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49

After passing through multiple Encoder layers, each Token can integrate contextual information from other Tokens in the input sequence.

3.2 Decoder

The Decoder of the original Transformer is likewise formed by stacking multiple Decoder Layers.

Each layer contains:

  1. Masked Multi-Head Self-Attention.
  2. Encoder-Decoder Cross-Attention.
  3. Feed Forward Network.
  4. Residual Connection.
  5. Layer Normalization.

Its structure is as follows:

flowchart TB
    A["Decoder Input"] --> B["Masked Multi-Head Self-Attention"]
    B --> C["Residual Connection and Layer Normalization"]
    C --> D["Encoder-Decoder Cross-Attention"]
    D --> E["Residual Connection and Layer Normalization"]
    E --> F["Feed Forward Network"]
    F --> G["Residual Connection and Layer Normalization"]
    G --> H["Decoder Output"]

    I["Encoder Output"] --> D

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Where:

  • Masked Self-Attention: restricts the current position to only access the current position and positions before it.
  • Cross-Attention: allows the Decoder to attend to the information output by the Encoder.
  • FFN: applies a non-linear transformation to the representation at each position.

3.3 The Original Transformer and Modern Large Language Models

Modern large language models do not necessarily adopt the complete Encoder-Decoder structure.

Based on their architecture, they can be divided into three categories:

Architecture Main Characteristics Typical Models
Encoder-only Bidirectional context modeling, suitable for understanding tasks BERT
Decoder-only Causal attention, suitable for autoregressive generation GPT, Llama, Qwen
Encoder-Decoder Encodes the input, then generates the output through the Decoder T5, the original Transformer

Encoder-only

It can utilize context information from both before and after the current position at the same time, and is commonly used for:

  • Text classification.
  • Named entity recognition.
  • Semantic understanding.
  • Feature extraction.

Decoder-only

It restricts the attention scope through a causal mask and predicts the next Token in left-to-right order, and is commonly used for:

  • Dialogue generation.
  • Code generation.
  • Text continuation.
  • General-purpose large language models.

Encoder-Decoder

It first encodes the input, then generates the target sequence based on the encoding result, and is commonly used for:

  • Machine translation.
  • Text summarization.
  • Text transformation.
  • Sequence-to-sequence tasks.

IV. Self-Attention: The Core Mechanism of the Transformer

Self-Attention is the most core component of the Transformer.

Its role is:

To dynamically aggregate information from other Tokens for each Token according to the degree of association between different Tokens in the sequence.

4.1 Why Do We Need Attention?

Consider the following sentence:

Xiaoming put the book on the table because it is relatively stable.

What "it" refers to here needs to be determined in combination with the context.

When the model processes "it", it needs to consider the words that appeared earlier and their semantic relationships.

Self-Attention allows the current position to directly compute association weights with other positions.

Unlike RNNs, which rely on step-by-step propagation, Attention can directly establish information interaction between distant positions.

4.2 Query, Key, Value

The core of Self-Attention is three vectors:

  • Query (Q): the query vector, representing what information the current position wants to find.
  • Key (K): the key vector, representing what information the current position can provide.
  • Value (V): the value vector, representing the content of the information actually provided by the current position.

For the input matrix:

X\in\mathbb{R}^{n\times d_{model}}

A linear transformation is performed through three learnable weight matrices:

Q=XW_Q
K=XW_K
V=XW_V

Where:

W_Q,W_K\in\mathbb{R}^{d_{model}\times d_k}
W_V\in\mathbb{R}^{d_{model}\times d_v}

Therefore:

Q\in\mathbb{R}^{n\times d_k}
K\in\mathbb{R}^{n\times d_k}
V\in\mathbb{R}^{n\times d_v}

Here, n represents the sequence length.

Note that Q, K, and V are not manually defined semantic labels, but different linear projections learned through training.

4.3 The Attention Computation Process

First, compute the dot product between Query and Key:

QK^T

This gives an n\times n matrix.

Each element in the matrix represents the matching score between a Query and a Key.

Then scale it:

\frac{QK^T}{\sqrt{d_k}}

Then apply Softmax to convert it into normalized weights:

A=Softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)

Finally, use the attention weights to compute a weighted sum of Value:

Output=AV

Complete formula:

\boxed{ Attention(Q,K,V) = Softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V }

Where:

  • QK^T: computes the relevance scores between Tokens.
  • \sqrt{d_k}: scaling factor.
  • Softmax: converts scores into a weight distribution.
  • V: provides the information actually aggregated.

4.4 Why divide by \sqrt{d_k}?

This is a very important design in Scaled Dot-Product Attention.

Assume that each dimension of Q and K is an independent random variable with mean 0 and variance 1.

The dot product:

q\cdot k=\sum_{i=1}^{d_k}q_ik_i

Under the above assumptions, its variance is approximately:

Var(q\cdot k)=d_k

As the dimension increases, the fluctuation in the dot product values also increases.

If larger dot products are fed directly into Softmax, the distribution may become too sharp.

For example:

Softmax([1,2,3])

and:

Softmax([10,20,30])

By comparison, the latter usually produces a more concentrated probability distribution.

A too sharp Softmax may make gradients smaller and affect training stability.

Therefore, we use:

\frac{QK^T}{\sqrt{d_k}}

to scale the dot product so that its numerical scale is more stable.

Note that this is a scale-control method based on statistical assumptions; it does not strictly guarantee constant variance for all input distributions.

4.5 Complete Attention Computation Flow

flowchart TB
    A["Input matrix X"] --> B["Linear projection"]
    B --> C["Generate Q, K, V"]
    C --> D["Compute the dot product of Q and K"]
    D --> E["Divide by √dₖ to scale"]
    E --> F["Compute attention weights with Softmax"]
    F --> G["Multiply by V"]
    G --> H["Obtain the Attention output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

4.6 A Simple Numerical Example

Suppose there are three Tokens:

I like AI

After linear projection, we obtain Q, K, and V.

To simplify the calculation, assume:

Q= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}
K= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}

Then:

QK^T= \begin{bmatrix} 1&0&1\\ 0&1&1\\ 1&1&2 \end{bmatrix}

Since:

d_k=2

After scaling:

S= \frac{1}{\sqrt{2}} \begin{bmatrix} 1&0&1\\ 0&1&1\\ 1&1&2 \end{bmatrix}

Apply Softmax to each row of the matrix to obtain the attention weight matrix:

A=Softmax(S)

Each row represents the attention distribution of one Query over all Keys.

Finally:

Output=AV

This means the output of each Token is obtained by aggregating the Values of multiple Tokens according to their weights.

V. Multi-Head Attention

A single Attention can only compute associations based on one set of Q, K, V.

But language contains many kinds of relationships:

  • Syntactic relationships.
  • Semantic relationships.
  • Coreference relationships.
  • Positional relationships.
  • Long-distance dependencies.
  • Domain associations.

Multi-Head Attention uses multiple independent attention heads to let the model learn different informational relationships in different representation subspaces.

5.1 Computation of Multi-Head Attention

For the i-th attention head:

head_i= Attention( XW_i^Q, XW_i^K, XW_i^V )

The outputs of multiple attention heads are concatenated:

Concat(head_1,head_2,\ldots,head_h)

Then passed through one more linear projection:

MultiHead(Q,K,V) = Concat(head_1,\ldots,head_h)W^O

Where:

  • h: the number of attention heads.
  • W_i^Q: the Query projection matrix of the i-th head.
  • W_i^K: the Key projection matrix of the i-th head.
  • W_i^V: the Value projection matrix of the i-th head.
  • W^O: the output projection matrix.

5.2 Structure of Multi-Head Attention

flowchart TB
    A["Input X"] --> B["Linear Projection"]
    B --> C["Attention Head 1"]
    B --> D["Attention Head 2"]
    B --> E["Attention Head 3"]
    B --> F["Attention Head h"]
    C --> G["Concat"]
    D --> G
    E --> G
    F --> G
    G --> H["Output Linear Projection"]
    H --> I["Multi-Head Attention Output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

Suppose:

d_{model}=768
h=12

If all heads have the same dimension, each head typically has:

d_k=d_v=\frac{768}{12}=64

Each attention head computes attention in a 64-dimensional subspace, and the results are finally concatenated back into 768 dimensions.

5.3 Why Are Multiple Heads Needed?

Different heads can learn different attention patterns.

For example, some heads may focus more on:

  • Adjacent Tokens.
  • Distant Tokens.
  • Subject–predicate relationships.
  • Relationships between pronouns and entities.
  • Specific structures within a sentence.

But one should not simply assume that every head corresponds to a well-defined linguistic function.

In actual training, the functions of multiple heads may overlap, and some heads may be underutilized.

The core value of Multi-Head Attention is to increase the diversity of attention representations.

5.4 MHA, MQA, and GQA

During inference in large language models, Keys and Values usually need to be cached; this is the KV Cache.

As the number of model layers and the context length increase, the KV Cache consumes a large amount of GPU memory.

Therefore, different attention structures have emerged:

Structure Query Heads Key/Value Heads Characteristics
MHA Multiple Same as the number of Query Heads Flexible expressive capacity, larger KV Cache
MQA Multiple 1 Multiple Query Heads share a single set of K/V
GQA Multiple Fewer than the Query Heads A compromise between expressive capacity and cache cost
  • MHA: Multi-Head Attention.
  • MQA: Multi-Query Attention.
  • GQA: Grouped-Query Attention.

GQA groups multiple Query Heads, with each group sharing one set of Keys and Values.

These structures are mainly used to improve the inference efficiency of large models, especially to reduce the GPU memory footprint and memory access pressure of the KV Cache.

6. Positional Encoding: How Does a Transformer Understand Order?

The Self-Attention of a Transformer does not inherently contain the sequential order of Tokens.

If only Token Embeddings are fed into Attention, the model cannot reliably identify the original sequence order from the input set alone.

Therefore, positional information needs to be introduced.

6.1 Why Is Positional Encoding Needed?

Consider:

cat chases dog
dog chases cat

The two sentences contain the same Tokens, but the order differs, and so does the meaning.

Without positional information, standard Self-Attention is permutation-equivariant with respect to the arrangement of input Tokens: if the input positions are rearranged, the output is rearranged accordingly, and the model does not automatically know the meaning of the original positions.

Therefore, positional information must be provided to the model.

6.2 Sinusoidal Positional Encoding

The original Transformer constructs positional encodings using sine and cosine functions.

For position pos and dimension i:

PE(pos,2i) = \sin\left( \frac{pos}{10000^{2i/d_{model}}} \right)
PE(pos,2i+1) = \cos\left( \frac{pos}{10000^{2i/d_{model}}} \right)

Where:

  • pos: the position of the Token in the sequence.
  • i: the vector dimension index.
  • d_{model}: the model hidden dimension.

Different dimensions have different frequencies, allowing the positional encoding to express positional variation at different scales.

The original Transformer adds the positional encoding to the Token Embedding:

X=Embedding(Token)+PE(position)

6.3 Learnable Positional Encoding

Another approach is to make the position vectors directly learnable parameters.

Assuming a maximum sequence length of L and a hidden dimension of d, the positional encoding matrix is:

P\in\mathbb{R}^{L\times d}

The model learns the vector corresponding to each position through training.

Advantages:

  • Simple to implement.
  • Position representations can be learned according to the task.

Limitations:

  • Usually constrained by the maximum position length seen during training.
  • Beyond the trained position range, good generalization cannot be directly guaranteed.

6.4 Relative Positional Encoding

Relative positional encoding focuses on the relative distance between two Tokens, rather than merely their respective absolute positions.

For example:

Token A and Token B are 2 positions apart
Token A and Token C are 5 positions apart

The model can use this relative relationship to establish attention associations.

Relative positional encoding and its variants are very important in modern Transformers.

6.5 RoPE: Rotary Positional Embedding

RoPE (Rotary Positional Embedding) is a positional encoding method widely used in modern Decoder-only large language models.

Its core idea is:

By applying a position-dependent rotation transformation to the Query and Key vectors, positional information is incorporated into the attention computation.

For a two-dimensional vector:

\mathbf{x}= \begin{bmatrix} x_1\\ x_2 \end{bmatrix}

At position m, the rotation transformation can be expressed as:

R(m\theta)= \begin{bmatrix} \cos(m\theta)&-\sin(m\theta)\\ \sin(m\theta)&\cos(m\theta) \end{bmatrix}

Then:

\mathbf{x}_m=R(m\theta)\mathbf{x}

RoPE groups different dimensions for rotation and applies it to both Query and Key.

In this way, the dot product between two positions can reflect relative positional information.

The advantages of RoPE include:

  • Integrate position information into the Attention computation.
  • Can naturally express relative positional relationships.
  • Suitable for autoregressive language models.
  • Can be combined with methods such as position interpolation to extend context length.

However, RoPE does not mean that a model can extend its context indefinitely. Long-context capability is also affected by factors such as training data, attention mechanisms, position frequency configuration, and inference strategies.

6.6 Comparison of Mainstream Position Encoding Methods

Type Core Idea Representative Characteristics
Sinusoidal Uses sine and cosine functions Used by the original Transformer
Learned Absolute Learns absolute position vectors Parametric and simple to implement
Relative Position Models relative distances between positions Focuses on relative relationships
RoPE Applies position-dependent rotations to Q and K Widely used in modern LLMs
ALiBi Adds position-dependent bias to attention scores Models positions via linear distance bias

7. Feed Forward Network: Feedforward Neural Network

Attention is responsible for information exchange between tokens.

But Attention alone is not enough; the Transformer also needs to apply further nonlinear transformations to the representation at each position.

This is the Feed Forward Network (FFN).

7.1 Basic Structure of FFN

The original Transformer uses two linear transformations, with a ReLU activation function in between:

FFN(x) = \max(0,xW_1+b_1)W_2+b_2

Where:

  • W_1: Weight matrix of the first layer.
  • b_1: Bias of the first layer.
  • W_2: Weight matrix of the second layer.
  • b_2: Bias of the second layer.

Usually, the hidden dimension is first expanded and then mapped back to the original dimension.

For example:

d_{model}=768
d_{ff}=3072

The computation process is:

768\rightarrow3072\rightarrow768

7.2 Differences Between Attention and FFN

Aspect Attention FFN
Main role Information exchange between tokens Applies nonlinear transformation to a single position
Input relationship Multiple positions in the same sequence Computed independently at each position
Core computation Q, K, V, Softmax Linear layers and activation functions
Information scope Can aggregate across tokens Usually does not directly interact across tokens
Role Context information fusion Feature transformation and enhanced representational capacity

Note that in a standard Transformer, the FFN in the same layer is usually applied independently to each token position, but its parameters are shared across all positions.

7.3 Changes in Modern FFN

Modern large language models do not necessarily continue to use the original ReLU FFN.

Common structures include:

  • GELU.
  • SwiGLU.
  • GEGLU.