1. What Is Multi-Head Attention?
Multi-Head Attention (MHA) is the attention mechanism in Transformers that strengthens the ability to exchange information.
Its core idea is:
Instead of using only one set of Query, Key, and Value to compute attention, it uses multiple sets of projection matrices to run Attention in parallel across multiple representation subspaces, and finally fuses the outputs of all attention heads.
In single-head Self-Attention, the model builds relationships between Tokens through one set of Q, K, V.
Multi-Head Attention, by contrast, uses multiple attention heads, and each Head has its own independent Q, K, V projection parameters.
For example, a model with 12 attention heads can run 12 different attention computations at the same time.
flowchart TB
A["Input hidden representation X"] --> B["Multiple independent linear projections"]
B --> C["Head 1"]
B --> D["Head 2"]
B --> E["Head 3"]
B --> F["Head 12"]
C --> G["Concat Concatenation"]
D --> G
E --> G
F --> G
G --> H["Output linear projection"]
H --> I["Multi-Head Attention Output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
Multi-head attention is not simply repeating one Attention computation many times; rather, through different learnable projection matrices, each Head learns how to match and aggregate information in a different representation subspace.
2. Why Do We Need Multi-Head Attention?
To understand Multi-Head Attention, you first need to understand the limitations of single-head attention.
2.1 How Single-Head Attention Is Computed
Standard Scaled Dot-Product Attention:
Where:
- Q: Query, the query vector.
- K: Key, the key vector.
- V: Value, the value vector.
- d_k: the feature dimension of Key.
Assume the input hidden dimension is:
Single-head attention can project Q, K, V directly into a 768-dimensional space:
Then it performs one Attention computation.
This means the entire attention mechanism uses only one set of projection spaces for matching and information aggregation.
2.2 The Limits of a Single Representation Space
Token relationships in natural language are diverse.
For example:
Artificial intelligence is changing the way software is developed.
The model may need to model all of the following at once:
- The semantic relationship between artificial intelligence and changing.
- The compositional relationship between software and development.
- The grammatical relationship between subject and predicate.
- The contextual associations between different positions.
These relationships cannot necessarily be fully expressed through a single projection space.
If there is only one attention head, all information matching has to be done through the same set of Q, K, V projections.
Multi-head attention allows the model to use multiple sets of projection parameters to learn different patterns of association.
Note that this does not mean each Head will automatically form a clear linguistic function. In practice, attention heads may overlap in function, and may also learn complex compositional features.
2.3 The Core Advantages of Multi-Head Attention
| Advantage | Description |
|---|---|
| Multi-subspace modeling | Learn associations in different projection spaces |
| Parallel computation | Multiple Heads can perform attention operations in parallel |
| Expressiveness | Increases the representational flexibility of the attention mechanism |
| Information fusion | Combine the results of different Heads into a unified representation |
| End-to-end learning | The projection parameters of each Head are optimized automatically through training |
3. The Overall Structure of Multi-Head Attention
Multi-Head Attention can be broken down into five main steps:
- Input the hidden representation.
- Generate multiple sets of Q, K, V.
- Each Head computes Attention independently.
- Concatenate the outputs of all Heads.
- Fuse the information through the output projection matrix.
flowchart TB
A["Input X"] --> B["Generate multiple sets of Q, K, V"]
B --> C["Multiple Attention Heads compute in parallel"]
C --> D["Concatenate outputs of each Head"]
D --> E["Output linear projection"]
E --> F["Obtain final output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
Below, we expand from the perspective of mathematical computation.
4. Step 1: Generate multiple sets of Q, K, V
Assume the input hidden representation is:
Where:
- n: sequence length.
- d_{model}: model hidden dimension.
Assume the number of attention heads is h.
Classic Multi-Head Attention usually evenly assigns the hidden dimension to each Head:
For example:
Then:
Each Head uses its own projection matrices:
The computation for the i-th Head is:
Where:
4.1 Why does each Head use its own projection matrices?
If all Heads share the same set of Q, K, V projection matrices, then the input representations they obtain will be the same.
Even if the subsequent computation uses the same structure, it is difficult for them to form truly independent attention representations.
By assigning independent parameters to each Head:
the model can learn different matching patterns in different projection spaces.
For example:
- Head 1 may focus more on a certain type of local correlation.
- Head 2 may focus more on long-distance information.
- Head 3 may focus more on specific contextual features.
These are possibilities to aid understanding, not fixed assignments of Head functions by the model.
4.2 Matrix implementation of multiple sets of projections
In actual engineering practice, a separate linear layer is usually not actually invoked for each Head.
A more common approach is to combine the projection matrices of all Heads:
Compute them all at once:
Then reorganize the results into multiple Heads.
flowchart TB
A["Input X"] --> B["Q Linear Projection"]
A --> C["K Linear Projection"]
A --> D["V Linear Projection"]
B --> E["Q Dimension Reshape"]
C --> F["K Dimension Reshape"]
D --> G["V Dimension Reshape"]
E --> H["Multiple Query Heads"]
F --> I["Multiple Key Heads"]
G --> J["Multiple Value Heads"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
This approach can fully exploit the parallel computing capability of matrix multiplication and reduce the scheduling overhead caused by many independent small-matrix operations.
5. Step 2: Each Head Independently Performs Attention
After generating multiple sets of Q, K, and V, each Head independently performs standard Scaled Dot-Product Attention.
The i-th Head:
Expanded:
Each Head has its own:
- Query representation.
- Key representation.
- Value representation.
- Attention score matrix.
- Attention weight matrix.
- Output representation.
5.1 Computation Dimensions of Each Head
Assume:
Then:
After Softmax:
Finally:
This yields:
The output sequence length of each Head remains unchanged, but the feature dimension is 64.
5.2 Parallel Computation of Multi-Head Attention
Assume there are 4 Heads:
flowchart TB
A["Input X"] --> B["Head 1: Attention"]
A --> C["Head 2: Attention"]
A --> D["Head 3: Attention"]
A --> E["Head 4: Attention"]
B --> F["head 1"]
C --> G["head 2"]
D --> H["head 3"]
E --> I["head 4"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
These Heads are mathematically independent of one another and can be executed in parallel.
On GPUs, practical implementations usually combine the Head dimension with the Batch dimension to organize computation, so as to use efficient batched matrix operations.
6. Step 3: Concatenate All Heads
After all Heads have finished computing, their outputs need to be combined.
Assume:
There are h heads in total.
Concatenation:
This gives:
In the classic Transformer:
Therefore:
6.1 Concatenation Is Not Summation
Concat and Sum are two different operations.
Suppose the outputs of two heads are:
Concat:
Sum:
Concat preserves the feature dimensions of different heads, whereas Sum directly adds the features at corresponding positions.
Multi-head attention uses Concat to preserve the output information of each head for the subsequent projection layer to fuse.
6.2 Dimensions After Concatenation
Assume:
Then:
After concatenation:
At this point, the hidden dimension is restored to 768.
VII. Step 4: Output Linear Projection
After concatenation, the result still needs to pass through an output projection matrix:
The final output is:
where:
In the classic configuration:
Therefore:
Finally:
7.1 Why Is an Output Projection Still Needed?
Concat merely places the features of different heads together; it does not actively learn how to fuse these features.
The output projection matrix W^O is responsible for performing a learnable linear combination of the concatenated features.
It can:
- Combine information from different heads.
- Learn relationships between different feature dimensions.
- Map the concatenation result to the model's hidden dimension.
- Provide a unified representation space for subsequent network layers.
7.2 Output Projection Is Not Merely Dimension Restoration
Although in the classic configuration the dimension after concatenation is already equal to d_{model}, the output projection still plays an important role.
Even if the input and output dimensions are the same, matrix multiplication still recombines the features of different heads.
Therefore, the output projection is not only a shape adjustment but also part of multi-head information fusion.
VIII. Complete Mathematical Formula of Multi-Head Attention
Combine all the steps.
8.1 A Single Head
The i-th head:
8.2 Multi-Head Concatenation
8.3 Output Projection
Complete formula:
where:
and:
Complete computation structure:
flowchart TB
A["Input hidden representation X"] --> B["Generate multiple sets of Q, K, V"]
B --> C["Each Head independently computes Scaled Dot-Product Attention"]
C --> D["Concat concatenation"]
D --> E["Output projection Wᴼ"]
E --> F["Final output O"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
9. Tensor Dimensions: How Does Multi-Head Attention Organize Data?
To understand Multi-Head Attention, you must grasp how tensor dimensions change.
Assume:
| Parameter | Value | Meaning |
|---|---|---|
| Batch Size | 32 | Batch size |
| Sequence Length | 128 | Sequence length |
| Hidden Dimension | 768 | Hidden dimension |
| Number of Heads | 12 | Number of attention heads |
| Head Dimension | 64 | Dimension per head |
Satisfying:
9.1 Input Tensor
9.2 Q, K, V Projection
Perform linear transformations separately:
9.3 Split the Head Dimension
Split the last dimension into:
Then adjust the dimension order:
Here, the meaning of each dimension is:
- 32: Batch.
- 12: Head.
- 128: Sequence Length.
- 64: Head Dimension.
9.4 Compute Attention Score
Each Head computes separately:
This yields:
The last two dimensions of the attention matrix correspond to the Query position and the Key position, respectively.
9.5 Softmax
This yields:
9.6 Aggregate Value
This yields:
9.7 Merge Heads
Merge the Head dimension with the Head Dimension:
9.8 Output Projection
After passing through W^O:
Complete dimension changes:
| Stage | Tensor Shape |
|---|---|
| Input X | [32, 128, 768] |
| Q/K/V Projection | [32, 128, 768] |
| Head Split | [32, 12, 128, 64] |
| Attention Score | [32, 12, 128, 128] |
| Softmax Weights | [32, 12, 128, 128] |
| Value Aggregation | [32, 12, 128, 64] |
| Head Merging | [32, 128, 768] |
| Output Projection | [32, 128, 768] |
10. Parameter Count of Multi-Head Attention
The parameter count of multi-head attention mainly comes from four sets of linear projections:
- Query projection.
- Key projection.
- Value projection.
- Output projection.
Assume:
and:
Then:
The parameter count of each matrix is approximately:
Therefore, the total parameter count is approximately:
If each linear layer includes a bias, this adds approximately:
10.1 Parameter Count Example
Assume:
Then the parameter count of the four projection matrices is:
If four sets of biases are included:
Total parameter count:
This does not include the parameters of other Transformer components, such as FFN, LayerNorm, etc.
10.2 Does the Number of Heads Increase the Total Parameter Count?
When d_{model} is kept unchanged and the dimension of each head is adjusted proportionally, the projection parameter count of standard MHA basically does not change with the number of heads.
For example:
| Hidden Dimension | Number of Heads | Head Dimension | Total Projection Parameter Count (Excluding Bias) |
|---|---|---|---|
| 768 | 8 | 96 | 2,359,296 |
| 768 | 12 | 64 | 2,359,296 |
| 768 | 16 | 48 | 2,359,296 |
As the number of heads increases, the dimension of each individual head decreases accordingly.
However, this does not mean the computational behavior is exactly the same. Different numbers of heads change the attention grouping, the partition of the projection space, and the model's representation.
11. Computational Complexity of Multi-Head Attention
The computation of Multi-Head Attention mainly includes:
- Linear projections of Q, K, and V.
- QK transposed dot product.
- Softmax.
- Multiplication of the attention weights by V.
- Output linear projection.
Assume:
- The sequence length is n.
- The hidden dimension is d.
- The number of heads is h.
- The dimension of each head is d/h.
11.1 Q, K, V Projections
The computational cost of the three linear projections is approximately:
11.2 QK Transposed Dot Product
Computational cost per head:
Total across all heads:
This gives:
11.3 Multiplication of Attention and V
The complexity is likewise approximately:
11.4 Output Projection
The complexity is approximately:
11.5 Total Complexity
Ignoring relatively small operations such as Softmax and Mask:
More specifically:
The constant coefficients here depend on how FLOPs are counted.
Note that:
- nd^2 mainly comes from linear projections.
- n^2d mainly comes from attention score computation and Value aggregation.
When the sequence length is very large, the quadratic term becomes an important computational and memory bottleneck.
12. Why Is the Attention Matrix Prone to Becoming a GPU Memory Bottleneck?
Multi-head attention needs to compute an attention matrix for each batch and each head:
The number of its elements is:
Assume:
Number of elements in the attention matrix:
This is approximately:
If each element uses FP16 or BF16 and occupies 2 bytes, simply storing one such matrix in full requires about 12 GiB of GPU memory.
Actual training may also involve Softmax intermediate results, data required for backpropagation, and other activations, so the GPU memory pressure is even greater.
12.1 FlashAttention
FlashAttention is an important technique optimized for attention computation.
Through tiled computation, online Softmax, and optimized GPU memory access, it reduces the materialization and storage of the full attention matrix.
Core objectives:
- Reduce costly GPU memory reads and writes.
- Lower the GPU memory usage of intermediate matrices.
- Improve GPU computational efficiency.
- Preserve the standard attention computation objective and obtain equivalent or approximate numerical results within the allowed numerical precision.
Note that FlashAttention does not change the basic Q, K, V structure of Multi-Head Attention; instead, it optimizes its computational implementation.
13. MHA, MQA, GQA: Three Architectures of Multi-Head Attention
Standard MHA uses an independent Key head and Value head for each Query head.
However, during large language model inference, the KV Cache occupies a large amount of GPU memory.
To reduce KV Cache costs, MQA and GQA have emerged in research and engineering practice.
13.1 MHA: Multi-Head Attention
Standard MHA:
- Multiple Query heads.
- Multiple Key heads.
- Multiple Value heads.
- The number of Query heads is the same as the number of KV heads.
Assume:
Each Query head has a corresponding KV head.
13.2 MQA: Multi-Query Attention
MQA retains multiple Query heads, but all Query heads share one set of Keys and Values.
For example:
Its advantages are:
- Reduce the storage size of the KV Cache.
- Lower the bandwidth requirement for reading KV data during inference.
- Improve efficiency in some autoregressive inference scenarios.
However, sharing KV also changes how the model expresses information and may introduce a trade-off between effectiveness and efficiency.
13.3 GQA: Grouped-Query Attention
GQA is a compromise architecture between MHA and MQA.
Multiple Query heads are grouped to share one set of Keys and Values.
For example:
Every 4 Query heads share one set of KV heads.
13.4 Comparison of the Three Architectures
| Architecture | Query Head | KV Head | KV Sharing Method |
|---|---|---|---|
| MHA | 16 | 16 | Each Query head uses KV independently |
| MQA | 16 | 1 | All Query heads share KV |
| GQA | 16 | 4 | Each group of Query heads shares KV |