Back to series
· 9 min read

Multi-Head Attention: From Parallel Attention Computation to Multi-Subspace Information Fusion


1. What Is Multi-Head Attention?

Multi-Head Attention (MHA) is the attention mechanism in Transformers that strengthens the ability to exchange information.

Its core idea is:

Instead of using only one set of Query, Key, and Value to compute attention, it uses multiple sets of projection matrices to run Attention in parallel across multiple representation subspaces, and finally fuses the outputs of all attention heads.

In single-head Self-Attention, the model builds relationships between Tokens through one set of Q, K, V.

Multi-Head Attention, by contrast, uses multiple attention heads, and each Head has its own independent Q, K, V projection parameters.

For example, a model with 12 attention heads can run 12 different attention computations at the same time.

flowchart TB
    A["Input hidden representation X"] --> B["Multiple independent linear projections"]

    B --> C["Head 1"]
    B --> D["Head 2"]
    B --> E["Head 3"]
    B --> F["Head 12"]

    C --> G["Concat Concatenation"]
    D --> G
    E --> G
    F --> G

    G --> H["Output linear projection"]
    H --> I["Multi-Head Attention Output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

Multi-head attention is not simply repeating one Attention computation many times; rather, through different learnable projection matrices, each Head learns how to match and aggregate information in a different representation subspace.

2. Why Do We Need Multi-Head Attention?

To understand Multi-Head Attention, you first need to understand the limitations of single-head attention.

2.1 How Single-Head Attention Is Computed

Standard Scaled Dot-Product Attention:

Attention(Q,K,V)= Softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V

Where:

  • Q: Query, the query vector.
  • K: Key, the key vector.
  • V: Value, the value vector.
  • d_k: the feature dimension of Key.

Assume the input hidden dimension is:

d_{model}=768

Single-head attention can project Q, K, V directly into a 768-dimensional space:

Q,K,V\in\mathbb{R}^{n\times768}

Then it performs one Attention computation.

This means the entire attention mechanism uses only one set of projection spaces for matching and information aggregation.

2.2 The Limits of a Single Representation Space

Token relationships in natural language are diverse.

For example:

Artificial intelligence is changing the way software is developed.

The model may need to model all of the following at once:

  • The semantic relationship between artificial intelligence and changing.
  • The compositional relationship between software and development.
  • The grammatical relationship between subject and predicate.
  • The contextual associations between different positions.

These relationships cannot necessarily be fully expressed through a single projection space.

If there is only one attention head, all information matching has to be done through the same set of Q, K, V projections.

Multi-head attention allows the model to use multiple sets of projection parameters to learn different patterns of association.

Note that this does not mean each Head will automatically form a clear linguistic function. In practice, attention heads may overlap in function, and may also learn complex compositional features.

2.3 The Core Advantages of Multi-Head Attention

Advantage Description
Multi-subspace modeling Learn associations in different projection spaces
Parallel computation Multiple Heads can perform attention operations in parallel
Expressiveness Increases the representational flexibility of the attention mechanism
Information fusion Combine the results of different Heads into a unified representation
End-to-end learning The projection parameters of each Head are optimized automatically through training

3. The Overall Structure of Multi-Head Attention

Multi-Head Attention can be broken down into five main steps:

  1. Input the hidden representation.
  2. Generate multiple sets of Q, K, V.
  3. Each Head computes Attention independently.
  4. Concatenate the outputs of all Heads.
  5. Fuse the information through the output projection matrix.
flowchart TB
    A["Input X"] --> B["Generate multiple sets of Q, K, V"]
    B --> C["Multiple Attention Heads compute in parallel"]
    C --> D["Concatenate outputs of each Head"]
    D --> E["Output linear projection"]
    E --> F["Obtain final output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49

Below, we expand from the perspective of mathematical computation.

4. Step 1: Generate multiple sets of Q, K, V

Assume the input hidden representation is:

X\in\mathbb{R}^{n\times d_{model}}

Where:

  • n: sequence length.
  • d_{model}: model hidden dimension.

Assume the number of attention heads is h.

Classic Multi-Head Attention usually evenly assigns the hidden dimension to each Head:

d_k=d_v=\frac{d_{model}}{h}

For example:

d_{model}=768
h=12

Then:

d_k=d_v=64

Each Head uses its own projection matrices:

W_i^Q\in\mathbb{R}^{d_{model}\times d_k}
W_i^K\in\mathbb{R}^{d_{model}\times d_k}
W_i^V\in\mathbb{R}^{d_{model}\times d_v}

The computation for the i-th Head is:

Q_i=XW_i^Q
K_i=XW_i^K
V_i=XW_i^V

Where:

Q_i,K_i\in\mathbb{R}^{n\times d_k}
V_i\in\mathbb{R}^{n\times d_v}

4.1 Why does each Head use its own projection matrices?

If all Heads share the same set of Q, K, V projection matrices, then the input representations they obtain will be the same.

Even if the subsequent computation uses the same structure, it is difficult for them to form truly independent attention representations.

By assigning independent parameters to each Head:

W_1^Q,W_2^Q,\ldots,W_h^Q

the model can learn different matching patterns in different projection spaces.

For example:

  • Head 1 may focus more on a certain type of local correlation.
  • Head 2 may focus more on long-distance information.
  • Head 3 may focus more on specific contextual features.

These are possibilities to aid understanding, not fixed assignments of Head functions by the model.

4.2 Matrix implementation of multiple sets of projections

In actual engineering practice, a separate linear layer is usually not actually invoked for each Head.

A more common approach is to combine the projection matrices of all Heads:

W^Q\in\mathbb{R}^{d_{model}\times(hd_k)}
W^K\in\mathbb{R}^{d_{model}\times(hd_k)}
W^V\in\mathbb{R}^{d_{model}\times(hd_v)}

Compute them all at once:

Q=XW^Q
K=XW^K
V=XW^V

Then reorganize the results into multiple Heads.

flowchart TB
    A["Input X"] --> B["Q Linear Projection"]
    A --> C["K Linear Projection"]
    A --> D["V Linear Projection"]

    B --> E["Q Dimension Reshape"]
    C --> F["K Dimension Reshape"]
    D --> G["V Dimension Reshape"]

    E --> H["Multiple Query Heads"]
    F --> I["Multiple Key Heads"]
    G --> J["Multiple Value Heads"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

This approach can fully exploit the parallel computing capability of matrix multiplication and reduce the scheduling overhead caused by many independent small-matrix operations.

5. Step 2: Each Head Independently Performs Attention

After generating multiple sets of Q, K, and V, each Head independently performs standard Scaled Dot-Product Attention.

The i-th Head:

head_i= Attention(Q_i,K_i,V_i)

Expanded:

head_i= Softmax\left( \frac{Q_iK_i^T}{\sqrt{d_k}} \right)V_i

Each Head has its own:

  • Query representation.
  • Key representation.
  • Value representation.
  • Attention score matrix.
  • Attention weight matrix.
  • Output representation.

5.1 Computation Dimensions of Each Head

Assume:

Q_i\in\mathbb{R}^{n\times64}
K_i\in\mathbb{R}^{n\times64}
V_i\in\mathbb{R}^{n\times64}

Then:

Q_iK_i^T\in\mathbb{R}^{n\times n}

After Softmax:

A_i\in\mathbb{R}^{n\times n}

Finally:

head_i=A_iV_i

This yields:

head_i\in\mathbb{R}^{n\times64}

The output sequence length of each Head remains unchanged, but the feature dimension is 64.

5.2 Parallel Computation of Multi-Head Attention

Assume there are 4 Heads:

flowchart TB
    A["Input X"] --> B["Head 1: Attention"]
    A --> C["Head 2: Attention"]
    A --> D["Head 3: Attention"]
    A --> E["Head 4: Attention"]

    B --> F["head 1"]
    C --> G["head 2"]
    D --> H["head 3"]
    E --> I["head 4"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

These Heads are mathematically independent of one another and can be executed in parallel.

On GPUs, practical implementations usually combine the Head dimension with the Batch dimension to organize computation, so as to use efficient batched matrix operations.

6. Step 3: Concatenate All Heads

After all Heads have finished computing, their outputs need to be combined.

Assume:

head_i\in\mathbb{R}^{n\times d_v}

There are h heads in total.

Concatenation:

H=Concat(head_1,head_2,\ldots,head_h)

This gives:

H\in\mathbb{R}^{n\times(hd_v)}

In the classic Transformer:

hd_v=d_{model}

Therefore:

H\in\mathbb{R}^{n\times d_{model}}

6.1 Concatenation Is Not Summation

Concat and Sum are two different operations.

Suppose the outputs of two heads are:

head_1=[1,2]
head_2=[3,4]

Concat:

Concat(head_1,head_2)=[1,2,3,4]

Sum:

head_1+head_2=[4,6]

Concat preserves the feature dimensions of different heads, whereas Sum directly adds the features at corresponding positions.

Multi-head attention uses Concat to preserve the output information of each head for the subsequent projection layer to fuse.

6.2 Dimensions After Concatenation

Assume:

n=128
h=12
d_v=64

Then:

head_i\in\mathbb{R}^{128\times64}

After concatenation:

H\in\mathbb{R}^{128\times768}

At this point, the hidden dimension is restored to 768.

VII. Step 4: Output Linear Projection

After concatenation, the result still needs to pass through an output projection matrix:

W^O\in\mathbb{R}^{(hd_v)\times d_{model}}

The final output is:

MultiHead(Q,K,V)= Concat(head_1,\ldots,head_h)W^O

where:

O=HW^O

In the classic configuration:

hd_v=d_{model}

Therefore:

W^O\in\mathbb{R}^{d_{model}\times d_{model}}

Finally:

O\in\mathbb{R}^{n\times d_{model}}

7.1 Why Is an Output Projection Still Needed?

Concat merely places the features of different heads together; it does not actively learn how to fuse these features.

The output projection matrix W^O is responsible for performing a learnable linear combination of the concatenated features.

It can:

  • Combine information from different heads.
  • Learn relationships between different feature dimensions.
  • Map the concatenation result to the model's hidden dimension.
  • Provide a unified representation space for subsequent network layers.

7.2 Output Projection Is Not Merely Dimension Restoration

Although in the classic configuration the dimension after concatenation is already equal to d_{model}, the output projection still plays an important role.

Even if the input and output dimensions are the same, matrix multiplication still recombines the features of different heads.

Therefore, the output projection is not only a shape adjustment but also part of multi-head information fusion.

VIII. Complete Mathematical Formula of Multi-Head Attention

Combine all the steps.

8.1 A Single Head

The i-th head:

head_i= Softmax\left( \frac{XW_i^Q(XW_i^K)^T}{\sqrt{d_k}} \right)XW_i^V

8.2 Multi-Head Concatenation

H=Concat(head_1,\ldots,head_h)

8.3 Output Projection

O=HW^O

Complete formula:

\boxed{ MultiHead(Q,K,V)= Concat(head_1,\ldots,head_h)W^O }

where:

\boxed{ head_i=Attention(Q_i,K_i,V_i) }

and:

\boxed{ Attention(Q_i,K_i,V_i)= Softmax\left( \frac{Q_iK_i^T}{\sqrt{d_k}} \right)V_i }

Complete computation structure:

flowchart TB
    A["Input hidden representation X"] --> B["Generate multiple sets of Q, K, V"]
    B --> C["Each Head independently computes Scaled Dot-Product Attention"]
    C --> D["Concat concatenation"]
    D --> E["Output projection Wᴼ"]
    E --> F["Final output O"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49

9. Tensor Dimensions: How Does Multi-Head Attention Organize Data?

To understand Multi-Head Attention, you must grasp how tensor dimensions change.

Assume:

Parameter Value Meaning
Batch Size 32 Batch size
Sequence Length 128 Sequence length
Hidden Dimension 768 Hidden dimension
Number of Heads 12 Number of attention heads
Head Dimension 64 Dimension per head

Satisfying:

768=12\times64

9.1 Input Tensor

X\in\mathbb{R}^{32\times128\times768}

9.2 Q, K, V Projection

Perform linear transformations separately:

Q,K,V\in\mathbb{R}^{32\times128\times768}

9.3 Split the Head Dimension

Split the last dimension into:

768=12\times64

Then adjust the dimension order:

Q,K,V\in \mathbb{R}^{32\times12\times128\times64}

Here, the meaning of each dimension is:

  • 32: Batch.
  • 12: Head.
  • 128: Sequence Length.
  • 64: Head Dimension.

9.4 Compute Attention Score

Each Head computes separately:

QK^T

This yields:

S\in \mathbb{R}^{32\times12\times128\times128}

The last two dimensions of the attention matrix correspond to the Query position and the Key position, respectively.

9.5 Softmax

A=Softmax(S)

This yields:

A\in \mathbb{R}^{32\times12\times128\times128}

9.6 Aggregate Value

O=AV

This yields:

O\in \mathbb{R}^{32\times12\times128\times64}

9.7 Merge Heads

Merge the Head dimension with the Head Dimension:

O\in\mathbb{R}^{32\times128\times768}

9.8 Output Projection

After passing through W^O:

O\in\mathbb{R}^{32\times128\times768}

Complete dimension changes:

Stage Tensor Shape
Input X [32, 128, 768]
Q/K/V Projection [32, 128, 768]
Head Split [32, 12, 128, 64]
Attention Score [32, 12, 128, 128]
Softmax Weights [32, 12, 128, 128]
Value Aggregation [32, 12, 128, 64]
Head Merging [32, 128, 768]
Output Projection [32, 128, 768]

10. Parameter Count of Multi-Head Attention

The parameter count of multi-head attention mainly comes from four sets of linear projections:

  • Query projection.
  • Key projection.
  • Value projection.
  • Output projection.

Assume:

d_{model}=d

and:

hd_k=hd_v=d

Then:

W^Q,W^K,W^V,W^O

The parameter count of each matrix is approximately:

d^2

Therefore, the total parameter count is approximately:

\boxed{ 4d^2 }

If each linear layer includes a bias, this adds approximately:

4d

10.1 Parameter Count Example

Assume:

d_{model}=768

Then the parameter count of the four projection matrices is:

4\times768^2
=2,359,296

If four sets of biases are included:

4\times768=3,072

Total parameter count:

2,362,368

This does not include the parameters of other Transformer components, such as FFN, LayerNorm, etc.

10.2 Does the Number of Heads Increase the Total Parameter Count?

When d_{model} is kept unchanged and the dimension of each head is adjusted proportionally, the projection parameter count of standard MHA basically does not change with the number of heads.

For example:

Hidden Dimension Number of Heads Head Dimension Total Projection Parameter Count (Excluding Bias)
768 8 96 2,359,296
768 12 64 2,359,296
768 16 48 2,359,296

As the number of heads increases, the dimension of each individual head decreases accordingly.

However, this does not mean the computational behavior is exactly the same. Different numbers of heads change the attention grouping, the partition of the projection space, and the model's representation.

11. Computational Complexity of Multi-Head Attention

The computation of Multi-Head Attention mainly includes:

  1. Linear projections of Q, K, and V.
  2. QK transposed dot product.
  3. Softmax.
  4. Multiplication of the attention weights by V.
  5. Output linear projection.

Assume:

  • The sequence length is n.
  • The hidden dimension is d.
  • The number of heads is h.
  • The dimension of each head is d/h.

11.1 Q, K, V Projections

The computational cost of the three linear projections is approximately:

3O(nd^2)

11.2 QK Transposed Dot Product

Computational cost per head:

O\left(n^2\frac{d}{h}\right)

Total across all heads:

h\times O\left(n^2\frac{d}{h}\right)

This gives:

O(n^2d)

11.3 Multiplication of Attention and V

The complexity is likewise approximately:

O(n^2d)

11.4 Output Projection

The complexity is approximately:

O(nd^2)

11.5 Total Complexity

Ignoring relatively small operations such as Softmax and Mask:

\boxed{ O(nd^2+n^2d) }

More specifically:

O(4nd^2+2n^2d)

The constant coefficients here depend on how FLOPs are counted.

Note that:

  • nd^2 mainly comes from linear projections.
  • n^2d mainly comes from attention score computation and Value aggregation.

When the sequence length is very large, the quadratic term becomes an important computational and memory bottleneck.

12. Why Is the Attention Matrix Prone to Becoming a GPU Memory Bottleneck?

Multi-head attention needs to compute an attention matrix for each batch and each head:

A\in\mathbb{R}^{B\times h\times n\times n}

The number of its elements is:

Bhn^2

Assume:

B=32,\quad h=12,\quad n=4096

Number of elements in the attention matrix:

32\times12\times4096^2

This is approximately:

6.44\times10^9

If each element uses FP16 or BF16 and occupies 2 bytes, simply storing one such matrix in full requires about 12 GiB of GPU memory.

Actual training may also involve Softmax intermediate results, data required for backpropagation, and other activations, so the GPU memory pressure is even greater.

12.1 FlashAttention

FlashAttention is an important technique optimized for attention computation.

Through tiled computation, online Softmax, and optimized GPU memory access, it reduces the materialization and storage of the full attention matrix.

Core objectives:

  • Reduce costly GPU memory reads and writes.
  • Lower the GPU memory usage of intermediate matrices.
  • Improve GPU computational efficiency.
  • Preserve the standard attention computation objective and obtain equivalent or approximate numerical results within the allowed numerical precision.

Note that FlashAttention does not change the basic Q, K, V structure of Multi-Head Attention; instead, it optimizes its computational implementation.

13. MHA, MQA, GQA: Three Architectures of Multi-Head Attention

Standard MHA uses an independent Key head and Value head for each Query head.

However, during large language model inference, the KV Cache occupies a large amount of GPU memory.

To reduce KV Cache costs, MQA and GQA have emerged in research and engineering practice.

13.1 MHA: Multi-Head Attention

Standard MHA:

  • Multiple Query heads.
  • Multiple Key heads.
  • Multiple Value heads.
  • The number of Query heads is the same as the number of KV heads.

Assume:

H_Q=H_K=H_V=16

Each Query head has a corresponding KV head.

13.2 MQA: Multi-Query Attention

MQA retains multiple Query heads, but all Query heads share one set of Keys and Values.

For example:

H_Q=16
H_{KV}=1

Its advantages are:

  • Reduce the storage size of the KV Cache.
  • Lower the bandwidth requirement for reading KV data during inference.
  • Improve efficiency in some autoregressive inference scenarios.

However, sharing KV also changes how the model expresses information and may introduce a trade-off between effectiveness and efficiency.

13.3 GQA: Grouped-Query Attention

GQA is a compromise architecture between MHA and MQA.

Multiple Query heads are grouped to share one set of Keys and Values.

For example:

H_Q=16
H_{KV}=4

Every 4 Query heads share one set of KV heads.

13.4 Comparison of the Three Architectures

Architecture Query Head KV Head KV Sharing Method
MHA 16 16 Each Query head uses KV independently
MQA 16 1 All Query heads share KV
GQA 16 4 Each group of Query heads shares KV