Back to series
· 10 min read

How Self-Attention Works: Understanding Query, Key, Value, and Attention Computation


I. What Is Self-Attention?

Self-Attention (the self-attention mechanism) is the core computational mechanism in Transformer for establishing information associations between different Tokens within a sequence.

Its basic idea is:

When processing a Token, dynamically aggregate information from other positions in the sequence according to the degree of association between the current Token and other Tokens.

Traditional sequence models usually pass information step by step through a fixed computational order, whereas Self-Attention can directly compute associations between different positions in the sequence.

For example:

Input:
Xiaoming put the book on the table because it is very stable.

Question:
What does "it" refer to?

The model needs to combine context to determine the relationship between the pronoun and other words.

Self-Attention allows the representation at the current position to attend to other positions in the sequence, thereby fusing relevant contextual information.

Note that Attention weights represent how the model allocates information across different positions in the current computational layer; they are not equivalent to real coreference relationships in the sense of human linguistics, nor can they simply be regarded as a complete explanation of the model.

II. The Core Computational Idea of Self-Attention

The input to Self-Attention is a set of Token vectors, and the output is still a set of vectors, but the output at each position incorporates information from other positions in the sequence.

Suppose the input sequence contains three Tokens:

Token 1: I
Token 2: like
Token 3: AI

The corresponding input vectors are:

X= \begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}

where:

X\in\mathbb{R}^{n\times d_{model}}

Self-Attention computes three different vectors for each Token:

  • Query (Q): what information the current Token wants to obtain.
  • Key (K): what information the current Token can provide.
  • Value (V): the content representation that the current Token actually provides.

Then, relevance scores are computed between the Query and all Keys, and the Values are weighted and aggregated according to these scores.

flowchart TB
    A["Input Token vector X"] --> B["Linear projection"]
    B --> C["Generate Query, Key, Value"]
    C --> D["Compute similarity between Query and Key"]
    D --> E["Scale attention scores"]
    E --> F["Softmax converts to weights"]
    F --> G["Weighted sum over Value"]
    G --> H["Output contextual representation"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

The entire process can be summarized as:

Output=Softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V

This is the basic formula of Scaled Dot-Product Attention.

It is broken down step by step below.

III. The Generation Mechanism of Query, Key, and Value

Q, K, and V are the three most important concepts in Self-Attention.

They are not three fixed attributes extracted directly from the original text; rather, they are obtained from the input hidden representation through three different learnable linear transformations.

3.1 Input Vectors

Let the hidden representation of the input sequence be:

X\in\mathbb{R}^{n\times d_{model}}

where:

  • n: sequence length.
  • d_{model}: model hidden dimension.

For example:

n=4,\quad d_{model}=512

then:

X\in\mathbb{R}^{4\times512}

This means the sequence contains 4 Tokens, and each Token corresponds to a 512-dimensional vector.

These vectors usually already contain Token Embedding, positional information, and features extracted by previous network layers.

3.2 Generating Q, K, and V via Linear Transformations

The model uses three different weight matrices:

W_Q\in\mathbb{R}^{d_{model}\times d_k}
W_K\in\mathbb{R}^{d_{model}\times d_k}
W_V\in\mathbb{R}^{d_{model}\times d_v}

and computes respectively:

Q=XW_Q
K=XW_K
V=XW_V

Therefore:

Q\in\mathbb{R}^{n\times d_k}
K\in\mathbb{R}^{n\times d_k}
V\in\mathbb{R}^{n\times d_v}

Real models usually also include learnable biases, but to simplify the formulas, bias terms are omitted here.

3.3 Why Are Three Different Projection Matrices Needed?

Because they serve different computational roles.

Vector Role Computational Meaning
Query Initiates an information query What information the current Token needs
Key Provides the basis for matching How the current Token matches the Query
Value Provides the actual content What information the current Token can contribute

This can be understood through a retrieval system:

  • Query: query conditions.
  • Key: index features.
  • Value: the actual retrieved content.

The model computes the degree of match between Query and Key, then aggregates Value according to the degree of match.

It should be emphasized that Q, K, and V are continuous vector representations learned by the model, and do not correspond to fixed semantic fields that humans can directly understand.

3.4 Q, K, and V Are Not Three Independent Inputs

A common misunderstanding is to think that Q, K, and V come from three different sets of raw data.

In Self-Attention:

Q=XW_Q,\quad K=XW_K,\quad V=XW_V

All three come from the same input X, just using different projection matrices.

This is also the meaning of Self in Self-Attention: the Query, Key, and Value used in the attention computation all come from representations of the same sequence.

In Cross-Attention, however, the Query and the Key, Value usually come from different sequences or different network modules.

4. Computing Attention Score: Establishing Associations Between Tokens

After generating Q, K, and V, the next step is to compute the degree of match between Query and Key.

4.1 Dot Product Calculation

For the Query of the i-th Token:

q_i

and the Key of the j-th Token:

k_j

compute their dot product:

s_{ij}=q_i k_j^T

The dot product result is used to measure the degree of match between the two vectors in the current representation space.

Generally speaking, the larger the dot product, the higher the attention score before Softmax.

However, the dot product is not a normalized probability, and it is not necessarily within a fixed range.

4.2 Matrix Form

Compute for the entire sequence at once:

S=QK^T

Assume:

Q\in\mathbb{R}^{n\times d_k}
K^T\in\mathbb{R}^{d_k\times n}

Then:

S\in\mathbb{R}^{n\times n}

where:

S_{ij}=q_i k_j^T

represents the matching score between the Query at position i and the Key at position j.

Therefore, the Attention Score matrix has the following structure:

  • Rows: positions of the Query.
  • Columns: positions of the Key.
  • Elements: matching scores between two positions.

For example, with a sequence length of 4:

S= \begin{bmatrix} s_{11}&s_{12}&s_{13}&s_{14}\\ s_{21}&s_{22}&s_{23}&s_{24}\\ s_{31}&s_{32}&s_{33}&s_{34}\\ s_{41}&s_{42}&s_{43}&s_{44} \end{bmatrix}

Each row corresponds to a Token's attention scores over the entire sequence.

4.3 Why Do We Need to Compute Relationships Between All Positions?

Because the semantic representation of a Token often depends on other Tokens.

For example:

Sentence A:
Apple released a new phone.

Sentence B:
I bought some apples.

The same word may have different meanings in different contexts.

Self-Attention allows the model to dynamically aggregate information from other positions based on the current context, so that the same Token forms different hidden representations in different contexts.

This is also an important reason why Transformer can handle complex contextual relationships.

5. Scaled Dot-Product: Why Divide by the Square Root?

Standard Self-Attention does not directly apply Softmax to QK^T; instead, it first divides by \sqrt{d_k}.

Formula:

S=\frac{QK^T}{\sqrt{d_k}}

This operation is called Scaled Dot-Product.

5.1 Why Does the Dot Product Grow with Dimension?

Assume each component of Query and Key is an independent random variable with mean 0 and variance 1.

The dot product is:

q\cdot k=\sum_{i=1}^{d_k}q_i k_i

Under the above independence assumption:

E[q\cdot k]=0

Its variance is approximately:

Var(q\cdot k)=d_k

Therefore, the standard deviation of the dot product is approximately:

\sqrt{d_k}

As the vector dimension increases, the fluctuation range of the dot product values also increases.

5.2 Why do excessively large scores affect Softmax?

Softmax formula:

Softmax(x_i)= \frac{e^{x_i}}{\sum_j e^{x_j}}

When the input score differences are too large, Softmax tends to produce a highly concentrated probability distribution.

For example:

Softmax([1,2,3])

and:

Softmax([10,20,30])

Compared with the first, the probability distribution produced by the second input is sharper.

When probabilities are highly concentrated, the gradients at non-maximum positions may become very small, which in turn affects gradient propagation during training.

5.3 The role of scaling

By dividing by:

\sqrt{d_k}

the numerical scale of the dot product can be controlled to some extent, making the input distribution to Softmax more stable.

For example:

d_k=64

Then the scaling factor is:

\sqrt{64}=8

Finally:

AttentionScore=\frac{QK^T}{8}

Note that this derivation relies on simplified statistical assumptions. In actual training, the distributions of Q and K change with model parameters, normalization methods, and training state.

6. Softmax: Converting Attention Scores into Weights

After scaling is complete, the row of scores corresponding to each Query needs to be converted into a weight distribution.

6.1 Softmax Computation

Suppose a Query corresponds to three Keys, and its scaled scores are:

S=[1,2,3]

Softmax:

\alpha_i= \frac{e^{s_i}}{\sum_j e^{s_j}}

The computed results are approximately:

\alpha=[0.0900,0.2447,0.6652]

This means the weighting ratios of the current Query over the three Values are approximately:

  • First position: 9.00%.
  • Second position: 24.47%.
  • Third position: 66.52%.

The attention weights satisfy:

\sum_j\alpha_j=1

and:

0\leq\alpha_j\leq1

Therefore, a row of attention weights can be understood as a probability distribution.

However, it is not the model’s probability that some fact is correct; it is the normalized weight in the current attention computation.

6.2 Applying Softmax Row by Row

For:

S\in\mathbb{R}^{n\times n}

Softmax is computed along the last dimension.

That is:

  • Each row corresponds to one Query.
  • Each row is normalized independently.
  • The weights in each row sum to 1.

This gives:

A=Softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)

where:

A\in\mathbb{R}^{n\times n}

A is the attention weight matrix.

7. Weighted Aggregation of Value: Generating Contextual Representations

After the attention weights are computed, the model still needs to actually retrieve information from other Tokens.

This step is performed through Value.

7.1 Weighted Sum

For the i-th Query, its output is:

o_i=\sum_{j=1}^{n}\alpha_{ij}v_j

where:

  • \alpha_{ij}: the attention weight of the i-th Query for the j-th Value.
  • v_j: the Value vector corresponding to the j-th Token.
  • o_i: the output vector at the i-th position.

Therefore, the output is not simply selecting a single most relevant Token, but performing a weighted sum over multiple Values according to the weights.

7.2 Matrix Form

Combining the computations for all positions:

O=AV

where:

A\in\mathbb{R}^{n\times n}
V\in\mathbb{R}^{n\times d_v}

Therefore:

O\in\mathbb{R}^{n\times d_v}

The final O is the representation after contextual fusion.

7.3 Complete Computation Formula

Combining the above steps:

\boxed{ Attention(Q,K,V)= Softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V }

The entire process can be broken down as:

flowchart TB
    A["Input X"] --> B["Q = XWQ"]
    A --> C["K = XWK"]
    A --> D["V = XWV"]
    B --> E["QK transpose dot product"]
    C --> E
    E --> F["Divide by √dk"]
    F --> G["Softmax"]
    G --> H["Attention weight matrix A"]
    H --> I["A × V"]
    D --> I
    I --> J["Context output O"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

8. Understanding Attention Through a Numerical Example

To understand the computation process more intuitively, suppose there are three Tokens:

I like AI

Suppose that after linear projection they yield the following simplified vectors:

Q= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}
K= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}
V= \begin{bmatrix} 1&0\\ 0&2\\ 2&1 \end{bmatrix}

For convenience of manual calculation, let:

d_k=2

8.1 Compute the transpose of QK

QK^T= \begin{bmatrix} 1&0&1\\ 0&1&1\\ 1&1&2 \end{bmatrix}

8.2 Scaling the Scores

S=\frac{QK^T}{\sqrt{2}}

This gives:

S\approx \begin{bmatrix} 0.707&0&0.707\\ 0&0.707&0.707\\ 0.707&0.707&1.414 \end{bmatrix}

8.3 Softmax Normalization

Compute Softmax row by row to obtain the approximate result:

A\approx \begin{bmatrix} 0.401&0.198&0.401\\ 0.198&0.401&0.401\\ 0.248&0.248&0.504 \end{bmatrix}

The weights in each row sum to approximately 1.

8.4 Multiply by V

O=AV

This gives:

O\approx \begin{bmatrix} 1.203&0.797\\ 1.000&1.203\\ 1.256&0.752 \end{bmatrix}

This means:

  • The output of the first Token incorporates the Value from all three positions.
  • The output of the second Token incorporates the Value from all three positions.
  • The output of the third Token also incorporates the Value from all three positions.

Although the three positions use the same input sequence, because each has a different Query, the attention weights they obtain are also different.

Therefore, each Token ultimately obtains a different contextual representation.

The Q, K, and V in this example are artificially constructed values and do not represent the trained parameters of a real language model.

9. Why Can Self-Attention Model Context?

The core value of Self-Attention is not merely computing similarity between Tokens; rather, it achieves contextual information fusion through dynamic weights.

9.1 Static Word Vectors and Contextual Representations

Traditional static word vectors usually provide a fixed vector representation for the same word.

For example:

Apple phone
I like eating apples

If only a single fixed word vector is used, it is difficult to directly reflect the difference in meaning of the same word in different contexts.

Self-Attention, by contrast, can dynamically adjust the information aggregation result according to the current input context.

The same Token can produce different Q, K, V, and attention weights in different sentences.

9.2 Layer-by-Layer Modeling with Multi-Layer Attention

A Transformer usually contains multiple Attention Layers.

Each layer recomputes Q, K, and V based on the hidden representation from the previous layer.

Therefore:

  • Shallow layers can learn local associations and basic patterns.
  • Middle layers can gradually integrate more complex contextual information.
  • Deep layers can form more abstract feature representations.

This is only a common way of understanding it; it does not mean that every layer has a fixed and strict semantic role.

10. Differences Between Self-Attention and Cross-Attention

Self-Attention and Cross-Attention both use Q, K, and V to compute attention, but their Q, K, and V come from different sources.

10.1 Self-Attention

In Self-Attention:

Q=XW_Q
K=XW_K
V=XW_V

Q, K, and V come from the same set of input representations.

For example, an Encoder performs Self-Attention on the input sequence.

10.2 Cross-Attention

In Cross-Attention, the Query comes from one sequence, while the Key and Value come from another sequence.

For example:

Q=H_{dec}W_Q
K=H_{enc}W_K
V=H_{enc}W_V

Where:

  • H_{dec}: Decoder hidden representation.
  • H_{enc}: Encoder output.

10.3 Comparison

Comparison aspect Self-Attention Cross-Attention
Query source Current sequence One sequence
Key source Current sequence Another sequence
Value source Current sequence Another sequence
Main role Information interaction within a sequence Information interaction between different sequences
Typical scenarios Encoder, Decoder Encoder-Decoder Decoder

Note that Self-Attention does not mean that all positions can necessarily access one another. The specific accessible range is also constrained by Padding Mask, Causal Mask, or other attention masks.

11. Mask: Controlling the Visible Range of Attention

In an actual Transformer, Attention does not necessarily allow all Tokens to access one another.

Mask is used to control which positions can participate in attention computation.

Usually, before Softmax, the scores of inaccessible positions are set to negative infinity.

S'=\frac{QK^T}{\sqrt{d_k}}+M

Where:

  • Accessible positions: the Mask value is usually 0.
  • Inaccessible positions: the Mask value is usually negative infinity.

Then:

A=Softmax(S')

The attention weights of inaccessible positions then become 0.

11.1 Padding Mask

Padding Mask is used to mask invalid Tokens padded during batching.

For example, input sequences have different lengths; to form a Batch of uniform shape, special Tokens usually need to be padded.

These padding positions should not be attended to by other positions as valid content.

11.2 Causal Mask

Causal Mask is used to implement autoregressive attention.

The current position can only access the current position and previous positions, and cannot access future Tokens.

For example, with a sequence length of 4:

M= \begin{bmatrix} 0&-\infty&-\infty&-\infty\\ 0&0&-\infty&-\infty\\ 0&0&0&-\infty\\ 0&0&0&0 \end{bmatrix}

The visibility relationships are as follows:

flowchart TB
    A["Position 1"] --> B["Can only access position 1"]
    C["Position 2"] --> D["Can access positions 1, 2"]
    E["Position 3"] --> F["Can access positions 1, 2, 3"]
    G["Position 4"] --> H["Can access positions 1, 2, 3, 4"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49

11.3 Uses of Different Masks

Mask Type Primary Purpose Typical Application
Padding Mask Mask padding positions Batch sequence processing
Causal Mask Mask future positions Autoregressive language models
Attention Mask A general term for attention visibility constraints Various Transformer models

In practice, the concrete data format of an Attention Mask may differ across frameworks. For example, some implementations use a boolean mask, while others use an additive mask, so its semantics cannot be judged from the variable name alone.

12. Multi-Head Self-Attention: Why Do We Need Multiple Heads?

A single Self-Attention computes attention through only one set of Q, K, V projection spaces.

To enhance the model's representational capacity, Transformer introduces Multi-Head Attention.

12.1 Limitations of Single-Head Attention

A single Attention Head uses one set of projection matrices:

Q=XW_Q
K=XW_K
V=XW_V

Then computes:

head=Attention(Q,K,V)

This structure performs attention computation in only one projection subspace.

However, multiple types of relational patterns may exist between Tokens, such as:

  • Syntactic relations.
  • Semantic relations.
  • Coreference relations.
  • Long-distance dependencies.
  • Positional associations.

Using multiple attention heads allows the model to learn different relational patterns in different representation subspaces.

12.2 Multi-Head Computation

The i-th Head:

head_i= Attention( XW_i^Q, XW_i^K, XW_i^V )

After computing multiple Heads separately, they are concatenated:

MultiHead(Q,K,V)= Concat(head_1,\ldots,head_h)W^O

Where:

  • h: the number of attention heads.
  • W_i^Q: the Query projection matrix of the i-th head.
  • W_i^K: the Key projection matrix of the i-th head.
  • W_i^V: the Value projection matrix of the i-th head.
  • W^O: the output projection matrix.

12.3 Multi-Head Structure Diagram

flowchart TB
    A["Input X"] --> B["Head 1"]
    A --> C["Head 2"]
    A --> D["Head 3"]
    A --> E["Head h"]

    B --> F["Concat"]
    C --> F
    D --> F
    E --> F

    F --> G["Output Linear Projection"]
    G --> H["Multi-Head Attention Output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

12.4 Multi-Head Dimension Calculation

Assume:

d_{model}=768
h=12

Then the dimension of each Head is typically:

d_k=d_v=\frac{768}{12}=64

The output of each Head:

head_i\in\mathbb{R}^{n\times64}

After concatenation:

Concat(head_1,\ldots,head_{12}) \in\mathbb{R}^{n\times768}

After the output projection, the dimension is usually still maintained as:

O\in\mathbb{R}^{n\times768}