I. Why Does Transformer Need Positional Encoding?
The core computation of Transformer relies on Self-Attention. Unlike RNN, which processes a sequence step by step over time, Transformer can process the entire sequence in parallel.
However, this parallel computation brings a problem: standard Self-Attention itself does not know the order of Tokens in the sequence.
For example:
I like artificial intelligence
Artificial intelligence likes me
The two sentences contain the same Tokens, differing only in order, yet their meanings can be completely different.
If the model cannot perceive the position of Tokens, it is difficult for it to accurately understand:
- The sequential relationship between words.
- The structural relationship between subject, predicate, and object.
- The distance between the current Token and the preceding and following Tokens.
- Local structure and long-distance dependencies in the sequence.
Therefore, Transformer needs a mechanism to introduce positional information into Token representations or attention computation.
1.1 Why Does Self-Attention Lack Position Awareness?
The formula for standard Self-Attention is:
Where:
- Q: Query, the query vector.
- K: Key, the key vector.
- V: Value, the value vector.
- d_k: The dimension of the Key vector.
Suppose the input sequence is:
Through linear transformation, we obtain:
If the order of the input Tokens is swapped, Q, K, and V will also be rearranged in the same way, and the attention computation results will be rearranged accordingly.
This means standard Self-Attention has permutation equivariance.
It can handle the relationships between Tokens, but it does not inherently distinguish whether a Token is the first, second, or last.
1.2 What Problem Does Positional Encoding Solve?
Positional encoding provides an additional positional representation for each Token.
Typically, Token Embedding can be combined with positional encoding:
Where:
- e_i: The word vector of the i-th Token.
- p_i: The position vector of the i-th position.
- h_i: The representation after adding positional information.
In this way, the model can utilize both the semantic information and the positional information of the Tokens.
Note: this is only one typical injection method for positional encoding. RoPE does not simply add a position vector to the Token Embedding; instead, it applies a rotation transformation to Query and Key during the attention computation stage.
II. Main Types of Positional Encoding
The development of positional encoding can be roughly divided into three categories:
- Absolute Positional Encoding.
- Relative Positional Encoding.
- Rotary Position Embedding (RoPE).
Their core difference lies in: how the model represents positions, and how positional relationships participate in attention computation.
2.1 Absolute Positional Encoding
Absolute positional encoding directly assigns a vector to each position in the sequence.
For example:
Token: I like artificial intelligence
Position: 0 1 2
Each position has a corresponding position vector:
Before being input into the model, the position vector is added to the Token Embedding:
Where:
- E: Token Embedding matrix.
- P: Positional encoding matrix.
- H: The input representation after fusing positional and semantic information.
Absolute positional encoding has two main implementations.
Method 1: Fixed Sinusoidal Positional Encoding
The original Transformer uses sine and cosine functions to construct position vectors.
For position pos and dimension index i:
Where:
- pos: The position of the Token in the sequence.
- i: Vector dimension index.
- d_{model}: Model hidden layer dimension.
- 10000: The frequency base used by the classic Transformer.
Different dimensions use different frequencies:
- Lower dimensions correspond to higher frequencies, so values change quickly as the position changes.
- Higher dimensions correspond to lower frequencies, so values change slowly as the position changes.
In this way, a position can be represented by multiple sets of sine and cosine values at different frequencies.
Why use multiple frequencies?
If all dimensions used the same frequency, the positional patterns the model could capture would be relatively limited.
A multi-frequency design allows the position vector to simultaneously encode:
- Fine-grained positional changes.
- Larger-range positional changes.
- Positional relationships at different scales.
The sine and cosine functions also have certain mathematical properties that allow relationships between different positions to be expressed through linear transformations.
Method 2: Learnable Positional Encoding
Learnable positional encoding no longer uses a fixed mathematical formula; instead, it directly maintains a trainable vector for each position:
Where:
- L: Preset maximum sequence length.
- d_{model}: Hidden layer dimension.
During training, the position vectors are updated together with the model's other parameters through backpropagation.
Advantages:
- Flexible representational capacity.
- It can learn position representations suited to the training data.
- Simple to implement.
Limitations:
- A maximum position length usually needs to be preset.
- It lacks inherent extrapolation ability beyond the range of positions seen during training.
- Position embeddings increase the model's parameter count.
2.2 Relative Positional Encoding
Absolute positional encoding focuses on which position in the sequence a Token occupies.
Relative positional encoding focuses more on:
How far apart two Tokens are, and the relative positional relationship between them.
For example:
Token: A B C D E
Position: 0 1 2 3 4
For Token C:
- A is 2 positions to the left of C.
- B is 1 position to the left of C.
- D is 1 position to the right of C.
- E is 2 positions to the right of C.
The relative distance between two Tokens can be defined as:
Where:
- i: The position of the Query.
- j: The position of the Key.
- r_{ij}: The positional offset of the Key relative to the Query.
A typical implementation adds a relative position bias to the Attention Score:
where b_{j-i} represents the position bias corresponding to the relative distance.
After Softmax:
In this way, when computing attention between Tokens, the model can take relative positional relationships into account.
T5 Relative Position Bias
T5 uses a relative position bias mechanism.
It does not set a separate parameter for every possible relative distance; instead, it divides distances into several buckets.
For example:
- Nearby positions use finer-grained buckets.
- Distant positions use coarser-grained buckets.
- Positions in different directions can be handled separately.
This controls the number of parameters while preserving the ability to model relationships at different distances.
Relative position bias is usually added to the Attention Logits rather than directly modifying the Token Embedding.
2.3 Comparison of Absolute and Relative Positional Encoding
| Comparison dimension | Absolute positional encoding | Relative positional encoding |
|---|---|---|
| Core focus | Absolute position of the Token | Distance between Tokens |
| Typical form | Position vector | Relative position bias or relational transformation |
| Injection point | Usually at the input Embedding stage | Usually at the attention computation stage |
| Position representation | Which Token it is | How many positions apart |
| Long-sequence extension | Depends on the specific design | Depends on the relative position modeling approach |
| Typical approach | Sinusoidal, Learned Embedding | T5 Relative Position Bias |
Note that relative positional encoding does not necessarily mean the model has no need for absolute position information at all. Different approaches differ in how they represent and inject positional information.
III. What Is RoPE?
RoPE is short for Rotary Position Embedding.
It was proposed by Su et al. in 2021, and its core idea is:
By applying position-dependent rotational transformations to the Query and Key vectors, the Attention Score naturally incorporates relative position information.
RoPE has an important difference from traditional position encoding:
Traditional absolute position encoding usually adds the position vector to the Token Embedding.
RoPE, on the other hand, applies a rotation matrix to the Query and Key during the Attention computation stage.
It does not simply add a position vector to the input representation.
3.1 The Core Idea of RoPE
We can start by understanding RoPE on a two-dimensional plane.
Suppose there is a two-dimensional vector:
If this vector is rotated by an angle \theta, we can use a two-dimensional rotation matrix:
The rotated vector is:
Expanding this:
RoPE applies this rotation idea to high-dimensional vectors.
For a Query and Key located at position m, a position-dependent rotation angle is used:
Where:
- m: the position index of the Token.
- \omega: the rotation frequency.
- \theta_m: the rotation angle corresponding to the current position.
Different dimensions use different frequencies, so each group of dimensions produces a different rotation speed.
3.2 Why Can RoPE Express Relative Position?
This is the most important mathematical property of RoPE.
Suppose:
- The Query is located at position m.
- The Key is located at position n.
After the RoPE transformation:
Their dot product is:
Using the property of matrix transposition:
Rotation matrices satisfy the orthogonality property:
And rotation matrices satisfy:
Therefore:
We finally obtain:
This formula is crucial.
It shows that:
The dot product of the Query and Key after the RoPE transformation depends only on the relative position difference between the two, rather than merely on their respective absolute positions.
For example:
- The Query is at position 10, and the Key is at position 15.
- The Query is at position 100, and the Key is at position 105.
The relative distance in both cases is 5.
For the same original vectors, their relative rotational relationships along the corresponding dimensions are consistent.
This is the mathematical basis for RoPE's ability to introduce relative position information into attention computation.
It should be emphasized that this conclusion applies to the dot-product structure after rotation. The model's complete attention output is also affected by factors such as the input content, other Tokens, the Mask, and network parameters.
IV. The High-Dimensional Rotation Mechanism of RoPE
In real models, the Query and Key are not two-dimensional vectors; they are vectors of tens, hundreds, or even higher dimensions.
RoPE splits high-dimensional vectors into multiple two-dimensional subspaces and rotates each of them separately.
4.1 Splitting High-Dimensional Vectors into Two-Dimensional Vectors
Suppose the dimension of a certain Attention Head is 8:
Using adjacent-dimension pairing, it can be split into:
Each group of two-dimensional vectors is rotated using a different frequency.
For the i-th group:
Therefore:
Each pair of dimensions can be understood as an independent two-dimensional rotation space.
4.2 Why do different dimensions use different frequencies?
RoPE uses different frequencies so that different dimensions have different sensitivities to position changes.
The frequency definition in classic RoPE is:
Where:
- i: index of the two-dimensional dimension pair.
- d: dimensions involved in rotation.
- \theta: frequency base; the classic configuration is usually 10000.
For position m, the rotation angle of the i-th group of dimensions is:
The role of different frequencies:
- Higher frequency: the rotation angle changes faster when position changes.
- Lower frequency: the rotation angle changes slower when position changes.
With multiple frequencies combined, the model can perceive positional relationships at different scales.
Here, two concepts need to be distinguished:
- \theta: frequency base, which is a configuration parameter of RoPE.
- \theta_{m,i}: the actual rotation angle used by a certain group of dimensions at a certain position.
They are not the same quantity.
4.3 Matrix form of RoPE
Combining all two-dimensional rotation matrices gives a high-dimensional block-diagonal matrix:
Each submatrix is a two-dimensional rotation matrix.
Finally:
From an implementation perspective, it is usually not necessary to actually create this large block-diagonal matrix.
Actual implementations use precomputed Sin and Cos tensors and perform rotation through element-wise operations, reducing memory and computation overhead.
V. Actual Location of RoPE in Transformer
RoPE is usually integrated into the computation process of Multi-Head Attention.
Taking standard Multi-Head Attention as an example:
Input hidden states:
They are respectively passed through linear projections:
Then Q, K, and V are split into multiple heads.
RoPE only acts on Q and K:
After that, attention is computed:
5.1 Why only rotate Q and K?
Attention Score is determined by the dot product of Query and Key:
It is precisely this similarity computation process that positional relationships need to affect.
RoPE rotates Q and K so that attention scores contain relative positional relationships.
Value, on the other hand, is responsible for providing the content information that is weighted and aggregated by attention:
Therefore, classic RoPE usually does not rotate V.
This does not mean that V contains no positional information. V comes from the hidden states of Transformer, and the hidden states have already been processed by previous network layers and may contain contextual and positional information.
5.2 Complete flow of RoPE in Attention
flowchart TB
A["Input hidden state X"] --> B["Linear projection Q, K, V"]
B --> C["Split into multiple Attention Heads"]
C --> D["Obtain Position ID"]
D --> E["Compute Sin and Cos for RoPE"]
C --> F["Original Q and K"]
E --> G["Rotary position transformation"]
F --> G
G --> H["Obtain rotated Q and K"]
H --> I["Compute QK transpose"]
C --> J["Keep original V"]
I --> K["Scale and add Attention Mask"]
J --> L["Multiply Softmax weights by V"]
K --> M["Softmax"]
M --> L
L --> N["Attention output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#E5F0E9,stroke:#83A995,color:#315D49
5.3 Differences Between RoPE and Causal Mask
These two mechanisms often appear together in Decoder-only Transformers, but they serve completely different purposes.
| Comparison Dimension | RoPE | Causal Mask |
|---|---|---|
| Core function | Inject positional information | Limit visible tokens |
| Target of processing | Query and Key | Attention Score |
| Main problem addressed | How the model perceives relative positions | How the model avoids seeing future tokens |
| Mathematical mechanism | Rotation transformation | Set disallowed positions to negative infinity |
| Whether it changes attention scope | Does not directly change it | Does change it |
Typical form of Causal Mask:
After Softmax, the attention weights for future positions become 0.
RoPE is responsible for positional relationships, while Causal Mask is responsible for information access constraints; the two cannot replace each other.
6. Simplified PyTorch Implementation of RoPE
One of the best ways to understand RoPE is to directly observe its tensor operations.
Below is a simplified version that uses adjacent-dimension pairing.
6.1 Generating Frequencies and Position Angles
import torch
def precompute_rope(
seq_len: int,
dim: int,
theta: float = 10000.0,
device: str = "cpu",
):
"""
Precompute the cos and sin for RoPE.
dim must be even.
Return shape: [seq_len, dim // 2]
"""
assert dim % 2 == 0
inv_freq = 1.0 / (
theta ** (
torch.arange(0, dim, 2, device=device).float() / dim
)
)
positions = torch.arange(
seq_len,
device=device,
dtype=torch.float32
)
freqs = torch.outer(positions, inv_freq)
cos = torch.cos(freqs)
sin = torch.sin(freqs)
return cos, sin
Where:
inv_freq: The rotation frequency for each pair of 2D dimensions.positions: The position indices of the tokens.freqs: The rotation angle corresponding to each position.cosandsin: The trigonometric values required for rotation.
6.2 Implementing 2D Rotation
def rotate_half(x):
"""
Pair adjacent dimensions:
(x0, x1), (x2, x3), ...
"""
x1 = x[..., ::2]
x2 = x[..., 1::2]
rotated = torch.stack(
(-x2, x1),
dim=-1
)
return rotated.flatten(-2)
def apply_rope(x, cos, sin):
"""
x: [batch, heads, seq_len, head_dim]
cos: [seq_len, head_dim // 2]
sin: [seq_len, head_dim // 2]
"""
cos = cos.repeat_interleave(2, dim=-1)
sin = sin.repeat_interleave(2, dim=-1)
cos = cos.unsqueeze(0).unsqueeze(0)
sin = sin.unsqueeze(0).unsqueeze(0)
return x * cos + rotate_half(x) * sin