1. What Is Rerank?
Rerank is a technique that re-orders the results of an initial retrieval pass.
In search systems and RAG architectures, a batch of candidates is typically recalled from a large document collection first using BM25, vector search, or hybrid search, and then a reranker model evaluates the relevance between each candidate document and the user query to reorder them.
It primarily solves one problem:
Initial retrieval can find relevant documents, but it cannot always judge accurately which documents most deserve to be returned first.
For example, a user query:
Why can KV Cache in Transformer reduce inference latency?
Initial retrieval might produce the following results:
| Initial Rank | Candidate Document | Characteristics of Initial Retrieval |
|---|---|---|
| 1 | Transformer's attention mechanism | Strong keyword match |
| 2 | How KV Cache works | Directly explains the query topic |
| 3 | Prefill and Decode in LLM inference | Covers the inference stage |
| 4 | Transformer model architecture | Related at the overall concept level |
| 5 | GPU memory optimization techniques | Related background knowledge |
Initial retrieval may rank documents containing large numbers of Transformer keywords near the top.
But judging from the user's question, what should really be returned first is the document that directly explains how KV Cache avoids redundant computation.
Rerank then further analyzes the semantic relationship between the query and each candidate document and recomputes relevance scores.
The final result might be:
| Reranked Order | Candidate Document |
|---|---|
| 1 | How KV Cache works |
| 2 | Prefill and Decode in LLM inference |
| 3 | Transformer's attention mechanism |
| 4 | GPU memory optimization techniques |
| 5 | Transformer model architecture |
Rerank is not responsible for finding documents across the entire knowledge base; it makes a finer-grained judgment over an existing candidate set.
2. Why Do We Need Rerank?
Modern retrieval systems typically adopt a two-stage or multi-stage retrieval architecture:
- Stage one: retrieval — quickly find candidate documents from massive amounts of data.
- Stage two: reranking — make a finer-grained relevance judgment on the candidate documents.
2.1 Limitations of Initial Retrieval
Whether it is BM25 or vector search, each has its own capability boundaries.
Problems with BM25:
BM25 computes relevance mainly based on term matching, term frequency, inverse document frequency, and document length.
It is good at exact matching, but poor at understanding complex semantic relationships.
For example:
Query:
How can we avoid recomputing historical tokens when an LLM generates an answer?
Document:
KV Cache caches the Key and Value corresponding to historical tokens and reuses them during subsequent autoregressive generation, thereby avoiding repeated attention projection computation for historical positions.
The two are highly related semantically, but their keyword expressions do not fully match.
Problems with Vector Search:
Vector search maps queries and documents into a vector space via embeddings and ranks them using cosine similarity, inner product, or distance.
It can capture semantic associations, but it may also rank documents that are conceptually similar and topically close—yet do not actually contain the answer—near the top.
For example:
- Query: How does KV Cache reduce the amount of computation in the Decode stage?
- Document A: The computational mechanism of KV Cache.
- Document B: GPU memory management techniques.
- Document C: The overall architecture of Transformer.
B and C may have high semantic relevance to the query, but they do not necessarily contain the answer the user actually needs.
2.2 The Value of Rerank
The core advantage of rerank is that it models the relationship between the Query and the Document more fully.
| Capability | Initial Retrieval | Rerank |
|---|---|---|
| Large-scale data retrieval | Good at it | Not suited to processing an entire knowledge base directly |
| Exact term matching | BM25 excels | Can combine with semantic judgment |
| Semantic relevance | Embedding retrieval | Enables finer-grained judgment |
| Query–document interaction | Usually weak | Usually strong |
| Ranking accuracy | Limited by the recall algorithm | Can be improved further |
| Computational cost | Usually lower | Usually higher |
The goal of rerank is not to replace initial retrieval, but to improve the ranking quality within the candidate set.
3. Where Rerank Fits in RAG
A typical RAG system can be divided into:
- Document parsing and chunking.
- Embedding vectorization.
- Index construction.
- User query.
- Initial retrieval.
- Rerank reordering.
- Context construction.
- LLM answer generation.
3.1 The Complete Workflow
flowchart TD
A["User Query"] --> B["Query Preprocessing"]
B --> C["Initial Retrieval"]
C --> D["BM25 Keyword Retrieval"]
C --> E["Vector Retrieval"]
D --> F["Candidate Document Set"]
E --> F
F --> G["Reranker Model"]
G --> H["Compute Query-Document Relevance"]
H --> I["Rerank"]
I --> J["Filter Top-K Documents"]
J --> K["Build Context"]
K --> L["LLM Generates Answer"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
3.2 Why place Rerank after recall?
Suppose the knowledge base contains:
- 1 million document chunks.
- Each query needs to recall 100 candidate chunks.
- In the end, only 5–10 highly relevant chunks are needed.
If a computationally expensive Reranker is used to score 1 million documents one by one, the computational overhead would be very large.
A more reasonable architecture is:
| Phase | Processed Objects | Primary Objective |
|---|---|---|
| Initial Retrieval | 1 million chunks | Quickly recall candidates |
| Candidate Set | 50–200 chunks | Narrow the search space |
| Rerank | 50–200 chunks | Precisely compute relevance |
| Final Filtering | 5–10 chunks | Build high-quality context |
The above numbers are only an architectural example; the actual configuration depends on the knowledge base size, model performance, query latency, and context budget.
Core principle:
First use efficient retrieval to narrow the search space, then use a high-precision model to optimize ranking.
IV. Core Architecture of Reranker: Bi-Encoder and Cross-Encoder
To understand Rerank, you first need to understand two types of text matching models:
- Bi-Encoder: dual encoder.
- Cross-Encoder: cross encoder.
Their most important difference is whether Query and Document jointly interact inside the model.
4.1 Bi-Encoder: Dual Encoder
Bi-Encoder encodes the query and document separately into vectors.
Suppose:
- Query is q.
- Document is d.
- Encoding model is f(\cdot).
Respectively, we obtain:
Then compute the similarity between the two vectors.
For example, cosine similarity:
Overall structure:
flowchart TD
A["Query"] --> B["Query Encoder"]
C["Document"] --> D["Document Encoder"]
B --> E["Query Embedding"]
D --> F["Document Embedding"]
E --> G["Similarity Computation"]
F --> G
G --> H["Similarity Score"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Characteristics of the Bi-Encoder:
- Query and Document are encoded independently.
- Document vectors can be computed and stored in advance.
- At query time, only the Query Embedding needs to be computed.
- A vector index can be used to perform ANN retrieval.
- Well suited for large-scale candidate recall.
But it also has one important limitation:
Query and Document do not interact deeply and directly during the encoding stage.
The model mainly judges whether the two are related based on their respective vector representations and the final similarity.
This may cause fine-grained semantic relationships to be lost.
4.2 Cross-Encoder: Cross Encoder
The Cross-Encoder concatenates the Query and the Document into a single joint input, which is then passed to the same model for encoding.
For example:
[CLS] Query [SEP] Document [SEP]
Within the Transformer's multi-layer attention computation, the model allows Query Tokens to interact directly with Document Tokens.
Finally, a relevance score is output through a classification head or a regression head.
Formal representation:
Where:
- q: the user query.
- d: a candidate document.
- g_\theta: the model that jointly models the Query and the Document.
- Score(q,d): the relevance score.
Overall structure:
flowchart TD
A["Query"] --> C["Concatenate Query and Document"]
B["Document"] --> C
C --> D["Joint Token Sequence"]
D --> E["Transformer Encoder"]
E --> F["Cross-Text Interaction Between Query and Document"]
F --> G["Pooling / Feature Extraction"]
G --> H["Relevance Prediction Head"]
H --> I["Relevance Score"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The Cross-Encoder can directly model:
- Whether the entities in the query correspond to the entities in the document.
- Whether the conditions in the query are satisfied by the document.
- Whether the document truly answers the question.
- Whether negation in the document changes the meaning.
- Whether there is a fine-grained semantic association between the query and the document.
For example:
Query:
Can Python multithreading fully utilize a multi-core CPU to execute CPU-bound tasks?
Document A:
The GIL in CPython restricts multiple threads from executing Python bytecode at the same time, so for CPU-bound tasks you can usually consider using multiprocessing.
Document B:
Python's threading module provides interfaces for creating and managing threads.
Both documents contain content related to Python and multithreading, but A answers the core condition in the query more directly.
Cross-Encoder can more fully identify this difference through joint attention computation over the Query and Document.
4.3 Comparison of Bi-Encoder and Cross-Encoder
| Dimension | Bi-Encoder | Cross-Encoder |
|---|---|---|
| Input method | Query and Document are input separately | Query and Document are input jointly |
| Encoding method | Independent encoding | Joint encoding |
| Cross-text interaction | No direct interaction during encoding | Direct interaction inside the Transformer |
| Document vector caching | Supported | Usually does not support reuse of independent document vectors |
| Large-scale ANN retrieval | Suitable | Not suitable for direct execution |
| Cost per match | Lower | Usually higher |
| Ranking accuracy | Suitable for coarse-grained recall | Suitable for fine-grained relevance judgment |
| Typical use | Retrieval | Reranking |
Note the following:
Bi-Encoder and Cross-Encoder are common architectures, but this does not mean that all Rerankers must use a Cross-Encoder.
Real systems also include LLM-based relevance judgment models, list-level ranking models, and other learning-to-rank architectures.
V. Why can Cross-Encoder improve ranking accuracy?
This is the most important part for understanding Rerank.
5.1 Information interaction limitations of independent encoding
Bi-Encoder computes separately:
Finally it uses:
In this structure, the representations of the Query and Document are mainly formed independently.
The model cannot, in each Transformer layer, directly and dynamically adjust the Document's token representations based on the specific content of the current Query.
5.2 Cross-Attention-style joint interaction
Cross-Encoder places both texts into the same Transformer sequence.
Let the hidden state of the joint input be:
In the self-attention layer:
Here Q, K, and V are all computed from the joint input.
Since Query and Document tokens are both within the scope of attention computation, Query tokens can attend to Document tokens, and Document tokens can also attend to Query tokens.
This means the model can dynamically learn:
- Which query term corresponds to which concept in the document.
- Whether the document contains the key conditions required by the query.
- Whether the document has semantic conflicts.
- Whether certain words are truly important in the current query context.
Strictly speaking, Cross-Encoder uses Self-Attention over the joint sequence, rather than necessarily using a separate Cross-Attention layer.
5.3 Computational complexity and engineering trade-offs
Assume:
- The Query length is L_q.
- The Document length is L_d.
- The Transformer hidden dimension is d.
The two parts of Bi-Encoder can be computed independently, and the encoding result on the document side can be pre-cached.
For N documents, at query time only one Query Embedding needs to be computed, and then similarity is computed with the existing document vectors.
Cross-Encoder performs joint computation for each Query-Document Pair.
The joint sequence length is approximately:
In the standard full-attention mechanism, the main complexity of attention computation is related to:
is related to it.
When scoring N candidate documents individually, the joint encoding computation needs to be performed N times.
Therefore, the cost of Cross-Encoder increases significantly with the number of candidates.
This is why production environments usually do not use Cross-Encoder to perform a full scan of the entire knowledge base.
VI. How does a Reranker calculate relevance scores?
The core output of a Reranker is the relevance score between the Query and Document.
Common approaches include binary classification, regression, and learning to rank.
6.1 Binary classification model
Classify Query-Document Pair into two categories:
- Relevant: relevant.
- Irrelevant: irrelevant.
The model outputs:
Relevance probabilities can be obtained via Softmax or Sigmoid.
For example:
Query: What is KV Cache?
Document A: KV Cache is used to cache the Key and Value of historical Tokens.
Relevant: 0.97
Document B: CNN is a common type of convolutional neural network.
Relevant: 0.02
In practice, a model may output class probabilities or relevance logits, depending on the training task and model implementation.
6.2 Regression Models
Model relevance as a continuous score:
For example:
| Document | Relevance Score |
|---|---|
| Document A | 8.92 |
| Document B | 6.35 |
| Document C | 2.18 |
A higher score generally means the model considers the document more relevant.
But note:
Reranker scores are usually not calibrated true probabilities.
Different models may use different output scales; even if two models both output values between 0 and 1, that does not mean their scores carry the same meaning.
6.3 Learning to Rank
Besides judging whether a single document is relevant, you can also train a model directly to learn the relative order among documents.
For example:
For the same query:
Where:
- d^+: a relevant document.
- d^-: an irrelevant document.
The training objective is for positive documents to score higher than negative documents.
This type of training is particularly well suited to search ranking tasks.
7. Reranker Training Mechanisms
A reranker's performance depends not only on the model architecture but also on the training data and objective function.
7.1 Training Data
A typical training sample consists of:
- Query: the query text.
- Positive Document: a relevant document.
- Negative Document: an irrelevant document.
- Relevance Label: a relevance label or grade.
Example:
| Query | Document | Label |
|---|---|---|
| What does KV Cache do? | Caches historical Key and Value to avoid recomputation | 1 |
| What does KV Cache do? | How computation works in CNN convolution layers | 0 |
| What impact does the Python GIL have? | The GIL's restrictions on thread execution | 1 |
| What impact does the Python GIL have? | Structure of an HTML document | 0 |
The quality of the training data directly affects the model's ability to judge relevance.
7.2 Constructing Negative Samples
Negative samples are not better simply because they are more random.
If a negative sample is completely unrelated to the query, the model can easily tell positives from negatives.
For example:
Query:
How do you implement a distributed lock with Redis?
Positive sample:
Use the SET NX EX command to implement a distributed lock with an expiration time.
Random negative sample:
CNN is a convolutional neural network.
This negative sample is too easy to distinguish.
More valuable are Hard Negatives:
Redis's RDB and AOF persistence mechanisms.
This document shares the Redis topic with the query but does not answer the distributed lock question.
Hard negatives help the model learn finer-grained relevance judgments.
Common sources of negative samples:
- Random negatives: drawn from unrelated documents.
- BM25 negatives: documents that are keyword-similar but semantically mismatched.
- Vector negatives: documents that are vector-similar but not truly relevant.
- Model hard negatives: documents that the current model incorrectly ranks near the top.
- Human-annotated negatives: documents confirmed by humans to be irrelevant or partially relevant.
7.3 Pairwise Ranking Loss
Pairwise loss is used to learn that positive samples should score higher than negative samples.
One common form is:
Where:
- s(q,d^+): the relevance score of the positive sample.
- s(q,d^-): the relevance score of the negative sample.
- \sigma: the Sigmoid function.
When the positive sample score is significantly higher than the negative sample score, the loss decreases.
If the negative sample score is higher than the positive sample score, the loss increases, and the model needs to adjust its parameters.
7.4 Listwise Ranking
The Listwise method learns a set of candidate documents corresponding to the same query as a whole.
Unlike Pairwise, which only focuses on the relative order of two documents, Listwise considers the ranking quality of the entire candidate list.
For example:
Query
├── Document A: Relevance Level 3
├── Document B: Relevance Level 2
├── Document C: Relevance Level 0
└── Document D: Relevance Level 1
The training objective is to enable the model to learn a more reasonable overall ranking.
Common methods include learning-to-rank methods based on ListNet, ListMLE, or NDCG optimization ideas.
Different Reranker models may adopt different training objectives, which does not mean that all models use the same Loss.
8. Complete Execution Flow of Rerank
A typical Reranker inference process can be divided into the following stages.
8.1 Input Construction
Combine the query and candidate documents into the model input.
For example:
Query:
What is KV Cache?
Document:
KV Cache is a caching mechanism used during large language model inference,
used to store the Key and Value corresponding to historical Tokens.
After Tokenizer processing, inputs required by the model such as Token ID and Attention Mask are generated.
8.2 Model Forward Computation
The model encodes the joint sequence through a Transformer Encoder.
This mainly includes:
- Token Embedding.
- Position Embedding or positional encoding mechanism.
- Multiple layers of Self-Attention.
- FFN feed-forward network.
- Pooling or special Token representation extraction.
- Relevance prediction head.
8.3 Relevance Scoring
The model outputs a relevance score based on the final hidden state.
For example:
Document A: 0.96
Document B: 0.72
Document C: 0.15
Document D: 0.83
8.4 Sorting and Filtering
Sort in descending order by score and select the final Top-K.
Complete process:
flowchart TD
A["User Query"] --> B["Retrieve candidate documents"]
B --> C["Construct Query-Document Pair"]
C --> D["Tokenizer Encoding"]
D --> E["Generate Token ID and Attention Mask"]
E --> F["Reranker Forward Computation"]
F --> G["Extract Joint Semantic Representation"]
G --> H["Relevance Prediction Head"]
H --> I["Output Relevance Score"]
I --> J["Sort by Score in Descending Order"]
J --> K["Top-K Filtering"]
K --> L["Return Reranking Results"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
9. Engineering Implementation of Reranker
9.1 Python Implementation: Using Cross-Encoder
You can use the CrossEncoder interface in Sentence Transformers to call an off-the-shelf reranking model.
from sentence_transformers import CrossEncoder
# Load a model that supports Cross-Encoder tasks
model = CrossEncoder(
"cross-encoder/ms-marco-MiniLM-L-6-v2"
)
query = "什么是 KV Cache?"
documents = [
"KV Cache 用于缓存历史 Token 的 Key 和 Value。",
"CNN 是一种常见的卷积神经网络。",
"大语言模型使用自回归方式逐个生成 Token。",
"KV Cache 可以减少 Decode 阶段的重复计算。"
]
# Build Query-Document pairs
pairs = [
(query, document)
for document in documents
]
# Perform relevance scoring
scores = model.predict(pairs)
# Sort by score descending
results = sorted(
zip(documents, scores),
key=lambda item: float(item[1]),
reverse=True
)
for document, score in results:
print(f"Score: {float(score):.4f}")
print(f"Document: {document}")
print()
Note:
- The example model is intended for English MS MARCO ranking tasks and is not suitable as a Chinese production model directly.
- For practical applications, choose a Reranker that matches the language, domain, and data distribution.
- CrossEncoder's
predictoutput may be a logit or a model-defined relevance score and should not be interpreted as a probability by default.
9.2 Batch Inference
Production environments usually do not call the model one item at a time; instead, they use batch inference.
from sentence_transformers import CrossEncoder
model = CrossEncoder(
"cross-encoder/ms-marco-MiniLM-L-6-v2"
)
query = "如何理解 Transformer 的 Self-Attention?"
documents = [
"Self-Attention 用于计算序列中不同 Token 之间的关联。",
"Transformer 使用多头注意力机制提取上下文信息。",
"CNN 通过卷积核提取局部特征。",
"LayerNorm 用于稳定神经网络训练过程。"
]
pairs = [(query, doc) for doc in documents]
scores = model.predict(
pairs,
batch_size=16
)
ranked = sorted(
zip(documents, scores),
key=lambda x: float(x[1]),
reverse=True
)
for rank, (doc, score) in enumerate(ranked, start=1):
print(rank, round(float(score), 4), doc)