1. What Is Hybrid Search?
Hybrid Search is an information retrieval method that combines multiple retrieval strategies. In modern AI applications, it typically refers to:
- Lexical Search: Retrieval based on how well the query terms match the terms in the documents; the typical algorithm is BM25.
- Vector Search: Converting the query and documents into vectors, then finding semantically relevant content by vector distance or similarity.
- Result Fusion: Merging, deduplicating, and ranking the results retrieved by different retrievers to form the final candidate set.
The two retrieval approaches solve different problems:
| Dimension | Lexical Search | Vector Search |
|---|---|---|
| Matching basis | Terms and their statistical features | Semantic relationships in vector space |
| Typical algorithms | BM25, TF-IDF | Dense Embedding, ANN |
| Exact terminology | Strong | Not necessarily reliable |
| Synonymous expressions | Relatively weak | Relatively strong |
| Semantic understanding | Limited | Relatively strong |
| Proper nouns | Usually good | Depends on embedding capability |
| Numbers, IDs, code | Usually good | Prone to semantic drift |
| Compute resources | Mainly consumes CPU and indexing resources | Requires vector storage and vector computation resources |
For example, consider the user query:
How do you solve the multithreading performance problem caused by the GIL in Python?
Lexical search might strongly match: - Python - GIL - multithreading - performance
Vector search might recall: - How CPython's global interpreter lock works - Performance differences between multithreading and multiprocessing - Parallel execution of CPU-bound tasks - The impact of the GIL on thread scheduling
The two retrieval results are complementary.
2. Why Is Hybrid Search Needed?
Using lexical search or vector search alone both have clear capability boundaries.
2.1 Limitations of Lexical Search
Lexical search algorithms such as BM25 primarily compute relevance based on term matching and statistical weights.
For example:
Query:
How can I improve model inference speed?
Document:
KV Cache stores the Key and Value of historical Tokens to reduce redundant computation during the Decode phase.
The two are highly related semantically, but they share no obvious keywords.
Lexical search may fail to fully recognize this semantic association.
Main problems:
- Synonym problem: Different words express the same or similar meaning.
- Semantic expression problem: The query and the document phrase things differently.
- Natural language problem: Users may use colloquial expressions, while documents use technical terminology.
- Context problem: Keyword matching struggles to directly understand complex semantic relationships.
2.2 Limitations of Vector Search
Vector search maps text into a vector space via embeddings and uses vector similarity to measure semantic relatedness.
But semantic similarity does not mean the retrieval target is exactly the same.
For example:
Query:
What is the installation command for PostgreSQL 16.2?
Vector search might recall:
- PostgreSQL installation tutorial
- PostgreSQL database architecture
- PostgreSQL version upgrade notes
- New features in the PostgreSQL 16 series
These documents are semantically related, but the user may need content that contains the exact version number and command.
Typical problems with vector search:
- Insufficient exact matching: Version numbers, error codes, product IDs, and the like may be weakened.
- Insufficient term discrimination: Similar technical terms may map to nearby vectors.
- Semantic noise: Recalled content is related but may not directly answer the question.
- Dependence on the vector model: The training data and expressive power of the embedding model affect retrieval results.
2.3 How the Two Retrieval Approaches Complement Each Other
Hybrid search is not simply concatenating two result lists; it uses different retrieval mechanisms to form complementary recall.
| Query type | Contribution of lexical search | Contribution of vector search |
|---|---|---|
| Technical terminology | Exact term matching | Expands related concepts |
| Natural language questions | Extracts key entities | Identifies overall semantics |
| Error code queries | Precisely hits the error code | Recalls related failure descriptions |
| Technical solution comparison | Hits specific technology names | Recalls conceptually related content |
| Synonymous expressions | May miss recalls | Supplements semantically related content |
| Version and parameter queries | Exactly matches numbers and parameters | Supplements contextual explanations |
Core idea:
Lexical search ensures precision, vector search expands semantic recall, and result fusion improves overall retrieval coverage.
3. Overall Architecture of Hybrid Search
A typical Hybrid Search system consists of the following modules:
- Query preprocessing: clean the query text and extract necessary information.
- Query dispatch: submit the same query to different retrievers.
- Keyword retrieval: recall candidate documents through an inverted index and BM25.
- Vector retrieval: retrieve semantically relevant documents through embeddings and ANN.
- Result fusion: merge multiple retrieval results.
- Deduplication and filtering: eliminate duplicate results and perform permission and metadata filtering.
- Reranking: use a reranker to finely rank candidate documents.
- Context construction: select the final document chunks and pass them to the LLM.
3.1 Overall Flowchart
flowchart TD
A["User Query"] --> B["Query Preprocessing"]
B --> C["Query Dispatch"]
C --> D["Keyword Retrieval (BM25)"]
C --> E["Vector Retrieval (Dense Search)"]
D --> F["Keyword Candidate Set"]
E --> G["Vector Candidate Set"]
F --> H["Result Fusion"]
G --> H
H --> I["Deduplication and Metadata Filtering"]
I --> J["Reranker Reranking"]
J --> K["Top-K Documents"]
K --> L["Build Context"]
L --> M["LLM Generates Answer"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The most important design here is: the two retrievers execute independently, and the fusion module processes the results in a unified way.
They can use different indexes, different recall counts, and even run in different storage systems.
3.2 Offline Index Construction Flow
Hybrid retrieval does not only include online queries; it also requires two sets of retrieval indexes to be built in advance.
flowchart TD
A["Raw Knowledge Documents"] --> B["Document Parsing"]
B --> C["Text Cleaning and Chunking"]
C --> D["Generate Chunk ID and Metadata"]
D --> E["Build Inverted Index"]
D --> F["Embedding Vectorization"]
E --> G["Keyword Index Store"]
F --> H["Vector Index Store"]
G --> I["Hybrid Retrieval System"]
H --> I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note the following:
- The inverted index stores information such as terms, document positions, and term frequencies.
- The vector index stores embedding vectors and their associated identifiers.
- The two sets of indexes must be linked through stable Chunk IDs or document IDs.
- When documents are updated or deleted, both sets of indexes must be maintained in sync.
- The data versions of the two index sets should remain consistent to avoid recalling content that is no longer valid.
IV. The Core of Hybrid Retrieval: Result Fusion
The raw scores from keyword retrieval and vector retrieval are usually not on the same scale.
For example:
| Document | BM25 Score | Vector Similarity |
|---|---|---|
| A | 12.5 | 0.81 |
| B | 8.7 | 0.92 |
| C | 15.2 | 0.68 |
| D | 5.3 | 0.89 |
A BM25 score can be greater than 10, whereas vector cosine similarity typically falls between -1 and 1.
Adding them directly:
would let the BM25 score hold a clear numerical advantage.
But this does not mean BM25 is necessarily more important — it simply means the two scoring systems have different numerical ranges.
Therefore, hybrid retrieval requires a sound fusion algorithm.
Common methods:
- Weighted Score Fusion
- Reciprocal Rank Fusion (RRF)
- Learning to Rank
V. Weighted Score Fusion
The basic idea of weighted score fusion is:
First convert the scores of different retrievers onto a combinable scale, then fuse them according to their weights.
5.1 Basic Formula
Where:
- S_k(d): the normalized score of keyword retrieval.
- S_v(d): the normalized score of vector retrieval.
- \alpha: the weight of keyword retrieval.
- 1-\alpha: the weight of vector retrieval.
- Score(d): the final fused score.
For example:
means keyword retrieval has a weight of 40% and vector retrieval has a weight of 60%.
Note: the weights represent the relative contribution within the fusion strategy; they do not represent the true proportion of retrieval quality.
5.2 Score Normalization
Common normalization methods include Min-Max and Z-Score.
Min-Max normalization:
This maps the scores in the current candidate set to the range 0 to 1.
Z-Score standardization:
Where:
- \mu: the mean of the scores.
- \sigma: the standard deviation of the scores.
Z-Score reflects the position of a score relative to the current distribution, but the result may be negative.
5.3 Weighted Fusion Example
Suppose normalization yields the following results:
| Document | Keyword Score | Vector Score |
|---|---|---|
| A | 0.90 | 0.70 |
| B | 0.60 | 0.95 |
| C | 1.00 | 0.40 |
Setting:
then:
Final ranking:
| Rank | Document | Fusion Score |
|---|---|---|
| 1 | B | 0.81 |
| 2 | A | 0.78 |
| 3 | C | 0.64 |
As can be seen, although B has a lower keyword score, its vector retrieval result is better, so it receives a higher overall ranking.
5.4 Limitations of Weighted Fusion
Weighted score fusion has several problems:
- The score distributions of different retrievers may vary with the query.
- Min-Max normalization is susceptible to extreme scores.
- Changes in the candidate set may cause normalization results to change.
- Weights need to be tuned on the actual dataset.
- Different models and distance metrics may have different scoring semantics.
Therefore, weighted fusion is suitable for retrieval systems whose scoring mechanisms are clear, calibrated, and stably comparable.
If the raw scores of different retrievers are difficult to calibrate, RRF can be considered.
6. RRF Reciprocal Rank Fusion
RRF (Reciprocal Rank Fusion) is a fusion algorithm based on rankings rather than raw scores.
It does not directly compare BM25 scores with vector similarity; instead, it calculates a fusion score based on a document's ranking in different retrieval results.
This is a very common fusion strategy in Hybrid Search.
6.1 Core Formula
Where:
- d: candidate document.
- n: number of retrievers.
- rank_i(d): the rank of the document in the ith retriever.
- k: smoothing constant.
Common implementations use k=60, but this is not a fixed requirement; it should be tuned according to the behavior of the retrieval system.
6.2 Why Use Reciprocal Rank?
The closer to the top the rank, the greater the contribution.
For example:
This means a document ranked 1st receives a higher contribution than a document ranked 10th.
At the same time, the contribution of lower-ranked results gradually decreases, preventing low-ranked results from having an excessive impact on the final score.
6.3 RRF Calculation Example
Suppose two retrievers return:
BM25 Retrieval Results
| Rank | Document |
|---|---|
| 1 | A |
| 2 | B |
| 3 | C |
| 4 | D |
Vector Retrieval Results
| Rank | Document |
|---|---|
| 1 | C |
| 2 | A |
| 3 | E |
| 4 | B |
Set:
Calculation results:
| Document | BM25 Rank | Vector Rank | RRF Score |
|---|---|---|---|
| A | 1 | 2 | 1/61+1/62 |
| B | 2 | 4 | 1/62+1/64 |
| C | 3 | 1 | 1/63+1/61 |
| D | 4 | Not retrieved | 1/64 |
| E | Not retrieved | 3 | 1/63 |
Final ranking:
- A
- C
- B
- E
- D
Both A and C rank near the top in both retrievers, so they receive higher fusion scores.
6.4 Advantages of RRF
- It does not require directly comparing the raw scores of different retrievers.
- It is insensitive to differences in score scales.
- It is simple to implement and has low computational cost.
- It can fuse multiple independent retrievers.
- It adapts well to different retrieval models.
6.5 Limitations of RRF
- It does not use the confidence information contained in raw scores.
- It considers only ranks, not the actual score gaps between ranks.
- It cannot automatically identify how reliable a given retriever is for a specific query.
- The parameter k affects how quickly the contribution of a rank decays.
- Enough candidate results must be retained; otherwise, potentially high-quality documents may be truncated before fusion.
For example:
One document ranks 1st in BM25 and another ranks 2nd, but their raw scores may be very close, or they may differ enormously.
RRF looks only at ranks and does not distinguish between these two cases.
7. Candidate Sets and Top-K Strategies in Hybrid Retrieval
The final performance of hybrid retrieval depends not only on the fusion algorithm but also on the number of results retrieved by each retriever.
Assume:
- BM25 retrieves 50 results.
- Vector retrieval retrieves 50 results.
- The two sets are merged and deduplicated.
- The final fused result retains 20 results.
The candidate set contains at most 100 results; the actual number depends on how much the two result sets overlap.
7.1 Why can't we truncate too early?
Suppose a document:
- ranks 80th in BM25.
- ranks 2nd in vector retrieval.
If BM25 retrieves only the top 10, that document can still enter the candidate set through vector retrieval.
But if it falls outside the retrieval window in both retrievers, it cannot enter the fusion stage.
Therefore, the number of results to retrieve must be determined jointly based on data scale, query complexity, latency budget, and downstream reranking capability.
7.2 Designing the Candidate Set Size
A common design approach:
| Stage | Example Count | Goal |
|---|---|---|
| BM25 retrieval | 50 | Ensure keyword coverage |
| Vector retrieval | 50 | Ensure semantic coverage |
| Merge and deduplicate | At most 100 | Form a unified candidate set |
| RRF fusion | At most 100 | Unified ranking |
| Reranker | 20–50 | Fine-grained ranking |
| Final context | 5–10 | Control context length |
These numbers are only examples, not universal optimal configurations.
7.3 Recall and Precision
Hybrid retrieval needs to pay attention to two metrics at the same time:
Recall@K:
It indicates how many relevant documents are covered among the top K retrieval results.
Precision@K (Precision):
It indicates how many of the top K results are actually relevant documents.
Typically:
- Increasing the number of candidates helps improve recall coverage, but may also introduce more noise.
- Improving ranking quality helps improve the top-ranked results, but does not necessarily improve first-stage recall.
The recall capability of the entire hybrid retrieval system cannot be judged solely by the performance of the final Top-5.
8. Reranker: The Fine-Ranking Mechanism After Hybrid Retrieval
Hybrid retrieval mainly addresses the recall of candidate documents.
However, results obtained through BM25, vector retrieval, and RRF may still contain ranking errors.
Therefore, a Reranker (re-ranking model) is usually introduced.
8.1 How the Reranker Works
Embedding retrieval generally uses a Bi-Encoder structure:
- The query is encoded separately.
- The document is encoded separately.
- Retrieval is performed through vector similarity.
This approach is suitable for large-scale retrieval, but the interaction between query and document is relatively limited.
A common implementation of the Reranker uses a Cross-Encoder:
- The Query and Document are input into the model together.
- The model uses an attention mechanism to establish fine-grained associations between the two.
- It outputs a relevance score.
Illustration:
flowchart TD
A["Query"] --> C["Cross-Encoder"]
B["Candidate Document"] --> C
C --> D["Joint Semantic Encoding"]
D --> E["Compute Relevance Score"]
E --> F["Re-rank"]
F --> G["Final Top-K"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
8.2 Why Is the Reranker Usually Placed After Fusion?
A Reranker is usually more computationally expensive than vector similarity calculation.
If Cross-Encoder scoring is performed directly on the entire knowledge base, the computational cost will be very high.
Therefore, BM25 and vector retrieval are usually first used for large-scale candidate recall, then fused through algorithms such as RRF, and finally fine-ranking is performed on a smaller candidate set.
It should be clear:
A Reranker can only re-rank candidate documents that have already been recalled; it cannot retrieve documents that were not recalled at all in the first stage.
Therefore, recall quality and re-ranking quality need to be optimized separately.
9. Python Implementation of Hybrid Retrieval
The following simplified Python code demonstrates the RRF fusion mechanism.
Assume that two ordered result lists have been obtained through BM25 and vector retrieval, respectively.
9.1 RRF Fusion Implementation
from collections import defaultdict
def reciprocal_rank_fusion(
result_lists,
k=60,
top_n=10
):
"""
result_lists:
Ordered result lists from multiple retrievers.
Each element is a document ID.
k:
RRF smoothing parameter.
top_n:
Number of results to return.
"""
scores = defaultdict(float)
for results in result_lists:
for rank, doc_id in enumerate(results, start=1):
scores[doc_id] += 1 / (k + rank)
ranked_results = sorted(
scores.items(),
key=lambda item: item[1],
reverse=True
)
return ranked_results[:top_n]
bm25_results = [
"doc_A",
"doc_B",
"doc_C",
"doc_D"
]
vector_results = [
"doc_C",
"doc_A",
"doc_E",
"doc_B"
]
results = reciprocal_rank_fusion(
[bm25_results, vector_results],
k=60,
top_n=5
)
for doc_id, score in results:
print(doc_id, round(score, 6))
The output results are arranged in descending order of fusion score.
9.2 Supporting Multiple Retrievers
RRF is not limited to BM25 and vector retrieval.
For example, it can simultaneously fuse:
- BM25.
- Dense Embedding retrieval.
- Sparse Embedding retrieval.
- Retrieval results from multiple different Embedding models.