1. What Is Semantic Search?
Semantic Search is an information retrieval technology based on content meaning. It does not only focus on whether the query text and the document contain the same keywords; instead, it attempts to understand whether the meanings expressed by the two are close.
Traditional keyword search focuses primarily on term matching:
- Whether the target keyword appears in the query.
- The frequency with which keywords appear in the document.
- The importance of keywords within the document.
- Whether there is a literal match between query terms and document terms.
Semantic search, by contrast, focuses on:
- Whether two pieces of text express similar meanings.
- Whether different wording describes the same issue.
- Whether there is a semantic association between the query and the document.
- Whether the text expresses the same intent in context.
For example, the user enters:
How can the response time of database queries be reduced?
Traditional keyword search may prioritize documents containing "database," "query," and "response time."
Semantic search, by contrast, may find:
- Methods for optimizing slow SQL queries.
- Database index design and execution plan analysis.
- Database connection pooling and concurrency control.
- Query caching and read/write splitting.
Even if the document does not contain the complete query keywords, as long as the semantic expression is close enough, it may still be retrieved.
1.1 Core Components of Semantic Search
Semantic search usually consists of three core components:
| Component | Primary Responsibility |
|---|---|
| Embedding model | Encodes text into high-dimensional vectors |
| Vector similarity calculation | Measures how close a query vector is to a document vector |
| Vector retrieval system | Quickly finds the most similar candidate results from a large number of vectors |
Among these, Embedding is responsible for constructing semantic representations, vector similarity is responsible for calculating semantic closeness, and the vector retrieval system is responsible for improving the efficiency of large-scale retrieval.
1.2 Differences Between Semantic Search and Keyword Search
| Comparison Dimension | Keyword Search | Semantic Search |
|---|---|---|
| Core basis | Term matching | Vector semantic similarity |
| Wording | Relies more on keywords | Can use different wording |
| Synonym handling | Relies on mechanisms such as dictionaries and expansion | Can learn semantic relationships through Embedding |
| Computation method | Inverted index, term scoring | Vector distance or similarity calculation |
| Main advantage | Precise matching and relatively strong interpretability | Stronger semantic generalization ability |
| Main limitation | Tends to miss relevant content expressed in different wording | May retrieve semantically similar but actually irrelevant content |
It should be noted that semantic search is not a complete replacement for keyword search.
For example, when searching for exact identifiers such as order numbers, user IDs, error codes, and product models, keyword search is usually more direct and reliable.
In practice, systems often combine the two to form hybrid retrieval.
2. Embedding: Mapping Text to Vector Space
The foundation of semantic search is Embedding (vector embedding).
An Embedding model converts text into a fixed-dimensional numerical vector, enabling computers to compare the semantic representations of different texts through mathematical operations.
2.1 Text Vectorization
Suppose there are three pieces of text:
- A: How can database query performance be optimized?
- B: How can SQL query speed be improved?
- C: How do you make tomato and egg stir-fry?
After processing by the Embedding model, three vectors are obtained:
Here, d represents the vector dimension.
Different Embedding models may have different output dimensions, such as 384, 768, 1024, 1536, or other dimensions.
These numbers are not the number of semantic categories; rather, they are the vector space dimensions the model uses to represent text features.
A text vector can be understood as:
Each dimension is a numerical feature learned by the model. Usually, a particular dimension cannot simply be interpreted as "database," "performance," or another clear human concept.
2.2 How Does Embedding Express Semantic Relationships?
The Embedding model learns semantic relationships between texts from training data.
The training process may involve:
- Contrastive learning.
- Construction of positive and negative samples.
- Query-document matching training.
- Semantic similarity learning.
- Ranking and relevance optimization.
After model training is complete, semantically close texts usually form relatively close representations in the vector space.
For example:
Text A: How can database query performance be optimized? Text B: How can SQL query speed be improved? Text C: How do you make tomato and egg stir-fry?
Ideally:
This means that the semantic association between A and B is stronger than that between A and C.
However, it should be noted that Embedding does not guarantee that all semantically similar texts will have higher similarity, nor does it guarantee that similarity can accurately reflect factual relevance. The final result depends on the model's training objective, domain data, text expression, and task type.
2.3 Query Vectors and Document Vectors
Semantic retrieval typically involves two types of vectors:
Query Vector (Query Embedding)
Generated from the question or query text entered by the user.
Document Vector (Document Embedding)
Generated from the documents, paragraphs, or text chunks to be retrieved.
In a practical RAG system, documents are usually processed offline first and their document vectors are saved; when a query is received online, the query vector is then generated.
Note that some embedding models use different encoding instructions or encoding paths for Query and Document, such as a query prefix and a document prefix. Even if both come from the same model, the encoding method specified by the model should be followed.
2.4 Basic Process of Semantic Retrieval
flowchart TD
A["Prepare knowledge documents"] --> B["Document chunking"]
B --> C["Embedding model"]
C --> D["Generate document embeddings"]
D --> E["Save vectors and metadata"]
F["User enters query"] --> G["Generate query embedding"]
G --> H["Vector similarity calculation"]
E --> H
H --> I["Sort by similarity"]
I --> J["Return Top-K results"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
3. What Is Cosine Similarity?
Cosine similarity is a commonly used similarity metric in vector retrieval.
It measures how close two vectors are in direction by computing the cosine of the angle between them.
Core formula:
Where:
- A: the first vector.
- B: the second vector.
- A \cdot B: the dot product of the two vectors.
- \|A\|: the magnitude of vector A.
- \|B\|: the magnitude of vector B.
3.1 Understanding Cosine Similarity from the Vector Angle
For two nonzero vectors, cosine similarity focuses only on the angle between their directions.
Here, \theta represents the angle between the two vectors.
Angle close to 0°: directions are close, similarity is close to 1 Angle of 90°: directions are orthogonal, similarity is 0 Angle close to 180°: directions are opposite, similarity is close to -1
The theoretical range of cosine similarity is:
It can usually be understood as follows:
| Cosine Similarity | Geometric Meaning |
|---|---|
| Close to 1 | The two vectors have similar directions |
| Close to 0 | The two vectors are approximately orthogonal |
| Close to -1 | The two vectors point in opposite directions |
It should be emphasized that a cosine similarity of 0 does not mean the two texts are completely unrelated; it only means that, in the current vector space and under the current metric, the two vectors are approximately orthogonal.
Likewise, a cosine similarity close to 1 does not mean the texts are identical at the factual level.
3.2 Why Use the Vector Angle Instead of Directly Comparing Vector Lengths?
Suppose the two vectors are:
Vector B is twice vector A:
They have different lengths, but exactly the same direction.
Calculate cosine similarity:
Therefore:
This shows that cosine similarity is insensitive to overall scaling of the vectors.
As long as two non-zero vectors have the same direction, even if their magnitudes differ, the cosine similarity remains 1.
This is also one of the important reasons it is suitable for comparing textual semantic similarity.
3.3 Geometric Intuition of Cosine Similarity
flowchart TD
A["Compare two non-zero vectors"] --> B["Calculate the angle between the vectors"]
B --> C{"Angle magnitude"}
C -->|"Close to 0°"| D["cosine close to 1"]
C -->|"Close to 90°"| E["cosine close to 0"]
C -->|"Close to 180°"| F["cosine close to -1"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#E5F0E9,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
4. Calculation Principles of Cosine Similarity
The calculation of cosine similarity can be broken down into two parts:
- Compute the vector dot product.
- Divide the dot product by the product of the magnitudes of the two vectors.
4.1 Vector Dot Product
The vector dot product (Dot Product) is also called the inner product.
For two d-dimensional vectors:
The dot product is defined as:
Expanded:
For example:
Then:
The dot product reflects the degree of numerical association between two vectors, but it is also affected by both vector direction and length.
Therefore, the magnitude of the dot product cannot be directly equated with cosine similarity.
4.2 Vector Magnitude
The vector magnitude (Vector Norm) represents the length of the vector.
For a vector:
Its L2 norm is:
For example:
Then:
Vector magnitude is used to eliminate the effect of vector length on the dot product result.
4.3 Complete Calculation Example
Given:
Step 1: Compute the dot product.
Step 2: Compute the vector magnitudes.
Step 3: Substitute into the cosine similarity formula.
The directions of the two vectors are very close, so the cosine similarity is close to 1.
4.4 Python Implementation
You can use NumPy to implement cosine similarity:
import numpy as np
def cosine_similarity(a, b):
a = np.asarray(a, dtype=np.float64)
b = np.asarray(b, dtype=np.float64)
if a.ndim != 1 or b.ndim != 1:
raise ValueError("输入必须是一维向量")
if a.shape != b.shape:
raise ValueError("向量维度必须一致")
norm_a = np.linalg.norm(a)
norm_b = np.linalg.norm(b)
if norm_a == 0 or norm_b == 0:
raise ValueError("零向量无法计算余弦相似度")
return np.dot(a, b) / (norm_a * norm_b)
a = [1, 2, 3]
b = [2, 3, 4]
score = cosine_similarity(a, b)
print(score)
Output:
0.9925833339709303
Here, float64 is used mainly to make the example calculation more stable. Actual Embedding models and vector databases may use float32, float16, or quantized data types.
5. Cosine Similarity and Vector Normalization
Vector normalization is an important operation in vector retrieval.
Its core goal is to adjust vectors to a uniform length, usually to unit length 1.
5.1 L2 Normalization
For a non-zero vector A, the L2 normalization formula is:
After normalization:
For example:
Its norm is:
After normalization:
At this point:
5.2 After Normalization, Cosine Similarity Can Be Simplified
Original cosine similarity:
If both A and B have already been normalized:
Then:
In other words, the dot product of two unit vectors is equal to their cosine similarity.
This brings an important engineering optimization:
If vectors have already been L2-normalized at both write and query time, inner product computation can be used instead of full cosine similarity computation.
However, the prerequisite is that the database's distance metric configuration, data preprocessing, and query vector handling must remain consistent.
5.3 Relationship Between Normalization, Inner Product, and Cosine Similarity
| Vector State | Relationship Between Inner Product and Cosine Similarity |
|---|---|
| Neither vector is normalized | Usually not equal |
| Both vectors have undergone L2 normalization | Exactly equal |
| Only one vector is normalized | Usually not equal |
| The two vectors use other normalization methods | Cannot be directly considered equal |
5.4 Python Normalization Implementation
import numpy as np
def l2_normalize(vector):
vector = np.asarray(vector, dtype=np.float32)
norm = np.linalg.norm(vector)
if norm == 0:
raise ValueError("零向量无法进行 L2 归一化")
return vector / norm
vector = np.array([3, 4], dtype=np.float32)
normalized = l2_normalize(vector)
print(normalized)
print(np.linalg.norm(normalized))
Output:
[0.6 0.8]
1.0
5.5 Normalization Is Not Always Necessary
Note that normalization is not a step that must be performed for all Embedding models.
Different models may have different training objectives and vector output methods:
- Some models output normalized vectors by default.
- Some models output unnormalized vectors.
- For some models, the recommended retrieval method is Cosine Similarity.
- Some models recommend using Dot Product.
- Some models require strictly following the specified Query / Document encoding method.
If the model itself treats vector norm as part of its training objective, extra normalization may change the original similarity ranking behavior.
Therefore, priority should be given to following the usage guidelines of the Embedding model, rather than unconditionally normalizing all vectors.
6. Differences Between Cosine Similarity, Euclidean Distance, and Inner Product
Vector retrieval does not only involve cosine similarity; there are also multiple distance metrics.
The most common include:
- Cosine Similarity: cosine similarity.
- Euclidean Distance: Euclidean distance.
- Inner Product: inner product.
They can all be used to measure relationships between vectors, but their mathematical meanings and ranking behaviors are different.
6.1 Cosine Similarity
Formula:
Characteristics:
- Focuses on vector direction.
- Insensitive to positive proportional scaling.
- Commonly used in text Embedding retrieval.
- Needs to handle the zero-vector issue.
Generally, the higher the score, the closer the directions.
6.2 Euclidean Distance
Euclidean Distance is used to measure the straight-line distance between two vectors in space.
Formula:
Expanded:
For example:
Then:
The smaller the Euclidean distance, the closer the two vectors are in space.
Unlike cosine similarity, Euclidean distance is affected by both vector direction and magnitude.
6.3 Inner Product
Inner product formula:
It directly computes the sum of the products of corresponding dimensions.
Characteristics:
- The calculation is simple.
- Vector length affects the result.
- It is very suitable for some retrieval models.
- It cannot directly replace cosine similarity when the conditions are not met.
6.4 Comparison of the Three Metrics
| Dimension | Cosine | Euclidean | Inner Product |
|---|---|---|---|
| Basis of calculation | Angle between vectors | Distance in vector space | Vector dot product |
| Whether affected by magnitude | Not affected by proportional scaling | Affected | Affected |
| Sorting direction | The larger, the more similar | The smaller, the closer | Usually, the larger, the more similar |
| Common uses | Semantic retrieval | Spatial distance retrieval | Vector matching and model-specified retrieval |
6.5 Why Is There Also a Relationship Between Euclidean Distance and Cosine Similarity After Normalization?
For two unit vectors A and B:
The square of the Euclidean distance:
Expanding:
Because:
Therefore:
And because the dot product of unit vectors equals cosine similarity:
From this, we can obtain:
When all vectors are L2-normalized, the larger the cosine similarity, the smaller the Euclidean distance, and the two produce the same nearest-neighbor ranking.
However, their raw scores are not the same.
6.6 An Important Engineering Pitfall
Suppose a vector database is configured to use L2 distance, but the business code treats the returned value as Cosine Similarity.
This may lead to:
- Mistaking a smaller distance for lower similarity.
- Handling the sorting direction incorrectly.
- Threshold judgment becoming completely invalid.
- Retrieval results being incorrectly filtered.
Different vector databases may define Score, Distance, and Similarity differently.
When using a database, you must confirm:
- The configured distance metric type.
- Whether the returned field represents distance or similarity.
- Whether a larger or smaller value indicates closeness.
- Whether vector normalization has been performed.
- Whether the distance has been transformed.
Do not assume that score is cosine similarity merely based on the field name.
VII. Complete Execution Flow of Semantic Retrieval
Semantic retrieval is not just about calculating the similarity between two vectors; it also involves data preparation, query encoding, filtering, candidate recall, and result ranking.
7.1 Offline Knowledge Ingestion
The offline phase is responsible for converting documents into searchable data.
Main steps:
- Acquire the original documents.
- Parse the document content.
- Split the documents into chunks.
- Use an embedding model to generate vectors.
- Save the vectors, text, and metadata.
- Build a vector index.
flowchart TD
A["Source Document"] --> B["Document Parsing"]
B --> C["Text Chunking"]
C --> D["Embedding Encoding"]
D --> E["Generate Document Vectors"]
E --> F["Write to Vector Database"]
F --> G["Build Vector Index"]
G --> H["Complete Knowledge Ingestion"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
7.2 Online Semantic Retrieval
After a user submits a query, the system needs to convert the query into a vector and find the closest document vectors in the vector database.
Main steps:
- Receive the user query.
- Use a matching embedding model to generate the query vector.
- Check the vector dimensions and model configuration.
- Apply metadata filtering based on permissions and business conditions.
- Use the vector index to recall candidate results.
- Calculate or retrieve similarity scores.
- Rank the candidate results.
- Return the Top-K documents.
flowchart TD
A["User Submits Query"] --> B["Query Embedding"]
B --> C{"Do Vector Dimensions Match Model Configuration?"}
C -->|"No"| X["Retrieval Failed: Configuration Mismatch"]
C -->|"Yes"| D["Apply Metadata and Permission Filtering"]
D --> E["ANN Vector Candidate Recall"]
E --> F{"Are Candidate Results Found?"}
F -->|"No"| Y["Return Empty Result"]
F -->|"Yes"| G["Similarity Calculation and Ranking"]
G --> H["Top-K Results"]
H --> I["Optional Reranker Reranking"]
I --> J["Return Retrieval Results"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
style X fill:#F7EAE7,stroke:#C98D7C,color:#754438
style Y fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
Note that the actual execution point of metadata filtering depends on the vector database implementation. Some systems support pre-retrieval filtering, some perform filtering during index traversal, and some support post-retrieval filtering. Different strategies affect recall and query performance.
7.3 Top-K Retrieval
Top-K means selecting the K results with the highest similarity from the candidate documents.
For example:
User query: How can database query performance be optimized?
Candidate results:
Document A: SQL index optimization methods
Document B: Database execution plan analysis
Document C: Database connection pool configuration
Document D: Python file reading methods
Document E: Database query cache design
Assuming K = 3, the system ultimately returns the three most relevant candidate documents.
Note:
- The larger the Top-K, the broader the recall scope is usually.
- A larger Top-K may also introduce more irrelevant content.
- For RAG, too much context increases token cost and noise.
- Top-K should be tuned in combination with recall, ranking quality, and context window.
8. How Should Similarity Scores Be Understood?
In semantic retrieval systems, results like the following are often seen:
[
{
"id": "doc_001",
"text": "SQL indexes can reduce the amount of data scanned by database queries.",
"score": 0.92
},
{
"id": "doc_002",
"text": "Database execution plans can help analyze slow queries.",
"score": 0.87
},
{
"id": "doc_003",
"text": "Python can read text through file objects.",
"score": 0.31
}
]
If score here explicitly means Cosine Similarity, then the higher the score, the closer the vector directions are.