1. What Is a Vector Space?
A vector space is a mathematical structure composed of vectors, with operations such as vector addition and scalar multiplication defined on it.
In machine learning and natural language processing, vector spaces provide a unified numerical representation for objects such as text, words, sentences, and images.
For example:
cat → [0.21, 0.83, -0.14, 0.56]
dog → [0.19, 0.79, -0.11, 0.52]
car → [0.91, -0.32, 0.45, -0.18]
These vectors themselves do not have directly readable natural-language meaning, but models can learn and express associations between objects through the mathematical relationships between vectors.
1.1 Basic Form of a Vector
An n-dimensional vector can be represented as:
where:
- \mathbf{x}: a vector.
- x_i: the i-th dimension of the vector.
- n: the vector dimension.
- \mathbb{R}^{n}: the vector space formed by n real numbers.
For example, a 4-dimensional vector:
can be viewed as a representation jointly formed by four numerical dimensions.
In machine learning, vector dimensions are usually determined by the model architecture and do not necessarily correspond to concepts that humans can explicitly name.
1.2 Basic Operations in a Vector Space
Vector addition
Two vectors with the same dimension can be added dimension by dimension:
Scalar multiplication
A vector can be multiplied by a real number:
Vector subtraction
Dot product
The dot product can measure how well two vectors agree in direction and is also an important basis for computing cosine similarity and attention weights.
2. Why Represent Semantics as Vectors?
Computers cannot directly understand natural language in the way humans do.
For a computing system, text is first and foremost a sequence of discrete symbols. To enable a model to process semantic relationships, discrete text needs to be transformed into a computable numerical representation.
One of the simplest approaches is to use integer encoding:
cat → 101
dog → 102
car → 103
However, this kind of encoding can only represent the identities of different objects; it cannot directly express the semantic relationships between them.
For example:
- Cats and dogs both belong semantically to the category of animals.
- Cars and bicycles both belong to the category of vehicles.
- Apples and bananas both belong to the category of fruits.
Integer encoding cannot directly reflect these relationships.
If vector representations are used, the model can, through training, make objects with similar semantics form certain geometric associations in the vector space.
Discrete text
↓
Numerical encoding
↓
Vector representation
↓
Compute relationships between vectors
↓
Perform tasks such as semantic matching, classification, and clustering
Therefore, the core value of vector representation is:
transforming semantic relationships that are difficult to compute directly into numerical relationships that can be processed through mathematical operations.
It should be noted that vectorization does not mean that a computer has achieved understanding in the human sense. It means that the model has learned a numerical representation that can support prediction and task processing.
3. Embedding: From Discrete Symbols to Continuous Vectors
Embedding is a technique for mapping discrete objects into a continuous vector space.
In natural language processing, Embedding can represent tokens, words, sentences, or documents as vectors.
3.1 Token Embedding
Large language models usually first use a Tokenizer to split text into tokens, and then convert the tokens into token IDs.
For example:
Raw text
↓
Tokenizer
↓
Token sequence
↓
Token ID sequence
↓
Embedding layer
↓
Vector sequence
Suppose the vocabulary contains the following tokens:
| Token | Token ID |
|---|---|
| I | 101 |
| like | 205 |
| artificial intelligence | 308 |
The model can use an Embedding matrix to map each Token ID to a fixed-dimensional vector.
Let the vocabulary size be V and the Embedding dimension be d. Then the Embedding matrix is:
For Token ID i, the corresponding vector is:
For example:
This indicates that the vocabulary contains 50,000 tokens, and each token corresponds to a 768-dimensional vector.
It is important to distinguish between the following:
- Token ID is a discrete index in the vocabulary.
- An embedding vector is a continuous numerical representation learned by the model.
- The vector dimension is not the number of tokens.
- The value of a Token ID does not indicate semantic magnitude or degree of similarity.
3.2 How Are Embeddings Learned?
Embeddings are usually learnable parameters during model training.
At the start of training, embedding parameters may receive initial values through random initialization.
As model training proceeds, the loss function produces gradients; backpropagation passes the gradients to the embedding parameters, and the optimizer then updates the parameters based on the gradients.
This process can be abstracted as follows:
flowchart TB
A["Input Token ID"] --> B["Look up the Embedding matrix"]
B --> C["Obtain initial vector representation"]
C --> D["Participate in model forward computation"]
D --> E["Compute training loss"]
E --> F["Backpropagate to compute gradients"]
F --> G["Update Embedding parameters"]
G --> H["Form a new vector representation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
Embeddings are not constructed by manually specifying the meaning of each dimension.
They gradually form representations useful for downstream tasks through training objectives and contextual relationships.
3.3 Static Embeddings and Context-Dependent Representations
Traditional word vector models usually learn a relatively fixed vector for each word.
For example, Word2Vec and GloVe are both classic static word vector methods.
However, natural language is ambiguous.
For example:
- An apple is a fruit.
- Apple released a new product.
The word “apple” has different meanings in the two sentences.
Static word vectors usually cannot dynamically adjust the representation of the same word according to context.
Modern Transformer models, by contrast, generate dynamic representations through contextual computation:
Where:
- x_i: the i-th token.
- \mathbf{h}_i: the hidden representation generated by the model for that token in the current context.
- f: the model’s contextual computation process.
Therefore, the same token can produce different hidden vectors in different contexts.
Note that Token Embedding and contextual hidden states are not the same concept:
| Type | Meaning |
|---|---|
| Token Embedding | The base vector corresponding to a Token ID |
| Position Embedding | A vector or positional encoding that represents position information |
| Hidden State | The contextual representation after computation by model layers |
| Sentence Embedding | Maps an entire sentence to a single vector |
| Document Embedding | Maps an entire document to a single vector |
IV. Geometric Relationships in Vector Space
The value of vector representations lies not only in converting text into numbers, but more importantly in using geometric relationships to measure the associations between different representations.
4.1 Vector Norm
The norm of a vector represents its length.
For a vector:
Its L2 norm is:
For example:
Then:
The norm can be used to measure the numerical scale of a vector.
In different models and applications, vector norms may have different meanings; one cannot simply interpret a larger norm as stronger semantics.
4.2 Euclidean Distance
Euclidean distance represents the straight-line distance between two vectors in space.
For two n-dimensional vectors:
The Euclidean distance is:
The smaller the distance, the closer the two vectors are under this metric.
Common applications:
- Vector clustering.
- K-Means.
- Nearest neighbor search.
- Distance computation in feature space.
However, Euclidean distance is affected by vector magnitude. If a task cares more about direction than numerical scale, cosine similarity is usually more suitable.
4.3 Cosine Similarity
Cosine Similarity measures the degree of similarity between the directions of two vectors.
It is defined as:
Its value range is:
It can usually be understood as follows:
| Cosine similarity | Geometric relationship |
|---|---|
| Close to 1 | Similar direction |
| Close to 0 | Approximately orthogonal directions |
| Close to -1 | Opposite directions |
For example:
The two vectors point in very similar directions, so their cosine similarity is high.
Cosine similarity is widely used in semantic search, text matching, and recommendation systems.
Note the following:
- Cosine similarity mainly measures direction and does not directly measure vector length.
- The cosine similarity of a zero vector is undefined, so practical implementations usually require special handling.
- High similarity does not mean the two texts are completely equivalent.
- Vectors produced by different embedding models usually cannot be compared directly.
4.4 Differences Between Dot Product and Cosine Similarity
The dot product and cosine similarity are closely related, but they are not the same.
where \theta is the angle between the two vectors.
The dot product is affected by both direction and magnitude, whereas cosine similarity normalizes the vector magnitude.
| Comparison item | Dot product | Cosine similarity |
|---|---|---|
| Considers direction | Yes | Yes |
| Considers magnitude | Yes | No, not after normalization |
| Value range | Not fixed | [-1, 1] |
| Common uses | Attention, vector retrieval | Semantic similarity, text matching |
If the vectors are all L2-normalized:
then:
This is also why many vector retrieval systems use normalized vectors and inner products for similarity search.
V. Semantic Space: How Do Vectors Express Semantic Relationships?
Semantic Space refers to a representation space learned by a model through vector representations that can reflect semantic associations.
Ideally, semantically similar objects will exhibit a certain similarity in the space.
For example:
- Cats, dogs, and rabbits may form a local region related to animals.
- Apples, bananas, and oranges may form a local region related to fruit.
- Cars, buses, and subways may form a local region related to transportation.
However, note that:
Semantic space is not a manually drawn map of conceptual categories; it is a high-dimensional representation structure formed by a model through its training objective.
It may encode multiple factors simultaneously, for example:
- Word meaning.
- Contextual relationships.
- Syntactic structure.
- Domain information.
- Task-related features.
- Statistical associations between words.
5.1 Semantic Similarity
Semantic Similarity indicates how close two texts are in meaning.
For example:
Text A: How do I learn Python programming?
Text B: Where should I start with Python?
Although the two sentences do not use exactly the same words, they express similar intents.
Based on semantic embeddings, they can be converted into vectors:
Then compute:
If two vectors have a high similarity, it means the model considers them to be strongly related in the current representation space.
5.2 Semantic Similarity Does Not Mean Semantic Equivalence
The following texts may have high topic similarity, but their actual meanings are not the same:
Text A: How to install Python?
Text B: How to uninstall Python?
They share a large amount of vocabulary and topic, but their operational goals are completely opposite.
This shows that:
- Lexical similarity does not equal semantic similarity.
- Topic similarity does not equal identical intent.
- Vector similarity cannot replace full semantic judgment.
An Embedding model's training objective, data distribution, and task type all affect its similarity behavior.
5.3 Semantic Directions and Vector Relationships
In classic word vector research, some word relationships have been observed to be expressible through vector arithmetic.
For example:
This phenomenon suggests that certain semantic relationships may appear as relatively stable geometric directions in a specific word vector space.
However, this should not be understood as:
- All semantic relationships have a linear structure.
- Every vector dimension corresponds to a clear human concept.
- Any model can reliably reproduce this relationship.
- Vector arithmetic is guaranteed to produce results that align with common sense.
The representation spaces of modern contextual Embeddings are more complex, and semantic relationships cannot necessarily be expressed through simple vector addition and subtraction.
Six, High-Dimensional Vector Spaces and the Curse of Dimensionality
Embeddings usually have relatively high dimensionality, such as 384, 768, 1024, 1536, or higher.
High-dimensional spaces can provide models with richer representational capacity, but they also bring computational and geometric problems.
6.1 Why Are High-Dimensional Representations Needed?
Natural language contains many complex relationships:
- Lexical semantics.
- Grammatical relationships.
- Contextual dependencies.
- Entity relationships.
- Domain knowledge.
- Task features.
Low-dimensional representations may be insufficient to fully express these relationships.
Increasing dimensionality can provide a larger representation space, but it does not mean that the higher the dimensionality, the better the results will necessarily be.
Actual performance also depends on:
- Model architecture.
- Training data.
- Training objective.
- Parameter scale.
- Negative sample construction.
- Vector normalization method.
- Downstream task.
6.2 The Curse of Dimensionality
The curse of dimensionality describes the phenomenon in which, as data dimensionality increases, many traditional geometric and statistical intuitions may fail.
For example, in low-dimensional spaces, nearby points are usually relatively easy to identify.
But in high-dimensional spaces:
- Data distribution may become sparse.
- The discriminative power of distance calculations may decrease.
- The difference in distance between the nearest and farthest neighbors may shrink.
- Models may need more data to cover the space.
Under specific conditions such as independent and identically distributed coordinates with similar scales, distances between high-dimensional random vectors may exhibit concentration.
This means that one cannot assume a stable semantic interpretation based solely on the magnitude of vector distance.
6.3 Vector Normalization
L2 normalization can map vectors onto the unit sphere:
After normalization:
It can reduce the impact of differences in vector magnitude on similarity computation.
However, normalization is not the default correct choice for all tasks.
If the model's training objective itself uses vector magnitude to express confidence or other information, blind normalization may lose useful information.
Seven, Sentence Embeddings and Document Embeddings
Token Embeddings mainly handle the representation of individual Tokens, whereas semantic search usually requires comparing complete sentences or documents.
Therefore, the Token sequence needs to be further converted into a fixed-dimensional vector.
7.1 Mean Pooling
Mean Pooling is a common way to generate sentence vectors.
Suppose a sentence produces n Token hidden vectors after passing through the model:
Mean pooling:
In practical implementations, an Attention Mask is usually needed to exclude Padding Tokens:
where m_i indicates whether that position is a valid Token.
Mean Pooling is simple and computationally inexpensive, but it does not guarantee that the resulting sentence vectors have good semantic retrieval capability.
7.2 CLS Pooling
Some models use a special CLS Token to represent the entire input sequence.
Through training, the model enables the hidden state at the CLS position to carry task-relevant sequence information.
This method depends on the model architecture and training objective; not all Transformers are suitable for directly using CLS Pooling.
7.3 Specially Trained Sentence Embeddings
Modern semantic search usually uses sentence Embedding models that have been specially trained.
Common training methods include:
- Contrastive Learning.
- Triplet Loss.
- Multiple-negatives ranking loss.
- Paired text similarity training.
- Query-document relevance training.
For example, contrastive learning can bring positive sample vectors closer together and push negative sample vectors farther apart.
A common form of the InfoNCE loss is:
Where:
- \mathbf{q}_i: query vector.
- \mathbf{d}_i^+: positive sample document vector matching the query.
- \mathbf{d}_j: candidate document vector, usually including negative samples.
- sim: similarity function.
- \tau: temperature parameter.
This training method makes the vector space better suited to semantic retrieval tasks.
8. Typical Applications of Vector Spaces in AI
8.1 Semantic Search
Traditional keyword search mainly relies on keyword matching.
For example:
User query: How can I improve program execution speed?
A document might contain:
Performance optimization: Methods for reducing program execution time
The two do not contain exactly the same keywords, but their meanings are similar.
Semantic search can convert the query and documents into vectors, then compute similarity.
flowchart TB
A["User enters query"] --> B["Query Embedding"]
B --> C["Generate query vector"]
D["Document collection"] --> E["Document chunking"]
E --> F["Document Embedding"]
F --> G["Generate document vectors"]
C --> H["Vector similarity calculation"]
G --> H
H --> I["Ranking and filtering"]
I --> J["Return relevant documents"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
Semantic search usually includes:
- Splitting documents into appropriate text chunks.
- Using an Embedding model to generate document vectors.
- Writing the vectors into a vector database or vector index.
- Converting the user query into a query vector.
- Retrieving candidate documents with higher similarity.
- Ranking by relevance and returning the results.
8.2 RAG Retrieval-Augmented Generation
RAG (Retrieval-Augmented Generation) combines external knowledge retrieval with large language model generation.
Vector retrieval is usually an important component of RAG.
The basic process is as follows: