1. What Is an Embedding?
Embedding is usually translated as embedding or vector embedding, and it is a representation method that maps discrete data into a low-dimensional continuous vector space.
In natural language processing, text is initially composed of discrete symbols such as characters, words, or tokens. These symbols themselves cannot directly participate in the matrix operations inside a neural network, so they need to be converted into numerical vectors.
Embedding accomplishes exactly this process.
For example:
Token Embedding
-----------------------------------------
apple → [0.21, 0.83, -0.14, ...]
banana → [0.18, 0.76, -0.09, ...]
computer → [0.92, -0.31, 0.57, ...]
Each token corresponds to a vector of fixed dimensionality.
These vectors are not hand-written features; they are parameters that the model learns through training. During training, the model continuously adjusts the vectors so that they can better serve downstream tasks.
1.1 The Core Role of Embedding
Embedding mainly solves three problems:
- Numerical representation: converting discrete symbols into continuous numerical values that a neural network can process.
- Semantic feature expression: expressing the semantics and features of objects through the numerical patterns in the vectors.
- Similarity computation: using the distance or similarity between vectors to measure the degree of association between different objects.
Embedding is not only a text-processing technique; it is also widely used in:
- Natural language processing: token, sentence, and document vectors.
- Computer vision: image, region, and visual patch vectors.
- Recommender systems: user, item, and behavior vectors.
- Multimodal models: unified or aligned representations of text, images, audio, and video.
- Search systems: semantic retrieval and vector databases.
2. Why Do We Need Embedding?
To understand Embedding, you first need to understand the difference between discrete data and continuous numerical values.
2.1 Token IDs Themselves Carry No Semantics
After tokenizer processing, text is converted into token IDs.
For example:
"I like artificial intelligence"
Tokenizer
↓
[I, like, artificial, intelligence]
Token ID
↓
[102, 356, 1287, 2048]
These IDs are merely indices in the vocabulary.
A token ID of 102 does not mean it is more important than token ID 356, nor does it mean that the closer two IDs are, the more similar their semantics are.
If token IDs are fed directly into a neural network as ordinary numerical values, meaningless numerical relationships are introduced.
For example:
- ID 100 and ID 101 do not necessarily have similar semantics.
- ID 100 and ID 1000 do not necessarily have a greater semantic difference.
- The magnitude of an ID does not represent the importance of the word.
Therefore, token IDs need to be mapped into vectors with learnable expressive capability.
2.2 The Limitations of One-Hot Encoding
One intuitive approach is to use One-Hot Encoding.
Assume the vocabulary size is 5:
| Token | One-Hot Vector |
|---|---|
| Apple | [1, 0, 0, 0, 0] |
| Banana | [0, 1, 0, 0, 0] |
| Computer | [0, 0, 1, 0, 0] |
| Orange | [0, 0, 0, 1, 0] |
| Book | [0, 0, 0, 0, 1] |
One-Hot encoding has two main problems.
Problem 1: Dimensionality is too high
If the vocabulary contains 100,000 tokens, each token requires a 100,000-dimensional vector.
This brings considerable storage and computational overhead.
Problem 2: It cannot directly express semantic relationships
The One-Hot vectors of any two different tokens are orthogonal:
Therefore, it cannot directly express:
- Apple and Orange are both fruits.
- Cat and Dog are both animals.
- Computer and Keyboard both belong to the computing domain.
One-Hot expresses only identity, not semantics.
2.3 How Embedding Improves on This
Embedding uses low-dimensional dense vectors in place of high-dimensional sparse vectors.
For example:
One-Hot:
Apple → [1, 0, 0, 0, 0, ...]
Banana → [0, 1, 0, 0, 0, ...]
Embedding:
Apple → [0.21, 0.83, -0.14, 0.45]
Banana → [0.18, 0.76, -0.09, 0.39]
Embedding has the following characteristics:
- The dimensionality is usually far smaller than the vocabulary size.
- Most dimensions of the vector contain non-zero numerical values.
- The vectors are learned through the training process.
- Certain semantic relationships can be expressed through vector operations.
Note that Embedding does not guarantee that all semantically similar objects will necessarily be close in every model. The structure of the vector space depends on the training objective, data distribution, model architecture, and training method.
3. The Mathematical Principles of Embedding
An embedding is essentially a learnable parameter matrix.
3.1 The Embedding Matrix
Assume:
- The vocabulary size is V.
- The embedding dimension is d.
Then the embedding matrix can be expressed as:
Where:
- V: the number of tokens in the vocabulary.
- d: the vector dimension corresponding to each token.
- E: the embedding parameter matrix that the model needs to learn.
For example:
Then the shape of the embedding matrix is:
In other words, every token in the vocabulary has a 768-dimensional vector.
3.2 Mapping from Token ID to Embedding
Suppose the input token ID is:
Then the model retrieves row 102 from the embedding matrix:
yielding:
This operation is called embedding lookup.
The process can be represented as:
flowchart TB
A["Input text"] --> B["Tokenizer"]
B --> C["Token ID sequence"]
C --> D["Embedding matrix"]
D --> E["Look up vector by Token ID"]
E --> F["Embedding vector sequence"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
From a computational perspective, embedding lookup can be understood as reading the corresponding row from the parameter matrix by index.
It can also be expressed as a one-hot matrix multiplication:
Where:
- x_i\in\mathbb{R}^{V}: the one-hot vector of the token.
- E\in\mathbb{R}^{V\times d}: the embedding matrix.
- e_i\in\mathbb{R}^{d}: the resulting embedding.
In practice, implementations usually do not actually construct the full one-hot vector; instead they look it up directly by index, avoiding unnecessary computation.
3.3 How Dimensions Change with Batched Input
Suppose the input has:
- Batch Size: B=2
- Sequence Length: L=4
- Embedding Dimension: d=768
The shape of the input token IDs is:
After embedding lookup:
we get:
The three dimensions represent:
| Dimension | Meaning |
|---|---|
| Batch | Number of samples processed at once |
| Sequence Length | Number of tokens contained in each sample |
| Embedding Dimension | Vector dimension of each token |
This is also a very common tensor shape in Transformers.
Note the distinction:
- Token IDs are discrete indices.
- Embeddings are continuous vectors.
- Sequence Length is the length of the sequence.
- Embedding Dimension is the feature dimension of a single token.
4. How Are Embeddings Trained?
Embedding vectors are usually not designed by hand; they are learned through a training objective and backpropagation.
4.1 Embeddings Are Themselves Model Parameters
In neural networks, the embedding matrix E is a trainable parameter.
During model training, the process usually goes through:
- Look up the embedding based on the Token ID.
- Feed the vector into subsequent network layers.
- Perform forward propagation.
- Compute the loss function based on the prediction result.
- Compute gradients through backpropagation.
- The optimizer updates the embedding matrix and other model parameters.
Training process:
flowchart TB
A["Token ID"] --> B["Embedding Lookup"]
B --> C["Obtain Embedding Vector"]
C --> D["Neural Network Computation"]
D --> E["Model Prediction"]
E --> F["Compute Loss"]
F --> G["Backpropagation"]
G --> H["Compute Embedding Gradients"]
H --> I["Optimizer Updates Parameters"]
I --> B
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Embedding parameters are continuously adjusted according to the training objective.
For example, in a language model, if certain vector representations help the model predict the next Token more accurately, then the training process will use gradient updates to gradually adapt the relevant parameters to this objective.
4.2 How Do Embeddings Acquire Semantic Information?
Embeddings do not automatically acquire meaning through some independent semantic annotation mechanism.
They mainly learn statistical patterns from training data and training objectives.
For example:
I like eating apples.
I like eating bananas.
I like eating oranges.
Apples are a kind of fruit.
Bananas are a kind of fruit.
Oranges are a kind of fruit.
These words often appear in similar contexts.
During training, the model gradually learns the statistical associations between them.
These associations may reflect:
- Semantic similarity.
- Grammatical roles.
- Contextual relationships.
- Topic relatedness.
- Task-related features.
But note:
Similar contexts do not mean identical semantics, and similar vectors do not mean two words can be substituted for each other.
For example:
- “Apple” and “banana” have similar category attributes.
- “Apple” and “phone” may have a strong association in some corpora.
- “Good” and “not good” are highly similar in literal form, but their semantic directions may be opposite.
What embeddings learn is the representation required by the training objective, not a naturally complete dictionary of word definitions.
5. From Word2Vec to Transformer Embeddings
Embedding technology has gone through multiple stages of development.
The main differences between stages lie in how vectors are trained and whether they can change dynamically according to context.
5.1 Word2Vec: Learning Word Vectors from Context
Word2Vec is a classic static word vector learning method.
It learns word vectors through the contextual relationships of words and mainly includes two training architectures.
CBOW (Continuous Bag of Words)
Predicts the center word based on context words.
For example:
I like [apples] and bananas
The model uses the surrounding words to predict the center word “apples.”
Skip-gram
Predicts the surrounding context words based on the center word.
For example:
Center word: apples
Predict:
I, like, and, bananas
The two methods have different directions:
| Method | Input | Prediction Target |
|---|---|---|
| CBOW | Context words | Center word |
| Skip-gram | Center word | Context words |
Word2Vec generates static word vectors.
The same word usually uses the same vector in different sentences.
For example:
I like apples.
Apple released a new phone.
In Word2Vec, the two occurrences of “apple” usually correspond to the same word vector.
This makes it difficult to directly distinguish the meanings of the same word in different contexts.
5.2 GloVe: Leveraging Global Word Co-occurrence Statistics
GloVe (Global Vectors for Word Representation) learns word vectors through global word co-occurrence statistics.
It focuses on the co-occurrence relationships of words across the entire corpus.
For example:
- "Doctor" often appears together with "hospital".
- "Program" often appears together with "code".
- "Airplane" often appears together with "airport".
GloVe learns vector representations through a word co-occurrence matrix and its statistical relationships.
Unlike Word2Vec, which mainly relies on local context prediction, GloVe places more emphasis on global co-occurrence statistics.
Both belong to the class of classic static word vector methods.
5.3 ELMo: Introducing Contextual Word Vectors
ELMo (Embeddings from Language Models) began to use contextual information to dynamically generate word vectors.
The same word can receive different representations in different sentences.
For example:
I went to the bank to handle some business.
There is a row of trees along the riverbank.
This assumes that the second sentence discusses a different context with a related meaning, and the model will generate different representations according to the context.
ELMo uses a bidirectional language model to extract contextual features, so that word vectors no longer depend entirely on the word itself.
5.4 BERT: Learning Contextual Representations with Transformer
BERT uses the Transformer Encoder architecture and learns Token representations through bidirectional context.
Its pretraining tasks mainly include:
- Masked Language Modeling (MLM): randomly masks some Tokens and asks the model to predict the masked content.
- Next Sentence Prediction (NSP): the inter-sentence relationship task used in the original BERT training.
For example:
Original text:
I like learning artificial intelligence.
Masked part:
I like learning [MASK] intelligence.
Model prediction:
artificial
The Token representations generated by BERT are influenced jointly by the left and right context.
Therefore, the same Token can receive different vectors in different sentences.
5.5 GPT: Learning Representations through Autoregressive Language Modeling
GPT uses the Transformer Decoder architecture and adopts an autoregressive language modeling objective.
Its core task is to predict the next Token based on the preceding context:
When processing a sequence, GPT converts Token IDs into vectors through the Embedding layer, and then computes contextual representations through multiple Transformer layers.
Unlike static word vectors, Token representations in a Transformer change with the network layer and the context.
Three concepts need to be distinguished:
- Token Embedding: the initial vector corresponding to a Token ID.
- Contextual Representation: the context-dependent representation computed by networks such as a Transformer.
- Output Embedding / Language Modeling Head: the output parameters or mapping structure used to map the model's hidden states into the vocabulary prediction space.
They may share some parameters in certain models, but they are not inherently the same concept.
5.6 Comparison of Different Embedding Techniques
| Technology | Core Method | Context-Dependent? |
|---|---|---|
| Word2Vec | Local context prediction | No |
| GloVe | Global word co-occurrence statistics | No |
| ELMo | Bidirectional language model | Yes |
| BERT | Transformer Encoder | Yes |
| GPT | Autoregressive Transformer | Yes |
The development trend can be summarized as:
flowchart TB
A["Static word vectors"] --> B["Word2Vec / GloVe"]
B --> C["Contextual representations"]
C --> D["ELMo"]
D --> E["Transformer representations"]
E --> F["BERT / GPT"]
F --> G["Multimodal Embedding"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
Note that Word2Vec, GloVe, ELMo, BERT, and GPT are not completely homogeneous models. They differ in architecture, training objectives, and representation methods.
6. The Relationship Between Embedding and Transformer
Embedding is one of the input foundations of the Transformer, but it is not equivalent to all vector representations inside the Transformer.
6.1 Input Embedding of the Transformer
In a typical Transformer architecture, Token IDs first pass through the Embedding layer and are converted into continuous vectors.
For an input sequence:
After Embedding:
where:
The resulting vector sequence then enters the Transformer network.
6.2 Why Are Positional Encodings Still Needed?
Embedding represents the content features of a Token, but ordinary Token Embedding itself does not contain information about the Token's position in the sequence.
For example:
I like artificial intelligence
Artificial intelligence likes me
The two sentences contain the same set of Tokens, but their order is different, and their meanings may also differ.
The self-attention mechanism of the Transformer does not inherently carry sequence-order information, so positional information needs to be introduced.
In the classic Transformer, Positional Encoding is used:
where:
- E(X): Token Embedding.
- P: positional encoding.
- H_0: the initial input representation of the Transformer.
Positional encoding can use fixed sinusoidal functions or learnable position vectors.
Modern Transformers may also use positional encoding or positional bias mechanisms such as RoPE (Rotary Positional Embedding) and ALiBi.
These methods are not all implemented through simple vector addition.
6.3 The Difference Between Embedding and Hidden States
This is a very important distinction when understanding the Transformer.
Suppose the model hidden dimension is d, and the input sequence length is L.
Token Embedding:
After the first Transformer layer:
After the second layer:
Finally, we obtain:
where:
- H_0: the initial Token Embedding (if the model also adds positional representations, this is the combined initial input).
- H_1: the output of the first Transformer layer.
- H_N: the hidden state of the last Transformer layer.
These hidden states contain contextual information after computation through multiple layers.
Therefore:
Embedding is the model's initial vector representation, while the hidden state is the result of the model performing contextual computation on the input.
flowchart TB
A["Token ID Sequence"] --> B["Token Embedding"]
B --> C["Add Positional Information"]
C --> D["Transformer Layer 1"]
D --> E["Transformer Layer 2"]
E --> F["Transformer Layer N"]
F --> G["Contextual Hidden State"]
G --> H["Downstream Task or Output Prediction"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
7. Embedding Space and Vector Similarity
One of the important values of Embedding is that it can express associations between objects through vector space.
7.1 Vector Space
Suppose each Token corresponds to a three-dimensional vector:
Then each Token can be viewed as a point in three-dimensional space.
If two vectors have similar features under some training objective, they may be relatively close in vector space.
Real models usually use Embeddings with hundreds or even thousands of dimensions.
For example:
- 384 dimensions.
- 768 dimensions.
- 1024 dimensions.
- 1536 dimensions.
- 3072 dimensions.
Higher dimensionality does not necessarily mean stronger semantic capability. Actual performance is also affected by factors such as model architecture, training data, objective function, vector quality, and inference method.
7.2 Cosine Similarity
To measure whether two Embeddings are similar, cosine similarity is commonly used.
Formula:
Where:
- A\cdot B: the dot product of the two vectors.
- \|A\|: the magnitude of vector A.
- \|B\|: the magnitude of vector B.
Cosine similarity focuses on the angle between the directions of two vectors.
Generally:
| Similarity | Meaning |
|---|---|
| Close to 1 | Directions are highly similar |
| Close to 0 | Directions are nearly orthogonal |
| Close to -1 | Directions are highly opposite |
However, these values cannot be directly interpreted as the probability of semantic similarity.
For example, a cosine similarity of 0.8 does not mean that two texts are 80% semantically identical.
7.3 Euclidean Distance
Euclidean distance measures the straight-line distance between two vectors.
Formula:
The smaller the distance, the closer the positions of the two vectors in the current space.
Cosine similarity and Euclidean distance focus on different things:
- Cosine similarity: focuses more on vector direction.
- Euclidean distance: affected by both direction and magnitude.
Once vectors are L2-normalized, there is a clear mathematical relationship between the two:
Therefore, for unit vectors, maximizing cosine similarity is equivalent to minimizing Euclidean distance.
7.4 Dot Product Similarity
The dot product is also commonly used in vector retrieval:
The dot product is affected by both vector direction and magnitude.
In vector databases and retrieval systems, common similarity computation methods include:
- Cosine Similarity.
- Inner Product.
- Euclidean Distance.
Which metric to choose depends on how the Embedding model was trained and on the configuration of the retrieval system.
8. Main Types of Embedding
Embeddings are not limited to token vectors. Different data types can each learn their own vector representations.
8.1 Token Embedding
Convert token IDs into vectors.
Mainly used for:
- Large language model input.
- Text encoding.
- Transformer networks.
8.2 Word Embedding
Map words to vectors.
Classic examples:
- Word2Vec.
- GloVe.
- FastText.
Word embeddings usually take words as the basic unit, whereas modern large language models generally use token-level representations.
8.3 Sentence Embedding
Encode an entire sentence into a fixed-dimensional vector.
For example:
Sentence A:
How do you learn machine learning?
Sentence B:
Where should you start with machine learning?
Sentence Embedding
↓
Sentence A → [0.21, 0.73, -0.15, ...]
Sentence B → [0.19, 0.70, -0.12, ...]
If the model is appropriately trained, these two vectors may have a high semantic similarity.
Sentence Embedding is commonly used for:
- Semantic search.
- Text clustering.
- Similar question matching.
- Text deduplication.
- FAQ retrieval.
8.4 Document Embedding
Convert a document or a longer piece of text into a vector.
Mainly used for:
- Document semantic retrieval.
- Knowledge base search.
- RAG.
- Document clustering.
- Content recommendation.
For long documents, it is usually necessary to chunk them first and then generate an Embedding for each chunk separately.
Otherwise, overly long text may exceed the model's input length limit, or cause local information to be diluted in the overall representation.
8.5 Image Embedding
Convert an image into a vector representation.
For example:
Input image
↓
Vision encoder
↓
Image Embedding
↓
[0.13, 0.72, -0.21, ...]
Image embeddings can be used for:
- Image-to-image search.
- Image classification.
- Image clustering.
- Image retrieval.
- Cross-modal matching.
8.6 Multimodal Embedding
Multimodal embeddings map data from different modalities into a vector space where they can be compared or interact with one another.
Common modalities include:
- Text.
- Image.
- Audio.
- Video.
For example, CLIP uses contrastive learning so that text and image representations can be compared for similarity in a shared space.
Text:
A dog running on the grass
↓
Text encoder
↓
Text Embedding
An image of a dog running on the grass
↓
Image encoder
↓
Image Embedding
↓
Compute cross-modal similarity
Note: multimodal Embedding does not necessarily mean that all modalities use the same encoder, nor does it mean that their original vector dimensions or internal structures are completely identical.
8.7 Comparison of Different Types
| Type | Input object | Typical use |
|---|---|---|
| Token Embedding | Token | Language model input |
| Word Embedding | Word | Word semantic analysis |
| Sentence Embedding | Sentence | Semantic matching |
| Document Embedding | Document | Document retrieval |
| Image Embedding | Image | Image retrieval |
| Audio Embedding | Audio | Audio recognition and retrieval |
| Multimodal Embedding | Multimodal data | Cross-modal retrieval and matching |
Nine, Applications of Embedding in Large Language Models
Embedding is not only an important part of the model input layer, but also a key technology for large language models to interact with external knowledge systems.