1. What Is a Tokenizer?
A Tokenizer, often referred to as a word segmenter or tokenizer, is the component in a natural language processing system responsible for converting text into a Token sequence.
In large language models, the Tokenizer sits between the raw text and the neural network.
Humans input natural language:
Artificial intelligence is changing the way software is developed.
What the model actually receives is usually not the raw string, but a Token ID sequence processed by the Tokenizer:
Raw text
↓
Tokenizer
↓
Token sequence
↓
Token ID
↓
Embedding
↓
Transformer
For example:
Input:
I like learning artificial intelligence.
Token:
["I", "like", "learning", "artificial", "intelligence", "."]
Token ID:
[102, 356, 1287, 2048, 3096, 17]
The above is only illustrative; the actual segmentation result and IDs depend on the Tokenizer used by the specific model.
1.1 Core Responsibilities of a Tokenizer
A Tokenizer mainly performs the following tasks:
- Text normalization: Process spaces, character forms, and special symbols in the text according to specific rules.
- Text preprocessing: Identify text structure and prepare for subsequent segmentation.
- Token segmentation: Split text into discrete units defined by the model.
- Token ID mapping: Convert Tokens into integer indices in the vocabulary.
- Special Token handling: Identify or insert control markers required by the task.
- Text decoding: Restore a Token ID sequence to text.
A Tokenizer is not responsible for understanding the meaning of text, nor does it directly perform language reasoning. It is responsible for establishing the mapping between text and the model's input representation.
1.2 Differences Between Tokenizer, Token, and Token ID
These three concepts are easy to confuse.
| Concept | Meaning | Example |
|---|---|---|
| Tokenizer | Text encoding and decoding tool | BPE Tokenizer |
| Token | Discrete unit after segmentation | artificial |
| Token ID | Integer index corresponding to a Token | 2048 |
It can be understood as follows:
- The Tokenizer is the processing tool.
- A Token is a discrete unit in the processing result.
- A Token ID is the Token's number in the vocabulary.
A Token ID is only an index and has no inherent semantic ordering.
For example:
Token Token ID
--------------------
Apple 102
Banana 356
Computer 1287
102 is not more important than 356, and the numerical difference between two IDs does not represent semantic distance either.
2. Why Do Large Language Models Need a Tokenizer?
Computers can store and process text, but neural networks usually need to convert inputs into numerical tensors.
The Tokenizer is responsible for converting text into discrete indices that the model can process.
2.1 Raw Text Cannot Be Used Directly as an Embedding Index
Suppose the user inputs:
I like machine learning.
The raw text is a string and cannot be used directly as a row index into the Embedding matrix.
The model first needs to convert it into Token IDs:
Raw text:
I like machine learning.
Token ID:
[102, 356, 1287, 2560, 17]
Then look it up through the Embedding matrix:
where:
- E: the Embedding parameter matrix.
- token\_id_i: the integer index of the i-th Token.
- e_i: the corresponding Embedding vector.
Therefore, the Tokenizer is an important step before text enters the Embedding layer.
2.2 Why Not Split Directly by Character?
One of the simplest methods is to split text by character.
For example:
Artificial intelligence
["A", "r", "t", "i", "f", "i", "c", "i", "a", "l", " ", "i", "n", "t", "e", "l", "l", "i", "g", "e", "n", "c", "e"]
This approach can reduce the problem of unknown characters, but it has some limitations:
- The sequence length may be longer.
- A single character may not have complete semantics.
- Language expression efficiency may be lower.
- The model needs to learn word structure through more context combinations.
If we split by whole words, we encounter another problem:
- Out-of-vocabulary words are difficult to handle.
- Inflection produces a large number of vocabulary entries.
- New words and technical terms require additional handling.
- The vocabulary may become extremely large.
For this reason, modern large language models generally adopt subword or other fine-grained tokenization methods, trading off vocabulary size, sequence length, and expressive capacity.
2.3 Why Not Just Use Whole Words?
Suppose the vocabulary contains only:
play
playing
played
player
Then these word forms may need to be stored separately.
If subword segmentation is used, they can be split into more basic units:
play
playing → play + ing
played → play + ed
player → play + er
This reduces vocabulary duplication and improves the ability to handle new words and inflections.
The actual segmentation result is determined by the vocabulary and algorithm of the specific tokenizer, and is not necessarily identical to the example above.
3. The Complete Tokenizer Workflow
A tokenizer does not simply split text on spaces.
A typical text encoding pipeline may involve multiple stages.
3.1 Overall Processing Pipeline
flowchart TB
A["Raw Text"] --> B["Text Normalization"]
B --> C["Pre-tokenization"]
C --> D["Subword Segmentation"]
D --> E["Token Sequence"]
E --> F["Vocabulary Lookup"]
F --> G["Token ID Sequence"]
G --> H["Add Special Tokens"]
H --> I["Model Input"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
The internal implementations of different tokenizers are not entirely the same.
Some use regular expressions for pre-tokenization, some operate at the byte level, some perform Unicode normalization, and others convert text directly into byte sequences before subword segmentation.
The pipeline above is therefore a conceptual model; it does not mean that all tokenizers perform every step in exactly the same order.
4. Stage One: Text Normalization
Text normalization is the process of applying necessary standardization to the input text.
The same character may have different Unicode representations, and text may also contain different kinds of spaces, line breaks, and special characters.
4.1 Unicode Character Representation
Unicode defines a unified character encoding system for different languages and symbols.
For example:
- Chinese characters.
- English letters.
- Digits.
- Punctuation marks.
- Mathematical symbols.
- Emoji.
Note, however, that a character as seen by a human does not necessarily correspond to a single Unicode code point.
For example, some emoji are composed of multiple code points.
Therefore, one cannot simply assume:
one visible character = one Unicode code point = one Token
These three concepts are not equivalent.
4.2 Unicode Normalization
Unicode provides several normalization forms:
| Form | Meaning |
|---|---|
| NFC | Canonical decomposition followed by canonical composition |
| NFD | Canonical decomposition |
| NFKC | Compatibility decomposition followed by canonical composition |
| NFKD | Compatibility decomposition |
For example, some accented characters can be represented by different Unicode sequences.
Normalization can unify some of these equivalent representations.
However, whether Unicode normalization is performed, and which form is used, depends on the tokenizer's design.
One cannot assume that all models automatically apply NFKC or NFC.
4.3 Handling of Spaces and Line Breaks
Spaces can be significant in a tokenizer.
For example:
hello world
and:
helloworld
will yield different encoding results in many tokenizers.
Likewise:
Hello
hello
A leading space can also affect token segmentation.
Some tokenizers encode space information into the token representation.
Therefore, spaces in the text must not be deleted arbitrarily, or the Token ID sequence may change.
4.4 Case Handling
Letter case in English text may affect the Tokenizer's encoding results.
For example:
apple
Apple
APPLE
They may correspond to different Tokens or Token combinations.
Whether to normalize case depends on the text processing strategy used during model training.
If the model uses a case-sensitive Tokenizer, arbitrarily converting text to lowercase may change the model input.
5. Stage 2: Pre-tokenization
Pre-tokenization divides the raw text into larger candidate segments, providing processing boundaries for the subsequent subword algorithm.
For example:
I love artificial intelligence.
It may be pre-tokenized as:
["I", "love", "artificial", "intelligence", "."]
But this does not mean that every pre-tokenization segment will ultimately correspond to one Token.
A single pre-tokenization segment may still be split into multiple subwords.
5.1 Why is pre-tokenization needed?
Suppose the input is:
I love machine learning.
The pre-tokenization stage first identifies:
I
love
machine
learning
.
Then the subword algorithm processes these segments separately.
This reduces unnecessary cross-boundary combinations and makes vocabulary construction and encoding easier to control.
5.2 Limitations of whitespace tokenization
Whitespace tokenization is fairly intuitive for English, but it is not suitable for languages such as Chinese.
For example:
我喜欢人工智能
Chinese text usually does not insert spaces between every word.
Therefore, the Tokenizer needs to adopt a segmentation strategy better suited to the target language.
5.3 Pre-tokens are not the same as final Tokens
It is necessary to distinguish clearly between:
- Pre-token: a segment produced during the pre-tokenization stage.
- Token: a discrete unit obtained after final encoding.
For example:
Pre-tokenization:
unbelievable
Subword tokenization:
un + believe + able
The actual result depends on the model vocabulary and algorithm.
6. Stage 3: Subword Tokenization
Subword tokenization is one of the core technologies in modern large language model Tokenizers.
It splits text into words, roots, affixes, or other reusable subword units.
The goal is to strike a balance among the following factors:
- Vocabulary size.
- Sequence length.
- Ability to handle out-of-vocabulary words.
- Encoding efficiency.
- Language coverage.
Mainstream subword algorithms include:
- BPE (Byte Pair Encoding).
- WordPiece.
- Unigram Language Model.
Their core ideas are similar, but their vocabulary construction methods and optimization objectives differ.
7. BPE: Byte Pair Encoding
BPE (Byte Pair Encoding) was originally a data compression algorithm and was later widely used for subword vocabulary construction in NLP.
Its core idea is:
Starting from smaller basic units, repeatedly merge frequently occurring adjacent units in the training corpus to gradually build a subword vocabulary.
7.1 Basic Idea of BPE
Suppose the training corpus contains:
low
lower
lowest
In the initial stage, words can be split into characters:
l o w
l o w e r
l o w e s t
Count the frequencies of adjacent character pairs.
For example:
l o
o w
w e
e r
e s
s t
If l o is the currently most frequent character pair, merge it into a new unit:
lo
Then continue counting and merging.
After multiple rounds of merging, the following may be formed:
low
lower
lowest
The final vocabulary will contain multiple levels, from basic units to high-frequency subwords.
7.2 BPE Training Process
flowchart TB
A["Prepare training corpus"] --> B["Split into basic units"]
B --> C["Count frequencies of adjacent units"]
C --> D["Select the most frequent adjacent units"]
D --> E["Perform merge"]
E --> F["Update vocabulary and corpus representation"]
F --> G{"Has the number of merges been reached?"}
G -->|"No"| C
G -->|"Yes"| H["Generate BPE vocabulary and merge rules"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
7.3 The BPE Encoding Process
After training is complete, the Tokenizer saves:
- The vocabulary.
- The merge rules.
- The relevant preprocessing configuration.
When encoding new text, the input is split into the corresponding subwords according to the learned merge rules.
For example:
unhappiness
It may be segmented into:
un + happiness
It may also be split further.
The actual segmentation is determined by the specific vocabulary and merge rules.
7.4 Byte-level BPE
Traditional character-level BPE may encounter the problem of out-of-vocabulary characters.
Byte-level BPE, by contrast, starts from UTF-8 byte sequences.
UTF-8 encodes Unicode text as one or more bytes.
For example:
A
corresponds to one byte:
0x41
Chinese characters usually correspond to multiple UTF-8 bytes.
Byte-level BPE can build its basic units on a set of bytes and then form larger tokens through merge rules.
Its advantages are:
- The base byte set is limited.
- It can cover any valid UTF-8 text.
- It is less likely to produce unknown tokens in the traditional sense because of new characters.
- It can handle different languages and symbols uniformly.
However, byte-level processing does not mean that every UTF-8 byte will ultimately become a separate token.
7.5 Characteristics of BPE
Advantages:
- The algorithm is intuitive.
- The vocabulary-building process is relatively easy to understand.
- It can reuse high-frequency subwords.
- It is suitable for multilingual and open-vocabulary scenarios.
Limitations:
- Merge frequency is not necessarily equivalent to semantic plausibility.
- Vocabulary size affects the encoding result.
- Specific implementations may involve different pre-tokenization and byte-processing rules.
8. WordPiece: A Segmentation Method Based on Subword Scores
WordPiece is another common subword algorithm and was once widely used in models such as BERT.
Its main difference from BPE lies in the selection criterion used when building the vocabulary.
8.1 Core Idea of WordPiece
WordPiece evaluates the plausibility of subword combinations by statistically analyzing word-combination relationships in language data.
Unlike simply choosing the most frequent adjacent character pair, WordPiece's classic vocabulary-learning approach selects candidate units to add to the vocabulary based on combination scores.
A common intuitive score formula is:
Where:
- P(ab): the statistical probability of the combination ab.
- P(a): the statistical probability of subword a.
- P(b): the statistical probability of subword b.
This formula illustrates the idea of combination association strength; actual training implementations may use more complex estimation and optimization strategies.
8.2 WordPiece Encoding Method
WordPiece usually uses a greedy longest-match strategy.
Suppose the vocabulary contains:
play
##ing
##er
Input:
playing
It may be segmented into:
play + ##ing
Where:
playis a word-initial subword.##ingindicates that this subword follows another subword.
## is a common form of subword boundary marker; the specific representation depends on the Tokenizer implementation.
If a fragment cannot be matched against the vocabulary, some WordPiece implementations map the entire pre-tokenized fragment to [UNK].
This differs from Byte-level BPE, which can cover arbitrary input through byte-level base units.
8.3 Characteristics of WordPiece
- Builds the vocabulary through subword combination statistics.
- Encoding often uses a longest-match strategy.
- Can handle morphological variation and some out-of-vocabulary words.
- Exact behavior depends on the vocabulary and preprocessing rules.
9. Unigram Language Model: A Probability-Based Subword Model
The Unigram Language Model is a subword modeling approach different from BPE and WordPiece.
Its core idea is:
Start from a relatively large candidate vocabulary, evaluate the contribution of candidate subwords through a probabilistic model, and progressively remove unimportant subwords.
9.1 Basic Idea of Unigram
Suppose the candidate vocabulary contains:
a
an
and
apple
app
le
The same piece of text may have multiple possible segmentations.
For example:
apple
There can be different candidate segmentation paths.
The Unigram model assigns probabilities to subwords and evaluates different segmentations based on the probability of the entire sequence.
For a segmentation sequence:
Under a simplified independence assumption:
The segmentation path with higher probability is usually chosen.
9.2 Vocabulary Pruning
Unigram training usually starts from a larger candidate vocabulary.
The main process:
- Initialize the candidate subword vocabulary.
- Estimate probabilities for candidate subwords.
- Compute the contribution of different subwords to the corpus encoding objective.
- Remove candidate subwords with smaller contributions.
- Re-estimate probabilities.
- Repeat until the target vocabulary size is reached.
9.3 Differences Between Unigram and BPE
| Comparison Dimension | BPE | Unigram |
|---|---|---|
| Initial vocabulary | Base units | Large candidate vocabulary |
| Core operation | Merge adjacent units | Remove candidate subwords |
| Optimization approach | High-frequency merging | Probabilistic model |
| Segmentation method | Based on merge rules | Select paths by probability |
| Typical characteristic | Intuitive merge process | Can express multiple segmentation possibilities |
10. SentencePiece: A Unified Tokenizer Implementation Framework
SentencePiece often appears alongside BPE and Unigram, but note:
SentencePiece is not an algorithm at exactly the same level as BPE, WordPiece, and Unigram.
It is a tool library for text tokenization and subword modeling that supports models such as BPE and Unigram.
10.1 Why Is SentencePiece Needed?
Traditional NLP pipelines usually rely on language-specific pre-tokenization tools.
For example:
- English is split using spaces and rules.
- Chinese relies on Chinese word segmentation tools.
- Japanese relies on Japanese word segmentation tools.
This approach introduces language-specific preprocessing dependencies.
SentencePiece aims to learn subword models directly from raw text as much as possible, reducing reliance on external word segmentation tools.
10.2 Characteristics of SentencePiece
- Can process raw text directly.
- Supports models such as BPE and Unigram.
- Supports multilingual text.
- Supports configurations such as character coverage.
- Can encode spaces as visible special marker forms.
For example, in common SentencePiece representations, a space may appear as:
▁Hello
▁world
Here, ▁ is used to represent a space boundary.
It is not an ordinary character in the raw text but a representational convention of SentencePiece.
10.3 Hierarchy of Algorithms and Tools
| Name | Positioning | Main Role |
|---|---|---|
| BPE | Subword algorithm | Builds vocabulary through merging |
| WordPiece | Subword algorithm | Builds vocabulary based on combination scores |
| Unigram | Subword probabilistic model | Learns segmentation through probability and vocabulary pruning |
| SentencePiece | Tool and implementation framework | Supports subword model training and encoding |
11. Comparison of Different Tokenizer Algorithms
The three mainstream subword algorithms have similar design goals but different training strategies.
| Comparison dimension | BPE | WordPiece | Unigram |
|---|---|---|---|
| Core mechanism | Frequency merging | Combination scoring | Probabilistic modeling |
| Vocabulary construction | Incremental addition | Incremental selection | Incremental pruning |
| Encoding method | Merge rules | Usually longest-match | Probabilistically optimal or near-optimal segmentation |
| Typical applications | Various implementations in the GPT series | The classic BERT implementation | The common SentencePiece model |
| Key characteristics | Intuitive process | Explicit subword matching | Can consider multiple segmentation paths |
Note that a specific model's Tokenizer often also includes configurations such as pre-tokenization, byte handling, special Tokens, normalization and post-processing.
Knowing only that BPE or WordPiece is used is not enough to fully reproduce a model's Tokenizer.
Twelve, Stage Four: Token ID Mapping
After subword segmentation is complete, the Tokenizer needs to map each Token to an integer ID.
12.1 Vocabulary
The Vocabulary stores the mapping relationship between Tokens and Token IDs.
For example:
| Token | Token ID |
|---|---|
| hello | 101 |
| world | 102 |
| machine | 103 |
| learning | 104 |
| ! | 105 |
The vocabulary can be understood as a mapping structure:
For example:
12.2 Token ID Mapping Process
Suppose the Tokenizer outputs:
["hello", "world", "!"]
Looking them up in the vocabulary:
hello → 101
world → 102
! → 105
The result is:
[101, 102, 105]
These are the discrete input indices that the model can use.
12.3 Vocabulary Size and the Embedding Matrix
Suppose:
- Vocabulary Size: V=50000.
- Embedding Dimension: d=768.
Embedding matrix:
Each Token ID can serve as a row index into the Embedding matrix.
flowchart TB
A["Token sequence"] --> B["Vocabulary lookup"]
B --> C["Token ID sequence"]
C --> D["Embedding parameter matrix"]
D --> E["Retrieve vector by ID"]
E --> F["Model input tensor"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note the following:
- Vocabulary Size determines the number of rows in the Embedding matrix.
- Embedding Dimension determines the dimensionality of each vector.
- A Token ID is not an Embedding vector.
- A Token ID does not encode any semantic ordering either.
Thirteen, Stage Five: Special Tokens
Ordinary Tokens are used to represent text content, while special Tokens are used to express model tasks, boundaries, control and structural information.
The design of special Tokens can vary greatly from one model to another.
13.1 Common Special Tokens
| Token | Common uses |
|---|---|
[UNK] |
Unknown token |
[PAD] |
Padding position |
[BOS] |
Beginning of sequence |
[EOS] |
End of sequence |
[CLS] |
Special position in classification tasks |
[SEP] |
Separates different text segments |
[MASK] |
Masked token |
[FIM_PREFIX] |
Prefix marker for fill-in-the-middle generation tasks |
[FIM_SUFFIX] |
Suffix marker for fill-in-the-middle generation tasks |
[FIM_MIDDLE] |
Middle marker for fill-in-the-middle generation tasks |
These names are common representations; this does not mean that all models include them, nor that their specific semantics are exactly the same across different models.
13.2 BOS and EOS
BOS (Beginning of Sequence) usually indicates the beginning of a sequence.
EOS (End of Sequence) usually indicates the end of a sequence.
For example:
[BOS] I like artificial intelligence [EOS]
They can help the model identify sequence boundaries.
However, not all models use separate BOS and EOS tokens.
13.3 PAD
PAD is used to pad sequences of different lengths to the same length.
For example:
Sequence A:
[101, 102, 103]
Sequence B:
[201, 202]
After padding:
[101, 102, 103]
[201, 202, PAD]
In batch training, PAD can make tensors have a uniform shape.
Usually, an Attention Mask is also needed to prevent the model from treating padding positions as valid content.
Whether PAD participates in loss calculation also needs to be set according to the specific training task.
13.4 MASK
MASK is commonly used in Masked Language Modeling.
For example:
Original text:
I like learning machine learning.
After masking:
I like learning [MASK] learning.
The model needs to predict the masked content based on context.
The classic BERT pretraining task uses this mechanism.
However, autoregressive GPT-style models usually do not use BERT-style [MASK] training methods.