Back to series
· 12 min read

How Tokenizers Work: From Text Segmentation to Token ID Encoding


1. What Is a Tokenizer?

A Tokenizer, often referred to as a word segmenter or tokenizer, is the component in a natural language processing system responsible for converting text into a Token sequence.

In large language models, the Tokenizer sits between the raw text and the neural network.

Humans input natural language:

Artificial intelligence is changing the way software is developed.

What the model actually receives is usually not the raw string, but a Token ID sequence processed by the Tokenizer:

Raw text
    ↓
Tokenizer
    ↓
Token sequence
    ↓
Token ID
    ↓
Embedding
    ↓
Transformer

For example:

Input:
I like learning artificial intelligence.

Token:
["I", "like", "learning", "artificial", "intelligence", "."]

Token ID:
[102, 356, 1287, 2048, 3096, 17]

The above is only illustrative; the actual segmentation result and IDs depend on the Tokenizer used by the specific model.

1.1 Core Responsibilities of a Tokenizer

A Tokenizer mainly performs the following tasks:

  1. Text normalization: Process spaces, character forms, and special symbols in the text according to specific rules.
  2. Text preprocessing: Identify text structure and prepare for subsequent segmentation.
  3. Token segmentation: Split text into discrete units defined by the model.
  4. Token ID mapping: Convert Tokens into integer indices in the vocabulary.
  5. Special Token handling: Identify or insert control markers required by the task.
  6. Text decoding: Restore a Token ID sequence to text.

A Tokenizer is not responsible for understanding the meaning of text, nor does it directly perform language reasoning. It is responsible for establishing the mapping between text and the model's input representation.

1.2 Differences Between Tokenizer, Token, and Token ID

These three concepts are easy to confuse.

Concept Meaning Example
Tokenizer Text encoding and decoding tool BPE Tokenizer
Token Discrete unit after segmentation artificial
Token ID Integer index corresponding to a Token 2048

It can be understood as follows:

  • The Tokenizer is the processing tool.
  • A Token is a discrete unit in the processing result.
  • A Token ID is the Token's number in the vocabulary.

A Token ID is only an index and has no inherent semantic ordering.

For example:

Token       Token ID
--------------------
Apple       102
Banana      356
Computer    1287

102 is not more important than 356, and the numerical difference between two IDs does not represent semantic distance either.

2. Why Do Large Language Models Need a Tokenizer?

Computers can store and process text, but neural networks usually need to convert inputs into numerical tensors.

The Tokenizer is responsible for converting text into discrete indices that the model can process.

2.1 Raw Text Cannot Be Used Directly as an Embedding Index

Suppose the user inputs:

I like machine learning.

The raw text is a string and cannot be used directly as a row index into the Embedding matrix.

The model first needs to convert it into Token IDs:

Raw text:
I like machine learning.

Token ID:
[102, 356, 1287, 2560, 17]

Then look it up through the Embedding matrix:

e_i=E[token\_id_i]

where:

  • E: the Embedding parameter matrix.
  • token\_id_i: the integer index of the i-th Token.
  • e_i: the corresponding Embedding vector.

Therefore, the Tokenizer is an important step before text enters the Embedding layer.

2.2 Why Not Split Directly by Character?

One of the simplest methods is to split text by character.

For example:

Artificial intelligence

["A", "r", "t", "i", "f", "i", "c", "i", "a", "l", " ", "i", "n", "t", "e", "l", "l", "i", "g", "e", "n", "c", "e"]

This approach can reduce the problem of unknown characters, but it has some limitations:

  • The sequence length may be longer.
  • A single character may not have complete semantics.
  • Language expression efficiency may be lower.
  • The model needs to learn word structure through more context combinations.

If we split by whole words, we encounter another problem:

  • Out-of-vocabulary words are difficult to handle.
  • Inflection produces a large number of vocabulary entries.
  • New words and technical terms require additional handling.
  • The vocabulary may become extremely large.

For this reason, modern large language models generally adopt subword or other fine-grained tokenization methods, trading off vocabulary size, sequence length, and expressive capacity.

2.3 Why Not Just Use Whole Words?

Suppose the vocabulary contains only:

play
playing
played
player

Then these word forms may need to be stored separately.

If subword segmentation is used, they can be split into more basic units:

play
playing → play + ing
played  → play + ed
player  → play + er

This reduces vocabulary duplication and improves the ability to handle new words and inflections.

The actual segmentation result is determined by the vocabulary and algorithm of the specific tokenizer, and is not necessarily identical to the example above.

3. The Complete Tokenizer Workflow

A tokenizer does not simply split text on spaces.

A typical text encoding pipeline may involve multiple stages.

3.1 Overall Processing Pipeline

flowchart TB
    A["Raw Text"] --> B["Text Normalization"]
    B --> C["Pre-tokenization"]
    C --> D["Subword Segmentation"]
    D --> E["Token Sequence"]
    E --> F["Vocabulary Lookup"]
    F --> G["Token ID Sequence"]
    G --> H["Add Special Tokens"]
    H --> I["Model Input"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

The internal implementations of different tokenizers are not entirely the same.

Some use regular expressions for pre-tokenization, some operate at the byte level, some perform Unicode normalization, and others convert text directly into byte sequences before subword segmentation.

The pipeline above is therefore a conceptual model; it does not mean that all tokenizers perform every step in exactly the same order.

4. Stage One: Text Normalization

Text normalization is the process of applying necessary standardization to the input text.

The same character may have different Unicode representations, and text may also contain different kinds of spaces, line breaks, and special characters.

4.1 Unicode Character Representation

Unicode defines a unified character encoding system for different languages and symbols.

For example:

  • Chinese characters.
  • English letters.
  • Digits.
  • Punctuation marks.
  • Mathematical symbols.
  • Emoji.

Note, however, that a character as seen by a human does not necessarily correspond to a single Unicode code point.

For example, some emoji are composed of multiple code points.

Therefore, one cannot simply assume:

one visible character = one Unicode code point = one Token

These three concepts are not equivalent.

4.2 Unicode Normalization

Unicode provides several normalization forms:

Form Meaning
NFC Canonical decomposition followed by canonical composition
NFD Canonical decomposition
NFKC Compatibility decomposition followed by canonical composition
NFKD Compatibility decomposition

For example, some accented characters can be represented by different Unicode sequences.

Normalization can unify some of these equivalent representations.

However, whether Unicode normalization is performed, and which form is used, depends on the tokenizer's design.

One cannot assume that all models automatically apply NFKC or NFC.

4.3 Handling of Spaces and Line Breaks

Spaces can be significant in a tokenizer.

For example:

hello world

and:

helloworld

will yield different encoding results in many tokenizers.

Likewise:

Hello
 hello

A leading space can also affect token segmentation.

Some tokenizers encode space information into the token representation.

Therefore, spaces in the text must not be deleted arbitrarily, or the Token ID sequence may change.

4.4 Case Handling

Letter case in English text may affect the Tokenizer's encoding results.

For example:

apple
Apple
APPLE

They may correspond to different Tokens or Token combinations.

Whether to normalize case depends on the text processing strategy used during model training.

If the model uses a case-sensitive Tokenizer, arbitrarily converting text to lowercase may change the model input.

5. Stage 2: Pre-tokenization

Pre-tokenization divides the raw text into larger candidate segments, providing processing boundaries for the subsequent subword algorithm.

For example:

I love artificial intelligence.

It may be pre-tokenized as:

["I", "love", "artificial", "intelligence", "."]

But this does not mean that every pre-tokenization segment will ultimately correspond to one Token.

A single pre-tokenization segment may still be split into multiple subwords.

5.1 Why is pre-tokenization needed?

Suppose the input is:

I love machine learning.

The pre-tokenization stage first identifies:

I
love
machine
learning
.

Then the subword algorithm processes these segments separately.

This reduces unnecessary cross-boundary combinations and makes vocabulary construction and encoding easier to control.

5.2 Limitations of whitespace tokenization

Whitespace tokenization is fairly intuitive for English, but it is not suitable for languages such as Chinese.

For example:

我喜欢人工智能

Chinese text usually does not insert spaces between every word.

Therefore, the Tokenizer needs to adopt a segmentation strategy better suited to the target language.

5.3 Pre-tokens are not the same as final Tokens

It is necessary to distinguish clearly between:

  • Pre-token: a segment produced during the pre-tokenization stage.
  • Token: a discrete unit obtained after final encoding.

For example:

Pre-tokenization:
unbelievable

Subword tokenization:
un + believe + able

The actual result depends on the model vocabulary and algorithm.

6. Stage 3: Subword Tokenization

Subword tokenization is one of the core technologies in modern large language model Tokenizers.

It splits text into words, roots, affixes, or other reusable subword units.

The goal is to strike a balance among the following factors:

  • Vocabulary size.
  • Sequence length.
  • Ability to handle out-of-vocabulary words.
  • Encoding efficiency.
  • Language coverage.

Mainstream subword algorithms include:

  1. BPE (Byte Pair Encoding).
  2. WordPiece.
  3. Unigram Language Model.

Their core ideas are similar, but their vocabulary construction methods and optimization objectives differ.

7. BPE: Byte Pair Encoding

BPE (Byte Pair Encoding) was originally a data compression algorithm and was later widely used for subword vocabulary construction in NLP.

Its core idea is:

Starting from smaller basic units, repeatedly merge frequently occurring adjacent units in the training corpus to gradually build a subword vocabulary.

7.1 Basic Idea of BPE

Suppose the training corpus contains:

low
lower
lowest

In the initial stage, words can be split into characters:

l o w
l o w e r
l o w e s t

Count the frequencies of adjacent character pairs.

For example:

l o
o w
w e
e r
e s
s t

If l o is the currently most frequent character pair, merge it into a new unit:

lo

Then continue counting and merging.

After multiple rounds of merging, the following may be formed:

low
lower
lowest

The final vocabulary will contain multiple levels, from basic units to high-frequency subwords.

7.2 BPE Training Process

flowchart TB
    A["Prepare training corpus"] --> B["Split into basic units"]
    B --> C["Count frequencies of adjacent units"]
    C --> D["Select the most frequent adjacent units"]
    D --> E["Perform merge"]
    E --> F["Update vocabulary and corpus representation"]
    F --> G{"Has the number of merges been reached?"}
    G -->|"No"| C
    G -->|"Yes"| H["Generate BPE vocabulary and merge rules"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

7.3 The BPE Encoding Process

After training is complete, the Tokenizer saves:

  • The vocabulary.
  • The merge rules.
  • The relevant preprocessing configuration.

When encoding new text, the input is split into the corresponding subwords according to the learned merge rules.

For example:

unhappiness

It may be segmented into:

un + happiness

It may also be split further.

The actual segmentation is determined by the specific vocabulary and merge rules.

7.4 Byte-level BPE

Traditional character-level BPE may encounter the problem of out-of-vocabulary characters.

Byte-level BPE, by contrast, starts from UTF-8 byte sequences.

UTF-8 encodes Unicode text as one or more bytes.

For example:

A

corresponds to one byte:

0x41

Chinese characters usually correspond to multiple UTF-8 bytes.

Byte-level BPE can build its basic units on a set of bytes and then form larger tokens through merge rules.

Its advantages are:

  • The base byte set is limited.
  • It can cover any valid UTF-8 text.
  • It is less likely to produce unknown tokens in the traditional sense because of new characters.
  • It can handle different languages and symbols uniformly.

However, byte-level processing does not mean that every UTF-8 byte will ultimately become a separate token.

7.5 Characteristics of BPE

Advantages:

  • The algorithm is intuitive.
  • The vocabulary-building process is relatively easy to understand.
  • It can reuse high-frequency subwords.
  • It is suitable for multilingual and open-vocabulary scenarios.

Limitations:

  • Merge frequency is not necessarily equivalent to semantic plausibility.
  • Vocabulary size affects the encoding result.
  • Specific implementations may involve different pre-tokenization and byte-processing rules.

8. WordPiece: A Segmentation Method Based on Subword Scores

WordPiece is another common subword algorithm and was once widely used in models such as BERT.

Its main difference from BPE lies in the selection criterion used when building the vocabulary.

8.1 Core Idea of WordPiece

WordPiece evaluates the plausibility of subword combinations by statistically analyzing word-combination relationships in language data.

Unlike simply choosing the most frequent adjacent character pair, WordPiece's classic vocabulary-learning approach selects candidate units to add to the vocabulary based on combination scores.

A common intuitive score formula is:

Score(a,b)= \frac{P(ab)}{P(a)P(b)}

Where:

  • P(ab): the statistical probability of the combination ab.
  • P(a): the statistical probability of subword a.
  • P(b): the statistical probability of subword b.

This formula illustrates the idea of combination association strength; actual training implementations may use more complex estimation and optimization strategies.

8.2 WordPiece Encoding Method

WordPiece usually uses a greedy longest-match strategy.

Suppose the vocabulary contains:

play
##ing
##er

Input:

playing

It may be segmented into:

play + ##ing

Where:

  • play is a word-initial subword.
  • ##ing indicates that this subword follows another subword.

## is a common form of subword boundary marker; the specific representation depends on the Tokenizer implementation.

If a fragment cannot be matched against the vocabulary, some WordPiece implementations map the entire pre-tokenized fragment to [UNK].

This differs from Byte-level BPE, which can cover arbitrary input through byte-level base units.

8.3 Characteristics of WordPiece

  • Builds the vocabulary through subword combination statistics.
  • Encoding often uses a longest-match strategy.
  • Can handle morphological variation and some out-of-vocabulary words.
  • Exact behavior depends on the vocabulary and preprocessing rules.

9. Unigram Language Model: A Probability-Based Subword Model

The Unigram Language Model is a subword modeling approach different from BPE and WordPiece.

Its core idea is:

Start from a relatively large candidate vocabulary, evaluate the contribution of candidate subwords through a probabilistic model, and progressively remove unimportant subwords.

9.1 Basic Idea of Unigram

Suppose the candidate vocabulary contains:

a
an
and
apple
app
le

The same piece of text may have multiple possible segmentations.

For example:

apple

There can be different candidate segmentation paths.

The Unigram model assigns probabilities to subwords and evaluates different segmentations based on the probability of the entire sequence.

For a segmentation sequence:

X=(x_1,x_2,\ldots,x_n)

Under a simplified independence assumption:

P(X)=\prod_{i=1}^{n}P(x_i)

The segmentation path with higher probability is usually chosen.

9.2 Vocabulary Pruning

Unigram training usually starts from a larger candidate vocabulary.

The main process:

  1. Initialize the candidate subword vocabulary.
  2. Estimate probabilities for candidate subwords.
  3. Compute the contribution of different subwords to the corpus encoding objective.
  4. Remove candidate subwords with smaller contributions.
  5. Re-estimate probabilities.
  6. Repeat until the target vocabulary size is reached.

9.3 Differences Between Unigram and BPE

Comparison Dimension BPE Unigram
Initial vocabulary Base units Large candidate vocabulary
Core operation Merge adjacent units Remove candidate subwords
Optimization approach High-frequency merging Probabilistic model
Segmentation method Based on merge rules Select paths by probability
Typical characteristic Intuitive merge process Can express multiple segmentation possibilities

10. SentencePiece: A Unified Tokenizer Implementation Framework

SentencePiece often appears alongside BPE and Unigram, but note:

SentencePiece is not an algorithm at exactly the same level as BPE, WordPiece, and Unigram.

It is a tool library for text tokenization and subword modeling that supports models such as BPE and Unigram.

10.1 Why Is SentencePiece Needed?

Traditional NLP pipelines usually rely on language-specific pre-tokenization tools.

For example:

  • English is split using spaces and rules.
  • Chinese relies on Chinese word segmentation tools.
  • Japanese relies on Japanese word segmentation tools.

This approach introduces language-specific preprocessing dependencies.

SentencePiece aims to learn subword models directly from raw text as much as possible, reducing reliance on external word segmentation tools.

10.2 Characteristics of SentencePiece

  • Can process raw text directly.
  • Supports models such as BPE and Unigram.
  • Supports multilingual text.
  • Supports configurations such as character coverage.
  • Can encode spaces as visible special marker forms.

For example, in common SentencePiece representations, a space may appear as:

▁Hello
▁world

Here, ▁ is used to represent a space boundary.

It is not an ordinary character in the raw text but a representational convention of SentencePiece.

10.3 Hierarchy of Algorithms and Tools

Name Positioning Main Role
BPE Subword algorithm Builds vocabulary through merging
WordPiece Subword algorithm Builds vocabulary based on combination scores
Unigram Subword probabilistic model Learns segmentation through probability and vocabulary pruning
SentencePiece Tool and implementation framework Supports subword model training and encoding

11. Comparison of Different Tokenizer Algorithms

The three mainstream subword algorithms have similar design goals but different training strategies.

Comparison dimension BPE WordPiece Unigram
Core mechanism Frequency merging Combination scoring Probabilistic modeling
Vocabulary construction Incremental addition Incremental selection Incremental pruning
Encoding method Merge rules Usually longest-match Probabilistically optimal or near-optimal segmentation
Typical applications Various implementations in the GPT series The classic BERT implementation The common SentencePiece model
Key characteristics Intuitive process Explicit subword matching Can consider multiple segmentation paths

Note that a specific model's Tokenizer often also includes configurations such as pre-tokenization, byte handling, special Tokens, normalization and post-processing.

Knowing only that BPE or WordPiece is used is not enough to fully reproduce a model's Tokenizer.

Twelve, Stage Four: Token ID Mapping

After subword segmentation is complete, the Tokenizer needs to map each Token to an integer ID.

12.1 Vocabulary

The Vocabulary stores the mapping relationship between Tokens and Token IDs.

For example:

Token Token ID
hello 101
world 102
machine 103
learning 104
! 105

The vocabulary can be understood as a mapping structure:

Vocabulary: Token\rightarrow ID

For example:

Vocabulary("hello")=101

12.2 Token ID Mapping Process

Suppose the Tokenizer outputs:

["hello", "world", "!"]

Looking them up in the vocabulary:

hello → 101
world → 102
!     → 105

The result is:

[101, 102, 105]

These are the discrete input indices that the model can use.

12.3 Vocabulary Size and the Embedding Matrix

Suppose:

  • Vocabulary Size: V=50000.
  • Embedding Dimension: d=768.

Embedding matrix:

E\in\mathbb{R}^{50000\times768}

Each Token ID can serve as a row index into the Embedding matrix.

flowchart TB
    A["Token sequence"] --> B["Vocabulary lookup"]
    B --> C["Token ID sequence"]
    C --> D["Embedding parameter matrix"]
    D --> E["Retrieve vector by ID"]
    E --> F["Model input tensor"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Note the following:

  • Vocabulary Size determines the number of rows in the Embedding matrix.
  • Embedding Dimension determines the dimensionality of each vector.
  • A Token ID is not an Embedding vector.
  • A Token ID does not encode any semantic ordering either.

Thirteen, Stage Five: Special Tokens

Ordinary Tokens are used to represent text content, while special Tokens are used to express model tasks, boundaries, control and structural information.

The design of special Tokens can vary greatly from one model to another.

13.1 Common Special Tokens

Token Common uses
[UNK] Unknown token
[PAD] Padding position
[BOS] Beginning of sequence
[EOS] End of sequence
[CLS] Special position in classification tasks
[SEP] Separates different text segments
[MASK] Masked token
[FIM_PREFIX] Prefix marker for fill-in-the-middle generation tasks
[FIM_SUFFIX] Suffix marker for fill-in-the-middle generation tasks
[FIM_MIDDLE] Middle marker for fill-in-the-middle generation tasks

These names are common representations; this does not mean that all models include them, nor that their specific semantics are exactly the same across different models.

13.2 BOS and EOS

BOS (Beginning of Sequence) usually indicates the beginning of a sequence.

EOS (End of Sequence) usually indicates the end of a sequence.

For example:

[BOS] I like artificial intelligence [EOS]

They can help the model identify sequence boundaries.

However, not all models use separate BOS and EOS tokens.

13.3 PAD

PAD is used to pad sequences of different lengths to the same length.

For example:

Sequence A:
[101, 102, 103]

Sequence B:
[201, 202]

After padding:
[101, 102, 103]
[201, 202, PAD]

In batch training, PAD can make tensors have a uniform shape.

Usually, an Attention Mask is also needed to prevent the model from treating padding positions as valid content.

Whether PAD participates in loss calculation also needs to be set according to the specific training task.

13.4 MASK

MASK is commonly used in Masked Language Modeling.

For example:

Original text:
I like learning machine learning.

After masking:
I like learning [MASK] learning.

The model needs to predict the masked content based on context.

The classic BERT pretraining task uses this mechanism.

However, autoregressive GPT-style models usually do not use BERT-style [MASK] training methods.

13.5 Structural Tokens of Chat Models