Back to series
· 10 min read

What Is a Token? Tokenization and Encoding Mechanisms


1. What Is a Token?

A token is the basic unit used by large language models when processing text.

Large language models do not directly read the text that humans see. Instead, they first use a tokenizer to split the text into a sequence of tokens, then convert each token into its corresponding integer ID, and finally feed it into the neural network for computation.

A token can be:

  • A complete English word.
  • Part of an English word.
  • A Chinese character.
  • A text fragment made up of multiple Chinese characters.
  • A punctuation mark.
  • A number or special symbol.
  • A special control token.

For example:

Artificial intelligence is changing the world.

might be tokenized as:

["Artificial", " intelligence", " is", " changing", " the", " world", "."]

Chinese text:

人工智能正在改变世界。

might be tokenized as:

["人工", "智能", "正在", "改变", "世界", "。"]

The above is only illustrative; the actual tokenization result depends on the tokenizer and vocabulary used by the model.

A token is not equal to a Chinese character, nor is it equal to a word.

It is a discrete text unit defined in the model's vocabulary.

2. Why Do Large Language Models Need Tokens?

Computers can store text, but neural networks cannot directly perform matrix operations according to the semantics of human language.

The core computations of neural networks are built on numbers, vectors, and matrices.

Therefore, natural language needs to be converted into a numerical representation that the model can process.

The whole process can be divided into three stages:

  1. Tokenization: split text into tokens.
  2. Encoding: map tokens to integer IDs.
  3. Vectorization: map token IDs to continuous vectors.

Overall flow:

flowchart TB
    A["Original text"] --> B["Tokenizer segmentation"]
    B --> C["Token sequence"]
    C --> D["Token ID encoding"]
    D --> E["Embedding vector mapping"]
    E --> F["Transformer neural network"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Three concepts need to be distinguished here:

Concept Meaning Data format
Token Basic unit after text segmentation Text fragment
Token ID Index of the token in the vocabulary Integer
Embedding Vector representation corresponding to the token ID Floating-point vector

For example:

Original text: 人工智能

Token:
["人工", "智能"]

Token ID:
[1024, 2056]

Embedding:
[
  [0.12, -0.31, 0.45, ...],
  [0.28,  0.16, 0.73, ...]
]

The IDs and vectors above are illustrative data.

Tokenization addresses how text is split; Token ID addresses how text is numbered; and Embedding addresses how the model performs computation through vectors.

3. What Is a Tokenizer?

A tokenizer is the component responsible for converting between text and token sequences.

It usually includes:

  • Preprocessing rules.
  • Tokenization algorithm.
  • Vocabulary.
  • Encoding logic.
  • Decoding logic.
  • Special token handling rules.

A tokenizer mainly handles conversion in two directions.

3.1 Encoding: Text Encoding

Encoding is the process of converting raw text into a sequence of token IDs.

For example:

Input:
I love artificial intelligence.

Suppose tokenization produces:

["I", " love", " artificial", " intelligence", "."] ```

Then map them according to the vocabulary:

```text [40, 1024, 2056, 3108, 13] ```

The actual processing flow:

```mermaid flowchart TB A["Original text"] --> B["Text preprocessing"] B --> C["Tokenization algorithm"] C --> D["Generate tokens"] D --> E["Look up vocabulary"] E --> F["Generate Token ID sequence"] style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F style B fill:#EAF3EC,stroke:#83A995,color:#315D49 style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style E fill:#EAF3EC,stroke:#83A995,color:#315D49 style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F ```

3.2 Decoding: Text Decoding

Decoding is the process of converting a Token ID sequence back into text.

For example:

```text Token ID: [40, 1024, 2056, 3108, 13] ```

Looking them up in the vocabulary gives:

```text ["I", " love", " artificial", " intelligence", "."] ```

Then, following the Tokenizer's decoding rules, they are merged and restored to:

```text I love artificial intelligence. ```

Note that the decoding process is not necessarily simple string concatenation.

Different Tokenizers may involve rules such as space restoration, byte decoding, and special Token handling.

3.3 The Relationship Between the Tokenizer and the Model

A Tokenizer and a language model are usually used as a matched pair.

A model is trained using a specific vocabulary and Token ID encoding scheme, and inference likewise requires a compatible Tokenizer.

If a mismatched Tokenizer is used, errors may occur in input encoding and output decoding even if the model weights are correct.

For example:

  • The vocabulary meanings corresponding to Token IDs are inconsistent.
  • The IDs of special Tokens are inconsistent.
  • Dialogue control markers are handled inconsistently.
  • The text decoding result is abnormal.

Therefore, the Tokenizer is not a peripheral tool that can be replaced at will; it is an important component of the model's input-output system.

IV. How Are Tokens Segmented?

The core question of Tokenization is:

According to what rules should a continuous piece of text be segmented into Tokens?

The simplest approach is to segment by character, word, or space, but these methods struggle to meet the needs of multilingual text, large-scale vocabularies, and unknown-word handling.

Currently, mainstream large language models generally use subword-level segmentation methods.

4.1 Character-Level Tokenization

Character-level tokenization splits text into individual characters.

For example:

```text 人工智能正在发展 ```

Segmentation result:

```text ["人", "工", "智", "能", "正", "在", "发", "展"] ```

Advantages:

  • The vocabulary size is relatively small.
  • Unknown characters are unlikely to occur.
  • Any sequence of characters can be composed.

Disadvantages:

  • The sequence length is relatively long.
  • A complete word may be split into multiple characters.
  • The model needs to learn the relationships between words through more positions.

4.2 Word-Level Tokenization

Word-level tokenization segments text by complete words.

For example:

```text 人工智能正在发展 ```

Segmentation result:

```text ["人工智能", "正在", "发展"] ```

Advantages:

  • Relatively complete word units are preserved.
  • The sequence length is relatively short.
  • It is fairly intuitive for common words.

Disadvantages:

  • The vocabulary size may be very large.
  • New words and rare words are difficult to handle.
  • Out-of-vocabulary words require additional handling.
  • There are many vocabulary composition issues for multilingual and morphologically complex languages.

4.3 Subword-Level Tokenization

Subword-level tokenization splits text into a combination of complete words and word fragments.

For example:

```text unbelievable ```

It may be split into:

```text ["un", "believ", "able"]

This preserves common word fragments and can also handle words that do not appear directly in the vocabulary by combining them.

Subword tokenization strikes a good balance between vocabulary size and sequence length, so it is widely used in modern large language models.

4.4 Comparison of Three Tokenization Methods

Tokenization method Segmentation unit Vocabulary size Sequence length Unknown word handling
Character-level Single character Small Long Usually easier
Word-level Complete words Large Short Relatively difficult
Subword-level Words or word fragments Medium Medium Usually more flexible

Modern tokenizers do not necessarily segment strictly according to word boundaries in the linguistic sense; instead, they determine tokens based on the learned vocabulary and algorithm rules.

5. Mainstream Tokenizer Algorithms

Common subword tokenization algorithms include:

  • BPE (Byte Pair Encoding).
  • WordPiece.
  • Unigram Language Model.
  • SentencePiece.

Their common goal is to build a finite-size vocabulary while representing various input texts as efficiently as possible.

5.1 BPE: Byte Pair Encoding

BPE is a subword construction algorithm based on frequency statistics and iterative merging.

It originated in the field of data compression and was later widely used in natural language processing.

Its basic idea is:

Start with smaller basic symbols and repeatedly merge the most common adjacent symbols in the corpus to gradually build a subword vocabulary.

Basic Process of BPE

Suppose the training corpus contains the following words:

low
lower
lowest
newer

Initially, the words can be split into characters:

l o w
l o w e r
l o w e s t
n e w e r

Then count the frequencies of adjacent symbol combinations.

For example:

l + o
o + w
w + e
e + r

If l + o has the highest frequency, merge them into a new symbol:

lo

After that, continue counting and merging.

After multiple rounds of iteration, the following may gradually form:

lo
low
er
est

Eventually, a vocabulary containing subword fragments of different lengths is obtained.

BPE Processing Flow

flowchart TB
    A["Prepare training corpus"] --> B["Split into basic symbols"]
    B --> C["Count adjacent symbol frequencies"]
    C --> D["Select highest-frequency combination"]
    D --> E["Merge adjacent symbols"]
    E --> F{"Has the vocabulary size been reached?"}
    F -->|"No"| C
    F -->|"Yes"| G["Obtain BPE vocabulary"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Note that the above is a character-level BPE illustration for ease of understanding.

Actual models may use byte-level BPE, which first converts text into a byte representation and then performs subword merging. This allows more flexible handling of different languages, special characters, and unknown characters.

Advantages of BPE

  • The vocabulary size is relatively controllable.
  • High-frequency words can form complete tokens.
  • Low-frequency words can be split into subwords.
  • It can handle a large number of unseen word combinations.
  • It is suitable for text scenarios such as multilingual content and code.

Limitations of BPE

  • High-frequency combinations affect vocabulary construction results.
  • It does not necessarily align with semantic boundaries in natural language.
  • Different corpora may produce different vocabularies.
  • Token efficiency may vary across different languages.

5.2 WordPiece

WordPiece is also a subword tokenization algorithm, commonly used in the BERT family of models.

Its core difference from BPE lies in the merging strategy.

BPE usually selects merge candidates based on the frequency of adjacent symbols, whereas WordPiece selects them based on how much the candidate combinations improve the language modeling objective or vocabulary score.

In simple terms:

  • BPE: focuses more on the frequency of adjacent symbol combinations.
  • WordPiece: focuses more on the contribution of a merge to the model's language modeling performance.

WordPiece often uses a special prefix to mark subwords that are not at the start of a word.

For example:

playing

It may be represented as:

["play", "##ing"]

Here, ##ing indicates that the fragment is not the beginning of a new word.

The actual segmentation depends on the specific vocabulary.

5.3 Unigram Language Model

Unigram is another subword modeling method.

Unlike BPE, which continuously merges from basic symbols, Unigram usually starts with a larger candidate subword vocabulary, evaluates the value of different subwords through a probabilistic model, and then gradually removes candidates with smaller contributions.

Its basic idea is:

  1. Build an initial candidate vocabulary.
  2. Estimate probabilities for candidate subwords.
  3. Calculate the impact of removing certain subwords on the overall corpus probability.
  4. Remove subwords with smaller impact.
  5. Repeat the iteration until the target vocabulary size is reached.

It usually uses a probabilistic model to handle multiple possible segmentations of the same text.

5.4 SentencePiece

SentencePiece is a language-independent subword tokenization tool and framework that can support algorithms such as BPE and Unigram.

Its important characteristics are:

  • Directly processes raw text.
  • Does not force reliance on spaces as word boundaries.
  • Supports multilingual text.
  • Can handle different writing systems in a unified way.
  • Supports subword training and encoding.

SentencePiece is commonly used in multilingual models and open-source large language models.

It is necessary to distinguish:

BPE, WordPiece, and Unigram are tokenization algorithms or modeling methods; SentencePiece is a tool framework that implements these tokenization methods.

They are not concepts entirely at the same classification level.

5.5 Comparison of Mainstream Algorithms

Algorithm Core Mechanism Vocabulary Construction Method Representative Applications
BPE Merging high-frequency adjacent symbols Stepwise merging Some Tokenizers in the GPT series
WordPiece Selecting subwords based on combination scores Stepwise expansion BERT
Unigram Filtering subwords based on a probabilistic model Stepwise pruning SentencePiece Unigram
SentencePiece Unified text processing and subword modeling Supports BPE and Unigram Multilingual models, etc.

It should be noted that the Tokenizer of a model family may change with versions, and its specific algorithm and configuration cannot be determined solely from the model name.

6. Vocabulary and Token ID

After a Tokenizer splits text into Tokens, it also needs to assign each Token a unique integer number.

This mapping is maintained by the vocabulary (Vocabulary).

6.1 What Is a Vocabulary?

The vocabulary is a set of mappings between Tokens and Token IDs.

For example:

Token Token ID
the 1
a 2
computer 3
artificial 4
intelligence 5
. 6

Actual model vocabularies usually contain tens of thousands or even more Tokens.

Vocabulary size is usually denoted as:

V=|Vocabulary|

where V represents the number of Tokens in the vocabulary.

6.2 What Is a Token ID?

A Token ID is the integer index of a Token in the vocabulary.

For example:

Text:
the computer

After passing through the Tokenizer:

Token:
["the", " computer"]

It is mapped to:

Token ID:
[1, 3]

It should be noted:

A Token ID is merely an index and does not indicate semantic size.

For example:

ID(\text{apple})=100
ID(\text{banana})=200

Therefore, one cannot conclude that banana is semantically greater than apple, nor that their semantic distance is 100.

The numerical value of a Token ID is usually just a number assigned during vocabulary construction.

6.3 Vocabulary Size and the Model Output Layer

Vocabulary size not only affects the Tokenizer but also directly affects the model's input and output structure.

Assume:

  • The vocabulary size is V.
  • The model hidden dimension is d.

An embedding matrix can usually be represented as:

E\in\mathbb{R}^{V\times d}

For autoregressive language models, the output projection layer usually needs to map hidden states to the vocabulary dimension:

W_{vocab}\in\mathbb{R}^{d\times V}

Thus, the model can compute a logit for every token in the vocabulary.

As the vocabulary size grows, it affects:

  • The number of embedding parameters.
  • The number of output projection layer parameters.
  • The dimensionality of the output logits.
  • The Softmax computation cost.
  • The model's training and inference resource requirements.

Some models use weight tying between the input embedding and output projection weights to reduce the number of parameters.

7. How Are Token IDs Converted to Embeddings?

A token ID is just an integer. When the model actually performs neural network computation, it usually needs to convert it into a continuous vector.

This process is handled by the embedding layer.

7.1 Embedding Matrix

Suppose the vocabulary size is:

V=50000

The hidden dimension is:

d=4096

Then the shape of the embedding matrix is:

E\in\mathbb{R}^{50000\times4096}

Each row corresponds to the vector representation of one token.

When the input token ID is 1024, the model retrieves row 1024:

e=E[1024]

This yields a 4096-dimensional vector.

7.2 Tensor Shapes for Batched Input

Suppose a single input contains:

  • Batch size: B.
  • Sequence length: T.
  • Hidden size: d.

The shape of the token ID tensor output by the tokenizer is usually:

[B,T]

After embedding, the tensor shape is:

[B,T,d]

For example:

B=2,\quad T=128,\quad d=4096

Then:

Token ID:
[2, 128]

Embedding:
[2, 128, 4096]

This means the model processes 2 samples at a time, each containing 128 tokens, with each token corresponding to a 4096-dimensional vector.

7.3 Embeddings and Positional Encoding

Token embeddings mainly provide the initial representation of tokens, but the model also needs to handle positional information.

In the traditional Transformer, a common approach is to add the token embedding and positional encoding:

X=E_{token}+E_{position}

In models that use RoPE, positional relationships are usually injected by applying a rotary transformation to the Query and Key in attention, rather than simply adding a position vector to the token embedding.

Different models implement this in different ways.

8. Special Tokens: How Does the Model Recognize Control Information?

In addition to ordinary text tokens, tokenizers usually define a number of special tokens.

Special tokens are used to represent text boundaries, task types, conversation structure, or control information.

Common types include:

Special Token Type Purpose
BOS Indicates the beginning of a sequence or message
EOS Indicates the end of a sequence
PAD Used to pad sequences of different lengths
UNK Represents an unknown token that cannot be recognized or mapped
SEP Separates different text segments or sequences
MASK Used for masked language modeling
Role Token Identifies the role of a message
End-of-Turn Indicates the end of a conversation turn
Tool Control Token Indicates a tool call or control flow

Different models do not necessarily use all of these tokens, and they may use different names and IDs.

8.1 BOS and EOS

BOS (Beginning of Sequence) usually indicates the start of a sequence.

EOS (End of Sequence) usually indicates the end of a sequence.

For example:

[BOS] Artificial intelligence is advancing [EOS]

The model can use these markers to identify the boundaries of the input sequence.

8.2 PAD

When batching texts of different lengths, sequences need to be padded to the same length.

For example:

Sequence A:
[101, 102, 103]

Sequence B:
[201, 202]

After padding:

Sequence A:
[101, 102, 103]

Sequence B:
[201, 202, PAD]

PAD allows samples of different lengths to form tensors of a uniform shape.

In actual computation, an attention mask is usually needed so that the model does not treat padded positions as valid text.

8.3 MASK

MASK is commonly used in Masked Language Modeling.

For example:

Original text:
Machine learning is an important branch of artificial intelligence.

Training input:
Machine learning is an important branch of [MASK].

The model needs to predict the masked Token based on the context.

BERT is a typical representative of this type of training objective.

Note that MASK is not a conventional generation token used by all autoregressive generative models.

8.4 Dialogue Control Tokens

In addition to ordinary text, chat models also need to represent control information such as message roles, message boundaries, and tool calls.

For example, a simplified dialogue structure might be:

<|im_start|>user
Hello
<|im_end|>
<|im_start|>assistant
Hello, how can I help you?
<|im_end|>

In this type of structure:

  • The message start token is used to mark a message.
  • Role information is used to distinguish between user, assistant, and tool.
  • The message end token is used to define message boundaries.
  • Some control tokens are used to indicate the end of an assistant turn or a handoff to a tool.

The formats used by actual models are not uniform.