Back to series
· 9 min read

What Is Next Token Prediction? How Do Large Language Models Learn to Predict the Next Token?


1. What Is Next Token Prediction?

Next Token Prediction, abbreviated as NTP, is commonly referred to in Chinese as next Token prediction.

It is one of the core training objectives of modern decoder-only large language models.

The core idea is very simple:

Given a sequence of tokens that has already appeared, let the model predict the next most likely token.

For example:

Input:
The weather today is very

Model predicts:
good

For another example:

Input:
The development of artificial intelligence is inseparable from

Model predicts:
data

The model does not directly learn the meaning of complete sentences, nor does it understand language through manually written rules; instead, through a large number of text samples, it repeatedly learns what the next token should be given the context.

It is important to note here: the unit the model predicts is a token, not necessarily a Chinese character, a word, or a complete phrase.

For example, after a sentence is processed by a Tokenizer, it may become:

Original text:
I like artificial intelligence.

Token sequence:
["I", "like", "artificial", "intelligence", "."]

It may also be split into other forms depending on the specific Tokenizer.

Therefore, Next Token Prediction is actually predicting the vocabulary element corresponding to a Token ID.

2. Why Can Predicting the Next Token Train a Large Language Model?

This is one of the most important questions for understanding large language models.

A natural question is:

If the model is only predicting the next token, why can it eventually answer questions, write code, translate, summarize, and even perform complex reasoning?

The answer lies in the fact that accurately predicting the next token requires the model to learn a large amount of complex information in the context.

2.1 Predicting the Next Token Is Not Just Memorizing Word Collocations

Suppose the model sees:

Newton proposed

It may predict:

the law of universal gravitation

This looks like it is memorizing fixed collocations.

But if the input becomes:

The classical mechanics theory proposed by Newton mainly studies

The content the model needs to predict will change.

It needs to use:

  • The semantics of the preceding text.
  • The relationships between words.
  • The grammatical structure of the sentence.
  • Relevant knowledge.
  • The communicative intent of the current context.

Let's look at another example:

Input:
If a number is divisible by 2, then it is an

The model may predict:

even number

For the prediction to be reasonable, the model needs to grasp certain mathematical concepts and patterns of linguistic expression.

This shows that although next-token prediction is a local prediction task, it places high demands on context modeling.

2.2 The Prediction Target Is Simple, the Learning Content Is Complex

To predict the next token, the model needs to gradually learn information at different levels:

Level of learning Content to master
Token level Co-occurrence patterns between tokens
Word level Word meanings and compositional relationships
Sentence level Grammar, syntax, and semantic structure
Paragraph level Contextual coherence and topical relationships
Knowledge level Associations among facts, concepts, and entities
Task level Patterns for question answering, translation, summarization, code generation, etc.

These abilities are not acquired item by item by separately setting multiple explicit training objectives; rather, they gradually emerge from a large number of sequence prediction tasks.

However, it should be emphasized:

Next Token Prediction does not guarantee that the model will necessarily acquire correct factual knowledge or reliable reasoning ability.

What it optimizes is token prediction performance under the training data distribution. Whether the model truly masters a certain ability is also affected by factors such as data quality, model scale, training methods, and subsequent optimization.

3. The Mathematical Principles of Next Token Prediction

The mathematical foundation of Next Token Prediction is conditional probability.

3.1 Understanding Language Models from Joint Probability

Suppose a token sequence is:

X=(x_1,x_2,\ldots,x_T)

where:

  • x_t: the t-th token.
  • T: the sequence length.

According to the chain rule in probability theory, the joint probability of the entire sequence can be decomposed as:

\boxed{ P(x_1,x_2,\ldots,x_T) = \prod_{t=1}^{T} P(x_t|x_1,\ldots,x_{t-1}) }

Written more concisely:

\boxed{ P(X)=\prod_{t=1}^{T}P(x_t|x_{<t}) }

where:

x_{<t}=(x_1,x_2,\ldots,x_{t-1})

denotes all tokens before the current position.

This is the basic form of probability decomposition for an autoregressive language model.

It decomposes the joint probability of the entire sequence into a series of conditional probabilities.

For example:

Token sequence:
I like learning AI

Corresponding conditional probabilities:
P(I)
P(like | I)
P(learning | I, like)
P(AI | I, like, learning)

The model needs to learn the conditional probability distribution at each position.

Here, P(x_1) can be understood as the probability of the first Token in the sequence; in actual training, it is usually handled uniformly through start markers such as BOS (Beginning of Sequence).

3.2 What Is a Conditional Probability Distribution?

Suppose the model has already seen:

I like

Many Tokens may appear next:

Candidate Token Predicted Probability
you 0.25
learning 0.20
artificial 0.15
programming 0.10
other Tokens 0.30

These probabilities form a conditional probability distribution:

P(x_t|x_{<t})

The model does not directly output a single deterministic answer; instead, it computes a score for each Token in the vocabulary and then converts these scores into probabilities.

Therefore, the essence of Next Token Prediction is:

Given the existing context, estimate the conditional probability that each Token in the vocabulary will be the next Token.

3.3 Why Is It Called Autoregressive?

Autoregressive, referred to in Chinese as 自回归.

Its core characteristic is:

The current prediction depends on content that has already appeared, and the newly generated result in turn becomes input for subsequent predictions.

For example:

Initial input:
The weather today

First prediction:
is very nice

Second prediction:
, suitable

Third prediction:
for going out

Fourth prediction:
for a walk.

During generation, each new Token enters the subsequent context.

flowchart TB
    A["Input existing Token sequence"] --> B["Compute context representation"]
    B --> C["Predict next Token"]
    C --> D["Select or sample Token"]
    D --> E["Append to sequence"]
    E --> F{"Reached stopping condition?"}
    F -->|"No"| B
    F -->|"Yes"| G["Generation ends"]
    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49

This is the basic mechanism of autoregressive generation.

4. How Does the Model Calculate the Probability of the Next Token?

To understand Next Token Prediction, it is necessary to further understand the computation process inside the Transformer.

A typical decoder-only large language model converts input Tokens into hidden vectors, then passes them through multiple Transformer Blocks, and finally computes prediction probabilities over the vocabulary.

4.1 Converting Token IDs to Embeddings

First, the text passes through a Tokenizer to obtain Token IDs.

For example:

Original text:
I like AI

Token:
["I", "like", "AI"]

Token ID:
[1024, 2587, 912]

The IDs here are for illustration only and do not represent the vocabulary indices of any real model.

The model converts Token IDs into vectors through an Embedding matrix:

e_t=Embedding(x_t)

Assuming the model's hidden dimension is 4096, each Token is mapped to a 4096-dimensional vector.

The input sequence can be represented as:

E\in\mathbb{R}^{B\times T\times d_{model}}

Where:

  • B: Batch Size.
  • T: Sequence length.
  • d_{model}: Hidden layer dimension.

4.2 Transformer Extracts Context Representations

Embeddings do not contain complete contextual information.

The model needs to pass through multiple layers of Transformer Blocks to establish connections between different Tokens.

A typical Decoder-only Block contains:

  • Causal Self-Attention.
  • Feed Forward Network.
  • Residual Connection.
  • LayerNorm or RMSNorm.

Among these, Causal Self-Attention is responsible for computing contextual information based on the Tokens before the current position.

After passing through multiple Transformer layers, the hidden states are obtained:

H=Transformer(E)

Where:

H\in\mathbb{R}^{B\times T\times d_{model}}

The hidden state at each position contains the contextual information accessible at that position.

For example, the hidden state at position 3 can use information from positions 1, 2, and 3, but cannot use position 4 or anything after it.

4.3 Mapping Hidden States to Logits

The hidden states output by the Transformer are not yet probabilities.

An output projection layer is needed to map the hidden dimension to the vocabulary size:

\boxed{ Z=HW_{vocab}+b }

Where:

  • H: Transformer hidden states.
  • W_{vocab}: Output projection matrix.
  • b: Optional bias term.
  • Z: Logits.

Assume:

d_{model}=4096

Vocabulary size:

V=100000

Then the shape of the output projection matrix is typically:

W_{vocab}\in\mathbb{R}^{4096\times100000}

Each position yields a Logits vector of length 100000.

It represents the unnormalized score corresponding to each Token in the vocabulary.

Note that some models use weight tying between the Embedding and the output projection; in this case, the input Embedding and the output projection are not two completely independent sets of parameters.

4.4 Softmax Converts Logits into Probabilities

Logits are not probabilities because they can be any real number.

The Softmax function can convert them into a probability distribution:

\boxed{ P(x_t=i|x_{<t}) = \frac{e^{z_i}} {\sum_{j=1}^{V}e^{z_j}} }

Where:

  • z_i: The Logit of the i-th Token.
  • V: Vocabulary size.
  • P(x_t=i|x_{<t}): The predicted probability that this Token becomes the next Token.

The Softmax output satisfies:

0\leq P(x_t=i|x_{<t})\leq1

And:

\sum_{i=1}^{V}P(x_t=i|x_{<t})=1

For example:

Token Logit Softmax Probability
Good 5.2 0.65
Not bad 4.1 0.20
Very good 3.0 0.10
Other — 0.05

The table above is illustrative data.

Through this probability distribution, the model expresses its prediction tendency toward different candidate Tokens.

4.5 Complete Computation Flow

flowchart TB
    A["Input text"] --> B["Tokenizer"]
    B --> C["Token ID"]
    C --> D["Token Embedding"]
    D --> E["Decoder-only Transformer"]
    E --> F["Context hidden states"]
    F --> G["Linear output projection"]
    G --> H["Logits"]
    H --> I["Softmax"]
    I --> J["Next Token probability distribution"]
    J --> K["Select or sample Token"]
    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style K fill:#E5F0E9,stroke:#83A995,color:#315D49

5. Training Phase: How Does the Model Learn to Predict the Next Token?

During inference, the model generates Tokens based on the context.

During training, the model is given a large amount of text, and it repeatedly predicts the true next Token and adjusts its parameters based on the prediction error.

5.1 How Are Training Samples Constructed?

Suppose the training text is:

I like learning AI

After passing through the Tokenizer, we get:

X=[x_1,x_2,x_3,x_4]

During training, we can construct:

Input context Prediction target
I like
I like learning
I like learning AI

The model needs to predict the next Token at every valid position.

In actual training, the model is usually not run separately for each position. Instead, the entire sequence is fed in at once, and all positions are computed in parallel through the causal attention Mask.

5.2 Why Can Training Be Computed in Parallel?

This is a very important engineering question.

Autoregressive generation appears to proceed Token by Token, but training does not require performing a full forward pass Token by Token.

The reason is:

The training text already exists in full, and all Tokens are known.

The model can take the entire sequence as input at once, while the Causal Mask restricts the information each position can access.

For example, if the sequence length is 4:

Token:
A B C D

Attention visibility range:

1 0 0 0
1 1 0 0
1 1 1 0
1 1 1 1

Although the input contains the full sequence, the following still holds:

  • Position A can only access A.
  • Position B can only access A and B.
  • Position C can only access A, B, and C.
  • Position D can access A, B, C, and D.

Therefore, predictions for all positions can be computed in parallel in the same forward pass without leaking future information.

This is also one of the important reasons why Transformer training is relatively efficient.

5.3 Teacher Forcing

In autoregressive language model training, Teacher Forcing is typically used.

Its core idea is:

When training each position, use the true historical Tokens from the dataset, rather than the Tokens the model has just predicted.

For example:

Full sequence:
<BOS> I like AI <EOS>

Model input:
<BOS> I like AI

Training labels:
I like AI <EOS>

At each position, the model predicts the next Token based on the true historical context.

This approach avoids the accumulation of prediction errors in the early stages of training and enables the model to learn the conditional probability distribution more stably.

A distinction needs to be made:

  • Teacher Forcing: uses true historical Tokens during training.
  • Autoregressive Generation: uses Tokens already generated by the model during inference.

The two differ in where their context comes from.

6. Loss Function: How Do We Measure Whether Predictions Are Correct?

Next Token Prediction typically uses cross-entropy loss (Cross Entropy Loss).

The model predicts a probability distribution, while the training data provides the next Token that actually occurs.

The loss function measures whether the model assigns a sufficiently high probability to the true Token.

6.1 Cross-Entropy Loss for a Single Token

Suppose the true next Token is:

AI

Model predicted probabilities:

Token Probability
artificial intelligence 0.8
machine learning 0.1
deep learning 0.05
other 0.05

The probability corresponding to the ground-truth token is:

P(y|context)=0.8

Cross-entropy loss for a single token:

\boxed{ L=-\log P(y|context) }

Therefore:

L=-\log(0.8)\approx0.223

If the model gives the ground-truth token a probability of only 0.01:

L=-\log(0.01)\approx4.605

The lower the predicted probability of the ground-truth token, the larger the loss.

6.2 Why use the negative logarithm?

The logarithmic function has the following properties:

  • The closer the probability is to 1, the closer the negative logarithm is to 0.
  • The closer the probability is to 0, the larger the negative logarithm.
  • The product of multiple conditional probabilities can be converted into a sum of logarithms.

From the joint probability:

P(X)=\prod_{t=1}^{T}P(x_t|x_{<t})

Taking the negative logarithm:

-\log P(X) = -\sum_{t=1}^{T} \log P(x_t|x_{<t})

Therefore, the negative log-likelihood of the entire sequence can be written as the sum of the prediction losses for each token.

6.3 Training loss for the entire sequence

For a sequence of length T:

\boxed{ L_{NTP} = -\sum_{t=1}^{T} \log P(x_t|x_{<t}) }

In actual training, the loss over valid tokens is usually averaged:

\boxed{ L_{NTP} = -\frac{1}{N} \sum_{t\in\mathcal{V}} \log P(x_t|x_{<t}) }

Where:

  • \mathcal{V}: the set of valid positions participating in loss calculation.
  • N: the number of valid tokens.

Padding tokens or positions not participating in training can be excluded via a label mask.

6.4 From loss to model parameter updates

After the loss function computes the prediction error, the gradients of the model parameters still need to be computed via backpropagation.

The optimizer updates the parameters according to the gradients, causing the model to gradually increase the predicted probability of the ground-truth token.

flowchart TB
    A["Input training token sequence"] --> B["Transformer forward pass"]
    B --> C["Obtain logits at each position"]
    C --> D["Compute cross-entropy loss"]
    D --> E["Backpropagation"]
    E --> F["Compute model parameter gradients"]
    F --> G["Optimizer updates parameters"]
    G --> H{"Continue training?"}
    H -->|"Yes"| A
    H -->|"No"| I["Obtain trained model"]
    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

Note:

  • Backpropagation is responsible for computing gradients.
  • The optimizer is responsible for updating parameters according to the gradients.
  • Cross-entropy is responsible for measuring the difference between predictions and ground-truth labels.

These three are different stages in the training process.