1. Why Do Large Language Models Need Sampling Strategies?
Large language models usually generate text in an autoregressive manner.
Each time, based on the existing context, the model predicts a probability distribution over the next Token, and then selects one of the candidate Tokens as the output.
Suppose the current context is:
The weather today is very
The model may produce the following candidate probabilities:
| Candidate Token | Probability |
|---|---|
| good | 0.55 |
| nice | 0.20 |
| hot | 0.12 |
| cold | 0.08 |
| terrible | 0.05 |
The model does not directly output the Token with the highest probability; instead, it can use different strategies to make the selection.
For example:
- Greedy Decoding: always select the Token with the highest probability.
- Random Sampling: sample randomly according to the probability distribution.
- Temperature: adjust the sharpness of the probability distribution.
- Top-K: keep only the K candidates with the highest probabilities.
- Top-P: keep only the candidate set whose cumulative probability reaches a specified threshold.
Together, these methods determine the determinism, diversity, and randomness of the text generated by the model.
2. From Logits to Token Probability Distribution
To understand Temperature, Top-K, and Top-P, we first need to understand the basic process by which the model generates Tokens.
1. What Are Logits?
After the Transformer finishes computing the current context, it obtains a Logits vector through the output layer.
Suppose the model vocabulary size is V. Then the Logits can be represented as:
where:
- V: the model vocabulary size.
- z_i: the Logit corresponding to the i-th Token.
- Logit: the unnormalized score given by the model to a candidate Token.
Logits can be any real number and are not restricted to the range 0 to 1.
For example:
These values are not probabilities and cannot be directly interpreted as the likelihood that the model will choose each Token.
2. Softmax Converts Logits into Probabilities
To obtain a probability distribution, we need to normalize via the Softmax function:
Softmax has two important properties:
- The probability of each Token is between 0 and 1.
- The probabilities of all candidate Tokens sum to 1.
For example:
| Token | Logit | Softmax Probability |
|---|---|---|
| A | 3.2 | 0.55 |
| B | 2.1 | 0.18 |
| C | 1.5 | 0.10 |
| D | 0.8 | 0.06 |
| E | -0.5 | 0.01 |
| Other | — | 0.10 |
The probability distribution obtained by Softmax is the basis for subsequent sampling strategies.
3. Overall Process of Token Selection
flowchart TB
A["Input current context"] --> B["Transformer forward pass"]
B --> C["Output Logits"]
C --> D["Temperature scaling"]
D --> E["Softmax probability distribution"]
E --> F["Top-K / Top-P filtering"]
F --> G["Renormalize probabilities"]
G --> H["Sample next Token"]
H --> I{"Stop condition met?"}
I -->|"No"| J["Append new Token to context"]
J --> B
I -->|"Yes"| K["Output generated result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#E5F0E9,stroke:#83A995,color:#315D49
Note: this is a common abstraction of the generation pipeline. Different inference frameworks may organize operations such as Temperature, Logits Processor, Top-K, and Top-P into different processing pipelines.
III. Temperature: Controlling the Sharpness of the Probability Distribution
Temperature is one of the most important parameters for controlling the randomness of generation.
It changes the probability distribution output by Softmax by adjusting the numerical scale of the Logits.
1. Mathematical Principle
The probability formula corresponding to Temperature is:
Where:
- z_i: the original Logit corresponding to the Token.
- T: the Temperature parameter.
- P(x_i): the adjusted Token probability.
The core function of Temperature is:
To change the probability differences among candidate Tokens by scaling the Logits.
2. The Lower the Temperature, the Sharper the Distribution
When:
the Logits are amplified.
Tokens that originally had higher probabilities will have their probabilities further increased; tokens with lower probabilities will have their probabilities further decreased.
For example, with the original Logits:
when Temperature is 1:
approximately:
when Temperature is 0.5:
approximately:
As can be seen, the advantage of the highest-probability candidate is markedly expanded.
| Token | T = 0.5 | T = 1.0 | T = 2.0 |
|---|---|---|---|
| A | 0.867 | 0.665 | 0.506 |
| B | 0.117 | 0.245 | 0.307 |
| C | 0.016 | 0.090 | 0.186 |
Therefore, a lower Temperature generally makes generation more stable, tending to select the Token that the model considers more likely.
But a low temperature does not mean absolute correctness, nor does it mean the model will not make errors.
3. The higher the Temperature, the flatter the distribution
When:
The Logits are compressed, and the probability differences among candidate Tokens shrink.
Lower-probability candidate Tokens get a greater chance of being sampled.
This usually leads to:
- Richer expression.
- More varied wording.
- Greater randomness in generation.
- Greater output variability.
- Higher risk of deviating from expectations.
As Temperature approaches infinity, the Softmax distribution corresponding to finite Logits gradually approaches a uniform distribution.
4. A few special cases of Temperature
| Temperature | Behavior |
|---|---|
| T=1 | Preserves the original Softmax distribution |
| 0<T<1 | Probability distribution becomes sharper |
| T>1 | Probability distribution becomes flatter |
| T\rightarrow 0^+ | Approaches selecting the Token corresponding to the highest Logit |
Temperature is usually required to be greater than 0.
When T=0, the division in the formula no longer applies. Engineering interfaces generally treat a Temperature of 0 as Greedy Decoding, rather than directly performing Softmax division by zero.
5. Temperature does not change the ranking of candidate Tokens
This is a very important detail.
For any two Tokens, if:
Since Temperature is positive:
Therefore, Temperature does not change the relative ranking of Logits.
It changes the probability distribution, not which Token has the higher original Logit.
For example:
Original ranking:
A > B > C
Temperature = 0.5:
A > B > C
Temperature = 2.0:
A > B > C
The ranking remains unchanged, but the sampling probability of each Token changes.
6. Use cases for Temperature
| Scenario | Common adjustment direction |
|---|---|
| Code generation | Lower Temperature |
| Mathematical computation | Lower Temperature |
| Technical Q&A | Lower or medium Temperature |
| Creative writing | Medium or higher Temperature |
| Story writing | Medium or higher Temperature |
| Diverse content generation | Higher Temperature |
The specific parameters need to be adjusted in light of the model, the task, and evaluation results; there is no fixed optimum that applies to all models.
IV. Top-K: Sampling Only from the K Tokens with the Highest Probability
Top-K is a candidate-set filtering strategy.
It does not change the original probability ranking of all tokens; instead, it keeps only the K tokens with the highest probabilities, sets the probabilities of all other tokens to 0, and then renormalizes the retained probabilities.
1. Mathematical Principle
Suppose the model's vocabulary is:
After Softmax, we obtain the probability distribution:
Sort the tokens by probability from high to low:
Top-K selects the K tokens with the highest probabilities:
After filtering, the probabilities are renormalized:
2. A Worked Example of Top-K
Suppose the model produces the following probability distribution:
| Token | Original probability |
|---|---|
| A | 0.40 |
| B | 0.25 |
| C | 0.15 |
| D | 0.10 |
| E | 0.06 |
| F | 0.04 |
Setting:
The model keeps only A, B, and C.
The sum of the probabilities after filtering is:
Renormalization:
Final distribution:
| Token | Original probability | Probability after Top-3 |
|---|---|---|
| A | 0.40 | 0.50 |
| B | 0.25 | 0.3125 |
| C | 0.15 | 0.1875 |
| D | 0.10 | 0 |
| E | 0.06 | 0 |
| F | 0.04 | 0 |
Top-K does not simply discard the low-probability tokens and then use the remaining original probabilities directly; renormalization is required.
3. How Does the Value of K Affect Generation?
- Small K: the candidate range shrinks, and the output becomes more concentrated.
- Large K: the candidate range expands, and generation diversity increases.
- K equal to the vocabulary size: no Top-K filtering is performed.
- K equal to 1: at each step only the highest-probability token is kept, which is equivalent to greedy selection after this filtering step.
Note that when K=1, even if random sampling is used afterward, the only token that can be selected is the single retained one.
4. Advantages and Limitations of Top-K
Pros:
- Simple to implement.
- Can eliminate a large number of low-probability candidates.
- Can limit the generation range.
- The parameter is intuitive and easy to debug.
Limitations:
- K is a fixed number and does not consider the actual shape of the current probability distribution.
- Under different contexts, the reasonable number of candidates may vary greatly.
- May remove some candidates that have low probability but are still reasonable.
- When the probability distribution is highly concentrated, a fixed K may retain too many low-quality candidates.
This is also an important reason why Top-P is widely adopted.
V. Top-P: Dynamically Select the Candidate Set Based on Cumulative Probability
Top-P is also known as Nucleus Sampling.
Unlike Top-K, which always keeps a fixed K candidates, Top-P dynamically determines how many Tokens to keep based on the cumulative probability.
Its core idea is:
After sorting probabilities from high to low, select the smallest candidate set whose cumulative probability reaches the specified threshold P.
1. Mathematical Principle
Assume the probability distribution is sorted:
Given threshold:
Find the smallest integer k such that:
Then the Top-P candidate set is:
After filtering, the probabilities also need to be renormalized:
2. Top-P Worked Example
Assume the probability distribution is as follows:
| Token | Probability | Cumulative Probability |
|---|---|---|
| A | 0.40 | 0.40 |
| B | 0.25 | 0.65 |
| C | 0.15 | 0.80 |
| D | 0.10 | 0.90 |
| E | 0.06 | 0.96 |
| F | 0.04 | 1.00 |
Set:
Start accumulating from the highest-probability Token:
- A: cumulative 0.40.
- A + B: cumulative 0.65.
- A + B + C: cumulative 0.80.
Therefore, the candidate set is:
Note: Top-P selects the smallest set whose cumulative probability reaches the threshold, not all Tokens whose individual probabilities are greater than P.
3. Why Is Top-P Dynamic?
Assume two contexts correspond to different probability distributions.
Scenario A: The probability distribution is highly concentrated
| Token | Probability | Cumulative Probability |
|---|---|---|
| A | 0.70 | 0.70 |
| B | 0.15 | 0.85 |
| C | 0.08 | 0.93 |
| D | 0.04 | 0.97 |
| E | 0.03 | 1.00 |
Reaching 0.90 requires only A, B, and C, for a total of 3 candidates.
Scenario B: The probability distribution is relatively dispersed
| Token | Probability | Cumulative Probability |
|---|---|---|
| A | 0.25 | 0.25 |
| B | 0.20 | 0.45 |
| C | 0.18 | 0.63 |
| D | 0.15 | 0.78 |
| E | 0.12 | 0.90 |
| F | 0.10 | 1.00 |
Reaching 0.90 requires A, B, C, D, and E — a total of 5 candidates.
This shows that:
Top-P automatically adjusts the number of candidates based on the current probability distribution.
When the model is very confident about a particular Token, the candidate set is usually small; when the model's judgment across multiple Tokens is more spread out, the candidate set is usually larger.
4. How does the P value affect generation?
- Small P: only a small number of candidates with higher probability mass are retained.
- Large P: more candidates are retained.
- P equal to 1: no Top-P filtering is performed.
- The closer P is to 0, the more the candidate set is concentrated on the highest-probability Tokens.
Note that actual implementations may set a minimum number of retained Tokens to avoid the candidate set becoming too small in extreme cases.
5. Advantages and Limitations of Top-P
Advantages:
- Dynamically adjusts the number of candidates based on the current probability distribution.
- Can adapt to the uncertainty of different contexts.
- Reduces the interference of large numbers of low-probability Tokens on sampling.
- Provides an adjustable balance between diversity and reasonableness.
Limitations:
- Requires sorting probabilities and calculating cumulative probabilities.
- The P value does not directly correspond to a fixed number of candidates.
- The candidate set varies considerably across different contexts.
- May still filter out some lower-probability but valuable candidates.
VI. What is the difference between Top-K and Top-P?
Both are probability filtering strategies, but the way they determine the candidate set is completely different.
| Comparison Dimension | Top-K | Top-P |
|---|---|---|
| Core Mechanism | Fixed number of candidates | Fixed cumulative probability threshold |
| Number of Candidates | Usually fixed | Dynamically varying |
| Basis for Judgment | Probability ranking | Cumulative probability |
| Main Parameter | K | P |
| When the distribution is concentrated | Still retains K candidates | Usually retains fewer candidates |
| When the distribution is spread out | Still retains K candidates | Usually retains more candidates |
| Advantage | Simple and controllable | Highly adaptive |
| Limitation | Does not consider the shape of the distribution | Number of candidates is not fixed |
1. The Core Difference Between the Two
This can be understood through the following illustration:
flowchart TB
A["Softmax probability distribution"] --> B["Candidate token ranking"]
B --> C["Top-K"]
B --> D["Top-P"]
C --> E["Keep a fixed K tokens"]
D --> F["Cumulative probability reaches P"]
E --> G["Renormalize"]
F --> H["Candidate set of dynamic size"]
H --> I["Renormalize"]
G --> J["Sample token"]
I --> J
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2. Can Top-K and Top-P be used at the same time?
Yes.
Some inference systems use Top-K and Top-P together.
For example:
Temperature = 0.8
Top-K = 50
Top-P = 0.9
One common approach is: