Back to series
· 8 min read

Temperature, Top-K and Top-P: How Do Large Language Models Control Randomness in Text Generation?


1. Why Do Large Language Models Need Sampling Strategies?

Large language models usually generate text in an autoregressive manner.

Each time, based on the existing context, the model predicts a probability distribution over the next Token, and then selects one of the candidate Tokens as the output.

Suppose the current context is:

The weather today is very

The model may produce the following candidate probabilities:

Candidate Token Probability
good 0.55
nice 0.20
hot 0.12
cold 0.08
terrible 0.05

The model does not directly output the Token with the highest probability; instead, it can use different strategies to make the selection.

For example:

  • Greedy Decoding: always select the Token with the highest probability.
  • Random Sampling: sample randomly according to the probability distribution.
  • Temperature: adjust the sharpness of the probability distribution.
  • Top-K: keep only the K candidates with the highest probabilities.
  • Top-P: keep only the candidate set whose cumulative probability reaches a specified threshold.

Together, these methods determine the determinism, diversity, and randomness of the text generated by the model.

2. From Logits to Token Probability Distribution

To understand Temperature, Top-K, and Top-P, we first need to understand the basic process by which the model generates Tokens.

1. What Are Logits?

After the Transformer finishes computing the current context, it obtains a Logits vector through the output layer.

Suppose the model vocabulary size is V. Then the Logits can be represented as:

z=(z_1,z_2,\ldots,z_V)

where:

  • V: the model vocabulary size.
  • z_i: the Logit corresponding to the i-th Token.
  • Logit: the unnormalized score given by the model to a candidate Token.

Logits can be any real number and are not restricted to the range 0 to 1.

For example:

z=[3.2,2.1,1.5,0.8,-0.5]

These values are not probabilities and cannot be directly interpreted as the likelihood that the model will choose each Token.

2. Softmax Converts Logits into Probabilities

To obtain a probability distribution, we need to normalize via the Softmax function:

P(x_i)= \frac{e^{z_i}} {\sum_{j=1}^{V}e^{z_j}}

Softmax has two important properties:

  • The probability of each Token is between 0 and 1.
  • The probabilities of all candidate Tokens sum to 1.
\sum_{i=1}^{V}P(x_i)=1

For example:

Illustrative data, used only to explain the role of Softmax

Token Logit Softmax Probability
A 3.2 0.55
B 2.1 0.18
C 1.5 0.10
D 0.8 0.06
E -0.5 0.01
Other — 0.10

The probability distribution obtained by Softmax is the basis for subsequent sampling strategies.

3. Overall Process of Token Selection

flowchart TB
    A["Input current context"] --> B["Transformer forward pass"]
    B --> C["Output Logits"]
    C --> D["Temperature scaling"]
    D --> E["Softmax probability distribution"]
    E --> F["Top-K / Top-P filtering"]
    F --> G["Renormalize probabilities"]
    G --> H["Sample next Token"]
    H --> I{"Stop condition met?"}
    I -->|"No"| J["Append new Token to context"]
    J --> B
    I -->|"Yes"| K["Output generated result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#E5F0E9,stroke:#83A995,color:#315D49

Note: this is a common abstraction of the generation pipeline. Different inference frameworks may organize operations such as Temperature, Logits Processor, Top-K, and Top-P into different processing pipelines.

III. Temperature: Controlling the Sharpness of the Probability Distribution

Temperature is one of the most important parameters for controlling the randomness of generation.

It changes the probability distribution output by Softmax by adjusting the numerical scale of the Logits.

1. Mathematical Principle

The probability formula corresponding to Temperature is:

P(x_i)= \frac{e^{z_i/T}} {\sum_{j=1}^{V}e^{z_j/T}}

Where:

  • z_i: the original Logit corresponding to the Token.
  • T: the Temperature parameter.
  • P(x_i): the adjusted Token probability.

The core function of Temperature is:

To change the probability differences among candidate Tokens by scaling the Logits.

2. The Lower the Temperature, the Sharper the Distribution

When:

0<T<1

the Logits are amplified.

Tokens that originally had higher probabilities will have their probabilities further increased; tokens with lower probabilities will have their probabilities further decreased.

For example, with the original Logits:

z=[3,2,1]

when Temperature is 1:

P=\operatorname{Softmax}([3,2,1])

approximately:

P=[0.665,0.245,0.090]

when Temperature is 0.5:

P=\operatorname{Softmax}([6,4,2])

approximately:

P=[0.867,0.117,0.016]

As can be seen, the advantage of the highest-probability candidate is markedly expanded.

Illustration of the probability distribution for the same set of Logits under different Temperature values

Token T = 0.5 T = 1.0 T = 2.0
A 0.867 0.665 0.506
B 0.117 0.245 0.307
C 0.016 0.090 0.186

Therefore, a lower Temperature generally makes generation more stable, tending to select the Token that the model considers more likely.

But a low temperature does not mean absolute correctness, nor does it mean the model will not make errors.

3. The higher the Temperature, the flatter the distribution

When:

T>1

The Logits are compressed, and the probability differences among candidate Tokens shrink.

Lower-probability candidate Tokens get a greater chance of being sampled.

This usually leads to:

  • Richer expression.
  • More varied wording.
  • Greater randomness in generation.
  • Greater output variability.
  • Higher risk of deviating from expectations.

As Temperature approaches infinity, the Softmax distribution corresponding to finite Logits gradually approaches a uniform distribution.

4. A few special cases of Temperature

Temperature Behavior
T=1 Preserves the original Softmax distribution
0<T<1 Probability distribution becomes sharper
T>1 Probability distribution becomes flatter
T\rightarrow 0^+ Approaches selecting the Token corresponding to the highest Logit

Temperature is usually required to be greater than 0.

When T=0, the division in the formula no longer applies. Engineering interfaces generally treat a Temperature of 0 as Greedy Decoding, rather than directly performing Softmax division by zero.

5. Temperature does not change the ranking of candidate Tokens

This is a very important detail.

For any two Tokens, if:

z_i>z_j

Since Temperature is positive:

\frac{z_i}{T}>\frac{z_j}{T}

Therefore, Temperature does not change the relative ranking of Logits.

It changes the probability distribution, not which Token has the higher original Logit.

For example:

Original ranking:
A > B > C

Temperature = 0.5:
A > B > C

Temperature = 2.0:
A > B > C

The ranking remains unchanged, but the sampling probability of each Token changes.

6. Use cases for Temperature

Scenario Common adjustment direction
Code generation Lower Temperature
Mathematical computation Lower Temperature
Technical Q&A Lower or medium Temperature
Creative writing Medium or higher Temperature
Story writing Medium or higher Temperature
Diverse content generation Higher Temperature

The specific parameters need to be adjusted in light of the model, the task, and evaluation results; there is no fixed optimum that applies to all models.

IV. Top-K: Sampling Only from the K Tokens with the Highest Probability

Top-K is a candidate-set filtering strategy.

It does not change the original probability ranking of all tokens; instead, it keeps only the K tokens with the highest probabilities, sets the probabilities of all other tokens to 0, and then renormalizes the retained probabilities.

1. Mathematical Principle

Suppose the model's vocabulary is:

V=\{x_1,x_2,\ldots,x_n\}

After Softmax, we obtain the probability distribution:

P=\{p_1,p_2,\ldots,p_n\}

Sort the tokens by probability from high to low:

p_{(1)}\geq p_{(2)}\geq\cdots\geq p_{(n)}

Top-K selects the K tokens with the highest probabilities:

S_K=\{x_{(1)},x_{(2)},\ldots,x_{(K)}\}

After filtering, the probabilities are renormalized:

P'(x_i)= \begin{cases} \frac{P(x_i)}{\sum_{x_j\in S_K}P(x_j)}, & x_i\in S_K\\ 0, & x_i\notin S_K \end{cases}

2. A Worked Example of Top-K

Suppose the model produces the following probability distribution:

Token Original probability
A 0.40
B 0.25
C 0.15
D 0.10
E 0.06
F 0.04

Setting:

K=3

The model keeps only A, B, and C.

The sum of the probabilities after filtering is:

0.40+0.25+0.15=0.80

Renormalization:

P'(A)=0.40/0.80=0.50
P'(B)=0.25/0.80=0.3125
P'(C)=0.15/0.80=0.1875

Final distribution:

Token Original probability Probability after Top-3
A 0.40 0.50
B 0.25 0.3125
C 0.15 0.1875
D 0.10 0
E 0.06 0
F 0.04 0

Top-K does not simply discard the low-probability tokens and then use the remaining original probabilities directly; renormalization is required.

3. How Does the Value of K Affect Generation?

  • Small K: the candidate range shrinks, and the output becomes more concentrated.
  • Large K: the candidate range expands, and generation diversity increases.
  • K equal to the vocabulary size: no Top-K filtering is performed.
  • K equal to 1: at each step only the highest-probability token is kept, which is equivalent to greedy selection after this filtering step.

Note that when K=1, even if random sampling is used afterward, the only token that can be selected is the single retained one.

4. Advantages and Limitations of Top-K

Pros:

  • Simple to implement.
  • Can eliminate a large number of low-probability candidates.
  • Can limit the generation range.
  • The parameter is intuitive and easy to debug.

Limitations:

  • K is a fixed number and does not consider the actual shape of the current probability distribution.
  • Under different contexts, the reasonable number of candidates may vary greatly.
  • May remove some candidates that have low probability but are still reasonable.
  • When the probability distribution is highly concentrated, a fixed K may retain too many low-quality candidates.

This is also an important reason why Top-P is widely adopted.

V. Top-P: Dynamically Select the Candidate Set Based on Cumulative Probability

Top-P is also known as Nucleus Sampling.

Unlike Top-K, which always keeps a fixed K candidates, Top-P dynamically determines how many Tokens to keep based on the cumulative probability.

Its core idea is:

After sorting probabilities from high to low, select the smallest candidate set whose cumulative probability reaches the specified threshold P.

1. Mathematical Principle

Assume the probability distribution is sorted:

p_{(1)}\geq p_{(2)}\geq\cdots\geq p_{(n)}

Given threshold:

0<P\leq1

Find the smallest integer k such that:

\sum_{i=1}^{k}p_{(i)}\geq P

Then the Top-P candidate set is:

S_P=\{x_{(1)},x_{(2)},\ldots,x_{(k)}\}

After filtering, the probabilities also need to be renormalized:

P'(x_i)= \begin{cases} \frac{P(x_i)}{\sum_{x_j\in S_P}P(x_j)}, & x_i\in S_P\\ 0, & x_i\notin S_P \end{cases}

2. Top-P Worked Example

Assume the probability distribution is as follows:

Token Probability Cumulative Probability
A 0.40 0.40
B 0.25 0.65
C 0.15 0.80
D 0.10 0.90
E 0.06 0.96
F 0.04 1.00

Set:

P=0.80

Start accumulating from the highest-probability Token:

  • A: cumulative 0.40.
  • A + B: cumulative 0.65.
  • A + B + C: cumulative 0.80.

Therefore, the candidate set is:

S_P=\{A,B,C\}

Note: Top-P selects the smallest set whose cumulative probability reaches the threshold, not all Tokens whose individual probabilities are greater than P.

3. Why Is Top-P Dynamic?

Assume two contexts correspond to different probability distributions.

All examples below use Top-P = 0.90

Scenario A: The probability distribution is highly concentrated

Token Probability Cumulative Probability
A 0.70 0.70
B 0.15 0.85
C 0.08 0.93
D 0.04 0.97
E 0.03 1.00

Reaching 0.90 requires only A, B, and C, for a total of 3 candidates.

Scenario B: The probability distribution is relatively dispersed

Token Probability Cumulative Probability
A 0.25 0.25
B 0.20 0.45
C 0.18 0.63
D 0.15 0.78
E 0.12 0.90
F 0.10 1.00

Reaching 0.90 requires A, B, C, D, and E — a total of 5 candidates.

This shows that:

Top-P automatically adjusts the number of candidates based on the current probability distribution.

When the model is very confident about a particular Token, the candidate set is usually small; when the model's judgment across multiple Tokens is more spread out, the candidate set is usually larger.

4. How does the P value affect generation?

  • Small P: only a small number of candidates with higher probability mass are retained.
  • Large P: more candidates are retained.
  • P equal to 1: no Top-P filtering is performed.
  • The closer P is to 0, the more the candidate set is concentrated on the highest-probability Tokens.

Note that actual implementations may set a minimum number of retained Tokens to avoid the candidate set becoming too small in extreme cases.

5. Advantages and Limitations of Top-P

Advantages:

  • Dynamically adjusts the number of candidates based on the current probability distribution.
  • Can adapt to the uncertainty of different contexts.
  • Reduces the interference of large numbers of low-probability Tokens on sampling.
  • Provides an adjustable balance between diversity and reasonableness.

Limitations:

  • Requires sorting probabilities and calculating cumulative probabilities.
  • The P value does not directly correspond to a fixed number of candidates.
  • The candidate set varies considerably across different contexts.
  • May still filter out some lower-probability but valuable candidates.

VI. What is the difference between Top-K and Top-P?

Both are probability filtering strategies, but the way they determine the candidate set is completely different.

Comparison Dimension Top-K Top-P
Core Mechanism Fixed number of candidates Fixed cumulative probability threshold
Number of Candidates Usually fixed Dynamically varying
Basis for Judgment Probability ranking Cumulative probability
Main Parameter K P
When the distribution is concentrated Still retains K candidates Usually retains fewer candidates
When the distribution is spread out Still retains K candidates Usually retains more candidates
Advantage Simple and controllable Highly adaptive
Limitation Does not consider the shape of the distribution Number of candidates is not fixed

1. The Core Difference Between the Two

This can be understood through the following illustration:

flowchart TB
    A["Softmax probability distribution"] --> B["Candidate token ranking"]
    B --> C["Top-K"]
    B --> D["Top-P"]
    C --> E["Keep a fixed K tokens"]
    D --> F["Cumulative probability reaches P"]
    E --> G["Renormalize"]
    F --> H["Candidate set of dynamic size"]
    H --> I["Renormalize"]
    G --> J["Sample token"]
    I --> J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

2. Can Top-K and Top-P be used at the same time?

Yes.

Some inference systems use Top-K and Top-P together.

For example:

Temperature = 0.8
Top-K = 50
Top-P = 0.9

One common approach is: