I. What Is LLM Inference?
Large language model inference refers to the process of using an already-trained model to compute output results based on the input provided by the user.
For example, the user inputs:
Please explain what a Transformer is.
The model needs to generate a reasonable answer based on the input content.
From a computational perspective, a large model does not directly understand the question and then write out the complete answer all at once. Instead, it uses neural network computation to produce a probability distribution over the next Token, then selects one Token according to the decoding strategy, and repeats this process continuously.
Core process:
flowchart TB
A["User Input Prompt"]
B["Tokenizer Encoding"]
C["Model Forward Pass"]
D["Compute Probability of Next Token"]
E["Decoding Strategy Selects Token"]
F["Append to Context"]
G{"Stop Generation?"}
H["Tokenizer Decoding"]
I["Return Complete Answer"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G -->|"No"| C
G -->|"Yes"| H
H --> I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Here, two concepts need to be distinguished:
- Training: Update model parameters through data and optimization algorithms.
- Inference: Use already-trained parameters to perform forward computation and generate prediction results.
Regular inference does not update model weights.
However, the inference process may involve the creation and updating of runtime data such as KV Cache, temporary activation values, and routing states. These changes are not equivalent to training model parameters.
II. Overall Architecture of LLM Inference
Taking a Decoder-only Transformer as an example, a single inference run mainly goes through the following parts:
- Input processing: Convert text into Token IDs.
- Embedding: Map Token IDs to vectors.
- Transformer: Compute contextual representations through multiple layers of Attention and FFN.
- LM Head: Project the hidden states to the vocabulary dimension to obtain Logits.
- Sampling / Decoding: Select the next Token based on the probability distribution.
- Loop generation: Append the new Token to the context and continue computing.
Overall structure:
flowchart TB
A["Input Text Prompt"]
B["Tokenizer"]
C["Token IDs"]
D["Token Embedding"]
E["Transformer Layers"]
F["Final Normalization"]
G["LM Head"]
H["Logits"]
I["Softmax"]
J["Decoding / Sampling"]
K["Next Token"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H --> I
I --> J
J --> K
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note:
- Tokenizer is not part of the main Transformer neural network.
- Embedding converts discrete Token IDs into continuous vectors.
- Transformer computes contextual information.
- LM Head maps hidden states to prediction scores over the vocabulary.
- Softmax converts Logits into a probability distribution.
- The decoding strategy selects the Token actually output from the probability distribution.
Next, we will break this down step by step.
III. Step 1: How Does a Prompt Enter the Model?
1. Text Passes Through the Tokenizer
The model cannot process raw strings directly; the text must first be split into Tokens by the Tokenizer and then mapped to integer IDs.
For example:
Input:
Artificial intelligence is changing the world
It might be encoded as:
Token:
["artificial", "intelligence", "is", "changing", "world"]
Token ID:
[1024, 5832, 219, 4381, 765]
The above is only for illustration; the actual Token segmentation and IDs depend on the vocabulary used by the model.
A Token ID is an index into the vocabulary; it does not indicate semantic magnitude or ordering among Tokens.
2. Converting Token IDs to Embeddings
Assume:
- The vocabulary size is V.
- The hidden dimension is d.
- The number of input Tokens is T.
Embedding matrix:
Input Token IDs:
The model looks up the Embedding table to obtain:
Finally:
If a positional encoding mechanism such as RoPE is used, positional information will be injected during subsequent Attention computation; other models may also use different positional representations.
3. Batch Dimension
In actual inference, it is usually not just one input that is processed; multiple requests are organized into a Batch.
Assume:
- Batch Size: B.
- Sequence length: T.
- Hidden Size: d.
Then the shape of the input hidden state is usually:
Where:
| Dimension | Meaning |
|---|---|
| B | Number of requests in a Batch |
| T | Token sequence length |
| d | Hidden vector dimension of each Token |
In real inference systems, the input lengths of different requests may differ, so issues such as Padding, sequence length management, and dynamic batching also need to be handled.
IV. Step 2: How Does a Transformer Compute Contextual Information?
After Token Embedding enters the Transformer, it needs to go through multiple layers of neural network computation.
Taking a typical Decoder-only Transformer as an example, each layer mainly contains:
- Causal Self-Attention.
- FFN or MoE.
- Residual connection.
- LayerNorm or RMSNorm.
1. Self-Attention
Self-Attention enables the current Token to obtain information from other Tokens in the context.
Input hidden state:
Obtained through three linear projections:
Where:
- Q (Query): query vector.
- K (Key): key vector.
- V (Value): value vector.
Attention computation:
Where:
- d_k: Key dimension.
- M: causal attention mask.
- QK^T: computes relevance scores between Tokens.
The causal mask ensures that the current position can only access itself and previous positions, and cannot access future Tokens.
For example:
Token sequence:
I like artificial intelligence technology
Information accessible at each position:
| Current Token | Accessible Tokens |
|---|---|
| I | I |
| like | I, like |
| artificial intelligence | I, like, artificial intelligence |
| technology | I, like, artificial intelligence, technology |
This is the Causal Attention of a Decoder-only model.
2. FFN Feedforward Network
Attention handles information exchange between Tokens, while FFN applies a nonlinear transformation to the hidden representation at each position.
Typical FFN:
It usually consists of an up-projection, an activation function, and a down-projection.
For example:
Actual models may use gated feedforward structures such as SwiGLU, rather than necessarily using a simple two-layer FFN.
3. Multi-Layer Transformer Computation
The computation result of one Transformer layer is passed to the next layer.
flowchart TB
A["输入隐藏状态"]
B["归一化"]
C["Causal Self-Attention"]
D["残差连接"]
E["归一化"]
F["FFN / MoE"]
G["残差连接"]
H["输出隐藏状态"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
Assuming the model has 32 Transformer layers, the input hidden state will go through 32 layers of computation.
Finally, we obtain:
This contains the final hidden representation of each Token under the current context.
5. Step 3: How Does the LM Head Predict the Next Token?
The hidden state output by the Transformer is not yet the final token probabilities.
The model needs to map the hidden vector to the vocabulary space through the LM Head.
1. Mapping Hidden States to Logits
Assume:
- The hidden dimension is d.
- The vocabulary size is V.
The LM Head weight matrix:
For the hidden state at the last position:
Compute:
This gives:
This vector is called logits.
Logits represent the model's unnormalized prediction scores for each token in the vocabulary.
For example, if the vocabulary has 100000 tokens, then the logits contain 100000 scores.
2. Softmax Converts to Probabilities
Logits are not probabilities themselves and need to pass through Softmax:
This gives the probability distribution of the next token.
Example:
| Candidate Token | Logits (illustrative) | Probability (illustrative) |
|---|---|---|
| is | 3.2 | 45% |
| of | 2.4 | 20% |
| a | 1.8 | 12% |
| model | 1.2 | 7% |
| Other tokens | — | 16% |
The actual probabilities need to be computed from the logits over the full vocabulary; only some candidates are shown here.
3. Why Doesn't the Model Directly Output the Highest-Probability Token?
If the highest-probability token is selected every time, this is Greedy Decoding.
However, actual inference can use different decoding strategies, such as:
- Greedy Search.
- Temperature Sampling.
- Top-K Sampling.
- Top-P Sampling.
- Beam Search.
These strategies affect the determinism, diversity, and computational cost of the final generated output.
6. Step 4: How Do Decoding Strategies Determine the Generated Content?
After the model completes the forward pass, it produces a token probability distribution.
Next, the next token must be selected according to the decoding strategy.
1. Greedy Decoding
Greedy decoding selects the token with the highest probability each time:
Advantages:
- Simple to implement.
- Deterministic output.
- No random sampling required.
Disadvantages:
- Prone to repetition or locally optimal results.
- Insufficient generation diversity.
- Does not guarantee that the overall sequence probability is optimal.
2. Temperature Sampling
Temperature is used to adjust the smoothness of the probability distribution.
Formula:
where:
- z_i: the original logit.
- T: Temperature.
Effects of different Temperature values:
| Temperature | Effect |
|---|---|
| T<1 | The probability distribution becomes sharper |
| T=1 | Maintains the original Softmax distribution |
| T>1 | The probability distribution becomes smoother |
Generally:
- Lower Temperature: more stable and conservative output.
- Higher Temperature: more random and diverse output.
However, Temperature does not increase the model's knowledge, nor does it guarantee a more accurate answer.
3. Top-K Sampling
Top-K keeps only the K tokens with the highest probabilities, then renormalizes them and samples from them.
For example:
Original candidates:
Token A: 40%
Token B: 25%
Token C: 15%
Token D: 10%
Token E: 10%
If:
Then only the following are kept:
- Token A.
- Token B.
- Token C.
The remaining candidates are excluded, and the remaining probabilities are renormalized.
4. Top-P Sampling
Top-P is also called Nucleus Sampling.
It ranks tokens from highest to lowest probability and selects the smallest candidate set whose cumulative probability reaches the threshold P.
For example:
Token A: 40%
Token B: 25%
Token C: 15%
Token D: 10%
Token E: 10%
If:
Then the candidate set is:
- Token A: 40%.
- Token B: 25%.
- Token C: 15%.
The cumulative probability reaches 80%, and subsequent tokens no longer participate in sampling.
5. Comparison of Different Decoding Strategies
| Strategy | Core Mechanism | Main Characteristics |
|---|---|---|
| Greedy | Selects the highest-probability token | Highly deterministic |
| Temperature | Adjusts the probability distribution | Controls randomness |
| Top-K | Keeps the top K candidates | Limits the candidate range |
| Top-P | Keeps candidates whose cumulative probability reaches the threshold | Dynamically adjusts the number of candidates |
| Beam Search | Keeps multiple candidate sequences | Performs sequence-level search |
Note the following:
- Temperature can be used in combination with Top-K and Top-P.
- Top-K and Top-P can also be used together.
- Beam Search differs from common stochastic sampling strategies in both goal and implementation.
VII. Step 5: Why Do Large Models Need KV Cache?
If the Attention for the entire historical sequence is recomputed every time a token is generated, the inference cost will be very high.
This is the problem that KV Cache needs to solve.
1. What Happens Without KV Cache?
Suppose the input prompt has 1,000 tokens.
When the model generates the first token, it needs to process these 1,000 tokens.
When generating the second token, the history length becomes 1,001.
Without a caching mechanism, the model needs to recompute the Key and Value corresponding to the historical tokens.
As the generation length increases, historical computations are repeatedly performed.
This produces a large amount of redundant computation.
2. Core Idea of KV Cache
In Transformer self-attention, each token computes:
where:
- Q: Query.
- K: Key.
- V: Value.
During autoregressive generation, the K and V of historical tokens do not change when new tokens are added, provided that computation conditions such as model parameters, historical tokens, and positions remain unchanged.
Therefore, the already computed K and V can be cached.
When generating a new token, only the Q, K, and V of the current token need to be computed, and the new K and V are appended to the cache.
3. KV Cache Workflow
flowchart TB
A["Input Prompt"]
B["Prefill Computation"]
C["Generate Past K / V"]
D["Save KV Cache"]
E["Input Latest Token"]
F["Compute Current Q / K / V"]
G["Read Past KV Cache"]
H["Perform Attention"]
I["Generate Next Token"]
J["Append Current K / V"]
K{"Continue Generating?"}
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H --> I
I --> J
J --> K
K -->|"Yes"| E
K -->|"No"| L["End Inference"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
4. KV Cache Memory Overhead
KV Cache reduces repeated computation, but it requires additional GPU memory to store historical K and V.
A simplified formula for estimating memory usage:
Where:
- L: Number of Transformer layers.
- B: Batch Size.
- T: Cached sequence length.
- H_{KV}: Number of Key/Value Heads.
- D_{head}: Dimension of each KV Head.
- S: Number of bytes per element.
- 2: Represents Key and Value, respectively.
For example:
- Layers: 32.
- Batch Size: 1.
- Context length: 8192.
- Number of KV Heads: 32.
- Head Dimension: 128.
- Data type: FP16, 2 bytes per element.
Then:
This is approximately:
This is the theoretical estimate for the full KV Cache of a single request, excluding framework management overhead, memory alignment, and other caches.
If GQA or MQA is used, the number of KV Heads decreases, and KV Cache usage is usually significantly reduced.
5. Limitations of KV Cache
KV Cache is not free.
It:
- Occupies additional GPU memory.
- Grows with context length.
- Increases with the number of concurrent requests.
- Adds complexity to cache allocation and management.
Therefore, LLM inference frameworks usually need to combine:
- PagedAttention.
- Continuous Batching.
- KV Cache quantization.
- GQA / MQA.
- Prefix Caching.
These techniques further optimize the inference system.
VIII. Step 6: What Are Prefill and Decode?
LLM inference is usually divided into two main stages:
- Prefill.
- Decode (token-by-token decoding).
The computational characteristics of the two differ significantly.
1. Prefill Stage
Prefill handles the complete input Prompt.
For example:
Please explain the attention mechanism of Transformer and explain why it can handle long-distance dependencies.
The Tokenizer converts it into a set of Token IDs.
The model performs a forward pass over the entire Prompt, obtains the hidden state at each position, and generates the corresponding K and V caches for each layer.
Characteristics of the Prefill phase:
- Processes multiple input Tokens at once.
- Can compute representations at different positions in the sequence in parallel.
- Requires a large number of matrix operations.
- Attention computation is closely related to the input sequence length.
- Establishes the KV Cache for the subsequent Decode phase.
Although the Causal Mask restricts the context each position can access, computations at different positions can still be executed in parallel through matrix operations.
2. Decode Phase
After Prefill is complete, the model begins generating Tokens step by step.
For example:
Prompt:
Transformer is a kind of
Generation:
neural
network
architecture
.
Each Decode step usually only needs to process the most recently generated Token.
The model uses the KV Cache to read historical K and V, completes the Attention computation for the current Token, and then predicts the next Token.
Characteristics of the Decode phase:
- Autoregressive generation.
- Usually executed Token by Token.
- Depends on the result generated in the previous step.
- Uses the KV Cache to avoid recomputing historical K and V.
- Is relatively sensitive to memory bandwidth, cache access, and single-step latency.
3. Comparison of Prefill and Decode
| Comparison Dimension | Prefill | Decode |
|---|---|---|
| Input | Full Prompt | Latest Token |
| Computation method | Multiple Tokens in parallel | Step-by-step generation |
| Primary goal | Process the input context | Generate subsequent Tokens |
| KV Cache | Establish cache | Read and append to cache |
| Computation characteristics | Large-scale matrix computation | Small-batch incremental computation |
| Performance focus | Prompt processing throughput | Token generation latency and throughput |
4. The Complete Relationship Between the Two Phases
flowchart TB
A["User submits Prompt"]
B["Tokenizer encoding"]
C["Prefill phase"]
D["Compute all Prompt Tokens"]
E["Establish KV Cache"]
F["Decode phase"]
G["Read KV Cache"]
H["Generate the next Token"]
I["Update KV Cache"]
J{"Whether the stop condition is reached"}
K["Return final text"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H --> I
I --> J
J -->|"No"| F
J -->|"Yes"| K
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note that Prefill and Decode are a division of computation phases, not two different models.
They use the same set of model parameters; only the input form and computation method differ.
IX. Step 7: When Does a Large Model Stop Generating?
A model does not generate Tokens indefinitely; there must be explicit stop conditions.
Common stopping methods include:
1. EOS Token
EOS (End of Sequence) is the end-of-sequence marker.
When the model generates the EOS Token, the inference system can determine that the current generation task has ended.
Different models may use different end Tokens or special Tokens.
2. Maximum Generation Length
Inference APIs typically support:
max_new_tokens = 1024
This means generating at most 1024 new Tokens.
Even if the model does not generate EOS, it will stop after reaching the length limit.
3. Stop Sequences
Some inference APIs support custom stop strings.
For example:
<END>
When the output meets a stop condition, the system terminates the current generation.
The specific implementation may check the stop condition at the token level or after text decoding.
4. Other Stop Conditions
These may also include:
- Request cancellation.
- Timeout.
- Server-side resource limits.
- End markers for specific structured outputs.
5. Complete Stop Check
flowchart TB
A["Generate a token"]
B{"Has EOS been generated?"}
C{"Has max_new_tokens been reached?"}
D{"Does it match a Stop Sequence?"}
E{"Is the request cancelled?"}
F["Continue generation"]
G["End generation"]
A --> B
B -->|"Yes"| G
B -->|"No"| C
C -->|"Yes"| G
C -->|"No"| D
D -->|"Yes"| G
D -->|"No"| E
E -->|"Yes"| G
E -->|"No"| F
F --> A
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#E5F0E9,stroke:#83A995,color:#315D49