1. Why Do We Need to Understand Inference Performance Metrics?
Large model inference performance depends not only on how fast the model generates an answer, but also on how long users have to wait, how many requests the system can process per second, and whether the service can remain stable as concurrency increases.
For example, two model services may behave completely differently:
- Service A: After the user sends a request, it starts returning content within 200 ms, but each subsequent Token takes 100 ms to generate.
- Service B: The user has to wait 1 s to see the first Token, but after that each Token takes only 20 ms.
If the user only needs simple Q&A, Service A may provide a better experience; if the user needs to generate long-form content, Service B may have an advantage.
Therefore, large model inference performance cannot be evaluated with just a single number.
Usually, the following metrics need to be considered:
| Metric | English | Meaning | Primary Measurement Target |
|---|---|---|---|
| Inference Latency | Latency | Time taken for a request from being sent to completion | Single-request response speed |
| First Token Latency | TTFT | Time from when the request is sent until the first Token is returned | First-response speed |
| Inter-Token Latency | TPOT | Average time interval between adjacent Tokens during generation | Sustained generation speed |
| Throughput | Throughput | Amount of data or number of requests processed per unit time | System processing capacity |
| Request Throughput | Request Throughput | Number of requests completed per unit time | Request processing capacity |
| Token Throughput | Token Throughput | Number of Tokens generated per unit time | Token processing capacity |
| End-to-End Latency | E2E Latency | Time from when the request is sent until the complete response ends | Complete interaction experience |
Among them, TTFT and TPOT are the two key metrics for understanding the streaming inference performance of large models.
2. Inference Latency
2.1 What Is Inference Latency?
Inference Latency refers to the time consumed by a model service to process one inference request.
In a complete online inference system, latency usually includes not only model computation time, but also stages such as request transmission, queuing, preprocessing, model execution, and response transmission.
The latency of a request can be broken down as:
Where:
- T_{\text{network}}: request transmission and network communication time.
- T_{\text{queue}}: time a request waits on the server side for scheduling.
- T_{\text{preprocess}}: Tokenizer, input validation, template assembly, and other preprocessing time.
- T_{\text{inference}}: model execution time.
- T_{\text{postprocess}}: result parsing, post-processing, and formatting time.
- T_{\text{return}}: time for the response to be transmitted to the client.
In actual deployments, there may also be additional time overhead from authentication, rate limiting, cache lookup, tool calls, and multi-model routing.
2.2 Composition of Inference Latency
Take a typical large model request as an example:
flowchart TD
A["Client sends request"] --> B["Network access"]
B --> C["Request queuing and scheduling"]
C --> D["Input preprocessing"]
D --> E["Model Prefill"]
E --> F["Model Decode"]
F --> G["Result post-processing"]
G --> H["Response returned"]
H --> I["Request completed"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note that in streaming response scenarios, the response is not necessarily returned only after the model has finished generating all content.
After the model generates the first token, it can send it to the client. The model then continues generating tokens and returns content incrementally.
Streaming inference therefore needs to be broken down further into TTFT, TPOT, and end-to-end latency.
2.3 Average Latency and Percentile Latency
Average latency is the arithmetic mean of the latencies of all requests:
Where:
- N: the number of requests.
- T_i: the latency of the i-th request.
However, averages tend to hide long-tail requests.
For example:
| Request | Latency |
|---|---|
| Request 1 | 100ms |
| Request 2 | 120ms |
| Request 3 | 130ms |
| Request 4 | 150ms |
| Request 5 | 2000ms |
The average latency is 500ms, but most requests actually complete within 200ms.
Production environments therefore usually also look at percentile latencies:
- P50: 50% of requests have a latency no higher than this value; also called the median.
- P90: 90% of requests have a latency no higher than this value.
- P95: 95% of requests have a latency no higher than this value.
- P99: 99% of requests have a latency no higher than this value.
- P99.9: 99.9% of requests have a latency no higher than this value.
For example:
| Metric | Latency |
|---|---|
| P50 | 180ms |
| P90 | 350ms |
| P95 | 500ms |
| P99 | 1200ms |
This means the system performs well on typical requests, but a small number of requests still have to wait longer.
Engineering practice: For online services, P95 and P99 often reflect the real user experience better than average latency.
III. TTFT: Time to First Token
3.1 What Is TTFT?
TTFT (Time To First Token) is the time elapsed from when the user sends a request to when the model returns the first generated token.
It primarily measures how long users must wait before they start seeing the model's response.
For large model services, TTFT is typically affected by the following factors:
- Queuing time after the request enters the system.
- Input tokenization and preprocessing time.
- Computation time of the prefill stage.
- Computation time of the first decode step.
- Sampling and transmission time of the first token.
3.2 The Relationship Between TTFT and Prefill
Large model inference typically consists of two stages:
- Prefill: process the full input prompt and compute the hidden states and KV cache for the input tokens.
- Decode: generate subsequent tokens one at a time based on the existing context.
The prefill stage must process the entire input sequence.
Suppose the user inputs 2,000 tokens; the model must complete the forward computation for these tokens before it can start generating the first token.
Therefore, the longer the prompt, the larger the prefill computation usually is, and the higher the TTFT may be.
flowchart TD
A["User submits prompt"] --> B["Request queuing"]
B --> C["Tokenization and preprocessing"]
C --> D["Prefill: Process input tokens"]
D --> E["Build KV Cache"]
E --> F["Execute first Decode step"]
F --> G["Generate first token"]
G --> H["Return first token"]
H --> I["TTFT ends"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Strictly speaking, the start and end points of TTFT timing need to be defined consistently in the load testing tool.
For example, the client may start timing from when it initiates the request, while the server may start timing when the request enters the inference engine. The TTFT values obtained by the two may not be the same.
3.3 Main Factors Affecting TTFT
| Factor | Impact mechanism | Optimization direction |
|---|---|---|
| Prompt length | More input tokens usually mean more Prefill work | Shorten context and compress Prompt |
| Request queuing | As concurrent requests increase, scheduling wait time rises | Continuous batching, appropriate rate limiting |
| Model size | As parameter count increases, compute and memory access overhead increase | Model optimization, quantization |
| GPU compute | Prefill computation is limited by hardware performance | Use higher-performance GPUs |
| KV Cache allocation | Insufficient GPU memory may cause waiting or preemption | Optimize GPU memory management |
| Prefill scheduling | Large Prefill tasks may affect other requests | Chunked Prefill |
| Network transmission | First-token return is affected by the network and service path | Optimize the network and response path |
3.4 Key Areas for TTFT Optimization
TTFT optimization is not equivalent to simply improving Decode speed.
This is because the main computational cost of TTFT is often concentrated in request queuing and the Prefill stage.
Common optimization methods include:
- Prompt compression: Reduce ineffective context and duplicate content.
- Prefix Caching: Reuse the KV Cache for identical prefixes to reduce repeated Prefill.
- Chunked Prefill: Split long Prompts into multiple computation chunks to prevent a single long Prefill from occupying the GPU for a long time.
- Continuous Batching: Dynamically schedule requests to reduce waiting time.
- Quantization and efficient kernels: Reduce model computation and memory access costs.
- Request priority scheduling: Provide lower waiting time for interactive requests.
- Load balancing: Avoid concentrating requests on a few GPU instances.
Note that Prefix Caching delivers benefits only when prefixes can be matched and the cache is still available; it does not mean that all Prompts can skip Prefill.
4. TPOT: Generation Time per Output Token
4.1 What Is TPOT?
TPOT (Time Per Output Token) represents the average time required per token when the model generates output tokens.
It mainly measures the sustained generation speed during the Decode stage.
The common calculation is:
Where:
- T_{\text{decode}}: The time from after the first token to the generation of the last token.
- N_{\text{output}}: The number of output tokens.
The reason for subtracting 1 is that the first token is already covered by TTFT, and TPOT mainly calculates the average generation time between subsequent tokens.
For example:
The model generates 101 tokens:
- First token latency: 500 ms.
- Time from after the first token to the last token: 2000 ms.
Then:
That is, the model generates one subsequent token every 20 ms on average.
4.2 TPOT and Token Generation Speed
TPOT and tokens per second (TPS) have an inverse relationship.
For a single request with a stable, continuous generation process:
For example:
| TPOT | Approximate generation speed |
|---|---|
| 100ms | 10 Token/s |
| 50ms | 20 Token/s |
| 25ms | 40 Token/s |
| 20ms | 50 Token/s |
| 10ms | 100 Token/s |
Note: the TPS here is the average generation speed of a single request, not the Token throughput of the entire inference cluster.
4.3 TPOT and the Decode Phase
The Decode phase has a clear autoregressive character:
When the model generates the next Token, it must depend on the Tokens already generated before it.
A typical process is as follows:
flowchart TD
A["Existing Prompt and KV Cache"] --> B["Decode Step 1"]
B --> C["Generate Token 1"]
C --> D["Update context and KV Cache"]
D --> E["Decode Step 2"]
E --> F["Generate Token 2"]
F --> G["Update context and KV Cache"]
G --> H["Decode Step 3"]
H --> I["Continue generating subsequent Tokens"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Each Decode Step typically requires:
- Reading the model weights.
- Reading the KV Cache corresponding to historical Tokens.
- Executing the model forward computation.
- Computing the probability distribution of the next Token.
- Executing the sampling or decoding strategy.
- Writing the KV information corresponding to the new Token into the cache.
As the context keeps growing, the KV Cache also keeps growing, and the memory-access cost of attention computation may rise accordingly.
4.4 Main Factors Affecting TPOT
| Factor | Mechanism of impact |
|---|---|
| Number of model parameters | The more parameters, the higher the weight-reading and computation cost usually is |
| GPU memory bandwidth | The Decode phase is often limited by weight and KV Cache memory access |
| KV Cache length | The longer the context, the greater the overhead of attention computation and cache reads may be |
| Number of concurrent requests | Changes in batch size affect the generation speed of a single request |
| Batch Size | A larger batch may improve GPU utilization, but may also increase per-request waiting |
| Attention Kernel | Different Kernels have different memory-access and computational efficiency |
| Quantization method | May reduce weight memory access, but also introduces dequantization overhead |
| Scheduling strategy | Resource contention between Prefill and Decode may affect the generation interval |
4.5 The Difference Between TPOT and ITL
ITL (Inter-Token Latency) denotes the time interval between adjacent output Tokens.
TPOT and ITL are easily confused in many performance benchmarks, but they need to be distinguished:
- ITL: can describe the actual time interval between two adjacent Tokens in a particular instance.
- TPOT: usually describes the average Token interval during the generation phase of a request.
For example, a request generates 5 Tokens, with time intervals of:
The average TPOT is:
But looking only at the average would ignore the noticeably high latency of the 4th token.
Therefore, when evaluating the streaming generation experience, you can look at average TPOT, P95 ITL, and P99 ITL together.
Different inference frameworks may have different timing boundaries for TPOT and ITL, so definitions must be unified before comparing data.
5. Relationship Between TTFT, TPOT, and End-to-End Latency
5.1 How Are the Three Related?
For a complete generation request:
- TTFT: the time the user waits for the first token.
- TPOT: the average generation time of subsequent tokens after the first token.
- E2E Latency: the total time the user waits for the entire response to complete.
Assume:
- TTFT = 500ms.
- Number of output tokens = 100.
- TPOT = 20ms.
Then:
Substituting:
That is, the entire request takes about 2.48 seconds.
This formula is suitable for approximate calculation of token generation time in streaming output. Actual systems may also have network transmission, output buffering, scheduling intervals, and post-processing overhead.
5.2 Different Metrics Correspond to Different Optimization Goals
| Optimization Goal | Priority Metrics | Typical Optimization Directions |
|---|---|---|
| Users see a response as soon as possible | TTFT | Caching, shortening the prompt, reducing queuing |
| Smoother continuous reading for users | TPOT, ITL | Optimize Decode, KV Cache, kernels |
| Complete requests finish as soon as possible | E2E Latency | Optimize Prefill and Decode simultaneously |
| The system handles more requests | Request Throughput | Concurrent scheduling, load balancing |
| The system generates more tokens | Token Throughput | Batching, GPU utilization, quantization |
| Stably meeting service commitments | P95, P99 | Rate limiting, isolation, resource reservation |
A common misconception is: pursuing only lower TPOT while ignoring TTFT.
For example, a system optimizes Decode to be very fast, but because queuing in the Prefill stage is severe, users still have to wait a long time to see the first token.
Another misconception is: pursuing only lower TTFT, causing the system to adopt overly aggressive low-concurrency scheduling, which reduces GPU utilization and lowers overall throughput as well.
Therefore, metric priorities need to be determined based on the business scenario.
6. Throughput
6.1 What Is Throughput?
Throughput indicates the amount of work completed by the system per unit time.
In large model inference, common throughput metrics include:
- Request Throughput: the number of requests processed or completed per second.
- Input Token Throughput: the number of input tokens processed per second.
- Output Token Throughput: the number of output tokens generated per second.
- Total Token Throughput: the total number of input and output tokens processed per second.
These metrics measure different dimensions and cannot directly substitute for one another.
6.2 Request Throughput
Request throughput usually indicates the number of requests completed per unit time:
For example, 1200 requests completed within 60 seconds:
This means the system completes an average of 20 requests per second.
QPS (Queries Per Second) can also be used to describe the number of queries processed per second.
In large model services, RPS and QPS are often used approximately interchangeably, but they should be distinguished according to the system's actual statistical basis:
- Number of requests received.
- Number of requests successfully processed.
- Number of requests completed.
For timed-out, failed, and canceled requests, whether they are counted in throughput must be clearly specified.
6.3 Token Throughput
Token Throughput indicates the number of tokens processed or generated per unit time.
Output token throughput:
Input token throughput:
Total token throughput:
For example, within 10 seconds, the system:
- Processes 100000 input tokens.
- Generates 20000 output tokens.
Then:
Note that input token throughput and output token throughput correspond to different computation stages:
- Input token throughput mainly reflects Prefill capability.
- Output token throughput mainly reflects Decode capability.
Although adding the two together directly can describe the overall scale of token processing, it is not suitable on its own for judging Decode speed.
6.4 Single-Request TPS vs. System TPS
This is a concept that is very easy to confuse in performance testing.
Suppose the system processes 10 requests simultaneously, and each request generates an average of 20 tokens/s.
Then:
That is:
- Single-request generation speed: 20 tokens/s.
- Total system output throughput: 200 tokens/s.
In real systems, the generation speed of each request is not necessarily the same, so total system throughput should be calculated based on the number of tokens across all requests rather than simply assuming that every request has the same speed.
6.5 Throughput and Concurrency
Increasing concurrency can usually improve GPU utilization.
However, throughput does not grow indefinitely with concurrency.
When GPU compute, memory bandwidth, KV Cache capacity, or scheduling capability reaches a bottleneck, continuing to increase concurrency may lead to:
- Increased request queuing.
- Higher TTFT.
- Higher TPOT.
- Worse P95 and P99 latency.
- OOM or request failures.
- System throughput entering a plateau or even declining.
The typical relationship can be represented as follows:
flowchart TD
A["Increase concurrent requests"] --> B["Expand batch size"]
B --> C["GPU utilization improves"]
C --> D["System throughput grows"]
D --> E{"Resource bottleneck reached?"}
E -->|"No"| F["Continue increasing throughput"]
E -->|"Yes"| G["Request queuing increases"]
G --> H["TTFT and TPOT degrade"]
H --> I["Tail latency rises"]
I --> J["Throughput plateaus or declines"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#EAE8FA,stroke:#A49BD5,color:#51477D
VII. Relationship Between Throughput and Latency
7.1 Throughput and Latency Are Not the Same Metric
Throughput measures the overall processing capability of the system, while latency measures the waiting time of a single request.
For example:
| System | Concurrent requests | Average TTFT | Single-request generation speed | System output throughput |
|---|---|---|---|---|
| A | 1 | 200ms | 50 tokens/s | 50 tokens/s |
| B | 10 | 500ms | 30 tokens/s | 300 tokens/s |
| C | 30 | 1500ms | 15 tokens/s | 450 tokens/s |
From the table, we can see that:
- A has a better single-request response experience.
- C has higher overall system output throughput.
- However, C's single-request experience is noticeably worse.
Therefore, service experience cannot be judged solely by system TPS, nor can cluster processing capability be judged solely by single-request TPOT.
7.2 Little's Law: Request Volume, Concurrency, and Latency
Little's Law is an important relationship in queuing theory:
Where:
- L: Average number of requests in the system, which can also be understood as the average number of in-flight requests.
- \lambda: Average request arrival rate.
- W: Average time a request stays in the system.
For example:
- 20 requests enter per second.
- Each request stays in the system for an average of 2 seconds.
Then:
On average, the system needs to accommodate 40 in-flight requests at the same time.
Little's Law applies to stable systems and requires consistent measurement definitions. It does not mean that the higher the concurrency, the higher the throughput; rather, it shows that there is a constraint relationship among the request arrival rate, the number of in-flight requests, and the average residence time.
7.3 Latency-Throughput Trade-off
System optimization usually involves the following relationships:
| Scheduling strategy | Advantages | Potential issues |
|---|---|---|
| Small batch, low concurrency | Lower per-request latency | GPU utilization and throughput may be insufficient |
| Large batch, high concurrency | Improves resource utilization and overall throughput | Per-request latency may increase |
| Dynamic batching | Adjusts batch size based on request changes | Scheduler implementation is complex |
| Continuous Batching | Requests can dynamically join and exit | Requires handling KV Cache and scheduling overhead |
| Priority scheduling | Guarantees critical requests | Low-priority requests may wait too long |
A reasonable goal is not to optimize a single metric to the extreme, but to find an appropriate performance balance point under business constraints.
8. Impact of Prefill and Decode on Performance Metrics
The two main stages of large model inference have different computational characteristics, so their impact on performance metrics also differs.
8.1 Prefill Stage
Prefill is responsible for processing the input prompt.
Main characteristics:
- Processes multiple input tokens at once.
- Has high parallelism.
- Computation usually increases with input length.
- Makes it easier to leverage the GPU's parallel computing capability.
- Mainly affects TTFT and input token throughput.
8.2 Decode Stage
Decode is responsible for generating output token by token.
Main characteristics:
- Autoregressive generation, with dependencies between tokens.
- Usually generates only one new token each time.
- Requires continuous reads of model weights and KV Cache.
- In many inference scenarios, it is limited by memory bandwidth.
- Mainly affects TPOT and output token throughput.
8.3 Performance Comparison of Prefill and Decode
| Dimension | Prefill | Decode |
|---|---|---|
| Input | Entire prompt | Existing context and latest token |
| Output | Initial KV Cache and the state required for the first generatable result | New token and incremental KV Cache |
| Computation mode | Large-scale parallelism | Step-by-step autoregressive |
| Common bottleneck | GPU compute capability | Memory bandwidth, KV Cache memory access |
| Main metrics | TTFT, input TPS | TPOT, output TPS |
| Optimization focus | Chunked Prefill, FlashAttention | KV Cache, continuous batching, quantization |
8.4 Resource Competition Between the Two Stages
When the system handles many requests simultaneously, Prefill and Decode may compete for GPU compute resources.
For example, a very long prompt entering the system may consume more compute resources, causing requests that are currently generating content to experience increased token intervals.
This situation may cause:
- TTFT increases: new requests wait for Prefill resources.
- TPOT increases: Decode of existing requests is disrupted.
- P95 and P99 latency increase: obvious performance differences appear among different requests.
Therefore, inference engines usually need to coordinate the execution order of Prefill and Decode.
Common techniques include:
- Chunked Prefill: Process long prompts in chunks.
- Continuous Batching: Dynamically schedule requests after each Decode round.
- Prefill-Decode Disaggregation: Execute Prefill and Decode on different compute resources.
- Token Budget: Limit the number of tokens that can be processed in each scheduling round.
9. Core Performance Optimization Techniques
9.1 Continuous Batching
Traditional static batching must wait for all requests in a batch to finish before proceeding to the next batch.
Continuous Batching, by contrast, allows requests to dynamically join or exit after each generation round.
Main benefits:
- Batch slots are released immediately after a request completes.
- New requests can fill idle slots.
- Reduces GPU idling caused by differing request lengths.
- Improves overall system token throughput.
However, Continuous Batching does not mean that latency will decrease for all requests. Its core value is improving resource utilization; actual TTFT and TPOT still depend on the scheduling strategy and resource contention.
9.2 PagedAttention and KV Cache Management
KV Cache is used to store the Key and Value of historical Tokens, avoiding recomputing all historical states each time a Token is generated.
PagedAttention manages KV Cache in blocks, reducing the fragmentation caused by contiguous memory allocation.
It can help the inference engine:
- Dynamically allocate KV Cache Blocks.
- Release the cache of completed requests on demand.
- Improve memory utilization.
- Support more concurrent requests.
- Reduce resource waste caused by memory fragmentation.
Its main role is to optimize KV Cache memory management, and it should not be confused with Continuous Batching.
The two are usually used together:
- Continuous Batching handles request scheduling.
- PagedAttention handles KV Cache memory management.
9.3 Prefix Caching
For requests that reuse the same prefix, the previously computed KV Cache can be reused.
For example, when multiple requests share the same system prompt or long document context, repeated Prefill work can be reduced.
Main benefits:
- Lower TTFT.
- Reduce repeated computation.
- Improve Prefill throughput.
Attention should be paid to the cache hit rate, cache capacity, prefix matching rules, and cache eviction policy.