Back to series
· 12 min read

Streaming Response Mechanism: How Do Large Language Models Achieve Real-Time Token-by-Token Output?


1. What Is Streaming?

Streaming (streaming response) is a communication mechanism that continuously transmits data to the client in chunks.

In the traditional non-streaming response mode, the server usually has to wait for the model to finish the entire response before returning the complete result to the client.

For example, the user asks:

Please explain how the Transformer works.

The model may need to generate hundreds or even thousands of tokens. In non-streaming mode, the user may have to wait until the entire response has been generated before seeing any content.

Streaming is different.

As the model generates each portion of content, the server can send it to the client, and the client continuously receives and displays it.

The main differences between the two modes are:

Comparison Dimension Non-Streaming Response Streaming Response
Delivery method Returns the complete result all at once Returns continuously in chunks
When content first becomes visible Usually requires waiting for full generation Can be displayed earlier
User experience Can feel like a long wait Closer to real-time interaction
Connection state Ends after the request completes Keeps the connection open during generation
Data processing Parsed all at once Parsed and accumulated continuously
Use cases Short text, batch processing Conversation, long text, real-time interaction

Two concepts should be distinguished:

  • Model generation: The model predicts and generates tokens step by step.
  • Streaming transmission: The server sends the already-generated content to the client in chunks.

Token-by-token generation inside the model does not mean the network must send one token at a time. The server can buffer and combine multiple tokens into a single data chunk for transmission.

2. The Overall Streaming Workflow

A typical large-model streaming response system includes a client, an application service, a model inference service, and a network transport layer.

flowchart TD
    A["User submits request"] --> B["Client initiates HTTP request"]
    B --> C["Application service receives request"]
    C --> D["Calls model inference service"]
    D --> E["Model performs Prefill"]
    E --> F["Model decodes step by step"]
    F --> G["Generates a new Token"]
    G --> H["Server assembles response data chunk"]
    H --> I["Sends over streaming connection"]
    I --> J["Client receives data chunk"]
    J --> K["Parses and updates UI"]
    K --> L{"Is generation complete?"}
    L -->|"No"| F
    L -->|"Yes"| M["Sends end event"]
    M --> N["Closes streaming connection"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#EAF3EC,stroke:#83A995,color:#315D49
    style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The whole process can be divided into several key stages:

  1. Establish connection: The client initiates a streaming request to the server.
  2. Model inference: The model processes the input and begins generating content.
  3. Data chunking: The server packages the generated content into one or more data chunks.
  4. Continuous transmission: Data chunks are sent continuously over the network connection.
  5. Client parsing: The client parses the data chunks and updates the UI.
  6. Completion handling: The model finishes generation, and the server sends an end marker and closes the connection.

Streaming does not change the model's core generation algorithm; it mainly changes how inference results are delivered.

3. Streaming and the Token Generation Mechanism

To understand Streaming, you first need to understand the autoregressive generation mechanism of large language models.

1. The Model Generates Token by Token

Taking a Decoder-only Transformer as an example, the model predicts the next Token based on the existing context.

Suppose the user input is:

The development direction of artificial intelligence is

The model may generate the following in sequence:

Multimodal
integration
and
agent
applications
.

Its generation process can be abstracted as:

flowchart TD
    A["Input Prompt"] --> B["Transformer computation"]
    B --> C["Generate next Token"]
    C --> D["Add Token to context"]
    D --> E{"Stop condition met?"}
    E -->|"No"| B
    E -->|"Yes"| F["Generation complete"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Each Decode round typically generates one new Token, which is then added to the context used for subsequent generation.

In actual inference, the model also uses KV Cache to reuse historical computation results, avoiding recomputing the Key and Value of historical Tokens.

2. A Token Is Not a Character

A Token may correspond to:

  • A complete word.
  • A Chinese character.
  • Multiple characters.
  • A word fragment.
  • Punctuation mark.
  • A special control marker.

For example, the following text:

Streaming response

may be split into multiple Tokens rather than generated one by one according to spaces or characters.

Therefore, a data chunk in Streaming does not necessarily correspond to a complete word, nor does it necessarily correspond to a complete sentence.

3. Token Generation and Network Transmission Are Not the Same Thing

This is the most common misunderstanding when trying to understand Streaming.

The model can generate Token by Token, but the inference service usually does not guarantee that network data is sent immediately every time a Token is generated.

Reasons include:

  • Network transmission has fixed overhead.
  • Multiple small packets may reduce transmission efficiency.
  • The server may perform buffering.
  • The inference framework may process output in batches.
  • Gateways, proxies, and clients may further buffer data.

For example, the model continuously generates:

Artificial
intelligence
is
changing
software
development
.

The server may merge these Tokens into:

Artificial intelligence is changing software development.

and then send it to the client.

Therefore:

A Token is the basic unit of model generation, while a Chunk is the data unit of streaming transmission; the two are not equivalent.

4. Core Transport Mechanisms of Streaming

Streaming is not just a model output method; it also involves application protocols, network transmission, and client-side parsing.

Common implementation methods include:

  1. HTTP Chunked Transfer Encoding.
  2. Server-Sent Events (SSE).
  3. WebSocket.
  4. Other streaming transmission protocols based on HTTP.

These technologies operate at different layers and cannot simply be regarded as the same protocol.

1. HTTP Chunked Transfer Encoding

HTTP Chunked Transfer Encoding is a chunked transfer encoding mechanism in HTTP/1.1.

In traditional HTTP responses, the server can usually specify the full length of the response body via Content-Length.

However, for dynamically generated content, the server often cannot determine the final response body length when generation begins.

Chunked Transfer Encoding allows the server to split the response body into multiple data chunks and send them step by step without needing to know the full length in advance.

Illustration:

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

5
Hello
6
 World
0

Here:

  • Each data chunk is preceded by the hexadecimal byte length of that chunk.
  • 5 indicates that the following data chunk contains 5 bytes.
  • 6 indicates that the following data chunk contains 6 bytes.
  • 0 indicates the end of response body transmission.

Note:

HTTP/1.1 Chunked Transfer Encoding is a transfer encoding mechanism and is not equivalent to SSE.

HTTP/2 does not use the HTTP/1.1 Transfer-Encoding: chunked mechanism; instead, it carries response data through HTTP/2 DATA Frames.

HTTP/3, in turn, transmits data based on QUIC.

The streaming response logic at the application layer can remain similar, but the underlying transport method may differ.

2. Server-Sent Events (SSE)

SSE is an HTTP-based mechanism for the server to push events unidirectionally to the client.

In LLM applications, SSE is a very common way to implement streaming responses.

Typical response headers:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

SSE uses a specific text format to structure events.

For example:

data: {"content":"Artificial intelligence"}

data: {"content":"is transforming"}

data: {"content":"software development"}

data: [DONE]

Each event usually ends with a blank line.

Common fields supported by SSE include:

Field Purpose
data Event data
event Event type
id Event identifier
retry Suggested client reconnection wait time

For example:

event: message
id: 101
data: {"content":"Hello"}

Clients can receive SSE events through the browser's EventSource interface.

However, the native EventSource is mainly suitable for GET requests and cannot freely set arbitrary request headers.

If an LLM API needs to send complex JSON requests via POST, the frontend usually uses fetch() to read the response stream, or performs SSE forwarding through a backend proxy.

3. WebSocket

WebSocket is a network protocol that supports bidirectional communication.

Unlike SSE, after a WebSocket connection is established, both the client and server can actively send messages.

Typical interaction:

flowchart TD
    A["Client initiates connection request"] --> B["HTTP Upgrade handshake"]
    B --> C["Establish WebSocket connection"]
    C --> D["Client sends message"]
    D --> E["Server processes request"]
    E --> F["Server sends streaming data"]
    F --> G["Client continuously receives"]
    G --> H{"Continue interaction?"}
    H -->|"Yes"| D
    H -->|"No"| I["Close WebSocket connection"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

WebSocket is better suited for:

  • Real-time voice interaction.
  • Bidirectional real-time communication.
  • Multi-turn continuous interaction.
  • Scenarios where the client needs to send control messages at any time.
  • Real-time state synchronization.

But WebSocket is not a mandatory solution for LLM streaming responses.

For ordinary text generation, HTTP SSE is usually sufficient.

4. Differences Between SSE and WebSocket

Comparison Dimension SSE WebSocket
Communication direction One-way push from server to client Bidirectional communication
Underlying transport HTTP WebSocket protocol
Data format UTF-8 text events Text or binary messages
Connection method HTTP long connection Connection established after Upgrade handshake
Automatic reconnection Supported by browser EventSource Usually requires application-layer implementation
Complexity Relatively simple Relatively complex
Typical scenarios Text streaming responses Real-time bidirectional interaction

When choosing, one should not simply assume that WebSocket is necessarily faster than SSE.

Factors that truly affect latency also include model inference speed, network quality, server-side buffering, proxy configuration, and client processing capability.

V. Server-Side Implementation of Streaming

The server needs to accomplish three core tasks:

  1. Receive the data generated by the model.
  2. Convert the data into a streaming response format.
  3. Continuously write to the HTTP response.

Using Python as an example, you can use FastAPI to build a simplified SSE service.

1. Basic SSE Service

import asyncio
import json

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()


async def generate_text():
    tokens = [
        "Streaming",
        " 是",
        " 一种",
        " 流式",
        "响应",
        "机制",
        "。"
    ]

    for token in tokens:
        event = {
            "content": token
        }

        yield f"data: {json.dumps(event, ensure_ascii=False)}\n\n"

        await asyncio.sleep(0.2)

    yield "data: [DONE]\n\n"


@app.get("/stream")
async def stream():
    return StreamingResponse(
        generate_text(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no"
        }
    )

Here, generate_text() is an asynchronous generator.

Each time yield is executed, a segment of data is provided to the response stream.

StreamingResponse is responsible for continuously sending this data to the client.

Several key points:

  • async def: Defines an asynchronous function.
  • yield: Produces response data incrementally.
  • text/event-stream: Declares the SSE content type.
  • \n\n: Marks the end of an SSE event.
  • X-Accel-Buffering: no: When using Nginx, this can be used to disable proxy buffering for this response.

The asyncio.sleep() in the example is only used to simulate the interval between token generations; a real system should consume the stream produced by the model inference service.

2. Connecting to a Real Model Inference Service

In real projects, the server usually needs to call the model API and forward the streaming data returned by the model to the frontend.

Overall structure:

flowchart TD
    A["Frontend request"] --> B["FastAPI service"]
    B --> C["Call model API"]
    C --> D["Receive model stream"]
    D --> E["Parse model events"]
    E --> F["Extract text or structured content"]
    F --> G["Wrap as SSE events"]
    G --> H["Continuously send to frontend"]
    H --> I["Frontend parses and displays"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Production environments also need to handle:

  • Connection timeouts to the model API.
  • Releasing resources after the client disconnects.
  • Model generation exceptions.
  • Proxy-layer buffering.
  • Request state for multi-turn conversations.
  • End events and error events.
  • Terminating the inference task after the client cancels the request.

Pay special attention to this: the client disconnecting does not mean the model inference service will necessarily stop generating automatically. The application needs to pass the cancellation signal to the underlying model request to avoid continuing to consume GPU resources meaninglessly.

VI. How Does the Client Receive Streaming Data?

The server is responsible for continuously sending, while the client needs to continuously read, parse, and update the page.

1. Reading the Response Stream with Fetch

The browser's fetch() can read an HTTP response body via ReadableStream.

Simplified example:

async function streamResponse() {
    const response = await fetch("/stream");

    if (!response.ok) {
        throw new Error("请求失败");
    }

    if (!response.body) {
        throw new Error("浏览器不支持响应流");
    }

    const reader = response.body.getReader();
    const decoder = new TextDecoder();

    let buffer = "";

    while (true) {
        const { value, done } = await reader.read();

        if (done) {
            break;
        }

        buffer += decoder.decode(value, {
            stream: true
        });

        const events = buffer.split("\n\n");

        buffer = events.pop() ?? "";

        for (const event of events) {
            const dataLine = event
                .split("\n")
                .find(line => line.startsWith("data:"));

            if (!dataLine) {
                continue;
            }

            const data = dataLine.slice(5).trim();

            if (data === "[DONE]") {
                return;
            }

            const result = JSON.parse(data);

            console.log(result.content);
        }
    }

    buffer += decoder.decode();

    if (buffer.trim()) {
        // Handle the last incomplete event according to the actual protocol
    }
}

This involves several important APIs:

API Purpose
response.body Get the response body stream
getReader() Get a stream reader
reader.read() Read the next data chunk
TextDecoder Decode bytes into text
decode() Decode byte data
stream: true Preserve decoding state for incomplete characters

2. Why do we need a buffer?

This is a very error-prone area in Streaming client development.

Network data chunks do not necessarily correspond one-to-one with SSE events.

For example, the server sends:

data: {"content":"Hello"}

data: {"content":"World"}

The client may receive any of the following forms:

Case 1: A complete event is received at once

data: {"content":"Hello"}

Case 2: Multiple events are received at once

data: {"content":"Hello"}

data: {"content":"World"}

Case 3: One event is split across multiple data chunks

First:

data: {"cont

Second:

ent":"Hello"}

If the client parses JSON directly after every reader.read(), parsing can fail.

This is because read() returns transport data chunks, which are not guaranteed to correspond to complete SSE events, and are even less guaranteed to correspond to complete JSON objects.

The correct approach is:

  1. Append new data to the buffer.
  2. Look for complete events using the SSE event delimiter.
  3. Parse only events that have been fully received.
  4. Keep the last incomplete fragment.
  5. Wait for subsequent data to continue assembling it.

Note that a real SSE parser should also correctly handle event, id, multi-line data, comment lines, CRLF line endings, and a final event that is not yet fully delimited when the stream ends.

In production, prefer a proven SSE parsing library rather than relying long-term on simplified string-splitting logic.

3. Why does TextDecoder need stream mode?

What travels over the network is bytes, not naturally complete UTF-8 characters.

A single Chinese character may consist of multiple UTF-8 bytes.

Suppose the byte sequence of a character is split into two parts by the network:

Chunk A: the first two bytes of the character
Chunk B: the remaining byte of the character

If each chunk is decoded independently, the first chunk may produce a replacement character because the character is incomplete.

Using:

decoder.decode(value, {
    stream: true
});

lets the decoder retain the bytes of an unfinished character and wait for subsequent data to arrive.

Therefore, a streaming client needs to handle two kinds of boundaries at the same time:

  • UTF-8 character boundaries.
  • SSE event boundaries.

These are boundaries at different levels, and they must not be conflated.

VII. The Relationship Between Streaming, Prefill, and Decode

Streaming is a transport-layer mechanism, while Prefill and Decode are computational phases in the LLM inference process.

The two belong to different layers, but they are closely related.

1. The Prefill Phase

Prefill is responsible for processing the full input prompt.

Its main work includes:

  • Tokenizer encoding.
  • Embedding mapping.
  • Transformer layer computation.
  • Generating the KV cache corresponding to the prompt.
  • Computing the predicted distribution for the first output token.

Prefill usually has to process the entire input sequence, so its computational cost is closely tied to prompt length.

In a typical autoregressive generation flow, the model must complete Prefill before it can begin generating the first output token.

2. The Decode Phase

Decode is responsible for generating subsequent tokens step by step.

Each Decode round typically:

  • Takes the new token that needs to be processed.
  • Reads the historical KV cache.
  • Performs Transformer computation.
  • Generates the next token.
  • Updates the KV cache.

The Decode phase is the main source of streamed text content.

3. How Streaming Maps to the Inference Phases

flowchart TD
    A["User submits prompt"] --> B["Tokenizer encoding"]
    B --> C["Prefill phase"]
    C --> D["Build KV cache"]
    D --> E["Compute first output token"]
    E --> F["Server sends first batch of data"]
    F --> G["Decode phase"]
    G --> H["Generate subsequent tokens"]
    H --> I["Server keeps sending data"]
    I --> J["Client keeps rendering"]
    J --> K{"Generation finished?"}
    K -->|"No"| G
    K -->|"Yes"| L["End streaming response"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F

One detail to note:

Prefill does not necessarily mean the user has to wait for all computation over the entire prompt to finish before seeing any response metadata. The server can send response headers, status events, or progress information in advance.

But for typical autoregressive text generation, the actual first generated token still requires the model to complete the necessary Prefill computation.

VIII. Key Performance Metrics for Streaming

When evaluating streaming response performance, you cannot look only at the total time of the whole request.

You need to observe model inference speed, time to first content, and network transmission behavior separately.

1. TTFT: Time To First Token

TTFT is the time elapsed from when the request starts until the client receives the first generated token.

It can be abstracted as:

TTFT = T_{\text{first token received}} - T_{\text{request start}}

TTFT typically includes: