Back to series
· 13 min read

Request and Response Mechanisms in Large Model APIs: From Request Packaging to Model Inference and Result Return


1. What Is the Request and Response Mechanism of an LLM API?

The request and response mechanism of an LLM API refers to the complete process in which the client submits model input according to the agreed protocol and data structure, the server performs authentication, parameter validation, model scheduling, and inference, and then returns the generated result to the client in the agreed format.

From an engineering perspective, it is not a simple function call but a remote procedure call (RPC) that spans the network, the server-side application, the model inference engine, and the data transmission layer.

A typical call involves the following stages:

flowchart TD
    A["Client constructs request"]
    B["HTTP request sent"]
    C["API gateway receives"]
    D["Authentication and parameter validation"]
    E["Model service scheduling"]
    F["Model executes inference"]
    G["Package response data"]
    H["HTTP response returned"]
    I["Client parses result"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

To understand this mechanism, it is necessary to distinguish three layers:

Layer Primary responsibility Focus
HTTP transport layer Handles data transmission for requests and responses Method, Headers, Body, Status Code
API application layer Handles authentication, parameter parsing, scheduling, and result packaging API Schema, Messages, Model, Usage
Model inference layer Handles the actual model computation Tokenizer, Prefill, Decode, Sampling

These three layers cooperate with one another, but their responsibilities are not the same.

For example, HTTP 200 indicates that the HTTP request was processed successfully; it does not mean the model's answer is necessarily correct. Likewise, an API returning JSON does not mean the model directly generated a JSON object internally.

2. Underlying Structure of an HTTP Request

LLM APIs are usually provided over HTTPS.

An HTTP request consists of parts such as the request line, request headers, and request body.

2.1 Request Line

The request line describes the request method, target path, and HTTP protocol version.

For example:

POST /v1/chat/completions HTTP/1.1

Here:

Part Meaning
POST HTTP request method
/v1/chat/completions Request resource path
HTTP/1.1 HTTP protocol version

Common methods include:

  • GET: Read a resource.
  • POST: Submit data or create a resource.
  • PUT: Update a resource in full.
  • PATCH: Partially update a resource.
  • DELETE: Delete a resource.

LLM inference endpoints usually use POST because they need to submit prompts, model parameters, and other input data.

2.2 Request Headers

Headers are used to convey meta-information about the request.

For example:

POST /v1/chat/completions HTTP/1.1
Host: api.example.com
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json
Accept: application/json

Common request headers:

Header Purpose
Host Target service host
Authorization Authentication
Content-Type Data type of the request body
Accept Data type the client expects to receive
User-Agent Client identifier
X-Request-ID Request tracking identifier, if supported by the service

The most important of these is:

Content-Type

Content-Type: application/json

Indicates that the request body uses JSON format.

Authorization

Authorization: Bearer YOUR_API_KEY

Indicates authentication using a Bearer Token.

The API key should not be exposed in public browser frontend or mobile app code. In production, the backend should be responsible for key management and model service calls.

2.3 Request Body

The request body carries the business data required for the model call.

A common Chat Completions-style request:

{
  "model": "example-model",
  "messages": [
    {
      "role": "system",
      "content": "你是一名专业的技术助手。"
    },
    {
      "role": "user",
      "content": "解释一下 Transformer 的工作原理。"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 1000,
  "stream": false
}

The request body contains two types of information:

  1. Task input: model name, message content, and context.
  2. Inference control parameters: temperature, output length, streaming toggle, etc.

Note that this uses a common OpenAI Chat Completions-compatible style. Different providers may use different fields, endpoint paths, and parameter definitions.

For example, the output length field may be max_tokens, max_completion_tokens, or max_output_tokens; you cannot assume they are fully interchangeable across all APIs.

III. Core Fields of the Request Body

The request body is the most important data part of a large model API. Only by understanding the field structure can you correctly control model behavior.

3.1 Model: Model Identifier

{
  "model": "example-model"
}

model specifies the model used for this request.

It usually determines:

  • Model architecture and parameter scale.
  • Supported input modalities.
  • Context window size.
  • Maximum output length.
  • Tool-calling capabilities.
  • Reasoning capabilities and latency characteristics.
  • Billing rules.

Model identifiers are usually defined by the provider, and there is no unified standard for model names across providers.

3.2 Messages: List of Chat Messages

messages describes the context for this model call.

For example:

{
  "messages": [
    {
      "role": "system",
      "content": "你是一名数据库专家。"
    },
    {
      "role": "user",
      "content": "什么是数据库索引?"
    }
  ]
}

Messages usually consist of multiple message objects.

Each message object must at least express the message source and content; the specific fields are defined by the API protocol.

Common roles include:

Role Meaning
system System-level instruction
developer Developer-level instruction
user User input
assistant Model output
tool External tool execution result

Not all providers use exactly the same set of roles or message structure.

3.3 Content: Message Content

content represents the actual content of the message.

The simplest case is a string:

{
  "role": "user",
  "content": "什么是 KV Cache?"
}

In multimodal APIs, content may also be an array composed of multiple content blocks:

{
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "请描述这张图片。"
    },
    {
      "type": "image_url",
      "image_url": {
        "url": "https://example.com/image.jpg"
      }
    }
  ]
}

This means message content is not necessarily plain text.

For vision, audio, and video models, the content structure may include:

  • Text fragments.
  • Image URLs or image data.
  • Audio data.
  • Video frames.
  • Timestamps.
  • Modality task markers.

The exact data structure depends on the model and the provider.

3.4 Temperature: Generation Randomness

Temperature controls the probability distribution during the sampling stage.

Let the model's output logits be z_i and the temperature be T; then the probability after temperature scaling is:

P(x_i)= \frac{\exp(z_i/T)}

Generally speaking:

  • The lower T is, the sharper the probability distribution.
  • The higher T is, the flatter the probability distribution.

Note that Temperature mainly affects the generation strategy; it does not change the parameters the model has already been trained on, nor can it directly improve the accuracy of the model's knowledge.

3.5 Output Length Limits

Common fields include:

{
  "max_tokens": 1000
}

It is used to limit the length of the generated content.

Different APIs may define this field differently:

  • Some limit the number of generated tokens.
  • Some use a dedicated maximum output token parameter.
  • Some also support controlling reasoning tokens and visible output tokens independently.

The context window and the maximum output length are not the same concept either.

The context window usually has to accommodate both the input content and the output content, whereas the maximum output length only controls the generated portion.

3.6 Stream: Whether to Enable Streaming Responses

{
  "stream": true
}

stream determines which response method the server uses.

  • false: usually waits for the complete result before returning.
  • true: usually keeps sending incremental events.

Different APIs may implement streaming calls through other parameters or a different endpoint.

IV. From JSON to Model Input: How Is Request Data Processed?

What the client sends is JSON data, but the model cannot directly understand JSON objects.

JSON is the data interchange format at the API layer; what the model actually processes is an input representation that has been parsed, normalized, and encoded.

4.1 JSON Parsing

The server first receives the HTTP body and parses the JSON data into internal objects.

For example:

{
  "model": "example-model",
  "messages": [
    {
      "role": "user",
      "content": "什么是 Transformer?"
    }
  ]
}

After parsing, the server can obtain:

request = {
    "model": "example-model",
    "messages": [
        {
            "role": "user",
            "content": "什么是 Transformer?"
        }
    ]
}

The Python dictionary here is only meant to illustrate the parsed data structure. The actual server may be implemented in Go, Java, Rust, C++, or other languages.

4.2 Parameter Validation

The server usually needs to check:

  • Whether the model exists.
  • Whether messages conform to the schema.
  • Whether role is valid.
  • Whether content meets the requirements.
  • Whether the token length exceeds the limit.
  • Whether temperature is within the allowed range.
  • Whether there is access permission for the corresponding model.
  • Whether the account quota is exceeded.

For example, a missing model name, an invalid message structure, or input exceeding the limit may all cause the request to be rejected before model inference begins.

4.3 Message Normalization

After validation passes, the server may convert the external API format into the format used internally by the model inference engine.

Different models may adopt different chat templates.

For example, messages may be organized into a sequence containing role boundaries and end markers:

<|system|>
You are a professional technical assistant.

<|user|>
What is a Transformer?

<|assistant|>

This is only for illustrating the concept; not all models use these tokens.

A model may use:

  • Chat Template.
  • Role Boundary Token.
  • Message Separator.
  • End-of-Turn Token.
  • Tool Call Token.

Message roles are not only ordinary text labels; they may also be represented by dedicated special tokens.

4.4 Tokenizer Encoding

After message normalization is complete, the text content must be converted by the Tokenizer into a sequence of token IDs.

For example:

Input text
    ↓
Tokenizer
    ↓
Token ID
    ↓
Embedding
    ↓
Transformer

Its mathematical representation can be abstracted as:

X = Tokenizer(M)

Where:

  • M: the normalized input messages.
  • Tokenizer: the tokenization and encoding process.
  • X: the token ID sequence.

It is then mapped to vectors through the Embedding layer:

E = Embedding(X)

Note that the model does not directly process the JSON key-value structure. The API service layer must first convert the request content into an input representation that the model can accept.

4.5 Complete Request Processing Flow

flowchart TD
    A["Receive HTTP Body"]
    B["Parse JSON"]
    C["Validate request fields"]
    D{"Is the request valid?"}
    E["Return parameter error"]
    F["Parse Messages"]
    G["Apply Chat Template"]
    H["Tokenizer Encoding"]
    I["Generate model input representation"]
    J["Submit to inference engine"]

    A --> B
    B --> C
    C --> D
    D -->|"No"| E
    D -->|"Yes"| F
    F --> G
    G --> H
    H --> I
    I --> J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

5. How Does the Server Execute Model Inference?

Only after a request passes validation at the API layer does it enter the actual model inference stage.

Taking a decoder-only autoregressive language model as an example, inference usually consists of two stages: Prefill and Decode.

5.1 Prefill: Processing the Input Context

The Prefill stage is responsible for processing the complete input sequence.

Its main tasks are:

  1. Convert the input tokens into embeddings.
  2. Execute computations for each Transformer layer.
  3. Compute the Query, Key, and Value required for Attention.
  4. Build the KV Cache corresponding to historical tokens.
  5. Compute the output distribution for the last input position.

For a prompt with input length N, Prefill can usually process multiple input positions in parallel.

Its main goal is to complete context encoding and establish the necessary cache state for subsequent generation.

5.2 Decode: Token-by-Token Generation

After Prefill is complete, the model enters the Decode stage.

Decode usually works in an autoregressive manner:

  1. Compute the probability distribution of the next token based on the current context.
  2. Select a token using Greedy, Sampling, or another decoding strategy.
  3. Add the new token to the generated sequence.
  4. Update the KV Cache.
  5. Continue predicting the next token.

Mathematically, this can be expressed as:

P(x_t|x_1,\ldots,x_{t-1})

The model predicts the next token based on the existing token sequence.

This process continues until a stopping condition is reached.

5.3 Relationship Between Prefill and Decode

flowchart TD
    A["Input Token sequence"]
    B["Prefill stage"]
    C["Process input context in parallel"]
    D["Build KV Cache"]
    E["Compute distribution for the first output Token"]
    F["Select the next Token"]
    G["Decode stage"]
    H["Read historical KV Cache"]
    I["Compute the current Token"]
    J["Append the new KV Cache"]
    K{"Is generation finished?"}
    L["Complete model output"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J
    J --> K
    K -->|"No"| G
    K -->|"Yes"| L

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Note that the output distribution at the last input position computed during Prefill can be used to select the first generated Token. The K/V of the first generated Token is usually computed and added to the cache later, when Decode takes that Token as input.

5.4 Model inference is not the same as an HTTP response

What the model generates internally usually still has to go through server-side processing before it becomes an API response.

This may involve:

  • Decoding Token IDs into text.
  • Handling special Tokens.
  • Parsing tool-call structures.
  • Constraining the output format.
  • Content safety checks.
  • Token usage statistics.
  • Determining the Finish Reason.

The model inference layer is responsible for generating the model output, and the API application layer is responsible for organizing the result into a data structure the client can understand.

Six. Response: The Structure of Response Data

After model inference completes, the server needs to construct a response.

In a typical Chat Completions–style API, the response usually contains:

  • A response ID.
  • The object type.
  • The model name.
  • The generated result.
  • The finish reason.
  • Token usage statistics.

6.1 Complete response example

{
  "id": "chatcmpl-example",
  "object": "chat.completion",
  "created": 1795090000,
  "model": "example-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Transformer is a neural network architecture based on the attention mechanism."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 18,
    "total_tokens": 38
  }
}

This is an illustrative response structure; the actual fields and optional content depend on the specific API.

6.2 Core fields of the Response

Field Meaning Purpose
id Response identifier Uniquely identifies the result of one call
object Object type Describes the response category
created Creation time Records when the response was generated
model Model name Identifies the model actually used
choices List of generated results Carries the model output
index Result index Identifies a candidate result
message Message object Carries the model message
finish_reason Finish reason Explains the generation end state
usage Usage statistics Counts Token consumption

Not every API returns all fields.

For example, some APIs use different response objects and may also place text content in other structures.

6.3 Choices: Model Generation Results

choices is usually used to carry model output results.

For example:

{
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "这是模型生成的回答。"
      },
      "finish_reason": "stop"
    }
  ]
}

Why use an array?

Some APIs support returning multiple candidate results in a single request, so an array carries multiple generation options.

However, not all model services support multiple candidate outputs, and not all APIs use choices.

6.4 Finish Reason: Why Generation Ends

finish_reason is used to describe why generation stopped.

Common values may include:

Value Meaning
stop Normal completion or stop condition hit
length Output length limit reached
tool_calls Tool call request generated
content_filter Content safety policy triggered

Enumeration values may differ across APIs.

Pay special attention:

A model generating a tool call request does not mean the external tool has already been executed.

Tool call requests still need to be parsed, validated, and executed by the application.

6.5 Usage: Token Usage Statistics

A common response may include:

{
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 18,
    "total_tokens": 38
  }
}

Where:

T_{total}=T_{input}+T_{output}
  • prompt_tokens: Number of input Tokens.
  • completion_tokens: Number of output Tokens.
  • total_tokens: Total number of Tokens.

Some APIs may also return detailed statistics for cached Tokens, reasoning Tokens, or multimodal Tokens.

Usage is not just a billing basis; it can also be used to:

  • Monitor request costs.
  • Estimate context consumption.
  • Optimize prompts.
  • Evaluate the usage efficiency of different models.
  • Detect abnormal calls.

7. Response Finish Reasons and Model Stopping Mechanisms

Why does a model stop generating?

It is not necessarily because it has completed a natural-language sentence; it may have triggered different termination conditions.

Common stop conditions include:

7.1 EOS Token

EOS (End of Sequence) is a special Token used to indicate the end of a sequence.

When the model generates an end marker that conforms to the current protocol, the generation process can end.

However, end markers may differ across models.

7.2 Stop Sequence

The client can provide stop sequences to the API.

For example:

{
  "stop": ["<END>"]
}

When the generated content matches the specified stop condition, the server may stop generating further content.

7.3 Maximum Output Length

When the number of generated Tokens reaches the maximum limit, the server terminates this generation.

This usually corresponds to a finish reason such as length.

7.4 Tool Call Termination

When the model decides to call a tool, it may first output a structured tool call request instead of directly generating the final natural-language answer.

After the application executes the tool and submits the result to the model, the model may continue generating subsequent content.

7.5 Stop Condition Flow

flowchart TD
    A["Start Token Generation"]
    B["Generate Next Token"]
    C{"EOS Triggered?"}
    D{"Stop Sequence Hit?"}
    E{"Output Length Limit Reached?"}
    F{"Tool Call Request Generated?"}
    G["Continue Generation"]
    H["Normal Completion"]
    I["Stop Sequence Completion"]
    J["Length Limit Completion"]
    K["Return Tool Call Request"]

    A --> B
    B --> C
    C -->|"Yes"| H
    C -->|"No"| D
    D -->|"Yes"| I
    D -->|"No"| E
    E -->|"Yes"| J
    E -->|"No"| F
    F -->|"Yes"| K
    F -->|"No"| G
    G --> B

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49

The figure above is a conceptual stop-decision process. A real server may use a different internal scheduling order, and tool calls may also be handled by a separate structured decoding process.

VIII. Transport Mechanisms for Non-Streaming and Streaming Responses

The response modes of large model APIs can mainly be divided into non-streaming and streaming.

The difference between the two is not whether the model generates Token by Token, but when the server transmits the generated results to the client.

8.1 Non-Streaming Response

In non-streaming mode, after the client initiates a request, it usually needs to wait for the model to complete this generation.

Then the server returns the complete response all at once.

sequenceDiagram
    participant C as Client
    participant S as API Service
    participant M as Model Inference Engine

    C->>S: POST Request
    S->>M: Submit model input
    M-->>S: Complete generated result
    S-->>C: HTTP 200 + JSON

Advantages:

  • Simple client implementation.
  • The response structure is usually complete.
  • Suitable for short texts and background tasks.

Disadvantages:

  • Long texts require a longer wait.
  • Users cannot see the generated content in advance.
  • The first-display latency may be high.

8.2 Streaming Response

In streaming mode, the server continuously sends incremental data while the model is generating.

The client receives and displays it incrementally.

sequenceDiagram
    participant C as Client
    participant S as API Service
    participant M as Model Inference Engine

    C->>S: POST request stream=true
    S->>M: Submit model input
    M-->>S: Generate the first Token
    S-->>C: Send Chunk 1
    M-->>S: Generate subsequent Tokens
    S-->>C: Send Chunk 2
    M-->>S: Continue generating
    S-->>C: Send Chunk 3
    M-->>S: Generation complete
    S-->>C: Send end event

Here, a Chunk does not necessarily correspond one-to-one with a single Token.

The server may combine multiple Tokens before sending, or may form data chunks of different sizes due to transmission buffering, encoding boundaries, and scheduling policies.

8.3 SSE Event Structure

A common SSE data format is as follows:

data: {"choices":[{"delta":{"content":"大模型"}}]}

data: {"choices":[{"delta":{"content":" API"}}]}

data: {"choices":[{"delta":{"content":" 的响应机制"}}]}

data: [DONE]

Streaming responses usually use delta to represent the newly added content in this chunk.

Compared with non-streaming responses:

Non-streaming Streaming
message delta
Full content Incremental content
Single response Multiple events
Complete JSON Event stream
Wait until generation finishes Can display while generating

8.4 SSE, HTTP Chunked, and WebSocket

These three concepts are easily confused.

Technology Core role Communication direction
SSE Server pushes events Server to client
HTTP Chunked Transfer HTTP/1.1 chunked transfer encoding Depends on the direction of the HTTP request and response
WebSocket Establishes a bidirectional message channel Bidirectional

SSE is an event-stream mechanism, HTTP Chunked is a transfer encoding method, and WebSocket is an independent bidirectional communication protocol.

HTTP/2 also supports streaming, so HTTP Chunked Transfer should not be regarded as a necessary condition for all HTTP streaming responses.

9. How Does the Client Parse API Responses?

The data returned by the server ultimately needs to be parsed by the client.

For non-streaming JSON, the parsing process is relatively simple.

9.1 Parsing a Complete Response in Python

import requests

response = requests.post(
    url,
    headers=headers,
    json=payload,
    timeout=60
)

response.raise_for_status()

data = response.json()

answer = data["choices"][0]["message"]["content"]

print(answer)

In actual production environments, it is recommended to perform safety checks on fields:

choices = data.get("choices", [])

if not choices:
    raise ValueError("模型没有返回候选结果")

message = choices[0].get("message", {})
answer = message.get("content")

if answer is None:
    raise ValueError("响应中没有文本内容")

print(answer)