1. What Is an LLM API?
A large model API (Application Programming Interface) is a standardized invocation interface that model service providers make available to developers. Developers send inputs such as text, images, and audio to the model through HTTP requests or SDKs, and receive the results generated by the model.
Calling an LLM API essentially encapsulates model inference capabilities as a remote service.
Traditional applications usually need to deploy and run models themselves. When calling through an API, however, the service provider is responsible for the model's computing, inference resources, and runtime environment.
1.1 Basic Architecture of an API Call
A typical LLM API call involves three main roles:
- Client: The application that initiates the request, such as a web frontend, mobile client, or backend service.
- API Server: Receives requests, completes authentication, parameter validation, and request scheduling.
- LLM: Performs inference and generates responses.
flowchart TD
A["Client application"]
B["API service"]
C["Authentication"]
D["Request parameter validation"]
E["Model inference"]
F["Generate response"]
G["Client receives result"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note: The client usually does not connect directly to the model's internal inference process; instead, it accesses the model through the API gateway or service entry provided by the service provider.
1.2 Differences Between Calling an API and Deploying a Model Locally
| Comparison Dimension | Calling a Cloud API | Deploying a Model Locally |
|---|---|---|
| Model runtime location | Service provider's cloud | Your own servers |
| Compute resources | Provided by the service provider | Prepare GPUs and other resources yourself |
| Initial cost | Usually lower | Usually higher |
| Operations responsibility | Service provider handles the underlying model service | Maintain the inference environment yourself |
| Data control | Must follow the service provider's data policy | Can control the deployment environment independently |
| Scalability | Depends on the service provider's quota and service capabilities | Depends on your own infrastructure |
| Billing method | Commonly Token or other usage-based billing | Hardware, energy, and operations costs |
The two approaches are not mutually exclusive. Enterprises can also adopt a hybrid architecture, assigning general tasks to cloud APIs and handling sensitive data or specific tasks in a privately deployed environment.
2. Core Components of an LLM API
Before actually writing code, you need to understand several core concepts involved in API calls.
2.1 Endpoint: API Address
An endpoint is the request entry point provided by an API service, usually a URL.
For example:
https://api.example.com/v1/chat/completions
Where:
https://: Uses the HTTPS protocol.api.example.com: API service domain./v1: API version path./chat/completions: Specific API resource.
Endpoints are not necessarily the same across different service providers.
Even if two service providers both support OpenAI-compatible interfaces, you cannot assume that all their paths, parameters, and response formats are completely identical.
2.2 API Key: Authentication Credential
An API Key is the authentication credential used when calling a model service.
It is usually used to:
- Identify the caller.
- Associate account and billing information.
- Control invocation permissions.
- Enforce rate limits.
- Track API usage.
Common request header:
Authorization: Bearer YOUR_API_KEY
Here, Bearer indicates the Bearer Token authentication method.
An API Key is a credential for accessing the model service; it is not a model parameter, nor is it the model itself.
Security requirements:
- Do not hard-code API Keys in frontend JavaScript.
- Do not commit API Keys to Git repositories.
- Do not print API Keys in full in logs.
- Store them using environment variables or a dedicated secret management service.
- If a leak is discovered, revoke and regenerate immediately.
2.3 Model: Specify the Model
The Model field specifies which model is invoked for this request.
For example:
{
"model": "example-model"
}
Different models may have different capabilities:
| Model type | Main capabilities |
|---|---|
| General-purpose language model | Text understanding, generation, and reasoning |
| Reasoning model | Complex problem analysis and multi-step reasoning |
| Vision-language model | Image understanding, OCR, and visual question answering |
| Audio model | Speech recognition, speech generation, and audio understanding |
| Multimodal model | Process text, images, audio, or video simultaneously |
Model names are usually defined by the provider. A model's context window, maximum output length, tool-calling capability, and range of multimodal support may also differ.
2.4 Request and Response
An API call usually consists of two parts:
- Request: The request sent by the client to the server.
- Response: The response returned by the server to the client.
The request contains information such as the model name, input content, and generation parameters.
The response contains information such as the model-generated content, finish reason, and token usage.
2.5 Token: The Basic Unit of Text Processing by Models
Large models usually do not directly use complete words or Chinese characters as the unit of computation; instead, they first use a Tokenizer to split text into tokens.
For example:
Input text:
Artificial intelligence is changing the way software is developed.
Tokenizer:
Token ID sequence
Embedding:
Vector representation
Transformer:
Context computation
Output:
Generated token sequence
The number of tokens affects:
- Context window usage.
- Model inference computation.
- KV Cache GPU memory usage.
- API billing.
- Request and response processing time.
Note that input tokens and output tokens may have different billing prices, and cached tokens may also follow different billing rules.
III. Basic Structure of an HTTP Request
Large model APIs are usually invoked over HTTPS.
A standard HTTP request mainly includes:
- HTTP Method.
- Request URL.
- Request Headers.
- Request Body.
3.1 HTTP Method
Common API requests use the POST method.
POST /v1/chat/completions HTTP/1.1
POST means submitting data to the server, which is suitable for passing prompts, model parameters, and multimodal input.
3.2 Request Headers
Headers describe the request's authentication information and data format.
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json
Common fields:
| Header | Purpose |
|---|---|
| Authorization | Identity authentication |
| Content-Type | Data format of the request body |
| Accept | Desired data format for the response |
| User-Agent | Identifies the client program |
| Custom tracing field | Correlate request logs and call chains |
Not all providers support the same custom headers.
3.3 Request Body
The request body is usually in JSON format.
The following is a common example of a Chat Completions-style request:
{
"model": "example-model",
"messages": [
{
"role": "system",
"content": "你是一个专业的技术助手。"
},
{
"role": "user",
"content": "解释一下 Transformer 的工作原理。"
}
],
"temperature": 0.7,
"max_tokens": 1000,
"stream": false
}
Main fields:
| Field | Purpose |
|---|---|
| model | Specify the model |
| messages | List of conversation messages |
| role | Message role |
| content | Message content |
| temperature | Controls randomness of generation |
| max_tokens | Limits the number of output tokens; the exact field and meaning depend on the API |
| stream | Whether to enable streaming responses |
Note: This uses a common OpenAI Chat Completions-compatible style example. Different APIs may use other fields such as max_output_tokens or max_completion_tokens; you cannot directly assume that the parameters are fully interchangeable.
4. Understanding Messages and Role
In chat-style APIs, messages is usually used to describe the conversation context.
A message generally contains:
{
"role": "user",
"content": "你好"
}
Here, role is used to distinguish the source and purpose of a message.
4.1 Common Message Roles
| Role | Meaning | Purpose |
|---|---|---|
| system | System instruction | Defines model behavior and constraints |
| developer | Developer instruction | Defines application-level rules |
| user | User message | Provides questions and input |
| assistant | Model message | Represents content generated by the model |
| tool | Tool result | Represents data returned by external tools |
Which roles are supported depends on the API protocol.
4.2 System Message
A System Message is usually used to define the model's overall behavior.
For example:
{
"role": "system",
"content": "你是一名资深 Python 工程师,回答时优先提供准确、可运行的代码。"
}
It can define:
- Model identity.
- Response style.
- Task boundaries.
- Output format.
- Business rules.
A System Message is not a model parameter update, nor does it permanently change the model. It only provides instructions within the context of the current request or conversation.
4.3 User Message
A User Message is the content actually submitted by the user.
{
"role": "user",
"content": "请使用 Python 实现一个快速排序算法。"
}
The model understands the user's intent in combination with other context in the current request.
4.4 Assistant Message
An Assistant Message represents content generated by the model.
{
"role": "assistant",
"content": "快速排序是一种基于分治思想的排序算法。"
}
In multi-turn conversations, applications usually need to save previous user messages and model replies so that subsequent requests can continue to use this context.
4.5 Tool Message
A Tool Message is used to carry the result after an external tool has executed.
For example, when the model requests a weather lookup, the application calls a weather service and then passes the result back to the model as a tool message.
Different APIs may use different message structures for tool call results; you cannot conflate the protocols of all providers.
5. Using Python to Call LLM APIs
Once you understand HTTP requests, you can use Python directly to make API calls.
5.1 Using the requests Library
requests is a commonly used HTTP client library in Python.
Install:
pip install requests
Prepare environment variables:
export LLM_API_KEY="your-api-key"
export LLM_BASE_URL="https://api.example.com/v1"
Python call example:
```python import os import requests api_key = os.environ["LLM_API_KEY"] base_url = os.environ["LLM_BASE_URL"] url = f"{base_url}/chat/completions" headers = { "Authorization": f"Bearer {api_key}", "Content-Type": "application/json" } payload = { "model": "example-model", "messages": [ { "role": "system", "content": "你是一名专业的技术助手。" }, { "role": "user", "content": "什么是 Transformer?" } ], "temperature": 0.7, "max_tokens": 1000 } response = requests.post( url, headers=headers, json=payload, timeout=60 ) response.raise_for_status() result = response.json() print(result) ```A few details are worth noting here:
headershandles authentication and declares the data format.json=payloadserializes the Python object into JSON.timeout=60limits how long the request waits.raise_for_status()throws an exception when the HTTP status indicates an error.response.json()parses the response body into a Python object.
In real development, you should not simply assume that the returned result always contains a particular fixed field; first check the response structure defined by the provider.
5.2 Parsing Model Responses
Take a typical Chat Completions–style response as an example:
{
"id": "chatcmpl-example",
"object": "chat.completion",
"model": "example-model",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Transformer 是一种基于注意力机制的神经网络架构。"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 20,
"completion_tokens": 18,
"total_tokens": 38
}
}
Extract the text:
answer = result["choices"][0]["message"]["content"]
print(answer)
Extract token usage:
usage = result.get("usage", {})
prompt_tokens = usage.get("prompt_tokens", 0)
completion_tokens = usage.get("completion_tokens", 0)
total_tokens = usage.get("total_tokens", 0)
print("输入 Token:", prompt_tokens)
print("输出 Token:", completion_tokens)
print("总 Token:", total_tokens)
The response fields here belong to an example protocol; not every model API uses the same structure.
5.3 Using the Official SDK
Besides sending HTTP requests directly, you can also use the SDK provided by the vendor.
Take the common OpenAI SDK calling style in Python as an example:
Install:
pip install openai
Set the environment variable:
export OPENAI_API_KEY="your-api-key"
Call:
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="example-model",
messages=[
{
"role": "system",
"content": "你是一名专业的技术助手。"
},
{
"role": "user",
"content": "解释一下什么是 KV Cache。"
}
],
temperature=0.7,
max_tokens=1000
)
print(response.choices[0].message.content)
The main value of the SDK is encapsulating the underlying HTTP communication:
- Automatically handles request serialization.
- Provides a unified response object.
- Provides exception types.
- Supports streaming iteration.
- Provides configuration capabilities such as timeouts and retries.
The SDK does not replace the model inference process, nor does it automatically guarantee that answers are correct.
If you use an API from a compatible provider, you usually also need to configure base_url and check the interfaces, fields, and model capabilities it supports.
6. Multi-turn Conversations: How to Maintain Context?
A common misconception is that the model API automatically remembers previous conversations.
In fact, many basic API calls use a stateless request model: each request processes only the data submitted with that request and does not automatically receive the entire prior conversation history from the application.
If you want the model to understand context, the application needs to explicitly maintain historical messages or use the session state management capabilities provided by the service provider.
6.1 Single-turn Conversation
A single-turn conversation submits only the current question:
{
"model": "example-model",
"messages": [
{
"role": "user",
"content": "什么是数据库索引?"
}
]
}
The model can answer the current question, but the next request will not naturally know the context of this question just because the API was called before.
6.2 Multi-turn Conversations
Suppose the user has a three-round conversation:
- What is a database index?
- Why can it improve query speed?
- Why is a B+ tree suitable for indexing?
The application needs to submit the relevant historical messages together:
{
"model": "example-model",
"messages": [
{
"role": "system",
"content": "你是一名数据库专家。"
},
{
"role": "user",
"content": "什么是数据库索引?"
},
{
"role": "assistant",
"content": "数据库索引是一种用于加速数据检索的数据结构。"
},
{
"role": "user",
"content": "它为什么能够提高查询速度?"
},
{
"role": "assistant",
"content": "索引可以帮助数据库缩小搜索范围,减少不必要的数据扫描。"
},
{
"role": "user",
"content": "B+ 树为什么适合做索引?"
}
]
}
The model uses the complete message context to understand “it” in the third-round question and the topic discussed earlier.
6.3 Maintenance Flow for Multi-turn Conversations
flowchart TD
A["Receive user message"]
B["Read conversation history"]
C["Construct Messages"]
D["Call LLM API"]
E["Receive model response"]
F["Save model reply"]
G["Return to user"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
6.4 Why Can't Conversation History Be Stored Indefinitely?
Carrying the full history with every request creates several problems:
- Input tokens keep growing.
- API call costs increase.
- Inference compute overhead increases.
- The context window may be exhausted.
- Early irrelevant information may interfere with the current task.
Common optimization strategies:
| Strategy | Main purpose |
|---|---|
| History truncation | Keep only the most recent few turns |
| Conversation summarization | Compress earlier history into a summary |
| RAG retrieval | Retrieve relevant history based on the current question |
| Key information extraction | Store structured information such as user preferences and task state |
| Context layering | Separate system instructions, long-term information, and short-term dialogue |
Real systems usually combine multiple strategies rather than simply accumulating Messages without limit.
7. Generation Parameters: How Do You Control Model Output?
Model APIs typically support a number of generation parameters that control output behavior.
7.1 Temperature
Temperature adjusts the randomness of the probability distribution during generation.
During sampling, the raw logits z_i can be scaled by a temperature T:
Generally speaking:
- Lower temperature: a sharper probability distribution, producing more stable results.
- Higher temperature: a flatter probability distribution, producing more diverse results.
Example:
{
"temperature": 0.2
}
Suitable for tasks that require high stability.
{
"temperature": 0.9
}
Can be used for generation tasks that need more expressive variation.
Temperature is not an accuracy switch. A low temperature does not guarantee factual correctness, and a high temperature does not mean errors are certain to occur.
7.2 Top-P
Top-P is also known as nucleus sampling.
The model sorts tokens by probability from highest to lowest, selects the candidate set whose cumulative probability reaches the specified threshold, and then samples from that set.
For example:
{
"top_p": 0.9
}
This means sampling from the candidate set whose cumulative probability reaches 0.9.
Both Temperature and Top-P affect the sampling distribution, so it is generally not advisable to adjust both substantially at the same time without experimental evidence.
7.3 Maximum Output Tokens
The maximum output token count limits the length of the model's generation.
Different APIs may use different parameter names, for example:
max_tokensmax_completion_tokensmax_output_tokens
For the exact names and supported ranges, refer to the provider's documentation.
It is important to distinguish:
- Input tokens: user messages, system instructions, and historical context.
- Output tokens: the content generated by the model.
The maximum output token count usually limits only the generated portion, and is not equal to the context window size of the entire request.
7.4 Stop Sequences
Stop Sequences are used to specify certain stop markers.
When the generated content reaches the stop condition defined by the provider, the model may end its output early.
For example:
{
"stop": ["<END>"]
}
Different APIs may differ in how they support and match stop sequences.
7.5 Comparison of Generation Parameters
| Parameter | What it controls | Main effect |
|---|---|---|
| Temperature | Probability distribution | Randomness of generation |
| Top-P | Candidate token set | Sampling range |
| Maximum output tokens | Output length | Response scale |
| Stop Sequences | Ending condition | Where generation terminates |
| Seed | Random number initialization | Helps reproduce results in supported environments |
Seed does not guarantee identical results across model versions, hardware environments, or server-side implementations.
8. Streaming: Streamed Responses
By default, the API may wait until the model has generated the complete answer before returning the response all at once.
This approach is called non-streaming.
For long-form generation, users have to wait a long time before seeing the first character.
Streaming, by contrast, allows the server to continuously send incremental results during generation.
8.1 Differences Between Non-Streaming and Streaming
| Comparison aspect | Non-streaming | Streaming |
|---|---|---|
| Response mode | Returns the complete result at once | Continuously returns incremental data |
| Initial display | Waits for the complete response | Can be displayed earlier |
| User experience | Noticeable waiting for long answers | Can be read progressively |
| Client-side processing | Parses the complete JSON | Continuously parses events |
| Use cases | Short text, background tasks | Chat, long-form text generation |
Streaming responses do not mean the model's computation itself is necessarily faster; rather, they allow the parts that have already been generated to be transmitted and displayed earlier.
8.2 SSE: Server-Sent Events
Many large model APIs use SSE-style event streams to transmit generated results.
A typical response may include:
Content-Type: text/event-stream
Event data example:
data: {"choices":[{"delta":{"content":"Transformer"}}]}
data: {"choices":[{"delta":{"content":" is a"}}]}
data: {"choices":[{"delta":{"content":" neural network architecture"}}]}
data: [DONE]
The client needs to parse events incrementally, rather than treating the entire response as a single complete JSON.
Several concepts need to be distinguished:
- SSE: A data format and mechanism for a server to push events to a client.
- HTTP Chunked Transfer: A chunked transfer encoding mechanism in HTTP/1.1.
- WebSocket: A protocol that supports bidirectional communication.
They are not the same concept. SSE can be transmitted via HTTP chunking and other methods, but SSE should not simply be equated with Chunked Transfer.
8.3 Streaming response flow
flowchart TD
A["Client sends request"]
B["Server starts inference"]
C["Generate new Token"]
D["Wrap incremental event"]
E["Send data chunk"]
F["Client parses event"]
G["Update UI"]
H{"Is generation complete?"}
I["End stream connection"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H -->|"No"| C
H -->|"Yes"| I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
8.4 Python streaming call example
Taking the Chat Completions style of a common SDK as an example:
from openai import OpenAI
client = OpenAI()
stream = client.chat.completions.create(
model="example-model",
messages=[
{
"role": "user",
"content": "详细解释一下 Transformer。"
}
],
stream=True
)
for chunk in stream:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
content = delta.content
if content:
print(content, end="", flush=True)
Here, stream=True indicates a request for a streaming response.
The client continuously receives new data chunks through an iterator.
In production environments, you also need to handle:
- Stream interruption.
- Network timeout.
- Client request cancellation.
- End marker.
- Tool call argument deltas.
- Server-side proxy buffering.
- Cases where some events lack text content.
8.5 Important Streaming Performance Metrics
| Metric | Meaning |
|---|---|
| TTFT | Time To First Token; time until the first token is returned |
| TPOT | Time Per Output Token; average generation time per output token |
| TPS | Tokens Per Second; token generation or processing rate; the exact counting basis must be clarified |
| E2E Latency | Total elapsed time from request initiation to response completion |
These metrics reflect different performance dimensions of model services; streaming experience cannot be judged by total response time alone.
9. Function Calling: Enabling Models to Call External Tools
Large models are primarily responsible for understanding and generating content; they do not inherently have permission to access databases, query weather, execute payments, or operate business systems.
Function Calling allows a model to generate structured call requests according to predefined tool descriptions.
The application parses the request, executes the actual tool, and then returns the execution result to the model.
9.1 Basic Principles of Function Calling
Suppose the user asks:
Help me check the weather in Beijing today.
The system defines a weather query tool:
{
"type": "function",
"function": {
"name": "get_weather",
"description": "查询指定城市的天气",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "城市名称"
}
},
"required": ["city"]
}
}
}
The model may generate a structured call request:
{
"name": "get_weather",
"arguments": {
"city": "北京"
}
}
Note: The model generating this content does not mean the weather query has been executed.
The actual call is handled by the application.
9.2 Complete Function Calling Workflow
flowchart TD
A["User makes a request"]
B["Application submits the message and tool definitions"]
C["Model determines whether a tool call is needed"]
D{"Is a tool needed?"}
E["Model generates a normal answer"]
F["Model generates a tool call request"]
G["Application parses the call arguments"]
H["Argument and permission validation"]
I["Execute the external tool"]
J["Return the tool execution result"]
K["Submit the tool result to the model"]
L["Model generates the final answer"]
A --> B
B --> C
C --> D
D -->|"No"| E
D -->|"Yes"| F
F --> G
G --> H
H --> I
I --> J
J --> K
K --> L
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#E5F0E9,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
9.3 Differences Between Function Calling and Regular Text Generation
| Comparison Dimension | Plain Text Generation | Function Calling |
|---|---|---|
| Output Content | Natural language text | Structured call request |
| Executes External Operations | Usually no | Tools executed by the application |
| Parameter Format | Free-form text | Usually constrained by a Schema |
| Source of Results | Generated by the model | Model and external tools together |
| Primary Use | Q&A, summarization, writing | Queries, computation, business operations |
Function Calling is an important foundation of Agent systems, but it is not itself a complete Agent.
Production systems also require tool permission control, execution state management, exception handling, and call auditing.
X. Multimodal API: Image, Audio, and Video Input
Modern large model APIs are no longer limited to text.
Different models and providers may support:
- Text input and generation.
- Image understanding.
- Audio recognition and understanding.
- Speech generation.
- Video understanding.
- Real-time audio and video interaction.
10.1 Image Input
Vision-language models (VLMs) can process images and combine them with text instructions to complete visual understanding tasks.
Typical tasks include:
- Image description.
- OCR text recognition.
- Chart analysis.
- Image question answering.
- Visual reasoning.
- Multi-image comparison.