Back to series
· 14 min read

How to Call LLM APIs: From HTTP Requests to Production-Grade AI Application Integration


1. What Is an LLM API?

A large model API (Application Programming Interface) is a standardized invocation interface that model service providers make available to developers. Developers send inputs such as text, images, and audio to the model through HTTP requests or SDKs, and receive the results generated by the model.

Calling an LLM API essentially encapsulates model inference capabilities as a remote service.

Traditional applications usually need to deploy and run models themselves. When calling through an API, however, the service provider is responsible for the model's computing, inference resources, and runtime environment.

1.1 Basic Architecture of an API Call

A typical LLM API call involves three main roles:

  • Client: The application that initiates the request, such as a web frontend, mobile client, or backend service.
  • API Server: Receives requests, completes authentication, parameter validation, and request scheduling.
  • LLM: Performs inference and generates responses.
flowchart TD
    A["Client application"]
    B["API service"]
    C["Authentication"]
    D["Request parameter validation"]
    E["Model inference"]
    F["Generate response"]
    G["Client receives result"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Note: The client usually does not connect directly to the model's internal inference process; instead, it accesses the model through the API gateway or service entry provided by the service provider.

1.2 Differences Between Calling an API and Deploying a Model Locally

Comparison Dimension Calling a Cloud API Deploying a Model Locally
Model runtime location Service provider's cloud Your own servers
Compute resources Provided by the service provider Prepare GPUs and other resources yourself
Initial cost Usually lower Usually higher
Operations responsibility Service provider handles the underlying model service Maintain the inference environment yourself
Data control Must follow the service provider's data policy Can control the deployment environment independently
Scalability Depends on the service provider's quota and service capabilities Depends on your own infrastructure
Billing method Commonly Token or other usage-based billing Hardware, energy, and operations costs

The two approaches are not mutually exclusive. Enterprises can also adopt a hybrid architecture, assigning general tasks to cloud APIs and handling sensitive data or specific tasks in a privately deployed environment.

2. Core Components of an LLM API

Before actually writing code, you need to understand several core concepts involved in API calls.

2.1 Endpoint: API Address

An endpoint is the request entry point provided by an API service, usually a URL.

For example:

https://api.example.com/v1/chat/completions

Where:

  • https://: Uses the HTTPS protocol.
  • api.example.com: API service domain.
  • /v1: API version path.
  • /chat/completions: Specific API resource.

Endpoints are not necessarily the same across different service providers.

Even if two service providers both support OpenAI-compatible interfaces, you cannot assume that all their paths, parameters, and response formats are completely identical.

2.2 API Key: Authentication Credential

An API Key is the authentication credential used when calling a model service.

It is usually used to:

  • Identify the caller.
  • Associate account and billing information.
  • Control invocation permissions.
  • Enforce rate limits.
  • Track API usage.

Common request header:

Authorization: Bearer YOUR_API_KEY

Here, Bearer indicates the Bearer Token authentication method.

An API Key is a credential for accessing the model service; it is not a model parameter, nor is it the model itself.

Security requirements:

  • Do not hard-code API Keys in frontend JavaScript.
  • Do not commit API Keys to Git repositories.
  • Do not print API Keys in full in logs.
  • Store them using environment variables or a dedicated secret management service.
  • If a leak is discovered, revoke and regenerate immediately.

2.3 Model: Specify the Model

The Model field specifies which model is invoked for this request.

For example:

{
  "model": "example-model"
}

Different models may have different capabilities:

Model type Main capabilities
General-purpose language model Text understanding, generation, and reasoning
Reasoning model Complex problem analysis and multi-step reasoning
Vision-language model Image understanding, OCR, and visual question answering
Audio model Speech recognition, speech generation, and audio understanding
Multimodal model Process text, images, audio, or video simultaneously

Model names are usually defined by the provider. A model's context window, maximum output length, tool-calling capability, and range of multimodal support may also differ.

2.4 Request and Response

An API call usually consists of two parts:

  • Request: The request sent by the client to the server.
  • Response: The response returned by the server to the client.

The request contains information such as the model name, input content, and generation parameters.

The response contains information such as the model-generated content, finish reason, and token usage.

2.5 Token: The Basic Unit of Text Processing by Models

Large models usually do not directly use complete words or Chinese characters as the unit of computation; instead, they first use a Tokenizer to split text into tokens.

For example:

Input text:
Artificial intelligence is changing the way software is developed.

Tokenizer:
Token ID sequence

Embedding:
Vector representation

Transformer:
Context computation

Output:
Generated token sequence

The number of tokens affects:

  • Context window usage.
  • Model inference computation.
  • KV Cache GPU memory usage.
  • API billing.
  • Request and response processing time.

Note that input tokens and output tokens may have different billing prices, and cached tokens may also follow different billing rules.

III. Basic Structure of an HTTP Request

Large model APIs are usually invoked over HTTPS.

A standard HTTP request mainly includes:

  1. HTTP Method.
  2. Request URL.
  3. Request Headers.
  4. Request Body.

3.1 HTTP Method

Common API requests use the POST method.

POST /v1/chat/completions HTTP/1.1

POST means submitting data to the server, which is suitable for passing prompts, model parameters, and multimodal input.

3.2 Request Headers

Headers describe the request's authentication information and data format.

Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

Common fields:

Header Purpose
Authorization Identity authentication
Content-Type Data format of the request body
Accept Desired data format for the response
User-Agent Identifies the client program
Custom tracing field Correlate request logs and call chains

Not all providers support the same custom headers.

3.3 Request Body

The request body is usually in JSON format.

The following is a common example of a Chat Completions-style request:

{
  "model": "example-model",
  "messages": [
    {
      "role": "system",
      "content": "你是一个专业的技术助手。"
    },
    {
      "role": "user",
      "content": "解释一下 Transformer 的工作原理。"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 1000,
  "stream": false
}

Main fields:

Field Purpose
model Specify the model
messages List of conversation messages
role Message role
content Message content
temperature Controls randomness of generation
max_tokens Limits the number of output tokens; the exact field and meaning depend on the API
stream Whether to enable streaming responses

Note: This uses a common OpenAI Chat Completions-compatible style example. Different APIs may use other fields such as max_output_tokens or max_completion_tokens; you cannot directly assume that the parameters are fully interchangeable.

4. Understanding Messages and Role

In chat-style APIs, messages is usually used to describe the conversation context.

A message generally contains:

{
  "role": "user",
  "content": "你好"
}

Here, role is used to distinguish the source and purpose of a message.

4.1 Common Message Roles

Role Meaning Purpose
system System instruction Defines model behavior and constraints
developer Developer instruction Defines application-level rules
user User message Provides questions and input
assistant Model message Represents content generated by the model
tool Tool result Represents data returned by external tools

Which roles are supported depends on the API protocol.

4.2 System Message

A System Message is usually used to define the model's overall behavior.

For example:

{
  "role": "system",
  "content": "你是一名资深 Python 工程师,回答时优先提供准确、可运行的代码。"
}

It can define:

  • Model identity.
  • Response style.
  • Task boundaries.
  • Output format.
  • Business rules.

A System Message is not a model parameter update, nor does it permanently change the model. It only provides instructions within the context of the current request or conversation.

4.3 User Message

A User Message is the content actually submitted by the user.

{
  "role": "user",
  "content": "请使用 Python 实现一个快速排序算法。"
}

The model understands the user's intent in combination with other context in the current request.

4.4 Assistant Message

An Assistant Message represents content generated by the model.

{
  "role": "assistant",
  "content": "快速排序是一种基于分治思想的排序算法。"
}

In multi-turn conversations, applications usually need to save previous user messages and model replies so that subsequent requests can continue to use this context.

4.5 Tool Message

A Tool Message is used to carry the result after an external tool has executed.

For example, when the model requests a weather lookup, the application calls a weather service and then passes the result back to the model as a tool message.

Different APIs may use different message structures for tool call results; you cannot conflate the protocols of all providers.

5. Using Python to Call LLM APIs

Once you understand HTTP requests, you can use Python directly to make API calls.

5.1 Using the requests Library

requests is a commonly used HTTP client library in Python.

Install:

pip install requests

Prepare environment variables:

export LLM_API_KEY="your-api-key"
export LLM_BASE_URL="https://api.example.com/v1"

Python call example:

```python import os import requests api_key = os.environ["LLM_API_KEY"] base_url = os.environ["LLM_BASE_URL"] url = f"{base_url}/chat/completions" headers = { "Authorization": f"Bearer {api_key}", "Content-Type": "application/json" } payload = { "model": "example-model", "messages": [ { "role": "system", "content": "你是一名专业的技术助手。" }, { "role": "user", "content": "什么是 Transformer?" } ], "temperature": 0.7, "max_tokens": 1000 } response = requests.post( url, headers=headers, json=payload, timeout=60 ) response.raise_for_status() result = response.json() print(result) ```

A few details are worth noting here:

  • headers handles authentication and declares the data format.
  • json=payload serializes the Python object into JSON.
  • timeout=60 limits how long the request waits.
  • raise_for_status() throws an exception when the HTTP status indicates an error.
  • response.json() parses the response body into a Python object.

In real development, you should not simply assume that the returned result always contains a particular fixed field; first check the response structure defined by the provider.

5.2 Parsing Model Responses

Take a typical Chat Completions–style response as an example:

{
  "id": "chatcmpl-example",
  "object": "chat.completion",
  "model": "example-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Transformer 是一种基于注意力机制的神经网络架构。"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 18,
    "total_tokens": 38
  }
}

Extract the text:

answer = result["choices"][0]["message"]["content"]

print(answer)

Extract token usage:

usage = result.get("usage", {})

prompt_tokens = usage.get("prompt_tokens", 0)
completion_tokens = usage.get("completion_tokens", 0)
total_tokens = usage.get("total_tokens", 0)

print("输入 Token:", prompt_tokens)
print("输出 Token:", completion_tokens)
print("总 Token:", total_tokens)

The response fields here belong to an example protocol; not every model API uses the same structure.

5.3 Using the Official SDK

Besides sending HTTP requests directly, you can also use the SDK provided by the vendor.

Take the common OpenAI SDK calling style in Python as an example:

Install:

pip install openai

Set the environment variable:

export OPENAI_API_KEY="your-api-key"

Call:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="example-model",
    messages=[
        {
            "role": "system",
            "content": "你是一名专业的技术助手。"
        },
        {
            "role": "user",
            "content": "解释一下什么是 KV Cache。"
        }
    ],
    temperature=0.7,
    max_tokens=1000
)

print(response.choices[0].message.content)

The main value of the SDK is encapsulating the underlying HTTP communication:

  • Automatically handles request serialization.
  • Provides a unified response object.
  • Provides exception types.
  • Supports streaming iteration.
  • Provides configuration capabilities such as timeouts and retries.

The SDK does not replace the model inference process, nor does it automatically guarantee that answers are correct.

If you use an API from a compatible provider, you usually also need to configure base_url and check the interfaces, fields, and model capabilities it supports.

6. Multi-turn Conversations: How to Maintain Context?

A common misconception is that the model API automatically remembers previous conversations.

In fact, many basic API calls use a stateless request model: each request processes only the data submitted with that request and does not automatically receive the entire prior conversation history from the application.

If you want the model to understand context, the application needs to explicitly maintain historical messages or use the session state management capabilities provided by the service provider.

6.1 Single-turn Conversation

A single-turn conversation submits only the current question:

{
  "model": "example-model",
  "messages": [
    {
      "role": "user",
      "content": "什么是数据库索引?"
    }
  ]
}

The model can answer the current question, but the next request will not naturally know the context of this question just because the API was called before.

6.2 Multi-turn Conversations

Suppose the user has a three-round conversation:

  1. What is a database index?
  2. Why can it improve query speed?
  3. Why is a B+ tree suitable for indexing?

The application needs to submit the relevant historical messages together:

{
  "model": "example-model",
  "messages": [
    {
      "role": "system",
      "content": "你是一名数据库专家。"
    },
    {
      "role": "user",
      "content": "什么是数据库索引?"
    },
    {
      "role": "assistant",
      "content": "数据库索引是一种用于加速数据检索的数据结构。"
    },
    {
      "role": "user",
      "content": "它为什么能够提高查询速度?"
    },
    {
      "role": "assistant",
      "content": "索引可以帮助数据库缩小搜索范围,减少不必要的数据扫描。"
    },
    {
      "role": "user",
      "content": "B+ 树为什么适合做索引?"
    }
  ]
}

The model uses the complete message context to understand “it” in the third-round question and the topic discussed earlier.

6.3 Maintenance Flow for Multi-turn Conversations

flowchart TD
    A["Receive user message"]
    B["Read conversation history"]
    C["Construct Messages"]
    D["Call LLM API"]
    E["Receive model response"]
    F["Save model reply"]
    G["Return to user"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

6.4 Why Can't Conversation History Be Stored Indefinitely?

Carrying the full history with every request creates several problems:

  • Input tokens keep growing.
  • API call costs increase.
  • Inference compute overhead increases.
  • The context window may be exhausted.
  • Early irrelevant information may interfere with the current task.

Common optimization strategies:

Strategy Main purpose
History truncation Keep only the most recent few turns
Conversation summarization Compress earlier history into a summary
RAG retrieval Retrieve relevant history based on the current question
Key information extraction Store structured information such as user preferences and task state
Context layering Separate system instructions, long-term information, and short-term dialogue

Real systems usually combine multiple strategies rather than simply accumulating Messages without limit.

7. Generation Parameters: How Do You Control Model Output?

Model APIs typically support a number of generation parameters that control output behavior.

7.1 Temperature

Temperature adjusts the randomness of the probability distribution during generation.

During sampling, the raw logits z_i can be scaled by a temperature T:

P(x_i)= \frac{\exp(z_i/T)}

Generally speaking:

  • Lower temperature: a sharper probability distribution, producing more stable results.
  • Higher temperature: a flatter probability distribution, producing more diverse results.

Example:

{
  "temperature": 0.2
}

Suitable for tasks that require high stability.

{
  "temperature": 0.9
}

Can be used for generation tasks that need more expressive variation.

Temperature is not an accuracy switch. A low temperature does not guarantee factual correctness, and a high temperature does not mean errors are certain to occur.

7.2 Top-P

Top-P is also known as nucleus sampling.

The model sorts tokens by probability from highest to lowest, selects the candidate set whose cumulative probability reaches the specified threshold, and then samples from that set.

For example:

{
  "top_p": 0.9
}

This means sampling from the candidate set whose cumulative probability reaches 0.9.

Both Temperature and Top-P affect the sampling distribution, so it is generally not advisable to adjust both substantially at the same time without experimental evidence.

7.3 Maximum Output Tokens

The maximum output token count limits the length of the model's generation.

Different APIs may use different parameter names, for example:

  • max_tokens
  • max_completion_tokens
  • max_output_tokens

For the exact names and supported ranges, refer to the provider's documentation.

It is important to distinguish:

  • Input tokens: user messages, system instructions, and historical context.
  • Output tokens: the content generated by the model.

The maximum output token count usually limits only the generated portion, and is not equal to the context window size of the entire request.

7.4 Stop Sequences

Stop Sequences are used to specify certain stop markers.

When the generated content reaches the stop condition defined by the provider, the model may end its output early.

For example:

{
  "stop": ["<END>"]
}

Different APIs may differ in how they support and match stop sequences.

7.5 Comparison of Generation Parameters

Parameter What it controls Main effect
Temperature Probability distribution Randomness of generation
Top-P Candidate token set Sampling range
Maximum output tokens Output length Response scale
Stop Sequences Ending condition Where generation terminates
Seed Random number initialization Helps reproduce results in supported environments

Seed does not guarantee identical results across model versions, hardware environments, or server-side implementations.

8. Streaming: Streamed Responses

By default, the API may wait until the model has generated the complete answer before returning the response all at once.

This approach is called non-streaming.

For long-form generation, users have to wait a long time before seeing the first character.

Streaming, by contrast, allows the server to continuously send incremental results during generation.

8.1 Differences Between Non-Streaming and Streaming

Comparison aspect Non-streaming Streaming
Response mode Returns the complete result at once Continuously returns incremental data
Initial display Waits for the complete response Can be displayed earlier
User experience Noticeable waiting for long answers Can be read progressively
Client-side processing Parses the complete JSON Continuously parses events
Use cases Short text, background tasks Chat, long-form text generation

Streaming responses do not mean the model's computation itself is necessarily faster; rather, they allow the parts that have already been generated to be transmitted and displayed earlier.

8.2 SSE: Server-Sent Events

Many large model APIs use SSE-style event streams to transmit generated results.

A typical response may include:

Content-Type: text/event-stream

Event data example:

data: {"choices":[{"delta":{"content":"Transformer"}}]}

data: {"choices":[{"delta":{"content":" is a"}}]}

data: {"choices":[{"delta":{"content":" neural network architecture"}}]}

data: [DONE]

The client needs to parse events incrementally, rather than treating the entire response as a single complete JSON.

Several concepts need to be distinguished:

  • SSE: A data format and mechanism for a server to push events to a client.
  • HTTP Chunked Transfer: A chunked transfer encoding mechanism in HTTP/1.1.
  • WebSocket: A protocol that supports bidirectional communication.

They are not the same concept. SSE can be transmitted via HTTP chunking and other methods, but SSE should not simply be equated with Chunked Transfer.

8.3 Streaming response flow

flowchart TD
    A["Client sends request"]
    B["Server starts inference"]
    C["Generate new Token"]
    D["Wrap incremental event"]
    E["Send data chunk"]
    F["Client parses event"]
    G["Update UI"]
    H{"Is generation complete?"}
    I["End stream connection"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H -->|"No"| C
    H -->|"Yes"| I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

8.4 Python streaming call example

Taking the Chat Completions style of a common SDK as an example:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="example-model",
    messages=[
        {
            "role": "user",
            "content": "详细解释一下 Transformer。"
        }
    ],
    stream=True
)

for chunk in stream:
    if not chunk.choices:
        continue

    delta = chunk.choices[0].delta
    content = delta.content

    if content:
        print(content, end="", flush=True)

Here, stream=True indicates a request for a streaming response.

The client continuously receives new data chunks through an iterator.

In production environments, you also need to handle:

  • Stream interruption.
  • Network timeout.
  • Client request cancellation.
  • End marker.
  • Tool call argument deltas.
  • Server-side proxy buffering.
  • Cases where some events lack text content.

8.5 Important Streaming Performance Metrics

Metric Meaning
TTFT Time To First Token; time until the first token is returned
TPOT Time Per Output Token; average generation time per output token
TPS Tokens Per Second; token generation or processing rate; the exact counting basis must be clarified
E2E Latency Total elapsed time from request initiation to response completion

These metrics reflect different performance dimensions of model services; streaming experience cannot be judged by total response time alone.

9. Function Calling: Enabling Models to Call External Tools

Large models are primarily responsible for understanding and generating content; they do not inherently have permission to access databases, query weather, execute payments, or operate business systems.

Function Calling allows a model to generate structured call requests according to predefined tool descriptions.

The application parses the request, executes the actual tool, and then returns the execution result to the model.

9.1 Basic Principles of Function Calling

Suppose the user asks:

Help me check the weather in Beijing today.

The system defines a weather query tool:

{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "查询指定城市的天气",
    "parameters": {
      "type": "object",
      "properties": {
        "city": {
          "type": "string",
          "description": "城市名称"
        }
      },
      "required": ["city"]
    }
  }
}

The model may generate a structured call request:

{
  "name": "get_weather",
  "arguments": {
    "city": "北京"
  }
}

Note: The model generating this content does not mean the weather query has been executed.

The actual call is handled by the application.

9.2 Complete Function Calling Workflow

flowchart TD
    A["User makes a request"]
    B["Application submits the message and tool definitions"]
    C["Model determines whether a tool call is needed"]
    D{"Is a tool needed?"}
    E["Model generates a normal answer"]
    F["Model generates a tool call request"]
    G["Application parses the call arguments"]
    H["Argument and permission validation"]
    I["Execute the external tool"]
    J["Return the tool execution result"]
    K["Submit the tool result to the model"]
    L["Model generates the final answer"]

    A --> B
    B --> C
    C --> D
    D -->|"No"| E
    D -->|"Yes"| F
    F --> G
    G --> H
    H --> I
    I --> J
    J --> K
    K --> L

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#E5F0E9,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F

9.3 Differences Between Function Calling and Regular Text Generation

Comparison Dimension Plain Text Generation Function Calling
Output Content Natural language text Structured call request
Executes External Operations Usually no Tools executed by the application
Parameter Format Free-form text Usually constrained by a Schema
Source of Results Generated by the model Model and external tools together
Primary Use Q&A, summarization, writing Queries, computation, business operations

Function Calling is an important foundation of Agent systems, but it is not itself a complete Agent.

Production systems also require tool permission control, execution state management, exception handling, and call auditing.

X. Multimodal API: Image, Audio, and Video Input

Modern large model APIs are no longer limited to text.

Different models and providers may support:

  • Text input and generation.
  • Image understanding.
  • Audio recognition and understanding.
  • Speech generation.
  • Video understanding.
  • Real-time audio and video interaction.

10.1 Image Input

Vision-language models (VLMs) can process images and combine them with text instructions to complete visual understanding tasks.

Typical tasks include:

  • Image description.
  • OCR text recognition.
  • Chart analysis.
  • Image question answering.
  • Visual reasoning.
  • Multi-image comparison.