1. What Is a Model Gateway?
As AI applications scale up, systems often need to connect to multiple model services at the same time, including:
- Commercial model APIs: OpenAI, Anthropic, Google Gemini, etc.
- Cloud provider model services: managed model APIs offered by various cloud platforms.
- Open-source model services: models deployed through inference frameworks such as vLLM, SGLang, and TGI.
- Enterprise internal models: fine-tuned models, specialized models, and privately deployed models.
If every business system directly integrates with different models, it easily leads to a large amount of duplicate code and management problems:
- Inconsistent interfaces: Request parameters, response structures, and error formats differ across models.
- Model coupling: Business code directly depends on specific model names and provider APIs.
- Lack of unified governance: Authentication, rate limiting, quotas, and cost control are scattered across individual business systems.
- Complex failure handling: When a model service fails, business systems must handle switching and degradation themselves.
- Lack of global observability: It is difficult to uniformly track model invocation volume, token consumption, latency, and error rates.
- High model upgrade costs: Replacing a model may require modifying multiple business services.
The Model Gateway is a unified access layer designed to solve these problems.
It encapsulates underlying model services behind a unified invocation interface, so business systems do not need to be directly aware of implementation details from different providers.
1.1 Core Positioning of a Model Gateway
The model gateway sits between business applications and model services:
flowchart TB
A["AI Business Applications"]
B["Model Gateway"]
C["Model Service A"]
D["Model Service B"]
E["Model Service C"]
F["Private Model Service"]
A --> B
B --> C
B --> D
B --> E
B --> F
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
Business applications only need to call the model gateway and no longer directly manage the connections, authentication, and request formats of each model provider.
1.2 Differences Between a Model Gateway and an API Gateway
Both sit at the request entry point, but they solve different problems.
| Comparison Dimension | API Gateway | Model Gateway |
|---|---|---|
| Core object | HTTP APIs, microservices | Large language models and inference services |
| Routing basis | URL, HTTP method, service instance | Model name, capabilities, cost, latency, context window |
| Request handling | Path forwarding, protocol conversion | Prompt, message, token, and model parameter adaptation |
| Load balancing | Service instance load | Model instances, providers, inference clusters |
| Cost management | Request volume, traffic, resources | Tokens, model pricing, inference cost |
| Failure handling | Service retries, circuit breaking | Model switching, degradation, context compatibility |
| Streaming | HTTP, WebSocket, etc. | SSE, streaming tokens, generation interruption |
| Observability | QPS, latency, error rate | Tokens, time to first token, generation speed, model invocation cost |
A model gateway can be built on top of API Gateway infrastructure, but ordinary API forwarding capabilities cannot simply be equated with a model gateway.
Model invocation has its own characteristics: request lengths vary, responses may be generated continuously, cost is tied to token count, and different models differ significantly in capabilities and context limits.
Therefore, a model gateway needs dedicated governance capabilities for model inference scenarios.
2. Core Architecture of a Model Gateway
A production-grade model gateway is typically composed of modules such as request entry, protocol adaptation, policy decision, model routing, execution scheduling, and response processing.
2.1 Overall Architecture
flowchart TB
A["Business Application"]
B["Unified API Access Layer"]
C["Authentication and Permission Verification"]
D["Request Parsing and Normalization"]
E["Policy Decision Engine"]
F["Model Router"]
G["Provider Adapter Layer"]
H["Model Serving Cluster"]
I["Response Normalization"]
J["Streaming and Non-Streaming Response"]
K["Logging, Metrics and Tracing"]
L["Configuration Center and Model Registry"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H --> I
I --> J
E -.-> L
F -.-> L
B -.-> K
G -.-> K
H -.-> K
J --> A
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
The responsibilities of each module are as follows:
| Module | Core Responsibilities |
|---|---|
| Unified API Access Layer | Receives business requests and provides a unified entry point for invocation |
| Authentication and Permission Verification | Verifies the caller's identity, tenant, and model permissions |
| Request Parsing and Normalization | Parses requests and converts them into a unified internal structure |
| Policy Decision Engine | Determines model selection, budget, priority, and constraint conditions |
| Model Router | Selects model services that meet the conditions |
| Provider Adapter Layer | Converts internal requests into provider protocols |
| Model Serving Cluster | Performs the actual model inference |
| Response Normalization | Unifies the response structures of different models |
| Streaming and Non-Streaming Response | Handles regular responses and incremental generation |
| Observability Module | Collects logs, metrics, traces, and cost data |
| Configuration Center | Manages models, routing, pricing, permissions, and policies |
2.2 Data Plane and Control Plane
Architecturally, a model gateway can be divided into a data plane and a control plane.
Data Plane
Responsible for actual request processing:
- Receiving requests.
- Validating requests.
- Performing model routing.
- Forwarding model calls.
- Processing model responses.
- Collecting invocation metrics.
The data plane sits on the critical request path and has high requirements for latency and availability.
Control Plane
Responsible for managing the operating policies of the data plane:
- Model registration and configuration.
- Routing rule management.
- Model pricing maintenance.
- Tenant quota management.
- Model permission management.
- Canary releases and policy updates.
- Provider health status management.
flowchart TB
A["Control Plane"]
B["Model Registry"]
C["Routing Policy Management"]
D["Permission and Quota Configuration"]
E["Pricing and Capability Configuration"]
F["Configuration Distribution"]
G["Data Plane"]
H["Request Ingress"]
I["Policy Enforcement"]
J["Model Routing"]
K["Model Invocation"]
L["Response Return"]
A --> B
A --> C
A --> D
A --> E
B --> F
C --> F
D --> F
E --> F
F --> G
G --> H
H --> I
I --> J
J --> K
K --> L
style A fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
This separation has several important benefits:
- Reduce request-path complexity: Model configuration and routing rules do not need to access a remote management service on every request.
- Improve availability: When the control plane is briefly unavailable, the data plane can continue using valid configuration from its local cache.
- Support centralized management: Multiple gateway instances can share a unified model catalog and policy configuration.
- Facilitate independent scaling: Request processing capacity and configuration management capacity can be scaled separately.
In production environments, control plane configuration changes are typically synchronized to the data plane through configuration push, event notifications, or periodic polling.
Note that configuration caching must include versioning, expiration, and rollback mechanisms; expired configuration must not be used indefinitely.
III. Unified Model Interface: Abstracting Provider Differences
One of the most fundamental capabilities of a model gateway is providing a unified model invocation interface.
3.1 Why Is a Unified Interface Needed?
APIs from different model providers may differ in the following ways:
- Request field names differ.
- Message roles and content structures differ.
- Parameter ranges and default values differ.
- Tool calling formats differ.
- Multimodal input structures differ.
- Error codes and error responses differ.
- Streaming response event formats differ.
If business systems call provider APIs directly, they must maintain these differences at the business layer.
The model gateway uses an internal unified data structure to isolate provider differences in the adaptation layer.
3.2 Defining a Unified Request Structure
Taking chat completion as an example, an internal request model can be abstracted as follows:
type ModelMessage = {
role: "system" | "user" | "assistant" | "tool";
content: string | Array<{
type: string;
text?: string;
imageUrl?: string;
}>;
};
type ModelRequest = {
model: string;
messages: ModelMessage[];
temperature?: number;
topP?: number;
maxTokens?: number;
stream?: boolean;
tools?: Array<{
type: "function";
function: {
name: string;
description?: string;
parameters: Record<string, unknown>;
};
}>;
metadata?: {
tenantId?: string;
userId?: string;
requestId?: string;
};
};
Here, ModelRequest is an internal abstraction structure of the gateway; it does not mean that all providers support every field in it.
For example:
- Some models do not support
temperature. - Some models use a different name to express the maximum number of generated tokens.
- Some models have differences in the parameter structure for tool calls.
- Multimodal input may require additional media type, resolution, or audio task configuration.
Therefore, a unified interface needs to be combined with model capability descriptions and parameter validation mechanisms.
3.3 Defining a Unified Response Structure
type ModelResponse = {
id: string;
model: string;
choices: Array<{
index: number;
message: {
role: "assistant";
content?: string;
toolCalls?: Array<{
id: string;
name: string;
arguments: string;
}>;
};
finishReason?: string;
}>;
usage?: {
promptTokens: number;
completionTokens: number;
totalTokens: number;
};
provider: string;
};
The unified response structure mainly solves three problems:
- Business systems do not need to understand the response formats of different providers.
- The gateway can uniformly collect token consumption and invocation results.
- When switching models, the business layer should require as few modifications as possible.
However, a unified interface should not discard model-specific capabilities in pursuit of consistency.
For provider-specific parameters, they can be preserved through extension fields, capability declarations, or dedicated interfaces.
3.4 The Boundary Between Unified Interfaces and Capability Abstraction
Model capabilities are not merely differences in input and output formats.
For example:
| Capability | Differences to consider |
|---|---|
| Text generation | Context length, generation length, sampling parameters |
| Structured output | Degree of JSON Schema support, constraint methods |
| Tool calling | Tool definition format, invocation behavior, parallel calling capability |
| Image understanding | Image formats, resolution, input limits |
| Audio understanding | Audio input formats, transcription and perception modes |
| Video understanding | Video frame input, timestamps, and sampling methods |
| Reasoning capability | Reasoning parameters, output structure, and token billing method |
| Embedding | Vector dimensions, normalization methods, input limits |
Therefore, the gateway cannot maintain only a model name and API address; it also needs to maintain model capability information.
4. Model Registry: Managing Model Metadata
The Model Registry is responsible for maintaining the models that the gateway can access and their related configurations.
Here, the Model Registry primarily refers to a runtime model catalog and is not necessarily equivalent to the model version repository in a machine learning training platform.
4.1 Model Metadata
A model record usually contains:
| Field | Description |
|---|---|
| modelId | Unique identifier within the gateway |
| provider | Model provider |
| providerModel | Actual model name at the provider |
| endpoint | Model service address |
| apiType | API protocol type |
| capabilities | Supported model capabilities |
| contextWindow | Maximum context window |
| maxOutputTokens | Maximum number of output tokens |
| inputPrice | Input token unit price |
| outputPrice | Output token unit price |
| status | Model enablement status |
| priority | Routing priority |
| timeout | Request timeout |
| credentialRef | Credential configuration reference |
Example model configuration:
{
"modelId": "general-chat",
"provider": "provider-a",
"providerModel": "model-x",
"endpoint": "https://api.example.com/v1",
"apiType": "openai-compatible",
"capabilities": [
"text-generation",
"tool-calling",
"streaming"
],
"contextWindow": 128000,
"maxOutputTokens": 8192,
"status": "enabled",
"priority": 10,
"timeout": 60000,
"credentialRef": "secret/model-provider-a"
}
The prices, context windows, and model capabilities in the example are only for demonstrating the configuration structure; actual values must be based on the latest documentation and deployment configuration of the corresponding model provider.
4.2 Logical Models and Physical Models
A model gateway needs to distinguish between logical models and physical models.
Logical Model
The model name exposed to business systems.
For example:
fast-chatadvanced-reasoningvision-modelembedding-model
Physical Model
The actual model service instance or vendor model that performs inference.
For example:
- A specific model version from a vendor.
- A model cluster deployed inside the enterprise.
- A model service instance on a cloud platform.
flowchart TB
A["Logical model: fast-chat"]
B["Routing policy"]
C["Physical model A"]
D["Physical model B"]
E["Physical model C"]
A --> B
B --> C
B --> D
B --> E
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
The business system calls fast-chat, and the gateway selects the actual model according to its policy.
This design frees business applications from having to care about the underlying model's vendor, deployment location, and specific version.
4.3 Model Capability Matrix
Routing based on model name alone is not enough.
For example, a request may require:
- Support for image input.
- Support for tool calling.
- Support for at least 64K context.
- Support for streaming output.
The gateway must therefore first filter out the models that meet these conditions.
A model capability matrix can be established:
| Model | Text generation | Image understanding | Tool calling | Streaming output | Long context |
|---|---|---|---|---|---|
| Model A | Yes | No | Yes | Yes | Yes |
| Model B | Yes | Yes | Yes | Yes | Yes |
| Model C | Yes | Yes | No | Yes | No |
| Model D | Yes | Yes | Yes | No | Yes |
The capability matrix can be used for:
- Pre-request validation.
- Filtering the set of candidate models.
- Intelligent routing.
- Determining compatibility when switching models.
- Impact analysis after model capabilities change.
Special attention is required: capability information should come from the vendor's official documentation, actual deployment configuration, and verified tests, rather than being inferred from the model name alone.
V. Model Routing: How Do You Choose the Right Model?
Model routing is one of the core capabilities of a model gateway.
It determines which model ultimately executes a request.
5.1 Routing Inputs
Model selection usually needs to take multiple factors into account:
| Factor | Description |
|---|---|
| Task type | General Q&A, code generation, reasoning, image understanding, etc. |
| Model capabilities | Whether the required input and output capabilities are supported |
| Context length | Whether the current request fits within the window |
| Latency requirements | Whether the business response-time target is met |
| Cost budget | Whether it fits the tenant or request budget |
| Model health status | Whether the service is available and not overloaded |
| Vendor restrictions | Region, data compliance, and access permissions |
| Business priority | Whether fallback models are permitted |
| User specification | Whether a fixed model or vendor is required |
Model routing is not simply about finding the cheapest model; it is constrained optimization across capability, quality, latency, cost, and availability.
5.2 Common Routing Strategies
1. Static routing
A fixed model is selected based on the model name or business configuration.
For example:
fast-chat → Model A
advanced-chat → Model B
vision-chat → Model C
embedding → Model D
Advantages:
- Simple logic.
- Stable behavior.
- Easy to troubleshoot.
Disadvantages:
- Cannot adjust to real-time load.
- Model failures require extra handling.
- Difficult to optimize cost dynamically.
Suitable for scenarios where business requirements are clear and model selection is stable.
2. Rule-Based Routing
Match routing conditions based on request attributes.
For example:
IF task = "vision"
THEN route to vision-model
IF task = "reasoning"
THEN route to reasoning-model
IF max_latency < 1000ms
THEN route to low-latency-model
Rule-based routing is suitable for enterprise scenarios with clear business constraints.
3. Cost-Based Routing
Among candidate models that meet capability and quality constraints, prioritize models with lower cost.
However, cost cannot be based only on input token price; the following also need to be considered:
- Output token price.
- Billing method for reasoning tokens.
- Price of cached tokens.
- Actual token consumption of different models.
- Additional costs caused by request failures and retries.
4. Latency-Based Routing
Select services based on real-time model latency, load, and availability.
Common metrics include:
- Average latency.
- P95 and P99 latency.
- Time To First Token (TTFT).
- Token generation speed.
- Request queue waiting time.
5. Quality-Based Routing
Select models with different capability levels based on task complexity.
For example:
- Use lightweight models for simple classification tasks.
- Use general-purpose models for ordinary Q&A.
- Use stronger models for complex code analysis.
- Use specially validated models for high-risk tasks.
Quality-based routing can use rules, classification models, or a dedicated routing model, but you cannot assume that the result is necessarily optimal based solely on the judgment of one routing model.
5.3 Routing Decision Process
flowchart TB
A["Receive model request"]
B["Parse task and constraints"]
C["Generate candidate model set"]
D["Are capabilities satisfied?"]
E["Are context and parameters compatible?"]
F["Is health status normal?"]
G["Calculate routing score"]
H["Select target model"]
I["Execute model call"]
J["Return routing failure"]
K["Record routing decision"]
A --> B
B --> C
C --> D
D -->|"Yes"| E
D -->|"No"| J
E -->|"Yes"| F
E -->|"No"| J
F -->|"Yes"| G
F -->|"No"| J
G --> H
H --> I
I --> K
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
The key here is to perform hard-constraint filtering first, then score and rank.
For example, if a model does not support image input, it cannot be selected to handle an image request even if it has the lowest price and latency.
5.4 Multi-Objective Routing Scoring
For candidate models that pass hard-constraint filtering, a composite score can be established.
For example:
Where:
- Q(m): Model quality score.
- L(m): Latency score.
- C(m): Cost score.
- A(m): Availability score.
- w_q,w_l,w_c,w_a: Weights for each dimension.
Note that lower latency and cost are generally better, so they should be converted into normalized metrics with consistent direction when scoring.
For example:
Final selection:
where M is the set of candidate models filtered by hard constraints.
This is an abstract scoring model, not a fixed algorithm that every gateway must adopt.
In production environments, the following also need to be considered:
- Whether metric weights change with the task.
- Whether latency and cost data are sufficiently stable.
- Whether model scores are backed by reliable evaluation data.
- Whether there is a hard upper limit in any dimension.
- Whether random exploration or traffic experiments need to be set up.
In particular, avoid directly mixing and comparing model quality scores across different tasks. Code generation, mathematical reasoning, and text classification require different quality evaluation standards.
6. Load Balancing: Distributing Model Service Traffic
Model routing addresses which model to choose, while load balancing addresses how requests are distributed across multiple model service instances.
When an enterprise deploys multiple instances of the same model, the gateway needs to distribute requests reasonably.
6.1 Common Load Balancing Strategies
| Strategy | How it works | Applicable scenarios |
|---|---|---|
| Round robin | Distributes requests in sequence | Instances have similar capabilities |
| Weighted round robin | Distributes requests according to instance weights | Instances have different performance |
| Random selection | Selects an instance at random | Simple traffic distribution |
| Least connections | Selects the instance with fewer current connections | Long-lived connections and streaming scenarios |
| Latency-aware | Prefers instances with faster responses | Latency-sensitive scenarios |
| Load-aware | Distributes based on queue length, GPU utilization, and other factors | Self-hosted inference clusters |
For large-model inference, simple round robin may not achieve ideal results.
The reason is that the computational cost of different requests varies greatly:
- Input prompt lengths differ.
- The number of output tokens differs.
- Prefill and Decode have different computational characteristics.
- Different requests use different amounts of KV Cache.
- An inference instance may be processing a large number of long-running requests.
Therefore, load balancing should ideally incorporate actual runtime metrics provided by the inference service.
6.2 Weighted Load Balancing
Suppose there are three inference instances:
| Instance | Weight |
|---|---|
| Instance A | 5 |
| Instance B | 3 |
| Instance C | 2 |
Ideally, the traffic distribution ratio is:
Weights can be set based on the number of GPUs, instance specifications, concurrency capacity, and historical processing capacity.
However, weights are not equal to actual available throughput. Instance load changes with request length, cache hit rate, and inference phase, so static weights usually need to be combined with real-time health status.
6.3 Model Instance Routing Flow
flowchart TB
A["Receive target model request"]
B["Get list of available instances"]
C["Filter out unhealthy instances"]
D["Read instance load"]
E["Calculate instance priority"]
F["Select target instance"]
G["Forward inference request"]
H["Update instance runtime metrics"]
I["No available instances"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
C -->|"No available instances"| I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
7. Fault Handling: Retries, Circuit Breaking, and Degradation
Model services are external dependencies and may experience:
- Network connection timeouts.
- API rate limiting.
- Model service overload.
- Upstream provider failure.
- Model returning errors.
- Requests taking too long to execute.
- Streaming connection interruption.
Without a unified fault-handling mechanism, an anomaly in one model service can affect the entire AI application.
7.1 Timeout Control
Model requests usually involve multiple time stages:
- Gateway processing time.
- Network connection time.
- Model queuing time.
- Prefill computation time.
- Decode generation time.
Therefore, you should not set only a single, overly short HTTP timeout.
You can distinguish:
- Connection timeout.
- First-token timeout.
- Overall request timeout.
- Idle stream timeout.
- Downstream client write timeout.
Different tasks should use different timeout budgets.
For example, short-text classification tasks and long-text generation tasks should not use exactly the same overall timeout policy.
7.2 Retry Mechanism
Retries are suitable for errors that are temporary and recoverable.
For example:
- The network connection temporarily fails.
- The upstream returns a retryable service error.
- A rate-limiting response explicitly indicates that retrying later is allowed.
However, the following cases are not suitable for blind retries:
- Request parameter errors.
- The model does not support the specified capability.
- Insufficient permissions.
- Insufficient balance or quota.
- A streaming request that has already started outputting content.
Retries also require attention to several issues:
First, control the number of retries.
Unlimited retries will amplify upstream failures.
Second, use exponential backoff.
For example:
Random jitter can be added to avoid many requests retrying at the same time.
Third, respect the retry budget.
If only 200ms remain in the total timeout for a request, you should not initiate a retry expected to take several seconds.
Fourth, distinguish whether the request has already produced side effects.
For ordinary model inference, duplicate requests usually mean extra cost; for tasks with tool calls or external side effects, repeated execution may cause real business problems.
7.3 Circuit Breaking Mechanism
A circuit breaker is used to prevent continuous calls to a faulty service.
Common states include:
- Closed: Normal calls.
- Open: Stop sending requests to the faulty service.
- Half-Open: Allow a small number of probe requests to determine whether the service has recovered.
stateDiagram-v2
[*] --> Closed
Closed --> Open: Consecutive failures reach threshold
Open --> HalfOpen: Circuit breaker wait time ends
HalfOpen --> Closed: Probe request succeeds
HalfOpen --> Open: Probe request fails
style Closed fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style Open fill:#F7EAE7,stroke:#C98D7C,color:#754438
style HalfOpen fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
Circuit breakers should be configured along reasonable fault-isolation dimensions, such as provider, model, or inference cluster.
If multiple completely independent model services share one circuit breaker state, a failure in one model may incorrectly affect other healthy models.
7.4 Fallback and Model Switching
When the primary model is unavailable, the gateway can try switching to a fallback model.
For example:
flowchart TB
A["Request primary model"]
B["Primary model call succeeds"]
C["Primary model call fails"]
D["Determine whether the error is switchable"]
E["Select fallback model"]
F["Fallback model call"]
G["Return model result"]
H["Return controlled error"]
I["Record fallback event"]
A --> B
A --> C
B --> G
C --> D
D -->|"Switch allowed"| E
D -->|"Switch not allowed"| H
E --> F
F -->|"Success"| G
F -->|"Failure"| H
G --> I
H --> I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style C fill:#F7EAE7,stroke:#C98D7C,color:#754438
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
Model fallback needs to consider compatibility:
- Does the fallback model support the same input types?
- Does it support the same tool-calling format?
- Does it have sufficient context length?
- Does it support the required structured output?
- Does its generation quality meet the minimum business requirements?
- Does it comply with data-region and privacy requirements?
Important: Not every failure should be treated as an error that can be resolved by switching models.
For example, authentication failures, parameter errors, and insufficient user quota usually should not be bypassed by switching models.
In addition, once a streaming response has already sent part of its content to the client, you cannot simply switch models and regenerate a complete answer; otherwise, duplicated content, semantic breaks, or protocol errors may occur.
VIII. Token Metering and Cost Management
An important difference between a model gateway and an ordinary API gateway is that it needs to understand model invocation costs.
Model invocation costs are usually related to input tokens, output tokens, cached tokens, reasoning tokens, and model type.
8.1 Token Usage Statistics
A single model request may produce:
- Prompt Tokens: the number of input tokens.
- Completion Tokens: the number of output tokens.
- Cached Tokens: the number of cached input tokens.
- Reasoning Tokens: reasoning tokens counted separately by some models.
- Total Tokens: the total number of tokens.
Different providers may have different token classifications and billing rules.
Therefore, the gateway should retain the provider's original usage information and convert it into a unified internal metering structure.
8.2 Cost Calculation
A simplified cost calculation model:
Where:
InputTokens: the number of input tokens.OutputTokens: the number of output tokens.InputPrice: price per million input tokens.OutputPrice: price per million output tokens.
If the provider prices cached tokens, reasoning tokens, or different input types separately, the calculation formula needs to be extended.
For example:
Where T_c and T_r represent cached tokens and separately priced reasoning tokens, respectively.
Actual billing should be based on the provider's billing definitions; you cannot simply calculate all tokens at the regular input or output price.
8.3 Multi-Tenant Quotas
Enterprises usually need to manage invocation quotas by tenant, user, application, or business line.
Common quotas include:
| Quota Type | Description |
|---|---|
| RPM | Requests per minute |
| TPM | Tokens per minute |
| Daily Quota | Daily invocation quota |
| Monthly Budget | Monthly budget |
| Concurrent Requests | Maximum number of concurrent requests |
| Max Input Tokens | Maximum input tokens per request |
| Max Output Tokens | Maximum output tokens per request |
Note that RPM and TPM are constraints on different dimensions.
A single request may consume only one invocation, but with a very large number of input and output tokens.
Therefore, rate limiting solely by request count cannot effectively control model costs.
8.4 Budget Control Process
flowchart TB
A["Receive tenant request"]
B["Identify tenant and application"]
C["Read quota and budget"]
D["Check RPM, TPM, and concurrency"]
E["Estimate cost of this request"]
F["Is the budget sufficient?"]
G["Execute model call"]
H["Settle actual usage"]
I["Update budget and metering records"]
J["Reject request"]
A --> B
B --> C
C --> D
D --> E
E --> F
F -->|"Allowed"| G
F -->|"Rejected"| J
G --> H
H --> I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
There is an easily overlooked problem in cost management: before a request is executed, the cost can only be estimated, and the actual number of output tokens usually cannot be determined until the model has finished.
Therefore, a production system needs to combine:
- Budget reservation before the request.
- A maximum output token limit.
- Usage monitoring while the request is executing.
- Settlement of the actual charges after the request completes.
- Billing handling for failed requests and partial output.
For concurrent budgets, you cannot simply rely on the logic of "read the balance first, then execute the request," otherwise multiple concurrent requests may all pass the check at the same time, causing total spending to exceed the budget.
A more reliable approach is to manage budgets through atomic quota reservation, transactions, or a centralized quota service.
9. Streaming Responses: Handling Continuously Generated Model Output
Large models usually take some time to complete an answer.
If the gateway has to wait for the model to generate the complete result before returning it, users must wait a long time before seeing any content.
Streaming responses allow the model to return generated results incrementally.
9.1 Non-streaming vs. streaming responses
Non-streaming response:
sequenceDiagram
participant App as Business application
participant Gateway as Model gateway
participant Model as Model service
App->>Gateway: Send complete request
Gateway->>Model: Forward request
Model-->>Gateway: Return complete result
Gateway-->>App: Return complete response
Streaming response:
sequenceDiagram
participant App as Business application
participant Gateway as Model gateway
participant Model as Model service
App->>Gateway: Send streaming request
Gateway->>Model: Forward streaming request
Model-->>Gateway: First data chunk
Gateway-->>App: Forward data chunk
Model-->>Gateway: Subsequent data chunks
Gateway-->>App: Keep forwarding
Model-->>Gateway: Completion event
Gateway-->>App: Send end event
9.2 Issues the gateway needs to handle
A streaming response is not simply a matter of forwarding SSE data; the following also need to be considered:
- Upstream stream connection management.
- Downstream client disconnects.
- Data chunk buffering.
- Heartbeats and idle timeouts.
- Conversion of streaming event formats.
- Token usage accounting.
- Propagation of streaming errors.
- Resource release.
In particular, avoid response buffering at the gateway layer that causes streaming data to accumulate to a certain size before it is sent.
Otherwise, even if the upstream model produces streaming output, users may still be unable to see the content in time.
9.3 Failure boundaries in streaming responses
Failure handling for streaming requests differs noticeably from that for ordinary requests.
If the model fails before returning the first token, the gateway usually still has a chance to switch to a backup model.
If the model has already returned part of the content: