Back to series
· 18 min read

Model Gateway Architecture and Design: Building a Unified, Stable, and Scalable LLM Access Layer


1. What Is a Model Gateway?

As AI applications scale up, systems often need to connect to multiple model services at the same time, including:

  • Commercial model APIs: OpenAI, Anthropic, Google Gemini, etc.
  • Cloud provider model services: managed model APIs offered by various cloud platforms.
  • Open-source model services: models deployed through inference frameworks such as vLLM, SGLang, and TGI.
  • Enterprise internal models: fine-tuned models, specialized models, and privately deployed models.

If every business system directly integrates with different models, it easily leads to a large amount of duplicate code and management problems:

  1. Inconsistent interfaces: Request parameters, response structures, and error formats differ across models.
  2. Model coupling: Business code directly depends on specific model names and provider APIs.
  3. Lack of unified governance: Authentication, rate limiting, quotas, and cost control are scattered across individual business systems.
  4. Complex failure handling: When a model service fails, business systems must handle switching and degradation themselves.
  5. Lack of global observability: It is difficult to uniformly track model invocation volume, token consumption, latency, and error rates.
  6. High model upgrade costs: Replacing a model may require modifying multiple business services.

The Model Gateway is a unified access layer designed to solve these problems.

It encapsulates underlying model services behind a unified invocation interface, so business systems do not need to be directly aware of implementation details from different providers.

1.1 Core Positioning of a Model Gateway

The model gateway sits between business applications and model services:

flowchart TB
    A["AI Business Applications"]
    B["Model Gateway"]
    C["Model Service A"]
    D["Model Service B"]
    E["Model Service C"]
    F["Private Model Service"]

    A --> B
    B --> C
    B --> D
    B --> E
    B --> F

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D

Business applications only need to call the model gateway and no longer directly manage the connections, authentication, and request formats of each model provider.

1.2 Differences Between a Model Gateway and an API Gateway

Both sit at the request entry point, but they solve different problems.

Comparison Dimension API Gateway Model Gateway
Core object HTTP APIs, microservices Large language models and inference services
Routing basis URL, HTTP method, service instance Model name, capabilities, cost, latency, context window
Request handling Path forwarding, protocol conversion Prompt, message, token, and model parameter adaptation
Load balancing Service instance load Model instances, providers, inference clusters
Cost management Request volume, traffic, resources Tokens, model pricing, inference cost
Failure handling Service retries, circuit breaking Model switching, degradation, context compatibility
Streaming HTTP, WebSocket, etc. SSE, streaming tokens, generation interruption
Observability QPS, latency, error rate Tokens, time to first token, generation speed, model invocation cost

A model gateway can be built on top of API Gateway infrastructure, but ordinary API forwarding capabilities cannot simply be equated with a model gateway.

Model invocation has its own characteristics: request lengths vary, responses may be generated continuously, cost is tied to token count, and different models differ significantly in capabilities and context limits.

Therefore, a model gateway needs dedicated governance capabilities for model inference scenarios.


2. Core Architecture of a Model Gateway

A production-grade model gateway is typically composed of modules such as request entry, protocol adaptation, policy decision, model routing, execution scheduling, and response processing.

2.1 Overall Architecture

flowchart TB
    A["Business Application"]
    B["Unified API Access Layer"]
    C["Authentication and Permission Verification"]
    D["Request Parsing and Normalization"]
    E["Policy Decision Engine"]
    F["Model Router"]
    G["Provider Adapter Layer"]
    H["Model Serving Cluster"]
    I["Response Normalization"]
    J["Streaming and Non-Streaming Response"]
    K["Logging, Metrics and Tracing"]
    L["Configuration Center and Model Registry"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J

    E -.-> L
    F -.-> L
    B -.-> K
    G -.-> K
    H -.-> K
    J --> A

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D

The responsibilities of each module are as follows:

Module Core Responsibilities
Unified API Access Layer Receives business requests and provides a unified entry point for invocation
Authentication and Permission Verification Verifies the caller's identity, tenant, and model permissions
Request Parsing and Normalization Parses requests and converts them into a unified internal structure
Policy Decision Engine Determines model selection, budget, priority, and constraint conditions
Model Router Selects model services that meet the conditions
Provider Adapter Layer Converts internal requests into provider protocols
Model Serving Cluster Performs the actual model inference
Response Normalization Unifies the response structures of different models
Streaming and Non-Streaming Response Handles regular responses and incremental generation
Observability Module Collects logs, metrics, traces, and cost data
Configuration Center Manages models, routing, pricing, permissions, and policies

2.2 Data Plane and Control Plane

Architecturally, a model gateway can be divided into a data plane and a control plane.

Data Plane

Responsible for actual request processing:

  • Receiving requests.
  • Validating requests.
  • Performing model routing.
  • Forwarding model calls.
  • Processing model responses.
  • Collecting invocation metrics.

The data plane sits on the critical request path and has high requirements for latency and availability.

Control Plane

Responsible for managing the operating policies of the data plane:

  • Model registration and configuration.
  • Routing rule management.
  • Model pricing maintenance.
  • Tenant quota management.
  • Model permission management.
  • Canary releases and policy updates.
  • Provider health status management.
flowchart TB
    A["Control Plane"]
    B["Model Registry"]
    C["Routing Policy Management"]
    D["Permission and Quota Configuration"]
    E["Pricing and Capability Configuration"]
    F["Configuration Distribution"]
    G["Data Plane"]
    H["Request Ingress"]
    I["Policy Enforcement"]
    J["Model Routing"]
    K["Model Invocation"]
    L["Response Return"]

    A --> B
    A --> C
    A --> D
    A --> E
    B --> F
    C --> F
    D --> F
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J
    J --> K
    K --> L

    style A fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F

This separation has several important benefits:

  1. Reduce request-path complexity: Model configuration and routing rules do not need to access a remote management service on every request.
  2. Improve availability: When the control plane is briefly unavailable, the data plane can continue using valid configuration from its local cache.
  3. Support centralized management: Multiple gateway instances can share a unified model catalog and policy configuration.
  4. Facilitate independent scaling: Request processing capacity and configuration management capacity can be scaled separately.

In production environments, control plane configuration changes are typically synchronized to the data plane through configuration push, event notifications, or periodic polling.

Note that configuration caching must include versioning, expiration, and rollback mechanisms; expired configuration must not be used indefinitely.


III. Unified Model Interface: Abstracting Provider Differences

One of the most fundamental capabilities of a model gateway is providing a unified model invocation interface.

3.1 Why Is a Unified Interface Needed?

APIs from different model providers may differ in the following ways:

  • Request field names differ.
  • Message roles and content structures differ.
  • Parameter ranges and default values differ.
  • Tool calling formats differ.
  • Multimodal input structures differ.
  • Error codes and error responses differ.
  • Streaming response event formats differ.

If business systems call provider APIs directly, they must maintain these differences at the business layer.

The model gateway uses an internal unified data structure to isolate provider differences in the adaptation layer.

3.2 Defining a Unified Request Structure

Taking chat completion as an example, an internal request model can be abstracted as follows:

type ModelMessage = {
  role: "system" | "user" | "assistant" | "tool";
  content: string | Array<{
    type: string;
    text?: string;
    imageUrl?: string;
  }>;
};

type ModelRequest = {
  model: string;
  messages: ModelMessage[];

  temperature?: number;
  topP?: number;
  maxTokens?: number;

  stream?: boolean;

  tools?: Array<{
    type: "function";
    function: {
      name: string;
      description?: string;
      parameters: Record<string, unknown>;
    };
  }>;

  metadata?: {
    tenantId?: string;
    userId?: string;
    requestId?: string;
  };
};

Here, ModelRequest is an internal abstraction structure of the gateway; it does not mean that all providers support every field in it.

For example:

  • Some models do not support temperature.
  • Some models use a different name to express the maximum number of generated tokens.
  • Some models have differences in the parameter structure for tool calls.
  • Multimodal input may require additional media type, resolution, or audio task configuration.

Therefore, a unified interface needs to be combined with model capability descriptions and parameter validation mechanisms.

3.3 Defining a Unified Response Structure

type ModelResponse = {
  id: string;
  model: string;

  choices: Array<{
    index: number;

    message: {
      role: "assistant";
      content?: string;

      toolCalls?: Array<{
        id: string;
        name: string;
        arguments: string;
      }>;
    };

    finishReason?: string;
  }>;

  usage?: {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
  };

  provider: string;
};

The unified response structure mainly solves three problems:

  1. Business systems do not need to understand the response formats of different providers.
  2. The gateway can uniformly collect token consumption and invocation results.
  3. When switching models, the business layer should require as few modifications as possible.

However, a unified interface should not discard model-specific capabilities in pursuit of consistency.

For provider-specific parameters, they can be preserved through extension fields, capability declarations, or dedicated interfaces.

3.4 The Boundary Between Unified Interfaces and Capability Abstraction

Model capabilities are not merely differences in input and output formats.

For example:

Capability Differences to consider
Text generation Context length, generation length, sampling parameters
Structured output Degree of JSON Schema support, constraint methods
Tool calling Tool definition format, invocation behavior, parallel calling capability
Image understanding Image formats, resolution, input limits
Audio understanding Audio input formats, transcription and perception modes
Video understanding Video frame input, timestamps, and sampling methods
Reasoning capability Reasoning parameters, output structure, and token billing method
Embedding Vector dimensions, normalization methods, input limits

Therefore, the gateway cannot maintain only a model name and API address; it also needs to maintain model capability information.


4. Model Registry: Managing Model Metadata

The Model Registry is responsible for maintaining the models that the gateway can access and their related configurations.

Here, the Model Registry primarily refers to a runtime model catalog and is not necessarily equivalent to the model version repository in a machine learning training platform.

4.1 Model Metadata

A model record usually contains:

Field Description
modelId Unique identifier within the gateway
provider Model provider
providerModel Actual model name at the provider
endpoint Model service address
apiType API protocol type
capabilities Supported model capabilities
contextWindow Maximum context window
maxOutputTokens Maximum number of output tokens
inputPrice Input token unit price
outputPrice Output token unit price
status Model enablement status
priority Routing priority
timeout Request timeout
credentialRef Credential configuration reference

Example model configuration:

{
  "modelId": "general-chat",
  "provider": "provider-a",
  "providerModel": "model-x",
  "endpoint": "https://api.example.com/v1",
  "apiType": "openai-compatible",
  "capabilities": [
    "text-generation",
    "tool-calling",
    "streaming"
  ],
  "contextWindow": 128000,
  "maxOutputTokens": 8192,
  "status": "enabled",
  "priority": 10,
  "timeout": 60000,
  "credentialRef": "secret/model-provider-a"
}

The prices, context windows, and model capabilities in the example are only for demonstrating the configuration structure; actual values must be based on the latest documentation and deployment configuration of the corresponding model provider.

4.2 Logical Models and Physical Models

A model gateway needs to distinguish between logical models and physical models.

Logical Model

The model name exposed to business systems.

For example:

  • fast-chat
  • advanced-reasoning
  • vision-model
  • embedding-model

Physical Model

The actual model service instance or vendor model that performs inference.

For example:

  • A specific model version from a vendor.
  • A model cluster deployed inside the enterprise.
  • A model service instance on a cloud platform.
flowchart TB
    A["Logical model: fast-chat"]
    B["Routing policy"]
    C["Physical model A"]
    D["Physical model B"]
    E["Physical model C"]

    A --> B
    B --> C
    B --> D
    B --> E

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D

The business system calls fast-chat, and the gateway selects the actual model according to its policy.

This design frees business applications from having to care about the underlying model's vendor, deployment location, and specific version.

4.3 Model Capability Matrix

Routing based on model name alone is not enough.

For example, a request may require:

  • Support for image input.
  • Support for tool calling.
  • Support for at least 64K context.
  • Support for streaming output.

The gateway must therefore first filter out the models that meet these conditions.

A model capability matrix can be established:

Model Text generation Image understanding Tool calling Streaming output Long context
Model A Yes No Yes Yes Yes
Model B Yes Yes Yes Yes Yes
Model C Yes Yes No Yes No
Model D Yes Yes Yes No Yes

The capability matrix can be used for:

  1. Pre-request validation.
  2. Filtering the set of candidate models.
  3. Intelligent routing.
  4. Determining compatibility when switching models.
  5. Impact analysis after model capabilities change.

Special attention is required: capability information should come from the vendor's official documentation, actual deployment configuration, and verified tests, rather than being inferred from the model name alone.


V. Model Routing: How Do You Choose the Right Model?

Model routing is one of the core capabilities of a model gateway.

It determines which model ultimately executes a request.

5.1 Routing Inputs

Model selection usually needs to take multiple factors into account:

Factor Description
Task type General Q&A, code generation, reasoning, image understanding, etc.
Model capabilities Whether the required input and output capabilities are supported
Context length Whether the current request fits within the window
Latency requirements Whether the business response-time target is met
Cost budget Whether it fits the tenant or request budget
Model health status Whether the service is available and not overloaded
Vendor restrictions Region, data compliance, and access permissions
Business priority Whether fallback models are permitted
User specification Whether a fixed model or vendor is required

Model routing is not simply about finding the cheapest model; it is constrained optimization across capability, quality, latency, cost, and availability.

5.2 Common Routing Strategies

1. Static routing

A fixed model is selected based on the model name or business configuration.

For example:

fast-chat        → Model A
advanced-chat    → Model B
vision-chat      → Model C
embedding        → Model D

Advantages:

  • Simple logic.
  • Stable behavior.
  • Easy to troubleshoot.

Disadvantages:

  • Cannot adjust to real-time load.
  • Model failures require extra handling.
  • Difficult to optimize cost dynamically.

Suitable for scenarios where business requirements are clear and model selection is stable.

2. Rule-Based Routing

Match routing conditions based on request attributes.

For example:

IF task = "vision"
    THEN route to vision-model

IF task = "reasoning"
    THEN route to reasoning-model

IF max_latency < 1000ms
    THEN route to low-latency-model

Rule-based routing is suitable for enterprise scenarios with clear business constraints.

3. Cost-Based Routing

Among candidate models that meet capability and quality constraints, prioritize models with lower cost.

However, cost cannot be based only on input token price; the following also need to be considered:

  • Output token price.
  • Billing method for reasoning tokens.
  • Price of cached tokens.
  • Actual token consumption of different models.
  • Additional costs caused by request failures and retries.

4. Latency-Based Routing

Select services based on real-time model latency, load, and availability.

Common metrics include:

  • Average latency.
  • P95 and P99 latency.
  • Time To First Token (TTFT).
  • Token generation speed.
  • Request queue waiting time.

5. Quality-Based Routing

Select models with different capability levels based on task complexity.

For example:

  • Use lightweight models for simple classification tasks.
  • Use general-purpose models for ordinary Q&A.
  • Use stronger models for complex code analysis.
  • Use specially validated models for high-risk tasks.

Quality-based routing can use rules, classification models, or a dedicated routing model, but you cannot assume that the result is necessarily optimal based solely on the judgment of one routing model.

5.3 Routing Decision Process

flowchart TB
    A["Receive model request"]
    B["Parse task and constraints"]
    C["Generate candidate model set"]
    D["Are capabilities satisfied?"]
    E["Are context and parameters compatible?"]
    F["Is health status normal?"]
    G["Calculate routing score"]
    H["Select target model"]
    I["Execute model call"]
    J["Return routing failure"]
    K["Record routing decision"]

    A --> B
    B --> C
    C --> D
    D -->|"Yes"| E
    D -->|"No"| J
    E -->|"Yes"| F
    E -->|"No"| J
    F -->|"Yes"| G
    F -->|"No"| J
    G --> H
    H --> I
    I --> K

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

The key here is to perform hard-constraint filtering first, then score and rank.

For example, if a model does not support image input, it cannot be selected to handle an image request even if it has the lowest price and latency.

5.4 Multi-Objective Routing Scoring

For candidate models that pass hard-constraint filtering, a composite score can be established.

For example:

Score(m)= w_q Q(m)+ w_l L(m)+ w_c C(m)+ w_a A(m)

Where:

  • Q(m): Model quality score.
  • L(m): Latency score.
  • C(m): Cost score.
  • A(m): Availability score.
  • w_q,w_l,w_c,w_a: Weights for each dimension.

Note that lower latency and cost are generally better, so they should be converted into normalized metrics with consistent direction when scoring.

For example:

L(m)=1-\operatorname{NormalizeLatency}(m)
C(m)=1-\operatorname{NormalizeCost}(m)

Final selection:

m^\*=\arg\max_{m\in M} Score(m)

where M is the set of candidate models filtered by hard constraints.

This is an abstract scoring model, not a fixed algorithm that every gateway must adopt.

In production environments, the following also need to be considered:

  • Whether metric weights change with the task.
  • Whether latency and cost data are sufficiently stable.
  • Whether model scores are backed by reliable evaluation data.
  • Whether there is a hard upper limit in any dimension.
  • Whether random exploration or traffic experiments need to be set up.

In particular, avoid directly mixing and comparing model quality scores across different tasks. Code generation, mathematical reasoning, and text classification require different quality evaluation standards.


6. Load Balancing: Distributing Model Service Traffic

Model routing addresses which model to choose, while load balancing addresses how requests are distributed across multiple model service instances.

When an enterprise deploys multiple instances of the same model, the gateway needs to distribute requests reasonably.

6.1 Common Load Balancing Strategies

Strategy How it works Applicable scenarios
Round robin Distributes requests in sequence Instances have similar capabilities
Weighted round robin Distributes requests according to instance weights Instances have different performance
Random selection Selects an instance at random Simple traffic distribution
Least connections Selects the instance with fewer current connections Long-lived connections and streaming scenarios
Latency-aware Prefers instances with faster responses Latency-sensitive scenarios
Load-aware Distributes based on queue length, GPU utilization, and other factors Self-hosted inference clusters

For large-model inference, simple round robin may not achieve ideal results.

The reason is that the computational cost of different requests varies greatly:

  • Input prompt lengths differ.
  • The number of output tokens differs.
  • Prefill and Decode have different computational characteristics.
  • Different requests use different amounts of KV Cache.
  • An inference instance may be processing a large number of long-running requests.

Therefore, load balancing should ideally incorporate actual runtime metrics provided by the inference service.

6.2 Weighted Load Balancing

Suppose there are three inference instances:

Instance Weight
Instance A 5
Instance B 3
Instance C 2

Ideally, the traffic distribution ratio is:

P(A)=\frac{5}{10}=50\%
P(B)=\frac{3}{10}=30\%
P(C)=\frac{2}{10}=20\%

Weights can be set based on the number of GPUs, instance specifications, concurrency capacity, and historical processing capacity.

However, weights are not equal to actual available throughput. Instance load changes with request length, cache hit rate, and inference phase, so static weights usually need to be combined with real-time health status.

6.3 Model Instance Routing Flow

flowchart TB
    A["Receive target model request"]
    B["Get list of available instances"]
    C["Filter out unhealthy instances"]
    D["Read instance load"]
    E["Calculate instance priority"]
    F["Select target instance"]
    G["Forward inference request"]
    H["Update instance runtime metrics"]
    I["No available instances"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    C -->|"No available instances"| I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F7EAE7,stroke:#C98D7C,color:#754438

7. Fault Handling: Retries, Circuit Breaking, and Degradation

Model services are external dependencies and may experience:

  • Network connection timeouts.
  • API rate limiting.
  • Model service overload.
  • Upstream provider failure.
  • Model returning errors.
  • Requests taking too long to execute.
  • Streaming connection interruption.

Without a unified fault-handling mechanism, an anomaly in one model service can affect the entire AI application.

7.1 Timeout Control

Model requests usually involve multiple time stages:

  1. Gateway processing time.
  2. Network connection time.
  3. Model queuing time.
  4. Prefill computation time.
  5. Decode generation time.

Therefore, you should not set only a single, overly short HTTP timeout.

You can distinguish:

  • Connection timeout.
  • First-token timeout.
  • Overall request timeout.
  • Idle stream timeout.
  • Downstream client write timeout.

Different tasks should use different timeout budgets.

For example, short-text classification tasks and long-text generation tasks should not use exactly the same overall timeout policy.

7.2 Retry Mechanism

Retries are suitable for errors that are temporary and recoverable.

For example:

  • The network connection temporarily fails.
  • The upstream returns a retryable service error.
  • A rate-limiting response explicitly indicates that retrying later is allowed.

However, the following cases are not suitable for blind retries:

  • Request parameter errors.
  • The model does not support the specified capability.
  • Insufficient permissions.
  • Insufficient balance or quota.
  • A streaming request that has already started outputting content.

Retries also require attention to several issues:

First, control the number of retries.

Unlimited retries will amplify upstream failures.

Second, use exponential backoff.

For example:

Delay_n=\min(Delay_{\max},Delay_0\times 2^n)

Random jitter can be added to avoid many requests retrying at the same time.

Third, respect the retry budget.

If only 200ms remain in the total timeout for a request, you should not initiate a retry expected to take several seconds.

Fourth, distinguish whether the request has already produced side effects.

For ordinary model inference, duplicate requests usually mean extra cost; for tasks with tool calls or external side effects, repeated execution may cause real business problems.

7.3 Circuit Breaking Mechanism

A circuit breaker is used to prevent continuous calls to a faulty service.

Common states include:

  • Closed: Normal calls.
  • Open: Stop sending requests to the faulty service.
  • Half-Open: Allow a small number of probe requests to determine whether the service has recovered.
stateDiagram-v2
    [*] --> Closed

    Closed --> Open: Consecutive failures reach threshold
    Open --> HalfOpen: Circuit breaker wait time ends
    HalfOpen --> Closed: Probe request succeeds
    HalfOpen --> Open: Probe request fails

    style Closed fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style Open fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style HalfOpen fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Circuit breakers should be configured along reasonable fault-isolation dimensions, such as provider, model, or inference cluster.

If multiple completely independent model services share one circuit breaker state, a failure in one model may incorrectly affect other healthy models.

7.4 Fallback and Model Switching

When the primary model is unavailable, the gateway can try switching to a fallback model.

For example:

flowchart TB
    A["Request primary model"]
    B["Primary model call succeeds"]
    C["Primary model call fails"]
    D["Determine whether the error is switchable"]
    E["Select fallback model"]
    F["Fallback model call"]
    G["Return model result"]
    H["Return controlled error"]
    I["Record fallback event"]

    A --> B
    A --> C
    B --> G
    C --> D
    D -->|"Switch allowed"| E
    D -->|"Switch not allowed"| H
    E --> F
    F -->|"Success"| G
    F -->|"Failure"| H
    G --> I
    H --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style C fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D

Model fallback needs to consider compatibility:

  1. Does the fallback model support the same input types?
  2. Does it support the same tool-calling format?
  3. Does it have sufficient context length?
  4. Does it support the required structured output?
  5. Does its generation quality meet the minimum business requirements?
  6. Does it comply with data-region and privacy requirements?

Important: Not every failure should be treated as an error that can be resolved by switching models.

For example, authentication failures, parameter errors, and insufficient user quota usually should not be bypassed by switching models.

In addition, once a streaming response has already sent part of its content to the client, you cannot simply switch models and regenerate a complete answer; otherwise, duplicated content, semantic breaks, or protocol errors may occur.


VIII. Token Metering and Cost Management

An important difference between a model gateway and an ordinary API gateway is that it needs to understand model invocation costs.

Model invocation costs are usually related to input tokens, output tokens, cached tokens, reasoning tokens, and model type.

8.1 Token Usage Statistics

A single model request may produce:

  • Prompt Tokens: the number of input tokens.
  • Completion Tokens: the number of output tokens.
  • Cached Tokens: the number of cached input tokens.
  • Reasoning Tokens: reasoning tokens counted separately by some models.
  • Total Tokens: the total number of tokens.

Different providers may have different token classifications and billing rules.

Therefore, the gateway should retain the provider's original usage information and convert it into a unified internal metering structure.

8.2 Cost Calculation

A simplified cost calculation model:

Cost = \frac{InputTokens}{10^6}\times InputPrice + \frac{OutputTokens}{10^6}\times OutputPrice

Where:

  • InputTokens: the number of input tokens.
  • OutputTokens: the number of output tokens.
  • InputPrice: price per million input tokens.
  • OutputPrice: price per million output tokens.

If the provider prices cached tokens, reasoning tokens, or different input types separately, the calculation formula needs to be extended.

For example:

\begin{aligned} Cost ={}& \frac{T_i \times P_i}{10^6} +\frac{T_o \times P_o}{10^6}\\ &+\frac{T_c \times P_c}{10^6} +\frac{T_r \times P_r}{10^6} \end{aligned}

Where T_c and T_r represent cached tokens and separately priced reasoning tokens, respectively.

Actual billing should be based on the provider's billing definitions; you cannot simply calculate all tokens at the regular input or output price.

8.3 Multi-Tenant Quotas

Enterprises usually need to manage invocation quotas by tenant, user, application, or business line.

Common quotas include:

Quota Type Description
RPM Requests per minute
TPM Tokens per minute
Daily Quota Daily invocation quota
Monthly Budget Monthly budget
Concurrent Requests Maximum number of concurrent requests
Max Input Tokens Maximum input tokens per request
Max Output Tokens Maximum output tokens per request

Note that RPM and TPM are constraints on different dimensions.

A single request may consume only one invocation, but with a very large number of input and output tokens.

Therefore, rate limiting solely by request count cannot effectively control model costs.

8.4 Budget Control Process

flowchart TB
    A["Receive tenant request"]
    B["Identify tenant and application"]
    C["Read quota and budget"]
    D["Check RPM, TPM, and concurrency"]
    E["Estimate cost of this request"]
    F["Is the budget sufficient?"]
    G["Execute model call"]
    H["Settle actual usage"]
    I["Update budget and metering records"]
    J["Reject request"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F -->|"Allowed"| G
    F -->|"Rejected"| J
    G --> H
    H --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438

There is an easily overlooked problem in cost management: before a request is executed, the cost can only be estimated, and the actual number of output tokens usually cannot be determined until the model has finished.

Therefore, a production system needs to combine:

  1. Budget reservation before the request.
  2. A maximum output token limit.
  3. Usage monitoring while the request is executing.
  4. Settlement of the actual charges after the request completes.
  5. Billing handling for failed requests and partial output.

For concurrent budgets, you cannot simply rely on the logic of "read the balance first, then execute the request," otherwise multiple concurrent requests may all pass the check at the same time, causing total spending to exceed the budget.

A more reliable approach is to manage budgets through atomic quota reservation, transactions, or a centralized quota service.


9. Streaming Responses: Handling Continuously Generated Model Output

Large models usually take some time to complete an answer.

If the gateway has to wait for the model to generate the complete result before returning it, users must wait a long time before seeing any content.

Streaming responses allow the model to return generated results incrementally.

9.1 Non-streaming vs. streaming responses

Non-streaming response:

sequenceDiagram
    participant App as Business application
    participant Gateway as Model gateway
    participant Model as Model service

    App->>Gateway: Send complete request
    Gateway->>Model: Forward request
    Model-->>Gateway: Return complete result
    Gateway-->>App: Return complete response

Streaming response:

sequenceDiagram
    participant App as Business application
    participant Gateway as Model gateway
    participant Model as Model service

    App->>Gateway: Send streaming request
    Gateway->>Model: Forward streaming request
    Model-->>Gateway: First data chunk
    Gateway-->>App: Forward data chunk
    Model-->>Gateway: Subsequent data chunks
    Gateway-->>App: Keep forwarding
    Model-->>Gateway: Completion event
    Gateway-->>App: Send end event

9.2 Issues the gateway needs to handle

A streaming response is not simply a matter of forwarding SSE data; the following also need to be considered:

  • Upstream stream connection management.
  • Downstream client disconnects.
  • Data chunk buffering.
  • Heartbeats and idle timeouts.
  • Conversion of streaming event formats.
  • Token usage accounting.
  • Propagation of streaming errors.
  • Resource release.

In particular, avoid response buffering at the gateway layer that causes streaming data to accumulate to a certain size before it is sent.

Otherwise, even if the upstream model produces streaming output, users may still be unable to see the content in time.

9.3 Failure boundaries in streaming responses

Failure handling for streaming requests differs noticeably from that for ordinary requests.

If the model fails before returning the first token, the gateway usually still has a chance to switch to a backup model.

If the model has already returned part of the content: