Back to series
· 19 min read

High-Availability Design for AI Model Services


I. Why AI Model Serving Needs High Availability Design

Compared with traditional web services, AI model serving has a more complex runtime environment and resource dependencies.

Traditional web services can usually scale horizontally by adding application instances, whereas AI model serving must also account for model weights, GPU memory, inference engines, KV Cache, model loading time, and coordination among compute resources.

A complete AI model serving system may depend on:

  • API Gateway: receives requests and performs authentication, routing, and rate limiting.
  • Model Router: selects inference instances based on model version, load, and resource status.
  • Model Server: loads models and executes inference.
  • GPU Worker: provides GPU compute resources.
  • Cache: caches model weights, Prefix Cache, KV Cache, or inference results.
  • Model Registry: manages model files, versions, and deployment configurations.
  • Object Storage: stores model weights, datasets, and inference results.
  • Scheduler: handles task scheduling and resource allocation.
  • Monitoring: collects runtime metrics, logs, and distributed tracing data.

A failure in any one of these critical links can affect user requests.

1.1 Typical Failures in AI Model Serving

Failure Type Typical Symptom Impact
GPU failure GPU Xid, device dropout, compute errors Inference instance unavailable
Out of GPU memory CUDA Out of Memory Request failure or process crash
Model loading failure Corrupted weights, download timeout Instance fails to start
Inference process crash Segmentation Fault, Runtime Error Current instance stops serving
Network failure Connection interruption between nodes Request timeout or lost results
Cache failure Redis unavailable, cache avalanche Increased latency, higher compute pressure
Traffic surge Concurrency exceeds processing capacity Request backlog, increased timeouts
Scheduler failure Unable to allocate GPU or Worker New requests cannot be executed
Model version failure New model has compatibility issues Inference errors or performance degradation
Dependency service failure Tokenizer, vector database, etc. unavailable Inference pipeline interrupted

High availability design does not mean guaranteeing that no component ever fails; rather, it ensures that after a failure occurs:

  1. The failure can be detected promptly.
  2. The scope of the failure can be isolated.
  3. Automatic failover or recovery can occur.
  4. The remaining healthy resources can be protected.
  5. Core business capabilities can be maintained as far as possible.

1.2 Core Metrics of High Availability

High availability design usually requires defining several key metrics.

Availability

Indicates the proportion of time within a specified period that the service can provide normal service.

Availability= \frac{Uptime}{TotalTime}\times100\%

It can also be calculated from downtime:

Availability= 1-\frac{Downtime}{TotalTime}

Common availability targets:

Availability Target Theoretical Annual Downtime
99% About 87.6 hours
99.9% About 8.76 hours
99.95% About 4.38 hours
99.99% About 52.56 minutes
99.999% About 5.26 minutes

The downtime figures here are theoretical budgets; whether they are actually met depends on the definition of the service level objective (SLO) and the statistical window.

SLO (Service Level Objective)

A service level objective defines the level of reliability the system must achieve.

For example:

  • Monthly availability of no less than 99.95%.
  • Online inference request success rate of no less than 99.9%.
  • P95 TTFT of no more than 2 seconds.
  • P99 inference request error rate below a specified threshold.

SLI (Service Level Indicator)

A service level indicator is the actual observed data used to measure an SLO.

For example:

  • Successful requests / total requests.
  • Normal service time / total service time.
  • P95 and P99 request latency.
  • Timeout rate.
  • Model inference failure rate.

RTO (Recovery Time Objective)

Recovery time objective, the maximum recovery time allowed after a failure occurs.

RPO (Recovery Point Objective)

Recovery point objective, the tolerable range of data loss when a failure occurs.

For stateless inference instances, it is usually unnecessary to restore historical inference progress; however, model configurations, task states, session records, and business data may require a strict RPO.

II. High Availability Architecture for AI Model Services

2.1 Problems with Single-Instance Deployment

The simplest model service architecture is an API service connected to a single model inference instance.

flowchart TD
    A["User Request"] --> B["API Service"]
    B --> C["Single Model Inference Instance"]
    C --> D["GPU"]
    D --> E["Return Result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#E5F0E9,stroke:#83A995,color:#315D49

This architecture is simple, but it has obvious single points of failure:

  • If the API service crashes, all requests cannot enter.
  • If the GPU fails, the model cannot continue inference.
  • If the model process runs OOM, the entire service becomes unavailable.
  • During model upgrades, the service may need to be taken offline.
  • When traffic exceeds the capacity of a single instance, all requests are affected.

This deployment is suitable for development, testing, and low-traffic internal applications, but not for production environments with clear SLA requirements.

2.2 Multi-Instance Redundant Architecture

The first step to improving availability is to deploy multiple independent model instances.

flowchart TD
    A["User Request"] --> B["API Gateway"]
    B --> C["Load Balancer"]
    C --> D["Model Server A"]
    C --> E["Model Server B"]
    C --> F["Model Server C"]
    D --> G["GPU Node A"]
    E --> H["GPU Node B"]
    F --> I["GPU Node C"]
    G --> J["Inference Result"]
    H --> J
    I --> J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

Multiple instances can achieve:

  • Switching to other instances when an instance fails.
  • Distributing traffic across multiple GPU nodes.
  • Supporting rolling upgrades.
  • Supporting dynamic scaling based on load.
  • Reducing the impact of a single-node failure on the overall service.

However, note that multiple instances do not necessarily mean true high availability.

If all three model instances are deployed on the same physical server, a server failure will still render all instances unavailable.

Therefore, redundancy must cover different fault domains.

2.3 Multi-Layer High Availability Architecture

Production-grade AI model services typically adopt a multi-layer redundancy design.

flowchart TD
    A["Client"] --> B["DNS / Global Traffic Management"]
    B --> C["Region A Entry"]
    B --> D["Region B Entry"]
    C --> E["API Gateway Cluster"]
    D --> F["API Gateway Cluster"]
    E --> G["Region A Model Router"]
    F --> H["Region B Model Router"]
    G --> I["Model Inference Instance Pool A"]
    H --> J["Model Inference Instance Pool B"]
    I --> K["GPU Resource Pool A"]
    J --> L["GPU Resource Pool B"]
    K --> M["Shared Model Artifacts and Configuration Management"]
    L --> M
    E --> N["Monitoring and Alerting"]
    F --> N
    G --> N
    H --> N

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style M fill:#E5F0E9,stroke:#83A995,color:#315D49
    style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

This architecture eliminates single points of failure at multiple levels:

Layer High Availability Measures
DNS / Traffic Entry Multi-region entry, health checks, traffic failover
API Gateway Multiple instances, stateless deployment
Model Router Multiple replicas, externalized state
Model Server Multi-instance redundancy
GPU Nodes Cross-host and cross-rack deployment
Model Artifacts Multi-replica storage, verification, and version management
Metadata Storage Primary-replica replication, clustering, or multi-region disaster recovery
Monitoring System Independent deployment, fault alerting

Two concepts need to be distinguished:

  • High Availability (HA): When a failure occurs, the system can continue to provide service or recover quickly.
  • Disaster Recovery (DR): When a region-level, data center-level, or large-scale infrastructure failure occurs, the system can recover business operations.

Multi-region deployment is a higher level of disaster recovery capability, but it also increases costs for data synchronization, networking, model distribution, and operations and maintenance.

3. Failure Domains and Redundancy Design

The key to a highly available architecture is not merely increasing the number of instances, but localizing failures as much as possible.

3.1 Failure Domains

A failure domain refers to a group of resources that may be affected simultaneously by the same failure cause.

Common failure domains include:

Failure Domain Typical Failure
Process level Python process crash, memory leak
Container level Container exit, health check failure
Host level Operating system crash, GPU failure
Rack level Power, switch, or network failure
Availability zone level Data center network, power, or infrastructure failure
Region level Large-scale cloud service or network failure

If multiple model instances share the same failure domain, their failures are not independent.

Therefore, during deployment, they should be distributed as much as possible:

  • Different physical hosts.
  • Different GPU nodes.
  • Different racks.
  • Different availability zones.
  • Different regions.

3.2 N+1 Redundancy

N+1 redundancy means the system requires N instances for normal operation, while one additional standby instance is prepared.

For example:

  • Normally requires 4 model instances.
  • 5 instances are actually deployed.
  • When one instance fails, the remaining 4 instances continue to handle the workload.

However, N+1 does not necessarily apply to all GPU services.

If GPU resources are very expensive, the following may be adopted:

  • N+1 hot standby.
  • Partial capacity reservation.
  • Multiple instances share redundant capacity.
  • Cross-region cold standby.

The choice should be based on the recovery time objective and cost.

3.3 Active-Active: Dual-Active Deployment

Active-Active means multiple instances process requests at the same time.

For example:

Region A: handles 50% of requests
Region B: handles 50% of requests

Advantages:

  • All nodes can provide actual service.
  • Resources are not left idle for long periods.
  • Failover is usually faster.
  • Traffic can be shared across regions.

Disadvantages:

  • Model versions need to remain compatible.
  • Data and cache consistency must be considered.
  • Cross-region traffic may increase network latency.
  • Multi-region resource scheduling is more complex.

For stateless inference services, Active-Active is usually relatively easy to implement.

However, for stateful sessions, task execution progress, and distributed inference, additional state synchronization and recovery mechanisms need to be designed.

3.4 Active-Standby: Primary-Standby Deployment

Active-Standby means the primary instance handles requests, and the standby instance takes over upon failure.

Normal state:
Primary -> handles all requests
Standby -> waits to take over

Failure state:
Primary -> unavailable
Standby -> takes over requests

Advantages:

  • The architecture is relatively simple.
  • Standby resources can be configured on demand.
  • Suitable for services with low traffic but requiring fast recovery.

Disadvantages:

  • Standby resources may remain idle for long periods.
  • Health checks and failover mechanisms are required.
  • The standby model may not yet be loaded onto the GPU.
  • Failover may involve significant cold-start time.

For large models, loading weights on the standby node may take a long time, so services that truly require a low RTO usually need to pre-warm standby instances.

4. Health Checks for Model Instances

A high-availability system must be able to determine whether a model instance can actually process requests.

Traditional web services can usually make a basic determination through HTTP health checks, but the health status of AI model services is more complex.

4.1 Three Basic Health Checks

Liveness Probe: Liveness Check

Checks whether the process is still alive.

For example:

  • Whether the HTTP service can respond.
  • Whether the main loop is still running.
  • Whether critical threads are functioning normally.
  • Whether a deadlock has occurred.

If the liveness probe fails, the platform can restart the container.

However, the liveness probe should not rely too heavily on model inference; otherwise, a brief period of GPU busyness may cause the container to restart repeatedly.

Readiness Probe: Readiness Check

Checks whether the instance is already capable of handling real requests.

Typical checks include:

  • Whether model weights have finished loading.
  • Whether the tokenizer is available.
  • Whether the GPU can be accessed normally.
  • Whether the inference engine has finished initializing.
  • Whether critical dependencies meet the requirements.
  • Whether it is currently draining or upgrading.

When readiness fails, the instance should be removed from the traffic pool, but it does not necessarily need to be restarted.

Startup Probe: Startup Check

Used to protect services with long startup times.

Large models may take a long time to:

  • Download model weights.
  • Load weights onto the GPU.
  • Initialize CUDA.
  • Create the KV Cache management structure.
  • Perform warm-up inference.

The startup probe can prevent the service from being prematurely judged as abnormal during startup.

4.2 Health Status of AI Model Services

It is recommended to divide the states of a model instance into:

State Meaning Accepts New Requests
STARTING Starting No
LOADING Loading model No
WARMING Warming up Usually no
READY Can serve normally Yes
DEGRADED Performance degraded or some capabilities are abnormal Depends on policy
DRAINING Draining requests No
UNHEALTHY Health check failed No
TERMINATED Stopped No

Health state transitions:

flowchart TD
    A["STARTING"] --> B["LOADING"]
    B --> C["WARMING"]
    C --> D{"Warm-up successful?"}
    D -->|Yes| E["READY"]
    D -->|No| F["UNHEALTHY"]
    E --> G{"Operating normally?"}
    G -->|Yes| E
    G -->|Partially abnormal| H["DEGRADED"]
    H --> I{"Recovered?"}
    I -->|Yes| E
    I -->|No| F
    E --> J["DRAINING"]
    H --> J
    J --> K["TERMINATED"]
    F --> L["Restart or reschedule"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style K fill:#E5F0E9,stroke:#83A995,color:#315D49
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49

4.3 Why health checks cannot rely solely on HTTP 200

Model services may encounter the following situations:

  • The HTTP process is healthy, but the GPU is already unavailable.
  • The model has finished loading, but a deadlock occurs inside the inference engine.
  • The service can return health check results, but the inference queue is severely backlogged.
  • The model can perform inference, but actual outputs remain abnormal.
  • The GPU is normal, but a critical Tokenizer file is missing.

Therefore, health checks should be split into multiple layers:

  1. Process health.
  2. Service health.
  3. Model health.
  4. GPU health.
  5. Inference capability health.
  6. Performance health.

Performance health, in particular, can be judged by latency, queue backlog, timeout rate, and resource pressure, rather than relying only on a boolean state.

V. Traffic Management and Load Balancing

After multiple model instances are deployed, you still need to decide which instance each request is sent to.

5.1 Load Balancing Strategies

Common strategies are as follows:

Strategy How it works Use cases
Round Robin Distributes requests in turn Instances have similar capacity
Weighted Round Robin Distributes requests by weight Instances have different specifications
Least Connections Selects the instance with fewer connections Request durations vary widely
Least Outstanding Requests Selects the instance with fewer outstanding requests Inference durations differ significantly
Latency-aware Selects instances based on historical latency Low latency is the goal
Resource-aware Routes based on GPU, VRAM, and queue status AI inference services
Consistent Hashing Maps stably by Key Cache locality is required

For AI model services, ordinary Round Robin is often insufficient.

This is because the computational cost of different requests can vary greatly.

For example:

  • Request A: 100 input tokens, 50 output tokens.
  • Request B: 10000 input tokens, 2000 output tokens.

If the two requests are treated as having the same workload, simple round-robin may cause some GPU nodes to accumulate many long requests.

A more reasonable approach is to schedule based on the following metrics:

  • Current number of active requests.
  • Number of queued requests.
  • GPU compute utilization.
  • GPU memory usage.
  • Available KV Cache space.
  • Number of input tokens per request.
  • Estimated number of output tokens.
  • Estimated execution time.
  • Prefix Cache hit status.

5.2 Request Routing Architecture

flowchart TD
    A["Request Incoming"] --> B["API Gateway"]
    B --> C["Authentication and Quota Check"]
    C --> D["Model Router"]
    D --> E["Get Instance Health Status"]
    E --> F["Get Resource and Queue Metrics"]
    F --> G["Check Model Version and Request Capabilities"]
    G --> H{"Is There an Available Instance?"}
    H -->|Yes| I["Calculate Routing Score"]
    H -->|No| J["Rate Limit, Queue, or Return Error"]
    I --> K["Select Target Instance"]
    K --> L["Forward Inference Request"]
    L --> M{"Is the Request Successful?"}
    M -->|Yes| N["Return Inference Result"]
    M -->|No| O["Determine Whether Retry Is Allowed"]
    O --> P{"Can It Be Safely Retried?"}
    P -->|Yes| D
    P -->|No| Q["Return Error or Degraded Result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#E5F0E9,stroke:#83A995,color:#315D49
    style O fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style P fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style Q fill:#F7EAE7,stroke:#C98D7C,color:#754438

5.3 Request Retries and Duplicate Execution

After a model instance fails, the Router may resend the request to another instance.

However, note: a request timeout does not mean the model did not execute.

For example:

  1. The request has already been sent to GPU node A.
  2. Node A completes inference.
  3. The network is interrupted while the result is being returned.
  4. API Gateway considers the request timed out.
  5. The request is resent to node B.
  6. The same task may be executed again.

This may lead to:

  • Repeated consumption of GPU resources.
  • Repeated execution of tool calls.
  • Repeated generation of business side effects.
  • Different instances returning different results.

Therefore, duplicate execution needs to be controlled through Request ID, Idempotency Key, and task state management.

For plain text inference, duplicate execution may mainly cause cost and latency issues; for Agent tool calls, database writes, and external API operations, it may create real business risks.

6. Timeout, Retry, and Circuit Breaker Mechanisms

A high-availability system cannot wait indefinitely for failure recovery, nor can it continue sending requests when a dependent service is failing.

6.1 Timeout: Timeout Control

Timeout means that once a request exceeds a preset time, it stops waiting for the current operation.

AI model services need to distinguish timeouts at different stages:

Timeout Type Meaning
Connect Timeout Connection establishment timeout
Queue Timeout Request queueing timeout
Model Load Timeout Model loading timeout
Prefill Timeout Input processing timeout
First Token Timeout First token timeout
Idle Timeout Streaming output has no data for an extended period
Total Request Timeout Total request duration timeout
Dependency Timeout Dependency service call timeout

For streaming inference, you cannot simply use a very short total request timeout.

Because generating longer text may take tens of seconds or even longer.

A more reasonable approach is:

  • Control connection establishment time.
  • Control queueing time.
  • Control first-token time.
  • Control the maximum idle time between two token outputs.
  • Set a reasonable upper limit on total execution time.

6.2 Retry: Retry Mechanism

Retries are suitable for failures that may recover temporarily.

Common retryable errors:

  • Temporary network connection failure.
  • GPU Worker is temporarily unavailable.
  • Service returns 503.
  • Downstream dependency experiences a brief timeout.
  • Requests are rejected due to excessive load.

Errors that should not be blindly retried:

  • Invalid parameter format.
  • Model version does not exist.
  • Authentication failed.
  • Insufficient user permissions.
  • Token count exceeds a hard limit.
  • Unsupported input format.

A common retry strategy is exponential backoff:

Delay_n=\min(Delay_{max},Delay_0\times2^n)

For example:

Retry count Base wait time
1st retry 1 second
2nd retry 2 seconds
3rd retry 4 seconds
4th retry 8 seconds

In practice, systems should also add random jitter to avoid a large number of requests being retried at the same time.

6.3 Circuit Breaker: Circuit Breaker Mechanism

A circuit breaker prevents continued access to a service that has already failed.

It generally has three states:

  • CLOSED: normal state, allowing requests to pass through.
  • OPEN: circuit-breaker state, directly rejecting requests or failing fast.
  • HALF-OPEN: half-open state, allowing a small number of probe requests.

Workflow:

flowchart TD
    A["Request received"] --> B["Circuit breaker"]
    B --> C{"Current state"}
    C -->|CLOSED| D["Call model service"]
    C -->|OPEN| E["Fail fast or degrade"]
    C -->|HALF-OPEN| F["Allow a small number of probe requests"]
    D --> G{"Call result"}
    G -->|Success| H["Success count and state recovery"]
    G -->|Failure| I["Increment failure count"]
    I --> J{"Has circuit-breaker threshold been exceeded?"}
    J -->|Yes| K["Enter OPEN"]
    J -->|No| L["Continue observing"]
    F --> M{"Is probe successful?"}
    M -->|Yes| N["Restore CLOSED"]
    M -->|No| K
    K --> O["Wait for circuit-breaker window"]
    O --> F

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49
    style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#E5F0E9,stroke:#83A995,color:#315D49
    style O fill:#EAF3EC,stroke:#83A995,color:#315D49

Circuit breakers need to be isolated by dependency or model instance.

For example:

  • A failure of Model A should not affect Model B.
  • A failure of GPU node A should not trip the circuit breaker for the entire inference cluster.
  • A Redis failure should not directly make all model services unavailable.

6.4 Differences Between Retry and Circuit Breaker

Mechanism Problem Solved Core Role
Timeout Requests wait too long Limit waiting time
Retry Temporary failures Attempt recovery
Circuit Breaker Persistent downstream failures Block invalid requests
Fallback Primary service unavailable Provide alternative capability
Bulkhead Resource interference Isolate fault scope

These mechanisms need to work together rather than replace one another.

7. Fault Isolation and Bulkhead Pattern

The Bulkhead Pattern originates from the compartment design concept of ships: when one area fails, it should not bring down the entire system.

In AI model services, the Bulkhead Pattern is especially important because different models and tasks vary greatly in resource consumption.

7.1 Isolation by Model

Different models use dedicated inference resource pools.

For example:

Model A
  -> GPU Pool A
  -> Worker A

Model B
  -> GPU Pool B
  -> Worker B

Model C
  -> GPU Pool C
  -> Worker C

This prevents an OOM or inference anomaly in one model from affecting other models.

7.2 Isolation by Task Type

Different types of AI tasks use different resource pools:

Task Type Recommended Isolation Method
Online LLM Inference Dedicated GPU Pool
Embedding Dedicated Inference Queue or Resource Pool
OCR Dedicated CPU/GPU Worker
Document Parsing CPU and Memory Resource Pool
Batch RAG Indexing Dedicated Asynchronous Task Pool
Model Training Dedicated Training Cluster

In particular, avoid large-scale offline tasks preempting online inference resources.

7.3 Isolation by Tenant

A multi-tenant AI platform needs to control resource usage by different tenants.

You can set:

  • Maximum concurrency per tenant.
  • Requests per minute.
  • Daily token quota.
  • Maximum context length.
  • Maximum number of output tokens.
  • GPU resource budget.
  • Priority and service level.

This prevents abnormal traffic from a single tenant from affecting other tenants.

7.4 Complete Structure of Resource Isolation

flowchart TD
    A["AI Model Service"] --> B["Request Classification"]
    B --> C["Tenant A"]
    B --> D["Tenant B"]
    B --> E["Background Tasks"]
    C --> F["Online Inference Resource Pool"]
    D --> G["Dedicated Inference Resource Pool"]
    E --> H["Offline Task Resource Pool"]
    F --> I["GPU Worker A"]
    G --> J["GPU Worker B"]
    H --> K["CPU / GPU Worker C"]
    I --> L["Dedicated Queues and Quotas"]
    J --> M["Dedicated Queues and Quotas"]
    K --> N["Dedicated Queues and Quotas"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#E5F0E9,stroke:#83A995,color:#315D49
    style M fill:#E5F0E9,stroke:#83A995,color:#315D49
    style N fill:#E5F0E9,stroke:#83A995,color:#315D49

8. Failover and Automatic Recovery

Failover refers to transferring requests to other available instances after the primary service or node fails.

8.1 Basic Process of Failover

A typical process includes:

  1. The monitoring system detects an anomaly.
  2. Health checks confirm that the instance is unavailable.
  3. Mark the anomalous instance as Unhealthy.
  4. Remove it from the load balancing pool.
  5. Route new requests to healthy instances.
  6. Retry requests that can be safely retried.
  7. Start a replacement instance.
  8. After the instance finishes warming up, add it back to the traffic pool.
flowchart TD
    A["Model instance running normally"] --> B["Health checks and metric monitoring"]
    B --> C{"Is the instance healthy?"}
    C -->|Yes| A
    C -->|No| D["Mark instance as Unhealthy"]
    D --> E["Remove from traffic pool"]
    E --> F["Route to other healthy instances"]
    F --> G["Check whether the request can be retried"]
    G --> H{"Is retry allowed?"}
    H -->|Yes| I["Resend the request"]
    H -->|No| J["Return an error or degrade"]
    D --> K["Start a replacement instance"]
    K --> L["Load model weights"]
    L --> M["Perform warm-up and health checks"]
    M --> N{"Is it ready?"}
    N -->|Yes| O["Rejoin the traffic pool"]
    N -->|No| P["Continue recovery or isolate"]
    I --> Q["Monitor recovery status"]
    J --> Q
    O --> Q
    P --> Q

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49
    style M fill:#EAF3EC,stroke:#83A995,color:#315D49
    style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style O fill:#E5F0E9,stroke:#83A995,color:#315D49
    style P fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style Q fill:#E5F0E9,stroke:#83A995,color:#315D49

8.2 Failure Detection Should Not Be Too Sensitive

If health checks are too sensitive, false positives may occur:

  • The GPU is briefly busy.
  • The network has a momentary jitter.
  • Model inference queue times become longer.
  • Occasional request timeouts.

These situations do not necessarily mean that the instance has actually failed.

It is recommended to use:

  • Number of consecutive failures.
  • Multiple time windows.
  • Combinations of metric thresholds.
  • Active probe requests.
  • Passive request failure rate.

For example, remove an instance only after multiple consecutive health check failures.

You can also set a recovery observation period to prevent an instance from immediately receiving a large number of requests right after it recovers.

8.3 Automatic Recovery and Manual Intervention

Not all failures should trigger automatic restarts.

For example:

  • A GPU driver anomaly may require a node restart.
  • Corrupted model weights need to be downloaded again.
  • A VRAM leak may require replacing the process.
  • Incorrect configuration may cause new instances to keep crashing.
  • Data corruption may require manual repair.

It is recommended to establish a tiered recovery strategy:

Failure Level Handling Method
Temporary network failure Automatic retry
Single-request inference failure Isolate the request and log the error
Model process anomaly Restart the process
GPU device anomaly Remove the node and reschedule
Corrupted model file Re-pull and verify the model
Configuration release error Roll back the configuration
Regional failure Start cross-region disaster recovery

9. High Availability Design for Model Deployment and Version Upgrades

Model upgrades are a very important source of risk in AI services.

A new model may exhibit the following:

  • Incompatible model weights.
  • Tokenizer mismatch.
  • The inference engine does not support new operators.
  • Degraded output quality.
  • Increased inference latency.
  • Increased VRAM usage.
  • Specific inputs cause the process to crash.

Therefore, model deployment needs to support progressive rollout and fast rollback capabilities.

9.1 Rolling Update

A rolling update is a deployment method that gradually replaces old instances.

For example, suppose there are originally 4 instances:

  1. Start a new-version instance first.
  2. Wait for the new instance to finish model loading and warm-up.
  3. Remove one old instance from the traffic pool.
  4. Wait for the old instance to finish processing in-flight requests.
  5. Update the next instance.
  6. End the release after all updates are complete.

This maintains a certain level of service capacity during the upgrade.

Note that during a rolling update, old and new versions may coexist, so the API protocol, model inputs and outputs, and dependent services must remain compatible.

9.2 Blue-Green Deployment

Blue-green deployment prepares two complete environments:

  • Blue: the current production environment.
  • Green: the new environment to be released.

After the new version is deployed, warmed up, and validated in the Green environment, traffic is switched over.

flowchart TD
    A["Current traffic"] --> B["Traffic ingress"]
    B --> C["Blue environment: old version"]
    D["Deploy new version"] --> E["Green environment"]
    E --> F["Load model and warm up"]
    F --> G["Run validation tests"]
    G --> H{"Validation passed?"}
    H -->|No| I["Stop release and keep Blue"]
    H -->|Yes| J["Switch production traffic to Green"]
    J --> K["Monitor new version metrics"]
    K --> L{"Any anomalies?"}
    L -->|No| M["Release complete"]
    L -->|Yes| N["Roll back to Blue"]
    N --> O["Restore old version traffic"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#E5F0E9,stroke:#83A995,color:#315D49
    style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style O fill:#EAF3EC,stroke:#83A995,color:#315D49

The advantage of blue-green deployment is a clear rollback path, but the disadvantage is that an additional environment must be prepared.

For large models, the additional GPU resources and model weight storage costs can be high.

9.3 Canary Release

Canary release first routes a small amount of traffic to the new version, then gradually increases the traffic proportion.

For example:

Stage 1: 1% of traffic goes to the new version
Stage 2: 5% of traffic goes to the new version
Stage 3: 20% of traffic goes to the new version
Stage 4: 50% of traffic goes to the new version
Stage 5: 100% of traffic goes to the new version

At each stage, monitor:

  • Request success rate.
  • P95/P99 latency.
  • TTFT and TPOT.
  • GPU memory usage.
  • Number of OOMs.
  • Model output quality.
  • Token cost.
  • User feedback.

For generative AI, you cannot monitor only system performance; you must also monitor model quality.

For example, although a new model may run normally, it may still exhibit:

  • Lower answer accuracy.
  • Higher hallucination rate.
  • Unstable JSON output format.
  • Lower tool-call success rate.
  • Degraded performance in specific languages.

Therefore, model releases need to combine automated evaluation with online metrics.

9.4 Comparison of Release Strategies

Strategy Advantages Disadvantages Use Cases
Rolling Update Relatively low resource overhead Rollback is relatively complex Routine model upgrades
Blue-Green Clear switching and rollback High resource overhead High-reliability model releases
Canary Risk is released gradually Requires traffic allocation and metric analysis Large-scale online model upgrades
Shadow Deployment Allows comparison of old and new models Increases compute cost Model quality validation

Shadow Deployment copies production requests to the new model but does not return the new model's results to users.

This allows observing the new model's latency, resource usage, and output differences under real traffic.