Back to series
· 18 min read

Agent Reliability, Fault Tolerance, and Exception Recovery


I. Why Does an Agent Need Reliability Mechanisms?

Traditional software typically executes according to deterministic program logic, with relatively controllable inputs, processing, and outputs.

Agents are different.

They need to autonomously plan tasks based on user goals, select tools, process results, and continuously adjust their strategy based on execution feedback. A single task may last minutes, hours, or even days.

For example, a user requests:

Analyze the company's operating data from the past three years, compile industry materials, generate a complete analysis report, and send it to management.

The Agent may need to:

  • Query multiple databases.
  • Read large numbers of business files.
  • Search external industry materials.
  • Invoke multiple analysis tools.
  • Generate charts and reports.
  • Save intermediate results.
  • Send the final report.

During execution, any step may fail:

  • A model API times out.
  • A tool service is unavailable.
  • A database connection drops.
  • A file read fails.
  • The model generates invalid parameters.
  • An external API returns an error.
  • Task execution exceeds the time limit.
  • Multi-agent collaboration produces conflicting results.
  • A service restart interrupts the task.
  • User approval is not completed for a long time.

Without reliability mechanisms, a single local failure could cause the entire task to fail.

More seriously, the Agent may have already performed some operations but be unable to accurately determine which succeeded and which failed, leading to problems such as duplicate writes, duplicate emails, or duplicate charges.

Therefore, an Agent system must have four core capabilities:

  1. Reliable execution: complete tasks stably under normal conditions.
  2. Fault isolation: local failures do not propagate indefinitely.
  3. Fault tolerance: adopt reasonable alternative strategies when anomalies occur.
  4. State recovery: continue execution from the correct position after a system interruption.

II. Understanding the Agent Reliability Model

1. Reliability, Availability, Fault Tolerance, and Recoverability

These concepts are often used interchangeably, but they address different questions.

Concept English Core Meaning
Reliability Reliability The ability of a system to continuously complete tasks correctly under specified conditions
Availability Availability The ability of a system to provide service when needed
Fault Tolerance Fault Tolerance The system can still maintain service when some components fail
Resilience Resilience The ability of a system to recover and adapt after being affected by a fault
Recoverability Recoverability The ability to restore state and continue execution after a fault
Observability Observability Understanding the internal state of a system through logs, metrics, and tracing

For example:

An Agent service may always be accessible but frequently generate incorrect results.

This means it has a certain level of availability, but not necessarily good reliability.

Another Agent may encounter a timeout when calling an external API, automatically switch to a backup service, and continue execution. This demonstrates fault tolerance.

If the service can recover a task from a persisted checkpoint after a restart, this demonstrates recoverability.

2. What Determines an Agent's Reliability?

An Agent's reliability is not simply model accuracy.

It can be understood from multiple dimensions:

R_{agent} = f(R_{model}, R_{tool}, R_{state}, R_{workflow}, R_{runtime})

Where:

  • R_{model}: reliability of model reasoning and output.
  • R_{tool}: reliability of tool invocation.
  • R_{state}: reliability of state management.
  • R_{workflow}: reliability of task orchestration.
  • R_{runtime}: reliability of the runtime environment.

These factors affect one another.

For example, the model generates the correct tool-call parameters, but the tool service times out, so the task still cannot be completed.

Or the tool executes successfully, but the system fails to save state in time, and after a process crash, the operation may be executed again.

Therefore, reliability must cover the entire execution chain.

3. The Agent's End-to-End Execution Chain

flowchart TD
    A["User submits task"] --> B["Task understanding and planning"]
    B --> C["Generate execution steps"]
    C --> D["Model reasoning"]
    D --> E["Select tool"]
    E --> F["Tool parameter validation"]
    F --> G["Tool execution"]
    G --> H["Result validation"]
    H --> I["Update task status"]
    I --> J{"Is the task complete?"}
    J -->|"No"| D
    J -->|"Yes"| K["Generate final result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Failures can occur at every stage, so task success cannot be checked only at the final result stage.

3. Common Types of Agent Failures

Before designing a fault-tolerant system, you first need to identify the sources of failure.

1. Model-Layer Failures

Model-layer failures mainly come from the model service and the reasoning process.

Failure Typical Symptom
API Timeout Model request timeout
Rate Limit Request rate exceeds the limit
Service Unavailable Model service unavailable
Context Overflow Input exceeds the context window
Invalid Output Output format does not meet requirements
Hallucination Generates information that does not exist
Tool Selection Error Selects the wrong tool
Reasoning Failure Logical error in the reasoning process

For example, the Agent asks the model to generate JSON:

{
  "tool": "search",
  "query": "AI Agent"
}

But the model actually returns:

I am going to call the search tool to look up information about AI Agent.

If the system directly parses it as JSON, a format error will occur.

Model output must undergo structural validation; you cannot assume the model will always follow the constraints.

2. Tool-Layer Failures

Agents rely on external tools to perform actual operations, so the tool layer is a failure-prone area.

Common problems include:

  • HTTP request timeout.
  • API returns 500.
  • API returns 429.
  • Database connection drops.
  • File does not exist.
  • Insufficient permissions.
  • Third-party service maintenance.
  • Tool return format changes.
  • Tool execution succeeds but the response is lost.

One particularly dangerous situation is:

The tool has already executed successfully, but the Agent did not receive a success response.

For example:

The Agent calls the email service to send an email.

The email service has already sent the email successfully, but the network is interrupted before the response is returned.

The Agent only sees a request timeout and cannot determine whether the email was actually sent.

If it retries directly, it may send two identical emails to the customer.

This type of problem needs to be solved through idempotency and execution status queries.

3. State-Layer Failures

The state layer is responsible for saving the Agent's task progress, context, and execution results.

Common failures include:

  • State is not saved in time.
  • State write fails.
  • Multiple Workers modify the state at the same time.
  • Checkpoint data is corrupted.
  • Expired state is loaded during recovery.
  • Conflict when merging results from parallel tasks.
  • State is inconsistent with the actual results in external systems.

For example:

The Agent has completed data analysis, but before the analysis results are saved, the service crashes.

After recovery, the Agent may re-execute the entire analysis process.

If the analysis process contains operations with side effects, this may lead to duplicate execution.

4. Workflow-Layer Failures

Complex Agents are usually composed of multiple steps.

For example:

Data collection
  → Data cleaning
  → Data analysis
  → Report generation
  → Review
  → Publishing

Possible issues:

  • A certain step fails.
  • A preceding step is not finished, but a later step runs ahead of time.
  • Task dependency relationships are wrong.
  • Parallel steps do not all complete.
  • A sub-agent does not respond for a long time.
  • The task enters an infinite loop.
  • The task status stays in “processing” for a long time.

Failures of this kind must be addressed through workflow state machines, timeout control, and task scheduling mechanisms.

5. Infrastructure Failures

The infrastructure layer includes:

  • Insufficient CPU resources.
  • Memory exhaustion.
  • Insufficient disk space.
  • Container exits abnormally.
  • Network outage.
  • Message queue unavailable.
  • Database failure.
  • Cloud service region failure.
  • Worker process crash.

These problems usually cannot be solved by modifying the Prompt; they require fault-tolerant design at the level of system architecture and infrastructure.

IV. Error Classification: Which Errors Are Recoverable?

Not all exceptions should be retried.

If error types are not distinguished, an Agent may keep repeating execution in situations that cannot be recovered, wasting resources and even widening the impact of the failure.

1. Transient Error

Transient errors can usually be recovered from by waiting or retrying.

For example:

  • A brief network interruption.
  • HTTP 429.
  • Service temporarily unavailable.
  • Database connection timeout.
  • Brief resource contention.

Suitable approaches:

  • Delayed retry.
  • Exponential backoff.
  • Random jitter.
  • Service switching.

2. Permanent Error

Permanent errors usually cannot be solved by simple retries.

For example:

  • Invalid API Key.
  • Insufficient user permissions.
  • Incorrect parameter format.
  • Resource does not exist.
  • The request violates business rules.
  • The tool name does not exist.

In this case, you should:

  • Stop the current operation immediately.
  • Return a clear error.
  • Fix the parameters or permissions.
  • Request additional information from the user when necessary.

3. Business Error

A business error means the operation itself does not comply with business rules.

For example:

  • Insufficient account balance.
  • Approval not granted.
  • The order status does not allow cancellation.
  • The user has exceeded the operation quota.
  • The current task is not allowed to be published.

Such exceptions should not be treated as technical failures and retried repeatedly.

4. Ambiguous Failure

Ambiguous failures are a category that requires particular attention in Agent systems.

For example:

  • A remote operation may have succeeded, but the response was lost.
  • The result of a database commit is unknown.
  • Whether an email was sent cannot be confirmed.
  • A payment request times out, but the third-party payment platform has not yet returned a final status.

In such cases, the system cannot simply assume the operation failed.

The correct handling approach is usually:

  1. Query the status of the remote operation.
  2. Use an idempotency key to check whether it has already been executed.
  3. Decide whether to continue based on the actual result.
  4. Escalate to manual handling when it cannot be confirmed.

5. Error Classification Table

Error type Example Recommended strategy
Transient error Network timeout Limited retries
Permanent error Invalid parameters Fail immediately
Business error Approval rejected Handle via business branch
Ambiguous failure Request response lost Query execution status
Resource error Out of memory Limit resources or reschedule
State error Corrupted Checkpoint Restore backup or manual intervention

6. Error Classification and Handling Flow

flowchart TD
    A["Catch Exception"] --> B["Extract Error Information"]
    B --> C["Exception Classification"]
    C --> D{"Error Type"}
    D -->|"Transient"| E["Limited Retry"]
    D -->|"Permanent"| F["Stop Execution and Return Error"]
    D -->|"Business Exception"| G["Enter Business Handling Branch"]
    D -->|"Uncertainty"| H["Query Remote Execution Status"]
    D -->|"Resource Exception"| I["Rate Limit or Reschedule"]
    E --> J{"Recovery Successful?"}
    J -->|"Yes"| K["Continue Task"]
    J -->|"No"| L["Degrade or Terminate"]
    H --> M{"Operation Result Confirmed?"}
    M -->|"Succeeded"| K
    M -->|"Failed"| E
    M -->|"Unable to Confirm"| N["Manual Intervention"]
    G --> O["Update Business Status"]
    I --> P["Re-execute After Resource Recovery"]
    F --> Q["Record Failure Status"]
    L --> Q
    N --> Q
    O --> Q
    P --> K

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style O fill:#EAF3EC,stroke:#83A995,color:#315D49
    style P fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style Q fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

V. Retry Mechanism: Automatic Recovery from Transient Failures

Retry is the most common fault-tolerance mechanism.

When an Agent fails to call a tool, it can reissue the request based on the exception type.

But retry is not simply repeated invocation.

A reasonable retry system needs to control:

  • Which errors can be retried.
  • Maximum retry count.
  • Retry interval.
  • Total execution time.
  • Whether there are side effects.
  • Whether to switch services.

1. Fixed-Interval Retry

Fixed-interval retry is the simplest approach.

For example:

First failure
  → Wait 2 seconds
  → Second retry
  → Wait 2 seconds
  → Third retry

The advantage is simple implementation.

The disadvantage is that when a large number of requests fail at the same time, all requests may be reissued at the same time, creating a traffic spike.

2. Exponential Backoff

Exponential backoff gradually increases the wait time based on the number of retries.

Formula:

T_n = \min(T_{max}, T_0 \times 2^n)

Where:

  • T_n: The wait time for the nth retry.
  • T_0: Initial wait time.
  • T_{max}: Maximum wait time.
  • n: Number of retries.

For example, with an initial wait of 1 second:

Retry count Wait time
First attempt 1 second
Second attempt 2 seconds
Third attempt 4 seconds
Fourth attempt 8 seconds
Fifth attempt 16 seconds

This approach reduces the sustained pressure on the service during an outage.

3. Random Jitter

If every request uses the same exponential backoff delay, they may still re-issue their requests at the same moment.

So random jitter is usually added:

T_n = Random(0, \min(T_{max}, T_0 \times 2^n))

This spreads the requests out and prevents a large number of them from recovering at the same time.

4. The Number of Retries Must Be Bounded

Unlimited retries lead to:

  • Continuous consumption of resources.
  • Tasks that can never finish.
  • A steadily growing number of requests.
  • Greater load on downstream services.
  • A backlog in the task queue.

Therefore you need to limit all of the following at once:

  • The maximum number of retries.
  • The timeout of a single request.
  • The overall task deadline.
  • The total retry budget.

For example:

import random
import time


def retry(
    operation,
    max_attempts=4,
    base_delay=1,
    max_delay=16
):
    for attempt in range(max_attempts):
        try:
            return operation()

        except TimeoutError:
            if attempt == max_attempts - 1:
                raise

            delay = min(
                max_delay,
                base_delay * (2 ** attempt)
            )

            delay = random.uniform(0, delay)

            time.sleep(delay)

    raise RuntimeError("重试失败")

This example only catches timeout exceptions.

In real systems, whether to retry should be decided by the exception type, and retry control should be combined with the overall task deadline so that the waiting time does not exceed the task budget.

5. The Retry-After Response Header

When an external API returns HTTP 429 or 503, it may provide a Retry-After response header.

For example:

HTTP/1.1 429 Too Many Requests
Retry-After: 10

This means the client is advised to wait 10 seconds before trying again.

An agent tool client should give priority to the retry advice explicitly provided by the server, while still being constrained by the overall deadline and the maximum number of retries.

6. Retry Flow

flowchart TD
    A["Start tool call"] --> B["Wait for result"]
    B --> C{"Succeeded?"}
    C -->|"Yes"| D["Return result"]
    C -->|"No"| E["Identify exception type"]
    E --> F{"Retry allowed?"}
    F -->|"No"| G["Log failure and stop"]
    F -->|"Yes"| H{"Retry budget exceeded?"}
    H -->|"Yes"| G
    H -->|"No"| I["Calculate backoff time"]
    I --> J["Add random jitter"]
    J --> K["Wait and re-execute"]
    K --> B

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49

VI. Idempotency: Preventing Side Effects Caused by Repeated Execution

The retry mechanism has an important prerequisite:

Repeatedly executing the same operation must not produce uncontrolled duplicate side effects.

This is idempotency.

1. What Is Idempotency?

If an operation produces the same final result whether it is executed once or multiple times, then the operation is idempotent.

Mathematically:

f(f(x)) = f(x)

For example:

Setting the user status to active, whether executed once or multiple times, always results in the final status being active.

However, the following operations are usually not naturally idempotent:

  • Creating an order.
  • Sending an email.
  • Charging a payment.
  • Creating a file.
  • Increasing the account balance.
  • Publishing content on social media.

For example:

account.balance += 100

Executing it twice increases the balance by 200, not 100.

2. Using an Idempotency Key

You can generate a unique idempotency key for each logical operation.

For example:

{
  "task_id": "task_001",
  "operation_id": "op_001",
  "idempotency_key": "task_001_send_report_001",
  "action": "send_email"
}

After the server receives the request:

  1. Check whether the idempotency key has already been processed.
  2. If it has been processed, return the previously saved result.
  3. If it has not been processed, execute the operation.
  4. Save the operation result and the idempotency key.
  5. Subsequent identical requests directly return the existing result.

Simplified example:

class IdempotencyStore:
    def __init__(self):
        self.results = {}

    def execute_once(self, key, operation):
        if key in self.results:
            return self.results[key]

        result = operation()

        self.results[key] = result

        return result

This code is only intended to illustrate the concept of an idempotency key.

In production, you cannot rely solely on an in-memory dictionary, because records will be lost after a process restart, and concurrent requests may pass the check at the same time.

You should use persistent storage that provides unique constraints and atomic operations.

3. Idempotency Keys Need to Be Bound to Operation Parameters

The same idempotency key should not be used for different business operations.

For example:

First request:

{
  "idempotency_key": "order_001",
  "amount": 100
}

Second request:

{
  "idempotency_key": "order_001",
  "amount": 1000
}

If the system only checks the idempotency key and does not check the parameters, it may return an incorrect result.

Therefore, you should store:

  • Idempotency key.
  • Operation type.
  • Digest of key business parameters.
  • Execution status.
  • Execution result.
  • Creation time.

If the same idempotency key corresponds to different parameters, the request should be rejected.

4. Idempotency State Machine

stateDiagram-v2
    [*] --> PENDING
    PENDING --> PROCESSING: Start execution
    PROCESSING --> SUCCEEDED: Execution succeeded
    PROCESSING --> FAILED: Explicit execution failure
    PROCESSING --> UNKNOWN: Execution result uncertain
    FAILED --> PROCESSING: Retry conditions met
    UNKNOWN --> SUCCEEDED: Query confirms success
    UNKNOWN --> FAILED: Query confirms failure
    UNKNOWN --> PROCESSING: Confirmed safe to retry
    SUCCEEDED --> [*]

Pay special attention to the following:

UNKNOWN cannot be directly equated with FAILED.

If the remote service has already completed the operation and only the response was lost, retrying directly may produce duplicate side effects.

VII. Circuit Breaker: Preventing Continuous Fault Propagation

When an Agent frequently calls a service that is already unavailable, continuous retries may further increase pressure on the downstream service.

A circuit breaker is used to temporarily stop calls when a service continues to fail.

1. The Three States of a Circuit Breaker

State Meaning
Closed Normal state; requests are allowed through
Open Open state; requests are rejected directly
Half-Open Half-open state; a small number of probe requests are allowed

2. Closed: Normal State

Under normal conditions, requests pass through the circuit breaker and call the service.

If the error rate continues to rise and reaches the configured threshold, it enters the Open state.

3. Open: Disconnected State

After the circuit breaker opens, it no longer sends regular requests to the failing service.

The system can:

  • Return an error immediately.
  • Use cached results.
  • Switch to a backup service.
  • Pause related tasks.
  • Put tasks into a waiting queue.

This prevents continuous requests from further overwhelming the downstream service.

4. Half-Open: Half-Open State

After a cooldown period, the system allows a small number of probe requests.

If the probe succeeds, it returns to the Closed state.

If the probe fails, it enters the Open state again.

5. Circuit Breaker State Transitions

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: Failure rate exceeds threshold
    Open --> HalfOpen: Cooldown period ends
    HalfOpen --> Closed: Probe request succeeds
    HalfOpen --> Open: Probe request fails

6. Circuit Breaker vs. Retry

Mechanism Problem Solved
Retry A single request temporarily fails
Circuit Breaker Downstream service continues to fail
Timeout Request has no response for a long time
Rate Limiter Request rate is too high
Bulkhead Failure spreads to other resource pools

They usually need to be used together.

For example:

When an Agent calls an external search API:

  • A single timeout is controlled by Timeout.
  • Temporary errors are handled by Retry.
  • Continuous service failures are protected by Circuit Breaker.
  • Excessive requests are limited by Rate Limiter.
  • Resources are isolated between different tools via Bulkhead.

8. Timeout Control: Preventing Tasks from Waiting Indefinitely

Agents often need to wait for external tools to return results.

If a tool does not have reasonable timeout settings, a request may occupy connections, threads, and task resources for a long time.

1. Common Timeout Types

Timeout Type Meaning
Connection Timeout Maximum wait time to establish a connection
Read Timeout Maximum time to wait for response data
Tool Timeout Maximum time for a single tool call
Model Timeout Maximum time for a model request
Step Timeout Maximum time for a single workflow step
Task Deadline Deadline for the entire task

2. Why Is an Overall Task Deadline Needed?

Suppose an Agent has 10 steps, and each step can run for at most 60 seconds.

If each step independently waits up to 60 seconds, the entire task could take 10 minutes.

If the user requires completion within 3 minutes, a per-step timeout cannot guarantee that the overall task meets the requirement.

Therefore, a unified Deadline is needed.

Each time a step is executed, the system must calculate the remaining time:

T_{remaining} = T_{deadline} - T_{now}

The timeout for the current step must not exceed the remaining task time.

3. Timeout Control Flow

flowchart TD
    A["Create Task"] --> B["Set Task Deadline"]
    B --> C["Execute Current Step"]
    C --> D["Calculate Remaining Time"]
    D --> E{"Deadline Exceeded?"}
    E -->|"Yes"| F["Cancel or Terminate Task"]
    E -->|"No"| G["Set Step Timeout"]
    G --> H["Call Model or Tool"]
    H --> I{"Completed Within Time?"}
    I -->|"Yes"| J["Save Step Result"]
    I -->|"No"| K["Trigger Timeout Exception"]
    K --> L["Check Whether Retry Is Allowed"]
    L --> M{"Time Budget Remaining?"}
    M -->|"Yes"| C
    M -->|"No"| F
    J --> N{"Task Complete?"}
    N -->|"No"| C
    N -->|"Yes"| O["Return Final Result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F

A timeout does not mean the remote operation has already stopped.

For example, the client stopping its wait on the email API does not mean the email service has stopped sending emails.

Therefore, for operations with side effects, after a timeout you still need to confirm the execution result through an idempotency key or a remote status query.

9. Checkpoint: Letting the Agent Resume from Where It Was Interrupted

For long-running Agents, keeping task state in memory alone is not enough.

Once the process crashes, the in-memory state is lost.

A Checkpoint is used to save the key state of an Agent during execution, so that a task can continue from a previous state.

1. What Does a Checkpoint Save?

Typical contents include:

  • Task ID.
  • The current execution node.
  • The current task status.
  • Completed steps.
  • Intermediate results.
  • Tool call records.
  • Retry count.
  • Pending operations.
  • User approval status.
  • Context summary.
  • Version and timestamp.

For example:

{
  "task_id": "task_001",
  "status": "running",
  "current_step": "analyze_data",
  "completed_steps": [
    "collect_data",
    "clean_data"
  ],
  "retry_count": 1,
  "state_version": 12,
  "checkpoint_at": "2026-11-21T10:00:00Z"
}

2. Why Can't We Just Save the Model Conversation?

An Agent's state is more than chat history.

For example:

  • The conversation record describes what the user said.
  • The task state describes how far execution has progressed.
  • The tool records describe which operations have been performed.
  • The Checkpoint describes the recoverable state at a given point in time.

If only the conversation history is saved, the system still cannot accurately determine:

  • Which tools have already been executed.
  • Which steps have already been completed.
  • Which operations have not yet been confirmed.
  • Which results have already been persisted.
  • Which tasks can be safely retried.

Therefore, a Checkpoint should save structured task state, not just chat messages.

3. The Basic Working Mechanism of a Checkpoint

flowchart TD
    A["Start Agent task"] --> B["Read current state"]
    B --> C["Execute workflow node"]
    C --> D["Produce node result"]
    D --> E["Update State"]
    E --> F["Persist Checkpoint"]
    F --> G{"Any remaining nodes?"}
    G -->|"Yes"| C
    G -->|"No"| H["Task complete"]
    F --> I["Process interruption occurs"]
    I --> J["Restore Worker"]
    J --> K["Load latest Checkpoint"]
    K --> L["Restore task state"]
    L --> C

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49

4. When to Save a Checkpoint

Different saving strategies can be adopted.

Save after each node completes

Save the state immediately after each workflow node executes successfully.

The advantage is finer recovery granularity.

The disadvantage is a higher write frequency.

Save at key steps

Save only at important nodes.

For example:

  • Data collection completed.
  • Data analysis completed.
  • Report generation completed.
  • External operations completed.

This approach reduces persistence overhead, but when a failure occurs, some work may need to be re-executed.

Scheduled saving

Save the state at fixed time intervals.

This suits long-running tasks, but you need to consider state consistency between the scheduled saves and the actual operations.

5. Consistency Between Checkpoints and External Side Effects

This is a difficult point in production-grade Agent design.

Suppose:

  1. The Agent calls a payment tool.
  2. The payment service has already charged the amount.
  3. The Agent has not yet saved a Checkpoint.
  4. The process crashes.

After recovery, the Agent may not know whether the payment was completed.

Therefore:

A Checkpoint can only restore local workflow state; it cannot automatically restore the real state of an external system.

Checkpoints need to be combined with idempotent operation records, remote state queries, and transaction mechanisms.

10. State Machine: Clarifying What Stage a Task Is In

Agent state management cannot rely on just a simple running or failed field.

Complex tasks require an explicit state machine.

1. Common Task States

State Meaning
CREATED The task has been created
PENDING Waiting for scheduling
RUNNING Executing
WAITING Waiting for an external event
RETRYING Retrying
PAUSED Paused intentionally
RECOVERING Recovering
SUCCEEDED Completed successfully
FAILED Execution failed
CANCELLED Cancelled by the user
TIMED_OUT Terminated due to timeout

2. Why Do We Need a State Machine?

Suppose a task is currently in SUCCEEDED.

The system should not switch it back to RUNNING unless a new execution version is explicitly created or the task is reopened.

Likewise, a task in CANCELLED must not automatically resume execution just because some delayed tool response arrives.

A state machine can restrict illegal state transitions.

3. State Transition Rules

```mermaid stateDiagram-v2 [*] --> CREATED CREATED --> PENDING: Submit for scheduling PENDING --> RUNNING: Worker starts execution RUNNING --> WAITING: Waiting for external event WAITING --> RUNNING: External event received RUNNING --> RETRYING: Transient exception RETRYING --> RUNNING: Retry succeeded RETRYING --> FAILED: Retries exhausted RUNNING --> PAUSED: Manually paused PAUSED --> RUNNING: Resume execution RUNNING --> RECOVERING: Runtime environment interrupted RECOVERING --> RUNNING: Recovery succeeded RECOVERING --> FAILED: Unable to recover RUNNING --> SUCCEEDED: Task completed RUNNING --> FAILED: Unrecoverable exception RUNNING --> TIMED_OUT: Deadline exceeded PENDING --> CANCELLED: User cancelled RUNNING --> CANCELLED: User cancelled SUCCEEDED --> [*] FAILED --> [*] CANCELLED --> [*] TIMED_OUT --> [*] ```

Transition validation for the state machine should be performed on the server side.

The model must not be allowed to write arbitrary task states directly.

4. State updates require concurrency control

If two Workers process the same task at the same time, state overwrites can occur.

For example:

  • Worker A reads version 10.
  • Worker B reads version 10.
  • Worker A updates to version 11.
  • Worker B overwrites version 11 using stale data.

You can use optimistic locking:

UPDATE agent_tasks
SET
    status = 'RUNNING',
    version = version + 1
WHERE
    task_id = 'task_001'
    AND version = 10;

If the number of updated rows is 0, it means the version has already changed, and you need to re-read the state and handle the conflict.

In production environments, you can also use task leases, Worker heartbeats, and fencing tokens to prevent an old Worker that has lost its lease from continuing to write state.

XI. Fault Recovery for Long-Running Tasks

The biggest difference between long-running Agents and ordinary request-response Agents is that a task may span multiple execution cycles.

For example:

  • The Worker restarts while the task is executing.
  • User approval is only completed several hours later.
  • An external service only recovers the next day.
  • The task is waiting for third-party data.
  • A system deployment upgrade causes the execution process to exit.

This means the Agent must have the ability to persist tasks and recover asynchronously.

1. Architecture for Long-Running Tasks

A reliable long-running Agent usually includes:

Component Purpose
Task Store Stores basic task information
State Store Stores execution state
Checkpoint Store Stores recovery checkpoints
Message Queue Schedules tasks
Worker Pool Executes tasks
Scheduler Manages task scheduling
Retry Manager Controls retries
Recovery Manager Detects and recovers abnormal tasks
Event Store Stores key execution events

2. Task Interruption and Recovery

flowchart TD
    A["Create long-running task"] --> B["Save task information"]
    B --> C["Enter task queue"]
    C --> D["Worker fetches task"]
    D --> E["Load task state"]
    E --> F["Execute current step"]
    F --> G["Save Checkpoint"]
    G --> H{"Completed normally?"}
    H -->|"Yes"| I["Mark task successful"]
    H -->|"No"| J{"Exception occurred?"}
    J -->|"No"| F
    J -->|"Yes"| K["Record failure event"]
    K --> L["Recovery Manager analyzes"]
    L --> M{"Can recover automatically?"}
    M -->|"Yes"| N["Reschedule task"]
    N --> D
    M -->|"No"| O["Pause task and alert"]
    O --> P["Manual handling"]
    P --> Q{"Allow continuation?"}
    Q -->|"Yes"| N
    Q -->|"No"| R["Mark task failed"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#EAF3EC,stroke:#83A995,color:#315D49
    style O fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style P fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style Q fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style R fill:#F7EAE7,stroke:#C98D7C,color:#754438

3. Worker Crash Detection

A Worker may exit for the following reasons:

  • The container is terminated.
  • The process crashes.
  • Out of memory.
  • Network interruption.
  • Node failure.

The system can use a heartbeat mechanism to determine whether a Worker is still alive.

For example:

{
  "worker_id": "worker_001",
  "task_id": "task_001",
  "lease_expires_at": "2026-11-21T10:05:00Z",
  "heartbeat_at": "2026-11-21T10:04:30Z"
}

If the lease expires, the scheduler can consider that the Worker no longer has the right to execute the task and reassign the task.

However, note:

The old Worker may have only experienced a temporary network interruption and may not have actually stopped.

Therefore, when reassigning tasks, you must use leases, fencing tokens, and idempotent execution mechanisms to prevent two Workers from producing side effects at the same time.

12. Saga: Handling Long-Running Cross-Service Transactions

An Agent may need to call multiple external services in sequence.

For example:

  1. Create an order.
  2. Deduct inventory.
  3. Deduct account balance.
  4. Create a shipping order.
  5. Send order notification.

These operations may be distributed across multiple independent services and cannot be committed uniformly using a single database transaction.

If the third step fails, how should the effects already produced by the first two steps be handled?

This is the problem that the Saga pattern is designed to solve.

1. What Is Saga?

Saga is a distributed transaction coordination pattern.

It splits a long transaction into multiple local transactions, each with a corresponding compensating operation.

For example:

Forward operation Compensating operation
Create order Cancel order
Deduct inventory Restore inventory
Freeze balance Unfreeze balance
Create shipping order Cancel shipping order

Note that a compensating operation is not necessarily the same as a database rollback.

For example, an email that has already been sent to a customer cannot truly be recalled; it can only be remedied through a follow-up notification.