1. Why Do Agents Need Reliability Mechanisms?
Traditional software typically executes according to deterministic program logic, and its input, processing, and output are relatively controllable.
Agents are different.
They must plan tasks autonomously based on user goals, select tools, process results, and continuously adjust their strategies according to execution feedback. A single task may last minutes, hours, or even days.
For example, a user might ask:
Analyze the company's operating data from the past three years, compile industry materials, generate a complete analysis report, and send it to management.
The Agent may need to:
- Query multiple databases.
- Read a large number of business files.
- Search for external industry materials.
- Call multiple analysis tools.
- Generate charts and reports.
- Save intermediate results.
- Send the final report.
During execution, any step may fail:
- Model API timeout.
- Tool service unavailable.
- Database connection interrupted.
- File read failure.
- The model generates invalid parameters.
- An external API returns an error.
- Task execution exceeds the time limit.
- Conflicting results arise in multi-Agent collaboration.
- A service restart interrupts the task.
- User approval is not completed for a long time.
Without reliability mechanisms, a single local failure can cause the entire task to fail.
More seriously, the Agent may have already performed some operations but be unable to accurately determine which succeeded and which failed, leading to duplicated writes, duplicated emails, or duplicated charges.
Therefore, an Agent system must have four core capabilities:
- Reliable execution: complete tasks reliably under normal conditions.
- Fault isolation: local failures do not propagate indefinitely.
- Fault-tolerant handling: adopt reasonable alternative strategies when exceptions occur.
- State recovery: after an interruption, the system can resume from the correct point.
2. Understanding the Agent Reliability Model
1. Reliability, Availability, Fault Tolerance, and Recoverability
These concepts are often used interchangeably, but they address different questions.
| Concept | English | Core meaning |
|---|---|---|
| Reliability | Reliability | The ability of a system to continuously complete tasks correctly under specified conditions |
| Availability | Availability | The ability of a system to provide service when needed |
| Fault tolerance | Fault Tolerance | The system can maintain service when some components fail |
| Resilience | Resilience | The ability of a system to recover and adapt after being affected by a failure |
| Recoverability | Recoverability | The ability to restore state and continue execution after a failure |
| Observability | Observability | Understanding the internal state of a system through logs, metrics, and traces |
For example:
An Agent service may always be accessible but frequently produce incorrect results.
This means it has a certain degree of availability, but not necessarily good reliability.
Another Agent may encounter a timeout when calling an external API, yet automatically switch to a backup service and continue execution; this demonstrates fault tolerance.
If the service can resume a task from a persisted checkpoint after a restart, that demonstrates recoverability.
2. What Determines the Reliability of an Agent?
An Agent's reliability is not simply the model's accuracy.
It can be understood from multiple dimensions:
Where:
- R_{model}: Reliability of model reasoning and output.
- R_{tool}: Reliability of tool invocation.
- R_{state}: Reliability of state management.
- R_{workflow}: Reliability of task orchestration.
- R_{runtime}: Reliability of the runtime environment.
These factors affect one another.
For example, the model may generate correct tool-call parameters, but if the tool service times out, the task still cannot be completed.
Or a tool may execute successfully, but the system fails to save state in time, so execution may be repeated after a process crash.
Therefore, reliability must cover the entire execution chain.
3. The Agent's End-to-End Execution Chain
flowchart TD
A["User submits task"] --> B["Task understanding and planning"]
B --> C["Generate execution steps"]
C --> D["Model inference"]
D --> E["Select tool"]
E --> F["Validate tool parameters"]
F --> G["Execute tool"]
G --> H["Validate result"]
H --> I["Update task status"]
I --> J{"Is the task complete?"}
J -->|"No"| D
J -->|"Yes"| K["Generate final result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Failures can occur at every stage, so you cannot check whether a task succeeded only at the final result stage.
III. Common Agent Failure Types
Before designing a fault-tolerant system, you first need to identify the sources of failure.
1. Model Layer Failures
Model layer failures mainly come from the model service and the reasoning process.
| Failure | Typical Symptom |
|---|---|
| API Timeout | Model request timed out |
| Rate Limit | Request rate exceeds the limit |
| Service Unavailable | Model service unavailable |
| Context Overflow | Input exceeds the context window |
| Invalid Output | Output format does not meet requirements |
| Hallucination | Generates non-existent information |
| Tool Selection Error | Selects the wrong tool |
| Reasoning Failure | Logical error in the reasoning process |
For example, the Agent asks the model to generate JSON:
{
"tool": "search",
"query": "AI Agent"
}
But the model actually returns:
I am going to call the search tool to look up information about AI Agent.
If the system parses it directly as JSON, a format error will occur.
Model output must undergo structural validation; you cannot assume the model will always follow constraints.
2. Tool Layer Failures
Agents rely on external tools to perform actual operations, so the tool layer is a high-incidence area for failures.
Common problems include:
- HTTP request timeout.
- The API returns 500.
- The API returns 429.
- Database connection dropped.
- File does not exist.
- Insufficient permissions.
- Third-party service maintenance.
- Tool response format changes.
- The tool executes successfully but the response is lost.
One particularly dangerous situation is:
The tool has already executed successfully, but the Agent does not receive a success response.
For example:
The Agent calls an email service to send an email.
The email service has already sent the email successfully, but the network is interrupted before the response is returned.
The Agent only sees a request timeout and cannot determine whether the email was actually sent.
If it retries directly, it may send the customer two identical emails.
This type of problem needs to be solved through idempotency and execution status queries.
3. State Layer Failures
The state layer is responsible for saving the Agent's task progress, context, and execution results.
Common failures include:
- State is not saved in time.
- State write fails.
- Multiple Workers modify the state at the same time.
- Checkpoint data is corrupted.
- Expired state is loaded during recovery.
- Conflicts occur when merging results from parallel tasks.
- The state is inconsistent with the actual results in the external system.
For example:
The Agent has completed data analysis, but before the analysis results are saved, the service crashes.
After recovery, the Agent may re-execute the entire analysis process.
If the analysis process includes operations with side effects, this may cause duplicate execution.
4. Workflow Layer Failures
Complex Agents are usually composed of multiple steps.
For example:
Data collection
→ Data cleaning
→ Data analysis
→ Report generation
→ Review
→ Publish
Possible issues include:
- A step fails.
- A preceding step has not completed, but a subsequent step executes early.
- Incorrect task dependencies.
- Not all parallel steps have completed.
- A sub-Agent is unresponsive for a long time.
- A task enters an infinite loop.
- A task remains in the processing state for a long time.
These failures need to be addressed through workflow state machines, timeout control, and task scheduling mechanisms.
5. Infrastructure Failures
The infrastructure layer includes:
- Insufficient CPU resources.
- Memory exhaustion.
- Insufficient disk space.
- Abnormal container exit.
- Network interruption.
- Message queue unavailable.
- Database failure.
- Cloud service region failure.
- Worker process crash.
These problems usually cannot be solved by modifying the Prompt; they depend on fault-tolerant design of the system architecture and infrastructure.
IV. Error Classification: Which Errors Are Recoverable?
Not all errors should be retried.
If error types are not distinguished, an Agent may keep repeating execution in an unrecoverable situation, wasting resources and even expanding the impact of the failure.
1. Transient Error
Transient errors can usually be recovered from by waiting or retrying.
Examples include:
- Brief network interruption.
- HTTP 429.
- Service temporarily unavailable.
- Database connection timeout.
- Brief resource contention.
Suitable strategies include:
- Delayed retry.
- Exponential backoff.
- Random jitter.
- Service failover.
2. Permanent Error
Permanent errors usually cannot be resolved by simple retries.
Examples include:
- Invalid API Key.
- Insufficient user permissions.
- Invalid parameter format.
- Resource does not exist.
- Request violates business rules.
- Tool name does not exist.
In this case, you should:
- Stop the current operation immediately.
- Return a clear error.
- Correct the parameters or permissions.
- Request additional information from the user when necessary.
3. Business Error
Business errors indicate that the operation itself does not comply with business rules.
Examples include:
- Insufficient account balance.
- Approval was not granted.
- The order status does not allow cancellation.
- The user has exceeded the operation quota.
- The current task is not allowed to be published.
Such errors should not be treated as technical failures and retried repeatedly.
4. Ambiguous Failure
Ambiguous failures are a category that requires particular attention in Agent systems.
Examples include:
- A remote operation may have succeeded, but the response was lost.
- The database commit result is unknown.
- It cannot be confirmed whether the email was sent successfully.
- The payment request timed out, but the third-party payment platform has not returned a final status.
In this case, the system cannot simply conclude that the operation failed.
The correct handling approach is usually as follows:
- Query the remote operation status.
- Use an idempotency key to check whether it has already been executed.
- Decide whether to continue based on the actual result.
- Escalate to manual handling when it cannot be confirmed.
5. Error Classification Table
| Error Type | Example | Recommended Strategy |
|---|---|---|
| Transient error | Network timeout | Limited retries |
| Permanent error | Invalid parameter | Fail immediately |
| Business error | Approval rejected | Business branch handling |
| Ambiguous failure | Request/response lost | Query execution status |
| Resource error | Insufficient memory | Limit resources or reschedule |
| State error | Checkpoint corrupted | Restore backup or manual intervention |
6. Error Classification and Handling Process
flowchart TD
A["Catch exception"] --> B["Extract error information"]
B --> C["Exception classification"]
C --> D{"Error type"}
D -->|"Transient"| E["Limited retry"]
D -->|"Permanent"| F["Stop execution and return an error"]
D -->|"Business exception"| G["Enter business processing branch"]
D -->|"Uncertainty"| H["Query remote execution status"]
D -->|"Resource exception"| I["Throttle or reschedule"]
E --> J{"Recovery successful?"}
J -->|"Yes"| K["Continue task"]
J -->|"No"| L["Degrade or terminate"]
H --> M{"Operation result confirmed?"}
M -->|"Succeeded"| K
M -->|"Failed"| E
M -->|"Cannot confirm"| N["Manual intervention"]
G --> O["Update business status"]
I --> P["Re-execute after restoring resources"]
F --> Q["Record failure status"]
L --> Q
N --> Q
O --> Q
P --> K
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
style O fill:#EAF3EC,stroke:#83A995,color:#315D49
style P fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style Q fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
V. Retry Mechanism: Automatically Recover from Transient Failures
Retry is the most common fault-tolerance mechanism.
When an Agent fails to call a tool, it can reissue the request based on the exception type.
But retry is not simply repeating the call.
A reasonable retry system needs to control:
- Which errors can be retried.
- Maximum number of retries.
- Retry interval.
- Total execution time.
- Whether there are side effects.
- Whether the service needs to be switched.
1. Fixed-interval retry
Fixed-interval retry is the simplest approach.
For example:
First failure
→ Wait 2 seconds
→ Second retry
→ Wait 2 seconds
→ Third retry
The advantage is simple implementation.
The disadvantage is that when many requests fail at the same time, all of them may be reissued at the same moment, creating a traffic spike.
2. Exponential Backoff
Exponential backoff gradually increases the wait time based on the number of retries.
Formula:
Where:
- T_n: Wait time for the n-th retry.
- T_0: Initial wait time.
- T_{max}: Maximum wait time.
- n: Number of retries.
For example, with an initial wait of 1 second:
| Retry attempt | Wait time |
|---|---|
| First | 1 second |
| Second | 2 seconds |
| Third | 4 seconds |
| Fourth | 8 seconds |
| Fifth | 16 seconds |
This approach reduces the sustained pressure on the service during a failure.
3. Random jitter
If all requests use the same exponential backoff timing, they may still reissue their requests at the same moment.
So random jitter is usually added:
This spreads the requests out and prevents a large number of requests from recovering simultaneously.
4. The number of retries must be limited
Unlimited retries lead to:
- Continuous resource consumption.
- Tasks that can never finish.
- A constantly growing number of requests.
- Greater pressure on downstream services.
- A backlog in the task queue.
Therefore the following must be limited at the same time:
- The maximum number of retries.
- The timeout for a single request.
- The overall task deadline.
- The total retry budget.
For example:
import random
import time
def retry(
operation,
max_attempts=4,
base_delay=1,
max_delay=16
):
for attempt in range(max_attempts):
try:
return operation()
except TimeoutError:
if attempt == max_attempts - 1:
raise
delay = min(
max_delay,
base_delay * (2 ** attempt)
)
delay = random.uniform(0, delay)
time.sleep(delay)
raise RuntimeError("重试失败")
This example only catches timeout exceptions.
In real engineering, whether to retry should be decided by the type of exception, and retry control should be combined with the overall task deadline so that the wait time does not exceed the task budget.
5. The Retry-After response header
When an external API returns HTTP 429 or 503, it may provide a Retry-After response header.
For example:
HTTP/1.1 429 Too Many Requests
Retry-After: 10
This indicates that the client is advised to wait 10 seconds before trying again.
An Agent tool client should give priority to the retry advice explicitly provided by the server, while still being constrained by the overall deadline and the maximum number of retries.
6. Retry flow
flowchart TD
A["Initiate tool call"] --> B["Wait for execution result"]
B --> C{"Succeeded?"}
C -->|"Yes"| D["Return execution result"]
C -->|"No"| E["Identify exception type"]
E --> F{"Retry allowed?"}
F -->|"No"| G["Record failure and stop"]
F -->|"Yes"| H{"Retry budget exceeded?"}
H -->|"Yes"| G
H -->|"No"| I["Compute backoff time"]
I --> J["Apply random jitter"]
J --> K["Wait and re-execute"]
K --> B
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
VI. Idempotency: Preventing Side Effects from Repeated Execution
The retry mechanism has an important prerequisite:
Repeatedly executing the same operation must not produce uncontrollable duplicate side effects.
This is idempotency.
1. What Is Idempotency?
If an operation produces the same final result whether it is executed once or multiple times, then the operation is idempotent.
Mathematically:
For example:
Setting a user's status to active — executed once or multiple times, the final status is always active.
But the following operations are usually not naturally idempotent:
- Creating an order.
- Sending an email.
- Charging a payment.
- Creating a file.
- Increasing an account balance.
- Publishing social media content.
For example:
account.balance += 100
Executing it twice adds 200, not 100.
2. Using an Idempotency Key
You can generate a unique idempotency key for each logical operation.
For example:
{
"task_id": "task_001",
"operation_id": "op_001",
"idempotency_key": "task_001_send_report_001",
"action": "send_email"
}
After the server receives the request:
- Check whether the idempotency key has already been processed.
- If it has been processed, return the previously saved result.
- If it has not been processed, execute the operation.
- Save the operation result and the idempotency key.
- Subsequent identical requests directly return the existing result.
Simplified example:
class IdempotencyStore:
def __init__(self):
self.results = {}
def execute_once(self, key, operation):
if key in self.results:
return self.results[key]
result = operation()
self.results[key] = result
return result
This code is only meant to illustrate the concept of an idempotency key.
Production environments cannot rely solely on an in-memory dictionary, because records are lost after a process restart, and concurrent requests may pass the check at the same time.
You should use persistent storage with unique constraints and atomic operation capabilities.
3. Idempotency Keys Must Be Bound to Operation Parameters
The same idempotency key should not be used for different business operations.
For example:
First request:
{
"idempotency_key": "order_001",
"amount": 100
}
Second request:
{
"idempotency_key": "order_001",
"amount": 1000
}
If the system only checks the idempotency key without checking the parameters, it may return an incorrect result.
Therefore you should save:
- The idempotency key.
- The operation type.
- A digest of the key business parameters.
- The execution status.
- The execution result.
- The creation time.
If the same idempotency key corresponds to different parameters, the request should be rejected.
4. Idempotency State Machine
stateDiagram-v2
[*] --> PENDING
PENDING --> PROCESSING: start execution
PROCESSING --> SUCCEEDED: execution succeeded
PROCESSING --> FAILED: explicit execution failure
PROCESSING --> UNKNOWN: execution result unknown
FAILED --> PROCESSING: retry conditions met
UNKNOWN --> SUCCEEDED: query confirms success
UNKNOWN --> FAILED: query confirms failure
UNKNOWN --> PROCESSING: confirmed safe to retry
SUCCEEDED --> [*]
Pay special attention:
UNKNOWN cannot be directly equated with FAILED.
If the remote service has already completed the operation and only the response was lost, retrying directly may produce duplicate side effects.
VII. Circuit Breaker: Avoiding Continuous Fault Propagation
When an Agent frequently calls a service that is already unavailable, continuous retries may further increase the pressure on the downstream service.
A circuit breaker is used to temporarily stop calls when a service keeps failing.
1. The Three States of a Circuit Breaker
| State | Meaning |
|---|---|
| Closed | Normal state, requests are allowed through |
| Open | Open state, requests are rejected directly |
| Half-Open | Half-open state, a small number of probe requests are allowed |
2. Closed: Normal State
Under normal conditions, requests pass through the circuit breaker and call the service.
If the error rate keeps rising and reaches the configured threshold, it enters the Open state.
3. Open: Disconnected State
After the circuit breaker opens, it no longer sends normal requests to the failing service.
The system can:
- Return an error quickly.
- Use cached results.
- Switch to a backup service.
- Pause related tasks.
- Place tasks in a waiting queue.
This prevents continuous requests from further overwhelming the downstream service.
4. Half-Open: Half-Open State
After a cooldown period, the system allows a small number of probe requests.
If the probe succeeds, it returns to the Closed state.
If the probe fails, it re-enters the Open state.
5. Circuit Breaker State Transitions
stateDiagram-v2
[*] --> Closed
Closed --> Open: failure rate exceeds threshold
Open --> HalfOpen: cooldown period ends
HalfOpen --> Closed: probe request succeeds
HalfOpen --> Open: probe request fails
6. Difference Between Circuit Breaker and Retry
| Mechanism | Problem Solved |
|---|---|
| Retry | A single request temporarily fails |
| Circuit Breaker | Downstream service sustains failures |
| Timeout | A request receives no response for a long time |
| Rate Limiter | Request rate is too high |
| Bulkhead | Failure spreads to other resource pools |
They usually need to be used together.
For example:
When an Agent calls an external search API:
- Single-request timeouts are controlled by Timeout.
- Transient errors are handled by Retry.
- Circuit Breaker protects against persistent service failures.
- Excessive requests are limited by Rate Limiter.
- Bulkhead isolates resources between different tools.
VIII. Timeout Control: Preventing Tasks from Waiting Indefinitely
An Agent often needs to wait for external tools to return results.
If a tool does not have a reasonable timeout setting, a single request may occupy connections, threads, and task resources for a long time.
1. Common Timeout Types
| Timeout Type | Meaning |
|---|---|
| Connection Timeout | Maximum wait time to establish a connection |
| Read Timeout | Maximum time to wait for response data |
| Tool Timeout | Maximum time for a single tool call |
| Model Timeout | Maximum time for a model request |
| Step Timeout | Maximum time for a single workflow step |
| Task Deadline | Deadline for the entire task |
2. Why Is an Overall Task Deadline Needed?
Suppose an Agent has 10 steps, and each step can run for at most 60 seconds.
If each step waits independently for 60 seconds, the entire task could take 10 minutes.
If the user requires completion within 3 minutes, a per-step timeout alone cannot guarantee that the overall task meets the requirement.
Therefore, a unified Deadline is needed.
Each time a step is executed, the system must calculate the remaining time:
The timeout for the current step must not exceed the remaining task time.
3. Timeout Control Flow
flowchart TD
A["Create task"] --> B["Set task Deadline"]
B --> C["Execute current step"]
C --> D["Calculate remaining time"]
D --> E{"Has the deadline been exceeded"}
E -->|"Yes"| F["Cancel or terminate the task"]
E -->|"No"| G["Set step timeout"]
G --> H["Call model or tool"]
H --> I{"Completed within the time limit"}
I -->|"Yes"| J["Save step result"]
I -->|"No"| K["Trigger timeout exception"]
K --> L["Determine whether retry is allowed"]
L --> M{"Is there still time budget"}
M -->|"Yes"| C
M -->|"No"| F
J --> N{"Is the task complete"}
N -->|"No"| C
N -->|"Yes"| O["Return final result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
A timeout does not mean the remote operation has stopped.
For example, the client stopping its wait on the email API does not mean the email service has stopped sending emails.
Therefore, for operations with side effects, after a timeout you still need to confirm the execution result through an idempotency key or a remote status query.
IX. Checkpoint: Letting the Agent Resume from the Interruption Point
For long-running Agents, saving task state in memory alone is not enough.
Once the process crashes, the in-memory state is lost.
A checkpoint is used to save the Agent's key state during execution, so that the task can continue from a previous state.
1. What does a Checkpoint save?
Typical contents include:
- Task ID.
- Current execution node.
- Current task status.
- Completed steps.
- Intermediate results.
- Tool call records.
- Retry count.
- Pending operations.
- User approval status.
- Context summary.
- Version and timestamp.
For example:
{
"task_id": "task_001",
"status": "running",
"current_step": "analyze_data",
"completed_steps": [
"collect_data",
"clean_data"
],
"retry_count": 1,
"state_version": 12,
"checkpoint_at": "2026-11-21T10:00:00Z"
}
2. Why can't we save only the model conversation?
An Agent's state is more than chat history.
For example:
- The conversation record describes what the user said.
- The task state describes where execution has currently reached.
- The tool record describes which operations have been performed.
- The checkpoint describes a recoverable state at a certain point in time.
If only the conversation history is saved, the system still cannot accurately determine:
- Which tools have already been executed.
- Which steps have already been completed.
- Which operations have not yet been confirmed.
- Which results have already been persisted.
- Which tasks can be safely retried.
Therefore, a checkpoint should save structured task state, not just chat messages.
3. The basic working mechanism of a Checkpoint
flowchart TD
A["Start Agent Task"] --> B["Read Current State"]
B --> C["Execute Workflow Node"]
C --> D["Produce Node Result"]
D --> E["Update State"]
E --> F["Persist Checkpoint"]
F --> G{"Are there more nodes?"}
G -->|"Yes"| C
G -->|"No"| H["Task Complete"]
F --> I["Process Interruption Occurs"]
I --> J["Recover Worker"]
J --> K["Load Latest Checkpoint"]
K --> L["Restore Task State"]
L --> C
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
4. When to Save a Checkpoint
Different saving strategies can be used.
Save after each node completes
Save the state immediately after each workflow node executes successfully.
The advantage is finer recovery granularity.
The disadvantage is a higher write frequency.
Save at key steps
Save only at important nodes.
For example:
- Data collection completed.
- Data analysis completed.
- Report generation completed.
- External operation completed.
This approach reduces persistence overhead, but some work may need to be re-executed when a failure occurs.
Scheduled saving
Save the state at fixed time intervals.
This suits long-running tasks, but you need to consider state consistency between the scheduled saves and the actual operations.
5. Consistency Between Checkpoints and External Side Effects
This is a difficult point in production-grade Agent design.
Suppose:
- The Agent calls a payment tool.
- The payment service has already charged the customer.
- The Agent has not yet saved a Checkpoint.
- The process crashes.
After recovery, the Agent may not know whether the payment has completed.
Therefore:
A Checkpoint can only restore the local workflow state; it cannot automatically restore the true state of external systems.
Checkpoints need to be combined with idempotent operation records, remote state queries, and transaction mechanisms.
10. State Machines: Clarifying Which Stage a Task Is In
Agent state management cannot rely on just a simple running or failed field.
Complex tasks require explicit state machines.
1. Common Task States
| State | Meaning |
|---|---|
| CREATED | The task has been created |
| PENDING | Waiting to be scheduled |
| RUNNING | Currently executing |
| WAITING | Waiting for an external event |
| RETRYING | Currently retrying |
| PAUSED | Paused intentionally |
| RECOVERING | Currently recovering |
| SUCCEEDED | Completed successfully |
| FAILED | Execution failed |
| CANCELLED | Cancelled by the user |
| TIMED_OUT | Terminated due to timeout |
2. Why Do We Need a State Machine?
Suppose a task is currently in SUCCEEDED.
The system should not switch it back to RUNNING unless it explicitly creates a new execution version or reopens the task.
Likewise, a task in CANCELLED must not automatically resume execution just because a delayed tool response arrives.
A state machine can restrict illegal state transitions.
3. State Transition Rules
stateDiagram-v2
[*] --> CREATED
CREATED --> PENDING: Submit for scheduling
PENDING --> RUNNING: Worker starts execution
RUNNING --> WAITING: Waiting for external event
WAITING --> RUNNING: External event received
RUNNING --> RETRYING: Transient error
RETRYING --> RUNNING: Retry succeeded
RETRYING --> FAILED: Retries exhausted
RUNNING --> PAUSED: Manually paused
PAUSED --> RUNNING: Resume execution
RUNNING --> RECOVERING: Runtime environment interrupted
RECOVERING --> RUNNING: Recovery succeeded
RECOVERING --> FAILED: Cannot recover
RUNNING --> SUCCEEDED: Task completed
RUNNING --> FAILED: Unrecoverable error
RUNNING --> TIMED_OUT: Deadline exceeded
PENDING --> CANCELLED: Cancelled by user
RUNNING --> CANCELLED: Cancelled by user
SUCCEEDED --> [*]
FAILED --> [*]
CANCELLED --> [*]
TIMED_OUT --> [*]
The state machine should validate transitions on the server side.
The model must not be allowed to write arbitrary task states directly.
4. State updates require concurrency control
If two Workers process the same task at the same time, state overwrites can occur.
For example:
- Worker A reads version 10.
- Worker B reads version 10.
- Worker A updates to version 11.
- Worker B overwrites version 11 using stale data.
You can use Optimistic Locking:
UPDATE agent_tasks
SET
status = 'RUNNING',
version = version + 1
WHERE
task_id = 'task_001'
AND version = 10;
If the number of updated rows is 0, the version has already changed, and you need to re-read the state and handle the conflict.
In production, you can also use task leases, Worker heartbeats, and fencing tokens to prevent an old Worker that has lost its lease from continuing to write state.
XI. Failure recovery for long-running tasks
The biggest difference between long-running Agents and ordinary request-response Agents is that a task may span multiple execution cycles.
For example:
- The Worker restarts while the task is executing.
- User approval is not completed until several hours later.
- An external service does not recover until the next day.
- The task waits for third-party data.
- A system deployment or upgrade causes the execution process to exit.
This means the Agent must have the ability to persist tasks and recover asynchronously.
1. Architecture for long-running tasks
A reliable long-running Agent usually includes:
| Component | Role |
|---|---|
| Task Store | Stores basic task information |
| State Store | Stores execution state |
| Checkpoint Store | Stores recovery checkpoints |
| Message Queue | Schedules tasks |
| Worker Pool | Executes tasks |
| Scheduler | Manages task scheduling |
| Retry Manager | Controls retries |
| Recovery Manager | Detects and recovers abnormal tasks |
| Event Store | Stores key execution events |
2. Task interruption and recovery
flowchart TD
A["Create long-running task"] --> B["Save task information"]
B --> C["Enter task queue"]
C --> D["Worker fetches task"]
D --> E["Load task state"]
E --> F["Execute current step"]
F --> G["Save checkpoint"]
G --> H{"Did it complete normally?"}
H -->|"Yes"| I["Mark task as successful"]
H -->|"No"| J{"Did an exception occur?"}
J -->|"No"| F
J -->|"Yes"| K["Record failure event"]
K --> L["Recovery Manager analyzes"]
L --> M{"Can it be recovered automatically?"}
M -->|"Yes"| N["Reschedule task"]
N --> D
M -->|"No"| O["Pause task and raise alert"]
O --> P["Manual handling"]
P --> Q{"Is continuation allowed?"}
Q -->|"Yes"| N
Q -->|"No"| R["Mark task as failed"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#EAF3EC,stroke:#83A995,color:#315D49
style O fill:#F7EAE7,stroke:#C98D7C,color:#754438
style P fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style Q fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style R fill:#F7EAE7,stroke:#C98D7C,color:#754438
3. Worker Crash Detection
A Worker may exit for the following reasons:
- The container is terminated.
- The process crashes.
- Memory is exhausted.
- The network is interrupted.
- The node fails.
The system can use a heartbeat mechanism to determine whether a Worker is still alive.
For example:
{
"worker_id": "worker_001",
"task_id": "task_001",
"lease_expires_at": "2026-11-21T10:05:00Z",
"heartbeat_at": "2026-11-21T10:04:30Z"
}
If the lease expires, the scheduler can consider that the Worker no longer has the right to execute the task and reassign the task.
However, note that:
The old Worker may only have experienced a temporary network interruption and may not have actually stopped.
Therefore, when reassigning tasks, you must use leases, fencing tokens, and idempotent execution mechanisms to prevent two Workers from causing side effects at the same time.
12. Saga: Handling Cross-Service Long Transactions
An Agent may need to call multiple external services in succession.
For example:
- Create an order.
- Deduct inventory.
- Deduct the account balance.
- Create a shipping order.
- Send an order notification.
These operations may be distributed across multiple independent services and cannot be committed uniformly using a single database transaction.
If the third step fails, how should the effects already produced by the first two steps be handled?
This is the problem that the Saga pattern is designed to solve.
1. What Is Saga?
Saga is a distributed transaction coordination pattern.
It splits a long transaction into multiple local transactions, each with a corresponding compensating action.
For example:
| Forward operation | Compensating operation |
|---|---|
| Create order | Cancel order |
| Deduct inventory | Restore inventory |
| Freeze balance | Unfreeze balance |
| Create shipping order | Cancel shipping order |
Note that a compensating operation is not necessarily the same as a database rollback.
For example, an email that has already been sent to a customer cannot truly be recalled; it can only be remedied through a follow-up notification.