I. Why Agents Need Loop Execution
Traditional programs usually execute according to a predefined flow; the program knows what to do next and returns a result after execution is complete.
Agents are different. They need to dynamically decide the next action based on the current task state, model reasoning results, and tool execution feedback.
For example, an Agent that automatically analyzes data and generates reports may need to:
- Understand the analysis goal proposed by the user.
- Determine whether the database needs to be queried.
- Call database tools to obtain data.
- Analyze the query results.
- Discover that the data is insufficient and continue calling other tools.
- Integrate the analysis results.
- Determine whether the task goal has been met.
- Generate the final report.
During execution, the next operation often depends on the result of the previous step, so it cannot rely entirely on a fixed linear flow.
An Agent needs an execution mechanism that can run continuously, make dynamic decisions, and adjust behavior based on feedback—this is the Agent Loop.
1.1 Basic Concepts of the Agent Loop
The Agent Loop is the Agent's cyclic execution mechanism; its core is to let the model observe the current state in each round of execution and decide the next operation.
A typical loop contains four phases:
| Phase | Core Responsibility |
|---|---|
| Observe | Observe the current state and execution results |
| Think | Analyze task progress and determine the next action |
| Act | Call tools or perform specific operations |
| Evaluate | Check execution results and determine whether to continue |
Among them, Evaluate is not necessarily a separate model call; it can also be completed jointly by rules, programs, or tool results.
1.2 Execution Flow of the Agent Loop
flowchart TD
A["Receive user task"] --> B["Initialize Agent state"]
B --> C["Observe current state"]
C --> D["Model reasoning and decision-making"]
D --> E{"Need to perform an action?"}
E -->|"Yes"| F["Call tools or perform actions"]
F --> G["Obtain execution results"]
G --> H["Update Agent state"]
H --> I{"Task completed?"}
I -->|"No"| C
I -->|"Yes"| J["Generate final result"]
E -->|"No"| J
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
The key to the loop is that after each execution, the Agent writes new information into the state and makes a new decision based on the updated state.
Without state updates, the Agent cannot reliably know what work it has already completed.
II. Core Execution Mechanisms of the Agent Loop
The Agent Loop is not merely repeatedly calling a large model; it is a control mechanism that includes decision-making, action execution, state updates, and termination determination.
2.1 Observe: Observing the Current State
The Observe phase is responsible for collecting the information currently available to the Agent.
It mainly includes:
- The task initially proposed by the user.
- Operations already completed.
- Historical model reasoning results.
- Tool call records.
- Tool execution return values.
- Current task progress.
- Remaining execution budget.
- Error messages and abnormal states.
For example:
state = {
"task": "分析最近一个月的销售数据",
"messages": [],
"tool_results": [],
"completed_steps": [],
"current_step": "query_sales",
"status": "running"
}
Observe does not mean handing all historical information directly to the model every round.
In long tasks, relevant messages, tool results, and task summaries should be selected according to the needs of the current decision, to avoid unbounded context growth.
2.2 Think: Model Reasoning and Decision-Making
The Think phase is responsible for deciding the next operation based on the current state.
The model may make the following decisions:
- Continue calling a tool.
- Adjust tool parameters.
- Perform a new analysis based on the results.
- Ask the user for additional information.
- Determine that the task has been completed.
- Identify errors and attempt to recover.
For example, the model might return:
{
"action": "call_tool",
"tool": "query_sales",
"arguments": {
"start_date": "2026-10-01",
"end_date": "2026-10-31"
},
"reason": "需要先获取销售数据"
}
The output here is a structured decision rather than a direct database query execution.
In production environments, you should perform schema validation, tool permission checks, and parameter validation on model output; model-generated content must never be treated directly as trusted instructions.
2.3 Act: Executing Concrete Operations
The Act stage is responsible for actually executing the action selected by the model.
Common operations include:
| Operation Type | Example |
|---|---|
| Tool invocation | Querying a database, searching the web |
| API request | Calling a business system interface |
| Code execution | Running Python to analyze data |
| File operations | Reading, generating, and modifying files |
| State operations | Updating task progress |
| Human interaction | Requesting user confirmation |
Note that the model deciding to call a tool does not mean the tool has been successfully executed.
A tool call needs to distinguish at least the following states:
- Pending: awaiting execution.
- Running: currently executing.
- Succeeded: execution succeeded.
- Failed: execution failed.
- Timeout: execution timed out.
- Cancelled: execution was cancelled.
Tool execution results must be fed back to the Agent before it can proceed to the next round of decision-making.
2.4 Evaluate: Assessing Execution Results
The Evaluate stage is responsible for determining whether the current result meets expectations.
For example:
- Did the tool execute successfully?
- Is the returned data complete?
- Has the current step been completed?
- Is a retry needed?
- Does the strategy need adjustment?
- Has the final task objective been met?
Evaluation can be carried out through a combination of mechanisms:
| Evaluation Method | Applicable Scenario |
|---|---|
| Rule-based judgment | Checking fields, status codes, counts |
| Model judgment | Judging whether the result meets the semantic goal |
| Tool verification | Verifying the result through queries or tests |
| Human confirmation | When important business operations are involved |
| Multi-layered verification | High-risk or high-value tasks |
For important tasks, you should not end the loop merely because the model says "it's done"; instead, confirm completion through verifiable business conditions.
III. Termination Conditions of the Agent Loop
Loop execution must have an explicit termination mechanism; otherwise, the Agent may call tools indefinitely, repeatedly reason, or keep attempting ineffective operations.
3.1 Common Termination Conditions
| Termination Condition | Description |
|---|---|
| Goal Achieved | The task objective has been achieved |
| Max Iterations | The maximum number of loops has been reached |
| Timeout | The task execution time has been exceeded |
| Token Budget | The model Token budget has been exceeded |
| Cost Budget | The task cost budget has been exceeded |
| User Cancelled | The user actively cancelled |
| Fatal Error | An unrecoverable error occurred |
| Human Approval | Pause pending human confirmation |
Two states need to be distinguished here:
- Completed: the task has been completed.
- Suspended: the task is temporarily paused and may be resumed later.
Reaching the maximum number of iterations does not mean the task has succeeded; it should usually be marked as completed-with-limits or suspended, with the current progress explained to the user.
3.2 Execution Flow with Termination Judgment
flowchart TD
A["Enter Agent Loop"] --> B["Read current state"]
B --> C["Check task execution budget"]
C --> D{"Is the budget exceeded?"}
D -->|"Yes"| E["Mark task as paused or failed"]
D -->|"No"| F["Execute model decision"]
F --> G["Execute tool or internal operation"]
G --> H["Update execution state"]
H --> I{"Is the task goal met?"}
I -->|"Yes"| J["Mark task as complete"]
I -->|"No"| K{"Has an unrecoverable error occurred?"}
K -->|"Yes"| E
K -->|"No"| L{"Is human confirmation required?"}
L -->|"Yes"| M["Pause and wait for human input"]
L -->|"No"| B
M --> N["Receive human decision"]
N --> O{"Is continuation allowed?"}
O -->|"Yes"| B
O -->|"No"| P["End task"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#F7EAE7,stroke:#C98D7C,color:#754438
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style O fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style P fill:#E5F0E9,stroke:#83A995,color:#315D49
3.3 Maximum Iterations and Execution Budget
In production environments, multiple limits usually need to be set at the same time.
MAX_ITERATIONS = 20
MAX_TOOL_CALLS = 30
MAX_EXECUTION_SECONDS = 300
MAX_RETRIES = 3
def should_stop(state):
if state["iteration"] >= MAX_ITERATIONS:
return True
if state["tool_call_count"] >= MAX_TOOL_CALLS:
return True
if state["elapsed_seconds"] >= MAX_EXECUTION_SECONDS:
return True
return False
In an actual implementation, execution time should be calculated by a monotonic clock or the task scheduler, rather than relying on the model to judge it on its own.
Budgets should also be configurable by task type. For example, a simple question-and-answer task may allow only a small number of model calls, whereas a complex research task allows more time and more tool calls.
IV. What Is Agent State
Agent state is the set of data that an Agent needs to maintain continuously during execution.
It includes not only conversation messages, but also task goals, execution progress, tool call results, error information, and control state.
4.1 Composition of Agent State
A relatively complete Agent state can be divided into the following categories:
| State Category | Typical Fields | Purpose |
|---|---|---|
| Task State | task_id, goal, status | Describes the task itself |
| Conversation State | messages | Stores conversation context |
| Execution State | iteration, current_step | Records execution progress |
| Tool State | tool_calls, tool_results | Records tool execution |
| Memory State | summary, memory_refs | Stores task-related memory |
| Error State | errors, retry_count | Records errors and retries |
| Control State | next_action, interrupt | Controls the execution flow |
| Budget State | token_usage, cost, deadline | Manages the execution budget |
Different Agent frameworks may design their fields differently, but the state responsibilities they need to cover are roughly the same.
4.2 Agent State Example
from typing import TypedDict, Literal, Any
class AgentState(TypedDict):
task_id: str
goal: str
messages: list[dict[str, Any]]
status: Literal[
"pending",
"running",
"waiting",
"completed",
"failed",
"cancelled"
]
iteration: int
current_step: str
tool_calls: list[dict[str, Any]]
tool_results: list[dict[str, Any]]
errors: list[dict[str, Any]]
retry_count: int
summary: str
token_usage: int
cost: float
The state here describes the execution context of an Agent task.
It is important to note that task state and model context are not the same concept:
- Agent state is the complete execution data.
- Model context is the information selectively provided each time the model is requested.
- Persisted state is data that needs to be retained across processes and over time.
The three are related, but should not be simply equated.
4.3 Changes in State During the Loop
Suppose the Agent needs to query sales data and generate a report:
| Execution Stage | iteration | current_step | status |
|---|---|---|---|
| Initialization | 0 | planning | pending |
| First reasoning | 1 | query_sales | running |
| Query complete | 1 | analyze_data | running |
| Data analysis | 2 | generate_report | running |
| Report generation | 3 | validation | running |
| Validation passed | 4 | completed | completed |
Each execution round may change multiple state fields.
Therefore, state management must not only save the final result but also record how the task changes from start to finish.
5. Why Agent State Must Be Persisted
If the Agent keeps state only in memory, task state may be lost after the program restarts, the server crashes, or the execution environment is destroyed.
For short-running, low-complexity tasks, in-memory state is usually sufficient.
However, for tasks that need to run for minutes or even hours, relying solely on memory introduces significant risks.
5.1 What Happens Without Persistence
Suppose the Agent is executing a complex data analysis task:
- It has already queried multiple data sources.
- It has already completed data cleaning.
- It has already generated intermediate analysis results.
- It is about to call a tool to generate a report.
- At this point, the server suddenly restarts.
If all state is stored in process memory, after the restart the Agent cannot accurately know:
- What the original task was.
- Which tools have already been executed.
- Which operations have already succeeded.
- Which steps have not yet been executed.
- Where it should continue from now.
An Agent may be forced to start over from the beginning, causing repeated computation, repeated calls, or even repeated business operations.
5.2 The Core Value of State Persistence
| Value | Description |
|---|---|
| Failure recovery | Restore tasks after process restart |
| Long-running task execution | Support continuous operation over time |
| Task suspension | Support resuming execution after human intervention |
| Execution tracking | Query historical states and execution records |
| Concurrency management | Support collaboration among multiple Workers |
| Audit analysis | Reconstruct the task execution process |
| Time travel | Return to historical states for inspection or recovery |
State persistence addresses the problem of preserving the Agent's execution context.
It does not automatically guarantee that external tool calls will not be duplicated, nor does it mean the entire system has transactional consistency.
6. Checkpoint: The Agent's Checkpoint Mechanism
Checkpoint is one of the most important mechanisms in Agent state persistence.
It means saving the current state as a recoverable checkpoint at a certain point in execution.
When a task reaches a specific stage, the system writes the state to persistent storage.
If a failure occurs later, execution can be recovered from the most recent valid checkpoint.
6.1 Basic Structure of a Checkpoint
A checkpoint typically needs to contain:
| Field | Purpose |
|---|---|
| checkpoint_id | Unique checkpoint identifier |
| task_id | Task it belongs to |
| thread_id | Execution session identifier |
| sequence | Checkpoint sequence number |
| parent_checkpoint_id | Previous checkpoint |
| state | Current state snapshot |
| metadata | Metadata such as version and time |
| created_at | Creation time |
Example:
{
"checkpoint_id": "cp_0003",
"task_id": "task_1001",
"thread_id": "thread_abc",
"sequence": 3,
"parent_checkpoint_id": "cp_0002",
"state": {
"status": "running",
"iteration": 3,
"current_step": "generate_report",
"retry_count": 0
},
"metadata": {
"schema_version": 1
},
"created_at": "2026-11-20T10:30:00Z"
}
Here, state is a snapshot of the state at a certain moment, not simple log text.
6.2 When Checkpoints Are Saved
It is not necessary to create a checkpoint after every statement executed.
Saving too frequently increases storage pressure and write costs, while saving too sparsely increases the recovery workload after a failure.
Common times to save include:
- After a round of Agent reasoning is completed.
- After a tool call is completed.
- After an important task phase is completed.
- When the Agent is about to enter a waiting state for human approval.
- When a key change occurs in the task state.
- When the task ends normally.
For tool calls with external side effects, special attention must be paid to the ordering relationship between checkpoints and tool execution.
6.3 Checkpoint Saving Process
flowchart TD
A["Agent starts execution"] --> B["Read current state"]
B --> C["Perform model inference"]
C --> D["Execute tool or task node"]
D --> E["Generate new state"]
E --> F{"Checkpoint condition met?"}
F -->|"No"| B
F -->|"Yes"| G["Generate state snapshot"]
G --> H["Write to persistent storage"]
H --> I{"Was the save successful?"}
I -->|"No"| J["Handle persistence exception"]
J --> K{"Allow retry?"}
K -->|"Yes"| H
K -->|"No"| L["Pause or mark as failed"]
I -->|"Yes"| M["Record checkpoint version"]
M --> B
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
6.4 Snapshots and Incremental Records
Checkpoints can use different save strategies.
Full Snapshot
Save the complete state each time.
Advantages:
- The recovery process is simple.
- There is no need to replay a large number of historical events.
- Reading the current state is relatively direct.
Disadvantages:
- When the state is large, storage overhead is significant.
- Frequent saves may produce a large amount of duplicate data.
Delta
Save only the parts that changed relative to the previous version.
Advantages:
- Reduce duplicate storage.
- Suitable for tasks with small state changes and long histories.
Disadvantages:
- Recovery requires merging multiple versions.
- It is necessary to handle delta chain corruption and version compatibility.
- Querying historical state is more complex.
In engineering practice, a strategy that combines periodic full snapshots with intermediate delta records can be used.
VII. Storage Architecture for Agent State Persistence
Agent state storage does not necessarily need to use a single database.
Different types of data have different access patterns and lifecycles, so storage media can be chosen based on data characteristics.
7.1 Common Storage Options
| Storage Type | Use Cases | Characteristics |
|---|---|---|
| Memory | Local development, temporary tasks | Simple, but cannot recover across processes |
| SQLite | Standalone applications, small-scale tasks | Simple deployment, supports transactions |
| PostgreSQL | Production task state | Good transaction, indexing, and concurrency capabilities |
| Redis | Cache, short-term state, locks | Fast access; pay attention to persistence configuration |
| Object Storage | Large state snapshots, files | Suitable for storing large objects |
| Event Store | Event history and auditing | Convenient for tracking state changes |
Real projects often use a combined architecture:
- PostgreSQL stores task metadata and checkpoint indexes.
- Redis stores short-term execution state and distributed locks.
- Object storage stores large files and snapshot attachments.
- Logs or event stores record the complete execution process.
7.2 Separate State Data from Business Data
Do not cram all business data directly into Agent State.
For example, an Agent is processing a 500 MB file.
If the complete file content is copied into the state snapshot every time a checkpoint is created, it will cause serious storage and performance problems.
A more reasonable approach is:
```python state = { "task_id": "task_1001", "status": "running", "current_step": "analyze_file", "file_ref": { "object_key": "uploads/report_001.pdf", "version": "v3" }, "summary": "正在分析销售报告", } ```The agent state stores file references, versions, and necessary metadata, while the files themselves live in object storage.
This design reduces checkpoint size and allows large files to have their own independent lifecycle management policies.
7.3 Recommended Overall Architecture
flowchart TD
A["User Request"] --> B["Agent Runtime"]
B --> C["Agent Loop"]
C --> D["State Manager"]
D --> E["Checkpoint Manager"]
E --> F["Persistent Storage"]
D --> G["Context Builder"]
G --> C
C --> H["Tool Execution Layer"]
H --> I["External API / Database"]
I --> J["Execution Result"]
J --> D
F --> K["Recovery Service"]
K --> D
D --> L["Task Status Query API"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#E5F0E9,stroke:#83A995,color:#315D49
Here it is important to distinguish between the Agent Runtime and persistent storage:
- The Runtime is responsible for executing the current task.
- The State Manager is responsible for maintaining runtime state.
- The Checkpoint Manager is responsible for generating and managing checkpoints.
- Persistent storage is responsible for reliably saving state.
- The Recovery Service is responsible for reloading historical state into the runtime environment.
8. Agent Failure Recovery Mechanisms
The real value of persistence lies in being able to recover tasks after the system encounters an exception.
But recovery is not simply a matter of reading the latest checkpoint and then unconditionally re-executing.
Especially when external tool calls are involved, it is necessary to determine whether the operation actually succeeded.
8.1 Common Failure Scenarios
| Failure Scenario | Possible Impact | Handling |
|---|---|---|
| Agent process crash | In-memory state lost | Restore from checkpoint |
| Model request timeout | Inference result unknown | Determine using the request identifier |
| Tool call timeout | External operation may have already executed | Query the execution result or retry idempotently |
| Database temporarily unavailable | Checkpoint cannot be saved | Retry, degrade, or pause |
| Worker terminated | Task interrupted | Reassigned by the scheduler |
| State version change | Old state cannot be read directly | Schema migration |
| Concurrent recovery | Same task executed multiple times | Distributed lock or lease mechanism |
8.2 Recovering from the Latest Checkpoint
A typical recovery process:
- Query the current state of the task.
- Obtain the most recent valid checkpoint.
- Verify the integrity of the checkpoint data.
- Check the state schema version.
- Load the state snapshot.
- Check for tool calls that have not yet completed.
- Resume the Agent Loop.
- Continue executing the subsequent steps.
flowchart TD
A["Task interruption detected"] --> B["Query task records"]
B --> C["Read the most recent valid checkpoint"]
C --> D{"Does the checkpoint exist?"}
D -->|"No"| E["Try to rebuild from the task's initial state"]
D -->|"Yes"| F["Verify snapshot integrity"]
F --> G{"Is the data valid?"}
G -->|"No"| H["Find an earlier valid checkpoint"]
H --> C
G -->|"Yes"| I["Check the state version"]
I --> J["Run Schema compatibility conversion"]
J --> K["Check for unfinished tool calls"]
K --> L["Restore Agent state"]
E --> L
L --> M["Continue Agent Loop"]
style A fill:#F7EAE7,stroke:#C98D7C,color:#754438
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#E5F0E9,stroke:#83A995,color:#315D49
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
8.3 Why Tool Calls Cannot Simply Be Repeated During Recovery
Suppose the Agent performs the following operation:
result = payment_tool(
order_id="order_1001",
amount=100
)
The payment service has already successfully charged the customer, but the Agent crashes before saving a checkpoint.
After restarting, the Agent may find no record of a successful payment in the current state, and therefore call the payment tool again.
If the payment API is not idempotent, this can result in a duplicate charge.
This shows that:
State persistence can only save the execution state known to the Agent; it cannot inherently guarantee consistency of operations between the Agent and external systems.
Therefore, it must be combined with idempotency keys, operation records, and external result queries.
9. Idempotency and Consistency of Tool Calls
The Agent's loop execution is non-deterministic, while external tools often have real business side effects.
As a result, tool calls are the part of state persistence design that is most prone to problems.
9.1 What Is Idempotency
Idempotency means that performing the same operation once or multiple times produces the same final business effect.
For example:
create_payment(
order_id="order_1001",
idempotency_key="task_1001_payment_01"
)
Even if the Agent resends the request due to a timeout or crash, the payment service should use the idempotency key to recognize this as the same business operation and avoid a duplicate charge.
Note that idempotency does not mean every request returns an identical response; rather, it requires that business side effects are not executed more than once.
9.2 Tool Call Records
It is recommended to create a separate record for each tool call:
tool_call = {
"call_id": "call_001",
"task_id": "task_1001",
"tool_name": "create_payment",
"idempotency_key": "task_1001_payment_01",
"status": "succeeded",
"request": {
"order_id": "order_1001",
"amount": 100
},
"result": {
"payment_id": "payment_001"
}
}
A tool call record should contain at least:
- A unique call ID.
- The task ID it belongs to.
- The tool name.
- The idempotency key.
- A summary of the request parameters.
- The execution status.
- The returned result or error.
- The start and end times.
For sensitive data, request parameters and results should be masked or encrypted.
9.3 The Tool Call State Machine
Tool calls can maintain their own state machine independently:
flowchart TD
A["Create tool call record"] --> B["Pending"]
B --> C["Running"]
C --> D{"Execution result"}
D -->|"Success"| E["Succeeded"]
D -->|"Failure"| F["Failed"]
D -->|"Timeout"| G["Unknown"]
G --> H["Query external execution result"]
H --> I{"Already executed"}
I -->|"Yes"| E
I -->|"No"| J["Check whether retry is allowed"]
J -->|"Allowed"| B
J -->|"Not allowed"| F
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#E5F0E9,stroke:#83A995,color:#315D49
style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
Among them, Unknown is a very important state.
A tool call timing out does not necessarily mean execution failed; it may also mean:
- The request has not yet reached the external service.
- The external service has already executed it, but the response was lost.
- The external service is still processing it.
- The external service failed to execute it, but the error response never came back.
Therefore, when you encounter Unknown, you should query the external execution result first, instead of retrying directly.
9.4 Why Exactly Once Cannot Be Promised Lightly
There are three common execution semantics in distributed systems:
| Semantics | Meaning | Risk |
|---|---|---|
| At-most-once | Executed at most once | Operations may be lost |
| At-least-once | Executed at least once | Operations may be executed repeatedly |
| Exactly-once | The business effect occurs exactly once | Requires additional coordination mechanisms |
In scenarios where an Agent interacts with external APIs, it is usually impossible to achieve true Exactly Once relying solely on local checkpoints.
A more practical engineering approach is:
- Use persistent call records.
- Generate a stable idempotency key for each business operation.
- Use idempotent interfaces supported by the external system.
- Query the execution result for operations that timed out.
- Send operations that cannot be confirmed to manual review.
- Reconcile regularly to fix inconsistent states.
The ultimate goal is controllable idempotency in terms of business effects, not simply guaranteeing that a network request is sent only once.
10. Concurrent Execution and State Consistency
When an Agent system has only one Worker, state updates are relatively simple.
But in production environments there are usually multiple Workers processing different tasks at the same time, and abnormal task scheduling may even cause the same task to be executed by multiple Workers simultaneously.
This leads to state overwrites, duplicate calls, and checkpoint conflicts.
10.1 The Concurrent Update Problem
Suppose the same Agent state is read by two Workers at the same time:
Initial iteration = 5
Worker A reads iteration = 5
Worker B reads iteration = 5
Worker A updates iteration = 6
Worker B updates iteration = 6
In the end, iteration in the database is still 6, not the expected 7.
This is the classic lost update problem.
10.2 Optimistic Locking
One solution is to use version numbers for concurrency control.
UPDATE agent_tasks
SET
state = :new_state,
version = version + 1
WHERE
task_id = :task_id
AND version = :expected_version;
If the number of affected rows is 0, the current version has already changed.
In that case, you need to re-read the latest state, and then decide whether to recompute or merge the update.
10.3 Distributed Locks and Leases
Another approach is to assign an execution lease to each task.
lease = {
"task_id": "task_1001",
"worker_id": "worker_03",
"lease_token": "token_abc",
"expires_at": "2026-11-20T10:35:00Z"
}
A Worker can execute a task only while holding a valid lease.
The lease mechanism needs to consider:
- Lease expiration time.
- Periodic renewal.
- Automatic release after a Worker crashes.
- False expiration judgments caused by network partitions.
- The issue of an old Worker continuing to write after it recovers.
Therefore, using only distributed locks is not enough. Writes to important state should ideally also use a Fencing Token or version checks to reject write operations from expired Workers.
11. Pausing and Resuming Long-Running Agent Tasks
Some tasks cannot be completed in a single continuous execution.
For example:
- Waiting for the user to confirm payment.
- Waiting for administrator approval.
- Waiting for an external system to return a processing result.
- Waiting for a scheduled task to trigger.
- Waiting for the user to provide additional information.
At this point, the Agent must not keep occupying Worker resources.
A more reasonable approach is to save the current state, pause execution, and resume once an external event arrives.
11.1 Suspended and Completed States
| State | Meaning | Recoverable |
|---|---|---|
| Pending | Waiting to execute | Yes |
| Running | Executing | Can be taken over by the recovery mechanism |
| Waiting | Waiting for external input | Yes |
| Suspended | Intentionally suspended | Yes |
| Completed | Completed | Usually not required |
| Failed | Execution failed | Depends on the recovery policy |
| Cancelled | Cancelled | Usually not allowed |
11.2 Recovery Process After Human Intervention
flowchart TD
A["Agent is executing a task"] --> B["Identify the need for human confirmation"]
B --> C["Save the current state"]
C --> D["Create a wait event"]
D --> E["Release Worker resources"]
E --> F["Wait for a user or administrator response"]
F --> G{"Is a valid response received?"}
G -->|"No"| F
G -->|"Yes"| H["Verify response permissions"]
H --> I{"Is continuation allowed?"}
I -->|"Yes"| J["Update Agent state"]
J --> K["Create recovery task"]
K --> L["Reload checkpoint"]
L --> M["Continue Agent Loop"]
I -->|"No"| N["Mark task as cancelled or terminated"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438