Back to series
· 16 min read

Agent Loop Execution and State Persistence


I. Why Agents Need Loop Execution

Traditional programs usually execute according to a predefined flow; the program knows what to do next and returns a result after execution is complete.

Agents are different. They need to dynamically decide the next action based on the current task state, model reasoning results, and tool execution feedback.

For example, an Agent that automatically analyzes data and generates reports may need to:

  1. Understand the analysis goal proposed by the user.
  2. Determine whether the database needs to be queried.
  3. Call database tools to obtain data.
  4. Analyze the query results.
  5. Discover that the data is insufficient and continue calling other tools.
  6. Integrate the analysis results.
  7. Determine whether the task goal has been met.
  8. Generate the final report.

During execution, the next operation often depends on the result of the previous step, so it cannot rely entirely on a fixed linear flow.

An Agent needs an execution mechanism that can run continuously, make dynamic decisions, and adjust behavior based on feedback—this is the Agent Loop.

1.1 Basic Concepts of the Agent Loop

The Agent Loop is the Agent's cyclic execution mechanism; its core is to let the model observe the current state in each round of execution and decide the next operation.

A typical loop contains four phases:

Phase Core Responsibility
Observe Observe the current state and execution results
Think Analyze task progress and determine the next action
Act Call tools or perform specific operations
Evaluate Check execution results and determine whether to continue

Among them, Evaluate is not necessarily a separate model call; it can also be completed jointly by rules, programs, or tool results.

1.2 Execution Flow of the Agent Loop

flowchart TD
    A["Receive user task"] --> B["Initialize Agent state"]
    B --> C["Observe current state"]
    C --> D["Model reasoning and decision-making"]
    D --> E{"Need to perform an action?"}
    E -->|"Yes"| F["Call tools or perform actions"]
    F --> G["Obtain execution results"]
    G --> H["Update Agent state"]
    H --> I{"Task completed?"}
    I -->|"No"| C
    I -->|"Yes"| J["Generate final result"]
    E -->|"No"| J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

The key to the loop is that after each execution, the Agent writes new information into the state and makes a new decision based on the updated state.

Without state updates, the Agent cannot reliably know what work it has already completed.


II. Core Execution Mechanisms of the Agent Loop

The Agent Loop is not merely repeatedly calling a large model; it is a control mechanism that includes decision-making, action execution, state updates, and termination determination.

2.1 Observe: Observing the Current State

The Observe phase is responsible for collecting the information currently available to the Agent.

It mainly includes:

  • The task initially proposed by the user.
  • Operations already completed.
  • Historical model reasoning results.
  • Tool call records.
  • Tool execution return values.
  • Current task progress.
  • Remaining execution budget.
  • Error messages and abnormal states.

For example:

state = {
    "task": "分析最近一个月的销售数据",
    "messages": [],
    "tool_results": [],
    "completed_steps": [],
    "current_step": "query_sales",
    "status": "running"
}

Observe does not mean handing all historical information directly to the model every round.

In long tasks, relevant messages, tool results, and task summaries should be selected according to the needs of the current decision, to avoid unbounded context growth.

2.2 Think: Model Reasoning and Decision-Making

The Think phase is responsible for deciding the next operation based on the current state.

The model may make the following decisions:

  • Continue calling a tool.
  • Adjust tool parameters.
  • Perform a new analysis based on the results.
  • Ask the user for additional information.
  • Determine that the task has been completed.
  • Identify errors and attempt to recover.

For example, the model might return:

{
  "action": "call_tool",
  "tool": "query_sales",
  "arguments": {
    "start_date": "2026-10-01",
    "end_date": "2026-10-31"
  },
  "reason": "需要先获取销售数据"
}

The output here is a structured decision rather than a direct database query execution.

In production environments, you should perform schema validation, tool permission checks, and parameter validation on model output; model-generated content must never be treated directly as trusted instructions.

2.3 Act: Executing Concrete Operations

The Act stage is responsible for actually executing the action selected by the model.

Common operations include:

Operation Type Example
Tool invocation Querying a database, searching the web
API request Calling a business system interface
Code execution Running Python to analyze data
File operations Reading, generating, and modifying files
State operations Updating task progress
Human interaction Requesting user confirmation

Note that the model deciding to call a tool does not mean the tool has been successfully executed.

A tool call needs to distinguish at least the following states:

  • Pending: awaiting execution.
  • Running: currently executing.
  • Succeeded: execution succeeded.
  • Failed: execution failed.
  • Timeout: execution timed out.
  • Cancelled: execution was cancelled.

Tool execution results must be fed back to the Agent before it can proceed to the next round of decision-making.

2.4 Evaluate: Assessing Execution Results

The Evaluate stage is responsible for determining whether the current result meets expectations.

For example:

  • Did the tool execute successfully?
  • Is the returned data complete?
  • Has the current step been completed?
  • Is a retry needed?
  • Does the strategy need adjustment?
  • Has the final task objective been met?

Evaluation can be carried out through a combination of mechanisms:

Evaluation Method Applicable Scenario
Rule-based judgment Checking fields, status codes, counts
Model judgment Judging whether the result meets the semantic goal
Tool verification Verifying the result through queries or tests
Human confirmation When important business operations are involved
Multi-layered verification High-risk or high-value tasks

For important tasks, you should not end the loop merely because the model says "it's done"; instead, confirm completion through verifiable business conditions.


III. Termination Conditions of the Agent Loop

Loop execution must have an explicit termination mechanism; otherwise, the Agent may call tools indefinitely, repeatedly reason, or keep attempting ineffective operations.

3.1 Common Termination Conditions

Termination Condition Description
Goal Achieved The task objective has been achieved
Max Iterations The maximum number of loops has been reached
Timeout The task execution time has been exceeded
Token Budget The model Token budget has been exceeded
Cost Budget The task cost budget has been exceeded
User Cancelled The user actively cancelled
Fatal Error An unrecoverable error occurred
Human Approval Pause pending human confirmation

Two states need to be distinguished here:

  • Completed: the task has been completed.
  • Suspended: the task is temporarily paused and may be resumed later.

Reaching the maximum number of iterations does not mean the task has succeeded; it should usually be marked as completed-with-limits or suspended, with the current progress explained to the user.

3.2 Execution Flow with Termination Judgment

flowchart TD
    A["Enter Agent Loop"] --> B["Read current state"]
    B --> C["Check task execution budget"]
    C --> D{"Is the budget exceeded?"}
    D -->|"Yes"| E["Mark task as paused or failed"]
    D -->|"No"| F["Execute model decision"]
    F --> G["Execute tool or internal operation"]
    G --> H["Update execution state"]
    H --> I{"Is the task goal met?"}
    I -->|"Yes"| J["Mark task as complete"]
    I -->|"No"| K{"Has an unrecoverable error occurred?"}
    K -->|"Yes"| E
    K -->|"No"| L{"Is human confirmation required?"}
    L -->|"Yes"| M["Pause and wait for human input"]
    L -->|"No"| B
    M --> N["Receive human decision"]
    N --> O{"Is continuation allowed?"}
    O -->|"Yes"| B
    O -->|"No"| P["End task"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style O fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style P fill:#E5F0E9,stroke:#83A995,color:#315D49

3.3 Maximum Iterations and Execution Budget

In production environments, multiple limits usually need to be set at the same time.

MAX_ITERATIONS = 20
MAX_TOOL_CALLS = 30
MAX_EXECUTION_SECONDS = 300
MAX_RETRIES = 3

def should_stop(state):
    if state["iteration"] >= MAX_ITERATIONS:
        return True

    if state["tool_call_count"] >= MAX_TOOL_CALLS:
        return True

    if state["elapsed_seconds"] >= MAX_EXECUTION_SECONDS:
        return True

    return False

In an actual implementation, execution time should be calculated by a monotonic clock or the task scheduler, rather than relying on the model to judge it on its own.

Budgets should also be configurable by task type. For example, a simple question-and-answer task may allow only a small number of model calls, whereas a complex research task allows more time and more tool calls.


IV. What Is Agent State

Agent state is the set of data that an Agent needs to maintain continuously during execution.

It includes not only conversation messages, but also task goals, execution progress, tool call results, error information, and control state.

4.1 Composition of Agent State

A relatively complete Agent state can be divided into the following categories:

State Category Typical Fields Purpose
Task State task_id, goal, status Describes the task itself
Conversation State messages Stores conversation context
Execution State iteration, current_step Records execution progress
Tool State tool_calls, tool_results Records tool execution
Memory State summary, memory_refs Stores task-related memory
Error State errors, retry_count Records errors and retries
Control State next_action, interrupt Controls the execution flow
Budget State token_usage, cost, deadline Manages the execution budget

Different Agent frameworks may design their fields differently, but the state responsibilities they need to cover are roughly the same.

4.2 Agent State Example

from typing import TypedDict, Literal, Any

class AgentState(TypedDict):
    task_id: str
    goal: str

    messages: list[dict[str, Any]]

    status: Literal[
        "pending",
        "running",
        "waiting",
        "completed",
        "failed",
        "cancelled"
    ]

    iteration: int
    current_step: str

    tool_calls: list[dict[str, Any]]
    tool_results: list[dict[str, Any]]

    errors: list[dict[str, Any]]
    retry_count: int

    summary: str

    token_usage: int
    cost: float

The state here describes the execution context of an Agent task.

It is important to note that task state and model context are not the same concept:

  • Agent state is the complete execution data.
  • Model context is the information selectively provided each time the model is requested.
  • Persisted state is data that needs to be retained across processes and over time.

The three are related, but should not be simply equated.

4.3 Changes in State During the Loop

Suppose the Agent needs to query sales data and generate a report:

Execution Stage iteration current_step status
Initialization 0 planning pending
First reasoning 1 query_sales running
Query complete 1 analyze_data running
Data analysis 2 generate_report running
Report generation 3 validation running
Validation passed 4 completed completed

Each execution round may change multiple state fields.

Therefore, state management must not only save the final result but also record how the task changes from start to finish.


5. Why Agent State Must Be Persisted

If the Agent keeps state only in memory, task state may be lost after the program restarts, the server crashes, or the execution environment is destroyed.

For short-running, low-complexity tasks, in-memory state is usually sufficient.

However, for tasks that need to run for minutes or even hours, relying solely on memory introduces significant risks.

5.1 What Happens Without Persistence

Suppose the Agent is executing a complex data analysis task:

  1. It has already queried multiple data sources.
  2. It has already completed data cleaning.
  3. It has already generated intermediate analysis results.
  4. It is about to call a tool to generate a report.
  5. At this point, the server suddenly restarts.

If all state is stored in process memory, after the restart the Agent cannot accurately know:

  • What the original task was.
  • Which tools have already been executed.
  • Which operations have already succeeded.
  • Which steps have not yet been executed.
  • Where it should continue from now.

An Agent may be forced to start over from the beginning, causing repeated computation, repeated calls, or even repeated business operations.

5.2 The Core Value of State Persistence

Value Description
Failure recovery Restore tasks after process restart
Long-running task execution Support continuous operation over time
Task suspension Support resuming execution after human intervention
Execution tracking Query historical states and execution records
Concurrency management Support collaboration among multiple Workers
Audit analysis Reconstruct the task execution process
Time travel Return to historical states for inspection or recovery

State persistence addresses the problem of preserving the Agent's execution context.

It does not automatically guarantee that external tool calls will not be duplicated, nor does it mean the entire system has transactional consistency.


6. Checkpoint: The Agent's Checkpoint Mechanism

Checkpoint is one of the most important mechanisms in Agent state persistence.

It means saving the current state as a recoverable checkpoint at a certain point in execution.

When a task reaches a specific stage, the system writes the state to persistent storage.

If a failure occurs later, execution can be recovered from the most recent valid checkpoint.

6.1 Basic Structure of a Checkpoint

A checkpoint typically needs to contain:

Field Purpose
checkpoint_id Unique checkpoint identifier
task_id Task it belongs to
thread_id Execution session identifier
sequence Checkpoint sequence number
parent_checkpoint_id Previous checkpoint
state Current state snapshot
metadata Metadata such as version and time
created_at Creation time

Example:

{
  "checkpoint_id": "cp_0003",
  "task_id": "task_1001",
  "thread_id": "thread_abc",
  "sequence": 3,
  "parent_checkpoint_id": "cp_0002",
  "state": {
    "status": "running",
    "iteration": 3,
    "current_step": "generate_report",
    "retry_count": 0
  },
  "metadata": {
    "schema_version": 1
  },
  "created_at": "2026-11-20T10:30:00Z"
}

Here, state is a snapshot of the state at a certain moment, not simple log text.

6.2 When Checkpoints Are Saved

It is not necessary to create a checkpoint after every statement executed.

Saving too frequently increases storage pressure and write costs, while saving too sparsely increases the recovery workload after a failure.

Common times to save include:

  • After a round of Agent reasoning is completed.
  • After a tool call is completed.
  • After an important task phase is completed.
  • When the Agent is about to enter a waiting state for human approval.
  • When a key change occurs in the task state.
  • When the task ends normally.

For tool calls with external side effects, special attention must be paid to the ordering relationship between checkpoints and tool execution.

6.3 Checkpoint Saving Process

flowchart TD
    A["Agent starts execution"] --> B["Read current state"]
    B --> C["Perform model inference"]
    C --> D["Execute tool or task node"]
    D --> E["Generate new state"]
    E --> F{"Checkpoint condition met?"}
    F -->|"No"| B
    F -->|"Yes"| G["Generate state snapshot"]
    G --> H["Write to persistent storage"]
    H --> I{"Was the save successful?"}
    I -->|"No"| J["Handle persistence exception"]
    J --> K{"Allow retry?"}
    K -->|"Yes"| H
    K -->|"No"| L["Pause or mark as failed"]
    I -->|"Yes"| M["Record checkpoint version"]
    M --> B

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style M fill:#E5F0E9,stroke:#83A995,color:#315D49

6.4 Snapshots and Incremental Records

Checkpoints can use different save strategies.

Full Snapshot

Save the complete state each time.

Advantages:

  • The recovery process is simple.
  • There is no need to replay a large number of historical events.
  • Reading the current state is relatively direct.

Disadvantages:

  • When the state is large, storage overhead is significant.
  • Frequent saves may produce a large amount of duplicate data.

Delta

Save only the parts that changed relative to the previous version.

Advantages:

  • Reduce duplicate storage.
  • Suitable for tasks with small state changes and long histories.

Disadvantages:

  • Recovery requires merging multiple versions.
  • It is necessary to handle delta chain corruption and version compatibility.
  • Querying historical state is more complex.

In engineering practice, a strategy that combines periodic full snapshots with intermediate delta records can be used.


VII. Storage Architecture for Agent State Persistence

Agent state storage does not necessarily need to use a single database.

Different types of data have different access patterns and lifecycles, so storage media can be chosen based on data characteristics.

7.1 Common Storage Options

Storage Type Use Cases Characteristics
Memory Local development, temporary tasks Simple, but cannot recover across processes
SQLite Standalone applications, small-scale tasks Simple deployment, supports transactions
PostgreSQL Production task state Good transaction, indexing, and concurrency capabilities
Redis Cache, short-term state, locks Fast access; pay attention to persistence configuration
Object Storage Large state snapshots, files Suitable for storing large objects
Event Store Event history and auditing Convenient for tracking state changes

Real projects often use a combined architecture:

  • PostgreSQL stores task metadata and checkpoint indexes.
  • Redis stores short-term execution state and distributed locks.
  • Object storage stores large files and snapshot attachments.
  • Logs or event stores record the complete execution process.

7.2 Separate State Data from Business Data

Do not cram all business data directly into Agent State.

For example, an Agent is processing a 500 MB file.

If the complete file content is copied into the state snapshot every time a checkpoint is created, it will cause serious storage and performance problems.

A more reasonable approach is:

```python state = { "task_id": "task_1001", "status": "running", "current_step": "analyze_file", "file_ref": { "object_key": "uploads/report_001.pdf", "version": "v3" }, "summary": "正在分析销售报告", } ```

The agent state stores file references, versions, and necessary metadata, while the files themselves live in object storage.

This design reduces checkpoint size and allows large files to have their own independent lifecycle management policies.

7.3 Recommended Overall Architecture

flowchart TD
    A["User Request"] --> B["Agent Runtime"]
    B --> C["Agent Loop"]
    C --> D["State Manager"]
    D --> E["Checkpoint Manager"]
    E --> F["Persistent Storage"]
    D --> G["Context Builder"]
    G --> C
    C --> H["Tool Execution Layer"]
    H --> I["External API / Database"]
    I --> J["Execution Result"]
    J --> D
    F --> K["Recovery Service"]
    K --> D
    D --> L["Task Status Query API"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style L fill:#E5F0E9,stroke:#83A995,color:#315D49

Here it is important to distinguish between the Agent Runtime and persistent storage:

  • The Runtime is responsible for executing the current task.
  • The State Manager is responsible for maintaining runtime state.
  • The Checkpoint Manager is responsible for generating and managing checkpoints.
  • Persistent storage is responsible for reliably saving state.
  • The Recovery Service is responsible for reloading historical state into the runtime environment.

8. Agent Failure Recovery Mechanisms

The real value of persistence lies in being able to recover tasks after the system encounters an exception.

But recovery is not simply a matter of reading the latest checkpoint and then unconditionally re-executing.

Especially when external tool calls are involved, it is necessary to determine whether the operation actually succeeded.

8.1 Common Failure Scenarios

Failure Scenario Possible Impact Handling
Agent process crash In-memory state lost Restore from checkpoint
Model request timeout Inference result unknown Determine using the request identifier
Tool call timeout External operation may have already executed Query the execution result or retry idempotently
Database temporarily unavailable Checkpoint cannot be saved Retry, degrade, or pause
Worker terminated Task interrupted Reassigned by the scheduler
State version change Old state cannot be read directly Schema migration
Concurrent recovery Same task executed multiple times Distributed lock or lease mechanism

8.2 Recovering from the Latest Checkpoint

A typical recovery process:

  1. Query the current state of the task.
  2. Obtain the most recent valid checkpoint.
  3. Verify the integrity of the checkpoint data.
  4. Check the state schema version.
  5. Load the state snapshot.
  6. Check for tool calls that have not yet completed.
  7. Resume the Agent Loop.
  8. Continue executing the subsequent steps.
flowchart TD
    A["Task interruption detected"] --> B["Query task records"]
    B --> C["Read the most recent valid checkpoint"]
    C --> D{"Does the checkpoint exist?"}
    D -->|"No"| E["Try to rebuild from the task's initial state"]
    D -->|"Yes"| F["Verify snapshot integrity"]
    F --> G{"Is the data valid?"}
    G -->|"No"| H["Find an earlier valid checkpoint"]
    H --> C
    G -->|"Yes"| I["Check the state version"]
    I --> J["Run Schema compatibility conversion"]
    J --> K["Check for unfinished tool calls"]
    K --> L["Restore Agent state"]
    E --> L
    L --> M["Continue Agent Loop"]

    style A fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style L fill:#E5F0E9,stroke:#83A995,color:#315D49
    style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F

8.3 Why Tool Calls Cannot Simply Be Repeated During Recovery

Suppose the Agent performs the following operation:

result = payment_tool(
    order_id="order_1001",
    amount=100
)

The payment service has already successfully charged the customer, but the Agent crashes before saving a checkpoint.

After restarting, the Agent may find no record of a successful payment in the current state, and therefore call the payment tool again.

If the payment API is not idempotent, this can result in a duplicate charge.

This shows that:

State persistence can only save the execution state known to the Agent; it cannot inherently guarantee consistency of operations between the Agent and external systems.

Therefore, it must be combined with idempotency keys, operation records, and external result queries.


9. Idempotency and Consistency of Tool Calls

The Agent's loop execution is non-deterministic, while external tools often have real business side effects.

As a result, tool calls are the part of state persistence design that is most prone to problems.

9.1 What Is Idempotency

Idempotency means that performing the same operation once or multiple times produces the same final business effect.

For example:

create_payment(
    order_id="order_1001",
    idempotency_key="task_1001_payment_01"
)

Even if the Agent resends the request due to a timeout or crash, the payment service should use the idempotency key to recognize this as the same business operation and avoid a duplicate charge.

Note that idempotency does not mean every request returns an identical response; rather, it requires that business side effects are not executed more than once.

9.2 Tool Call Records

It is recommended to create a separate record for each tool call:

tool_call = {
    "call_id": "call_001",
    "task_id": "task_1001",
    "tool_name": "create_payment",
    "idempotency_key": "task_1001_payment_01",
    "status": "succeeded",
    "request": {
        "order_id": "order_1001",
        "amount": 100
    },
    "result": {
        "payment_id": "payment_001"
    }
}

A tool call record should contain at least:

  • A unique call ID.
  • The task ID it belongs to.
  • The tool name.
  • The idempotency key.
  • A summary of the request parameters.
  • The execution status.
  • The returned result or error.
  • The start and end times.

For sensitive data, request parameters and results should be masked or encrypted.

9.3 The Tool Call State Machine

Tool calls can maintain their own state machine independently:

flowchart TD
    A["Create tool call record"] --> B["Pending"]
    B --> C["Running"]
    C --> D{"Execution result"}
    D -->|"Success"| E["Succeeded"]
    D -->|"Failure"| F["Failed"]
    D -->|"Timeout"| G["Unknown"]
    G --> H["Query external execution result"]
    H --> I{"Already executed"}
    I -->|"Yes"| E
    I -->|"No"| J["Check whether retry is allowed"]
    J -->|"Allowed"| B
    J -->|"Not allowed"| F

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#E5F0E9,stroke:#83A995,color:#315D49
    style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Among them, Unknown is a very important state.

A tool call timing out does not necessarily mean execution failed; it may also mean:

  • The request has not yet reached the external service.
  • The external service has already executed it, but the response was lost.
  • The external service is still processing it.
  • The external service failed to execute it, but the error response never came back.

Therefore, when you encounter Unknown, you should query the external execution result first, instead of retrying directly.

9.4 Why Exactly Once Cannot Be Promised Lightly

There are three common execution semantics in distributed systems:

Semantics Meaning Risk
At-most-once Executed at most once Operations may be lost
At-least-once Executed at least once Operations may be executed repeatedly
Exactly-once The business effect occurs exactly once Requires additional coordination mechanisms

In scenarios where an Agent interacts with external APIs, it is usually impossible to achieve true Exactly Once relying solely on local checkpoints.

A more practical engineering approach is:

  1. Use persistent call records.
  2. Generate a stable idempotency key for each business operation.
  3. Use idempotent interfaces supported by the external system.
  4. Query the execution result for operations that timed out.
  5. Send operations that cannot be confirmed to manual review.
  6. Reconcile regularly to fix inconsistent states.

The ultimate goal is controllable idempotency in terms of business effects, not simply guaranteeing that a network request is sent only once.


10. Concurrent Execution and State Consistency

When an Agent system has only one Worker, state updates are relatively simple.

But in production environments there are usually multiple Workers processing different tasks at the same time, and abnormal task scheduling may even cause the same task to be executed by multiple Workers simultaneously.

This leads to state overwrites, duplicate calls, and checkpoint conflicts.

10.1 The Concurrent Update Problem

Suppose the same Agent state is read by two Workers at the same time:

Initial iteration = 5

Worker A reads iteration = 5
Worker B reads iteration = 5

Worker A updates iteration = 6
Worker B updates iteration = 6

In the end, iteration in the database is still 6, not the expected 7.

This is the classic lost update problem.

10.2 Optimistic Locking

One solution is to use version numbers for concurrency control.

UPDATE agent_tasks
SET
    state = :new_state,
    version = version + 1
WHERE
    task_id = :task_id
    AND version = :expected_version;

If the number of affected rows is 0, the current version has already changed.

In that case, you need to re-read the latest state, and then decide whether to recompute or merge the update.

10.3 Distributed Locks and Leases

Another approach is to assign an execution lease to each task.

lease = {
    "task_id": "task_1001",
    "worker_id": "worker_03",
    "lease_token": "token_abc",
    "expires_at": "2026-11-20T10:35:00Z"
}

A Worker can execute a task only while holding a valid lease.

The lease mechanism needs to consider:

  • Lease expiration time.
  • Periodic renewal.
  • Automatic release after a Worker crashes.
  • False expiration judgments caused by network partitions.
  • The issue of an old Worker continuing to write after it recovers.

Therefore, using only distributed locks is not enough. Writes to important state should ideally also use a Fencing Token or version checks to reject write operations from expired Workers.


11. Pausing and Resuming Long-Running Agent Tasks

Some tasks cannot be completed in a single continuous execution.

For example:

  • Waiting for the user to confirm payment.
  • Waiting for administrator approval.
  • Waiting for an external system to return a processing result.
  • Waiting for a scheduled task to trigger.
  • Waiting for the user to provide additional information.

At this point, the Agent must not keep occupying Worker resources.

A more reasonable approach is to save the current state, pause execution, and resume once an external event arrives.

11.1 Suspended and Completed States

State Meaning Recoverable
Pending Waiting to execute Yes
Running Executing Can be taken over by the recovery mechanism
Waiting Waiting for external input Yes
Suspended Intentionally suspended Yes
Completed Completed Usually not required
Failed Execution failed Depends on the recovery policy
Cancelled Cancelled Usually not allowed

11.2 Recovery Process After Human Intervention

flowchart TD
    A["Agent is executing a task"] --> B["Identify the need for human confirmation"]
    B --> C["Save the current state"]
    C --> D["Create a wait event"]
    D --> E["Release Worker resources"]
    E --> F["Wait for a user or administrator response"]
    F --> G{"Is a valid response received?"}
    G -->|"No"| F
    G -->|"Yes"| H["Verify response permissions"]
    H --> I{"Is continuation allowed?"}
    I -->|"Yes"| J["Update Agent state"]
    J --> K["Create recovery task"]
    K --> L["Reload checkpoint"]
    L --> M["Continue Agent Loop"]
    I -->|"No"| N["Mark task as cancelled or terminated"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style M fill:#E5F0E9,stroke:#83A995,color:#315D49
    style N fill:#F7EAE7,stroke:#C98D7C,color:#754438