Back to series
· 15 min read

Computer Use: How Do Agents Operate Computers?


I. What Is Computer Use?

Computer Use (the ability to use a computer) refers to an AI agent interacting with computer systems through a graphical user interface (GUI), observing the screen, identifying controls, moving the mouse, clicking buttons, typing text, scrolling pages, and operating applications just like a human does.

Traditional AI agents interact with external systems mainly through APIs, tool calls, and code execution, whereas Computer Use extends the scope of interaction to desktop applications, browsers, office software, and other systems that do not expose APIs.

For example, a user might say:

Open the browser, search for a product, compare prices across different websites, and compile the results into an Excel file.

An agent with Computer Use capability needs to complete the following:

  1. Open the browser.
  2. Go to a search website.
  3. Enter the keywords and run the search.
  4. Observe the search results.
  5. Open the relevant product pages.
  6. Extract the product names and prices.
  7. Open Excel.
  8. Create a spreadsheet and fill in the data.
  9. Save the file and report the results.

The whole process does not necessarily rely on the target application providing a dedicated API; instead, it is carried out by simulating how a human operates the graphical interface.

1. Differences Between Computer Use and Traditional Tool Calling

Comparison dimension Traditional tool calling Computer Use
Interaction method APIs, function calls Mouse, keyboard, screen
Target of operation Well-defined interfaces Graphical user interface
Input form Structured parameters Screenshots, interface state
Output form JSON, text, structured data Screen changes, operation results
Environment dependency Interfaces and permissions Operating system, windows, and interface
Stability Usually higher Affected by interface changes
Applicable scope Systems with APIs Systems with an operable interface
Error handling API error codes Visual feedback and state judgment

The core of traditional tool calling is invoking interfaces that have already been defined, whereas the core of Computer Use is understanding the current interface and deciding what action to perform next.

2. Differences Between Computer Use and RPA

Both Computer Use and RPA (Robotic Process Automation) can operate a computer through the graphical interface, but the two work in different ways.

Comparison dimension Traditional RPA AI Computer Use
Execution logic Predefined process Dynamic planning based on goals
Operation path Fixed steps Can be adjusted according to the interface
Page recognition Coordinates, control selectors Visual understanding, control recognition
Exception handling Rules and preset branches Re-decides based on the new state
Adaptability Depends on process configuration Can handle some unknown situations
Suitable tasks High-frequency, fixed, repetitive tasks Multi-step, dynamic, unstructured tasks

For example, traditional RPA usually requires defining in advance:

  • Which button to click.
  • Which input field to type into.
  • How long to wait.
  • Which page to navigate to.
  • Which branch to execute when an exception occurs.

AI Computer Use, by contrast, can determine from the screen content what page it is currently on, and then decide where to click.

Note that AI Computer Use does not mean completely replacing RPA. For stable, repetitive processes with clear rules, traditional RPA is often more efficient. The two can also be used in combination.

II. The Overall Architecture of Computer Use

Computer Use is usually not simply a matter of having a large model output mouse coordinates; rather, it is made up of multiple parts working together, including perception, decision-making, action execution, environment feedback, and state management.

1. Core Architecture

flowchart TD
    A["User Task Goal"] --> B["Agent Controller"]
    B --> C["Task Planning and Step Decomposition"]
    C --> D["Computer Use Agent"]
    D --> E["Screen Observation and Visual Understanding"]
    E --> F["UI Element Recognition"]
    F --> G["Action Decision"]
    G --> H["Action Executor"]
    H --> I["Operating System and Applications"]
    I --> J["UI State Changes"]
    J --> K["Re-screenshot and Observe"]
    K --> L{"Task Completed?"}
    L -->|"No"| E
    L -->|"Yes"| M["Result Verification and Task Delivery"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The entire system can be divided into six core modules.

Module Primary Responsibilities
Agent Controller Manage task lifecycle and execution state
Task Planner Break complex tasks into executable steps
Perception Module Capture screen information and understand the interface
Decision Module Determine the next mouse and keyboard operations
Action Executor Convert actions into operating system input
State Verification Module Check action results and determine whether to continue

Among them, the three most critical stages are perception, decision-making, and execution.

2. Perception, Decision-Making, and Execution

The basic working mode of Computer Use can be summarized as:

  • Perception: What is currently displayed on the screen?
  • Decision-making: What should be done next to complete the task?
  • Execution: How can the decision be converted into mouse and keyboard operations?
  • Verification: Did the operation actually produce the expected result?

This is a closed-loop interactive system, rather than a one-off generation of a series of mouse coordinates.

flowchart TD
    A["Capture current screen"] --> B["Visual understanding"]
    B --> C["Recognize target control"]
    C --> D["Reason with task goal"]
    D --> E["Generate action instruction"]
    E --> F["Execute mouse or keyboard actions"]
    F --> G["Observe operation result"]
    G --> H{"Does the result meet expectations?"}
    H -->|"Yes"| I["Update task status"]
    H -->|"No"| J["Analyze failure cause"]
    J --> K["Adjust operation strategy"]
    K --> A
    I --> L{"Is the task complete?"}
    L -->|"No"| A
    L -->|"Yes"| M["Output final result"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F

3. Screen Perception: How Do Agents Understand Computer Interfaces?

The first core capability of Computer Use is screen perception.

Humans can directly see windows, buttons, text boxes, and menus, and understand their purposes. Large models, however, need to convert screen information into input that can be reasoned about.

1. Screenshots

The most basic method is to capture an image of the current desktop or application window via a screenshot.

For example:

import pyautogui

# Capture the current screen screenshot
screenshot = pyautogui.screenshot()

# Save the screenshot
screenshot.save("screen.png")

The screenshot can be used as input to a vision model, allowing the model to analyze the current interface.

However, a screenshot not only contains the target control, but may also contain:

  • Multiple application windows.
  • Browser tabs.
  • Pop-ups and notification messages.
  • Desktop icons.
  • Irrelevant background content.
  • Mouse pointer and scrollbars.

Therefore, the agent needs to identify the parts relevant to the current task from a large amount of visual information.

2. Visual Understanding

Visual understanding is the process of converting screen images into structured semantic information.

For example, the current screen shows a login page:

Login Page

Username: [________________]

Password:  [________________]

        [Login]

Forgot password?

The vision model needs to recognize:

UI Element Recognition Result Purpose
Username input field Input Enter account
Password input field Input Enter password
Login button Button Submit login
Forgot password Link Recover password

The model not only needs to recognize the positions of elements, but also needs to understand their semantics and purposes.

For example:

  • Which input field is used to enter the username?
  • Which button is used to submit the form?
  • Is the current page already logged in?
  • Is an error message displayed?
  • Is there a pop-up that needs to be handled?

This requires the vision model to have certain OCR, object detection, layout understanding, and semantic reasoning capabilities.

3. OCR and Visual Models

Computer Use can use different technologies to obtain interface information.

Technology Main role Characteristics
OCR Extract on-screen text Good for recognizing text and labels
Object detection Locate buttons and controls Can identify element regions
Image understanding models Understand the overall meaning of a page Suited to complex interfaces
UI Automation Retrieve native control information Can read control properties
DOM parsing Retrieve web page structure Suited to browser environments

OCR can recognize text, but it cannot necessarily understand the function that the text corresponds to.

For example, there are two buttons on the screen:

  • Confirm
  • Cancel

OCR can recognize the text on both buttons, but visual understanding is still needed to determine which button should be clicked.

In engineering practice, you should not hand every task over to a vision model. For browser pages, prefer using the DOM and the accessibility tree to obtain structured information; for desktop applications where structured information is unavailable, then use visual understanding.

4. Coordinate Systems and Element Localization

Computer Use frequently needs to convert recognition results into screen coordinates.

Suppose the model recognizes a login button whose position is:

{
  "element": "登录按钮",
  "type": "button",
  "center": {
    "x": 640,
    "y": 480
  }
}

Here, x and y represent the position of the button's center point on the screen.

If the screen resolution is 1920 × 1080, the coordinate origin is usually located at the top-left corner:

  • The X axis increases to the right.
  • The Y axis increases downward.
  • The top-left corner of the screen is (0, 0).

Coordinate-based localization requires attention to several issues:

  1. The screenshot resolution may differ from the actual screen resolution.
  2. System scaling may affect coordinate mapping.
  3. Multi-monitor environments may have different coordinate origins.
  4. Moving or resizing a window may change the positions of controls.
  5. Page scrolling may change the screen coordinates of the target element.

Therefore, coordinates should not be treated as a long-term valid identifier for a control.

A more reliable approach is to re-observe the interface before every critical operation and determine the operation position based on the current screenshot.

IV. Action Execution: How Does the Agent Control the Mouse and Keyboard?

After completing screen perception, the agent needs to convert decisions into concrete computer input.

Common operations include:

Action Role
click Click the mouse
double_click Double-click the mouse
move Move the mouse
drag Drag the mouse
scroll Scroll the page
key_press Press a keyboard key
type_text Type text
hotkey Execute a key combination
screenshot Capture the current screen

1. Mouse Operations

Take Python's PyAutoGUI as an example:

import pyautogui

# Move the mouse
pyautogui.moveTo(640, 480, duration=0.3)

# Click the mouse
pyautogui.click(640, 480)

# Double-click
pyautogui.doubleClick(640, 480)

# Right-click
pyautogui.rightClick(640, 480)

# Drag
pyautogui.moveTo(300, 300)
pyautogui.dragTo(600, 500, duration=0.5)

In essence, these operations convert coordinates and actions into input events that the operating system can recognize.

2. Keyboard Operations

Keyboard input can be divided into ordinary text input, key presses, and key combinations.

import pyautogui

# Type text
pyautogui.write("Hello Computer Use", interval=0.05)

# Press Enter
pyautogui.press("enter")

# Copy using a keyboard shortcut
pyautogui.hotkey("ctrl", "c")

# Paste using a keyboard shortcut
pyautogui.hotkey("ctrl", "v")

# Select all
pyautogui.hotkey("ctrl", "a")

Note that write() is mainly suitable for inputting ordinary characters. Chinese input, special characters, input method switching, and keyboard layouts on different operating systems may require additional handling.

3. Action Abstraction Layer

In actual engineering, you should not let a large model directly generate arbitrary Python code and execute it.

A more reasonable design is to provide a set of constrained action interfaces.

For example:

from typing import Literal
from pydantic import BaseModel, Field


class ComputerAction(BaseModel):
    action: Literal[
        "click",
        "double_click",
        "move",
        "scroll",
        "type",
        "press",
        "hotkey",
        "screenshot"
    ]

    x: int | None = None
    y: int | None = None

    text: str | None = None
    key: str | None = None

    direction: Literal[
        "up",
        "down",
        "left",
        "right"
    ] | None = None

    amount: int = Field(default=0, ge=0, le=1000)

The model can output structured actions:

{
  "action": "click",
  "x": 640,
  "y": 480
}

The executor is responsible for validating parameters and executing them:

import pyautogui


def execute_action(action: ComputerAction):
    if action.action == "click":
        if action.x is None or action.y is None:
            raise ValueError("点击操作缺少坐标")

        pyautogui.click(action.x, action.y)

    elif action.action == "type":
        if action.text is None:
            raise ValueError("输入操作缺少文本")

        pyautogui.write(action.text)

    elif action.action == "press":
        if action.key is None:
            raise ValueError("按键操作缺少按键")

        pyautogui.press(action.key)

    elif action.action == "screenshot":
        return pyautogui.screenshot()

    else:
        raise ValueError("暂不支持该动作")

It should be emphasized here: the example only demonstrates how to abstract actions; a production system also needs coordinate boundary validation, permission checks, timeout control, operation logs, and exception recovery.

V. The Core Execution Loop of Computer Use

The most important mechanism of Computer Use is not a single click action, but the loop of continuously observing the environment, executing actions, and updating state.

1. The Observe-Think-Act Loop

The core mechanism can be abstracted into three stages:

  • Observe: Observe the current screen.
  • Think: Analyze the goal and current state, and decide the next action.
  • Act: Execute the action and obtain feedback.
flowchart TD
    A["Observe: Observe the screen"] --> B["Identify the current interface state"]
    B --> C["Think: Analyze the task goal"]
    C --> D["Choose the next action"]
    D --> E["Act: Perform mouse or keyboard operations"]
    E --> F["Obtain the new screen state"]
    F --> G{"Is the goal achieved?"}
    G -->|"No"| A
    G -->|"Yes"| H["Task completed"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

For example, a user asks the agent to search for a technical article in a browser.

On the first observation, the browser displays the search home page.

The agent decides to click the search box and enter keywords.

After execution, the browser displays the search results.

The agent observes the page again, identifies the search results list, and then selects the target article.

Each action changes the environment, so the next decision should be based on the latest interface state rather than always relying on the initial screenshot.

2. Why Can't All Actions Be Generated at Once?

Suppose the agent plans to execute:

Open browser
Click address bar
Enter URL
Press Enter
Wait for the page to load
Click login
Enter account
Enter password
Click submit

If all actions are generated at once and executed back-to-back, many problems arise:

  • The browser may not open successfully.
  • The address bar may not receive focus.
  • The page may fail to load.
  • The login button may have moved.
  • A CAPTCHA may appear on the login page.
  • An error message may appear after submission.

If an earlier action fails, all subsequent actions may become invalid.

Therefore, a more reliable way to execute is to split the task into multiple stages, with necessary state verification at each stage.

3. Action Execution Granularity

Not every action needs its own separate call to the large model.

For example:

Click the search box
Enter keywords
Press Enter

These three actions are clearly related and can be executed as one short batch of operations.

However, which result to open after the search completes must be judged anew based on the page content.

In engineering practice, you can adopt:

  • Deterministic operations: batch execution is allowed.
  • State-dependent operations: re-observe after execution.
  • High-risk operations: must be executed individually and verified.
  • Irreversible operations: require additional authorization or human confirmation.

This balances efficiency and reliability.

VI. Computer Use and Multimodal Large Models

Computer Use generally requires the participation of a multimodal large model, because the system must both understand screen images and generate operation commands based on natural-language goals.

1. Multimodal Input

Unlike ordinary text tasks, the input to Computer Use typically includes:

  • The user's task description.
  • The current screen screenshot.
  • The history of operations.
  • The current task state.
  • The list of available actions.
  • Execution environment information.

For example:

{
  "task": "Open the browser and search for a Python tutorial",
  "screen": {
    "width": 1920,
    "height": 1080,
    "image": "current screen screenshot"
  },
  "history": [
    "Launch the browser"
  ],
  "available_actions": [
    "click",
    "type",
    "press",
    "scroll",
    "screenshot"
  ]
}

The model needs to combine this information to determine the next action.

2. Multimodal Output

Model output is not necessarily just natural language; it can also be a structured action.

For example:

{
  "reasoning_summary": "The search page is already open; keywords need to be entered",
  "action": {
    "type": "click",
    "coordinate": {
      "x": 720,
      "y": 320
    }
  }
}

The executor then sends the action to the operating system.

Here reasoning_summary is only used to describe a presentable basis for the decision; it should not require the model to expose its complete internal reasoning process.

3. Coordinate Prediction and Visual Grounding

The model can represent the target location in different ways.

Absolute coordinates

{
  "x": 640,
  "y": 480
}

This directly expresses a position on the screen.

The advantage is simplicity; the drawback is that it is easily affected by screen resolution, window position, and scaling factor.

Relative coordinates

{
  "x_ratio": 0.5,
  "y_ratio": 0.45
}

This expresses coordinates as proportions of the screen width and height.

Conversion formulas:

x = x_{ratio} \times W
y = y_{ratio} \times H

Where:

  • W: screen width.
  • H: screen height.

Relative coordinates can reduce the impact of resolution changes, but they cannot solve the problems of layout changes and element movement.

Bounding box grounding

{
  "element": "search button",
  "bbox": [600, 430, 700, 480]
}

Where the bounding box is expressed as:

bbox = [x_1,y_1,x_2,y_2]

The executor can choose the center point of the bounding box to click:

x_c = \frac{x_1+x_2}{2}
y_c = \frac{y_1+y_2}{2}

A bounding box can provide more spatial information than a single-point coordinate.

4. Limitations of Multimodal Models

Vision models are not absolutely reliable.

Common problems include:

  • Small-font recognition errors.
  • Icon semantic understanding errors.
  • Confusion between similar buttons.
  • Target loss due to screen occlusion.
  • Unstable screenshot states caused by page animations.
  • Inaccurate understanding of complex tables and dense layouts.
  • Mistaking visually adjacent elements for having the same function.

Therefore, production systems should combine visual recognition, DOM, accessibility tree, and operation result verification as much as possible, rather than relying entirely on the model guessing coordinates.

7. Browser Automation and Desktop Automation

Computer Use can be applied to different computer operation environments, but there are obvious differences in the technical implementation of browsers and desktop applications.

1. Browser Automation

Browsers usually have richer structured information.

For example:

  • DOM nodes.
  • Element attributes.
  • CSS selectors.
  • Accessibility Tree.
  • Page URL.
  • Network requests and loading states.

Therefore, web page elements can be obtained directly through tools such as Playwright.

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)

    page = browser.new_page()

    page.goto("https://example.com")

    print(page.title())

    page.screenshot(path="page.png")

    browser.close()

If you want to operate the search box:

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)

    page = browser.new_page()

    page.goto("https://example.com")

    search_box = page.locator("input[name='q']")

    search_box.fill("Python 教程")

    search_box.press("Enter")

    page.wait_for_load_state("domcontentloaded")

    browser.close()

Here DOM positioning is used, rather than relying on fixed screen coordinates.

This approach is usually more stable, and it is also easier to determine whether the operation succeeded.

2. Desktop Automation

Desktop applications are more complicated.

For example:

  • Excel.
  • Word.
  • Photoshop.
  • Local file manager.
  • Enterprise internal desktop clients.
  • Legacy business systems without open APIs.

These software programs may not have a directly accessible web DOM and require:

  • Windows UI Automation.
  • macOS Accessibility API.
  • Linux AT-SPI.
  • Image recognition and OCR.
  • Mouse and keyboard simulation tools.

For desktop software, Computer Use can identify controls through screenshots and perform mouse and keyboard operations.

3. How to Choose Between the Two Approaches?

Task Scenario Recommended Approach
Open web pages and navigate Browser automation
Fill out web forms DOM or accessibility tree
Click complex controls in web pages DOM first, visual assistance
Operate local Excel Office API or desktop automation
Operate traditional desktop software UI Automation or visual operations
Operate software without open interfaces Computer Use
Complex cross-application workflows Combination of API, browser automation, and Computer Use

An important engineering principle is:

Tasks that can be completed through stable APIs do not need to be forced to use Computer Use; only when interface interaction truly adds value should visual operations be used.

8. Typical Application Scenarios of Computer Use

The advantage of Computer Use lies in cross-application operations and handling tasks that lack standard APIs.

1. Browser Information Collection

User request:

Open the browser, search for recently published AI technology articles, and organize the titles and links.

The agent can:

  1. Open the browser.
  2. Go to the search website.
  3. Enter keywords.
  4. Obtain search results.
  5. Identify article titles.
  6. Open the target pages.
  7. Organize the information.
  8. Output the results.

For information that can be obtained through a search API or web page DOM, prioritize structured interfaces; Computer Use can be used for supplementary steps that require real interface interaction.

2. Office Software Automation

For example:

Organize the sales data in the download directory into an Excel report and generate a bar chart.

The agent can:

  • Open the file manager.
  • Locate the data file.
  • Open Excel.
  • Create a worksheet.
  • Enter the data.
  • Format the spreadsheet.
  • Create a chart.
  • Save the file.

However, for file reading, data processing, and chart generation, using Python, Excel API, or Office automation interfaces directly is usually more efficient. Computer Use is better suited to supplement operations that must be completed through a graphical interface.

3. Cross-Application Tasks

For example:

Retrieve customer information from the browser, organize it in Excel, and then send it to the specified contact via email.

This is a typical cross-application task.

flowchart TD
    A["Receive user task"] --> B["Open browser"]
    B --> C["Get customer information"]
    C --> D["Extract and organize data"]
    D --> E["Open Excel"]
    E --> F["Create data table"]
    F --> G["Save file"]
    G --> H["Open email client"]
    H --> I["Create email"]
    I --> J["Add attachment"]
    J --> K{"Is sending authorized?"}
    K -->|"No"| L["Wait for human confirmation"]
    L --> M{"Approved?"}
    M -->|"No"| N["Cancel sending"]
    M -->|"Yes"| O["Send email"]
    K -->|"Yes"| O
    O --> P["Verify sending result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style P fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Note that operations such as sending emails, submitting orders, and modifying business data may have real consequences. Agents should verify recipients, attachments, and content before execution, and require human approval based on the risk level.

IX. Exception Handling and Failure Recovery in Computer Use

Graphical interface operations are subject to significant uncertainty; one cannot assume that every action will execute successfully.

Production-grade Computer Use must have exception detection, action retry, state recovery, and task termination mechanisms.

1. Common Exceptions

Exception Type Possible Cause Handling Method
Target element not found Page layout changed Take a new screenshot and identify
Click has no effect Coordinate offset or control not loaded Relocate and verify
Page load timeout Network or service exception Wait, retry, or terminate
Input failed Input field did not receive focus Click again and check
Popup obstruction New window or prompt appears Identify the popup and decide
Incorrect page state Incorrect operation sequence Replan the steps
Insufficient operation permissions System denies the operation Request authorization or exit
Unclear operation result Lack of clear feedback Re-observe and verify

2. Exception Handling Process

flowchart TD
    A["Perform computer operation"] --> B["Obtain operation feedback"]
    B --> C{"Succeeded?"}
    C -->|"Yes"| D["Update task status"]
    C -->|"No"| E["Identify exception type"]
    E --> F{"Can it recover automatically?"}
    F -->|"Yes"| G["Re-observe the current interface"]
    G --> H["Adjust operation strategy"]
    H --> I{"Retry limit exceeded?"}
    I -->|"No"| A
    I -->|"Yes"| J["Record failure and stop"]
    F -->|"No"| K{"Requires manual handling?"}
    K -->|"Yes"| L["Request manual intervention"]
    K -->|"No"| J
    L --> M{"Manual handling completed"}
    M -->|"Yes"| G
    M -->|"No"| J
    D --> N["Continue executing the task"]

    style A fill:#EAF3EC,stroke:#83A995,color:#315D49
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F