I. What Is Computer Use?
Computer Use (the ability to use a computer) refers to an AI agent interacting with computer systems through a graphical user interface (GUI), observing the screen, identifying controls, moving the mouse, clicking buttons, typing text, scrolling pages, and operating applications just like a human does.
Traditional AI agents interact with external systems mainly through APIs, tool calls, and code execution, whereas Computer Use extends the scope of interaction to desktop applications, browsers, office software, and other systems that do not expose APIs.
For example, a user might say:
Open the browser, search for a product, compare prices across different websites, and compile the results into an Excel file.
An agent with Computer Use capability needs to complete the following:
- Open the browser.
- Go to a search website.
- Enter the keywords and run the search.
- Observe the search results.
- Open the relevant product pages.
- Extract the product names and prices.
- Open Excel.
- Create a spreadsheet and fill in the data.
- Save the file and report the results.
The whole process does not necessarily rely on the target application providing a dedicated API; instead, it is carried out by simulating how a human operates the graphical interface.
1. Differences Between Computer Use and Traditional Tool Calling
| Comparison dimension | Traditional tool calling | Computer Use |
|---|---|---|
| Interaction method | APIs, function calls | Mouse, keyboard, screen |
| Target of operation | Well-defined interfaces | Graphical user interface |
| Input form | Structured parameters | Screenshots, interface state |
| Output form | JSON, text, structured data | Screen changes, operation results |
| Environment dependency | Interfaces and permissions | Operating system, windows, and interface |
| Stability | Usually higher | Affected by interface changes |
| Applicable scope | Systems with APIs | Systems with an operable interface |
| Error handling | API error codes | Visual feedback and state judgment |
The core of traditional tool calling is invoking interfaces that have already been defined, whereas the core of Computer Use is understanding the current interface and deciding what action to perform next.
2. Differences Between Computer Use and RPA
Both Computer Use and RPA (Robotic Process Automation) can operate a computer through the graphical interface, but the two work in different ways.
| Comparison dimension | Traditional RPA | AI Computer Use |
|---|---|---|
| Execution logic | Predefined process | Dynamic planning based on goals |
| Operation path | Fixed steps | Can be adjusted according to the interface |
| Page recognition | Coordinates, control selectors | Visual understanding, control recognition |
| Exception handling | Rules and preset branches | Re-decides based on the new state |
| Adaptability | Depends on process configuration | Can handle some unknown situations |
| Suitable tasks | High-frequency, fixed, repetitive tasks | Multi-step, dynamic, unstructured tasks |
For example, traditional RPA usually requires defining in advance:
- Which button to click.
- Which input field to type into.
- How long to wait.
- Which page to navigate to.
- Which branch to execute when an exception occurs.
AI Computer Use, by contrast, can determine from the screen content what page it is currently on, and then decide where to click.
Note that AI Computer Use does not mean completely replacing RPA. For stable, repetitive processes with clear rules, traditional RPA is often more efficient. The two can also be used in combination.
II. The Overall Architecture of Computer Use
Computer Use is usually not simply a matter of having a large model output mouse coordinates; rather, it is made up of multiple parts working together, including perception, decision-making, action execution, environment feedback, and state management.
1. Core Architecture
flowchart TD
A["User Task Goal"] --> B["Agent Controller"]
B --> C["Task Planning and Step Decomposition"]
C --> D["Computer Use Agent"]
D --> E["Screen Observation and Visual Understanding"]
E --> F["UI Element Recognition"]
F --> G["Action Decision"]
G --> H["Action Executor"]
H --> I["Operating System and Applications"]
I --> J["UI State Changes"]
J --> K["Re-screenshot and Observe"]
K --> L{"Task Completed?"}
L -->|"No"| E
L -->|"Yes"| M["Result Verification and Task Delivery"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The entire system can be divided into six core modules.
| Module | Primary Responsibilities |
|---|---|
| Agent Controller | Manage task lifecycle and execution state |
| Task Planner | Break complex tasks into executable steps |
| Perception Module | Capture screen information and understand the interface |
| Decision Module | Determine the next mouse and keyboard operations |
| Action Executor | Convert actions into operating system input |
| State Verification Module | Check action results and determine whether to continue |
Among them, the three most critical stages are perception, decision-making, and execution.
2. Perception, Decision-Making, and Execution
The basic working mode of Computer Use can be summarized as:
- Perception: What is currently displayed on the screen?
- Decision-making: What should be done next to complete the task?
- Execution: How can the decision be converted into mouse and keyboard operations?
- Verification: Did the operation actually produce the expected result?
This is a closed-loop interactive system, rather than a one-off generation of a series of mouse coordinates.
flowchart TD
A["Capture current screen"] --> B["Visual understanding"]
B --> C["Recognize target control"]
C --> D["Reason with task goal"]
D --> E["Generate action instruction"]
E --> F["Execute mouse or keyboard actions"]
F --> G["Observe operation result"]
G --> H{"Does the result meet expectations?"}
H -->|"Yes"| I["Update task status"]
H -->|"No"| J["Analyze failure cause"]
J --> K["Adjust operation strategy"]
K --> A
I --> L{"Is the task complete?"}
L -->|"No"| A
L -->|"Yes"| M["Output final result"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
3. Screen Perception: How Do Agents Understand Computer Interfaces?
The first core capability of Computer Use is screen perception.
Humans can directly see windows, buttons, text boxes, and menus, and understand their purposes. Large models, however, need to convert screen information into input that can be reasoned about.
1. Screenshots
The most basic method is to capture an image of the current desktop or application window via a screenshot.
For example:
import pyautogui
# Capture the current screen screenshot
screenshot = pyautogui.screenshot()
# Save the screenshot
screenshot.save("screen.png")
The screenshot can be used as input to a vision model, allowing the model to analyze the current interface.
However, a screenshot not only contains the target control, but may also contain:
- Multiple application windows.
- Browser tabs.
- Pop-ups and notification messages.
- Desktop icons.
- Irrelevant background content.
- Mouse pointer and scrollbars.
Therefore, the agent needs to identify the parts relevant to the current task from a large amount of visual information.
2. Visual Understanding
Visual understanding is the process of converting screen images into structured semantic information.
For example, the current screen shows a login page:
Login Page
Username: [________________]
Password: [________________]
[Login]
Forgot password?
The vision model needs to recognize:
| UI Element | Recognition Result | Purpose |
|---|---|---|
| Username input field | Input | Enter account |
| Password input field | Input | Enter password |
| Login button | Button | Submit login |
| Forgot password | Link | Recover password |
The model not only needs to recognize the positions of elements, but also needs to understand their semantics and purposes.
For example:
- Which input field is used to enter the username?
- Which button is used to submit the form?
- Is the current page already logged in?
- Is an error message displayed?
- Is there a pop-up that needs to be handled?
This requires the vision model to have certain OCR, object detection, layout understanding, and semantic reasoning capabilities.
3. OCR and Visual Models
Computer Use can use different technologies to obtain interface information.
| Technology | Main role | Characteristics |
|---|---|---|
| OCR | Extract on-screen text | Good for recognizing text and labels |
| Object detection | Locate buttons and controls | Can identify element regions |
| Image understanding models | Understand the overall meaning of a page | Suited to complex interfaces |
| UI Automation | Retrieve native control information | Can read control properties |
| DOM parsing | Retrieve web page structure | Suited to browser environments |
OCR can recognize text, but it cannot necessarily understand the function that the text corresponds to.
For example, there are two buttons on the screen:
- Confirm
- Cancel
OCR can recognize the text on both buttons, but visual understanding is still needed to determine which button should be clicked.
In engineering practice, you should not hand every task over to a vision model. For browser pages, prefer using the DOM and the accessibility tree to obtain structured information; for desktop applications where structured information is unavailable, then use visual understanding.
4. Coordinate Systems and Element Localization
Computer Use frequently needs to convert recognition results into screen coordinates.
Suppose the model recognizes a login button whose position is:
{
"element": "登录按钮",
"type": "button",
"center": {
"x": 640,
"y": 480
}
}
Here, x and y represent the position of the button's center point on the screen.
If the screen resolution is 1920 × 1080, the coordinate origin is usually located at the top-left corner:
- The X axis increases to the right.
- The Y axis increases downward.
- The top-left corner of the screen is
(0, 0).
Coordinate-based localization requires attention to several issues:
- The screenshot resolution may differ from the actual screen resolution.
- System scaling may affect coordinate mapping.
- Multi-monitor environments may have different coordinate origins.
- Moving or resizing a window may change the positions of controls.
- Page scrolling may change the screen coordinates of the target element.
Therefore, coordinates should not be treated as a long-term valid identifier for a control.
A more reliable approach is to re-observe the interface before every critical operation and determine the operation position based on the current screenshot.
IV. Action Execution: How Does the Agent Control the Mouse and Keyboard?
After completing screen perception, the agent needs to convert decisions into concrete computer input.
Common operations include:
| Action | Role |
|---|---|
| click | Click the mouse |
| double_click | Double-click the mouse |
| move | Move the mouse |
| drag | Drag the mouse |
| scroll | Scroll the page |
| key_press | Press a keyboard key |
| type_text | Type text |
| hotkey | Execute a key combination |
| screenshot | Capture the current screen |
1. Mouse Operations
Take Python's PyAutoGUI as an example:
import pyautogui
# Move the mouse
pyautogui.moveTo(640, 480, duration=0.3)
# Click the mouse
pyautogui.click(640, 480)
# Double-click
pyautogui.doubleClick(640, 480)
# Right-click
pyautogui.rightClick(640, 480)
# Drag
pyautogui.moveTo(300, 300)
pyautogui.dragTo(600, 500, duration=0.5)
In essence, these operations convert coordinates and actions into input events that the operating system can recognize.
2. Keyboard Operations
Keyboard input can be divided into ordinary text input, key presses, and key combinations.
import pyautogui
# Type text
pyautogui.write("Hello Computer Use", interval=0.05)
# Press Enter
pyautogui.press("enter")
# Copy using a keyboard shortcut
pyautogui.hotkey("ctrl", "c")
# Paste using a keyboard shortcut
pyautogui.hotkey("ctrl", "v")
# Select all
pyautogui.hotkey("ctrl", "a")
Note that write() is mainly suitable for inputting ordinary characters. Chinese input, special characters, input method switching, and keyboard layouts on different operating systems may require additional handling.
3. Action Abstraction Layer
In actual engineering, you should not let a large model directly generate arbitrary Python code and execute it.
A more reasonable design is to provide a set of constrained action interfaces.
For example:
from typing import Literal
from pydantic import BaseModel, Field
class ComputerAction(BaseModel):
action: Literal[
"click",
"double_click",
"move",
"scroll",
"type",
"press",
"hotkey",
"screenshot"
]
x: int | None = None
y: int | None = None
text: str | None = None
key: str | None = None
direction: Literal[
"up",
"down",
"left",
"right"
] | None = None
amount: int = Field(default=0, ge=0, le=1000)
The model can output structured actions:
{
"action": "click",
"x": 640,
"y": 480
}
The executor is responsible for validating parameters and executing them:
import pyautogui
def execute_action(action: ComputerAction):
if action.action == "click":
if action.x is None or action.y is None:
raise ValueError("点击操作缺少坐标")
pyautogui.click(action.x, action.y)
elif action.action == "type":
if action.text is None:
raise ValueError("输入操作缺少文本")
pyautogui.write(action.text)
elif action.action == "press":
if action.key is None:
raise ValueError("按键操作缺少按键")
pyautogui.press(action.key)
elif action.action == "screenshot":
return pyautogui.screenshot()
else:
raise ValueError("暂不支持该动作")
It should be emphasized here: the example only demonstrates how to abstract actions; a production system also needs coordinate boundary validation, permission checks, timeout control, operation logs, and exception recovery.
V. The Core Execution Loop of Computer Use
The most important mechanism of Computer Use is not a single click action, but the loop of continuously observing the environment, executing actions, and updating state.
1. The Observe-Think-Act Loop
The core mechanism can be abstracted into three stages:
- Observe: Observe the current screen.
- Think: Analyze the goal and current state, and decide the next action.
- Act: Execute the action and obtain feedback.
flowchart TD
A["Observe: Observe the screen"] --> B["Identify the current interface state"]
B --> C["Think: Analyze the task goal"]
C --> D["Choose the next action"]
D --> E["Act: Perform mouse or keyboard operations"]
E --> F["Obtain the new screen state"]
F --> G{"Is the goal achieved?"}
G -->|"No"| A
G -->|"Yes"| H["Task completed"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
For example, a user asks the agent to search for a technical article in a browser.
On the first observation, the browser displays the search home page.
The agent decides to click the search box and enter keywords.
After execution, the browser displays the search results.
The agent observes the page again, identifies the search results list, and then selects the target article.
Each action changes the environment, so the next decision should be based on the latest interface state rather than always relying on the initial screenshot.
2. Why Can't All Actions Be Generated at Once?
Suppose the agent plans to execute:
Open browser
Click address bar
Enter URL
Press Enter
Wait for the page to load
Click login
Enter account
Enter password
Click submit
If all actions are generated at once and executed back-to-back, many problems arise:
- The browser may not open successfully.
- The address bar may not receive focus.
- The page may fail to load.
- The login button may have moved.
- A CAPTCHA may appear on the login page.
- An error message may appear after submission.
If an earlier action fails, all subsequent actions may become invalid.
Therefore, a more reliable way to execute is to split the task into multiple stages, with necessary state verification at each stage.
3. Action Execution Granularity
Not every action needs its own separate call to the large model.
For example:
Click the search box
Enter keywords
Press Enter
These three actions are clearly related and can be executed as one short batch of operations.
However, which result to open after the search completes must be judged anew based on the page content.
In engineering practice, you can adopt:
- Deterministic operations: batch execution is allowed.
- State-dependent operations: re-observe after execution.
- High-risk operations: must be executed individually and verified.
- Irreversible operations: require additional authorization or human confirmation.
This balances efficiency and reliability.
VI. Computer Use and Multimodal Large Models
Computer Use generally requires the participation of a multimodal large model, because the system must both understand screen images and generate operation commands based on natural-language goals.
1. Multimodal Input
Unlike ordinary text tasks, the input to Computer Use typically includes:
- The user's task description.
- The current screen screenshot.
- The history of operations.
- The current task state.
- The list of available actions.
- Execution environment information.
For example:
{
"task": "Open the browser and search for a Python tutorial",
"screen": {
"width": 1920,
"height": 1080,
"image": "current screen screenshot"
},
"history": [
"Launch the browser"
],
"available_actions": [
"click",
"type",
"press",
"scroll",
"screenshot"
]
}
The model needs to combine this information to determine the next action.
2. Multimodal Output
Model output is not necessarily just natural language; it can also be a structured action.
For example:
{
"reasoning_summary": "The search page is already open; keywords need to be entered",
"action": {
"type": "click",
"coordinate": {
"x": 720,
"y": 320
}
}
}
The executor then sends the action to the operating system.
Here reasoning_summary is only used to describe a presentable basis for the decision; it should not require the model to expose its complete internal reasoning process.
3. Coordinate Prediction and Visual Grounding
The model can represent the target location in different ways.
Absolute coordinates
{
"x": 640,
"y": 480
}
This directly expresses a position on the screen.
The advantage is simplicity; the drawback is that it is easily affected by screen resolution, window position, and scaling factor.
Relative coordinates
{
"x_ratio": 0.5,
"y_ratio": 0.45
}
This expresses coordinates as proportions of the screen width and height.
Conversion formulas:
Where:
- W: screen width.
- H: screen height.
Relative coordinates can reduce the impact of resolution changes, but they cannot solve the problems of layout changes and element movement.
Bounding box grounding
{
"element": "search button",
"bbox": [600, 430, 700, 480]
}
Where the bounding box is expressed as:
The executor can choose the center point of the bounding box to click:
A bounding box can provide more spatial information than a single-point coordinate.
4. Limitations of Multimodal Models
Vision models are not absolutely reliable.
Common problems include:
- Small-font recognition errors.
- Icon semantic understanding errors.
- Confusion between similar buttons.
- Target loss due to screen occlusion.
- Unstable screenshot states caused by page animations.
- Inaccurate understanding of complex tables and dense layouts.
- Mistaking visually adjacent elements for having the same function.
Therefore, production systems should combine visual recognition, DOM, accessibility tree, and operation result verification as much as possible, rather than relying entirely on the model guessing coordinates.
7. Browser Automation and Desktop Automation
Computer Use can be applied to different computer operation environments, but there are obvious differences in the technical implementation of browsers and desktop applications.
1. Browser Automation
Browsers usually have richer structured information.
For example:
- DOM nodes.
- Element attributes.
- CSS selectors.
- Accessibility Tree.
- Page URL.
- Network requests and loading states.
Therefore, web page elements can be obtained directly through tools such as Playwright.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.goto("https://example.com")
print(page.title())
page.screenshot(path="page.png")
browser.close()
If you want to operate the search box:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.goto("https://example.com")
search_box = page.locator("input[name='q']")
search_box.fill("Python 教程")
search_box.press("Enter")
page.wait_for_load_state("domcontentloaded")
browser.close()
Here DOM positioning is used, rather than relying on fixed screen coordinates.
This approach is usually more stable, and it is also easier to determine whether the operation succeeded.
2. Desktop Automation
Desktop applications are more complicated.
For example:
- Excel.
- Word.
- Photoshop.
- Local file manager.
- Enterprise internal desktop clients.
- Legacy business systems without open APIs.
These software programs may not have a directly accessible web DOM and require:
- Windows UI Automation.
- macOS Accessibility API.
- Linux AT-SPI.
- Image recognition and OCR.
- Mouse and keyboard simulation tools.
For desktop software, Computer Use can identify controls through screenshots and perform mouse and keyboard operations.
3. How to Choose Between the Two Approaches?
| Task Scenario | Recommended Approach |
|---|---|
| Open web pages and navigate | Browser automation |
| Fill out web forms | DOM or accessibility tree |
| Click complex controls in web pages | DOM first, visual assistance |
| Operate local Excel | Office API or desktop automation |
| Operate traditional desktop software | UI Automation or visual operations |
| Operate software without open interfaces | Computer Use |
| Complex cross-application workflows | Combination of API, browser automation, and Computer Use |
An important engineering principle is:
Tasks that can be completed through stable APIs do not need to be forced to use Computer Use; only when interface interaction truly adds value should visual operations be used.
8. Typical Application Scenarios of Computer Use
The advantage of Computer Use lies in cross-application operations and handling tasks that lack standard APIs.
1. Browser Information Collection
User request:
Open the browser, search for recently published AI technology articles, and organize the titles and links.
The agent can:
- Open the browser.
- Go to the search website.
- Enter keywords.
- Obtain search results.
- Identify article titles.
- Open the target pages.
- Organize the information.
- Output the results.
For information that can be obtained through a search API or web page DOM, prioritize structured interfaces; Computer Use can be used for supplementary steps that require real interface interaction.
2. Office Software Automation
For example:
Organize the sales data in the download directory into an Excel report and generate a bar chart.
The agent can:
- Open the file manager.
- Locate the data file.
- Open Excel.
- Create a worksheet.
- Enter the data.
- Format the spreadsheet.
- Create a chart.
- Save the file.
However, for file reading, data processing, and chart generation, using Python, Excel API, or Office automation interfaces directly is usually more efficient. Computer Use is better suited to supplement operations that must be completed through a graphical interface.
3. Cross-Application Tasks
For example:
Retrieve customer information from the browser, organize it in Excel, and then send it to the specified contact via email.
This is a typical cross-application task.
flowchart TD
A["Receive user task"] --> B["Open browser"]
B --> C["Get customer information"]
C --> D["Extract and organize data"]
D --> E["Open Excel"]
E --> F["Create data table"]
F --> G["Save file"]
G --> H["Open email client"]
H --> I["Create email"]
I --> J["Add attachment"]
J --> K{"Is sending authorized?"}
K -->|"No"| L["Wait for human confirmation"]
L --> M{"Approved?"}
M -->|"No"| N["Cancel sending"]
M -->|"Yes"| O["Send email"]
K -->|"Yes"| O
O --> P["Verify sending result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style P fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note that operations such as sending emails, submitting orders, and modifying business data may have real consequences. Agents should verify recipients, attachments, and content before execution, and require human approval based on the risk level.
IX. Exception Handling and Failure Recovery in Computer Use
Graphical interface operations are subject to significant uncertainty; one cannot assume that every action will execute successfully.
Production-grade Computer Use must have exception detection, action retry, state recovery, and task termination mechanisms.
1. Common Exceptions
| Exception Type | Possible Cause | Handling Method |
|---|---|---|
| Target element not found | Page layout changed | Take a new screenshot and identify |
| Click has no effect | Coordinate offset or control not loaded | Relocate and verify |
| Page load timeout | Network or service exception | Wait, retry, or terminate |
| Input failed | Input field did not receive focus | Click again and check |
| Popup obstruction | New window or prompt appears | Identify the popup and decide |
| Incorrect page state | Incorrect operation sequence | Replan the steps |
| Insufficient operation permissions | System denies the operation | Request authorization or exit |
| Unclear operation result | Lack of clear feedback | Re-observe and verify |
2. Exception Handling Process
flowchart TD
A["Perform computer operation"] --> B["Obtain operation feedback"]
B --> C{"Succeeded?"}
C -->|"Yes"| D["Update task status"]
C -->|"No"| E["Identify exception type"]
E --> F{"Can it recover automatically?"}
F -->|"Yes"| G["Re-observe the current interface"]
G --> H["Adjust operation strategy"]
H --> I{"Retry limit exceeded?"}
I -->|"No"| A
I -->|"Yes"| J["Record failure and stop"]
F -->|"No"| K{"Requires manual handling?"}
K -->|"Yes"| L["Request manual intervention"]
K -->|"No"| J
L --> M{"Manual handling completed"}
M -->|"Yes"| G
M -->|"No"| J
D --> N["Continue executing the task"]
style A fill:#EAF3EC,stroke:#83A995,color:#315D49
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F