Back to series
· 16 min read

Browser Agent: How Do Agents Operate a Browser?


I. What Is a Browser Agent?

A Browser Agent is an AI agent that can understand a user's goal and autonomously control a browser to complete tasks.

Traditional browser automation usually relies on pre-written scripts. Developers must explicitly specify which URL to visit, which element to click, what content to fill in, and how to handle page navigation.

Browser Agents are different. The user only needs to describe the task goal, and the agent can decide the next action on its own based on the current state of the web page, then adjust its subsequent strategy according to the execution results.

For example, a user might say:

Search for recently published technical articles about AI Agents, filter out 5 valuable pieces of content, extract the title, author, publication date, and link, and organize them into a table.

A Browser Agent needs to:

  1. Understand the user's search goal.
  2. Choose an appropriate search website.
  3. Open the browser and visit the search page.
  4. Enter the search keywords.
  5. Analyze the search results.
  6. Identify relevant technical articles.
  7. Open the target articles and extract information.
  8. Determine whether the information is complete and whether it is duplicated.
  9. Organize it into a structured table.
  10. Return the final result.

The whole process involves web navigation, page understanding, element localization, information extraction, and task state management.

1. Differences Between Browser Agents and Ordinary Browser Automation

Comparison Dimension Traditional Browser Automation Browser Agent
Execution goal Predefined operation steps A task goal described by the user
Execution path Fixed scripts Dynamic planning
Page recognition CSS Selector, XPath DOM, visual understanding, semantic localization
Decision-making method Preset rules Model reasoning and strategy selection
Exception handling Fixed exception branches Adjustment based on page state
Adaptability Easily breaks when the page changes Can adapt to some changes
Information extraction Fixed fields and selectors Dynamic extraction based on the task
Use cases Repetitive workflows Unstructured web tasks

Traditional automation focuses on how to execute specified steps, whereas a Browser Agent focuses more on how to accomplish the user's goal.

2. Differences Between Browser Agents and Computer Use

Browser Agents and Computer Use both belong to the ability of agents to operate computers, but they differ in operation scope and interaction method.

Comparison Dimension Browser Agent Computer Use
Operation scope Browsers and web pages The entire desktop environment
Primary targets Web pages, tabs, browser controls Browsers, desktop software, operating systems
Core technologies DOM, Playwright, browser tools Screenshots, vision models, mouse and keyboard
Element localization CSS, XPath, semantic localization Coordinates, visual recognition, control tree
Interaction method Page-level operations and browser control Graphical interface operations
Typical tasks Searching, scraping, filling out web pages Cross-application operations and desktop automation

The two are not mutually exclusive.

A Browser Agent can use tools such as Playwright to perform web page operations, and can also invoke Computer Use when needed to control the browser through the mouse and keyboard.

A Browser Agent focuses on the autonomous completion of browser tasks, while Computer Use focuses on general operations of the computer's graphical interface.

II. Overall Architecture of a Browser Agent

A Browser Agent typically consists of a task planner, a browser controller, a page perception module, an action executor, an information extraction module, and a state management module.

1. Core Architecture

flowchart TD
    A["User Task Goal"] --> B["Agent Controller"]
    B --> C["Task Understanding and Planning"]
    C --> D["Browser Agent"]
    D --> E["Browser Control Layer"]
    E --> F["Page State Acquisition"]
    F --> G["DOM and Visual Understanding"]
    G --> H["Page Element Localization"]
    H --> I["Action Decision"]
    I --> J["Browser Action Execution"]
    J --> K["Web Environment"]
    K --> L["Operation Result Feedback"]
    L --> M{"Task Completed?"}
    M -->|"No"| F
    M -->|"Yes"| N["Information Organization and Result Delivery"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The responsibilities of each module are as follows:

Module Main Responsibility
Agent Controller Manage task execution and overall state
Task Planner Break down natural language goals into operation steps
Browser Control Layer Manage browsers, tabs, and pages
Page Perception Module Acquire DOM, screenshots, and page state
Element Localization Module Determine the web page elements to operate on
Action Execution Module Execute clicks, input, scrolling, and navigation
Information Extraction Module Extract structured data from web pages
State Management Module Save operation history, task progress, and exception information

2. Three-Layer Structure of Browser Agent

From an engineering implementation perspective, Browser Agent can be divided into three layers.

Intelligent Decision Layer

Responsible for understanding tasks, breaking down steps, selecting tools, analyzing pages, and determining the next action.

Browser Control Layer

Responsible for managing browser instances, tabs, page loading, network status, and browser context.

Web Page Interaction Layer

Responsible for locating DOM elements, clicking buttons, filling out forms, extracting text, and performing page operations.

flowchart TB
    A["Intelligent Decision Layer"]
    A1["Task Understanding"]
    A2["Step Planning"]
    A3["Tool Selection"]
    A4["Result Evaluation"]
    B["Browser Control Layer"]
    B1["Browser Instance"]
    B2["Tab Management"]
    B3["Page Lifecycle"]
    B4["Session and Context"]
    C["Web Interaction Layer"]
    C1["DOM Parsing"]
    C2["Element Location"]
    C3["Page Operations"]
    C4["Information Extraction"]

    A --> B
    B --> C
    A1 --> A
    A2 --> A
    A3 --> A
    A4 --> A
    B1 --> B
    B2 --> B
    B3 --> B
    B4 --> B
    C1 --> C
    C2 --> C
    C3 --> C
    C4 --> C

    style A fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style A1 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style A2 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style A3 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style A4 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B1 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B2 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B3 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B4 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style C1 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C2 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C3 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C4 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

The three layers collaborate through tool interfaces and state data. The intelligent decision layer does not need to directly operate the browser's low-level internals; instead, it accomplishes tasks through the capabilities provided by the browser control layer.

III. Browser Control: How Does an Agent Connect to a Browser?

A Browser Agent first needs to acquire browser control capability.

It must be able to create browser instances, open web pages, manage tabs, retrieve page content, and perform browser operations.

1. Basic Principles of Browser Automation

Browser automation tools typically communicate with the browser through debugging protocols or automation interfaces provided by the browser.

Taking Chromium as an example, common control methods include:

  • Chrome DevTools Protocol (CDP).
  • WebDriver protocol.
  • Playwright's own browser control mechanism.
  • Interaction interfaces provided by browser extensions.

These technologies allow external programs to access browser pages, read the DOM, execute JavaScript, simulate input, and capture screenshots.

2. Playwright

Playwright is one of the commonly used browser automation tools today, supporting Chromium, Firefox, and WebKit.

It can:

  • Launch a browser.
  • Create a browser context.
  • Open web pages.
  • Locate page elements.
  • Click and type.
  • Monitor page loading.
  • Retrieve page text.
  • Save screenshots.

Installation:

pip install playwright
playwright install

Basic example:

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=False
    )

    context = browser.new_context()

    page = context.new_page()

    page.goto("https://example.com")

    print(page.title())

    print(page.url)

    browser.close()

Where:

Object Role
Browser Browser instance
BrowserContext Independent browser context
Page Browser tab
Locator Page element locator
Frame Frame within a page
BrowserType Browser type

3. The Role of BrowserContext

BrowserContext is a very important concept in browser automation.

A single browser instance can create multiple independent BrowserContexts, and each context can have its own cookies, cache, and session storage.

For example:

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=False
    )

    context_a = browser.new_context()
    context_b = browser.new_context()

    page_a = context_a.new_page()
    page_b = context_b.new_page()

    page_a.goto("https://example.com")
    page_b.goto("https://example.org")

    browser.close()

This kind of isolation is suitable for:

  • Multi-user parallel tasks.
  • Browser operations with different accounts.
  • Agent session isolation.
  • Independent test environments.
  • Preventing different tasks from sharing login state.

Note that BrowserContext primarily isolates browser session data; it is not equivalent to an OS-level security sandbox.

4. Browser Connection and Lifecycle Management

In a production environment, you should not restart the browser for every action performed.

Usually a browser manager is responsible for:

  1. Creating browser instances.
  2. Allocating browser contexts.
  3. Creating and managing tabs.
  4. Maintaining connection state.
  5. Handling browser crashes.
  6. Releasing resources that are no longer in use.

The browser lifecycle can be represented as:

flowchart TD
    A["Receive browser task"] --> B["Check browser instance"]
    B --> C{"Is the browser available?"}
    C -->|"No"| D["Launch browser"]
    C -->|"Yes"| E["Reuse browser"]
    D --> F["Create BrowserContext"]
    E --> F
    F --> G["Create or select Page"]
    G --> H["Execute web task"]
    H --> I{"Are there follow-up tasks?"}
    I -->|"Yes"| H
    I -->|"No"| J["Release context or retain session"]
    J --> K["Task complete"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F

IV. Web Perception: How Does an Agent Understand a Web Page?

After the browser opens a web page, the Browser Agent needs to obtain page information and convert it into context that the model can understand.

Unlike Computer Use, which mainly relies on screenshots, a Browser Agent can usually obtain the web page structure directly.

1. What Is the DOM?

The DOM (Document Object Model) is the tree structure formed after the browser parses an HTML document.

For example:

<html>
  <body>
    <h1>AI 技术文章</h1>

    <input
      type="text"
      placeholder="请输入关键词"
    />

    <button>搜索</button>

    <a href="/article/1">
      查看文章
    </a>
  </body>
</html>

The browser converts HTML into a DOM node tree.

This can be simplified as:

Document
└── html
    └── body
        ├── h1
        │   └── "AI Technical Article"
        ├── input
        ├── button
        │   └── "Search"
        └── a
            └── "View Article"

The DOM can provide clearer structural information than screenshots.

For example, an Agent can know:

  • which buttons are on the page.
  • which elements are input fields.
  • what text is associated with a button.
  • which address a link points to.
  • whether an element is visible.
  • whether an element is interactive.

2. The Difference Between DOM and HTML

HTML is a markup language that describes the structure of a web page, while the DOM is the object model formed after the browser parses HTML.

The two cannot simply be regarded as the same thing.

HTML describes the initial document structure, while the DOM is the structural state at browser runtime.

When JavaScript modifies the page, the DOM may change, while the original HTML text does not necessarily change in sync.

For example:

document.querySelector("h1").textContent =
    "新的文章标题";

This code modifies the DOM node in the current page.

Therefore, a Browser Agent should focus on the runtime state of the current page and must not always rely on the initially retrieved HTML.

3. Retrieving DOM Content

Playwright can be used to read page content:

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch()

    page = browser.new_page()

    page.goto("https://example.com")

    html = page.content()

    print(html)

    browser.close()

If you only need the page text:

text = page.locator("body").inner_text()

print(text)

If you need to retrieve links:

links = page.locator("a").evaluate_all("""
elements => elements.map(element => ({
    text: element.innerText,
    href: element.href
}))
""")

print(links)

These APIs can help an Agent obtain structured data from the page.

4. Accessibility Tree

In addition to the DOM, the browser can also provide an Accessibility Tree.

It is an accessibility information structure formed by the browser based on the structure, semantics, and accessibility attributes of the web page.

For example:

<button aria-label="搜索文章">
    <svg></svg>
</button>

Although this button has no visible text, it provides clear semantic information through aria-label.

The accessibility tree can help an agent understand:

  • element roles.
  • element names.
  • element states.
  • hierarchical relationships between elements.
  • semantics of interactive elements.

Common roles include:

Role Meaning
button Button
textbox Text input field
link Link
checkbox Checkbox
radio Radio button
heading Heading
combobox Combo box
list List

For well-structured web pages, the Accessibility Tree is often more suitable for an agent to perform semantic localization than relying solely on screenshots.

5. Page Screenshots and Visual Understanding

Although the DOM and the accessibility tree are very useful, they cannot solve all problems.

For example:

  • The page does not correctly set semantic attributes.
  • Important content is drawn via Canvas.
  • The page uses complex custom controls.
  • The web page contains an image CAPTCHA.
  • Page layout relationships affect operation decisions.

In such cases, screenshots and vision models need to be combined.

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch()

    page = browser.new_page(
        viewport={
            "width": 1280,
            "height": 800
        }
    )

    page.goto("https://example.com")

    page.screenshot(
        path="page.png",
        full_page=True
    )

    browser.close()

In engineering practice, a hybrid perception strategy can be adopted:

  • DOM is used for structure retrieval.
  • Accessibility Tree is used for semantic localization.
  • OCR is used to recognize text in images.
  • Vision models are used to understand layout and complex interfaces.
  • Screenshots are used to help determine the actual display state of the page.

V. Web Element Localization: How Does the Agent Find the Target?

After web page perception, the Agent needs to find the element that actually needs to be operated on.

For example, the user requests:

Click the search button.

The model must find the corresponding search button on the page and confirm that it is the target required by the current task, rather than a button with the same name elsewhere.

1. CSS Selector

CSS Selector is a way to locate DOM elements through CSS selector syntax.

For example:

page.locator("#search")

means locating the element with id="search".

page.locator(".search-button")

means locating the element with the search-button class name.

page.locator("input[name='keyword']")

means locating the input box with the specified name.

Advantages:

  • Precise localization.
  • High execution efficiency.
  • Suitable for pages with a stable structure.

Disadvantages:

  • Depends on the page DOM structure.
  • May fail after the page is redesigned.
  • Not necessarily the most readable for models.

2. XPath

XPath is a query language used to locate nodes in an XML or HTML tree structure.

For example:

page.locator("//button[text()='搜索']")

You can also use:

page.locator(
    "//input[@placeholder='请输入关键词']"
)

XPath can locate based on text, attributes, and node relationships.

However, complex XPath expressions easily depend on page hierarchy, causing localization issues after the page structure changes.

3. Semantic Localization

Compared with CSS and XPath, semantic localization is closer to human expression.

For example:

page.get_by_role(
    "button",
    name="搜索"
).click()

You can also use:

page.get_by_placeholder(
    "请输入关键词"
).fill("AI Agent")

Or:

page.get_by_text(
    "查看文章"
).click()

The advantages of semantic localization are:

  • The expression is more natural.
  • It is easier to correspond to user tasks.
  • There is no need to remember complex DOM hierarchies.
  • It is suitable for large models to select operation targets based on page semantics.

However, semantic localization also needs to handle issues such as elements with the same name, fuzzy matching, and non-unique element matches.

4. Priority of Localization Strategies

In production environments, the following strategies can be adopted:

Priority Locating method Applicable scenario
1 Unique ID or stable attribute The page provides a stable identifier
2 Role and Accessible Name Control semantics are clear
3 Label and Placeholder Form elements
4 Text locator Page text is clear
5 CSS Selector Precise structural locating is required
6 XPath Complex node relationships
7 Visual coordinates Reliable structural information cannot be obtained

This is not an absolute technical ranking. Actual selection should be based on element uniqueness, stability, and page semantic quality.

5. Verification after locating

Locating an element does not mean it can be operated directly.

You need to check:

  • Whether the element exists.
  • Whether the element is unique.
  • Whether the element is visible.
  • Whether the element is interactive.
  • Whether the element is obscured by other content.
  • Whether the current element belongs to the correct page or area.

For example:

search_button = page.get_by_role(
    "button",
    name="搜索"
)

count = search_button.count()

if count != 1:
    raise RuntimeError(
        f"搜索按钮数量异常:{count}"
    )

if not search_button.is_visible():
    raise RuntimeError(
        "搜索按钮不可见"
    )

if not search_button.is_enabled():
    raise RuntimeError(
        "搜索按钮不可用"
    )

search_button.click()

This can reduce incorrect operations caused by locating errors.

VI. Web Page Action Execution: How Does the Agent Operate the Page?

The action execution layer of the Browser Agent is responsible for converting the model's decisions into browser operations.

1. Page navigation

Page navigation is the foundation of browser operations.

page.goto(
    "https://example.com",
    wait_until="domcontentloaded"
)

Common wait conditions include:

Wait condition Meaning
commit The browser receives the response and starts loading the document
domcontentloaded DOM content has finished loading
load The page and related resources trigger the load event
networkidle Network activity reaches an idle state

Note that networkidle is not suitable for all websites. Pages with long-lived connections, polling, or continuous network requests may never reach an idle state.

In actual engineering, wait conditions should be selected according to the specific task, and conditions such as element visibility, URL changes, or business state should be used to determine whether the page is ready.

2. Click operation

page.get_by_role(
    "button",
    name="搜索"
).click()

Playwright's Locator usually automatically performs actionability checks and waits within a reasonable range for the element to meet the conditions.

However, a successful click does not mean the target business task has been completed.

For example, after clicking the submit button, you still need to verify whether a submission success prompt appears on the page.

3. Entering text

search_box = page.get_by_placeholder(
    "请输入关键词"
)

search_box.fill("Browser Agent")

fill() sets the input field content to the specified text.

If you need to simulate character-by-character input, you can use:

search_box.press_sequentially(
    "Browser Agent",
    delay=80
)

The two have different applicable scenarios:

  • fill(): quickly set the input field content.
  • press_sequentially(): simulates character-by-character input, suitable for scenarios that need to trigger keyboard input events.

4. Form filling

page.get_by_label(
    "用户名"
).fill("demo_user")

page.get_by_label(
    "邮箱"
).fill("demo@example.com")

page.get_by_role(
    "button",
    name="提交"
).click()

In real tasks, a form may contain:

  • Text box.
  • Password box.
  • Dropdown list.
  • Radio button.
  • Checkbox.
  • Date picker.
  • File upload control.

The agent needs to choose the correct operation method based on the field type.

5. Page Scrolling

page.evaluate("""
window.scrollBy({
    top: 600,
    behavior: "smooth"
})
""")

You can also scroll the page to a specific element:

target = page.get_by_text(
    "相关文章"
)

target.scroll_into_view_if_needed()

The second method is better suited to target-element-driven automation.

6. Tab Management

A Browser Agent may need to open multiple web pages at the same time.

For example:

  • Search results page.
  • Article detail page.
  • Reference page.
  • Data organization page.

Playwright can listen for new pages:

with page.expect_popup() as popup_info:
    page.get_by_text(
        "打开详情"
    ).click()

new_page = popup_info.value

print(new_page.url)

You can also listen for new tabs using a browser context:

new_pages = []

context.on(
    "page",
    lambda page: new_pages.append(page)
)

The Agent needs to clearly distinguish the tab corresponding to the current task to avoid performing operations on the wrong page.

VII. Browser Agent's Core Execution Loop

The most important mechanism of a Browser Agent is continuously obtaining page state, analyzing the task, executing actions, and verifying results.

1. Observe-Think-Act

Its basic pattern can be summarized as:

  • Observe: Get the current web page state.
  • Think: Understand the page and determine the next step.
  • Act: Perform browser operations.
  • Verify: Check whether the operation produced the expected result.
flowchart TD
    A["Observe: Get page state"] --> B["Parse DOM and page content"]
    B --> C["Think: Analyze task progress"]
    C --> D["Determine target element and action"]
    D --> E["Act: Perform browser action"]
    E --> F["Verify: Check operation result"]
    F --> G{"Is the goal reached?"}
    G -->|"No"| A
    G -->|"Yes"| H["Organize task results"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

2. Why Can't All Browser Actions Be Generated at Once?

Suppose the user requests:

Log in to the website, search for a specified product, add it to the shopping cart, and then submit the order.

If the agent generates all actions at once:

  1. Open the website.
  2. Click login.
  3. Enter account and password.
  4. Submit login.
  5. Search for products.
  6. Click the product.
  7. Add to cart.
  8. Submit order.

Any change in a step can cause subsequent operations to fail.

For example:

  • Login fails.
  • The website displays a CAPTCHA.
  • Search results are empty.
  • The product is no longer available.
  • The cart button is disabled.
  • A confirmation page appears before order submission.

Therefore, a Browser Agent must re-evaluate subsequent steps based on the result of each key operation.

3. Action Batches and State Verification

Not every browser action requires a separate call to the model.

For example:

Locate the search box
Fill in keywords
Press Enter

These operations have a clear sequential relationship and can be executed as a single batch.

However, after the search is complete, which result to choose depends on the actual page content, so re-observation is required.

Recommended action granularity:

Operation Type Recommended Execution Method
Consecutive deterministic operations Can be executed in batches
Page navigation Check the URL and loading status after execution
Dynamic search results Re-read the page
Form submission Execute separately and verify
Deletion and payment Mandatory risk check
Unclear results Pause and re-observe

The core principle is: deterministic operations can be executed in batches, while operations that depend on environmental changes must be verified promptly.

VIII. Browser Agent Information Extraction Mechanisms

A Browser Agent is not only able to operate web pages; it also needs to extract valuable information from them.

For example, a user asks it to collect the titles, authors, publication dates, and links of 10 technical articles.

This requires converting web page content into structured data.

1. Page Text Extraction

The simplest approach is to obtain the visible text on the web page.

content = page.locator(
    "body"
).inner_text()

print(content)

This approach is suitable for quickly obtaining the main text of a page, but it may contain:

  • Navigation bar.
  • Footer.
  • Ads.
  • Related recommendations.
  • Duplicate content.
  • Irrelevant notification messages.

Therefore, the entire page text cannot be handed directly to the model without processing.

2. Structured DOM Extraction

If the page structure is relatively stable, specified fields can be extracted using selectors.

For example:

articles = page.locator(
    ".article-card"
).evaluate_all("""
elements => elements.map(element => ({
    title: element.querySelector(".title")?.innerText,
    author: element.querySelector(".author")?.innerText,
    date: element.querySelector(".date")?.innerText,
    link: element.querySelector("a")?.href
}))
""")

print(articles)

Output structure:

[
  {
    "title": "AI Agent 技术实践",
    "author": "作者 A",
    "date": "2026-11-20",
    "link": "https://example.com/article/1"
  },
  {
    "title": "Browser Agent 架构设计",
    "author": "作者 B",
    "date": "2026-11-19",
    "link": "https://example.com/article/2"
  }
]

The fields and sample data above are only used to illustrate the structured extraction method.

3. Model-Assisted Extraction

If page structures differ across websites, you can first retrieve the relevant content and then pass it to the model for semantic extraction.

For example, ask the model to output:

{
  "title": "文章标题",
  "author": "作者名称",
  "published_at": "发布日期",
  "url": "文章链接",
  "summary": "文章摘要"
}

The model is responsible for understanding unstructured content, and the program is responsible for validating the output format.

Model output should not be directly treated as trustworthy data; you also need to check:

  • Whether the title is empty.
  • Whether the URL is valid.
  • Whether the date format is correct.
  • Whether the fields come from the target page.
  • Whether there are duplicate articles.
  • Whether any fields were inferred and completed by the model.

4. Complete Information Extraction Process

flowchart TD
    A["Open the target webpage"] --> B["Retrieve page content"]
    B --> C["Extract relevant DOM regions"]
    C --> D["Clean irrelevant information"]
    D --> E["Model semantic extraction"]
    E --> F["Generate structured data"]
    F --> G["Validate field formats"]
    G --> H{"Is the data complete?"}
    H -->|"No"| I["Re-extract or visit the detail page"]
    I --> B
    H -->|"Yes"| J["Deduplicate and merge data"]
    J --> K["Output the final result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F

5. Data Sources and Credibility

Data extracted by a Browser Agent should preserve source information as much as possible.

The following data structure is recommended:

article = {
    "title": "Browser Agent 架构设计",
    "author": "作者名称",
    "published_at": "2026-11-20",
    "url": "https://example.com/article/1",
    "source": "example.com",
    "extracted_at": "2026-11-21T08:00:00",
    "confidence": 0.95
}

Where:

  • source: Data source.
  • extracted_at: Extraction time.
  • confidence: The system's internal confidence assessment.

It should be noted that the confidence output by the model is not inherently equal to the actual accuracy rate. Production systems should calibrate confidence through sample verification and evaluation data.

IX. Combining Browser Agent with RAG

Browser Agent can be combined with RAG (Retrieval-Augmented Generation) to enable web search, information collection, and knowledge base updates.

The two solve different problems:

  • Browser Agent is responsible for accessing web pages and acquiring external information.
  • RAG is responsible for knowledge retrieval, context organization, and augmented generation.

1. Browser Agent + RAG Architecture

flowchart TD
    A["User asks a question"] --> B["Agent analyzes the question"]
    B --> C["Generate web search strategy"]
    C --> D["Browser Agent"]
    D --> E["Search engine"]
    E --> F["Obtain web page results"]
    F --> G["Visit target page"]
    G --> H["Extract web page content"]
    H --> I["Cleaning and document parsing"]
    I --> J["Text chunking"]
    J --> K["Generate Embedding"]
    K --> L["Write to vector database"]
    L --> M["RAG retrieval"]
    M --> N["Context construction"]
    N --> O["Large model generates answer"]
    O --> P["Return answer and sources"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49
    style M fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style N fill:#EAF3EC,stroke:#83A995,color:#315D49
    style O fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style P fill:#DCEFE5,stroke:#4F8B70,color:#24563F