1. Why Do We Need Agentic RAG?
Traditional RAG (Retrieval-Augmented Generation) typically uses a fixed retrieval pipeline:
- Receive the user's question.
- Convert the question into a vector.
- Perform vector retrieval.
- Retrieve the top-K text chunks.
- Pass the text chunks to the large language model.
- Generate the final answer.
This architecture is simple and stable, and it suits most basic knowledge Q&A scenarios.
But with complex questions, a fixed retrieval chain has clear limitations.
For example, a user asks:
Compare the reasoning capabilities, deployment costs, and applicable scenarios of current mainstream open-source large models, and analyze how they differ when it comes to model selection for enterprise knowledge base systems.
This question actually contains multiple sub-tasks:
- Identify the mainstream open-source models.
- Look up the models' reasoning capabilities.
- Obtain the models' deployment requirements.
- Analyze hardware and operating costs.
- Compare the applicable scenarios of different models.
- Make a selection in the context of the enterprise knowledge base architecture.
- Aggregate information from multiple sources and draw a conclusion.
A single vector retrieval pass can hardly guarantee that the recalled information is complete and accurate.
Traditional RAG also has several typical problems:
- It does not proactively determine whether the current question requires retrieval.
- It does not choose different retrieval strategies based on question complexity.
- When retrieval results are insufficient, it usually does not proactively launch another round of retrieval.
- It cannot dynamically adjust the query based on retrieval quality.
- There is no clear task planning across multiple sub-questions.
- It lacks the ability to handle retrieval failures, conflicting evidence, and missing information on its own.
The core idea of Agentic RAG is:
Make the large language model no longer just a consumer of retrieval results, but the decision-maker and coordinator of the retrieval process.
Based on the current task, the system can dynamically decide:
- Is retrieval needed?
- What should be retrieved?
- Which retrieval tool should be used?
- Does the query need to be rewritten?
- Does the question need to be split?
- Is the current evidence sufficient?
- Should retrieval continue?
- Can the final answer be generated?
This evolves RAG from a fixed pipeline into an intelligent retrieval system capable of planning, execution, observation, and adjustment.
2. What Is Agentic RAG?
Agentic RAG is an architectural pattern that brings Agent capabilities into a RAG system.
It uses a large language model for task planning and decision-making, calls retrieval tools to obtain information, evaluates the current state based on the results returned by those tools, and decides on the next action.
2.1 Core Components of Agentic RAG
A typical Agentic RAG system contains the following modules:
| Module | Main Responsibilities |
|---|---|
| LLM | Understand questions, plan tasks, decide retrieval strategy |
| Agent Controller | Manage execution state and task flow |
| Query Planner | Decompose questions and generate retrieval plans |
| Retriever | Execute vector, keyword, graph retrieval, and other operations |
| Tool Registry | Register and manage callable tools |
| Evidence Evaluator | Evaluate the relevance and sufficiency of retrieved evidence |
| Memory | Store conversation context and task state |
| Generator | Generate the final answer based on the evidence |
| Guardrails | Control permissions, security boundaries, and execution limits |
Among these, the most important change is the introduction of a decision loop.
In traditional RAG, the retrieval process is usually predetermined, whereas Agentic RAG can dynamically change subsequent actions based on execution results.
2.2 The Basic Workflow of Agentic RAG
flowchart TD
A["User question"]
B["Agent understands the question"]
C["Determine whether retrieval is needed"]
D{"Is external knowledge needed?"}
E["Generate answer directly"]
F["Create retrieval plan"]
G["Select retrieval tool"]
H["Execute retrieval"]
I["Observe retrieval results"]
J["Evaluate evidence quality"]
K{"Is evidence sufficient?"}
L["Adjust query strategy"]
M["Integrate evidence"]
N["Generate final answer"]
A --> B --> C --> D
D -->|"No"| E
D -->|"Yes"| F --> G --> H --> I --> J --> K
K -->|"No"| L --> G
K -->|"Yes"| M --> N
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#83A995,color:#315D49
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#DCEFE5,stroke:#83A995,color:#315D49
This process demonstrates three core characteristics of Agentic RAG:
- Dynamic planning: decide which retrieval tasks to execute based on the question.
- Feedback-driven: determine the next action based on the results returned by tools.
- Iterative execution: when evidence is insufficient, continue retrieval instead of generating an answer immediately.
Note that Agentic RAG does not mean that all tasks must be completed autonomously by the Agent.
Practical systems usually combine rules, state machines, and model decisions to constrain key steps.
3. Differences Between Traditional RAG and Agentic RAG
The most important difference between the two is not whether they use large language models, but whether the retrieval process has dynamic decision-making capability.
3.1 Architectural differences
Traditional RAG:
flowchart LR
A["User question"]
B["Fixed query processing"]
C["Fixed retrieval strategy"]
D["Top-K recall"]
E["Context construction"]
F["LLM generation"]
G["Final answer"]
A --> B --> C --> D --> E --> F --> G
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#83A995,color:#315D49
Agentic RAG:
flowchart TD
A["User Question"]
B["Agent Planning"]
C["Select Retrieval Strategy"]
D["Call Retrieval Tool"]
E["Observe Results"]
F["Evaluate Quality"]
G{"Does it meet task requirements?"}
H["Re-plan"]
I["Integrate Evidence"]
J["Generate Answer"]
A --> B --> C --> D --> E --> F --> G
G -->|"No"| H --> C
G -->|"Yes"| I --> J
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
style I fill:#DCEFE5,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#83A995,color:#315D49
3.2 Core Differences Comparison
| Comparison Dimension | Traditional RAG | Agentic RAG |
|---|---|---|
| Retrieval strategy | Fixed | Dynamic selection |
| Query processing | Usually single-pass processing | Can be rewritten and decomposed over multiple rounds |
| Task planning | Predefined workflow | Agent-based dynamic planning or hybrid planning |
| Number of retrievals | Usually once or a fixed number of times | Dynamically determined based on the task |
| Evidence evaluation | Usually relies on fixed ranking | An independent evaluation step can be added |
| Tool invocation | Fixed retrieval components | Multiple tools can be selected |
| Failure handling | Predefined fallback strategy | Can adjust based on error feedback |
| Complex questions | Relies on single-retrieval performance | Tasks can be split and solved step by step |
| Execution cost | Relatively controllable | May be higher |
| System complexity | Relatively simple | Higher |
3.3 Agentic RAG Is Not Necessarily Better Than Traditional RAG
Agentic RAG adds model calls, task planning, tool scheduling, and feedback evaluation.
This means it may bring:
- More Token consumption.
- Higher response latency.
- More model calls.
- More complex state management.
- Less predictable execution paths.
- Higher system debugging costs.
For simple questions, traditional RAG is usually more efficient.
For example:
What is the company's employee annual leave policy?
If there is a clear annual leave policy document in the knowledge base, you can simply retrieve it and generate an answer; there is no need to start a complex Agent loop.
Therefore, the truly reasonable design is:
Use fixed RAG for simple questions and Agentic RAG for complex questions.
IV. Adaptive Retrieval
Adaptive retrieval is one of the core capabilities of Agentic RAG.
It emphasizes dynamically adjusting retrieval behavior based on question characteristics, retrieval results, and execution status.
Traditional retrieval usually uses fixed parameters:
- Fixed Top-K.
- Fixed retriever.
- Fixed query expression.
- Fixed recall strategy.
- Fixed reranking pipeline.
Adaptive retrieval, by contrast, can adjust these parameters and strategies according to the actual situation.
4.1 Core Dimensions of Adaptive Retrieval
| Adaptive Dimension | What Is Adjusted |
|---|---|
| Retrieval decision | Whether retrieval is needed |
| Retrieval depth | Single-round retrieval or multi-round retrieval |
| Retrieval method | Vector, keyword, graph, or hybrid retrieval |
| Query strategy | Original query, query rewriting, multi-query |
| Top-K | Dynamically adjust the number of retrieved results |
| Retrieval scope | Specified documents, knowledge base, or external sources |
| Reranking | Whether to enable Reranker |
| Evidence evaluation | Determine whether evidence is relevant, sufficient, and credible |
| Stopping condition | Decide whether to continue searching |
4.2 Retrieval Decision: Is Retrieval Needed?
Not all questions require RAG.
You can determine whether retrieval is needed based on the type of question, the model’s existing knowledge, real-time requirements, and dependence on external data.
For example:
| User question | Retrieval strategy |
|---|---|
| What is HTTP? | Can be answered directly |
| What are the company’s internal reimbursement standards? | Must retrieve from the internal knowledge base |
| What is the USD-to-RMB exchange rate today? | Requires a real-time data tool |
| Analyze this technical document | Requires reading the document and retrieving related content |
| Compare the latest prices of multiple models | Requires retrieving the latest public information |
Two cases need to be distinguished here:
- The model can answer based on its existing knowledge.
- The model has sufficiently reliable and currently valid evidence to answer.
These two are not the same.
For internal knowledge, real-time information, business data, and high-risk facts, retrieval must not be skipped based solely on the model’s subjective judgment.
4.3 Dynamic Selection of Retrieval Strategies
Different questions are suited to different retrieval methods.
| Retrieval method | Applicable scenarios |
|---|---|
| Dense Retrieval | Semantically similar, natural-language questions |
| BM25 | Keywords, identifiers, proper nouns |
| Hybrid Search | Requires both semantics and keywords |
| Reranker | Improves ranking quality after initial retrieval |
| Multi-Query | The original query is insufficiently expressive |
| HyDE | Large expression gap between the query and target documents |
| Graph Retrieval | Multi-entity, multi-hop relationships |
| Web Search | Requires public, real-time information |
| SQL | Precise querying of structured business data |
The Agent can select one or more retrievers based on the question.
For example:
The user asks:
Why has an order not been shipped yet?
The system may need to:
- Query the order status through SQL.
- Retrieve logistics exception records.
- Retrieve order processing rules.
- Correlate the results based on time and business status.
- Synthesize the evidence to explain the cause.
Using vector retrieval alone makes it difficult to complete this kind of cross-data-source query.
4.4 Dynamic Top-K
Traditional RAG often uses a fixed Top-K parameter.
For example:
retriever.search(
query=query,
top_k=5
)
However, the amount of evidence required can vary greatly across different questions.
- Simple fact queries: may require only 2–3 chunks.
- Multi-condition questions: may require 5–10 chunks.
- Multi-entity analysis: may require more scattered evidence.
- Comprehensive analysis of long documents: may require staged retrieval.
Adaptive retrieval can dynamically adjust Top-K based on evidence quality.
def adaptive_retrieve(query, retriever):
top_k = 3
max_top_k = 15
while top_k <= max_top_k:
results = retriever.search(
query=query,
top_k=top_k
)
score = evaluate_evidence(results)
if score.is_sufficient:
return results
top_k += 3
return results
This example is only a simplified version of the logic.
A real system cannot determine evidence sufficiency solely based on a vector similarity threshold; it also needs to consider:
- Whether it covers the key conditions in the question.
- Whether there is mutually independent evidence.
- Whether there are conflicts between sources.
- Whether it contains outdated information.
- Whether it can support the final conclusion.
4.5 The Complete Process of Adaptive Retrieval
flowchart TD
A["Receive Question"]
B["Question Classification"]
C{"Need External Evidence?"}
D["Answer Directly"]
E["Select Retrieval Strategy"]
F["Execute Initial Retrieval"]
G["Evaluate Recall Results"]
H{"Sufficient Evidence?"}
I["Rewrite Query"]
J["Adjust Top-K"]
K["Switch Retriever"]
L["Execute Second Retrieval"]
M["Evidence Integration"]
N["Generate Answer"]
A --> B --> C
C -->|"No"| D
C -->|"Yes"| E --> F --> G --> H
H -->|"No"| I
H -->|"No"| J
H -->|"No"| K
I --> L
J --> L
K --> L
L --> G
H -->|"Yes"| M --> N
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#DCEFE5,stroke:#83A995,color:#315D49
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style M fill:#DCEFE5,stroke:#83A995,color:#315D49
style N fill:#DCEFE5,stroke:#83A995,color:#315D49
You need to add a maximum iteration count, a timeout, and cost limits, so that retrieval does not run indefinitely when evidence is insufficient.
V. Core Patterns of Agentic RAG
Agentic RAG is not a single architecture.
Depending on how tasks are planned and executed, a variety of retrieval patterns can be designed.
5.1 Query Routing
Query Routing is the most basic Agentic RAG pattern.
The system first analyzes the question, then selects the appropriate data source or retriever.
For example:
- Technical questions: technical knowledge base.
- Enterprise policy questions: internal document library.
- Order questions: business database.
- Latest industry information: Web Search.
- Multi-entity relationship questions: Graph Database.
flowchart TD
A["User Question"]
B["Question Classification and Intent Recognition"]
C{"Select Data Source"}
D["Technical Knowledge Base"]
E["Enterprise Document Library"]
F["Business Database"]
G["External Search"]
H["Retrieval Results"]
I["Unified Result Processing"]
J["Generate Answer"]
A --> B --> C
C -->|"Technical question"| D
C -->|"Policy question"| E
C -->|"Business question"| F
C -->|"Real-time question"| G
D --> H
E --> H
F --> H
G --> H
H --> I --> J
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#DCEFE5,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#83A995,color:#315D49
Query routing does not necessarily require a complex Agent.
For systems with clear business scenarios and stable data sources, rule-based routing is usually more controllable.
When question semantics are complex and there are many data sources, you can let an LLM assist with routing decisions.
5.2 Query Rewriting
A question as the user phrases it is not necessarily suitable for direct retrieval.
For example:
Why is it so slow?
The system lacks a clear retrieval target.
If the preceding discussion was about a certain database, the question can be rewritten using the conversational context as:
What are the common causes of response latency in PostgreSQL under high-concurrency write scenarios?
The goal of query rewriting is to transform a natural-language question into an expression better suited to retrieval.
Common approaches:
- Supply context.
- Eliminate references.
- Add key entities.
- Clarify the query scope.
- Add time and conditions.
- Convert a complex question into multiple retrieval queries.
The risk of query rewriting is altering the user's original intent.
Therefore, the system should retain the original question and treat the rewritten result as retrieval support information, rather than directly replacing the user's real need.
5.3 Multi-Query
Multi-Query generates several differently phrased queries from one question to improve recall coverage.
For example:
Original question:
How does RAG reduce hallucinations in large models?
It can generate:
Query 1: How does RAG reduce model hallucinations through external knowledge?
Query 2: How does retrieval-augmented generation constrain the evidence a model's answers are based on?
Query 3: How can knowledge bases and evidence citations be used to improve answer credibility?
Query 4: What limitations does RAG have regarding factual consistency and answer verifiability?
After that, retrieval is executed in parallel, and the results are deduplicated, fused, and ranked.
flowchart TD
A["Original question"]
B["Generate multiple queries"]
C["Query one"]
D["Query two"]
E["Query three"]
F["Parallel retrieval"]
G["Result one"]
H["Result two"]
I["Result three"]
J["Deduplicate results"]
K["Fuse results"]
L["Reranker ranking"]
M["Final evidence"]
A --> B
B --> C
B --> D
B --> E
C --> F
D --> F
E --> F
F --> G
F --> H
F --> I
G --> J
H --> J
I --> J
J --> K --> L --> M
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style M fill:#DCEFE5,stroke:#83A995,color:#315D49
The advantage of Multi-Query is that it broadens semantic coverage.
But a larger number of queries also brings more noise, duplicate content, and retrieval cost.
5.4 Decomposition
Complex questions often contain multiple pieces of information that need to be retrieved separately.
For example:
Compare the message-handling mechanisms, reliability, and applicable scenarios of Redis, Kafka, and RabbitMQ.
This can be split into:
- Redis's message-handling mechanism.
- Kafka's message-handling mechanism.
- RabbitMQ's message-handling mechanism.
- A comparison of the three in terms of message reliability.
- The typical applicable scenarios for each of the three.
- Forming a selection recommendation based on business constraints.
Each sub-question is retrieved independently and finally aggregated.
flowchart TD
A["Complex question"]
B["Task decomposition"]
C["Sub-question 1: Redis"]
D["Sub-question 2: Kafka"]
E["Sub-question 3: RabbitMQ"]
F["Sub-question 4: Reliability comparison"]
G["Sub-question 5: Scenario analysis"]
H["Parallel retrieval"]
I["Subtask results"]
J["Result integration"]
K["Generate comprehensive answer"]
A --> B
B --> C
B --> D
B --> E
B --> F
B --> G
C --> H
D --> H
E --> H
F --> H
G --> H
H --> I --> J --> K
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#83A995,color:#315D49
style K fill:#DCEFE5,stroke:#83A995,color:#315D49
Question decomposition needs to preserve the dependencies between subtasks.
For example, selection recommendations must be built on the preceding analysis of mechanisms, performance, and business conditions.
You cannot simply split the question into multiple independent queries and then merge the results unconditionally.
5.5 Iterative Retrieval: Iterative Retrieval
Iterative retrieval is an important capability of Agentic RAG.
Instead of generating an answer immediately after a single retrieval, it judges based on the current evidence whether further retrieval is needed.
For example:
The first round of retrieval finds:
- The message reliability mechanism of Kafka has been found.
- The message acknowledgment mechanism of RabbitMQ has been found.
- The persistence mechanism of Redis Streams is missing.
The system can initiate a second round of retrieval for the missing information.
After the second round is complete, it then evaluates whether the overall evidence is sufficient.
5.6 Corrective RAG: Corrective Retrieval
Corrective RAG (CRAG) emphasizes evaluating retrieval results and taking corrective measures based on the evaluation results.
Its core idea is:
- When retrieval results are relevant, continue using the evidence.
- When retrieval results are irrelevant, retrieve again.
- When internal knowledge is insufficient, other information sources can be sought according to the strategy.
- Filter low-quality evidence to avoid passing it directly to the generation model.
It is necessary to distinguish: CRAG is a design approach for correcting retrieval quality; it is not equivalent to all Agentic RAG systems, nor does it mean that web search is necessarily required.
5.7 Self-RAG: Self-Reflective Retrieval
Through the model's self-evaluation mechanism, Self-RAG determines when to retrieve, how to use retrieval results, and whether the generated content is supported by evidence.
Its focus includes:
- Whether retrieval is needed.
- Whether the retrieval results are relevant.
- Whether the generated content is supported by evidence.
- Whether the answer meets the requirements of the question.
Self-RAG and ordinary Agentic RAG have different emphases:
- Agentic RAG places more emphasis on the Agent's planning, tool invocation, and execution control.
- Self-RAG places more emphasis on retrieval decisions and self-reflection during the generation process.
The two can be combined, but they cannot simply be regarded as the same method.
6. Evidence Evaluation: How Does an Agent Judge Whether Retrieval Results Are Reliable?
Whether Agentic RAG can truly outperform traditional RAG depends to a large extent on evidence evaluation capability.
If the system cannot judge whether the current retrieval results are reliable enough, then even with multiple rounds of retrieval, it may just repeatedly search.
6.1 Core Dimensions of Evidence Evaluation
| Evaluation Dimension | Question to Determine |
|---|---|
| Relevance | Is the evidence relevant to the question? |
| Coverage | Does it cover the key conditions in the question? |
| Sufficiency | Is the current evidence sufficient to answer the question? |
| Consistency | Are there conflicts between different pieces of evidence? |
| Freshness | Is the information outdated? |
| Authority | Is the source credible? |
| Traceability | Can the original source be traced? |
6.2 Relevance Does Not Equal Sufficiency
This is a very important concept in Agentic RAG.
Suppose the user asks:
How does Kafka ensure message reliability?
The retrieval results include:
- Kafka's partitioning mechanism.
- Kafka's consumer group mechanism.
- Kafka's message acknowledgment mechanism.
All of these topics are related to Kafka.
However, if key content such as producer acknowledgment, consumer offset commits, and replication mechanisms is missing, the evidence still cannot be considered sufficient.
Therefore, similarity scores cannot simply be used to determine whether a task has been completed.
6.3 Evidence Sufficiency Assessment
The question can be broken down into several necessary information points, and evidence coverage can be checked for each.
For example:
| Necessary information point | Covered |
|---|---|
| Producer acknowledgment mechanism | Yes |
| Broker replication mechanism | Yes |
| Consumer offset commit | No |
| Consumption failure handling | No |
| Duplicate consumption issue | Yes |
At this point, even if multiple relevant documents have been retrieved, a complete technical conclusion cannot be generated directly.
Further retrieval can be initiated for the missing information points.
6.4 Evidence Scoring Model
A comprehensive evidence scoring model can be constructed:
Where:
- R: Relevance.
- C: Coverage.
- S_f: Evidence sufficiency.
- T: Source credibility.
- A: Timeliness.
- w_r,w_c,w_s,w_t,w_a: Corresponding weights.
The weights need to be determined according to the business scenario; it is not appropriate to directly use a single set of universal fixed values.
For example:
- Technical principle Q&A pays more attention to relevance and coverage.
- Financial information pays more attention to source credibility and timeliness.
- Enterprise policy Q&A pays more attention to version validity and source authority.
This scoring model can serve as a reference for engineering design, but scores from different dimensions should not be treated directly as objective probabilities.
6.5 Decision Process for Evidence Assessment
flowchart TD
A["Retrieve search results"]
B["Relevance assessment"]
C{"Is it relevant?"}
D["Filter irrelevant results"]
E["Coverage assessment"]
F{"Are key contents covered?"}
G["Locate missing information"]
H["Supplementary retrieval"]
I["Source credibility and timeliness assessment"]
J{"Is the evidence credible and valid?"}
K["Check evidence conflicts"]
L{"Are there unresolved conflicts?"}
M["Proceed to manual review or retain uncertainty"]
N["Form a usable evidence set"]
O["Pass to the generation model"]
A --> B --> C
C -->|"No"| D
C -->|"Yes"| E
D --> E
E --> F
F -->|"No"| G --> H --> A
F -->|"Yes"| I --> J
J -->|"No"| G
J -->|"Yes"| K --> L
L -->|"Yes"| M
L -->|"No"| N --> O
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F7EAE7,stroke:#C98D7C,color:#754438
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#F7EAE7,stroke:#C98D7C,color:#754438
style N fill:#DCEFE5,stroke:#83A995,color:#315D49
style O fill:#DCEFE5,stroke:#83A995,color:#315D49
In actual engineering, evidence assessment can be performed jointly by rules, cross-encoders, dedicated evaluation models, or LLMs.
For high-risk tasks, one cannot rely solely on the same generation model to self-evaluate its own output.
7. The Execution Mechanism of Agentic RAG: Plan, Act, Observe, Reflect
The way Agentic RAG runs is closely tied to the Agent's execution loop.
A common abstract model is:
- Plan: plan the task.
- Act: execute actions.
- Observe: observe the results.
- Reflect: reflect and adjust.
7.1 Plan: Task Planning
The Agent first understands the user's goal and breaks a complex problem into tasks that can be executed.
For example:
Analyze whether a certain enterprise is suited to using GraphRAG to build a knowledge management system.
This can be broken down into:
- Understand the scale of the enterprise's existing knowledge base.
- Analyze the degree of association between documents.
- Determine whether there is a need for multi-entity, multi-hop relationship queries.
- Analyze the cost of building a knowledge graph.
- Compare traditional RAG with GraphRAG.
- Provide architecture recommendations.
The planning stage needs to clarify:
- Task objectives.
- List of subtasks.
- Task dependencies.
- Required tools.
- Execution order.
- Completion conditions.
7.2 Act: Executing Actions
The Agent invokes different tools according to the plan.
For example:
- Vector Search.
- BM25 Search.
- Graph Query.
- SQL Query.
- Web Search.
- Document Reader.
- Reranker.
Tool invocation usually needs to go through a unified tool registration and parameter validation mechanism.
7.3 Observe: Observing Results
The Agent receives the data returned by the tools.
The observations include:
- Whether the query succeeded.
- How many results were returned.
- Whether the results are empty.
- Whether the results are relevant.
- Whether a timeout occurred.
- Whether there are data conflicts.
- Whether further retrieval is needed.
Successful tool execution does not mean the task has been completed.
For example, a graph database returning 20 relationships does not mean those relationships can answer the user's question.
7.4 Reflect: Reflection and Adjustment
Based on the observations, the Agent judges whether the current strategy is effective.
Possible adjustments include:
- Modify the query conditions.
- Rewrite the keywords.
- Expand the retrieval scope.
- Switch retrievers.
- Adjust Top-K.
- Split subtasks.
- End the current task.
- Return an explanation that the evidence is insufficient.
7.5 The Agent Execution Loop
flowchart TD
A["Receive user goal"]
B["Plan: formulate a plan"]
C["Act: call tools"]
D["Observe: observe results"]
E["Reflect: evaluate execution status"]
F{"Is the task complete?"}
G["Update plan"]
H["Integrate results"]
I["Generate final answer"]
A --> B --> C --> D --> E --> F
F -->|"No"| G --> C
F -->|"Yes"| H --> I
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#83A995,color:#315D49
This loop can continue for multiple rounds, but maximum execution steps, maximum elapsed time, and maximum cost must be set.
Otherwise, the Agent may keep calling tools because it cannot determine the task's completion conditions.
8. System Architecture Design for Agentic RAG
A production-grade Agentic RAG system needs to separate intelligent decision-making from deterministic execution.
The large language model must not be allowed to directly control all underlying resources.
8.1 Layered Architecture
| Architecture Layer | Core Components | Responsibilities |
|---|---|---|
| Access Layer | API Gateway, Session Management | Receive requests and manage user context |
| Agent Layer | Planner, Controller | Task planning and process control |
| Decision Layer | Router, Evaluator | Select retrieval strategies and evaluate evidence |
| Tool Layer | Retriever, SQL, Graph, Web | Execute external operations |
| Knowledge Layer | Vector DB, Graph DB, Document Store | Store knowledge and indexes |
| Generation Layer | LLM, Context Builder | Build context and generate answers |
| Governance Layer | Guardrails, Observability | Permissions, security, monitoring, and auditing |