Back to series
· 13 min read

The Complete RAG Workflow: A Full-Pipeline Analysis from Knowledge Ingestion and Vector Retrieval to Answer Generation


I. Overview of the Complete RAG Workflow

RAG (Retrieval-Augmented Generation) is not a simple vector search; it is a complete engineering pipeline that spans from knowledge data processing to model answers.

A complete RAG system usually consists of two stages:

  • Offline knowledge indexing (Indexing Pipeline): responsible for collection, parsing, cleaning, chunking, vectorization, and index building.
  • Online retrieval and generation (Retrieval and Generation Pipeline): responsible for understanding the user's question, retrieving relevant knowledge, building context, and generating an answer.

The two stages are connected through the knowledge base and the index.

1.1 Complete Architecture

flowchart TB
    subgraph OFF["Offline Knowledge Indexing"]
        A["Raw Knowledge Data"]
        B["Data Collection and Connection"]
        C["Document Parsing"]
        D["Text Cleaning and Structuring"]
        E["Chunking Document Splitting"]
        F["Embedding Vectorization"]
        G["Vector Index"]
        H["Text and Metadata Storage"]
        A --> B --> C --> D --> E --> F --> G
        E --> H
    end

    subgraph ON["Online Retrieval and Generation"]
        I["User Question"]
        J["Question Understanding and Rewriting"]
        K["Query Embedding"]
        L["Hybrid Retrieval"]
        M["Candidate Result Filtering"]
        N["Reranking"]
        O["Context Construction"]
        P["Large Language Model"]
        Q["Answer and Citations"]
        I --> J --> K --> L --> M --> N --> O --> P --> Q
    end

    G --> L
    H --> L

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49
    style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style N fill:#EAF3EC,stroke:#83A995,color:#315D49
    style O fill:#EAF3EC,stroke:#83A995,color:#315D49
    style P fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style Q fill:#E5F0E9,stroke:#83A995,color:#315D49

1.2 Responsibilities of the Two Pipelines

Dimension Offline Knowledge Indexing Online Retrieval and Generation
Execution timing When documents are added or updated Every time the user asks a question
Core objective Build a searchable knowledge index Find the evidence needed for the current question
Main input PDF, Word, web pages, databases, etc. User question and conversation context
Core processing Parsing, cleaning, chunking, vectorization Rewriting, retrieval, filtering, ranking
Main output Vector index and knowledge chunks Model answer and citations
Performance focus Throughput, data quality, indexing efficiency Retrieval latency, recall quality, generation speed

The two pipelines should not be conflated.

Document indexing can usually be executed asynchronously, whereas online retrieval must complete within the response-time budget of the user request.


II. Stage One: The Offline Knowledge Indexing Pipeline

The task of the offline stage is to turn raw knowledge into searchable data.

It determines what the RAG system can retrieve, whether the retrieval results are complete, and whether knowledge updates are reliable.

2.1 Step One: Data Collection

Data ingestion is the entry point of knowledge indexing.

Common data sources:

Data Source Data Format Collection Method
Local files PDF, DOCX, TXT, Markdown File upload, directory scanning
Enterprise cloud drives Documents, spreadsheets, attachments API, sync connectors
Websites HTML, web page content Web Crawler, API
Databases Business records, structured data SQL, CDC, business APIs
Code repositories Source code, README, comments Git, code parsers
Knowledge management systems Pages, documents, knowledge entries System API

The data collection layer needs to address several issues:

Data source identification

Every document should have a stable document_id; the file name alone must not be used as the unique identifier.

Incremental sync

You should not re-collect every document each time. Whether an update is needed can be determined based on modification time, version number, content hash, or event notifications.

Permission information

At ingestion time, you need to capture the document's tenant, department, access level, and authorization scope, so they can serve as the basis for permission filtering during retrieval.

Data versioning

The same document may have multiple versions, and it must be possible to identify additions, modifications, and deletions.

It is recommended to record the following for each document:

{
  "document_id": "doc_10001",
  "source_id": "source_001",
  "source_type": "file",
  "file_name": "technical-guide.pdf",
  "content_hash": "sha256-value",
  "version": 3,
  "tenant_id": "tenant_001",
  "updated_at": "2026-11-20T10:00:00Z"
}

Here, content_hash can be used to determine whether the content has actually changed, avoiding repeated parsing and vectorization of documents that have not changed.

2.2 Step 2: Document parsing

The raw files collected usually cannot be used directly for semantic retrieval; they must first be converted into processable content.

Different formats require different parsers.

Document Type Parsing Focus
PDF Page text, reading order, tables, page numbers
DOCX Headings, paragraphs, tables, images
HTML Body text extraction, heading hierarchy, links
Markdown Headings, code blocks, lists, blockquotes
XLSX Worksheets, column names, row data
PPTX Slide titles, body text, notes
Code files File paths, classes, functions, comments

For example, a PDF technical document might contain:

Page 1: Product introduction
Page 2: System architecture
Page 3: Deployment process
Page 4: Configuration parameters
Page 5: Troubleshooting

After parsing, you should preserve as much of the following as possible:

  • The original document ID.
  • Page numbers.
  • Heading hierarchy.
  • Paragraph order.
  • Table structure.
  • Code block boundaries.
  • The association between images and body text.

If you extract only plain text, important structural information may be lost.

For example:

Parameter Type Default Value
timeout integer 30
retry integer 3

If parsing turns this into a continuous string with no column names or boundaries, the model may be unable to accurately understand the relationship between parameters and values.

Scanned PDFs also require OCR; for documents containing a large number of charts, a vision model may be needed to extract chart information.

2.3 Step 3: Text cleaning and structuring

After document parsing, cleaning is usually also required.

Common operations:

  • Remove repeated headers and footers.
  • Strip out navigation bars and advertising content.
  • Fix abnormal line breaks.
  • Handle garbled characters and special characters.
  • Clean up duplicated text.
  • Normalize heading and paragraph formatting.
  • Preserve necessary table and code structures.

Cleaning must not go too far.

For technical documents, the following content usually has high retrieval value:

HTTP 429
ERR_CONNECTION_RESET
PostgreSQL 16.2
timeout = 30
retry_count = 3

If all digits, punctuation, underscores, and special characters are removed, critical technical information may be destroyed.

After structuring, a document can preserve its heading path:

{
  "document_id": "doc_10001",
  "title": "Redis 技术指南",
  "sections": [
    {
      "heading": "缓存机制",
      "subheading": "缓存淘汰策略",
      "content": "Redis 支持多种缓存淘汰策略,包括 LRU 和 LFU。"
    }
  ]
}

The title path can be retained as contextual information during subsequent Chunking.

2.4 Step 4: Chunking Document Splitting

Chunking is the process of splitting a long document into multiple independently retrievable segments.

Each segment is called a Chunk.

Suppose the original document is:

# Redis Caching Mechanisms

## I. Cache Eviction Policies

Redis supports multiple cache eviction policies.
Common policies include LRU, LFU, and random eviction.

## II. Cache Expiration Mechanism

Redis supports setting expiration times for Keys.

It can be split into:

Chunk 1:
Title: Redis Caching Mechanisms
Section: Cache Eviction Policies
Content: Redis supports multiple cache eviction policies.
Common policies include LRU, LFU, and random eviction.

Chunk 2:
Title: Redis Caching Mechanisms
Section: Cache Expiration Mechanism
Content: Redis supports setting an expiration time for a Key.

Each Chunk should have its own unique identifier and retain an association with the original document.

Common Splitting Strategies

Strategy Principle Applicable Scenarios
Fixed length Split by Token count General text
Overlap Adjacent segments retain overlapping content Continuous narrative
Split by paragraph Split at natural paragraph boundaries Articles, reports
Split by heading Split according to section hierarchy Technical documentation, manuals
Semantic splitting Split according to topic changes Long-form natural language
Parent-Child Retrieve with small chunks, supplement context with large chunks Complex knowledge bases

Chunk Size and Overlap

Assume:

  • Chunk Size = 500 Tokens.
  • Chunk Overlap = 100 Tokens.

Then adjacent Chunks will retain 100 Tokens of overlapping content.

flowchart TB
    A["Original Document Tokens 1-1200"] --> B["Chunk 1: 1-500"]
    A --> C["Chunk 2: 401-900"]
    A --> D["Chunk 3: 801-1200"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49

Overlap can reduce semantic loss at boundaries, but it increases the number of vectors and the probability of duplicate retrieval.

Chunk Size is not necessarily better when smaller:

  • Too small: context is incomplete, making it easy to lose qualifying conditions.
  • Too large: retrieval granularity becomes coarser, making it easy to introduce irrelevant content.
  • Reasonable size: can independently express a relatively complete unit of knowledge.

In actual projects, parameters should be adjusted according to the document type and evaluation results.

2.5 Step 5: Generate Embedding Vectors

After Chunking is complete, an Embedding model is needed to convert each text segment into a vector.

Assume:

Redis supports two common cache eviction policies: LRU and LFU.

After passing through the Embedding model:

E(x)=[v_1,v_2,\ldots,v_d]

Where:

  • x: Chunk text.
  • E: Embedding model.
  • d: Vector dimension.
  • v_i: The ith component of the vector.

For example:

E(x)=[0.12,-0.34,0.56,\ldots]

A vector represents semantic features learned by the model, not a list of keywords, and each dimension cannot be directly interpreted as a specific concept.

Embedding Processing Workflow

flowchart TB
    A["Chunk text"] --> B["Embedding model"]
    B --> C["Generate fixed-dimensional vector"]
    C --> D["Attach Chunk ID"]
    D --> E["Prepare to write to vector index"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The following must be ensured:

  • Document vectors and query vectors are in a compatible vector space.
  • Vector dimensions meet the index requirements.
  • The embedding model version is traceable.
  • When the model is changed, it is possible to identify whether old vectors need to be regenerated.
  • Batch vectorization can handle failures and retries.

2.6 Step 6: Save Vectors, Text, and Metadata

After vectors are generated, the data needs to be written to the knowledge base.

A complete record for a Chunk usually contains three parts:

  1. Vector: the vector.
  2. Text: original text or normalized text.
  3. Metadata: document source and business attributes.

Example:

{
  "id": "chunk_10001",
  "vector": [0.12, -0.34, 0.56],
  "text": "Redis 支持 LRU 和 LFU 两种常见缓存淘汰策略。",
  "metadata": {
    "document_id": "doc_10001",
    "title": "Redis 技术指南",
    "section": "缓存淘汰策略",
    "page": 12,
    "tenant_id": "tenant_001",
    "version": 3
  }
}

In actual systems, vectors are usually stored in a vector database or a database that supports vector indexing, while original documents and other business information can also be stored in object storage or a relational database.

Relationships Between Data

flowchart TB
    A["Document"] --> B["Chunk 1"]
    A --> C["Chunk 2"]
    A --> D["Chunk 3"]
    B --> E["Embedding Vector 1"]
    C --> F["Embedding Vector 2"]
    D --> G["Embedding Vector 3"]
    B --> H["Metadata"]
    C --> I["Metadata"]
    D --> J["Metadata"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

2.7 Step 7: Build a Retrieval Index

After knowledge data is written, a retrieval index needs to be built.

Common index types:

  • Flat: exact vector search.
  • HNSW: graph-based approximate nearest neighbor search.
  • IVF: clustering-based inverted index.
  • IVF-PQ: combines clustering and vector quantization.

If hybrid retrieval is used, a keyword index also needs to be built, such as a BM25 inverted index.

At this point, the offline knowledge ingestion process is largely complete.

Note that successfully writing data does not necessarily mean it can be retrieved immediately. Different databases may have different mechanisms for index construction, refresh, and visibility, so index readiness must be handled explicitly.


III. Phase 2: Online Retrieval and Generation Process

When a user asks a question, the system begins executing the online RAG pipeline.

The core goal of the online process is to find the most relevant and trustworthy knowledge that the user has permission to access, within limited latency and context budget.

3.1 Step 1: Receive the User's Question

Suppose the user asks:

How does Redis's LFU eviction strategy work?

The system needs to obtain:

  • The current user identity.
  • The current session ID.
  • The user's question.
  • The knowledge base scope.
  • The model configuration.
  • The conversation history.

Example request:

{
  "conversation_id": "conv_001",
  "query": "How does Redis's LFU eviction policy work?",
  "knowledge_base_id": "kb_001"
}

The user identity and permissions must not be trusted directly from fields in the request body; they should be obtained from authentication information verified on the server side.

3.2 Step 2: Query Processing

The user's original question is not necessarily suitable for direct retrieval.

For example:

How does it evict data?

If the preceding context of the current conversation is about Redis, the system can perform anaphora resolution using the conversation history:

How does Redis's cache eviction policy work?

Common query processing steps include:

  • Text normalization.
  • Keyword extraction.
  • Anaphora resolution.
  • Query rewriting.
  • Multi-query generation.
  • Query classification.
  • Knowledge base routing.

Query Rewrite

The main goal of Query Rewrite is to convert a natural language question into a query expression that is better suited for retrieval.

For example:

Original question Rewritten query
How does it evict data? Redis cache eviction mechanism
What's the difference between LFU and LRU? Differences between Redis LFU and LRU eviction policies
How do I configure it? How to configure Redis maxmemory-policy

Query rewriting can be done with rules, with a model, or with a combination of both.

The original question needs to be preserved, to avoid the rewriting process changing what the user actually wants to ask.

3.3 Step 3: Query Embedding

If vector retrieval is used, the processed query needs to be converted into a vector.

q=E(Q)

Where:

  • Q: the user query.
  • E: the embedding model.
  • q: the query vector.

For example:

How does Redis's LFU eviction policy work?

After embedding, we get:

q=[q_1,q_2,\ldots,q_d]

Only when the query vector and the document vectors generated in the offline stage are in a compatible vector space can meaningful similarity search be performed.

3.4 Step 4: Execute Knowledge Retrieval

There are usually several approaches in the retrieval stage.

Vector retrieval

Find semantically relevant Chunks based on the similarity between the query vector and the document vectors.

Keyword retrieval

Find documents containing key terms using algorithms such as BM25.

Hybrid retrieval

Combine vector retrieval with keyword retrieval to balance semantic expression and exact matching.

For example:

User question:
How does Redis's LFU eviction policy work?

Vector retrieval:
1. Redis LFU mechanism
2. Redis LRU mechanism
3. Redis memory management

Keyword retrieval:
1. LFU eviction algorithm
2. maxmemory-policy
3. Redis high-frequency access statistics

The results of the two retrieval methods can be merged and then passed to the subsequent ranking module.

3.5 Step 5: Metadata Filtering — Permission and Metadata Filtering

The preliminary retrieval results may contain documents that the user should not be able to access, and may also contain content that does not meet the current business conditions.

For example:

tenant_id = tenant_001
document_type = technical
version = 3

Filtering can be based on:

  • Tenant ID.
  • User permissions.
  • Department.
  • Document type.
  • Document version.
  • Time range.
  • Knowledge base scope.

Pay special attention: permission filtering should be performed at the retrieval stage as much as possible, rather than waiting until after the model has generated a response.

If retrieval results are cached, the cache key must also include the necessary tenant and permission context, to prevent retrieval content that should not be shared from being shared between different users.

3.6 Step 6: Reranking

The candidate results obtained in the retrieval stage may not fully match the question.

The Reranker performs a more fine-grained relevance judgment on the candidate documents.

For example:

Candidate content Initial ranking After reranking
Redis LFU mechanism 1 1
Redis LRU mechanism 2 3
Redis memory management 3 2
Redis persistence mechanism 4 4

Reranking can use a Cross-Encoder or other ranking models.

A typical approach is to feed the Query and candidate Chunk into the model together to obtain a relevance score:

s_i=R(Q,D_i)

Where:

  • Q: the query text.
  • D_i: the candidate document.
  • R: the reranking model.
  • s_i: the relevance score.

Finally, select the final candidate results based on the scores.

The cost of a Reranker is usually higher than that of simple vector retrieval, so a common architecture is to first recall a batch of candidates and then rerank a smaller set.

3.7 Step Seven: Context Construction

After retrieval is complete, the final candidate snippets need to be organized into context that the model can understand.

You cannot simply concatenate all retrieval results indiscriminately.

Considerations include:

  • Relevance ordering.
  • Content deduplication.
  • Document source.
  • Section context.
  • Token budget.
  • Time and version information.
  • Whether there are conflicts between contents.

For example, the final retrieved results are:

[Reference 1]
Redis LFU performs approximate eviction based on access frequency.

[Reference 2]
Redis LFU uses an approximate counting mechanism and combines it with time decay to handle access frequency.

[Reference 3]
maxmemory-policy can configure different memory eviction policies.

The system can combine these materials into:

Reference materials:

Reference 1:
Redis LFU performs approximate eviction based on access frequency.

Reference 2:
Redis LFU uses an approximate counting mechanism and combines it with time decay to handle access frequency.

Reference 3:
maxmemory-policy can configure different memory eviction policies.

User question:
How does Redis's LFU eviction strategy work?

3.8 Step Eight: Call the Large Language Model to Generate an Answer

After the context is prepared, the backend calls the large language model.

The prompt usually contains:

  1. System Prompt.
  2. Conversation history.
  3. Retrieved knowledge snippets.
  4. The current user question.
  5. Output format requirements.

For example:

System:
You are a professional technical assistant.

Please follow these rules:
1. Prioritize answering based on the reference materials.
2. Do not fabricate information that does not exist in the materials.
3. If the materials are insufficient, state so clearly.
4. Key conclusions should provide corresponding sources.

Context:
[Reference 1]
Redis LFU performs approximate eviction based on access frequency.

[Reference 2]
Redis LFU uses an approximate counting mechanism and combines it with time decay to handle access frequency.

Question:
How does Redis's LFU eviction strategy work?

The model generates a natural language answer based on this context.

Note that RAG does not mean the model can only use retrieved information. Unless strict constraints and verification mechanisms are adopted, the model may still supplement the content with parametric knowledge.

3.9 Step Nine: Answer Validation and Citation

After the model finishes generating, further processing can be performed:

  • Check whether citation IDs exist.
  • Check whether the citations are within the scope accessible to the current user.
  • Validate the output format.
  • Check for sensitive information.
  • Perform evidence verification on key conclusions.
  • Decide whether returning is allowed based on business rules.

For example:

{
  "answer": "Redis LFU performs approximate eviction based on the access frequency of keys.",
  "citations": [
    {
      "document_id": "doc_10001",
      "chunk_id": "chunk_10002",
      "title": "Redis Technical Guide",
      "page": 12
    }
  ]
}

Citation information should be managed by the server based on actual retrieval results; document IDs or page numbers generated by the model itself should not be fully trusted.

3.10 Step Ten: Return the Answer and Record Logs

The final result is returned to the frontend.

{
  "conversation_id": "conv_001",
  "message_id": "msg_002",
  "answer": "Redis LFU 根据键的访问频率进行近似淘汰。",
  "citations": [
    {
      "title": "Redis 技术指南",
      "page": 12
    }
  ],
  "usage": {
    "input_tokens": 1200,
    "output_tokens": 300
  }
}

The backend also needs to record:

  • Request ID.
  • Query content or a redacted query summary.
  • Number of retrieval hits.
  • Candidate document IDs.
  • Reranker results.
  • Model version.
  • Token usage.
  • Retrieval latency.
  • Model generation latency.
  • Error messages.

This information can be used for subsequent analysis of retrieval quality and system performance.


4. Complete Sequence of Online RAG

The preceding workflow broke down the individual steps from a functional perspective. Below, we examine a complete request from the perspective of system interaction.

4.1 Sequence Diagram

sequenceDiagram
    autonumber
    participant U as User
    participant API as RAG API
    participant Q as Query Processor
    participant E as Embedding
    participant V as Vector Database
    participant R as Reranker
    participant L as LLM
    participant D as Data Store

    U->>API: Send question
    API->>API: Authenticate and authorize
    API->>D: Fetch conversation history
    D-->>API: Return history messages
    API->>Q: Process user question
    Q-->>API: Return retrieval query
    API->>E: Generate query vector
    E-->>API: Return Query Vector
    API->>V: Execute vector or hybrid retrieval
    V-->>API: Return candidate chunks
    API->>API: Metadata and permission filtering
    API->>R: Rerank candidate results
    R-->>API: Return ranked results
    API->>API: Build augmented context
    API->>L: Send prompt
    L-->>API: Return generated result
    API->>API: Validate citations and output
    API->>D: Save messages and citations
    API-->>U: Return answer

4.2 Latency Breakdown of an Online Request

RAG response time is not just model generation time.

It can be abstracted as:

T_{total}= T_{auth}+ T_{query}+ T_{embedding}+ T_{retrieval}+ T_{rerank}+ T_{context}+ T_{generation}+ T_{post}

Where:

  • T_{auth}: authentication and permission checks.
  • T_{query}: question processing and rewriting.
  • T_{embedding}: query vectorization.
  • T_{retrieval}: knowledge retrieval.
  • T_{rerank}: reranking.
  • T_{context}: context construction.
  • T_{generation}: model generation.
  • T_{post}: post-processing and persistence.

In actual implementations, some operations can be executed in parallel, such as vector retrieval and keyword retrieval.

However, dependencies must be preserved: until retrieval results are returned, reranking and final context construction based on those results cannot be completed.