Back to series
· 15 min read

Architecture Design for Multimodal RAG


I. What Is Multimodal RAG?

Traditional RAG (Retrieval-Augmented Generation) mainly handles text data.

A typical scenario: the user asks a question, the system retrieves relevant text snippets from the knowledge base, and passes the retrieval results as context to a large language model to generate an answer.

But in real-world business, knowledge does not always exist in plain text form.

For example:

  • Product manuals contain a large number of structural diagrams and equipment photos.
  • Medical materials contain text, medical images, and examination reports.
  • Video surveillance contains continuous footage, speech, and temporal information.
  • Enterprise knowledge bases contain PDFs, Excel files, PPTs, images, and audio recordings.
  • Industrial equipment materials contain drawings, instrument readings, and operation videos.
  • E-commerce product information contains titles, descriptions, product images, and demo videos.

If only the text is extracted, a large amount of key knowledge may be lost.

Multimodal RAG incorporates data from different modalities into a unified knowledge retrieval system, enabling the system to retrieve based on text, images, audio, video, and other information, and to provide the relevant evidence to a multimodal large model.

1.1 Differences Between Multimodal RAG and Traditional RAG

Comparison Dimension Traditional RAG Multimodal RAG
Data type Mainly text Text, images, audio, video, etc.
Document parsing Text extraction and chunking Multimodal parsing and content understanding
Vector representation Text embedding Unimodal and cross-modal embeddings
Retrieval method Keywords, text vectors Text, image, audio, and cross-modal retrieval
Result content Text chunks Text, images, audio clips, video frames, etc.
Context assembly Text context Multimodal context
Model capability Text generation models Multimodal understanding and generation models
Main challenges Text recall and relevance Modality alignment, temporal association, cross-modal retrieval

Multimodal RAG is not simply adding an image field to traditional RAG; rather, it requires redesigning data parsing, knowledge representation, index organization, retrieval fusion, and model input mechanisms.

1.2 Core Goals of Multimodal RAG

It mainly addresses five categories of problems:

  1. Multimodal knowledge ingestion: the ability to parse data in different formats.
  2. Unified knowledge representation: establishing associations among text, images, audio, and video.
  3. Cross-modal retrieval: supporting scenarios such as querying images with text, querying text with images, and querying videos with text.
  4. Multimodal context fusion: organizing retrieval results into input that the model can understand.
  5. Traceable evidence: the ability to locate the knowledge source, document location, video timestamp, and original media content.

II. Overall Architecture of Multimodal RAG

Multimodal RAG can be divided into a data processing pipeline and an online retrieval-and-generation pipeline.

The data processing pipeline is responsible for converting raw multimodal data into retrievable knowledge assets.

The online pipeline is responsible for understanding the user's question, retrieving multimodal evidence, organizing context, and calling the model to generate results.

2.1 Overall System Architecture

flowchart TB
    A["Multimodal Data Sources"] --> B["Data Ingestion Layer"]
    B --> C["Multimodal Parsing Layer"]
    C --> D["Content Normalization and Knowledge Extraction"]
    D --> E["Multimodal Knowledge Chunking"]
    E --> F["Feature Extraction and Embedding"]
    E --> G["Metadata and Relationships"]
    F --> H["Vector Index"]
    G --> I["Structured Index"]
    D --> J["Raw Media Storage"]
    H --> K["Multimodal Retrieval Service"]
    I --> K
    J --> K

    L["User Query"] --> M["Query Understanding and Modality Recognition"]
    M --> K
    K --> N["Cross-modal Recall and Result Fusion"]
    N --> O["Reranking and Evidence Filtering"]
    O --> P["Multimodal Context Assembly"]
    P --> Q["Multimodal Large Model"]
    Q --> R["Answer and Evidence Return"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style M fill:#EAF3EC,stroke:#83A995,color:#315D49
    style N fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style O fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style P fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style Q fill:#EAF3EC,stroke:#83A995,color:#315D49
    style R fill:#DCEFE5,stroke:#4F8B70,color:#24563F

2.2 Data Processing Pipeline

The data processing pipeline is offline or near-real-time processing.

Main tasks:

  • Acquire raw files.
  • Identify data types.
  • Extract text and media content.
  • Generate multimodal knowledge chunks.
  • Extract modality features.
  • Build vector indexes.
  • Store metadata and media relationships.

After data processing is complete, the knowledge can be used by the online retrieval service.

2.3 Online Retrieval Pipeline

The online pipeline is mainly responsible for:

  • Identifying user query intent.
  • Determining which modalities the query contains.
  • Selecting retrieval strategies.
  • Performing multi-path recall.
  • Merging and ranking results.
  • Retrieving raw media evidence.
  • Constructing model context.
  • Returning answers and citations.

The online pipeline needs to focus on controlling retrieval latency, context length, and media transmission costs.


3. Multimodal Data Ingestion and Parsing

The first foundational task of multimodal RAG is to transform data from different sources into a unified knowledge processing object.

3.1 Multimodal Data Types

Modality Data Format Main Processing Tasks
Text TXT, Markdown, HTML Text extraction, cleaning, and chunking
Image JPG, PNG, WebP OCR, image understanding, region detection
Audio WAV, MP3, AAC Speech recognition, speaker recognition, audio understanding
Video MP4, MOV, WebM Frame extraction, shot segmentation, speech extraction
Table CSV, Excel Table structure parsing, field extraction
Presentation PPT, PPTX Page parsing, text and image extraction
PDF PDF documents Text, layout, image, and table parsing
Structured Data JSON, database records Schema parsing, entity and relationship extraction

Different modalities require different parsing techniques and cannot all be processed with a text extractor.

3.2 Document Parsing

Documents such as PDF, Word, and PPT usually contain multiple types of content:

  • Title.
  • Body text.
  • Tables.
  • Images.
  • Captions.
  • Headers and footers.
  • Page layout.
  • Section hierarchy.

Parsing needs to preserve the structural relationships between content elements.

For example, an architecture diagram may appear within a section and be accompanied by explanatory text.

If only the image is extracted and the section information is lost, subsequent retrieval will find it difficult to determine what system the image describes.

It is recommended to build a document structure tree:

{
  "document_id": "doc_1001",
  "title": "System Architecture Design",
  "pages": [
    {
      "page_number": 1,
      "sections": [
        {
          "section_id": "sec_001",
          "title": "Overall Architecture",
          "blocks": [
            {
              "block_id": "block_001",
              "type": "text",
              "content": "The system consists of an access layer, a service layer, and a data layer."
            },
            {
              "block_id": "block_002",
              "type": "image",
              "asset_id": "asset_001"
            }
          ]
        }
      ]
    }
  ]
}

This preserves the relationships among text, images, and sections.

3.3 Image Parsing

Image processing typically involves multiple stages:

  1. Image format identification.
  2. Resolution and dimension checks.
  3. OCR text recognition.
  4. Image content understanding.
  5. Object and region recognition.
  6. Image description generation.
  7. Image feature vector generation.
  8. Metadata storage.

Different technologies handle different tasks.

OCR

Responsible for recognizing text in images, suitable for:

  • Screenshots.
  • Scanned documents.
  • Table images.
  • Receipts.
  • Text annotations in drawings.

Vision-language model (VLM)

Responsible for understanding the semantic content in images, for example:

  • Image description.
  • Scene recognition.
  • Object relationships.
  • Chart interpretation.
  • Visual question answering.

Object detection and region recognition

Used to identify specific objects or regions in images.

For example, in images of industrial equipment, it is necessary to locate gauges, buttons, and abnormal components.

Image parsing results can include:

{
  "asset_id": "image_001",
  "type": "image",
  "source_document": "doc_1001",
  "description": "Shows a system architecture consisting of an access layer, a business service layer, and a data layer.",
  "ocr_text": [
    "API Gateway",
    "Business Service",
    "Database"
  ],
  "objects": [
    {
      "label": "API Gateway",
      "bbox": [0.1, 0.2, 0.3, 0.4]
    }
  ],
  "page_number": 3
}

Here, bbox is used to describe the position of the target in the image, and the specific coordinate format needs to be uniformly agreed upon within the system.

3.4 Audio Parsing

Audio data requires consideration of speech content, temporal information, and speaker information.

Common processing:

  • Voice activity detection (VAD).
  • Automatic speech recognition (ASR).
  • Speaker diarization.
  • Timestamp extraction.
  • Audio event recognition.
  • Emotion or tone analysis (as required by the business).

The ASR results should not be saved only as a complete text passage; the time ranges of segments should also be preserved.

For example:

{
  "asset_id": "audio_001",
  "segments": [
    {
      "segment_id": "seg_001",
      "start": 0.0,
      "end": 3.5,
      "speaker": "speaker_1",
      "text": "Today I will introduce the overall architecture of the system."
    },
    {
      "segment_id": "seg_002",
      "start": 3.6,
      "end": 8.2,
      "speaker": "speaker_1",
      "text": "The system mainly consists of three core modules."
    }
  ]
}

Timestamps make it possible to later locate the specific segments in the original audio and video.

3.5 Video Parsing

Video is one of the more complex data types in multimodal RAG.

A video contains not only images but also:

  • Consecutive video frames.
  • Audio tracks.
  • Subtitles.
  • Timing information.
  • Scene changes.
  • Shot transitions.
  • Actions and events.

A video cannot simply be treated as a single image, nor can it be handled by extracting only the full speech transcript.

Video parsing typically includes:

Video metadata extraction

  • Duration.
  • Resolution.
  • Frame rate.
  • Encoding format.
  • Audio track information.

Shot and scene segmentation

Identify different scenes based on changes in the visual content.

Keyframe extraction

Select frames from the video that can represent the content of a scene.

Audio processing

Extract speech, speakers, and audio events.

Timeline alignment

Associate video frames, speech segments, and subtitles on a unified timeline.

Video data can be organized into the following structure:

{
  "asset_id": "video_001",
  "duration": 120.0,
  "scenes": [
    {
      "scene_id": "scene_001",
      "start": 0.0,
      "end": 35.0,
      "keyframes": [
        {
          "timestamp": 5.2,
          "asset_id": "frame_001"
        },
        {
          "timestamp": 18.5,
          "asset_id": "frame_002"
        }
      ],
      "transcript": [
        {
          "start": 2.0,
          "end": 8.0,
          "text": "First, I will go through the device installation steps."
        }
      ]
    }
  ]
}

Keyframes and speech segments from a video must retain the original video identifier and timestamps; otherwise, retrieval results cannot accurately locate the original content.


4. Multimodal Knowledge Representation and Data Models

One of the core challenges of multimodal RAG is how to organize data from different modalities into knowledge objects that are retrievable, linkable, and traceable.

Converting all data into text does not by itself mean that multimodal knowledge modeling has been accomplished.

4.1 Unified Knowledge Objects

Knowledge can be uniformly abstracted into objects such as Asset, Block, Chunk, Embedding, and Relation.

Object Responsibility
Asset Raw file or media asset
Block A content unit within a document
Chunk A retrievable knowledge fragment
Embedding Modality feature vector
Relation Relationships between different knowledge objects
Metadata Attributes such as source, time, and permissions

For example:

Document
  ├── Section
  │    ├── Text Block
  │    ├── Image Block
  │    └── Table Block
  └── Video Asset
       ├── Scene
       │    ├── Keyframe
       │    └── Transcript Segment
       └── Audio Segment

This structure preserves both the hierarchical relationships of the content and the cross-modal associations.

4.2 Multimodal Chunk

A traditional text chunk mainly contains a passage of text.

A multimodal chunk, by contrast, can contain different types of content.

For example:

{
  "chunk_id": "chunk_001",
  "document_id": "doc_1001",
  "content_type": "multimodal",
  "text": "系统整体架构包含接入层、业务层和数据层。",
  "assets": [
    {
      "asset_id": "image_001",
      "type": "image"
    }
  ],
  "metadata": {
    "page": 3,
    "section": "整体架构",
    "tenant_id": "tenant_001"
  }
}

Note that the logical content of a Chunk and the actual media files should be stored separately.

The Chunk stores the associations, while the original images, audio, and video are stored in object storage or a media storage system.

4.3 Multimodal Relationships

There are numerous associations between different modalities.

For example:

  • A text description corresponds to an image.
  • An image belongs to a PDF page.
  • A video frame belongs to a video scene.
  • An audio segment corresponds to a period of time in a video.
  • A table belongs to a document section.
  • A region in an image corresponds to an OCR result.

These relationships can be modeled as a graph structure:

flowchart TD
    A["Document"] --> B["Section"]
    B --> C["Text Chunk"]
    B --> D["Image Asset"]
    B --> E["Table Asset"]
    D --> F["Image Embedding"]
    D --> G["OCR Result"]
    D --> H["Image Caption"]
    I["Video Asset"] --> J["Scene"]
    J --> K["Keyframe"]
    J --> L["Transcript Segment"]
    K --> M["Frame Embedding"]
    L --> N["Text Embedding"]
    G --> O["Cross-modal Relation"]
    H --> O
    C --> O

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style M fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style N fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style O fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Relationship modeling can be implemented using a relational database, document database, or graph database; it does not necessarily require introducing a standalone graph database.

4.4 Multimodal Metadata

Multimodal data requires richer metadata than ordinary text.

Images may require:

  • Width and height.
  • Image format.
  • OCR content.
  • Image description.
  • Image source.
  • Page location.

Audio may require:

  • Sample rate.
  • Number of channels.
  • Duration.
  • Language.
  • Speaker.
  • Time segment.

Video may require:

  • Video duration.
  • Frame rate.
  • Scene range.
  • Keyframe timestamps.
  • Audio track association.
  • Subtitle segment.

A unified metadata model can use common fields plus modality-specific extension fields, avoiding forcing all fields into the same simple structure.


5. Multimodal Embedding and Feature Space

The foundation of multimodal retrieval is converting different modalities into computable feature representations.

Traditional text RAG uses text Embedding.

Multimodal RAG, by contrast, may use:

  • Text Embedding.
  • Image Embedding.
  • Audio Embedding.
  • Video Frame Embedding.
  • Cross-modal Joint Embedding.

5.1 Unimodal Embedding

Each modality can use a dedicated model to extract features.

For example:

E_t=f_t(Text)
E_i=f_i(Image)
E_a=f_a(Audio)
E_v=f_v(Video)

Where:

  • f_t: text encoder.
  • f_i: image encoder.
  • f_a: audio encoder.
  • f_v: video encoder.

Different encoders may produce vectors of different dimensions and may also reside in completely different feature spaces.

Therefore, you cannot assume that vectors from all modalities can be directly compared using cosine similarity.

5.2 Cross-modal Joint Embedding

Some multimodal models use joint training to map features from different modalities into a shared semantic space.

For example:

E_t=f_t(Text)
E_i=f_i(Image)

If the model has undergone cross-modal contrastive learning, text and image vectors can be compared for similarity in the shared space.

For example:

Text:
A red car parked by the side of the road.

Image:
A photo containing a red car and a road.

The system can find the corresponding image through cross-modal vector similarity.

5.3 Contrastive Learning Mechanism

Cross-modal representation learning often uses contrastive learning.

Core idea:

  • Matched text and images are used as positive samples.
  • Mismatched text and images are used as negative samples.
  • Increase similarity between positive samples.
  • Decrease similarity between negative samples.

A common form of contrastive loss:

L=-\log \frac{ \exp(sim(z_t,z_i)/\tau) }{ \sum_j\exp(sim(z_t,z_j)/\tau) }

Where:

  • z_t: text vector.
  • z_i: matching image vector.
  • sim: similarity function.
  • \tau: temperature parameter.

This type of training can bring semantically similar content from different modalities closer together.

Note that a shared feature space does not mean all details can be fully preserved. For example, small text, precise numbers, and fine-grained spatial relationships in images may still require OCR or dedicated visual understanding models.

5.4 Can vectors from different modalities be stored together?

This depends on the feature space of the Embedding model.

Common approaches:

Separate vector collections

Different modalities use separate vector collections.

Advantages:

  • Dimensions can differ.
  • Models can be upgraded independently.
  • Retrieval strategies are more flexible.

Shared vector collection

A shared index is only suitable when vector dimensions, distance metrics, and feature spaces are compatible.

Advantages:

  • Facilitates unified retrieval.
  • Can simplify some index management.

Disadvantages:

  • Similarity distributions may differ across modalities.
  • There may be recall bias between modalities.
  • Model upgrades require handling compatibility.

From an engineering perspective, logically separated indexes with a unified retrieval interface are more recommended.


6. Multimodal Retrieval Strategies

Retrieval in multimodal RAG is no longer limited to text querying text; it also needs to support combined retrieval across different modalities.

6.1 Multimodal Retrieval Matrix

Query Modality Target Modality Typical Scenario
Text Text Query technical documentation
Text Image Find images based on a description
Image Image Find similar images
Image Text Find documentation based on an image
Text Video Search video clips
Video Frame Text Find descriptions based on frames
Audio Text Find meeting materials based on recordings
Text Audio Find audio clips based on content
Text Table Query business metrics and reports
Multimodal Multimodal Comprehensively analyze video, audio, and documents

Real-world systems do not necessarily need to support all combinations; the choice should be based on business scenarios.

6.2 Text Querying Images

For example:

User:
Find images showing a server room.

Retrieval process:

  1. Embed the query text.
  2. Use a cross-modal vector index to search for images.
  3. Filter by permissions and metadata.
  4. Return candidate images.
  5. Use a vision model or reranking model to evaluate relevance.
  6. Return the images and their source information.

6.3 Text Queries over Images

For example:

The user uploads a system architecture diagram:
What is the request entry point in this diagram?

Two approaches can be used:

Direct Visual Understanding

Input the image into a vision-language model and directly analyze the image content.

Retrieval-Augmented Understanding

First vectorize the image, retrieve relevant architecture documents, then pass the image and the retrieved textual materials together to the model.

The second approach can use external knowledge to supplement information that the image itself cannot provide.

6.4 Text Queries over Video

For example:

User:
Find the segment in the video that introduces the device startup process.

Video retrieval can combine:

  • Video keyframes.
  • ASR transcript text.
  • Scene descriptions.
  • Video chapter information.
  • Timestamps.

Relying only on video frames may fail to find key content mentioned in the speech, while relying only on ASR may miss the operational steps shown in the visuals.

Therefore, multi-channel recall is needed.

6.5 Multi-Channel Retrieval Fusion

Multimodal retrieval can be performed separately as:

  • Text vector retrieval.
  • Image vector retrieval.
  • Audio text retrieval.
  • Video frame retrieval.
  • Keyword retrieval.
  • Structured condition queries.

Scores returned by different retrieval channels are usually not on the same scale.

Raw scores cannot be added directly.

The following fusion methods can be used:

Rank Fusion

Use methods such as RRF to fuse based on the rankings from different retrieval channels.

RRF(d)= \sum_{i=1}^{n} \frac{1}{k+rank_i(d)}

Normalized Score Fusion

First convert the scores from different retrieval channels into a comparable range, then perform weighted fusion.

Learning to Rank

Use training data to learn relevance weights for different modalities.

For complex business scenarios, the weights of different retrieval channels can also be dynamically adjusted based on query intent.

6.6 Multimodal Retrieval Execution Flow

flowchart TD
    A["User submits query"] --> B["Query modality identification"]
    B --> C["Query semantic analysis"]
    C --> D["Select retrieval channels"]
    D --> E["Text retrieval"]
    D --> F["Image retrieval"]
    D --> G["Audio retrieval"]
    D --> H["Video retrieval"]
    D --> I["Structured query"]
    E --> J["Candidate result aggregation"]
    F --> J
    G --> J
    H --> J
    I --> J
    J --> K["Permission and metadata filtering"]
    K --> L["Cross-modal score fusion"]
    L --> M["Reranking"]
    M --> N["Return multimodal evidence"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style M fill:#EAF3EC,stroke:#83A995,color:#315D49
    style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F

7. Multimodal Reranking and Evidence Selection

Multimodal retrieval usually returns a large number of candidate results.

These results may come from different modalities and may contain duplicates, related items, and conflicts.

Therefore, a second-pass relevance evaluation is required.

7.1 Why Is a Multimodal Reranker Needed?

For example, a user asks:

Show the server deployment topology and explain the connections between the nodes.

The retrieval results may include:

  • A passage of text describing the deployment architecture.
  • A server topology diagram.
  • A network configuration table.
  • A video introducing the deployment process.
  • An unrelated photo of server equipment.

These results may all contain relevant keywords, but they are not equally helpful for the current question.

A reranker needs to combine query intent with the evidence content to judge relevance.

7.2 Implementation Approaches to Reranking

Text reranking

Feed the query and text snippets into a cross-encoder to assess semantic relevance.

Visual reranking

Feed the query together with the image into a vision-language model to determine whether the image truly satisfies the query conditions.

Cross-modal reranking

Use a model that supports multimodal input such as text and images to evaluate candidate content in a unified way.

Rule-based reranking

Adjust based on source credibility, time, document version, and business priority.

7.3 Evidence Selection Strategy

It is recommended to handle retrieval relevance and evidence quality separately.

Relevance judgment:

  • Is it relevant to the question?
  • Does it cover the information the user needs?
  • Does it contain the key entities?
  • Does it contain the necessary temporal or spatial information?

Evidence quality judgment:

  • Is the source credible?
  • Is it the currently valid version?
  • Does it have complete context?
  • Can the original content be located?
  • Are there any conflicts?
  • Has it passed permission checks?

The evidence finally selected should have sufficient relevance and credibility.


8. Multimodal Context Assembly

Retrieved content cannot simply be spliced into the model input indiscriminately.

Multimodal input needs to be organized according to model capabilities, the context window, and task requirements.

8.1 Context Composition

A multimodal context may contain:

System Instructions
    System rules and safety constraints

User Query
    The user's original question

Text Evidence
    Relevant text snippets

Image Evidence
    Image content and reference information

Audio Evidence
    Audio transcript and time ranges

Video Evidence
    Video frames and timestamps

Structured Evidence
    Tables, JSON, and structured records

Source Metadata
    Source, version, page number, permissions, and other information

Different models may support different input structures, so the gateway needs to adapt to each model's capabilities.

8.2 Image Context

Image context can be organized in several ways:

Pass the original image directly

Suitable for tasks that require recognizing fine image details.

Pass an image description

Suitable for semantic search and general image understanding.

Pass OCR results

Suitable for text, numbers, and table content within images.

Pass image regions

Suitable for tasks that focus only on specific objects or regions.

The different approaches vary in information fidelity and token cost.

If an image contains small fonts, precise numbers, or complex charts, relying solely on automatically generated image descriptions may lose important information.

8.3 Audio Context

Depending on the task, you can choose:

  • Raw audio.
  • ASR transcript text.
  • Speech segments with timestamps.
  • Speaker information.
  • Audio event descriptions.

For example, meeting Q&A can usually prioritize the transcript and timestamps; if the user asks about tone of voice, background sounds, or a specific audio event, raw audio may be needed.

8.4 Video Context

Video is usually not suitable for passing all raw frames directly into the model.

You can filter first:

  • Keyframes.
  • Relevant scenes.
  • Corresponding speech segments.
  • Time ranges.
  • Video summaries.

For example:

{
  "video_id": "video_001",
  "evidence": [
    {
      "start": 15.0,
      "end": 28.0,
      "keyframes": [
        "frame_001",
        "frame_002"
      ],
      "transcript": "The operator begins starting the equipment."
    }
  ]
}

The model can understand the relevant segments from this information without having to process the entire video.

8.5 Context Budget Management

The context cost of a multimodal model comes not only from text tokens but potentially also from image resolution, image tiling, audio length, and the number of video frames.

Budgets therefore need to be allocated across different content types:

  • Text token budget.
  • Image count and resolution budget.
  • Audio duration budget.
  • Video frame count budget.
  • Total input cost budget.

When the context exceeds the budget, you can:

  1. Remove duplicate evidence.
  2. Deprioritize less relevant content.
  3. Replace some raw content with summaries.
  4. Reduce the number of video frames.
  5. Preferentially retain high-credibility evidence.
  6. Keep raw media for critical regions.

Multimodal content cannot simply be truncated, otherwise the associations between images, video frames, and timestamps may be broken.


9. Generation Mechanism of Multimodal RAG

The multimodal model is responsible for synthesizing the user's question and the retrieved evidence to generate the final answer.

9.1 Model Inputs and Outputs

Inputs may include:

  • A text question.
  • Relevant text snippets.
  • Images.
  • Audio.
  • Video frames.
  • Tables.
  • Structured data.

Outputs may include:

  • A text answer.
  • Image descriptions.
  • Structured JSON.
  • Timestamp references.
  • An evidence list.
  • Multimodal content.

Whether a model can directly accept a given medium, whether it supports structured output, and whether it supports mixed input across multiple modalities must all be determined through model capability declarations and actual testing.

9.2 Evidence-Constrained Generation