I. What Is Multimodal RAG?
Traditional RAG (Retrieval-Augmented Generation) mainly handles text data.
A typical scenario: the user asks a question, the system retrieves relevant text snippets from the knowledge base, and passes the retrieval results as context to a large language model to generate an answer.
But in real-world business, knowledge does not always exist in plain text form.
For example:
- Product manuals contain a large number of structural diagrams and equipment photos.
- Medical materials contain text, medical images, and examination reports.
- Video surveillance contains continuous footage, speech, and temporal information.
- Enterprise knowledge bases contain PDFs, Excel files, PPTs, images, and audio recordings.
- Industrial equipment materials contain drawings, instrument readings, and operation videos.
- E-commerce product information contains titles, descriptions, product images, and demo videos.
If only the text is extracted, a large amount of key knowledge may be lost.
Multimodal RAG incorporates data from different modalities into a unified knowledge retrieval system, enabling the system to retrieve based on text, images, audio, video, and other information, and to provide the relevant evidence to a multimodal large model.
1.1 Differences Between Multimodal RAG and Traditional RAG
| Comparison Dimension | Traditional RAG | Multimodal RAG |
|---|---|---|
| Data type | Mainly text | Text, images, audio, video, etc. |
| Document parsing | Text extraction and chunking | Multimodal parsing and content understanding |
| Vector representation | Text embedding | Unimodal and cross-modal embeddings |
| Retrieval method | Keywords, text vectors | Text, image, audio, and cross-modal retrieval |
| Result content | Text chunks | Text, images, audio clips, video frames, etc. |
| Context assembly | Text context | Multimodal context |
| Model capability | Text generation models | Multimodal understanding and generation models |
| Main challenges | Text recall and relevance | Modality alignment, temporal association, cross-modal retrieval |
Multimodal RAG is not simply adding an image field to traditional RAG; rather, it requires redesigning data parsing, knowledge representation, index organization, retrieval fusion, and model input mechanisms.
1.2 Core Goals of Multimodal RAG
It mainly addresses five categories of problems:
- Multimodal knowledge ingestion: the ability to parse data in different formats.
- Unified knowledge representation: establishing associations among text, images, audio, and video.
- Cross-modal retrieval: supporting scenarios such as querying images with text, querying text with images, and querying videos with text.
- Multimodal context fusion: organizing retrieval results into input that the model can understand.
- Traceable evidence: the ability to locate the knowledge source, document location, video timestamp, and original media content.
II. Overall Architecture of Multimodal RAG
Multimodal RAG can be divided into a data processing pipeline and an online retrieval-and-generation pipeline.
The data processing pipeline is responsible for converting raw multimodal data into retrievable knowledge assets.
The online pipeline is responsible for understanding the user's question, retrieving multimodal evidence, organizing context, and calling the model to generate results.
2.1 Overall System Architecture
flowchart TB
A["Multimodal Data Sources"] --> B["Data Ingestion Layer"]
B --> C["Multimodal Parsing Layer"]
C --> D["Content Normalization and Knowledge Extraction"]
D --> E["Multimodal Knowledge Chunking"]
E --> F["Feature Extraction and Embedding"]
E --> G["Metadata and Relationships"]
F --> H["Vector Index"]
G --> I["Structured Index"]
D --> J["Raw Media Storage"]
H --> K["Multimodal Retrieval Service"]
I --> K
J --> K
L["User Query"] --> M["Query Understanding and Modality Recognition"]
M --> K
K --> N["Cross-modal Recall and Result Fusion"]
N --> O["Reranking and Evidence Filtering"]
O --> P["Multimodal Context Assembly"]
P --> Q["Multimodal Large Model"]
Q --> R["Answer and Evidence Return"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style O fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style P fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style Q fill:#EAF3EC,stroke:#83A995,color:#315D49
style R fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2.2 Data Processing Pipeline
The data processing pipeline is offline or near-real-time processing.
Main tasks:
- Acquire raw files.
- Identify data types.
- Extract text and media content.
- Generate multimodal knowledge chunks.
- Extract modality features.
- Build vector indexes.
- Store metadata and media relationships.
After data processing is complete, the knowledge can be used by the online retrieval service.
2.3 Online Retrieval Pipeline
The online pipeline is mainly responsible for:
- Identifying user query intent.
- Determining which modalities the query contains.
- Selecting retrieval strategies.
- Performing multi-path recall.
- Merging and ranking results.
- Retrieving raw media evidence.
- Constructing model context.
- Returning answers and citations.
The online pipeline needs to focus on controlling retrieval latency, context length, and media transmission costs.
3. Multimodal Data Ingestion and Parsing
The first foundational task of multimodal RAG is to transform data from different sources into a unified knowledge processing object.
3.1 Multimodal Data Types
| Modality | Data Format | Main Processing Tasks |
|---|---|---|
| Text | TXT, Markdown, HTML | Text extraction, cleaning, and chunking |
| Image | JPG, PNG, WebP | OCR, image understanding, region detection |
| Audio | WAV, MP3, AAC | Speech recognition, speaker recognition, audio understanding |
| Video | MP4, MOV, WebM | Frame extraction, shot segmentation, speech extraction |
| Table | CSV, Excel | Table structure parsing, field extraction |
| Presentation | PPT, PPTX | Page parsing, text and image extraction |
| PDF documents | Text, layout, image, and table parsing | |
| Structured Data | JSON, database records | Schema parsing, entity and relationship extraction |
Different modalities require different parsing techniques and cannot all be processed with a text extractor.
3.2 Document Parsing
Documents such as PDF, Word, and PPT usually contain multiple types of content:
- Title.
- Body text.
- Tables.
- Images.
- Captions.
- Headers and footers.
- Page layout.
- Section hierarchy.
Parsing needs to preserve the structural relationships between content elements.
For example, an architecture diagram may appear within a section and be accompanied by explanatory text.
If only the image is extracted and the section information is lost, subsequent retrieval will find it difficult to determine what system the image describes.
It is recommended to build a document structure tree:
{
"document_id": "doc_1001",
"title": "System Architecture Design",
"pages": [
{
"page_number": 1,
"sections": [
{
"section_id": "sec_001",
"title": "Overall Architecture",
"blocks": [
{
"block_id": "block_001",
"type": "text",
"content": "The system consists of an access layer, a service layer, and a data layer."
},
{
"block_id": "block_002",
"type": "image",
"asset_id": "asset_001"
}
]
}
]
}
]
}
This preserves the relationships among text, images, and sections.
3.3 Image Parsing
Image processing typically involves multiple stages:
- Image format identification.
- Resolution and dimension checks.
- OCR text recognition.
- Image content understanding.
- Object and region recognition.
- Image description generation.
- Image feature vector generation.
- Metadata storage.
Different technologies handle different tasks.
OCR
Responsible for recognizing text in images, suitable for:
- Screenshots.
- Scanned documents.
- Table images.
- Receipts.
- Text annotations in drawings.
Vision-language model (VLM)
Responsible for understanding the semantic content in images, for example:
- Image description.
- Scene recognition.
- Object relationships.
- Chart interpretation.
- Visual question answering.
Object detection and region recognition
Used to identify specific objects or regions in images.
For example, in images of industrial equipment, it is necessary to locate gauges, buttons, and abnormal components.
Image parsing results can include:
{
"asset_id": "image_001",
"type": "image",
"source_document": "doc_1001",
"description": "Shows a system architecture consisting of an access layer, a business service layer, and a data layer.",
"ocr_text": [
"API Gateway",
"Business Service",
"Database"
],
"objects": [
{
"label": "API Gateway",
"bbox": [0.1, 0.2, 0.3, 0.4]
}
],
"page_number": 3
}
Here, bbox is used to describe the position of the target in the image, and the specific coordinate format needs to be uniformly agreed upon within the system.
3.4 Audio Parsing
Audio data requires consideration of speech content, temporal information, and speaker information.
Common processing:
- Voice activity detection (VAD).
- Automatic speech recognition (ASR).
- Speaker diarization.
- Timestamp extraction.
- Audio event recognition.
- Emotion or tone analysis (as required by the business).
The ASR results should not be saved only as a complete text passage; the time ranges of segments should also be preserved.
For example:
{
"asset_id": "audio_001",
"segments": [
{
"segment_id": "seg_001",
"start": 0.0,
"end": 3.5,
"speaker": "speaker_1",
"text": "Today I will introduce the overall architecture of the system."
},
{
"segment_id": "seg_002",
"start": 3.6,
"end": 8.2,
"speaker": "speaker_1",
"text": "The system mainly consists of three core modules."
}
]
}
Timestamps make it possible to later locate the specific segments in the original audio and video.
3.5 Video Parsing
Video is one of the more complex data types in multimodal RAG.
A video contains not only images but also:
- Consecutive video frames.
- Audio tracks.
- Subtitles.
- Timing information.
- Scene changes.
- Shot transitions.
- Actions and events.
A video cannot simply be treated as a single image, nor can it be handled by extracting only the full speech transcript.
Video parsing typically includes:
Video metadata extraction
- Duration.
- Resolution.
- Frame rate.
- Encoding format.
- Audio track information.
Shot and scene segmentation
Identify different scenes based on changes in the visual content.
Keyframe extraction
Select frames from the video that can represent the content of a scene.
Audio processing
Extract speech, speakers, and audio events.
Timeline alignment
Associate video frames, speech segments, and subtitles on a unified timeline.
Video data can be organized into the following structure:
{
"asset_id": "video_001",
"duration": 120.0,
"scenes": [
{
"scene_id": "scene_001",
"start": 0.0,
"end": 35.0,
"keyframes": [
{
"timestamp": 5.2,
"asset_id": "frame_001"
},
{
"timestamp": 18.5,
"asset_id": "frame_002"
}
],
"transcript": [
{
"start": 2.0,
"end": 8.0,
"text": "First, I will go through the device installation steps."
}
]
}
]
}
Keyframes and speech segments from a video must retain the original video identifier and timestamps; otherwise, retrieval results cannot accurately locate the original content.
4. Multimodal Knowledge Representation and Data Models
One of the core challenges of multimodal RAG is how to organize data from different modalities into knowledge objects that are retrievable, linkable, and traceable.
Converting all data into text does not by itself mean that multimodal knowledge modeling has been accomplished.
4.1 Unified Knowledge Objects
Knowledge can be uniformly abstracted into objects such as Asset, Block, Chunk, Embedding, and Relation.
| Object | Responsibility |
|---|---|
| Asset | Raw file or media asset |
| Block | A content unit within a document |
| Chunk | A retrievable knowledge fragment |
| Embedding | Modality feature vector |
| Relation | Relationships between different knowledge objects |
| Metadata | Attributes such as source, time, and permissions |
For example:
Document
├── Section
│ ├── Text Block
│ ├── Image Block
│ └── Table Block
└── Video Asset
├── Scene
│ ├── Keyframe
│ └── Transcript Segment
└── Audio Segment
This structure preserves both the hierarchical relationships of the content and the cross-modal associations.
4.2 Multimodal Chunk
A traditional text chunk mainly contains a passage of text.
A multimodal chunk, by contrast, can contain different types of content.
For example:
{
"chunk_id": "chunk_001",
"document_id": "doc_1001",
"content_type": "multimodal",
"text": "系统整体架构包含接入层、业务层和数据层。",
"assets": [
{
"asset_id": "image_001",
"type": "image"
}
],
"metadata": {
"page": 3,
"section": "整体架构",
"tenant_id": "tenant_001"
}
}
Note that the logical content of a Chunk and the actual media files should be stored separately.
The Chunk stores the associations, while the original images, audio, and video are stored in object storage or a media storage system.
4.3 Multimodal Relationships
There are numerous associations between different modalities.
For example:
- A text description corresponds to an image.
- An image belongs to a PDF page.
- A video frame belongs to a video scene.
- An audio segment corresponds to a period of time in a video.
- A table belongs to a document section.
- A region in an image corresponds to an OCR result.
These relationships can be modeled as a graph structure:
flowchart TD
A["Document"] --> B["Section"]
B --> C["Text Chunk"]
B --> D["Image Asset"]
B --> E["Table Asset"]
D --> F["Image Embedding"]
D --> G["OCR Result"]
D --> H["Image Caption"]
I["Video Asset"] --> J["Scene"]
J --> K["Keyframe"]
J --> L["Transcript Segment"]
K --> M["Frame Embedding"]
L --> N["Text Embedding"]
G --> O["Cross-modal Relation"]
H --> O
C --> O
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style M fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style N fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style O fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
Relationship modeling can be implemented using a relational database, document database, or graph database; it does not necessarily require introducing a standalone graph database.
4.4 Multimodal Metadata
Multimodal data requires richer metadata than ordinary text.
Images may require:
- Width and height.
- Image format.
- OCR content.
- Image description.
- Image source.
- Page location.
Audio may require:
- Sample rate.
- Number of channels.
- Duration.
- Language.
- Speaker.
- Time segment.
Video may require:
- Video duration.
- Frame rate.
- Scene range.
- Keyframe timestamps.
- Audio track association.
- Subtitle segment.
A unified metadata model can use common fields plus modality-specific extension fields, avoiding forcing all fields into the same simple structure.
5. Multimodal Embedding and Feature Space
The foundation of multimodal retrieval is converting different modalities into computable feature representations.
Traditional text RAG uses text Embedding.
Multimodal RAG, by contrast, may use:
- Text Embedding.
- Image Embedding.
- Audio Embedding.
- Video Frame Embedding.
- Cross-modal Joint Embedding.
5.1 Unimodal Embedding
Each modality can use a dedicated model to extract features.
For example:
Where:
- f_t: text encoder.
- f_i: image encoder.
- f_a: audio encoder.
- f_v: video encoder.
Different encoders may produce vectors of different dimensions and may also reside in completely different feature spaces.
Therefore, you cannot assume that vectors from all modalities can be directly compared using cosine similarity.
5.2 Cross-modal Joint Embedding
Some multimodal models use joint training to map features from different modalities into a shared semantic space.
For example:
If the model has undergone cross-modal contrastive learning, text and image vectors can be compared for similarity in the shared space.
For example:
Text:
A red car parked by the side of the road.
Image:
A photo containing a red car and a road.
The system can find the corresponding image through cross-modal vector similarity.
5.3 Contrastive Learning Mechanism
Cross-modal representation learning often uses contrastive learning.
Core idea:
- Matched text and images are used as positive samples.
- Mismatched text and images are used as negative samples.
- Increase similarity between positive samples.
- Decrease similarity between negative samples.
A common form of contrastive loss:
Where:
- z_t: text vector.
- z_i: matching image vector.
- sim: similarity function.
- \tau: temperature parameter.
This type of training can bring semantically similar content from different modalities closer together.
Note that a shared feature space does not mean all details can be fully preserved. For example, small text, precise numbers, and fine-grained spatial relationships in images may still require OCR or dedicated visual understanding models.
5.4 Can vectors from different modalities be stored together?
This depends on the feature space of the Embedding model.
Common approaches:
Separate vector collections
Different modalities use separate vector collections.
Advantages:
- Dimensions can differ.
- Models can be upgraded independently.
- Retrieval strategies are more flexible.
Shared vector collection
A shared index is only suitable when vector dimensions, distance metrics, and feature spaces are compatible.
Advantages:
- Facilitates unified retrieval.
- Can simplify some index management.
Disadvantages:
- Similarity distributions may differ across modalities.
- There may be recall bias between modalities.
- Model upgrades require handling compatibility.
From an engineering perspective, logically separated indexes with a unified retrieval interface are more recommended.
6. Multimodal Retrieval Strategies
Retrieval in multimodal RAG is no longer limited to text querying text; it also needs to support combined retrieval across different modalities.
6.1 Multimodal Retrieval Matrix
| Query Modality | Target Modality | Typical Scenario |
|---|---|---|
| Text | Text | Query technical documentation |
| Text | Image | Find images based on a description |
| Image | Image | Find similar images |
| Image | Text | Find documentation based on an image |
| Text | Video | Search video clips |
| Video Frame | Text | Find descriptions based on frames |
| Audio | Text | Find meeting materials based on recordings |
| Text | Audio | Find audio clips based on content |
| Text | Table | Query business metrics and reports |
| Multimodal | Multimodal | Comprehensively analyze video, audio, and documents |
Real-world systems do not necessarily need to support all combinations; the choice should be based on business scenarios.
6.2 Text Querying Images
For example:
User:
Find images showing a server room.
Retrieval process:
- Embed the query text.
- Use a cross-modal vector index to search for images.
- Filter by permissions and metadata.
- Return candidate images.
- Use a vision model or reranking model to evaluate relevance.
- Return the images and their source information.
6.3 Text Queries over Images
For example:
The user uploads a system architecture diagram:
What is the request entry point in this diagram?
Two approaches can be used:
Direct Visual Understanding
Input the image into a vision-language model and directly analyze the image content.
Retrieval-Augmented Understanding
First vectorize the image, retrieve relevant architecture documents, then pass the image and the retrieved textual materials together to the model.
The second approach can use external knowledge to supplement information that the image itself cannot provide.
6.4 Text Queries over Video
For example:
User:
Find the segment in the video that introduces the device startup process.
Video retrieval can combine:
- Video keyframes.
- ASR transcript text.
- Scene descriptions.
- Video chapter information.
- Timestamps.
Relying only on video frames may fail to find key content mentioned in the speech, while relying only on ASR may miss the operational steps shown in the visuals.
Therefore, multi-channel recall is needed.
6.5 Multi-Channel Retrieval Fusion
Multimodal retrieval can be performed separately as:
- Text vector retrieval.
- Image vector retrieval.
- Audio text retrieval.
- Video frame retrieval.
- Keyword retrieval.
- Structured condition queries.
Scores returned by different retrieval channels are usually not on the same scale.
Raw scores cannot be added directly.
The following fusion methods can be used:
Rank Fusion
Use methods such as RRF to fuse based on the rankings from different retrieval channels.
Normalized Score Fusion
First convert the scores from different retrieval channels into a comparable range, then perform weighted fusion.
Learning to Rank
Use training data to learn relevance weights for different modalities.
For complex business scenarios, the weights of different retrieval channels can also be dynamically adjusted based on query intent.
6.6 Multimodal Retrieval Execution Flow
flowchart TD
A["User submits query"] --> B["Query modality identification"]
B --> C["Query semantic analysis"]
C --> D["Select retrieval channels"]
D --> E["Text retrieval"]
D --> F["Image retrieval"]
D --> G["Audio retrieval"]
D --> H["Video retrieval"]
D --> I["Structured query"]
E --> J["Candidate result aggregation"]
F --> J
G --> J
H --> J
I --> J
J --> K["Permission and metadata filtering"]
K --> L["Cross-modal score fusion"]
L --> M["Reranking"]
M --> N["Return multimodal evidence"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F
7. Multimodal Reranking and Evidence Selection
Multimodal retrieval usually returns a large number of candidate results.
These results may come from different modalities and may contain duplicates, related items, and conflicts.
Therefore, a second-pass relevance evaluation is required.
7.1 Why Is a Multimodal Reranker Needed?
For example, a user asks:
Show the server deployment topology and explain the connections between the nodes.
The retrieval results may include:
- A passage of text describing the deployment architecture.
- A server topology diagram.
- A network configuration table.
- A video introducing the deployment process.
- An unrelated photo of server equipment.
These results may all contain relevant keywords, but they are not equally helpful for the current question.
A reranker needs to combine query intent with the evidence content to judge relevance.
7.2 Implementation Approaches to Reranking
Text reranking
Feed the query and text snippets into a cross-encoder to assess semantic relevance.
Visual reranking
Feed the query together with the image into a vision-language model to determine whether the image truly satisfies the query conditions.
Cross-modal reranking
Use a model that supports multimodal input such as text and images to evaluate candidate content in a unified way.
Rule-based reranking
Adjust based on source credibility, time, document version, and business priority.
7.3 Evidence Selection Strategy
It is recommended to handle retrieval relevance and evidence quality separately.
Relevance judgment:
- Is it relevant to the question?
- Does it cover the information the user needs?
- Does it contain the key entities?
- Does it contain the necessary temporal or spatial information?
Evidence quality judgment:
- Is the source credible?
- Is it the currently valid version?
- Does it have complete context?
- Can the original content be located?
- Are there any conflicts?
- Has it passed permission checks?
The evidence finally selected should have sufficient relevance and credibility.
8. Multimodal Context Assembly
Retrieved content cannot simply be spliced into the model input indiscriminately.
Multimodal input needs to be organized according to model capabilities, the context window, and task requirements.
8.1 Context Composition
A multimodal context may contain:
System Instructions
System rules and safety constraints
User Query
The user's original question
Text Evidence
Relevant text snippets
Image Evidence
Image content and reference information
Audio Evidence
Audio transcript and time ranges
Video Evidence
Video frames and timestamps
Structured Evidence
Tables, JSON, and structured records
Source Metadata
Source, version, page number, permissions, and other information
Different models may support different input structures, so the gateway needs to adapt to each model's capabilities.
8.2 Image Context
Image context can be organized in several ways:
Pass the original image directly
Suitable for tasks that require recognizing fine image details.
Pass an image description
Suitable for semantic search and general image understanding.
Pass OCR results
Suitable for text, numbers, and table content within images.
Pass image regions
Suitable for tasks that focus only on specific objects or regions.
The different approaches vary in information fidelity and token cost.
If an image contains small fonts, precise numbers, or complex charts, relying solely on automatically generated image descriptions may lose important information.
8.3 Audio Context
Depending on the task, you can choose:
- Raw audio.
- ASR transcript text.
- Speech segments with timestamps.
- Speaker information.
- Audio event descriptions.
For example, meeting Q&A can usually prioritize the transcript and timestamps; if the user asks about tone of voice, background sounds, or a specific audio event, raw audio may be needed.
8.4 Video Context
Video is usually not suitable for passing all raw frames directly into the model.
You can filter first:
- Keyframes.
- Relevant scenes.
- Corresponding speech segments.
- Time ranges.
- Video summaries.
For example:
{
"video_id": "video_001",
"evidence": [
{
"start": 15.0,
"end": 28.0,
"keyframes": [
"frame_001",
"frame_002"
],
"transcript": "The operator begins starting the equipment."
}
]
}
The model can understand the relevant segments from this information without having to process the entire video.
8.5 Context Budget Management
The context cost of a multimodal model comes not only from text tokens but potentially also from image resolution, image tiling, audio length, and the number of video frames.
Budgets therefore need to be allocated across different content types:
- Text token budget.
- Image count and resolution budget.
- Audio duration budget.
- Video frame count budget.
- Total input cost budget.
When the context exceeds the budget, you can:
- Remove duplicate evidence.
- Deprioritize less relevant content.
- Replace some raw content with summaries.
- Reduce the number of video frames.
- Preferentially retain high-credibility evidence.
- Keep raw media for critical regions.
Multimodal content cannot simply be truncated, otherwise the associations between images, video frames, and timestamps may be broken.
9. Generation Mechanism of Multimodal RAG
The multimodal model is responsible for synthesizing the user's question and the retrieved evidence to generate the final answer.
9.1 Model Inputs and Outputs
Inputs may include:
- A text question.
- Relevant text snippets.
- Images.
- Audio.
- Video frames.
- Tables.
- Structured data.
Outputs may include:
- A text answer.
- Image descriptions.
- Structured JSON.
- Timestamp references.
- An evidence list.
- Multimodal content.
Whether a model can directly accept a given medium, whether it supports structured output, and whether it supports mixed input across multiple modalities must all be determined through model capability declarations and actual testing.