I. What Is a Multimodal Large Language Model?
Traditional language models mainly process text: they split text into tokens, map them into vectors, model contextual relationships through a Transformer, and ultimately predict the next token.
But real-world information is not limited to text; it also includes:
- Image: objects, scenes, spatial relationships, text, and visual details.
- Audio: speech, music, ambient sounds, tone, and emotional cues.
- Video: continuous frames, motion changes, scene transitions, and temporal relationships.
- Spatial data: position, direction, depth, pose, and motion trajectories.
- Text: natural language, code, structured descriptions, and instructions.
A Multimodal Large Language Model (MLLM) can process information from two or more modalities and establish semantic associations between different modalities.
For example, a user uploads an image and asks:
What color is the vehicle in the image? Where is it parked?
The model needs to complete several tasks:
- Identify the vehicle in the image.
- Extract visual attributes such as the vehicle's color.
- Understand the spatial relationship between the vehicle and its surrounding environment.
- Understand the user's text question.
- Associate visual information with the text question.
- Organize a natural language answer.
This is not merely a simple combination of image recognition and text generation, but rather information representation, alignment, and joint reasoning across different modalities.
1.1 Unimodal Models and Multimodal Models
| Comparison Dimension | Unimodal Model | Multimodal Model |
|---|---|---|
| Input data | One primary data type | Multiple data types |
| Data representation | Single-modality encoding | Multi-modality encoding |
| Feature space | Mainly modeled within the same modality | Requires cross-modal association |
| Training objective | Intra-modal prediction or generation | Intra-modal and cross-modal objectives |
| Output form | Usually a single modality | Can support multiple modalities |
| Typical tasks | Text generation, image classification | Image-text question answering, speech dialogue, video understanding |
Note: A multimodal model is not necessarily able to handle all modalities, nor does it mean that all inputs and outputs can be combined arbitrarily. Its specific capabilities depend on the model architecture, training data, and training objectives.
1.2 Core Components of a Multimodal Large Language Model
A typical multimodal large language model usually includes the following components:
- Modality Encoder: Converts raw data such as images, audio, and video into feature representations.
- Projector: Maps features from different modalities into a representation space that the model can process.
- Fusion Module: Establishes information interaction between different modalities.
- LLM Backbone: Uses a Transformer to model context, perform reasoning, and organize output.
- Modality Decoder: Converts the model's internal representations into target modalities such as images and audio.
- Task Control Module: Determines input parsing, modality routing, generation method, and output format based on the task.
Not all models use exactly the same components. Some models directly use a unified Transformer to process multiple modalities, while others perform modality conversion through dedicated encoders and decoders.
II. Overall Workflow of Multimodal Models
At a high level, multimodal models need to address three core questions:
- How can data from different modalities be converted into representations that the model can process?
- How can representations from different modalities be linked?
- How can results in the corresponding modality be generated according to task requirements?
2.1 Typical Processing Architecture
The following is a typical architecture using independent modality encoders and an LLM backbone:
flowchart TD
A["Multimodal input"] --> B["Input type recognition and preprocessing"]
B --> C["Modality routing"]
C --> D["Text encoding"]
C --> E["Image encoding"]
C --> F["Audio encoding"]
C --> G["Video frame and temporal encoding"]
D --> H["Feature projection and representation alignment"]
E --> H
F --> H
G --> H
H --> I["Cross-modal fusion"]
I --> J["Transformer backbone network"]
J --> K["Context understanding and task reasoning"]
K --> L["Output task routing"]
L --> M["Text generation"]
L --> N["Image generation"]
L --> O["Audio generation"]
L --> P["Video or other structured output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style O fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style P fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
This is a conceptual architecture and does not mean that all multimodal models strictly follow this structure.
For example:
- Image understanding models may use a visual encoder to project image features into the input space of a language model.
- Speech dialogue models may use mixed modeling of audio tokens and text tokens.
- Image generation models may use diffusion models or autoregressive visual token decoders.
- Native multimodal models may process sequences of multiple modalities within a unified Transformer.
3. How Are Different Modalities Converted into Tokens?
A key question for multimodal models is: how can a single model process text, images, audio, and video when their data structures are completely different?
The core idea is to convert different modalities into vectors or discrete token sequences, and then feed them into the model through a unified or compatible representation interface.
However, different modalities are not encoded in the same way.
3.1 Text: Token IDs and Embeddings
Text models typically use a tokenizer to split text into tokens and map them to integer IDs.
For example:
Input text:
Multimodal large model
Tokenizer:
["Multi", "modal", "large", "model"]
Token ID:
[1024, 2381, 512, 819]
Then, through an embedding layer, the token IDs are mapped to continuous vectors:
The resulting text vectors can be used as input to the Transformer.
Note that the actual tokenization method of a tokenizer is determined by its vocabulary; the segmentation above is only for illustrating the principle.
3.2 Images: From Pixels to Visual Tokens
The raw data of an image is usually represented as a pixel matrix.
An RGB image can be represented as:
where:
- H: image height.
- W: image width.
- 3: the three RGB color channels.
Directly inputting all pixels into a language model is not realistic, so a vision encoder is usually needed to extract features.
Taking Vision Transformer (ViT) as an example, an image is split into multiple patches.
Suppose the input image size is:
The patch size is:
Then the number of patches is:
Each patch is converted into a vector, then passed through linear projection and positional encoding to form a visual token sequence.
flowchart TD
A["Input RGB image"] --> B["Image size and preprocessing"]
B --> C["Split into patches"]
C --> D["Flatten patches"]
D --> E["Linear projection"]
E --> F["Add positional encoding"]
F --> G["Vision Transformer"]
G --> H["Visual feature sequence"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Patch Embedding can be simplified as:
Where:
- P_i: the ith image patch.
- Flatten: flattens the patch.
- W_p: learnable projection matrix.
- b_p: bias parameter.
- z_i: visual vector corresponding to the patch.
Subsequently, the Vision Transformer models relationships between different patches through Self-Attention.
For example:
- Which regions belong to the same object.
- Spatial relationships between objects.
- Local textures and the overall scene in the image.
- Relationships between text regions and other visual objects.
Note: A visual token is not necessarily equal to a single patch. Actual models may use overlapping patches, multi-scale features, region features, dynamic resolution, cropped tiles, or other visual encoding methods.
For high-resolution images, the model may also split the image into multiple local regions to preserve finer visual information.
3.3 Audio: From Waveform to Audio Tokens
Audio differs from images; it is essentially a signal that changes over time.
Digital audio can be represented as a discrete sample sequence:
where each x_t represents the amplitude at a certain sampling moment.
Audio models usually do not directly process the raw waveform as ordinary text tokens; instead, they extract features through an audio encoder.
Common processing methods include:
- Waveform encoding: directly process the sampled signal.
- Spectral encoding: convert audio into a time-frequency representation.
- Mel Spectrogram: map the spectrum onto the Mel frequency scale.
- Neural audio coding: use neural networks to extract continuous or discrete audio representations.
- Speech recognition encoding: extract features suitable for understanding speech content.
Taking Mel Spectrogram as an example:
flowchart TD
A["Raw audio waveform"] --> B["Framing and windowing"]
B --> C["Short-time Fourier transform STFT"]
C --> D["Power spectrum calculation"]
D --> E["Mel filter bank"]
E --> F["Log compression"]
F --> G["Mel Spectrogram"]
G --> H["Audio encoder"]
H --> I["Audio feature sequence"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The short-time Fourier transform (STFT) is used to analyze the frequency distribution of audio within different time windows:
Where:
- m: the time frame index.
- k: the frequency index.
- N: the window length.
- H: the hop length.
- w[n]: the window function.
The Mel spectrogram can convert audio into time-frequency features that are better suited to perception and model processing.
However, not all multimodal audio models use the Mel spectrogram. Some models use raw waveforms or learned audio encoders directly, while others encode audio into discrete tokens.
3.4 Video: From Continuous Frames to Spatiotemporal Tokens
Video can be understood as image frames arranged in temporal order.
A video can be represented as:
where I_t denotes the t-th video frame.
A video model needs to understand not only the content of each frame, but also the temporal relationships between frames.
A typical processing pipeline includes:
- Video decoding.
- Sampling video frames at time intervals.
- Visually encoding each frame.
- Adding temporal position or timing information.
- Building cross-frame relationships.
- Forming a video feature sequence.
flowchart TD
A["Input video"] --> B["Video decoding"]
B --> C["Sample frames over time"]
C --> D["Visual image encoding"]
D --> E["Add temporal information"]
E --> F["Temporal feature modeling"]
F --> G["Video spatiotemporal representation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Video understanding is fundamentally different from understanding a single image.
For example:
- A single image can identify the posture of a person who is running.
- A video can analyze how that person's motion changes from standing still to starting to run.
- Continuous video can also determine the order in which actions occur, their duration, and changes in the scene.
A video model needs to model spatial and temporal information simultaneously:
where \mathcal{T} denotes temporal information.
Video processing may also use 3D convolutions, spatiotemporal Transformers, sparse frame sampling, keyframe extraction, or video-specific tokenizers, rather than being limited to frame-by-frame image encoding.
IV. Cross-modal Alignment: How Do Different Modalities Establish Semantic Connections?
Converting different modalities into vectors does not mean they naturally share the same semantic space.
For example:
- The "cat" in text is a language token.
- A cat in an image is a visual region made up of a large number of pixels.
- A cat's meow in audio is a segment of acoustic signal.
Their raw data structures differ, and the vector distributions produced by their encoders may also differ.
Multimodal models need to solve the problem of cross-modal alignment.
4.1 What Is Cross-modal Alignment?
The goal of cross-modal alignment is to establish learnable correspondences, in the model's representation space, between information that expresses the same or related semantics across different modalities.
For example:
- Text: "An orange cat is sitting on the sofa."
- Image: a photo of an orange cat sitting on a sofa.
The model needs to learn the correspondence between text semantics and image content.
One common approach is contrastive learning.
4.2 Basic Principles of Contrastive Learning
Taking image-text contrastive learning as an example, features are extracted separately using an image encoder and a text encoder:
The features are then mapped into a comparable vector space and normalized.
Cosine similarity is used to compute the degree of matching between an image and a text:
The training objective is to:
- Increase the similarity of matching image-text pairs.
- Decrease the similarity of non-matching image-text pairs.
For example, a batch contains 4 image-text pairs:
| Image | Text | Matching relationship |
|---|---|---|
| Image A | Description A | Positive sample |
| Image B | Description B | Positive sample |
| Image C | Description C | Positive sample |
| Image D | Description D | Positive sample |
| Image A | Description B | Negative sample |
| Image A | Description C | Negative sample |
The model learns cross-modal correspondences from a large number of positive and negative samples.
A common symmetric contrastive loss can be expressed as:
where the image-to-text loss is:
Where:
- N: number of samples in the batch.
- s(v_i,t_j): similarity between image and text.
- \tau: temperature parameter.
This training approach allows images and text to establish semantic associations in a shared representation space.
CLIP is a representative example of image-text contrastive learning. It learns semantic correspondences between images and text from large-scale image-text paired data.
Note: Contrastive learning is only one method for cross-modal alignment. Multimodal large models may also use matching loss, generative training, cross-attention, supervised fine-tuning, and other methods to establish alignment relationships.
4.3 Alignment Is Not the Same as Being Identical
Cross-modal alignment does not mean making images, text, and audio become completely identical vectors.
Its more important goal is to establish semantic associations.
For example:
- An image of a dog.
- The text “dog”.
- A recording of a dog barking.
The three representations still retain the information characteristics of their respective modalities, but the model can understand the semantic connections between them.
This is also why multimodal models need both shared representations and the retention of modality-specific information.
5. Cross-modal Fusion: How Do Different Modalities Participate Together in Reasoning?
After completing modality encoding and semantic alignment, the model also needs to combine information from different modalities.
This process is called cross-modal fusion.
Common fusion methods mainly fall into three categories.
5.1 Early Fusion
Early fusion usually merges tokens from different modalities into the same sequence at earlier layers of the model.
For example:
Text Tokens:
[T1, T2, T3]
Image Tokens:
[I1, I2, I3, I4]
Fusion sequence:
[T1, T2, T3, I1, I2, I3, I4]
It is then processed by a unified Transformer.
flowchart TD
A["Text Token"] --> C["Unified Sequence Construction"]
B["Image Token"] --> C
C --> D["Unified Transformer"]
D --> E["Cross-modal Self-Attention"]
E --> F["Joint Semantic Representation"]
style A fill:#EAF3EC,stroke:#83A995,color:#315D49
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Advantages:
- Different modalities can directly perform Self-Attention.
- Information exchange between modalities is relatively flexible.
- A unified sequence modeling architecture can be built.
Disadvantages:
- The number of multimodal tokens may be very large.
- Attention computation cost and GPU memory usage are relatively high.
- Sequence lengths differ greatly across modalities, which may affect training efficiency.
5.2 Late Fusion
Late fusion first processes different modalities independently, then fuses the results at the high-level representation or task output stage.
For example:
Image encoder → Image features ┐
├→ Feature fusion → Task output
Text encoder → Text features ┘
Its characteristics are:
- Different modalities can use specialized encoders.
- Feature extraction within each modality is relatively independent.
- Fusion occurs at a higher level.
- It has strong capabilities for independently processing different modalities.
However, if fusion occurs too late, the model may have difficulty fully utilizing fine-grained cross-modal interaction information.
5.3 Intermediate Fusion
Intermediate fusion establishes cross-modal interaction at multiple stages of the encoding process.
For example, image features and text features are first encoded separately, and then information is exchanged through Cross-Attention.
flowchart TD
A["Image input"] --> B["Visual encoder"]
C["Text input"] --> D["Text encoder"]
B --> E["Visual features"]
D --> F["Text features"]
E --> G["Cross-Attention"]
F --> G
G --> H["Cross-modal feature fusion"]
H --> I["Joint reasoning"]
style A fill:#EAF3EC,stroke:#83A995,color:#315D49
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
5.4 How does Cross-Attention achieve cross-modal interaction?
Cross-Attention is an important mechanism for multimodal fusion.
Suppose text features serve as Query, and image features serve as Key and Value:
Then cross-modal attention can be expressed as:
Here:
- Query comes from the text modality.
- Key and Value come from the image modality.
- The Attention weights indicate the degree to which text positions attend to image features.
- The output is a text representation after fusing image information.
For example, the user asks:
What is the object on the left in the image?
The text Query can use the attention mechanism to extract information related to “left” and “object” from the visual features.
The key value of Cross-Attention is that one modality can dynamically extract relevant features based on query information provided by another modality.
However, Attention weights should not be directly interpreted as causal explanations in the human sense or as strict visual grounding results. They are merely an internal information-weighting mechanism of the model.
6. Main Architecture Types of Multimodal Large Models
Multimodal models do not have only one architecture. According to how modalities enter the language model and whether they share a backbone network, they can be roughly divided into the following categories.
6.1 Encoder + Projection Layer + Language Model
This is a relatively common vision-language model architecture.
Typical structure:
- The visual encoder is responsible for extracting image features.
- The projection layer is responsible for dimensionality mapping.
- The language model is responsible for understanding and generating text.
flowchart TD
A["Input image"] --> B["Visual encoder"]
B --> C["Visual features"]
C --> D["Projector layer"]
E["User text"] --> F["Tokenizer and Embedding"]
D --> G["Language model input"]
F --> G
G --> H["LLM Transformer"]
H --> I["Text generation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F