1. What Is a Vision-Language Model?
A Vision-Language Model (VLM) is a type of multimodal model capable of processing images and natural language jointly.
Traditional language models mainly process text, whereas vision-language models can receive both images and textual instructions, and generate language responses based on visual content.
For example, a user uploads an image containing cars, pedestrians, and traffic lights, and asks:
How many cars are in the image? What color is the traffic light? What are the pedestrians doing?
The model needs to:
- Extract visual features from the image.
- Recognize the cars, pedestrians, and traffic lights.
- Understand the spatial relationships between the objects.
- Parse the textual question posed by the user.
- Establish an association between the textual question and the visual features.
- Generate a natural language answer based on the visual information.
This type of task involves visual perception, language understanding, and cross-modal reasoning.
1.1 How VLMs Differ from Traditional Computer Vision Models
Traditional computer vision models are usually designed for well-defined visual tasks, such as image classification, object detection, image segmentation, and OCR.
VLMs, by contrast, further introduce language understanding capabilities, enabling the model to accomplish more flexible visual tasks through natural language instructions.
| Comparison Dimension | Traditional Computer Vision Models | Vision-Language Models |
|---|---|---|
| Input | Images or video | Images and text |
| Output | Classes, coordinates, masks, etc. | Natural language or structured results |
| Task definition | Usually predetermined | Can be specified via natural language instructions |
| Semantic understanding | Focused on visual tasks | Joint visual and linguistic semantics |
| Cross-modal interaction | Usually limited | One of the core capabilities |
| Generalization ability | Influenced by the training task | Usually has stronger task transfer capability |
It should be noted that VLMs do not mean traditional vision models have become obsolete. In tasks such as object detection, industrial defect detection, and fine-grained image segmentation, specialized vision models still hold significant value.
1.2 Core Capabilities of VLMs
Vision-language models can typically support the following tasks:
- Image Captioning: Generate a natural language description based on an image.
- Visual Question Answering (VQA): Answer user questions based on an image.
- Visual Grounding: Locate target regions in an image based on a textual description.
- OCR and document understanding: Recognize text in images and its structural relationships.
- Chart Understanding: Understand the data, trends, and relationships in charts.
- Spatial relationship understanding: Identify the position, orientation, distance, and interrelationships of objects.
- Multi-image comparison: Analyze the differences between different images.
- Visual reasoning: Combine visual evidence to complete multi-step problem analysis.
These capabilities are not necessarily all present in every VLM; this depends on the model architecture, training data, and training objectives.
2. Overall Architecture of VLMs
Many current vision-language models adopt the following basic architecture:
- Vision Encoder: the visual encoder.
- Projector: the visual feature projection layer.
- LLM: the language model.
- Output Head: the output layer.
Its core idea is:
The vision encoder is responsible for understanding the image, the projection layer is responsible for connecting visual features with the language model, and the language model is responsible for understanding, reasoning, and generating in conjunction with user instructions.
2.1 Typical Architecture
```mermaid flowchart TD A["Input Image"] --> B["Image Preprocessing"] B --> C["Vision Encoder"] C --> D["Visual Features"] D --> E["Projector Layer"] F["User Text"] --> G["Tokenizer"] G --> H["Text Embedding"] E --> I["Multimodal Feature Combination"] H --> I I --> J["LLM Transformer"] J --> K["Context Understanding and Visual Reasoning"] K --> L["LM Head"] L --> M["Natural Language Answer"] style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style C fill:#EAF3EC,stroke:#83A995,color:#315D49 style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style H fill:#EAF3EC,stroke:#83A995,color:#315D49 style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F style J fill:#EAF3EC,stroke:#83A995,color:#315D49 style K fill:#EAF3EC,stroke:#83A995,color:#315D49 style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F ``` This is a typical architecture connecting a vision encoder with a language model; it does not mean that all VLMs adopt the same structure. For example, some models use an independent vision encoder and language model that interact through Cross-Attention, while others use a unified multimodal Transformer that directly processes Tokens from different modalities. ### 2.2 Responsibilities of Each Component | Component | English Name | Core Responsibility | |---|---|---| | Vision Encoder | Vision Encoder | Extract visual features from images | | Feature Projector | Projector | Map visual features into the input space of the language model | | Language Model | LLM | Understand text instructions, associate visual information, and perform reasoning | | Output Layer | LM Head | Predict the next text Token based on hidden states | Among these, the Vision Encoder and the Projector are the two components that deserve particular attention when understanding a traditional VLM architecture. ## 3. Vision Encoder: How Does a Visual Encoder Understand an Image? The Vision Encoder is the visual perception module of a VLM. Its main task is to convert a raw image into a feature representation that the model can process. Unlike text, an image is essentially a pixel matrix. A language model cannot directly understand raw pixels, so features must be extracted through a visual encoder. ### 3.1 Basic Representation of an Image An RGB image can be represented as:- P_i: the i-th image patch.
- W_p: learnable projection matrix.
- b_p: bias.
- z_i: Patch Embedding.
Positional encoding is then added so that the model can perceive the spatial position of different patches within the image.
flowchart TD
A["Input Image"] --> B["Image Preprocessing"]
B --> C["Split into Patches"]
C --> D["Patch Embedding"]
D --> E["Add Positional Encoding"]
E --> F["Transformer Encoder"]
F --> G["Visual Feature Sequence"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
3.3 How Does Self-Attention Extract Visual Relationships?
Patch Embedding yields a set of visual tokens.
These tokens need to establish relationships with one another through Self-Attention.
Where:
- Q: Query vector.
- K: Key vector.
- V: Value vector.
- d_k: the dimension of the Key.
Through Self-Attention, the model can establish information associations between different image regions.
For example:
- Determining whether multiple patches belong to the same object.
- Understanding the relationship between an object and its background.
- Identifying the connection between local details and the overall scene.
- Establishing spatial-semantic associations between objects.
Note that the visual encoder extracts feature representations learned by the model; it does not directly produce a human-readable list of objects.
3.4 Why Do Different VLMs Use Different Visual Encoders?
Common visual encoders include:
- ViT: a Transformer-based visual encoder.
- SigLIP: a vision-language encoding approach trained with image-text semantic alignment.
- CLIP Vision Encoder: obtains visual representations through image-text contrastive learning.
- ConvNeXt: a convolutional-network-based visual encoding architecture.
- EVA: obtains visual features through large-scale visual pretraining.
Different visual encoders differ in the following aspects:
- Adaptability to image resolution.
- Ability to extract local details.
- Ability to understand global semantics.
- Pretraining data and objectives.
- Feature dimensionality.
- Computational cost.
When choosing a visual encoder, you need to consider whether its pretraining approach matches the downstream VLM task.
4. Projector: How Do You Connect Visual Features to a Language Model?
The visual features output by the visual encoder cannot necessarily be fed directly into the language model.
The reason is that the visual encoder and the language model usually have different hidden dimensions, representation distributions, and training objectives.
For example:
- Vision Encoder output dimension: 1024.
- LLM hidden dimension: 4096.
The two have inconsistent dimensions and cannot be combined directly in the form of ordinary text embeddings.
The role of the Projector is precisely to accomplish this feature-space mapping.
4.1 Linear Projection
The simplest approach is to use a linear layer:
Where:
- X_v: visual encoder output.
- W_p: projection matrix.
- b_p: bias.
- Z_v: the mapped visual features.
Suppose:
Then:
The projected visual features then have a representation shape aligned with the hidden dimension of the language model.
4.2 MLP Projection
Besides a linear layer, a multilayer perceptron (MLP) can also be used for nonlinear mapping.
For example:
An MLP can learn more complex feature mappings.
Compared with single-layer linear projection, it has stronger nonlinear expressive power, but it also increases the number of parameters and computational overhead.
4.3 The Projector Is More Than Dimensionality Conversion
This is a point that is easy to overlook when understanding VLMs.
The Projector does not merely convert 1024 dimensions into 4096 dimensions.
It also needs to help visual features adapt to the language model's representation space so that the language model can effectively use this visual information.
For example:
- Map visual region features into vectors suitable for processing by the language model.
- Preserve visual semantics relevant to language tasks.
- Establish a learnable connection between visual representations and language representations.
Therefore, the training quality of the Projector directly affects whether visual information can be effectively used by the language model.
4.4 Q-Former: Another Way to Connect Visual Features
BLIP-2 proposed the Q-Former (Querying Transformer) architecture.
Unlike simply projecting all visual features directly into the language model, Q-Former uses a set of learnable queries to extract task-relevant information from the vision encoder output.
Basic process:
- The vision encoder extracts image features.
- A set of learnable queries serves as query input.
- The queries read visual features through cross-attention.
- It outputs a relatively fixed number of compressed visual representations.
- Map the visual representations into the language model space.
flowchart TD
A["Input Image"] --> B["Vision Encoder"]
B --> C["Visual Feature Sequence"]
D["Learnable Queries"] --> E["Q-Former"]
C --> E
E --> F["Visual Information Extraction and Compression"]
F --> G["Feature Projection"]
G --> H["LLM"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The value of Q-Former lies in extracting relatively compact visual representations through a learnable query mechanism, reducing the pressure of directly passing a large number of visual tokens.
5. Visual Tokens: How Do Images Enter the LLM?
Understanding visual tokens is the key to understanding the VLM inference mechanism.
The input to a traditional language model is usually:
Text
↓
Tokenizer
↓
Token ID
↓
Embedding
↓
Transformer
A VLM, by contrast, needs to introduce visual features in addition to text tokens.
5.1 Differences Between Text Tokens and Visual Tokens
| Comparison Dimension | Text Token | Visual Token |
|---|---|---|
| Raw Data | Text | Image |
| Generation Method | Tokenizer | Vision Encoder |
| Basic Representation | Discrete Token ID | Usually Continuous Vectors |
| Vector Source | Token Embedding | Vision Encoder |
| Number | Determined by Text Length | Determined by Resolution, Patch, and Encoding Strategy |
| Semantic Content | Linguistic Information | Visual Information |
There is an important concept here:
Visual tokens are not necessarily discrete integer IDs.
In many VLMs, visual tokens are actually continuous vectors output by the vision encoder.
After passing through the Projector, they are embedded as a set of vectors into the language model's input sequence.
5.2 How Are Text and Visual Tokens Combined?
Suppose the user asks:
What animals are in this picture?
After the text passes through the Tokenizer, we get:
[What, animals, are, in, this, picture]
After the image passes through the vision encoder and projection layer, we get:
[V1, V2, V3, V4, ..., VN]
The model can combine the two representations:
[Text Tokens]
+
[Visual Feature Sequence]
+
[Task Control Tokens]
Conceptually, this can be expressed as:
Where:
- X_{\text{text}}: text features.
- X_{\text{vision}}: visual features.
- [;]: denotes sequence concatenation.
Real models may also add modality markers, separators, positional encodings, and special task tokens.
Note that the actual ordering of the sequence depends on the model design; it is not necessarily always text first and images after.
5.3 Why Does the Number of Visual Tokens Matter?
Suppose an image is encoded into 576 visual tokens, and the user's question contains 30 text tokens.
Then the effective length of the model's input sequence may reach:
If multiple images are provided as input, or if a video contains multiple sampled frames, the number of tokens increases further.
The computational complexity of standard global Self-Attention scales with the square of the sequence length:
Therefore, the number of visual tokens directly affects:
- Prefill computation cost.
- Attention memory usage.
- KV Cache size.
- Inference latency.
- Context window usage.
This is also why modern VLMs adopt techniques such as image slicing, patch merging, visual token compression, and dynamic resolution.
6. How Do VLMs Achieve Cross-Modal Understanding?
A vision encoder can extract image features, but that does not mean the model has already understood the user's question.
The model still needs to establish a connection between the text question and the visual features.
This process is called cross-modal interaction.
6.1 Self-Attention and Cross-Attention
Common ways in which VLMs exchange information across modalities include:
- Enabling vision–text interaction through Self-Attention within a unified sequence.
- Letting text features read visual features through Cross-Attention.
- Exchanging modality information through other dedicated fusion modules.
Take Cross-Attention as an example:
Suppose the text representation serves as the Query and the visual features serve as the Key and Value:
Then:
Here, the text Query can extract relevant information from the visual features according to the current question.
For example:
User question:
What is the object on the table on the left side of the image?
The model needs to focus on:
- The left-side region.
- The region where the table is located.
- The local objects on the table.
- Visual details relevant to the question.
Cross-Attention provides a dynamic information retrieval mechanism that allows language features to read relevant content from visual features according to the question.
6.2 The Complete Process of Visual Question Answering
Take the case where a user asks about an object in an image:
flowchart TD
A["User uploads image"] --> B["Vision Encoder"]
B --> C["Visual feature extraction"]
D["User enters question"] --> E["Tokenizer"]
E --> F["Text feature representation"]
C --> G["Multimodal input fusion"]
F --> G
G --> H["LLM Transformer"]
H --> I["Joint visual and textual reasoning"]
I --> J["Generate answer"]
J --> K["Output natural language"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Here it is necessary to distinguish visual perception from language reasoning:
- The Vision Encoder is primarily responsible for extracting visual information.
- The LLM is responsible for combining the visual representation with text instructions to complete the task.
- The two are connected through projection, fusion, or cross-attention mechanisms.
It cannot simply be assumed that the LLM itself already possesses complete pixel-level visual perception capabilities.
VII. The Training Mechanism of VLMs
The training objective of a VLM is not only to enable the model to recognize images, but also to enable it to understand the relationship between images and language.
A typical training process includes:
- Vision encoder pre-training.
- Image-text alignment training.
- Vision-language connector module training.
- Multimodal instruction fine-tuning.
- Preference alignment and safety training.
Training pipelines differ from model to model; some stages may be merged, and some models directly train the vision encoder and language model jointly.
7.1 Vision Encoder Pre-training
The vision encoder first needs to learn the visual structures and semantic features in images.
Common training approaches include:
- Image classification.
- Self-supervised visual learning.
- Image contrastive learning.
- Masked image modeling.
- Joint image-text pre-training.
For example, ViT can learn visual features from large-scale image data.
If the vision encoder already possesses strong visual representation capabilities, the difficulty of subsequent VLM training can be reduced.
7.2 Image-Text Alignment Training
The goal of image-text alignment training is to establish a semantic connection between images and their corresponding text.
CLIP is a typical representative.
It uses an image encoder and a text encoder to extract features separately:
Then, through contrastive learning, matching image-text features are brought closer together, while non-matching ones are pushed further apart.
Similarity can be measured using cosine similarity:
This approach enables the model to learn the correspondence between images and natural language descriptions.
Note that CLIP itself mainly learns image-text representation alignment; it is not a complete conversational VLM. It typically cannot directly perform open-ended multi-turn image question answering the way generative vision-language models can.
7.3 Vision-Language Connector Module Training
The vision encoder and the language model usually have different representation spaces.
Therefore, connector modules such as a Projector or Q-Former need to be trained so that visual features can be effectively utilized by the language model.
Common training approaches:
- Freeze the vision encoder and language model, and train only the connector module.
- Freeze the vision encoder, and jointly train the Projector and language model.
- Jointly fine-tune the vision encoder, connector module, and language model.
Freezing some modules can reduce training costs and also preserve the capabilities of existing models.
7.4 Multimodal Instruction Fine-Tuning
A vision-language model needs to understand tasks that users pose in natural language.
For example:
| Image Input | User Instruction | Training Objective |
|---|---|---|
| Image of a cat | Describe the content of the image | Generate an image description |
| Traffic scene | Identify the color of the traffic light | Answer a visual question |
| Document image | Extract the text in the image | Output OCR content |
| Mathematical chart | Analyze the data trend | Generate an analysis result |
| Multiple images | Compare the differences between two images | Output a comparison conclusion |
Instruction fine-tuning enables the model to develop from a mere image-text association capability into a model capable of executing natural language visual tasks.
7.5 Training Objectives
A VLM can use multiple loss functions.
For example:
Image-text contrastive loss:
Used to learn the semantic alignment relationship between images and text.
Autoregressive language modeling loss:
Where:
- I: image input.
- X: text instruction.
- y_t: target output token.
- y_{<t}: previously generated tokens.
This objective requires the model to predict the correct output sequence based on the image and text context.
During multi-task training, multiple objectives can be combined:
Where \lambda_1,\lambda_2,\lambda_3 are the weights of the different loss terms.
Actual models do not necessarily use all of the above losses simultaneously; this depends on the training stage and model design.
VIII. The Inference Process of VLMs
Having understood the training mechanism, let us now look at how the model completes a visual question answering task in practical application.
Suppose a user uploads an image and asks:
How many people are in this image? What is each of them doing?
8.1 Processing Steps in the Inference Stage
Step 1: Image preprocessing
The system reads the image and performs operations such as resizing, cropping, and normalization.
Step 2: Visual feature extraction
The Vision Encoder converts the image into a sequence of visual features.
Step 3: Feature Projection
The projector maps visual features into a representation space that the language model can process.
Step 4: Text Encoding
The tokenizer converts the user's question into text tokens, and the embedding layer generates text vectors.
Step 5: Multimodal Feature Combination
Visual features and text features are combined according to the model architecture.
Step 6: Language Model Inference
The Transformer uses the visual information and textual context to generate output representations.
Step 7: Text Generation
The LM head produces a probability distribution over the next token, and a token is then selected according to the decoding strategy.
Step 8: Output the Answer
The model keeps generating until the stopping condition is met.