Back to series
· 14 min read

Comparing GPT, Claude, Gemini, DeepSeek, and Qwen Technology Stacks: Architectures, Training, and Ecosystems of Five Major LLMs


I. Overview of the Five Major Model Systems

The field of large models has already formed several representative technology systems.

They compete not only on model capability, but also continue to evolve in model architecture, training methods, inference optimization, multimodal technology, tool calling, and deployment ecosystems.

Five representative model systems:

Model System Developer Core Positioning Main Technical Directions
GPT OpenAI General-purpose foundation models and agent platform Reasoning, code, multimodality, tool calling
Claude Anthropic General models and complex task execution Deep reasoning, code, long-horizon agents, safety alignment
Gemini Google DeepMind Natively multimodal model system Multimodal understanding, reasoning, long context, agents
DeepSeek DeepSeek High-efficiency models and reasoning technology MoE, MLA, reinforcement learning, inference optimization
Qwen Alibaba Qwen Team Open models and a full-stack multimodal ecosystem Open weights, MoE, vision-language, audio, agents

First, one concept needs to be made clear:

GPT, Claude, Gemini, DeepSeek, and Qwen are not five individual models, but five model families or technology systems.

Each system contains models of different sizes, task orientations, and deployment approaches.

For example:

  • GPT includes models positioned at different reasoning levels, speeds, and costs.
  • Claude includes different series such as Opus, Sonnet, and Haiku.
  • Gemini includes models with different positioning such as Pro and Flash.
  • DeepSeek includes general-purpose, reasoning, and differently sized models.
  • Qwen includes language, vision, audio, code, and multimodal models.

Therefore, when comparing technology systems, one cannot judge an entire model family based solely on the leaderboard results of a single version.

II. GPT: A Model System Built Around General Capabilities and a Tool Ecosystem

2.1 Basic Positioning of GPT

GPT is the name of OpenAI's large language model series. Its technical路线 begins with general language modeling and gradually extends to complex reasoning, multimodal understanding, code generation, and agent tasks.

GPT is commonly understood as Generative Pre-trained Transformer:

  • Generative: generative modeling.
  • Pre-trained: pre-training.
  • Transformer: a model technology lineage developed from the Transformer architecture.

It should be noted that the complete internal architecture of specific GPT models has not been fully disclosed. One cannot infer that all models use exactly the same network structure or training configuration simply because the name contains Transformer.

The GPT technology system can be understood from several aspects.

2.2 Core Technical Directions

1. Autoregressive Language Modeling

GPT's foundational capabilities come from language model training.

For an input sequence:

X=(x_1,x_2,\ldots,x_T)

An autoregressive language model factorizes the joint probability through conditional probabilities:

P(X)=\prod_{t=1}^{T}P(x_t|x_{<t})

The model predicts the next token based on the preceding tokens, learning language patterns, knowledge associations, and contextual representations through large-scale training.

This forms the basis of GPT's text generation capability.

2. Reasoning Capability

The GPT series has gradually strengthened its ability to handle complex tasks, including:

  • Mathematical reasoning.
  • Code analysis and generation.
  • Multi-step problem solving.
  • Scientific and professional knowledge analysis.
  • Planning under multiple constraints.
  • Tool calling and result verification.

Reasoning models typically invest additional compute before generating the final answer, performing more complex internal processing.

Two concepts need to be distinguished:

  • Model scale: how many parameters the model contains.
  • Inference compute: how much compute the model invests when handling a single request.

Reasoning capability is not determined solely by parameter scale; it is also affected by training methods, inference strategies, and compute budget.

3. Multimodal Capability

The GPT technology system is no longer limited to text generation; it also involves:

  • Image understanding.
  • Speech input and output.
  • Real-time voice interaction.
  • Image generation and editing.
  • Video-related understanding and processing.
  • Combining visual information with tool operation.

The goal of multimodal models is to let the model process different types of information, rather than merely converting images or audio into text before handing it to a language model.

However, different GPT models do not have exactly the same input and output modalities and capabilities; the capability documentation for the specific model should be taken as the reference.

4. Tool Calling and Agents

One of GPT's important technical directions is combining language models with external tools.

Depending on the task, the model can invoke:

  • Web Search.
  • Function Calling.
  • Code Interpreter.
  • File retrieval.
  • Data analysis tools.
  • External business APIs.
  • Computer operation tools.

The model no longer merely generates a piece of text; it can participate in a task workflow that includes planning, execution, and verification.

2.3 GPT's Agent Execution Mechanism

flowchart TB
    A["Receive user task"] --> B["Understand goals and constraints"]
    B --> C["Analyze task complexity"]
    C --> D["Formulate execution plan"]
    D --> E["Choose tools or answer directly"]
    E --> F{"Are external tools needed?"}
    F -->|"Yes"| G["Call tool"]
    G --> H["Receive tool results"]
    H --> I["Analyze execution results"]
    I --> J{"Is the task complete?"}
    J -->|"No"| D
    J -->|"Yes"| K["Organize final result"]
    F -->|"No"| K
    K --> L["Return to user"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style L fill:#E5F0E9,stroke:#83A995,color:#315D49

The key to this kind of architecture is not simply adding more tools, but establishing a reliable interaction mechanism between the model and the external execution environment.

The issues that need to be handled include:

  • Whether the tool selection is correct.
  • Whether the parameters conform to the interface definition.
  • How to recover after a tool call fails.
  • How to maintain state across multiple calls.
  • How to validate the data returned by tools.
  • How to prevent the model from performing unauthorized operations.

2.4 The GPT Technology Ecosystem

GPT's advantages come not only from the model itself, but also from its surrounding development platforms.

These mainly include:

  • ChatGPT: a user-facing AI product.
  • OpenAI API: the interface for model invocation and application development.
  • Responses API: supports model responses, tool calls, and multi-step interaction.
  • Agents SDK: an agent development toolkit.
  • Codex: an agent product for software development tasks.
  • GPT series models: general-purpose reasoning, text, code, and multimodal models.

OpenAI's official model documentation has already integrated models, tool calls, reasoning, file processing, and agent development into a unified API system.

As of November 2026, the official model catalog includes models with different positioning, such as GPT-6.2 Astra and GPT-6.2 Sol. Specific capabilities, interfaces, and usage restrictions should be based on the corresponding model documentation.

III. Claude: Centered on Complex Tasks, Code Engineering, and Safety Alignment

3.1 Claude's Basic Positioning

Claude is a series of large language models developed by Anthropic.

Claude's technical approach emphasizes:

  • Complex reasoning.
  • Long-text understanding.
  • Software engineering tasks.
  • Multi-step task execution.
  • Model behavior alignment.
  • Safety and controllability.

Similar to GPT, Claude's complete internal model architecture has not been fully disclosed. Therefore, a technical analysis of Claude should focus on its publicly available training philosophy, capability design, and product interfaces, rather than speculating about undisclosed network structures.

3.2 Constitutional AI

An important technical feature of Claude is Constitutional AI.

Traditional model alignment methods usually rely on human feedback, letting the model learn the response styles that humans prefer.

Constitutional AI, by contrast, introduces a set of explicit principles to guide the model in evaluating and improving its own responses.

The core idea is:

  1. Formulate a set of behavioral principles.
  2. Have the model check its own responses against the principles.
  3. Generate critiques and revisions based on the principles.
  4. Use these results to train the model.
  5. Further improve model behavior through subsequent reinforcement learning.

Its basic process can be abstracted as:

flowchart TB
    A["Input training questions"] --> B["Model generates initial answer"]
    B --> C["Evaluate against behavioral principles"]
    C --> D["Identify issues in the answer"]
    D --> E["Generate revised answer"]
    E --> F["Construct training feedback"]
    F --> G["Run model training"]
    G --> H["Improve model behavior"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

The behavioral principles here can include:

  • Helpful.
  • Honest.
  • Harmless.
  • Respect users.
  • Avoid assisting with illegal or harmful behavior.
  • Appropriately express limitations when uncertain.

Constitutional AI does not mean that a model can automatically guarantee safety, nor does it mean that all answers strictly comply with the principles. It is a training and alignment method used to improve model behavior.

Anthropic has publicly shared the Constitutional AI research method and continues to publish Claude's behavioral principles and related documentation.

3.3 Claude's Model Tiers

Claude uses model families with different positioning to meet different performance and cost requirements.

The traditional naming system mainly includes:

Series Design positioning Typical tasks
Opus High-capability model Complex reasoning, software engineering, long-horizon tasks
Sonnet Balanced capability and speed Daily development, analysis, knowledge work
Haiku High-efficiency model Classification, extraction, fast responses

As of November 2026, the official model catalog also includes new series such as Fable; the earlier three-tier naming system should not be regarded as a permanently fixed product structure.

3.4 Claude's Software Engineering Capabilities

One important application area for Claude is complex software engineering.

Typical tasks include:

  • Read large codebases.
  • Understand dependencies between modules.
  • Locate complex bugs.
  • Design refactoring plans.
  • Modify multiple files.
  • Write tests.
  • Analyze the causes of test failures.
  • Iteratively fix issues.

Unlike simple code generation, complex software engineering tasks require the model to understand longer contexts and maintain consistent task goals across multiple steps.

For example:

flowchart TB
    A["Receive software development requirements"] --> B["Understand codebase structure"]
    B --> C["Locate relevant modules"]
    C --> D["Analyze dependencies"]
    D --> E["Develop a modification plan"]
    E --> F["Modify code"]
    F --> G["Run tests"]
    G --> H{"Do tests pass?"}
    H -->|"No"| I["Analyze errors and fix"]
    I --> G
    H -->|"Yes"| J["Check code quality"]
    J --> K["Submit changes"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#E5F0E9,stroke:#83A995,color:#315D49

These tasks place higher demands on the model:

  • Long-context information retrieval.
  • Code semantic understanding.
  • Cross-file dependency analysis.
  • Multi-step planning.
  • Tool invocation.
  • Verification of execution results.

3.5 Claude's Technical Ecosystem

Claude's main ecosystem includes:

  • Claude: a user-facing AI assistant.
  • Claude API: a model development interface.
  • Claude Code: an agent tool for software development.
  • Claude Platform: a platform for model development and management.
  • MCP: Model Context Protocol, used to connect models with external tools, data sources, and services.

It is important to note that MCP is not a model architecture exclusive to Claude, but an open protocol that can be adopted by different models and applications.

What it addresses is the question of how model applications can connect to external resources in a relatively standardized way.

IV. Gemini: Centered on Native Multimodality and Google's Full-Stack Ecosystem

4.1 Gemini's Basic Positioning

Gemini is a series of large models developed by Google DeepMind.

Its key technical direction is native multimodality.

Traditional multimodal systems often use multiple independent models to process text, images, and audio separately, and then combine them into a single system through interfaces.

Gemini's technical direction instead emphasizes considering multiple modalities during the model training and design stages, enabling the model to process different types of information and to perform understanding and reasoning within a unified model system.

When Gemini was first released, Google emphasized the native multimodal training approach.

4.2 What Is Native Multimodality?

Multimodal models need to handle different types of data.

Common modalities include:

  • Text.
  • Images.
  • Audio.
  • Video.
  • Structured data.

Different modalities are represented in different ways.

For example:

Modality Raw Data Form Main Processing Tasks
Text Token sequences Language understanding and generation
Images Pixels or visual features Object recognition, scene understanding
Audio Waveforms or audio features Speech recognition, semantic understanding
Video Consecutive image frames and temporal information Temporal understanding, event analysis

Multimodal models need to map these different data representations into a representation space where joint modeling can be performed.

This can be abstracted as:

X_m=E_m(I_m)

Where:

  • I_m: the raw input of a certain modality.
  • E_m: the corresponding modality encoder or processing module.
  • X_m: the converted representation.

Representations of different modalities can interact within the model:

H=F(X_{text},X_{image},X_{audio},X_{video})

Here F denotes the joint processing procedure of the multimodal model.

This is merely an abstract expression of multimodal modeling, and does not mean that all versions of Gemini use exactly the same encoders or fusion structures.

4.3 Joint Multimodal Understanding

Suppose a user inputs a video and asks a question:

"In this video, at what time does the person start to leave the room?"

The model needs to process:

  1. The people and scene information in the video frames.
  2. The temporal order within the video.
  3. The changes in the person's position at different points in time.
  4. The natural language question posed by the user.
  5. The semantic relationship between the video content and the question.
flowchart TB
    A["Receive multimodal input"] --> B["Text information processing"]
    A --> C["Image and video processing"]
    A --> D["Audio information processing"]
    B --> E["Multimodal representation"]
    C --> E
    D --> E
    E --> F["Cross-modal information interaction"]
    F --> G["Joint semantic understanding"]
    G --> H["Multi-step reasoning"]
    H --> I["Generate structured or natural language results"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

The value of native multimodality lies not only in being able to recognize images or transcribe audio, but more importantly in being able to leverage the correlated information across different modalities to accomplish complex tasks.

4.4 Gemini's Long-Context Capabilities

Long context is one of the key directions in Gemini's technology system.

The context window determines the upper limit on the number of input tokens a model can process in a single request.

Long context can be used for:

  • Analyzing large codebases.
  • Reading lengthy technical documentation.
  • Processing large volumes of meeting records.
  • Analyzing long videos.
  • Linking information across documents.
  • Planning complex tasks.

But a distinction needs to be made:

A large context window does not mean the model can utilize all information in the context with equal accuracy.

Long context also faces:

  • Attention computation cost.
  • Locating key information.
  • Information interference.
  • Long-distance dependencies.
  • Contradictory information within the context.
  • Inference latency and memory pressure.

Therefore, long-context capability depends not only on window length, but also on the model's information retrieval, compression, and reasoning abilities.

4.5 Gemini's Google Technology Ecosystem

Gemini's technical advantages also come from Google's infrastructure and product ecosystem.

These mainly include:

  • Gemini: the user-facing AI products and model system.
  • Gemini API: the model development interface.
  • Google AI Studio: a tool for model experimentation and application development.
  • Vertex AI / Gemini Enterprise Agent Platform: enterprise-grade model platform.
  • Google Search: the search and information retrieval ecosystem.
  • Android: the mobile device and system-level application ecosystem.
  • Google Cloud: cloud computing and enterprise deployment infrastructure.
  • Gemma: the open model series released by Google.

Note that Gemini and Gemma follow different model openness strategies.

Gemini is primarily provided as part of Google's model service system, while Gemma is an open model series released separately.

As of November 2026, Google's model catalog already covers models with different positioning, such as Gemini 4, Gemini 3.1 Pro, and Flash, and has expanded to audio, video, and Agent-related capabilities.

V. DeepSeek: Centered on MoE, Reasoning Training, and High Efficiency

5.1 Basic Positioning of DeepSeek

DeepSeek is a large model series developed by DeepSeek (深度求索).

Important characteristics of its technology system include:

  • Large-scale MoE architecture.
  • Efficient Attention design.
  • Reinforcement learning and reasoning training.
  • Training and inference efficiency optimization.
  • Open models and technical reports.

Compared with systems such as GPT and Claude, which use hosted model services as their primary access point, DeepSeek has publicly released the weights, technical reports, and model cards for multiple models, providing researchers with more materials for analysis and deployment.

However, the degree of openness, licenses, and available capabilities of different versions still need to be checked separately.

5.2 DeepSeek's MoE Architecture

MoE (Mixture of Experts) is a very important architectural direction in DeepSeek's technology system.

In a traditional Dense Transformer, the model's main FFN sublayers typically perform the corresponding computation for every token.

MoE, by contrast, divides the FFN portion into multiple expert networks and uses a Router to select a subset of experts for different tokens.

Basic structure:

  • Router: responsible for selecting experts.
  • Experts: responsible for performing different computations.
  • Gating: responsible for computing expert weights.
  • Top-K: selects the experts participating in the computation for the current token.

Its output can be expressed as:

y=\sum_{i\in TopK(x)}g_i(x)E_i(x)

Where:

  • x: the input token representation.
  • E_i(x): the computation result of the i-th expert.
  • g_i(x): the routing weight of the corresponding expert.

The core value of MoE is:

The model can have a large number of total parameters, but each token only activates a subset of the expert parameters.

Therefore, the total parameter count of the model is not the same as the number of activated parameters per token.

5.3 Technical Features of DeepSeek-V3

DeepSeek-V3 is an important representative for studying DeepSeek's technical roadmap.

Its public technical report discloses the following designs:

Technology Function
DeepSeekMoE Sparse expert network
MLA Multi-head Latent Attention, reduces KV representation and cache overhead
Auxiliary-Loss-Free Load Balancing Improves expert load balancing
Multi-Token Prediction Introduces a multi-token prediction auxiliary training objective
FP8 Training Improves training compute efficiency
DualPipe Optimizes pipeline parallelism and compute-communication overlap

The configuration disclosed in the DeepSeek-V3 technical report is:

  • Total parameter count: 671B.
  • Activated parameters per token: approximately 37B.
  • Pre-training data scale: 14.8T tokens.
  • Uses the DeepSeekMoE and MLA architectures.

These figures correspond to a specific version of DeepSeek-V3 and should not be directly applied to all subsequent DeepSeek models.

5.4 MLA: Multi-head Latent Attention

MLA is an important Attention optimization technique proposed by DeepSeek.

Traditional Multi-Head Attention needs to store historical Key and Value during inference to support attention computation for subsequent tokens.

For long-context models, KV Cache may occupy a large amount of GPU memory.

The core idea of MLA is to compress Key and Value related information through a low-dimensional latent representation, reducing the amount of data that needs to be cached during inference.

It can be abstracted as:

c_t^{KV}=W^{DKV}h_t

Where:

  • h_t: the hidden state of the current token.
  • W^{DKV}: the low-dimensional projection matrix.
  • c_t^{KV}: the compressed latent KV representation.

In subsequent attention computation, the required Key and Value representations are restored through the corresponding projections.

It should be noted that MLA does not simply perform ordinary compression on traditional K and V tensors; it structurally designs the attention representation and computation method.

Its main goals include:

  • Reduce KV Cache storage.
  • Lower memory access pressure during inference.
  • Improve long-context inference efficiency.
  • Reduce the deployment cost of large-scale models.

5.5 DeepSeek-R1 and Reinforcement Learning for Reasoning Training

DeepSeek-R1 is an important representative for understanding DeepSeek's reasoning approach.

It emphasizes improving the model's complex reasoning ability through reinforcement learning.

Reinforcement learning training can be abstracted as:

  1. The model receives a question.
  2. The model generates candidate answers or a reasoning process.
  3. A reward model or rules evaluate the result.
  4. The model policy is optimized according to the reward signal.
  5. Task performance is improved through multiple rounds of training.

The reinforcement learning objective can be expressed as:

\max_{\theta} \mathbb{E}_{y\sim\pi_\theta(\cdot|x)} [R(x,y)]

Where:

  • \pi_\theta: the policy model parameterized by \theta.
  • x: the input question.
  • y: the result generated by the model.
  • R(x,y): the reward function.

In reasoning tasks, rewards can come from:

  • Whether the mathematical answer is correct.
  • Whether the code passes the tests.
  • Whether logical constraints are satisfied.
  • Whether the format meets the requirements.
  • Whether the result satisfies specific rules.

The research approach of DeepSeek-R1 demonstrates the application value of reinforcement learning in mathematics, code, and complex reasoning tasks.

However, it should be noted that reinforcement learning is not simply about making the model "think longer." Training effectiveness depends on reward design, sampling strategy, training stability, and data distribution.

5.6 DeepSeek's Technical Ecosystem

It mainly includes:

  • DeepSeek Chat: a user-facing product.
  • DeepSeek API: a model invocation interface.
  • DeepSeek-V series: general-purpose language models.
  • DeepSeek-R series: the reasoning model line.
  • DeepSeek technical reports: public architecture and training research.
  • Open weights and model community: supports local deployment, research, and secondary development.

DeepSeek's official transparency center continuously publishes model cards, technical reports, and model-related information.

6. Qwen: Centered on Open Weights and a Full-Stack Model Ecosystem

6.1 Qwen's Basic Positioning

Qwen (Tongyi Qianwen) is a series of large language models and multimodal models developed by Alibaba's Tongyi Qianwen team.

Qwen's technical system has two obvious characteristics:

  1. Broad model coverage.
  2. A relatively rich open-weight model ecosystem.

Its model directions include not only general-purpose language models but also:

  • Mathematics and reasoning.
  • Code generation.
  • Vision-language models.
  • Audio models.
  • Multimodal models.
  • Embedding.
  • Reranker.
  • Agent and tool calling.

Qwen has both open-weight models and proprietary model services provided through its cloud platform; the two should not be conflated.

6.2 Qwen's Model System

Qwen is not just a text model; it has gradually formed multiple model branches.

Model line Primary tasks
Qwen general-purpose language model Dialogue, text understanding and generation
Qwen Reasoning Mathematics, logic and complex reasoning
Qwen Coder Code generation and program analysis
Qwen-VL Image understanding and vision-language tasks
Qwen-Omni Multimodal understanding and interaction
Qwen-Audio Speech recognition and audio understanding
Qwen-Embedding Text or multimodal vector representations
Qwen-Reranker Re-ranking of retrieval results

The value of this system is that developers can choose different models according to the specific task, rather than making a single general-purpose model take on all the work.

6.3 Dense and MoE developing in parallel

Qwen's model system includes both Dense models and MoE models.

Dense models

Each token typically passes through the corresponding complete Dense FFN sublayer in the model.

Main characteristics:

  • The structure is relatively straightforward.
  • The deployment approach is easier to understand.
  • The amount of computation has a relatively direct relationship with the parameter scale.

MoE models

A Router dynamically selects a subset of experts to participate in computation.

Main characteristics:

  • The total parameter scale can be large.
  • The number of activated parameters per token can be small.
  • Expert routing and communication need to be handled.
  • All-to-All communication requires attention in multi-device deployment.

Different model versions of Qwen may adopt different structures; one should not assume that the entire Qwen series is MoE.

6.4 Qwen's multimodal technologies

Qwen's multimodal system covers directions such as vision, audio and video.

For example, a vision-language model needs to process at the same time:

  • Image pixel information.
  • Visual features.
  • Text instructions.
  • The semantic relationship between image and text.

An audio model, meanwhile, needs to process:

  • Speech signals.
  • Acoustic features.
  • Linguistic content.
  • Speaker and timing information.
  • Speech generation.

The overall processing of a multimodal model can be abstracted as: