I. What Is Generative Artificial Intelligence?
Generative Artificial Intelligence (Generative AI), or generative AI for short, is an important technical direction in the field of artificial intelligence.
Its core capability is:
Learning the statistical patterns and structures in large amounts of data, and generating new content according to input conditions.
This content can be:
- Text: articles, reports, summaries, conversations.
- Images: illustrations, photos, design drawings.
- Audio: speech, music, sound effects.
- Video: short videos, animations, video clips.
- Code: program code, SQL, scripts.
- 3D content: models, scenes, and spatial structures.
For example:
User input:
Please generate a technical article about asynchronous programming in Python.
Generative AI can generate a structured article according to the instruction.
User input:
Generate an image of a data center in a futuristic technology style.
An image generation model can generate the corresponding image based on the text description.
User input:
Write a Python function that implements quicksort.
A code generation model can generate the corresponding program code.
These tasks share one thing in common:
The model does not simply look up an existing answer from a database; instead, it computes and generates new content based on learned patterns and the current input.
1. Core Characteristics of Generative AI
| Feature | Description |
|---|---|
| Data-driven | Learns content patterns from large amounts of data |
| Model-driven | Uses a trained generative model to produce results |
| Conditional generation | Generates content based on text, images, or other inputs |
| Probabilistic modeling | Constructs outputs through probability distributions or other generative mechanisms |
| Content diversity | The same input may produce different results under different generation conditions |
| Controllable generation | The generation process can be controlled through prompts, parameters, or conditional information |
Note that generative AI is not simply a content copying tool.
Its core lies in learning data distributions, structures, and relationships, and then using a model to generate new samples that meet the conditions.
However, generated results are not necessarily true, correct, or reliable. The fact that a model can generate seemingly plausible content does not mean it has mastered the facts behind all that content.
II. What Is the Difference Between Generative AI and Traditional Artificial Intelligence?
To understand generative AI, we first need to understand the difference between traditional AI and generative AI in terms of task objectives.
Traditional AI systems are often used for recognition, classification, prediction, and decision-making.
For example:
- Determining whether there is a cat in an image.
- Determining whether an email is spam.
- Predicting tomorrow's sales.
- Determining whether a transaction is anomalous.
- Recommending products based on user behavior.
Generative AI, by contrast, focuses more on content generation.
For example:
- Generating an image of a cat based on a description.
- Generating an email based on requirements.
- Generating code based on a problem.
- Generating a summary based on an article.
- Generating a video based on a text description.
1. Discriminative Models and Generative Models
In machine learning, discriminative models and generative models are often used to distinguish different modeling approaches.
Discriminative model:
It mainly learns the relationship between input data and target labels, or directly learns the decision boundary between classes.
For example:
Given an email, determine whether it is spam or a legitimate email.
It can be expressed as:
Where:
- x: input data.
- y: target class.
- P(y|x): the conditional probability of belonging to a certain class given the input.
Generative model:
It usually learns the probability distribution of the data itself, or learns the generative distribution of data under given conditions.
For example:
Learning the distribution of a large number of face images, and then generating new images that conform to that distribution.
It can be expressed as:
A conditional generation task can be expressed as:
Where:
- x: generated data.
- c: generation condition.
- P(x|c): the probability distribution of generated data under the given condition.
This is a classic probabilistic modeling perspective.
Note that modern generative models do not necessarily all explicitly compute the full data probability distribution. Some models learn the generation process through denoising, adversarial training, or other objectives.
2. Comparison of the Two Types of Models
| Comparison Dimension | Discriminative Models | Generative Models |
|---|---|---|
| Core objective | Recognition, classification, and prediction | Generating new content |
| Primary modeling target | The relationship between inputs and labels | Data distribution or the generation process |
| Typical output | Categories, scores, predicted values | Text, images, audio, etc. |
| Typical techniques | Logistic regression, SVM, classification networks | GPT, GAN, VAE, Diffusion |
| Typical tasks | Spam classification | Text and image generation |
The two are not absolutely mutually exclusive.
A single model can simultaneously possess capabilities such as representation learning, discrimination, and generation. Generative models can also be used for classification and prediction tasks.
For example, in addition to generating text, large language models can also perform text classification, information extraction, and question-answering tasks.
III. The Overall Technical System of Generative AI
Generative AI is not a single model, but a technical system composed jointly of data, algorithms, model architectures, training methods, and inference mechanisms.
From a technical perspective, it can be broken down into the following parts:
- Data: provides learning material for the model.
- Algorithms: define how the model learns and generates.
- Model architecture: defines the computational structure of the model.
- Model training: learns model parameters through an optimization process.
- Model inference: uses the trained model to generate content.
- Application systems: integrate generation capabilities into real business operations.
1. Generative AI Technical Architecture
flowchart TB
A["Generative AI"]
A --> B["Data Layer"]
A --> C["Algorithm and Model Layer"]
A --> D["Training and Inference Layer"]
A --> E["Application Layer"]
B --> B1["Text Data"]
B --> B2["Image Data"]
B --> B3["Audio and Video Data"]
B --> B4["Code and Structured Data"]
C --> C1["Autoregressive Models"]
C --> C2["GAN"]
C --> C3["VAE"]
C --> C4["Diffusion"]
D --> D1["Pre-training"]
D --> D2["Fine-tuning"]
D --> D3["Inference and Sampling"]
E --> E1["Text Generation"]
E --> E2["Image Generation"]
E --> E3["Audio Generation"]
E --> E4["Video Generation"]
E --> E5["Code Generation"]
classDef root fill:#DCEFE5,stroke:#4F8B70,color:#24563F,stroke-width:2px
classDef group fill:#EAF3EC,stroke:#83A995,color:#315D49
classDef item fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
class A root
class B,C,D,E group
class B1,B2,B3,B4,C1,C2,C3,C4,D1,D2,D3,E1,E2,E3,E4,E5 item
2. Core Technical Components of Generative AI
| Technical Component | Main Role |
|---|---|
| Data engineering | Data collection, cleaning, filtering, and preprocessing |
| Tokenization | Converting input into the Tokens used by the model |
| Model architecture | Defining the computational structure of the model |
| Loss function | Measuring the difference between model predictions and training targets |
| Optimization algorithm | Updating model parameters |
| Pre-training | Learning general patterns in data |
| Fine-tuning | Adapting to specific tasks or behavioral requirements |
| Inference engine | Executing the model's forward computation |
| Sampling algorithm | Controlling the selection process for generated results |
| Application engineering | Integrating model capabilities into products and business systems |
Together, these elements constitute the technical foundation of generative AI.
IV. Main Technical Approaches of Generative AI
Generative AI can be implemented using different model architectures and training methods.
Currently representative technical approaches include:
- Autoregressive models.
- Generative adversarial networks (GANs).
- Variational autoencoders (VAEs).
- Diffusion models.
Their generation mechanisms differ markedly.
1. Autoregressive Models: Predicting the Next Element Step by Step
Autoregressive models are an important technical approach for current text generation and large language models.
Their core idea is:
Based on the content already generated, predict the next Token, and then repeat this process continuously.
For example, the model needs to generate:
Artificial intelligence is changing the way software is developed.
The model might generate it according to the following process:
Artificial
Artificial intelligence
Artificial intelligence is
Artificial intelligence is changing
Artificial intelligence is changing software
Artificial intelligence is changing software development
Artificial intelligence is changing software development methods
The segmentation here is only to illustrate the generation process. Real models usually generate in units of tokens, and a token does not necessarily correspond to a complete Chinese character or word.
1.1 Mathematical Principles of Autoregressive Generation
Suppose a piece of text consists of multiple tokens:
According to the chain rule of probability, the joint probability can be decomposed as:
It can also be written as:
Where:
- x_t: The token at the current position.
- x_{<t}: All tokens before the current position.
- P(x_t|x_{<t}): The conditional probability of the next token given the context.
At each step, the model outputs a probability distribution over the vocabulary, and then selects the next token according to a sampling strategy.
1.2 Autoregressive Generation Process
flowchart TB
A["Input Prompt"] --> B["Tokenization"]
B --> C["Input to model"]
C --> D["Compute next token probability distribution"]
D --> E["Sample and select token"]
E --> F["Append to context"]
F --> G{"End generation?"}
G -->|"No"| C
G -->|"Yes"| H["Output full content"]
classDef start fill:#DCEFE5,stroke:#4F8B70,color:#24563F
classDef process fill:#EAF3EC,stroke:#83A995,color:#315D49
classDef decision fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
class A,H start
class B,C,D,E,F process
class G decision
1.3 Typical Techniques
- GPT-series language models.
- Autoregressive code generation models.
- Some audio and video generation models.
The advantage of autoregressive models is that the generation process has explicit context dependencies, making them suitable for sequence data such as language and code.
Their limitation is that the generation process usually must be executed token by token, so generating long sequences incurs certain latency and computational cost.
2. GAN: Generative Adversarial Network
GAN (Generative Adversarial Network) is a generative model trained by having two networks compete against each other.
It contains two core parts:
Generator
Responsible for generating data from random input.
Discriminator
Responsible for determining whether the input data comes from real data or from the generator.
The two networks compete with each other during training:
- The generator strives to produce samples that are closer to real data.
- The discriminator strives to distinguish real data from generated data.
As training progresses, the generator gradually improves the quality of its generations.
2.1 Basic Structure of GAN
flowchart TB
A["Random noise"] --> B["Generator"]
B --> C["Generated samples"]
D["Real samples"] --> E["Discriminator"]
C --> E
E --> F["Real or generated discrimination result"]
F --> G["Adversarial loss"]
G --> B
G --> E
classDef data fill:#DCEFE5,stroke:#4F8B70,color:#24563F
classDef model fill:#EAF3EC,stroke:#83A995,color:#315D49
classDef process fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
class A,D,C data
class B,E model
class F,G process
2.2 Training Objective of GAN
The classic objective function of GAN can be expressed as:
Where:
Where:
- G: Generator.
- D: Discriminator.
- x: Real samples.
- z: Random noise.
- G(z): Samples generated by the generator.
The discriminator tries to distinguish real samples from generated samples, while the generator tries to make generated samples be identified by the discriminator as real data.
2.3 Typical Applications of GANs
- Face generation.
- Image style transfer.
- Image inpainting.
- Image enhancement.
- Super-resolution reconstruction.
2.4 Main Limitations of GANs
- Training may be unstable.
- The generator and discriminator need to maintain a balanced training.
- Mode collapse may occur.
- There are challenges in training and control on large-scale complex data.
GAN is an important technology in the development of generative AI, but current mainstream high-quality image generation technologies have largely shifted to routes such as diffusion models.
3. VAE: Variational Autoencoder
VAE (Variational Autoencoder) is a generative model based on probabilistic latent variable modeling.
It learns a probabilistic representation of data in latent space to achieve data reconstruction and new sample generation.
VAE mainly consists of two parts:
- Encoder: encoder.
- Decoder: decoder.
The encoder maps input data to the probability distribution of latent variables, and the decoder generates data from the latent variables.
3.1 Basic Structure of VAE
flowchart LR
A["Input data"] --> B["Encoder"]
B --> C["Latent variable distribution"]
C --> D["Sample latent variables"]
D --> E["Decoder"]
E --> F["Reconstructed or generated data"]
classDef data fill:#DCEFE5,stroke:#4F8B70,color:#24563F
classDef process fill:#EAF3EC,stroke:#83A995,color:#315D49
classDef latent fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
class A,F data
class B,E process
class C,D latent
3.2 Core Principles of VAE
Suppose the input data is x, and the latent variable is z.
The encoder learns:
which represents the approximate posterior distribution of the latent variables given the input data.
The decoder learns:
which represents the conditional distribution for generating the input data given the latent variables.
VAE is usually trained by maximizing the evidence lower bound (ELBO):
It can also be understood as minimizing:
where:
- Reconstruction loss: measures the difference between the generated result and the original input.
- KL divergence: constrains the difference between the latent distribution and the prior distribution.
The core value of VAE is learning a latent space with a continuous probabilistic structure, enabling the model to generate new data through sampling.
3.3 Typical Applications of VAE
- Image generation.
- Data compression.
- Latent representation learning.
- Anomaly detection.
- Data distribution modeling.
4. Diffusion: Diffusion Model
Diffusion Model is an important technical route in the field of modern image generation and is also widely used in audio and video generation.
Its core idea is:
First learn how to gradually remove noise from data, then start from random noise and generate new data through a reverse denoising process.
4.1 Two Processes of Diffusion Models
Diffusion models usually contain two core processes.
Forward Diffusion Process (Forward Process)
Gradually add noise to real data, making the data gradually approach a preset noise distribution.
Reverse Denoising Process (Reverse Process)
Train a neural network to learn how to gradually remove noise and recover data matching the target distribution from random noise.
flowchart LR
A["Real image"] --> B["Add a small amount of noise"]
B --> C["Medium noise"]
C --> D["Heavy noise"]
D --> E["Nearly random noise"]
E --> F["Reverse denoising"]
F --> G["Gradually recover structure"]
G --> H["Generated image"]
classDef data fill:#DCEFE5,stroke:#4F8B70,color:#24563F
classDef process fill:#EAF3EC,stroke:#83A995,color:#315D49
classDef noise fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
class A,H data
class B,C,F,G process
class D,E noise
The figure above is a simplified illustration of the diffusion model principle. The actual training and sampling process also includes mechanisms such as noise scheduling, time-step encoding, and a denoising network.
4.2 Mathematical Principles of Diffusion Models
In the common DDPM framework, the forward process can be expressed as:
Where:
- x_t: Noisy data at time step t.
- \beta_t: Noise strength at the current time step.
- I: Identity matrix.
Through reparameterization, a noisy sample at any time step can be obtained directly from the original data:
Where:
- x_0: Original data.
- \epsilon: Gaussian noise.
- \alpha_t=1-\beta_t.
- \bar{\alpha}_t=\prod_{s=1}^{t}\alpha_s.
During training, the model typically learns to predict the added noise:
Where:
- \epsilon_\theta: Noise predicted by the model.
- t: Diffusion time step.
Once the model has learned denoising capability, it can start from random noise and gradually generate new images.
4.3 Conditional Generation with Diffusion Models
In text-to-image tasks, the model needs not only to learn denoising but also to understand text conditions.
For example:
An orange cat sits on the windowsill, sunlight streams into the room, realistic photography style.
The text encoder converts the prompt into a conditional representation that the model can use, and the diffusion model controls the generation process based on these conditions.
Typical techniques:
- Stable Diffusion.
- Latent Diffusion.
- Denoising Diffusion Probabilistic Models(DDPM)。
- Denoising Diffusion Implicit Models(DDIM)。
4.4 Main Applications of Diffusion Models
- Text-to-image.
- Image editing.
- Image inpainting.
- Image super-resolution.
- Video generation.
- Audio generation.
- 3D content generation.
5. Comparison of Four Generative Models
| Technique | Core principle | Generation method | Typical applications |
|---|---|---|---|
| Autoregressive models | Conditional probability modeling | Predict the next element step by step | Text, code |
| GAN | Adversarial competition between generator and discriminator | The generator directly generates samples | Image generation |
| VAE | Probabilistic latent space modeling | Sampling and decoding latent variables | Images, representation learning |
| Diffusion | Learn the reverse denoising process | Gradually generate data from noise | Images, video, audio |
These four techniques represent important approaches in generative models, but they are not the entirety of generative AI.
In practical systems, there are also Flow Matching, Normalizing Flows, Mixture-of-Experts architectures, and other generative modeling methods.
V. Why Are Large Language Models Considered Generative AI?
Large language models (LLMs) are one of the most important technical representatives of generative AI.
For example, the GPT series of models are mainly based on the Transformer architecture and are trained on large-scale text data to learn the statistical patterns and contextual relationships of language.
1. What Is a Large Language Model?
A large language model is a language model trained on large-scale data and with a large number of parameters.
It can perform:
- Text understanding.
- Text generation.
- Conversational interaction.
- Code generation.
- Text summarization.
- Information extraction.
- Translation.
- Question answering.
- A certain degree of reasoning and problem solving.
Note that LLM is not an umbrella term for all generative AI.
It mainly refers to language models, whereas generative AI also includes generative models for images, audio, video, and other data modalities.
2. What Is a Transformer?
A Transformer is a neural network architecture based on the attention mechanism.
One of its most important core technologies is the self-attention mechanism (Self-Attention).
Self-attention can calculate the information weight of each position based on the associations between different positions in the input sequence.
Its core computation is:
Where:
- Q: Query, the query matrix.
- K: Key, the key matrix.
- V: Value, the value matrix.
- d_k: Key vector dimension.
Through the attention mechanism, the model can integrate information from different positions based on contextual relationships.
A Transformer also usually includes:
- Multi-head attention (Multi-Head Attention).
- Feed-forward network (Feed Forward Network).
- Residual connection (Residual Connection).
- Layer normalization (Layer Normalization).
- Positional representation mechanism (Positional Representation).
3. Basic Structure of a Large Language Model
flowchart TB
A["Input text"] --> B["Tokenization"]
B --> C["Token Embedding"]
C --> D["Positional representation"]
D --> E["Transformer network"]
E --> F["Output layer"]
F --> G["Next Token probability distribution"]
G --> H["Generate Token"]
classDef data fill:#DCEFE5,stroke:#4F8B70,color:#24563F
classDef process fill:#EAF3EC,stroke:#83A995,color:#315D49
classDef model fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
class A,H data
class B,C,D,F,G process
class E model
This is a schematic of a typical autoregressive language model structure. Different models may use Encoder-Decoder, Decoder-only, or other architectures.
4. How Does a Large Language Model Generate Text?
For example, with the user input:
The development directions of artificial intelligence include
The model calculates the probability distribution of the next Token based on the current context.
For example:
| Candidate Token | Example probability |
|---|---|
| Machine learning | 0.35 |
| Deep learning | 0.25 |
| Natural language processing | 0.20 |
| Other Token | 0.20 |
The numbers above are only used to illustrate the probability distribution and do not represent the output of any actual model.
The model selects a Token according to a sampling strategy, then adds it to the context and continues predicting the next Token.
By repeating this process, a complete text is eventually formed.
Therefore, the core generation mechanism of a large language model can be summarized as:
Understand the contextual representation, predict the next Token, and generate the sequence in a loop.
6. How Is Generative AI Trained?
The capabilities of generative AI do not arise out of thin air; they are gradually learned through data, model architectures, and training algorithms.
Different generative models have different training objectives, but overall the process usually involves stages such as data preparation, pretraining, post-training, and model evaluation.
1. Data Preparation
Data is the foundation for training models.
Different models require different types of data:
| Model type | Main training data |
|---|---|
| Large language model | Text, code, documents |
| Image generation model | Images and text descriptions |
| Speech generation model | Audio, text, speech annotations |
| Video generation model | Video, text, temporal information |
| Multimodal model | Text, images, audio, video, etc. |
Data preparation typically includes:
- Data collection.
- Data cleaning.
- Duplicate data handling.
- Low-quality data filtering.
- Format standardization.
- Privacy and copyright compliance handling.
- Tokenization or other modality encoding.
Data quality directly affects model training performance.
2. Pre-training
Pre-training is the process of enabling a model to learn general patterns from large-scale data.
Take autoregressive language models as an example:
The model learns vocabulary, syntax, context, and other statistical patterns in language by predicting the next Token.
The training objective can usually be expressed as:
Where:
- x_t: the current Token in the training text.
- x_{<t}: the preceding context.
- P_\theta: the conditional probability predicted by the model.
- T: sequence length.
The model continuously adjusts its parameters to reduce the loss of the prediction objective.
Note that different generative models do not have the same pre-training objectives.
For example:
- GPT-style autoregressive models usually use next-Token prediction.
- BERT-style models often use masked language modeling, but their typical use is representation learning and understanding tasks, not standard autoregressive text generation models.
- Diffusion models usually learn noise prediction or denoising objectives.
3. Supervised Fine-Tuning (SFT)
A pre-trained model has already learned a large number of general data patterns, but it may not be able to reliably complete tasks according to user requirements.