I. What Are Discriminative Models and Generative Models?
In machine learning, discriminative models and generative models are two important modeling approaches.
Their core difference is:
- Discriminative models: focus on learning how to determine the output from the input.
- Generative models: focus on learning the distribution of the data itself, or how data are generated under given conditions.
For example, suppose there is a set of images of cats and dogs.
We want to train a model that can recognize the animal in an image.
Discriminative models focus on:
Given an image, does it belong to a cat or a dog?
Generative models focus on:
What data distributions do cat images and dog images have respectively? How can a new image of a cat or dog be generated?
The two solve different problems and have different learning objectives.
1. The Core Idea of Discriminative Models
Discriminative models mainly learn the conditional probability relationship between the input x and the target label y.
Their typical modeling objective is:
where:
- x: input data.
- y: target label.
- P(y|x): the conditional probability that the output belongs to class y given input x.
For example, given an input image:
the model calculates:
The model ultimately determines that the animal in the image is a cat.
Discriminative models usually focus on classification boundaries, prediction accuracy, and the input-output mapping relationship, and do not need to fully learn the generation process of the input data.
2. The Core Idea of Generative Models
Generative models typically learn the probability distribution of the data, or learn the distribution from which data are generated under given conditions.
Common modeling objectives include:
and:
A conditional generation task can be represented as:
where:
- x: data sample.
- y: label.
- c: generation condition.
- P(x): the marginal probability distribution of the data.
- P(x,y): the joint probability distribution of inputs and labels.
- P(x|c): the conditional probability distribution of data generated under the given condition.
For example, the model learns from a large number of cat images and captures statistical patterns such as the shape, texture, color, and spatial structure of cats.
When the generation condition is input:
A white cat sitting on the windowsill.
the model can generate a new image that matches the description based on the learned distribution.
Note that not all generative models directly learn P(x). For example, a conditional generative model may directly learn P(x|c), and a diffusion model may also learn the generative distribution indirectly through a denoising objective.
II. The Mathematical Essence of the Two Types of Models
To understand discriminative models and generative models, the most important thing is to understand the relationships between probability distributions.
1. Discriminative Models: Learning Conditional Probability
Discriminative models directly learn:
They focus on:
Given input data, what is the probability that the output belongs to each of the different classes?
For example, in a spam email classification task:
Input:
Congratulations, you have won a large bonus. Please click the link immediately to claim it.
Model output:
| Category | Predicted probability |
|---|---|
| Spam email | 0.98 |
| Legitimate email | 0.02 |
The model only needs to determine which category the input belongs to; it does not need to generate a complete email, nor does it necessarily need to learn the full data distributions of legitimate and spam emails.
Common discriminative modeling methods include:
- Logistic regression.
- Support vector machine (SVM).
- Decision tree.
- Random forest.
- Discriminative neural networks.
2. Generative Models: Learning Data Distributions
Generative models can learn:
They can also learn:
For example, suppose the dataset contains images of cats and dogs.
The model can learn:
- What statistical characteristics do images of cats usually have?
- What statistical characteristics do images of dogs usually have?
- What are the differences in data distributions across different classes?
- What kind of joint relationship exists between input data and classes?
If the model learns the joint probability distribution:
Then according to Bayes' formula, we can obtain:
Further using the decomposition of joint probability:
we can obtain:
This means that a generative model can not only be used to generate data, but can also use the learned probability distribution to complete classification tasks.
3. The relationship between the two modeling approaches
flowchart TB
A["Training data"] --> B["Model learning"]
B --> C["Discriminative modeling"]
B --> D["Generative modeling"]
C --> E["Learn conditional probability P(y|x)"]
D --> F["Learn data distribution P(x)"]
D --> G["Learn joint distribution P(x,y)"]
D --> H["Learn conditional generative distribution P(x|c)"]
E --> I["Classification and prediction"]
F --> J["Data generation"]
G --> K["Generation and classification"]
H --> L["Conditional content generation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
style K fill:#E5F0E9,stroke:#83A995,color:#315D49
style L fill:#E5F0E9,stroke:#83A995,color:#315D49
Here, three concepts need to be distinguished:
| Probability form | Meaning | Typical use |
|---|---|---|
| (P(y | x)) | Given input, predict the label |
| P(x) | The probability distribution of the data itself | Unconditional data generation |
| P(x,y) | The joint distribution of inputs and labels | Joint modeling, classification, and generation |
| (P(x | c)) | Given a condition, generate data |
This is also the most important mathematical difference between the two types of models.
III. How Discriminative Models Work
Discriminative models mainly learn the mapping relationship between input and output from training data.
Take spam email classification as an example.
1. Prepare training data
Suppose we have the following training samples:
| Email content | Label |
|---|---|
| Claim your free bonus immediately | Spam |
| This week's project meeting schedule | Legitimate email |
| Click the link to get a reward | Spam |
| Please review this week's work report | Legitimate email |
| Congratulations on receiving a free gift | Spam |
The model needs to learn the relationship between email content and class labels.
2. Feature extraction
In traditional machine learning, it is usually necessary to convert text into numerical features.
For example:
- Whether it contains "free".
- Whether it contains "bonus".
- Whether it contains "click the link".
- Email length.
- Number of links.
- Frequency of specific words.
obtaining the feature vector:
where each x_i represents a feature.
Modern neural networks can also learn feature representations directly from text; they do not necessarily require manually designing these features.
3. Build a classification model
Taking logistic regression as an example:
where:
- w: model weights.
- x: input features.
- b: bias term.
- z: the result of the linear computation.
For binary classification tasks, the Sigmoid function can be used:
where:
The model outputs a probability between 0 and 1.
For example:
When the predicted probability exceeds a set threshold, the email is classified as spam.
4. Compute the loss and update the parameters
Binary classification tasks commonly use the binary cross-entropy loss:
where:
- y: the true label.
- p: the model's predicted probability.
- L: the loss value.
During training, the optimization algorithm continually adjusts the parameters so that the predictions come as close as possible to the true labels.
5. The training workflow of a discriminative model
flowchart TB
A["Prepare labeled training data"] --> B["Feature extraction or representation learning"]
B --> C["Feed into the discriminative model"]
C --> D["Predict class probabilities"]
D --> E["Compute the classification loss"]
E --> F["Compute gradients"]
F --> G["Update model parameters"]
G --> H{"Has the training condition been met?"}
H -->|"No"| C
H -->|"Yes"| I["Obtain the trained classification model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
The training objective of a discriminative model is usually to improve performance on a classification or prediction task, rather than to learn the complete data generation process.
IV. How Generative Models Work
The core of a generative model is to learn the data distribution and then generate new data samples based on the learned distribution.
Let us again take images of cats and dogs as an example.
1. Prepare the training data
A generative model can use a large number of images as training data.
For example:
- Images of cats.
- Images of dogs.
- Images of animals in various poses.
- Images under different lighting, backgrounds, and shooting angles.
If it is a conditional generative model, it may also need text descriptions or class labels corresponding to the images.
2. Learn the data distribution
The model learns representative statistical patterns from the training data.
For images, these patterns may include:
- Color distribution.
- Edges and textures.
- Shape structures.
- Spatial relationships between parts and the whole.
- Correlations among different visual elements.
The model does not simply memorize all the images; rather, it learns the data distribution or the generation process through a parameterized model.
3. Generate data according to conditions
For example, the user inputs:
A white cat sitting on a windowsill, with sunlight streaming into the room.
The model converts the text description into generation conditions and then produces an image through the generation process.
The generated result does not need to be exactly identical to any image in the training set; rather, it is expected to conform to the learned data patterns and the input conditions.
4. The training workflow of a generative model
There are many kinds of generative models, and their training workflows are not entirely the same.
Below, taking a diffusion model as an example:
flowchart TB
A["Prepare training images"] --> B["Data preprocessing"]
B --> C["Randomly select a time step"]
C --> D["Add noise to the image"]
D --> E["Input to denoising model"]
E --> F["Predict noise"]
F --> G["Compute denoising loss"]
G --> H["Backpropagation and parameter update"]
H --> I{"Have the training conditions been met?"}
I -->|"No"| C
I -->|"Yes"| J["Obtain the trained generative model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
The figure above shows a typical training approach for diffusion models; it does not mean that all generative models use noise addition and denoising for training.
5. How do generative models produce new content?
Different generative models have different generation mechanisms.
| Model | Generation mechanism |
|---|---|
| Autoregressive model | Predict the next Token step by step based on context |
| GAN | Map random input to data samples through the generator |
| VAE | Sample from the latent space, then generate data through the decoder |
| Diffusion | Start from noise and gradually perform the denoising process |
For example, when a large language model generates text, it usually predicts the next Token based on the existing context and repeats this process continuously.
When a diffusion model generates an image, it usually starts from random noise and gradually restores the image structure.
Although the two have different generation methods, both are important technical routes in generative modeling.
V. Core Difference Between Discriminative Models and Generative Models
The most important difference between the two types of models is not what programming language is used or whether neural networks are adopted, but rather the different modeling objectives and learning targets.
1. Core comparison
| Comparison dimension | Discriminative model | Generative model |
|---|---|---|
| English name | Discriminative Model | Generative Model |
| Core objective | Judgment, classification, prediction | Learn distributions, generate data |
| Typical modeling objective | (P(y | x)) |
| Focus | Relationship between input and output | Data distribution or generation process |
| Typical tasks | Classification, regression, recognition | Text, image, audio generation |
| Data requirements | Supervised tasks usually require labels | Training data can be labeled or unlabeled |
| Output form | Class, probability, predicted value | Text, images, audio, etc. |
| Typical models | Logistic regression, SVM, classification networks | GPT, GAN, VAE, Diffusion |
| Training objective | Classification loss, regression loss, etc. | Likelihood, adversarial loss, denoising loss, etc. |
Note in particular: being a generative model does not mean it must generate images or text; it can also be used for tasks such as classification, anomaly detection, and probability estimation.
2. Understanding from the input-output perspective
Discriminative model:
Input: an animal image
Model: classification model
Output:
Cat: 0.92
Dog: 0.08
Final result: Cat
Generative model:
Input: a text description
Model: image generation model
Output: a new image matching the description
Discriminative models usually map inputs to labels or predicted values.
Generative models, in turn, generate new data based on the learned distribution or generation mechanism.
3. Understanding from the learning objective perspective
Discriminative models focus on:
Generative models focus on:
Or:
The former selects the most likely class based on the input; the latter samples and generates data from the learned distribution.
VI. Why Can Generative Models Also Perform Classification?
This is a point that easily causes confusion when understanding the two types of models.
Since generative models are mainly used to generate content, why can they also perform classification?
The key lies in:
The data distribution learned by a generative model can be transformed into classification results through probabilistic inference.
1. Using Bayes' Formula for Classification
Suppose we need to determine whether an email is spam.