Back to series
· 8 min read

What is the difference between discriminative and generative models?


I. What Are Discriminative Models and Generative Models?

In machine learning, discriminative models and generative models are two important modeling approaches.

Their core difference is:

  • Discriminative models: focus on learning how to determine the output from the input.
  • Generative models: focus on learning the distribution of the data itself, or how data are generated under given conditions.

For example, suppose there is a set of images of cats and dogs.

We want to train a model that can recognize the animal in an image.

Discriminative models focus on:

Given an image, does it belong to a cat or a dog?

Generative models focus on:

What data distributions do cat images and dog images have respectively? How can a new image of a cat or dog be generated?

The two solve different problems and have different learning objectives.

1. The Core Idea of Discriminative Models

Discriminative models mainly learn the conditional probability relationship between the input x and the target label y.

Their typical modeling objective is:

P(y|x)

where:

  • x: input data.
  • y: target label.
  • P(y|x): the conditional probability that the output belongs to class y given input x.

For example, given an input image:

x=\text{A picture of an animal}

the model calculates:

P(\text{cat}|x)=0.9
P(\text{dog}|x)=0.1

The model ultimately determines that the animal in the image is a cat.

Discriminative models usually focus on classification boundaries, prediction accuracy, and the input-output mapping relationship, and do not need to fully learn the generation process of the input data.

2. The Core Idea of Generative Models

Generative models typically learn the probability distribution of the data, or learn the distribution from which data are generated under given conditions.

Common modeling objectives include:

P(x)

and:

P(x,y)

A conditional generation task can be represented as:

P(x|c)

where:

  • x: data sample.
  • y: label.
  • c: generation condition.
  • P(x): the marginal probability distribution of the data.
  • P(x,y): the joint probability distribution of inputs and labels.
  • P(x|c): the conditional probability distribution of data generated under the given condition.

For example, the model learns from a large number of cat images and captures statistical patterns such as the shape, texture, color, and spatial structure of cats.

When the generation condition is input:

A white cat sitting on the windowsill.

the model can generate a new image that matches the description based on the learned distribution.

Note that not all generative models directly learn P(x). For example, a conditional generative model may directly learn P(x|c), and a diffusion model may also learn the generative distribution indirectly through a denoising objective.

II. The Mathematical Essence of the Two Types of Models

To understand discriminative models and generative models, the most important thing is to understand the relationships between probability distributions.

1. Discriminative Models: Learning Conditional Probability

Discriminative models directly learn:

P(y|x)

They focus on:

Given input data, what is the probability that the output belongs to each of the different classes?

For example, in a spam email classification task:

Input:

Congratulations, you have won a large bonus. Please click the link immediately to claim it.

Model output:

Category Predicted probability
Spam email 0.98
Legitimate email 0.02

The model only needs to determine which category the input belongs to; it does not need to generate a complete email, nor does it necessarily need to learn the full data distributions of legitimate and spam emails.

Common discriminative modeling methods include:

  • Logistic regression.
  • Support vector machine (SVM).
  • Decision tree.
  • Random forest.
  • Discriminative neural networks.

2. Generative Models: Learning Data Distributions

Generative models can learn:

P(x)

They can also learn:

P(x,y)

For example, suppose the dataset contains images of cats and dogs.

The model can learn:

  • What statistical characteristics do images of cats usually have?
  • What statistical characteristics do images of dogs usually have?
  • What are the differences in data distributions across different classes?
  • What kind of joint relationship exists between input data and classes?

If the model learns the joint probability distribution:

P(x,y)

Then according to Bayes' formula, we can obtain:

P(y|x)=\frac{P(x,y)}{P(x)}

Further using the decomposition of joint probability:

P(x,y)=P(x|y)P(y)

we can obtain:

P(y|x)= \frac{P(x|y)P(y)}{P(x)}

This means that a generative model can not only be used to generate data, but can also use the learned probability distribution to complete classification tasks.

3. The relationship between the two modeling approaches

Schematic of probability relationships

flowchart TB
    A["Training data"] --> B["Model learning"]

    B --> C["Discriminative modeling"]
    B --> D["Generative modeling"]

    C --> E["Learn conditional probability P(y|x)"]
    D --> F["Learn data distribution P(x)"]
    D --> G["Learn joint distribution P(x,y)"]
    D --> H["Learn conditional generative distribution P(x|c)"]

    E --> I["Classification and prediction"]
    F --> J["Data generation"]
    G --> K["Generation and classification"]
    H --> L["Conditional content generation"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49
    style K fill:#E5F0E9,stroke:#83A995,color:#315D49
    style L fill:#E5F0E9,stroke:#83A995,color:#315D49

Here, three concepts need to be distinguished:

Probability form Meaning Typical use
(P(y x)) Given input, predict the label
P(x) The probability distribution of the data itself Unconditional data generation
P(x,y) The joint distribution of inputs and labels Joint modeling, classification, and generation
(P(x c)) Given a condition, generate data

This is also the most important mathematical difference between the two types of models.

III. How Discriminative Models Work

Discriminative models mainly learn the mapping relationship between input and output from training data.

Take spam email classification as an example.

1. Prepare training data

Suppose we have the following training samples:

Email content Label
Claim your free bonus immediately Spam
This week's project meeting schedule Legitimate email
Click the link to get a reward Spam
Please review this week's work report Legitimate email
Congratulations on receiving a free gift Spam

The model needs to learn the relationship between email content and class labels.

2. Feature extraction

In traditional machine learning, it is usually necessary to convert text into numerical features.

For example:

  • Whether it contains "free".
  • Whether it contains "bonus".
  • Whether it contains "click the link".
  • Email length.
  • Number of links.
  • Frequency of specific words.

obtaining the feature vector:

x=[x_1,x_2,\ldots,x_n]

where each x_i represents a feature.

Modern neural networks can also learn feature representations directly from text; they do not necessarily require manually designing these features.

3. Build a classification model

Taking logistic regression as an example:

z=w^Tx+b

where:

  • w: model weights.
  • x: input features.
  • b: bias term.
  • z: the result of the linear computation.

For binary classification tasks, the Sigmoid function can be used:

P(y=1|x)=\sigma(z)

where:

\sigma(z)=\frac{1}{1+e^{-z}}

The model outputs a probability between 0 and 1.

For example:

P(\text{spam}|x)=0.95

When the predicted probability exceeds a set threshold, the email is classified as spam.

4. Compute the loss and update the parameters

Binary classification tasks commonly use the binary cross-entropy loss:

L=-[y\log(p)+(1-y)\log(1-p)]

where:

  • y: the true label.
  • p: the model's predicted probability.
  • L: the loss value.

During training, the optimization algorithm continually adjusts the parameters so that the predictions come as close as possible to the true labels.

5. The training workflow of a discriminative model

flowchart TB
    A["Prepare labeled training data"] --> B["Feature extraction or representation learning"]
    B --> C["Feed into the discriminative model"]
    C --> D["Predict class probabilities"]
    D --> E["Compute the classification loss"]
    E --> F["Compute gradients"]
    F --> G["Update model parameters"]
    G --> H{"Has the training condition been met?"}
    H -->|"No"| C
    H -->|"Yes"| I["Obtain the trained classification model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

The training objective of a discriminative model is usually to improve performance on a classification or prediction task, rather than to learn the complete data generation process.

IV. How Generative Models Work

The core of a generative model is to learn the data distribution and then generate new data samples based on the learned distribution.

Let us again take images of cats and dogs as an example.

1. Prepare the training data

A generative model can use a large number of images as training data.

For example:

  • Images of cats.
  • Images of dogs.
  • Images of animals in various poses.
  • Images under different lighting, backgrounds, and shooting angles.

If it is a conditional generative model, it may also need text descriptions or class labels corresponding to the images.

2. Learn the data distribution

The model learns representative statistical patterns from the training data.

For images, these patterns may include:

  • Color distribution.
  • Edges and textures.
  • Shape structures.
  • Spatial relationships between parts and the whole.
  • Correlations among different visual elements.

The model does not simply memorize all the images; rather, it learns the data distribution or the generation process through a parameterized model.

3. Generate data according to conditions

For example, the user inputs:

A white cat sitting on a windowsill, with sunlight streaming into the room.

The model converts the text description into generation conditions and then produces an image through the generation process.

The generated result does not need to be exactly identical to any image in the training set; rather, it is expected to conform to the learned data patterns and the input conditions.

4. The training workflow of a generative model

There are many kinds of generative models, and their training workflows are not entirely the same.

Below, taking a diffusion model as an example:

flowchart TB
    A["Prepare training images"] --> B["Data preprocessing"]
    B --> C["Randomly select a time step"]
    C --> D["Add noise to the image"]
    D --> E["Input to denoising model"]
    E --> F["Predict noise"]
    F --> G["Compute denoising loss"]
    G --> H["Backpropagation and parameter update"]
    H --> I{"Have the training conditions been met?"}
    I -->|"No"| C
    I -->|"Yes"| J["Obtain the trained generative model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

The figure above shows a typical training approach for diffusion models; it does not mean that all generative models use noise addition and denoising for training.

5. How do generative models produce new content?

Different generative models have different generation mechanisms.

Model Generation mechanism
Autoregressive model Predict the next Token step by step based on context
GAN Map random input to data samples through the generator
VAE Sample from the latent space, then generate data through the decoder
Diffusion Start from noise and gradually perform the denoising process

For example, when a large language model generates text, it usually predicts the next Token based on the existing context and repeats this process continuously.

When a diffusion model generates an image, it usually starts from random noise and gradually restores the image structure.

Although the two have different generation methods, both are important technical routes in generative modeling.

V. Core Difference Between Discriminative Models and Generative Models

The most important difference between the two types of models is not what programming language is used or whether neural networks are adopted, but rather the different modeling objectives and learning targets.

1. Core comparison

Comparison dimension Discriminative model Generative model
English name Discriminative Model Generative Model
Core objective Judgment, classification, prediction Learn distributions, generate data
Typical modeling objective (P(y x))
Focus Relationship between input and output Data distribution or generation process
Typical tasks Classification, regression, recognition Text, image, audio generation
Data requirements Supervised tasks usually require labels Training data can be labeled or unlabeled
Output form Class, probability, predicted value Text, images, audio, etc.
Typical models Logistic regression, SVM, classification networks GPT, GAN, VAE, Diffusion
Training objective Classification loss, regression loss, etc. Likelihood, adversarial loss, denoising loss, etc.

Note in particular: being a generative model does not mean it must generate images or text; it can also be used for tasks such as classification, anomaly detection, and probability estimation.

2. Understanding from the input-output perspective

Discriminative model:

Input: an animal image

Model: classification model

Output:
Cat: 0.92
Dog: 0.08

Final result: Cat

Generative model:

Input: a text description

Model: image generation model

Output: a new image matching the description

Discriminative models usually map inputs to labels or predicted values.

Generative models, in turn, generate new data based on the learned distribution or generation mechanism.

3. Understanding from the learning objective perspective

Discriminative models focus on:

\hat{y}=\arg\max_y P(y|x)

Generative models focus on:

x\sim P(x)

Or:

x\sim P(x|c)

The former selects the most likely class based on the input; the latter samples and generates data from the learned distribution.

VI. Why Can Generative Models Also Perform Classification?

This is a point that easily causes confusion when understanding the two types of models.

Since generative models are mainly used to generate content, why can they also perform classification?

The key lies in:

The data distribution learned by a generative model can be transformed into classification results through probabilistic inference.

1. Using Bayes' Formula for Classification

Suppose we need to determine whether an email is spam.