Back to series
· 15 min read

Supervised, Unsupervised, and Reinforcement Learning: The Three Core Paradigms of Machine Learning


Machine learning is not a single algorithmic system, but rather a collection of multiple distinct learning paradigms. These paradigms differ noticeably in their data sources, feedback mechanisms, optimization objectives, and training methods.

Among them, Supervised Learning, Unsupervised Learning, and Reinforcement Learning are the three most representative core paradigms of machine learning.

Each addresses a different type of problem:

  • Supervised learning: uses labeled data to learn the mapping between inputs and target outputs.
  • Unsupervised learning: uses data without explicit labels to discover the internal structure, distribution, and latent patterns within the data.
  • Reinforcement learning: interacts with an environment and learns how to act based on reward or penalty signals in order to maximize long-term gain.

Understanding these three learning paradigms not only helps build a holistic view of machine learning, but also serves as an important foundation for understanding deep learning, large language models, agents, and automated decision-making systems.


I. What Is a Learning Paradigm in Machine Learning?

1.1 Definition of a Learning Paradigm

A learning paradigm describes how a machine acquires experience from data or an environment, builds a model, and gradually improves its task performance.

The core difference between learning paradigms lies not in which neural network or algorithm is used, but in:

  1. What kind of data is used during learning?
  2. Are there explicit target labels?
  3. Through what signal does the model receive feedback?
  4. What is the model's optimization objective?
  5. Does the learning process involve continuous interaction with an environment?

For example, neural networks can be used to implement supervised learning, unsupervised learning, and reinforcement learning alike.

Therefore, learning paradigms, model architectures, and specific algorithms are concepts at different levels.

1.2 Comparison of the Three Major Paradigms

Comparison Dimension Supervised Learning Unsupervised Learning Reinforcement Learning
English name Supervised Learning Unsupervised Learning Reinforcement Learning
Core data Labeled data Data without explicit labels Environment interaction data
Source of feedback Ground-truth labels The data's own structure or the training objective Reward signals
Core objective Learn the mapping from input to output Discover latent patterns in the data Learn an optimal decision policy
Typical tasks Classification, regression Clustering, dimensionality reduction, density estimation Control, decision-making, path planning
Learning approach Adjust the model based on prediction error Optimize based on structure or a custom objective Adjust the policy or value estimate based on reward
Typical algorithms Linear regression, SVM, random forests K-Means, PCA, DBSCAN Q-Learning, PPO, SAC
Typical applications Spam detection, price prediction User segmentation, anomaly detection Robot control, game agents

1.3 Overall Relationship Among the Three Paradigms

flowchart TB
    A["Machine Learning"] --> B["Supervised Learning"]
    A --> C["Unsupervised Learning"]
    A --> D["Reinforcement Learning"]

    B --> B1["Labeled data"]
    B1 --> B2["Learn the mapping from input to target output"]
    B2 --> B3["Classification and regression"]

    C --> C1["Data without explicit labels"]
    C1 --> C2["Discover internal structure in the data"]
    C2 --> C3["Clustering and dimensionality reduction"]

    D --> D1["Continuous interaction with the environment"]
    D1 --> D2["Obtain rewards or penalties"]
    D2 --> D3["Learn a decision policy"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style B1 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B2 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B3 fill:#E5F0E9,stroke:#83A995,color:#315D49
    style C1 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C2 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C3 fill:#E5F0E9,stroke:#83A995,color:#315D49
    style D1 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D2 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D3 fill:#E5F0E9,stroke:#83A995,color:#315D49

II. Supervised Learning

2.1 What Is Supervised Learning?

Supervised learning is a machine learning method that trains models using labeled data.

Labeled data means that each training sample contains not only input features but also the corresponding correct target value.

For example:

House Area Number of Rooms District House Price
80 m² 2 District A 2 million
100 m² 3 District B 2.8 million
120 m² 3 District A 3.5 million
150 m² 4 District C 4.2 million

Where:

  • Input features (Features): house area, number of rooms, and district.
  • Label: house price.
  • Model: learns the relationship between the input features and the house price.

The model is trained on a large number of labeled samples and gradually learns a mapping function:

f_\theta:X\rightarrow Y

Where:

  • X: input feature space.
  • Y: target output space.
  • f_\theta: the model controlled by parameters \theta.
  • \theta: the parameters that the model needs to learn.

After training is complete, the model can predict the price based on new house features.

2.2 Core Principles of Supervised Learning

The essence of supervised learning is: measure the error of the model's predictions using the true labels, and continuously adjust the model parameters so that the predictions gradually approach the true target.

A typical training process includes:

  1. Input training samples.
  2. The model performs forward computation.
  3. Obtain prediction results.
  4. Compare the prediction results with the true labels.
  5. Compute the loss function.
  6. Update model parameters based on gradients.
  7. Repeat training until the expected result is achieved.
flowchart TB
    A["Prepare labeled training data"] --> B["Input features X"]
    B --> C["Model forward computation"]
    C --> D["Obtain prediction Ŷ"]
    D --> E["Compare with true label Y"]
    E --> F["Compute loss function"]
    F --> G["Backpropagate to compute gradients"]
    G --> H["Optimizer updates model parameters"]
    H --> I{"Is the training condition met?"}
    I -->|"No"| B
    I -->|"Yes"| J["Obtain the trained model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Note the following:

Supervised learning does not necessarily use neural networks, nor does it necessarily use backpropagation.

For example:

  • Linear regression can be solved using the least squares method.
  • Decision trees can split using information gain or the Gini index.
  • Random forests ensemble multiple decision trees.
  • Neural networks typically use backpropagation and gradient-based optimization.

Although their optimization mechanisms differ, they all rely on labeled data to learn the target mapping.

2.3 Two Major Tasks in Supervised Learning

Supervised learning mainly includes classification and regression.

2.3.1 Classification

The goal of a classification task is to predict the class to which a sample belongs.

For example:

  • Spam detection: spam / legitimate email.
  • Image recognition: cat / dog / bird.
  • User churn prediction: churned / not churned.
  • Risk identification: high risk / low risk.

Classification models typically learn conditional probabilities:

P(Y|X)

where X is the input and Y is the class label.

A binary classification model may output:

P(Y=1|X)=0.92

This means the model estimates the probability that the sample belongs to the positive class as 92%.

Multiclass tasks typically use Softmax to convert model outputs into a class probability distribution.

Common classification algorithms:

  • Logistic Regression
  • Decision Tree
  • Random Forest
  • Support Vector Machine
  • Naive Bayes
  • Gradient Boosting
  • Neural Network

2.3.2 Regression

The goal of a regression task is to predict continuous numerical values.

For example:

  • House price prediction.
  • Product sales prediction.
  • User spending amount prediction.
  • Temperature prediction.
  • Remaining useful life prediction for equipment.

Regression models typically learn:

\hat{y}=f_\theta(x)

where \hat{y} is the model prediction.

Common regression loss functions include:

Mean Squared Error (MSE)

MSE=\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2

Mean Absolute Error (MAE)

MAE=\frac{1}{N}\sum_{i=1}^{N}|y_i-\hat{y}_i|

MSE penalizes larger prediction errors more heavily; MAE is relatively less sensitive to outlier errors.

2.4 Core Components of Supervised Learning

Component Description
Training Data Data used to train the model
Feature Features input to the model
Label Target value corresponding to a sample
Model A function that learns the relationship between input and output
Loss Function Measures prediction error
Optimizer Adjusts parameters according to the optimization objective
Validation Set Used for model selection and hyperparameter tuning
Test Set Used to evaluate the model's final generalization performance

The training objective of supervised learning can usually be expressed as:

\theta^*=\arg\min_\theta \frac{1}{N}\sum_{i=1}^{N} L(f_\theta(x_i),y_i)

where:

  • x_i: the input of the i-th training sample.
  • y_i: the corresponding ground-truth label.
  • f_\theta(x_i): the model prediction.
  • L: the loss function.
  • \theta^*: the optimal parameters obtained from training.

In actual training, a regularization term may also be added to reduce the risk of overfitting:

\theta^*=\arg\min_\theta \left[ \frac{1}{N}\sum_{i=1}^{N}L(f_\theta(x_i),y_i) +\lambda R(\theta) \right]

2.5 Advantages and Disadvantages of Supervised Learning

Advantages:

  • The training objective is clear, and evaluation metrics are easy to define.
  • When label quality is high and the data distribution is reasonable, prediction performance is usually good.
  • It can be directly applied to business tasks such as classification and regression.
  • Model training and evaluation processes are relatively mature.

Disadvantages:

  • It requires a large amount of labeled data.
  • High-quality annotation can be very costly.
  • Label errors can affect model learning.
  • When the training data distribution differs from the real-world environment, generalization performance may degrade.
  • Models may easily learn biases in the data rather than genuinely useful patterns.

2.6 Typical Applications of Supervised Learning

Application Area Typical Tasks
Finance Credit scoring, fraud detection
Healthcare Medical image classification, disease risk prediction
E-commerce Product sales forecasting, user purchase prediction
Internet Spam content detection, ad click-through rate prediction
Industry Product defect detection, equipment failure prediction
Natural language processing Text classification, sentiment analysis, named entity recognition
Computer vision Object detection, image classification, semantic segmentation

III. Unsupervised Learning

3.1 What Is Unsupervised Learning?

Unsupervised learning is a method of training on data that lacks explicit human-provided labels.

Unlike supervised learning, unsupervised learning does not provide a predefined correct answer for each sample in advance.

For example, an e-commerce platform has a large amount of user behavior data:

User Number of Views Number of Purchases Average Spending
A 120 15 800 yuan
B 20 2 100 yuan
C 95 12 750 yuan
D 15 1 80 yuan
E 60 8 450 yuan

These data are not explicitly labeled with which type each user belongs to.

Unsupervised learning can analyze similarities among user features and divide users into different groups.

For example:

  • High-frequency spending users.
  • Low-frequency spending users.
  • Potential high-value users.
  • Price-sensitive users.

Rather than predicting based on categories provided by humans, the model discovers latent structure from the data.

3.2 Core Principles of Unsupervised Learning

Unsupervised learning does not mean the model has no optimization objective.

It simply lacks the human-provided target labels used in traditional supervised learning.

The model still needs to learn data features through some objective function, for example:

  • Minimize clustering distance.
  • Maximize data likelihood.
  • Minimize reconstruction error.
  • Maximize consistency between representations.
  • Learn the low-dimensional structure of the data.

Different algorithms may adopt completely different objective functions.

For example, K-Means optimizes the clustering result by minimizing the squared distance between samples and their assigned cluster centers.

J=\sum_{i=1}^{N}\|x_i-\mu_{c_i}\|^2

Where:

  • x_i: the ith sample.
  • \mu_{c_i}: the center of the cluster to which the sample belongs.
  • c_i: the cluster index to which the sample belongs.
  • J: the clustering objective function.

3.3 Main Tasks of Unsupervised Learning

3.3.1 Clustering

Clustering is a method of grouping similar samples into the same group.

Core idea:

  • Samples within the same cluster should be as similar as possible.
  • Samples in different clusters should be as distinct as possible.

Typical algorithms:

K-Means

Divide the data into a pre-specified number K of clusters, and complete clustering by iteratively updating cluster centers.

Main steps:

  1. Specify the number of clusters K.
  2. Initialize K cluster centers.
  3. Assign each sample to the nearest center.
  4. Recalculate the center based on the samples within the cluster.
  5. Repeat assignment and update until a stopping condition is met.

DBSCAN

A density-based clustering algorithm.

Main characteristics:

  • Does not require the number of clusters to be specified in advance.
  • Can discover clusters with non-spherical structures.
  • Can identify noise points and outliers.
  • Is relatively sensitive to parameter selection and data distributions with different densities.

Hierarchical Clustering

Builds a hierarchy by continuously merging or splitting groups of samples.

Commonly used to explore hierarchical relationships among data.

3.3.2 Dimensionality Reduction

Dimensionality reduction is a method of mapping high-dimensional data into a lower-dimensional space while preserving as much important information as possible.

For example, a single user behavior record may contain hundreds of features.

High-dimensional data can bring about:

  • Increased computational cost.
  • Feature redundancy.
  • Accumulated noise.
  • Difficulty in visualization.
  • Degradation of distance metrics in high-dimensional space, among other issues.

Dimensionality reduction can help models extract more compact data representations.

Typical algorithms:

PCA (Principal Component Analysis)

It projects the original data into a lower-dimensional space by finding the orthogonal directions along which the data variance is largest.

The core of PCA is performing eigenvalue decomposition on the data covariance matrix, or using singular value decomposition (SVD) to obtain the principal components.

Note that: maximum variance does not equal maximum task-relevant information. PCA is a linear dimensionality reduction method and is not necessarily suitable for all data structures.

t-SNE

Mainly used for nonlinear visualization of high-dimensional data, preserving local neighborhood relationships as much as possible.

It is suitable for exploring data distributions, but the on-plot distances between different clusters, as well as cluster areas and densities, should not be directly interpreted as strict global structure.

UMAP

A dimensionality reduction method based on manifold learning and topological structure, commonly used for high-dimensional data visualization and representation exploration.

3.3.3 Density Estimation

The goal of density estimation is to learn the probability distribution of the data:

p(x)

By estimating the probability density of data appearing in different regions, one can understand the data distribution and use it for tasks such as anomaly detection and generative modeling.

Common methods:

  • Kernel Density Estimation.
  • Gaussian Mixture Model.
  • Normalizing Flow.

Among them, the Gaussian mixture model approximates complex data distributions through a weighted combination of multiple Gaussian distributions:

p(x)=\sum_{k=1}^{K}\pi_k\mathcal{N}(x|\mu_k,\Sigma_k)

Here \pi_k is the mixture weight, satisfying:

\sum_{k=1}^{K}\pi_k=1

3.3.4 Representation Learning

The goal of representation learning is to automatically learn meaningful feature representations from raw data.

For example:

  • Converting images into semantic vectors.
  • Converting text into embeddings.
  • Converting user behavior into user vectors.
  • Compressing complex data into latent space representations.

Typical methods include:

  • Autoencoder.
  • Contrastive Learning.
  • Self-supervised pre-training.
  • Some generative representation learning methods.

Representation learning is an important foundation of modern deep learning and large model technology.

3.4 Typical Training Workflow of Unsupervised Learning

flowchart TB
    A["Prepare raw data"] --> B["Data cleaning and preprocessing"]
    B --> C["Extract or construct features"]
    C --> D["Select unsupervised learning algorithm"]
    D --> E["Learn data structure or latent representation"]
    E --> F["Optimize model objective function"]
    F --> G["Obtain clustering results or data representation"]
    G --> H["Analyze and evaluate results"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

It should be noted that the results of unsupervised learning do not necessarily have a single correct answer.

For example, user segmentation can be divided along different dimensions such as spending power, purchase frequency, and product preference. Different segmentation approaches may all have business value.

Therefore, in addition to technical metrics such as the silhouette coefficient and reconstruction error, unsupervised learning also requires combining business context to judge whether the results are meaningful.

3.5 Advantages and Disadvantages of Unsupervised Learning

Advantages:

  • Does not depend on large amounts of manually labeled data.
  • Can discover unknown data structures.
  • Can assist in feature extraction and data exploration.
  • Suitable for handling large-scale data with missing labels.
  • Can serve as a preprocessing step for other machine learning tasks.

Disadvantages:

  • Lacking explicit labels, results are harder to verify directly.
  • Different algorithms may yield different structures.
  • Business interpretation of algorithm results may be relatively difficult.
  • Relatively sensitive to feature scale, distance metrics, and hyperparameters.
  • The discovered data structures do not necessarily have practical business value.

3.6 Typical Applications of Unsupervised Learning

Application Area Typical Tasks
E-commerce Customer segmentation, product clustering
Finance Anomaly transaction detection, risk pattern exploration
Cybersecurity Anomalous traffic identification
Healthcare Patient cohort analysis
Recommender systems User and item representation learning
Computer vision Image feature extraction, data clustering
Natural language processing Text clustering, semantic representation learning

4. Reinforcement Learning

4.1 What is Reinforcement Learning?

Reinforcement learning is a machine learning method in which an agent continuously interacts with an environment and learns a decision-making policy based on reward signals.

Unlike supervised learning, reinforcement learning usually does not have a standard answer explicitly provided at every step.

The agent needs to:

  1. Observe the current state of the environment.
  2. Select an action according to the current policy.
  3. Execute the action.
  4. Receive feedback from the environment.
  5. Adjust the decision-making policy based on the reward and the subsequent state.

For example, a robot learning how to walk:

  • The robot is currently in a certain body posture.
  • The model selects joint movement commands.
  • The robot executes the action.
  • The environment provides feedback on whether the robot maintains balance and whether it is moving toward the goal.
  • The model adjusts its policy based on the reward.

Reinforcement learning focuses not on whether a single prediction is accurate, but on whether a sequence of decisions can achieve a higher long-term return.

4.2 Core Components of Reinforcement Learning

Reinforcement learning usually includes the following core concepts:

Concept English Description
Agent Agent The entity that makes decisions and learns
Environment Environment The external system with which the agent interacts
State State Describes the current situation of the environment
Action Action The behavior the agent can take
Reward Reward Feedback provided by the environment on the result of an action
Policy Policy The rule for selecting actions based on the state
Value Function Value Function Measures the long-term return of a state or action
Episode Episode An interaction process from start to termination

4.3 Basic Interaction Mechanism of Reinforcement Learning

Reinforcement learning can be understood through the following process:

flowchart TB
    A["Agent"] -->|"Execute Action"| B["Environment"]
    B -->|"Return State"| A
    B -->|"Return Reward"| A
    A --> C["Evaluate Current Decision"]
    C --> D["Update Policy"]
    D --> A

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#E5F0E9,stroke:#83A995,color:#315D49

This is not a one-time prediction process, but a closed loop of continuous interaction and ongoing policy improvement.

4.4 Markov Decision Process (MDP)

Reinforcement learning usually uses a Markov Decision Process (MDP) to describe the problem.

A standard MDP can be represented as:

\mathcal{M}=(S,A,P,R,\gamma)

Where:

  • S: state space.
  • A: action space.
  • P(s'|s,a): state transition probability.
  • R(s,a,s'): reward function.
  • \gamma: discount factor.

The Markov property means that, given the current state, the probability distribution of future states no longer depends on the complete history, but on the current state and current action.

P(S_{t+1}|S_t,A_t,\ldots,S_0) = P(S_{t+1}|S_t,A_t)

There is an important prerequisite here:

The state must contain sufficient information to satisfy the Markov property.

If the current observation cannot fully describe the environment state, the problem may be better modeled as a partially observable Markov decision process (POMDP).

4.5 Core Optimization Objective of Reinforcement Learning

The goal of reinforcement learning is usually to maximize the long-term cumulative reward, not merely to maximize the reward of the current step.

Define the discounted cumulative return:

G_t=\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}

Where:

  • G_t: the cumulative return starting from time step t.
  • R_{t+k+1}: the reward received in the future.
  • \gamma: the discount factor, usually satisfying 0\leq\gamma<1.

The discount factor determines how important future rewards are relative to the current reward.

  • When \gamma is small, the policy places more emphasis on near-term rewards.
  • When \gamma is large, the policy places more emphasis on long-term returns.

The objective of the policy can be expressed as:

\pi^*=\arg\max_\pi \mathbb{E}_{\pi}[G_t]

where \pi denotes the policy and \pi^* denotes the optimal policy.

4.6 Value Function and Action Value Function

Reinforcement learning needs to evaluate the long-term returns of different states and actions.

状态价值函数(State Value Function)

V^\pi(s)= \mathbb{E}_\pi[G_t|S_t=s]

It represents the expected cumulative return obtainable starting from state s under policy \pi.

动作价值函数(Action Value Function)

Q^\pi(s,a)= \mathbb{E}_\pi[G_t|S_t=s,A_t=a]

It represents the expected cumulative return obtainable when, in state s, action a is taken first and the policy \pi is then followed.

The difference between the two:

  • V(s): evaluates how good the current state is overall.
  • Q(s,a): evaluates how good taking a particular action in the current state is.

4.7 Bellman Equation

The Bellman equation is an important mathematical foundation of reinforcement learning.

It expresses the value of the current state as a combination of the immediate reward and the value of future states.

For the state value function:

V^\pi(s)= \sum_a\pi(a|s) \sum_{s'}P(s'|s,a) \left[ R(s,a,s')+\gamma V^\pi(s') \right]

For the action value function:

Q^\pi(s,a)= \sum_{s'}P(s'|s,a) \left[ R(s,a,s')+ \gamma\sum_{a'}\pi(a'|s')Q^\pi(s',a') \right]

The Bellman equation reveals the core idea of reinforcement learning:

The value of a current decision depends not only on the current reward, but also on the future state it leads to and the returns that follow.

This is also an important reason why reinforcement learning differs from simple immediate-feedback optimization.

4.8 Main Algorithms of Reinforcement Learning

Reinforcement learning algorithms can be classified from different perspectives.

4.8.1 Value-Based Methods

These methods focus on learning the state value or the action value, and then select actions based on the value.

Q-Learning

Q-Learning is a classic model-free, off-policy reinforcement learning algorithm.

Its update formula:

Q(s,a)\leftarrow Q(s,a)+ \alpha \left[ r+\gamma\max_{a'}Q(s',a')-Q(s,a) \right]

Where:

  • \alpha: the learning rate.
  • r: the current reward.
  • \gamma: the discount factor.
  • s': the new state after taking the action.

By continuously revising the Q values, it gradually approaches the optimal action value function.

SARSA

SARSA is a typical on-policy algorithm.

Its update formula:

Q(s,a)\leftarrow Q(s,a)+ \alpha \left[ r+\gamma Q(s',a')-Q(s,a) \right]

The key differences from Q-Learning:

  • Q-Learning updates using the maximum Q value in the next state.
  • SARSA updates using the next action actually selected by the policy.

DQN(Deep Q-Network)

DQN introduces deep neural networks into Q-Learning, using a neural network to approximate the action value function:

Q(s,a;\theta)

Classic DQN uses Experience Replay and a Target Network to improve training stability.

4.8.2 Policy-Based Methods

These methods learn the policy directly:

\pi_\theta(a|s)

The model directly outputs an action probability distribution or a continuous action based on the current state.

Policy Gradient

The expected return of the policy is optimized via gradient ascent:

\nabla_\theta J(\theta) = \mathbb{E} \left[ \nabla_\theta\log\pi_\theta(a|s)G_t \right]

The core idea is to increase the probability of actions that yield higher returns and decrease the probability of actions that perform poorly.

PPO (Proximal Policy Optimization)

PPO is one of the most widely used policy optimization algorithms today.

By limiting the magnitude of updates between the old and new policies, it reduces the risk of training instability caused by excessively large policy updates.

PPO is commonly used in robot control, game agents, and the post-training pipelines of some large models.

4.8.3 Actor-Critic Methods

Actor-Critic combines policy learning with value evaluation.

  • Actor: responsible for selecting actions based on the state.
  • Critic: responsible for evaluating the value of a state or action.

The Actor updates its policy based on the evaluation signal provided by the Critic.

Typical algorithms:

  • A2C
  • A3C
  • DDPG
  • TD3
  • SAC

Among them:

  • DDPG is mainly designed for continuous action spaces.
  • TD3 mitigates value overestimation and training instability through several mechanisms.
  • SAC balances reward maximization and policy exploration through a maximum entropy objective.