1. Introduction
A neural network computes predictions through forward propagation, measures prediction error with a loss function, and then computes the gradients of the loss with respect to the model parameters through backpropagation.
However, after computing the gradients, a key question still needs to be addressed:
How should the model parameters be adjusted based on the gradients so that the loss function gradually decreases?
This is the problem that optimization algorithms must solve.
Gradient descent is the most basic parameter optimization method. Building on it, researchers have proposed optimization algorithms such as stochastic gradient descent (SGD), Momentum, AdaGrad, RMSProp, and Adam to improve training efficiency, convergence speed, and training stability.
This article focuses on the following:
- The mathematical principles of gradient descent and the parameter update mechanism.
- The differences between Batch Gradient Descent, SGD, and Mini-batch SGD.
- Learning rate and its impact on the training process.
- The Momentum optimization mechanism.
- The adaptive learning rate mechanisms of AdaGrad and RMSProp.
- Adam's first moment, second moment, and bias correction.
- The differences between Adam, AdamW, and SGD.
- Learning rate scheduling, gradient clipping, and optimizer selection.
- Engineering practices for optimizers in PyTorch.
2. What Problem Do Optimization Algorithms Solve?
The goal of neural network training is to find a set of model parameters that make the loss function as small as possible.
Suppose the model parameters are:
The training objective of the model can be expressed as:
Where:
- \theta: model parameters.
- L(\theta): loss function.
- \theta^*: the parameters that minimize the loss function.
In actual training, the loss function is usually composed of the losses of multiple samples in the training data:
Where:
- N: number of training samples.
- x_i: the i-th input sample.
- y_i: the true label.
- f(x_i;\theta): model prediction.
- \ell: loss function for a single sample.
For complex neural networks, it is usually impossible to directly obtain an analytical solution for the optimal parameters, so numerical optimization algorithms must be used to continuously adjust the parameters.
The core task of an optimization algorithm is to:
- Compute the model loss based on the current parameters.
- Compute the gradients of the loss with respect to the parameters.
- Determine the parameter adjustment direction based on the gradients.
- Update the model parameters.
- Repeat the above process until the training stopping condition is met.
2.1 The Complete Model Training Process
flowchart TB
A["Initialize model parameters"]
B["Load training data"]
C["Forward propagation"]
D["Compute loss function"]
E["Backpropagation to compute gradients"]
F["Optimizer updates parameters"]
G{"Has the stopping condition been met?"}
H["Save the trained model"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G -->|"No"| B
G -->|"Yes"| H
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Three concepts need to be distinguished:
- Loss function: defines the objective that the model needs to optimize.
- Backpropagation: computes the gradients of the objective function with respect to the model parameters.
- Optimizer: updates the model parameters based on the gradients and its own state.
Together, these three form the parameter learning mechanism of a neural network.
3. Gradient Descent: The Most Basic Parameter Optimization Algorithm
Gradient descent is one of the most classic numerical optimization algorithms.
Its core idea is:
Adjust the parameters in the direction opposite to the gradient of the loss function, so that the loss function gradually decreases.
3.1 The Directional Meaning of the Gradient
Suppose the loss function is:
Its gradient is:
The gradient indicates the direction in which the function increases fastest at the current position.
Therefore, the negative gradient direction corresponds to the direction of fastest local decrease.
Parameter update formula:
where:
- \theta_t: the parameter at iteration t.
- \theta_{t+1}: the updated parameter.
- \eta: learning rate.
- \nabla_\theta L(\theta_t): the gradient at the current parameter position.
The learning rate determines the step size of each parameter update.
3.2 One-Dimensional Gradient Descent Example
Suppose we need to minimize:
Derivative:
Set:
First gradient computation:
Update the parameter:
Second gradient computation:
Update:
Continue iterating:
| Iteration | Parameter w | Loss L(w) |
|---|---|---|
| 0 | 0.0000 | 9.0000 |
| 1 | 0.6000 | 5.7600 |
| 2 | 1.0800 | 3.6864 |
| 3 | 1.4640 | 2.3593 |
| 4 | 1.7712 | 1.5099 |
| 5 | 2.0170 | 0.9664 |
As the parameter gradually approaches the optimum:
the loss also gradually approaches zero.
3.3 Intuitive Process of Gradient Descent
flowchart TB
A["Current position: relatively high loss"]
B["Compute gradient at current position"]
C["Determine negative gradient direction"]
D["Move parameter according to learning rate"]
E["Reach new parameter position"]
F{"Whether stopping condition is met"}
A --> B
B --> C
C --> D
D --> E
E --> F
F -->|"No"| B
F -->|"Yes"| G["Obtain current optimization result"]
style A fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
3.4 Limitations of Gradient Descent
Standard gradient descent needs to compute the gradient over the entire training set before each parameter update.
When the dataset is very large, this brings a high computational cost.
For example:
- The training set has 1 million samples.
- Every parameter update requires processing all samples.
- Completing a single parameter update requires a large amount of computation.
In addition, standard gradient descent has the following problems:
- The parameter update frequency is low.
- The computational cost of a single iteration is high.
- As the dataset size grows, memory and computational pressure increase.
- For non-convex loss functions, it may still be affected by issues such as saddle points and flat regions.
Therefore, deep learning usually adopts gradient optimization methods based on mini-batches of data.
IV. Batch GD, SGD, and Mini-batch SGD
Gradient descent can be divided into three main forms according to the number of training samples used for each parameter update.
4.1 Batch Gradient Descent
Batch gradient descent updates parameters using all training samples to compute the gradient each time.
Formula:
where N is the number of samples in the training set.
Characteristics:
- Each update uses all training data.
- The gradient is usually closer to the gradient of the full training set objective.
- The update direction is relatively stable.
- The larger the dataset, the higher the cost of a single update.
It is suitable for scenarios with a small amount of data that can be processed within the allowed range of memory or computational resources.
4.2 Stochastic Gradient Descent
Stochastic gradient descent (SGD) usually refers to computing the gradient using one randomly sampled training example each time.
Formula:
where i denotes the index of the randomly selected sample.
Characteristics:
- Each update requires only one sample.
- The parameter update frequency is high.
- The computational cost per update is low.
- The gradient estimate has relatively large noise.
- The loss curve may fluctuate noticeably.
The randomness of SGD comes from using only part of the data each time to estimate the gradient of the full training objective.
When samples are drawn randomly and the corresponding conditions are satisfied, the single-sample gradient can serve as an unbiased estimate of the overall gradient.
4.3 Mini-batch SGD
Mini-batch stochastic gradient descent (Mini-batch SGD) computes the gradient using a batch of training samples each time.
Suppose the batch size is B:
where:
- B: batch size.
- \mathcal B_t: the set of samples used in the current iteration.
- \eta: learning rate.
It achieves a good balance between computational efficiency and gradient stability.
Modern deep learning training usually adopts the Mini-batch SGD form.
Note that in practical engineering, people often directly refer to training with mini-batches as SGD, especially when using improved versions such as Momentum.
4.4 Comparison of the Three Approaches
| Comparison Dimension | Batch GD | SGD | Mini-batch SGD |
|---|---|---|---|
| Number of samples per update | All samples | 1 | A batch of samples |
| Computational cost per update | High | Low | Moderate |
| Update frequency | Low | High | Relatively high |
| Gradient fluctuation | Relatively small | Relatively large | Moderate |
| Parallel computation | Yes | Limited by single-sample granularity | Suitable for batch parallelism |
| Deep learning applications | Relatively rare | Single-sample training is less common | Very common |
4.5 Impact of Batch Size on Training
Batch size is an important hyperparameter during training.
Smaller batch size:
- Less computation per update.
- More parameter updates.
- Noisier gradient estimates.
- May provide a certain regularization effect.
- Loss fluctuations during training may be more pronounced.
Larger batch size:
- Better suited to GPU batch parallel computation.
- Gradient estimates per update are usually more stable.
- Fewer parameter updates per epoch.
- May result in higher GPU memory usage.
- Under the same training configuration, it does not necessarily achieve better generalization.
It cannot simply be assumed that the larger the batch size, the better; in practice, it needs to be adjusted together with the learning rate, model size, hardware resources, and training data.
V. Learning Rate: The Key Hyperparameter Determining the Magnitude of Parameter Updates
The learning rate (LR) is one of the most important hyperparameters in gradient optimization.
It determines the magnitude by which the optimizer adjusts the parameters according to the gradient each time.
Taking gradient descent as an example:
Here, \eta is the learning rate.
5.1 Learning Rate Too Large
If the learning rate is too large, the magnitude of each parameter update may exceed a reasonable range.
Typical symptoms:
- The loss function oscillates violently.
- Parameters bounce back and forth around the optimum.
- The loss rises instead of falling.
- Training may diverge outright.
5.2 Learning Rate Too Small
If the learning rate is too small, parameter updates are too small in magnitude.
Typical symptoms:
- The loss decreases very slowly.
- Training requires more iterations.
- Training may linger in flat regions for a long time.
- It is difficult to reach a satisfactory result under a limited training budget.
5.3 Illustration of the Learning Rate's Impact
flowchart TB
A["Choose a learning rate"]
B["Learning rate too large"]
C["Learning rate appropriate"]
D["Learning rate too small"]
E["Loss oscillates or training diverges"]
F["Loss decreases steadily"]
G["Convergence is slow"]
A --> B
A --> C
A --> D
B --> E
C --> F
D --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
5.4 Fixed Learning Rate vs. Dynamic Learning Rate
Fixed learning rate:
The same learning rate is used throughout the entire training process.
Dynamic learning rate:
The learning rate is adjusted according to training progress, loss changes, or a preset scheduling strategy.
Common learning rate scheduling strategies:
| Scheduling Strategy | Core Mechanism |
|---|---|
| Step Decay | Reduce the learning rate every specified number of Epochs |
| Exponential Decay | Decay according to an exponential rule |
| Cosine Annealing | Gradually reduce the learning rate along a cosine curve |
| ReduceLROnPlateau | Reduce the learning rate when the validation metric plateaus |
| Warmup | Gradually increase the learning rate in the early stage of training |
| Cosine Warmup | Warm up first, then apply cosine decay |
Learning rate scheduling is not a gradient computation rule of the optimizer itself, but a strategy for controlling how the learning rate changes during training.
6. Momentum: Introducing Historical Gradient Information
Ordinary SGD updates parameters using only the gradient of the current batch each time.
If the gradient directions of different batches fluctuate considerably, parameter updates may oscillate noticeably.
Momentum improves on this problem by introducing historical gradient information.
6.1 The Basic Idea of Momentum
Momentum can be understood as:
Considering not only the current gradient, but also the update direction accumulated previously.
Its common form is:
Where:
- g_t: the current gradient.
- v_t: the momentum variable.
- \beta: the momentum coefficient.
- \eta: the learning rate.
The momentum coefficient is usually set to a value close to 1, such as 0.9.
6.2 How Does Momentum Improve Training?
Suppose the gradient directions of several consecutive batches are largely consistent.
Ordinary SGD uses only the current gradient each time.
Momentum continuously accumulates historical gradients, strengthening the effect of updates along consistent directions.
If the gradient direction changes repeatedly, the historical gradients also smooth the current update.
Therefore, Momentum can generally:
- Accelerate parameter updates along stable directions.
- Reduce oscillation in some directions.
- Improve optimization performance on certain narrow, elongated loss surfaces.
- Increase training efficiency.
6.3 Comparison of SGD and Momentum
Ordinary SGD:
Momentum:
Momentum does not simply amplify the learning rate; it introduces a historical gradient state.
Note that different optimizer implementations may use different momentum definitions and initialization methods, so the formula must be kept consistent with the specific implementation.
7. AdaGrad: Adjusting the Learning Rate Based on Historical Gradients
Ordinary SGD uses a uniform learning rate.
However, across different parameter dimensions, the magnitude and frequency of gradients may differ greatly.
For example:
- Some parameters frequently receive relatively large gradients.
- Some parameters rarely receive gradients.
- The gradient scales differ across parameter dimensions.
AdaGrad (Adaptive Gradient Algorithm) adaptively adjusts the learning rate for different parameter dimensions by accumulating the squares of historical gradients.
7.1 AdaGrad Update Formula
First, accumulate the squared gradients:
Then update the parameters:
Where:
- r_t: Accumulated sum of squared historical gradients.
- g_t: Current gradient.
- \eta: Initial learning rate.
- \epsilon: Numerical stability term.
Here, squaring and taking the square root are usually element-wise operations.
7.2 Core Characteristics of AdaGrad
For parameters with larger historical gradients:
will gradually increase, and the corresponding effective learning rate will decrease.
For parameters with smaller historical gradients, the effective learning rate is relatively large.
Therefore, AdaGrad can adjust the update magnitude for different parameter dimensions.
7.3 Limitations of AdaGrad
AdaGrad continuously accumulates the squares of historical gradients.
As training progresses:
it usually keeps increasing.
This may cause the effective learning rate to keep shrinking and even become very small in the later stages of training.
Therefore, in some continuous training scenarios, AdaGrad may suffer from overly rapid learning rate decay.
8. RMSProp: Addressing AdaGrad’s Learning Rate Decay Problem
RMSProp (Root Mean Square Propagation) accumulates squared gradients via an exponential moving average.
Unlike AdaGrad, it does not accumulate all historical squared gradients without bound; instead, it gives higher weight to recent gradients.
8.1 RMSProp Update Formula
First, compute the exponential moving average of the squared gradients:
Then update the parameters:
Where:
- s_t: Exponential moving average of squared gradients.
- \rho: Decay coefficient.
- \eta: Learning rate.
- \epsilon: Numerical stability term.
8.2 Differences Between RMSProp and AdaGrad
| Comparison Dimension | AdaGrad | RMSProp |
|---|---|---|
| Gradient Statistics Method | Accumulated squared gradients | Exponential moving average |
| Historical Information | Continuous accumulation | Higher weight on recent information |
| Effective Learning Rate | Usually keeps decreasing | Can adjust dynamically with gradient statistics |
| Main Advantage | Adapts to sparse gradients | Mitigates overly rapid learning rate decay |
Through its moving average mechanism, RMSProp prevents the effective learning rate from continuously shrinking in one direction due to unbounded accumulation of historical squared gradients.
9. Adam: Combining Momentum and Adaptive Learning Rates
Adam (Adaptive Moment Estimation) is a widely used optimization algorithm in deep learning.
It combines two ideas:
- Momentum: uses first-moment information from historical gradients.
- RMSProp: uses second-moment information from historical squared gradients.
Adam simultaneously maintains first- and second-moment estimates of the gradients and applies bias correction.
9.1 Core Variables of Adam
Let the current gradient be:
First-moment estimate:
Second-moment estimate:
Where:
- m_t: First-moment estimate of the gradient.
- v_t: Second raw moment estimate of the squared gradient.
- \beta_1: First-moment decay coefficient.
- \beta_2: Second-moment decay coefficient.
Note: Here, v_t is the exponential moving average of the squared gradient, not a statistical variance computed directly in the strict sense.
9.2 Why Is Bias Correction Needed?
Adam typically initializes the first- and second-moment variables to zero:
In the early stage of training, since the historical state has not yet accumulated, the moment estimates tend to be biased toward zero.
To address this, Adam introduces bias correction:
Where:
- \hat{m}_t: the bias-corrected first moment estimate.
- \hat{v}_t: the bias-corrected second moment estimate.
Bias correction mitigates the effect of zero initialization on the moment estimates in the early stage of training.
9.3 Adam Parameter Update Formula
The final parameter update:
Where:
- \eta: the learning rate.
- \hat{m}_t: the bias-corrected first moment.
- \hat{v}_t: the bias-corrected second moment.
- \epsilon: the numerical stability term.
This is the core update mechanism of Adam.
9.4 The Complete Computation Flow of Adam
flowchart TB
A["Compute the current gradient g"]
B["Update the first moment m"]
C["Update the second moment v"]
D["Apply first moment bias correction"]
E["Apply second moment bias correction"]
F["Compute the adaptive update amount"]
G["Update the model parameters"]
A --> B
A --> C
B --> D
C --> E
D --> F
E --> F
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
9.5 Adam's Core Advantages
Combined First and Second Moments
The first moment uses directional information from historical gradients, while the second moment adjusts the update scale across different parameter dimensions.
Adaptive Learning Rate