Back to series
· 10 min read

Gradient Descent, SGD, and Adam: Understanding Parameter Optimization Algorithms in Neural Networks


1. Introduction

A neural network computes predictions through forward propagation, measures prediction error with a loss function, and then computes the gradients of the loss with respect to the model parameters through backpropagation.

However, after computing the gradients, a key question still needs to be addressed:

How should the model parameters be adjusted based on the gradients so that the loss function gradually decreases?

This is the problem that optimization algorithms must solve.

Gradient descent is the most basic parameter optimization method. Building on it, researchers have proposed optimization algorithms such as stochastic gradient descent (SGD), Momentum, AdaGrad, RMSProp, and Adam to improve training efficiency, convergence speed, and training stability.

This article focuses on the following:

  • The mathematical principles of gradient descent and the parameter update mechanism.
  • The differences between Batch Gradient Descent, SGD, and Mini-batch SGD.
  • Learning rate and its impact on the training process.
  • The Momentum optimization mechanism.
  • The adaptive learning rate mechanisms of AdaGrad and RMSProp.
  • Adam's first moment, second moment, and bias correction.
  • The differences between Adam, AdamW, and SGD.
  • Learning rate scheduling, gradient clipping, and optimizer selection.
  • Engineering practices for optimizers in PyTorch.

2. What Problem Do Optimization Algorithms Solve?

The goal of neural network training is to find a set of model parameters that make the loss function as small as possible.

Suppose the model parameters are:

\theta=\{W,b\}

The training objective of the model can be expressed as:

\boxed{ \theta^*=\arg\min_{\theta}L(\theta) }

Where:

  • \theta: model parameters.
  • L(\theta): loss function.
  • \theta^*: the parameters that minimize the loss function.

In actual training, the loss function is usually composed of the losses of multiple samples in the training data:

L(\theta)=\frac{1}{N}\sum_{i=1}^{N} \ell(f(x_i;\theta),y_i)

Where:

  • N: number of training samples.
  • x_i: the i-th input sample.
  • y_i: the true label.
  • f(x_i;\theta): model prediction.
  • \ell: loss function for a single sample.

For complex neural networks, it is usually impossible to directly obtain an analytical solution for the optimal parameters, so numerical optimization algorithms must be used to continuously adjust the parameters.

The core task of an optimization algorithm is to:

  1. Compute the model loss based on the current parameters.
  2. Compute the gradients of the loss with respect to the parameters.
  3. Determine the parameter adjustment direction based on the gradients.
  4. Update the model parameters.
  5. Repeat the above process until the training stopping condition is met.

2.1 The Complete Model Training Process

flowchart TB
    A["Initialize model parameters"]
    B["Load training data"]
    C["Forward propagation"]
    D["Compute loss function"]
    E["Backpropagation to compute gradients"]
    F["Optimizer updates parameters"]
    G{"Has the stopping condition been met?"}
    H["Save the trained model"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G -->|"No"| B
    G -->|"Yes"| H

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Three concepts need to be distinguished:

  • Loss function: defines the objective that the model needs to optimize.
  • Backpropagation: computes the gradients of the objective function with respect to the model parameters.
  • Optimizer: updates the model parameters based on the gradients and its own state.

Together, these three form the parameter learning mechanism of a neural network.


3. Gradient Descent: The Most Basic Parameter Optimization Algorithm

Gradient descent is one of the most classic numerical optimization algorithms.

Its core idea is:

Adjust the parameters in the direction opposite to the gradient of the loss function, so that the loss function gradually decreases.

3.1 The Directional Meaning of the Gradient

Suppose the loss function is:

L(\theta)

Its gradient is:

\nabla_\theta L(\theta)

The gradient indicates the direction in which the function increases fastest at the current position.

Therefore, the negative gradient direction corresponds to the direction of fastest local decrease.

Parameter update formula:

\boxed{ \theta_{t+1} = \theta_t-\eta\nabla_\theta L(\theta_t) }

where:

  • \theta_t: the parameter at iteration t.
  • \theta_{t+1}: the updated parameter.
  • \eta: learning rate.
  • \nabla_\theta L(\theta_t): the gradient at the current parameter position.

The learning rate determines the step size of each parameter update.

3.2 One-Dimensional Gradient Descent Example

Suppose we need to minimize:

L(w)=(w-3)^2

Derivative:

\frac{dL}{dw}=2(w-3)

Set:

w_0=0
\eta=0.1

First gradient computation:

g_0=2(0-3)=-6

Update the parameter:

w_1=0-0.1\times(-6)=0.6

Second gradient computation:

g_1=2(0.6-3)=-4.8

Update:

w_2=0.6-0.1\times(-4.8)=1.08

Continue iterating:

Iteration Parameter w Loss L(w)
0 0.0000 9.0000
1 0.6000 5.7600
2 1.0800 3.6864
3 1.4640 2.3593
4 1.7712 1.5099
5 2.0170 0.9664

As the parameter gradually approaches the optimum:

w^*=3

the loss also gradually approaches zero.

3.3 Intuitive Process of Gradient Descent

flowchart TB
    A["Current position: relatively high loss"]
    B["Compute gradient at current position"]
    C["Determine negative gradient direction"]
    D["Move parameter according to learning rate"]
    E["Reach new parameter position"]
    F{"Whether stopping condition is met"}

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F -->|"No"| B
    F -->|"Yes"| G["Obtain current optimization result"]

    style A fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

3.4 Limitations of Gradient Descent

Standard gradient descent needs to compute the gradient over the entire training set before each parameter update.

When the dataset is very large, this brings a high computational cost.

For example:

  • The training set has 1 million samples.
  • Every parameter update requires processing all samples.
  • Completing a single parameter update requires a large amount of computation.

In addition, standard gradient descent has the following problems:

  • The parameter update frequency is low.
  • The computational cost of a single iteration is high.
  • As the dataset size grows, memory and computational pressure increase.
  • For non-convex loss functions, it may still be affected by issues such as saddle points and flat regions.

Therefore, deep learning usually adopts gradient optimization methods based on mini-batches of data.


IV. Batch GD, SGD, and Mini-batch SGD

Gradient descent can be divided into three main forms according to the number of training samples used for each parameter update.

4.1 Batch Gradient Descent

Batch gradient descent updates parameters using all training samples to compute the gradient each time.

Formula:

\theta_{t+1} = \theta_t-\eta \frac{1}{N} \sum_{i=1}^{N} \nabla_\theta\ell_i(\theta_t)

where N is the number of samples in the training set.

Characteristics:

  • Each update uses all training data.
  • The gradient is usually closer to the gradient of the full training set objective.
  • The update direction is relatively stable.
  • The larger the dataset, the higher the cost of a single update.

It is suitable for scenarios with a small amount of data that can be processed within the allowed range of memory or computational resources.

4.2 Stochastic Gradient Descent

Stochastic gradient descent (SGD) usually refers to computing the gradient using one randomly sampled training example each time.

Formula:

\theta_{t+1} = \theta_t-\eta \nabla_\theta\ell_i(\theta_t)

where i denotes the index of the randomly selected sample.

Characteristics:

  • Each update requires only one sample.
  • The parameter update frequency is high.
  • The computational cost per update is low.
  • The gradient estimate has relatively large noise.
  • The loss curve may fluctuate noticeably.

The randomness of SGD comes from using only part of the data each time to estimate the gradient of the full training objective.

When samples are drawn randomly and the corresponding conditions are satisfied, the single-sample gradient can serve as an unbiased estimate of the overall gradient.

4.3 Mini-batch SGD

Mini-batch stochastic gradient descent (Mini-batch SGD) computes the gradient using a batch of training samples each time.

Suppose the batch size is B:

\boxed{ \theta_{t+1} = \theta_t-\eta \frac{1}{B} \sum_{i\in\mathcal B_t} \nabla_\theta\ell_i(\theta_t) }

where:

  • B: batch size.
  • \mathcal B_t: the set of samples used in the current iteration.
  • \eta: learning rate.

It achieves a good balance between computational efficiency and gradient stability.

Modern deep learning training usually adopts the Mini-batch SGD form.

Note that in practical engineering, people often directly refer to training with mini-batches as SGD, especially when using improved versions such as Momentum.

4.4 Comparison of the Three Approaches

Comparison Dimension Batch GD SGD Mini-batch SGD
Number of samples per update All samples 1 A batch of samples
Computational cost per update High Low Moderate
Update frequency Low High Relatively high
Gradient fluctuation Relatively small Relatively large Moderate
Parallel computation Yes Limited by single-sample granularity Suitable for batch parallelism
Deep learning applications Relatively rare Single-sample training is less common Very common

4.5 Impact of Batch Size on Training

Batch size is an important hyperparameter during training.

Smaller batch size:

  • Less computation per update.
  • More parameter updates.
  • Noisier gradient estimates.
  • May provide a certain regularization effect.
  • Loss fluctuations during training may be more pronounced.

Larger batch size:

  • Better suited to GPU batch parallel computation.
  • Gradient estimates per update are usually more stable.
  • Fewer parameter updates per epoch.
  • May result in higher GPU memory usage.
  • Under the same training configuration, it does not necessarily achieve better generalization.

It cannot simply be assumed that the larger the batch size, the better; in practice, it needs to be adjusted together with the learning rate, model size, hardware resources, and training data.


V. Learning Rate: The Key Hyperparameter Determining the Magnitude of Parameter Updates

The learning rate (LR) is one of the most important hyperparameters in gradient optimization.

It determines the magnitude by which the optimizer adjusts the parameters according to the gradient each time.

Taking gradient descent as an example:

\theta_{t+1} = \theta_t-\eta\nabla_\theta L(\theta_t)

Here, \eta is the learning rate.

5.1 Learning Rate Too Large

If the learning rate is too large, the magnitude of each parameter update may exceed a reasonable range.

Typical symptoms:

  • The loss function oscillates violently.
  • Parameters bounce back and forth around the optimum.
  • The loss rises instead of falling.
  • Training may diverge outright.

5.2 Learning Rate Too Small

If the learning rate is too small, parameter updates are too small in magnitude.

Typical symptoms:

  • The loss decreases very slowly.
  • Training requires more iterations.
  • Training may linger in flat regions for a long time.
  • It is difficult to reach a satisfactory result under a limited training budget.

5.3 Illustration of the Learning Rate's Impact

flowchart TB
    A["Choose a learning rate"]
    B["Learning rate too large"]
    C["Learning rate appropriate"]
    D["Learning rate too small"]
    E["Loss oscillates or training diverges"]
    F["Loss decreases steadily"]
    G["Convergence is slow"]

    A --> B
    A --> C
    A --> D
    B --> E
    C --> F
    D --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

5.4 Fixed Learning Rate vs. Dynamic Learning Rate

Fixed learning rate:

The same learning rate is used throughout the entire training process.

Dynamic learning rate:

The learning rate is adjusted according to training progress, loss changes, or a preset scheduling strategy.

Common learning rate scheduling strategies:

Scheduling Strategy Core Mechanism
Step Decay Reduce the learning rate every specified number of Epochs
Exponential Decay Decay according to an exponential rule
Cosine Annealing Gradually reduce the learning rate along a cosine curve
ReduceLROnPlateau Reduce the learning rate when the validation metric plateaus
Warmup Gradually increase the learning rate in the early stage of training
Cosine Warmup Warm up first, then apply cosine decay

Learning rate scheduling is not a gradient computation rule of the optimizer itself, but a strategy for controlling how the learning rate changes during training.


6. Momentum: Introducing Historical Gradient Information

Ordinary SGD updates parameters using only the gradient of the current batch each time.

If the gradient directions of different batches fluctuate considerably, parameter updates may oscillate noticeably.

Momentum improves on this problem by introducing historical gradient information.

6.1 The Basic Idea of Momentum

Momentum can be understood as:

Considering not only the current gradient, but also the update direction accumulated previously.

Its common form is:

v_t=\beta v_{t-1}+g_t
\theta_{t+1}=\theta_t-\eta v_t

Where:

  • g_t: the current gradient.
  • v_t: the momentum variable.
  • \beta: the momentum coefficient.
  • \eta: the learning rate.

The momentum coefficient is usually set to a value close to 1, such as 0.9.

6.2 How Does Momentum Improve Training?

Suppose the gradient directions of several consecutive batches are largely consistent.

Ordinary SGD uses only the current gradient each time.

Momentum continuously accumulates historical gradients, strengthening the effect of updates along consistent directions.

If the gradient direction changes repeatedly, the historical gradients also smooth the current update.

Therefore, Momentum can generally:

  • Accelerate parameter updates along stable directions.
  • Reduce oscillation in some directions.
  • Improve optimization performance on certain narrow, elongated loss surfaces.
  • Increase training efficiency.

6.3 Comparison of SGD and Momentum

Ordinary SGD:

\theta_{t+1}=\theta_t-\eta g_t

Momentum:

v_t=\beta v_{t-1}+g_t
\theta_{t+1}=\theta_t-\eta v_t

Momentum does not simply amplify the learning rate; it introduces a historical gradient state.

Note that different optimizer implementations may use different momentum definitions and initialization methods, so the formula must be kept consistent with the specific implementation.


7. AdaGrad: Adjusting the Learning Rate Based on Historical Gradients

Ordinary SGD uses a uniform learning rate.

However, across different parameter dimensions, the magnitude and frequency of gradients may differ greatly.

For example:

  • Some parameters frequently receive relatively large gradients.
  • Some parameters rarely receive gradients.
  • The gradient scales differ across parameter dimensions.

AdaGrad (Adaptive Gradient Algorithm) adaptively adjusts the learning rate for different parameter dimensions by accumulating the squares of historical gradients.

7.1 AdaGrad Update Formula

First, accumulate the squared gradients:

r_t=r_{t-1}+g_t^2

Then update the parameters:

\boxed{ \theta_{t+1} = \theta_t- \frac{\eta}{\sqrt{r_t}+\epsilon}g_t }

Where:

  • r_t: Accumulated sum of squared historical gradients.
  • g_t: Current gradient.
  • \eta: Initial learning rate.
  • \epsilon: Numerical stability term.

Here, squaring and taking the square root are usually element-wise operations.

7.2 Core Characteristics of AdaGrad

For parameters with larger historical gradients:

\sqrt{r_t}

will gradually increase, and the corresponding effective learning rate will decrease.

For parameters with smaller historical gradients, the effective learning rate is relatively large.

Therefore, AdaGrad can adjust the update magnitude for different parameter dimensions.

7.3 Limitations of AdaGrad

AdaGrad continuously accumulates the squares of historical gradients.

As training progresses:

r_t=r_{t-1}+g_t^2

it usually keeps increasing.

This may cause the effective learning rate to keep shrinking and even become very small in the later stages of training.

Therefore, in some continuous training scenarios, AdaGrad may suffer from overly rapid learning rate decay.


8. RMSProp: Addressing AdaGrad’s Learning Rate Decay Problem

RMSProp (Root Mean Square Propagation) accumulates squared gradients via an exponential moving average.

Unlike AdaGrad, it does not accumulate all historical squared gradients without bound; instead, it gives higher weight to recent gradients.

8.1 RMSProp Update Formula

First, compute the exponential moving average of the squared gradients:

s_t=\rho s_{t-1}+(1-\rho)g_t^2

Then update the parameters:

\boxed{ \theta_{t+1} = \theta_t- \frac{\eta}{\sqrt{s_t}+\epsilon}g_t }

Where:

  • s_t: Exponential moving average of squared gradients.
  • \rho: Decay coefficient.
  • \eta: Learning rate.
  • \epsilon: Numerical stability term.

8.2 Differences Between RMSProp and AdaGrad

Comparison Dimension AdaGrad RMSProp
Gradient Statistics Method Accumulated squared gradients Exponential moving average
Historical Information Continuous accumulation Higher weight on recent information
Effective Learning Rate Usually keeps decreasing Can adjust dynamically with gradient statistics
Main Advantage Adapts to sparse gradients Mitigates overly rapid learning rate decay

Through its moving average mechanism, RMSProp prevents the effective learning rate from continuously shrinking in one direction due to unbounded accumulation of historical squared gradients.


9. Adam: Combining Momentum and Adaptive Learning Rates

Adam (Adaptive Moment Estimation) is a widely used optimization algorithm in deep learning.

It combines two ideas:

  1. Momentum: uses first-moment information from historical gradients.
  2. RMSProp: uses second-moment information from historical squared gradients.

Adam simultaneously maintains first- and second-moment estimates of the gradients and applies bias correction.

9.1 Core Variables of Adam

Let the current gradient be:

g_t=\nabla_\theta L(\theta_t)

First-moment estimate:

m_t=\beta_1m_{t-1}+(1-\beta_1)g_t

Second-moment estimate:

v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2

Where:

  • m_t: First-moment estimate of the gradient.
  • v_t: Second raw moment estimate of the squared gradient.
  • \beta_1: First-moment decay coefficient.
  • \beta_2: Second-moment decay coefficient.

Note: Here, v_t is the exponential moving average of the squared gradient, not a statistical variance computed directly in the strict sense.

9.2 Why Is Bias Correction Needed?

Adam typically initializes the first- and second-moment variables to zero:

m_0=0
v_0=0

In the early stage of training, since the historical state has not yet accumulated, the moment estimates tend to be biased toward zero.

To address this, Adam introduces bias correction:

\hat{m}_t= \frac{m_t}{1-\beta_1^t}
\hat{v}_t= \frac{v_t}{1-\beta_2^t}

Where:

  • \hat{m}_t: the bias-corrected first moment estimate.
  • \hat{v}_t: the bias-corrected second moment estimate.

Bias correction mitigates the effect of zero initialization on the moment estimates in the early stage of training.

9.3 Adam Parameter Update Formula

The final parameter update:

\boxed{ \theta_{t+1} = \theta_t- \eta \frac{\hat{m}_t} {\sqrt{\hat{v}_t}+\epsilon} }

Where:

  • \eta: the learning rate.
  • \hat{m}_t: the bias-corrected first moment.
  • \hat{v}_t: the bias-corrected second moment.
  • \epsilon: the numerical stability term.

This is the core update mechanism of Adam.

9.4 The Complete Computation Flow of Adam

flowchart TB
    A["Compute the current gradient g"]
    B["Update the first moment m"]
    C["Update the second moment v"]
    D["Apply first moment bias correction"]
    E["Apply second moment bias correction"]
    F["Compute the adaptive update amount"]
    G["Update the model parameters"]

    A --> B
    A --> C
    B --> D
    C --> E
    D --> F
    E --> F
    F --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

9.5 Adam's Core Advantages

Combined First and Second Moments

The first moment uses directional information from historical gradients, while the second moment adjusts the update scale across different parameter dimensions.

Adaptive Learning Rate