I. Introduction
Neural networks compute predictions through forward propagation and measure prediction error through a loss function. But merely knowing whether the model predicts well does not automatically make the model better.
The model also needs to know:
- Which parameters affected the current loss?
- How large is each parameter's influence on the loss?
- How should the parameters be adjusted to reduce the loss?
- When the network contains dozens or even hundreds of layers, how can the gradients of all parameters be computed efficiently?
These questions involve one of the most central mechanisms in neural network training: backpropagation.
The mathematical foundation of backpropagation is the chain rule in calculus. It efficiently computes the partial derivatives of the loss function with respect to each parameter in the network by propagating gradients layer by layer from the output end to the input end.
This article focuses on:
- The basic concepts of derivatives, partial derivatives, and gradients.
- The mathematical principles of the chain rule.
- Computational graphs and gradient propagation mechanisms.
- The complete derivation of backpropagation in neural networks.
- Gradient computation in multilayer networks.
- The matrix form of backpropagation.
- Automatic differentiation and reverse-mode differentiation.
- Vanishing gradients, exploding gradients, and their remedies.
- The relationship between backpropagation and optimizers.
II. Why Do Neural Networks Need Backpropagation?
Let us first review the basic training process of a neural network.
Suppose the model parameters are:
The model receives input data X and obtains the prediction result through forward propagation:
Then it computes the loss:
If the loss is relatively large, it indicates that the current model parameters are not ideal enough, and the weights and biases need to be adjusted.
The problem, however, is that neural networks usually contain a large number of parameters.
For example:
- A simple fully connected network may have tens of thousands of parameters.
- A deep learning model may have millions or even billions of parameters.
- The parameter scale of large language models may reach tens of billions or more.
If we tried to adjust parameters one by one and then observe the change in loss, the computational cost would be extremely high.
Therefore, an efficient method is needed to compute the influence of each parameter on the final loss.
This influence is represented by the gradient:
Backpropagation is the core algorithm for computing these gradients.
2.1 Forward Propagation and Backpropagation
Forward propagation:
- Starts with the input data.
- Computes layer by layer according to the network structure.
- Obtains the model prediction.
- Computes the final loss.
Backpropagation:
- Starts with the loss function.
- Computes gradients according to the chain rule.
- Propagates step by step from the output layer backward.
- Obtains the gradients of parameters at each layer.
Together, the two constitute the core computational process in neural network training.
flowchart TB
A["Input data"]
B["Forward propagation"]
C["Model prediction"]
D["Compute loss"]
E["Backpropagation"]
F["Compute weight and bias gradients"]
G["Optimizer updates parameters"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Pay special attention:
Backpropagation is responsible for computing gradients, not directly for updating model parameters.
Parameter updates are performed by optimizers such as SGD and Adam.
III. Mathematical Foundations of Backpropagation: Derivatives and Partial Derivatives
Before understanding backpropagation, we need to grasp several calculus concepts.
3.1 Derivatives
A derivative describes the rate of change of a function near a certain point.
Suppose:
The derivative of the function with respect to x is:
For example:
Taking the derivative:
When x=3:
This means that near x=3, when the input changes slightly, the output's rate of change is approximately 6.
From a geometric perspective, the derivative represents the slope of the tangent line to the curve at a point.
3.2 Partial Derivatives
When a function contains multiple variables, partial derivatives are needed to describe the function's rate of change with respect to one variable.
Suppose:
When taking the partial derivative with respect to x, treat y as a constant:
When taking the partial derivative with respect to y, treat x as a constant:
Loss functions in neural networks typically depend on a large number of parameters, so it is necessary to compute the partial derivative of the loss with respect to each parameter.
3.3 Gradient
Suppose the loss function is:
Its gradient is defined as:
The gradient contains the partial derivatives of the loss function with respect to all parameters.
It has two important implications:
- The gradient direction indicates the direction in which the function increases fastest locally.
- The negative gradient direction indicates the direction in which the function decreases fastest locally.
Therefore, the gradient can provide directional information for optimizing model parameters.
3.4 Gradients and Parameter Updates
Take gradient descent as an example:
Where:
- \theta_t: the current parameter.
- \eta: the learning rate.
- \nabla_\theta L: the loss gradient.
If the gradient corresponding to a parameter is positive, gradient descent tends to decrease that parameter.
If the gradient is negative, gradient descent tends to increase that parameter.
However, the specific update magnitude also depends on the learning rate and the optimizer's update rule.
4. Chain Rule: The Core Mathematical Principle of Backpropagation
The chain rule is the most important mathematical foundation of backpropagation.
The problem it solves is:
When a variable indirectly affects another variable through multiple functions, how do you compute the derivative of the final variable with respect to the original variable?
4.1 Differentiating a Composition of Two Functions
Suppose:
Then:
According to the chain rule:
In other words:
The derivative of the final output with respect to the original input equals the product of the derivatives of the intermediate functions.
4.2 A Simple Example
Suppose:
When:
First compute:
According to the chain rule:
Take the derivatives separately:
Therefore:
Substitute u=8:
Alternatively, expand the original function:
The result is likewise 48.
4.3 Chain Differentiation of Multi-Layer Functions
If a function contains multiple intermediate variables:
Then:
This is precisely the core computation method of backpropagation in neural networks.
Each layer of a neural network can be viewed as a function.
The final loss depends on the outputs of all preceding layers, so the chain rule can be applied layer by layer along the computation path.
5. Computational Graphs: Understanding How Gradients Propagate
A computational graph is a representation that breaks down mathematical computations into nodes and connections.
Where:
- Nodes represent variables or operations.
- Directed edges represent data dependencies.
- The forward direction represents numerical computation.
- The backward direction represents gradient computation.
5.1 A Simple Computational Graph
Suppose:
The corresponding computation graph:
flowchart LR
X["Input x"]
W["Weight w"]
M["Multiply u=xw"]
B["Bias b"]
A["Add y=u+b"]
S["Square L=y²"]
X --> M
W --> M
M --> A
B --> A
A --> S
style X fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style W fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style B fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style A fill:#EAF3EC,stroke:#83A995,color:#315D49
style S fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
Assume:
Forward computation:
Next we need to compute the gradient of the loss with respect to each variable.
5.2 Backward Computation from the Output End
First:
Therefore:
Since:
we have:
By the chain rule:
And since:
we have:
Finally:
This gives:
| Variable | Gradient |
|---|---|
| x | 42 |
| w | 28 |
| b | 14 |
This example demonstrates the basic mechanism of backpropagation:
starting from the final loss, the gradients of each variable are computed step by step in the reverse direction along the computation graph.
5.3 Gradient Accumulation
In a computation graph, a variable may affect multiple downstream computation nodes at the same time.
For example:
Here x affects u through two computation paths.
By the differentiation rules:
From the perspective of the computation graph, the gradients from the two paths must be added together.
This is also an important rule in backpropagation:
When a variable affects the final loss through multiple paths, the gradients passed back along those paths must be accumulated.
Six. Backpropagation in Neural Networks
Now we apply the chain rule to a neural network.
Consider a simple two-layer fully connected neural network:
Loss function:
After the forward pass is complete, we need to compute:
These gradients are used to update the network parameters.
6.1 Output Layer Gradients
First, define the output layer error term:
If the output layer uses Softmax, the loss function uses multiclass cross-entropy, and the labels use standard one-hot encoding, then:
The expression here uses the single-sample or per-sample computation form.
For batch data, the way the loss is summed or averaged must also be taken into account in practice.
Output layer weight gradient:
Output layer bias gradient:
For batch-averaged loss, the bias gradient must be summed along the batch dimension and divided by the batch size.
6.2 Hidden Layer Gradients
After the output layer gradients have been computed, propagation must continue backward.
The hidden layer activation values are:
Therefore:
By the chain rule:
Where:
- \delta^{[2]}: output layer error term.
- W^{[2]}: output layer weights.
- f'(Z^{[1]}): derivative of the hidden layer activation function.
- \odot: element-wise multiplication.
Hidden layer weight gradient:
Hidden layer bias gradient:
For batch input, the bias gradient needs to be summed or averaged over the batch dimension.
6.3 Overall Flow of Backpropagation
flowchart TB
A["Forward propagation completed"]
B["Compute output layer error term"]
C["Compute output layer weight gradients"]
D["Compute output layer bias gradients"]
E["Propagate gradients to hidden layer"]
F["Combine with activation function derivative"]
G["Compute hidden layer weight gradients"]
H["Compute hidden layer bias gradients"]
I["Obtain gradients for all parameters"]
A --> B
B --> C
B --> D
C --> E
D --> E
E --> F
F --> G
F --> H
G --> I
H --> I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
7. Complete Mathematical Derivation of Backpropagation
To understand backpropagation more deeply, we start from a concrete neural network and derive the gradients of each parameter below.
Assume the network contains:
- Two input features.
- Two hidden layer neurons.
- One output neuron.
- The hidden layer uses ReLU.
- The output layer uses linear activation.
- The loss function uses mean squared error.
7.1 Forward Computation
Input:
Hidden layer:
Output layer:
Loss:
Here, the squared error with a 1/2 coefficient is used to simplify the derivative expression.
7.2 Compute the Output Layer Gradients
First, differentiate the loss:
Since the output layer is a linear transformation:
Therefore:
7.3 Compute the Hidden Layer Gradients
The hidden layer output is:
The derivative of ReLU is:
At z=0, it is non-differentiable; practical implementations usually adopt a conventional subgradient.
The hidden layer error term is:
Thus:
At this point, the gradients of all parameters have been computed.
7.4 Why Compute from Back to Front?
The hidden layer parameters do not directly participate in the final loss calculation.
They influence the output layer through the hidden layer output, which in turn influences the final loss.
Therefore, the output layer gradients must be computed first, and then the gradients are propagated back to the hidden layer according to the chain rule.
This is why backpropagation uses a reverse computation order.
VIII. Backpropagation in Multi-Layer Neural Networks
For a neural network with L layers, the computation of each layer can be expressed uniformly as:
where:
The final output is:
Backpropagation starts from the last layer to compute the error term.
Define:
For a hidden layer:
Weight gradient:
Bias gradient:
The formulas above use the single-sample form or the form without averaging over the batch.
For data with batch size m:
Here it is assumed that the batch loss uses the sample average, and the error term for each sample is calculated according to the corresponding loss definition.
8.1 Recursive Relationship of Backpropagation
From a mathematical structure perspective, backpropagation has a recursive nature:
- The gradient of the last layer is computed directly from the loss function.
- The gradient of the current layer depends on the gradient of the next layer.
- Each layer combines the derivative of its own activation function.
- Gradients propagate layer by layer toward the earlier layers.
This recursive relationship allows multi-layer neural networks to use a unified computation rule to solve for gradients.
8.2 Difference Between Parameter Gradients and Error Terms
These two concepts are easy to confuse.