I. Why Does AI Need Calculus?
One of the core tasks of machine learning is to adjust model parameters through training data so that the model's predictions are as close as possible to the true results.
Whether it is linear regression, logistic regression, deep neural networks, or large language models, the training process can be abstracted as an optimization problem:
Where:
- \theta: model parameters, such as weights and biases.
- L(\theta): the model loss function, used to measure the difference between the model's predictions and the true results.
- \theta^*: the parameters that minimize the loss function.
The problem is: model parameters are usually enormous in number, and the loss function may be very complex. How do we know which parameters should be adjusted, in which direction, and by how much?
This is exactly where calculus comes into play.
Calculus provides the core mathematical tools for model optimization:
| Mathematical Concept | Role in AI |
|---|---|
| Function | Describes the relationship between model inputs, parameters, and outputs |
| Limit | Understanding continuous change and infinitesimal change |
| Derivative | Measures the rate of change of a function at a given point |
| Partial derivative | Measures the impact of change in one parameter among many |
| Gradient | Aggregates change information across multiple parameter directions |
| Chain rule | Breaks down complex computations into a continuous propagation of local derivatives |
| Integral | Describes continuous accumulation, used in scenarios such as probabilistic modeling and expectation computation |
| Taylor expansion | Approximately analyzes how a function changes within a local region |
Among these, derivatives, partial derivatives, gradients, and the chain rule are the most important foundations for understanding the training mechanism of neural networks.
II. Functions and Model Parameters: The Starting Point of Optimization
2.1 What Is a Function?
A function describes the mapping relationship between inputs and outputs.
For example:
When the input is x=2:
In machine learning, the model itself can be viewed as a function:
Where:
- x: input data.
- \theta: model parameters.
- \hat{y}: model prediction result.
Take linear regression as an example:
Where:
- w: weight.
- b: bias.
- x: input feature.
- \hat{y}: predicted value.
The process of model training is to find suitable w and b so that the predicted values are as close as possible to the true values.
2.2 Loss Function
Suppose the true value is y and the model prediction is \hat{y}. The mean squared error can be used as the loss function:
Substituting the linear model:
At this point, the loss function is no longer merely a function of the input data, but a function of the model parameters w,b.
Training a model means finding a set of parameters that makes the loss function as small as possible.
2.3 From Functions to Optimization Problems
Suppose the model has only one parameter w, and the loss function is:
Its minimum occurs at:
But for complex models, the loss function may contain millions or even billions of parameters, and it is impossible to find the optimal solution directly through simple algebraic calculation.
Therefore, it is necessary to study the patterns of how the function changes at different parameter positions.
This leads to derivatives.
III. Derivatives: Measuring the Rate of Change of a Function
3.1 The Basic Definition of a Derivative
A derivative describes how fast the output changes when the input undergoes a small change near a certain point of the function.
For the function:
Its derivative at x is defined as:
Where:
- \Delta x: a small change in the input.
- f(x+\Delta x)-f(x): the change in the output.
- The ratio of the two: the average rate of change.
- As \Delta x approaches 0, we get the instantaneous rate of change.
Geometrically, the derivative represents the slope of the tangent line to the function's curve at a given point.
3.2 What Does the Sign of a Derivative Mean?
Suppose:
Take the derivative:
The derivative means different things at different positions:
| Position | Derivative | Meaning |
|---|---|---|
| x<0 | Negative | The function decreases as x increases |
| x=0 | 0 | The tangent slope at this point is 0 |
| x>0 | Positive | The function increases as x increases |
The sign of the derivative tells us the direction of the function's local change.
In model optimization, this is extremely important:
- Positive derivative: as the parameter increases, the loss tends to increase.
- Negative derivative: as the parameter increases, the loss tends to decrease.
- Zero derivative: the first-order rate of change at the current point is zero, but it is not necessarily a minimum.
3.3 Common Derivative Formulas
Understanding the common differentiation rules is the foundation for learning gradient descent and backpropagation later on.
| Function | Derivative |
|---|---|
| f(x)=c | f'(x)=0 |
| f(x)=x | f'(x)=1 |
| f(x)=x^n | f'(x)=nx^{n-1} |
| f(x)=e^x | f'(x)=e^x |
| f(x)=\ln x | f'(x)=1/x |
| f(x)=\sin x | f'(x)=\cos x |
| f(x)=\cos x | f'(x)=-\sin x |
A few important differentiation rules:
Constant multiple rule
Sum and difference rule
Product rule
Quotient rule
Chain rule
The chain rule is the most important mathematical foundation for backpropagation in neural networks, and we will cover it in detail later.
IV. Partial Derivatives: Handling Multiple Model Parameters
4.1 Why Do We Need Partial Derivatives?
Real models usually contain a large number of parameters.
For example:
This is a loss function with two variables.
When w and b change simultaneously, the loss function also changes.
But we want to understand separately:
- If only w is adjusted, how does the loss change?
- If only b is adjusted, how does the loss change?
This requires partial derivatives.
A partial derivative represents the rate of change of a multivariable function with respect to one of its variables while holding the other variables constant.
For:
Partial derivative with respect to w:
Partial derivative with respect to b:
The symbol \partial denotes a partial derivative, used to distinguish it from the ordinary derivative of a single-variable function.
4.2 Geometric Meaning of Partial Derivatives
The derivative of a single-variable function describes the slope of the tangent line at a point on a curve.
A multivariable function can be understood as a surface.
For example:
Taking the partial derivative with respect to x means fixing y and observing the local rate of change of the surface along the x direction.
Taking the partial derivative with respect to y means fixing x and observing the local rate of change of the surface along the y direction.
4.3 Role of Partial Derivatives in Model Training
Take the mean squared error of linear regression as an example:
Partial derivative with respect to the weight w:
Partial derivative with respect to the bias b:
These two partial derivatives respectively reflect the effect of the weight and bias on the current loss.
The model optimizer can use this information to adjust the parameters separately.
5. Gradient: Directional Information for Multi-Parameter Optimization
5.1 What Is a Gradient?
The gradient is a vector formed by all the partial derivatives of a multivariable function.
For:
Its gradient is:
Where:
- \nabla: gradient operator.
- \theta_i: the i-th parameter of the model.
- \frac{\partial L}{\partial\theta_i}: the partial derivative of the loss function with respect to this parameter.
The gradient contains two important pieces of information:
- Direction: the direction in which the function increases fastest.
- Magnitude: the maximum local rate of change of the function in that direction.
Therefore, the gradient can help us determine how the model parameters should be adjusted.
5.2 Gradient and Loss Descent
Note the following:
- The gradient points in the direction of the fastest local increase of the function.
- The negative gradient points in the direction of the fastest local decrease of the function.
Therefore, if the goal is to minimize the loss function, the parameters are usually updated in the direction of the negative gradient.
For a two-dimensional loss function:
The gradient is:
The negative gradient direction is:
This means adjusting w and b at the same time, so that the loss function moves in the descending direction.
5.3 What Does the Magnitude of the Gradient Mean?
The magnitude of the gradient is:
The gradient norm reflects the maximum local rate of change of the function near the current point.
In general:
- Large gradient: the loss changes quickly near the current point.
- Small gradient: the loss changes slowly near the current point.
- Zero gradient: the current point is a first-order stationary point.
But a large gradient does not mean a larger parameter update step must be used. The actual update also depends on the learning rate, curvature, parameter scale, and the optimizer strategy.
5.4 Is a Zero Gradient Necessarily the Optimal Solution?
Not necessarily.
Consider three typical cases:
| Type | First Derivative | Meaning |
|---|---|---|
| Local minimum | 0 | Surrounding function values are no smaller than the current value |
| Local maximum | 0 | Surrounding function values are no larger than the current value |
| Saddle point | 0 | Rises in some directions and falls in others |
For example:
attains a minimum at x=0.
And:
attains a maximum at x=0.
Now consider:
The gradient is zero at (0,0), but the function increases along the x direction and decreases along the y direction, so this is a saddle point.
For deep neural networks, the loss function is usually high-dimensional and non-convex, and both saddle points and complex flat regions are issues that need to be considered during optimization.
Six. Gradient Descent: How Are Model Parameters Updated?
6.1 The Core Idea of Gradient Descent
Gradient descent is an optimization algorithm that uses gradient information to find a local minimum of a function.
Core idea:
- Initialize the model parameters.
- Compute the loss under the current parameters.
- Compute the gradient of the loss function with respect to the parameters.
- Update the parameters along the negative gradient direction.
- Repeat the above process until the stopping condition is met.
Parameter update formula:
Where:
- \theta_t: the current parameters.
- \theta_{t+1}: the updated parameters.
- \eta: the learning rate.
- \nabla L(\theta_t): the gradient at the current parameter position.
6.2 A Simple Numerical Example
Suppose the loss function is:
Take the derivative:
Set the initial parameter:
Learning rate:
First update:
Second update:
As the updates are repeated, the parameter gradually approaches the optimal value w=3.
6.3 The Gradient Descent Process
flowchart TB
A["Initialize model parameters"] --> B["Input training data"]
B --> C["Forward propagation computes predictions"]
C --> D["Compute loss function"]
D --> E["Compute parameter gradients"]
E --> F["Update parameters using the learning rate"]
F --> G{"Is the stopping condition met?"}
G -->|"No"| C
G -->|"Yes"| H["Output the trained model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
6.4 Why Is the Learning Rate So Important?
The learning rate determines the step size of each parameter update.
If the learning rate is too small:
- The parameter updates are tiny.
- Convergence may be very slow.
- More training iterations are required.
If the learning rate is too large:
- It may overshoot the better region.
- The loss function may oscillate back and forth.
- In severe cases it may fail to converge, or even diverge numerically.
A larger learning rate is not always better, nor is a smaller, more stable one always better; the choice needs to be tuned to the shape of the loss surface and the optimization algorithm.
7. The Chain Rule: The Mathematical Foundation of Neural Network Backpropagation
This is one of the most important applications of calculus in deep learning.
7.1 Why Do We Need the Chain Rule?
A neural network is usually composed of multiple computational layers.
The output of a later layer depends on the output of the previous layer, and the final loss depends on the network output.
For example:
Suppose:
Then the loss function L is not determined directly by x, but depends on x indirectly through several intermediate variables.
We want to find:
By the chain rule:
In other words, the derivative of a complex function can be broken down into the product of several local derivatives.
7.2 A Concrete Example
Suppose:
Expand it:
Differentiate directly:
We can also use the chain rule:
Therefore:
Substituting z=3x^2:
Both methods give the same result.
7.3 How Is the Chain Rule Applied to Neural Networks?
A simple neural network can be abstracted as:
flowchart LR
A["Input x"] --> B["Linear transformation"]
B --> C["Activation function"]
C --> D["Next layer computation"]
D --> E["Predicted output"]
E --> F["Loss function"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
During the forward propagation stage:
- Data enters the network from the input layer.
- Each layer performs calculations such as linear transformations and activation functions.
- Finally, the model's prediction result is obtained.
- The loss is computed based on the prediction result.
During the backpropagation stage:
- Gradients are computed starting from the loss function.
- The chain rule is used to propagate gradients backward layer by layer.
- The influence of each layer's parameters on the loss is computed.
- The gradients are passed to the optimizer to update the parameters.
7.4 Weight Gradients in Backpropagation
Consider a simple neuron:
where \sigma is the activation function.
Assume the loss is:
We need the influence of weight w on the loss:
By the chain rule:
Compute them separately:
Finally:
This formula demonstrates the basic mechanism of backpropagation: the influence of the loss on the output is propagated layer by layer to the weight parameters.
7.5 Why Are Deep Networks Prone to Vanishing and Exploding Gradients?
Backpropagation requires applying the chain rule repeatedly.
When the network has many layers, gradients may involve the product of a large number of local derivatives.
If the absolute values of these derivatives remain less than 1 for a long time, the product may rapidly approach 0.
This is the vanishing gradient problem.
If the absolute values of these derivatives remain greater than 1 for a long time, the product may grow rapidly.
This is the exploding gradient problem.
flowchart TB
A["Deep network backpropagation"] --> B["Repeatedly compute local gradients"]
B --> C{"Numerical change in the gradient product"}
C -->|"Continuously shrinking"| D["Vanishing gradient"]
C -->|"Continuously growing"| E["Exploding gradient"]
C -->|"Remains within a reasonable range"| F["Normal gradient propagation"]
D --> G["Parameters in earlier layers are difficult to update effectively"]
E --> H["Training instability or numerical overflow"]
F --> I["Parameter optimization proceeds normally"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F7EAE5,stroke:#C98D79,color:#704334
style E fill:#F7EAE5,stroke:#C98D79,color:#704334
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
Common mitigation methods include:
| Problem | Common Approaches |
|---|---|
| Vanishing gradient | Activation functions such as ReLU, proper initialization, residual connections |
| Exploding gradient | Gradient clipping, proper initialization, learning rate adjustment |
| Difficulty training deep networks | Residual structures, normalization methods, improved optimizers |
| Numerical instability | Reasonable data scaling, numerically stable computation |
Note: these methods address training problems at different levels, and they cannot guarantee that vanishing or exploding gradients will never occur in every network.
8. Integration: Continuous Accumulation and Probability Computation
Derivatives study rates of change, while integrals study accumulated quantities.
The two are closely related core concepts in calculus.
8.1 Definite Integral
For the function:
The definite integral over the interval [a,b]:
can be understood as the signed area between the function's curve and the horizontal axis.
More rigorously, the definite integral describes the accumulated result of the function over the interval.
For example:
8.2 Indefinite Integral
The indefinite integral is used to find an antiderivative of a function.
where:
C is the constant of integration.
For example: