1. Preface
Neural networks generate predictions through forward propagation, but whether those predictions meet expectations requires a clear standard of measurement.
For example:
- Regression task: the predicted house price is 2 million, but the actual price is 2.2 million — how large is the error?
- Classification task: the model predicts a 90% probability that an image belongs to a cat, but the true class is dog — how heavily should the model be penalized?
- Language model: the model predicts a probability distribution over the next Token — how much does it differ from the true Token?
These questions all involve a core concept in machine learning: the loss function.
A loss function measures the discrepancy between the model's predictions and the true targets. Optimization algorithms then use the gradient information provided by the loss function to adjust the model parameters, so that the model gradually gains better predictive capability.
To understand loss functions and optimization objectives, you need to master the following:
- The distinction between loss functions, cost functions, and optimization objectives.
- The main loss functions used in regression and classification tasks.
- The relationship between cross-entropy, negative log-likelihood, and probabilistic modeling.
- Gradients of loss functions and model parameter updates.
- Regularization, weight decay, and generalization capability.
- Loss combination and weight design in multi-task learning.
- Training loss, validation loss, and model selection.
2. What Is a Loss Function?
A loss function is a mathematical function used to measure the discrepancy between the model's predictions and the true targets.
Suppose:
- The input data is X.
- The true labels are Y.
- The model parameters are \theta.
- The model predictions are \hat{Y}.
Forward propagation can be expressed as:
The loss function is expressed as:
Where:
- f: the neural network model.
- \theta: the model parameters.
- \ell: the loss function.
- L: the loss value for the current sample or current batch.
The loss value is usually a scalar, describing the model's prediction error on the current data.
Generally speaking, provided the loss is reasonably defined, a smaller loss means the model's predictions better match the training objective.
2.1 Single-Sample Loss
For a single sample:
For example, house price prediction:
- True house price: 2.2 million.
- Model prediction: 2 million.
If squared error is used:
Here the loss value is 400, in squared price units.
2.2 Batch Loss
In actual training, the model usually processes multiple samples at once.
Suppose a batch contains m samples; then the batch mean loss can be expressed as:
Depending on the task, summation, weighted averaging, or other aggregation methods may also be used.
Batch loss consolidates the errors of multiple samples into a single optimization objective, which facilitates the subsequent computation of gradients.
2.3 The Basic Roles of a Loss Function
A loss function serves three main purposes:
- Error measurement: measuring the discrepancy between model predictions and true targets.
- Basis for optimization: providing the objective for gradient computation and parameter updates.
- Model comparison: comparing the optimization performance of models under the same data and the same loss definition.
Note that the loss value does not necessarily equal the actual business error. For example, MSE has units that are the square of the target variable's units, whereas MAE has the same units as the target variable.
3. Loss Function, Cost Function, and Optimization Objective
Machine learning literature frequently features concepts such as Loss, Cost Function, and Objective Function.
These terms are related, but not entirely equivalent.
| Concept | English | Meaning |
|---|---|---|
| Loss function | Loss Function | Measures the prediction error of a single sample or a single batch |
| Cost function | Cost Function | Usually denotes the average loss over the entire dataset |
| Optimization objective | Objective Function | The objective that actually needs to be minimized or maximized during model training |
| Regularization term | Regularization Term | Imposes constraints on model complexity or parameter magnitude |
| Evaluation metric | Evaluation Metric | A metric used to measure the model's actual performance |
Across different papers and frameworks, these terms may overlap in usage, so their specific mathematical definitions cannot be judged from the names alone.
3.1 From Single-Sample Loss to an Overall Objective
Suppose the training dataset contains N samples:
The loss for a single sample is:
The empirical risk over the entire dataset is:
This is the basic form of Empirical Risk Minimization (ERM).
The goal of training a model is usually to find a set of parameters:
Where:
- \theta: model parameters.
- J(\theta): overall training objective.
- \theta^*: parameters that make the objective function attain a relatively small value.
3.2 Optimization Objective with Regularization
If only the training error is minimized, the model may overfit the training data.
Therefore, a regularization term can be added to the objective function:
Where:
- The first term: data fitting loss.
- R(\theta): regularization function.
- \lambda: regularization strength.
At this point, the model needs to consider two aspects simultaneously:
- Fit the training data as well as possible.
- Control model complexity or parameter scale.
This constitutes a very important optimization idea in machine learning.
IV. Common Loss Functions in Regression Tasks
The goal of regression tasks is to predict continuous numerical values, such as:
- House price prediction.
- Sales prediction.
- Temperature prediction.
- User spending amount prediction.
- Equipment remaining useful life prediction.
Regression problems usually need to measure the difference between predicted numerical values and true numerical values.
4.1 Mean Squared Error (MSE)
Mean Squared Error is one of the most common regression loss functions.
Formula:
Where:
- y_i: true value.
- \hat{y}_i: predicted value.
- N: number of samples.
Characteristics:
- Squares the error.
- The larger the error, the faster the penalty grows.
- Relatively sensitive to outliers.
- Good mathematical properties, convenient for differentiation and optimization.
Per-sample loss:
Derivative with respect to the predicted value:
The larger the error, the larger the absolute value of the gradient is usually.
Applicable scenarios:
Regression tasks where larger prediction errors require stronger penalties and outliers will not seriously interfere with training.
4.2 Root Mean Squared Error (RMSE)
Root Mean Squared Error:
RMSE is the square root of MSE.
Characteristics:
- Has the same unit as the prediction target.
- Remains sensitive to larger errors.
- Commonly used for evaluating regression model performance.
Note: RMSE can also be used directly as an optimization objective in some training implementations, but its gradient form differs from that of MSE, and numerical handling needs to be considered near the point where the overall error is zero.
4.3 Mean Absolute Error (MAE)
Mean Absolute Error:
Characteristics:
- Takes the absolute value of the error.
- Does not amplify large errors by squaring as MSE does.
- Relatively insensitive to outliers.
- Non-differentiable at the point where the error is zero, but subgradients can be used for optimization.
Per-sample loss:
Its gradient with respect to the predicted value is:
When \hat{y}=y, the subgradient definition is used.
4.4 Huber Loss
Huber Loss combines the characteristics of MSE and MAE.
Definition:
Where:
\delta is the threshold that controls the piecewise position.
Characteristics:
- Uses squared error in the small-error region.
- Uses a linear penalty in the large-error region.
- More robust to outliers than MSE.
- Keeps the first derivative continuous at the piecewise junction.
When to use:
When the data contains some outliers but you still want to retain the optimization properties of squared error in the small-error region.
4.5 Comparison of Regression Loss Functions
| Loss function | Error handling | Sensitivity to outliers | Characteristics |
|---|---|---|---|
| MSE | Squared error | High | Large errors are penalized heavily |
| RMSE | Square root after squared error | High | Same units as the target variable |
| MAE | Absolute error | Lower | More robust to outliers |
| Huber | Piecewise squared and linear | Moderate | Balances smoothness and robustness |
V. Common Loss Functions in Classification Tasks
The goal of a classification task is to determine which class a sample belongs to.
For example:
- Spam detection.
- Image classification.
- User churn prediction.
- Risk identification.
- Text sentiment classification.
Classification models typically output class scores or a probability distribution, and a loss function then measures the difference between the prediction and the true class.
5.1 Binary Cross Entropy
Binary classification tasks usually use binary cross entropy loss (BCE).
Assume:
- The true label is y\in\{0,1\}.
- The model predicts the probability of the positive class as p.
The loss function is:
When the true label is 1:
When the true label is 0:
It can be seen that:
- When the true class is positive, the higher the model's predicted probability for the positive class, the smaller the loss.
- When the true class is negative, the higher the model's predicted probability for the negative class, the smaller the loss.
- When the predicted probability for the true class is too low, the loss grows rapidly.
For example:
True label:
The loss corresponding to different predicted probabilities:
| Predicted probability p | Loss -\log p |
|---|---|
| 0.99 | 0.0101 |
| 0.90 | 0.1054 |
| 0.70 | 0.3567 |
| 0.50 | 0.6931 |
| 0.10 | 2.3026 |
| 0.01 | 4.6052 |
The more confidently the model predicts the wrong class, the larger the penalty it receives.
5.2 From Sigmoid to Binary Cross Entropy
A binary classification neural network usually outputs a logit first:
which is then converted into a probability via Sigmoid:
Substituting the probability into BCE:
This can be further written as:
This expression is binary cross entropy in logits form.
In practice, prefer the numerically stable logits version of the implementation rather than manually computing the Sigmoid first and then applying the log.
For example, in PyTorch:
import torch
import torch.nn as nn
criterion = nn.BCEWithLogitsLoss()
logits = torch.tensor([0.8, -0.5, 1.2])
targets = torch.tensor([1.0, 0.0, 1.0])
loss = criterion(logits, targets)
What is passed in here is logits, not probabilities that have gone through Sigmoid.
5.3 Categorical Cross Entropy
Multi-class tasks typically use Softmax to convert the model output into a class probability distribution.
Suppose the model has C classes and outputs logits:
Softmax:
where:
For a one-hot encoded label y, the cross-entropy is:
If the true class is class c, then:
That is, multiclass cross-entropy mainly penalizes the model for assigning insufficient probability to the true class.
5.4 Combination of Softmax and Cross-Entropy
Multiclass neural networks usually use:
However, in practical frameworks, the logits version of the cross-entropy loss is usually used directly.
For example, in PyTorch:
import torch
import torch.nn as nn
criterion = nn.CrossEntropyLoss()
logits = torch.tensor([
[2.0, 1.0, 0.1],
[0.2, 0.5, 2.1]
])
targets = torch.tensor([0, 2])
loss = criterion(logits, targets)
There are two key points here:
logitsare raw scores that have not been processed by Softmax.targetsare class indices, and do not have to be manually converted to one-hot encoding.
Internally, PyTorch's CrossEntropyLoss performs numerically stable LogSoftmax and negative log-likelihood computation.
5.5 Multilabel Classification and Binary Cross-Entropy
Multiclass classification and multilabel classification must not be confused.
Multiclass classification:
A sample can belong to only one class.
For example:
- Cat.
- Dog.
- Bird.
Softmax + Cross Entropy is commonly used.
Multilabel classification:
A sample can have multiple labels at the same time.
For example, an image may contain the following at the same time:
- Person.
- Car.
- Building.
Typically, an independent Sigmoid is used for each label, and then the loss is calculated using BCE.
Therefore:
| Task | Output method | Common loss |
|---|---|---|
| Binary classification | Single Sigmoid | BCE |
| Mutually exclusive multiclass classification | Softmax | Cross Entropy |
| Multilabel classification | Multiple independent Sigmoids | BCE |
VI. Cross-Entropy, Negative Log-Likelihood, and Probabilistic Modeling
Cross-entropy is a very important loss function in deep learning.
Understanding cross-entropy requires further understanding probability distributions and maximum likelihood estimation.
6.1 What Is Cross-Entropy?
In information theory, cross-entropy (Cross Entropy) is used to measure the difference between two probability distributions.
Suppose:
- The true distribution is P.
- The model-predicted distribution is Q.
Cross-entropy is defined as:
where:
- P(x): the probability assigned by the true distribution to event x.
- Q(x): the probability predicted by the model.
When the true distribution is deterministic, cross-entropy can be simplified to the negative logarithm of the predicted probability corresponding to the true class.
6.2 Cross-Entropy and KL Divergence
KL divergence (Kullback-Leibler Divergence) is used to measure the information difference between two probability distributions:
It has the following relationship with cross-entropy:
where H(P) is the entropy of the true distribution itself.
When the true distribution P is fixed, H(P) does not change with the model parameters.
Therefore:
That is, under the condition that the true distribution is fixed, minimizing cross-entropy is equivalent to minimizing the KL divergence between the model distribution and the true distribution.
Note that KL divergence is generally asymmetric and therefore cannot be directly regarded as a distance in the ordinary sense.
6.3 Cross-Entropy and Maximum Likelihood Estimation
Suppose the model gives the probability for sample x_i's true label y_i:
Under the assumption of independent and identically distributed samples, the likelihood function of the entire dataset is:
Maximum Likelihood Estimation (MLE) seeks to find the parameters that maximize the likelihood:
For computational convenience, the logarithm of the likelihood is usually taken:
Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood:
This is an important source of cross-entropy loss in classification tasks.
Therefore, cross-entropy is not an arbitrarily designed error formula, but an optimization objective with a clear probabilistic and statistical foundation.
7. How Does the Loss Function Affect Model Parameter Optimization?
The loss function defines the optimization objective, but how model parameters are adjusted still depends on gradient computation and optimization algorithms.
7.1 Relationship Between the Loss Function and Model Parameters
Suppose the model predicts:
The loss function is:
The loss depends not only on the prediction results but also indirectly on the model parameters.
Changing the parameter \theta changes the model's prediction results, and ultimately the loss also changes.
Therefore, the core of the training process is to find an appropriate combination of parameters that reduces the loss as much as possible.
7.2 What Is a Gradient?
The gradient represents the collection of partial derivatives of a multivariate function with respect to each parameter.
For the parameter vector:
The gradient of the loss function is:
The gradient describes how sensitive the loss function is to changes in each parameter.
When differentiable, the negative gradient direction is the direction of fastest local descent.
7.3 Gradient Descent
The most basic parameter optimization method is gradient descent.
Update formula:
where:
- \theta_t: current parameters.
- \eta: learning rate.
- \nabla_\theta L: the loss gradient at the current parameters.
The learning rate determines the step size of each parameter update.
If the learning rate is too large, it may cause the optimization process to oscillate or even diverge.
If the learning rate is too small, it may lead to slow training.
7.4 Complete Process from Loss to Parameter Update
flowchart TB
A["Input training data"]
B["Model forward pass"]
C["Obtain predictions"]
D["Loss function computes error"]
E["Compute loss gradient"]
F["Backpropagation computes parameter gradients"]
G["Optimizer updates parameters"]
H["Proceed to next training round"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H --> B
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
Here it is necessary to distinguish:
- Loss function: defines the optimization objective.
- Automatic differentiation and backpropagation: compute gradients.
- Optimizer: updates parameters based on gradients.
Together, the three form the core mechanism of model parameter optimization.
8. Regularization: Why Can't We Just Minimize Training Loss?
The goal of model training should not merely be to make the training set loss as small as possible.
If a model overfits the training data, it may perform very well on the training set but poorly on new data.
This phenomenon is called overfitting.
Regularization is an important method for controlling model complexity and improving generalization ability.
8.1 L1 Regularization
L1 regularization is usually defined as the sum of the absolute values of the parameters:
The corresponding optimization objective is:
Characteristics:
- It tends to produce sparse parameters.
- It can drive some parameters to zero.
- It is often used for feature selection and sparse modeling.
8.2 L2 Regularization
L2 regularization is usually defined as the sum of the squares of the parameters:
The corresponding optimization objective is:
Characteristics:
- It penalizes large parameters.
- It tends to make the parameters smaller overall.
- Unlike L1, it usually does not directly produce a large number of parameters that are exactly zero.
In neural network training, L2 regularization is closely related to weight decay.