1. Why Does Good Training Performance Not Mean the Model Is Truly Excellent?
The goal of machine learning is not for the model to memorize the training data, but for it to learn generalizable patterns from limited samples and apply those patterns to data it has never seen.
A model may achieve near-100% accuracy on the training set yet perform poorly on new data; alternatively, it may perform unsatisfactorily on both the training and test sets.
These two situations correspond to overfitting and underfitting, respectively.
Understanding these two problems requires distinguishing three concepts:
- Training Performance: how the model performs on the training data.
- Generalization Ability: the model's ability to predict new data that was not involved in training.
- Generalization Error: the expected prediction error of the model over the target data distribution.
What machine learning models truly pursue is good generalization ability, not merely a reduction in training error.
2. What Is Overfitting?
2.1 Definition of Overfitting
Overfitting refers to a model adapting too closely to the training data: it learns not only the genuine patterns in the data, but also the noise, randomness, and irrelevant details present in the training data.
The model performs very well on the training set but shows a marked decline on the validation or test set.
Typical signs:
- Training error keeps decreasing.
- Validation error decreases at first, then begins to rise.
- Training set accuracy is very high, while test set accuracy is noticeably low.
- The model is highly sensitive to details in the training samples.
- The model's performance may drop markedly when the data distribution changes slightly.
2.2 How Overfitting Arises
Suppose the real data is generated by some unknown function:
Where:
- f(x): the true data-generating pattern.
- \epsilon: random noise.
- x: input features.
- y: target value.
The model cannot know the true function directly; it can only estimate it from limited training samples.
If the model has excessive expressive power, it may fit the random fluctuations in the training samples as well.
For example, the real data may approximately follow a linear pattern, but if the model fits it with a high-degree polynomial, it may form a complex curve that passes near every training sample while failing to describe the overall trend accurately.
2.3 An Intuitive Example of Overfitting
- Underfitting: the model is too simple to capture the main patterns in the data.
- Reasonable fit: the model has learned the main patterns without chasing the noise excessively.
- Overfitting: the model follows the training samples too closely, and the curve shows unnecessary complex fluctuations.
2.4 Why Do Models Overfit?
Common causes include:
1. Model complexity is too high
Too many model parameters, too many network layers, or an overly complex decision boundary may give the model expressive power far beyond what the current data requires.
2. Insufficient training data
Limited samples cannot adequately cover the true data distribution, so the model easily mistakes features that appear by chance for stable patterns.
3. Excessive noise in the training data
Noise such as incorrect labels, outliers, and collection errors may be learned directly by the model.
4. Training for too long
In some training tasks, the model first learns the main patterns; continued optimization afterward may gradually fit the details and noise in the training data.
However, more training epochs do not necessarily lead to overfitting; the specifics depend on the model, the data, the optimization method, and the regularization strategy.
5. Too many feature dimensions
When the number of features is too large relative to the number of samples, the model may exploit spurious correlations to fit the training set.
6. Data leakage
Information that should not have been available is used during training, such as test set information, future data, or features derived directly from the label. In such cases, evaluation results may be inflated, and the model's performance in real environments drops markedly.
3. What Is Underfitting?
3.1 Definition of Underfitting
Underfitting refers to a model failing to adequately learn the useful patterns in the training data, resulting in poor performance on both the training set and unseen data.
Typical signs:
- Training error is high.
- Validation error is also high.
- Even after adding more training epochs, the model still fails to achieve satisfactory results.
- The model cannot capture nonlinear relationships or complex features in the data.
- The performance gap between the training set and the validation set is usually small.
3.2 How Underfitting Arises
The model's expressive power is insufficient to represent the complex relationships in the real data.
For example, the real data has a clearly nonlinear relationship, but the model uses only a simple linear function:
No matter how w and b are adjusted, the model cannot accurately fit the complex nonlinear pattern.
This is not because the optimization algorithm failed to find suitable parameters, but possibly because the model's own expressive power is limited.
3.3 Common Causes of Underfitting
1. The model is too simple
The model structure is insufficient to express the complex relationships in the data.
2. Insufficient feature expressiveness
The input features lack key information, so the model cannot obtain the effective signal needed to make predictions.
3. Regularization is too strong
Regularization methods such as L1, L2, and Dropout limit model complexity, but excessive strength may prevent the model from learning adequately.
4. Insufficient Training
Too few training epochs means the optimization process has not yet fully converged.
5. Unreasonable Optimization Configuration
An excessively large learning rate may cause unstable training, while an excessively small learning rate may make training progress slowly.
6. Data Quality Problems
Issues such as incorrect labels, missing features, and improper preprocessing can also prevent the model from learning effective patterns.
IV. What Is Model Generalization?
4.1 Definition of Generalization Ability
Generalization refers to the ability of a model to apply the patterns learned during training to new samples that did not participate in training.
Training data is only a finite sample from the target data distribution, and the model needs to estimate the overall patterns from limited data.
Suppose the training data comes from some unknown distribution:
The model obtains a prediction function through training samples:
Ideally, we want the model's expected risk on the true data distribution to be as small as possible:
Where:
- P(X,Y): the true data distribution.
- L: the loss function.
- f_\theta: the model determined by the parameters \theta.
- R(\theta): the expected risk of the model on the target distribution.
However, the true data distribution is usually unknown, so the full expected risk cannot be computed directly.
In actual training, we use a finite set of training samples to compute the empirical risk:
The training process typically minimizes the empirical risk, whereas what the model really needs is a lower risk on the target distribution.
The difference between the two is the key to understanding the generalization problem.
4.2 Empirical Risk and True Risk
| Concept | Meaning |
|---|---|
| Empirical risk | The average loss of the model on a finite set of training samples |
| True risk | The expected loss of the model on the target data distribution |
| Generalization gap | The difference between the true risk and the empirical risk |
Error on a test set or validation set is usually used to approximate and evaluate the model's generalization performance.
Note that test set error is only an estimate of the risk on the target distribution and is not equal to the true risk itself. The quality of the estimate depends on whether the data is representative, whether the sample size is sufficient, and whether the evaluation process is independent.
4.3 The Core Goal of Model Generalization
The goal of model training is not to drive training error infinitely low, but to strike a balance between model complexity and the ability to fit the data.
flowchart TB
A["Limited training data"] --> B["Model learns data patterns"]
B --> C["Forms model parameters"]
C --> D["Evaluate using unseen data"]
D --> E{"Is generalization performance good?"}
E -->|"Yes"| F["Model has good generalization ability"]
E -->|"No"| G["Analyze overfitting, underfitting, or data problems"]
G --> H["Adjust the model and training strategy"]
H --> B
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
V. Bias and Variance: The Theoretical Basis for Understanding Overfitting and Underfitting
Bias and variance provide an important theoretical framework for understanding the model generalization problem.
5.1 What Is Bias?
Bias describes the systematic difference between the model's predictions and the true patterns.
High bias means the model has a strong systematic error and cannot adequately express the true patterns in the data.
Typical situations:
- The model structure is too simple.
- The expressive power of the features is insufficient.
- The model assumptions do not match the true relationships in the data.
- Both training error and validation error are relatively high.
Underfitting is usually associated with high bias.
5.2 What Is Variance?
Variance describes how sensitive the model is to changes in the training data.
If training the same type of model on different training samples causes the model's predictions to change noticeably, the model has high variance.
High-variance models are easily affected by randomness in the training samples.
Typical situations:
- Model complexity is too high.
- The number of training samples is insufficient.
- The data contains a lot of noise.
- The model is overly sensitive to individual samples.
Overfitting is usually associated with high variance.
5.3 The Bias-Variance Tradeoff
For squared error loss, under the classical assumptions, the prediction error can be decomposed as:
Where:
- Bias²: the systematic error of the model.
- Variance: how sensitive the model is to changes in the training data.
- Noise: random noise in the data generation process that the model cannot eliminate.
This decomposition helps us understand the relationship between model complexity and generalization ability.
Typically:
- Low model complexity: higher bias, lower variance.
- High model complexity: bias may decrease, variance may increase.
- A reasonable model: strikes an appropriate balance between bias and variance.
Note that this is a classical theoretical analysis framework, not a precise formula that can be applied directly under all models, loss functions, and training conditions.
5.4 Bias-Variance Illustration
As model complexity increases, bias typically trends downward while variance trends upward. Together they affect the model's prediction error.
Model complexity is not a matter of the higher the better, nor the lower the better.
6. How to Tell Overfitting from Underfitting
The most direct approach is to look at the loss and evaluation metrics on both the training set and the validation set at the same time.
6.1 Training Loss and Validation Loss
| Training Set Performance | Validation Set Performance | Possible Problem |
|---|---|---|
| High loss | High loss | Underfitting, insufficient optimization, or data problems |
| Low loss | High loss | Overfitting or a difference in data distribution |
| Low loss | Low loss | Good performance under the current split |
| High loss | Low loss | Need to check the data split, randomness, and evaluation pipeline |
The last case is uncommon, but it can occur when the training set involves strong data augmentation, Dropout, or other perturbations used only during training. In that case, the loss in training mode may be higher than the loss in validation mode.
6.2 Judging from the Loss Curves
Typical curve for overfitting:
- Training loss keeps decreasing.
- Validation loss decreases at first.
- Validation loss begins to rise after reaching its minimum.
Typical curve for underfitting:
- Training loss remains high throughout.
- Validation loss is also high.
- The model has not yet adequately learned the patterns in the data.
Typical curve for a good fit:
- Training loss decreases steadily.
- Validation loss decreases in tandem and levels off.
- A reasonably small gap is maintained between the two.
6.3 Don't Look Only at Training Accuracy
High training accuracy does not mean the model generalizes well.
For example:
| Metric | Model A | Model B |
|---|---|---|
| Training accuracy | 99.8% | 96.5% |
| Validation accuracy | 78.0% | 94.2% |
Although Model A performs better on the training set, Model B performs clearly better on the validation set.
If the goal is to deploy to a real environment, generalization performance should usually be prioritized over training-set metrics.
7. How to Address Overfitting
The core problem with overfitting is that the model learns too much of the randomness in the training data.
Solutions include limiting model complexity, adding effective data, introducing regularization, and improving validation strategies.
7.1 Add More Training Data
Adding high-quality, representative training samples helps the model learn more stable patterns in the data.
But increasing the amount of data does not mean simply duplicating existing samples.
What is truly valuable is expanding the coverage of the data distribution, for example:
- Different scenarios.
- Different users.
- Different devices.
- Different time periods.
- Different classes and boundary samples.
If the newly added data highly duplicates existing data, the actual benefit may be limited.
7.2 Data Augmentation
Data augmentation generates more varied training inputs by applying reasonable transformations to training samples.
Common methods in computer vision:
- Random cropping.
- Random flipping.
- Random rotation.
- Color jittering.
- Mixup.
- CutMix.
Common methods in natural language processing:
- Synonym replacement.
- Back-translation.
- Constrained text rewriting.
- Token-level perturbation.
Data augmentation must preserve the semantic correctness of the task label.
For example, in image classification, horizontal flipping usually does not change the class of a cat, but in tasks such as traffic sign recognition, flipping may change the meaning of the sign.
7.3 L1 Regularization
L1 regularization adds a penalty on the absolute values of the parameters to the original loss function:
Where:
- L_{\text{data}}: the original task loss.
- \theta_i: model parameters.
- \lambda: regularization strength.
L1 regularization tends to drive some parameters to zero, so it is often used for sparse modeling and feature selection.
7.4 L2 Regularization
L2 regularization adds a squared-parameter penalty to the loss function:
It tends to constrain the overall magnitude of the parameters, reducing the model's excessive reliance on individual features and training samples.
Differences between L1 and L2:
| Aspect | L1 | L2 |
|---|---|---|
| Penalty term | Absolute value of parameters | Squared parameters |
| Effect on parameters | Tends to produce sparse parameters | Tends to shrink parameters |
| Feature selection | Has a certain sparse-selection effect | Usually does not push parameters to exactly zero |
| Common use | Sparse modeling | Controlling parameter magnitude |
It is important to distinguish the L2 penalty in the loss function from Weight Decay in the optimizer. For plain SGD, the two correspond directly under common settings; for adaptive optimizers such as Adam, a coupled L2 penalty is not equivalent to decoupled weight decay, and AdamW is the optimizer designed precisely for this.
7.5 Dropout
Dropout is a common regularization technique in neural networks.
During training, the outputs of a random subset of neurons are set to zero, reducing the network's dependence on specific neurons or local feature combinations.
flowchart TB
A["Input features"] --> B["Neural network hidden layers"]
B --> C["Randomly generate Dropout mask"]
C --> D["Set outputs of some neurons to zero"]
D --> E["Continue forward propagation"]
E --> F["Compute loss and update parameters"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Typical behavior of Dropout:
- Training phase: randomly mask some activations.
- Inference phase: disable Dropout.
- Common implementation: use Inverted Dropout, scaling the retained activations during training.
In PyTorch:
import torch.nn as nn
model = nn.Sequential(
nn.Linear(128, 256),
nn.ReLU(),
nn.Dropout(p=0.3),
nn.Linear(256, 10)
)
# Training mode: enable Dropout
model.train()
# Inference mode: disable Dropout
model.eval()
Note that model.eval() does more than disable Dropout; it also makes Batch Normalization use running statistics.
7.6 Early Stopping
Early Stopping monitors validation set performance to prevent the model from continuing to train after validation performance has already deteriorated.
Basic process:
flowchart TB
A["Start training"] --> B["Run one training epoch"]
B --> C["Calculate validation set loss"]
C --> D{"Is it better than the historical best?"}
D -->|"Yes"| E["Save current best model"]
D -->|"No"| F["Increment unimproved epoch count"]
E --> G["Reset patience counter"]
G --> H{"Have training termination conditions been met?"}
F --> H
H -->|"No"| B
H -->|"Yes"| I["Restore best model parameters"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
Common parameters:
patience: The number of consecutive epochs allowed without improvement in the validation metric.min_delta: The minimum change required to consider the metric improved.best_score: Best historical validation metric.best_checkpoint: Saved best model parameters.
Early Stopping can usually be regarded as an implicit regularization technique, but it cannot replace proper data splitting and model evaluation.
7.7 Reduce model complexity
If the model's representational capacity far exceeds the data scale, you can try:
- Reduce the number of network layers.
- Reduce the number of neurons in hidden layers.
- Reduce unnecessary features.
- Limit decision tree depth.
- Increase the minimum number of samples per leaf node in decision trees.
- Reduce unnecessary model parameters.
However, lower complexity is not always better—over-restricting can lead to underfitting.
7.8 Use cross-validation
Cross-validation evaluates the model's stability under different combinations of training and validation samples by splitting the data multiple times.
Take K-fold cross-validation as an example:
flowchart TB
A["Full training data"] --> B["Split into K subsets"]
B --> C["Select one subset as validation set"]
B --> D["Use remaining K-1 subsets as training set"]
C --> E["Train and evaluate model"]
D --> E
E --> F["Switch validation subset"]
F --> G{"Have K rounds been completed?"}
G -->|"No"| C
G -->|"Yes"| H["Aggregate evaluation results from K rounds"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The main value of cross-validation:
- Reduce the randomness caused by a single data split.
- Evaluate the model's sensitivity to data splitting.
- Provide a more stable basis for hyperparameter selection.
However, not all data is suitable for ordinary K-fold cross-validation:
- Time series: use chronological splitting or rolling validation.
- User behavior data: group by user when necessary to avoid information leakage from the same user.
- Grouped data such as medical and device data: perform Group Split according to the business scenario.
- Highly imbalanced classification: consider stratified cross-validation.
8. How to solve underfitting?
The core problem of underfitting is that the model has not fully learned the useful patterns in the data.
Unlike overfitting, underfitting usually requires improving the model's learning capacity or the training process.
8.1 Increase model complexity
You can try:
- Increase the number of network layers.
- Increase the number of neurons in hidden layers.
- Use a network architecture better suited to the task.
- Introduce nonlinear activation functions.
- Use a model with greater representational capacity.
For example, when a linear model cannot represent complex nonlinear relationships, you can try polynomial features, tree models, or neural networks.
8.2 Add effective features
If the input features lack key information, even a sufficiently complex model cannot make accurate predictions.
Effective information can be increased through feature engineering:
- Construct interaction features.
- Extract temporal features.
- Incorporate domain knowledge.
- Use more expressive feature encodings.
- Use data representations suited to text, images, and audio.
8.3 Weakening Regularization
If the strength of L1, L2, or Dropout is too high, the model may be overly constrained.
You can moderately reduce:
- The L1 regularization coefficient.
- The L2 regularization coefficient.
- Weight Decay.
- The Dropout rate.
When making adjustments, you need to observe performance on both the training set and the validation set; you cannot judge based on training loss alone.
8.4 Increasing Training Epochs
If the model is still learning effectively, an insufficient number of training epochs may lead to underfitting.
You can increase the number of Epochs and control it in combination with a learning rate scheduler and Early Stopping.
8.5 Adjusting the Learning Rate and Optimizer
The learning rate affects the process of updating model parameters.
- Learning rate too large: may cause the loss to oscillate or even diverge.
- Learning rate too small: may cause optimization to progress slowly.
- Unsuitable optimizer: may affect training stability and convergence efficiency.
You can try adjusting the learning rate, using a learning rate decay strategy, or switching to a different optimizer.
9. Comparison of Strategies for Handling Overfitting and Underfitting
| Dimension | Overfitting | Underfitting |
|---|---|---|
| Training error | Usually relatively low | Usually relatively high |
| Validation error | Noticeably higher than training error | Usually also relatively high |
| Model behavior | Memorizes training details | Unable to fully learn the patterns |
| Model complexity | May be too high | May be insufficient |
| Data strategy | Add effective samples, data augmentation | Add effective features, improve data quality |
| Model strategy | Reduce complexity | Increase representational capacity |
| Regularization | Appropriately strengthen | Appropriately weaken |
| Training epochs | Consider early stopping | May appropriately increase |
| Core objective | Improve generalization ability | Improve fitting ability |
It should be emphasized that overfitting and underfitting are diagnoses of model behavior; you cannot directly determine a unique cause just by seeing a certain kind of curve. Data quality, label noise, training configuration, distribution shifts, and the evaluation pipeline can all affect the results.
10. Training Set, Validation Set, and Test Set
Evaluating a model's generalization ability is inseparable from a proper dataset split.