In machine learning, a model not only needs to learn the patterns in the training data, but also needs to prove that it can handle data it has never seen.
If the same batch of data is used both to train and to evaluate a model, the resulting numbers often fail to reflect the model's true generalization ability. The model may simply have memorized the training samples rather than genuinely learning the patterns underlying the data.
For this reason, machine learning typically splits a dataset into three parts:
- Training Set: used to train the model and learn its parameters.
- Validation Set: used for model selection, hyperparameter tuning, and monitoring the training process.
- Test Set: used for the final evaluation of the model's generalization performance on unseen data.
These three types of datasets form the foundation of machine learning experiment design and model evaluation.
1. Why Do We Need to Split the Dataset?
1. The Goal of Model Training Is Not to Memorize Data
The core goal of machine learning is to learn generalizable patterns from a limited number of samples.
Consider a house price prediction task:
Input features:
- House area
- Number of rooms
- Geographic location
- Age of the house
- Surrounding amenities
Prediction target:
- House sale price
The model learns the relationship between features and house prices from the training data.
But the training data is only a subset of samples from the real world. When the model is actually deployed, it faces house data that has never appeared before.
Therefore, two questions need to be distinguished:
- Can the model fit the data it has already seen well?
- Can the model correctly predict data it has never seen?
The first question concerns fitting ability, and the second concerns generalization ability.
2. Training Error Is Not the Same as Generalization Error
Training Error: the prediction error of the model on the training set.
Generalization Error: the expected error the model produces on unseen samples under the target data distribution.
In general:
- Training error can be computed directly during model training.
- Generalization error cannot be obtained directly, because all samples in the true data distribution are not visible.
- A test set can serve as an approximate basis for evaluating generalization ability, but it is not equal to the true generalization error.
If a model overfits the training data, it may show very low training error and very high test error.
flowchart TB
A["Complete Dataset"] --> B["Split the Dataset"]
B --> C["Training Set"]
B --> D["Validation Set"]
B --> E["Test Set"]
C --> F["Model Learning"]
D --> G["Model Selection and Tuning"]
F --> G
G --> H["Determine the Final Model"]
E --> I["Final Performance Evaluation"]
H --> I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2. Training Set: The Data from Which the Model Learns Its Parameters
1. Core Definition
The Training Set is the collection of data used to train a machine learning model.
Through the samples in the training set, the model learns the relationship between input features and the prediction target, continuously adjusting its internal parameters so that the loss function gradually decreases.
For a supervised learning task, this can be expressed as:
where:
- x_i: the input features of the i-th sample.
- y_i: the label corresponding to the i-th sample.
- N: the number of training samples.
- D_{train}: the training set.
The model can be expressed as:
where:
- f_\theta: the model determined by the parameters \theta.
- \hat{y}: the model's prediction result.
- \theta: the parameters the model needs to learn.
2. How Does the Training Set Participate in Model Training?
The core process of model training is: input a training sample, produce a prediction, compute the loss, and update the parameters.
Taking supervised learning as an example:
flowchart TB
A["Load training samples"] --> B["Feed into model"]
B --> C["Forward propagation"]
C --> D["Generate predictions"]
D --> E["Compute loss function"]
E --> F["Backpropagation to compute gradients"]
F --> G["Optimizer updates parameters"]
G --> H{"Stop condition reached?"}
H -->|"No"| A
H -->|"Yes"| I["Obtain trained model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Model training is usually centered on an optimization objective:
Where:
- L: the loss function of the model on the training set.
- \theta: model parameters.
- \theta^*: the parameters obtained after optimization.
For deep learning models, the training process usually includes:
- Forward propagation: compute the model's predictions.
- Loss calculation: measure the difference between predictions and true labels.
- Backpropagation: compute the gradients of the loss with respect to model parameters.
- Parameter update: adjust model parameters through the optimizer.
3. What should be noted about the training set?
The training set directly determines what data the model learns from, so data quality is very important.
Key points to focus on:
- Sample representativeness: Training data should cover real business scenarios as much as possible.
- Label quality: Incorrect labels can affect the direction of model learning.
- Class distribution: Severe class imbalance may cause the model to favor the majority class.
- Data coverage: Training data should contain sufficiently diverse samples.
- Data leakage: Training data must not contain information that should be unknown at prediction time.
The size of the training set also affects model performance. When there is insufficient data, the model may be unable to learn stable patterns; when data quality is problematic, simply increasing the amount of data may not improve the model.
III. Validation Set: Data for Model Selection and Hyperparameter Tuning
1. Core Definition
Validation Set is a dataset used to evaluate the model training process, select model architecture, and tune hyperparameters.
The validation set does not directly participate in the gradient updates of regular model parameters, but it affects which model is ultimately selected during development.
It primarily solves one problem:
How can we choose, among multiple candidate models, the one with better generalization ability?
2. Why is a validation set needed in addition to the training set?
Suppose we are training a classification model and need to compare different learning rates:
- Model A: learning rate 0.001.
- Model B: learning rate 0.01.
- Model C: learning rate 0.1.
If we only observe the training set loss, we cannot adequately determine which set of parameters performs better on unseen data.
Therefore, we can periodically use the validation set to evaluate model performance during training.
For example:
| Model | Training Set Accuracy | Validation Set Accuracy |
|---|---|---|
| Model A | 96% | 94% |
| Model B | 99% | 91% |
| Model C | 100% | 83% |
Model C performs best on the training set but worst on the validation set.
This suggests that Model C may be severely overfitting.
If validation set performance is used as the basis for model selection, Model A may be the more reasonable choice.
3. Main Uses of the Validation Set
(1) Hyperparameter Tuning
Hyperparameters are parameters set by the developer before or during training; they are not model parameters learned directly through the current training process.
Common hyperparameters include:
- Learning rate
- Batch Size
- Number of epochs
- Number of network layers
- Hidden layer dimensions
- Regularization coefficient
- Dropout rate
- Maximum decision tree depth
- Number of K-Means clusters K
By comparing the effects of different hyperparameter combinations on the validation set, a more suitable configuration can be selected.
(2) Model architecture selection
For example, compare:
- Logistic regression and random forest.
- CNN and Transformer.
- Neural networks with different numbers of layers.
- Different feature combinations.
- Different preprocessing methods.
The validation set can provide a consistent basis for model selection.
(3) Early Stopping
Early stopping is a training strategy used to prevent overfitting.
During training, the loss on the training set may continue to decrease, while the loss on the validation set may first decrease and then begin to rise.
When the validation metric shows no improvement for a certain number of iterations, training can be stopped early and the model parameters can be restored to those that performed best on the validation set.
flowchart TB
A["Start training"] --> B["Update parameters using the training set"]
B --> C["Evaluate on the validation set"]
C --> D{"Have validation metrics improved?"}
D -->|"Yes"| E["Save current best model"]
E --> F["Continue training"]
D -->|"No"| G["Accumulate the non-improvement count"]
G --> H{"Has the early stopping threshold been reached?"}
H -->|"No"| F
H -->|"Yes"| I["Stop training"]
F --> B
I --> J["Restore best model parameters"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
(4) Model selection and checkpoint saving
During training, multiple model checkpoints can be saved, and the best version can be selected based on validation set metrics.
For example:
- Lowest validation loss.
- Highest validation accuracy.
- Best F1 Score.
- Lowest validation MAE.
Note that evaluation metrics differ across tasks, so model performance should not be judged solely by accuracy.
4. The validation set is not completely independent
This is an easily overlooked issue.
Although the validation set does not participate in regular gradient updates, developers repeatedly observe validation results and adjust the model based on them.
For example:
- Check validation accuracy.
- Modify the model architecture.
- Train again.
- Check validation accuracy again.
- Continue tuning hyperparameters.
As the number of adjustments increases, the development process effectively becomes increasingly adapted to the validation set.
Therefore, some degree of overfitting to the validation set can also occur.
The validation set is not absolutely independent final evaluation data, which is an important reason why the test set must be kept separate.
IV. Test Set: Data for Final Model Evaluation
1. Core Definition
The test set (Test Set) is the dataset used to evaluate the performance of the final model.
The test set should remain isolated during model development and hyperparameter selection, and should not be used repeatedly to guide model adjustments.
Its core task is:
To provide a relatively independent performance evaluation once the model and training plan have been determined.
2. Role of the Test Set
Suppose model training has been completed and the final model has been determined using the validation set.
At this point, use the test set for prediction:
Then calculate the model's evaluation metrics on the test set.
For example, for a classification task:
For regression tasks, mean squared error can be used:
Test results can help determine:
- Whether the model has good generalization ability.
- Whether the model has obvious overfitting.
- Whether the model meets business launch requirements.
- Whether the model's performance differs across categories, groups of people, or scenarios.
3. Why Can't the Test Set Be Used Repeatedly for Hyperparameter Tuning?
Suppose that during model development, the test set results are repeatedly checked:
- The accuracy of the first test was 90%.
- After modifying the model, the accuracy was 91%.
- After modifying the model again, the accuracy was 92%.
- Through continuous adjustments based on the test results, 94% was eventually achieved.
Although the final test accuracy improved, the test set had already been involved in model selection.
In practice, the developer has already used feedback from the test set to optimize the model.
At this point, the test set is no longer completely independent evaluation data, and the final result may overestimate the model's true generalization ability.
The correct approach is:
- Use the training set to train the model.
- Use the validation set to tune the model.
- Determine the model and training plan.
- Finally, use the test set for final evaluation.
If the test results are unsatisfactory and it is necessary to re-enter the model development process, then one should recognize that the original test set has already participated in development decisions. To obtain a new independent evaluation, additional new test data must be held out, or a new evaluation set must be established on subsequent data.
4. The Test Set Is Not a Full Guarantee After Model Deployment
The test set can only evaluate the model's performance on a specific data sample and data distribution.
The real production environment may also experience:
- Changes in data distribution (Data Drift).
- Changes in the relationship between features and labels (Concept Drift).
- Changes in user behavior.
- Changes in the external environment.
- Changes in data collection and preprocessing logic.
Therefore, after the model is deployed, it is still necessary to continuously monitor model performance and re-evaluate and retrain it based on actual business conditions.
V. Core Differences Between the Training Set, Validation Set, and Test Set
The difference between the three lies not in the data format, but in the responsibilities each assumes and whether they are allowed to participate in model development decisions.
| Comparison Dimension | Training Set | Validation Set | Test Set |
|---|---|---|---|
| English Name | Training Set | Validation Set | Test Set |
| Core Purpose | Learn model parameters | Model selection and hyperparameter tuning | Final performance evaluation |
| Participates in Gradient Updates | Yes | Usually no | No |
| Used for Hyperparameter Tuning | Indirectly involved in the training plan | Yes | No |
| Used for Early Stopping | No, provides training data | Yes | No |
| Used for Final Evaluation | No | No | Yes |
| Usage Frequency | High | Repeatedly used during development | Used during final evaluation |
Two concepts need to be distinguished in particular:
- Model parameters (Parameters): Learned by the model from training data, such as weights and biases in a neural network.
- Hyperparameters: Set by developers or determined through search, such as learning rate, Batch Size, and number of network layers.
The training set is mainly used to learn model parameters, and the validation set is mainly used to help select hyperparameters and model design.
VI. What Proportions Should a Dataset Be Split Into?
1. Common Split Ratios
In machine learning, there is no fixed split ratio that applies to all tasks.
Common schemes are as follows:
| Split Scheme | Training Set | Validation Set | Test Set | Applicable Scenario |
|---|---|---|---|---|
| 70 / 15 / 15 | 70% | 15% | 15% | Conventional machine learning tasks |
| 80 / 10 / 10 | 80% | 10% | 10% | Relatively sufficient data volume |
| 80 / 20 | 80% | No independent validation set | 20% | Simple baseline or combined with cross-validation |
| 90 / 5 / 5 | 90% | 5% | 5% | Small data volume and need more training data |
The ratios are only empirical references and should be determined in combination with data scale, task type, and evaluation objectives.
2. Impact of Data Volume on Splitting Strategy
When the data volume is small:
If there are only a few hundred or a few thousand samples, directly splitting into three datasets may result in insufficient samples in the validation and test sets.
In this case, K-Fold Cross Validation can be considered to improve the utilization efficiency of limited data.
When the data volume is large:
If there are millions or even hundreds of millions of samples, it is usually unnecessary for the validation and test sets to account for a high proportion.
Even if only a small proportion is used, enough evaluation samples may still be obtained.
Deep learning tasks:
Deep learning tasks such as image, text, and speech often require a large amount of training data. The data split ratio needs to be determined in combination with model complexity, data scale, and training cost.
3. Data Volume and Evaluation Reliability
Too few samples in the test set can cause large random fluctuations in evaluation metrics.
For example, a binary classification model correctly predicts 18 out of 20 test samples, achieving an accuracy of 90%.
But if the prediction result for just one sample changes, the accuracy changes by 5 percentage points.
If the test set contains 100,000 samples, the effect of a single sample on overall accuracy is much smaller.
Therefore, when designing a test set, you should consider not only the proportion but also:
- Number of samples.
- Class distribution.
- Data representativeness.
- Stability of evaluation metrics.
- Confidence intervals and statistical uncertainty.
VII. Common Methods for Dataset Splitting
Not all datasets are suitable for direct random splitting.
Data structure, temporal relationships, and sample associations all influence the splitting strategy.
1. Random Split
Random split is the most common dataset splitting method.
Shuffle all samples, then allocate them to the training set, validation set, and test set according to the specified ratio.
flowchart TB
A["Original dataset"] --> B["Randomly shuffle samples"]
B --> C["Split by ratio"]
C --> D["Training set"]
C --> E["Validation set"]
C --> F["Test set"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
Applicable scenarios:
- Samples are mutually independent.
- Data distribution is relatively stable.
- There is no obvious temporal order dependency.
- There are no user, device, or entity associations that need to be isolated.
Note: A random split is not necessarily reasonable. If the data has temporal associations, user associations, or duplicate samples, a random split may cause data leakage.
2. Stratified Split
Stratified split means that when splitting a dataset, the class proportions in each subset are kept as consistent as possible with those in the original dataset.
For example, the original dataset contains:
- Normal samples: 90%.
- Anomalous samples: 10%.
If random splitting is used directly, the proportion of anomalous samples in a subset may be noticeably low.
Stratified splitting can keep the class distributions of the training, validation, and test sets as similar as possible.
Applicable scenarios:
- Binary classification tasks.
- Multiclass classification tasks.
- Class-imbalanced datasets.
- Tasks where the minority class has few samples.
Note: Stratified splitting is mainly used for classification tasks. Multilabel tasks require splitting methods suited to multilabel distributions and cannot simply apply ordinary single-label stratification logic.
3. Time-based Split
For tasks such as time series, financial forecasting, sales forecasting, and user behavior prediction, data cannot simply be split randomly.
Because when a model makes real predictions, it can only use information available before the current moment.
For example:
- Training set: 2022 to 2024.
- Validation set: first half of 2025.
- Test set: second half of 2025.
flowchart TB
A["Data sorted by time"] --> B["Historical data"]
B --> C["Training set"]
B --> D["Validation set"]
B --> E["Test set"]
C --> F["Earlier time period"]
D --> G["Middle time period"]
E --> H["Later time period"]
F --> I["Train model"]
G --> J["Tune model"]
H --> K["Simulate future prediction"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The core principle of time-based splitting is:
The training data must be earlier in time than the validation data, and the validation data must be earlier in time than the test data.
Feature generation time, label observation windows, and prediction windows also need to be considered. For example, when predicting user churn over the next 7 days, you cannot use behavioral data that is only available after the prediction time point.
4. Group Split
When multiple samples come from the same user, device, patient, or entity, grouped splitting needs to be considered.
For example, a medical dataset contains multiple patients, and each patient may have multiple examination records.
With random splitting, the following can happen:
- Some of patient A's records end up in the training set.
- Other records from patient A end up in the test set.
The model may identify test samples through patient-specific information instead of learning transferable medical patterns.
This can make test results look artificially high.
Grouped splitting requires that samples in the same group go into only one data subset.
Applicable scenarios:
- Medical patient data.
- User behavior data.
- Device monitoring data.
- Consecutive frames in a video.
- Multiple records for the same entity.
5. Combining stratification, grouping, and time constraints
Complex tasks may have multiple constraints at the same time.
For example:
- Users need to be isolated from one another.
- The proportions of each class need to be as balanced as possible.
- The data also has a temporal order.
In this case, the splitting scheme needs to be designed according to the actual evaluation goal, rather than mechanically using a single method.
The core of data splitting is not to pursue a fixed ratio, but to ensure that the evaluation process can simulate the data the model will truly face in the future.
8. Cross-validation: making fuller use of limited data
1. What is cross-validation?
Cross-validation is a method that repeatedly trains and evaluates a model by splitting the training data multiple times.
The most common form is K-fold cross-validation.
Suppose the data is divided into 5 parts:
- 1st time: the 1st part is used as the validation set, and the remaining 4 parts as the training set.
- 2nd time: the 2nd part is used as the validation set, and the remaining 4 parts as the training set.
- Continue in this way.
- Finally, 5 evaluation results are obtained.
flowchart TB
A["Complete development data"] --> B["Split into K folds"]
B --> C["Fold 1 validation"]
B --> D["Fold 2 validation"]
B --> E["Fold 3 validation"]
B --> F["Fold K validation"]
C --> G["Model training and evaluation"]
D --> G
E --> G
F --> G
G --> H["Aggregate K evaluation results"]
H --> I["Calculate average metrics and variation"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F