I. Deep Learning Training: From Theoretical Models to Engineering Systems
Training a deep learning model is essentially the process of continuously adjusting the parameters of a neural network through data and optimization algorithms, so that the model's predictions on a given task gradually approach the target results.
From a mathematical perspective, the training process can be abstracted as:
where:
- \theta: the model's learnable parameters.
- D: the training dataset.
- L: the loss function of the model on the dataset.
- \theta^*: the optimized model parameters.
But in real engineering practice, model training is far more than executing a single gradient descent step.
A complete deep learning training system typically has to handle:
- Data collection and data cleaning.
- Dataset splitting and preprocessing.
- Model architecture design and parameter initialization.
- Forward propagation and loss computation.
- Backpropagation and gradient computation.
- Parameter updates by the optimizer.
- Validation set evaluation and hyperparameter tuning.
- Model checkpoint saving and restoration.
- Test set evaluation and model selection.
- Model export, deployment, and monitoring.
Together, these steps constitute the training engineering of deep learning.
II. The Complete Deep Learning Training Workflow
2.1 Overall Training Workflow
From an engineering perspective, deep learning training can be divided into four stages: data preparation, model training, model evaluation, and model delivery.
flowchart TB
A["Raw data"] --> B["Data cleaning and preprocessing"]
B --> C["Training set / validation set / test set"]
C --> D["Build Dataset and DataLoader"]
D --> E["Define model architecture"]
E --> F["Initialize model parameters"]
F --> G["Forward propagation"]
G --> H["Compute loss function"]
H --> I["Backpropagation to compute gradients"]
I --> J["Optimizer updates parameters"]
J --> K{"Validation or stopping condition reached?"}
K -->|"No"| G
K -->|"Yes"| L["Validation set evaluation and model selection"]
L --> M["Final evaluation on the test set"]
M --> N["Save model and training configuration"]
N --> O["Model deployment and inference"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style N fill:#EAF3EC,stroke:#83A995,color:#315D49
style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2.2 What Does a Training Step Consist Of?
In a typical mini-batch training step, the following operations are mainly performed:
flowchart TB
A["Get a Batch"] --> B["Clear Historical Gradients"]
B --> C["Model Forward Pass"]
C --> D["Compute Loss"]
D --> E["Loss.backward()"]
E --> F["Gradient Clipping (Optional)"]
F --> G["optimizer.step()"]
G --> H["Record Training Metrics"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
These steps are the foundation for understanding deep learning training code:
zero_grad(): Clears the gradients accumulated from the previous iteration.forward(): Computes predictions based on the current parameters.loss: Measures the difference between predictions and targets.backward(): Computes the gradients of the loss with respect to the model parameters.step(): Updates the model parameters based on the gradients.
It is important to be clear: backpropagation is responsible for computing gradients, and the optimizer is responsible for updating parameters based on the gradients. They are not the same thing.
III. Stage One: Data Preparation and Preprocessing
Data quality directly affects what a model can learn.
Deep learning models usually need raw data to be converted into tensors suitable for neural network computation and organized according to the responsibilities of training, validation, and testing.
3.1 Data Collection
Data takes different forms for different tasks.
| Task Type | Data Form | Example |
|---|---|---|
| Image Recognition | Images, labels | Images and categories |
| Text Classification | Token sequences, labels | Sentences and sentiment categories |
| Object Detection | Images, bounding boxes | Images and object coordinates |
| Speech Recognition | Audio, text | Speech and transcribed text |
| Time Series Forecasting | Continuous numeric sequences | Historical sensor data |
| Large Language Model Training | Token sequences | IDs after tokenizing text |
During data collection, the following areas require special attention:
- Whether data sources are legal and reliable.
- Whether data covers real business scenarios.
- Whether labels are accurate.
- Whether duplicate samples exist.
- Whether outliers and missing values exist.
- Whether data across different categories is severely imbalanced.
- Whether private or sensitive data is included.
Data volume does not equal data quality. A large amount of duplicated, erroneous, or biased data does not necessarily improve model capability.
3.2 Data Cleaning
The goal of data cleaning is to improve data quality and reduce interference from invalid information during training.
Common operations include:
Missing Value Handling
- Delete samples with severe missingness.
- Impute using mean, median, and other methods.
- Use dedicated missing-value markers.
- Handle according to business semantics.
Outlier Handling
- Statistical distribution analysis.
- Quantile detection.
- Rule-based validation.
- Decide whether to clip or delete based on the task.
Duplicate Data Handling
- Exact duplicate sample detection.
- Near-duplicate data detection.
- Cross-partition duplicate checks for datasets.
Label Quality Checks
- Label format validation.
- Label range checks.
- Unified category mapping.
- Manual review on sampled data.
Special attention is needed: data cleaning rules must not be fitted using information from the test set. For example, the mean and standard deviation required for standardization should be computed only from the training set.
3.3 Data Preprocessing
Different types of data usually require different preprocessing methods.
| Data Type | Common Preprocessing |
|---|---|
| Image | Resize, Crop, Normalize, data augmentation |
| Text | Tokenization, Padding, Truncation |
| Audio | Resampling, frame segmentation, Mel Spectrogram |
| Numeric features | Standardization, normalization, missing value handling |
| Time series | Sliding windows, temporal alignment, trend handling |
Taking image data as an example:
Where:
- x: original pixel value.
- \mu: mean computed from the training set.
- \sigma: standard deviation computed from the training set.
- x': normalized pixel value.
Note that the training and inference phases should use the same preprocessing configuration.
3.4 Data Augmentation
Data augmentation increases the diversity of training data by applying reasonable transformations to training samples.
Common methods:
- Random image cropping.
- Horizontal flipping.
- Color perturbation.
- Random rotation.
- Text replacement and perturbation.
- Audio noise injection.
- Time series window perturbation.
The purpose of data augmentation is not simply to create more data, but to introduce reasonable variations so that the model learns more stable features.
The augmentation strategy needs to be determined according to the task. For example, image rotation in a digit recognition task may change the semantics of the digit and cannot be used unconditionally.
3.5 Dataset Splitting
Common dataset splitting methods:
- Training Set.
- Validation Set.
- Test Set.
| Dataset | Primary Use | Whether Involved in Parameter Gradient Updates |
|---|---|---|
| Training set | Learn model parameters | Yes |
| Validation set | Model selection, hyperparameter tuning | No |
| Test set | Final evaluation of generalization ability | No |
Common ratios include:
- 70% / 15% / 15%.
- 80% / 10% / 10%.
- The training set accounts for a larger proportion, while the validation and test sets are adjusted according to the data scale.
These ratios are not fixed standards.
Data splitting needs to consider task characteristics:
- Classification tasks can use stratified sampling.
- Time series tasks should be split in chronological order.
- User behavior data needs to consider isolation at the user level.
- Highly correlated samples from the same subject should not be arbitrarily scattered across the training and test sets.
If there is information overlap between the training set and the test set, data leakage may occur, resulting in inflated evaluation results.
IV. Stage Two: Dataset and DataLoader
In PyTorch, data is usually managed through Dataset and DataLoader.
The two have different responsibilities.
4.1 Dataset: Dataset Abstraction
Dataset is responsible for describing how to access a single sample.
Usually, you need to implement:
__len__(): returns the dataset size.__getitem__(): returns a sample according to the index.
Example:
import torch
from torch.utils.data import Dataset
class CustomDataset(Dataset):
def __init__(self, features, labels):
self.features = features
self.labels = labels
def __len__(self):
return len(self.features)
def __getitem__(self, index):
x = self.features[index]
y = self.labels[index]
return x, y
features = torch.randn(1000, 20)
labels = torch.randint(0, 3, (1000,))
dataset = CustomDataset(features, labels)
print(len(dataset))
# 1000
x, y = dataset[0]
print(x.shape)
# [20]
Dataset is usually not responsible for organizing entire batches, but provides an interface for reading individual samples.
4.2 DataLoader: Batch Loading
DataLoader is responsible for organizing samples from a Dataset into batches and supports:
- Batch Size.
- Shuffle.
- Multiprocess loading.
- Data collation (Collate).
- Pin Memory.
- Handling of incomplete batches.
Example:
from torch.utils.data import DataLoader
train_loader = DataLoader(
dataset,
batch_size=32,
shuffle=True,
num_workers=4,
pin_memory=True,
drop_last=True
)
for inputs, labels in train_loader:
print(inputs.shape)
print(labels.shape)
break
Output shapes:
torch.Size([32, 20])
torch.Size([32])
4.3 The Meaning of Batch Size
Assume the training set has 1000 samples and the Batch Size is 32.
Number of iterations per Epoch:
If drop_last=True is set, the samples in the final incomplete batch are discarded, so only the following are actually processed:
Therefore, Epoch, Batch, Iteration, and Optimizer Step must be clearly distinguished.
- Epoch: One complete pass over the training dataset.
- Batch: A set of data fed into the model at one time.
- Iteration: The processing of one training batch.
- Optimizer Step: One parameter update performed by the optimizer.
In a typical training loop, one Iteration corresponds to one Optimizer Step, but this is not necessarily the case with gradient accumulation.
5. Stage 3: Model Construction and Parameter Initialization
After data preparation is complete, the neural network structure must be defined.
The model structure determines how inputs are transformed into outputs and what kind of functional relationships the model can express.
5.1 Defining the Model Structure
Take a multilayer perceptron (MLP) as an example:
import torch.nn as nn
class MLP(nn.Module):
def __init__(self, input_dim, hidden_dim, num_classes):
super().__init__()
self.network = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, num_classes)
)
def forward(self, x):
return self.network(x)
The model contains:
- Input layer.
- Hidden layers.
- Activation functions.
- Output layer.
Here, forward() defines the model's computation process from input to output.
5.2 Parameter Initialization
The weights of a neural network usually need to be initialized properly.
Common initialization methods:
| Initialization Method | Main Characteristics | Common Use Cases |
|---|---|---|
| Zero Initialization | All parameters initialized to zero | Not suitable for ordinary network weights |
| Xavier Initialization | Controls the scale of activations and gradients across layers | Sigmoid, Tanh, etc. |
| He Initialization | Accounts for ReLU-type activation functions | ReLU, Leaky ReLU |
| Orthogonal Initialization | Initializes as an orthogonal matrix | Some recurrent networks |
If all neurons are initialized with exactly the same weights, a symmetry problem may arise, preventing the neurons from learning different features.
5.3 Model Parameters and Hyperparameters
Two concepts need to be distinguished.
Model Parameters:
Variables that are updated via gradients during training.
For example:
- Weight matrices.
- Biases.
- Embedding parameters.
- Attention projection matrices.
Hyperparameters:
Parameters configured by the developer before or during training.
For example:
- Learning Rate.
- Batch Size.
- Epoch.
- Hidden Size.
- Dropout.
- Weight Decay.
- Optimizer type.
Model parameters are learned from the training data by the optimization algorithm, while hyperparameters are usually determined through experiments, search, or experience.
6. Stage 4: Forward Propagation and Loss Computation
6.1 Forward Propagation
Forward propagation is the process in which the model computes prediction results based on its current parameters.
For a multilayer neural network:
Where:
- x: input data.
- W: weights.
- b: bias.
- f: activation function.
- \hat y: the model's prediction result.
In PyTorch:
outputs = model(inputs)
This single line of code typically executes the complete forward computation defined inside the model.
6.2 Loss Function
The loss function measures the difference between the model's predictions and the true targets.
Common loss functions:
| Task | Common Loss Functions |
|---|---|
| Regression | MSE, MAE, Huber Loss |
| Binary classification | Binary Cross Entropy |
| Multi-class classification | Cross Entropy |
| Multi-label classification | BCEWithLogitsLoss |
| Object detection | Classification loss, bounding-box regression loss |
| Language model | Token-level Cross Entropy |
For example, for a multi-class classification task:
criterion = nn.CrossEntropyLoss()
loss = criterion(outputs, labels)
Note here:
CrossEntropyLoss typically takes raw Logits that have not gone through Softmax, rather than probabilities that have already been through Softmax.
6.3 Loss for One Batch
For a batch containing N samples, the common average loss can be expressed as:
Where:
- N: Batch Size.
- \hat y_i: Predicted result.
- y_i: Ground-truth label.
- \ell: Loss for a single sample.
Training is the process of gradually reducing the loss function on the training objective through parameter updates.
VII. Stage Five: Backpropagation and Gradient Computation
7.1 What Is Backpropagation?
Backpropagation is an algorithm for computing the gradients of the loss function with respect to the model parameters.
For example:
Backpropagation computes:
These gradients are used to tell the optimizer:
- What effect a change in a parameter has on the loss.
- Which parameters need to be increased.
- Which parameters need to be decreased.
- The local sensitivity of the loss to changes in each parameter.
7.2 The Chain Rule
Neural networks are usually composed of multiple functions combined together.
Suppose:
Then:
Using the chain rule, a neural network computes, layer by layer, the gradients of the output loss with respect to the parameters of each layer.
7.3 Engineering Implementation of Backpropagation
PyTorch computes gradients using automatic differentiation:
loss.backward()
In a normal training scenario, model parameters automatically record the computation graph.
After calling backward(), gradients are typically accumulated into the parameter's .grad attribute.
Note: by default, PyTorch does not automatically clear existing gradients on each backpropagation.
Therefore, it is usually necessary to call the following before the next iteration:
optimizer.zero_grad()
7.4 Gradient Accumulation
By default, one batch usually performs one backpropagation and one optimizer update.
However, when GPU memory is limited, you can accumulate the gradients of several small batches and then perform a single parameter update.
For example:
accumulation_steps = 4
optimizer.zero_grad(set_to_none=True)
for step, (inputs, labels) in enumerate(train_loader):
outputs = model(inputs)
loss = criterion(outputs, labels)
(loss / accumulation_steps).backward()
if (step + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad(set_to_none=True)
If each small batch has a size of 8 and you accumulate 4 times, then—given the same number of samples and correct loss scaling, among other conditions—the gradient effect is approximately that of a single update with a Batch Size of 32.
But note:
- Gradient accumulation does not reduce the total amount of forward and backward computation.
- Layers that depend on the current batch's statistics, such as BatchNorm, may behave differently from a true large batch.
- Random operations such as Dropout also make the two not exactly equivalent.
- Gradients from the final incomplete accumulation window need to be handled separately and must not simply be discarded.
VIII. Stage Six: The Optimizer Updates Parameters
8.1 The Role of the Optimizer
After backpropagation computes the gradients, an optimizer is still needed to update the model parameters based on those gradients.
The most basic gradient descent update formula:
Where:
- \theta_t: current parameters.
- \eta: learning rate.
- \nabla_{\theta}L: gradient of the loss with respect to the parameters.
- \theta_{t+1}: updated parameters.
8.2 Common Optimizers
| Optimizer | Core Mechanism | Common Features |
|---|---|---|
| SGD | Uses gradients to update parameters | Simple structure |
| SGD + Momentum | Introduces momentum | Improves the stability of the update direction |
| RMSprop | Moving average of squared gradients | Adaptively adjusts the step size |
| Adam | First- and second-moment estimates | Widely used |
| AdamW | Decoupled weight decay | Commonly used in Transformer |
8.3 PyTorch Optimizer Configuration
optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-3,
weight_decay=1e-4
)
Here:
model.parameters(): the model parameters to be optimized.lr: learning rate.weight_decay: weight decay coefficient.
Call:
optimizer.step()
Performs one parameter update.
The optimizer only updates parameters that participate in optimization and have valid gradients. Frozen parameters or parameters not involved in the current computation graph will not be updated in the usual way.
8.4 Learning Rate Scheduler
The learning rate has an important impact on the training process.
In the early stage of training, a larger learning rate can be used to accelerate convergence; later, the learning rate is gradually reduced to allow more fine-grained parameter adjustments.
Common strategies:
- StepLR.
- MultiStepLR.
- ExponentialLR.
- CosineAnnealingLR.
- ReduceLROnPlateau.
- Warmup + Cosine Decay.
Example:
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
optimizer,
T_max=100
)
The timing for calling the scheduler needs to be determined according to its specific type.
For schedulers that update by epoch, usually call scheduler.step() after each epoch; for schedulers that update by training step, call it after the corresponding optimizer update.
IX. Seventh Stage: Complete Training Loop
Combining the preceding steps forms the actual training loop.
9.1 Complete PyTorch Training Example
The following uses a simple multi-class classification task to integrate data loading, model training, validation, optimizer updates, and checkpoint saving.
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
# -------------------------
# 1. Prepare the data
# -------------------------
X_train = torch.randn(1000, 20)
y_train = torch.randint(0, 3, (1000,))
X_val = torch.randn(200, 20)
y_val = torch.randint(0, 3, (200,))
train_dataset = TensorDataset(X_train, y_train)
val_dataset = TensorDataset(X_val, y_val)
train_loader = DataLoader(
train_dataset,
batch_size=32,
shuffle=True
)
val_loader = DataLoader(
val_dataset,
batch_size=64,
shuffle=False
)
# -------------------------
# 2. Define the model
# -------------------------
class MLP(nn.Module):
def __init__(self):
super().__init__()
self.network = nn.Sequential(
nn.Linear(20, 128),
nn.ReLU(),
nn.Linear(128, 64),
nn.ReLU(),
nn.Linear(64, 3)
)
def forward(self, x):
return self.network(x)
device = torch.device(
"cuda" if torch.cuda.is_available() else "cpu"
)
model = MLP().to(device)
# -------------------------
# 3. Define the loss function and optimizer
# -------------------------
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-3,
weight_decay=1e-4
)
# -------------------------
# 4. Training and validation
# -------------------------
num_epochs = 20
best_val_loss = float("inf")
for epoch in range(num_epochs):
# ===== Training phase =====
model.train()
train_loss_sum = 0.0
train_samples = 0
for inputs, labels in train_loader:
inputs = inputs.to(device)
labels = labels.to(device)
optimizer.zero_grad(set_to_none=True)
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
nn.utils.clip_grad_norm_(
model.parameters(),
max_norm=1.0
)
optimizer.step()
batch_size = inputs.size(0)
train_loss_sum += loss.item() * batch_size
train_samples += batch_size
train_loss = train_loss_sum / train_samples
# ===== Validation phase =====
model.eval()
val_loss_sum = 0.0
val_samples = 0
correct = 0
with torch.no_grad():
for inputs, labels in val_loader:
inputs = inputs.to(device)
labels = labels.to(device)
outputs = model(inputs)
loss = criterion(outputs, labels)
batch_size = inputs.size(0)
val_loss_sum += loss.item() * batch_size
val_samples += batch_size
predictions = outputs.argmax(dim=1)
correct += (
predictions == labels
).sum().item()
val_loss = val_loss_sum / val_samples
val_accuracy = correct / val_samples
print(
f"Epoch {epoch + 1:02d} | "
f"Train Loss: {train_loss:.4f} | "
f"Val Loss: {val_loss:.4f} | "
f"Val Acc: {val_accuracy:.4f}"
)
# ===== Save the best model =====
if val_loss < best_val_loss:
best_val_loss = val_loss
torch.save(
{
"epoch": epoch + 1,
"model_state_dict": model.state_dict(),
"optimizer_state_dict": optimizer.state_dict(),
"best_val_loss": best_val_loss
},
"best_model.pth"
)
The data used here is randomly generated; it is only for demonstrating the structure of the training code and does not mean the model has actual classification capability.
9.2 Why do training and validation need to be handled separately?
Training phase:
model.train()
Enables training mode.
Validation phase:
model.eval()
Enables evaluation mode.
These two methods affect the behavior of certain network layers.
For example:
Dropout
- During training, randomly drops some activations.
- During inference and evaluation, random dropout is disabled.
BatchNorm
- During training, uses the statistics of the current batch and updates the running statistics.
- During evaluation, usually uses the running statistics accumulated during training.
In addition:
with torch.no_grad():
disables gradient recording in the current code block, reducing GPU memory usage during validation.
model.eval() and torch.no_grad() serve different purposes and cannot replace each other.
10. Eighth Stage: Validation Set Evaluation and Model Selection
A decrease in training loss does not necessarily mean the model will perform well on real data.
The model needs to be evaluated on a validation set to assess its generalization ability.
10.1 Common Evaluation Metrics
Different tasks require different metrics.
Classification Tasks
- Accuracy.
- Precision.
- Recall.
- F1 Score.
- ROC-AUC.
- PR-AUC.
Regression Tasks
- MAE.
- MSE.
- RMSE.
- R^2.
Object Detection Tasks
- IoU.
- Precision.
- Recall.
- mAP.
Language Models
- Cross Entropy Loss.
- Perplexity.
- Task-related evaluation metrics.
Metric selection should be based on the task objectives, rather than simply pursuing a single numerical value.
10.2 Performance on the Training Set and Validation Set
Common situations:
| Training Set Performance | Validation Set Performance | Possible Issue |
|---|---|---|
| Good | Good | The model may have good generalization ability |
| Good | Poor | Possible overfitting |
| Poor | Poor | Possible underfitting or optimization failure |
| Poor | Good | Possible data distribution discrepancy or evaluation issue |
Judgments should be made comprehensively in light of the data distribution, loss curves, and evaluation metrics.
10.3 Early Stopping
Early Stopping is a training stopping strategy.
Stop training when the validation set metric does not improve for several consecutive epochs.
For example:
patience = 5
If the validation loss does not improve for 5 consecutive epochs, you can consider stopping training and restoring the checkpoint with the best validation set performance.
Early stopping can reduce ineffective training and, to some extent, mitigate overfitting.
10.4 Training Curve Analysis
Usually, it is necessary to observe the following at the same time:
- Training Loss.
- Validation Loss.
- Training Accuracy.
- Validation Accuracy.
- Learning Rate.
Typical patterns:
Training loss keeps decreasing while validation loss begins to rise:
Overfitting may have occurred.
Both training loss and validation loss are high:
The model may be underfitting, the learning rate may be inappropriate, or there may be a problem with data processing.
Training loss suddenly becomes NaN:
This may involve issues such as an excessively large learning rate, numerical instability, anomalous data, or mixed-precision overflow.
11. Ninth Stage: Model Saving, Restoration, and Experiment Management
In practical engineering, training may last for hours, days, or even longer.
Therefore, reliable mechanisms for model saving and restoration need to be established.
11.1 Saving Model Parameters
torch.save(
model.state_dict(),
"model_weights.pth"
)
This is a common way to save model parameters in PyTorch.
When loading:
model = MLP()
state_dict = torch.load(
"model_weights.pth",
map_location="cpu",
weights_only=True
)
model.load_state_dict(state_dict)
model.eval()