Back to series
· 13 min read

Deep Learning Training Workflow and Engineering Implementation: From Data Preparation to Model Deployment


I. Deep Learning Training: From Theoretical Models to Engineering Systems

Training a deep learning model is essentially the process of continuously adjusting the parameters of a neural network through data and optimization algorithms, so that the model's predictions on a given task gradually approach the target results.

From a mathematical perspective, the training process can be abstracted as:

\theta^*=\arg\min_{\theta}L(\theta;D)

where:

  • \theta: the model's learnable parameters.
  • D: the training dataset.
  • L: the loss function of the model on the dataset.
  • \theta^*: the optimized model parameters.

But in real engineering practice, model training is far more than executing a single gradient descent step.

A complete deep learning training system typically has to handle:

  1. Data collection and data cleaning.
  2. Dataset splitting and preprocessing.
  3. Model architecture design and parameter initialization.
  4. Forward propagation and loss computation.
  5. Backpropagation and gradient computation.
  6. Parameter updates by the optimizer.
  7. Validation set evaluation and hyperparameter tuning.
  8. Model checkpoint saving and restoration.
  9. Test set evaluation and model selection.
  10. Model export, deployment, and monitoring.

Together, these steps constitute the training engineering of deep learning.

II. The Complete Deep Learning Training Workflow

2.1 Overall Training Workflow

From an engineering perspective, deep learning training can be divided into four stages: data preparation, model training, model evaluation, and model delivery.

flowchart TB
    A["Raw data"] --> B["Data cleaning and preprocessing"]
    B --> C["Training set / validation set / test set"]
    C --> D["Build Dataset and DataLoader"]
    D --> E["Define model architecture"]
    E --> F["Initialize model parameters"]
    F --> G["Forward propagation"]
    G --> H["Compute loss function"]
    H --> I["Backpropagation to compute gradients"]
    I --> J["Optimizer updates parameters"]
    J --> K{"Validation or stopping condition reached?"}
    K -->|"No"| G
    K -->|"Yes"| L["Validation set evaluation and model selection"]
    L --> M["Final evaluation on the test set"]
    M --> N["Save model and training configuration"]
    N --> O["Model deployment and inference"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49
    style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style N fill:#EAF3EC,stroke:#83A995,color:#315D49
    style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F

2.2 What Does a Training Step Consist Of?

In a typical mini-batch training step, the following operations are mainly performed:

flowchart TB
    A["Get a Batch"] --> B["Clear Historical Gradients"]
    B --> C["Model Forward Pass"]
    C --> D["Compute Loss"]
    D --> E["Loss.backward()"]
    E --> F["Gradient Clipping (Optional)"]
    F --> G["optimizer.step()"]
    G --> H["Record Training Metrics"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

These steps are the foundation for understanding deep learning training code:

  • zero_grad(): Clears the gradients accumulated from the previous iteration.
  • forward(): Computes predictions based on the current parameters.
  • loss: Measures the difference between predictions and targets.
  • backward(): Computes the gradients of the loss with respect to the model parameters.
  • step(): Updates the model parameters based on the gradients.

It is important to be clear: backpropagation is responsible for computing gradients, and the optimizer is responsible for updating parameters based on the gradients. They are not the same thing.

III. Stage One: Data Preparation and Preprocessing

Data quality directly affects what a model can learn.

Deep learning models usually need raw data to be converted into tensors suitable for neural network computation and organized according to the responsibilities of training, validation, and testing.

3.1 Data Collection

Data takes different forms for different tasks.

Task Type Data Form Example
Image Recognition Images, labels Images and categories
Text Classification Token sequences, labels Sentences and sentiment categories
Object Detection Images, bounding boxes Images and object coordinates
Speech Recognition Audio, text Speech and transcribed text
Time Series Forecasting Continuous numeric sequences Historical sensor data
Large Language Model Training Token sequences IDs after tokenizing text

During data collection, the following areas require special attention:

  • Whether data sources are legal and reliable.
  • Whether data covers real business scenarios.
  • Whether labels are accurate.
  • Whether duplicate samples exist.
  • Whether outliers and missing values exist.
  • Whether data across different categories is severely imbalanced.
  • Whether private or sensitive data is included.

Data volume does not equal data quality. A large amount of duplicated, erroneous, or biased data does not necessarily improve model capability.

3.2 Data Cleaning

The goal of data cleaning is to improve data quality and reduce interference from invalid information during training.

Common operations include:

Missing Value Handling

  • Delete samples with severe missingness.
  • Impute using mean, median, and other methods.
  • Use dedicated missing-value markers.
  • Handle according to business semantics.

Outlier Handling

  • Statistical distribution analysis.
  • Quantile detection.
  • Rule-based validation.
  • Decide whether to clip or delete based on the task.

Duplicate Data Handling

  • Exact duplicate sample detection.
  • Near-duplicate data detection.
  • Cross-partition duplicate checks for datasets.

Label Quality Checks

  • Label format validation.
  • Label range checks.
  • Unified category mapping.
  • Manual review on sampled data.

Special attention is needed: data cleaning rules must not be fitted using information from the test set. For example, the mean and standard deviation required for standardization should be computed only from the training set.

3.3 Data Preprocessing

Different types of data usually require different preprocessing methods.

Data Type Common Preprocessing
Image Resize, Crop, Normalize, data augmentation
Text Tokenization, Padding, Truncation
Audio Resampling, frame segmentation, Mel Spectrogram
Numeric features Standardization, normalization, missing value handling
Time series Sliding windows, temporal alignment, trend handling

Taking image data as an example:

x'=\frac{x-\mu}{\sigma}

Where:

  • x: original pixel value.
  • \mu: mean computed from the training set.
  • \sigma: standard deviation computed from the training set.
  • x': normalized pixel value.

Note that the training and inference phases should use the same preprocessing configuration.

3.4 Data Augmentation

Data augmentation increases the diversity of training data by applying reasonable transformations to training samples.

Common methods:

  • Random image cropping.
  • Horizontal flipping.
  • Color perturbation.
  • Random rotation.
  • Text replacement and perturbation.
  • Audio noise injection.
  • Time series window perturbation.

The purpose of data augmentation is not simply to create more data, but to introduce reasonable variations so that the model learns more stable features.

The augmentation strategy needs to be determined according to the task. For example, image rotation in a digit recognition task may change the semantics of the digit and cannot be used unconditionally.

3.5 Dataset Splitting

Common dataset splitting methods:

  • Training Set.
  • Validation Set.
  • Test Set.
Dataset Primary Use Whether Involved in Parameter Gradient Updates
Training set Learn model parameters Yes
Validation set Model selection, hyperparameter tuning No
Test set Final evaluation of generalization ability No

Common ratios include:

  • 70% / 15% / 15%.
  • 80% / 10% / 10%.
  • The training set accounts for a larger proportion, while the validation and test sets are adjusted according to the data scale.

These ratios are not fixed standards.

Data splitting needs to consider task characteristics:

  • Classification tasks can use stratified sampling.
  • Time series tasks should be split in chronological order.
  • User behavior data needs to consider isolation at the user level.
  • Highly correlated samples from the same subject should not be arbitrarily scattered across the training and test sets.

If there is information overlap between the training set and the test set, data leakage may occur, resulting in inflated evaluation results.

IV. Stage Two: Dataset and DataLoader

In PyTorch, data is usually managed through Dataset and DataLoader.

The two have different responsibilities.

4.1 Dataset: Dataset Abstraction

Dataset is responsible for describing how to access a single sample.

Usually, you need to implement:

  • __len__(): returns the dataset size.
  • __getitem__(): returns a sample according to the index.

Example:

import torch
from torch.utils.data import Dataset


class CustomDataset(Dataset):

    def __init__(self, features, labels):
        self.features = features
        self.labels = labels

    def __len__(self):
        return len(self.features)

    def __getitem__(self, index):
        x = self.features[index]
        y = self.labels[index]

        return x, y


features = torch.randn(1000, 20)
labels = torch.randint(0, 3, (1000,))

dataset = CustomDataset(features, labels)

print(len(dataset))
# 1000

x, y = dataset[0]

print(x.shape)
# [20]

Dataset is usually not responsible for organizing entire batches, but provides an interface for reading individual samples.

4.2 DataLoader: Batch Loading

DataLoader is responsible for organizing samples from a Dataset into batches and supports:

  • Batch Size.
  • Shuffle.
  • Multiprocess loading.
  • Data collation (Collate).
  • Pin Memory.
  • Handling of incomplete batches.

Example:

from torch.utils.data import DataLoader

train_loader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4,
    pin_memory=True,
    drop_last=True
)

for inputs, labels in train_loader:
    print(inputs.shape)
    print(labels.shape)
    break

Output shapes:

torch.Size([32, 20])
torch.Size([32])

4.3 The Meaning of Batch Size

Assume the training set has 1000 samples and the Batch Size is 32.

Number of iterations per Epoch:

Steps= \left\lceil\frac{1000}{32}\right\rceil =32

If drop_last=True is set, the samples in the final incomplete batch are discarded, so only the following are actually processed:

\left\lfloor\frac{1000}{32}\right\rfloor=31

Therefore, Epoch, Batch, Iteration, and Optimizer Step must be clearly distinguished.

  • Epoch: One complete pass over the training dataset.
  • Batch: A set of data fed into the model at one time.
  • Iteration: The processing of one training batch.
  • Optimizer Step: One parameter update performed by the optimizer.

In a typical training loop, one Iteration corresponds to one Optimizer Step, but this is not necessarily the case with gradient accumulation.

5. Stage 3: Model Construction and Parameter Initialization

After data preparation is complete, the neural network structure must be defined.

The model structure determines how inputs are transformed into outputs and what kind of functional relationships the model can express.

5.1 Defining the Model Structure

Take a multilayer perceptron (MLP) as an example:

import torch.nn as nn


class MLP(nn.Module):

    def __init__(self, input_dim, hidden_dim, num_classes):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),

            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),

            nn.Linear(hidden_dim, num_classes)
        )

    def forward(self, x):
        return self.network(x)

The model contains:

  1. Input layer.
  2. Hidden layers.
  3. Activation functions.
  4. Output layer.

Here, forward() defines the model's computation process from input to output.

5.2 Parameter Initialization

The weights of a neural network usually need to be initialized properly.

Common initialization methods:

Initialization Method Main Characteristics Common Use Cases
Zero Initialization All parameters initialized to zero Not suitable for ordinary network weights
Xavier Initialization Controls the scale of activations and gradients across layers Sigmoid, Tanh, etc.
He Initialization Accounts for ReLU-type activation functions ReLU, Leaky ReLU
Orthogonal Initialization Initializes as an orthogonal matrix Some recurrent networks

If all neurons are initialized with exactly the same weights, a symmetry problem may arise, preventing the neurons from learning different features.

5.3 Model Parameters and Hyperparameters

Two concepts need to be distinguished.

Model Parameters:

Variables that are updated via gradients during training.

For example:

  • Weight matrices.
  • Biases.
  • Embedding parameters.
  • Attention projection matrices.

Hyperparameters:

Parameters configured by the developer before or during training.

For example:

  • Learning Rate.
  • Batch Size.
  • Epoch.
  • Hidden Size.
  • Dropout.
  • Weight Decay.
  • Optimizer type.

Model parameters are learned from the training data by the optimization algorithm, while hyperparameters are usually determined through experiments, search, or experience.

6. Stage 4: Forward Propagation and Loss Computation

6.1 Forward Propagation

Forward propagation is the process in which the model computes prediction results based on its current parameters.

For a multilayer neural network:

a^{(1)}=f(W^{(1)}x+b^{(1)})
a^{(2)}=f(W^{(2)}a^{(1)}+b^{(2)})
\hat y=W^{(3)}a^{(2)}+b^{(3)}

Where:

  • x: input data.
  • W: weights.
  • b: bias.
  • f: activation function.
  • \hat y: the model's prediction result.

In PyTorch:

outputs = model(inputs)

This single line of code typically executes the complete forward computation defined inside the model.

6.2 Loss Function

The loss function measures the difference between the model's predictions and the true targets.

Common loss functions:

Task Common Loss Functions
Regression MSE, MAE, Huber Loss
Binary classification Binary Cross Entropy
Multi-class classification Cross Entropy
Multi-label classification BCEWithLogitsLoss
Object detection Classification loss, bounding-box regression loss
Language model Token-level Cross Entropy

For example, for a multi-class classification task:

criterion = nn.CrossEntropyLoss()

loss = criterion(outputs, labels)

Note here:

CrossEntropyLoss typically takes raw Logits that have not gone through Softmax, rather than probabilities that have already been through Softmax.

6.3 Loss for One Batch

For a batch containing N samples, the common average loss can be expressed as:

L= \frac{1}{N} \sum_{i=1}^{N} \ell(\hat y_i,y_i)

Where:

  • N: Batch Size.
  • \hat y_i: Predicted result.
  • y_i: Ground-truth label.
  • \ell: Loss for a single sample.

Training is the process of gradually reducing the loss function on the training objective through parameter updates.

VII. Stage Five: Backpropagation and Gradient Computation

7.1 What Is Backpropagation?

Backpropagation is an algorithm for computing the gradients of the loss function with respect to the model parameters.

For example:

L=L(\theta)

Backpropagation computes:

\nabla_{\theta}L = \frac{\partial L}{\partial\theta}

These gradients are used to tell the optimizer:

  • What effect a change in a parameter has on the loss.
  • Which parameters need to be increased.
  • Which parameters need to be decreased.
  • The local sensitivity of the loss to changes in each parameter.

7.2 The Chain Rule

Neural networks are usually composed of multiple functions combined together.

Suppose:

y=f(g(x))

Then:

\frac{dy}{dx} = \frac{df}{dg} \cdot \frac{dg}{dx}

Using the chain rule, a neural network computes, layer by layer, the gradients of the output loss with respect to the parameters of each layer.

7.3 Engineering Implementation of Backpropagation

PyTorch computes gradients using automatic differentiation:

loss.backward()

In a normal training scenario, model parameters automatically record the computation graph.

After calling backward(), gradients are typically accumulated into the parameter's .grad attribute.

Note: by default, PyTorch does not automatically clear existing gradients on each backpropagation.

Therefore, it is usually necessary to call the following before the next iteration:

optimizer.zero_grad()

7.4 Gradient Accumulation

By default, one batch usually performs one backpropagation and one optimizer update.

However, when GPU memory is limited, you can accumulate the gradients of several small batches and then perform a single parameter update.

For example:

accumulation_steps = 4

optimizer.zero_grad(set_to_none=True)

for step, (inputs, labels) in enumerate(train_loader):

    outputs = model(inputs)
    loss = criterion(outputs, labels)

    (loss / accumulation_steps).backward()

    if (step + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)

If each small batch has a size of 8 and you accumulate 4 times, then—given the same number of samples and correct loss scaling, among other conditions—the gradient effect is approximately that of a single update with a Batch Size of 32.

But note:

  • Gradient accumulation does not reduce the total amount of forward and backward computation.
  • Layers that depend on the current batch's statistics, such as BatchNorm, may behave differently from a true large batch.
  • Random operations such as Dropout also make the two not exactly equivalent.
  • Gradients from the final incomplete accumulation window need to be handled separately and must not simply be discarded.

VIII. Stage Six: The Optimizer Updates Parameters

8.1 The Role of the Optimizer

After backpropagation computes the gradients, an optimizer is still needed to update the model parameters based on those gradients.

The most basic gradient descent update formula:

\theta_{t+1} = \theta_t-\eta\nabla_{\theta}L

Where:

  • \theta_t: current parameters.
  • \eta: learning rate.
  • \nabla_{\theta}L: gradient of the loss with respect to the parameters.
  • \theta_{t+1}: updated parameters.

8.2 Common Optimizers

Optimizer Core Mechanism Common Features
SGD Uses gradients to update parameters Simple structure
SGD + Momentum Introduces momentum Improves the stability of the update direction
RMSprop Moving average of squared gradients Adaptively adjusts the step size
Adam First- and second-moment estimates Widely used
AdamW Decoupled weight decay Commonly used in Transformer

8.3 PyTorch Optimizer Configuration

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=1e-3,
    weight_decay=1e-4
)

Here:

  • model.parameters(): the model parameters to be optimized.
  • lr: learning rate.
  • weight_decay: weight decay coefficient.

Call:

optimizer.step()

Performs one parameter update.

The optimizer only updates parameters that participate in optimization and have valid gradients. Frozen parameters or parameters not involved in the current computation graph will not be updated in the usual way.

8.4 Learning Rate Scheduler

The learning rate has an important impact on the training process.

In the early stage of training, a larger learning rate can be used to accelerate convergence; later, the learning rate is gradually reduced to allow more fine-grained parameter adjustments.

Common strategies:

  • StepLR.
  • MultiStepLR.
  • ExponentialLR.
  • CosineAnnealingLR.
  • ReduceLROnPlateau.
  • Warmup + Cosine Decay.

Example:

scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer,
    T_max=100
)

The timing for calling the scheduler needs to be determined according to its specific type.

For schedulers that update by epoch, usually call scheduler.step() after each epoch; for schedulers that update by training step, call it after the corresponding optimizer update.

IX. Seventh Stage: Complete Training Loop

Combining the preceding steps forms the actual training loop.

9.1 Complete PyTorch Training Example

The following uses a simple multi-class classification task to integrate data loading, model training, validation, optimizer updates, and checkpoint saving.

import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader


# -------------------------
# 1. Prepare the data
# -------------------------

X_train = torch.randn(1000, 20)
y_train = torch.randint(0, 3, (1000,))

X_val = torch.randn(200, 20)
y_val = torch.randint(0, 3, (200,))

train_dataset = TensorDataset(X_train, y_train)
val_dataset = TensorDataset(X_val, y_val)

train_loader = DataLoader(
    train_dataset,
    batch_size=32,
    shuffle=True
)

val_loader = DataLoader(
    val_dataset,
    batch_size=64,
    shuffle=False
)


# -------------------------
# 2. Define the model
# -------------------------

class MLP(nn.Module):

    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(20, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 3)
        )

    def forward(self, x):
        return self.network(x)


device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

model = MLP().to(device)


# -------------------------
# 3. Define the loss function and optimizer
# -------------------------

criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=1e-3,
    weight_decay=1e-4
)


# -------------------------
# 4. Training and validation
# -------------------------

num_epochs = 20
best_val_loss = float("inf")

for epoch in range(num_epochs):

    # ===== Training phase =====
    model.train()

    train_loss_sum = 0.0
    train_samples = 0

    for inputs, labels in train_loader:

        inputs = inputs.to(device)
        labels = labels.to(device)

        optimizer.zero_grad(set_to_none=True)

        outputs = model(inputs)

        loss = criterion(outputs, labels)

        loss.backward()

        nn.utils.clip_grad_norm_(
            model.parameters(),
            max_norm=1.0
        )

        optimizer.step()

        batch_size = inputs.size(0)

        train_loss_sum += loss.item() * batch_size
        train_samples += batch_size

    train_loss = train_loss_sum / train_samples


    # ===== Validation phase =====
    model.eval()

    val_loss_sum = 0.0
    val_samples = 0
    correct = 0

    with torch.no_grad():

        for inputs, labels in val_loader:

            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)

            loss = criterion(outputs, labels)

            batch_size = inputs.size(0)

            val_loss_sum += loss.item() * batch_size
            val_samples += batch_size

            predictions = outputs.argmax(dim=1)

            correct += (
                predictions == labels
            ).sum().item()

    val_loss = val_loss_sum / val_samples
    val_accuracy = correct / val_samples

    print(
        f"Epoch {epoch + 1:02d} | "
        f"Train Loss: {train_loss:.4f} | "
        f"Val Loss: {val_loss:.4f} | "
        f"Val Acc: {val_accuracy:.4f}"
    )


    # ===== Save the best model =====
    if val_loss < best_val_loss:

        best_val_loss = val_loss

        torch.save(
            {
                "epoch": epoch + 1,
                "model_state_dict": model.state_dict(),
                "optimizer_state_dict": optimizer.state_dict(),
                "best_val_loss": best_val_loss
            },
            "best_model.pth"
        )

The data used here is randomly generated; it is only for demonstrating the structure of the training code and does not mean the model has actual classification capability.

9.2 Why do training and validation need to be handled separately?

Training phase:

model.train()

Enables training mode.

Validation phase:

model.eval()

Enables evaluation mode.

These two methods affect the behavior of certain network layers.

For example:

Dropout

  • During training, randomly drops some activations.
  • During inference and evaluation, random dropout is disabled.

BatchNorm

  • During training, uses the statistics of the current batch and updates the running statistics.
  • During evaluation, usually uses the running statistics accumulated during training.

In addition:

with torch.no_grad():

disables gradient recording in the current code block, reducing GPU memory usage during validation.

model.eval() and torch.no_grad() serve different purposes and cannot replace each other.

10. Eighth Stage: Validation Set Evaluation and Model Selection

A decrease in training loss does not necessarily mean the model will perform well on real data.

The model needs to be evaluated on a validation set to assess its generalization ability.

10.1 Common Evaluation Metrics

Different tasks require different metrics.

Classification Tasks

  • Accuracy.
  • Precision.
  • Recall.
  • F1 Score.
  • ROC-AUC.
  • PR-AUC.

Regression Tasks

  • MAE.
  • MSE.
  • RMSE.
  • R^2.

Object Detection Tasks

  • IoU.
  • Precision.
  • Recall.
  • mAP.

Language Models

  • Cross Entropy Loss.
  • Perplexity.
  • Task-related evaluation metrics.

Metric selection should be based on the task objectives, rather than simply pursuing a single numerical value.

10.2 Performance on the Training Set and Validation Set

Common situations:

Training Set Performance Validation Set Performance Possible Issue
Good Good The model may have good generalization ability
Good Poor Possible overfitting
Poor Poor Possible underfitting or optimization failure
Poor Good Possible data distribution discrepancy or evaluation issue

Judgments should be made comprehensively in light of the data distribution, loss curves, and evaluation metrics.

10.3 Early Stopping

Early Stopping is a training stopping strategy.

Stop training when the validation set metric does not improve for several consecutive epochs.

For example:

patience = 5

If the validation loss does not improve for 5 consecutive epochs, you can consider stopping training and restoring the checkpoint with the best validation set performance.

Early stopping can reduce ineffective training and, to some extent, mitigate overfitting.

10.4 Training Curve Analysis

Usually, it is necessary to observe the following at the same time:

  • Training Loss.
  • Validation Loss.
  • Training Accuracy.
  • Validation Accuracy.
  • Learning Rate.

Typical patterns:

Training loss keeps decreasing while validation loss begins to rise:

Overfitting may have occurred.

Both training loss and validation loss are high:

The model may be underfitting, the learning rate may be inappropriate, or there may be a problem with data processing.

Training loss suddenly becomes NaN:

This may involve issues such as an excessively large learning rate, numerical instability, anomalous data, or mixed-precision overflow.

11. Ninth Stage: Model Saving, Restoration, and Experiment Management

In practical engineering, training may last for hours, days, or even longer.

Therefore, reliable mechanisms for model saving and restoration need to be established.

11.1 Saving Model Parameters

torch.save(
    model.state_dict(),
    "model_weights.pth"
)

This is a common way to save model parameters in PyTorch.

When loading:

model = MLP()

state_dict = torch.load(
    "model_weights.pth",
    map_location="cpu",
    weights_only=True
)

model.load_state_dict(state_dict)

model.eval()