Back to series
· 12 min read

Learning Rate, Batch Size, and Epochs: Understanding the Three Core Hyperparameters in Model Training


I. Introduction

Neural network training depends not only on the model architecture, training data, and loss function, but is also shaped by a series of training hyperparameters.

Among them, the learning rate, batch size, and number of epochs are the three most fundamental and most important training configurations.

Each controls a different aspect:

  • Learning rate: how large a step the model should take at each parameter update.
  • Batch size: how many training samples are used per parameter update.
  • Epochs: how many times the entire training dataset must be fully learned.

These three parameters look simple, yet they directly affect the model's training speed, convergence process, memory consumption, and generalization ability.

They are also closely interconnected. For example, adjusting the batch size changes the number of parameter updates per epoch; adjusting the learning rate affects the size of each update; and adjusting the number of epochs affects the training process the model goes through and the total number of updates.

This article focuses on:

  • The mathematical meaning of the learning rate and its effects.
  • How batch size is computed and the principles for choosing it.
  • The differences between epoch, iteration, and step.
  • How the three hyperparameters interact.
  • Learning rate scheduling and batch size adjustment strategies.
  • The relationship between overfitting, underfitting, and the number of epochs.
  • The differences between large-batch and small-batch training.
  • Configuring training parameters in PyTorch, with practical examples.

II. An Overall View of the Three Core Hyperparameters

Let us start from the basic process of model training to understand these three parameters.

Suppose the training set contains 10,000 samples:

  • The batch size is set to 100.
  • 100 samples are read at a time.
  • The loss and gradients for the current batch are computed.
  • The optimizer updates the model parameters once.
  • This repeats until all 10,000 samples have been processed, completing one epoch.

Training for 20 epochs means the model traverses the entire training set 20 times.

In this process:

  • Batch size determines how much data is read each time.
  • Learning rate determines the step size of each parameter update.
  • Epoch determines how many full passes are made over the training set.

2.1 How the Three Relate to One Another

flowchart TB
    A["Model training configuration"]
    B["Learning rate"]
    C["Batch size"]
    D["Training epochs"]
    E["Controls the magnitude of each parameter update"]
    F["Controls the amount of data per gradient computation"]
    G["Controls the number of full dataset passes"]
    H["Jointly shape the training process"]
    I["Convergence speed, training stability, and generalization"]

    A --> B
    A --> C
    A --> D
    B --> E
    C --> F
    D --> G
    E --> H
    F --> H
    G --> H
    H --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

These three parameters cannot be understood in isolation; they must be considered together with model size, dataset size, hardware conditions, and the optimizer.


III. Learning Rate: Controlling the Size of Each Parameter Update

The learning rate (LR) is a core hyperparameter in optimization algorithms.

It determines how far the optimizer adjusts the model parameters based on the current gradient.

3.1 Mathematical Definition of the Learning Rate

Take the most basic gradient descent as an example:

\theta_{t+1} = \theta_t-\eta\nabla_\theta L(\theta_t)

Where:

  • \theta_t: the current model parameters.
  • \theta_{t+1}: the updated model parameters.
  • \eta: the learning rate.
  • \nabla_\theta L(\theta_t): the gradient of the current loss with respect to the parameters.

The learning rate determines the scaling factor applied to the gradient in the parameter update formula.

For example:

\theta=2

The current gradient is:

g=4

If the learning rate is:

\eta=0.1

Then:

\theta_{\text{new}}=2-0.1\times4=1.6

If the learning rate is changed to:

\eta=0.01

Then:

\theta_{\text{new}}=2-0.01\times4=1.96

As you can see, with the same gradient but different learning rates, the resulting parameter update magnitude also differs.

3.2 The Impact of a Learning Rate That Is Too Large

When the learning rate is too large, the model may repeatedly overshoot the target position near the bottom of the loss function.

Typical symptoms:

  • The loss function oscillates up and down.
  • Parameter updates are too large.
  • Training loss cannot decrease steadily.
  • The training process may diverge.
  • Numerical anomalies such as NaN or Inf may appear.

On some complex loss surfaces, an excessively large learning rate may also push the parameters into regions with higher loss.

3.3 The Impact of a Learning Rate That Is Too Small

When the learning rate is too small, each parameter update is very limited.

Typical symptoms:

  • Loss decreases slowly.
  • More training time is required.
  • With a limited training budget, the model may be hard to optimize sufficiently.
  • The parameters may remain in flat regions for a long time.

A learning rate that is too small does not mean the model will necessarily fail to converge; it may simply need more iterations to reach the same optimization effect.

3.4 Choosing a Learning Rate

Common example values for the learning rate:

Learning Rate Typical Characteristics
0.1 Larger step size; training stability needs attention
0.01 A common initial value to try
0.001 Common in the default configuration of optimizers such as Adam
0.0001 A smaller update step size
0.00001 Often used in some fine-tuning tasks or in more sensitive training phases

These values are only meant to build intuition; they are not fixed standards.

Different optimizers, model architectures, parameter initializations, and training data may require different learning rates.

In particular, note the following:

The same learning rate value does not represent the same actual parameter update magnitude in SGD and in Adam.

The reason is that adaptive optimizers such as Adam adjust the update magnitude of each parameter dimension based on historical gradients.

3.5 Learning Rate and Training Outcomes

flowchart TB
    A["Set the learning rate"]
    B["Learning rate too large"]
    C["Learning rate appropriate"]
    D["Learning rate too small"]
    E["Loss oscillates or diverges"]
    F["Stable optimization, loss gradually decreases"]
    G["Updates too small, convergence is slow"]

    A --> B
    A --> C
    A --> D
    B --> E
    C --> F
    D --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

IV. Batch Size: Deciding How Many Training Samples to Use Each Time

Batch size refers to the number of training samples the model uses each time it computes the loss and gradients.

For example:

Suppose the training set contains 10,000 examples.

If the batch size is set to 100, then each training iteration uses 100 samples to compute the gradient.

To process the entire training dataset, you need:

\frac{10000}{100}=100

iterations.

This assumes the dataset size is divisible by the batch size.

4.1 The Mathematical Expression of Batch Size

Suppose the loss function for the full training set is:

L(\theta)=\frac{1}{N}\sum_{i=1}^{N}\ell_i(\theta)

where:

  • N: the number of training samples.
  • \ell_i(\theta): the loss of the i-th sample.

Each step uses a mini-batch containing B samples:

\mathcal B_t=\{i_1,i_2,\ldots,i_B\}

The batch loss:

L_{\mathcal B_t}(\theta) = \frac{1}{B} \sum_{i\in\mathcal B_t}\ell_i(\theta)

The corresponding gradient:

g_t= \frac{1}{B} \sum_{i\in\mathcal B_t} \nabla_\theta\ell_i(\theta)

The optimizer updates the model parameters using this batch gradient.

So batch size essentially controls:

How many training samples are used for each gradient estimate.

4.2 Three Typical Forms of Batch Size

Depending on how many samples participate in the gradient computation each time, three forms can be distinguished.

Type Batch Size Description
Batch Gradient Descent All training samples Computes the gradient using the full dataset each time
Stochastic Gradient Descent 1 Computes the gradient using a single sample each time
Mini-batch SGD Greater than 1 and less than the dataset size Computes the gradient using a batch of samples each time

Modern deep learning typically uses mini-batch training.

4.3 Small-batch Training

For example:

B=16

16 samples are used each time.

Characteristics:

  • The amount of computation per step is relatively small.
  • Memory usage is relatively low.
  • The number of parameter updates per epoch is relatively high.
  • The gradient estimate tends to be noisier.
  • The training process may fluctuate noticeably.

This suits some scenarios with limited memory, large models, or large datasets.

4.4 Large-batch Training

For example:

B=1024

1024 samples are used each time.

Characteristics:

  • The amount of computation per step is relatively large.
  • It is easier to exploit batch-parallel computation on GPUs.
  • The gradient estimate is usually more stable.
  • Memory usage is higher.
  • The number of parameter updates per epoch is relatively low.

Large-batch training is not necessarily better than small-batch training.

In some tasks, large-batch training must be combined with learning rate adjustment, warmup, and other optimization strategies; otherwise, training may become unstable or generalization may degrade.

4.5 Batch Size and Gradient Noise

If the gradient of each sample differs, then the gradients computed from different batches will also differ.

Small batches usually have a larger gradient estimation variance.

Under simplifying conditions such as independent and identically distributed samples with finite variance, the variance of the batch-averaged gradient usually decreases as the batch size increases, approximately proportional to 1/B.

Intuitive understanding:

  • The smaller the batch size, the fewer samples are drawn each time, and the more pronounced the gradient fluctuation usually is.
  • The larger the batch size, the stronger the averaging effect on the gradient, and the more stable the estimate usually is.

However, the gradient variance in actual training is also affected by sample correlation, data distribution, model state, and sampling strategy.

4.6 Impact of Batch Size on Memory

Batch size directly affects the amount of data processed in each forward and backward pass.

When the model architecture and input size remain unchanged, increasing the batch size usually increases the memory occupied by intermediate data such as activations.

Training memory usually consists of the following parts:

  • Model parameters.
  • Parameter gradients.
  • Optimizer states.
  • Intermediate activations from the forward pass.
  • Temporary computation space.

Among these, optimizer states can also occupy a large amount of memory.

For example, Adam typically needs to maintain first-moment and second-moment states, so its optimizer state memory is usually higher than that of basic SGD, which does not maintain these states.

Note that increasing the batch size does not mean that all memory usage increases proportionally; the actual situation depends on the model architecture, implementation, mixed precision, and memory optimization techniques.


V. Training Rounds: Epoch, Iteration, and Step

A training round (Epoch) means the model traverses the entire training dataset once.

This is one of the most confusing concepts for understanding the training process.

It is necessary to clearly distinguish:

  • Epoch
  • Iteration
  • Step

5.1 Epoch

An epoch means the model performs one full training traversal over the training set.

Suppose:

  • The training set has 10,000 samples.
  • The batch size is 100.

When the model has processed all 10,000 training samples, one epoch is complete.

If set to:

Epoch=20

it means the model is scheduled to traverse the training set completely 20 times.

5.2 Iteration

An iteration usually means one training iteration, that is, processing one batch and completing the corresponding gradient computation and parameter update.

Suppose:

N=10000
B=100

Then one epoch requires:

Iteration=\frac{10000}{100}=100

iterations.

If training for 20 epochs:

Total\ Iterations=100\times20=2000

5.3 Step

In deep learning engineering, a step usually means one optimizer update, but its exact meaning needs to be determined in the context of the framework and the training code.

Especially in the gradient accumulation scenario:

  • One batch may perform one forward pass and one backward pass.
  • The gradients of multiple batches may be accumulated.
  • Only after reaching the specified number of accumulation steps is one optimizer update performed.

Therefore:

Iteration and optimizer step usually correspond in ordinary training, but they are not necessarily the same in scenarios such as gradient accumulation.

5.4 Relationship Among the Three Concepts

flowchart TB
    A["Training Set"]
    B["Epoch 1"]
    C["Batch 1"]
    D["Batch 2"]
    E["Batch 3"]
    F["..."]
    G["Batch N"]
    H["Epoch 2"]
    I["Continue iterating over the training set"]
    J["Training finished"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

5.5 How Many Iterations Are in One Epoch?

Suppose:

N=1050
B=100

If the last incomplete batch is not dropped:

StepsPerEpoch= \left\lceil\frac{1050}{100}\right\rceil =11

Where:

  • The first 10 batches each contain 100 data samples.
  • The last 1 batch contains 50 data samples.

If drop_last=True is set, the last 50 data samples are dropped:

StepsPerEpoch= \left\lfloor\frac{1050}{100}\right\rfloor =10

Therefore, when calculating the number of training steps, you need to consider whether to keep the last incomplete batch.

5.6 Epoch Is Not Equal to the Number of Parameter Updates

An epoch represents a pass through the dataset.

The number of parameter updates depends on:

  • Dataset size.
  • Batch Size.
  • Whether to drop the last batch.
  • Whether gradient accumulation is used.
  • Whether some optimizer updates are skipped.

In standard Mini-batch training:

\boxed{ StepsPerEpoch= \left\lceil\frac{N}{B}\right\rceil }

When the last batch is dropped:

StepsPerEpoch= \left\lfloor\frac{N}{B}\right\rfloor

Without gradient accumulation, one batch usually corresponds to one optimizer update.


6. Interactions Among the Three Hyperparameters

Learning rate, Batch Size, and Epoch are not independent parameters.

Changing one of them may affect the optimal configuration of the other two.

6.1 Batch Size and Number of Parameter Updates

Suppose the training set has:

N=10000

Training epochs:

Epoch=10

Compare different Batch Size:

Batch Size Updates per Epoch Total Updates
16 625 6250
32 313 3130
64 157 1570
128 79 790
256 40 400

Here it is assumed that the last batch is not dropped and that there is no gradient accumulation.

It can be seen that:

The larger the Batch Size, the fewer parameter updates per Epoch there usually are.

This means that even with the same Epoch, training processes with different Batch Sizes are not equivalent.

6.2 Why Does the Same Epoch Not Mean the Same Amount of Training?

Suppose two models are both trained for 10 Epochs:

  • Model A: Batch Size = 32.
  • Model B: Batch Size = 256.

Model A updates parameters significantly more times per Epoch.

Therefore, although both models traverse the training set completely 10 times, they undergo different numbers of parameter updates.

If training configurations need to be compared fairly, the following should also be considered:

  • Total number of optimizer updates.
  • Number of training samples actually processed.
  • Total training compute.
  • Training time.
  • Learning rate schedule progress.

6.3 Batch Size and Learning Rate

When adjusting Batch Size, it is usually also necessary to re-evaluate the learning rate.

Reasons include:

  1. Batch size affects the noise level of gradient estimates.
  2. Gradient noise affects the trajectory of parameter updates.
  3. Under different batch sizes, the model may require different learning rates.
  4. The optimization stability of large-batch training may depend on strategies such as Warmup.

For SGD, under certain conditions, the linear learning rate scaling rule can be used as a starting point for experiments:

\eta_{\text{new}} = \eta_{\text{old}} \frac{B_{\text{new}}}{B_{\text{old}}}

For example, if Batch Size increases from 128 to 256, you can try doubling the learning rate.

But this is not a universally valid mathematical law.

The actual effect is also affected by model architecture, optimizer, data scale, training duration, and learning rate scheduling method.

For adaptive optimizers such as Adam, this rule also cannot be applied mechanically.

6.4 Epoch and Learning Rate

Increasing the number of training epochs means the model has more opportunities to use the training data to adjust parameters.

However, as training progresses, the model may gradually approach a better region of the current optimization objective.

At this point, if the learning rate is still too large, it may be difficult to perform fine optimization stably.

Therefore, common practice is:

  • Use an appropriate learning rate in the early stage of training.
  • Adjust it during the middle stage according to the training situation.
  • Gradually reduce the learning rate in the later stage.
  • Decide whether to continue training based on validation set metrics.

This is why learning rate scheduling is used together with the number of training epochs.


7. Learning Rate Scheduling Strategies

In actual deep learning training, a completely fixed learning rate is rarely used throughout.

A more common approach is to dynamically adjust the learning rate according to training progress.

7.1 Step Decay

Step Decay reduces the learning rate at fixed intervals.

For example:

\eta_t= \eta_0\gamma^k

Where:

  • \eta_0: initial learning rate.
  • \gamma: decay factor.
  • k: number of decay steps already completed.

Example:

Epoch 1-10:  0.1
Epoch 11-20: 0.01
Epoch 21-30: 0.001

Suitable for scenarios where the training period is relatively clear.

7.2 Exponential Decay

Exponential decay:

\eta_t=\eta_0e^{-kt}

where k controls the decay rate.

The learning rate gradually decreases as training progresses.

7.3 Cosine Annealing

Cosine annealing adjusts the learning rate using a cosine function.

A common form is:

\eta_t= \eta_{\min} + \frac{1}{2} (\eta_{\max}-\eta_{\min}) \left( 1+\cos\frac{\pi t}{T} \right)

Where:

  • \eta_{\max}: Maximum learning rate.
  • \eta_{\min}: Minimum learning rate.
  • T: Scheduling period.
  • t: Current scheduling progress.

This approach allows the learning rate to decrease smoothly.

7.4 Warmup

Warmup is a learning rate warm-up strategy used at the beginning of training.

It does not directly use the target learning rate at the start of training; instead, it gradually increases the learning rate.

For example:

Early training: 0.00001
Mid-warmup: 0.0001
End of warmup: 0.001

Warmup is commonly used in large model training and large-batch training.

It can reduce the risk of instability when using a relatively large learning rate early in training.

7.5 Warmup + Cosine

This is a common learning rate schedule combination in modern deep learning training.

The overall process:

  1. Gradually increase the learning rate early in training.
  2. Reach the preset peak learning rate.
  3. Enter the cosine decay phase.
  4. Gradually reduce the learning rate to the configured minimum.
flowchart TB
    A["Training start"]
    B["Warmup: learning rate warm-up"]
    C["Reach peak learning rate"]
    D["Cosine decay"]
    E["Approach minimum learning rate"]
    F["Training end"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Note that Warmup and Cosine are learning rate scheduling strategies, not standalone optimizers.

They can be combined with optimizers such as SGD, Adam, and AdamW.


8. Training Epochs, Overfitting, and Underfitting

Epochs do not only control training duration; they also affect how well the model fits the training data.

8.1 Underfitting

Underfitting means the model has not yet sufficiently learned the patterns in the training data.

Typical signs:

  • Training loss is relatively high.
  • Validation loss is also relatively high.
  • Performance on both the training set and validation set is unsatisfactory.

Possible causes:

  • Insufficient model capacity.
  • Insufficient training epochs.
  • Unreasonable learning rate configuration.
  • Insufficient feature representation capability.
  • Excessive regularization.

Increasing the number of training epochs can sometimes improve underfitting, but it may not solve issues such as insufficient model capacity.

8.2 Overfitting

Overfitting means the model adapts excessively to specific patterns in the training data, leading to degraded generalization ability.

Typical signs:

  • Training loss continues to decrease.
  • Validation loss stops decreasing or even begins to rise.
  • Performance on the training set is noticeably better than on the validation set.

Possible causes:

  • Excessive model capacity.
  • Insufficient training data.
  • Too many training epochs.
  • Insufficient regularization.
  • There is a discrepancy between the training data distribution and real application scenarios.

8.3 Training Loss and Validation Loss

During training, it is usually necessary to monitor both:

  • Training Loss: loss on the training set.
  • Validation Loss: loss on the validation set.

They can help determine whether the model is still learning effectively and whether overfitting has occurred.

flowchart TB
    A["Start training"]
    B["Record training loss"]
    C["Record validation loss"]
    D{"Training and validation performance"}
    E["High training loss and high validation loss"]
    F["Training loss decreases, validation loss improves in tandem"]
    G["Training loss continues to decrease, validation loss rises"]
    H["Consider underfitting and model configuration"]
    I["Continue monitoring the training process"]
    J["Consider overfitting and early stopping"]
    K["Select the model based on the validation set"]

    A --> B
    B --> C
    C --> D
    D --> E
    D --> F
    D --> G
    E --> H
    F --> I
    G --> J
    H --> K
    I --> K
    J --> K

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F