I. Introduction
Forward propagation is the most fundamental computation process in neural networks and the core mechanism by which a model makes predictions.
Whether it is a simple multilayer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer, its core computation can be abstracted as follows: data enters the network from the input layer, passes through the linear transformations and nonlinear activations of each layer in sequence, and finally produces the model output.
Forward propagation primarily solves one problem:
Given input data and the current model parameters, how is the model's prediction result computed?
To understand forward propagation, one needs to grasp the following core concepts:
- The layer structure and data flow of neural networks.
- The computation process of linear transformations and activation functions.
- Changes in weights, biases, and tensor dimensions.
- Forward computation for a single sample and for batch data.
- The relationship between the output layer and different tasks.
- The connection between forward propagation, loss computation, and backpropagation.
This article aims to provide an in-depth understanding of the forward propagation mechanism of neural networks from three perspectives: mathematical formulation, matrix operations, and practical computation.
II. What Is Forward Propagation?
Forward propagation refers to the computation process in which input data follows the structure of the neural network, starting from the input layer, passing through the hidden layers in sequence, and finally reaching the output layer.
Suppose a neural network contains:
- Input layer: receives raw features.
- Hidden layers: extract and transform feature representations.
- Output layer: generates the final prediction result.
Its basic flow is as follows:
flowchart TB
A["Input data X"]
B["First layer linear transformation"]
C["First layer activation function"]
D["Second layer linear transformation"]
E["Second layer activation function"]
F["Output layer computation"]
G["Model prediction result"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The essence of forward propagation is a process of function composition.
Suppose the network contains two hidden layers and one output layer; it can be expressed as:
Where:
| Symbol | Meaning |
|---|---|
| X | Input data |
| W^{[l]} | Weight matrix of layer l |
| b^{[l]} | Bias vector of layer l |
| Z^{[l]} | Linear transformation result of layer l |
| A^{[l]} | Activation value of layer l |
| f_l | Hidden-layer activation function |
| g | Output-layer activation function |
| \hat{Y} | Model prediction result |
Here we use the matrix convention in which batch data is organized by rows; that is, each row represents one sample.
Note: Forward propagation is not simply passing the input data to the next layer; rather, each layer applies a parameterized transformation to its input. As the number of network layers increases, the data representation also changes gradually.
III. Core Computation of Forward Propagation: Linear Transformation
In a neural network, each layer usually first performs a linear transformation.
For a fully connected layer, the computation formula is:
Where:
- X: input to the current layer.
- W: weight matrix of the current layer.
- b: bias vector of the current layer.
- Z: result after linear transformation.
Strictly speaking, an expression that includes a bias term is an affine transformation, not a pure linear transformation. In engineering practice, it is usually collectively referred to as linear layer computation.
3.1 Calculation of a Single Neuron
Suppose a neuron receives three input features:
The corresponding weights are:
The bias is b.
Then the calculation result of this neuron is:
Then it passes through the activation function:
Here, a is the activation value output by the neuron.
3.2 Matrix Computation of Multiple Neurons
In an actual neural network, a layer usually contains multiple neurons.
Assume:
- The number of input features is 3.
- The number of neurons in the current layer is 4.
Then the dimension of the weight matrix is:
The dimension of the bias vector is:
The input matrix is:
where m denotes the number of samples in a batch.
After the linear transformation:
we obtain:
Here, the bias vector is broadcast along the batch dimension, so that the neurons corresponding to each sample have the same bias parameter added.
3.3 Why Use Matrix Computation?
If computation is performed neuron by neuron and sample by sample, the process becomes very cumbersome.
Matrix operations can organize a large number of multiplications and additions in a unified way, fully leveraging the parallel computing capabilities of CPUs and GPUs.
For example:
A single matrix multiplication can complete the linear computation for all samples and all neurons in a batch.
This is also one of the reasons deep learning frameworks make extensive use of tensor operations.
IV. Activation Functions: Introducing Nonlinear Expressiveness
After the linear transformation is completed, the neural network usually applies an activation function.
The computation process is:
where f is the activation function.
4.1 Why Are Activation Functions Needed?
This is a key question for understanding neural networks.
Suppose a network contains multiple layers of linear transformations:
Substituting yields:
Since matrix multiplication is still a linear transformation, multiple layers of linear transformations can ultimately still be merged into a single linear transformation.
Even if the network is increased to dozens of layers, if no nonlinear activation function is introduced in between, its overall expressive capacity remains limited to linear transformations.
After adding activation functions:
Since activation functions are usually nonlinear, the entire network can no longer be simply merged into a single linear transformation.
This enables the neural network to learn complex nonlinear mapping relationships.
4.2 Common Activation Functions
ReLU
Characteristics:
- Simple computation.
- The gradient is 1 in the positive region.
- The output is 0 in the negative region.
- May suffer from the dying ReLU problem, where neurons output 0 for a long time.
Sigmoid
Characteristics:
- The output range is (0,1).
- Commonly used in the output layer of binary classification models.
- Prone to gradient saturation when the absolute value of the input is large.
Tanh
Characteristics:
- The output range is (-1,1).
- The output is centered around 0.
- Gradient saturation may also occur when the absolute value of the input is large.
GELU
where \Phi(x) is the cumulative distribution function of the standard normal distribution.
GELU is a smooth nonlinear activation function and is widely used in network architectures such as Transformer.
4.3 Hidden Layers and Output Layers Use Different Activation Functions
Activation functions in hidden layers are mainly responsible for enhancing the network's nonlinear expressive capacity.
The activation function of the output layer needs to be selected according to the specific task.
| Task | Common output method | Output meaning |
|---|---|---|
| Regression | Linear output | Continuous value |
| Binary classification | Sigmoid | Probability of the positive class |
| Multi-class classification | Softmax | Probability for each class |
| Multi-label classification | Multiple independent Sigmoid | Independent probability for each label |
Pay special attention to the following:
- Softmax is usually used for multi-class tasks where classes are mutually exclusive.
- In multi-label classification, multiple labels can hold simultaneously, so independent Sigmoid functions are usually used.
- Regression tasks do not necessarily require an activation function; whether to use one depends on the range and constraints of the prediction target.
V. Understanding Forward Propagation Through a Complete Example
The following uses a simple fully connected neural network to calculate the forward propagation process step by step.
Assume the network structure is as follows:
- Input layer: 2 features.
- Hidden layer: 2 neurons.
- Output layer: 1 neuron.
- Hidden layer activation function: ReLU.
- Output layer: linear output.
Network structure:
flowchart TB
A["Input layer<br/>x1, x2"]
B["Hidden layer<br/>Neurons h1, h2"]
C["Output layer<br/>Neuron y"]
D["Prediction result"]
A --> B
B --> C
C --> D
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
5.1 Input data
Assume the input is:
Corresponding to two features:
5.2 Hidden layer linear computation
Assume the hidden layer weight matrix is:
The bias vector is:
Compute:
Substitute:
This gives:
That is:
5.3 Hidden layer activation computation
Using ReLU:
Since both numbers are greater than 0:
At this point, the hidden layer has completed the first transformation of the input features.
5.4 Output layer linear computation
Assume the output layer weights are:
The bias is:
Compute:
Substitute:
This gives:
Since this is a regression task, the output layer uses a linear output:
5.5 Complete calculation process
The entire forward propagation process can be summarized as:
This example demonstrates the most basic computation mechanism of a neural network:
The input data is transformed by weights and biases, then processed by the activation function, and finally produces the prediction result.
6. Batch Forward Propagation: From Single Sample to Matrix Computation
When actually training a neural network, we usually do not process only one sample at a time; instead, we use mini-batch training.
Suppose:
- The batch size is 32.
- The number of input features is 10.
- The number of hidden-layer neurons is 64.
- The output dimension is 3.
Then the shape of the input tensor is:
First-layer weights:
First-layer bias:
Compute:
This gives:
After applying the activation function:
The output shape is still:
If the output layer has 3 neurons, then:
Finally, we obtain:
6.1 Why Are Tensor Dimensions Important?
Most computations in neural networks depend on matching tensor dimensions.
For a fully connected layer:
where:
- m: number of samples in the batch.
- n: number of input features.
- h: number of neurons in the current layer.
If the input dimension of the weight matrix does not match the output dimension of the previous layer, matrix multiplication cannot be performed.
Therefore, when debugging a neural network, tensor shape is one of the most important pieces of information to check.
6.2 Dimensional Relationships Between Different Layers
For a network containing multiple fully connected layers:
where:
- n_0: number of input features.
- n_1: number of neurons in the first hidden layer.
- n_2: number of neurons in the second hidden layer.
- n_3: output dimension.
The corresponding weight matrix dimensions are:
The input and output dimensions of adjacent layers must remain consistent.
7. Forward Propagation and Loss Functions
Forward propagation yields model predictions, not a measure of training performance.
To determine whether the predictions meet expectations, we also need to compare the predicted values with the true labels.
This process is handled by the loss function.
Suppose:
- The true label is Y.
- The model prediction is \hat{Y}.
The loss function can be expressed as:
It measures the difference between the model’s predictions and the true targets.
7.1 Regression Tasks: Mean Squared Error
For regression problems, mean squared error (MSE) is commonly used:
where:
- y_i: true value.
- \hat{y}_i: predicted value.
- m: number of samples.
Generally, the larger the prediction error, the larger the loss.
7.2 Classification Tasks: Cross-Entropy Loss
For classification tasks, cross-entropy loss is commonly used.
Binary cross-entropy:
Multi-class cross-entropy:
where:
- C: number of classes.
- y_{ik}: the one-hot encoded component of the true label.
- \hat{y}_{ik}: the class probability predicted by the model.
In real-world engineering, classification models typically use the numerically stable logits version of the cross-entropy loss, avoiding the numerical issues caused by explicit Softmax and Log operations.
7.3 Relationship Between Forward Propagation and Loss Computation
Two concepts need to be distinguished: