Back to series
· 7 min read

Neural Network Forward Propagation: The Complete Computation from Input Data to Model Output


I. Introduction

Forward propagation is the most fundamental computation process in neural networks and the core mechanism by which a model makes predictions.

Whether it is a simple multilayer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer, its core computation can be abstracted as follows: data enters the network from the input layer, passes through the linear transformations and nonlinear activations of each layer in sequence, and finally produces the model output.

Forward propagation primarily solves one problem:

Given input data and the current model parameters, how is the model's prediction result computed?

To understand forward propagation, one needs to grasp the following core concepts:

  • The layer structure and data flow of neural networks.
  • The computation process of linear transformations and activation functions.
  • Changes in weights, biases, and tensor dimensions.
  • Forward computation for a single sample and for batch data.
  • The relationship between the output layer and different tasks.
  • The connection between forward propagation, loss computation, and backpropagation.

This article aims to provide an in-depth understanding of the forward propagation mechanism of neural networks from three perspectives: mathematical formulation, matrix operations, and practical computation.


II. What Is Forward Propagation?

Forward propagation refers to the computation process in which input data follows the structure of the neural network, starting from the input layer, passing through the hidden layers in sequence, and finally reaching the output layer.

Suppose a neural network contains:

  • Input layer: receives raw features.
  • Hidden layers: extract and transform feature representations.
  • Output layer: generates the final prediction result.

Its basic flow is as follows:

flowchart TB
    A["Input data X"]
    B["First layer linear transformation"]
    C["First layer activation function"]
    D["Second layer linear transformation"]
    E["Second layer activation function"]
    F["Output layer computation"]
    G["Model prediction result"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The essence of forward propagation is a process of function composition.

Suppose the network contains two hidden layers and one output layer; it can be expressed as:

\begin{aligned} Z^{[1]} &= XW^{[1]}+b^{[1]}\\ A^{[1]} &= f_1(Z^{[1]})\\ Z^{[2]} &= A^{[1]}W^{[2]}+b^{[2]}\\ A^{[2]} &= f_2(Z^{[2]})\\ Z^{[3]} &= A^{[2]}W^{[3]}+b^{[3]}\\ \hat{Y} &= g(Z^{[3]}) \end{aligned}

Where:

Symbol Meaning
X Input data
W^{[l]} Weight matrix of layer l
b^{[l]} Bias vector of layer l
Z^{[l]} Linear transformation result of layer l
A^{[l]} Activation value of layer l
f_l Hidden-layer activation function
g Output-layer activation function
\hat{Y} Model prediction result

Here we use the matrix convention in which batch data is organized by rows; that is, each row represents one sample.

Note: Forward propagation is not simply passing the input data to the next layer; rather, each layer applies a parameterized transformation to its input. As the number of network layers increases, the data representation also changes gradually.


III. Core Computation of Forward Propagation: Linear Transformation

In a neural network, each layer usually first performs a linear transformation.

For a fully connected layer, the computation formula is:

Z=XW+b

Where:

  • X: input to the current layer.
  • W: weight matrix of the current layer.
  • b: bias vector of the current layer.
  • Z: result after linear transformation.

Strictly speaking, an expression that includes a bias term is an affine transformation, not a pure linear transformation. In engineering practice, it is usually collectively referred to as linear layer computation.

3.1 Calculation of a Single Neuron

Suppose a neuron receives three input features:

X=[x_1,x_2,x_3]

The corresponding weights are:

W= \begin{bmatrix} w_1\\ w_2\\ w_3 \end{bmatrix}

The bias is b.

Then the calculation result of this neuron is:

z=x_1w_1+x_2w_2+x_3w_3+b

Then it passes through the activation function:

a=f(z)

Here, a is the activation value output by the neuron.

3.2 Matrix Computation of Multiple Neurons

In an actual neural network, a layer usually contains multiple neurons.

Assume:

  • The number of input features is 3.
  • The number of neurons in the current layer is 4.

Then the dimension of the weight matrix is:

W\in\mathbb{R}^{3\times4}

The dimension of the bias vector is:

b\in\mathbb{R}^{1\times4}

The input matrix is:

X\in\mathbb{R}^{m\times3}

where m denotes the number of samples in a batch.

After the linear transformation:

Z=XW+b

we obtain:

Z\in\mathbb{R}^{m\times4}

Here, the bias vector is broadcast along the batch dimension, so that the neurons corresponding to each sample have the same bias parameter added.

3.3 Why Use Matrix Computation?

If computation is performed neuron by neuron and sample by sample, the process becomes very cumbersome.

Matrix operations can organize a large number of multiplications and additions in a unified way, fully leveraging the parallel computing capabilities of CPUs and GPUs.

For example:

Z=XW+b

A single matrix multiplication can complete the linear computation for all samples and all neurons in a batch.

This is also one of the reasons deep learning frameworks make extensive use of tensor operations.


IV. Activation Functions: Introducing Nonlinear Expressiveness

After the linear transformation is completed, the neural network usually applies an activation function.

The computation process is:

A=f(Z)

where f is the activation function.

4.1 Why Are Activation Functions Needed?

This is a key question for understanding neural networks.

Suppose a network contains multiple layers of linear transformations:

H_1=XW_1
H_2=H_1W_2

Substituting yields:

H_2=XW_1W_2

Since matrix multiplication is still a linear transformation, multiple layers of linear transformations can ultimately still be merged into a single linear transformation.

Even if the network is increased to dozens of layers, if no nonlinear activation function is introduced in between, its overall expressive capacity remains limited to linear transformations.

After adding activation functions:

H_1=f(XW_1+b_1)
H_2=f(H_1W_2+b_2)

Since activation functions are usually nonlinear, the entire network can no longer be simply merged into a single linear transformation.

This enables the neural network to learn complex nonlinear mapping relationships.

4.2 Common Activation Functions

ReLU

ReLU(x)=\max(0,x)

Characteristics:

  • Simple computation.
  • The gradient is 1 in the positive region.
  • The output is 0 in the negative region.
  • May suffer from the dying ReLU problem, where neurons output 0 for a long time.

Sigmoid

\sigma(x)=\frac{1}{1+e^{-x}}

Characteristics:

  • The output range is (0,1).
  • Commonly used in the output layer of binary classification models.
  • Prone to gradient saturation when the absolute value of the input is large.

Tanh

tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}

Characteristics:

  • The output range is (-1,1).
  • The output is centered around 0.
  • Gradient saturation may also occur when the absolute value of the input is large.

GELU

GELU(x)=x\Phi(x)

where \Phi(x) is the cumulative distribution function of the standard normal distribution.

GELU is a smooth nonlinear activation function and is widely used in network architectures such as Transformer.

4.3 Hidden Layers and Output Layers Use Different Activation Functions

Activation functions in hidden layers are mainly responsible for enhancing the network's nonlinear expressive capacity.

The activation function of the output layer needs to be selected according to the specific task.

Task Common output method Output meaning
Regression Linear output Continuous value
Binary classification Sigmoid Probability of the positive class
Multi-class classification Softmax Probability for each class
Multi-label classification Multiple independent Sigmoid Independent probability for each label

Pay special attention to the following:

  • Softmax is usually used for multi-class tasks where classes are mutually exclusive.
  • In multi-label classification, multiple labels can hold simultaneously, so independent Sigmoid functions are usually used.
  • Regression tasks do not necessarily require an activation function; whether to use one depends on the range and constraints of the prediction target.

V. Understanding Forward Propagation Through a Complete Example

The following uses a simple fully connected neural network to calculate the forward propagation process step by step.

Assume the network structure is as follows:

  • Input layer: 2 features.
  • Hidden layer: 2 neurons.
  • Output layer: 1 neuron.
  • Hidden layer activation function: ReLU.
  • Output layer: linear output.

Network structure:

flowchart TB
    A["Input layer<br/>x1, x2"]
    B["Hidden layer<br/>Neurons h1, h2"]
    C["Output layer<br/>Neuron y"]
    D["Prediction result"]

    A --> B
    B --> C
    C --> D

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F

5.1 Input data

Assume the input is:

X= \begin{bmatrix} 1 & 2 \end{bmatrix}

Corresponding to two features:

x_1=1,\quad x_2=2

5.2 Hidden layer linear computation

Assume the hidden layer weight matrix is:

W^{[1]}= \begin{bmatrix} 0.5 & 0.3\\ 0.2 & 0.4 \end{bmatrix}

The bias vector is:

b^{[1]}= \begin{bmatrix} 0.1 & 0.2 \end{bmatrix}

Compute:

Z^{[1]}=XW^{[1]}+b^{[1]}

Substitute:

Z^{[1]}= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 0.5 & 0.3\\ 0.2 & 0.4 \end{bmatrix} + \begin{bmatrix} 0.1 & 0.2 \end{bmatrix}

This gives:

Z^{[1]}= \begin{bmatrix} 1.0 & 1.3 \end{bmatrix}

That is:

z_1=1.0,\quad z_2=1.3

5.3 Hidden layer activation computation

Using ReLU:

A^{[1]}=ReLU(Z^{[1]})

Since both numbers are greater than 0:

A^{[1]}= \begin{bmatrix} 1.0 & 1.3 \end{bmatrix}

At this point, the hidden layer has completed the first transformation of the input features.

5.4 Output layer linear computation

Assume the output layer weights are:

W^{[2]}= \begin{bmatrix} 0.6\\ 0.8 \end{bmatrix}

The bias is:

b^{[2]}=0.1

Compute:

Z^{[2]}=A^{[1]}W^{[2]}+b^{[2]}

Substitute:

Z^{[2]}= \begin{bmatrix} 1.0 & 1.3 \end{bmatrix} \begin{bmatrix} 0.6\\ 0.8 \end{bmatrix} +0.1

This gives:

Z^{[2]}=1.74

Since this is a regression task, the output layer uses a linear output:

\hat{Y}=1.74

5.5 Complete calculation process

The entire forward propagation process can be summarized as:

\begin{aligned} X &= \begin{bmatrix}1&2\end{bmatrix}\\ Z^{[1]} &= XW^{[1]}+b^{[1]}\\ A^{[1]} &= ReLU(Z^{[1]})\\ Z^{[2]} &= A^{[1]}W^{[2]}+b^{[2]}\\ \hat{Y} &= Z^{[2]}=1.74 \end{aligned}

This example demonstrates the most basic computation mechanism of a neural network:

The input data is transformed by weights and biases, then processed by the activation function, and finally produces the prediction result.


6. Batch Forward Propagation: From Single Sample to Matrix Computation

When actually training a neural network, we usually do not process only one sample at a time; instead, we use mini-batch training.

Suppose:

  • The batch size is 32.
  • The number of input features is 10.
  • The number of hidden-layer neurons is 64.
  • The output dimension is 3.

Then the shape of the input tensor is:

X\in\mathbb{R}^{32\times10}

First-layer weights:

W^{[1]}\in\mathbb{R}^{10\times64}

First-layer bias:

b^{[1]}\in\mathbb{R}^{64}

Compute:

Z^{[1]}=XW^{[1]}+b^{[1]}

This gives:

Z^{[1]}\in\mathbb{R}^{32\times64}

After applying the activation function:

A^{[1]}=f(Z^{[1]})

The output shape is still:

A^{[1]}\in\mathbb{R}^{32\times64}

If the output layer has 3 neurons, then:

W^{[2]}\in\mathbb{R}^{64\times3}
b^{[2]}\in\mathbb{R}^{3}

Finally, we obtain:

\hat{Y}\in\mathbb{R}^{32\times3}

6.1 Why Are Tensor Dimensions Important?

Most computations in neural networks depend on matching tensor dimensions.

For a fully connected layer:

X_{m\times n}\cdot W_{n\times h} = Z_{m\times h}

where:

  • m: number of samples in the batch.
  • n: number of input features.
  • h: number of neurons in the current layer.

If the input dimension of the weight matrix does not match the output dimension of the previous layer, matrix multiplication cannot be performed.

Therefore, when debugging a neural network, tensor shape is one of the most important pieces of information to check.

6.2 Dimensional Relationships Between Different Layers

For a network containing multiple fully connected layers:

n_0\rightarrow n_1\rightarrow n_2\rightarrow n_3

where:

  • n_0: number of input features.
  • n_1: number of neurons in the first hidden layer.
  • n_2: number of neurons in the second hidden layer.
  • n_3: output dimension.

The corresponding weight matrix dimensions are:

W^{[1]}\in\mathbb{R}^{n_0\times n_1}
W^{[2]}\in\mathbb{R}^{n_1\times n_2}
W^{[3]}\in\mathbb{R}^{n_2\times n_3}

The input and output dimensions of adjacent layers must remain consistent.


7. Forward Propagation and Loss Functions

Forward propagation yields model predictions, not a measure of training performance.

To determine whether the predictions meet expectations, we also need to compare the predicted values with the true labels.

This process is handled by the loss function.

Suppose:

  • The true label is Y.
  • The model prediction is \hat{Y}.

The loss function can be expressed as:

L=L(Y,\hat{Y})

It measures the difference between the model’s predictions and the true targets.

7.1 Regression Tasks: Mean Squared Error

For regression problems, mean squared error (MSE) is commonly used:

MSE=\frac{1}{m}\sum_{i=1}^{m}(y_i-\hat{y}_i)^2

where:

  • y_i: true value.
  • \hat{y}_i: predicted value.
  • m: number of samples.

Generally, the larger the prediction error, the larger the loss.

7.2 Classification Tasks: Cross-Entropy Loss

For classification tasks, cross-entropy loss is commonly used.

Binary cross-entropy:

L=-\frac{1}{m}\sum_{i=1}^{m} [y_i\log(\hat{y}_i)+(1-y_i)\log(1-\hat{y}_i)]

Multi-class cross-entropy:

L=-\frac{1}{m}\sum_{i=1}^{m} \sum_{k=1}^{C}y_{ik}\log(\hat{y}_{ik})

where:

  • C: number of classes.
  • y_{ik}: the one-hot encoded component of the true label.
  • \hat{y}_{ik}: the class probability predicted by the model.

In real-world engineering, classification models typically use the numerically stable logits version of the cross-entropy loss, avoiding the numerical issues caused by explicit Softmax and Log operations.

7.3 Relationship Between Forward Propagation and Loss Computation

Two concepts need to be distinguished: