I. Basic Concepts of Neural Networks
A neural network is a type of machine learning model inspired by biological nervous systems and built on parameterized computational units.
It forms computational layers from multiple neurons, passes input data through a series of linear and nonlinear transformations, gradually extracts features, establishes a mapping relationship between the data and the target, and ultimately completes tasks such as classification, regression, and generation.
Neural networks are the core foundation of deep learning.
Several concepts need to be clarified:
| Concept | Meaning |
|---|---|
| Neuron | The basic computational unit that performs weighted summation and nonlinear transformation |
| Weight | A parameter that controls how much each input feature affects the computation result |
| Bias | A learnable parameter added to the linear transformation |
| Activation Function | Introduces nonlinear capabilities to neural networks |
| Layer | A computational stage consisting of a set of neurons or tensor operations |
| Forward Propagation | The process in which input passes through the network layer by layer to produce an output |
| Loss Function | A function that measures the difference between model predictions and the target |
| Backpropagation | Uses the chain rule to compute gradients of the loss with respect to model parameters |
| Optimizer | An algorithm that updates model parameters based on gradients |
The core idea of a neural network can be summarized as:
By using a large number of learnable parameters and nonlinear computations, it builds a model capable of learning complex mapping relationships from data.
II. Why Can Neural Networks Learn?
Traditional programs usually have developers explicitly write the processing rules:
Input data
↓
Execute predefined rules
↓
Output result
Neural networks, however, take a different approach.
Developers define the network structure, loss function, and training method, and the model continuously adjusts its internal parameters through training data so that predictions gradually approach the target.
For example, to identify whether an image contains a cat:
- Input: image pixel data.
- Output: the probability that the image belongs to the cat category.
- Parameters: the weights and biases inside the neural network.
- Training objective: reduce the loss between predictions and true labels.
The model does not rely on developers writing cat-recognition rules one by one; instead, it learns feature representations that can distinguish different categories from a large number of samples.
2.1 Basic Learning Process of a Neural Network
flowchart TB
A["Input training data"] --> B["Neural network forward propagation"]
B --> C["Generate prediction results"]
C --> D["Compute loss function"]
D --> E["Backpropagation computes gradients"]
E --> F["Optimizer updates parameters"]
F --> G{"Is the stopping condition met?"}
G -->|"No"| B
G -->|"Yes"| H["Obtain the trained model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
This process is repeated over multiple training steps until it reaches a preset number of training epochs, a loss threshold, a validation set stopping condition, or other stopping criteria.
Note the following:
- Forward propagation computes the prediction results.
- The loss function measures the prediction error.
- Backpropagation computes the gradients.
- The optimizer updates the parameters.
Backpropagation and parameter optimization are two distinct steps and must not be conflated.
III. Artificial Neurons: The Basic Computational Units of Neural Networks
An artificial neuron is the most basic computational unit in a neural network.
It receives multiple inputs, performs a weighted sum of the inputs, and then produces an output through an activation function.
3.1 Mathematical Expression of a Neuron
Suppose a neuron receives n inputs:
Each input corresponds to a weight:
There is also a bias parameter b.
First, compute the linear combination:
Then, pass it through the activation function:
Where:
- x_i: input features.
- w_i: weights corresponding to the inputs.
- b: bias.
- z: weighted sum result.
- f: activation function.
- a: final output of the neuron.
Overall process:
flowchart TB
A["Input feature x1"] --> D["Weighted sum"]
B["Input feature x2"] --> D
C["Input feature xn"] --> D
W["Weight parameter W"] --> D
P["Bias parameter b"] --> D
D --> E["Linear result z"]
E --> F["Activation function f"]
F --> G["Output a"]
style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style W fill:#EAF3EC,stroke:#83A995,color:#315D49
style P fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
3.2 The Role of Weights
Weights determine the degree to which different input features affect the neuron's output.
For example:
Here:
- The weight of x_1 is 0.8.
- The weight of x_2 is 0.2.
All else being equal, x_1 has a greater impact on the linear result.
Weights are core parameters that need to be learned during neural network training.
By adjusting weights, the model gradually changes the mapping relationship between input features and output results.
Note that the magnitude of a weight cannot be directly equated with feature importance. The scale, correlation, network structure, and nonlinear effects of input features all affect the actual interpretation.
3.3 The Role of Bias
Bias allows a neuron to still have a nonzero linear output when the input is zero.
Without bias:
The linear transformation must pass through the origin, limiting its expressive power.
With bias added, the model can adjust the position of the linear transformation, giving it greater fitting capability.
3.4 Why Do We Need Activation Functions?
This is one of the most important foundations for understanding neural networks.
If a neural network contains only linear transformations, then no matter how many layers are stacked, the final result is still just a linear transformation.
For example:
Substituting gives:
After simplification:
It can still be written as:
This means that multiple layers of linear computation do not automatically gain complex nonlinear expressive power.
Therefore, neural networks need to introduce nonlinear activation functions between layers.
Activation functions enable the network to learn more complex mappings and are an important reason why deep neural networks have strong expressive power.
4. Common Activation Functions
Activation functions determine how a neuron converts the linear computation result into an output.
Different activation functions have different numerical ranges, gradient properties, and applicable scenarios.
4.1 Sigmoid
Sigmoid function:
Output range:
Characteristics:
- Maps any real number to between 0 and 1.
- The output can be interpreted as a probability, but whether it is well calibrated as a probability also depends on the model and training process.
- Commonly used in the output layer of binary classification models.
- Prone to gradient saturation in regions of large positive or negative values.
Derivative:
When the absolute value of the input is large, the derivative approaches zero, which may cause vanishing gradients.
4.2 Tanh
Hyperbolic tangent function:
Output range:
Characteristics:
- The output is centered around zero.
- Compared with Sigmoid, the output contains both positive and negative values.
- It also has the gradient saturation problem in regions with large positive or negative values.
- It was once widely used in traditional recurrent neural networks.
4.3 ReLU
ReLU (Rectified Linear Unit):
Characteristics:
- When the input is greater than zero, the output is the input itself.
- When the input is less than zero, the output is zero.
- Simple computation and high training efficiency.
- Widely used in many feedforward neural networks and convolutional neural networks.
One problem with ReLU is Dying ReLU.
When a neuron remains in the negative region for a long time, its gradient is zero, and it may be unable to recover through gradient descent.
4.4 Leaky ReLU
Leaky ReLU retains a small slope in the negative region:
where \alpha is a small positive number.
It can mitigate the problem of ReLU having zero gradient in the negative region.
4.5 GELU
GELU (Gaussian Error Linear Unit) is commonly used in modern neural network architectures such as Transformer.
It is defined as:
where \Phi(x) is the cumulative distribution function of the standard normal distribution.
GELU applies a smooth nonlinear transformation to the input and, unlike ReLU, does not use simple hard threshold cutoff.
4.6 Softmax
Softmax is usually used in the output layer of multi-class classification tasks.
For an input vector:
Softmax is defined as:
The output satisfies:
Therefore, the output can be interpreted as a probability distribution over mutually exclusive classes.
Note that Softmax is used in a different scenario from hidden-layer activation functions such as ReLU and GELU. It is usually used to convert the logits of a classification model into class probabilities.
4.7 Activation Function Comparison
| Activation Function | Output Range | Main Characteristics | Common Uses |
|---|---|---|---|
| Sigmoid | (0, 1) | Probabilistic output, prone to saturation | Binary classification output layer |
| Tanh | (-1, 1) | Zero-centered, prone to saturation | Traditional RNN |
| ReLU | [0, +∞) | Simple and efficient, may die | CNN, MLP |
| Leaky ReLU | (-∞, +∞) | Retains gradient in the negative interval | Some feedforward networks |
| GELU | (-∞, +∞) | Smooth nonlinearity | Transformer |
| Softmax | (0, 1) | Outputs a normalized probability distribution | Multi-class classification output layer |
V. Basic Structure of Neural Networks
A neural network usually consists of an input layer, hidden layers, and an output layer.
5.1 Input Layer
The input layer is responsible for receiving raw data or preprocessed data.
The input format differs by task:
- Tabular data: numerical features, categorical features, etc.
- Image data: pixel matrices or image tensors.
- Text data: Token ID, embedding vectors.
- Audio data: waveform or spectral features.
- Time series: sequences of features ordered by time.
The input layer itself usually does not perform complex learnable transformations; it is mainly responsible for passing the input data to subsequent network structures.
5.2 Hidden Layer
Hidden layers are the main part of a neural network where feature representations are learned.
They consist of one or more computational layers that process data layer by layer through weights, biases, and activation functions.
For example:
Each layer generates a new feature representation based on the output of the previous layer.
In many networks, shallow layers more easily learn local, simple patterns, while deep layers can combine these representations to form more abstract features.
However, this does not mean that all neural networks learn strictly from simple to complex; the actual feature hierarchy depends on the network architecture, training data, and optimization process.
5.3 Output Layer
The output layer is responsible for converting the hidden-layer representations into the prediction results required by the task.
Common output forms:
| Task | Output Form | Common Output Activation |
|---|---|---|
| Regression | Continuous value | Linear output |
| Binary classification | Probability of a single class | Sigmoid |
| Mutually exclusive multi-class classification | Probability distribution over multiple classes | Softmax |
| Multi-label classification | Probabilities of multiple independent labels | Multiple Sigmoid functions |
A distinction has to be made here:
- Multi-class classification: a sample usually belongs to only one class.
- Multi-label classification: a sample can carry several labels at the same time.
For this reason, Softmax cannot be treated as the universal output method for every classification task.
5.4 Three-Layer Structure Diagram
flowchart LR
subgraph I["Input Layer"]
A1["Feature 1"]
A2["Feature 2"]
A3["Feature 3"]
end
subgraph H["Hidden Layer"]
B1["Neuron"]
B2["Neuron"]
B3["Neuron"]
B4["Neuron"]
end
subgraph O["Output Layer"]
C1["Prediction"]
end
A1 --> B1
A1 --> B2
A1 --> B3
A1 --> B4
A2 --> B1
A2 --> B2
A2 --> B3
A2 --> B4
A3 --> B1
A3 --> B2
A3 --> B3
A3 --> B4
B1 --> C1
B2 --> C1
B3 --> C1
B4 --> C1
style A1 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style A2 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style A3 fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style B1 fill:#EAF3EC,stroke:#83A995,color:#315D49
style B2 fill:#EAF3EC,stroke:#83A995,color:#315D49
style B3 fill:#EAF3EC,stroke:#83A995,color:#315D49
style B4 fill:#EAF3EC,stroke:#83A995,color:#315D49
style C1 fill:#DCEFE5,stroke:#4F8B70,color:#24563F
This is a simplified fully connected network structure. A real neural network may contain tens, hundreds, or even more computational modules, and it does not necessarily use a fully connected arrangement.
6. Forward Propagation: How Does a Neural Network Compute Its Output?
Forward propagation is the process in which input data enters the network, passes through the computation of each layer, and finally produces a prediction.
It is the foundation of both inference and training computation in neural networks.
6.1 Single-Layer Neural Network
For an input vector:
A fully connected layer can be expressed as:
It then passes through an activation function:
Where:
- W: the weight matrix.
- x: the input vector.
- b: the bias vector.
- z: the result of the linear transformation.
- a: the output after activation.
6.2 Multi-Layer Neural Network
Suppose the network contains L hidden or output computational layers:
For layer l:
We finally obtain:
Where:
- l: the index of the current layer.
- W^{(l)}: the weight matrix of the current layer.
- b^{(l)}: the bias of the current layer.
- a^{(l-1)}: the output of the previous layer.
- f^{(l)}: the activation function of the current layer.
- \hat{y}: the model's prediction.
These equations describe the basic computation of a standard feedforward neural network.
6.3 Tensor Dimensions
Real neural networks usually carry out batch computation in the form of matrices or tensors.
Suppose:
- The batch size is B.
- The input feature dimension is d_{in}.
- The output feature dimension is d_{out}.
Adopting the convention that batch data is arranged by rows:
Then:
We get:
Here, the bias b is broadcast along the batch dimension.
Understanding tensor shapes is very important because most computations in neural networks are built on matrix multiplication, tensor operations, and broadcasting mechanisms.
7. Backpropagation: How Do Neural Networks Learn Parameters?
The parameters of a neural network usually cannot be determined directly through simple rules; they need to be continuously adjusted based on training data.
Backpropagation is one of the core algorithms for training neural networks.
Using the chain rule, it propagates the effect of the final loss on the output backward layer by layer, computing the gradients of the weights and biases in each layer.
7.1 Loss Function
A loss function measures the difference between the model's predictions and the true targets.
For example, regression tasks often use mean squared error (MSE):
Classification tasks often use cross-entropy loss.
For a multi-class classification task with a single sample:
where:
- y_i: indicator value of the true class.
- \hat{y}_i: predicted class probability from the model.
- C: number of classes.
The loss function turns the quality of the model's predictions into an optimizable numerical objective.
7.2 Chain Rule
Suppose:
Then, according to the chain rule:
Since:
Therefore:
This shows that the gradient of the loss function with respect to the weight can be obtained by multiplying several local derivatives.
For multi-layer networks, backpropagation systematically applies this computational pattern to the entire computational graph.
7.3 Basic Flow of Backpropagation
flowchart TB
A["Input training data"] --> B["Forward propagation"]
B --> C["Compute predictions"]
C --> D["Compute loss"]
D --> E["Compute output layer gradients"]
E --> F["Backpropagate layer by layer"]
F --> G["Compute parameter gradients for each layer"]
G --> H["Optimizer updates parameters"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
7.4 Parameter Updates
After computing the gradients, the optimizer adjusts the parameters according to the gradients.
The most basic gradient descent update formula is:
where:
- w_t: current parameter.
- \eta: learning rate.
- \frac{\partial L}{\partial w_t}: gradient of the current parameter.
- w_{t+1}: updated parameter.
Bias parameters are updated in a similar way.
The goal of gradient descent is to adjust parameters along the local direction in which the loss function decreases fastest.
In practical deep learning training, commonly used optimizers also include:
- SGD.
- Momentum.
- RMSprop.
- Adam.
- AdamW.
Among them, Adam adjusts the magnitude of parameter updates using first- and second-order gradient moment estimates; AdamW decouples weight decay from the gradient update.
Note that backpropagation is responsible for computing gradients, while optimizers such as SGD and Adam are responsible for using those gradients to update the parameters.
8. Key Concepts in Neural Network Training
To understand neural networks, you cannot stop at the network architecture; you also need to master several key concepts in the training process.
8.1 Parameters
Parameters are numerical values that the model learns automatically during training.
Typical parameters include:
- Weight matrices.
- Bias vectors.
- Learnable embedding vectors in some architectures.
- Learnable scaling and shifting parameters in normalization layers.
The number of parameters is usually one of the important indicators of a model's scale.
8.2 Hyperparameters
Hyperparameters are control variables configured by the developer before or during training.
Common hyperparameters:
| Hyperparameter | Function |
|---|---|
| Learning Rate | Controls the step size of parameter updates |
| Batch Size | Number of samples used per parameter update |
| Epoch | Number of complete passes over the training set |
| Hidden Size | Feature dimension of the hidden layers |
| Number of Layers | Number of network layers |
| Dropout Rate | Proportion of random dropout |
| Weight Decay | Strength of weight decay |
The difference between parameters and hyperparameters:
- Parameters are learned through the training process.
- Hyperparameters are determined by the training configuration or the tuning process.
8.3 Batch, Iteration and Epoch
These three concepts are easy to confuse.
Batch: A group of training samples fed into the model at one time for computation.
Iteration: A single training update step, usually including forward propagation, loss computation, backpropagation, and the optimizer update.
Epoch: One complete pass of the model over the training dataset.
Suppose the training set has 10,000 samples, the Batch Size is 100, and the last incomplete batch is not dropped: