I. What Is a CNN?
CNN (Convolutional Neural Network) is a class of deep learning models centered on the convolution operation, mainly used to process data with a grid structure.
Its most typical application is image processing.
An RGB image can be represented as a three-dimensional tensor:
where:
- H: image height.
- W: image width.
- C: number of channels.
- RGB color images usually have 3 channels.
Traditional fully connected neural networks usually need to flatten an image into a one-dimensional vector before performing matrix operations.
For example, a 224\times224\times3 image contains the following after flattening:
input features.
If it is directly connected to a fully connected layer with 1024 neurons, the weight parameters alone number:
This does not yet include bias parameters.
For data such as images, which have a clear spatial structure, directly using fully connected layers produces a large number of parameters and also makes it difficult to effectively exploit local spatial relationships.
A CNN slides convolution kernels over the image, performs feature extraction on local regions, and reduces the number of model parameters through parameter sharing.
Its core ideas are:
- Local connectivity: each convolution kernel focuses only on a local region of the input.
- Weight sharing: the same convolution kernel uses the same parameters at different spatial positions.
- Hierarchical feature extraction: progressively combine low-level visual features into high-level semantic features.
II. Core Structure of a CNN
A typical CNN usually consists of the following modules:
- Input Layer
- Convolution Layer
- Activation Function
- Pooling Layer
- Fully Connected Layer
- Output Layer
Structure of a typical image classification CNN:
flowchart TB
A["Input image"] --> B["Convolution layer"]
B --> C["Activation function ReLU"]
C --> D["Pooling layer"]
D --> E["Convolution layer"]
E --> F["Activation function ReLU"]
F --> G["Pooling layer"]
G --> H["Flatten"]
H --> I["Fully connected layer"]
I --> J["Softmax classification output"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Note that this is only one way of organizing a classic CNN.
Modern CNNs do not necessarily strictly follow a fixed structure of convolution, activation, pooling, convolution, activation, pooling. For example, ResNet uses residual connections, MobileNet uses depthwise separable convolutions, and some networks use global average pooling to replace the large fully connected layer after flattening.
III. The Convolution Operation: The Core of CNN
3.1 What Is Convolution?
Convolution is the central operation in a CNN, used to extract local features from input data.
In two-dimensional image processing, a convolution kernel (Kernel) is a relatively small weight matrix.
For example, a 3\times3 convolution kernel:
This convolution kernel can perform weighted computation on a local region in the image, producing a stronger response to edges in a specific direction.
Suppose a local region in the input image is:
The convolution kernel is:
Multiply the corresponding positions and then sum:
This result is one output value obtained from the convolution operation over that local region.
The convolution kernel continuously moves over the input image, repeatedly performing the same computation on different local regions, ultimately forming a feature map (Feature Map).
3.2 Convolution Kernels and Feature Extraction
Different convolution kernels can learn different types of local patterns.
Common visual features include:
- Horizontal edges.
- Vertical edges.
- Diagonal edges.
- Color changes.
- Texture.
- Local shapes.
It should be noted that convolution kernels in modern CNNs are usually not manually specified edge-detection operators, but parameters automatically learned from training data through backpropagation and optimization algorithms.
During training, the model adjusts the convolution kernel weights according to the task loss, gradually developing feature extraction capabilities useful for the target task.
3.3 Convolution in CNNs and Convolution in Mathematics
Strictly speaking, the convolution operations commonly found in deep learning frameworks are usually cross-correlation (Cross-Correlation), rather than the strict convolution in the mathematical definition that requires flipping the convolution kernel.
Two-dimensional cross-correlation can be expressed as:
Where:
- X: input feature map.
- K: convolution kernel.
- K_h,K_w: convolution kernel height and width.
- Y: output feature map.
Strict mathematical convolution requires spatially flipping the convolution kernel.
However, in CNNs, the convolution kernel parameters are learned through training. Using cross-correlation does not change the model's expressive capacity, so deep learning frameworks usually adopt this implementation approach.
IV. Parameters of Convolutional Layers
A convolutional layer not only contains convolution kernels, but also involves multiple important parameters that control how convolution is computed.
4.1 Kernel Size: Convolution Kernel Size
Kernel Size indicates the spatial size of the convolution kernel.
Common configurations:
| Convolution Kernel | Characteristics |
|---|---|
| 1×1 | Channel information fusion and dimension adjustment |
| 3×3 | Common local feature extraction |
| 5×5 | Larger local spatial range |
| 7×7 | Larger initial receptive area |
Larger convolution kernels can cover a larger spatial range in a single operation, but usually require more computation.
Multiple 3\times3 convolutional layers can gradually expand the receptive field while introducing more nonlinear transformations.
For example, two consecutive 3\times3 convolutional layers, with a stride of 1 and no dilated convolution, have a theoretical receptive field of 5\times5.
4.2 Stride: Step Size
Stride indicates the pixel distance the convolution kernel moves each time.
For example:
- Stride = 1: move 1 position each time.
- Stride = 2: move 2 positions each time.
- Stride = 3: move 3 positions each time.
The larger the stride, the fewer times the convolution kernel moves, and the output feature map is usually smaller.
4.3 Padding: Padding
Padding indicates adding extra pixels around the input feature map, usually using zero padding (Zero Padding).
Common purposes:
- Control the output feature map size.
- Preserve information at edge positions.
- Allow spatial resolution to be maintained after multiple convolution layers.
Common types:
| Type | Description |
|---|---|
| Valid | No additional padding |
| Same | Pad according to the target output size, usually keeping the output spatial size unchanged (when Stride=1) |
| Zero Padding | Pad the perimeter with zero values |
4.4 Output Size Calculation
The output size of a convolutional layer is jointly determined by the input size, convolution kernel size, padding, and stride.
In the height direction:
In the width direction:
Where:
- H_{\text{in}},W_{\text{in}}: input height and width.
- K_h,K_w: convolution kernel height and width.
- P_h,P_w: padding size.
- S_h,S_w: stride.
- D_h,D_w: dilation rate (Dilation).
- \lfloor\cdot\rfloor: round down (floor).
If dilated convolution is not used, D_h=D_w=1.
Example:
Input size:
Convolution kernel:
Padding:
Stride:
Then the output height is:
The width works the same way, so the output size remains:
If you change it to Stride = 2:
The output size becomes:
5. Input Channels and Output Channels
Understanding channels is the key to mastering the computation process of a CNN.
5.1 Input Channels
The number of input channels indicates how many channels the input feature map contains.
For example:
- Grayscale image: 1 channel.
- RGB image: 3 channels.
- Intermediate-layer feature maps: may contain 64, 128, 256, or other numbers of channels.
5.2 Output Channels
The number of output channels indicates how many feature maps a convolutional layer generates.
One output channel usually corresponds to one independent set of convolution kernel parameters.
Suppose the input feature map is:
Using 16 3\times3 convolution kernels, with stride 1 and Padding 1.
Each output channel needs to cover all 3 input channels.
Therefore, the spatial and channel dimensions of each convolution kernel are:
This ultimately generates 16 output channels:
5.3 Calculating the Number of Parameters in a Convolutional Layer
Suppose:
- Number of input channels: C_{\text{in}}
- Number of output channels: C_{\text{out}}
- Convolution kernel size: K_h\times K_w
- Each output channel has one bias parameter.
The number of parameters in the convolutional layer is:
For example:
- Input channels: 3.
- Output channels: 64.
- Convolution kernel: 3\times3.
Then the number of parameters is:
This shows that, through local connectivity and parameter sharing, a convolutional layer can process a fairly large image with relatively few parameters.
If bias=False is set, the final bias parameters are not included.
6. Feature Maps
A feature map is the output result of the computation performed by a convolutional layer.
It contains not only the values, but also how those values are distributed across spatial positions.
For example:
After passing through a convolutional layer:
This means:
- The spatial height is 112.
- The spatial width is 112.
- It contains 64 feature channels.
Each channel can be understood as a different kind of feature response map.
During training, different channels may gradually develop different responses to different visual patterns.
6.1 Feature Representations in Different Network Layers
CNNs usually form hierarchical feature representations gradually through multiple layers of convolution.
Shallow features:
- Edges.
- Color changes.
- Orientation.
- Simple textures.
Mid-level features:
- Local shapes.
- Texture combinations.
- Local object structure.
Deep features:
- Object parts.
- Complex shapes.
- Semantic patterns related to categories.
Note that this is a generalization intended to aid understanding. CNN channels do not necessarily correspond to a single semantic concept that humans can explicitly name, and feature mixing may also occur between different layers.
7. Pooling Layer
7.1 What Is Pooling?
Pooling is an operation that aggregates local regions, commonly used to reduce the spatial size of feature maps.
It can reduce subsequent computation and make feature representations more stable to position changes within a certain range.
Common pooling methods:
- Max pooling.
- Average pooling.
- Global average pooling.
7.2 Max Pooling
Max pooling selects the maximum value in a specified region.
For example, input region:
Using a 2\times2 pooling kernel, Stride = 2.
This gives:
Max pooling preserves the feature with the strongest response in each local region.
7.3 Average Pooling
Average pooling computes the average of all elements in a local region.
For example:
After average pooling:
Max pooling emphasizes the strongest response, while average pooling preserves the overall average information of the region.
7.4 Global Average Pooling
Global average pooling computes the average over the entire spatial region for each channel.
Suppose the input feature map is:
After global average pooling:
Each channel ultimately retains only a single value.
It is often used in classification networks to replace the flattening operation before a large fully connected layer, reducing the number of parameters.
7.5 Is Pooling Always Necessary?
Not necessarily.
Traditional CNNs often use max pooling to reduce spatial dimensions, but modern networks can also perform downsampling through convolutions with a stride greater than 1.
For example:
- ResNet uses convolutions with a stride of 2 for downsampling.
- MobileNet uses depthwise convolutions and stride to control spatial dimensions.
- Some classification networks use global average pooling instead of a traditional large fully connected classification head.
Pooling is a common design choice, not a module that every CNN must include.
8. Receptive Field
The receptive field is an important concept for understanding hierarchical feature extraction in CNNs.
8.1 What Is a Receptive Field?
The receptive field indicates which regions of the input data can affect a given neuron in the neural network.
For a feature position in a convolutional network, it usually corresponds to a local region in the input image.
Shallow convolutional layers have smaller receptive fields and mainly focus on local details.
As the number of network layers increases, the receptive field gradually expands, and deep features can integrate spatial information over a larger range.
8.2 How Does the Receptive Field Expand?
Suppose two consecutive 3\times3 convolutional layers are used, both with Stride = 1.
Theoretical receptive field of the first layer:
Second layer:
Therefore, the theoretical receptive field of two consecutive 3\times3 convolutional layers is:
Three consecutive 3\times3 convolutional layers:
Therefore, the theoretical receptive field is:
8.3 Receptive Field Calculation Formula
For a general convolutional network, the theoretical receptive field can be calculated recursively.
Let:
- r_l: theoretical receptive field of layer l.
- j_l: the interval when adjacent positions in the feature map of layer l are mapped back to the input image.
- k_l: convolution kernel size of the current layer.
- s_l: stride of the current layer.
- d_l: dilation rate of the current layer.
Initial conditions:
Recurrence formulas:
These formulas help us calculate the theoretical receptive field after the combined effect of multiple convolution layers, strides, and dilated convolutions.
It is necessary to distinguish:
- Theoretical receptive field: the input coverage range derived from the computational graph structure.
- Effective receptive field: the input region that actually has a strong influence on the output.
The effective receptive field is usually smaller than the theoretical receptive field, and different positions do not influence the output uniformly.
Nine, Important Variants of CNN
CNN does not have only one form, namely ordinary convolution. With the development of computer vision tasks, a variety of convolution structures have emerged to reduce computation, expand the receptive field, or enhance feature representation capability.
9.1 1×1 Convolution
1×1 convolution is also called pointwise convolution.
It does not directly expand the spatial receptive range; instead, it performs a linear combination of different channels at the same spatial position.
Assume the input is:
After passing through C_{\text{out}} 1\times1 convolution kernels:
Common uses:
- Adjust the number of channels.
- Fuse information from different channels.
- Reduce the computation of subsequent convolutions.
- Build bottleneck structures.
9.2 Dilated Convolution
Dilated convolution expands the receptive field by introducing intervals between convolution kernel elements, without necessarily increasing the number of kernel parameters proportionally.
For example, for a 3\times3 convolution kernel:
- Dilation = 1: ordinary convolution.
- Dilation = 2: intervals are added between the sampling positions of the convolution kernel.
- Dilation = 3: the sampling range is further expanded.
Effective kernel size:
Where:
- K: original convolution kernel size.
- D: dilation rate.
For a 3\times3 convolution kernel with Dilation = 2:
Its effective spatial coverage is 5\times5, but the actual number of learnable parameters is still only 9 (one convolution kernel for each input channel).
Common applications:
- Semantic segmentation.
- Scene understanding.
- Vision tasks that require larger contextual information.
9.3 Depthwise Separable Convolution
Depthwise separable convolution splits ordinary convolution into two steps:
- Depthwise Convolution: each input channel performs spatial convolution independently.
- Pointwise Convolution: use 1\times1 convolution to fuse channel information.
Number of parameters in ordinary convolution:
Number of parameters in depthwise separable convolution:
When bias is ignored, the ratio of their parameter counts is:
When the number of channels is large, depthwise separable convolution can usually significantly reduce the number of parameters and the amount of computation.
Representative application: MobileNet.
9.4 Transposed Convolution
Transposed convolution is often used in network structures that need to expand the spatial size of feature maps.
Typical applications:
- Image generation.
- Semantic segmentation decoders.
- Feature map upsampling.
It is not a strict inverse operation of ordinary convolution, nor can it guarantee recovery of the original input information.
In essence, it computes the transpose structure of the linear transformation corresponding to convolution.
When using transposed convolution, pay attention to the relationship among output size, Padding, Stride, and Output Padding.
9.5 Residual Connection
Residual connections are one of the core designs of ResNet.
A standard network aims to learn directly:
The residual structure expresses the target transformation as:
where:
- x: input.
- F(x): the transformation learned by the residual branch.
- F(x)+x: the final output.
Residual connections provide a more direct propagation path for gradients, which helps train deeper networks.
10. Classic CNN Network Architectures
The development of CNNs has gone through several important stages.
10.1 LeNet
LeNet is one of the early classic convolutional neural networks, mainly used for handwritten digit recognition.
Key features:
- Convolutional layers extract local features.
- Pooling layers reduce spatial dimensions.
- Fully connected layers perform classification.
LeNet demonstrates the basic structure of convolutional neural networks for image recognition tasks.
10.2 AlexNet
AlexNet achieved a major breakthrough in the 2012 ImageNet competition, promoting the development of deep learning in computer vision.
Key features:
- Uses a deeper convolutional network.
- Uses the ReLU activation function.
- Uses Dropout to alleviate overfitting.
- Uses GPUs to accelerate training.
- Uses data augmentation to improve generalization.
10.3 VGG
The design characteristic of VGG is to use multiple consecutive 3\times3 convolutional layers to build the network.
Advantages:
- Regular structure.
- The network design is easy to understand.
- Suitable for studying hierarchical feature extraction in convolutional networks.
Disadvantages:
- Large number of parameters.
- High computational cost.
- Higher training cost for deep networks.
10.4 GoogLeNet / Inception
The Inception network uses convolution operations of different sizes in parallel to extract features over different spatial ranges.
Key features:
- Multi-scale feature extraction.
- Uses 1\times1 convolutions for channel compression.
- Reduces computational cost.
- Improves the network's feature representation capability.
10.5 ResNet
ResNet (Residual Network) uses residual connections to address optimization difficulties in training deep networks.
Typical residual block:
Its design allows the network to learn residual transformations based on existing features.
Representative versions:
- ResNet-18
- ResNet-34
- ResNet-50
- ResNet-101
- ResNet-152
10.6 MobileNet
MobileNet targets mobile devices and resource-constrained environments.
Key features:
- Depthwise separable convolution.
- Reduces the number of parameters.
- Lowers computational cost.
- Supports mobile vision tasks.
MobileNet is suitable for application scenarios that focus on inference latency, model size, and computational resources.
10.7 EfficientNet
EfficientNet uses compound scaling to coordinately adjust:
- network depth.
- network width.
- input image resolution.
Its core idea is not merely to increase a single dimension, but to coordinate the relationship between model size and input resolution.
11. Computational Cost and Parameter Count of CNN
To understand the engineering performance of CNNs, it is necessary to distinguish between parameter count and computational cost.
11.1 Parameter Count (Parameters)
Parameter count refers to the number of weights and biases that need to be learned in the model.
Parameter count of a standard convolutional layer:
Parameter count mainly affects:
- Model storage space.
- Model file size.
- State storage required for parameter optimization.
- Part of the resource overhead for training and inference.