Back to series
· 9 min read

Why GPUs Are Suited for Deep Learning: From Parallel Computing to CUDA Core Architecture


1. Why Does Deep Learning Need GPUs?

The core computation of deep learning models is mainly concentrated on a large number of numerical operations, such as:

  • Matrix Multiplication
  • Vector Operations
  • Convolution
  • Tensor Operations
  • Activation function computation
  • Gradient computation and parameter updates

Take a simple fully connected layer as an example:

Y=XW+b

Assume:

  • The input matrix X has shape B\times D
  • The weight matrix W has shape D\times H
  • The bias vector b has shape H

Then the output matrix Y has shape:

Y\in \mathbb{R}^{B\times H}

Each output element requires a multiply-add computation of length D.

As the batch size, input dimension, and hidden layer dimension continue to increase, the amount of computation grows rapidly.

For matrix multiplication:

C=A\times B

If:

A\in \mathbb{R}^{M\times K}
B\in \mathbb{R}^{K\times N}

Then the computational complexity is approximately:

O(MKN)

A large amount of computation in deep learning can be converted into similar matrix and tensor operations.

The advantage of GPUs is that they can execute a large number of mutually independent or weakly dependent computational tasks simultaneously.

2. Architectural Differences Between CPUs and GPUs

CPUs and GPUs do not simply differ in performance level; they are processors designed for different computational tasks.

1. CPU: Designed for General-Purpose Computing and Complex Control

The CPU (Central Processing Unit) is mainly responsible for general-purpose computing.

Typical characteristics:

  • Relatively few cores.
  • Each core has strong computing capability.
  • Good at complex control logic and branch judgment.
  • Has a large cache hierarchy.
  • Good at low-latency, serial, or complex dependency tasks.

Typical CPU work includes:

  • Operating system scheduling.
  • Program control flow.
  • File and network I/O.
  • Data preprocessing.
  • Complex logical judgment.
  • Task scheduling and resource management.

2. GPU: Designed for Large-Scale Parallel Computing

The GPU (Graphics Processing Unit) was originally mainly used for graphics rendering.

Graphics rendering requires processing large amounts of vertex, pixel, and texture data simultaneously, and such tasks are highly parallel.

GPUs gradually developed architectures suitable for general-purpose parallel computing.

Typical characteristics:

  • Has a large number of compute units.
  • Can schedule a large number of threads simultaneously.
  • Suitable for executing the same or similar computational operations.
  • Has high memory bandwidth.
  • Provides specialized hardware for operations such as matrix multiplication.

3. CPU vs. GPU Comparison

Comparison Dimension CPU GPU
Number of cores Relatively few Usually a large number of compute cores
Single-core capability Relatively strong Usually emphasizes overall throughput more
Design goal General-purpose computing, low latency Large-scale parallelism, high throughput
Control logic Good at complex branching Better suited to regular parallel tasks
Cache hierarchy Usually larger Emphasizes parallel computing and data throughput more
Memory System memory Dedicated video memory or unified memory architecture
Tasks it excels at System scheduling, logic processing Matrix computation, graphics rendering, tensor operations

The design philosophies of the two can be summarized as:

flowchart TB
    A["Computation task"] --> B{"Task type"}

    B -->|"Complex control, branch decisions"| C["CPU"]
    B -->|"Many rules, repetitive computations"| D["GPU"]

    C --> E["A few high-performance cores"]
    E --> F["Low-latency general-purpose computing"]

    D --> G["Many parallel computing units"]
    G --> H["High-throughput parallel computing"]

    F --> I["Suitable for general-purpose programs"]
    H --> J["Suitable for deep learning computations"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

Note: Modern CPUs also have multiple cores, SIMD vector instructions, and parallel computing capabilities, and modern GPUs also have complex scheduling and caching mechanisms. The difference between the two lies in hardware resource allocation and design focus, not in the idea that CPUs can only execute serially and GPUs can only execute in parallel.

III. Why Are GPUs Good at Parallel Computing?

The key to understanding a GPU is understanding how it organizes and executes a large number of computational tasks.

1. Data Parallelism

Many computations in deep learning exhibit data parallelism.

For example, applying ReLU to a tensor containing 1 million elements:

ReLU(x)=\max(0,x)

The computation for each element can be completed independently:

y_i=\max(0,x_i)

Computing y_1 does not need to wait for y_2; there is no data dependency between different elements.

Therefore, these computational tasks can be assigned to different GPU threads for parallel execution.

flowchart TB
    A["Input tensor"] --> B["Split into multiple data elements"]

    B --> C["Thread 1: compute x1"]
    B --> D["Thread 2: compute x2"]
    B --> E["Thread 3: compute x3"]
    B --> F["Thread N: compute xN"]

    C --> G["Output tensor"]
    D --> G
    E --> G
    F --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49

Data parallelism is well suited to:

  • Matrix multiplication.
  • Convolution computation.
  • Tensor addition and subtraction.
  • Activation functions.
  • Batch normalization.
  • Some loss function computations.

2. SIMT Execution Model

A common GPU programming execution model is SIMT (Single Instruction, Multiple Threads).

Its core idea is:

Multiple threads execute the same instruction logic but process different data.

For example:

y[i] = x[i] * 2

A large number of threads can execute this logic simultaneously, with each thread processing a different index.

In NVIDIA GPUs, threads are typically organized and scheduled in warps.

A warp usually contains 32 threads.

Threads in the same warp execute according to a unified instruction stream but have their own thread state and data.

Note:

  • SIMT is not equivalent to SIMD.
  • SIMD uses a single instruction to operate on multiple data elements.
  • SIMT organizes parallel execution from the perspective of multiple threads.
  • GPU hardware can achieve efficient computation through mechanisms such as vectorization and grouped execution.

3. Warp and Thread Scheduling

In NVIDIA GPUs, a warp is an important scheduling unit.

A warp usually contains 32 threads, and threads are organized and executed on Streaming Multiprocessors (SMs).

When a Warp is waiting for data due to memory access or other reasons, the SM can switch to executing other ready Warps.

This mechanism helps hide memory access latency.

For example:

flowchart TB
    A["GPU"] --> B["SM 0"]
    A --> C["SM 1"]
    A --> D["SM N"]

    B --> E["Warp 0"]
    B --> F["Warp 1"]
    B --> G["Warp N"]

    E --> H["Thread 0 ... Thread 31"]
    F --> I["Thread 0 ... Thread 31"]
    G --> J["Thread 0 ... Thread 31"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

There are three levels to understand here:

Level Meaning
GPU The entire graphics processor
SM A streaming multiprocessor inside the GPU
Warp A scheduling and execution unit composed of multiple threads

The number of Warps, the number of SMs, and the resource configuration per SM depend on the GPU architecture.

4. Branch Divergence: Limitations of GPU Parallel Computing

GPUs cannot efficiently parallelize all computational tasks.

If threads in the same Warp execute different branches, branch divergence may occur.

For example:

if x[i] > 0:
    y[i] = x[i] * 2
else:
    y[i] = x[i] + 1

If some threads in the same Warp enter if and others enter else, the hardware may need to execute different paths in stages and mask threads that do not participate in the current path.

This reduces parallel execution efficiency.

Therefore, GPUs are usually better suited for:

  • Large amounts of repetitive computation.
  • Regular data access patterns.
  • Few dependencies between threads.
  • Tasks with relatively simple branching logic.

IV. Why Is Matrix Multiplication a Core Advantage of GPUs?

Matrix multiplication is one of the most important computational operations in deep learning.

Fully connected layers, attention mechanisms, Embedding projections, and many other modules in neural networks all involve matrix multiplication.

1. The Computation Process of Matrix Multiplication

Given:

C=A\times B

Where:

A\in \mathbb{R}^{M\times K}
B\in \mathbb{R}^{K\times N}

Output:

C\in \mathbb{R}^{M\times N}

Each output element:

C_{ij}=\sum_{k=1}^{K}A_{ik}B_{kj}

The total number of multiply-add operations is approximately:

2MKN

Here, the FLOPs convention counts one multiplication and one addition separately.

When the matrix scale is large, the computational cost is very substantial.

2. How Does the GPU Accelerate Matrix Multiplication?

The GPU splits matrix computation into a large number of small-scale computational tasks.

These tasks can be assigned to different thread blocks and compute units.

A typical process includes:

  1. Divide the matrix into multiple tiles.
  2. Load the required data into faster on-chip memory.
  3. Perform local matrix multiply-add.
  4. Accumulate intermediate results.
  5. Write the final results back to GPU memory.

This approach improves data reuse and reduces repeated accesses to GPU memory.

3. Tensor Core: Hardware Specifically Designed to Accelerate Matrix Computation

In modern NVIDIA GPUs, Tensor Core is a hardware unit specifically used to accelerate matrix operations.

It has a different focus from traditional CUDA Cores.

Hardware Unit Primary Responsibility
CUDA Core General scalar, vector, integer, and floating-point operations
Tensor Core High-throughput matrix multiply-add operations
SM Organizes and schedules threads and contains various compute and storage resources

Tensor Core can efficiently perform matrix multiply-add operations:

D=A\times B+C

It supports multiple data precisions, with the exact range depending on the GPU architecture, for example:

  • FP16
  • BF16
  • TF32
  • FP8
  • INT8
  • Other lower-precision formats

Not all GPUs support all of the above formats.

The value of Tensor Cores lies in offloading the matrix operations that occur frequently in deep learning to dedicated hardware, thereby achieving higher computational throughput.

V. Why Is VRAM Bandwidth So Important?

GPUs have strong compute capability, but compute units need a continuous supply of data.

If data cannot be read from memory in time, compute units may sit idle waiting.

Therefore, VRAM bandwidth is an important indicator of GPU performance.

1. VRAM and System Memory

GPUs usually have dedicated VRAM, such as GDDR or HBM.

Type Primary use
System memory (RAM) CPU runs programs and stores ordinary data
GPU memory (VRAM) Stores data, model parameters, and intermediate results required for GPU computation
HBM High-bandwidth memory, commonly used in high-performance computing and data-center GPUs

VRAM capacity determines how much data the GPU can hold.

VRAM bandwidth determines how much data can be transferred per unit time.

These two metrics should not be confused.

2. VRAM Bandwidth

VRAM bandwidth is usually expressed in GB/s or TB/s.

Theoretical bandwidth can be approximated as:

Bandwidth=\text{data transfer rate}\times \text{bus width}

In actual calculations, the bus width must be converted from bits to bytes.

For example, assume:

  • The VRAM data rate is 20 Gbps.
  • The bus width is 256 bits.

The theoretical bandwidth is:

20\times 256\div 8=640\ GB/s

Actual available bandwidth is usually lower than the theoretical value, depending on access patterns, cache hit rates, and hardware implementation.

3. Compute Density and the Roofline Model

To determine whether a computational task is limited by compute capability or by memory bandwidth, the Roofline Model can be used.

One core metric is Arithmetic Intensity:

AI=\frac{FLOPs}{Bytes}

It represents how many floating-point operations are performed for each Byte of data read or written.

The upper performance limit can be approximated as:

Performance\leq \min(PeakCompute,\ Bandwidth\times AI)

where:

  • PeakCompute: the theoretical peak compute capability of the hardware.
  • Bandwidth: memory bandwidth.
  • AI: arithmetic intensity.

This allows us to distinguish two types of tasks:

Compute-bound

The amount of computation is relatively large, and performance is mainly limited by GPU compute unit performance.

Memory-bound

The amount of data transfer is relatively large, and performance is mainly limited by VRAM bandwidth or memory access efficiency.

This is also why actual GPU performance cannot be judged by TFLOPS alone.

4. Why Can't Small Models Always Fully Utilize the GPU?

There is overhead in launching compute tasks, allocating threads, and transferring data on the GPU.

If the computational task is too small:

  • GPU compute units may not be fully utilized.
  • Kernel launch overhead may account for a high proportion.
  • Data transfer between the CPU and GPU may become a bottleneck.
  • Data preparation and synchronization time may exceed actual computation time.

Therefore, the GPU is not the best choice for every program.

For many small-scale, low-parallelism tasks, the CPU may be more suitable.

VI. How Does the GPU Accelerate Neural Network Training?

Deep learning training includes not only forward propagation but also backpropagation and parameter updates.

The GPU can process many of the matrix and tensor operations in parallel.

1. Forward Propagation

Take the fully connected layer from earlier as an example:

Y=XW+b

The GPU can perform matrix multiplication and bias addition in parallel.

2. Loss Computation

Assume mean squared error is used:

L=\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat y_i)^2

The errors for different samples can be computed in parallel, and the final loss is then obtained through a reduction operation.

3. Backpropagation

The core of backpropagation is the chain rule.

Take a linear layer as an example:

Y=XW+b

Assume the gradient of the loss with respect to the output is:

G=\frac{\partial L}{\partial Y}

Then:

\frac{\partial L}{\partial W}=X^TG
\frac{\partial L}{\partial X}=GW^T
\frac{\partial L}{\partial b}=\sum_{i=1}^{B}G_i

These gradient computations also involve many matrix multiplications and reduction operations, making them well suited to GPU acceleration.

4. Parameter Update

Take SGD as an example:

W_{t+1}=W_t-\eta \frac{\partial L}{\partial W_t}

where:

  • W_t: current model parameters.
  • \eta: learning rate.
  • \frac{\partial L}{\partial W_t}: current gradient.

Parameter updates are usually also performed by tensor operations on the GPU.

5. Complete Training Process

flowchart TB
    A["Training data"] --> B["Load data to GPU"]
    B --> C["Forward propagation"]
    C --> D["Compute loss"]
    D --> E["Backpropagation"]
    E --> F["Compute parameter gradients"]
    F --> G["Optimizer updates parameters"]
    G --> H{"Training complete?"}

    H -->|"No"| C
    H -->|"Yes"| I["Obtain the trained model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The focus of GPU acceleration is not just forward propagation, but large-scale tensor computation throughout the entire training process.

VII. CUDA: The Key Technology for General-Purpose GPU Computing

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model introduced by NVIDIA.

It enables developers to use NVIDIA GPUs to perform general-purpose computing tasks, rather than being limited to graphics rendering.

1. The Role of CUDA

CUDA provides:

  • GPU programming model.
  • GPU thread organization and scheduling mechanisms.
  • GPU memory management.
  • Kernel authoring and execution.
  • GPU math libraries.
  • Runtime interfaces for coordinated execution with the CPU.

Deep learning frameworks can use CUDA to invoke GPUs to perform computations.

2. CUDA Kernel

A kernel is a function that runs on the GPU.

For example, a simple vector addition kernel:

__global__ void vector_add(
    const float* A,
    const float* B,
    float* C,
    int N
) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;

    if (i < N) {
        C[i] = A[i] + B[i];
    }
}

Where: