I. What Is CUDA?
CUDA (Compute Unified Device Architecture) is NVIDIA's general-purpose GPU computing platform and programming model.
Before CUDA, GPUs were mainly used for graphics rendering, and developers typically used GPUs through graphics APIs.
CUDA opened GPU computing power to general-purpose programs, enabling developers to write GPU parallel programs in languages such as C and C++.
CUDA is not just a programming language, nor a standalone library, but a complete GPU computing ecosystem.
It mainly includes:
- CUDA programming model: defines how GPU threads and computing tasks are organized.
- CUDA C/C++: used to write GPU kernels.
- CUDA Runtime: provides interfaces for device management, memory allocation, and kernel launch.
- CUDA Driver API: provides lower-level GPU control capabilities.
- CUDA Toolkit: includes compilers, development tools, and related libraries.
- CUDA Libraries: provide high-performance computing capabilities such as matrix computation and deep learning.
1. Why Is CUDA Needed?
CPUs and GPUs have different design goals.
CPUs focus more on single-thread performance, complex control logic, and low-latency tasks.
GPUs focus more on throughput, executing similar computing tasks simultaneously through a large number of compute units.
For example, suppose you need to perform the same squaring operation on an array of one million elements:
A CPU can use multiple cores to process in parallel, but the number of cores is relatively limited.
A GPU, by contrast, can launch a large number of threads and have different threads process different elements.
flowchart TD
A["Input one million data elements"] --> B{"Select compute device"}
B -->|"CPU"| C["A few high-performance cores"]
B -->|"GPU"| D["A large number of parallel compute threads"]
C --> E["Process data in batches"]
D --> F["Multiple threads process different elements simultaneously"]
E --> G["Merge computation results"]
F --> G
G --> H["Obtain output array"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2. The Core Value of CUDA
CUDA mainly addresses three problems:
First, it enables GPUs to perform general-purpose computing.
Developers are no longer limited to graphics rendering; they can perform matrix multiplication, image processing, scientific computing, and neural network computation on GPUs.
Second, it provides a unified parallel programming model.
Developers can split tasks into a large number of threads, and the GPU automatically schedules and executes them.
Third, it provides a mature high-performance computing ecosystem.
CUDA provides libraries such as cuBLAS, cuDNN, and NCCL, reducing the work developers need to do to repeatedly implement low-level computations.
II. GPU Hardware Execution Architecture
To understand CUDA, you first need to understand how GPUs organize computing resources.
A GPU is not a processor with a large number of independent CPU cores; rather, it consists of multiple Streaming Multiprocessors (SMs).
Each SM contains multiple compute units, registers, shared memory, and scheduling resources.
1. Main Components of a GPU
A typical NVIDIA GPU includes:
| Component | Function |
|---|---|
| SM | GPU's main computing and scheduling unit |
| CUDA Core | Performs general-purpose computation such as integer and floating-point operations |
| Tensor Core | Accelerates tensor operations such as matrix multiplication |
| Warp Scheduler | Schedules Warp execution |
| Register File | Stores thread-private registers |
| Shared Memory | Provides high-speed storage shared by threads within an SM |
| L1 Cache | Caches locally accessed data |
| L2 Cache | Cache shared by multiple SMs on the GPU |
| Global Memory | Global data storage in GPU memory |
The exact hardware quantities and organization vary depending on GPU architecture and model.
2. What Is an SM?
An SM is the core hardware unit for CUDA parallel execution.
A GPU usually contains multiple SMs; each SM can host multiple thread blocks at the same time and schedule different warps according to resource availability.
During execution, a thread block is handled by a single SM and is not split across multiple SMs for simultaneous execution.
An SM can manage multiple thread blocks at the same time, but how many thread blocks it can host simultaneously depends on:
- The number of registers available to the SM.
- Shared memory capacity.
- The number of threads per thread block.
- The thread and thread block limits of the GPU architecture.
For example, suppose a GPU has 80 SMs and the program launches 1000 thread blocks.
The GPU does not place all thread blocks into hardware execution at once; instead, it schedules them in batches according to the resource conditions of the SMs.
flowchart TD
A["GPU"] --> B["SM 0"]
A --> C["SM 1"]
A --> D["SM 2"]
A --> E["SM ..."]
B --> F["Warp Scheduler"]
C --> G["Warp Scheduler"]
D --> H["Warp Scheduler"]
F --> I["CUDA Core / Tensor Core"]
G --> J["CUDA Core / Tensor Core"]
H --> K["CUDA Core / Tensor Core"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
3. CUDA Core and Tensor Core
CUDA Core is the basic computing unit in a GPU that performs general-purpose computation.
Tensor Core, by contrast, is specially optimized for operations such as matrix multiplication and matrix multiply-add.
In deep learning, a common form of computation is:
or matrix multiply-add:
Tensor Core can execute operations corresponding to the supported data types and matrix shapes with higher throughput.
Commonly supported precisions include:
- FP16
- BF16
- TF32
- FP8
- INT8
- some lower-bit-width integer formats
The specific supported range depends on the GPU architecture.
Note that Tensor Core is not a simple replacement for CUDA Core. Ordinary operators, control logic, and other computations may still be handled by CUDA Core and other execution units.
III. CUDA Programming Model: Thread, Block, and Grid
The most important concept in CUDA is the thread organization model.
Developers write a GPU Kernel, define what work each thread is responsible for, and then organize a large number of threads through Grids and Blocks.
CUDA uses three core concepts:
- Thread: a thread.
- Block: a thread block.
- Grid: a collection of thread blocks.
1. Thread: Thread
Thread is the most basic execution unit in the CUDA programming model.
Each thread has its own:
- Thread ID.
- Registers.
- Local variables.
- Execution state.
For example, when performing an array squaring computation, each thread can be made to process one array element.
__global__ void square(float* input, float* output, int n) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
if (idx < n) {
output[idx] = input[idx] * input[idx];
}
}
Where:
threadIdx.x: the index of the thread within the current Block.blockIdx.x: the index of the Block within the Grid.blockDim.x: the number of threads per Block along the x dimension.
Through the global index idx, a thread determines the position of the data it processes.
2. Block: Thread Block
A Block is a group of threads that can execute cooperatively.
Threads within the same Block can:
- Share data through Shared Memory.
- Coordinate using synchronization primitives.
- Perform cooperative computation through Warp-level instructions.
For example, a Block contains 256 threads:
dim3 block(256);
This means a one-dimensional thread block contains 256 threads.
A Block is the basic organizational unit of GPU scheduling.
A Block is assigned to one SM for execution, and the threads within the Block are not split across multiple SMs.
3. Grid: Grid
A Grid is all the Blocks contained in a single Kernel launch.
Suppose you need to process 10,000 elements, with 256 threads per Block:
Therefore, 40 Blocks need to be launched.
The last Block may have some threads with no corresponding data, so boundary checks are needed to avoid out-of-bounds access.
4. Three-Level Organizational Relationship
flowchart TD
A["Grid"] --> B["Block 0"]
A --> C["Block 1"]
A --> D["Block 2"]
A --> E["Block ..."]
B --> F["Thread 0"]
B --> G["Thread 1"]
B --> H["Thread ..."]
C --> I["Thread 0"]
C --> J["Thread 1"]
C --> K["Thread ..."]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
These can be understood as:
| Level | Meaning | Main Role |
|---|---|---|
| Grid | All thread blocks | Describes the complete computational task |
| Block | A group of threads | Organizes collaboration and data sharing |
| Thread | A single thread | Performs the specific computation |
IV. Warp: The GPU's True Execution Scheduling Unit
After understanding Thread, Block, and Grid, you also need to understand Warp.
Warp is an important unit by which GPU hardware schedules threads.
In NVIDIA CUDA GPUs, a Warp typically contains 32 threads.
Threads in the same Warp use the SIMT (Single Instruction, Multiple Threads) execution model.
This means threads in a Warp typically execute the same instruction but can operate on different data.
1. What Is SIMT?
SIMT is one of the core execution models of CUDA GPUs.
Suppose a Warp has 32 threads, and each thread is responsible for one element in an array.
All threads execute the same square-operation instruction but access different data locations.
Thread 0 -> input[0] -> output[0]
Thread 1 -> input[1] -> output[1]
Thread 2 -> input[2] -> output[2]
...
Thread 31 -> input[31] -> output[31]
This approach enables the GPU to efficiently process large amounts of computations of the same type.
2. Relationship Between Warp and Block
Suppose a Block contains 256 threads:
This Block can be divided into 8 Warps.
If a Block has 128 threads, it contains 4 Warps.
If the number of threads is not divisible by 32, some threads in the last Warp may be inactive.
3. Warp Divergence: Branch Divergence
Warp Divergence is a very important concept in CUDA performance optimization.
Suppose the 32 threads in a Warp execute a conditional:
if (threadIdx.x % 2 == 0) {
result = a + b;
} else {
result = a * b;
}
Half of the threads perform addition, and the other half perform multiplication.
Because these threads belong to the same Warp, the hardware may need to execute the different branches separately and use thread masks to control which threads are active.
This reduces execution efficiency.
Note that modern GPUs have more complex optimization mechanisms for branch execution, and the actual impact depends on the architecture, branch conditions, and code structure.
Common situations that can easily cause performance issues:
- Different threads within the same Warp frequently execute different branches.
- The branches contain complex computations.
- Thread execution paths differ significantly.
Optimization directions:
- Try to make threads in the same Warp follow similar execution paths.
- Reduce unnecessary conditional branches.
- Group data for different computation paths appropriately.
4. Warp Shuffle
Warp Shuffle is a mechanism for threads within a Warp to exchange register data.
For example, one thread can read the register value of another thread in the same Warp.
Common uses:
- Reduction within a Warp.
- Prefix sum computation.
- Data rearrangement.
- Cooperative computation within a Warp.
Compared with using Shared Memory, Warp Shuffle can reduce memory access and synchronization overhead in scenarios where it applies.
V. The CUDA Memory System
A GPU does not have only one kind of storage space, namely video memory.
CUDA provides a multi-level memory system, and different memories differ in access speed, capacity, scope, and lifetime.
Understanding the memory system is the key to writing high-performance CUDA programs.
1. Memory Hierarchy
| Memory Type | Location | Access Scope | Characteristics |
|---|---|---|---|
| Register | Inside the SM | A single thread | Fast, limited capacity |
| Local Memory | GPU memory system | A single thread | Thread-private, usually located in video memory |
| Shared Memory | Inside the SM | The same Block | Shared by threads, fast |
| L1 Cache | Inside the SM | Accessed by threads in the SM | Caches local data |
| L2 Cache | GPU level | Shared by multiple SMs | Caches global data |
| Global Memory | Video memory | All GPU threads | Large capacity, relatively high latency |
| Constant Memory | GPU memory system | Read-only for all threads | Suitable for broadcast access |
| Texture Memory | GPU cache system | GPU threads | Has a specific cache access mechanism |
2. Register: Registers
Registers are thread-private high-speed storage space.
Each thread can use registers to store:
- Intermediate computation results.
- Local variables.
- Indices.
- Temporary values.
Register access is extremely fast, but the total number of registers per SM is limited.
If a Kernel uses a large number of registers per thread, it may reduce the number of Warps that can reside simultaneously in an SM.
This affects the GPU's Occupancy.
3. Shared Memory
Shared Memory is a high-speed storage space that threads within the same Block can access together.
It resides inside the SM and is suitable for holding data that needs to be accessed repeatedly by multiple threads.
For example, in matrix multiplication, part of the matrix data can be loaded into Shared Memory and then reused by multiple threads.
This reduces repeated access to Global Memory.
Example:
__shared__ float tile[16][16];
This declares a shared two-dimensional array within a Block.
Note the following:
- The scope of Shared Memory is the Block.
- Different Blocks cannot directly share the same block of Shared Memory.
- Thread synchronization must be considered before use.
- Shared Memory has limited capacity.
4. Global Memory
Global Memory is usually the GPU's video memory.
It has a large capacity, and all GPU threads can access it through pointers.
For example:
__global__ void add(float* a, float* b, float* c, int n) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
if (idx < n) {
c[idx] = a[idx] + b[idx];
}
}
Here, a, b, and c usually point to data in Global Memory.
Global Memory has higher access latency than registers and Shared Memory, so access efficiency should be improved as much as possible.
5. Memory Access Performance
The CUDA memory system can be understood from three aspects: access latency, capacity, and sharing scope.
flowchart TD
A["CUDA Memory Hierarchy"] --> B["Thread-Private Storage"]
A --> C["Block-Shared Storage"]
A --> D["GPU Global Storage"]
B --> E["Register"]
B --> F["Local Memory"]
C --> G["Shared Memory"]
C --> H["L1 Cache"]
D --> I["L2 Cache"]
D --> J["Global Memory"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
Note that although Local Memory has “Local” in its name, it is not high-speed on-chip storage within an SM. It usually resides in the device memory hierarchy, may be cached, and its access cost is often significantly higher than that of registers.
When register pressure is too high, the compiler may place some variables in Local Memory; this situation is called Register Spilling.
6. CUDA Kernel Execution Mechanism
A Kernel is a function in CUDA that runs on the GPU.
Developers declare a Kernel using __global__ and invoke it with a special launch syntax.
1. Kernel Function
A simple vector addition:
__global__ void vector_add(
float* a,
float* b,
float* c,
int n
) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
if (idx < n) {
c[idx] = a[idx] + b[idx];
}
}
Here:
__global__: indicates that this is a GPU Kernel.a,b: input arrays.c: output array.idx: the data index handled by the current thread.if (idx < n): prevents threads from accessing locations outside the array bounds.
2. Kernel Launch
CUDA uses the following syntax to launch a Kernel:
vector_add<<<gridSize, blockSize>>>(a, b, c, n);
Where:
gridSize: the number of Blocks in the Grid.blockSize: the number of threads per Block.
For example:
int n = 10000;
int blockSize = 256;
int gridSize = (n + blockSize - 1) / blockSize;
vector_add<<<gridSize, blockSize>>>(a, b, c, n);
The calculation gives:
The GPU schedules threads to execute the Kernel according to the organization of the Grid and Blocks.
3. Kernel Execution Process
flowchart TD
A["CPU prepares input data"] --> B["Allocate GPU memory"]
B --> C["Copy input data to GPU"]
C --> D["Configure Grid and Block"]
D --> E["Launch CUDA Kernel"]
E --> F["GPU schedules thread blocks to SMs"]
F --> G["Warp schedules threads to execute instructions"]
G --> H["Read and write GPU memory and complete computation"]
H --> I["Check execution results"]
I --> J["Copy results back to CPU"]
J --> K["Release GPU resources"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
There is an important concept here: kernel launches are usually asynchronous.
After the CPU submits a kernel launch request, it does not necessarily wait for the GPU to finish the computation before continuing to execute subsequent code.
If you need to ensure the computation is complete, you can use a synchronization operation.
For example:
cudaDeviceSynchronize();
In actual development, you also need to check for kernel launch errors and asynchronous execution errors.
7. CUDA Thread Synchronization Mechanisms
Many threads on the GPU may access shared data at the same time.
If there are data dependencies between multiple threads, synchronization mechanisms are needed to ensure execution order and data correctness.
1. __syncthreads()
__syncthreads() is a block-level synchronization primitive.
It causes threads participating in execution within the same block to wait at a specified point until all threads in the block have reached the synchronization point.
For example:
__global__ void shared_example(float* input, float* output) {
__shared__ float cache[256];
int tid = threadIdx.x;
int idx = blockIdx.x * blockDim.x + tid;
cache[tid] = input[idx];
__syncthreads();
output[idx] = cache[tid] * 2.0f;
}
Here, data is first written to shared memory, then synchronization ensures the data is ready, and finally the computation is performed.
Note the following:
- It can only synchronize threads within the same block.
- Different blocks cannot directly rely on
__syncthreads()for synchronization. - Threads in the same block must reach the synchronization point with correct control flow.
- If some threads skip the synchronization point, it may lead to incorrect program behavior or deadlock.
2. Atomic Operations
When multiple threads modify the same variable at the same time, a data race may occur.
For example:
counter += 1;
If multiple threads perform this operation simultaneously, updates may be lost.
CUDA provides atomic operations:
atomicAdd(&counter, 1);
It can guarantee that the addition operation at the specified memory location is atomic.
Common atomic operations:
atomicAddatomicSubatomicExchatomicMinatomicMaxatomicCAS
Atomic operations can ensure correctness, but when many threads contend for the same address, they can also become a performance bottleneck.
3. Stream: CUDA Streams
A CUDA stream is an ordered queue of GPU operations.
Operations in the same stream are executed in the order they are submitted.
Operations in different streams can execute concurrently when dependencies and hardware conditions permit.
Common uses:
- Overlapping data transfer and computation.
- Executing multiple independent kernels in parallel.
- Multi-request task scheduling.
- Improving GPU utilization.
For example:
cudaStream_t stream1;
cudaStream_t stream2;
cudaStreamCreate(&stream1);
cudaStreamCreate(&stream2);
kernel_a<<<grid, block, 0, stream1>>>();
kernel_b<<<grid, block, 0, stream2>>>();
These two kernels are submitted to different streams, but whether they actually execute concurrently still depends on GPU resources, kernel characteristics, and dependencies.
8. Key Optimization Techniques for CUDA Parallel Computing
Writing a CUDA program is not just about splitting a task into a large number of threads; it also requires threads, memory, and compute resources to work together efficiently.
1. Coalesced Memory Access: Coalescing Memory Accesses
The GPU memory system tries to coalesce memory accesses from multiple threads into fewer memory transactions.
Suppose a warp contains 32 threads, and each thread reads contiguous 4-byte data:
Thread 0 -> data[0]
Thread 1 -> data[1]
Thread 2 -> data[2]
...
Thread 31 -> data[31]
This kind of contiguous access is usually conducive to memory transaction coalescing.
If the addresses accessed by threads are highly scattered, more memory transactions may be required, reducing bandwidth utilization.
Therefore, contiguous access and a reasonable data layout are very important.
2. Shared Memory Tiling
Matrix multiplication requires repeatedly reading data from matrices.
If every thread reads data directly from global memory, a large number of duplicate accesses may occur.
Tiling splits the matrix into small tiles, first loads them into shared memory, and then allows multiple threads within a block to reuse them.
For example:
The matrix can be split into multiple submatrices:
This approach improves data reuse and reduces global memory accesses.
3. Occupancy: Occupancy Rate
Occupancy is the ratio of the number of warps actually resident on an SM to the maximum number of resident warps allowed by the hardware.
Formula:
Factors affecting occupancy include:
- The number of registers used by each thread.
- The shared memory used by each block.
- The number of threads per block.
- SM hardware resource limits.
Increasing occupancy helps hide memory access latency, but it does not mean that higher occupancy necessarily leads to better performance.
If increasing occupancy reduces data reuse or causes more redundant computation, final performance may actually decrease.
4. Kernel Fusion: Operator Fusion
Suppose a computational task consists of three operations:
If multiplication and addition are executed separately, multiple kernels may need to be launched, and intermediate results may be read and written multiple times.
Kernel Fusion combines multiple operations into a single kernel for execution.
flowchart TD
A["Input data x"] --> B["Execute multiplication kernel separately"]
B --> C["Write intermediate result"]
C --> D["Launch addition kernel"]
D --> E["Write final result"]
A --> F["Fused kernel"]
F --> G["Execute multiplication and addition once"]
G --> H["Write final result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F