Back to series
· 10 min read

Why Does a CPU Need a Cache?


1. The Essence of CPU Instruction Execution

The core task of the CPU (Central Processing Unit) is to execute machine instructions.

High-level language code written by programmers in C, Java, Python, and other languages must ultimately be converted, through compilation, interpretation, or runtime execution mechanisms, into operations that the processor can execute.

Take C as an example:

int a = 10;
int b = 20;
int c = a + b;

This code expresses three basic operations:

  1. Define variable a.
  2. Define variable b.
  3. Add a and b, and store the result in c.

But the CPU does not directly understand int, +, or variable names.

At the machine level, these operations are represented as a series of machine instructions.

For example, a hypothetical instruction sequence:

MOV R1, #10
MOV R2, #20
ADD R3, R1, R2
STORE [ADDR], R3

Where:

  • MOV: data transfer.
  • ADD: integer addition.
  • STORE: write data to memory.
  • R1, R2, R3: general-purpose registers.
  • #10, #20: immediate values.
  • [ADDR]: memory address operand.

The above is pseudocode assembly for ease of understanding and does not correspond to the complete instruction encoding of any real processor.

The process by which the CPU executes instructions is essentially:

Fetch the instruction, decode the instruction, read the operands, perform the operation, and save the result.

This process is usually called the instruction cycle.

2. Basic Stages of Instruction Execution

Traditional computer organization principles usually divide the execution of an instruction into several stages:

  1. Fetch.
  2. Decode.
  3. Execute.
  4. Memory access.
  5. Write back.

Different instructions do not necessarily go through all stages. For example, integer addition between registers usually does not need to access data memory, whereas load instructions require memory access.

2.1 Instruction Execution Flow

flowchart TD
    A["取指 Fetch"] --> B["译码 Decode"]
    B --> C["读取操作数"]
    C --> D["执行 Execute"]
    D --> E{"是否需要访存?"}
    E -->|是| F["访存 Memory Access"]
    E -->|否| G["写回 Write Back"]
    F --> G
    G --> H["更新处理器状态"]
    H --> I["继续执行后续指令"]

    classDef stage fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef decision fill:#FFF4E5,stroke:#D99A35,color:#704510
    classDef result fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A,B,C,D,F,G stage
    class E decision
    class H,I result

This is the traditional abstract model of instruction execution. Modern CPUs usually further split instructions into micro-operations and process multiple instructions in parallel through mechanisms such as pipelining and out-of-order execution.

3. The First Stage: Fetch

Fetch is the first stage of CPU instruction execution.

The CPU needs to know where the next instruction to be executed is located.

This location is recorded by the program counter (PC).

3.1 What Is the PC?

The PC is a register in the CPU used to hold the address of the next instruction to be fetched.

In different processor architectures, the exact semantics of the PC and when it is updated may differ.

For example:

PC = 0x1000

This indicates that the processor currently needs to fetch an instruction from the corresponding address.

The basic process of the fetch stage:

  1. Determine the instruction address according to the PC.
  2. Fetch the instruction from the instruction cache or a lower storage level.
  3. Send the instruction to subsequent processing logic.
  4. Update the fetch address, usually continuing to fetch the next sequential instruction.

3.2 How Does the PC Change?

Suppose the program instructions are located at consecutive addresses:

Instruction Address Instruction
0x1000 MOV R1, #10
0x1004 MOV R2, #20
0x1008 ADD R3, R1, R2
0x100C STORE [ADDR], R3

If each instruction is fixed at 4 bytes in length, then during sequential execution, the PC usually increments by 4.

flowchart TD
    A["PC = 0x1000"] --> B["Fetch MOV R1, #10"]
    B --> C["PC = 0x1004"]
    C --> D["Fetch MOV R2, #20"]
    D --> E["PC = 0x1008"]
    E --> F["Fetch ADD R3, R1, R2"]
    F --> G["PC = 0x100C"]
    G --> H["Fetch STORE instruction"]

    classDef address fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef fetch fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A,C,E,G address
    class B,D,F,H fetch

Note:

Not all CPUs have instructions that are 4 bytes long.

For example:

  • x86/x86-64 instruction length is variable.
  • ARM A32 instructions are usually 4 bytes.
  • ARM Thumb instructions can use 16-bit or 32-bit encodings.
  • AArch64 instructions are usually 4 bytes.

Therefore, how the PC is updated depends on the specific instruction architecture.

3.3 Where Are Instructions Fetched From?

The CPU does not always access main memory directly to fetch instructions.

Modern CPUs usually have an instruction cache (Instruction Cache, I-Cache).

The instruction fetch logic preferentially fetches instructions from the I-Cache.

If there is an I-Cache miss, the corresponding data must be fetched from a lower-level cache or main memory.

flowchart TD
    A["PC provides instruction address"] --> B["Query L1 I-Cache"]
    B --> C{"Cache hit?"}
    C -->|Yes| D["Fetch instruction"]
    C -->|No| E["Query lower-level cache"]
    E --> F{"Hit?"}
    F -->|Yes| G["Fetch instruction cache line"]
    F -->|No| H["Access main memory"]
    H --> I["Load instruction cache line"]
    G --> D
    I --> D
    D --> J["Send to instruction processing front end"]

    classDef access fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef decision fill:#FFF4E5,stroke:#D99A35,color:#704510
    classDef result fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A,B,E,H,I access
    class C,F decision
    class D,G,J result

3.4 Instruction Prefetch

To reduce waiting, modern CPUs also use prefetching mechanisms.

The prefetcher tries to fetch instruction data that may be executed soon in advance, so that instructions are already in the cache or front-end buffer structures when they are actually needed.

Prefetching is usually based on program execution patterns, such as sequential instruction fetch and branch prediction results.

Prefetching is not guaranteed to be accurate every time. If prefetched data is not used, it may waste cache space and memory bandwidth.

4. Stage Two: Decode

The fetch stage obtains machine instruction encodings.

The CPU also needs to parse these encodings to determine what operation is actually to be performed.

This process is called instruction decoding.

4.1 What Is a Machine Instruction?

A machine instruction is a binary encoding defined by the processor's instruction set architecture (ISA).

For example, an instruction may represent:

ADD R3, R1, R2

Its semantics are:

Add the values in registers R1 and R2, and write the result to R3.

However, a real machine instruction does not directly store the string ADD R3, R1, R2; instead, it uses binary bits to represent information such as the opcode, register numbers, and addressing mode.

4.2 What Information Does an Instruction Usually Contain?

A machine instruction may contain the following information:

Field Purpose
Opcode Opcode, indicating what operation to perform
Source Operand Source operand
Destination Operand Destination operand
Addressing Mode Addressing mode
Immediate Immediate value
Condition Information related to conditional execution or condition evaluation

The specific fields are determined by the ISA and instruction format; not all instructions contain all fields.

4.3 What Does the Decoder Do?

The decoder identifies the instruction type and operand information from the machine instruction encoding and generates the control information required for subsequent execution.

For example:

ADD R3, R1, R2

After decoding, the processor needs to determine:

  • Operation type: integer addition.
  • Source operands: R1, R2.
  • Destination register: R3.
  • Execution resource: integer arithmetic unit.
  • Result handling: write the addition result to the destination register.
flowchart TD
    A["Machine instruction encoding"] --> B["Instruction decoder"]
    B --> C["Identify opcode"]
    B --> D["Parse source operand"]
    B --> E["Parse destination operand"]
    B --> F["Parse addressing mode"]
    C --> G["Generate control information"]
    D --> G
    E --> G
    F --> G
    G --> H["Enter subsequent execution flow"]

    classDef input fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef decode fill:#FFF4E5,stroke:#D99A35,color:#704510
    classDef output fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A input
    class B,C,D,E,F,G decode
    class H output

4.4 x86 Instructions and Micro-operations (μops)

Modern x86 CPUs have complex instruction sets, and some instructions may contain multiple operations.

To improve execution efficiency, many processors decode complex x86 instructions into simpler micro-operations (Micro-operations, μops).

For example, a complex instruction with a memory operand may internally be split into multiple micro-operations such as load, compute, and store.

flowchart TD
    A["x86 machine instruction"] --> B["Instruction decoding"]
    B --> C["Generate one or more μops"]
    C --> D["Scheduling and execution"]
    D --> E["Execution unit"]
    E --> F["Result handling"]

    classDef instruction fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef decode fill:#FFF4E5,stroke:#D99A35,color:#704510
    classDef execute fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A instruction
    class B,C decode
    class D,E,F execute

Note:

  • Not all x86 instructions are split into multiple μops.
  • Some simple instructions can map directly to a single micro-operation.
  • Some processors also support micro-operation fusion (Micro-op Fusion).
  • The decoding and micro-operation implementations of different CPU microarchitectures may differ.

The ISA defines the software-visible instruction semantics, while μops are an internal implementation detail of a specific processor microarchitecture.

5. Stage Three: Reading Operands

After decoding is complete, the CPU needs to obtain the data required to execute the instruction.

Operands may come from:

  • Registers.
  • Immediate values in the instruction.
  • Memory.

5.1 Register Operands

Registers are high-speed storage units inside the CPU.

For example:

ADD R3, R1, R2

The CPU needs to read the values of R1 and R2, then perform the addition.

flowchart TD
    A["Read instruction ADD R3, R1, R2"] --> B["Read register R1"]
    A --> C["Read register R2"]
    B --> D["Obtain operand A"]
    C --> E["Obtain operand B"]
    D --> F["Send to integer execution unit"]
    E --> F

    classDef source fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef result fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A,B,C source
    class D,E,F result

Register access is usually much faster than accessing main memory, so compilers make use of registers as efficiently as possible.

5.2 Immediate Operands

An immediate is a constant directly encoded in the instruction.

For example:

MOV R1, #10

Here, 10 is the immediate.

The CPU does not need to access memory separately to read this constant; instead, it parses it from the instruction encoding.

5.3 Memory Operands

Some instructions need to access memory.

For example:

LOAD R1, [0x1000]

This means reading data from the specified memory address into R1.

A real processor may use an Address Generation Unit (AGU) to calculate the effective address, and then use a Load/Store Unit (LSU) to complete the memory access.

flowchart TD
    A["Memory access instruction"] --> B["Decode address expression"]
    B --> C["Address Generation Unit (AGU)"]
    C --> D["Calculate effective address"]
    D --> E["Access data cache"]
    E --> F{"L1 D-Cache hit?"}
    F -->|Yes| G["Read data"]
    F -->|No| H["Access lower-level cache or main memory"]
    H --> G
    G --> I["Send data to execution flow"]

    classDef address fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef decision fill:#FFF4E5,stroke:#D99A35,color:#704510
    classDef result fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A,B,C,D,E,H address
    class F decision
    class G,I result

Note:

A memory operand does not mean the CPU accesses main memory every time. The data may already be present in L1, L2, or L3 cache.

VI. Stage Four: Execute

The execute stage is the stage where the instruction operation is actually completed.

The CPU selects the corresponding execution unit based on the instruction type.

Modern CPUs usually have multiple execution units with different functions.

6.1 Integer Arithmetic Unit (ALU)

The ALU (Arithmetic Logic Unit) is responsible for integer arithmetic and logical operations.

Common operations include:

  • Addition.
  • Subtraction.
  • AND operation.
  • OR operation.
  • XOR operation.
  • Comparison.
  • Shift.

For example:

ADD R3, R1, R2

The execution logic can be abstracted as:

R3=R1+R2

6.2 Floating-Point Unit (FPU)

The FPU (Floating Point Unit) is used to perform floating-point operations.

For example:

float a = 1.5f;
float b = 2.5f;
float c = a * b;

Floating-point operations generally follow floating-point standards such as IEEE 754, but the specific instruction support and execution details depend on the processor architecture.

6.3 SIMD Execution Unit

SIMD (Single Instruction Multiple Data) allows a single instruction to perform the same type of operation on multiple data elements.

For example:

A = [1, 2, 3, 4]
B = [5, 6, 7, 8]

A + B = [6, 8, 10, 12]

On processors that support the corresponding SIMD instructions, vector registers and vector execution units can be used to process multiple data elements in parallel.

Common technologies include:

  • x86 SSE.
  • x86 AVX.
  • ARM NEON.
  • ARM SVE.

6.4 Branch Execution Unit

Branch instructions change the program execution order.

For example:

if (a > b) {
    c = a;
} else {
    c = b;
}

The CPU needs to determine whether the condition holds and determine which instruction should be executed next.

Modern processors usually use branch prediction mechanisms to predict the branch direction in advance and reduce pipeline stalls.

6.5 Relationship Between Different Execution Units

flowchart TD
    A["Instruction scheduler"] --> B["Integer ALU"]
    A --> C["Floating-point FPU"]
    A --> D["SIMD vector unit"]
    A --> E["Load/Store Unit (LSU)"]
    A --> F["Branch execution unit"]

    B --> G["Execution result"]
    C --> G
    D --> G
    E --> G
    F --> G

    classDef dispatch fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef unit fill:#FFF4E5,stroke:#D99A35,color:#704510
    classDef result fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A dispatch
    class B,C,D,E,F unit
    class G result

Different processors have different numbers of execution ports and execution units, and instructions of the same type may also be handled by multiple execution units.

VII. Stage Five: Memory Access

The memory access stage mainly targets instructions that need to read from or write to memory.

For example:

LOAD R1, [0x1000]
STORE [0x2000], R1

The first instruction reads data from memory.

The second instruction writes register data to memory.

7.1 Load Operation

Load means reading data from memory.

LOAD R1, [ADDR]

Basic process:

  1. Calculate the effective address.
  2. Look up the data cache.
  3. If the cache hits, fetch the data.
  4. If the cache misses, request the data from lower-level storage.
  5. Provide the read result to subsequent execution logic.

7.2 Store Operation

Store means writing data to memory.

STORE [ADDR], R1

The CPU needs to:

  1. Calculate the effective address.
  2. Obtain the data to be written.
  3. Perform cache write and coherence-related operations.
  4. Handle lower-level storage updates according to the cache policy.

Modern processors usually have structures such as a Store Buffer to temporarily hold write operations that have not yet completed lower-level storage processing.

Therefore, instruction execution completion and final data write to DRAM do not necessarily occur at the same moment.

8. Sixth Stage: Write Back

The write-back stage is responsible for saving the execution result to the target location.

For example:

ADD R3, R1, R2

After execution completes, the result needs to be written to R3.

If it is a load instruction:

LOAD R1, [ADDR]

then the data read needs to be written to R1.

If it is a store instruction, the process of writing data to the cache or storage system is handled by the corresponding memory access mechanism, and it does not necessarily go through the register write-back stage in the traditional sense.

8.1 Instruction Execution Result

flowchart TD
    A["Execution unit completes operation"] --> B["Produces result"]
    B --> C{"What is the target location?"}
    C -->|Register| D["Write to target register"]
    C -->|Memory| E["Enter Store processing flow"]
    C -->|Status flag| F["Update relevant status"]
    D --> G["Complete corresponding result processing"]
    E --> G
    F --> G

    classDef execute fill:#E8F1FF,stroke:#3973C6,color:#17365D
    classDef decision fill:#FFF4E5,stroke:#D99A35,color:#704510
    classDef result fill:#EAF7F0,stroke:#39966B,color:#174D35

    class A,B execute
    class C decision
    class D,E,F,G result

9. Why Can't Modern CPUs Simply Execute One Instruction to Completion Before Executing the Next?

The preceding sections introduced the traditional instruction execution process.

If the CPU had to work strictly in the following way:

decode only after fetch completes, execute only after decode completes, and fetch the next instruction only after execution completes, then a large amount of hardware resources might sit idle.

For example, while executing a data load instruction that needs to wait for memory, the CPU's other execution units may not be fully utilized.

To improve processor throughput, modern CPUs introduce techniques such as pipelining, out-of-order execution, and branch prediction.

9.1 Instruction Pipeline

The core idea of a pipeline is:

Split instruction execution into multiple stages so that multiple instructions are in different stages at the same time.

This can be understood as a production assembly line:

  • The first instruction is executing.
  • The second instruction is decoding.
  • The third instruction is being fetched.

Different instructions can occupy different pipeline stages at the same time.

flowchart TB
    subgraph C1["Cycle 1"]
        A1["Instruction 1: Fetch"]
    end

    subgraph C2["Cycle 2"]
        A2["Instruction 1: Decode"]
        B2["Instruction 2: Fetch"]
    end

    subgraph C3["Cycle 3"]
        A3["Instruction 1: Execute"]
        B3["Instruction 2: Decode"]
        C3I["Instruction 3: Fetch"]
    end

    subgraph C4["Cycle 4"]
        A4["Instruction 1: Memory Access"]
        B4["Instruction 2: Execute"]
        C4I["Instruction 3: Decode"]
        D4["Instruction 4: Fetch"]
    end

    subgraph C5["Cycle 5"]
        A5["Instruction 1: Write Back"]
        B5["Instruction 2: Memory Access"]
        C5I["Instruction 3: Execute"]
        D5["Instruction 4: Decode"]
    end

    classDef stage fill:#E8F1FF,stroke:#3973C6,color:#17365D

    class A1,A2,B2,A3,B3,C3I,A4,B4,C4I,D4,A5,B5,C5I,D5 stage

The figure above shows an idealized five-stage pipeline.

Actual processors may use more pipeline stages and may also process multiple instructions, micro-operations, and execution threads simultaneously.

Pipelines mainly improve instruction throughput; they do not mean that the execution latency of a single instruction is necessarily reduced.

9.2 Pipeline Conflicts

Pipelines cannot always run smoothly.

There are three common types of conflicts:

Type English Meaning
Structural hazard Structural Hazard Multiple instructions compete for the same hardware resource
Data hazard Data Hazard Data dependencies exist between instructions
Control hazard Control Hazard Branch instructions make the subsequent execution path uncertain

Data hazards

For example:

ADD R1, R2, R3
SUB R4, R1, R5

The second instruction needs to use R1.

If the first instruction has not yet produced the result, the second instruction cannot use that value directly.

This situation is a data dependency.