Back to series
· 9 min read

Preprocessing and Compilation


1. What Are Preprocessing and Compilation?

The source code written by programmers usually cannot be executed directly by the CPU; it must be processed by a compilation toolchain and converted into code that the target platform can execute.

Taking C as an example, a typical compilation process can be divided into:

  • Preprocessing: Handles preprocessing directives such as header files, macro definitions, and conditional compilation.
  • Compilation: Analyzes the source code, checks syntax and semantics, generates an intermediate representation, and performs optimization.
  • Assembly: Converts assembly instructions into machine instructions and generates object files.
  • Linking: Integrates object files and libraries, resolves symbol references, and generates an executable file.

Overall flow:

flowchart TB
    A["Source Code<br/>main.c"] --> B["Preprocessing<br/>Preprocessing"]
    B --> C["Compilation<br/>Compilation"]
    C --> D["Assembly Code<br/>Assembly Code"]
    D --> E["Assembly<br/>Assembly"]
    E --> F["Object File<br/>Object File"]
    F --> G["Linking<br/>Linking"]
    H["Other Object Files and Libraries"] --> G
    G --> I["Executable<br/>Executable"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style D fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style E fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style G fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style H fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

Note that preprocessing, compilation, assembly, and linking are conceptually distinct stages. An actual toolchain may combine and execute several of these stages.

This article mainly uses C and the GCC/Clang toolchain as examples to introduce the internal mechanisms of preprocessing and compilation.

2. Preprocessing

2.1 What Is Preprocessing?

Preprocessing is an independent stage in the C/C++ compilation process and is mainly responsible for handling preprocessing directives that begin with #.

Preprocessing occurs before formal compilation. Its core task is to perform text-level processing on the source code and generate preprocessed source code.

Common preprocessing directives include:

Directive Purpose
#include Include header files
#define Define macros
#undef Undefine macros
#if Conditional compilation
#ifdef Determine whether a macro is defined
#ifndef Determine whether a macro is not defined
#elif Other conditional branches
#else Default branch of conditional compilation
#endif End conditional compilation
#error Generate a preprocessing error
#pragma Pass specific directives to the compiler

The preprocessor mainly processes source code according to preprocessing rules; it is not responsible for complete C syntax analysis and type checking.

2.2 Header File Expansion (#include)

#include is used to include the contents of a specified file into the current source file.

For example:

#include <stdio.h>

int main(void) {
    printf("Hello World\n");
    return 0;
}

When the preprocessor encounters:

#include <stdio.h>

It will search for the corresponding file according to the header file search rules and incorporate its contents into the preprocessing result of the current translation unit.

Header file inclusion generally comes in two forms:

1. System header files

#include <stdio.h>

This form is generally used to include header files provided by the standard library or the system.

2. User Header Files

#include "config.h"

This form is typically used to include the project's own header files.

The search order and specific rules for the two forms are determined by the implementation and compiler configuration; you cannot simply assume that angle brackets search only system directories and double quotes search only the current directory.

2.3 The Practical Meaning of Header File Expansion

C typically uses header files to declare functions, types, macros, and constants.

For example:

math_utils.h:

#ifndef MATH_UTILS_H
#define MATH_UTILS_H

int add(int a, int b);

#endif

main.c:

#include "math_utils.h"

int main(void) {
    return add(10, 20);
}

After preprocessing, main.c will contain the effective contents of the header file.

You can think of it as:

#ifndef MATH_UTILS_H
#define MATH_UTILS_H

int add(int a, int b);

#endif

int main(void) {
    return add(10, 20);
}

The actual output may also contain information such as line number markers.

Note: header file expansion does not mean that the function implementation is automatically brought in as well. Header files usually provide declarations, while the function definitions may reside in separate .c files, which must later be combined through linking.

2.4 Macro Definitions and Macro Expansion (#define)

Macro definitions are used to define replacement rules for identifiers.

For example:

#define MAX_SIZE 100

int buffer[MAX_SIZE];

The preprocessor will replace MAX_SIZE with 100.

After expansion:

int buffer[100];

Macros can also take parameters.

For example:

#define SQUARE(x) ((x) * (x))

int result = SQUARE(5);

After expansion:

int result = ((5) * (5));

Pay special attention here: a macro is essentially a replacement mechanism at the preprocessing stage, not an ordinary function call.

For example:

#define SQUARE(x) ((x) * (x))

int result = SQUARE(i++);

After expansion this is equivalent to:

int result = ((i++) * (i++));

The i in the expression is modified multiple times, which may result in undefined behavior.

Therefore, macro parameters should be used with care; a macro cannot simply be treated as a function.

2.5 Several Important Mechanisms in Macro Expansion

1. The Stringizing Operator (#)

It can convert a macro parameter into a string literal.

#define TO_STRING(x) #x

const char *name = TO_STRING(hello);

Expansion result:

const char *name = "hello";

2. The Token-Pasting Operator (##)

It can paste two preprocessing tokens together into a new token.

#define MAKE_VAR(name) value_##name

int MAKE_VAR(count) = 10;

Expansion result:

int value_count = 10;

3. Variadic Macros

They can accept a variable number of arguments.

#define LOG(format, ...) \
    printf("[LOG] " format "\n", __VA_ARGS__)

Usage:

LOG("value = %d", 100);

In real projects, when using variadic macros you need to be aware of differences between language standards and compilers regarding support for empty arguments.

2.6 Conditional Compilation

Conditional compilation is used to determine, based on macro definitions or conditional expressions, which code takes part in subsequent compilation.

For example:

#define DEBUG_MODE 1

#if DEBUG_MODE
    printf("Debug mode\n");
#else
    printf("Release mode\n");
#endif

When DEBUG_MODE is 1, the preprocessor keeps the first branch and excludes the second branch.

Conditional compilation is commonly used for:

  • Distinguish between Debug and Release builds.
  • Support different operating systems.
  • Support different CPU architectures.
  • Control feature toggles.
  • Avoid duplicate inclusion of header files.

For example:

#ifdef _WIN32
    // Windows platform code
#elif defined(__linux__)
    // Linux platform code
#else
    // Code for other platforms
#endif

Note that conditional compilation selects code branches during the preprocessing stage; it is not an if decision at program runtime.

2.7 Header Guards (Include Guard)

If the same header file is included multiple times, it may cause issues such as duplicate declarations and type redefinitions.

A common solution is to use a header guard macro:

#ifndef CONFIG_H
#define CONFIG_H

#define MAX_CONNECTIONS 100

struct Config {
    int port;
    int timeout;
};

#endif

Its working mechanism:

flowchart TB
    A["Encounter a header file"] --> B{"Is the guard macro already defined?"}
    B -->|"No"| C["Define guard macro"]
    C --> D["Expand header file contents"]
    D --> E["Continue processing source code"]
    B -->|"Yes"| F["Skip header file contents"]
    F --> E

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

Some compilers also support:

#pragma once

This is used to prevent the same header file from being processed multiple times in a single translation unit.

#pragma once is simple to use, but it is a compiler-supported extension; the traditional Include Guard is more consistent with the standard preprocessing directive mechanism.

III. Viewing Preprocessing Results

The most direct way to understand preprocessing is to look at the actual output generated by the preprocessor.

Using GCC:

gcc -E main.c -o main.i

Where:

  • -E: Run only the preprocessing stage.
  • main.c: Input source file.
  • -o main.i: Specify the preprocessed output file.

You can also output directly to the terminal:

gcc -E main.c

Preprocessing results typically include:

  • Expanded header file contents.
  • Macros after replacement.
  • Code retained after conditional compilation.
  • Line markers used to record source file locations.

If you only want to view macro definitions:

gcc -dM -E main.c

This helps you understand the macros already defined by the compiler and in the current compilation environment.

3.1 A Complete Preprocessing Example

Source code:

#include <stdio.h>

#define VERSION 2
#define MESSAGE "Hello"

int main(void) {
    printf("%s %d\n", MESSAGE, VERSION);
    return 0;
}

Run:

gcc -E main.c -o main.i

In the preprocessing result, the part related to the user code can be simplified as follows:

int main(void) {
    printf("%s %d\n", "Hello", 2);
    return 0;
}

The actually generated .i file will also contain declarations expanded from stdio.h and other preprocessing information.

IV. Compilation

4.1 What Is Compilation?

After preprocessing is complete, the compiler begins language-level analysis of the processed source code and converts it into a code representation closer to the target platform.

The compilation stage mainly performs:

  1. Lexical analysis.
  2. Syntax analysis.
  3. Semantic analysis.
  4. Intermediate representation generation.
  5. Code optimization.
  6. Target code generation.

Note that the internal implementation of a specific compiler may vary. For example, GCC and Clang are not entirely the same in their intermediate representations, optimization pipelines, and internal data structures.

V. Lexical Analysis

5.1 What Is Lexical Analysis?

Lexical analysis is the first step in how a compiler processes source code.

It splits a continuous sequence of characters into individual lexical units (tokens) that each carry meaning.

For example:

int result = a + 10;

The lexer can identify:

Token Type
int Keyword
result Identifier
= Operator
a Identifier
+ Operator
10 Integer literal
; Separator

5.2 What Lexical Analysis Does

It mainly covers:

  • Recognizing keywords.
  • Recognizing identifiers.
  • Recognizing constants and string literals.
  • Recognizing operators.
  • Recognizing separators.
  • Ignoring or handling whitespace and comments.

For example:

int count = 100;

The lexer does not treat the whole statement as a single unit; it splits it into multiple tokens.

5.3 Lexical Analysis and Regular Expressions

Lexical analysis can usually be implemented with a finite automaton.

Many lexical rules can be described with regular expressions, for example:

  • Identifiers: start with a letter or underscore, followed by letters, digits, or underscores.
  • Integers: composed of digits.
  • Whitespace: spaces, tabs, newlines, and so on.

Real-world lexers typically use techniques such as DFA (deterministic finite automaton) to recognize tokens efficiently.

Lexical analysis mainly answers:

Which kind of lexical unit does the current character sequence belong to?

It is not responsible for judging whether the structure of a complete statement is valid.

6. Syntax Analysis

6.1 What Is Syntax Analysis?

The parser receives the token sequence produced by lexical analysis and, following the grammar rules of the programming language, builds the syntactic structure of the program.

For example:

int result = a + 10;

The parser needs to recognize:

  • This is a variable declaration.
  • The variable type is int.
  • The variable name is result.
  • The initializer expression is a + 10.
  • The addition expression contains two operands.

6.2 Abstract Syntax Tree (AST)

Syntax analysis usually produces an Abstract Syntax Tree (AST).

For example:

int result = a + 10;

It can be abstracted as:

VariableDeclaration
├── Type: int
├── Name: result
└── Initializer
    └── Addition
        ├── Identifier: a
        └── IntegerLiteral: 10

The AST retains the core syntactic structure of the program and omits some surface-level syntactic details that do not affect semantic analysis.

6.3 Why Do We Need an AST?

The AST is an important foundation for the compiler to understand the structure of a program.

It can support:

  • Type checking.
  • Variable scope analysis.
  • Expression optimization.
  • Code generation.
  • Static analysis.
  • Code formatting and refactoring.

For example, the following two expressions:

a + b * c
a + (b * c)

When operator precedence is unambiguous, they have the same core expression structure.

An AST can represent operator precedence and associativity as a tree structure, rather than simply preserving the character sequence.

7. Semantic Analysis

7.1 What Is Semantic Analysis?

Semantic analysis checks whether a program conforms to the semantic rules of the programming language.

Syntax analysis can only determine whether the code structure conforms to the grammar; it cannot guarantee that the code is correct in terms of types, scope, symbol references, and so on.

For example:

int number = "hello";

This code is basically valid in terms of syntactic structure, but the initializer expression is incompatible with the variable type.

Semantic analysis needs to identify problems of this kind.

7.2 Main Contents of Semantic Analysis

Check Description
Type checking Checks whether expression and variable types are compatible
Scope checking Checks whether a variable is within a valid scope
Symbol resolution Determines the declaration that an identifier reference corresponds to
Function call checking Checks the number and types of arguments
Return value checking Checks function return expressions
Access control Checks access permissions of types or members, depending on the language

7.3 Symbol Table

The symbol table is an important data structure in the compiler, used to record information about identifiers in the program.

For example:

int number = 10;

int add(int a, int b) {
    return a + b;
}

The compiler might maintain information like this:

Symbol name Type Scope Other information
number int File scope Global variable
add Function File scope Return type is int
a int Function parameter scope Parameter
b int Function parameter scope Parameter

The symbol table helps the compiler accomplish the following:

  • Variable lookup.
  • Type checking.
  • Function call resolution.
  • Scope management.
  • Associating addresses and storage information.

7.4 Type Checking and Implicit Conversion

The compiler needs to determine whether the type of an expression is legal, and to perform the necessary type conversions according to the rules of the language.

For example:

int a = 10;
double b = 3.5;

double result = a + b;

Following the usual arithmetic conversion rules, C converts a to the appropriate floating-point type, then performs the addition.

Semantic analysis not only checks whether the types are compatible, but also needs to determine the types of expressions and the conversion rules.

VIII. Intermediate Representation (IR)

8.1 What Is an Intermediate Representation?

An intermediate representation is a form of program representation used internally by the compiler, situated between the source language and the target machine code.

Compilers typically convert the AST into an IR that is better suited to optimization and code generation.

Why is an IR needed?

If a compiler generated machine code for every CPU architecture directly from every source language, it would need to maintain a large amount of separate conversion logic.

For example:

  • C to x86-64.
  • C to ARM64.
  • Rust to x86-64.
  • Rust to ARM64.

With an intermediate representation, the compilation process can be split into:

  • Front end: converting the source language into IR.
  • Middle end: analyzing and optimizing the IR.
  • Back end: converting the IR into code for the target platform.