Back to series
· 13 min read

Build Tools and Build Systems


1. What Are Build Tools and Build Systems?

Source code in software development usually cannot be deployed or delivered directly. A project may contain hundreds of source files, third-party dependencies, configuration files, resource files, and test code, all of which must go through a series of processes such as compilation, linking, packaging, testing, and code generation before the final artifact can be produced.

These processing activities are collectively referred to as Build.

Build-related technologies can be divided into two levels:

  • Build Tool: Software that executes specific build tasks, such as compiling code, compressing resources, generating files, and packaging programs.
  • Build System: A mechanism that organizes and manages the complete build process, responsible for task dependencies, execution order, incremental builds, caching, parallel scheduling, and build result management.

The two are not entirely independent product categories. Some tools have both task orchestration and build management capabilities, such as Gradle, Bazel, and Buck.

1. What Problems Do Build Tools Solve?

Build tools mainly solve the automation of repetitive engineering tasks.

For example, a Java project may require:

  1. Check project dependencies.
  2. Compile Java source code.
  3. Process resource files.
  4. Run unit tests.
  5. Package into a JAR file.
  6. Generate other files required for release.

If performed entirely manually, it is not only inefficient but also prone to missing steps and environment differences.

Build tools define these operations as tasks that can be executed repeatedly.

2. What Problems Do Build Systems Solve?

As project scale grows, simple task execution is no longer sufficient.

For example:

  • Which tasks must be executed first?
  • Which tasks can be executed simultaneously?
  • After modifying a source file, which tasks need to be re-run?
  • How can repeated compilation be avoided?
  • How can different developers obtain consistent build results?
  • How can previous build caches be reused in a CI/CD environment?

These issues fall under the responsibility of the build system.

3. Differences Between Build Tools and Build Systems

Comparison Dimension Build Tool Build System
Core responsibility Execute specific build operations Manage the complete build process
Focus Task execution Task relationships and execution strategy
Task dependencies Can be supported Usually a core capability
Incremental builds Supported by some tools Usually a key focus
Parallel execution Supported by some tools Usually scheduled in a unified manner
Cache management May be supported Usually has more complete mechanisms
Typical representatives GCC, Clang, Webpack Bazel, Buck, Gradle

Note: This division is an abstraction at the level of responsibilities; it does not mean that each product can only belong to one of these categories.


2. The Basic Process of Software Build

Building is not simply executing a compilation command once; it is an engineering process composed of multiple stages.

The specific steps differ across languages and projects, but typically include the following phases:

  1. Environment preparation: Check the toolchain, configuration, and runtime environment.
  2. Dependency resolution: Determine the external libraries and their versions required by the project.
  3. Code generation: Generate source files based on configuration, interface definitions, or other inputs.
  4. Compilation: Convert source code into object code or intermediate products.
  5. Resource processing: Copy, transform, and compress static resources.
  6. Linking: Combine object files and libraries to produce an executable or library.
  7. Testing: Run automated tests.
  8. Packaging: Produce JARs, ZIPs, images, or other delivery artifacts.
  9. Release: Upload artifacts to an artifact repository or deployment environment.

1. Typical Build Workflow

flowchart TD
    A["Source code and project configuration"] --> B["Environment and toolchain check"]
    B --> C["Dependency resolution"]
    C --> D["Code generation"]
    D --> E["Compilation"]
    E --> F["Resource processing"]
    F --> G["Linking"]
    G --> H["Automated testing"]
    H --> I{"Do the tests pass?"}
    I -->|"Yes"| J["Package build artifacts"]
    I -->|"No"| K["Report build failure"]
    J --> L["Artifact verification"]
    L --> M["Release or deploy"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style D fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style J fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style K fill:#FBE9E9,stroke:#D9AAAA,color:#334155
    style L fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style M fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

This flow is a conceptual example; not all projects execute in exactly the same order.

For example, a Java project may compile first and then run tests; a frontend project may treat resource processing and code bundling as the main stages; a C/C++ project typically has to go through compilation and linking.

2. Build Inputs and Build Outputs

A build system generally needs to clearly identify its inputs and outputs.

Common inputs:

  • Source code files.
  • Configuration files.
  • Dependency libraries.
  • Compilers and their versions.
  • Build scripts.
  • Environment variables.
  • Code generators.
  • Build parameters.

Common outputs:

  • Object files.
  • Executable files.
  • Dynamic and static libraries.
  • Bytecode.
  • Archive files such as JAR, WAR, and ZIP.
  • Frontend static assets.
  • Container images.
  • Test reports and build metadata.

A build system needs to know which inputs affect which outputs; otherwise it is difficult to accurately determine whether a rebuild is required.


III. Common Build Tools

Different language ecosystems have different build tools, and they differ noticeably in design goals, build flow, and applicable scenarios.

1. C/C++: Make, CMake, Ninja

C/C++ projects generally need to handle source file compilation, object file generation, library linking, and toolchain configuration for different platforms.

Make

Make is a classic build automation tool; it typically describes targets, dependencies, and commands to execute through a Makefile.

Example:

CC = gcc
CFLAGS = -Wall -O2

app: main.o utils.o
    $(CC) main.o utils.o -o app

main.o: main.c
    $(CC) $(CFLAGS) -c main.c -o main.o

utils.o: utils.c
    $(CC) $(CFLAGS) -c utils.c -o utils.o

clean:
    rm -f *.o app

Three main targets are defined here:

  • app: the final executable.
  • main.o: compiled from main.c.
  • utils.o: compiled from utils.c.

Make determines which commands to execute based on targets and their dependencies.

CMake

CMake itself is mainly a cross-platform build configuration and generation tool. Based on the project structure described in CMakeLists.txt, it generates Makefiles, Ninja files, or other files required by native build systems.

Example:

cmake_minimum_required(VERSION 3.20)

project(MyApp LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_executable(my_app
    main.cpp
    utils.cpp
)

Typical usage:

cmake -S . -B build
cmake --build build

Need to distinguish:

  • CMake: describes and generates build configurations.
  • Make: executes build rules in a Makefile.
  • Ninja: executes tasks according to Ninja build files.

Ninja

Ninja is a build tool focused on execution speed. Its build files are usually generated by tools such as CMake.

It focuses on efficiently executing predefined build rules rather than providing a complex build configuration language.

2. Java: Maven and Gradle

Java projects usually involve dependency management, source compilation, testing, resource processing, and JAR/WAR packaging.

Maven

Maven is centered on the Project Object Model (POM) and uses pom.xml to describe project configuration, dependencies, and plugins.

Example:

<project>
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>demo</artifactId>
    <version>1.0.0</version>

    <properties>
        <maven.compiler.release>17</maven.compiler.release>
    </properties>
</project>

Common commands:

mvn clean
mvn compile
mvn test
mvn package
mvn install

Maven provides a convention-over-configuration project structure and lifecycle mechanism.

Gradle

Gradle uses a DSL to describe build logic and supports Groovy DSL and Kotlin DSL.

Example:

plugins {
    java
}

group = "com.example"
version = "1.0.0"

java {
    toolchain {
        languageVersion.set(
            JavaLanguageVersion.of(17)
        )
    }
}

Common commands:

./gradlew clean
./gradlew build
./gradlew test
./gradlew jar

Important features of Gradle include:

  • Task-based build model.
  • Flexible plugin system.
  • Incremental builds.
  • Build cache.
  • Parallel task execution.
  • Java Toolchains.

Maven places more emphasis on a standardized lifecycle and conventions; Gradle provides more flexible build logic and a task model.

3. JavaScript / TypeScript: Webpack, Vite, esbuild

The build priorities for frontend projects usually include:

  • Module dependency analysis.
  • TypeScript or JSX transformation.
  • CSS processing.
  • Image and font asset processing.
  • Code minification.
  • Tree Shaking.
  • Code splitting.
  • Production bundling.

Webpack

Webpack is a bundling tool centered on the module dependency graph.

It uses Loaders to process different types of modules and Plugins to extend the build lifecycle.

Example:

const path = require("path");

module.exports = {
  mode: "production",

  entry: "./src/index.js",

  output: {
    filename: "bundle.js",
    path: path.resolve(__dirname, "dist")
  }
};

Vite

Vite is a build tool for modern frontend development.

During development, it typically uses native ES Modules to provide module services and relies on dependency pre-bundling to improve the development experience; for production builds, it uses its production build toolchain to bundle and optimize assets.

Common commands:

npm run dev
npm run build
npm run preview

esbuild

esbuild is a high-performance JavaScript, TypeScript, and CSS build tool implemented in Go.

It has fast parsing, transformation, and bundling speeds and is often integrated into other tools.

Note: Vite, Webpack, and esbuild do not have exactly the same positioning. They each emphasize different aspects of development servers, plugin systems, module processing, and production builds.

4. Python: Setuptools, Hatch, Build

Builds in Python projects usually involve source distribution packages (sdist) and wheel binary distribution packages.

Common tools include:

  • Setuptools: a traditional and widely used Python build backend.
  • Hatchling: the build backend in the Hatch ecosystem.
  • Flit: a tool for building and publishing Python packages.
  • Build: a frontend tool that invokes build backends through the PEP 517 interface.

For example:

python -m build

This command typically generates, based on the project configuration:

dist/
├── example-1.0.0.tar.gz
└── example-1.0.0-py3-none-any.whl

A distinction needs to be made here:

  • pip: mainly responsible for installing Python packages.
  • Build: responsible for invoking build backends to generate distribution packages.
  • Setuptools, Hatchling: responsible for the actual build work.

5. Go: Go Build

Go provides built-in build commands and does not require configuring a separate build tool the way many languages do.

Common commands:

go build
go test
go install
go clean

For example:

go build -o app ./cmd/server

The Go toolchain handles package dependencies, compilation, linking, and the build cache.

6. Rust: Cargo

Cargo is Rust's official build tool and package manager.

It not only manages dependencies, but is also responsible for build tasks, testing, running, documentation generation, and publishing-related operations.

Common commands:

cargo build
cargo build --release
cargo test
cargo run
cargo doc

Rust projects typically use Cargo.toml to describe the project and its dependency configuration.

Cargo invokes the Rust compiler rustc to perform the actual compilation work.


IV. The Core Architecture of a Build System

A build system does not simply execute commands from top to bottom according to a script; rather, it manages the build process through tasks, dependencies, and scheduling mechanisms.

1. Tasks

A task is the basic unit of execution in a build system.

For example:

  • CompileTask: a compilation task.
  • TestTask: a testing task.
  • PackageTask: a packaging task.
  • GenerateTask: a code generation task.
  • CopyResourceTask: a resource copying task.

Each task typically contains:

  • The task name.
  • Input files.
  • Output files.
  • Dependent tasks.
  • Execution logic.
  • Execution parameters.

2. Task Dependency Graph

Dependencies usually exist between build tasks.

For example:

compileA ─┐
          ├──> package
compileB ─┘

package depends on compileA and compileB.

A build system can organize task relationships into a directed acyclic graph (DAG).

flowchart TD
    A["Read project configuration"] --> B["Resolve dependencies"]
    B --> C["Compile module A"]
    B --> D["Compile module B"]
    C --> E["Run tests"]
    D --> E
    E --> F["Generate release package"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style E fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

In this example:

  • Compiling module A and module B do not depend on each other, so they can be executed in parallel.
  • The test task must wait for both modules to finish compiling.
  • The packaging task must wait for the test task to finish.

Real build systems may also configure testing as an optional stage; the exact dependencies depend on the build rules.

3. Task Scheduling

Task scheduling is responsible for determining the execution order of tasks and the allocation of resources.

Main considerations:

  • Whether task dependencies are satisfied.
  • The number of CPU cores.
  • Memory limits.
  • I/O resources.
  • Task priority.
  • The safety of parallel execution.

A build system generally looks in the dependency graph for tasks whose execution conditions are already satisfied, and hands them over to the executor.

For example:

flowchart TD
    A["Build task graph"] --> B["Check task dependencies"]
    B --> C{"Are there any executable tasks?"}
    C -->|"Yes"| D["Add to scheduling queue"]
    D --> E["Allocate execution resources"]
    E --> F["Execute task"]
    F --> G["Record execution result"]
    G --> B
    C -->|"No"| H["Check incomplete tasks"]
    H --> I{"Are all tasks complete?"}
    I -->|"Yes"| J["Build complete"]
    I -->|"No"| K["Report dependency error or task failure"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style C fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style D fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style G fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style H fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style J fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style K fill:#FBE9E9,stroke:#D9AAAA,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

If a build task graph contains cyclic dependencies, the build system usually needs to report an error rather than schedule indefinitely.

4. Executor

The executor is responsible for actually running tasks.

It may invoke:

  • Compilers.
  • Script interpreters.
  • Shell commands.
  • Code generators.
  • Test frameworks.
  • External tools.

The build system is responsible for managing task relationships, while the executor is responsible for performing the actual work. Together they complete the build process.


V. Incremental Build

Incremental builds are an important capability of modern build systems.

Its goal is to re-execute only the affected build tasks when a project changes, rather than rebuilding the entire project every time.

1. Why Are Incremental Builds Needed?

Suppose a project contains 1000 source files.

If only one file is modified but all files are recompiled every time, a large amount of unnecessary work is produced.

Incremental builds analyze changes in inputs and reuse previous build results as much as possible.

2. Timestamp-Based Incremental Builds

Traditional Make usually uses file timestamps to determine whether a target needs to be updated.

For example:

app: main.o utils.o
    gcc main.o utils.o -o app

main.o: main.c
    gcc -c main.c -o main.o

utils.o: utils.c
    gcc -c utils.c -o utils.o

When the modification time of main.c is later than that of main.o, Make re-executes the corresponding compilation rule.

This approach is simple and efficient, but it has certain limitations:

  • Timestamp precision may affect the determination.
  • If file contents change but timestamps are not updated correctly, changes may be missed.
  • Non-file inputs such as toolchains and environment variables may not be fully tracked.
  • The output dependencies of some build tasks are not easy to express using timestamps alone.

3. Content-Hash-Based Incremental Builds

More advanced build systems may use content hashes to identify input changes.

For example:

Source file contents
    +
Compiler version
    +
Compilation parameters
    +
Dependency files
    ↓
Build input digest

When the digests of all relevant inputs remain consistent, the system can reuse previous build results.

This mechanism makes it easier to achieve precise cache decisions than relying solely on timestamps.

However, it depends on whether build inputs are fully declared. If a task secretly reads undeclared files or environment variables, incorrect cache hits may occur even if a hashing mechanism is in place.

4. Incremental Build Process

flowchart TD
    A["Start build"] --> B["Read build task"]
    B --> C["Collect task inputs"]
    C --> D["Compute input state"]
    D --> E{"Have the inputs changed?"}
    E -->|"No"| F["Reuse existing output"]
    E -->|"Yes"| G["Re-execute task"]
    G --> H["Save new build result"]
    F --> I["Check next task"]
    H --> I
    I --> J{"Are there more tasks?"}
    J -->|"Yes"| B
    J -->|"No"| K["Build complete"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style F fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style G fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style J fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style K fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

6. Build Cache

Incremental builds determine which tasks need to be re-executed, while build caching further addresses how to reuse results that have already been computed.

The two are closely related, but not identical.

1. What Is a Build Cache?

A build cache stores the results produced by task execution so they can be reused directly in subsequent builds.

For example:

First build:

compile-A → perform compilation → save result

Second build:

compile-A → check cache → hit → reuse result

If the inputs have not changed, the same compilation task does not need to be executed again.

2. Local Cache and Remote Cache

Type Storage Location Use Case
Local cache Developer's local machine Repeated local builds
Remote cache Shared server or object storage Multi-person collaboration, CI/CD
Distributed cache Multi-node cache system Large projects and multiple build nodes

Remote caching allows different developers or CI nodes to reuse build results, but it must address issues such as permissions, integrity verification, cache isolation, and invalidation management.

3. Cache Key

The cache system needs to determine whether the current task has the same inputs as a previous task.

Therefore, a cache key is usually constructed.

CacheKey = Hash(
    source file contents,
    dependency contents,
    compiler version,
    compilation parameters,
    environment configuration,
    build rules
)

The actual algorithm and the fields included are determined by the specific build system.

If the cache key omits an important input, it may lead to incorrect reuse.

For example:

The same source code may produce different results under different compiler versions or compilation parameters.

If the cache key does not include these differences, it may incorrectly hit an old cache.

4. Cache Invalidation and Cache Pollution

Cache invalidation means that an existing cache is no longer applicable to the current inputs.

Cache pollution means that incorrect, expired, or untrusted results are stored in the cache, affecting subsequent builds.

Common causes include:

  • Incomplete declaration of build inputs.
  • Environment variables are not included in the cache key.
  • Dependency files are modified externally.
  • Different platforms incorrectly share a cache.
  • The build script has undeclared external side effects.
  • The cache write process is incomplete.

In engineering practice, you should make the inputs and outputs of build tasks as explicit as possible, and apply access control and integrity verification to shared caches.


7. Reproducible Build

Reproducible builds are an important foundation for software supply chain security and engineering reliability.

They emphasize that, under the same specified build conditions and with the same inputs, repeated builds should produce consistent results.

1. Why Are Reproducible Builds Needed?

Suppose a developer builds a software package locally and then releases it to production.

If the artifact produced by rebuilding in the CI environment differs from the local result, the source of the difference must be further identified.

Possible causes include:

  • Different dependency versions.
  • Different compiler versions.
  • Different build environments.
  • Timestamps are embedded in the artifact.
  • File order is not fixed.
  • Different environment variables.
  • The build script reads external state.

Reproducible builds can help development teams locate these issues.

2. Key Conditions for Reproducible Builds

Factor Approach
Dependency versions Pin dependency versions and use lock files
Compiler version Pin the toolchain version
Build environment Use standardized containers or build images
Timestamps Use a consistent timestamp strategy
File ordering Use deterministic ordering
Environment variables Explicitly declare and control environment variables
External resources Pin input versions and verify their contents
Build scripts Avoid implicit dependencies and uncontrolled side effects

3. Reproducible Build Process

flowchart TD
    A["Determine source code version"] --> B["Lock dependency versions"]
    B --> C["Pin toolchain"]
    C --> D["Prepare standardized environment"]
    D --> E["Execute build"]
    E --> F["Generate build artifacts"]
    F --> G["Compute artifact digests"]
    G --> H["Repeat build in an independent environment"]
    H --> I["Compare artifact digests"]
    I --> J{"Are the results consistent?"}
    J -->|"Yes"| K["Validation passed"]
    J -->|"No"| L["Analyze build differences"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style G fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style J fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style K fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style L fill:#FBE9E9,stroke:#D9AAAA,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

Note that reproducible builds are not just about repeatedly running the same command. They require the key inputs and environmental factors in the build process to be sufficiently explicit and controllable.


VIII. Advanced Capabilities of Build Systems