Back to series
· 10 min read

Package Managers and Dependency Management


I. What Is a Package Manager?

A package manager is a tool for managing software packages and their dependencies.

In modern software development, a project usually does not rely entirely on code written by itself; it needs to use standard libraries, third-party libraries, frameworks, tools, and other components.

Manually downloading, configuring, and updating these components every time creates a lot of repetitive work and easily leads to issues such as version inconsistency, dependency conflicts, and environment differences.

Package managers exist to solve these problems.

It is usually responsible for:

  • Searching for and retrieving packages.
  • Installing and uninstalling packages.
  • Resolving dependencies between packages.
  • Managing package versions.
  • Downloading and verifying packages.
  • Updating project dependencies.
  • Maintaining dependency manifests and lockfiles.
  • Managing package repositories and publishing workflows.

1. What Is a Package?

A package is a software component organized in a certain format for distribution and reuse.

A package usually contains:

Component Description
Source code or binary files The actual implementation of the package
Metadata Information such as name, version, and author
Dependency declarations Other packages required for running or building
Entry point information Module entry point or program entry point
License The licensing terms of the package
Verification information Used to verify file integrity
Documentation Installation instructions, usage, and API documentation

For example, a Python project can be published as a package via PyPI, a JavaScript project can be published via npm, and a Rust library can be published via crates.io.

Note that a package is not necessarily the same as a library. A package may also contain command-line tools, compiler plugins, build tools, applications, or other resources.

2. Basic Workflow of a Package Manager

A package manager usually looks up the package repository based on the dependencies declared by the project, resolves version constraints, retrieves the packages, and installs them into the specified environment.

flowchart TD
    A["Project declares dependencies"] --> B["Package manager reads configuration"]
    B --> C["Query package repository"]
    C --> D["Resolve version constraints"]
    D --> E["Compute dependency relationships"]
    E --> F["Download packages"]
    F --> G["Verify integrity"]
    G --> H["Install dependencies"]
    H --> I["Update dependency records"]
    I --> J["Project uses dependencies"]

    style A fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style D fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style G fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style J fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

During actual execution, a package manager may also handle caching, platform adaptation, build scripts, environment isolation, dependency conflicts, and other issues.

II. Package Managers for Major Languages

Different languages have different package management ecosystems. The way package managers are used, their configuration files, and their dependency resolution rules also differ.

1. Python: pip, uv, Poetry

Common package management tools in the Python ecosystem include:

Tool Primary Use
pip Install and manage Python packages
venv Create Python virtual environments
Poetry Project dependency, environment, and build management
uv High-performance Python package management and project tool
conda Environment and package management, often used in scientific computing

pip is the most common package installation tool for Python.

# Install a third-party library
python -m pip install requests

# Install a specified version
python -m pip install requests==2.32.3

# Uninstall a package
python -m pip uninstall requests

# List installed packages
python -m pip list

# Show dependency information
python -m pip show requests

# Export packages in the current environment
python -m pip freeze > requirements.txt

Python projects typically use requirements.txt to record dependencies, and can also use pyproject.toml together with tools such as Poetry and uv to manage project configuration and dependencies.

It is important to distinguish: pip is mainly responsible for package installation, while venv is responsible for creating virtual environments. They are not the same tool.

2. Java: Maven, Gradle

The Java ecosystem mainly uses Maven and Gradle to manage dependencies and build tasks.

Tool Characteristics
Maven Uses XML configuration, with clear conventions and a mature ecosystem
Gradle Supports Groovy and Kotlin DSL, with flexible build logic

Maven uses pom.xml to declare projects and dependencies.

<dependencies>
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>33.3.1-jre</version>
    </dependency>
</dependencies>

Gradle can use Kotlin DSL:

dependencies {
    implementation("com.google.guava:guava:33.3.1-jre")
    testImplementation("org.junit.jupiter:junit-jupiter:5.11.0")
}

Maven and Gradle are not only dependency management tools; they also handle project building, testing, packaging, and publishing.

3. JavaScript: npm, Yarn, pnpm

The JavaScript ecosystem mainly manages npm packages through package managers.

Common tools:

Tool Characteristics
npm The default, common tool in the Node.js ecosystem, with a mature ecosystem
Yarn Provides dependency management and workspace capabilities
pnpm Improves dependency reuse efficiency through content-addressable storage and linking mechanisms

Common npm commands:

# Initialize project
npm init -y

# Install dependencies
npm install axios

# Install dev dependencies
npm install --save-dev typescript

# Uninstall dependencies
npm uninstall axios

# View dependency tree
npm ls

# Update dependencies that match version constraints
npm update

Common project files:

  • package.json: Project metadata, scripts, and dependency declarations.
  • package-lock.json: Locks the npm dependency resolution results.
  • node_modules: Directory of locally installed dependencies.

npm, Yarn, and pnpm use different lock file formats. Generally, you should maintain the corresponding lock file according to the package manager chosen for the project, and avoid having multiple tools arbitrarily modify the same dependency tree.

4. Go: Go Modules

Go uses Go Modules to manage project modules and dependencies.

Common commands:

# Initialize module
go mod init example.com/myapp

# Add or upgrade dependencies
go get github.com/gin-gonic/gin

# Tidy dependencies
go mod tidy

# View dependency relationships
go mod graph

# View module information
go list -m all

Main files:

File Purpose
go.mod Declares the module path and dependency requirements
go.sum Records checksum information for dependency content

Go Modules manages dependencies using module versions and supports module proxies, checksum databases, and version selection mechanisms.

5. Rust: Cargo

Rust officially uses Cargo to manage projects, dependencies, and builds.

Common commands:

# Create project
cargo new myapp

# Add dependencies
cargo add serde

# Build project
cargo build

# Run tests
cargo test

# View dependency tree
cargo tree

# Update dependencies
cargo update

Main files:

File Purpose
Cargo.toml Declares project metadata and dependencies
Cargo.lock Locks dependency resolution results
target/ Build artifact directory

Cargo is responsible not only for downloading dependencies, but also for compiling, testing, building, and publishing.

6. C/C++: Conan, vcpkg

Dependency management in the C/C++ ecosystem is more complex than in many modern languages because it involves factors such as compilers, operating systems, processor architectures, ABI, and build systems.

Common tools:

Tool Key features
Conan Cross-platform C/C++ dependency management
vcpkg C/C++ package management tool maintained by Microsoft
CMake Cross-platform build system that can also integrate dependency discovery and retrieval

Conan example:

# Install Conan
pip install conan

# Install dependencies
conan install . --output-folder=build

vcpkg example:

# Install a specified library
vcpkg install fmt

# Install the library for a specified platform configuration
vcpkg install fmt:x64-windows

C/C++ projects especially need to ensure that the build configurations of dependencies are consistent, such as compiler version, runtime library, static or dynamic linking, and target platform architecture.

III. Core Concepts of Dependency Management

Dependency management refers to the process of declaring, resolving, installing, updating, locking, and maintaining the software components that a project depends on.

Package managers are important tools for carrying out dependency management, but dependency management itself is a broader engineering practice.

1. What Is a Dependency?

A dependency is an external component that a project needs when running, building, or testing.

For example, a web service may depend on:

  • Web framework.
  • Database driver.
  • JSON serialization library.
  • Logging component.
  • Authentication component.
  • Testing framework.

Dependencies can be categorized by purpose of use.

Type Meaning Example
Runtime dependency Required when the program is running Web framework, database driver
Development dependency Required during development Code formatting tool
Test dependency Required when running tests pytest, JUnit
Build dependency Required for compilation and packaging Compiler plugin, code generator
Optional dependency Required only when certain features are enabled Image processing extension

Different package managers do not use exactly the same names and semantics for dependency categories.

2. Direct Dependencies and Transitive Dependencies

A direct dependency is a dependency that the project explicitly declares.

A transitive dependency is another component that a direct dependency further requires.

For example:

flowchart TD
    A["Business project"] --> B["Web framework"]
    A --> C["Logging library"]
    B --> D["HTTP utility library"]
    B --> E["Serialization library"]
    D --> F["Network infrastructure components"]
    E --> G["Encoding utility library"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style E fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style G fill:#FFF5DE,stroke:#E8D29A,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

In this example:

  • The web framework and logging library are direct dependencies of the project.
  • The HTTP utility library and serialization library are transitive dependencies of the web framework.
  • The network infrastructure components and encoding utility library are deeper-level transitive dependencies.

Transitive dependencies can form a complex dependency tree.

Therefore, the number of components actually used by a project is usually much larger than the number directly declared in configuration files.

3. Dependency Graph

A dependency graph is a data structure that describes the dependency relationships between software packages.

Usually, each software package can be viewed as a node, and each dependency relationship as a directed edge.

For example:

flowchart TD
    A["Application"] --> B["Library A"]
    A --> C["Library B"]
    B --> D["Library C"]
    C --> D
    D --> E["Library E"]
    C --> F["Library F"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style E fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

A dependency graph can help developers:

  • Understand the project's dependency structure.
  • Identify duplicate dependencies.
  • Analyze dependency conflicts.
  • Determine the scope of impact of vulnerabilities.
  • Assess the impact of upgrades.
  • Identify components that are no longer used.

A dependency graph may be a directed acyclic graph (DAG), but different ecosystems handle cyclic dependencies differently. Some tools reject certain cyclic dependencies, while other systems allow specific forms of circular references.

4. Version Management and Semantic Versioning

Software packages need version numbers to distinguish different releases.

Version management is an important part of dependency management, because different versions may have API changes, behavior changes, and compatibility differences.

1. Semantic Versioning

Semantic versioning typically uses:

MAJOR.MINOR.PATCH

For example:

3.5.2

The three numbers represent the following, respectively:

Part Meaning Example
MAJOR Incompatible API changes Upgrade from 3.x to 4.x
MINOR Backward-compatible feature additions Upgrade from 3.5 to 3.6
PATCH Backward-compatible bug fixes Upgrade from 3.5.2 to 3.5.3

The goal of semantic versioning is to let developers understand the compatibility impact of changes from the version number.

However, it depends on project maintainers following the specification correctly, and it cannot guarantee that all packages are released strictly according to semantic versioning.

2. Pre-release Versions

Packages may also publish pre-release versions, for example:

1.0.0-alpha.1
1.0.0-beta.2
1.0.0-rc.1
1.0.0

Common stages:

  • Alpha: an early test version; features may be incomplete.
  • Beta: features are relatively complete, but issues may still exist.
  • RC: a release candidate, usually close to the final release.
  • Stable: the final stable release.

Production environments usually need to use pre-release versions with caution, unless the project explicitly needs to validate new features in advance.

3. Version Constraints

Dependency declarations usually do not simply specify a single version; they can also set an allowed version range.

Common syntax:

Syntax Common meaning
1.2.3 Exact version; specific behavior depends on the tool
>=1.2.3 At least the specified version
<2.0.0 Below the specified version
^1.2.3 Allows updates within the compatible range
~1.2.3 Usually limited to a smaller update range
* Matches a broad range

For example:

{
  "dependencies": {
    "axios": "^1.7.0",
    "lodash": "~4.17.0"
  }
}

Under npm's semantic versioning rules:

  • ^1.7.0 usually allows updating to a compatible 1.x version.
  • ~4.17.0 usually allows updating to a 4.17.x patch version.

Version constraint syntax and resolution rules vary by package manager; the rules of one tool cannot be directly applied to other ecosystems.

4. Version Conflicts

Dependency conflicts can occur when different dependencies impose incompatible version requirements on the same package.

For example:

flowchart TD
    A["Project"] --> B["Component A"]
    A --> C["Component B"]
    B --> D["Shared Library 1.x"]
    C --> E["Shared Library 2.x"]
    D --> F["Inconsistent Version Constraints"]
    E --> F
    F --> G["Dependency Resolution Failure or Runtime Risk"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style E fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style G fill:#FBE9E9,stroke:#D9AAAA,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

Dependency conflicts may manifest as:

  • Unable to find a version that satisfies all constraints.
  • Type or symbol incompatibility during compilation.
  • Class or method not found at runtime.
  • Behavioral differences between versions.
  • Plugin incompatible with the host program.

Resolution methods include unifying versions, upgrading upstream dependencies, excluding conflicting dependencies, using compatible versions, or running different versions through isolation mechanisms.

V. Dependency Resolution Mechanism

Dependency resolution is one of the core functions of a package manager.

When a project declares multiple dependencies, the package manager needs to find a set of concrete package versions that satisfy the constraints.

1. Inputs to Dependency Resolution

Dependency resolution usually needs to consider:

  • Direct dependencies of the project.
  • Version constraints of each dependency.
  • Transitive dependency declarations.
  • Available versions in package repositories.
  • Operating system and processor architecture.
  • Optional features and platform conditions.
  • Compatibility rules between dependencies.

2. Basic Process of Dependency Resolution

flowchart TD
    A["Read Project Dependency Declarations"] --> B["Fetch Candidate Package Versions"]
    B --> C["Read Transitive Dependencies"]
    C --> D["Build Dependency Constraints"]
    D --> E["Select Candidate Version"]
    E --> F{"Are All Constraints Satisfied?"}
    F -->|"Yes"| G["Generate Dependency Resolution Result"]
    F -->|"No"| H["Try Other Version Combinations"]
    H --> I{"Is There a Feasible Solution?"}
    I -->|"Yes"| E
    I -->|"No"| J["Report Dependency Conflict"]
    G --> K["Download and Install"]

    style A fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style D fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style G fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style H fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style J fill:#FBE9E9,stroke:#D9AAAA,color:#334155
    style K fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

This is the dependency resolution process at a conceptual level. Different package managers use different algorithms and may adopt backtracking search, constraint solving, version-first strategies, or other algorithms.

3. Version Selection Strategies

Package managers may adopt different version selection strategies:

Latest Compatible Version

Choose a newer version while satisfying the constraints.

Minimal Version Selection

Prefer lower versions that satisfy the requirements to reduce unnecessary upgrades.

Lockfile Priority

When a lockfile already exists, prefer the concrete versions already resolved in the lockfile.

Unified Version Strategy

Try to use a consistent version of the same dependency throughout the project.

Multiple Versions Coexisting

In ecosystems that allow it, the same project can use multiple versions of a package at the same time.

Strategies differ among package managers, and you cannot assume that all tools will automatically choose the latest version.

VI. Lockfiles and Reproducible Builds

Lockfiles are an important part of modern dependency management systems.

Dependency declarations usually describe the range of allowed versions, while a lockfile records the specific versions ultimately determined by a particular dependency resolution.

1. Why Are Lockfiles Needed?

Suppose the project declares:

{
  "dependencies": {
    "axios": "^1.7.0"
  }
}

When installing at different times, different compatible versions may be resolved.

For example:

  • When developer A installs, version 1.7.2 is resolved.
  • When developer B installs, version 1.8.0 is resolved.

Although both versions satisfy the constraint, they may behave differently.

A lock file can record exact versions, reducing this discrepancy.

2. Common Lock Files

Ecosystem Lock File
Python poetry.lock, uv.lock
JavaScript package-lock.json, pnpm-lock.yaml, yarn.lock
Rust Cargo.lock
Go go.sum
Java Maven and Gradle can use mechanisms such as dependency locking

Note especially: go.sum mainly records checksum information for module contents and is not equivalent to npm's package-lock.json. Go's version selection also depends on module information such as go.mod.

3. Main Purposes of Lock Files

  • Pin dependency resolution results.
  • Reduce dependency differences between development and production environments.
  • Improve the consistency of continuous integration builds.
  • Facilitate dependency vulnerability audits.
  • Record dependency upgrade changes.
  • Support troubleshooting build issues.

4. Reproducible Build

Reproducible builds emphasize obtaining consistent build results under controlled conditions using the same source code, dependencies, and build environment.

A lock file is only an important part of achieving reproducible builds.

Other factors also need to be considered:

  • Compiler version.
  • Operating system and architecture.
  • Build tool version.
  • Environment variables.
  • System-level dependencies.
  • Timestamps and randomness.
  • Build scripts and external resources.

A fully reproducible build cannot rely on a lock file alone.