Back to series
· 9 min read

ABI and Calling Convention


I. What Is an ABI?

ABI stands for Application Binary Interface.

It defines how compiled programs interact with other program modules, runtime environments, and operating systems at the binary level.

If API (Application Programming Interface) describes the interface at the source code level, then ABI describes the interface at the machine code level.

For example, two C source files can each be compiled into object files and then combined by the linker into an executable program. They can work together correctly not only because the function names match, but also because both sides must follow the same ABI rules.

1. Differences Between API and ABI

Comparison Dimension API ABI
Full name Application Programming Interface Application Binary Interface
Level Source code level Binary level
Intended for Developers, compilers Compilers, linkers, runtimes
Main contents Function declarations, classes, parameter types Calling conventions, data layout, symbol rules
Dependencies Programming languages and library interfaces CPU architecture, operating system, toolchain
Compatibility Source-code compatibility Binary compatibility

For example:

int add(int a, int b);

This is a function declaration at the API level.

But the compiler must also determine:

  • Where are parameters a and b placed—in which registers or stack locations?
  • Which register holds the return value?
  • Does the caller or the callee restore the stack space?
  • Which registers may the function modify?
  • How are function symbols named and exported?

These questions fall within the scope of ABI.

2. Core Components of ABI

ABI is not just a set of function calling rules; it is a set of conventions at the binary level.

It mainly includes:

Component Role
Calling Convention Specifies how parameters, return values, and registers are used
Data type layout Specifies the size, alignment, and representation of primitive types
Structure layout Specifies field ordering, padding, and alignment rules
Name Mangling Specifies the binary symbol names of functions and variables
Stack frame layout Specifies how stack space is organized during function calls
Register usage rules Specifies which registers are saved by the caller or callee
Exception handling and unwinding mechanisms Specifies binary collaboration rules for exception propagation, stack unwinding, and related matters
Object and runtime conventions Specifies the binary representation and interaction of runtime objects for some languages

Note: ABI specifications differ across platforms. Even with the same CPU instruction set, different operating systems and toolchains may use different ABIs.

II. What Is a Calling Convention?

A Calling Convention is an important part of ABI.

It specifies how the two parties in a function call cooperate to complete a function call. It mainly involves:

  1. How function parameters are passed.
  2. How return values are passed.
  3. Which registers may be modified.
  4. Which registers must be saved and restored.
  5. Who is responsible for cleaning up the stack space used during the call.
  6. How the stack pointer and stack frame are maintained.

The core goal of a calling convention is to make the Caller and Callee form consistent expectations about the function call process.

1. What Roles Are Involved in a Function Call?

flowchart TB
    A["Caller"] --> B["Prepare parameters according to the calling convention"]
    B --> C["Execute the function call instruction"]
    C --> D["Callee"]
    D --> E["Execute the function logic"]
    E --> F["Prepare the return value"]
    F --> G["Return to the caller"]
    G --> H["Caller continues execution"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style C fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style D fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style E fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style H fill:#E8EDF2,stroke:#B8C3CE,color:#334155

    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

A calling convention does not specify what a function actually does; rather, it specifies how the two parties to a function call exchange data and maintain the execution environment.

2. Parameter Passing Methods

Function arguments are usually passed through registers or the stack.

Register-based parameter passing

Place function arguments in designated registers.

Advantages:

  • Fast access.
  • No additional stack memory reads or writes are required.
  • Suitable for common calls with a small number of arguments.

Stack-based parameter passing

Write function arguments into stack space prepared by the caller, and the callee reads them according to the convention.

Advantages:

  • Can support more arguments.
  • Not directly limited by the number of available argument registers.
  • Suitable for some complex argument layouts.

In practice, calling conventions usually use a mix of registers and the stack rather than choosing only one.

3. Return Value Passing

Common ways of passing return values include:

  • Register return: simple return values such as integers and pointers are usually returned through a designated register.
  • Multi-register return: some platforms allow larger simple aggregate values to be returned through multiple registers.
  • Implicit return pointer: more complex or larger structures may be returned through a memory address provided by the caller.

For example, in the common x86-64 ABI, integer return values usually use the RAX register.

However, this does not mean that all types and all platforms use the same return method.

3. Register Preservation Rules

Register preservation rules are a very important part of a calling convention.

During a function call, both the caller and the callee may use CPU registers. To prevent function calls from corrupting data that other code depends on, the responsibility for preserving registers must be specified.

They are usually divided into two categories.

1. Caller-saved: Saved by the Caller

Also called volatile registers.

If the caller wants the value in a register to remain valid after the function call, it must save it itself before the call.

The callee may freely modify these registers.

For example:

Caller:
    Place important data in register R10
    Save the original value of R10
    Call the function

Callee:
    May modify R10

Caller:
    Function returns
    Restore the original value of R10

The core meaning of caller-saved is that values that need to be preserved across a function call are the caller's responsibility to protect.

2. Callee-saved: Saved by the Callee

Also called non-volatile registers.

If the callee needs to use these registers, it must first save their original values and restore them before returning.

For example:

Caller:
    Call the function

Callee:
    Save the original value of RBX
    Use RBX to perform calculations
    Restore the original value of RBX
    Return

Caller:
    Can continue using the original RBX value

3. Comparison of the Two Rules

Aspect Caller-saved Callee-saved
Responsibility for saving Caller Callee
Can the callee modify it? Yes Yes, but the original value must be restored
When to save Before the call Before the callee uses it
When to restore After the call returns Before the callee returns
Typical use Temporary calculation data Data that must be preserved across calls

Note:

Caller-saved and callee-saved do not indicate whether a register can be modified; rather, they specify who is responsible for ensuring that the original value is preserved after modification.

4. Stack Frames and Function Calls

A stack frame is a segment of stack space used during the execution of a function.

During a function call, the program may need to allocate space for local variables, temporary data, saved register values, and so on.

A typical stack frame may contain:

  • Return address.
  • Saved registers.
  • Local variables.
  • Temporary calculation space.
  • Some arguments passed on the stack.

The exact layout depends on the ABI, compiler optimizations, and the function implementation.

1. How a Stack Frame Is Formed

Taking a common x86-64 function call as an example, a function call may involve the following process:

flowchart TB
    A["Caller prepares arguments"] --> B["Execute call instruction"]
    B --> C["Save return address"]
    C --> D["Enter called function"]
    D --> E["Set up stack frame or adjust stack space"]
    E --> F["Save registers that need to be preserved"]
    F --> G["Execute function logic"]
    G --> H["Restore registers and stack space"]
    H --> I["Execute ret instruction"]
    I --> J["Return to caller"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style F fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style G fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style H fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976

    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

This is a typical flow that is easy to understand; not every function must execute every step.

For example:

  • The compiler may omit the frame pointer.
  • Leaf functions may not need additional stack space.
  • Inline functions may not produce an actual function call.
  • Optimized functions may keep all local variables in registers.

2. x86-64 Stack Frame Diagram

In the common stack that grows toward lower addresses, the stack space can be abstractly represented as:

High address
┌────────────────────────────────┐
│ Caller stack arguments         │
├────────────────────────────────┤
│ Return address                 │
├────────────────────────────────┤
│ Saved frame pointer (optional) │
├────────────────────────────────┤
│ Saved registers                │
├────────────────────────────────┤
│ Local variables                │
├────────────────────────────────┤
│ Temp space / alignment padding │
└────────────────────────────────┘
Low address

This is a conceptual layout and does not represent a fixed memory order. The actual stack frame may also be affected by argument allocation strategies, stack alignment, compiler optimizations, and security mechanisms.

3. Stack Alignment

The CPU or ABI usually requires the stack pointer to satisfy specific alignment constraints.

For example, the System V AMD64 ABI specifies a 16-byte stack alignment requirement for ordinary function calls. After the call instruction pushes the return address, the stack pointer at the function entry has a corresponding relationship with the alignment state before the call.

The significance of stack alignment includes:

  • Satisfying the alignment requirements of SIMD instructions for memory access.
  • Improving the efficiency of certain data accesses.
  • Ensuring consistent stack layout across different compiled modules.
  • Meeting the ABI requirements for function call entry.

If stack alignment conventions are not followed when writing assembly by hand, some instructions may execute abnormally, or the called function may encounter problems that are difficult to diagnose.

V. Calling Conventions on Mainstream Platforms

Calling conventions are closely related to the CPU architecture, operating system, and ABI.

Even within x86-64, different platforms have different calling conventions.

1. System V AMD64 ABI

The System V AMD64 ABI is a common basis for calling conventions in x86-64 environments that adopt the System V ABI, such as Linux and macOS.

The first six integer and pointer arguments usually use the following registers:

Parameter order Register
1st RDI
2nd RSI
3rd RDX
4th RCX
5th R8
6th R9
Subsequent arguments Usually passed on the stack

Floating-point arguments are usually passed using XMM registers.

Common integer return values use RAX.

Some other rules:

  • RBX, RBP, and R12–R15 are usually callee-saved registers.
  • RAX, RCX, RDX, RSI, RDI, and R8–R11 are usually caller-saved registers.
  • RSP must be maintained in accordance with the ABI.
  • Function calls must satisfy the specified stack alignment requirements.

Note that floating-point arguments and integer arguments follow different register classification rules; aggregate types such as structs must also go through the ABI's argument classification algorithm, and the table above cannot simply be applied based on the number of arguments.

2. Microsoft x64 Calling Convention

The Windows x64 platform mainly uses the Microsoft x64 calling convention.

The first four arguments of integer and pointer types usually use:

Argument Order Register
1st RCX
2nd RDX
3rd R8
4th R9
Subsequent arguments Usually passed on the stack

Floating-point arguments usually use the XMM0–XMM3 registers corresponding to the argument positions.

Integer return values commonly use RAX.

Windows x64 also has an important characteristic: the caller needs to reserve 32 bytes of Shadow Space for the callee.

Even if the first four arguments are passed in registers, the caller still needs to reserve room for this space.

3. Comparison of the Two Calling Conventions

Comparison Dimension System V AMD64 Microsoft x64
Main platforms Linux, macOS and related environments Windows x64
Integer argument registers RDI, RSI, RDX, RCX, R8, R9 RCX, RDX, R8, R9
Number of integer register arguments 6 4
Floating-point argument registers XMM registers, allocated by argument classification XMM0–XMM3, allocated by argument position
Shadow space No corresponding fixed 32-byte requirement 32-byte Shadow Space
Integer return value RAX RAX
Stack alignment Follows the System V ABI specification Follows the Microsoft x64 ABI specification

Although both run on the x86-64 instruction set, their argument register allocation, stack space usage, and register preservation rules are not the same.

Therefore, assembly call code from one platform cannot be used directly on another.

6. How Do Different Programming Languages Handle Calling Conventions?

Different languages implement function calls through compilers, runtimes, and ABI specifications.

1. C Language

C typically uses the default calling convention specified by the target platform.

For example:

int add(int a, int b) {
    return a + b;
}

In an x86-64 Linux environment, the compiler usually follows the System V AMD64 ABI.

In a Windows x64 environment, it usually follows the Microsoft x64 calling convention.

Therefore, the same C code, when compiled on different platforms, may produce machine code that uses different argument registers and stack layouts.

2. C++ Language

In addition to calling conventions, C++ also involves name mangling.

For example:

int add(int a, int b);

double add(double a, double b);

C++ supports function overloading, so the compiler needs to encode information such as function names and parameter types into symbol names, so that the linker can distinguish between different functions.

This means:

  • The function name in the source code is not necessarily the same as the binary symbol name.
  • Different compilers may use different name mangling rules.
  • Even if the calling conventions are the same, differing name mangling rules can still cause linking to fail.

If you need a C++ function to be exported with C linkage rules, you can use:

extern "C" {
    int add(int a, int b);
}

extern "C" mainly affects language linkage and name mangling rules; it does not mean that the platform ABI, data layout, and calling conventions can be ignored.

3. Rust Language

Rust uses its own Rust ABI by default.

The Rust ABI does not promise to be as stable as the C ABI for long-term use by external binary modules, so cross-language interfaces usually explicitly specify the C ABI:

#[no_mangle]
pub extern "C" fn add(a: i32, b: i32) -> i32 {
    a + b
}

Here:

  • extern "C": Specifies the use of the target platform's C calling convention.
  • #[no_mangle]: Requires that Rust's default symbol name mangling not be performed when exporting.
  • pub: Controls visibility at the Rust level.

Note that the specific syntax for symbol attributes may differ between Rust versions and editions; in newer Rust versions, some scenarios require #[unsafe(no_mangle)].

4. High-Level Languages Such as Python and Java

Function calls in high-level languages do not necessarily correspond directly to C ABI function calls.

For example:

  • Python function calls are usually executed by the interpreter or runtime and involve object references, argument binding, and runtime management.
  • Java method calls generally follow the execution mechanisms defined by the JVM specification and, after JIT compilation, are mapped to the machine-level calling rules of the target platform.
  • The Java Native Interface (JNI) defines the interface conventions for interaction between Java and native code.

High-level language runtimes can use their own calling mechanisms internally, but when interacting with the operating system, dynamic libraries, or other languages, they usually need to follow explicit binary interface conventions.

VII. ABI Compatibility and Cross-Language Calls

ABI compatibility determines whether different compiled modules can correctly cooperate at the binary level.

Even if two modules use different programming languages, as long as they follow compatible ABIs and have consistent data layouts, symbols, and runtime conventions, cross-language calls can be achieved.