I. What Is an ABI?
ABI stands for Application Binary Interface.
It defines how compiled programs interact with other program modules, runtime environments, and operating systems at the binary level.
If API (Application Programming Interface) describes the interface at the source code level, then ABI describes the interface at the machine code level.
For example, two C source files can each be compiled into object files and then combined by the linker into an executable program. They can work together correctly not only because the function names match, but also because both sides must follow the same ABI rules.
1. Differences Between API and ABI
| Comparison Dimension | API | ABI |
|---|---|---|
| Full name | Application Programming Interface | Application Binary Interface |
| Level | Source code level | Binary level |
| Intended for | Developers, compilers | Compilers, linkers, runtimes |
| Main contents | Function declarations, classes, parameter types | Calling conventions, data layout, symbol rules |
| Dependencies | Programming languages and library interfaces | CPU architecture, operating system, toolchain |
| Compatibility | Source-code compatibility | Binary compatibility |
For example:
int add(int a, int b);
This is a function declaration at the API level.
But the compiler must also determine:
- Where are parameters
aandbplaced—in which registers or stack locations? - Which register holds the return value?
- Does the caller or the callee restore the stack space?
- Which registers may the function modify?
- How are function symbols named and exported?
These questions fall within the scope of ABI.
2. Core Components of ABI
ABI is not just a set of function calling rules; it is a set of conventions at the binary level.
It mainly includes:
| Component | Role |
|---|---|
| Calling Convention | Specifies how parameters, return values, and registers are used |
| Data type layout | Specifies the size, alignment, and representation of primitive types |
| Structure layout | Specifies field ordering, padding, and alignment rules |
| Name Mangling | Specifies the binary symbol names of functions and variables |
| Stack frame layout | Specifies how stack space is organized during function calls |
| Register usage rules | Specifies which registers are saved by the caller or callee |
| Exception handling and unwinding mechanisms | Specifies binary collaboration rules for exception propagation, stack unwinding, and related matters |
| Object and runtime conventions | Specifies the binary representation and interaction of runtime objects for some languages |
Note: ABI specifications differ across platforms. Even with the same CPU instruction set, different operating systems and toolchains may use different ABIs.
II. What Is a Calling Convention?
A Calling Convention is an important part of ABI.
It specifies how the two parties in a function call cooperate to complete a function call. It mainly involves:
- How function parameters are passed.
- How return values are passed.
- Which registers may be modified.
- Which registers must be saved and restored.
- Who is responsible for cleaning up the stack space used during the call.
- How the stack pointer and stack frame are maintained.
The core goal of a calling convention is to make the Caller and Callee form consistent expectations about the function call process.
1. What Roles Are Involved in a Function Call?
flowchart TB
A["Caller"] --> B["Prepare parameters according to the calling convention"]
B --> C["Execute the function call instruction"]
C --> D["Callee"]
D --> E["Execute the function logic"]
E --> F["Prepare the return value"]
F --> G["Return to the caller"]
G --> H["Caller continues execution"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
style C fill:#FFF5DE,stroke:#E8D29A,color:#334155
style D fill:#F3EDFC,stroke:#C4B2E3,color:#334155
style E fill:#E5F4EC,stroke:#A3CBB5,color:#334155
style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#E8EDF2,stroke:#B8C3CE,color:#334155
linkStyle default stroke:#A3B8AC,stroke-width:1.5px
A calling convention does not specify what a function actually does; rather, it specifies how the two parties to a function call exchange data and maintain the execution environment.
2. Parameter Passing Methods
Function arguments are usually passed through registers or the stack.
Register-based parameter passing
Place function arguments in designated registers.
Advantages:
- Fast access.
- No additional stack memory reads or writes are required.
- Suitable for common calls with a small number of arguments.
Stack-based parameter passing
Write function arguments into stack space prepared by the caller, and the callee reads them according to the convention.
Advantages:
- Can support more arguments.
- Not directly limited by the number of available argument registers.
- Suitable for some complex argument layouts.
In practice, calling conventions usually use a mix of registers and the stack rather than choosing only one.
3. Return Value Passing
Common ways of passing return values include:
- Register return: simple return values such as integers and pointers are usually returned through a designated register.
- Multi-register return: some platforms allow larger simple aggregate values to be returned through multiple registers.
- Implicit return pointer: more complex or larger structures may be returned through a memory address provided by the caller.
For example, in the common x86-64 ABI, integer return values usually use the RAX register.
However, this does not mean that all types and all platforms use the same return method.
3. Register Preservation Rules
Register preservation rules are a very important part of a calling convention.
During a function call, both the caller and the callee may use CPU registers. To prevent function calls from corrupting data that other code depends on, the responsibility for preserving registers must be specified.
They are usually divided into two categories.
1. Caller-saved: Saved by the Caller
Also called volatile registers.
If the caller wants the value in a register to remain valid after the function call, it must save it itself before the call.
The callee may freely modify these registers.
For example:
Caller:
Place important data in register R10
Save the original value of R10
Call the function
Callee:
May modify R10
Caller:
Function returns
Restore the original value of R10
The core meaning of caller-saved is that values that need to be preserved across a function call are the caller's responsibility to protect.
2. Callee-saved: Saved by the Callee
Also called non-volatile registers.
If the callee needs to use these registers, it must first save their original values and restore them before returning.
For example:
Caller:
Call the function
Callee:
Save the original value of RBX
Use RBX to perform calculations
Restore the original value of RBX
Return
Caller:
Can continue using the original RBX value
3. Comparison of the Two Rules
| Aspect | Caller-saved | Callee-saved |
|---|---|---|
| Responsibility for saving | Caller | Callee |
| Can the callee modify it? | Yes | Yes, but the original value must be restored |
| When to save | Before the call | Before the callee uses it |
| When to restore | After the call returns | Before the callee returns |
| Typical use | Temporary calculation data | Data that must be preserved across calls |
Note:
Caller-saved and callee-saved do not indicate whether a register can be modified; rather, they specify who is responsible for ensuring that the original value is preserved after modification.
4. Stack Frames and Function Calls
A stack frame is a segment of stack space used during the execution of a function.
During a function call, the program may need to allocate space for local variables, temporary data, saved register values, and so on.
A typical stack frame may contain:
- Return address.
- Saved registers.
- Local variables.
- Temporary calculation space.
- Some arguments passed on the stack.
The exact layout depends on the ABI, compiler optimizations, and the function implementation.
1. How a Stack Frame Is Formed
Taking a common x86-64 function call as an example, a function call may involve the following process:
flowchart TB
A["Caller prepares arguments"] --> B["Execute call instruction"]
B --> C["Save return address"]
C --> D["Enter called function"]
D --> E["Set up stack frame or adjust stack space"]
E --> F["Save registers that need to be preserved"]
F --> G["Execute function logic"]
G --> H["Restore registers and stack space"]
H --> I["Execute ret instruction"]
I --> J["Return to caller"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#FFF5DE,stroke:#E8D29A,color:#334155
style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#E5F4EC,stroke:#A3CBB5,color:#334155
style F fill:#E5F4EC,stroke:#A3CBB5,color:#334155
style G fill:#F3EDFC,stroke:#C4B2E3,color:#334155
style H fill:#E5F4EC,stroke:#A3CBB5,color:#334155
style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
linkStyle default stroke:#A3B8AC,stroke-width:1.5px
This is a typical flow that is easy to understand; not every function must execute every step.
For example:
- The compiler may omit the frame pointer.
- Leaf functions may not need additional stack space.
- Inline functions may not produce an actual function call.
- Optimized functions may keep all local variables in registers.
2. x86-64 Stack Frame Diagram
In the common stack that grows toward lower addresses, the stack space can be abstractly represented as:
High address
┌────────────────────────────────┐
│ Caller stack arguments │
├────────────────────────────────┤
│ Return address │
├────────────────────────────────┤
│ Saved frame pointer (optional) │
├────────────────────────────────┤
│ Saved registers │
├────────────────────────────────┤
│ Local variables │
├────────────────────────────────┤
│ Temp space / alignment padding │
└────────────────────────────────┘
Low address
This is a conceptual layout and does not represent a fixed memory order. The actual stack frame may also be affected by argument allocation strategies, stack alignment, compiler optimizations, and security mechanisms.
3. Stack Alignment
The CPU or ABI usually requires the stack pointer to satisfy specific alignment constraints.
For example, the System V AMD64 ABI specifies a 16-byte stack alignment requirement for ordinary function calls. After the call instruction pushes the return address, the stack pointer at the function entry has a corresponding relationship with the alignment state before the call.
The significance of stack alignment includes:
- Satisfying the alignment requirements of SIMD instructions for memory access.
- Improving the efficiency of certain data accesses.
- Ensuring consistent stack layout across different compiled modules.
- Meeting the ABI requirements for function call entry.
If stack alignment conventions are not followed when writing assembly by hand, some instructions may execute abnormally, or the called function may encounter problems that are difficult to diagnose.
V. Calling Conventions on Mainstream Platforms
Calling conventions are closely related to the CPU architecture, operating system, and ABI.
Even within x86-64, different platforms have different calling conventions.
1. System V AMD64 ABI
The System V AMD64 ABI is a common basis for calling conventions in x86-64 environments that adopt the System V ABI, such as Linux and macOS.
The first six integer and pointer arguments usually use the following registers:
| Parameter order | Register |
|---|---|
| 1st | RDI |
| 2nd | RSI |
| 3rd | RDX |
| 4th | RCX |
| 5th | R8 |
| 6th | R9 |
| Subsequent arguments | Usually passed on the stack |
Floating-point arguments are usually passed using XMM registers.
Common integer return values use RAX.
Some other rules:
RBX,RBP, andR12–R15are usually callee-saved registers.RAX,RCX,RDX,RSI,RDI, andR8–R11are usually caller-saved registers.RSPmust be maintained in accordance with the ABI.- Function calls must satisfy the specified stack alignment requirements.
Note that floating-point arguments and integer arguments follow different register classification rules; aggregate types such as structs must also go through the ABI's argument classification algorithm, and the table above cannot simply be applied based on the number of arguments.
2. Microsoft x64 Calling Convention
The Windows x64 platform mainly uses the Microsoft x64 calling convention.
The first four arguments of integer and pointer types usually use:
| Argument Order | Register |
|---|---|
| 1st | RCX |
| 2nd | RDX |
| 3rd | R8 |
| 4th | R9 |
| Subsequent arguments | Usually passed on the stack |
Floating-point arguments usually use the XMM0–XMM3 registers corresponding to the argument positions.
Integer return values commonly use RAX.
Windows x64 also has an important characteristic: the caller needs to reserve 32 bytes of Shadow Space for the callee.
Even if the first four arguments are passed in registers, the caller still needs to reserve room for this space.
3. Comparison of the Two Calling Conventions
| Comparison Dimension | System V AMD64 | Microsoft x64 |
|---|---|---|
| Main platforms | Linux, macOS and related environments | Windows x64 |
| Integer argument registers | RDI, RSI, RDX, RCX, R8, R9 | RCX, RDX, R8, R9 |
| Number of integer register arguments | 6 | 4 |
| Floating-point argument registers | XMM registers, allocated by argument classification | XMM0–XMM3, allocated by argument position |
| Shadow space | No corresponding fixed 32-byte requirement | 32-byte Shadow Space |
| Integer return value | RAX | RAX |
| Stack alignment | Follows the System V ABI specification | Follows the Microsoft x64 ABI specification |
Although both run on the x86-64 instruction set, their argument register allocation, stack space usage, and register preservation rules are not the same.
Therefore, assembly call code from one platform cannot be used directly on another.
6. How Do Different Programming Languages Handle Calling Conventions?
Different languages implement function calls through compilers, runtimes, and ABI specifications.
1. C Language
C typically uses the default calling convention specified by the target platform.
For example:
int add(int a, int b) {
return a + b;
}
In an x86-64 Linux environment, the compiler usually follows the System V AMD64 ABI.
In a Windows x64 environment, it usually follows the Microsoft x64 calling convention.
Therefore, the same C code, when compiled on different platforms, may produce machine code that uses different argument registers and stack layouts.
2. C++ Language
In addition to calling conventions, C++ also involves name mangling.
For example:
int add(int a, int b);
double add(double a, double b);
C++ supports function overloading, so the compiler needs to encode information such as function names and parameter types into symbol names, so that the linker can distinguish between different functions.
This means:
- The function name in the source code is not necessarily the same as the binary symbol name.
- Different compilers may use different name mangling rules.
- Even if the calling conventions are the same, differing name mangling rules can still cause linking to fail.
If you need a C++ function to be exported with C linkage rules, you can use:
extern "C" {
int add(int a, int b);
}
extern "C" mainly affects language linkage and name mangling rules; it does not mean that the platform ABI, data layout, and calling conventions can be ignored.
3. Rust Language
Rust uses its own Rust ABI by default.
The Rust ABI does not promise to be as stable as the C ABI for long-term use by external binary modules, so cross-language interfaces usually explicitly specify the C ABI:
#[no_mangle]
pub extern "C" fn add(a: i32, b: i32) -> i32 {
a + b
}
Here:
extern "C": Specifies the use of the target platform's C calling convention.#[no_mangle]: Requires that Rust's default symbol name mangling not be performed when exporting.pub: Controls visibility at the Rust level.
Note that the specific syntax for symbol attributes may differ between Rust versions and editions; in newer Rust versions, some scenarios require #[unsafe(no_mangle)].
4. High-Level Languages Such as Python and Java
Function calls in high-level languages do not necessarily correspond directly to C ABI function calls.
For example:
- Python function calls are usually executed by the interpreter or runtime and involve object references, argument binding, and runtime management.
- Java method calls generally follow the execution mechanisms defined by the JVM specification and, after JIT compilation, are mapped to the machine-level calling rules of the target platform.
- The Java Native Interface (JNI) defines the interface conventions for interaction between Java and native code.
High-level language runtimes can use their own calling mechanisms internally, but when interacting with the operating system, dynamic libraries, or other languages, they usually need to follow explicit binary interface conventions.
VII. ABI Compatibility and Cross-Language Calls
ABI compatibility determines whether different compiled modules can correctly cooperate at the binary level.
Even if two modules use different programming languages, as long as they follow compatible ABIs and have consistent data layouts, symbols, and runtime conventions, cross-language calls can be achieved.