1. What Is Model Quantization?
Model quantization is the technique of mapping the high-precision numerical values in a model to low-precision values.
In deep learning models, weights, activations, and intermediate computation results are typically represented using FP32, FP16, or BF16.
For example, a single FP32 parameter occupies 4 bytes:
FP32: 0.123456789
FP16: 0.1235
INT8: 31
INT4: 7
The core goals of quantization are:
- Reduce the storage space for model weights.
- Reduce GPU memory usage.
- Lower the bandwidth required to read weights.
- Use low-precision hardware instructions to increase computational throughput.
- Reduce model deployment costs.
- Improve inference efficiency within an acceptable range of accuracy loss.
To be clear: quantization is not just about shrinking data types; it is a systems engineering effort involving numerical mapping, error control, compute kernels, and hardware execution.
1.1 Why Do Large Models Need Quantization?
Large models have a very large number of parameters, and model weights are typically a major source of memory overhead in inference deployment.
Suppose a model has 70B parameters:
| Data Type | Theoretical Storage per Parameter | Theoretical Weight Footprint |
|---|---|---|
| FP32 | 4 Bytes | 280 GB |
| FP16 | 2 Bytes | 140 GB |
| BF16 | 2 Bytes | 140 GB |
| FP8 | 1 Byte | 70 GB |
| INT8 | 1 Byte | 70 GB |
| INT4 | 0.5 Byte | 35 GB |
The figures above are theoretical weight sizes in decimal units, excluding overheads such as quantization Scale, Zero Point, alignment padding, runtime workspace, and KV Cache.
As can be seen from the table:
- Compared with FP32, FP16 reduces weight storage by about 50%.
- Compared with FP16, INT8 reduces theoretical weight storage by about 50%.
- Compared with FP16, INT4 reduces theoretical weight storage by about 75%.
This means a model that originally required multiple GPUs to deploy may, after quantization, fit into fewer GPUs — or even run on a single card.
1.2 The Three Main Targets of Quantization
Model quantization typically involves three types of data.
| Target | Meaning | Purpose of Quantization |
|---|---|---|
| Weight Quantization | Weight quantization | Reduce model parameter storage |
| Activation Quantization | Activation quantization | Reduce the storage and compute overhead of intermediate tensors |
| KV Cache Quantization | KV Cache quantization | Reduce cache memory for long-context inference |
The three differ in quantization difficulty.
Weights are usually fixed before model deployment, so their distribution can be measured in advance and quantization completed ahead of time.
Activations depend on the input data, and their distribution may vary from request to request, requiring calibration data, dynamic statistics, or special quantization methods.
KV Cache, meanwhile, keeps growing throughout the generation process, and its quantization must also account for Attention's sensitivity to the precision of historical Keys and Values.
2. Numerical Precision Fundamentals: FP32, FP16, BF16, FP8, and INT
To understand quantization, you first need to understand how different numerical formats are represented.
2.1 The Basic Structure of Floating-Point Numbers
An IEEE 754 floating-point number usually consists of three parts:
- Sign: the sign bit.
- Exponent: the exponent bits.
- Mantissa: the mantissa bits, also called the Fraction.
A floating-point number can be abstractly expressed as:
where:
- s: the sign bit.
- e: the exponent encoding.
- bias: the exponent offset.
- m: the mantissa part.
The exponent bits determine the numerical range, and the mantissa bits determine the numerical precision.
The more exponent bits there are, the larger the dynamic range that can typically be represented.
The more mantissa bits there are, the finer the numerical variation that can typically be represented.
2.2 Common Floating-Point Formats
| Format | Total bits | Exponent bits | Mantissa bits | Characteristics |
|---|---|---|---|---|
| FP32 | 32 | 8 | 23 | High precision, relatively large range |
| FP16 | 16 | 5 | 10 | Small storage footprint, but a relatively narrow dynamic range |
| BF16 | 16 | 8 | 7 | Dynamic range close to FP32 |
| FP8 E4M3 | 8 | 4 | 3 | Relatively high precision, smaller range |
| FP8 E5M2 | 8 | 5 | 2 | Larger range, lower precision |
FP8 has two common formats:
- E4M3: 4 exponent bits and 3 mantissa bits; generally better suited to computations with higher precision requirements.
- E5M2: 5 exponent bits and 2 mantissa bits; generally has a larger dynamic range.
Actual FP8 formats also have special encodings, exponent biases, and finite-value conventions; the full numerical range cannot be inferred from the exponent and mantissa bits alone.
2.3 Integer Formats
INT8 and INT4 are fixed-point integer representations.
| Format | Bits | Signed integer range |
|---|---|---|
| INT8 | 8 | -128 to 127 |
| INT4 | 4 | -8 to 7 |
| INT2 | 2 | -2 to 1 |
Integers themselves have no floating-point exponent or mantissa.
They can only represent a finite number of discrete values, so floating-point numbers need to be mapped into the integer space using Scale and Zero Point.
2.4 Differences Between FP16 and INT8
FP16 and INT8 are both low-precision representations of 16 bits or less, but their numerical structures are completely different.
FP16:
- Uses floating-point representation.
- Has exponent and mantissa bits.
- Can represent values of different orders of magnitude.
- Can usually participate directly in floating-point matrix operations.
INT8:
- Uses integer representation.
- Requires Scale and Zero Point to establish numerical mapping.
- Has a limited numerical range.
- Usually requires integer multiply-add or quantized compute kernels.
Therefore, INT8 is not simply FP16 truncated to 8 bits.
3. Mathematical Principles of Linear Quantization
Linear quantization is the most basic method for integer weight quantization such as INT8 and INT4.
It maps floating-point numbers to a finite range of integers through a linear mapping.
3.1 Quantization: Quantization Formula
The most common affine quantization (Affine Quantization) can be expressed as:
where:
- x: original floating-point number.
- q: quantized integer.
- s: Scale, the scaling factor.
- z: Zero Point, the zero-point offset.
- q_{\min}: minimum integer value.
- q_{\max}: maximum integer value.
round: rounding operation.clip: clips to the valid integer range.
Dequantization formula:
where \hat{x} is the approximate floating-point value recovered after quantization.
Note: Dequantization cannot recover the original values; it can only recover an approximation of the quantized values.
3.2 Calculating the Scale
For asymmetric quantization, if the floating-point data range is:
and the integer range is:
the Scale is usually calculated as:
Zero Point:
Practical implementations usually also clamp the Zero Point to the integer range.
3.3 Symmetric Quantization
Symmetric Quantization usually sets the Zero Point to 0.
For signed integers:
Quantization formula:
Dequantization:
Advantages of symmetric quantization:
- No additional Zero Point is needed.
- Integer matrix multiplication is easier to optimize.
- The computation process is relatively simple.
- It suits many weight tensors.
However, if the data distribution clearly deviates from zero, symmetric quantization may not fully utilize the integer representation range.
3.4 Quantization Calculation Example
Suppose the weight range is:
Using symmetric INT8 quantization, take:
Then:
Original weight:
Quantization result:
Dequantization:
Quantization error:
This is the numerical error introduced by quantization.
3.5 Where Does Quantization Error Come From?
The main sources include:
- Rounding error: error produced when floating-point numbers are mapped to discrete integers.
- Clipping error: values outside the quantization range are clipped.
- Parameter estimation error: the Scale and Zero Point are not accurate enough.
- Distribution shift: the calibration data and the real inference data are inconsistent.
- Accumulated error: errors from multi-layer quantization propagate layer by layer.
Among these, clipping error is usually more severe than ordinary rounding error.
For example, if a weight is 12 but the quantization range can only represent up to 1, then that value will be clipped, and after dequantization it can still only yield a value close to 1.
4. Quantization Granularity: Per-Tensor, Per-Channel, and Group-wise
Quantization requires not only choosing INT8 or INT4 but also deciding the scope over which the Scale applies.
Quantization Granularity directly affects quantization error, metadata overhead, and computational efficiency.
4.1 Per-Tensor Quantization
Per-Tensor means the entire tensor shares one Scale and Zero Point.
For example, for a weight matrix:
all elements share the same set of quantization parameters.
Advantages:
- Lowest parameter overhead.
- Simple implementation.
- Easy to use high-performance matrix computation kernels.
Disadvantages:
- The value distributions of different channels may differ greatly.
- Extreme values may compress the effective quantization range of other elements.
- Precision loss may be significant under INT4.
4.2 Per-Channel Quantization
Per-Channel computes the Scale independently for each output channel.
For a linear layer:
Suppose:
where:
- N: output dimension.
- K: input dimension.
An independent Scale can be assigned to each output channel:
This better adapts to the weight distributions of different output channels.
Advantages:
- Usually more accurate than Per-Tensor.
- Reduces the error caused by differences in value distributions across channels.
- Suitable for many INT8 weight quantization scenarios.
Disadvantages:
- More quantization parameters must be stored.
- The computation kernel must support channel-level Scales.
- Different matrix layouts affect how the Scale is applied.
4.3 Group-wise Quantization
Group-wise Quantization divides a channel or matrix into multiple small groups, and each group computes its Scale independently.
For example:
A row of the weight matrix:
Group 1: 128 weights, sharing Scale 1
Group 2: 128 weights, sharing Scale 2
Group 3: 128 weights, sharing Scale 3
Group 4: 128 weights, sharing Scale 4
Common Group Sizes:
- 32.
- 64.
- 128.
- 256.
The specific size depends on the quantization algorithm, model structure, and compute kernel.
The smaller the Group Size:
- The Scale is closer to the local data distribution.
- Quantization error may be lower.
- The amount of metadata is larger.
- Kernel reads and computation management are more complex.
The larger the Group Size:
- Scale overhead is lower.
- Computation implementation may be simpler.
- Distribution differences across local regions may increase quantization error.
4.4 Comparison of the Three Granularities
| Method | Scale Scope | Metadata Overhead | Accuracy Performance |
|---|---|---|---|
| Per-Tensor | Entire tensor | Lowest | Depends on overall distribution |
| Per-Channel | Each channel | Relatively low | Usually better |
| Group-wise | Each group | Medium | Flexible; can further reduce error |
4.5 Schematic Diagram of Quantization Granularity
flowchart TD
A["Original weight matrix"] --> B{"Choose quantization granularity"}
B --> C["Per-Tensor"]
B --> D["Per-Channel"]
B --> E["Group-wise"]
C --> F["Entire matrix shares Scale"]
D --> G["Independent Scale per output channel"]
E --> H["Independent Scale per group of weights"]
F --> I["Low metadata overhead"]
G --> J["Adapts to channel distribution differences"]
H --> K["Controls local quantization error"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
V. INT8 Quantization: 8-Bit Integer Computation
INT8 is a very common low-precision integer computation format today.
It uses 8-bit signed integers to represent values, and in theory each parameter occupies 1 Byte.
5.1 Basic Characteristics of INT8
- The integer range is usually -128 to 127.
- Weight storage takes about half as much as FP16.
- Integer matrix multiplication instructions can be used.
- A Scale is needed to restore the numerical scale.
- It can usually achieve model compression with relatively low precision loss.
INT8 is commonly used in model inference in two ways:
- Weights use INT8, and activations use FP16 or BF16.
- Both weights and activations use INT8.
The first is usually called W8A16.
The second is usually called W8A8.
5.2 W8A16: INT8 Weights, FP16 Activations
W8A16 means:
- Weight: INT8.
- Activation: FP16.
Model weights are stored in INT8. During matrix computation, depending on the compute kernel, the weights are converted or fused into a form suitable for computation.
Illustration:
Its advantages are:
- Weight VRAM usage is significantly reduced.
- Activations still retain higher precision.
- Lower quantization requirements for activation distributions.
- It is easier to achieve good deployment results on some GPUs.
But note: W8A16 does not guarantee that all hardware can directly execute native mixed matrix multiplication of INT8 weights and FP16 activations.
In practice, it may be implemented through dequantization, fused kernels, or specific mixed-precision kernels.
5.3 W8A8: INT8 Weights, INT8 Activations
W8A8 means:
- Weight: INT8.
- Activation: INT8.
Matrix multiplication:
Practical integer matrix multiplication usually uses a wider accumulation type, such as INT32.
The computation result needs to be combined with the input and weight scales to restore the numerical scale.
For symmetric quantization:
Therefore:
Here, the integer multiply-accumulate result is usually first accumulated into INT32 and then converted to the target output format through scaling.
Advantages of W8A8:
- Both weights and activations can be compressed.
- Reduces data transfer and storage overhead.
- May achieve higher throughput on hardware that supports INT8 Tensor Cores.
Challenges:
- Activation distributions change dynamically.
- Outliers may affect the Scale.
- Activation quantization error may affect model output.
- More complex quantization and calibration strategies are required.
5.4 Accumulation Precision of INT8
An easily overlooked issue is:
INT8 inputs do not mean the entire computation process uses INT8.
For example:
If a_k and w_k are both INT8, then an individual product can be handled in the integer compute path, but the accumulation process usually requires a wider integer type.
If INT8 is used for accumulation throughout, values can easily overflow.
Therefore, a common compute path is:
INT8 × INT8
↓
INT32 accumulation
↓
Scale restoration
↓
FP16 / BF16 / FP32 output
This design reduces the storage cost of input data while avoiding severe overflow problems caused by low-bit-width accumulation.
6. INT4 Quantization: 4-bit Weight Compression
INT4 uses 4 bits to represent an integer.
Its core value is further reducing model weight storage and read overhead.
6.1 Numeric Range of INT4
Signed INT4 typically represents:
Each parameter theoretically occupies only half a byte.
Since a byte has 8 bits, two INT4 values can be packed into one byte.
For example:
High 4 bits: Weight A
Low 4 bits: Weight B
This method is called Packing.
Actual quantization formats may use different encoding methods, such as symmetric signed encoding, asymmetric encoding, or custom low-bit-width formats.
6.2 Main Advantages of INT4
- Weight storage is only one quarter that of FP16.
- Theoretically significantly reduces GPU memory requirements.
- Reduces weight read bandwidth.
- Facilitates single-GPU deployment of large models.
- May provide significant benefits in scenarios where weight reading is the bottleneck.
6.3 Main Challenges of INT4
INT4 has only 16 discrete values.
This means the representable values are very limited.
For example:
If the entire tensor is mapped to the limited integer range of INT4, relatively large quantization steps are needed to represent different floating-point values.
Compared with INT8, INT4 usually has larger rounding error.
In addition, if a tensor contains a small number of extreme values, the quantization range may be dominated by these extreme values, causing most ordinary weights to use fewer effective quantization levels.
6.4 Group-wise INT4
To reduce INT4 quantization error, practical models usually adopt Group-wise Quantization.
For example:
- Every 128 weights form a group.
- The Scale is computed separately for each group.
- Different groups have independent quantization ranges.
This avoids a small number of outliers in the entire matrix affecting all weights.
A typical INT4 weight representation may include:
Quantized weights: INT4 Packed Values
Scaling parameters: FP16 / BF16 Scale
Zero-point parameters: Optional
Group size: Group Size
Therefore, the actual memory footprint of an INT4 model is not equal to the number of parameters multiplied by 0.5 byte.
Scale, Zero Point, alignment, and other metadata overhead also need to be accounted for.
6.5 INT4 Dequantization Process
Suppose a group of weights uses symmetric INT4 quantization:
Dequantization:
In actual GPU inference, it is not necessarily required to first fully dequantize the entire weight matrix into FP16.
A more efficient approach is:
- Read Packed INT4 from GPU memory.
- Unpack it inside the kernel.
- Fetch the corresponding Group Scale.
- Convert in registers or along the compute path.
- Perform matrix multiplication.
- Output the accumulation result.
This approach avoids creating an additional full FP16 weight copy.
7. Comparison of INT8 and INT4
Both INT8 and INT4 are forms of integer weight quantization, but they differ significantly in precision, storage, and computation methods.
| Comparison dimension | INT8 | INT4 |
|---|---|---|
| Bit width | 8 Bit | 4 Bit |
| Theoretical storage | 1 Byte/parameter | 0.5 Byte/parameter |
| Integer representation levels | 256 encoded values | 16 encoded values |
| Quantization error | Usually lower | Usually higher |
| Scale overhead | Relatively low | Requires attention for group-wise |
| Weight read volume | Lower | Even lower |
| Hardware support | Relatively broad | Depends on hardware and Kernel |
| Deployment complexity | Relatively low | Usually higher |
| Applicable scenarios | Balance between accuracy and performance | GPU memory-constrained, large-model compression |
7.1 INT4 Is Not Necessarily Faster Than INT8
This is a very important engineering misconception in model quantization.
INT4 weights have a smaller data volume, but that does not mean inference will necessarily be faster.
The actual speed depends on:
- Whether the GPU supports efficient INT4 computation.
- INT4 unpacking overhead.
- Scale read overhead.
- Whether dequantization is fused into matrix multiplication.
- Whether the Kernel fully utilizes Tensor Cores.
- Whether it is limited by memory bandwidth.
- Batch Size and matrix shape.
If the unpacking and dequantization overhead of the INT4 Kernel is large, it may not fully exploit the bandwidth advantage of lower bit width.
7.2 Choosing Between INT4 and INT8
Selection can usually be based on the following goals:
| Goal | Recommended approach |
|---|---|
| Minimize GPU memory usage | INT4 |
| Control quantization error as much as possible | INT8 |
| Deploy a larger model on a single GPU | INT4 |
| Higher requirements for output quality | INT8 or higher precision |
| Need a mature integer computation path | INT8 |
| GPU has efficient low-bit-width computation support | Evaluate INT4 |
Ultimately, quality evaluation and performance testing on real business datasets should be the deciding factors.
8. FP8: Low-Precision Floating-Point Computing
FP8 and INT8 both use 8-bit representation, but their computational logic is different.
FP8 retains a floating-point exponent and mantissa, and can represent values across different orders of magnitude.
INT8, by contrast, needs to be mapped to a finite integer range through Scale and Zero Point.
8.1 Characteristics of FP8
FP8 is mainly used to reduce the data precision and storage overhead of floating-point computation.
Common formats:
| Format | Structure | Key characteristics |
|---|---|---|
| E4M3 | 4-bit exponent, 3-bit mantissa | Relatively high precision |
| E5M2 | 5-bit exponent, 2-bit mantissa | Larger dynamic range |
Advantages of FP8:
- A single element occupies 1 Byte.
- It can preserve the exponent structure of floating-point numbers.
- It is suitable for certain matrix multiplications and deep learning computations.
- Higher throughput can be achieved on GPUs that support FP8 Tensor Cores.
8.2 Differences Between FP8 and INT8
| Dimension | FP8 | INT8 |
|---|---|---|
| Numeric type | Floating point | Integer |
| Exponent structure | Yes | No |
| Scale | Common implementations require scale management | Usually required |
| Zero Point | Typically does not use affine integer zero points | Can be used |
| Numeric range | Depends on the floating-point format | Depends on integer encoding and Scale |
| Main application | Low-precision floating-point computation | Integer computation for weights and activations |
| Hardware execution | FP8 Tensor Cores, etc. | INT8 Tensor Cores, etc. |
8.3 Scale Management for FP8
Although FP8 has a more flexible dynamic range than INT8, it is still limited.
If input values exceed the representable range, overflow or saturation will occur.
If values are too small, underflow and precision loss may also occur.
Therefore, practical computation usually requires a scaling factor:
After the computation:
For matrix multiplication, the inputs and outputs may have different scaling parameters.
FP8 training and inference systems may also use dynamic Scale, delayed Scale, or Scale management strategies based on historical statistics.
9. Weight Quantization, Activation Quantization, and KV Cache Quantization
Although all three types of quantization are low-precision optimizations, they act in different locations and present different engineering challenges.
9.1 Weight Quantization
Weight Quantization is the process of converting model parameters into a lower-precision format.
For example:
Original model:
W1: FP16
W2: FP16
W3: FP16
After quantization:
W1: INT4
W2: INT4
W3: INT4
Weight quantization is usually completed before model deployment.
Main benefits:
- Reduce model file size.
- Reduce GPU memory used for weights.
- Reduce weight read bandwidth.
Common approaches:
- INT8 Weight Quantization.
- INT4 Weight Quantization.
- GPTQ.
- AWQ.
- GGUF quantization format.
- BitsAndBytes low-bit weight quantization.
9.2 Activation Quantization
Activation Quantization is the process of converting intermediate computation results of the model into a lower-precision format.
For example:
Embedding
↓
Attention
↓
Activation Quantization
↓
FFN
↓
Activation Quantization
↓
Next layer
Activation values depend on the input data, model layer, and runtime state.
Therefore, activation quantization is usually more complex than static weight quantization.
Main challenges:
- Input distributions vary with requests.
- Numerical ranges differ significantly across layers.
- Outliers can degrade quantization accuracy.
- Calibration or dynamic scales are required.
- Quantization and dequantization operations may introduce additional overhead.
9.3 KV Cache Quantization
The KV Cache stores the Key and Value of historical tokens.
In long-context, high-concurrency inference, the KV Cache may consume a large amount of GPU memory.
Therefore, the KV Cache can be converted from FP16 or BF16 to FP8, INT8, or other low-precision formats.
Its main benefits are:
- Reduce cache memory usage.
- Increase the number of concurrent requests supported by the same GPU.
- Reduce read bandwidth for historical Keys and Values.
However, KV Cache quantization requires particular attention to Attention errors.
Historical Keys and Values participate in the Attention computation for every subsequent token, so quantization errors may continuously affect the generated results.
9.4 Comparison of the Three Types of Quantization
| Quantization Target | Quantization Timing | Main Benefit | Main Risk |
|---|---|---|---|
| Weight | Usually before deployment | Reduce weight memory | Model capability degradation |
| Activation | During execution or configured before deployment | Reduce intermediate tensor overhead | Distribution shifts and outliers |
| KV Cache | During inference | Reduce context cache usage | Accumulation of Attention errors |
10. Mainstream Quantization Algorithms: GPTQ, AWQ, SmoothQuant, and bitsandbytes
The differences among quantization algorithms mainly lie in how they determine quantization parameters, how they handle outliers, and how they reduce model output error.
10.1 GPTQ
GPTQ is a post-training quantization method for Transformer weights.
The core idea of GPTQ is that when quantizing weights, it considers not only the difference between the current weights and the quantized result, but also the impact of weight error on the model output.
It uses approximate second-order information to guide the quantization process.
Core Principle
For a linear layer:
The quantized weights are:
Output error:
If some directions of the input activations X are more important, then a weight error of the same magnitude may cause output changes of different degrees.
GPTQ uses calibration data to compute input-related statistics, approximately measuring the impact of weight error on the output.
A common objective can be written as:
where \|\cdot\|_F denotes the Frobenius norm.
GPTQ reduces this objective through approximate second-order optimization and a step-by-step quantization strategy.
Main Characteristics of GPTQ
- It belongs to Post-Training Quantization, PTQ.
- It usually does not require fully retraining the model.
- It relies on representative calibration data.
- It is commonly used for INT4 weight quantization.
- It can be combined with Group-wise Quantization.
Limitations of GPTQ
- The quantization process requires computing statistical matrices.
- The quantization stage may consume considerable CPU/GPU memory.
- The choice of calibration data affects the results.
- Inference speed depends on whether the corresponding Kernel is efficient.
10.2 AWQ
AWQ stands for Activation-aware Weight Quantization.
The core idea of AWQ is that the importance of weights is related to input activation values.
Different weights do not have the same impact on model output.
If the activation values of some input channels are large, the quantization error on the corresponding weights may be more significant.
AWQ uses activation statistics to identify important channels and protects the quantization accuracy of key weights through scaling transformations.
Core Idea
For a linear layer:
Scale the input channels:
Adjust the weights accordingly:
Therefore:
Under ideal mathematical conditions, the model computation results remain unchanged.
Then quantization is applied to the adjusted weights.
This transformation can change the numerical distribution of the weights, allowing the quantization process to better protect important channels.
Main Features of AWQ
- Uses activation statistics to guide weight quantization.
- Commonly used for low-bit weight quantization.
- Does not require full model fine-tuning.
- Commonly used for INT4 deployment.
- Suitable for combination with high-performance low-bit matrix computation kernels.
Limitations of AWQ
- Requires calibration data.
- Requires handling weight scaling and model architecture adaptation.
- Support varies across operators and model architectures.
- Final inference performance depends on the kernel implementation.
10.3 SmoothQuant
SmoothQuant mainly addresses the problem of outliers in activations.
In Transformers, activations may have a small number of channels with particularly large magnitudes.
If INT8 quantization is applied directly to activations, these outliers may reduce the effective quantization precision of most other values.
The core idea of SmoothQuant is to shift part of the quantization difficulty from activations to weights.
It adjusts the numerical distributions of inputs and weights through an equivalent scaling transformation.
For:
we can construct:
Thus:
Without changing the mathematical computation results, activations become easier to quantize.
Main Features of SmoothQuant
- Focuses on activation outliers.
- Commonly used for W8A8.
- Can improve INT8 activation quantization for Transformers.
- Reduces the dynamic quantization burden at inference through offline scaling.
10.4 bitsandbytes
bitsandbytes is a commonly used low-bit quantization library in the deep learning ecosystem.
Common capabilities include:
- 8-bit optimizer-related features.
- 8-bit model weight loading.
- 4-bit weight quantization.
- NF4.
- FP4.
- Integration with tools such as Transformers.
NF4 (NormalFloat4) is a 4-bit data representation designed for approximately normally distributed weights.
It is not ordinary uniform INT4.
NF4 designs quantization levels according to the target distribution, making the limited 4-bit codes better suited to represent weights of a specific distribution.
10.5 Comparison of Mainstream Algorithms
| Algorithm | Core Idea | Common Target | Requires Calibration Data |
|---|---|---|---|
| GPTQ | Uses approximate second-order information to reduce weight error | INT4 / INT8 weights | Usually required |
| AWQ | Protects weight channels based on activation importance | INT4 weights | Usually required |
| SmoothQuant | Shifts the impact of activation outliers to weights | W8A8 | Usually required |
| bitsandbytes | Provides low-bit data formats and computation support | INT8 / 4-bit | Depends on the specific scheme |
11. Quantization Calibration: Calibration
Quantization is not simply choosing INT8 or INT4 and then directly converting the model.
It is necessary to determine parameters such as data distribution, Scale, Zero Point, and outlier handling methods.
This process is usually called Calibration.
11.1 Why Is Calibration Needed?
Suppose the actual distribution of activations in a certain layer is:
If calibration incorrectly assumes:
then the quantization range will be severely wasted.
A large number of normal values can only use a small number of integer codes, reducing effective precision.
Conversely, if the calibration range is too narrow, large values in real inference data may be clipped.
11.2 Calibration Dataset
The calibration dataset should be as close as possible to real business inputs.
For example:
- General conversation.
- Code generation.
- Mathematical reasoning.
- Long context.
- Multilingual input.
- Structured data.
- Tool-call requests.
If the model is mainly used for code generation but is calibrated only with ordinary chat data, it may not sufficiently cover the real activation distribution.
11.3 Calibration Methods
Common statistical methods include:
Min-Max
Determine the quantization range based on the minimum and maximum values.
The advantage is simplicity.
The disadvantage is susceptibility to outliers.
Percentile
Use percentiles to determine the quantization range.
For example, ignore the extreme 0.1% or 1% of data.
The advantage is that it can reduce the impact of outliers.
The disadvantage is that extreme values may be truncated.
Histogram
Compute an activation histogram and select the quantization range based on the distribution.
This can more finely balance quantization error and clipping error.
MSE Optimization
Search for Scale to minimize the mean squared error between data before and after quantization.
This method directly optimizes reconstruction error, but it may not fully reflect the final task quality of the model.
11.4 Calibration Process
flowchart TD
A["Prepare pretrained model"] --> B["Prepare representative calibration data"]
B --> C["Run model forward pass"]
C --> D["Collect weight and activation statistics"]
D --> E["Analyze numerical distribution and outliers"]
E --> F["Determine quantization granularity"]
F --> G["Compute Scale and Zero Point"]
G --> H["Perform quantization conversion"]
H --> I["Evaluate quantization error"]
I --> J{"Does the error meet requirements?"}
J -->|"No"| K["Adjust calibration data or quantization strategy"]
K --> C
J -->|"Yes"| L["Save quantized model"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
12. PTQ and QAT: Post-Training Quantization and Quantization-Aware Training
Model quantization mainly falls into two approaches:
- PTQ: Post-Training Quantization (post-training quantization).
- QAT: Quantization-Aware Training (quantization-aware training).
12.1 PTQ: Post-Training Quantization
PTQ performs quantization after model training is complete.
Typical process:
- Load an already trained FP16 or BF16 model.
- Prepare calibration data.
- Collect weight or activation distributions.
- Compute quantization parameters.
- Convert weights and related computation paths.
- Evaluate model quality.
Advantages:
- No need to perform full model training again.
- Lower cost.
- Relatively simple deployment process.
- Suitable for quickly compressing existing models.
Disadvantages:
- Accuracy loss may be larger at low bit widths.
- Calibration data affects the results.
- Higher adaptation requirements for some model architectures.
GPTQ, AWQ, and SmoothQuant are all typical post-training quantization approaches.
12.2 QAT: Quantization-Aware Training
QAT simulates quantization error during model training or fine-tuning.
During training, the model can still use high-precision parameters for gradient updates, but the forward pass simulates low-precision quantization behavior.
A typical approach is to use Fake Quantization.
Fake Quantization does not necessarily actually store parameters as INT8; instead, it performs the following in the forward pass:
Where:
- Q: quantization.
- DQ: dequantization.
In this way, the training process can perceive quantization error and adapt to low-precision computation through parameter updates.
12.3 Straight-Through Estimator
The quantization function includes a rounding operation:
Strictly speaking, the rounding function is non-differentiable almost everywhere.
Therefore, QAT usually adopts the Straight-Through Estimator, or STE for short.
STE uses an approximate gradient during backpropagation, allowing gradients to pass through the quantization operation.
A simplified form:
In practice, implementations may set gradients to zero outside the quantization range, to prevent values outside the valid range from continuously affecting training.
12.4 Comparison of PTQ and QAT
| Comparison Dimension | PTQ | QAT |
|---|---|---|
| Execution timing | After training is complete | During training or fine-tuning |
| Requires training | Usually not required | Required |
| Computational cost | Relatively low | Relatively high |
| Adaptation to quantization error | Relies on calibration and algorithms | The model can learn to adapt |
| Ultra-low bit width | Requires refined algorithms | Has the potential for better accuracy |
| Deployment complexity | Relatively low | Training pipeline is more complex |