1. Why Do We Need to Optimize Large Models?
Large language models typically have enormous numbers of parameters. As model scale grows, training, inference, and deployment face increasingly high resource requirements.
The main challenges include:
- Gap between model capabilities and business requirements: General-purpose models are not necessarily familiar with domain-specific knowledge, business rules, and task formats.
- High training cost: Fully updating billions or even hundreds of billions of parameters requires substantial compute resources.
- High inference cost: Model parameters, activations, and KV Cache all consume GPU memory, and inference is also constrained by compute capacity and memory bandwidth.
- Constrained deployment environments: Edge devices, private servers, and mobile terminals may not be able to run models at their original scale.
- Response latency and throughput requirements: Production systems usually need to balance model quality, latency, concurrency, and cost.
Three important model optimization techniques have emerged around these issues:
| Technique | Core goal | Primary target of modification | Typical benefit |
|---|---|---|---|
| Fine-Tuning | Adapt the model to a specific task | Model parameters or adapter parameters | Improve task adaptability |
| Knowledge Distillation | Transfer the teacher model's capabilities to the student model | Training objectives and parameters of the student model | Obtain a smaller, easier-to-deploy model |
| Quantization | Reduce numerical representation precision | Numerical representations such as weights, activations, and cache | Reduce GPU memory, storage, and some computational costs |
These three are not substitutes for one another.
- Fine-tuning addresses how a model adapts to a specific task.
- Distillation addresses how to transfer the capabilities of an existing model to another model.
- Quantization addresses how to store and execute a model at lower numerical precision.
2. Large Model Fine-Tuning
2.1 What Is Fine-Tuning?
Fine-tuning means continuing training a pretrained model on data for a specific task or domain so that the model's parameters adapt to the new task requirements.
During pretraining, the model learns general language abilities, knowledge associations, and pattern representations from massive amounts of data.
During fine-tuning, the model continues to learn a specific data distribution, task requirements, and output format.
For example:
- General-purpose model: can answer questions in a wide range of domains.
- Legal-domain model: is more familiar with legal terminology, legal documents, and related task formats.
- Code model: is better at code generation, code completion, and program analysis.
- Enterprise internal model: better aligns with the company's business processes and output standards.
Fine-tuning does not train a model from scratch; instead, it uses existing model parameters as initialization.
2.2 Basic Principles of Fine-Tuning
Suppose the pretrained model parameters are:
Fine-tuning uses a specific dataset:
where:
- x_i: input data, such as an instruction or a question.
- y_i: target output, such as an answer or label.
- N: number of training samples.
The goal of fine-tuning is to minimize the loss on the training data:
where:
- f_\theta: the model whose parameters are \theta.
- L: the loss function.
- \theta^\*: the parameters obtained after fine-tuning.
For autoregressive language models, the common objective is still Next Token Prediction:
The model predicts the next token in the target sequence based on context, computes the difference between the prediction and the training target through the loss function, and then updates the parameters via backpropagation.
Note that different fine-tuning tasks may have different data formats, supervision signals, and loss designs, but the core idea is to use a new training objective to adjust model behavior.
2.3 Complete Fine-Tuning Process
flowchart TB
A["Select a pretrained model"] --> B["Prepare domain or task data"]
B --> C["Data cleaning and quality checks"]
C --> D["Construct training samples"]
D --> E["Configure fine-tuning strategy"]
E --> F["Model forward pass"]
F --> G["Compute training loss"]
G --> H["Backpropagation"]
H --> I["Update trainable parameters"]
I --> J{"Training condition met?"}
J -->|"No"| F
J -->|"Yes"| K["Save fine-tuned model"]
K --> L["Validation and evaluation"]
L --> M["Deployment and continuous monitoring"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
2.4 Full Fine-Tuning: Full Fine-Tuning
Full fine-tuning means updating all or nearly all trainable parameters of the model during training.
For a model with 7B parameters:
this means that about 7 billion parameters need to participate in training.
The basic process is:
- Load the pretrained model.
- Feed in training samples.
- Perform a forward pass.
- Compute the prediction loss.
- Compute gradients via backpropagation.
- Update model parameters with the optimizer.
Parameter updates can be abstracted as:
Where:
- \eta: learning rate.
- \nabla_\theta L: gradient of the loss function with respect to the model parameters.
Advantages:
- Model parameters can fully adapt to the new task.
- The representational capacity of different model layers can be adjusted.
- Suitable for scenarios with sufficient data, substantial task differences, and adequate resources.
Disadvantages:
- High GPU memory overhead during training.
- High computational cost.
- High cost of training and saving the full model.
- When data quality is poor, overfitting or catastrophic forgetting can easily occur.
Full fine-tuning does not necessarily outperform parameter-efficient fine-tuning. The actual results depend on data quality, task characteristics, model size, and training configuration.
2.5 Parameter-Efficient Fine-Tuning: PEFT
Parameter-Efficient Fine-Tuning (PEFT) is a class of methods for reducing fine-tuning costs.
Its core idea is:
Freeze most parameters of the pretrained model as much as possible, and train only a small number of newly added or selected parameters.
Common methods include:
| Method | Core idea | Main characteristics |
|---|---|---|
| LoRA | Uses low-rank matrices to represent weight updates | Few parameters; widely used |
| QLoRA | Freezes the quantized base model and trains via LoRA | Reduces fine-tuning memory requirements |
| Adapter | Adds small adapter modules to network layers | Can save adapters for different tasks |
| Prefix Tuning | Learns trainable continuous prefix representations | Does not directly update the main model |
| Prompt Tuning | Learns soft prompt vectors | Fewer trainable parameters |
| BitFit | Updates only bias parameters | Simple to implement, but limited in scope |
Core Principle of LoRA
The basic idea of LoRA (Low-Rank Adaptation) is not to directly update the original weight matrix, but to approximate the weight change using two low-rank matrices.
Suppose the original weight matrix is:
Full fine-tuning needs to learn a complete weight update matrix:
LoRA decomposes it into:
Where:
And:
The final weights are:
Where:
- W_0: the frozen pretrained weights.
- A, B: the low-rank matrices that need to be trained.
- r: the rank of the low-rank matrices.
- \alpha: the scaling factor.
Originally, d\times k parameters needed to be learned; now only the following need to be learned:
When r is much smaller than the matrix dimensions, the number of trainable parameters can be reduced significantly.
The important value of LoRA is that it separates task adaptation capability from the enormous parameters of the base model.
A single base model can be paired with multiple LoRA adapters, loading different adaptation parameters for different business scenarios.
Note that LoRA does not have to be used on every model layer, and a larger rank does not necessarily mean better results. The actual effect depends on the target modules, the rank, the scaling factor, the training data, and the characteristics of the task.
2.6 Key Issues in Fine-Tuning
Data quality is more important than data quantity.
Large amounts of duplicated, incorrect, or mutually conflicting data may cause the model to learn the wrong patterns.
During fine-tuning, you also need to pay attention to:
- Deduplicating data, to avoid training distribution shift caused by duplicate samples.
- Keeping the training set, validation set, and test set isolated, to avoid data leakage.
- Setting a reasonable learning rate, to avoid damaging pretrained capabilities.
- Monitoring overfitting through the validation set.
- Controlling the ratio of domain data to general data, to reduce catastrophic forgetting.
- Correctly setting roles, end-of-turn markers, and loss masks for multi-turn conversation data.
In instruction fine-tuning in particular, loss is usually computed only on the Assistant portions that the model needs to generate, rather than treating all input text as prediction targets. This practice is commonly called Label Masking.
3. LLM Distillation: Knowledge Distillation
3.1 What Is Model Distillation?
Model distillation is a knowledge transfer technique.
It uses a more capable teacher model to guide a student model's learning, so that the student model can acquire as much of the teacher model's ability as possible while being smaller in scale and requiring fewer computational resources.
For example:
- Teacher model: a high-performance model with a larger parameter scale.
- Student model: a model with a smaller parameter scale and lower inference cost.
The goal of distillation is not to simply copy the teacher model's parameters, but to let the student model learn the supervisory information provided by the teacher model.
The biggest difference between fine-tuning and distillation:
- Fine-tuning: adjusting a model using task data.
- Distillation: training a student model using knowledge or supervisory signals produced by a teacher model.
Distillation also typically requires a training process, so the student model's parameters are updated. From this perspective, distillation can use fine-tuning-style optimization methods, but the source of its supervisory signals and its training objective are more special.
3.2 Teacher Model and Student Model
Distillation usually involves two models:
Teacher Model
- Usually has a larger parameter scale or stronger task capability.
- Provides predictions, probability distributions, hidden representations, or reasoning examples for training samples.
- Provides the supervisory signals that the student model needs to learn.
Student Model
- Usually smaller in scale.
- Learns the knowledge provided by the teacher model.
- The goal is to achieve the best possible performance at a lower computational cost.
Note that the teacher model does not necessarily have to have more parameters than the student model. The core of distillation is the knowledge transfer relationship, not a strict ranking by parameter scale.
3.3 Basic Principles of Distillation
Suppose the teacher model outputs logits:
and the student model outputs logits:
Using Softmax and a temperature parameter T, the logits can be converted into soft probability distributions:
Where:
- p_t^{(T)}: the teacher model's soft probability distribution.
- p_s^{(T)}: the student model's soft probability distribution.
- T: the distillation temperature.
As T increases, the probability distribution usually becomes smoother, and the relative relationships among low-probability classes become easier to reveal.
For example, the teacher model's predictions for three classes might be:
| Category | Regular probability distribution | Soft distribution at higher temperature |
|---|---|---|
| A | 0.90 | 0.65 |
| B | 0.08 | 0.25 |
| C | 0.02 | 0.10 |
Soft labels not only tell the student model what the correct answer is, but also contain the teacher model's relative judgments about other candidate classes.
This is also an important feature that distinguishes distillation from training with only hard labels.
3.4 KL Divergence: Making the Student Model Imitate the Teacher Model
A common distillation loss is KL divergence:
After expansion:
KL divergence measures the difference between two probability distributions.
As the student model's predicted distribution gradually approaches that of the teacher model, the distillation loss decreases.
Because temperature changes the smoothness of the probability distribution, common implementations multiply the distillation loss by T^2 to adjust the gradient scale at different temperatures:
3.5 Distillation Loss and Ground-Truth Label Loss
In practice, the student model is usually not trained solely to imitate the teacher model.
This is because the teacher model can also make mistakes. If the student model completely copies the teacher model's outputs, it may inherit the teacher model's errors.
Therefore, a common approach is to combine the ground-truth label loss and the distillation loss:
where:
- L_{KD}: the distillation loss between the teacher model and the student model.
- L_{CE}: the cross-entropy loss between the student model and the ground-truth labels.
- \lambda: the weight of the two training objectives.
This combination enables the student model to learn both the teacher model's behavior and the ground-truth task labels at the same time.
For language models, teacher-generated responses, token-level probability distributions, or intermediate representations can also be used as supervision signals; the specific loss form depends on the distillation scheme.
3.6 Main Types of Distillation
| Distillation Type | Transferred Content | Core Idea |
|---|---|---|
| Response distillation | Text generated by the model | Learn the teacher model's way of answering |
| Logits distillation | Output-layer logits | Learn the teacher model's output distribution |
| Feature distillation | Hidden representations in intermediate layers | Learn the teacher model's internal representations |
| Attention distillation | Attention weights or related structures | Learn the teacher model's attention patterns |
| Data distillation | Training data generated by the teacher model | Use the teacher model to construct training samples |
| Reasoning-process distillation | Reasoning examples or intermediate reasoning steps | Learn the problem-solving process demonstrated by the teacher model |
These types are not mutually exclusive; a distillation scheme can combine multiple supervision signals.
Note in particular that the reasoning process output by a model does not necessarily fully reflect the model's true internal computation process. Reasoning-process distillation transfers reasoning examples generated by the teacher model; it should not be simply understood as directly copying its internal reasoning mechanism.
3.7 Workflow of Large-Model Distillation
flowchart TB
A["Prepare teacher model"] --> B["Prepare distillation data"]
B --> C["Teacher model generates supervision signals"]
C --> D["Generate soft labels or example responses"]
D --> E["Prepare student model"]
E --> F["Student model forward pass"]
F --> G["Compute distillation loss"]
G --> H["Combine with ground-truth label loss"]
H --> I["Backpropagate and update student model"]
I --> J{"Training conditions met?"}
J -->|"No"| F
J -->|"Yes"| K["Save student model"]
K --> L["Independently evaluate model capabilities"]
L --> M["Deploy student model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
3.8 Special Challenges in Large Model Distillation
Traditional classification models usually have a limited number of output classes, whereas the output vocabulary of large language models can reach tens of thousands or even more.
For autoregressive language models, the teacher model and the student model may have different:
- Tokenizers.
- Vocabulary sizes.
- Network architectures.
- Hidden layer dimensions.
- Number of attention heads.
- Context windows.
This means it is not always possible to directly compare the Logits or hidden-layer vectors of the two models.
For example, the 1000th Token ID in the teacher model and the 1000th Token ID in the student model may represent completely different Tokens.
If the two have different vocabularies, it is usually necessary to use approaches such as text-level response distillation, a unified Token space, and aligned representation spaces, rather than directly applying Logits distillation over the same indices.
In addition, distillation also faces:
- Teacher model error propagation: The student model may learn the teacher model's incorrect answers.
- Capacity compression loss: The student model has limited capacity, making it difficult to fully absorb the teacher model's knowledge.
- Insufficient data coverage: The distillation data cannot cover all of the teacher model's capabilities.
- Distribution shift: The inputs the student model encounters during actual operation may differ from the distillation training data.
- Training cost: Generating large amounts of supervision data with the teacher model itself requires computational resources.
Distillation does not guarantee that the student model will match the teacher model's performance on all tasks.
4. Large Model Quantization: Quantization
4.1 What Is Quantization?
Quantization refers to converting high-precision values in model parameters or computations into lower-precision numerical representations to reduce storage and computation overhead.
For example:
- FP32: 32-bit floating point.
- FP16: 16-bit floating point.
- BF16: 16-bit floating point.
- INT8: 8-bit integer.
- INT4: 4-bit integer.
If a model contains 7 billion parameters:
- FP32 weights theoretically require about 28 GB.
- FP16 weights theoretically require about 14 GB.
- INT8 weights theoretically require about 7 GB.
- INT4 weights theoretically require about 3.5 GB.
This only calculates the weight data itself and does not include additional overhead such as quantization scale factors, zero points, alignment padding, metadata, activations, and KV Cache.
The core goal of quantization is:
Use fewer bits to represent model values while preserving model performance as much as possible.
4.2 Basic Principles of Quantization
Taking linear quantization as an example, map floating-point numbers to an integer range.
A common affine quantization form is:
where:
- x: original floating-point number.
- q: quantized integer.
- s: Scale, the scaling factor.
- z: Zero Point, the zero point.
- q_{\min},q_{\max}: lower and upper bounds of the quantized integer.
During dequantization:
where \hat{x} is the approximately restored floating-point number after quantization.
Quantization inevitably introduces numerical error:
Quantization design needs to control the error as much as possible to avoid the error having a noticeable impact on model output.
4.3 Symmetric and Asymmetric Quantization
Symmetric Quantization
The numerical range is distributed symmetrically around zero, usually using:
Its computation is simple, and it is suitable for many weight tensors.
Asymmetric Quantization
The quantization interval is allowed not to be centered on zero, and the original values are mapped through Scale and Zero Point.
When the original value distribution is asymmetric, asymmetric quantization may make fuller use of the integer representation range.
| Aspect | Symmetric Quantization | Asymmetric Quantization |
|---|---|---|
| Zero point | Usually 0 | Can be nonzero |
| Parameters | Mainly uses Scale | Uses Scale and Zero Point |
| Computation | Relatively simple | Requires handling zero-point offset |
| Applicable case | Weights with a relatively symmetric distribution | Values with an asymmetric distribution |
4.4 Classification by Quantization Granularity
Quantization can change not only data precision but also the granularity at which Scale and Zero Point are computed.
| Quantization granularity | Description | Characteristics |
|---|---|---|
| Per-Tensor | The entire tensor shares quantization parameters | Simple to implement, but precision may be affected by extreme values |
| Per-Channel | Independent quantization parameters per channel | Usually better adapts to differences between channels |
| Per-Group | Each group of elements shares quantization parameters | A trade-off among precision, storage, and computation |
For example, a weight matrix can be quantized separately by output channel, or consecutive weights can be divided into multiple Groups, with each group using an independent Scale.
Group-wise quantization is very common in low-bit model deployment.
4.5 Post-Training Quantization: PTQ
Post-Training Quantization (PTQ) refers to converting a model's weights or other values to a low-precision representation after the model has been trained.
Typical workflow:
- Load the fully trained model.
- Analyze the distributions of weights and activations.
- Determine the quantization precision and granularity.
- Compute quantization parameters such as Scale and Zero Point.
- Perform quantization.
- Evaluate accuracy.
- Adjust the quantization configuration based on the results.
Some PTQ methods require a small amount of calibration data to estimate the distribution and range of activations, but they do not require full gradient training like regular fine-tuning.
The advantages of PTQ are low cost and relatively simple implementation.
The disadvantage is that quantization error can directly affect model output, especially with extremely low-bit quantization.
4.6 Quantization-Aware Training: QAT
Quantization-Aware Training (QAT) simulates the quantization process during training so that the model adapts to quantization error in advance during training.
Its basic idea is:
- Simulate low-precision quantization in the forward pass.
- Use approximate gradients to update parameters in the backward pass.
- Enable the model to learn to adapt to the numerical changes caused by quantization during training.