I. What Is SFT?
SFT (Supervised Fine-Tuning) is an important stage in large language model training.
It uses supervised data constructed through manual writing, manual filtering, or other methods to further train a pretrained model, enabling the model to learn to complete tasks according to user instructions.
The pretraining stage mainly learns:
Given a piece of context, predict the next Token.
The SFT stage further learns:
Given an instruction, generate a response that meets the instruction's requirements.
For example, a pretrained model has been exposed to a large amount of natural language text and may have learned how to continue writing an article, but it does not necessarily know that it should answer directly when a user asks a question.
After SFT, the model is exposed to a large amount of data similar to the following:
Input instruction:
Please explain what the HTTP protocol is and list its main characteristics.
Expected response:
HTTP is an application-layer protocol used to transfer hypertext between clients and servers.
Main characteristics:
1. Based on the request-response model.
2. Usually runs on top of transport foundations such as TCP or QUIC.
3. It is stateless.
4. Supports multiple request methods and status codes.
5. Can provide secure transport via HTTPS using TLS.
Through a large number of similar training samples, the model gradually learns the mapping between instructions and responses.
1. Core Objectives of SFT
SFT mainly addresses several problems that base models have in practical interaction:
| Problem | SFT training objective |
|---|---|
| Does not understand user intent | Learn to identify task requirements in instructions |
| Poor at answering directly | Learn to organize content in a question-answer format |
| Does not follow output formats | Learn format constraints such as JSON, Markdown, and code |
| Poor at multi-turn dialogue | Learn to understand the current question in the context of the conversation history |
| Lacks stable task patterns | Learn tasks such as classification, summarization, translation, and code generation |
| Outputs are verbose or off-topic | Learn response styles that better meet the target requirements |
The focus of SFT is not to learn language from scratch, but to further shape the model's behavior based on its existing language capabilities.
2. Where SFT Fits in Large Language Model Training
A typical large language model training process is as follows:
flowchart TB
A["Large-scale pretraining data"] --> B["Pretraining"]
B --> C["Base Model"]
C --> D["Construct instruction supervision data"]
D --> E["SFT supervised fine-tuning"]
E --> F["Instruction-tuned model"]
F --> G["Preference optimization and reinforcement learning"]
G --> H["Aligned model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
Note that this is a typical process, and not all models are trained strictly in this order. Some models may use multi-round SFT, continual pretraining, synthetic data training, or other combined strategies.
II. What Is the Difference Between SFT and Pretraining?
SFT is built on top of pretrained models, but the two differ clearly in data format, training objective, and model behavior.
1. Different Training Data
Pretraining mainly uses large-scale raw text.
For example:
Artificial intelligence is a discipline that studies how to make machines exhibit intelligent behavior.
Machine learning is an important branch of artificial intelligence.
Deep learning is a method of machine learning.
These texts do not necessarily contain explicit questions and answers.
SFT, by contrast, mainly uses supervised data with explicit task relationships.
For example:
Instruction: What is machine learning?
Response: Machine learning is an important branch of artificial intelligence.
It trains models using data so that they can learn patterns from the data
and make predictions or decisions on new data.
The core difference between the two is:
- Pretraining data mainly provides language and knowledge content.
- SFT data additionally provides task requirements and target outputs.
2. Different Training Objectives
Pretraining usually uses an autoregressive language modeling objective:
The model needs to predict every target Token in the training text.
SFT usually uses a conditional language modeling objective:
Where:
- x: the user instruction or input context.
- y_t: the token of the target response at position t.
- y_{<t}: everything in the target response before the current token.
- \theta: the model parameters.
SFT aims to make the model more likely to predict the correct response token, given the instruction and the existing prefix of the response.
3. The Models Behave Differently
| Comparison dimension | Pretrained model | SFT model |
|---|---|---|
| Core objective | Learn general language modeling | Learn to perform tasks by following instructions |
| Data form | Large-scale raw text | Instructions and target responses |
| Input form | Text context | Instruction, dialogue, or task input |
| Output tendency | Continue generating text | Generate the response the task requires |
| Dialogue ability | Not necessarily stable | Usually better suited to instruction interaction |
| Format adherence | Not necessarily reliable | Can be reinforced through supervised data |
There is an important distinction here:
SFT does not change the basic mechanism by which the model predicts the next token.
What it changes is the conditional relationship expressed by the training data and the training objective.
The model still learns through Next Token Prediction; it is just that what it now predicts are responses that satisfy the instruction.
3. How Are SFT Datasets Constructed?
The effectiveness of SFT depends heavily on the quality of the supervised data.
Even if the base model has a very large number of parameters, errors in the instruction data, low-quality responses, or insufficient task coverage can still hurt fine-tuning results.
1. Basic Instruction Data Structure
The simplest SFT data usually contains two parts:
- Instruction: the user instruction.
- Output: the expected response.
For example:
{
"instruction": "Explain what a database index is",
"output": "A database index is a data structure used to speed up data queries..."
}
Some datasets also include input context:
{
"instruction": "Summarize the following passage",
"input": "Databases use indexes to reduce the amount of data that must be scanned during a query...",
"output": "Database indexes speed up data retrieval by maintaining an additional data structure."
}
This kind of structure can be abstracted as:
where:
- x_i: the i-th training input.
- y_i: the corresponding target response.
- N: the number of training samples.
During actual training, the input and output are usually serialized into a unified dialogue format and then passed to the Tokenizer for encoding.
2. Common Types of SFT Data
| Data type | Example task | Training value |
|---|---|---|
| Knowledge Q&A | Explain a technical concept | Improve question-answering ability |
| Instruction execution | Complete a task as required | Improve instruction-following ability |
| Text generation | Write an article, write an email | Learn how content is organized |
| Summarization and extraction | Summarize a long text | Learn information compression |
| Translation | Chinese-English translation | Improve cross-lingual ability |
| Classification and judgment | Determine the category of a text | Learn structured judgment |
| Code generation | Write code from requirements | Improve programming task ability |
| Logical reasoning | Math and logic problems | Learn patterns of specific reasoning tasks |
| Multi-turn dialogue | Continuous contextual conversation | Learn dialogue state management |
| Formatted output | JSON, XML, Markdown | Improve structured output ability |
A dataset should not chase quantity alone; it also needs to cover enough task types and input forms.
3. Why Does Diversity Matter?
Suppose an SFT dataset contains one million examples, but 90% of them are simple knowledge Q&A.
The model may then perform well on knowledge Q&A tasks, but it will not necessarily be good at:
- complex instructions;
- code generation;
- multi-turn dialogue;
- structured output;
- long-text analysis;
- multi-step tasks.
This shows that data volume does not directly represent how well tasks are covered.
Practical data construction usually needs to consider:
- Task type distribution.
- Instruction complexity distribution.
- Response length distribution.
- Language and domain distribution.
- Ratio of simple tasks to complex tasks.
- Consistency of data quality across different tasks.
4. Data Sources
SFT data may come from:
- high-quality instruction data written by humans;
- question-answer data annotated by humans;
- professional task data written by experts;
- curated public instruction datasets;
- synthetic data generated using strong models;
- samples from user interaction data that have been authorized, filtered, and processed.
Synthetic data has already become an important data source, but it cannot be assumed that all model-generated responses are high-quality answers.
Synthetic data needs to go through quality assessment, fact verification, deduplication, and safety filtering.
IV. SFT Data Quality Control
SFT is not simply a matter of collecting questions and answers and then training on them.
Data quality control usually involves multiple stages.
flowchart TB
A["Collect raw instruction data"] --> B["Format normalization"]
B --> C["Check instruction validity"]
C --> D["Check response quality"]
D --> E["Fact and logic verification"]
E --> F["Duplicate data detection"]
F --> G["Safety and privacy filtering"]
G --> H["Task classification and quality tiering"]
H --> I["Train/validation set split"]
I --> J["Form the SFT dataset"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
1. Instruction Quality
A good instruction should clearly state the task requirements.
An instruction of lower quality:
Introduce it.
This instruction lacks a clear object and output requirements.
A better instruction:
Please explain how database indexes work,
focusing on the query process of B+ trees,
and illustrate it with a simple example.
High-quality instructions can reduce task ambiguity, enabling the model to learn a clearer relationship between input and output.
2. Response Quality
Target responses are what the model directly learns to imitate.
If the responses in the training data contain errors, the model may learn incorrect knowledge or expression patterns.
Response quality usually needs to be checked for:
- whether the facts are correct;
- whether the reasoning process is sound;
- whether it actually answers the question;
- whether there are obvious omissions;
- whether it conforms to the specified format;
- whether it contains unnecessary content;
- whether there are safety and compliance issues.
3. Data Deduplication
Duplicate data causes certain instructions and responses to be over-reinforced during training.
SFT data usually needs:
- deduplication of exactly identical samples;
- deduplication of similar instructions;
- detection of similar responses;
- cross-source duplicate detection.
But near-duplicates are not always worthless.
For example, the same piece of knowledge may need to be trained through different wording, different levels of difficulty, and different contexts. Deleting these samples entirely could reduce the model's ability to adapt to variations in instructions.
4. Train/Validation Set Split
SFT usually requires splitting data into a training set and a validation set.
The training set is used to compute loss and update parameters, while the validation set is used to observe the model's performance on data that was not involved in parameter updates.
It is necessary to avoid having highly similar data appear in both the training set and the validation set; otherwise, validation metrics may be overly optimistic.
V. Chat Templates and Special Tokens
Real large language models do not simply concatenate user questions and responses directly.
To represent roles, message boundaries, and conversation structure, dedicated chat templates are usually used.
1. Why Are Chat Templates Needed?
Consider the following multi-turn conversation:
User: What is Python?
Assistant: Python is a high-level programming language.
User: Is it suitable for artificial intelligence?
Assistant: Yes. Python has a rich ecosystem for machine learning and deep learning.
The model needs to know:
- which content comes from the user;
- which content comes from the assistant;
- which message has ended;
- whether the current turn has ended;
- whether a tool needs to be called.
If the text is merely concatenated, it is difficult for the model to reliably distinguish these structures.
Therefore, many models use dedicated Chat Templates.
2. Common Structural Markers
Different models define special tokens differently. Below are some common structural markers as examples:
| Marker | Common role |
|---|---|
im_start |
Begins a message |
im_sep |
Separates the message header from the body |
im_end |
Ends a message |
fim_suffix |
Ends the assistant turn |
fim_middle |
Marks specific fill or middle content |
meta_sep |
Introduces channel metadata |
meta_start |
Introduces format metadata |
The actual tokens corresponding to these names usually have dedicated Token IDs. The strings in the table above are merely visible representations provided for ease of understanding; not all models use the same markers.
3. Example of a conversation sequence
Take a common structured conversation format as an example:
<|im_start|>user<|im_sep|>
What is a Transformer?
<|im_end|>
<|im_start|>assistant<|im_sep|>
A Transformer is a neural network architecture based on the attention mechanism.
<|im_end|>
Here:
userindicates the user role.assistantindicates the assistant role.im_startindicates the start of a message.im_sepindicates the separation between the message header and the body.im_endindicates the end of the current message.
For a complete assistant answer, a special marker indicating the end of the assistant turn may also be used.
4. The difference between Message End and Turn End
These two concepts are very important.
- Message end: the current message has ended, but the assistant may continue to output more.
- Assistant turn end: the assistant's current round of answering is complete.
For example, the assistant may first output analysis channel content and then output the final answer. The end of an intermediate message does not mean the end of the entire assistant turn.
Different models have their own conventions for the markers of message end, turn end, and tool handoff; the rules of one template cannot be applied directly to all models.
6. The Core Training Mechanism of SFT
SFT still uses autoregressive language modeling; the only difference is that the training input and the target answer have an explicit instruction relationship.
1. Language modeling under instruction conditioning
Suppose a training example is:
where:
- x: the user instruction and necessary context.
- y=(y_1,y_2,\ldots,y_T): the target answer.
The model learns:
By the chain rule of probability:
The model must predict the next target Token based on the user instruction and the answer prefix already generated.
2. Teacher Forcing
SFT training typically uses Teacher Forcing.
Its core idea is:
At each position of the target answer during training, the model uses the ground-truth historical answer Tokens from the dataset as context, rather than the Tokens the model itself has just generated.
For example, suppose the target answer is:
Python is a programming language.
During training:
| Current prediction position | Answer prefix provided to the model | Target Token |
|---|---|---|
| 1 | empty | Python |
| 2 | Python | is |
| 3 | Python is | a |
| 4 | Python is a | programming language |
| 5 | Python is a programming language | . |
This allows the loss for the entire answer sequence to be computed in parallel.
But at actual inference time, the model has no ground-truth answer to serve as historical context and can only use the Tokens it has already generated.
This is an important difference between training and inference.
3. The forward computation process of SFT
flowchart TB
A["User instruction and context"] --> B["Construct chat template"]
B --> C["Tokenizer encoding"]
C --> D["Input to Transformer"]
D --> E["Generate logits for each position"]
E --> F["Align with target response tokens"]
F --> G["Compute supervised loss"]
G --> H["Backpropagation"]
H --> I["Optimizer updates parameters"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
7. SFT Loss Function: Train Only the Response or the Entire Sequence?
This is a very important detail in SFT implementation.
A conversation usually contains:
- System prompt;
- User instruction;
- Assistant response;
- Special tokens;
- Possible tool messages.
But not all tokens should participate in supervised loss computation.
1. Compute Loss Only on the Assistant Response
In common instruction fine-tuning, the loss is usually computed only on the assistant's target response.
For example:
System: You are a professional programming assistant.
User: Explain what Python is.
Assistant: Python is a high-level programming language.
During training:
- System prompt: serves as conditional context;
- User instruction: serves as conditional context;
- Assistant response: serves as the supervision target.
The corresponding loss can be written as:
where:
- m_t=1: this position participates in loss computation.
- m_t=0: this position does not participate in loss computation.
Usually, the positions corresponding to user messages and system messages are set so that response loss is not computed.
2. Why Not Train on All Tokens Directly?
If user instructions, system prompts, and assistant responses are all used as prediction targets, the model will also learn to imitate user input.
This does not necessarily align with the goal of instruction fine-tuning.
For example, the model may spend a large amount of training resources predicting user questions, instead of focusing on learning how to generate responses that meet the requirements.
Computing loss only on the target response can make the training objective more focused on assistant behavior.
However, this is not an absolute rule for all SFT approaches. Some training methods assign different weights to different roles or tokens, or even supervise the complete conversation.
3. The Role of Loss Mask
Loss Mask is used to control which tokens participate in loss computation.
For example:
| Token Type | Participates in Loss |
|---|---|
| System content | Usually no |
| User content | Usually no |
| Assistant response | Usually yes |
| Assistant end marker | Determined by training configuration |
| Padding Token | No |
| Tool output | Depends on the training objective and data design |
A special distinction needs to be made here:
Attention Mask and Loss Mask are not the same thing.
- Attention Mask: controls which positions the model can attend to.
- Loss Mask: controls which positions participate in loss computation.
Attention Mask affects the model's range of visible information, while Loss Mask affects the supervised objective for parameter optimization.
The two may exist at the same time, but they serve different responsibilities.
4. PyTorch Loss Computation Example
Below is a simplified example, assuming the model outputs have already been aligned with the label positions:
```python import torch import torch.nn.functional as F # logits: [batch_size, seq_len, vocab_size] # labels: [batch_size, seq_len] # loss_mask: [batch_size, seq_len] def sft_loss(logits, labels, loss_mask): vocab_size = logits.size(-1) # Flatten the sequence dimension logits = logits.reshape(-1, vocab_size) labels = labels.reshape(-1) loss_mask = loss_mask.reshape(-1).float() # Compute cross-entropy token by token token_loss = F.cross_entropy( logits, labels, reduction="none", ignore_index=-100 ) # Only count valid supervised positions valid_mask = loss_mask * (labels != -100).float() loss = ( token_loss * valid_mask ).sum() / valid_mask.sum().clamp_min(1.0) return loss ``` This example illustrates several key points: 1. The model outputs logits, not probabilities after an early softmax. 2. Cross-entropy is computed token by token. 3. The loss mask controls which positions participate in the loss computation. 4. Padding or ignored positions do not participate in the loss. 5. Normalization is performed using the number of valid tokens. In real-world engineering, you also need to handle label shifting, special tokens, samples of varying lengths, and loss normalization under distributed training. --- ## 8. Parameter Update Strategies for SFT SFT can adopt different parameter update approaches. ### 1. Full Fine-Tuning Full fine-tuning updates all trainable parameters in the model. Let the pretrained model parameters be: