Back to series
· 13 min read

SFT Instruction Fine-Tuning: How Do Large Language Models Learn to Understand Instructions and Generate Responses That Meet Requirements?


I. What Is SFT?

SFT (Supervised Fine-Tuning) is an important stage in large language model training.

It uses supervised data constructed through manual writing, manual filtering, or other methods to further train a pretrained model, enabling the model to learn to complete tasks according to user instructions.

The pretraining stage mainly learns:

Given a piece of context, predict the next Token.

The SFT stage further learns:

Given an instruction, generate a response that meets the instruction's requirements.

For example, a pretrained model has been exposed to a large amount of natural language text and may have learned how to continue writing an article, but it does not necessarily know that it should answer directly when a user asks a question.

After SFT, the model is exposed to a large amount of data similar to the following:

Input instruction:

Please explain what the HTTP protocol is and list its main characteristics.

Expected response:

HTTP is an application-layer protocol used to transfer hypertext between clients and servers.

Main characteristics:
1. Based on the request-response model.
2. Usually runs on top of transport foundations such as TCP or QUIC.
3. It is stateless.
4. Supports multiple request methods and status codes.
5. Can provide secure transport via HTTPS using TLS.

Through a large number of similar training samples, the model gradually learns the mapping between instructions and responses.

1. Core Objectives of SFT

SFT mainly addresses several problems that base models have in practical interaction:

Problem SFT training objective
Does not understand user intent Learn to identify task requirements in instructions
Poor at answering directly Learn to organize content in a question-answer format
Does not follow output formats Learn format constraints such as JSON, Markdown, and code
Poor at multi-turn dialogue Learn to understand the current question in the context of the conversation history
Lacks stable task patterns Learn tasks such as classification, summarization, translation, and code generation
Outputs are verbose or off-topic Learn response styles that better meet the target requirements

The focus of SFT is not to learn language from scratch, but to further shape the model's behavior based on its existing language capabilities.

2. Where SFT Fits in Large Language Model Training

A typical large language model training process is as follows:

flowchart TB
    A["Large-scale pretraining data"] --> B["Pretraining"]
    B --> C["Base Model"]
    C --> D["Construct instruction supervision data"]
    D --> E["SFT supervised fine-tuning"]
    E --> F["Instruction-tuned model"]
    F --> G["Preference optimization and reinforcement learning"]
    G --> H["Aligned model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

Note that this is a typical process, and not all models are trained strictly in this order. Some models may use multi-round SFT, continual pretraining, synthetic data training, or other combined strategies.


II. What Is the Difference Between SFT and Pretraining?

SFT is built on top of pretrained models, but the two differ clearly in data format, training objective, and model behavior.

1. Different Training Data

Pretraining mainly uses large-scale raw text.

For example:

Artificial intelligence is a discipline that studies how to make machines exhibit intelligent behavior.
Machine learning is an important branch of artificial intelligence.
Deep learning is a method of machine learning.

These texts do not necessarily contain explicit questions and answers.

SFT, by contrast, mainly uses supervised data with explicit task relationships.

For example:

Instruction: What is machine learning?

Response: Machine learning is an important branch of artificial intelligence.
It trains models using data so that they can learn patterns from the data
and make predictions or decisions on new data.

The core difference between the two is:

  • Pretraining data mainly provides language and knowledge content.
  • SFT data additionally provides task requirements and target outputs.

2. Different Training Objectives

Pretraining usually uses an autoregressive language modeling objective:

L_{\text{pretrain}} = -\sum_{t=1}^{T} \log P_{\theta}(x_t\mid x_{<t})

The model needs to predict every target Token in the training text.

SFT usually uses a conditional language modeling objective:

L_{\text{SFT}} = -\sum_{t=1}^{T} \log P_{\theta}(y_t\mid x,y_{<t})

Where:

  • x: the user instruction or input context.
  • y_t: the token of the target response at position t.
  • y_{<t}: everything in the target response before the current token.
  • \theta: the model parameters.

SFT aims to make the model more likely to predict the correct response token, given the instruction and the existing prefix of the response.

3. The Models Behave Differently

Comparison dimension Pretrained model SFT model
Core objective Learn general language modeling Learn to perform tasks by following instructions
Data form Large-scale raw text Instructions and target responses
Input form Text context Instruction, dialogue, or task input
Output tendency Continue generating text Generate the response the task requires
Dialogue ability Not necessarily stable Usually better suited to instruction interaction
Format adherence Not necessarily reliable Can be reinforced through supervised data

There is an important distinction here:

SFT does not change the basic mechanism by which the model predicts the next token.

What it changes is the conditional relationship expressed by the training data and the training objective.

The model still learns through Next Token Prediction; it is just that what it now predicts are responses that satisfy the instruction.


3. How Are SFT Datasets Constructed?

The effectiveness of SFT depends heavily on the quality of the supervised data.

Even if the base model has a very large number of parameters, errors in the instruction data, low-quality responses, or insufficient task coverage can still hurt fine-tuning results.

1. Basic Instruction Data Structure

The simplest SFT data usually contains two parts:

  • Instruction: the user instruction.
  • Output: the expected response.

For example:

{
  "instruction": "Explain what a database index is",
  "output": "A database index is a data structure used to speed up data queries..."
}

Some datasets also include input context:

{
  "instruction": "Summarize the following passage",
  "input": "Databases use indexes to reduce the amount of data that must be scanned during a query...",
  "output": "Database indexes speed up data retrieval by maintaining an additional data structure."
}

This kind of structure can be abstracted as:

D=\{(x_i,y_i)\}_{i=1}^{N}

where:

  • x_i: the i-th training input.
  • y_i: the corresponding target response.
  • N: the number of training samples.

During actual training, the input and output are usually serialized into a unified dialogue format and then passed to the Tokenizer for encoding.

2. Common Types of SFT Data

Data type Example task Training value
Knowledge Q&A Explain a technical concept Improve question-answering ability
Instruction execution Complete a task as required Improve instruction-following ability
Text generation Write an article, write an email Learn how content is organized
Summarization and extraction Summarize a long text Learn information compression
Translation Chinese-English translation Improve cross-lingual ability
Classification and judgment Determine the category of a text Learn structured judgment
Code generation Write code from requirements Improve programming task ability
Logical reasoning Math and logic problems Learn patterns of specific reasoning tasks
Multi-turn dialogue Continuous contextual conversation Learn dialogue state management
Formatted output JSON, XML, Markdown Improve structured output ability

A dataset should not chase quantity alone; it also needs to cover enough task types and input forms.

3. Why Does Diversity Matter?

Suppose an SFT dataset contains one million examples, but 90% of them are simple knowledge Q&A.

The model may then perform well on knowledge Q&A tasks, but it will not necessarily be good at:

  • complex instructions;
  • code generation;
  • multi-turn dialogue;
  • structured output;
  • long-text analysis;
  • multi-step tasks.

This shows that data volume does not directly represent how well tasks are covered.

Practical data construction usually needs to consider:

  1. Task type distribution.
  2. Instruction complexity distribution.
  3. Response length distribution.
  4. Language and domain distribution.
  5. Ratio of simple tasks to complex tasks.
  6. Consistency of data quality across different tasks.

4. Data Sources

SFT data may come from:

  • high-quality instruction data written by humans;
  • question-answer data annotated by humans;
  • professional task data written by experts;
  • curated public instruction datasets;
  • synthetic data generated using strong models;
  • samples from user interaction data that have been authorized, filtered, and processed.

Synthetic data has already become an important data source, but it cannot be assumed that all model-generated responses are high-quality answers.

Synthetic data needs to go through quality assessment, fact verification, deduplication, and safety filtering.


IV. SFT Data Quality Control

SFT is not simply a matter of collecting questions and answers and then training on them.

Data quality control usually involves multiple stages.

flowchart TB
    A["Collect raw instruction data"] --> B["Format normalization"]
    B --> C["Check instruction validity"]
    C --> D["Check response quality"]
    D --> E["Fact and logic verification"]
    E --> F["Duplicate data detection"]
    F --> G["Safety and privacy filtering"]
    G --> H["Task classification and quality tiering"]
    H --> I["Train/validation set split"]
    I --> J["Form the SFT dataset"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

1. Instruction Quality

A good instruction should clearly state the task requirements.

An instruction of lower quality:

Introduce it.

This instruction lacks a clear object and output requirements.

A better instruction:

Please explain how database indexes work,
focusing on the query process of B+ trees,
and illustrate it with a simple example.

High-quality instructions can reduce task ambiguity, enabling the model to learn a clearer relationship between input and output.

2. Response Quality

Target responses are what the model directly learns to imitate.

If the responses in the training data contain errors, the model may learn incorrect knowledge or expression patterns.

Response quality usually needs to be checked for:

  • whether the facts are correct;
  • whether the reasoning process is sound;
  • whether it actually answers the question;
  • whether there are obvious omissions;
  • whether it conforms to the specified format;
  • whether it contains unnecessary content;
  • whether there are safety and compliance issues.

3. Data Deduplication

Duplicate data causes certain instructions and responses to be over-reinforced during training.

SFT data usually needs:

  • deduplication of exactly identical samples;
  • deduplication of similar instructions;
  • detection of similar responses;
  • cross-source duplicate detection.

But near-duplicates are not always worthless.

For example, the same piece of knowledge may need to be trained through different wording, different levels of difficulty, and different contexts. Deleting these samples entirely could reduce the model's ability to adapt to variations in instructions.

4. Train/Validation Set Split

SFT usually requires splitting data into a training set and a validation set.

The training set is used to compute loss and update parameters, while the validation set is used to observe the model's performance on data that was not involved in parameter updates.

It is necessary to avoid having highly similar data appear in both the training set and the validation set; otherwise, validation metrics may be overly optimistic.


V. Chat Templates and Special Tokens

Real large language models do not simply concatenate user questions and responses directly.

To represent roles, message boundaries, and conversation structure, dedicated chat templates are usually used.

1. Why Are Chat Templates Needed?

Consider the following multi-turn conversation:

User: What is Python?

Assistant: Python is a high-level programming language.

User: Is it suitable for artificial intelligence?

Assistant: Yes. Python has a rich ecosystem for machine learning and deep learning.

The model needs to know:

  • which content comes from the user;
  • which content comes from the assistant;
  • which message has ended;
  • whether the current turn has ended;
  • whether a tool needs to be called.

If the text is merely concatenated, it is difficult for the model to reliably distinguish these structures.

Therefore, many models use dedicated Chat Templates.

2. Common Structural Markers

Different models define special tokens differently. Below are some common structural markers as examples:

Marker Common role
im_start Begins a message
im_sep Separates the message header from the body
im_end Ends a message
fim_suffix Ends the assistant turn
fim_middle Marks specific fill or middle content
meta_sep Introduces channel metadata
meta_start Introduces format metadata

The actual tokens corresponding to these names usually have dedicated Token IDs. The strings in the table above are merely visible representations provided for ease of understanding; not all models use the same markers.

3. Example of a conversation sequence

Take a common structured conversation format as an example:

<|im_start|>user<|im_sep|>
What is a Transformer?
<|im_end|>
<|im_start|>assistant<|im_sep|>
A Transformer is a neural network architecture based on the attention mechanism.
<|im_end|>

Here:

  • user indicates the user role.
  • assistant indicates the assistant role.
  • im_start indicates the start of a message.
  • im_sep indicates the separation between the message header and the body.
  • im_end indicates the end of the current message.

For a complete assistant answer, a special marker indicating the end of the assistant turn may also be used.

4. The difference between Message End and Turn End

These two concepts are very important.

  • Message end: the current message has ended, but the assistant may continue to output more.
  • Assistant turn end: the assistant's current round of answering is complete.

For example, the assistant may first output analysis channel content and then output the final answer. The end of an intermediate message does not mean the end of the entire assistant turn.

Different models have their own conventions for the markers of message end, turn end, and tool handoff; the rules of one template cannot be applied directly to all models.


6. The Core Training Mechanism of SFT

SFT still uses autoregressive language modeling; the only difference is that the training input and the target answer have an explicit instruction relationship.

1. Language modeling under instruction conditioning

Suppose a training example is:

(x,y)

where:

  • x: the user instruction and necessary context.
  • y=(y_1,y_2,\ldots,y_T): the target answer.

The model learns:

P_{\theta}(y\mid x)

By the chain rule of probability:

P_{\theta}(y\mid x) = \prod_{t=1}^{T} P_{\theta}(y_t\mid x,y_{<t})

The model must predict the next target Token based on the user instruction and the answer prefix already generated.

2. Teacher Forcing

SFT training typically uses Teacher Forcing.

Its core idea is:

At each position of the target answer during training, the model uses the ground-truth historical answer Tokens from the dataset as context, rather than the Tokens the model itself has just generated.

For example, suppose the target answer is:

Python is a programming language.

During training:

Current prediction position Answer prefix provided to the model Target Token
1 empty Python
2 Python is
3 Python is a
4 Python is a programming language
5 Python is a programming language .

This allows the loss for the entire answer sequence to be computed in parallel.

But at actual inference time, the model has no ground-truth answer to serve as historical context and can only use the Tokens it has already generated.

This is an important difference between training and inference.

3. The forward computation process of SFT

flowchart TB
    A["User instruction and context"] --> B["Construct chat template"]
    B --> C["Tokenizer encoding"]
    C --> D["Input to Transformer"]
    D --> E["Generate logits for each position"]
    E --> F["Align with target response tokens"]
    F --> G["Compute supervised loss"]
    G --> H["Backpropagation"]
    H --> I["Optimizer updates parameters"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

7. SFT Loss Function: Train Only the Response or the Entire Sequence?

This is a very important detail in SFT implementation.

A conversation usually contains:

  • System prompt;
  • User instruction;
  • Assistant response;
  • Special tokens;
  • Possible tool messages.

But not all tokens should participate in supervised loss computation.

1. Compute Loss Only on the Assistant Response

In common instruction fine-tuning, the loss is usually computed only on the assistant's target response.

For example:

System: You are a professional programming assistant.
User: Explain what Python is.
Assistant: Python is a high-level programming language.

During training:

  • System prompt: serves as conditional context;
  • User instruction: serves as conditional context;
  • Assistant response: serves as the supervision target.

The corresponding loss can be written as:

L_{\text{SFT}} = -\sum_{t=1}^{T} m_t \log P_{\theta}(y_t\mid x,y_{<t})

where:

  • m_t=1: this position participates in loss computation.
  • m_t=0: this position does not participate in loss computation.

Usually, the positions corresponding to user messages and system messages are set so that response loss is not computed.

2. Why Not Train on All Tokens Directly?

If user instructions, system prompts, and assistant responses are all used as prediction targets, the model will also learn to imitate user input.

This does not necessarily align with the goal of instruction fine-tuning.

For example, the model may spend a large amount of training resources predicting user questions, instead of focusing on learning how to generate responses that meet the requirements.

Computing loss only on the target response can make the training objective more focused on assistant behavior.

However, this is not an absolute rule for all SFT approaches. Some training methods assign different weights to different roles or tokens, or even supervise the complete conversation.

3. The Role of Loss Mask

Loss Mask is used to control which tokens participate in loss computation.

For example:

Token Type Participates in Loss
System content Usually no
User content Usually no
Assistant response Usually yes
Assistant end marker Determined by training configuration
Padding Token No
Tool output Depends on the training objective and data design

A special distinction needs to be made here:

Attention Mask and Loss Mask are not the same thing.

  • Attention Mask: controls which positions the model can attend to.
  • Loss Mask: controls which positions participate in loss computation.

Attention Mask affects the model's range of visible information, while Loss Mask affects the supervised objective for parameter optimization.

The two may exist at the same time, but they serve different responsibilities.

4. PyTorch Loss Computation Example

Below is a simplified example, assuming the model outputs have already been aligned with the label positions:

```python import torch import torch.nn.functional as F # logits: [batch_size, seq_len, vocab_size] # labels: [batch_size, seq_len] # loss_mask: [batch_size, seq_len] def sft_loss(logits, labels, loss_mask): vocab_size = logits.size(-1) # Flatten the sequence dimension logits = logits.reshape(-1, vocab_size) labels = labels.reshape(-1) loss_mask = loss_mask.reshape(-1).float() # Compute cross-entropy token by token token_loss = F.cross_entropy( logits, labels, reduction="none", ignore_index=-100 ) # Only count valid supervised positions valid_mask = loss_mask * (labels != -100).float() loss = ( token_loss * valid_mask ).sum() / valid_mask.sum().clamp_min(1.0) return loss ``` This example illustrates several key points: 1. The model outputs logits, not probabilities after an early softmax. 2. Cross-entropy is computed token by token. 3. The loss mask controls which positions participate in the loss computation. 4. Padding or ignored positions do not participate in the loss. 5. Normalization is performed using the number of valid tokens. In real-world engineering, you also need to handle label shifting, special tokens, samples of varying lengths, and loss normalization under distributed training. --- ## 8. Parameter Update Strategies for SFT SFT can adopt different parameter update approaches. ### 1. Full Fine-Tuning Full fine-tuning updates all trainable parameters in the model. Let the pretrained model parameters be:
\theta_0
After SFT training, we obtain:
\theta_{\text{SFT}}
The optimization objective can be expressed as:
\theta_{\text{SFT}} = \arg\min_{\theta} L_{\text{SFT}}(\theta)
During training, trainable parameters such as embeddings, attention, FFN, normalization layers, and the output projection may all change. **Advantages:** - The parameter update scope is complete; - The model's internal representations can be fully adjusted; - High flexibility for adapting to complex domains. **Disadvantages:** - High memory requirements; - High training cost; - Full model weights need to be saved; - Overfitting may occur on small datasets; - Fine-tuning separately for different tasks may require maintaining multiple sets of model weights. ### 2. Parameter-Efficient Fine-Tuning (PEFT) Parameter-Efficient Fine-Tuning (PEFT) trains only a subset of the model's parameters or newly added parameters. Its core idea is: Freeze most of the original pretrained model's parameters as much as possible, and adapt to the task using only a small number of trainable parameters. Common methods include: - LoRA; - QLoRA; - Adapter; - Prefix Tuning; - Prompt Tuning. ### 3. The Basic Principle of LoRA LoRA (Low-Rank Adaptation) is a very common parameter-efficient fine-tuning method. Suppose the original linear layer weight is:
W_0\in\mathbb{R}^{d\times k}
Full fine-tuning requires directly updating the entire matrix. LoRA instead freezes the original weight W_0 and trains only a low-rank increment:
\Delta W=BA
where:
A\in\mathbb{R}^{r\times k}
B\in\mathbb{R}^{d\times r}
and:
r\ll \min(d,k)
The final weight is:
W=W_0+\frac{\alpha}{r}BA
where: