I. What Is Large Language Model Alignment?
Pretraining gives large language models the ability to do language modeling, associate knowledge, and generate text, while SFT teaches the model to understand instructions and generate responses as required.
But this does not mean the responses the model generates will necessarily match human preferences.
For example, a user asks:
Please explain what a database index is.
The model might generate two kinds of responses:
Response A:
A database index is a data structure used to speed up data queries. By maintaining additional index information, it reduces the amount of data that must be scanned during a query. Common index structures include B+ trees and hash indexes.
Response B:
A database index is a thing inside a database that can help the database run faster. It is very important and can improve database performance.
Both responses attempt to explain database indexes, but Response A is more accurate and specific, and its information is organized more clearly.
If only SFT is used, the model typically learns to imitate the standard responses in the training data. But when a question has multiple reasonable answers, pure supervised learning struggles to fully express the preference relationships among different answers.
This is the problem that preference alignment needs to solve.
1. What Is Alignment?
Large language model alignment refers to using training and other constraint mechanisms to make the model's output behavior better match the intended goals.
Common goals include:
- Helpfulness: responses can actually solve the problem.
- Truthfulness: minimize incorrect information as much as possible.
- Safety: avoid generating harmful content.
- Instruction Following: complete tasks according to user requirements.
- Response Quality: responses are clear, accurate, and well structured.
- Preference Consistency: among multiple reasonable responses, favor the more preferred one.
Note that alignment does not mean the model can guarantee that all responses are correct, nor does it mean human preferences are naturally equal to objective facts.
What the model learns is the preferences expressed by specific data, annotation standards, and reward mechanisms.
2. What Are RLHF and DPO?
Both RLHF and DPO can be used to optimize language models based on preference data, but they take different optimization paths.
RLHF (Reinforcement Learning from Human Feedback)
In Chinese it is usually called reinforcement learning from human feedback.
The classic RLHF pipeline usually includes:
- Use an SFT model to generate multiple candidate responses.
- Humans compare and rank the candidate responses.
- Use the preference data to train a reward model.
- Use a reinforcement learning algorithm (such as PPO) to optimize the language model.
- Adjust the model's behavior according to the reward signal.
DPO (Direct Preference Optimization)
In Chinese it is usually called direct preference optimization.
DPO directly uses preference data to optimize the language model; it does not require training a separate reward model, nor does it require online reinforcement learning like the classic RLHF-PPO pipeline.
The core differences between the two:
| Comparison Dimension | RLHF (Classic PPO Route) | DPO |
|---|---|---|
| Preference data | Required | Required |
| Reward model | Usually required | No separate training needed |
| Reinforcement learning | Uses algorithms such as PPO | No traditional PPO pipeline needed |
| Reference model | Usually required | Usually required |
| Optimization approach | Reward-driven policy optimization | Direct optimization on preference pairs |
| Engineering complexity | Usually higher | Usually lower |
| Training stability | Needs to handle RL stability issues | Usually easier to implement |
| Applicable scenarios | Complex reward optimization, online interaction, etc. | Offline preference data optimization |
Here, RLHF specifically refers to the classic reward-model-plus-PPO route. In a broad sense, RLHF can also include other reinforcement learning methods based on human feedback, while DPO is a direct preference optimization method.
II. From SFT to Preference Alignment
To understand RLHF and DPO, we first need to clarify the capability boundaries of SFT.
1. SFT Learns Target Responses
SFT data usually takes the form:
Where:
- x_i: the input instruction.
- y_i: the target response.
- N: the number of training samples.
The model is trained by maximizing the conditional probability of the target response:
The corresponding negative log-likelihood loss is:
The core of SFT is imitating the given target response.
2. Preference Data Expresses Relative Quality
Preference data is different from SFT data.
SFT usually tells the model:
How this question should be answered.
Preference data, on the other hand, tells the model:
For the same question, which response is more worth choosing.
For example:
Input:
Explain database indexes.
Candidate Answer A:
A database index is an auxiliary data structure that can reduce the range of data scanned during a query.
Common index structures include B+ trees and hash indexes.
Candidate Answer B:
A database index is just a feature inside a database.
Human annotators considered A better than B.
This preference data can be represented as:
Where:
- x: the input question.
- y_w: the preferred answer (Winner).
- y_l: the non-preferred answer (Loser).
The model no longer learns only a single standard answer; instead, it learns the relative preference relationships between answers.
3. Basic Form of Preference Data
Common preference data structure:
{
"prompt": "Explain database indexes",
"chosen": "A database index is an auxiliary data structure that can speed up data queries.",
"rejected": "A database index is just a feature in a database."
}
Actual datasets may also include:
- annotator information;
- scores;
- multiple candidate answers;
- rejection reasons;
- safety labels;
- task types;
- data sources.
Preference data can come from human annotation as well as expert evaluation, model-assisted evaluation, and rule-based validation.
However, the quality of preference signals from different sources may vary, so quality control is needed.
4. Overall Relationship from SFT to Preference Alignment
flowchart TB
A["Large-scale pretraining"] --> B["Base model"]
B --> C["SFT supervised fine-tuning"]
C --> D["Instruction model"]
D --> E["Generate multiple candidate answers"]
E --> F["Human or model preference evaluation"]
F --> G["Construct preference data"]
G --> H["RLHF or DPO"]
H --> I["Preference-aligned model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
III. How Is Preference Data Constructed?
Preference data is the common foundation of RLHF and DPO.
Data quality directly affects the preference direction the model ultimately learns.
1. Generating Candidate Answers
First, use a model that has already undergone SFT to generate multiple candidate answers for the same input.
For example:
Prompt:
Please explain the TCP three-way handshake.
The model generates:
Answer A:
The TCP three-way handshake consists of three stages: SYN, SYN-ACK, and ACK,
and is used to establish a connection between the client and the server.
Answer B:
TCP establishes a connection by sending SYN, SYN-ACK, and ACK,
and during this process negotiates connection parameters such as the initial sequence number.
Answer C:
When a TCP connection is established, the client sends SYN, the server responds with ACK,
and then the client confirms the connection.
The strategy for generating candidate answers affects the quality of preference data.
If all candidate answers are very similar, it is difficult for annotators to provide valuable distinguishing information.
Therefore, a certain degree of answer diversity is usually required when constructing data.
2. Human Preference Annotation
Human annotators compare candidate answers.
Common annotation methods:
Pairwise Comparison
Directly choose the better of two answers.
Multi-candidate Ranking
Rank multiple candidate answers.
Graded Scoring
Score accuracy, completeness, safety, and expression quality separately.
Dimension-based Evaluation
Annotate preferences separately for different evaluation dimensions.
Pairwise comparison is a very common data format in classical preference training.
3. Preference Annotation Criteria
Preferences cannot rely solely on subjective impressions; clear evaluation criteria usually need to be established.
| Dimension | Evaluation Focus |
|---|---|
| Correctness | Whether there are factual errors |
| Completeness | Whether core issues are covered |
| Relevance | Whether it stays closely tied to user needs |
| Clarity | Whether the expression is easy to understand |
| Instruction Following | Whether format and constraints are satisfied |
| Safety | Whether harmful content exists |
| Conciseness | Whether there is unnecessary redundancy |
Different tasks may have different evaluation weights.
For example, code generation tasks place more emphasis on code correctness and executability, while creative writing tasks may place more emphasis on expression quality and creativity.
4. Noise in Preference Data
Human annotations are not always consistent.
For example:
- Different annotators have different levels of knowledge;
- Different preferences for answer length;
- Different understandings of technical terms;
- Different judgments of risk and safety;
- There may be multiple reasonable answers to the same question.
Therefore, preference data may contain noise and subjectivity.
Common quality control methods include:
- Establish unified annotation guidelines;
- Use multiple annotators for cross-evaluation;
- Review samples with disagreements;
- Filter out low-confidence samples;
- Record inter-annotator agreement;
- Use experts to validate data in high-risk domains.
IV. RLHF: Reinforcement Learning from Human Feedback
RLHF is an important approach in large language model alignment technology.
Classic RLHF usually adopts three main stages:
- Supervised fine-tuning (SFT).
- Reward modeling.
- Reinforcement learning optimization (usually using PPO).
Its core idea is:
First, train a reward model using human preference data so that it can evaluate response quality; then, through reinforcement learning, make the language model tend to generate high-reward responses.
1. Overall Architecture of RLHF
flowchart TB
A["Pretrained Base Model"] --> B["SFT Supervised Fine-Tuning"]
B --> C["SFT Model"]
C --> D["Generate Multiple Candidate Responses"]
D --> E["Human Preference Annotation"]
E --> F["Preference Dataset"]
F --> G["Train Reward Model"]
C --> H["Policy Model Initialization"]
H --> I["Generate Response"]
I --> J["Reward Model Scoring"]
J --> K["KL Penalty and Reward Calculation"]
K --> L["PPO Updates the Policy Model"]
L --> I
L --> M["Aligned Model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
In the figure, the policy model is responsible for generating responses, the reward model is responsible for evaluating responses, and PPO adjusts the policy model according to the reward signal.
In actual training, the reward model is usually kept frozen while the policy model is continuously updated; a reference model is also used to constrain the policy model from deviating too far from the initial SFT model.
V. Reward Model
The reward model is a core component in classic RLHF.
Its task is not to directly generate responses, but to score a given input and response.
1. Input and Output of the Reward Model
The reward model usually receives:
where:
- x: user input.
- y: model response.
and outputs a scalar reward:
where:
- \phi: reward model parameters.
- r_{\phi}: the reward value predicted by the model.
For example:
| Response | Reward model output |
|---|---|
| Response A | 2.8 |
| Response B | 0.7 |
| Response C | -1.2 |
These scores are only illustrative.
The reward value itself usually has no fixed absolute meaning; it is mainly used to compare the relative quality of different responses.
2. How is the reward model trained?
The reward model is trained using preference data.
Suppose:
- y_w: the preferred response.
- y_l: the dispreferred response.
We want the reward model to satisfy:
One classic approach is to use the Bradley-Terry preference model.
It defines the probability that response y_w is preferred over y_l as:
By simplification:
where:
is the Sigmoid function.
This formula expresses an important idea:
The reward model does not need to directly predict the absolute scores given by humans; instead, it learns the relative difference between preferred and dispreferred responses.
3. Reward model loss function
For a preference sample, the loss function is:
It can also be written as:
If the reward model gives a higher score to the preferred response, the loss decreases.
If the reward model incorrectly gives a higher score to the dispreferred response, the loss increases.
For the entire dataset:
4. Structure of the reward model
In a classic implementation, the reward model can be initialized from a pretrained language model or an SFT model.
After concatenating the input and response, they are fed into a Transformer, and the hidden state of the last valid token is passed to a scalar output head.
Simplified structure:
flowchart TB
A["Prompt and Response"] --> B["Tokenizer Encoding"]
B --> C["Transformer"]
C --> D["Hidden State of Last Valid Token"]
D --> E["Linear Reward Head"]
E --> F["Scalar Reward"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#E5F0E9,stroke:#83A995,color:#315D49
The reward model usually needs a dedicated scoring head rather than directly using the language model's vocabulary output layer.
5. Limitations of the reward model
The reward model is only an approximate model of human preferences; it does not mean it truly understands response quality.
It may exhibit:
- giving higher scores to verbose responses;
- preferring expressions that appear more confident on the surface;
- failing to accurately identify subtle factual errors;
- being exploited by the policy model;
- giving unreliable scores to responses outside the training distribution.
Particular attention should be paid to reward hacking (Reward Hacking).
Once the policy model discovers the scoring patterns of the reward model, it may learn to generate responses that receive high scores but are not actually of high quality.
This is also why RLHF requires additional constraints and evaluation.
VI. PPO: Using Reinforcement Learning to Optimize Language Models
After the reward model is trained, reinforcement learning can be used to optimize the language model.
Classic RLHF usually uses PPO (Proximal Policy Optimization).
1. Treating the language model as a policy model
There are several core concepts in reinforcement learning:
| Reinforcement Learning Concept | Corresponding Concept in Large Language Models |
|---|---|
| Agent | Language model |
| Policy | Token generation policy |
| State | Current input and generated context |
| Action | Currently generated token |
| Reward | Reward provided by the reward model |
| Episode | A single answer generation process |
| Trajectory | Complete token generation sequence |
Suppose the user input is:
Explain what an operating system is.
The model generates the answer step by step:
An operating system is system software that manages computer hardware and software resources.
Each time a token is generated, it can be regarded as the policy model performing one action.
Therefore, a language model can be represented as an autoregressive policy:
where:
- \theta: Policy model parameters.
- x: User input.
- y_{<t}: Prefix of the answer generated so far.
- y_t: Currently generated token.
2. Where Does the Reward Signal Come From?
After the model generates a complete answer, the input and answer can be passed to the reward model for scoring.
For example:
However, if only the final score from the reward model is used, the policy model may deviate excessively from the original SFT model.
Therefore, classic RLHF usually introduces a KL penalty.
3. Why Is a KL Penalty Needed?
Suppose the policy model keeps pursuing high scores given by the reward model.
Without constraints, the model may gradually deviate from the original SFT model and produce:
- Unnatural expressions;
- Repeated content;
- Abnormally verbose answers;
- Degradation of language ability;
- Behavior that exploits loopholes in the reward model.
To address this, a reference model can be used to constrain the policy model.
The reference model is usually a frozen copy of the SFT model.
Define the KL divergence between the policy model and the reference model:
KL divergence measures the difference between two probability distributions.
For discrete distributions:
In language model training, the KL penalty is usually used to constrain the policy model from deviating too far from the reference model.
4. Optimization Objective of RLHF
The objective can be abstracted as:
where:
- r_{\phi}(x,y): Reward given by the reward model.
- \pi_{\theta}: Current policy model.
- \pi_{\mathrm{ref}}: Reference model.
- \beta: KL penalty coefficient.
This objective expresses two directions that balance each other:
- Increase the reward of the answer.
- Keep the model from deviating too far from the reference model.
The larger \beta is, the stronger the penalty on policy deviation usually is; the smaller it is, the more substantial policy adjustment the model is allowed to make.
Practical PPO implementations may use per-token KL estimation, reward shaping, and other training details, and do not necessarily directly compute the exact KL divergence over the full vocabulary.
7. Core Mathematical Mechanism of PPO
The main goal of PPO is to update the policy model stably and avoid an update causing too large a change in policy.
1. Policy Probability Ratio
Suppose:
- \pi_{\theta_{\text{old}}: Policy before the update.
- \pi_{\theta}: Current policy to be optimized.
For an action token, define the probability ratio:
where:
- s_t: the current context state.
- a_t: the token currently being generated.
This ratio reflects how much the probability of a given action has changed under the new policy relative to the old one.
2. Advantage Function
The reward model typically provides a score at the level of the final response, but PPO needs to estimate how much each step's action contributes to the final outcome.
This is where the Advantage function comes in.
where:
- Q(s_t,a_t): the expected return after taking action a_t in state s_t.
- V(s_t): the baseline value of being in state s_t.
Intuitively:
- A_t>0: the current action is better than expected.
- A_t<0: the current action is worse than expected.
- A_t\approx0: the current action is close to the baseline level.
In PPO, advantage estimates are usually used to guide policy updates.
3. PPO Clipped Objective
The classic clipped objective of PPO is:
where:
- \rho_t(\theta): the probability ratio between the new and old policies.
- A_t: the advantage estimate.
- \epsilon: the clipping range.
The clipping operation keeps the policy probability ratio within a certain range, reducing the risk of an excessively large policy update.
For example:
Suppose:
Then the probability ratio is usually constrained to:
Note that PPO's clipped objective is not a simple truncation of the policy probabilities; instead, the min operation in the objective function limits excessive optimization gains.
4. Why Is PPO Suited to Policy Optimization?
Traditional policy gradient methods can make policy performance unstable when a single update is too large.
PPO limits the update magnitude through its clipped objective, which generally makes it easier to implement and debug than some earlier policy optimization methods.
But PPO does not guarantee stable training.
In large language model settings, the following still need to be handled:
- quality of the reward model;
- KL penalty coefficient;
- advantage estimation;
- value model training;
- batch size;
- learning rate;
- generation length;
- distributed training and sampling efficiency.
5. Main Model Components of PPO in RLHF
Classic PPO-RLHF typically includes the following components:
| Component | Role | Updated |
|---|---|---|
| Policy Model | Generates responses | Yes |
| Reference Model | Constrains policy drift | No |
| Reward Model | Scores responses | Usually no |
| Value Model / Critic | Estimates state values | Yes |
The Policy Model and Value Model can be separate models, or they can be implemented by sharing part of the network structure.
8. The Complete RLHF-PPO Training Pipeline
Combining the reward model, the policy model, and PPO yields the classic RLHF training pipeline.
flowchart TB
A["Load SFT Policy Model"] --> B["Copy Reference Model"]
A --> C["Initialize Value Model"]
D["Trained Reward Model"] --> F["Reward Evaluation"]
E["Prompt Batch"] --> G["Policy Model Generates Response"]
G --> F
B --> H["Compute KL Penalty"]
F --> I["Construct Training Reward"]
H --> I
I --> J["Compute Advantage"]
C --> J
J --> K["PPO Clipped Objective"]
K --> L["Update Policy Model"]
K --> M["Update Value Model"]
L --> N{"Continue Training?"}
M --> N
N -->|"Yes"| E
N -->|"No"| O["Obtain RLHF-Aligned Model"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style O fill:#E5F0E9,stroke:#83A995,color:#315D49
Typical training steps:
- Sample inputs from the Prompt dataset.
- The Policy Model generates a response based on the input.
- The Reward Model scores the response.
- The Reference Model is used to calculate the degree of policy deviation.
- Combine the reward and KL penalty to construct the training signal.
- The Value Model estimates state values.
- Compute Advantage.
- Use PPO to update the Policy Model.
- Update the Value Model according to the training strategy.
- Repeat sampling and optimization.
1. Why do we need a Value Model?
PPO usually requires a Critic to estimate state values.
When a language model generates a response, it may contain hundreds or even thousands of tokens.
If only the reward of the final response is known, it is difficult to directly determine each token's contribution to the outcome.
The Value Model estimates state values and provides a baseline for Advantage calculation.
2. Why is RLHF computationally expensive?
Classic RLHF-PPO may simultaneously require:
- a policy model;
- a reference model;
- a reward model;
- a value model.
These models may all have a large number of parameters.
In addition, the following are also required:
- multiple rounds of text generation;
- reward model inference;
- value estimation;
- policy gradient computation;
- distributed parameter updates.
Therefore, the training system for classic PPO-RLHF is usually more complex than ordinary SFT, and has higher GPU memory, computation, and communication overhead.
9. DPO: Direct Preference Optimization
DPO (Direct Preference Optimization) is another important preference alignment method.
Its core idea is:
Instead of training a separate reward model and then optimizing the policy with PPO, it directly uses preference data to optimize the language model.
The key to DPO is that it starts from a reward maximization objective with a KL constraint, derives the relationship between policy probabilities and rewards, and then constructs an objective function that can be trained directly with preference data.
1. What problem does DPO solve?
Classic RLHF-PPO needs to maintain multiple model components:
- a policy model;
- a reference model;
- a reward model;
- a value model.
DPO, however, does not need to train a separate reward model, nor does it need to train the Value Model in PPO.
It directly uses:
such preference data for optimization.
where:
- x: input prompt.
- y_w: preferred response.
- y_l: dispreferred response.
The training objective is to increase the probability of preferred responses relative to dispreferred responses.
2. Core Idea of DPO
DPO does not simply require:
Because if we only keep increasing the probability of preferred responses, the model may become overly biased.
DPO introduces a reference model and compares the relative probability changes of the current policy and the reference policy for the two responses.
Its core form is:
It focuses on:
Whether the current model, relative to the reference model, is more inclined toward the preferred response rather than the dispreferred response.
10. Mathematical Principles and Derivation of DPO
The most important part of DPO is deriving the direct preference optimization loss from the KL-constrained reward optimization objective.
1. Starting from Reward Maximization
Suppose we want to find a policy \pi such that responses generated under input x receive higher reward.
At the same time, to prevent the policy from deviating too far from the reference model \pi_{\text{ref}}, a KL penalty is introduced:
where:
- r(x,y): response reward.
- \pi: policy to be optimized.
- \pi_{\text{ref}}: reference policy.
- \beta: KL regularization coefficient.
This objective considers both reward and policy deviation.
2. Deriving the Optimal Policy
Expand the KL divergence:
Substitute into the objective function:
while satisfying the probability normalization constraint:
Using Lagrange multipliers to solve, we can obtain the optimal policy:
where:
Z(x) is a normalization factor, which can also be understood as the partition function.
3. Deriving the Reward from the Optimal Policy
Take the logarithm of the optimal policy formula:
Rearranging gives:
This formula is crucial.
It shows that, under the above KL-regularized reward maximization framework, the reward can be expressed by the log probability ratio between the optimal policy and the reference policy, plus a normalization term that depends only on input x.
DPO uses exactly this relationship to avoid explicitly training a reward model.
4. Constructing the Preference Probability
The classic Bradley-Terry model assumes:
Substitute the reward expression above.
Since the two responses have the same input x, the normalization term \beta\log Z(x) cancels out.
Therefore:
Replacing the policy to be optimized \pi^* with the parameterized model \pi_{\theta} yields the preference probability model of DPO.
5. DPO Loss Function
This finally gives: