Back to series
· 15 min read

RLHF and DPO Alignment Techniques: How Do Large Language Models Learn Human Preferences?


I. What Is Large Language Model Alignment?

Pretraining gives large language models the ability to do language modeling, associate knowledge, and generate text, while SFT teaches the model to understand instructions and generate responses as required.

But this does not mean the responses the model generates will necessarily match human preferences.

For example, a user asks:

Please explain what a database index is.

The model might generate two kinds of responses:

Response A:

A database index is a data structure used to speed up data queries. By maintaining additional index information, it reduces the amount of data that must be scanned during a query. Common index structures include B+ trees and hash indexes.

Response B:

A database index is a thing inside a database that can help the database run faster. It is very important and can improve database performance.

Both responses attempt to explain database indexes, but Response A is more accurate and specific, and its information is organized more clearly.

If only SFT is used, the model typically learns to imitate the standard responses in the training data. But when a question has multiple reasonable answers, pure supervised learning struggles to fully express the preference relationships among different answers.

This is the problem that preference alignment needs to solve.

1. What Is Alignment?

Large language model alignment refers to using training and other constraint mechanisms to make the model's output behavior better match the intended goals.

Common goals include:

  • Helpfulness: responses can actually solve the problem.
  • Truthfulness: minimize incorrect information as much as possible.
  • Safety: avoid generating harmful content.
  • Instruction Following: complete tasks according to user requirements.
  • Response Quality: responses are clear, accurate, and well structured.
  • Preference Consistency: among multiple reasonable responses, favor the more preferred one.

Note that alignment does not mean the model can guarantee that all responses are correct, nor does it mean human preferences are naturally equal to objective facts.

What the model learns is the preferences expressed by specific data, annotation standards, and reward mechanisms.

2. What Are RLHF and DPO?

Both RLHF and DPO can be used to optimize language models based on preference data, but they take different optimization paths.

RLHF (Reinforcement Learning from Human Feedback)

In Chinese it is usually called reinforcement learning from human feedback.

The classic RLHF pipeline usually includes:

  1. Use an SFT model to generate multiple candidate responses.
  2. Humans compare and rank the candidate responses.
  3. Use the preference data to train a reward model.
  4. Use a reinforcement learning algorithm (such as PPO) to optimize the language model.
  5. Adjust the model's behavior according to the reward signal.

DPO (Direct Preference Optimization)

In Chinese it is usually called direct preference optimization.

DPO directly uses preference data to optimize the language model; it does not require training a separate reward model, nor does it require online reinforcement learning like the classic RLHF-PPO pipeline.

The core differences between the two:

Comparison Dimension RLHF (Classic PPO Route) DPO
Preference data Required Required
Reward model Usually required No separate training needed
Reinforcement learning Uses algorithms such as PPO No traditional PPO pipeline needed
Reference model Usually required Usually required
Optimization approach Reward-driven policy optimization Direct optimization on preference pairs
Engineering complexity Usually higher Usually lower
Training stability Needs to handle RL stability issues Usually easier to implement
Applicable scenarios Complex reward optimization, online interaction, etc. Offline preference data optimization

Here, RLHF specifically refers to the classic reward-model-plus-PPO route. In a broad sense, RLHF can also include other reinforcement learning methods based on human feedback, while DPO is a direct preference optimization method.


II. From SFT to Preference Alignment

To understand RLHF and DPO, we first need to clarify the capability boundaries of SFT.

1. SFT Learns Target Responses

SFT data usually takes the form:

D_{\text{SFT}}=\{(x_i,y_i)\}_{i=1}^{N}

Where:

  • x_i: the input instruction.
  • y_i: the target response.
  • N: the number of training samples.

The model is trained by maximizing the conditional probability of the target response:

\max_{\theta} \sum_{i=1}^{N} \log P_{\theta}(y_i\mid x_i)

The corresponding negative log-likelihood loss is:

L_{\text{SFT}} = -\sum_{i=1}^{N} \log P_{\theta}(y_i\mid x_i)

The core of SFT is imitating the given target response.

2. Preference Data Expresses Relative Quality

Preference data is different from SFT data.

SFT usually tells the model:

How this question should be answered.

Preference data, on the other hand, tells the model:

For the same question, which response is more worth choosing.

For example:

Input:

Explain database indexes.

Candidate Answer A:

A database index is an auxiliary data structure that can reduce the range of data scanned during a query.
Common index structures include B+ trees and hash indexes.

Candidate Answer B:

A database index is just a feature inside a database.

Human annotators considered A better than B.

This preference data can be represented as:

(x,y_w,y_l)

Where:

  • x: the input question.
  • y_w: the preferred answer (Winner).
  • y_l: the non-preferred answer (Loser).

The model no longer learns only a single standard answer; instead, it learns the relative preference relationships between answers.

3. Basic Form of Preference Data

Common preference data structure:

{
  "prompt": "Explain database indexes",
  "chosen": "A database index is an auxiliary data structure that can speed up data queries.",
  "rejected": "A database index is just a feature in a database."
}

Actual datasets may also include:

  • annotator information;
  • scores;
  • multiple candidate answers;
  • rejection reasons;
  • safety labels;
  • task types;
  • data sources.

Preference data can come from human annotation as well as expert evaluation, model-assisted evaluation, and rule-based validation.

However, the quality of preference signals from different sources may vary, so quality control is needed.

4. Overall Relationship from SFT to Preference Alignment

flowchart TB
    A["Large-scale pretraining"] --> B["Base model"]
    B --> C["SFT supervised fine-tuning"]
    C --> D["Instruction model"]
    D --> E["Generate multiple candidate answers"]
    E --> F["Human or model preference evaluation"]
    F --> G["Construct preference data"]
    G --> H["RLHF or DPO"]
    H --> I["Preference-aligned model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

III. How Is Preference Data Constructed?

Preference data is the common foundation of RLHF and DPO.

Data quality directly affects the preference direction the model ultimately learns.

1. Generating Candidate Answers

First, use a model that has already undergone SFT to generate multiple candidate answers for the same input.

For example:

Prompt:
Please explain the TCP three-way handshake.

The model generates:

Answer A:
The TCP three-way handshake consists of three stages: SYN, SYN-ACK, and ACK,
and is used to establish a connection between the client and the server.
Answer B:
TCP establishes a connection by sending SYN, SYN-ACK, and ACK,
and during this process negotiates connection parameters such as the initial sequence number.
Answer C:
When a TCP connection is established, the client sends SYN, the server responds with ACK,
and then the client confirms the connection.

The strategy for generating candidate answers affects the quality of preference data.

If all candidate answers are very similar, it is difficult for annotators to provide valuable distinguishing information.

Therefore, a certain degree of answer diversity is usually required when constructing data.

2. Human Preference Annotation

Human annotators compare candidate answers.

Common annotation methods:

Pairwise Comparison

Directly choose the better of two answers.

Multi-candidate Ranking

Rank multiple candidate answers.

Graded Scoring

Score accuracy, completeness, safety, and expression quality separately.

Dimension-based Evaluation

Annotate preferences separately for different evaluation dimensions.

Pairwise comparison is a very common data format in classical preference training.

3. Preference Annotation Criteria

Preferences cannot rely solely on subjective impressions; clear evaluation criteria usually need to be established.

Dimension Evaluation Focus
Correctness Whether there are factual errors
Completeness Whether core issues are covered
Relevance Whether it stays closely tied to user needs
Clarity Whether the expression is easy to understand
Instruction Following Whether format and constraints are satisfied
Safety Whether harmful content exists
Conciseness Whether there is unnecessary redundancy

Different tasks may have different evaluation weights.

For example, code generation tasks place more emphasis on code correctness and executability, while creative writing tasks may place more emphasis on expression quality and creativity.

4. Noise in Preference Data

Human annotations are not always consistent.

For example:

  • Different annotators have different levels of knowledge;
  • Different preferences for answer length;
  • Different understandings of technical terms;
  • Different judgments of risk and safety;
  • There may be multiple reasonable answers to the same question.

Therefore, preference data may contain noise and subjectivity.

Common quality control methods include:

  • Establish unified annotation guidelines;
  • Use multiple annotators for cross-evaluation;
  • Review samples with disagreements;
  • Filter out low-confidence samples;
  • Record inter-annotator agreement;
  • Use experts to validate data in high-risk domains.

IV. RLHF: Reinforcement Learning from Human Feedback

RLHF is an important approach in large language model alignment technology.

Classic RLHF usually adopts three main stages:

  1. Supervised fine-tuning (SFT).
  2. Reward modeling.
  3. Reinforcement learning optimization (usually using PPO).

Its core idea is:

First, train a reward model using human preference data so that it can evaluate response quality; then, through reinforcement learning, make the language model tend to generate high-reward responses.

1. Overall Architecture of RLHF

flowchart TB
    A["Pretrained Base Model"] --> B["SFT Supervised Fine-Tuning"]
    B --> C["SFT Model"]
    C --> D["Generate Multiple Candidate Responses"]
    D --> E["Human Preference Annotation"]
    E --> F["Preference Dataset"]
    F --> G["Train Reward Model"]
    C --> H["Policy Model Initialization"]
    H --> I["Generate Response"]
    I --> J["Reward Model Scoring"]
    J --> K["KL Penalty and Reward Calculation"]
    K --> L["PPO Updates the Policy Model"]
    L --> I
    L --> M["Aligned Model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style L fill:#EAF3EC,stroke:#83A995,color:#315D49
    style M fill:#E5F0E9,stroke:#83A995,color:#315D49

In the figure, the policy model is responsible for generating responses, the reward model is responsible for evaluating responses, and PPO adjusts the policy model according to the reward signal.

In actual training, the reward model is usually kept frozen while the policy model is continuously updated; a reference model is also used to constrain the policy model from deviating too far from the initial SFT model.


V. Reward Model

The reward model is a core component in classic RLHF.

Its task is not to directly generate responses, but to score a given input and response.

1. Input and Output of the Reward Model

The reward model usually receives:

(x,y)

where:

  • x: user input.
  • y: model response.

and outputs a scalar reward:

r_{\phi}(x,y)\in\mathbb{R}

where:

  • \phi: reward model parameters.
  • r_{\phi}: the reward value predicted by the model.

For example:

Response Reward model output
Response A 2.8
Response B 0.7
Response C -1.2

These scores are only illustrative.

The reward value itself usually has no fixed absolute meaning; it is mainly used to compare the relative quality of different responses.

2. How is the reward model trained?

The reward model is trained using preference data.

Suppose:

  • y_w: the preferred response.
  • y_l: the dispreferred response.

We want the reward model to satisfy:

r_{\phi}(x,y_w)>r_{\phi}(x,y_l)

One classic approach is to use the Bradley-Terry preference model.

It defines the probability that response y_w is preferred over y_l as:

P(y_w\succ y_l\mid x) = \frac{ e^{r_{\phi}(x,y_w)} }{ e^{r_{\phi}(x,y_w)}+ e^{r_{\phi}(x,y_l)} }

By simplification:

P(y_w\succ y_l\mid x) = \sigma \left( r_{\phi}(x,y_w)-r_{\phi}(x,y_l) \right)

where:

\sigma(z)=\frac{1}{1+e^{-z}}

is the Sigmoid function.

This formula expresses an important idea:

The reward model does not need to directly predict the absolute scores given by humans; instead, it learns the relative difference between preferred and dispreferred responses.

3. Reward model loss function

For a preference sample, the loss function is:

L_{\text{RM}} = -\log \sigma \left( r_{\phi}(x,y_w)-r_{\phi}(x,y_l) \right)

It can also be written as:

L_{\text{RM}} = -\log \frac{ e^{r_{\phi}(x,y_w)} }{ e^{r_{\phi}(x,y_w)}+ e^{r_{\phi}(x,y_l)} }

If the reward model gives a higher score to the preferred response, the loss decreases.

If the reward model incorrectly gives a higher score to the dispreferred response, the loss increases.

For the entire dataset:

L_{\text{RM}} = -\mathbb{E}_{(x,y_w,y_l)\sim D} \left[ \log\sigma \left( r_{\phi}(x,y_w)-r_{\phi}(x,y_l) \right) \right]

4. Structure of the reward model

In a classic implementation, the reward model can be initialized from a pretrained language model or an SFT model.

After concatenating the input and response, they are fed into a Transformer, and the hidden state of the last valid token is passed to a scalar output head.

Simplified structure:

flowchart TB
    A["Prompt and Response"] --> B["Tokenizer Encoding"]
    B --> C["Transformer"]
    C --> D["Hidden State of Last Valid Token"]
    D --> E["Linear Reward Head"]
    E --> F["Scalar Reward"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#E5F0E9,stroke:#83A995,color:#315D49

The reward model usually needs a dedicated scoring head rather than directly using the language model's vocabulary output layer.

5. Limitations of the reward model

The reward model is only an approximate model of human preferences; it does not mean it truly understands response quality.

It may exhibit:

  • giving higher scores to verbose responses;
  • preferring expressions that appear more confident on the surface;
  • failing to accurately identify subtle factual errors;
  • being exploited by the policy model;
  • giving unreliable scores to responses outside the training distribution.

Particular attention should be paid to reward hacking (Reward Hacking).

Once the policy model discovers the scoring patterns of the reward model, it may learn to generate responses that receive high scores but are not actually of high quality.

This is also why RLHF requires additional constraints and evaluation.


VI. PPO: Using Reinforcement Learning to Optimize Language Models

After the reward model is trained, reinforcement learning can be used to optimize the language model.

Classic RLHF usually uses PPO (Proximal Policy Optimization).

1. Treating the language model as a policy model

There are several core concepts in reinforcement learning:

Reinforcement Learning Concept Corresponding Concept in Large Language Models
Agent Language model
Policy Token generation policy
State Current input and generated context
Action Currently generated token
Reward Reward provided by the reward model
Episode A single answer generation process
Trajectory Complete token generation sequence

Suppose the user input is:

Explain what an operating system is.

The model generates the answer step by step:

An operating system is system software that manages computer hardware and software resources.

Each time a token is generated, it can be regarded as the policy model performing one action.

Therefore, a language model can be represented as an autoregressive policy:

\pi_{\theta}(y_t\mid x,y_{<t})

where:

  • \theta: Policy model parameters.
  • x: User input.
  • y_{<t}: Prefix of the answer generated so far.
  • y_t: Currently generated token.

2. Where Does the Reward Signal Come From?

After the model generates a complete answer, the input and answer can be passed to the reward model for scoring.

For example:

r_{\phi}(x,y)=2.5

However, if only the final score from the reward model is used, the policy model may deviate excessively from the original SFT model.

Therefore, classic RLHF usually introduces a KL penalty.

3. Why Is a KL Penalty Needed?

Suppose the policy model keeps pursuing high scores given by the reward model.

Without constraints, the model may gradually deviate from the original SFT model and produce:

  • Unnatural expressions;
  • Repeated content;
  • Abnormally verbose answers;
  • Degradation of language ability;
  • Behavior that exploits loopholes in the reward model.

To address this, a reference model can be used to constrain the policy model.

The reference model is usually a frozen copy of the SFT model.

Define the KL divergence between the policy model and the reference model:

D_{\mathrm{KL}} \left( \pi_{\theta}(\cdot\mid s) \parallel \pi_{\mathrm{ref}}(\cdot\mid s) \right)

KL divergence measures the difference between two probability distributions.

For discrete distributions:

D_{\mathrm{KL}}(P\parallel Q) = \sum_i P(i)\log\frac{P(i)}{Q(i)}

In language model training, the KL penalty is usually used to constrain the policy model from deviating too far from the reference model.

4. Optimization Objective of RLHF

The objective can be abstracted as:

\max_{\pi_{\theta}} \mathbb{E}_{x\sim D,y\sim\pi_{\theta}(\cdot\mid x)} \left[ r_{\phi}(x,y) - \beta D_{\mathrm{KL}} \left( \pi_{\theta}(\cdot\mid x) \parallel \pi_{\mathrm{ref}}(\cdot\mid x) \right) \right]

where:

  • r_{\phi}(x,y): Reward given by the reward model.
  • \pi_{\theta}: Current policy model.
  • \pi_{\mathrm{ref}}: Reference model.
  • \beta: KL penalty coefficient.

This objective expresses two directions that balance each other:

  1. Increase the reward of the answer.
  2. Keep the model from deviating too far from the reference model.

The larger \beta is, the stronger the penalty on policy deviation usually is; the smaller it is, the more substantial policy adjustment the model is allowed to make.

Practical PPO implementations may use per-token KL estimation, reward shaping, and other training details, and do not necessarily directly compute the exact KL divergence over the full vocabulary.


7. Core Mathematical Mechanism of PPO

The main goal of PPO is to update the policy model stably and avoid an update causing too large a change in policy.

1. Policy Probability Ratio

Suppose:

  • \pi_{\theta_{\text{old}}: Policy before the update.
  • \pi_{\theta}: Current policy to be optimized.

For an action token, define the probability ratio:

\rho_t(\theta) = \frac{ \pi_{\theta}(a_t\mid s_t) }{ \pi_{\theta_{\text{old}}}(a_t\mid s_t) }

where:

  • s_t: the current context state.
  • a_t: the token currently being generated.

This ratio reflects how much the probability of a given action has changed under the new policy relative to the old one.

2. Advantage Function

The reward model typically provides a score at the level of the final response, but PPO needs to estimate how much each step's action contributes to the final outcome.

This is where the Advantage function comes in.

A_t=Q(s_t,a_t)-V(s_t)

where:

  • Q(s_t,a_t): the expected return after taking action a_t in state s_t.
  • V(s_t): the baseline value of being in state s_t.

Intuitively:

  • A_t>0: the current action is better than expected.
  • A_t<0: the current action is worse than expected.
  • A_t\approx0: the current action is close to the baseline level.

In PPO, advantage estimates are usually used to guide policy updates.

3. PPO Clipped Objective

The classic clipped objective of PPO is:

L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min \left( \rho_t(\theta)A_t, \operatorname{clip} (\rho_t(\theta),1-\epsilon,1+\epsilon)A_t \right) \right]

where:

  • \rho_t(\theta): the probability ratio between the new and old policies.
  • A_t: the advantage estimate.
  • \epsilon: the clipping range.

The clipping operation keeps the policy probability ratio within a certain range, reducing the risk of an excessively large policy update.

For example:

Suppose:

\epsilon=0.2

Then the probability ratio is usually constrained to:

[0.8,1.2]

Note that PPO's clipped objective is not a simple truncation of the policy probabilities; instead, the min operation in the objective function limits excessive optimization gains.

4. Why Is PPO Suited to Policy Optimization?

Traditional policy gradient methods can make policy performance unstable when a single update is too large.

PPO limits the update magnitude through its clipped objective, which generally makes it easier to implement and debug than some earlier policy optimization methods.

But PPO does not guarantee stable training.

In large language model settings, the following still need to be handled:

  • quality of the reward model;
  • KL penalty coefficient;
  • advantage estimation;
  • value model training;
  • batch size;
  • learning rate;
  • generation length;
  • distributed training and sampling efficiency.

5. Main Model Components of PPO in RLHF

Classic PPO-RLHF typically includes the following components:

Component Role Updated
Policy Model Generates responses Yes
Reference Model Constrains policy drift No
Reward Model Scores responses Usually no
Value Model / Critic Estimates state values Yes

The Policy Model and Value Model can be separate models, or they can be implemented by sharing part of the network structure.


8. The Complete RLHF-PPO Training Pipeline

Combining the reward model, the policy model, and PPO yields the classic RLHF training pipeline.

flowchart TB
    A["Load SFT Policy Model"] --> B["Copy Reference Model"]
    A --> C["Initialize Value Model"]
    D["Trained Reward Model"] --> F["Reward Evaluation"]
    E["Prompt Batch"] --> G["Policy Model Generates Response"]
    G --> F
    B --> H["Compute KL Penalty"]
    F --> I["Construct Training Reward"]
    H --> I
    I --> J["Compute Advantage"]
    C --> J
    J --> K["PPO Clipped Objective"]
    K --> L["Update Policy Model"]
    K --> M["Update Value Model"]
    L --> N{"Continue Training?"}
    M --> N
    N -->|"Yes"| E
    N -->|"No"| O["Obtain RLHF-Aligned Model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style O fill:#E5F0E9,stroke:#83A995,color:#315D49

Typical training steps:

  1. Sample inputs from the Prompt dataset.
  2. The Policy Model generates a response based on the input.
  3. The Reward Model scores the response.
  4. The Reference Model is used to calculate the degree of policy deviation.
  5. Combine the reward and KL penalty to construct the training signal.
  6. The Value Model estimates state values.
  7. Compute Advantage.
  8. Use PPO to update the Policy Model.
  9. Update the Value Model according to the training strategy.
  10. Repeat sampling and optimization.

1. Why do we need a Value Model?

PPO usually requires a Critic to estimate state values.

When a language model generates a response, it may contain hundreds or even thousands of tokens.

If only the reward of the final response is known, it is difficult to directly determine each token's contribution to the outcome.

The Value Model estimates state values and provides a baseline for Advantage calculation.

2. Why is RLHF computationally expensive?

Classic RLHF-PPO may simultaneously require:

  • a policy model;
  • a reference model;
  • a reward model;
  • a value model.

These models may all have a large number of parameters.

In addition, the following are also required:

  • multiple rounds of text generation;
  • reward model inference;
  • value estimation;
  • policy gradient computation;
  • distributed parameter updates.

Therefore, the training system for classic PPO-RLHF is usually more complex than ordinary SFT, and has higher GPU memory, computation, and communication overhead.


9. DPO: Direct Preference Optimization

DPO (Direct Preference Optimization) is another important preference alignment method.

Its core idea is:

Instead of training a separate reward model and then optimizing the policy with PPO, it directly uses preference data to optimize the language model.

The key to DPO is that it starts from a reward maximization objective with a KL constraint, derives the relationship between policy probabilities and rewards, and then constructs an objective function that can be trained directly with preference data.

1. What problem does DPO solve?

Classic RLHF-PPO needs to maintain multiple model components:

  • a policy model;
  • a reference model;
  • a reward model;
  • a value model.

DPO, however, does not need to train a separate reward model, nor does it need to train the Value Model in PPO.

It directly uses:

(x,y_w,y_l)

such preference data for optimization.

where:

  • x: input prompt.
  • y_w: preferred response.
  • y_l: dispreferred response.

The training objective is to increase the probability of preferred responses relative to dispreferred responses.

2. Core Idea of DPO

DPO does not simply require:

P_{\theta}(y_w\mid x) > P_{\theta}(y_l\mid x)

Because if we only keep increasing the probability of preferred responses, the model may become overly biased.

DPO introduces a reference model and compares the relative probability changes of the current policy and the reference policy for the two responses.

Its core form is:

\log \frac{P_{\theta}(y_w\mid x)} {P_{\text{ref}}(y_w\mid x)} - \log \frac{P_{\theta}(y_l\mid x)} {P_{\text{ref}}(y_l\mid x)}

It focuses on:

Whether the current model, relative to the reference model, is more inclined toward the preferred response rather than the dispreferred response.


10. Mathematical Principles and Derivation of DPO

The most important part of DPO is deriving the direct preference optimization loss from the KL-constrained reward optimization objective.

1. Starting from Reward Maximization

Suppose we want to find a policy \pi such that responses generated under input x receive higher reward.

At the same time, to prevent the policy from deviating too far from the reference model \pi_{\text{ref}}, a KL penalty is introduced:

\max_{\pi} \mathbb{E}_{y\sim\pi(\cdot\mid x)} [r(x,y)] - \beta D_{\mathrm{KL}} \left( \pi(\cdot\mid x) \parallel \pi_{\text{ref}}(\cdot\mid x) \right)

where:

  • r(x,y): response reward.
  • \pi: policy to be optimized.
  • \pi_{\text{ref}}: reference policy.
  • \beta: KL regularization coefficient.

This objective considers both reward and policy deviation.

2. Deriving the Optimal Policy

Expand the KL divergence:

D_{\mathrm{KL}}(\pi\parallel\pi_{\text{ref}}) = \sum_y \pi(y\mid x) \log \frac{\pi(y\mid x)} {\pi_{\text{ref}}(y\mid x)}

Substitute into the objective function:

\max_{\pi} \sum_y\pi(y\mid x)r(x,y) - \beta \sum_y \pi(y\mid x) \log \frac{\pi(y\mid x)} {\pi_{\text{ref}}(y\mid x)}

while satisfying the probability normalization constraint:

\sum_y\pi(y\mid x)=1

Using Lagrange multipliers to solve, we can obtain the optimal policy:

\pi^*(y\mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y\mid x) \exp\left(\frac{r(x,y)}{\beta}\right)

where:

Z(x) = \sum_y \pi_{\text{ref}}(y\mid x) \exp\left(\frac{r(x,y)}{\beta}\right)

Z(x) is a normalization factor, which can also be understood as the partition function.

3. Deriving the Reward from the Optimal Policy

Take the logarithm of the optimal policy formula:

\log\pi^*(y\mid x) = \log\pi_{\text{ref}}(y\mid x) + \frac{r(x,y)}{\beta} - \log Z(x)

Rearranging gives:

r(x,y) = \beta \log \frac{\pi^*(y\mid x)} {\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x)

This formula is crucial.

It shows that, under the above KL-regularized reward maximization framework, the reward can be expressed by the log probability ratio between the optimal policy and the reference policy, plus a normalization term that depends only on input x.

DPO uses exactly this relationship to avoid explicitly training a reward model.

4. Constructing the Preference Probability

The classic Bradley-Terry model assumes:

P(y_w\succ y_l\mid x) = \sigma \left( r(x,y_w)-r(x,y_l) \right)

Substitute the reward expression above.

Since the two responses have the same input x, the normalization term \beta\log Z(x) cancels out.

Therefore:

P(y_w\succ y_l\mid x) = \sigma \left( \beta \log \frac{\pi^*(y_w\mid x)} {\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi^*(y_l\mid x)} {\pi_{\text{ref}}(y_l\mid x)} \right)

Replacing the policy to be optimized \pi^* with the parameterized model \pi_{\theta} yields the preference probability model of DPO.

5. DPO Loss Function

This finally gives:

\boxed{ L_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim D} \left[ \log\sigma \left( \beta \log \frac{\pi_{\theta}(y_w\mid x)} {\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_{\theta}(y_l\mid x)} {\pi_{\text{ref}}(y_l\mid x)} \right) \right] }