DPO — Direct Preference Optimization: Your Language Model is Secretly a Reward Model

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF).

However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model.

In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss.

The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for fitting a reward model, sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods.

Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

Large unsupervised language models (LMs) trained on very large datasets acquire surprising capabilities. However, these models are trained on data generated by humans with a wide variety of goals, priorities, and skillsets. Some of these goals and skillsets may not be desirable to imitate; for example, while we may want our AI coding assistant to understand common programming mistakes in order to correct them, nevertheless, when generating code, we would like to bias our model toward the (potentially rare) high-quality coding ability present in its training data.

Similarly, we might want our language model to be aware of a common misconception believed by 50% of people, but we certainly do not want the model to claim this misconception to be true in 50% of queries about it! In other words, selecting the model's desired responses and behavior from its very wide knowledge and abilities is crucial to building AI systems that are safe, performant, and controllable.

While existing methods typically steer LMs to match human preferences using reinforcement learning (RL), we will show that the RL-based objective used by existing methods can be optimized exactly with a simple binary cross-entropy objective, greatly simplifying the preference learning pipeline.

At a high level, existing methods instill the desired behaviors into a language model using curated sets of human preferences. This preference learning stage occurs after an initial stage of large-scale unsupervised pre-training. While the most straightforward approach is supervised fine-tuning on human demonstrations of high quality responses, the most successful class of methods is reinforcement learning from human (or AI) feedback (RLHF/RLAIF). RLHF methods fit a reward model to a dataset of human preferences and then use RL to optimize a language model policy to produce responses assigned high reward without drifting excessively far from the original model.

While RLHF produces models with impressive conversational and coding abilities, the RLHF pipeline is considerably more complex than supervised learning, involving training multiple LMs and sampling from the LM policy in the loop of training, incurring significant computational costs.

In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning. We propose Direct Preference Optimization (DPO), an algorithm that implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a KL-divergence constraint) but is simple to implement and straightforward to train.

Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective. Like existing algorithms, DPO relies on a theoretical preference model (such as the Bradley-Terry model) that measures how well a given reward function aligns with empirical preference data.

However, while existing methods use the preference model to define a preference loss to train a reward model and then train a policy that optimizes the learned reward model, DPO uses a change of variables to define the preference loss as a function of the policy directly.

Self-supervised language models of increasing scale learn to complete some tasks zero-shot or with few-shot prompts. However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and human-written completions. This `instruction-tuning' procedure enables LLMs to generalize to instructions outside of the instruction-tuning set and generally increase their usability.

Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation, summarization, story-telling, and instruction-following. These methods first optimize a neural network reward function for compatibility with the dataset of preferences under a preference model such as the Bradley-Terry model, then fine-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE, proximal policy optimization (PPO), or variants.

Outside of the context of language, learning policies from preferences has been studied in both bandit and reinforcement learning settings. Contextual bandit learning using preferences or rankings of actions, rather than rewards, is known as a contextual dueling bandit (CDB). In the absence of absolute rewards, theoretical analysis of CDBs substitutes the notion of an optimal policy with a von Neumann winner, a policy whose expected win rate against any other policy is at least 50%.

However, in the CDB setting, preference labels are given online, while in learning from human preferences, we typically learn from a fixed batch of offline preference-annotated action pairs. Similarly, preference-based RL (PbRL) learns from binary preferences generated by an unknown `scoring' function rather than rewards. Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve first explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it. We instead present a single stage policy learning approach that directly optimizes a policy to satisfy preferences.

We review the RLHF pipeline. It usually includes three phases: 1) supervised fine-tuning (SFT); 2) preference sampling and reward learning; and 3) RL optimization.

SFT: RLHF typically begins by fine-tuning a pre-trained LM with supervised learning on high-quality data for the downstream task(s) of interest (dialogue, summarization, etc.), to obtain a model $\pi^{\text{SFT}}$.

Reward Modelling Phase: In the second phase the SFT model is prompted with prompts $x$ to produce pairs of answers $(y_1, y_2)\sim\pi^{\text{SFT}}(y\mid x)$. These are then presented to human labelers who express preferences for one answer, denoted as $y_w\succ y_l\mid x$ where $y_w$ and $y_l$ denote the preferred and dispreferred completion amongst $(y_1, y_2)$ respectively.

The preferences are assumed to be generated by some latent reward model $r^*(y, x)$, which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) model being a popular choice (although more general Plackett-Luce ranking models are also compatible with the framework if we have access to several ranked answers). The BT model stipulates that the human preference distribution $p^*$ can be written as:

$$p^*(y_1\succ y_2\mid x)=\frac{\exp\left(r^*(x,y_1)\right)}{\exp\left(r^*(x,y_1)\right)+\exp\left(r^*(x,y_2)\right)} \tag{1}$$

Assuming access to a static dataset of comparisons $\mathcal{D}=\{x^{(i)}, y_w^{(i)}, y_l^{(i)}\}_{i=1}^N$ sampled from $p^*$, we can parametrize a reward model $r_\phi(x,y)$ and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss:

$$\mathcal{L}_R(r_\phi,\mathcal{D})=-\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[\log\sigma\left(r_\phi(x,y_w)-r_\phi(x,y_l)\right)\right] \tag{2}$$

where $\sigma$ is the logistic function. In the context of LMs, the network $r_\phi(x,y)$ is often initialized from the SFT model $\pi^{\text{SFT}}(y\mid x)$ with the addition of a linear layer on top of the final transformer layer that produces a single scalar prediction for the reward value. To ensure a reward function with lower variance, prior works normalize the rewards, such that $\mathbb{E}_{x,y\sim\mathcal{D}}\left[r_\phi(x,y)\right]=0$ for all $x$.

RL Fine-Tuning Phase: During the RL phase, the learned reward function is used to provide feedback to the language model. Following prior works, the optimization is formulated as

$$\max_{\pi_\theta}\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(y\mid x)}\left[r_\phi(x,y)\right]-\beta\,\mathbb{D}_{\mathrm{KL}}\left[\pi_\theta(y\mid x)\,\|\,\pi^{\text{ref}}(y\mid x)\right] \tag{3}$$

where $\beta$ is a parameter controlling the deviation from the base reference policy $\pi^{\text{ref}}$, namely the initial SFT model $\pi^{\text{SFT}}$. In practice, the language model policy $\pi_\theta$ is also initialized to $\pi^{\text{SFT}}$.

The added constraint is important, as it prevents the model from deviating too far from the distribution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers. Due to the discrete nature of language generation, this objective is not differentiable and is typically optimized with reinforcement learning.

The standard approach has been to construct the reward function $r(x,y)=r_\phi(x,y)-\beta\left(\log\pi_\theta(y\mid x)-\log\pi^{\text{ref}}(y\mid x)\right)$, and maximize using PPO.

Motivated by the challenges of applying reinforcement learning algorithms on large-scale problems such as fine-tuning language models, our goal is to derive a simple approach for policy optimization using preferences directly. Unlike prior RLHF methods, which learn a reward and then optimize it via RL, our approach leverages a particular choice of reward model parameterization that enables extraction of its optimal policy in closed form, without an RL training loop.

Our key insight is to leverage an analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functions into a loss function over policies. This change-of-variables approach avoids fitting an explicit, standalone reward model, while still optimizing under existing models of human preferences, such as the Bradley-Terry model. In essence, the policy network represents both the language model and the (implicit) reward.

Deriving the DPO objective. We start with the same RL objective as prior work, Eq. (3), under a general reward function $r$. It is straightforward to show that the optimal solution to the KL-constrained reward maximization objective takes the form:

$$\pi_r(y\mid x)=\frac{1}{Z(x)}\pi^{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right) \tag{4}$$

where $Z(x)=\sum_y \pi^{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)$ is the partition function.

Even if we use the MLE estimate $r_\phi$ of the ground-truth reward function $r^*$, it is still expensive to estimate the partition function $Z(x)$, which makes this representation hard to utilize in practice. However, we can rearrange Eq. (4) to express the reward function in terms of its corresponding optimal policy $\pi_r$, the reference policy $\pi^{\text{ref}}$, and the unknown partition function $Z(\cdot)$. Specifically, we first take the logarithm of both sides and then with some algebra we obtain:

$$r(x,y)=\beta\log\frac{\pi_r(y\mid x)}{\pi^{\text{ref}}(y\mid x)}+\beta\log Z(x) \tag{5}$$

We can apply this reparameterization to the ground-truth reward $r^*$ and corresponding optimal model $\pi^*$. Fortunately, the Bradley-Terry model depends only on the difference of rewards between two completions, i.e., $p^*(y_1\succ y_2\mid x)=\sigma\left(r^*(x,y_1)-r^*(x,y_2)\right)$. Substituting Eq. (5) into Eq. (1), the partition function cancels, and we can express the human preference probability in terms of only the optimal policy $\pi^*$ and reference policy $\pi^{\text{ref}}$.

Thus, the optimal RLHF policy $\pi^*$ under the Bradley-Terry model satisfies the preference model:

$$p^*(y_1\succ y_2\mid x)=\frac{1}{1+\exp\left(\beta\log\frac{\pi^*(y_2\mid x)}{\pi^{\text{ref}}(y_2\mid x)}-\beta\log\frac{\pi^*(y_1\mid x)}{\pi^{\text{ref}}(y_1\mid x)}\right)} \tag{6}$$

Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parametrized policy $\pi_\theta$. Analogous to the reward modeling approach (i.e. Eq. (2)), our policy objective becomes:

(see the full-width equation block below)

This way, we fit an implicit reward using an alternative parameterization, whose optimal policy is simply $\pi_\theta$. Moreover, since our procedure is equivalent to fitting a reparametrized Bradley-Terry model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution.

What does the DPO update do? For a mechanistic understanding of DPO, it is useful to analyze the gradient of the loss function $\mathcal{L}_{\text{DPO}}$. The gradient with respect to the parameters $\theta$ can be written as:

(see the full-width equation block below)

where $\hat{r}_\theta(x,y)=\beta\log\frac{\pi_\theta(y\mid x)}{\pi^{\text{ref}}(y\mid x)}$ is the reward implicitly defined by the language model $\pi_\theta$ and reference model $\pi^{\text{ref}}$.

Intuitively, the gradient increases the likelihood of the preferred completions $y_w$ and decreases the likelihood of dispreferred completions $y_l$. Importantly, the examples are weighed by how much higher the implicit reward model rates the dispreferred completions, scaled by $\beta$, i.e., how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naïve version of this method without the weighting coefficient can cause the language model to degenerate.

DPO outline. The general DPO pipeline is as follows: 1) Sample completions $y_1,y_2\sim\pi^{\text{ref}}(\cdot\mid x)$ for every prompt $x$, label with human preferences to construct the offline dataset $\mathcal{D}=\{x^{(i)},y_w^{(i)},y_l^{(i)}\}_{i=1}^N$ and 2) optimize the language model $\pi_\theta$ to minimize $\mathcal{L}_{\text{DPO}}$ for the given $\pi^{\text{ref}}$ and $\mathcal{D}$ and desired $\beta$.

In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using $\pi^{\text{SFT}}$, we initialize $\pi^{\text{ref}}=\pi^{\text{SFT}}$ whenever available. However, when $\pi^{\text{SFT}}$ is not available, we initialize $\pi^{\text{ref}}$ by maximizing likelihood of preferred completions $(x,y_w)$, that is, $\pi^{\text{ref}}=\arg\max_\pi\mathbb{E}_{x,y_w\sim\mathcal{D}}\left[\log\pi(y_w\mid x)\right]$. This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and $\pi^{\text{ref}}$ used by DPO.

In this section, we give further interpretation of the DPO method, provide theoretical backing, and relate advantages of DPO to issues with actor-critic algorithms used for RLHF (such as PPO).

Your Language Model Is Secretly a Reward Model. DPO is able to bypass both fitting an explicit reward and performing RL to learn the policy using a single maximum likelihood objective. Note the optimization objective Eq. (7) is equivalent to a Bradley-Terry model with a reward parameterization $r^*(x,y)=\beta\log\frac{\pi^*_\theta(y\mid x)}{\pi^{\text{ref}}(y\mid x)}$ and we optimize our parametric model $\pi_\theta$, equivalently to the reward model optimization in Eq. (2) under the change of variables.

In this section we will build the theory behind this reparameterization, show that it does not constrain the class of learned reward models, and allows for the exact recovery of the optimal policy. We begin by defining an equivalence relation between reward functions.

Definition. We say that two reward functions $r(x,y)$ and $r'(x,y)$ are equivalent iff $r(x,y)-r'(x,y)=f(x)$ for some function $f$.

It is easy to see that this is indeed an equivalence relation, which partitions the set of reward functions into classes. We can state the following two lemmas:

Lemma 1. Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution.

Lemma 2. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem.

The proofs are straightforward and we defer them to the appendix. The first lemma is a well-known under-specification issue with the Plackett-Luce family of models. Due to this under-specification, we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. (2). The second lemma states that all reward functions from the same class yield the same optimal policy, hence for our final objective, we are only interested in recovering an arbitrary reward function from the optimal class.

Theorem. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization $r(x,y)=\beta\log\frac{\pi(y\mid x)}{\pi^{\text{ref}}(y\mid x)}$ for some model $\pi(y\mid x)$ and a given reference model $\pi^{\text{ref}}(y\mid x)$.

Proof sketch. Consider any reward function $r(x,y)$, which induces a corresponding optimal model $\pi_r(y\mid x)$, specified by Eq. (4). We show that a reward function from the equivalence class of $r$ can be represented using the reparameterization. We define the projection $f(r;\pi^{\text{ref}},\beta)(x,y)=r(x,y)-\beta\log\sum_y\pi^{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)$. Since the added normalization term is only a function of the prefix $x$, it is a reward function in the equivalence class of $r(x,y)$. Replacing $r$ with the RHS of Eq. (5) gives $f(r;\pi^{\text{ref}},\beta)(x,y)=\beta\log\frac{\pi_r(y\mid x)}{\pi^{\text{ref}}(y\mid x)}$.

That is, the projection $f$ produces a member of the equivalence class of $r$ with the desired form, and we do not lose any generality in our reward model from the proposed reparameterization.

Instability of Actor-Critic Algorithms. We can also use our framework to diagnose instabilities with standard actor-critic algorithms used for RLHF, such as PPO. We follow the RLHF pipeline and focus on the RL fine-tuning step. We can draw connections to the control-as-inference framework for the constrained RL problem in Eq. (3). We assume a parameterized model $\pi_\theta(y\mid x)$ and minimize $\mathbb{D}_{\text{KL}}\left[\pi_\theta(y\mid x)\,\|\,\pi^*(y\mid x)\right]$ where $\pi^*$ is the optimal policy from Eq. (4) induced by the reward function $r_\phi(y,x)$. With some algebra this leads to the optimization objective:

(see the full-width equation block below)

This is the same objective optimized in prior works using the DPO-equivalent reward for the reward class of $r_\phi$. In this setting, we can interpret the normalization term in $f(r_\phi,\pi^{\text{ref}},\beta)$ as the soft value function of the reference policy $\pi^{\text{ref}}$. While this term does not affect the optimal solution, without it, the policy gradient of the objective could have high variance, making learning unstable.

We can accommodate for the normalization term using a learned value function, but that can also be difficult to optimize. Alternatively, prior works have normalized rewards using a human completion baseline, essentially a single sample Monte-Carlo estimate of the normalizing term. In contrast the DPO reparameterization yields a reward function that does not require any baselines.

In this section, we empirically evaluate DPO's ability to train policies directly from preferences. First, in a well-controlled text-generation setting, we ask: how efficiently does DPO trade off maximizing reward and minimizing KL-divergence with the reference policy, compared to common preference learning algorithms such as PPO? Next, we evaluate DPO's performance on larger models and more difficult RLHF tasks, including summarization and dialogue. We find that with almost no tuning of hyperparameters, DPO tends to perform as well or better than strong baselines like RLHF with PPO as well as returning the best of $N$ sampled trajectories under a learned reward function.

Tasks. Our experiments explore three different open-ended text generation tasks. In controlled sentiment generation, $x$ is a prefix of a movie review from the IMDb dataset, and the policy must generate $y$ with positive sentiment. For this experiment we generate preference pairs over generations using a pre-trained sentiment classifier, where $p(\text{positive}\mid x,y_w)>p(\text{positive}\mid x,y_l)$. For SFT, we fine-tune GPT-2-large until convergence on reviews from the train split of the IMDB dataset.

In summarization, $x$ is a forum post from Reddit; the policy must generate a summary $y$ of the main points in the post. We use the Reddit TL;DR summarization dataset along with human preferences gathered by Stiennon et al. We use an SFT model fine-tuned on human-written forum post summaries with the TRLX framework for RLHF.

Finally, in single-turn dialogue, $x$ is a human query, which may be anything from a question about astrophysics to a request for relationship advice. A policy must produce an engaging and helpful response $y$. We use the Anthropic Helpful and Harmless dialogue dataset, containing 170k dialogues between a human and an automated assistant. In this setting, no pre-trained SFT model is available; we therefore fine-tune an off-the-shelf language model on only the preferred completions to form the SFT model.

Evaluation. Our experiments use two different approaches to evaluation. In order to analyze the effectiveness of each algorithm in optimizing the constrained reward maximization objective, in the controlled sentiment generation setting we evaluate each algorithm by its frontier of achieved reward and KL-divergence from the reference policy; this frontier is computable because we have access to the ground-truth reward function (a sentiment classifier).

However, in the real world, the ground truth reward function is not known; therefore, we evaluate algorithms with their win rate against a baseline policy, using GPT-4 as a proxy for human evaluation of summary quality and response helpfulness. For summarization, we use reference summaries in the test set as the baseline; for dialogue, we use the preferred response in the test dataset as the baseline. We conduct a human study to justify our usage of GPT-4 for evaluation in §6.4. We find GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement.

Methods. In addition to DPO, we evaluate several existing approaches: zero-shot prompting with GPT-J (summarization) and 2-shot prompting with Pythia-2.8B (dialogue); the SFT model; Preferred-FT, fine-tuned with supervised learning on the chosen completion $y_w$; Unlikelihood, which maximizes the probability of $y_w$ and minimizes the probability of $y_l$ with an optional coefficient $\alpha\in[0,1]$; and PPO with the learned reward, as well as PPO-GT using the ground-truth reward in the sentiment setting.

The KL-constrained reward maximization objective used in typical RLHF algorithms balances exploitation of reward while restricting the policy from deviating far from the reference policy. Therefore, when comparing algorithms, we must take into account both reward achieved as well as the KL discrepancy; achieving slightly higher reward but with much higher KL is not necessarily desirable.

Figure 2 shows the reward-KL frontier for various algorithms in the sentiment setting. We execute multiple training runs for each algorithm, using a different hyperparameter for policy conservativeness in each run (target KL $\in\{3,6,9,12\}$ for PPO, $\beta\in\{0.05,0.1,1,5\}$, $\alpha\in\{0.05,0.1,0.5,1\}$ for unlikelihood, random seeds for preferred-FT). This sweep includes 22 runs in total. After each 100 training steps until convergence, we evaluate each policy on a set of test prompts, computing the average reward under the true reward function as well as the average sequence-level KL with the reference policy.

We find that DPO produces by far the most efficient frontier, achieving the highest reward while still achieving low KL. This result is particularly notable for multiple reasons. First, DPO and PPO optimize the same objective, but DPO is notably more efficient; DPO's reward/KL tradeoff strictly dominates PPO. Second, DPO achieves a better frontier than PPO, even when PPO can access ground truth rewards (PPO-GT).

Next, we evaluate fine-tuning performance of DPO on summarization and single-turn dialogue. For summarization, automatic evaluation metrics such as ROUGE can be poorly correlated with human preferences, and prior work has found that fine-tuning LMs using PPO on human preferences provides more effective summaries. We evaluate different methods by sampling completions on the test split of TL;DR and computing the average win rate against reference completions in the test set. The completions for all methods are sampled at temperatures varying from 0.0 to 1.0.

We find that DPO has a win rate of approximately 61% at a temperature of 0.0, exceeding the performance of PPO at ~57% at its optimal sampling temperature of 0.0. DPO also achieves a higher maximum win rate compared to the best of $N$ baseline. We note that we did not meaningfully tune DPO's $\beta$ hyperparameter, so these results may underestimate DPO's potential. Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade to that of the base GPT-J model at high temperatures.

On single-turn dialogue, we evaluate on the subset of the Anthropic HH test split with one step of human-assistant interaction. GPT-4 evaluations use the preferred completions on the test as the reference. As there is no standard SFT model for this task, we start with a pre-trained Pythia-2.8B, use Preferred-FT to train a reference model on the chosen completions, and then train using DPO. We also compare against the best of 128 Preferred-FT completions and a 2-shot prompted version of Pythia-2.8B. DPO is the only computationally efficient method that improves over the preferred completions in the Anthropic HH dataset, and provides similar or better performance to the computationally demanding Best of 128 baseline.

To further compare the performance of PPO and DPO under distribution shifts, we evaluate the PPO and DPO policies from our Reddit TL;DR summarization experiment on a different distribution, news articles in the test split of the CNN/DailyMail dataset, using the best sampling temperatures from TL;DR (0 and 0.25). We computed the GPT-4 win rate against the ground-truth summaries in the datasets, using the same GPT-4 (C) prompt we used for Reddit TL;DR, but replacing the words “forum post” with “news article”.

For this new distribution, DPO continues to outperform the PPO policy by a significant margin. This experiment provides initial evidence that DPO policies can generalize similarly well to PPO policies, even though DPO does not use the additional unlabeled Reddit TL;DR prompts that PPO uses.

We conduct a human study to verify the reliability of GPT-4's judgments, using the results of the TL;DR summarization experiment and two different GPT-4 prompts. The GPT-4 (S) (simple) prompt simply asks for which summary better-summarizes the important information in the post. The GPT-4 (C) (concise) prompt also asks for which summary is more concise; we evaluate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than humans do with the GPT-4 (S) prompt.

We perform three comparisons, using the highest (DPO, temp. 0.25), the lowest (PPO, temp. 1.0), and a middle-performing (SFT, temp. 0.25) method with the aim of covering a diversity of sample qualities; all three methods are compared against greedily-sampled PPO (its best-performing temperature).

We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations. Overall, the GPT-4 (C) prompt generally provides win rates more representative of humans; we therefore use this prompt for the main results in §6.2.

Learning from preferences is a powerful, scalable framework for training capable, aligned language models. We have introduced DPO, a simple training paradigm for training language models from preferences without reinforcement learning. Rather than coercing the preference learning problem into a standard RL setting in order to use off-the-shelf RL algorithms, DPO identifies a mapping between language model policies and reward functions that enables training a language model to satisfy human preferences directly, with a simple cross-entropy loss, without reinforcement learning or loss of generality.

With virtually no tuning of hyperparameters, DPO performs similarly or better than existing RLHF algorithms, including those based on PPO; DPO thus meaningfully reduces the barrier to training more language models from human preferences.

Limitations & Future Work. Our results raise several important questions. How does the DPO policy generalize out of distribution, compared with learning from an explicit reward function? Our initial results suggest DPO policies can generalize similarly to PPO-based models, but more comprehensive study is needed. Can training with self-labeling from the DPO policy similarly make effective use of unlabeled prompts? How does reward over-optimization manifest in the direct preference optimization setting, and is the slight decrease in performance in Figure 3-right an instance of it? While we evaluate models up to 6B parameters, scaling DPO to state-of-the-art models orders of magnitude larger is an exciting direction. Regarding evaluations, we find that the win rates computed by GPT-4 are impacted by the prompt; future work may study the best way to elicit high-quality judgments from automated systems. Finally, many possible applications of DPO exist beyond training language models from human preferences, including training generative models in other modalities.