# Robust Preference Optimization
through Reward Model Distillation

Adam Fisch∗ Jacob Eisenstein∗ Vicky Zayats∗ Alekh Agarwal

Ahmad Beirami Chirag Nagpal Pete Shaw Jonathan Berant∗

Google DeepMind

###### Abstract

Language model (LM) post-training (or alignment) involves maximizing a reward
function that is derived from preference annotations. Direct Preference
Optimization (DPO) is a popular offline alignment method that trains a policy
directly on preference data without the need to train a reward model or apply
reinforcement learning. However, typical preference datasets have only a
single, or at most a few, annotation per preference pair, which causes DPO to
overconfidently assign rewards that trend towards infinite magnitude. This
frequently leads to degenerate policies, sometimes causing even the
probabilities of the _preferred_ generations to go to zero. In this work, we
analyze this phenomenon and propose _distillation_ to get a better proxy for
the true preference distribution over generation pairs: we train the LM to
produce probabilities that match the distribution induced by a reward model
trained on the preference data. Moreover, to account for uncertainty in the
reward model we are distilling from, we optimize against a _family of reward
models_ that, as a whole, is likely to include at least one reasonable proxy
for the preference distribution. Our results show that distilling from such a
family of reward models leads to improved robustness to distribution shift in
preference annotations, while preserving the simple supervised nature of DPO.

11footnotetext: Core contributor. Correspondence to<EMAIL_ADDRESS>
##  1 Introduction

Language model (LM) post-training (or alignment) aims to steer language model
policies towards responses that agree with human preferences. Early state-of-
the-art approaches have focused on reward learning from human feedback. In
this paradigm, preference annotations are used to train reward models, which
then guide the optimization of the language model policy through online
reinforcement learning (an approach broadly referred to as RLHF). Recent
research on offline “Direct Preference Optimization” [DPO; 23] and extensions
thereof [3; 31], however, has demonstrated that it is also possible to
directly optimize policies on the preference data, which bypasses the need for
a separate reward model—and its offline nature also leads to faster, and
simpler, training frameworks.

While this direct approach to preference optimization is attractive in terms
of its simplicity and efficiency, it also raises important questions about the
effectiveness and robustness of the resulting policies—as well as the broader
utility of using an explicit reward model. In this paper, we argue that
explicit reward modeling can, in fact, offer substantial practical advantages
that are not captured by DPO’s formulation. In particular, we theoretically
show that relying solely on the preference data can be a precarious strategy,
with few natural brakes in place to prevent policies trained under the DPO
objective from careening off towards degenerate policies when the preference
data exhibits certain idiosyncratic properties. On the other hand, explicit
reward models can easily be regularized and understood—regardless of whether
they are Bradley-Terry models [4], margin-based ranking models [40], or simply
any other kind of function that correlates well with human preferences [31;
17].

Taking a step back from pure direct preference optimization, we propose a
method that merges the best of both worlds: an efficient reward model
distillation algorithm that (i) operates effectively in the offline setting,
(ii) makes minimal assumptions about the true, optimal reward we aim to
maximize, and (iii) demonstrates greater robustness to the specific
distribution of prompt/response data used for policy alignment. Drawing
inspiration from prior knowledge distillation techniques [14; 26; 35; 10], we
leverage the same change of variables trick employed in DPO to express the
language model policy in terms of its implicit reward model [23]. We then
train the policy to match our desired, explicit reward via an $L_{2}$ loss
that directly regresses the pairwise differences in target rewards for any two
generation pairs $(x,y_{1})$ and $(x,y_{2})$. We theoretically establish the
equivalence between optimizing this simple distillation loss over a
sufficiently diverse offline dataset of unlabeled examples and optimizing the
traditional online RLHF objective with reinforcement learning.

Our reward model distillation approach, however, is not immune to some of the
same challenges facing DPO-style learning of policies. In particular, reward
model distillation requires having a reliable reward model—but having a
reliable reward requires having a reliable method for extracting a reward
model from a potentially noisy preference dataset. To address the uncertainty
surrounding the “right” reward model, we introduce a pessimistic extension to
our approach. This extension aims to maximize the worst-case improvement of
our model across a plausible family of reward models (e.g., those sufficiently
consistent with annotated preference data). This strategy aligns with that of
existing work in conservative offline reinforcement learning [5; 16].
Interestingly, we derive that this pessimistic objective can be equivalently
expressed and optimized by adding a simple additional KL-divergence
regularization to the original distillation objective.

Empirically, we find that reward model distillation, particularly pessimistic
reward model distillation, leads to similar performance to prior direct
preference optimization methods in settings where the preference datasets used
are unbiased, but significantly better performance in settings where the
preference datasets are biased, when compared to DPO and the Identity
Preference Optimization (IPO) framework of [3], which was introduced as a more
robust alternative to DPO. To further support these empirical observations, we
provide an extensive theoretical analysis that both (i) sheds more light on
the degenerative tendencies of DPO and issues inherent to its objective, and
(ii) highlights relative advantages of our explicitly regularized approaches.

##  2 Preliminaries

We begin with a brief review of Direct Preference Optimization (DPO) [23] and
its analysis. Proofs of all theoretical results provided here, and in the rest
of the paper, are deferred to Appendix A.

###  2.1 The preference alignment problem

Let $x$ be an input prompt, and let $y\sim\pi_{\theta}(\cdot\mid x)$ be the
language model policy $\pi_{\theta}$’s response to $x$. Given some reward
function $r^{*}(x,y)$ and another reference policy $\pi_{\mathrm{ref}}(y\mid
x)$, the goal of alignment is to solve for the “aligned” policy
$\pi_{\theta^{*}}(y\mid x)$ that maximizes the following RLHF objective, i.e.,

$\pi_{\theta^{*}}(y\mid
x)=\arg\\!\max_{\pi_{\theta}}\mathbb{E}_{\mu(x)}\left[\mathbb{E}_{\pi_{\theta}(y\mid
x)}[r^{*}(x,y)]-\beta\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}(\cdot\mid
x)\|\pi_{\mathrm{ref}}(\cdot\mid x)]\right],$ (1)

where $\mu(x)$ is a fixed distribution over prompts, and the KL-divergence
term prevents the aligned policy from being dramatically different from the
anchoring reference policy, $\pi_{\mathrm{ref}}(y\mid x)$. Here, the reward
function $r^{*}$ is typically not known in advance, but rather inferred from
collected human preference data in the form of $(x,y^{w},y^{\ell})$, where $x$
is the prompt, $y^{w}$ is the “winning”, or preferred, response, and
$y^{\ell}$ is the “losing”, or dispreferred, response. A common approach is to
assume that pairs $(y_{1},y_{2})$ follow a Bradley-Terry model [4], under
which the probability that $y_{1}$ is preferred to $y_{2}$ given the reward
function $r^{*}$ and prompt $x$ is $p^{*}(y_{1}\succ y_{2}\mid
x)=\sigma(r^{*}(x,y_{1})-r^{*}(x,y_{2}))$, where $\sigma(\cdot)$ is the
sigmoid function and $\succ$ denotes preference. Under this model, we can use
the preference data $(x,y^{w},y^{\ell})\sim\mathcal{D}_{\mathrm{pref}}$ to
estimate $r^{*}$ via maximum likelihood estimation, i.e.,

$\hat{r}\in\arg\\!\min_{r}\mathbb{E}_{(y^{w},y^{\ell},x)\sim\mathcal{D}_{\mathrm{pref}}}\left[-\log\sigma(r(x,y^{w})-r_{\phi}(x,y^{\ell}))\right].$
(2)

With $\hat{r}$ in hand, Eq. (1) can be optimized using standard reinforcement
learning algorithms [27; 29; 6].

###  2.2 Direct preference optimization

DPO is a simple approach for offline policy optimization that uses preferences
to directly align the language model policy, without training an intermediate
reward model. Specifically, DPO leverages the fact that the optimal solution
to the KL-constrained objective in (1) takes the form [15]

$\pi_{\theta^{*}}(y\mid x)=\frac{1}{{Z}(x)}\pi_{\mathrm{ref}}(y\mid
x)\exp\left(\frac{1}{\beta}r^{*}(x,y)\right),$ (3)

where ${Z}(x)=\sum_{y}\pi_{\mathrm{ref}}(y\mid
x)\exp(\frac{1}{\beta}r^{*}(x,y))$ is the partition function. DPO
reparameterizes the true reward function $r^{*}$ in terms of the optimal
policy $\pi_{\theta^{*}}$ that it induces, i.e.,

$r^{*}(x,y)=\beta\log\left(\frac{\pi_{\theta^{*}}(y\mid
x)}{\pi_{\mathrm{ref}}(y\mid x)}\right)+\beta\log{Z}(x).$ (4)

Under the Bradley-Terry model, the likelihood that $y_{1}\succ y_{2}$ can then
be written as

$p^{*}(y_{1}\succ y_{2}\mid
x)=\sigma\left(\beta\log\frac{\pi_{\theta^{*}}(y_{1})\pi_{\mathrm{ref}}(y_{2})}{\pi_{\theta^{*}}(y_{2})\pi_{\mathrm{ref}}(y_{1})}\right),$
(5)

where now $\pi_{\theta^{*}}$ can be directly estimated on
$\mathcal{D}_{\mathrm{pref}}$ following the objective in (2), in place of the
intermediate reward model $\hat{r}$, i.e., $\pi_{\hat{\theta}}(y\mid
x)\in\arg\\!\min_{\pi_{\theta}}\mathcal{L}_{\mathrm{dpo}}(\pi_{\theta};\mathcal{D}_{\mathrm{pref}})$
where

$\mathcal{L}_{\mathrm{dpo}}(\pi_{\theta};\mathcal{D}_{\mathrm{pref}})=\mathbb{E}_{(y^{w},y^{\ell},x)\sim\mathcal{D}_{\mathrm{pref}}}\left[-\log\sigma\left(\beta\log\frac{\pi_{\theta^{*}}(y^{w})\pi_{\mathrm{ref}}(y^{\ell})}{\pi_{\theta^{*}}(y^{\ell})\pi_{\mathrm{ref}}(y^{w})}\right)\right].$
(6)

###  2.3 Pitfalls of direct preference optimization

As argued in [3], the Bradley-Terry assumption that DPO strongly relies on for
maximum likelihood estimation is sensitive to the underlying preference data.
Specifically, if we have any two responses $y_{1}$ and $y_{2}$ where
$p^{*}(y_{1}\succ y_{2}\mid x)=1$, then the Bradley-Terry model dictates that
$r^{*}(y_{1})-r^{*}(y_{2})=+\infty$, and therefore $\pi_{\theta^{*}}(y_{2}\mid
x)=0$ for _any_ finite KL-regularization strength $\beta$.

We can illustrate this phenomenon on a broader level with the following
example.

######  Assumption 1.

Suppose we are given a preference dataset of (context-free) pairs
$\mathcal{D}_{\mathrm{pref}}=\\{(y^{w}_{i},y^{\ell}_{i})\\}_{i=1}^{n}$, the
pairs $(y^{w}_{i},y^{\ell}_{i})$ are mutually disjoint in both the elements.
Further suppose that we optimize the DPO objective on
$\mathcal{D}_{\mathrm{pref}}$ with a single parameter $\theta_{y}$ for each
$y$.

######  Proposition 1.

Under Assumption 1, for any $(y,y^{\prime})$ such that $y=y^{w}_{i}$ and
$y^{\prime}=y^{\ell}_{i}$ for some $i$, we have
$\frac{\pi_{{\theta}^{*}}(y)\pi_{\mathrm{ref}}(y^{\prime})}{\pi_{{\theta}^{*}}(y^{\prime})\pi_{\mathrm{ref}}(y)}\to\infty$,
for all global minimizers $\pi_{\theta^{*}}$ of the DPO objective in (6), for
any $\beta>0$.

######  Corollary 1.

Under Assumption 1, further assume that $0<\pi_{\mathrm{ref}}(y)<1$ for all
$y$. Then $\pi_{\theta^{*}}$ is a global minimizer of the DPO objective in (6)
iff $\pi_{\theta^{*}}(\mathcal{C}(y^{\ell})^{c})\to 1$ with
$\pi_{\theta^{*}}(y^{w}_{i})>0$ $\forall i\in[n]$, where
$\mathcal{C}(y^{\ell})^{c}$ is the complement of the set of all responses $y$
that appear as a dispreferred $y^{\ell}_{i}$ for any $i\in[n]$.

Additional analysis of the training dynamics of DPO is also provided in §5. A
significant, and non-obvious, implication of Corollary 1 is that the set of
global optima of the DPO loss also includes policies that can shift nearly all
probability mass to responses that never even appear in the training set—and
even assign near zero probability to all of the training data responses that
do in fact correspond to winning generations, $y^{w}$, a phenomenon that has
been observed empirically [e.g., 20]. Stated differently, Corollary 1 implies
that any $\theta^{*}$ merely satisfying $\pi_{\theta^{*}}(y^{\ell}_{i})=0$
with $\pi_{\theta^{*}}(y^{w}_{i})>0$ $\forall i\in[n]$ is a global minimizer
of the DPO objective in this setting. Though simplistic, the scenario in
Assumption 1 is closer to reality than might first be appreciated: in many
practical situations we can almost always expect the finite-sample preference
data to contain one (or at most a few) preference annotations per example
$(x,y_{1},y_{2})$, while the policies $\pi_{\theta}$ can have billions of
parameters ($\gg n$). Of course, this issue can also be viewed as a classic
instance of overfitting—with the additional caveat that as opposed to
_overpredicting_ responses within the training set, we might overfit to
_almost never_ producing anything like the “good” responses that do appear
within the training set. Furthermore, without additional regularization
(beyond $\beta$), we can expect this degeneration to easily happen in typical
preference datasets.

##  3 Uncertainty-aware reward model distillation

As discussed in the previous section, a core issue in preference optimization
is that the true preference distribution $p^{*}(y_{1}\succ y_{2}\mid x)$ is
not known. Attempting to infer it from finite-sample preference data (that may
further be biased or out-of-distribution with respect to the target domain)
can then result in a failure to learn reasonable policies. In this section, we
now propose an inherently regularized approach to direct preference
optimization that uses uncertainty-aware reward model distillation.

###  3.1 Reward model distillation

Suppose for the moment that the reward function $r^{*}$ was in fact known, and
did not have to be inferred from sampled preference data. Under this setting,
we can then define an efficient offline optimization procedure that is similar
in spirit to DPO, but no longer relies directly on a preference dataset.
Concretely, given unlabeled samples $(x,y_{1},y_{2})\sim\rho$ (where the
number of samples can be potentially unlimited), we can define a simple
“distillation” loss, $\mathcal{L}_{\mathrm{distill}}(r^{*},\pi_{\theta})$, as
follows:

$\displaystyle\mathcal{L}_{\mathrm{distill}}(r^{*},\pi_{\theta};\rho)=\mathbb{E}_{\rho(x,y_{1},y_{2})}\left[\left(r^{*}(x,y_{1})-r^{*}(x,y_{2})-\beta\log\frac{\pi_{\theta}(y_{1}\mid
x)\pi_{\mathrm{ref}}(y_{2}\mid x)}{\pi_{\theta}(y_{2}\mid
x)\pi_{\mathrm{ref}}(y_{1}\mid x)}\right)^{2}\right].$ (7)

Intuitively, the distillation loss seeks to exactly match _differences_ in
reward model scores across all generation pairs $(x,y_{1},y_{2})$. It is then
easy to see that under the Bradley-Terry model, this is equivalent to matching
the strength of the preference relationship, $y_{1}\succ y_{2}$. Furthermore,
by only matching differences, we can still conveniently ignore the log
partition term, $\log Z(x)$, in the implicit reward formulation for
$\pi_{\theta}$ as shown in (4), as it is constant across different $y$ for any
given $x$. Finally, similar to the motivation in DPO, we can show that
minimizing $\mathcal{L}_{\mathrm{distill}}(r^{*},\pi_{\theta};\rho)$ indeed
results in an optimally aligned policy $\pi_{\theta^{*}}$, as long as the data
distribution $\rho$ has sufficient support.

######  Theorem 1.

Let $\mathcal{Y}$ denote the set of all possible responses for any model
$\pi_{\theta}.$ Assume that $\mathrm{supp}(\pi_{\mathrm{ref}}(y\mid
x))=\mathcal{Y}$, i.e., the reference policy may generate any outcome with
non-zero probability. Further, let
$\mathrm{supp}(\rho(x,y_{1},y_{2}))=\mathrm{supp}(\mu(x))\times\mathcal{Y}\times\mathcal{Y}$.
Let $\pi_{\theta^{*}}(y\mid
x)\in\arg\\!\min_{\pi_{\theta}}\mathcal{L}_{\mathrm{distill}}(r^{*},\pi_{\theta};\rho)$
be a minimizer over all possible policies, of the implicit reward distillation
loss in (7), for which $r^{*}(x,y)$ is assumed to be deterministic, and finite
everywhere. Then for any $\beta>0$, $\pi_{\theta^{*}}$ also maximizes the
alignment objective in (1).

The above result holds for a broad class of data distributions
$\rho(x,y_{1},y_{2})$, and makes no assumptions on $r^{*}$ (e.g., it is no
longer necessary for it to be defined using a Bradley-Terry model). In fact,
this result can also be seen as strict generalization of the IPO framework of
[3] when taking $r^{*}(x,y)\triangleq\mathbf{1}\\{y=y_{w}\\}$, if labeled
pairs $(x,y_{w},y_{l})$ are provided instead of the unlabeled pairs
$(x,y_{1},y_{2})$.

Of course, the true reward $r^{*}$ is usually not known in practice. Still, as
in standard RLHF, we can go about constructing good proxies by using the
preference data to identify plausible target reward models
$r_{\mathrm{tgt}}$—further guided by any amount of regularization and
inductive bias that we desire. A natural choice is to first learn
$r_{\mathrm{tgt}}$ on the preference data $\mathcal{D}_{\mathrm{pref}}$ using
standard methods, and then reuse $\mathcal{D}_{\mathrm{pref}}$ to distill
$\pi_{\theta}$, which is similar to classical settings in teacher-based model
distillation [14; 26]. Furthermore, as $r_{\mathrm{tgt}}$ is a real-valued
model, at a bare minimum it is guaranteed to induce a regularized Bradley-
Terry preference distribution $p_{\mathrm{tgt}}(y_{1}\succ y_{2}\mid
x)>0,~{}\forall
x,y_{1},y_{2}\in\mathcal{X}\times\mathcal{Y}\times\mathcal{Y}$, and thereby
avoid some of the degeneracies identified in §2.3 for the maximum likelihood
estimate under DPO.

###  3.2 Pessimistic reward model distillation

Choosing a single reward model $r_{\mathrm{tgt}}$ for anchoring the LM policy
can naturally still lead to degenerate behavior if $r_{\mathrm{tgt}}$ is a
poor approximation of the true $r^{*}$ that accurately reflects human
preferences. However, we can easily extend our framework to handle uncertainty
in the right target reward function by defining a confidence _set_ of $k\geq
1$ plausible target reward models,
$\mathcal{S}=\left\\{r_{\mathrm{tgt}}^{1},\ldots,r_{\mathrm{tgt}}^{k}\right\\},$
and training $\pi_{\theta^{*}}(y\mid x)$ to maximize the following
“pessimistic” form of the objective in (1):

$\max_{\pi_{\theta}}\underset{r_{\mathrm{tgt}}^{i}\in\mathcal{S}}{\min}\mathbb{E}_{\mu(x)}\Big{[}\underbrace{\mathbb{E}_{\pi_{\theta}(y\mid
x)}[r_{\mathrm{tgt}}^{i}(x,y)]-\mathbb{E}_{\pi_{\mathrm{ref}}(y\mid
x)}[r_{\mathrm{tgt}}^{i}(x,y)]}_{\text{ advantage over baseline
policy}}-\beta\mathbb{D}_{\mathrm{KL}}(\pi_{\theta}(\cdot\mid
x)\|\pi_{\mathrm{ref}}(\cdot\mid x))\Big{]}.$ (8)

Figure 1: A toy illustration of Theorem 2, which states that the optimal
$\pi_{\theta^{*}}$ for (8) is the policy in $\mathcal{P}_{\beta}(\mathcal{S})$
with the lowest forward-KL from $\pi_{\textrm{SFT}}$. The set
$\mathcal{P}_{\beta}(\mathcal{S})$ contains a (potentially infinite) set of
policies $\pi_{1},\pi_{2},\dots$ corresponding to target reward models. Here,
$\pi_{\textrm{SFT}}$ assigns equal mass to $y^{w}$ and $y^{\ell}$,
$\pi_{\textrm{MLE}}$ is the MLE solution for the DPO objective, which puts all
probability mass on $y^{w}$, and $\pi_{3}$ is the policy in
$\mathcal{P}_{\beta}(\mathcal{S})$ with lowest forward-KL.

In this pessimistic objective we are no longer optimizing $\pi_{\theta}$ for a
single reward, but optimizing $\pi_{\theta}$ to produce generations that are
scored favorably on average, even by the worst-case reward model in the set
$\mathcal{S}$, relative to the generations of the baseline policy
$\pi_{\mathrm{ref}}$.111It is useful to optimize the advantage as it cancels
the effects of constant differences between reward models in $\mathcal{S}$. We
are also free to use any baseline policy $\pi_{\mathrm{base}}$; we pick
$\pi_{\mathrm{ref}}$ for simplicity and ease of analysis in §5. When the set
$\mathcal{S}=\\{r^{*}\\}$ consists of only the ground-truth reward, the
objective (8) is equivalent to standard RLHF (1), up to a constant offset
independent of $\theta$. More generally, whenever $\mathcal{S}$ includes a
good proxy $\widetilde{r}$ for $r^{*}$, the pessimistic advantage evaluation
ensures that the the policy $\pi^{*}_{\theta}$ that maximizes eq. 8 still has
a large advantage over $\pi_{\mathrm{ref}}$ under all $r\in\mathcal{S}$,
including $\widetilde{r}$. This use of pessimism to handle uncertainty in the
knowledge of the true reward is related to similar techniques in the offline
RL literature [16; 5].

For the objective to be meaningful, the set $\mathcal{S}$ has to be chosen
carefully. When $\mathcal{S}$ is small, it might not include any good proxy
for $r^{*}$. Conversely, if $\mathcal{S}$ is too rich, it forces
$\pi_{\theta^{*}}$ to be nearly identical to $\pi_{\mathrm{ref}}$, since any
deviations from $\pi_{\mathrm{ref}}$ might be penalized by some reward model
in $\mathcal{S}$. Consequently, we want to design $\mathcal{S}$ to be the
smallest possible set which contains a reasonable approximation to $r^{*}$.

To optimize (8), it turns out that we can formulate it as an equivalent
constrained offline optimization problem, that we will show to conveniently
admit a similar loss form as (7).

######  Theorem 2 (Pessimistic distillation).

Define the constrained minimizer

$\pi_{\theta^{*}}(y\mid
x)\in\underset{\pi_{\theta}\in\mathcal{P}_{\beta}(\mathcal{S})}{\arg\\!\min}~{}\beta\mathbb{E}_{\mu(x)}\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}(\cdot\mid
x)\|\pi_{\theta}(\cdot\mid x)),$ (9)

where $\mathcal{P}_{\beta}(\mathcal{S})$ is the set of all possible policies
with implicit reward models that are consistent with any target reward model
$r_{\mathrm{tgt}}^{i}\in\mathcal{S}$, i.e.,
$\mathcal{P}_{\beta}(\mathcal{S})\triangleq\\{\pi_{\theta_{i}}\\}_{i=1}^{|\mathcal{S}|}$
where $\pi_{\theta_{i}}\propto\pi_{\mathrm{ref}}(y\mid
x)\exp{\frac{1}{\beta}r_{\mathrm{tgt}}^{i}(x,y)}$. Then for any $\beta>0$,
$\pi_{\theta^{*}}$ also maximizes the pessimistic alignment objective in (8).

To unpack this result, Theorem 2 stipulates that the $\pi_{\theta}$ that
maximizes the pessimistic objective in (8) is the policy in
$\mathcal{P}_{\beta}(\mathcal{S})$ that is closest in _forward_ KL-divergence
to $\pi_{\mathrm{ref}}$ (see Figure 1).222Note that the objective in (9)
minimizes the forward KL-divergence
$\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}(\cdot\mid
x)\|\pi_{\theta}(\cdot\mid x))$ even though the pessimistic objective in (8)
is regularized with reverse KL-divergence
$\mathbb{D}_{\mathrm{KL}}(\pi_{\theta}(\cdot\mid
x)\|\pi_{\mathrm{ref}}(\cdot\mid x))$. In addition, this policy also maximizes
the expected reward of one of the $r_{\mathrm{tgt}}^{i}\in\mathcal{S}$ (minus
the additional weighted reverse KL-divergence penalty term). Intuitively, the
forward KL-divergence term serves the role of biasing the model towards
optimizing for reward models that are similar to the implicit reward that
$\pi_{\mathrm{ref}}$ already maximizes. Otherwise, there might exist a target
reward model $r_{\mathrm{tgt}}^{i}\in\mathcal{S}$ for which the advantage of
$\pi_{\theta}$ relative to $\pi_{\mathrm{ref}}$ will be low, or even negative
(a solution that we would like to avoid).

####  3.2.1 Optimization

The constraint in (9) can then be relaxed and approximately optimized by
introducing an objective with a Lagrangian-style penalty with strength
$\alpha>0$ on a form of distillation loss as (7), i.e.,

$\displaystyle\min_{\pi_{\theta}}\beta\mathbb{E}_{\mu(x)}\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}(y\mid
x)\|\pi_{\theta}(y\mid
x))+\alpha\min_{r^{i}_{\mathrm{tgt}}\in\mathcal{S}}\mathcal{L}_{\mathrm{distill}}(r^{i}_{\mathrm{tgt}},\pi_{\theta};\rho),$
(10)

where for convenience we divide by $\alpha$ and instead optimize333In
practice, we also compute and optimize the $\min$ over reward models per each
mini-batch of examples.

$\displaystyle\mathcal{L}_{\mathrm{pdistill}}(\mathcal{S},\pi_{\theta};\rho)=\min_{r^{i}_{\mathrm{tgt}}\in\mathcal{S}}\mathcal{L}_{\mathrm{distill}}(r^{i}_{\mathrm{tgt}},\pi_{\theta};\rho)+\gamma\mathbb{E}_{\mu(x)}\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}(\cdot\mid
x)\|\pi_{\theta}(\cdot\mid x)),$ (11)

where $\gamma=\beta/\alpha$. In reality, minimizing (11) for $\gamma>0$ is
equivalent to solving the constrained optimization problem in (9) with an
implicitly larger set of possible reward models
$\mathcal{S}_{\gamma}\supseteq\mathcal{S}$ indexed by $\gamma$. More
specifically, $\mathcal{S}_{\gamma}$ also contains all reward models
$\tilde{r}$ that are approximately consistent with the anchoring reward models
$r_{\mathrm{tgt}}^{i}$ contained in $\mathcal{S}$, as the following result
states.

######  Proposition 2 (Soft pessimistic distillation).

Assume the same conditions as Theorem 1. Then for any $0<\gamma<\infty$, there
exists a $\lambda\geq 0$ such that $\pi_{\theta^{*}}(y\mid
x)\in\arg\\!\min_{\pi_{\theta}}\mathcal{L}_{\mathrm{pdistill}}(\mathcal{S},\pi_{\theta};\rho)$,
where $\pi_{\theta^{*}}$ is a minimizer over all possible policies, is a
solution to (9) for the effective reward model set

$\mathcal{S}_{\gamma}=\bigcup_{r_{\mathrm{tgt}}^{i}\in\mathcal{S}}\Big{\\{}\tilde{r}\colon\mathbb{E}_{\rho(x,y_{1},y_{2})}\left[(r_{\mathrm{tgt}}^{i}(x,y_{1})-r_{\mathrm{tgt}}^{i}(x,y_{2})-\tilde{r}(x,y_{1})+\tilde{r}(x,y_{2}))^{2}\right]\leq\lambda\Big{\\}}.$
(12)

As a result, optimizing (11) even when using the singleton
$\mathcal{S}=\\{r_{\mathrm{tgt}}\\}$ yields an implicitly pessimistic
objective, in which the pessimism is over all reward models $\tilde{r}$ that
are consistent up to $\lambda$ with $r_{\mathrm{tgt}}$.

###  3.3 Pessimistic DPO

We can also observe that Proposition 2 can be leveraged to obtain an
alternative, implicitly pessimistic, objective that uses DPO directly instead
of distillation. Consider the following regularized DPO loss:

$\mathcal{L}_{\mathrm{pdpo}}(\pi_{\theta};\mathcal{D}_{\mathrm{pref}})=\mathcal{L}_{\mathrm{dpo}}(\pi_{\theta};\mathcal{D}_{\mathrm{pref}})+\gamma\mathbb{E}_{\mu(x)}\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}(y\mid
x)\|\pi_{\theta}(y\mid x)).$ (13)

Following a similar analysis as in Proposition 2, we can derive that this
implicitly corresponds to maximizing the pessimistic objective in (8) for the
reward model set

$\mathcal{S}_{\gamma}=\Big{\\{}r_{\pi_{\theta}}\colon\mathcal{L}_{\mathrm{dpo}}(\pi_{\theta};\mathcal{D}_{\mathrm{pref}})\leq\min_{\pi^{\prime}_{\theta}}\mathcal{L}_{\mathrm{dpo}}(\pi^{\prime}_{\theta};\mathcal{D}_{\mathrm{pref}})+\lambda\Big{\\}},$
(14)

where $r_{\pi_{\theta}}(x,y)\triangleq\beta\log\pi_{\theta}(y\mid
x)/\pi_{\mathrm{ref}}(y\mid x)+\beta\log Z(x)$ is the implicit reward model
defined by $\pi_{\theta}$. $\mathcal{S}_{\gamma}$ then corresponds to the set
of reward models $r_{\pi_{\theta}}$ that are all approximate minimizers of the
DPO loss. This not only includes the MLE, but also all other estimators that
obtain nearly the same loss. In principle, this can be expected to help
ameliorate some of the issues of §2.3: since driving the reward to $\pm\infty$
only marginally decreases the $\mathcal{L}_{\mathrm{dpo}}$ loss past a certain
point, the set $\mathcal{S}$ will also include finite reward functions
$|r_{\pi_{\theta}}(x,y)|<\infty$ for any $\gamma>0$. These rewards would then
be preferred if they induce a policy with a smaller (forward) KL-divergence to
$\pi_{\mathrm{ref}}$ than the degenerate, infinite rewards.

##  4 Experimental results

The main motivation for reward distillation and pessimism is to increase
alignment robustness in challenging settings where it is difficult to learn
good policies directly from the preference data. To demonstrate the
effectiveness of our approach, we run experiments on the popular TL;DR
summarization task [29; 32], in which we simulate a scenario where the
preference data has a spurious correlation between the _length_ of a summary
and whether or not it is preferred.444Length has been repeatedly shown in the
past to correlate with reward [28; 21].

###  4.1 Experimental setup

We first train an “oracle” reward model on the TL;DR preference data training
set [29] and relabel all preference pairs with this oracle. This enables us to
use the oracle reward model for evaluation, without worrying about the gap to
true human preferences. After relabeling, longer responses (where longer is
defined as $y_{1}$ having at least $10\%$ more tokens than $y_{2}$) are
preferred in $61\%$ of the examples.

To test the effect of a spurious correlation on preference-based policy
optimization, we select as a training set 30K examples from the relabeled data
such that the longer output is preferred in $\rho$ fraction of examples, with
$\rho\in\\{0.2,0.3,0.4,0.5,0.6,0.7,0.8\\}.$ Each such training set is denoted
$\mathcal{D}_{\rho}$. At each $\mathcal{D}_{\rho}$, we compare our approach to
DPO [23] and IPO [3], which are currently the most commonly used offline
alignment methods. We test the following variants of distillation and
pessimism:

  * •

Distilled DPO (d-DPO): Trains a reward model $r_{\rho}$ on
$\mathcal{D}_{\rho}$, and then optimizes
$\mathcal{L}_{\mathrm{distill}}(r_{\rho},\pi_{\theta};\rho)$.

  * •

Pessimistic DPO (p-DPO): A pessimistic version of DPO as described in §3.3,
trained on $\mathcal{D}_{\rho}$.

  * •

Pessimistic Distilled DPO (pd-DPO): Combines the above two by training a
reward model $r_{\rho}$ on $\mathcal{D}_{\rho}$ and optimizing the pessimistic
distillation objective (Eq. (11)) with confidence set
$\mathcal{S}=\\{r_{\mathrm{tgt}}\\}$.

  * •

Pessimistic Ensemble DPO (e-DPO): To create ensembles of reward models, we
subsample from each $\mathcal{D}_{\rho}$ five preference datasets,
$\mathcal{D}_{\rho,b}$, at $b\in\mathcal{B}=\\{0.2,0.4,0.5,0.6,0.8\\}$, such
that the fraction of pairs where the longer response is preferred is $b$, and
train reward models $r_{\rho,b}$ on those subsets. Consequently, sensitivity
to length should vary across ensemble members. We then apply the same
procedure as pd-DPO above, with a confidence set
$\mathcal{S}_{\rho}=\\{r_{\rho,b}\\}_{b=1}^{\mathcal{B}}$.

All reward models and policies are initialized from Palm-2-XS [2]. Policies
also go through a supervised finetuning step on human-written summaries from
the original TL;DR training set [32] prior to alignment, and we term this
policy $\pi_{\textrm{SFT}}$. We evaluate performance by sampling summaries for
test set prompts, evaluating the average reward according to the oracle reward
model, and computing the advantage in average reward compared to
$\pi_{\textrm{SFT}}$ (before alignment). We train policies for $10^{4}$ steps
with batch size $16$ and learning rate $10^{-6}$, and reward models for $3k$
steps with batch size $64$ and learning rate $4\times 10^{-6}$. We use the
validation set for model selection during policy training and to choose the
following hyperparameters. For all DPO variants, we sweep over
$\beta\in\\{.01,.1,1,3,10,30,100\\}$. For IPO, we sweep over
$\tau\in\\{0.01,0.1,1,3,5,10,25\\}$. For all pessimistic methods we anneal
$\gamma=\alpha/\beta$ from $10^{-4}$ to $10^{-2}$ linearly during the $10k$
training steps.

###  4.2 Results

Figure 2: Main results, showing the advantage in oracle reward compared to the
initial finetuned policy. Errorbars correspond to bootstrap 95% confidence
intervals for finite sample variance. Ensemble DPO (e-DPO) is significantly
better than DPO and IPO in the challenging setup where shorter responses are
preferred ($\rho\leq 0.5$), and is generally the best-performing method
overall in this regime. Distilled DPO (d-DPO) performs best when longer
responses are preferred ($\rho>0.6$).

We present the results of our experiment in Figure 2. As can be seen in the
plot, the more challenging setting is when $\rho<0.5$, which corresponds to a
sample of preference annotations in which shorter outputs are generally
preferred. This distribution shift is more difficult because as mentioned the
oracle reward model (trained on human annotations) has a bias in favor of
longer outputs [28]. Nevertheless we get sizable improvements compared to the
reference policy $\pi_{\textrm{SFT}}$ for all length bias values.

All approaches that invoke distillation (d-DPO, e-DPO, dp-DPO) outperform IPO
and DPO ($p<.01$ by a Wald test) for $\rho\leq 0.5$, where shorter responses
are preferred. Pessimistic ensemble DPO (e-DPO) performs particularly well in
these settings, generally outperforming all methods that use a single reward
model. When longer responses are preferred ($\rho>0.6)$, single reward
distillation (d-DPO) leads to the highest performance, significantly
outperforming both DPO and IPO ($p<.01$ by a Wald test). Interestingly, p-DPO
does not provide empirical benefits relative to the distillation based
methods, indicating that the distillation loss itself is quite important. For
the effect of hyper-parameter selection, see Figure D.1. In DPO-based methods,
the optimal value of $\beta$ is inversely correlated with the bias; in IPO the
same holds for the $\tau$ hyperparameter.

To better understand the utility of reward ensembles in e-DPO, in particular
when $\rho<0.5$, we examine the role of each reward model in the ensemble
across different biases. Specifically, given the final e-DPO policy per length
bias, for each example we identify the reward model $r_{\rho,b}$ that best
matches the implicit reward of this policy, i.e., for which reward model is
$\mathcal{L}_{\mathrm{distill}}$ minimized on that example (see Eq. (7) and
(11)). We find that when the policy is trained on data where shorter
preference are preferred ($\rho<.5$), the reward model that best matches the
policy often has the opposite bias ($b$ is high), and vice versa. Thus, the
success of e-DPO may be explained by its ability to distill from reward models
that do not suffer from the bias in the policy training data, which is
particularly helpful when $\rho\leq.5$ as this bias is also not shared by the
oracle RM. We provide the full distribution over reward models for all $\rho$
and $\beta$ in App. C. Overall, these results demonstrate the efficacy of
training a policy by distilling from a reward model in the presence of
distribution shifts, and that a careful design of an ensemble to mitigate
spurious correlations can lead to further performance gains.555We also
experimented with an ensemble where members are different checkpoints across
training of a reward model on the preference data and did not observe any
empirical gains from this form of ensemble.

##  5 Theoretical analysis

This section characterizes problems with the DPO objective and solutions
offered by pessimistic DPO and distillation, focusing on the simplified
scenario in which we optimize with respect to a single preference pairs
$(y^{w},y^{\ell})$. Once again, all proofs are deferred to Appendix A.

In its Lagrangian formulation, pessimistic DPO adds a forward KL term to the
DPO objective (§3.3). For the sake of analysis, we assume that the preference
annotations are sampled from the reference distribution,
$\mu(x)\times\pi_{\mathrm{ref}}(y\mid x)\times\pi_{\mathrm{ref}}(y\mid x)$.
Then a finite-sample approximation of the forward KL term is
$\hat{\Omega}(\Theta):=\sum_{(y^{w},y^{\ell})\in\mathcal{D}_{\mathrm{Pref}}}-(\log\pi_{\theta}(y^{\ell})+\log\pi_{\theta}(y^{w})).$
By applying this finite-sample approximation, _p-DPO has a finite optimum,
unlike DPO_ , as shown in Proposition 1. Note that this analysis is limited in
two ways: (1) as mentioned, we compute the KL term over the completions in the
preference data; (2) we directly optimize the probability ratios
$\psi_{w}=\pi_{\theta}(y^{w})/\pi_{\mathrm{ref}}(y^{w})$ and
$\psi_{\ell}=\pi_{\theta}(y^{\ell})/\pi_{\mathrm{ref}}(y^{\ell})$, rather than
optimizing them jointly through the parameters. For sufficiently expressive
$\pi_{\theta}$, however, this approximation captures the behavior of the two
algorithms reasonably well.

######  Proposition 3.

Let $\hat{\mathcal{L}}_{\mathrm{pdpo}}$ represent a finite-sample
approximation to $\mathcal{L}_{\mathrm{pdpo}}$ with the empirical forward KL
term $\hat{\Omega}(\Theta).$ For a fixed $\hat{\pi}_{\theta}(y_{i}^{w})$ and
$\alpha>1$, the
$\arg\\!\min_{\pi_{\theta}(y^{\ell})}\hat{\mathcal{L}}_{\mathrm{pdpo}}$ is
$\min\left(1-\hat{\pi}_{\theta}(y_{i}^{w}),\hat{\pi}_{\theta}(y_{i}^{\ell})\right)$,
with
$\log\hat{\pi}_{\theta}(y_{i}^{\ell})=-\frac{1}{\beta}\log\left(\alpha-1\right)+\log\hat{\pi}_{\theta}(y_{i}^{w})+\log\frac{\pi_{\mathrm{ref}}(y_{i}^{\ell})}{\pi_{\mathrm{ref}}(y_{i}^{w})}.$

The optimum in Proposition 3 corresponds to
$\log\psi_{w}/\psi_{\ell}={\beta}^{-1}\log(\alpha-1)$. Recall that IPO seeks
to assign a constant value to this ratio by minimizing
$(\log\frac{\psi_{w}}{\psi_{\ell}}-\tau^{-1})^{2}$; the (unconstrained) optima
are identical for $\tau^{-1}:={\beta}^{-1}\log(\alpha-1)$, but the loss
surfaces are different (see Appendix B). DPO sets
$\pi_{\theta}(y^{\ell}_{i})\to 0$, as shown in Corollary 1; this is due not
only to competition from $\pi_{\theta}(y^{w}_{i})$ but from DPO penalizing
positive probability on $y^{\ell}_{i}$. Analysis of the distilled loss gives a
similar result:

######  Proposition 4.

For any fixed $\hat{\pi}_{\theta}(y_{i}^{w})$ and $\beta>0,$ the $\arg\\!\min$
of the distilled DPO objective (eq. 7) is
$\min(1-\hat{\pi}_{\theta}(y_{i}^{w}),\hat{\pi}_{\theta}(y_{i}^{\ell})$, with
$\log\hat{\pi}_{\theta}(y_{i}^{\ell})=\frac{1}{\beta}(r_{t}(x,y_{i}^{\ell})-r_{t}(x,y_{i}^{w}))+\log\hat{\pi}_{\theta}(y_{i}^{w})+\log\frac{\pi_{\mathrm{ref}}(y_{i}^{\ell})}{\pi_{\mathrm{ref}}(y_{i}^{w})}$.

While the setting is simplistic, the results are comforting: here the
additional regularization effects of both distillation and pessimism (in the
case of p-DPO) clearly help to avoid degenerate optima.

##### Why DPO can drive $\pi(y^{w})$ to zero.

In §2.3 we pointed out a peculiarity of the DPO global optima: in certain
cases, it can include policies where $\pi(y^{w})$ may be nearly $0$ for all
$y^{w}$ in the training set. This undesirable behavior has also been observed
in practice [20; 22; 30]. For intuition on why this may happen, consider the
simplified case where the policy is a bag-of-words model,
$\pi_{\theta}(y)\propto\exp\left(c(y)\cdot\theta\right)$ for $c(y)$
representing a vector of counts in $y$ and $\theta_{i}$ representing the
unnormalized log-probability of token $i$. Then we can formally show that DPO
optimization monotonically decreases an upper bound on the probability of the
_preferred_ completion,
$\tilde{\pi}_{\theta^{(t-1)}}(y^{w})\geq\tilde{\pi}_{\theta^{(t)}}(y^{w})\geq\pi_{\theta^{(t)}}(y^{w}).$

######  Proposition 5.

Let $y^{w},y^{\ell}\in\mathcal{V}^{n}$ be preferred vs. dispreferred outputs
of length $n$, respectively, with
$\pi_{\mathrm{ref}}(y^{w}),\pi_{\mathrm{ref}}(y^{\ell})>0$ and corresponding
count vectors $c(y^{w}),c(y^{\ell}).$ Let
$\log\pi_{\theta}(y)=c(y)\cdot\theta-nZ(\theta)$ for
$Z(\theta)=\log\sum^{\mathcal{V}}_{i}e^{\theta_{i}}$, with upper bound
$\log\tilde{\pi}_{\theta}(y)=c(y)\cdot\theta-n\max_{j}\theta_{j}$. Let
$\theta^{(t)}$ represent the parameters of $\pi$ after $t$ steps of gradient
descent on $\mathcal{L}_{\mathrm{dpo}}(\\{y^{\ell},y^{w},x\\})$, with
$\theta^{(0)}=0$. Then
$\pi_{\theta^{(t)}}(y^{w})\leq\tilde{\pi}_{\theta^{(t)}}(y^{w})\leq\tilde{\pi}_{\theta^{(t-1)}}(y^{w})$
for all $t$.

##### Where does the probability mass go?

If $\pi_{\theta^{(t)}}(y^{w})$ decreases in $t$, what other strings become
more probable? In the following proposition, we show that under the bag-of-
words model, DPO optimization moves probability mass away from $y^{w}$ to
sequences that contain only the tokens that maximize the difference between
$y^{w}$ and $y^{\ell}$. This is a concrete example of the type of undesirable
optima described in §2.3, now shown here to be realizable.

######  Proposition 6.

Let $y^{w}$ and $y^{\ell}$ be preferred / dispreferred outputs of length $n$.
Let $\Delta=c(y^{w})-c(y^{\ell})$ be the difference in unigram counts. Let
$\hat{y}=[i,i,\ldots,i]$, for $i\in\text{arg}\max\Delta,$ with
$\|c(\hat{y})\|_{1}=n$. Then
$\pi_{\theta^{(t)}}(y^{w})-\pi_{\theta^{(t)}}(\hat{y})=\tau(t)k$ for some
$k\leq 0$ and some non-decreasing $\tau:\mathbb{Z}_{+}\to\mathbb{R}_{+}$.

We have $k=0$ when $c(y^{w})=c(\hat{y})$, and $k\ll 0$ when
$\|c(y^{w})\|_{2}\ll\|c(\hat{y})\|_{2}=n$ (dense $c(y^{w})$) and
$\|\Delta\|_{2}=\|\Delta\|_{\infty}$ (sparse $\Delta$). This implies that when
$y^{w}$ and $y^{\ell}$ are similar, $\pi_{\theta}(y^{w})$ will degrade more
rapidly. Early stopping will therefore tradeoff between reaching the
degenerate solution on such cases, and underfitting other cases in which
$y^{w}$ and $y^{\ell}$ are more distinct.

##  6 Related work

Recent work in offline alignment has focused on DPO [23] as a simpler
alternative for aligning language models from preference data. Subsequent work
has identified issues with DPO, including weak regularization [3] and a
tendency to decrease the probability of winning generations during training
[20]. Other methods have explored various avenues for improvement. These
include analyzing the impact of noise on DPO alignment [11], proposing to
update the reference policy during training [12], and suggesting a variant of
IPO with a per-context margin [1]. Additional research has focused on token-
level alignment methods [38; 22] and on developing a unified view of various
offline alignment methods [31]. This work builds upon several these findings,
and provides further analysis, as well as a solution based on pessimism and
reward distillation.

While offline alignment methods are popular, recent evidence suggests that
online alignment methods such as RLHF [6; 29], may lead to more favorable
outcomes [13; 30; 8; 34]. Notably, Zhu et al. [41] proposed iterative data
smoothing, which uses a trained model to softly label data during RLHF.
Whether online or offline, however, policies are still succeptible to
overfitting to certain degenerate phenomena. To this end, reward ensembles
have been widely investigated recently as a mechanism for tackling reward
hacking in RLHF [9; 7; 39; 25], and in the context of multi-objective
optimization [19; 24]. We use an ensemble of rewards to represent the
uncertainty with respect to reward models that are suitable given preference
data. Moskovitz et al. [19] focus on “composite” rewards, with the goal of
achieving high task reward while ensuring that every individual component is
above some threshold—also by applying a Lagrangian relaxation. In this work,
we also consider multiple reward models, but we only focus on cases where
there is no known, obvious reward decomposition.

Finally, the question of using a small amount of offline data to learn high-
quality policies, instead of online access to reward feedback, has been widely
studied in the offline reinforcement learning (RL) literature. The predominant
approach here is to use pessimism, that is, to learn a policy with the highest
reward under all plausible environment models consistent with the data, with
an extensive theoretical [18; 37; 33] and empirical [16; 5; 36] body of
supporting work. The key insight in this literature is that without pessimism,
the RL algorithm learns undesirable behaviors which are not explicitly ruled
out in the training data, and pessimism provides a robust way of preventing
such undesirable extrapolations, while still preserving generalization within
the support of the data.

##  7 Conclusion

LM alignment is crucial for deploying safe and helpful assistants, but is
difficult due to lack of access to perfect preference oracles. We presented a
thorough theoretical analysis of some of the degeneracies that DPO is
susceptible to when learning from sampled human preference data. Furthermore,
our findings suggest that explicit reward modeling remains a powerful vehicle
for introducing regularization into post-training. By distilling the reward
assigned by a single, explicit reward model—or a family of explicit reward
models—directly into the implicit reward maximized by our policies using
offline data, we demonstrated that we can achieve improved robustness to
variations in preference dataset quality, while maintaining the simplicity of
the DPO framework.

Limitations.  The empirical results in the paper are based on one dataset and
form of distribution shift. For deeper understanding of pessimism and
ensembling, additional settings should be explored. The theoretical aspects of
the paper are sometimes based on restrictive assumptions and simplifications.
Nonetheless, they provide potential explanations for phenomena observed in
real-world settings.

Broader impact.  We introduce new ideas to the active field of research on
preference-based post-training, which we hope will help facilitate the
alignment of large models, and improve understanding of current
approaches—ultimately supporting the development of capable and reliable AI
systems.

Acknowledgements.  We thank Alexander D’Amour and Chris Dyer for helpful
comments and feedback on the manuscript. This research also benefited from
discussions with Victor Veitch, Mandar Joshi, Kenton Lee, Kristina Toutanova,
David Gaddy, Dheeru Dua, Yuan Zhang, Tianze Shi, and Anastasios Angelopoulos.

## References

  * Amini et al. [2024] Afra Amini, Tim Vieira, and Ryan Cotterell.  Direct preference optimization with an offset.  _arXiv preprint arXiv:2402.10571_ , 2024.
  * Anil et al. [2023] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu.  Palm 2 technical report, 2023.
  * Azar et al. [2024] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello.  A general theoretical paradigm to understand learning from human preferences.  In _International Conference on Artificial Intelligence and Statistics_ , pages 4447–4455. PMLR, 2024.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.  ISSN 00063444.  URL http://www.jstor.org/stable/2334029.
  * Cheng et al. [2022] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal.  Adversarially trained actor critic for offline reinforcement learning.  In _International Conference on Machine Learning_ , pages 3852–3878. PMLR, 2022.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Coste et al. [2023] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger.  Reward model ensembles help mitigate overoptimization.  _arXiv preprint arXiv:2310.02743_ , 2023.
  * Dong et al. [2024] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang.  Rlhf workflow: From reward modeling to online rlhf.  _arXiv preprint arXiv:2405.07863_ , 2024.
  * Eisenstein et al. [2023] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al.  Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking.  _arXiv preprint arXiv:2312.09244_ , 2023.
  * Furlanello et al. [2018] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.  Born again neural networks.  In Jennifer Dy and Andreas Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 1607–1616. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/furlanello18a.html.
  * Gao et al. [2024] Yang Gao, Dana Alon, and Donald Metzler.  Impact of preference noise on the alignment performance of generative language models.  _arXiv preprint arXiv:2404.09824_ , 2024.
  * Gorbatovski et al. [2024] Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov.  Learn your reference model for real good alignment.  _arXiv preprint arXiv:2404.09656_ , 2024.
  * Guo et al. [2024] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al.  Direct language model alignment from online ai feedback.  _arXiv preprint arXiv:2402.04792_ , 2024.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  In _NIPS Deep Learning and Representation Learning Workshop_ , 2015.  URL http://arxiv.org/abs/1503.02531.
  * Korbak et al. [2022] Tomasz Korbak, Ethan Perez, and Christopher Buckley.  RL with KL penalties is better viewed as bayesian inference.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1083–1091, 2022.
  * Kumar et al. [2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Lee et al. [2023] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_ , 2023.
  * Liu et al. [2020] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill.  Provably good batch off-policy reinforcement learning without great exploration.  _Advances in neural information processing systems_ , 33:1264–1274, 2020.
  * Moskovitz et al. [2023] Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer.  Confronting reward model overoptimization with constrained rlhf.  _arXiv preprint arXiv:2310.04373_ , 2023.
  * Pal et al. [2024] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White.  Smaug: Fixing failure modes of preference optimisation with dpo-positive.  _arXiv preprint arXiv:2402.13228_ , 2024.
  * Park et al. [2024] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn.  Disentangling length from quality in direct preference optimization.  _arXiv preprint arXiv:2403.19159_ , 2024.
  * Rafailov et al. [2024a] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.  From $r$ to $q^{*}$: Your language model is secretly a q-function.  _arXiv preprint arXiv:2404.12358_ , 2024a.
  * Rafailov et al. [2024b] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024b.
  * Rame et al. [2024] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord.  Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Ramé et al. [2024] Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret.  WARM: On the benefits of weight averaged reward models.  _arXiv preprint arXiv:2401.12187_ , 2024.
  * Romero et al. [2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In _In Proceedings of ICLR_ , 2015.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Singhal et al. [2023] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett.  A long way to go: Investigating length correlations in rlhf.  _arXiv preprint arXiv:2310.03716_ , 2023.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Tajwar et al. [2024] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar.  Preference fine-tuning of llms should leverage suboptimal, on-policy data.  _arXiv preprint arXiv:2404.14367_ , 2024.
  * Tang et al. [2024] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot.  Generalized preference optimization: A unified approach to offline alignment.  _arXiv preprint arXiv:2402.05749_ , 2024.
  * Völske et al. [2017] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein.  TL;DR: Mining Reddit to learn automatic summarization.  In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, _Proceedings of the Workshop on New Frontiers in Summarization_ , pages 59–63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/W17-4508.  URL https://aclanthology.org/W17-4508.
  * Xie et al. [2021] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal.  Bellman-consistent pessimism for offline reinforcement learning.  _Advances in neural information processing systems_ , 34:6683–6694, 2021.
  * Xu et al. [2024] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu.  Is DPO superior to PPO for LLM alignment? a comprehensive study.  _arXiv preprint arXiv:2404.10719_ , 2024.
  * Yang et al. [2019] Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L. Yuille.  Training deep neural networks in generations: a more tolerant teacher educates better students.  In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence_ , AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019.  ISBN 978-1-57735-809-1.  doi: 10.1609/aaai.v33i01.33015628.  URL https://doi.org/10.1609/aaai.v33i01.33015628.
  * Yu et al. [2021] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn.  Combo: Conservative offline model-based policy optimization.  _Advances in neural information processing systems_ , 34:28954–28967, 2021.
  * Zanette et al. [2021] Andrea Zanette, Martin J Wainwright, and Emma Brunskill.  Provable benefits of actor-critic methods for offline reinforcement learning.  _Advances in neural information processing systems_ , 34:13626–13640, 2021.
  * Zeng et al. [2024] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang.  Token-level direct preference optimization.  _arXiv preprint arXiv:2404.11999_ , 2024.
  * Zhai et al. [2023] Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang.  Uncertainty-penalized reinforcement learning from human feedback with diverse reward lora ensembles.  _arXiv preprint arXiv:2401.00243_ , 2023.
  * Zhao et al. [2023] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023.
  * Zhu et al. [2024] Banghua Zhu, Michael I Jordan, and Jiantao Jiao.  Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf.  _arXiv preprint arXiv:2401.16335_ , 2024.

##  Appendix A Proofs

###  A.1 Proof of Proposition 1

###### Proof.

Since all the preference pairs $(y,y^{\prime})$ are mutually disjoint, and
$\theta_{y}$ is specific to each $y$, the DPO objective over
$\mathcal{D}_{\mathrm{pref}}$ is convex in
$\Delta=\\{\Delta_{1},\ldots,\Delta_{n}\\}$, where

$\Delta_{i}=\beta\log\frac{\pi_{\theta}(y_{i}^{w})\pi_{\mathrm{ref}}(y_{i}^{\ell})}{\pi_{\theta}(y_{i}^{\ell})\pi_{\mathrm{ref}}(y_{i}^{w})}.$
(15)

Furthermore, the different $\Delta_{i}$ are completely independent from each
other due to the preference pairs being disjoint, so they can be optimized
over separately.

In particular, for every $i$ we have that

$\lim_{\Delta_{i}\rightarrow\infty}-\log\left(\sigma\left(\Delta_{i}\right)\right)=0,$
(16)

which implies that $\Delta^{*}=\\{\infty\\}^{n}$ is the unique global
minimizer of the DPO loss over $\mathcal{D}_{\mathrm{pref}}$ in the space of
$\Delta$’s, and any $\theta^{*}$ that is a global minimizer must therefore
satisfy

$\log\frac{\pi_{\theta}(y_{i}^{w})\pi_{\mathrm{ref}}(y_{i}^{\ell})}{\pi_{\theta}(y_{i}^{\ell})\pi_{\mathrm{ref}}(y_{i}^{w})}=\infty.$
(17)

∎

###  A.2 Proof of Corollary 1

###### Proof.

Following the same argument of the proof of Proposition 1, we have that all
global minimizers $\theta^{*}$ of the DPO satisfy $\Delta_{i}^{*}=\infty$,
which in turn implies that

$\frac{\pi_{\theta^{*}}(y_{i}^{w})\pi_{\mathrm{ref}}(y_{i}^{\ell})}{\pi_{{\theta^{*}}}(y_{i}^{\ell})\pi_{\mathrm{ref}}(y_{i}^{w})}=\infty.$
(18)

Since $\pi_{\mathrm{ref}}(y)$ is assumed to satisfy
$0<\pi_{\mathrm{ref}}(y)<1$ for all $y$, this implies that all $\theta^{*}$
satisfy

$\frac{\pi_{\theta^{*}}(y_{i}^{w})}{\pi_{\theta^{*}}(y_{i}^{\ell})}=\infty,$
(19)

which further implies that $\pi_{\theta^{*}}(y_{i}^{\ell})=0$ and
$\pi_{\theta^{*}}(y_{i}^{w})>0$ for all $i\in[n]$, as
$\pi_{\theta^{*}}(y_{i}^{w})\leq 1$ for any $y_{i}^{w}$. Aggregating

$\mathcal{C}(y_{\ell})=\\{y\colon\exists i\in[n]\text{ s.t }y_{i}^{\ell}=y\\}$
(20)

then gives that

$\displaystyle\pi_{\theta^{*}}(\mathcal{C}(y_{\ell}))$
$\displaystyle=\sum_{y\in\mathcal{C}(y_{\ell})}\pi_{\theta^{*}}(y)=0\Longrightarrow\pi_{\theta^{*}}(\mathcal{C}(y_{\ell})^{c})=1.$
(21)

∎

To prove the converse, let $\pi_{\theta^{\prime}}$ be a policy that satisfies
$\pi_{\theta^{\prime}}(\mathcal{C}(y^{\ell})^{c})=1$, with
$\pi_{\theta^{\prime}}(y_{i}^{w})>0$, $\forall i\in[n]$,. As
$\pi_{\theta^{\prime}}(y)\geq 0$ for all $y$, this implies that
$\pi_{\theta^{\prime}(y_{i}^{\ell})}=0$ $\forall i\in[n]$. Then, we have

$\frac{\pi_{\theta^{\prime}}(y_{i}^{w})}{\pi_{\theta^{\prime}}(y_{i}^{\ell})}=\infty,$
(22)

which by Proposition 1 implies that $\pi_{\theta^{\prime}}$ is a global
optimum.

###  A.3 Proof of Theorem 1

###### Proof.

We know that the optimal policy for the RLHF objective (1) is given by
$\pi_{\theta^{*}}(y|x)\propto\pi_{\mathrm{ref}}(y|x)\exp(r^{*}(x,y)/\beta)$.
Plugging this policy into the distillation objective (7), we see that
$\mathcal{L}_{\text{distill}}(r^{*},\pi_{\theta^{*}},\rho)=0$ for all $\rho$.
In fact, the loss is equal to 0 pointwise, meaning that $\pi_{\theta^{*}}$ is
a global minimizer of the distillation objective (7). Further, let $\pi$ be
some other minimizer of $\mathcal{L}_{\text{distill}}(r^{*},\cdot,\rho)$. Then
$\pi$ also has to attain a loss of $0$ at all $(x,y,y^{\prime})$ in the
support of $\rho$, meaning that
$\log\pi(y|x)-\log\pi(y^{\prime}|x)=\log\pi_{\theta^{*}}(y|x)-\log\pi_{\theta^{*}}(y|x)$
for all $(x,y,y^{\prime})$ in the support of $\rho$. Consequently, the two
policies coincide in the support of $\rho$ (due to the normalization
constraint, there is no additional offset term allowed as the support of
$\rho$ covers all of $\mathcal{Y}$). Finally, noting that the support of the
chosen $\rho$ is such that $\pi_{\theta^{*}}$ puts no mass outside its support
due to the KL constraint in (1), we complete the proof. ∎

###  A.4 Proof of Theorem 2

###### Proof.

Consider the pessimistic objective:

$\max_{\pi_{\theta}}~{}\underset{r_{\mathrm{tgt}}\in\mathcal{S}}{\min}~{}\mathbb{E}_{\mu(x)}\Big{[}{\mathbb{E}_{\pi_{\theta}(y\mid
x)}[r_{\mathrm{tgt}}(x,y)]-\mathbb{E}_{\pi_{\mathrm{ref}}(y\mid
x)}[r_{\mathrm{tgt}}(x,y)]}\Big{]}-\beta\mathbb{D}_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}}).$
(23)

As it is linear in $r_{\mathrm{tgt}}$ and convex in $\pi$, we can switch the
order of $\min$ and $\max$:

$\underset{r_{\mathrm{tgt}}\in\mathcal{S}}{\min}~{}\left[\underset{\pi\in\Pi}{\max}~{}\mathbb{E}_{\mu(x)}\Big{[}\mathbb{E}_{\pi(y\mid
x)}[r_{\mathrm{tgt}}(x,y)]-\mathbb{E}_{\pi_{\mathrm{ref}}(y\mid
x)}[r_{\mathrm{tgt}}(x,y)]\Big{]}-\beta\mathbb{D}_{\mathrm{KL}}(\pi\|\pi_{\mathrm{ref}})\right].$
(24)

Note that every $r_{\mathrm{tgt}}\in\mathcal{S}$ can be written in terms of
the KL-constrained policy $\pi^{*}_{r_{\mathrm{tgt}}}$ it induces, i.e.,

$r_{\mathrm{tgt}}(x,y)=\beta\log\frac{\pi^{*}_{r_{\mathrm{tgt}}}(y\mid
x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x,r_{\mathrm{tgt}}),$ (25)

where

$\displaystyle\pi^{*}_{r_{\mathrm{tgt}}}$
$\displaystyle=\arg\\!\max_{\pi_{\theta}}\mathbb{E}_{\mu(x)}\mathbb{E}_{\pi_{\theta}(y\mid
x)}[r_{\mathrm{tgt}}(x,y)]-\beta\mathbb{D}_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}})$
(26)

which has the form

$\pi^{*}_{r_{\mathrm{tgt}}}(y\mid
x)=\frac{1}{Z(x,r_{\mathrm{tgt}})}\pi_{\mathrm{ref}}(y\mid
x)\exp\left(\frac{1}{\beta}r_{\mathrm{tgt}}(x,y)\right)$ (27)

where $Z(x,r_{\mathrm{tgt}})$ is the partition function:

$Z(x,r_{\mathrm{tgt}})=\sum_{y\in\mathcal{Y}}\pi_{\mathrm{ref}}(y\mid
x)\exp\left(\frac{1}{\beta}r_{\mathrm{tgt}}(x,y)\right).$ (28)

Substituting $\pi^{*}_{r_{\mathrm{tgt}}}$ in for $\max_{\pi}$ and writing
$r_{\mathrm{tgt}}$ in terms of $\pi^{*}_{r_{\mathrm{tgt}}}$, we get the
simplified objective

$\displaystyle\underset{r_{\mathrm{tgt}}\in\mathcal{S}}{\min}$
$\displaystyle~{}\left[\underset{\pi\in\Pi}{\max}~{}\mathbb{E}_{\mu(x)}\Big{[}\mathbb{E}_{\pi(y\mid
x)}[r_{\mathrm{tgt}}(x,y)]-\mathbb{E}_{\pi_{\mathrm{ref}}(y\mid
x)}[r_{\mathrm{tgt}}(x,y)]\Big{]}-\beta\mathbb{D}_{\mathrm{KL}}(\pi\|\pi_{\mathrm{ref}})\right]$
$\displaystyle=\underset{r_{\mathrm{tgt}}\in\mathcal{S}}{\min}~{}\bigg{[}\mathbb{E}_{\mu(x)}\bigg{[}\mathbb{E}_{\pi^{*}_{r_{\mathrm{tgt}}}(y\mid
x)}\bigg{[}\beta\log\frac{\pi^{*}_{r_{\mathrm{tgt}}}(y\mid
x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x,r_{\mathrm{tgt}})\bigg{]}$
$\displaystyle\hskip 68.28644pt~{}~{}-\mathbb{E}_{\pi_{\mathrm{ref}}(y\mid
x)}\bigg{[}\beta\log\frac{\pi^{*}_{r_{\mathrm{tgt}}}(y\mid
x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x,r_{\mathrm{tgt}})\bigg{]}$ (29)
$\displaystyle\hskip
68.28644pt~{}~{}-\beta\mathbb{D}_{\mathrm{KL}}(\pi^{*}_{r_{\mathrm{tgt}}}\|\pi_{\mathrm{ref}}\mid
x)\bigg{]}\bigg{]}$
$\displaystyle=\underset{r_{\mathrm{tgt}}\in\mathcal{S}}{\min}~{}\beta\bigg{[}\mathbb{E}_{\mu(x)}\bigg{[}\mathbb{D}_{\mathrm{KL}}(\pi^{*}_{r_{\mathrm{tgt}}}\|\pi_{\mathrm{ref}}\mid
x)+\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}\|\pi^{*}_{r_{\mathrm{tgt}}}\mid
x)-\mathbb{D}_{\mathrm{KL}}(\pi^{*}_{r_{\mathrm{tgt}}}\|\pi_{\mathrm{ref}}\mid
x)\bigg{]}\bigg{]}$
$\displaystyle=\min_{r_{\mathrm{tgt}}\in\mathcal{S}}\beta\mathbb{E}_{\mu(x)}\left[\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}\|\pi^{*}_{r_{\mathrm{tgt}}}\mid
x)\right].$

∎

###  A.5 Proof of Proposition 2

###### Proof.

The proof is a standard Lagrangian duality argument, which we reproduce here
for completeness. For two functions $f(z)$ and $g(z)$, let us define

$\displaystyle z^{*}=\arg\\!\min_{z}f(z)+\alpha g(z).$ (30)

Let us also consider the constrained problem

$\displaystyle z^{\prime}=\arg\\!\min_{z}f(z)\quad\mbox{s.t. }g(z)\leq
g(z^{*}).$ (31)

Suppose by contradiction that $z^{*}$ is not a minimizer of (31). Since
$z^{*}$ is feasible for the constraint by construction, we get that
$f(z^{\prime})<f(z^{*})$. Consequently, we further have

$f(z^{\prime})+\alpha g(z^{\prime})<f(z^{*})+\alpha g(z^{*}),$

where the inequality follows from the feasibility of $z^{\prime}$ in (31).
This contradicts the optimality of $z^{*}$ in (30), meaning that $z^{*}$ must
be a minimizer of (31). Applying this general result with
$f=\beta\mathbb{E}_{\mu(x)}\mathbb{D}_{\mathrm{KL}}(\pi_{\mathrm{ref}}(y\mid
x)\|\pi_{\theta}(y\mid x))$,
$g=\min_{r^{i}_{\mathrm{tgt}}\in\mathcal{S}}\mathcal{L}_{\mathrm{distill}}(r^{i}_{\mathrm{tgt}},\pi_{\theta};\rho)$,
and $z=\pi_{\theta}$ completes the proof, since we recognize the set
$\mathcal{S}_{\gamma}$ in (12) to be equivalent to
$\bigcup_{r_{\mathrm{tgt}}^{i}\in\mathcal{S}}\mathcal{L}_{\mathrm{distill}}(r^{i}_{\mathrm{tgt}},\pi_{\theta};\rho)\leq\lambda$.
∎

###  A.6 Proof of Proposition 3

###### Proof.

We differentiate $\mathcal{L}_{\mathrm{pdpo}}$ with respect to
$\psi_{\ell}=\pi_{\theta}(y^{\ell})/\pi_{\mathrm{ref}}(y^{\ell})$ with $i$
implicit, obtaining,

$\displaystyle\frac{\partial\mathcal{L}_{\mathrm{pdpo}}}{\partial\psi_{\ell}}=$
$\displaystyle\beta\frac{\psi_{\ell}^{\beta}}{\psi_{w}^{\beta}+\psi_{\ell}^{\beta}}\psi_{\ell}^{-1}-\frac{\beta}{\alpha}\psi_{\ell}^{-1}=\beta\psi_{\ell}^{-1}\left(\frac{\psi_{\ell}^{\beta}}{\psi_{w}^{\beta}+\psi_{\ell}^{\beta}}-\alpha^{-1}\right)$
(32)

which is zero when,

$\displaystyle\alpha\psi_{\ell}^{\beta}=$
$\displaystyle\psi^{\beta}_{w}+\psi^{\beta}_{\ell}$ (33)
$\displaystyle\psi_{\ell}=$
$\displaystyle\left(\frac{1}{\alpha-1}\right)^{1/\beta}\psi_{w}$ (34)
$\displaystyle\log\psi_{\ell}=$
$\displaystyle-\frac{1}{\beta}\log(\alpha-1)+\log\psi_{w}$ (35)
$\displaystyle\log\pi_{\hat{\theta}}(y^{\ell})=$
$\displaystyle\log\pi_{\mathrm{ref}}(y^{\ell})-\frac{1}{\beta}\log\left(\alpha-1\right)+\log\pi_{\theta}(y^{w})-\log\pi_{\mathrm{ref}}(y^{w}).$
(36)

By the second-order condition, the critical point is a minimum. The objective
$\mathcal{L}_{\mathrm{pdpo}}$ is the sum of two components: the negative log
sigmoid term for $\mathcal{L}_{i}$ and the negative log probability for
$\hat{\Omega}$. Because each component is a convex function of $\psi_{i}$, so
is $\mathcal{L}_{\mathrm{pdpo}}$. As a result, the local minimum
$\log\hat{\pi}_{\theta}(y^{\ell})$ is also a global minimum. ∎

###  A.7 Proof of Proposition 4

###### Proof.

This follows directly from differentiating eq. 7 with respect to
$\pi_{\theta}(y_{2}).$ ∎

###  A.8 Proof of Proposition 5

###### Proof.

Let $\Delta=[c(y^{w})-c(y^{\ell})]$ and
$\rho=\pi_{\mathrm{ref}}(y^{w})/\pi_{\mathrm{ref}}(y^{\ell})$. The theorem
assumes $|y^{w}|=|y^{\ell}|$. Then
$\mathcal{L}_{\mathrm{dpo}}=-\log\sigma\left(\beta(\Delta\cdot\theta)+\beta\log\rho\right).$
The derivative with respect to $\theta$ is,

$\displaystyle\frac{\partial\mathcal{L}_{\beta}(\theta)}{\partial\theta}=$
$\displaystyle-(1-\sigma(\beta(\Delta\cdot\theta)+\beta\log\rho))\beta\Delta=-\Pr(y^{\ell}\succ
y^{w};\theta)\beta\Delta\prec 0.$ (37)

Let $\delta_{t}=\beta\Pr(y^{\ell}\succ y^{w};\theta^{(t)}).$ Then,

$\displaystyle\tilde{\pi}_{\theta^{(t)}}=$ $\displaystyle\theta^{(t)}\cdot
c(y^{w})-n\max_{j}\theta^{(t)}_{j}$ (38) $\displaystyle=$
$\displaystyle(\theta^{(t-1)}+\delta_{t}\Delta)\cdot
c(y^{w})-n\max_{j}(\theta^{(t-1)}_{j}+\delta_{t}\Delta_{j})$ (39)
$\displaystyle=$ $\displaystyle\theta^{(t-1)}\cdot
c(y^{w})-n\max_{j}\theta^{(t-1)}_{j}+\delta_{t}\Delta\cdot
c(y^{w})-n\delta_{t}\max_{j}\Delta_{j}$ (40) $\displaystyle=$
$\displaystyle\tilde{\pi}_{\theta^{(t-1)}}+\delta_{t}\left(\Delta\cdot
c(y^{w})-n\max_{j}\Delta_{j}\right)$ (41) $\displaystyle=$
$\displaystyle\tilde{\pi}_{\theta^{(t-1)}}+\delta_{t}\sum_{j}^{\mathcal{V}}c_{j}(y^{w})(\Delta_{j}-\max_{j^{\prime}}\Delta_{j^{\prime}})\leq\tilde{\pi}_{\theta^{(t-1)}}.$
(42)

We obtain
$\max_{j}\left(\theta_{j}^{(t-1)}+\delta_{t}\Delta_{j}\right)=\max_{j}\theta_{j}^{(t-1)}+\max_{j}\delta_{t}\Delta_{j}$
from the fact that $\theta^{(0)}=0$ and therefore $j\in\text{arg}\max\Delta$
implies $j\in\text{arg}\max\theta^{(t^{\prime})}$ for all $t^{\prime}>0$. The
second-to-last step uses ${n=\sum_{j}^{\mathcal{V}}c_{j}(y^{w})}$ and the
final step uses $\Delta_{j}\leq\max_{j}^{\prime}\Delta_{j^{\prime}}$. Finally,
we have $\pi_{\theta^{(t)}}(y)\leq\tilde{\pi}_{\theta^{(t)}}(y^{w})$ because
$Z(\theta)=\log\sum_{j}\exp\theta_{j}\geq\log\max_{j}\exp\theta_{j}=\max_{j}\theta_{j}.$
∎

###  A.9 Proof of Proposition 6

###### Proof.

Applying gradient descent with learning rate $\eta$ to the gradient from
Equation 37, at each step $t$ the parameters are,

$\displaystyle\theta^{(t)}=$
$\displaystyle\theta^{(t-1)}+\eta\beta\Pr(y^{\ell}\succ
y^{w};\theta^{(t-1)})\Delta=\left(\sum^{t}_{t^{\prime}=1}\eta\beta\Pr(y^{\ell}\succ
y^{w};\theta^{(t^{\prime})})\right)\Delta=\tau(t)\Delta.$ (43)

Plugging these parameters into the likelihoods,

$\displaystyle\ell_{\theta^{(t)}}(c(y^{w}))-\ell_{\theta^{(t)}}(\hat{y})$
$\displaystyle=c(y^{w})\cdot\theta^{(t)}-nZ(\theta^{(t)})-c(\hat{y})\cdot\theta^{(t)}+nZ(\theta^{(t)})$
(44)
$\displaystyle=\left(c(y^{w})-c(\hat{y})\right)\cdot\theta^{(t)}=\left(c(y^{w})-c(\hat{y})\right)\cdot(\tau(t)\Delta)$
(45) $\displaystyle=\tau(t)(c(y^{w})\cdot\Delta-n\max\Delta)=\tau(t)k,$ (46)

with $k\leq 0$ by
$c(y^{w})\cdot\Delta\leq||c(y^{w})||_{1}\times||\Delta||_{\infty}=n\max\Delta.$
∎

##  Appendix B Transitive closure

Both p-DPO and IPO target a constant ratio for $\log\psi_{w}/\psi_{l}$.
However, the loss surfaces are different. To see this, we consider a
simplified setting with three possible outputs, $y_{1}$, $y_{2}$, $y_{3}$. We
observe either $\mathcal{D}=\\{(y_{1}\prec y_{2}),(y_{2}\prec y_{3})\\}$ or
$\overline{\mathcal{D}}=\mathcal{D}\cup\\{(y_{1}\prec y_{3})\\}$. If we treat
this problem as a multi-arm bandit, the goal is to assign a weight to each
arm, which we denote $\psi_{i}=\log\pi_{\theta}(y_{i}|x)+Z_{x},$ with $Z_{x}$
an underdetermined log-partition function.

######  Proposition 7.

Let $\mathcal{D}=\\{(i,i+1):i\in 1,2,\ldots,n\\}$ for $n>2$. Let
$\overline{\mathcal{D}}$ be the dataset arising from the transitive closure of
$\mathcal{D}.$ Assume $\pi_{\mathrm{ref}}$ is indifferent to all
$(y_{i},y_{j})$. Let
$\psi^{(\mathcal{D})}_{\infty}=\max_{i}\psi^{(\mathcal{D})}_{i}-\min_{i}\psi^{(\mathcal{D})}_{i}$.
Then
$\psi^{(\mathcal{D})}_{\infty}=(n-1)\tau^{-1}>\psi^{(\mathcal{\overline{D}})}_{\infty}=2\frac{n-1}{n}\tau^{-1}.$

###### Proof.

For $\mathcal{D}$, the IPO objective can be minimized at zero, so that
$\psi^{(\mathcal{D})}_{\infty}=(n-1)\tau^{-1}$. For $\overline{\mathcal{D}}$,
each adjacent pair of completions is separated by $\gamma$, and the objective
is $\sum_{i=1}^{n-1}(n-i)(i\gamma-\tau^{-1})^{2}.$ The minimum is
$\gamma=\frac{n(n+1)(n-1)/6}{n^{2}(n+1)(n-1)/12}\tau^{-1}=\frac{2}{n}\tau^{-1}$,
so that
$\psi^{(\overline{\mathcal{D}})}_{\infty}=(n-1)\gamma=2\frac{n-1}{n}\tau^{-1}<(n-1)\tau^{-1}=\psi^{(\mathcal{D})}_{\infty}$
for $n>2$. ∎

Intuitively, the observation of ${(y_{1}\prec y_{3})}$ should increase
confidence that $y_{3}$ is superior to $y_{1},$ but in IPO it has the opposite
effect, drawing their scores closer together. While pessimistic DPO also has a
target ratio between each preference pair, its loss surface is different: in
particular, it does not increase quadratically as we move away from the
target. We find empirically that pessimistic DPO is robust to the transitive
closure of preference annotations in the multi-arm bandit setting, as shown in
Figure B.1. As discussed above, DPO will set $\psi_{1}\to-\infty$ because
$y_{1}$ is never preferred.

In our experiments we solve the p-DPO and IPO objectives for both
$\mathcal{D}=\\{(y_{1},y_{2}),(y_{2},y_{3})\\}$ and
$\mathcal{\overline{D}}=\mathcal{D}\cup\\{(y_{1},y_{3})\\}$, solving with
respect to $\\{\pi_{\theta}(y_{i})\\}.$ IPO is solved analytically as a
quadratic program; for p-DPO we used projected gradient descent. We consider
$\beta\in(1,3,10,30)$ and $\alpha\in(5,10,20,50,100,1000)$. As shown in Figure
B.1, there are significant differences in the IPO solutions with and without
transitive closure, while for p-DPO these differences are imperceptible.

Figure B.1: Effect of transitive closure on p-DPO and IPO solutions to
preference learning in a multi-arm bandit. Each column shows the learned
policy probability for a given arm, based on the preferences $y_{1}\prec
y_{2}\prec y_{3}$. The top row shows that in p-DPO, the probabilities are not
materially affected by the transitive closure $y_{1}\prec y_{3}$. The bottom
row shows that in IPO, transitive closure causes the probabilities to be
compressed. In each subfigure, we sweep a range of effective values of
$\tau^{-1}$, shown on the x-axis.

##  Appendix C Distribution over reward models for e-DPO

Figure C.1 investigates the reason for the success of e-DPO, especially when
$\rho<.5$. For every length bias, we show across all training examples the
fraction of cases where a certain reward model, $r_{\rho,b}$, best matched the
implicit reward of the final e-DPO policy. The policy matches different reward
models in different examples. Moreover, there is inverse correlation between
the data bias for policy training ($\rho$) and the data bias for training the
reward models ($b$). This suggests that the ensemble in e-DPO helps as the
policy is distilling from reward models that do not share the data bias of the
policy training set.

Figure C.1: We show for every length bias, $\rho$, the distribution over
reward models that best match the final policy trained by e-DPO across all
training examples. We observe that the e-DPO policy matches different reward
models across examples. Moreover, when the policy is trained with data biased
towards preferring short responses, the reward model that was trained on
longer responses is often preferred and vice versa.

##  Appendix D Hyperparameters

Validation set performance across the range of hyperparameter settings is
shown in Figure D.1. In pilot studies we found that these results were
relatively robust to variation in the random seed, but did not conduct
extensive investigation of this effect across all methods and hyperparameters
due to cost.

Figure D.1: Validation set results across hyperparameters for each method. For
all methods, different values of $\rho$ induce different optimal
hyperparameters $\beta$ and $\tau^{-1}$.

##  Appendix E Compute resources

We train policies on 32 TPU v3 chips and reward models on 16 TPU v3 chips. We
obtain roughly 0.1 steps per second when training, for both the policy and
reward models.