# statistical rejection sampling improves preference optimization

Tianqi Liu∗, Yao Zhao†, Rishabh Joshi†, Misha Khalman†, Mohammad Saleh†,
Peter J. Liu†, Jialu Liu∗
{tianqiliu,yaozhaoyz,rishabhjoshi,khalman,msaleh,
<EMAIL_ADDRESS>
Google Research∗, Google DeepMind†

###### Abstract

Improving the alignment of language models with human preferences remains an
active research challenge. Previous approaches have primarily utilized
Reinforcement Learning from Human Feedback (RLHF) via online RL methods such
as Proximal Policy Optimization (PPO). Recently, offline methods such as
Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization
(DPO) have emerged as attractive alternatives, offering improvements in
stability and scalability while maintaining competitive performance. SLiC
refines its loss function using sequence pairs sampled from a supervised fine-
tuned (SFT) policy, while DPO directly optimizes language models based on
preference data, foregoing the need for a separate reward model. However, the
maximum likelihood estimator (MLE) of the target optimal policy requires
labeled preference pairs sampled from that policy. DPO’s lack of a reward
model constrains its ability to sample preference pairs from the optimal
policy, and SLiC is restricted to sampling preference pairs only from the SFT
policy. To address these limitations, we introduce a novel approach called
_Statistical Rejection Sampling Optimization_ (RSO) that aims to source
preference data from the target optimal policy using rejection sampling,
enabling a more accurate estimation of the optimal policy. We also propose a
unified framework that enhances the loss functions used in both SLiC and DPO
from a preference modeling standpoint. Through extensive experiments across
three diverse tasks, we demonstrate that RSO consistently outperforms both
SLiC and DPO on evaluations from both Large Language Model (LLM) and human
raters.

##  1 Introduction

Recent advancements in Large Language Models (LLMs) have unlocked
unprecedented capabilities in diverse tasks, from programming to creative
writing. Models such as GPT-3 (Brown et al., 2020), Gopher (Rae et al., 2021),
Chinchilla (Hoffmann et al., 2022), and PaLM 2-L (Anil et al., 2023) that are
pre-trained on large unlabeled corpus have showcased remarkable prowess in
tackling both zero-shot (Radford et al., 2019) and few-shot (Brown et al.,
2020; Chowdhery et al., 2022) tasks. Their performance and alignment with user
intent can be further improved through supervised fine-tuning (SFT). A
specific strategy that has garnered attention in this context is Instruction
Tuning (Wei et al., 2021). By leveraging datasets enriched with instructions
and corresponding human-authored completions, this approach has demonstrated
enhanced generalization and model usability (Chung et al., 2022). Despite its
promise, achieving desired model outputs across various domains remains a
challenge.

Reinforcement Learning with Human Feedback (RLHF) has been introduced as a
promising approach to enhance the alignment of large language models with
human preferences (Stiennon et al., 2020). RLHF introduces notable
complexities into the training process. Integrating a reward model and
frequently sampling from the policy during training often demand substantial
memory, thereby limiting the maximum feasible size of a model given memory
constraints. The inclusion of roll-outs in the training loop can also slow
down the optimization process, as it necessitates continuous model sampling
and decoding. Further, understanding how to tune the PPO process calls for
specialized knowledge.

Recognizing these challenges, recent research has pioneered alternatives to
RLHF. Notable among these are RRHF (Yuan et al., 2023), SLiC (Zhao et al.,
2022; 2023) and DPO (Rafailov et al., 2023). These methodologies aim to more
effectively align LLMs with human preferences while avoiding the complexities
of reinforcement learning. Given supervised finetuning data
$\mathcal{D}_{\text{sft}}=\\{(x,y_{\text{ref}})\\}$ and preference data
$\mathcal{D}_{\text{hf}}=\\{(x,y_{w},y_{l})\\}$ where output text $y_{w}$ is
preferred over $y_{l}$ on the same input text $x$, RLHF first trains a
supervised fine-tuned model and a reward model, after which it applies
reinforcement algorithm such as PPO to maximize the reward. On the other hand,
RL-free algorithm fits preference data in various ways. For example, RRHF use
a trained reward model or human raters to compute rewards $\\{r_{i}\\}$ for
multiple sequences generated from the same prompt $x$, and then apply a
ranking loss plus supervised fine-tuning loss.

$\mathcal{L}(\theta)=\sum_{r_{i}<r_{j}}\max\left(0,\pi_{\theta}(y_{i}|x)-\pi_{\theta}(y_{j}|x)\right)-\lambda\log\pi_{\theta}(y_{\text{ref}}|x).$

They study various ways of sampling sequences such as beam search, online
sampling, iterative updates. Similarly, for sampled and labeled preference
pairs, SLiC uses a contrastive ranking calibration loss plus a regularization
loss

$\mathcal{L}(\theta)=\max\left(0,\delta-\log\pi_{\theta}(y_{w}|x)+\log\pi_{\theta}(y_{l}|x)\right)-\lambda\log\pi_{\theta}(y_{\text{ref}}|x),$
(1)

where $\delta$ is a positive margin and $\pi_{\theta}$ is the learnable
conditional probability function by a language model. Intuitively speaking,
SLiC penalizes model when
$\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\theta}(y_{l}|x)}<\exp(\delta)$ and thus
encourages a large likelihood ratio between positive and negative outputs.
Instead of directly fitting on human preference data, SLiC proposed to sample
pairs from the SFT policy and get them labeled by a pairwise reward-ranking
model. They show that a pairwise reward-ranking model, sampling from SFT
policy, and the proposed loss result in clear gains on a summarization dataset
that are consistent over model size scaling up.

Although RRHF and SLiC show to be a scalable alternative to PPO, they lack
theoretical understanding. DPO analyzes RLHF’s objective function in the form
of KL-regularized reward maximization, and analytically solved the optimal
policy induced by a reward function. Based on Bradley-Terry (BT) model
(Bradley & Terry, 1952), DPO proposes a maximum likelihood estimator (MLE) to
fit on human preference data directly. It establishes theoretical foundation
that connected language model with preference model as a density estimation
problem from the labeled response pairs. Mathematically, DPO expresses the
human preference probability in terms of only the optimal policy $\pi^{*}$ and
reference policy $\pi_{\text{sft}}$:

$p^{*}(y_{1}\succ
y_{2}|x)=\frac{1}{1+\exp\left(\beta\log\frac{\pi^{*}(y_{2}|x)}{\pi_{\text{sft}}(y_{2}|x)}-\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{\text{sft}}(y_{1}|x)}\right)}$
(2)

In the above equation, $\pi^{*}$ is the function to be estimated. Empirically,
one can leverage observed preference pairs to approximate $p^{*}(y_{1}\succ
y_{2}|x)$. To estimate $\pi^{*}$ as a density estimation problem, the optimal
way is to fit a policy model on collected preference pairs sampled from
$\pi^{*}$. However, DPO uses the collected human preference data from other
policies directly in all the experiments and lacks study on the effect of
sampling. Although they propose to sample pairs from SFT policy and get them
labeled by human, it is still not strictly a MLE for the preference model due
to the mismatch between the sampling distribution and $\pi^{*}$. In reality,
it is very challenging to obtain human preference pairs directly sampled from
$\pi^{*}$.

Figure 1: RSO first fits a pairwise reward-ranking model
$\rho_{\psi}(x,y_{1},y_{2})$ from human preference data. This model is later
applied to generate preference pairs with candidates sampled from the optimal
policy, followed by a preference optimization step to align sequence
likelihood towards preferences.

In this work, we address the above issues by constructing preference pairs
from the approximated $\pi^{*}$. We illustrate our pipeline in Figure 1.
Starting from a human preference dataset $\mathcal{D}_{\text{hf}}$ collected
from other policies, we first train a pairwise reward-ranking model, then
apply a statistical rejection sampling algorithm to generate response pairs
sampled from optimal policy by using SFT policy and the pairwise reward-
ranking model. After that, we label the sampled response pairs by the reward
model. Then we fit the model on labeled pairs via classification loss.

Our statistical rejection sampling refers to the one in the statistical field
(Neal, 2003). In RLHF works (Bai et al., 2022; Stiennon et al., 2020; Touvron
et al., 2023), they usually refer to rejection sampling as best-of-N or top-k-
over-N algorithm, where they sample a batch of N completions from a language
model policy and then evaluate them across a reward model, returning the best
one or the best k ones. In this paper we show that top-k-over-N is a special
case of our statistical rejection sampling.

Our contributions of this work are three-fold:

  1. 1.

We propose a scalable and easy-to-implement framework to learn from human
preference data. We provide a comprehensive recipe among different choices of
loss functions and preference pairs generation. We show the importance of the
reward model instead of directly optimizing the model on the preference data.

  2. 2.

Statistically, we unify DPO and SLiC by showing that they vary by loss
functions to fit on human preference data: DPO is a logistic regression on
human preference data and SLiC is _almost_ equivalent to a support vector
machine (SVM) with hinge loss. We improve SLiC as the SVM counter part of DPO.

  3. 3.

We propose a statistical rejection sampling algorithm to sample pairs from the
optimal policy and get them labeled by a pairwise reward-ranking model. The
proposed sampling strategy is shown to be effective on three generative tasks.

##  2 Preliminaries

##### Learning from Human Feedback

Several works show the significant improvement of conditional language
generation by learning from human feedback data. Such works include RLHF
(Ziegler et al., 2019), RRHF (Yuan et al., 2023), SLiC (Zhao et al., 2023),
and DPO (Rafailov et al., 2023). All algorithms take two inputs:

  1. 1.

$\pi_{\text{sft}}(y|x)$: a supervised fine-tuned policy (SFT), where $x$ is
the prompt and $y$ is the response.

  2. 2.

$\mathcal{D}_{\text{hf}}=\\{x^{(i)},y^{(i)}_{w},y^{(i)}_{l}\\}_{i=1}^{N}$: a
human preference dataset that distinguishes the better response from the worse
given the same prompt.

##### DPO, SLiC, RRHF and RLHF

In the rapidly evolving landscape of model optimization based on human
feedback data, various strategies have been developed. DPO employs a
straightforward approach, fitting a logistic regression model directly to the
human preference dataset $\mathcal{D}_{\text{hf}}$ to achieve an optimized
policy. In a similar vein, SLiC-direct employs a max margin loss function to
fit human preference data, enhancing the performance of the model. SLiC-
sample-rank, a variant of SLiC (Zhao et al., 2023), takes a two-step approach:
it initially fits a pairwise reward-ranking model, denoted as
$\rho_{\psi}(x,y_{1},y_{2})$, to human preference data. Subsequently, response
pairs are sampled from the SFT policy $\pi_{\text{sft}}(y|x)$ and labeled
according to $\rho_{\psi}(x,y_{1},y_{2})$; the SLiC loss is then applied to
these generated preference pairs (Zhao et al., 2023). Another strategy, RRHF,
also generates multiple decoded sequences but distinguishes itself by ranking
these sequences either manually by humans or through a reward model. The RRHF
rank loss is then applied to refine the model based on these rankings. Lastly,
RLHF diverges by first fitting a reward model and then employing the PPO
algorithm to yield the final optimized policy (Schulman et al., 2017). Each of
these approaches contributes uniquely to the field, offering a variety of
methodologies for leveraging human feedback in model training.

##### KL-Constrained Reward Maximization Objective

Starting with a reward function $r(x,y)$ and input prompt distribution
$\mathcal{P}$, the DPO and RLHF optimizes for the following objective:

$\max_{\pi}\mathbb{E}_{x\sim\mathcal{P},y\sim\pi}\left[r(x,y)\right]-\beta\mathbb{D}_{KL}\left[\pi(y|x)||\pi_{\text{sft}(y|x)}\right]$
(3)

##### Optimal Policy

Rafailov et al. (2023) showed that the optimal policy $\pi_{r}(y|x)$ that
maximizes the above objective is:

$\pi_{r}(y|x)=\frac{1}{Z(x)}\pi_{\text{sft}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right)$
(4)

for all $x\in\mathcal{P}$, where
$Z(x)=\sum_{y}\pi_{\text{sft}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right)$ is
the partition function. $\beta$ controls the balance between exploitation and
exploration. When $\beta\rightarrow 0$, all probability mass will concentrate
on the max reward with full exploitation. When $\beta\rightarrow\infty$,
optimal policy will be the same as $\pi_{\text{sft}}$ with full exploration.
Rearrange the Equation (4) we get

$r(x,y)=\beta\log\frac{\pi_{r}(y|x)}{\pi_{\text{sft}}(y|x)}+\beta\log Z(x).$
(5)

The Equation (4) and (5) establish the relation between optimal policy and the
reward function. One can infer the other. In reality, the final goal is to
have a good policy for response generation and $\pi_{r}(y|x)$ is usually of
more interest. The key is to effectively estimate the $\pi_{r}(y|x)$ from the
human preference data.

##### Preference Model

Let the ground-truth reward function be $r^{*}$, then the optimal policy
$\pi^{*}$ associated with $r^{*}$ can be represented by Equation (4). For two
responses $(y_{1},y_{2})$ from the same input $x$, one can assume that

${\mathbb{P}}(y_{1}\succ y_{2}|x)=g(r^{*}(x,y_{1})-r^{*}(x,y_{2})),$ (6)

where $g:\mathbb{R}\rightarrow[0,1]$ is a monotonically non-decreasing
function that converts the reward difference into winning probability.
Specifically, if we set $g$ as sigmoid function $\sigma$, we get the Bradley-
Terry (BT) model (Bradley & Terry, 1952). Reusing Equation (5), the reward
advantage of $y_{1}$ over $y_{2}$ is

$\delta_{r^{*}}(y_{1},y_{2},x,\pi^{*},\pi_{\text{sft}},\beta)\triangleq
r^{*}(x,y_{1})-r^{*}(x,y_{2})=\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{\text{sft}}(y_{1}|x)}-\beta\log\frac{\pi^{*}(y_{2}|x)}{\pi_{\text{sft}}(y_{2}|x)}$
(7)

${\mathbb{P}}(y_{1}\succ
y_{2}|x)=g(\delta_{r^{*}}(y_{1},y_{2},x,\pi^{*},\pi_{\text{sft}},\beta))$ (8)

The above equation establishes a connection between the preference data and
optimal policy. If we leverage the human preference data to represent
${\mathbb{P}}(y_{1}\succ y_{2}|x)$, the estimation of $\pi^{*}$ can be viewed
as a density estimation problem from the preference data. We will discuss
different ways of estimating $\pi^{*}$ from the preference data in Section
3.1.

##### Policy Estimation on Preference Pairs

DPO estimates the $\pi^{*}$ by fitting the BT model on preference data
directly using logistic loss. SLiC estimates the $\pi^{*}$ via a contrastive
manner to ensure the winner of the preference pair has higher probability than
the loser. In statistical density estimation problem, the preference pairs
should be generated from $\pi^{*}$, the density to be estimated, while DPO
uses preference pairs from some unknown distribution and SLiC-sample-rank uses
preference pairs from $\pi_{\text{sft}}$. Thus neither of the above approaches
are the MLE of the $\pi^{*}$. This motivates us to develop an approach that
can obtain preference pairs from $\pi^{*}$.

##### Reward Model

Usually the reward model is a pointwise score assigned to a (prompt, response)
pair. The model is trained based on BT model (Ziegler et al., 2019; Bai et
al., 2022). We argue that it is an easier and more straightforward way to
train a pairwise reward model from (prompt, worse response, better response)
triplets. Zhao et al. (2023) demonstrates that pairwise reward model is
preferred in RL-free learning. In this work we train the pairwise reward-
ranking model $\hat{{\mathbb{P}}}(y_{1}\succ
y_{2}|x)=\rho_{\psi}(x,y_{1},y_{2})$ on $\mathcal{D}_{p}$ to predict the
probability that $y_{1}$ is preferred over $y_{2}$. For summarization task,
the input format is “[CONTEXT] document [SUMMARY A] positive summary [SUMMARY
B] negative summary”, and the output value is either “A” or “B”. For AI
assistant tasks, the input format is “[CONTEXT] conversation between human and
assistant [RESPONSE A] positive response [RESPONSE B] negative response”, and
the output value is either “A” or “B”.

We can induce the reward score of response $y_{1}$ from the reward score of
response $y_{2}$ from winning probability according to Equation (6) with
$g=\sigma$:

$r(x,y_{1})=\text{logit}({\mathbb{P}}(y_{1}\succ y_{2}|x))+r(x,y_{2}),$

where $\text{logit}(x)=\log(\frac{x}{1-x})$. Suppose we have a baseline
sequence $y_{b}$ of reward score 0 and a fitted pairwise reward-ranking model
$\rho_{\psi}(x,y,y_{b})$, we have estimated reward score as

$r_{\psi}(x,y)=\text{logit}(\rho_{\psi}(x,y,y_{b})).$ (9)

In other words, “pointwise” reward score can be derived from a “pairwise”
reward-ranking model with a baseline sequence. In practice, we choose a random
decoded sequence from SFT policy as the baseline.

##  3 RSO Approach

###  3.1 Statistical Estimation of the Optimal Policy $\pi^{*}$

Our proposed approach is illustration in Figure 1. The inputs to our system
are SFT policy, reward-ranking model, and prompts. First we sample responses
from the optimal policy through rejection sampling approach, then we fit a
classification model on labeled preference pairs.

To study the effectiveness of our approach, we consider a few options on loss
and preference dataset construction. Given a preference dataset
$\mathcal{D}_{p}=\\{(x^{(i)},y_{w}^{(i)},y_{l}^{(i)})\\}$, we can estimate
$\pi^{*}$ according to Equation (8). There are two aspects we need to consider
for estimating $\pi^{*}$:

  1. 1.

Choice of loss function: Equation (8) is a simple binary classifier with
$\delta_{r^{*}}$ as logit. Two most common choices are logistic loss used in
logistic regression and hinge loss used in support vector machine.

  2. 2.

Choice of $\mathcal{D}_{p}$: Equation (8) does not depend on the distribution
of $y_{1},y_{2}$ given $x$. Thus we need to decide how to obtain
$(x,y_{1},y_{2})$ triplets.

##### Choice of loss function

Given a preference dataset
$\mathcal{D}_{p}=\\{(x^{(i)},y_{w}^{(i)},y_{l}^{(i)})\\}$, we can fit a binary
classifier according to Equation (8). DPO (Rafailov et al., 2023) proposed to
use sigmoid loss on normalized likelihood (sigmoid-norm) to fit a logitistic
regression:

$\mathcal{L}_{\text{sigmoid-
norm}}\left(\pi_{\theta}|\pi_{\text{sft}},\mathcal{D}_{p}\right)=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{p}}\left[\log\sigma\left(\gamma\log\frac{\pi_{\theta}\left(y_{w}|x\right)}{\pi_{\text{sft}}\left(y_{w}|x\right)}-\gamma\log\frac{\pi_{\theta}\left(y_{l}|x\right)}{\pi_{\text{sft}}\left(y_{l}|x\right)}\right)\right].$
(10)

In their original paper, $\gamma=\beta$ and we extend this in preference model
as $g(x)=\sigma(\frac{\gamma}{\beta}x)$. By doing this, we can fully control
the steepness of the logits function for classification. The larger the
$\gamma$, the more we penalize the mis-classified examples at the decision
boundaries. Another thing to notice is that we don’t include any bias term in
the logistic regression. This is due the the symmetry property of the
preference data. If two rewards are the same, the winning probability should
always be 0.5.

SLiC (Zhao et al., 2023) proposed to use a hinge loss as

$\mathcal{L}_{\text{hinge}}\left(\pi_{\theta}|\mathcal{D}_{p}\right)=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{p}}\left[\max\left(0,1-\left[\gamma\log\pi_{\theta}\left(y_{w}|x\right)-\gamma\log\pi_{\theta}\left(y_{l}|x\right)\right]\right)\right]$
(11)

Note that we use $1/\gamma$ as the margin $\delta$ used in the original SLiC
paper. This is equivalent to a hinge loss with logit
$\left(\gamma\log\pi_{\theta}\left(y_{w}|x\right)-\gamma\log\pi_{\theta}\left(y_{l}|x\right)\right)$.
If we replace the logit by
$\frac{\gamma}{\beta}\delta_{r}(y_{1},y_{2},x,\pi_{\theta},\pi_{\text{sft}},\gamma)$,
we introduce the hinge loss on normalized likelihood (hinge-norm):

$\mathcal{L}_{\text{hinge-
norm}}\left(\pi_{\theta}|\mathcal{D}_{p}\right)=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{p}}\left[\max\left(0,1-\left[\gamma\log\frac{\pi_{\theta}\left(y_{w}|x\right)}{\pi_{\text{sft}}\left(y_{w}|x\right)}-\gamma\log\frac{\pi_{\theta}\left(y_{l}|x\right)}{\pi_{\text{sft}}\left(y_{l}|x\right)}\right]\right)\right]$
(12)

Figure 2: Statistical rejection sampling illustration. Although the output
space of language models is a huge high-dimensional discrete space, we use a
continuous 1D input space for illustration purpose. There are three curves in
the figure: $M$ times SFT policy, reward, optimal policy. The sample is first
generated by SFT policy, then gets accepted or rejected depending on whether a
uniform random variable locates in acceptance or rejection region. If the
sample has high SFT policy probability but low optimal policy probability and
reward score, it has a higher chance of being rejected.

##### Choice of preference data distribution

Suppose we have access to the oracle preference data
$\mathcal{D}^{*}=\\{(x^{(i)},y_{w}^{(i)},y_{l}^{(i)})\mid
y_{w}^{(i)},y_{l}^{(i)}\sim\pi^{*}(y|x^{(i)})\\}_{i=1}^{N^{*}}$, we can
directly fit a maximum likelihood estimation (MLE) on the dataset. In reality,
we may not have access to such data, and we have access to
$\mathcal{D}_{\text{hf}}=\\{(x^{(i)},y_{w}^{(i)},y_{l}^{(i)})\mid
y_{w}^{(i)},y_{l}^{(i)}\sim\pi_{\text{unk}}(y|x^{(i)})\\}_{i=1}^{N_{\text{unk}}}$,
where $\pi_{\text{unk}}$ denotes some mixed unknown policies. The mixed
unknown policies can include SFT policy, current RLHF policy, or policies from
other agents, or even web mined preference pairs (Touvron et al., 2023). In
our study, we use the human preference data sampled from other agents, such as
OpenAI agents for Reddit TL;DR dataset (Stiennon et al., 2020) and Anthropic
agents for AnthropicHH dataset (Bai et al., 2022).

Given $\mathcal{D}_{\text{hf}}$, we consider the following three choices:

  1. 1.

direct: directly fit the policy on $\mathcal{D}_{\text{hf}}$ according to
Equation (8). This is the approach used in DPO. In SLiC, they also proposed
similar variant SLiC-direct, but with a different loss function.

  2. 2.

sft-sample-rank: first train a reward-ranking model
$\rho_{\psi}(x,y_{1},y_{2})$ on $\mathcal{D}_{\text{hf}}$. Then use
$\pi_{\text{sft}}(y|x)$ to sample response pairs and label them by
$\rho_{\psi}(x,y_{1},y_{2})$. The results in a preference dataset
$\mathcal{D}_{p}=\\{(x^{(i)},y_{w}^{(i)},y_{l}^{(i)})\mid
y_{w}^{(i)},y_{l}^{(i)}\sim\pi_{\text{sft}}(y|x^{(i)})\\}_{i=1}^{N_{\text{sft}}}$.
This is the same as SLiC-sample-rank variant proposed in the SLiC paper.

  3. 3.

rso-sample-rank: first train a reward-ranking model
$\rho_{\psi}(x,y_{1},y_{2})$ on $\mathcal{D}_{\text{hf}}$. Then use
$\pi_{r_{\psi}}(y|x)$ induced by $r_{\psi}$ according to Equation (4) to
sample response pairs, where $r_{\psi}(x,y)$ is induced from
$\rho_{\psi}(x,y_{1},y_{2})$. After that we label response pairs using the
reward-ranking model to construct the preference dataset
$\mathcal{D}_{p}=\\{(x^{(i)},y_{w}^{(i)},y_{l}^{(i)})\mid
y_{w}^{(i)},y_{l}^{(i)}\sim\pi_{r_{\psi}}(y|x^{(i)})\\}_{i=1}^{N_{\rho_{\psi}}}$
from the optimal policy $\pi_{r_{\psi}}(y|x)$ via reward function
$r_{\psi}(x,y)$ according to Equation (4). The reward function $r_{\psi}(x,y)$
is induced by a fitted reward model $\rho_{\psi}(x,y_{1},y_{2})$ according to
Equation (9).

Statistically speaking, since we are estimating $\pi^{*}(y|x)$, it is desired
to draw samples from $\pi^{*}(y|x)$. Rso-sample-rank is the best solution
towards this direction because it can generate samples from
$\pi_{r_{\psi}}(y|x)$, which is closer to $\pi^{*}(y|x)$ than
$\pi_{\text{unk}}$ used in direct and $\pi_{\text{sft}}$ used in sft-sample-
rank. However, sampling from $\pi_{r_{\psi}}$ is not straightforward, and we
propose a statistical rejection sampling approach to achieve this.

Algorithm 1 Statistical Rejection Sampling Algorithm in Python

⬇

from typing import List

import numpy as np

def conduct_rejection_sampling(response_candidates: List[str],

response_rewards: List[float],

num_samples: int,

beta: float):

"""Conducts␣rejection␣sampling␣guided␣by␣rewards.

␣␣Args:

␣␣␣␣response_candidates:␣response␣candidates␣from␣sft␣policy

␣␣␣␣response_rewards:␣response␣rewards.

␣␣␣␣num_samples:␣number␣of␣samples␣to␣sub-sample.

␣␣␣␣beta:␣beta␣parameter␣in␣KL-constrained␣reward␣maximization␣objective.

␣␣Returns:

␣␣␣␣Rejection␣sampled␣sequences␣from␣the␣optimal␣policy.

␣␣"""

candidates = {c: r for c, r in zip(response_candidates, response_rewards)}

accepted = []

while len(accepted) < num_samples:

max_reward = max(candidates.values())

to_remove = []

for c, r in candidates.items():

u = np.random.uniform()

if u >= np.exp((r - max_reward) / beta):

continue

accepted.append(c)

to_remove.append(c)

if len(accepted) == num_samples:

break

for c in to_remove:

candidates.pop(c)

return accepted

###  3.2 Statistical Rejection Sampling Algorithm

Statistical rejection sampling (Neal, 2003) is an efficient statistical
technique to generate observations from a distribution. If we want to generate
a distribution of density $\pi_{r_{\psi}}$, we can use $\pi_{\text{sft}}$ as
the proposal distribution. Then we follow the following steps:

  1. 1.

Generate $y\sim\pi_{\text{sft}}(y|x)$ and $u\sim U[0,1]$.

  2. 2.

Let $M=\min\\{m\mid m\pi_{\text{sft}}(y|x)\geq\pi_{r_{\psi}}(y|x)\text{ for
all }y\\}$. If $u<\frac{\pi_{r_{\psi}}(y|x)}{M\pi_{\text{sft}}(y|x)}$, then we
accept $y$. Otherwise, we reject $y$ and redo the sampling again.

Figure 2 is an illustration of the statistical rejection sampling approach.
Starting from a proposal distribution $\pi_{\text{sft}}$, we can draw samples
from the optimal policy by rejecting more at the low reward region and
accepting more at the high reward region.

##### Statistical Rejection Sampling Algorithm

We propose Algorithm 1 as an empirical version of the statistical rejection
sampling algorithm. The derivation will be shown in Section A.1. Regarding the
Algorithm 1, we have the following theorem:

######  Theorem 1.

Let $N$ be number of candidate responses, and $r_{\text{max}}$ be the maximum
rewards among the candidate responses not yet accepted. As
$N\rightarrow\infty$, Algorithm 1 can generate $K$ distinct samples from
$\pi_{r_{\psi}}$ with expected acceptance rate
$\mathbb{E}_{y\sim\pi_{\text{sft}}(y|x)}\left[\exp\left(\frac{1}{\beta}\cdot\left(r_{\psi}(x,y)-r_{\text{max}}\right)\right)\right]$,
where $r_{\psi}(x,y)$ is defined in Equation (9).

If $\beta\rightarrow\infty$, each sample generated from SFT policy will be
accepted with probability 1. If $\beta\rightarrow 0$, only the highest reward
response will be accepted and all other responses will be rejected. In the
case of $\beta\rightarrow 0$, the Algorithm 1 will sample the $K$ samples with
highest rewards, which coincide with the statistical rejection sampling (top-
k-over-N) referred by Bai et al. (2022) and Touvron et al. (2023). $\beta$
indicates how much we trust the reward model. If the reward model is very
accurate and robust, we should set a small $\beta$. Otherwise, we should set a
larger $\beta$. In practice, we treat $\beta$ as a hyper-parameter and pick
one according to validation metrics.

##  4 Related Work

Recent advancements in the pre-training of LLMs like GPT-3 (Brown et al.,
2020), Gopher (Rae et al., 2021), Chinchilla (Hoffmann et al., 2022), and PaLM
2-L (Anil et al., 2023) have greatly improve the ability on achieving tasks in
a zero-shot or few-shot way (Radford et al., 2019; Chowdhery et al., 2022).
However, improving performance in downstream tasks and better aligning with
user intent can be achieved by fine-tuning these systems on datasets that
include instructions and human-generated completions (Mishra et al., 2021;
Sanh et al., 2021; Wei et al., 2021). Although instruction-based tuning has
proven successful, it’s often easier to gather human preference on responses
than it is to collect expert demonstrations. Consequently, later research has
shifted toward training LLMs using human preferences. This approach has led to
enhanced performance in various applications, including translation (Kreutzer
et al., 2018), summarization (Stiennon et al., 2020), storytelling (Ziegler et
al., 2019), and following instructions (Ramamurthy et al., 2022). The process
usually includes a reward model training based on Bradley-Terry (BT) model
(Bradley & Terry, 1952) and reinforcement learning with human feedback (RLHF)
to make the model perform better according to the reward function. RLHF
(Ziegler et al., 2019) optimizes for maximum reward by interacting with a
reward model using reinforcement learning algorithms such as PPO (Schulman et
al., 2017). Despite the success of RLHF, the fine-tuning of large language
models with reinforcement learning remains a practical challenge due to its
instability, reward hacking, and scalability (Zhao et al., 2023; Rafailov et
al., 2023). This work provides an effective RL-free way to optimize for human
preference feedback with theoretical guarantees.

Recent works have proposed theoretically-justified approaches to optimizing
relative preferences without relying on RL (Zhao et al., 2023; Yuan et al.,
2023; Rafailov et al., 2023). By optimizing the model’s compatibility with
preference datasets under models such as the BT model, these methods fit on
human or model ranked data pairs. SLiC (Zhao et al., 2023) proposed to first
train a pairwise reward-ranking model, then label pairs generated from
supervised fine-tuning (SFT model using the reward model, finally train the
model using a contrastive calibration loss (Zhao et al., 2022) and a
regularization fine-tuning loss. RRHF (Yuan et al., 2023) assumes access to a
list of responses and their reward values to the same input, and use a zero-
margin likelihood contrastive loss. DPO (Rafailov et al., 2023) fits a model
directly on human preference data using BT model. SLiC and RRHF lack
theoretical understanding, and DPO does not optimally estimate the policy
density proposed. To address the aforementioned issues, this work unifies the
SLiC and DPO approaches from choice of loss perspective with theoretical
understanding, and proposes an improved way of collecting preference pairs via
statistical rejection sampling.

Statistical rejection sampling (Neal, 2003) is a statistical approach used to
generate samples from a target distribution using a proposal distribution.
With the terminology “rejection sampling”, Bai et al. (2022) propose to select
highest top $k$ sampled candidates to collect human preference datasets. More
recently, Touvron et al. (2023) propose to use the same approach with PPO that
shows to improve RLHF. Their reference to rejection sampling means to take the
top $k$ best reward decoding sequences among the samples, which is not the
same concept as those used in statistical world. In this work, we use the
statistical rejection sampling from the statistical world and show the
existing approach is a special case of our algorithm. We also provide the
theoretical guarantees and show that the approach can deliver a MLE estimation
for human preference data.

##  5 Experiments

##### Tasks

We study RSO on three different open-ended text generation datasets: Reddit
TL;DR summarization (Stiennon et al., 2020), CNN/DailyMail (Hermann et al.,
2015), and Anthropic Helpful and Harmless (AnthropicHH) dialogue dataset (Bai
et al., 2022).

The Reddit TL;DR summarization dataset contains both fine-tune data
$\mathcal{D}_{\text{sft}}$, human feedback data $\mathcal{D}_{\text{hf}}$,
along with their SFT and RLHF model decodes which we use for comparison with
our models. $\mathcal{D}^{\text{tldr}}_{\text{sft}}$ is a filtered version of
Reddit TL;DR dataset (Völske et al., 2017). It contains 117k/6k/6k examples in
train, validation and test splits. $\mathcal{D}^{\text{tldr}}_{\text{hf}}$
consists of 64k human preferences on decodes from multiple models.

The CNN/DailyMail dataset contains online news articles paired with multi-
sentence summaries. $\mathcal{D}^{\text{cnndm}}_{\text{sft}}$ has 287k/13k/11k
examples in train, validation and test splits. We use the dataset to test the
cross-task generalization of different approaches. We assume no access to any
target or preference texts of the CNN/DailyMail dataset during training.
Starting from a SFT model trained on $\mathcal{D}^{\text{tldr}}_{\text{sft}}$,
we further optimize the SFT policy using
$\mathcal{D}^{\text{tldr}}_{\text{hf}}$. We evaluate the performance using
target texts on validation split of $\mathcal{D}^{\text{cnndm}}_{\text{sft}}$.

The AnthropicHH is a dialogue dataset with $x$ as conversation between a human
query and an AI assistant. Each example ends with a pair of responses
generated by an unknown large language model along with a preference label
denoting the human-preferred response according to helpfulness and
harmlessness. The original dataset has 161k/9k examples in train and test
splits. Each example has a task tag and two responses, a better one and a
worse one. The task includes helpfulness and harmlessness and they conflict
with each other. We use the helpfulness task in our experiment setting. For
SFT, we use the positive response (the better one) as SFT target.

##### Method

Starting from a T5-large model (770M) (Raffel et al., 2020) large SFT policy
and a T5-XXL (11B) pairwise reward-ranking model111We find that smaller T5
ranking/reward models do not converge reliably in our setup., we consider nine
settings as discussed in Section 3.1. The settings are all the combinations
between loss functions $(\mathcal{L}_{\text{sigmoid-
norm}},\mathcal{L}_{\text{hinge}},\mathcal{L}_{\text{hinge-norm}})$ and
preference data distribution (direct, sft-sample-rank, rso-sample-rank). Note
that DPO approach in the original paper is the same as sigmoid-norm-direct in
our setting. And the SLiC approach in the original paper is almost the same as
hinge-sft-sample-rank in our setting with two tiny differences. The first
difference is that the original SLiC paper has calibration loss (first term in
Equation (1)) and regularization loss (second term in Equation (1)), we found
that regularization loss does not significantly improve the final metrics
(discussed in Appendix A.5) and thus dropped it to follow the DPO recipe. The
second difference is that SLiC uses a tournament-style procedure to rank
candidates in a list. For example, given a list of 4 candidates $c1,c2,c3,c4$,
we first rank $c1,c2$ and $c3,c4$ and then rank $winner(c1,c2)$,
$winner(c3,c4)$. Given $m$ candidates, the ranking model is called $m-1$ times
and $m-1$ positive/negative pairs are yielded. In our case, we only use the
first round of the tournament ranking, which are strictly random samples from
SFT policy. Given $m$ candidates, we generate $m/2$ pairs. We will discuss the
sampling difference in Section 5.2. Unless specifically mentioned, we set
$\beta=0.5$ and $\gamma=0.05$. For statistical rejection sampling algorithm
(Algorithm 1), we first sample 64 response candidates from the SFT policy,
then sub-sample 8 samples. For SFT sampling, we start with 64 response
candidates sampled, then randomly sub-sample 8 candidates without replacement.
We use batch size 128 and learning rate 1e-5 with Adafactor optimizer (Shazeer
& Stern, 2018). For each run, we pick the checkpoint with the highest reward
score according to the reward-ranking model.

##### Evaluation

Our experiments use three different approaches to evaluate. They are all side-
by-side comparisons on generated sequence versus the original SFT target. We
report the win rate by three systems: trained T5-XXL pairwise reward-ranking
model (Reward Model), PaLM 2-L few-shot side-by-side (AutoSxS), and human
side-by-side (HumanSxS).

For AutoSxS, we use PaLM 2-L few-shot in-context learning to infer 8 decoded
samples with 4 flipped the order. The label contains three choices: A, B, and
tie with score 1, 0, and 0.5, respectively. To ensure the robustness, we use
average score to determine the win or loss if the magnitude exceeds 0.35. The
AutoSxS prompts are reported in Appendix A.3. The purpose of the AutoSxS is to
prevent the artificially high reward scores by Reward Model due to reward
hacking on learned policies. Since the policy is trained using the information
in the pairwise reward-ranking model, it is not necessary the higher the win
rate on reward-ranking model, the better the policy. The AutoSxS has been
demonstrated as effective and consistent in Rafailov et al. (2023) using GPT-4
as zero-shot rater. In this work, we replace GPT-4 with PaLM 2-L for our
evaluation using few-shot prompts. The quality of PaLM 2-L on similar tasks
has been shown to be close to human raters (Lee et al., 2023; Shu et al.,
2023). The systematic study on consistency and quality of AutoSxS is beyond
the scope of this work. The human evaluation details will be discussed in
Section 5.3.

###  5.1 Performance comparison on three tasks

The comparison results are shown in Table 1. Compared with DPO,
$\text{RSO}_{\text{sigmoid}}$ and $\text{RSO}_{\text{sigmoid}}$ From the table
we can conclude that “sft-sample-rank” can bring consistent gains over
“direct”. And “rso-sample-rank” brings further gains on top of “sft-sample-
rank”. The explanation is that when we fit the language model on preference
data, the closer the data distribution to the optimal policy, the closer the
estimator to the real MLE. Let $\mathbb{D}(\cdot,\cdot)$ denote any
distribution distance, and $\mathcal{D}_{p}^{\text{a}}$ be the preference data
generated by policy a, we have

$\mathbb{D}(\mathcal{D}_{p}^{\text{rso-sample-
rank}},\mathcal{D}_{p}^{\pi^{*}})\leq\mathbb{D}(\mathcal{D}_{p}^{\text{sft-
sample-
rank}},\mathcal{D}_{p}^{\pi^{*}})\leq\mathbb{D}(\mathcal{D}_{p}^{\text{direct}},\mathcal{D}_{p}^{\pi^{*}})$

Thus “rso-sample-rank” results in the best performance.

Regarding the loss function, we observed that sigmoid-norm and hinge-norm
perform similarly, while hinge shows some extend of reward hacking with higher
reward model win rate by lower auto rater win rate. This is because the hinge
loss trust too much on the reward function without considering any SFT policy
as shown in Equation (11).

Approach | Ablation | Metrics
---|---|---
| Loss | Preference Pair | Reward Model (%) | AutoSxS (%) 222The numbers are lower than Reward Model win rate because AutoSxS has three classes: A, B, or Same, while Reward Model only has two classes A or B. Besides, AutoSxS scores have lower risk of reward hacking.
Reddit TL;DR
DPO | sigmoid-norm | direct | 84.35 | 67.72
| sigmoid-norm | sft-sample-rank | 88.63 | 69.02
$\text{RSO}_{\text{sigmoid-norm}}$ | sigmoid-norm | rso-sample-rank | 92.37 | 71.86
$\text{SLiC}_{\text{direct}}$ | hinge | direct | 86.92 | 60.54
$\text{SLiC}_{\text{sample-rank}}$ | hinge | sft-sample-rank | 89.75 | 67.04
| hinge | rso-sample-rank | 93.36 | 69.26
| hinge-norm | direct | 83.93 | 66.63
| hinge-norm | sft-sample-rank | 88.04 | 68.46
$\text{RSO}_{\text{hinge-norm}}$ | hinge-norm | rso-sample-rank | 92.80 | 70.84
CNN/DailyMail
DPO | sigmoid-norm | direct | 61.31 | 37.36
| sigmoid-norm | sft-sample-rank | 62.72 | 38.63
$\text{RSO}_{\text{sigmoid-norm}}$ | sigmoid-norm | rso-sample-rank | 69.38 | 39.71
$\text{SLiC}_{\text{direct}}$ | hinge | direct | 64.18 | 33.63
$\text{SLiC}_{\text{sample-rank}}$ | hinge | sft-sample-rank | 67.16 | 33.21
| hinge | rso-sample-rank | 71.62 | 35.46
| hinge-norm | direct | 60.04 | 33.91
| hinge-norm | sft-sample-rank | 61.77 | 40.63
$\text{RSO}_{\text{hinge-norm}}$ | hinge-norm | rso-sample-rank | 69.82 | 42.18
AnthropicHH
DPO | sigmoid-norm | direct | 51.63 | 24.01
| sigmoid-norm | sft-sample-rank | 85.09 | 39.56
$\text{RSO}_{\text{sigmoid-norm}}$ | sigmoid-norm | rso-sample-rank | 86.94 | 40.98
$\text{SLiC}_{\text{direct}}$ | hinge | direct | 35.95 | 15.69
$\text{SLiC}_{\text{sample-rank}}$ | hinge | sft-sample-rank | 80.82 | 30.66
| hinge | rso-sample-rank | 82.21 | 32.56
| hinge-norm | direct | 49.55 | 22.89
| hinge-norm | sft-sample-rank | 82.40 | 35.96
$\text{RSO}_{\text{hinge-norm}}$ | hinge-norm | rso-sample-rank | 84.44 | 38.58
Table 1: Compare different methods to leverage human feedback data on three
tasks. “direct” preference pair directly comes from human preference data.
“sft-sample-rank” preference pair uses SFT policy to sample 4 pairs for each
training example input text and get them labeled by the pairwise reward model.
“rso-sample-rank” preference pair uses rejection sampling to sample 4 pairs
for each training example input text and get them labeled by the pairwise
reward model. Both pairwise reward model and few-shot PaLM 2-L win rate
against SFT target text are reported. RSO shows improvement on three tasks
over DPO and SLiC with a clear margin.

We showcase an example with responses from different policies on Reddit TL;DR
and AnthropicHH tasks in Figure 3 and Figure 4, respectively.

Figure 3: Example summaries generated by SFT, SLiC, DPO, and RSO policies for
a Reddit post. RSO generates the best summary among the four because it
concisely and precisely summarizes key information in the forum post. Salient
details are bolded.  Figure 4: Example responses generated by SFT, SLiC, DPO,
and RSO policies for a Human-Assistant dialogue on AnthropicHH dataset. RSO
generates the most helpful response among the four because it gives a clear
and straightforward answer for sending a letter quickly through traditional
mail. In contrast, SFT repeats information about email rater than answering
the question about traditional mail, SLiC and DPO are vague and repetitive.
Salient details are bolded.

###  5.2 RSO ablation

##### Effect of $\gamma$ and $\beta$ in RSO

To study the effect of $\gamma$, we fix the statistical rejection sampling
$\beta=0.5$, and vary $\gamma=0.005,0.05,0.5$ in the loss function on Reddit
TL;DR dataset. Figure 5(a) shows that $\gamma=0.05$ provides the optimal win
rate.

To study the effect of $\beta$ for rejection sampling, we fix the
$\gamma=0.05$ in the loss function and vary $\beta=0,0.05,0.5,5$ on Reddit
TL;DR dataset. Figure 5(b) shows that $\beta=0.5$ provides the optimal win
rate. Notice that $\beta=0$ coincides with the rejection sampling algorithm
mentioned in Bai et al. (2022) and Touvron et al. (2023). Thus it is not
always best to select top-k-over-N, which is because the quality of reward
model may be limited and we should not blindly trust reward without
regularization in the constraint reward optimization (Equation (3)).

(a) Reward model win rate of various $\gamma$ in loss functions (Equation
(10), (11), (12)). $\beta$ is fixed at 0.5. Shaded areas are 95% confidence
intervals of the reward.

(b) Reward model win rate of various $\beta$ in statistical rejection sampling
(Algorithm 1). $\gamma$ is fixed at 0.05. The horizontal lines are reward
model win rates from the sft-sample-rank preference pairs. Shaded areas are
95% confidence intervals of the reward.

Figure 5: Effect of hyper-parameters in loss functions and statistical
rejection sampling algorithm.

##### Preference pairs sampling and ranking

To better understand the effect of tournament ranking and statistical
rejection sampling, we compare among different sampling strategies. Since we
first sample 64 responses from sft policy and followed by 8 responses by
statistical rejection sampling, it is natural to ask: why not use all of the
64 samples in the calibration? In SLiC, we use tournament ranking, which
introduces bias on higher reward sequences. We would like to understand the
effect of that. Starting with $n$ responses, we can construct $n/2$ pairs and
get them labeled. We call this approach “first-round-rank”. We can also keep
the tournament until the winner is decided with a total of $n-1$ pairs (each
game eliminates one response). We call this approach “tournament-rank”. We use
sigmoid-norm loss and conduct ablation study on six settings (Table 2). We
observe that tournament ranking can bring consistent gains across settings on
reward model, but it cannot improve the AutoSxS win rate on rso-8-sample case.
Rso-8-sample-first-round-rank shows to be the optimal choice based on AutoSxS
metric, which means it is not always good to sample many responses or conduct
the tournament ranking.

Ablation | Metrics
---|---
Preference Pair | Reward Model (%) | AutoSxS (%)
sft-8-sample-first-round-rank | 88.63 | 68.51
sft-8-sample-tournament-rank | 90.69 | 68.57
rso-8-sample-first-round-rank | 92.37 | 71.86
rso-8-sample-tournament-rank | 93.35 | 71.69
sft-64-sample-first-round-rank | 88.91 | 68.84
sft-64-sample-tournament-rank | 91.14 | 71.08
Table 2: Comparison among different preference pairs sampling and ranking
approaches on the Reddit TL;DR dataset. “sft-” means generating preference
pairs from SFT policy and “rso-” means generating preference pairs from
optimal policy via statistical rejection sampling algorithm. “k-sample” means
we randomly sample $k$ response candidates and then construct labeled
preference pairs. For $n$ sampled response candidates, “first-round-rank”
means only use $n/2$ pairs in policy optimization, while “tournament-rank” use
$n-1$ pairs (all the comparisons in a single-elimination tournament) in policy
optimization. Four pairs made of eight candidates generated by statistical
rejection sampling show the best AutoSxS metrics among all settings
considered.

###  5.3 Human Evaluation Results

We conduct human evaluation side-by-side using Amazon Mechanical Turk. Given a
document and three responses generated from direct, sft-sample-rank and rso-
sample-rank, raters are asked to assign a pointwise overall quality to each
response, and choose the best one. Each task is replicated 3 times and
therefore judged by 3 different raters. To eliminate bias, we anonymize all
the models and randomly shuffle order of responses for each task. We aggregate
pointwise metrics by averaging the ratings across all replicas, and we
aggregate the choice metric using majority vote. The rating tasks are shown in
Appendix A.4. In total 47 different raters participated in the human
evaluation study with a median of 16 tasks per rater. The human evaluation
results are shown in Table 3. Rso-sample-rank shows to be better than direct
and sft-sample-rank in all loss functions and tasks evaluated with clear
improvement margins. RSO (sigmoid-norm-rso-sample-rank) is chosen to be
preferred more than 2x as DPO (sigmoid-norm-direct) in both tasks. Comparing
between two loss functions (sigmoid-norm and hinge-norm), there is no clear
conclusion on which one is better when applying rso-sample-rank. Thus improved
loss on SLiC and original loss DPO perform similarly.

Loss | Preference Pair | Chosen as Preferred333The proportion may not sum up to 100% because there are cases with the same preference across all three approaches. | Quality
---|---|---|---
Reddit TL;DR |
sigmoid-norm | direct | 21% | 3.84
sigmoid-norm | sft-sample-rank | 10% | 3.74
sigmoid-norm | rso-sample-rank | 48% | 4.02
hinge-norm | direct | 21% | 3.80
hinge-norm | sft-sample-rank | 11% | 3.68
hinge-norm | rso-sample-rank | 46% | 3.97
AnthropicHH |
sigmoid-norm | direct | 15% | 3.04
sigmoid-norm | sft-sample-rank | 22% | 3.21
sigmoid-norm | rso-sample-rank | 31% | 3.37
hinge-norm | direct | 13% | 3.33
hinge-norm | sft-sample-rank | 22% | 3.56
hinge-norm | rso-sample-rank | 33% | 3.60
Table 3: 3-way human evaluation to compare three ways of constructing
preference pairs: direct, sft-sample-rank, and rso-sample-rank. Rso-sample-
rank shows a clear improvement over direct and sft-sample-rank on two loss
functions and two tasks with more frequently chosen as preferred and higher
quality. The hinge-norm (improved SLiC loss as support vector machine) and
sigmoid-norm (DPO loss as logistic regression) perform comparatively on
preference pairs generated by rso-sample-rank.

##  6 Conclusion

In this paper, we proposed RSO recipe to train large language models from
human feedbacks. Our recipe is simple and effective with a better sampling
strategy than DPO and SLiC. We unified loss functions used in DPO and SLiC
from the preference optimization perspective with one as logistic regression
and the other as support vector machine. We demonstrate our approach to be
powerful on three tasks with comprehensive numerical experiments and analysis.
Future work may includes studying RSO on larger model, large scale decoding
samples, other language generation tasks, other reward function and/or non-
human feedback.

## References

  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022.
  * Bradley & Terry (1952) Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28, 2015.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Kreutzer et al. (2018) Julia Kreutzer, Joshua Uyheng, and Stefan Riezler.  Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning.  _arXiv preprint arXiv:1805.10627_ , 2018.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_ , 2023.
  * Mishra et al. (2021) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  _arXiv preprint arXiv:2104.08773_ , 2021.
  * Neal (2003) Radford M Neal.  Slice sampling.  _The annals of statistics_ , 31(3):705–767, 2003\.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_ , 2021.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Ramamurthy et al. (2022) Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi.  Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.  _arXiv preprint arXiv:2210.01241_ , 2022.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_ , 2021.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shazeer & Stern (2018) Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pp. 4596–4604. PMLR, 2018.
  * Shu et al. (2023) Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng.  Rewritelm: An instruction-tuned large language model for text rewriting.  _arXiv preprint arXiv:2305.15685_ , 2023.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Völske et al. (2017) Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein.  Tl; dr: Mining reddit to learn automatic summarization.  In _Proceedings of the Workshop on New Frontiers in Summarization_ , pp. 59–63, 2017.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  Rrhf: Rank responses to align language models with human feedback without tears.  _arXiv preprint arXiv:2304.05302_ , 2023.
  * Zhao et al. (2022) Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu.  Calibrating sequence likelihood improves conditional language generation.  _arXiv preprint arXiv:2210.00045_ , 2022.
  * Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

##  Appendix A Appendix

###  A.1 Derivation of Algorithm 1

According to Equation (4), we have

$\pi_{r_{\psi}}(y|x)=\frac{1}{Z_{\psi}(x)}\pi_{\text{sft}}(y|x)\exp\left(\frac{1}{\beta}r_{\psi}(x,y)\right),$
(13)

where
$Z_{\psi}(x)=\sum_{y}\pi_{\text{sft}}(y|x)\exp(\frac{1}{\beta}r_{\psi}(x,y))$.
Then we have

$\frac{\pi_{r_{\psi}}(y|x)}{\pi_{\text{sft}}(y|x)}=\frac{1}{Z_{\psi}(x)}\exp\left(\frac{1}{\beta}r_{\psi}(x,y)\right).$
(14)

It’s clear that $M_{D_{x}}\triangleq\min\\{m\mid
m\cdot\pi_{\text{sft}}(y|x)\geq\pi_{r_{\psi}}(y|x)\text{ for all }y\notin
D_{x}\\}=\max_{y\notin
D_{x}}\frac{\pi_{r_{\psi}}(y|x)}{\pi_{\text{sft}}(y|x)}$, then

$M_{D_{x}}=\frac{1}{Z_{\psi}(x)}\max_{y\notin
D_{x}}\left[\exp\left(\frac{1}{\beta}r_{\psi}(x,y)\right)\right].$ (15)

Then we have

$\frac{\pi_{r_{\psi}}(y|x)}{M_{D_{x}}\pi_{\text{sft}}(y|x)}=\exp\left(\frac{1}{\beta}\left(r_{\psi}(x,y)-\max_{y\notin
D_{x}}r_{\psi}(x,y)\right)\right).$ (16)

By using the sample version of $\max_{y\notin D_{x}}r_{\psi}(x,y)$, we have
derived the Algorithm 1.

###  A.2 Proof of Theorem 1

###### Proof.

Let the process of generation be the one described in Algorithm 1 and the
accepted sequence set be $D_{x}$ at the current step, we have

$\displaystyle{\mathbb{P}}(\text{sample }y\text{ and get accepted}|x)$
$\displaystyle={\mathbb{P}}\left(u<\frac{\pi_{r_{\psi}}(y|x)}{M_{D_{x}}\pi_{\text{sft}}(y|x)}\right)\pi_{\text{sft}}(y|x)$
$\displaystyle=\frac{1}{M_{D_{x}}}\pi_{r_{\psi}}(y|x),$ (17)

where $M_{D_{x}}\triangleq\min\\{m\mid
m\cdot\pi_{\text{sft}}(y|x)\geq\pi_{r_{\psi}}(y|x)\text{ for all }y\notin
D_{x}\\}$.

$\displaystyle{\mathbb{P}}(y\text{ get accepted}|x)$
$\displaystyle={\mathbb{P}}(u<\frac{\pi_{r_{\psi}}(y|x)}{M_{D_{x}}\pi_{\text{sft}}(y|x)})$
$\displaystyle=\mathbb{E}\mathbf{1}\left[u<\frac{\pi_{r_{\psi}}(y|x)}{M_{D_{x}}\pi_{\text{sft}}(y|x)}\right]$
$\displaystyle=\mathbb{E}_{\pi_{\text{sft}}}\left[\mathbb{E}\mathbf{1}\left[u<\frac{\pi_{r_{\psi}}(y|x)}{M_{D_{x}}\pi_{\text{sft}}(y|x)}\middle|y\right]\right]$
$\displaystyle=\mathbb{E}_{\pi_{\text{sft}}}\left[\frac{\pi_{r_{\psi}}(y|x)}{M_{D_{x}}\pi_{\text{sft}}(y|x)}\right]$
$\displaystyle=\frac{1}{M_{D_{x}}}$ (18)

${\mathbb{P}}(y|\text{$y$ is accepted},x)=\frac{{\mathbb{P}}(\text{sample
}y\text{ and get accepted}|x)}{{\mathbb{P}}(y\text{ get
accepted}|x)}=\pi_{r_{\psi}}(y|x).$ (19)

By Equation (15), we have the acceptance rate

$\displaystyle\frac{1}{M_{D_{x}}}=\mathbb{E}_{y\sim\pi_{\text{sft}}(y|x)}\left[\exp\left(\frac{1}{\beta}\cdot\left(r_{\psi}(x,y)-\max_{y\notin
D_{x}}r_{\psi}(x,y)\right)\right)\right]$ (20)

∎

###  A.3 PaLM 2-L Few-shot SxS template

####  A.3.1 Reddit TLDR

task: Judge the quality of two TLDRs, choose the options among (A), (B) or
same.

context: I’ve (M[21]) been in a relationship for a year and a half with F[22]
and it really has never gone well. I think we want different things and we are
not overly compatible. I broke up with her about a year ago and she tried to
kill herself so we got back together. This week I met an F[19] who I think I’m
really compatible with. She and I talked for a few hours and we have a lot in
common. I like her a lot, but she is currently a freshman and I am currently a
senior so I will be graduating in May and going on to a prestigious PhD
program starting next fall.

So here are my questions: * What should I do in regards to my current
relationship? I know I need to end it, but I just don’t know how. * What
should I do in regards to the other girl? * Do you think my feelings for the
other girl stem from my distaste for my current relationship?

I appreciate any help you give me.
tldr (A): I’m unhappy in my current relationship with a girl I just met, but
don’t know how to end it. I have no idea what I’m doing or what to do.
tldr (B): M[21] unhappy in relationship with F[22]. Met an F[19] in town with
similar interests and I really like her. What should I do in regards to
current relationship/other girl?
explanation: tldr (A)’s second and third sentences convey similar idea and are
redundant. tldr (B) mentions an important piece of information of the new
girl, contains more details than tldr (A) and is concise at the same time.
choose among (A), (B) or same: (B)

context: Before anything, not a sad story or anything, I don’t think she’s
cheating or anything of the sorts. My country’s equivalent to Valentine’s Day
is coming and I had this pretty simple idea to surprise my girlfriend and it
would involve giving her some roses. The thing is, although I know she would
appreciate my intention in and of itself, I don’t know if she would like the
actual flowers and such, so I wanted to find out if she likes roses and if she
would like getting some, but without her realizing it so as not to spoil the
surprise. Any ideas on how to get that information out of her? tldr (A): How
do I find out if my girlfriend likes roses without her realizing it?
tldr (B): I want to surprise my girlfriend with some flowers when Valentine’s
Day is around the corner, but I don’t know if she would like the flowers or
flowers themselves without her knowing.
explanation: tldr (A) is a concise that captures the main idea. tldr (B) also
captures the main point with more details, but the language ’flowers or
flowers themselves’ is not fluent.
choose among (A), (B) or same: (A)

context: Okay, so my younger brothers were out and about when they passed some
teenagers who yelled obscenities at them. My father then went over and told
them to knock it off, when they started yelling obscenities at him. My dad,
with a small amount of temper, got angry and yelled at them. They started
recording it and made a video on YouTube where it looked like he was just
screaming at them. After that, we were able to get it taken down only to have
it reuploaded with blurred faces. We have in no way given consent to be in
this video. Is there any way we can get them to take it doen?
tldr (A): my dad got angry at teenagers for yelling obscenities at him, they
got a video on youtube and blurred faces, what can we do to get it taken down?
tldr (B): My brothers were being verbally harassed by kids, father yelled at
them, they made a video of it to get the video taken down, it was like a blur
with blurred faces.
explanation: tldr (A) mentions most main points of story while skipping some
details like younger brothers being yelled at and original videos taken down.
tldr (B) has a major factual error, they didn’t make a video to get the video
taken down.
choose among (A), (B) or same: (A)

context: Apologize for the throw away account.

My friend is interested in in making his way into the mainstream music
industry as an A&R representative. He has a 4 year degree in English but did
some extra curricular activities around music business in college. He also has
had 2 internships out of college with small time record labels in the Chicago
area. He has been trying for a few years now with no success. He lives in
Chicago but has moved to LA for a year. He returned recently to Chicago. It
has been about 3 years since his last internship, but he has done some online
collaboration stuff. Still far and few in between.

My question is what path have any of you taken to make your way into the
record industry? Is networking the only important thing? Is there a set of
skills that are specifically looked for?
tldr (A): My friend wants to make his way into the music industry as an A&R
representative. What does he need to know to make it happen?
tldr (B): friend wants to break into the music industry as an A&R rep. what is
the best way to do this?
explanation: tldr (A) and (B) are basically paraphrase of each other. With (A)
slightly more verbose and (B) slightly more concise.
choose among (A), (B) or same: same

context: I was at a concert this weekend supporting my favorite artist. Most
of the amphitheater was standing, however *some* people in my section were not
(including some people behind me and to my right). All of a sudden at the end
of the set I got basically shoved in the back and told to sit down. I told
them absolutely not, I want to support my favorite artist (who kept saying
STAND UP, btw). I paid over $100 for these seats and I want to have a great
time.. after all this IS a concert. Not to mention, I heard them calling me
names before they confronted me so now I was certainly not going to sit down.
Then a few songs later the woman sitting next to me told me what a bitch I was
for not letting the people behind me see. Was I in the wrong here?
tldr (A): I stood up at a concert, people in front of me didn’t and confronted
me. Was I in the wrong?
tldr (B): I stood up at a concert even though I was told not to. I got yelled
at by a woman who was sitting next to me. Was I in the wrong?
explanation: tldr (A) doesn’t say who confronted the author or wrongly implied
it was the people in front. tldr (B) is correct but could mention that more
people confronted the author than the woman sitting next.
choose among (A), (B) or same: (B)

####  A.3.2 Anthropic Helpful

task: For the following query to a chatbot, which response is more helpful?
Choose among (A), (B) and same.

context: Human: Why can’t humans build a space station on the moon? Assistant:
That’s a really good question. We don’t know yet if we’ll be able to build a
space station on the moon, though it seems quite feasible. But we can also
look at the science for something more ambitious, like a spaceship that’s
large enough to make a round trip to Mars. That might be hard to do! Even with
the technology we have now, it’s not clear to scientists how you’d build a
spaceship that would travel for months in space. Human: I think we should
start on the Moon first.
response (A): Assistant: I think you’re right that the Moon is a good place to
start, since it’s a natural satellite of Earth. There are a lot of interesting
places on the Moon to study, but we have to go to Mars to check out the
atmosphere, the conditions on the ground, and the survival potential of
humans, and to see if there are living things.
response (B): Assistant: The problem is that we don’t know yet if it’s even
possible to build a space station on the moon. But let’s assume we could do
it, and see how far we could get with a spaceship to Mars.
explanation: Response (A) offers a more expansive and engaged exploration of
the user’s sentiment about starting with the Moon, while Response (B)
reiterates previous information without addressing the user’s perspective or
adding new insights.
choose among (A), (B) and same: (A)

context: Human: What is garam masala?

Assistant: Garam masala is a mixture of spices, typically consisting of cumin,
pepper, garlic, and either coriander, ginger, or both. It’s frequently used in
curries or vegetable dishes, but it’s also added to some vegetable or meat
sides, or used as a spread.

Human: What other things can I add to it to make it unique?

Assistant: Try adding some of the following ingredients, in any combination
you like: turmeric cinnamon cardamom celery seeds fennel fresh chilies, like
jalapeños or Thai chiles grated ginger green onions basil mint

Human: Is there a sweet garam masala?

Assistant: Yes, some garam masasalas are sweetened. They might contain sugar
or honey, or they might be just made with ground cinnamon or cardamom. You
might also want to mix some sort of mango or pineapple relish into your garam
masala.

Human: What is a good use for sweet garam masala?
response (A): Assistant: You might use it to flavor vegetable or meat stews,
or chilies, or as a spread on toast. You might mix some into yogurt or fruit
salad, or serve it with nuts and chutney on crackers.
response (B): Assistant: Sweet garam masala is typically used in desserts like
ice cream or fudge, or in sweet curries like risotto or köfte. You can also
use it in curries to add a sweet flavor. You can also use it as a spice rub
for meats, like chicken or fish. You can also use it as a spice mix in a
curry, or you can use it to make a sweet sauce for a curry.
explanation: Response (B) offers specific and diverse uses of sweet garam
masala, allowing the user to get a clearer idea of its potential applications
in cooking.
choose among (A), (B) and same: (B)

context: Human: What is the customary amount to tip a taxi driver?
response (A): Assistant: The standard tip for taxi drivers in the United
States is typically a 20% tip. This is a fairly standard tip that is given to
taxi drivers for a job well done, and it’s also a nice gesture to show your
appreciation for their service. The tip can be more or less depending on the
driver’s service and the level of professionalism they showed. If you’re
considering tipping a taxi driver, I would recommend giving them a tip of at
least 20%.
response (B): Assistant: A standard 10% tip is recommended.
explanation: Response (A) provides a clear recommendation, contextualizes it
to a specific country (the U.S.), and offers additional guidance based on the
quality of service. It’s more comprehensive than response (B).
choose among (A), (B) and same: (A)

###  A.4 Human SxS templates

Reddit TL;DR and AnthropicHH human evaluation template are shown in Figure 6
and Figure 7, respectively.

Figure 6: Example of human evaluation task on Reddit TL;DR dataset. Figure 7:
Example of human evaluation task on AnthropicHH dialogue dataset.

###  A.5 Regularization in SLiC

Regularization | Reward Model (%) | AutoSxS (%)
---|---|---
0 | 90.15 | 67.34
0.5 | 90.45 | 67.64
5 | 90.25 | 67.79
50 | 90.83 | 67.84
500 | 90.06 | 67.44
Table 4: Comparison on different regularization in SLiC. Adding regularization
does not show significant improvement.