# On Diversified Preferences of Large Language Model Alignment

Dun Zeng  Yong Dai  Pengyu Cheng  Tianhao Hu  Wanshun Chen  Nan Du  Zenglin Xu

###### Abstract

Aligning large language models (LLMs) with human preferences has been
recognized as the key to improving LLMs’ interaction quality. However, in this
pluralistic world, human preferences can be diversified by people’s different
tastes, which hinders the effectiveness of LLM alignment methods. In this
paper, we provide the first quantitative analysis to verify the existence of
diversified preferences in commonly used human feedback datasets. To mitigate
the alignment ineffectiveness caused by diversified preferences, we propose a
novel Multi-Objective Reward learning method (MORE), which can automatically
adjust the learning gradients across different preference data sources. In
experiments, we evaluate MORE with the Pythia-1.4B model on five mixed human
preference datasets, on which our method achieves superior performance
compared with other baselines in terms of preference accuracy and prediction
calibration.

Machine Learning, ICML

##  1 Introduction

Large language models (LLMs), such as ChatGPT (OpenAI, 2023) and LLaMA
(Touvron et al., 2023a, b), have significantly accelerated the development
process toward artificial general intelligence (AGI). Among the key factors
for such great achievement, the alignment technique, which finetunes LLMs with
human feedback (Christiano et al., 2017), has played an essential role in
training LLMs’ responses to follow human values (e.g., helpfulness and
harmlessness) (Askell et al., 2021). Among the LLM alignment algorithms,
reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) has
become the mainstream solution, which first learns a reward model (RM)
representing human preferences and then updates LLMs via the proximal policy
optimization (PPO) (Schulman et al., 2017) toward generating responses with
higher RM scores. However, RLHF has recently been challenged for its
cumbersome computational complexity and training instability. To avoid the
reinforcement learning scheme of RLHF, alternative alignment methods have been
sequentially proposed, such as RAFT (Dong et al., 2023b), DPO (Rafailov et
al., 2023), RRHF (Yuan et al., 2023), and APO (Cheng et al., 2023b).

Figure 1: Illustration of Diversified Preferences: Training RM on one
preference data source (e.g., RM${}_{\text{Helpful}}$) can damage its original
performance (RM${}_{\text{Raw}}$) on the other sets.

The performance of these LLM alignment methods highly depends on the quality
of human preference data $(x,y_{w},y_{l})$, where $x$ is the input query to
the LLM, and response $y_{w}$ is preferred to response $y_{l}$ under the human
annotation (Ouyang et al., 2022). Ideally, the preference data sets should
uniformly be helpful, harmless, benevolent, and unbiased to guide the LLM
alignment. However, in real-world scenarios, individuals can have diversified
preferences on the same topic based on their different experiences,
educational backgrounds, religions, and cultures. Even for the same person,
his or her expected model answer to a particular question can vary depending
on different scenarios (Cheng et al., 2023a). This causes a serious
inconvenience in preference data annotation and collection processes, for
different annotators can make considerable disagreements on the selection of
better responses (Bai et al., 2022a). The annotation disagreement
significantly hinders the effectiveness of alignment methods (Davani et al.,
2022; Wan et al., 2023). Hence, how to resolve divergences and conflicts in
preference data is vitally important, yet it has not received adequate
attention and remained unexplored in the LLM alignment domain.

To further identify the diversified preferences quantitatively, we select
several commonly used human feedback datasets, train an RM on each of them,
and then test the performance on the other sets. The datasets we used are
Summarize (Stiennon et al., 2020), Webgpt (Nakano et al., 2021a),
Helpful&Harmless (Bai et al., 2022a), and OASST1 (Köpf et al., 2023). The
correspondingly learned RM${}_{\text{Summ}}$, RM${}_{\text{Webgpt}}$,
RM${}_{\text{Helpful}}$, RM${}_{\text{Harmless}}$ and RM${}_{\text{Oasst1}}$
are fine-tuned on a Pythia-1.4B (Biderman et al., 2023) base
RM${}_{\text{Raw}}$. We plot the preference accuracy of each RM on all of the
five testing sets in Figure 1. We observe that fine-tuning a reward model on a
single preference dataset (e.g., RM${}_{\text{Helpful}}$) damages its original
performance (performance of RM${}_{\text{Raw}}$) on the other preference sets.
This indicates that different human preference datasets have different data
distributions, which even include preference conflicts (Cheng et al., 2023a).
Therefore, directly utilizing human feedback from different sources can hinder
the LLM alignment processes because of the preference divergence.

To address the problem of diversified preferences, inspired by multi-objective
optimization methods (Sener & Koltun, 2018; Zeng et al., 2023), we regard RMs
as a shared reward additionally with a customized reward drift. The shared
reward represents the shared human preferences (or general preferences) and
the reward drift contains individual or domain-specific preference information
(Cheng et al., 2023a). To capture the shared (general) preference information,
we introduce a Multi-Objective Reward training scheme (MORE), which treats
feedback data from different sources as different learning objectives, then
re-weight these objectives by minimizing the mean gradient of their
corresponding reward drifts. With multi-objective reward learning, RMs can
capture a broader range of preferences and mitigate the impact of preference
conflicts. The main contributions of this paper are:

  * •

We conduct a comprehensive analysis of how diversified preferences affect
reward modeling in LLM alignment. We find that diversified preferences can
cause high variances in the RM output distributions, which negatively impact
the performance of LLM alignment.

  * •

We model RMs as a shared reward plus reward drift and propose a Multi-
Objective Reward (MORE) training scheme, which makes self-adaption to the RM
learning gradient by minimizing the reward drifts.

  * •

We empirically verified the effectiveness of MORE with Pythia-1.4B (Biderman
et al., 2023) RMs on multiple preference datasets, where MORE outperforms
other baseline training strategies for multi-source preference learning and
alignment.

Figure 2: Multi-objective reward model training scheme (MORE), which consists
of four steps: (1) collect a diversified batch of data from the mixed dataset;
(2) calculate the RM gradient for each preference source; (3) minimize the
reward drift to determine the scalar
$(\lambda_{1},\lambda_{2},\dots,\lambda_{K})$ for MORE loss; (4) update the RM
with the re-weighted RM loss.

##  2 Preliminary

#### Large Language Model Alignment

Parameterized by $\theta$, a reward model (RM) is a mapping
$r_{\theta}:\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R}$, which provides
a real-valued reward score $r_{\theta}(x,y)$ evaluating a textual response
$y=(y_{1},y_{2},\dots,y_{M})\in\mathcal{Y}$ corresponding to an input prompt
$x=(x_{1},x_{2},\dots,x_{N})\in\mathcal{X}$. Given a sample
$(x,y_{w},y_{l})\sim\mathcal{D}$ from a preference dataset $\mathcal{D}$,
$r_{\theta}$ is expected to provide a preference score with
$r_{\theta}(x,y_{w})>r_{\theta}(x,y_{l})$, representing the response $y_{w}$
is preferred. Following the Bradley-Terry model (David, 1963), the RM learning
objective on on the preference dataset $\mathcal{D}=\\{(x,y_{w},y_{l})\\}$ is
defined as:

$\textstyle\mathcal{L}_{\text{rank}}(\theta;\mathcal{D})=-\mathbb{E}_{\mathcal{D}}\log(\sigma\left(r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})\right))$
(1)

where $\sigma(\cdot)$ is the Sigmoid function. With a well-learned reward
$r_{\theta}(x,y)$, LLM alignment optimizes the generation policy
$\pi_{\phi}(y|x)$ by maximizing the expected reward value:

$\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(y|x)}[r_{\theta}(x,y)]-\beta\mathbb{D}_{\text{KL}}[\pi(y|x)\|\pi_{\text{ref}}(y|x)],$
(2)

where $\mathbb{D}_{\text{KL}}[\pi(y|x)\|\pi_{\text{ref}}(y|x)]$ is the KL
divergence regularizer between current policy $\pi_{\phi}$ and a reference
$\pi_{\text{ref}}$, preventing the optimization from instability and
degeneration.

#### Domain Generalization

Machine learning methods suffer from performance degeneration when the source
domain data and the target domain data follow different distributions, which
has been recognized as the domain shift problem (Pan & Yang, 2009; Csurka,
2017; Wang et al., 2021). To address this problem, domain generalization is
proposed to minimize the domain shift across domains. In this direction,
existing methods aim to learn the domain invariant representation to reduce
the discrepancy between representations of multiple source domains (Zhou et
al., 2022a).

#### Multi-objective Optimization

Multi-objective Optimization (MOO) (Gunantara, 2018) is a branch of methods
addressing learning problems involving multiple conflicting objectives. In
real-world scenarios, it commonly encounters situations where multiple
objectives need to be considered simultaneously, often with trade-offs between
them. In the practice of machine learning, most MOO methods (Sener & Koltun,
2018; Zeng et al., 2023) apply linear scalarization (Barrett & Narayanan,
2008) to merge multiple objectives into one, and then automatically adjust the
objective coefficients to balance the conflicts among different tasks.

##  3 Empirical Study of Diversified Preferences

We start with an empirical analysis of diversified preferences in reward
modeling on multiple sources
$\mathcal{D}=\\{\mathcal{D}_{1},\dots,\mathcal{D}_{K}\\}$, where each data
source $D_{k}$ contains the preference comparison pairs from different tasks
(Dong et al., 2023a), domains (Cheng et al., 2023a), or individuals (Bai et
al., 2022a). As mentioned in Section 1, we selected Summarize (Stiennon et
al., 2020), Webgpt (Nakano et al., 2021a), Helpful&Harmless (Bai et al.,
2022a), and OASST1 (Köpf et al., 2023) as the different preference sources to
empirical analysis the phenomena of diversified preferences. We use
Pythia-1.4B (Biderman et al., 2023) as the RM base, and fine-tuned RMs with
comparisons from each source. The learning rate is set to $5e^{-5}$. All RMs
are trained with batch size 64 in one epoch.

The reward distribution of different reward models is diversified on the same
dataset. We summarize the reward difference (the reward difference is the
difference in reward value between the winning sample and the losing sample
given by a reward model) provided by RMs, as shown in Figure 3. Additional
results on other test sets are provided in Appendix C. Compared with the
results of raw model $\text{RM}_{\text{Raw}}$, we observe that training on
different datasets results in diverse reward values (right plot) and
distribution shift (left plot). Specifically, the reward value distribution of
$\text{RM}_{\text{Harmless}}$ shifts from the $\text{RM}_{\text{Raw}}$ in a
certain degree. While the reward value distributions of
$\text{RM}_{\text{Helpful}}$, $\text{RM}_{\text{Webgpt}}$,
$\text{RM}_{\text{Oasst1}}$ and $\text{RM}_{\text{Summ.}}$ shifts to the a
different direction. Moreover, despite of the distribution of
$\text{RM}_{\text{Helpful}}$, $\text{RM}_{\text{Webgpt}}$,
$\text{RM}_{\text{Oasst1}}$ and $\text{RM}_{\text{Summ.}}$ are similar, the
mean-variance of their reward values are quite different.

Furthermore, when considering the accuracy gains illustrated in Figure 1, the
observed shift in reward distribution indicates that the learned reward values
from preference datasets are diversified. To effectively capture the shared
reward values across these diversified preferences, it becomes necessary to
formulate a new problem approach for reward modeling on diverse preference
datasets.

Figure 3: Statistics of reward values on Harmless test set. Left: the
distribution of reward differences. The box plot depicts the statistics of
reward differences on the test set. Right: The solid box indicates the
statistics of reward values on correct rewarded samples, while the hollow box
indicates the wrong rewarded samples.

##  4 Multi-objective Reward Learning

###  4.1 Preference Diversity as Reward Drift

We denote $r^{*}(\cdot,\cdot)$ as the shared reward function, which (ideally)
provides reward values reflecting the shared values among people. As the
collected human-feedback datasets are limited and implicitly biased, they
usually can not cover the whole shared value. Therefore, training an RM
$r_{\theta}$ on a limited preference dataset can be viewed as drifting from an
optimal reward. We form the reward drift of a reward model parameterized by
$\theta$ from a data level:

$\textstyle r_{\theta}(x,y)=r_{\theta}^{*}(x,y)+\tilde{r}_{\theta}(x,y)$ (3)

where $\tilde{r}_{\theta}(x,y)$ is the reward drift learned from the given
dataset. Then, we investigate the vanilla ranking loss for reward modeling.
Substituting reward function in equation 1 with the drifted form equation 3,
we have

$\displaystyle\mathcal{L}_{\text{rank}}(\theta;\mathcal{D})=-$
$\displaystyle\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}[\log(\sigma(r_{\theta}^{*}(x,y_{w})-r_{\theta}^{*}(x,y_{l})$
$\displaystyle\quad+\tilde{r}_{\theta}(x,y_{w})-\tilde{r}_{\theta}(x,y_{l})))].$
(4)

Hence, updating the RM to minimize the rank loss will enlarge the reward
differences (input of the Sigmoid function). Simultaneously, the reward drift
is also enlarged.

###  4.2 Reward Drift Modeling

We discuss the unique form of learning general preferences from diversified
data sources in this section. Firstly, we justify the reward agreement and
disagreement in preference modeling:

######  Definition 1 (Reward agreement and disagreement).

Given a set of diverse preference datasets
$\\{\mathcal{D}_{1},\dots,\mathcal{D}_{K}\\}$, we can obtain RM sets
$\\{\theta_{1},\dots,\theta_{K}\\}$, which are trained solely on every single
dataset. Then, given an comparison sample
$(x,y_{w},y_{l})\in\mathcal{X}\times\mathcal{Y}$, we say this is an agreed
sample if $r_{\theta_{j}}(x,y_{w})>r_{\theta_{j}}(x,y_{l})$ for all $j\in[K]$.
Otherwise, if there is a pair $(i,j)$ such that
$r_{\theta_{i}}(x,y_{w})>r_{\theta_{i}}(x,y_{l})$ and
$r_{\theta_{j}}(x,y_{w})<r_{\theta_{j}}(x,y_{l})$, this sample is disagreed.

Compared with the previous discussion (Davani et al., 2022; Basile et al.,
2021; Wang et al., 2023a; Zhao et al., 2023a) in label disagreement, we are
focusing on the rewards disagreement across the RMs.

Letting $\theta$ be the RM trained on mixed diverse datasets
$\mathcal{D}=\\{\mathcal{D}_{1},\dots,\mathcal{D}_{K}\\}$, the
$r_{\theta}(x,y)$ can be viewed as a multi-task learner with shared parameters
(Sener & Koltun, 2018). Then, the reward value provided by $r_{\theta}(x,y)$
can be decomposed into voting format weighted by an implicit $\lambda$:

$\textstyle
r_{\theta}(x,y)=\sum_{i=1}^{K}\lambda_{i}r_{\theta_{i}}(x,y)=r_{\theta}^{*}(x,y)+\sum_{i=1}^{K}\lambda_{i}\tilde{r}_{\theta_{i}}(x,y)$
(5)

where the shared reward $r^{*}_{\theta}(\cdot,\cdot)$ is the same with
arbitrary $\lambda$, and $\tilde{r}_{\theta_{i}}(\cdot,\cdot)$ is the reward
drift. We interpret that the $\tilde{r}_{\theta_{i}}(\cdot,\cdot)$ is provided
by subset of parameters $\theta_{i}$, representing the preferences from the
$i$-th dataset $\mathcal{D}_{i}$. For example, the $\theta$ can be implemented
as an ensemble model, where $\\{{\theta_{i}}\\},i\in[N]$ is the base models.
Therefore, it is natural to adjust the weight $\lambda$ in an ensemble manner
(Coste et al., 2023; Jang et al., 2023; Touvron et al., 2023a; Eisenstein et
al., 2023) to mitigate the reward drift such that
$\min\sum_{i=1}^{K}\lambda_{i}\tilde{r}_{\theta_{i}}(x,y)$. Moreover, if the
datasets are extremely diverse, the data samples are very likely to be
disagreed by $K$ diverse preference by Definition 1. Hence, there always
exists an combination $\lambda$ such that
$\sum_{i=1}^{K}\lambda_{i}\tilde{r}_{\theta_{i}}(x,y)=0$.

###  4.3 Training Scheme: MORE

Compared with an ensemble of a set of RMs (Jang et al., 2023; Eisenstein et
al., 2023), we are more interested in training a single RM that learns the
shared preference. Our idea is to reduce the model update on reward drift
during RM training via linear scalarization (Barrett & Narayanan, 2008).

MORE loss function. Inspired by previous analysis, finding proper weights
$\lambda$ for mitigating reward drifts is a feasible way. Analogously, we
propose training an RM to capture the shared preference distribution across
multiple datasets with the training objective:

$\displaystyle\textstyle\mathcal{L}_{\text{MORE}}(\theta;\mathcal{D})=\sum_{i=1}^{K}\lambda_{i}\mathcal{L}_{\text{rank}}(\theta,\mathcal{D}_{i}),$
(6)

where $\sum_{i}^{K}\lambda_{i}=1$. Compared with vanilla ranking loss in
Equation 1, the above loss additionally focuses on the combination relation
across preferences. It suggests training a better RM via adjusting training
weights $\lambda$. Moreover, this formulation also covers typical training
strategies. For example, directly mixing diverse preference datasets
$\mathcal{D}=\\{\mathcal{D}_{1},\dots,\mathcal{D}_{K}\\}$ and training a RM
induces $\lambda_{i}=\frac{|\mathcal{D}_{i}|}{|\mathcal{D}|}$ (McMahan et al.,
2017). Therefore, if the number of data samples from a single preference is
greatly larger than other preferences, the RM is likely to drift to the
preference with more samples. Excluding data quantity, the weight is also
decided by the quality of data samples in the training process (Katharopoulos
& Fleuret, 2018).

#### Computing $\lambda$

We propose to use adaptive weighting methods to balance the bias across
different domains and adjust the reward modeling process. Straightforwardly,
the mitigation task in Equation 9 can be efficiently solved by the Frank-Wolfe
solver (Jaggi, 2013; Sener & Koltun, 2018; Zhou et al., 2022b; Zeng et al.,
2023), while the computing cost for solving it is proportional to the size of
parameters $\theta$. To avoid expensive computation, we only utilize gradients
of the reward head $\theta_{\text{rm}}\in\mathcal{R}^{h}$ from each
preference. In detail, we obtain the hidden states
$z=r_{\theta_{\text{lm}}}(x)$ before the reward head and compute the gradient
of the reward head solely with data $(z,y_{w},y_{l})$. In summary, the
$\lambda$ is computed by:

$\textstyle\lambda=\arg\min_{\lambda}\left\|\sum_{i=1}^{K}\lambda_{i}g_{i}\right\|^{2},\
\sum_{i=1}^{K}\lambda_{i}=1,\lambda_{i}\geq 0,$ (7)

where $g_{i}$ is a momentum gradient related to
$\nabla\mathcal{L}_{\text{rank}}(\theta_{\text{rm}})$. During the training
process $t\in[T]$, the gradients is marked with
$\\{g_{1}^{t},\dots,g_{K}^{t}\\},t\in[T]$. Moreover, gradients of the reward
head are cached and updated in a momentum manner with $\alpha$:

$\textstyle
g_{i}^{t}=(1-\alpha)g_{i}^{t-1}+\alpha\frac{1}{|\mathcal{B}_{i}|}\sum_{\mathcal{B}_{i}}\nabla_{\theta_{\text{rm}}}\mathcal{L}_{\text{rank}}(\theta_{\text{rm}};\mathcal{B}_{i}).$
(8)

The cached gradient $g_{i}^{t}$ preserves the history updating information. It
stabilizes the results $\lambda^{t}$ by involving the information from the
past data samples (Zeng et al., 2023).

#### Outline

The MORE loss requires simple modification on batch data sampling and
additional computing $\lambda$. In detail, the pipeline of MORE is given in
Figure 2 and described as follows: 1) Sample a diverse batch data
$\mathcal{B}=\\{\mathcal{B}_{i}\\}_{i=1}^{K}$,
$\mathcal{B}_{i}=\\{x,y_{w},y_{l}\\}_{b=1}^{B}$ and input the batch data
forward the RM and obtain the hidden states $\\{z_{i}\\}_{b=1}^{B}$ before the
reward head $\theta_{\text{rm}}$. 2) Compute the gradient of reward with data
$\\{z,y_{w},y_{l}\\}$ and update the cache $\\{g_{i}^{t}\\}$ following
equation 8. 3) Compute the weights $\lambda$ by equation 7. Finally, we
substitute the loss weights in equation 6 as the final loss for the optimizer
to conduct backward and model updating. This procedure mitigates the potential
reward drift from the overall reward.

Theoretical explanation The MORE loss minimizes the ranking loss after solving
a reward drift mitigation task. Let batch data
$\mathcal{B}=\\{x^{(b)},y_{w}^{(b)},y_{l}^{(b)}\\}_{b=1}^{B}\sim\mathcal{D}$
be the sampled batch data from diverse datasets. Furthermore,
$\mathcal{B}_{i}\sim\mathcal{D}_{i}\subset\mathcal{B},\forall i\in[K]$ is the
subset of batch data from the $i$-th preference dataset. We have the gradient
of MORE loss:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{MORE}}(\theta;\mathcal{B})=\sum_{b=1}^{B}\left[-\nabla_{\theta}\log(\sigma(r_{\theta}^{*}(x^{(b)},y_{w}^{(b)}-y_{l}^{(b)})))\right]$
(9)
$\displaystyle\quad+K\underbrace{\min\sum_{i=1}^{K}\lambda_{i}\sum_{j=1}^{|\mathcal{B}_{i}|}\left[-\nabla_{\theta}\log(\sigma(\tilde{r}_{\theta}(x^{(j)},y_{w}^{(j)}-y_{l}^{(j)})))\right]}_{\text{Reward
Drift Mitigation Task}},$

where we let
$r_{\theta}(x,y_{w}-y_{l})=r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})$ and adjust
$\lambda$ (equation 7) to minimize the effect of reward drifts. Furthermore,
$\mathcal{L}_{\text{MORE}}$ shares the same magnitude of vanilla loss function
$\mathcal{L}_{\text{rank}}$ in expectation over the whole training data set.
The point can be justified by comparing it with vanilla ranking loss as shown
in Appendix A. The main difference falls in the loss of the mini-batch
descending process.

Table 1: Reward model performance on diverse datasets. ∗Lower is better.

Training | Testing Dataset (Acc %) | Metrics
---|---|---
Dataset | Policy | Helpful | Harmless | Oasst1 | Webgpt | Summ. | Avg. Acc | W.Avg. Acc | ECEa∗
- | Raw | 49.78 | 47.18 | 51.15 | 49.84 | 49.88 | 49.56 | 49.47 | 0.239
Helpful | - | 66.75 | 35.89 | 62.85 | 60.06 | 53.34 | 55.77 | 55.68 | 0.179
Harmless | - | 35.67 | 67.21 | 43.10 | 38.41 | 47.18 | 46.31 | 47.06 | 0.296
Oasst1 | - | 59.09 | 43.29 | 66.61 | 66.31 | 55.82 | 58.22 | 57.47 | 0.119
Webgpt | - | 60.96 | 41.13 | 64.00 | 67.37 | 49.11 | 56.51 | 55.32 | 0.128
Summ. | - | 60.07 | 42.56 | 58.68 | 60.36 | 58.44 | 56.02 | 56.18 | 0.206
ALL | Ensemble | 60.92 | 44.89 | 65.83 | 67.83 | 56.82 | 59.26 | 56.88 | 0.139
ALL | Hybrid | 57.14 | 52.63 | 66.30 | 66.15 | 58.25 | 60.09 | 59.60 | 0.078
ALL | Sequential | 57.95 | 53.71 | 66.71 | 67.68 | 57.07 | 60.62 | 60.36 | 0.084
ALL | MORE | 58.46 | 61.85 | 65.88 | 66.31 | 59.28 | 62.35 | 61.68 | 0.074

##  5 Experiments

In this section, we conduct an empirical evaluation of MORE for training RMs
on diverse preference datasets. Firstly, we provide a detailed description of
our experimental setup in Section 5.1. Then, we demonstrate the performance
enhancement achieved by employing MORE in contrast to traditional RM training
policies in Section 5.2. Next, we examine the distribution of reward values
generated by the trained RMs. Through analysis of reward values, we
demonstrate how the diverse preference dataset affects reward modeling.

###  5.1 Experiment Setup

#### Datasets

The human preference datasets we used include Helpful&Harmless (Bai et al.,
2022a), OASST1 (Köpf et al., 2023), Webgpt (Nakano et al., 2021a), and
Summarize (Stiennon et al., 2020). To avoid the preference unbalance from the
size of training sets, we randomly sample 17,106 samples from Helpful,
Harmless, OASST1 and Summarize training sets to align with the Webgpt dataset.
Besides, we note the unbalance of training data can be addressed by resampling
methods (Estabrooks et al., 2004). The corresponding datasets contain (2,352,
2,312, 957, 901, 3,620) test samples respectively.

#### Baselines

We compare our method with conventional fine-tuning strategies (Dong et al.,
2023a) in the training language models, which include sequential training and
hybrid training. We note that hybrid training scheme is essentially a special
case of MORE with a static $\lambda=\frac{1}{K}$. Moreover, the sequential
training scheme only samples a batch of data from single preference datasets
during training. We also compare with the naive ensemble, which merely
averages the reward values of independently trained rewards following equation
5.

#### Evaluation metric

We use the preference accuracy on test datasets for each domain. If an RM
$r(x,y_{w})>r(x,y_{l})$ for a test sample $(x,y_{w},y_{l})$, we denote it as a
correct prediction. The preference accuracy is then computed as the proportion
of correct predictions within all testing response pairs. However, preference
accuracy only provides pairwise comparisons of responses and does not reflect
the degree of preference for each response. Following Bai et al. (2022a);
Cheng et al. (2023b), we examine the probability calibration to test if the
learned RMs accurately represent the human preference distribution. This is
measured by the Expected Calibration Error (ECE) (Naeini et al., 2015; Zhu et
al., 2023).

#### Training setup

Due to the limitation of computational resources, we only select Pythia-1.4B
(Biderman et al., 2023) as the LM base for RM training. Experiments with a
large scale of the base model are in progress. We use the last token embedding
of the output hidden states as the pooled hidden representation, then add one
linear layer (RM head) with the scale-value output on it to predict reward
scores. All RM training batch size is set to 64. The max input sequence length
is 512. All RMs are fine-tuned with one epoch. We use optimizer AdamW
(Loshchilov & Hutter, 2017) with learning rate $5e^{-5}$. We set $\alpha=0.5$
for MORE.

Table 2: Sequence of Harmless dataset potentially affects the performance.
Seq.@$i$ indicates the Harmless is $i$-th utilized dataset in sequential
training.

Training | Testing Dataset (Acc %) | Metrics
---|---|---
Dataset | Policy | Helpful | Harmless | Oasst1 | Webgpt | Summ. | Avg. Acc | W.Avg. Acc | ECE
ALL | Seq.@1 | 56.50 | 59.64 | 65.88 | 65.54 | 59.22 | 61.35 | 61.06 | 0.107
ALL | Seq.@2 | 58.80 | 59.29 | 66.40 | 67.22 | 58.01 | 61.94 | 61.64 | 0.075
ALL | Seq.@3 | 58.24 | 53.20 | 64.31 | 67.22 | 58.34 | 60.26 | 60.00 | 0.087
ALL | Seq.@4 | 57.95 | 53.71 | 66.71 | 67.68 | 57.07 | 60.62 | 60.36 | 0.084
ALL | Seq.@5 | 60.45 | 60.51 | 67.03 | 66.31 | 58.06 | 62.46 | 62.20 | 0.076
ALL | MORE | 58.46 | 61.85 | 65.88 | 66.31 | 59.28 | 62.35 | 61.68 | 0.074

Figure 5: Statistics of reward values on testsets provided by the RM trained
with different policies on all mixed datasets. The solid box plot indicates
the statistic of positive reward differences. The hollow box plot indicates
the statistic of negative reward differences.

###  5.2 Reward Modeling on Multiple Preference Datasets

This section evaluates the performance of the RM trained on diverse preference
datasets. The results are given in Table 1. Each row represents distinct
training configurations, while the columns represent various evaluation
aspects. The term “Avg.” denotes the arithmetic mean of accuracy across all
test domains, while “W.Avg.” signifies the weighted average of accuracy,
considering the size of the respective testsets.

#### Main performance

The MORE achieves better testing accuracy and lower ECE values, indicating
that the policy provides an accurate reward value on the test. In the last
four rows of the table, we train an RM on mixed datasets with different
policies. The ensemble, hybrid, and sequential policies show similar test
accuracy on single domains and average accuracy. However, the Harmless
dataset, which has different preferences compared to the other four datasets,
experiences the largest accuracy degeneration with these policies. In
contrast, MORE exhibits the lowest performance degeneration on the Harmless
test.

#### Agreement & disagreement analysis

The agreement or disagreement of preferences can be reflected in the testing
accuracy. These findings highlight the importance of identifying and
understanding the agreement or disagreement across collected datasets in
practical applications. By comparing the experimental results of the first six
rows, we can observe that when the model is trained on datasets from other
domains, the test accuracy on the Harmless dataset decreases. This indicates a
disagreement in preferences between the Harmless dataset and the other
datasets.

In contrast, we observe a phenomenon of agreement when training the RM on one
domain, which leads to an increase in test accuracy on the other preferences.
For example, training the RM on any of the preference datasets such as
Helpful, Oasst1, Webgpt, or Summarize will increase the test accuracy on the
other preferences, even if the model has never seen the preference data
samples.

#### Sequential training analysis

We also find that sequential training is sensitive to the sequence of
preference datasets, as shown in Table 2. When adjusting the position of the
Harmless dataset while maintaining a fixed sequence for the other datasets
(Helpful, Oasst1, Webgpt, Summarize), we observe that sequential training
slightly outperforms MORE in terms of testing accuracy when the Harmless
dataset is placed at the end. However, MORE performs better on the ECE metric.
Therefore, while sequential training has the potential to achieve comparable
performance with MORE, it requires additional tuning efforts to adjust the
dataset sequence, which can be troublesome when dealing with large and diverse
datasets.

#### Why sequential training works better when Harmless at the last?

The training performance is influenced by the proportion of agreement and
disagreement data. Previous literature (Touvron et al., 2023a; Bai et al.,
2022b) has shown that there is a disagreement between the Helpful and Harmless
datasets. Additionally, our observations indicate that the Oasst1, Webgpt, and
Summarize datasets share distributional similarities with Helpful (Figure 3)
and exhibit similar testing accuracy (Table 1). As a result, the Harmless
dataset disagrees with the other datasets, and its proportion in the mixed
datasets is relatively small. Due to the smaller proportion of Harmless data,
the model updating process allocates less attention to this dataset compared
to the others. Sequential training can achieve better performance by adjusting
the sequence of datasets. For instance, the recently seen samples are less
likely to be forgotten (Wang et al., 2023b). In contrast, MORE successfully
enhances performance by adaptively re-weighting losses of different
preferences during training using Equation 6. MORE automatically captures the
agreement and disagreement through Equation 9 and balances the model updating
process accordingly.

###  5.3 Analysis on Diversified Reward Distributions

Reward accuracy and ECE only measure the quality of reward values provided by
RMs. How preference diversity plays into the reward model has not been
discussed. This section presents our analysis on the statistics of reward
values, highlighting the impact of diverse preferences on reward models.

#### Experiment setup

We first partition the testset into an “agreement testset” and a “disagreement
testset”. In detail, we let
$\\{\theta_{\text{helpful}},\theta_{\text{harmless}},\theta_{\text{oasst}},\theta_{\text{webgpt}},\theta_{\text{summ.}}\\}$
represent a preference model set. We utilize each preference model in the set
to infer every data sample from the testset. We obtain the agreement testset
which comprises data samples that agreed by the preference model set following
Definition 1. The remaining samples form the disagreement testset. We analyze
statistics and distribution of the reward difference (i.e.,
$r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})$) in reward values. It is worth
noting that a positive reward difference
$r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})>0$ indicates that the model
accurately assigns a higher reward value to the winning sample.

#### Analysis of reward differences on each preference

Our findings indicate that the reward value distributions of different reward
models are generally distinct. When examining the RM trained on different
single preference datasets, we observe a significant diversity in reward
values. In Figure LABEL:fig:reward_dist, we present the reward difference
values of the first 50 test samples from the Harmless dataset as an example
(Figure LABEL:fig:reward_a). Notably, even on the agreed samples, the baseline
model provides different rewards. This observation suggests that the
preference distribution learned from a single dataset by the reward model is
highly diverse compared to others. Furthermore, Figure LABEL:fig:reward_b
provides statistics of the reward difference across the entire test set,
further supporting our statements regarding the diversity of reward values.

In brief conclusion, despite presenting all the datasets to align with human
preferences, the reward models trained on each dataset produce diverse results
for the same data samples. This phenomenon suggests that the human preferences
captured in these datasets are diverse from each other. Additionally, the
difficulty of training the reward model on different preference datasets
varies. Furthermore, even when a reward model correctly assigns rewards to a
pair of comparison samples, the reward values can still exhibit significant
differences. These observations highlight the need for further research on the
task of reward modeling and the distribution of reward values. Such studies
may help improve the accuracy of reward models in accurately capturing the
degree of preferences.

#### Analysis of reward difference on training policies

MORE prevents the RM from outputting extreme reward values and maintains the
reward accuracy (shown in Table 1). Specifically, we observe the reward values
provided by RMs trained with different policies, as depicted in Figure 5.
Compared with the sequential training scheme, MORE outputs reward values and
variance with lower mean and variance. Connecting with our methodology, MORE
mitigated the reward drifting. Hence, it tends to output a lower absolute
reward signal as more accurate reward values according to ECE in Table 1.
Furthermore, the hybrid training scheme also yields lower reward values
compared to MORE. However, the reward accuracy is poorer in this case, which
indicates that compressing the mean and variance of reward differences may
degenerate the performance. These results suggest that the too-high or low
reward differences may not be good in testing metrics. In contrast, MORE
generates appropriate reward values, thereby enhancing the quality of reward
modeling.

##  6 Discussion

We have proposed a simple yet effective method for RM to enhance the
performance of learned RMs using diverse preference datasets. Our evaluation
focuses on assessing the capabilities of trained RMs without completing the
RLHF pipeline, which leaves us with some unanswered questions for future work.

#### Limitations

In this version, we have exclusively evaluated the MORE approach on the base
model Pythia-1.4B. Further works on larger or smaller models are in progress.
Besides, we randomly sample training samples from part of the dataset to avoid
the effects of unbalanced datasets. The experiment of MORE handling the
unbalancing mixed datasets will be presented in the next version. Throughout
our experiments, we have found it necessary to utilize equation 8 in MORE,
where we set $\alpha=0.5$ for all experiments. Further sensitive study on the
value of setting $\alpha$ is in progress.

#### Future work

We have demonstrated the alignment ability of the reward model using the ECE
metric (Zhu et al., 2023). However, the ultimate performance of the LLM, fine-
tuned by the RM, remains a crucial aspect of our investigation. In our future
work, we plan to complete the RL component in the RLHF pipeline. Additionally,
we aim to answer several key questions regarding the reward model and MORE: 1)
How do the number of diverse datasets and the scaling of base models impact
the performance of MORE? 2) Does the RM trained by MORE yield improved results
in RL tasks? 3) Can learning feedback from mixed diverse preference datasets
enhance the performance of the LLM? Furthermore, we will delve deeper into
understanding the implications of preference diversity in reward modeling.
Gaining insights into the effects of preference diversity is crucial for
designing algorithms that exhibit enhanced data efficiency.

#### Extension and Applications

The concept of learning shared preferences from mixed diverse preference
datasets can be extended to DPO by incorporating the MORE loss (Rafailov et
al., 2023). Furthermore, the approach of learning shared preferences through
MORE can be implemented as a preference modeling pre-trained (PMP) paradigm
(Askell et al., 2021). Subsequently, the RM trained using MORE can be easily
fine-tuned to specific preferences (Cheng et al., 2023a). This flexibility
allows for the adaptation of our approach to various applications and opens up
new possibilities for preference modeling.

##  7 Related Work

RLHF has become the mainstream approach to align language models towards
helpfulness, and harmlessness (Leike et al., 2018; Nakano et al., 2021b;
Ouyang et al., 2022; Bai et al., 2022a). They all utilize an RM to align
machine learning systems with human performance, which directly decides the
performance of preference alignment. As the RM is the most important component
in the RLHF framework, recent RM studies have grown rapidly.

The original goal of RM is to provide a scalar score to a model response and
indicate the quality in equation 2, especially helpfulness and harmlessness.
Due to the trade-off in quality aspects (Touvron et al., 2023a; Bai et al.,
2022a), it can be challenging for a single RM to perform well in all aspects.
Our work related to previous works handling multiple rewards and potential
disagreement in preferences. For instance, Llama-2 (Touvron et al., 2023a)
utilizes two separate RMs, one optimized for helpfulness and another for
harmlessness. They mitigate the magnitude bias of the reward scalar with a
margin loss, which provides a large margin for pairs with distinct responses,
and a smaller one for those with similar responses. Multiple RMs can be
utilized as majority voting or averaging (Jaques et al., 2020; Jang et al.,
2023) in the PPO (Schulman et al., 2017). Wang et al. (2023a) introduces a
Bayesian-based approach called d-PM to align language model with human
preferences with disagreement. Cheng et al. (2023a) proposes to train a
customized RM from the general RM to avoid disagreement from different
preference domains. Furthermore, our theoretical intuition follows recent work
DPO (Rafailov et al., 2023) and SLiC-HF (Zhao et al., 2023b) for preference
alignment, which explores more straightforward methods to align language
models with human preferences. Beyond the methodology, they have shown the
RLHF framework is working as likelihood calibration tasks (Deng et al., 2020;
Wang et al., 2023a; Azar et al., 2023), which proves that the reward values
provided by the RM are also important.

##  8 Conclusion

This paper addresses the problem of diversified preferences in reward modeling
for LLM alignment. Our experimental results demonstrate that RMs trained on
different preference data sources exhibit substantial diversity in terms of
reward distributions. This diversity derives from the intrinsic
characteristics of the datasets, posing a challenge for RMs bridging the gaps
among preference sources and better learning shared preference. To tackle this
challenge, we propose a straightforward yet highly effective multi-objective
reward model training scheme (MORE). Empirical results on five preference
datasets shows that MORE can enhance the RM training to better capture and
represent the shared preferences among multiple data sources. Furthermore, our
proposed method outperforms conventional training schemes across three
different evaluation metrics, highlighting its effectiveness and superiority.

## References

  * Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al.  A general language assistant as a laboratory for alignment.  _arXiv preprint arXiv:2112.00861_ , 2021.
  * Azar et al. (2023) Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos.  A general theoretical paradigm to understand learning from human preferences.  _arXiv preprint arXiv:2310.12036_ , 2023.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Barrett & Narayanan (2008) Leon Barrett and Srini Narayanan.  Learning all optimal policies with multiple criteria.  In _Proceedings of the 25th international conference on Machine learning_ , pp. 41–47, 2008.
  * Basile et al. (2021) Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, et al.  We need to consider disagreement in evaluation.  In _Proceedings of the 1st workshop on benchmarking: past, present and future_ , pp. 15–21. Association for Computational Linguistics, 2021.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pp. 2397–2430. PMLR, 2023.
  * Cheng et al. (2023a) Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du.  Everyone deserves a reward: Learning customized human preferences.  _arXiv preprint arXiv:2309.03126_ , 2023a.
  * Cheng et al. (2023b) Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, and Nan Du.  Adversarial preference optimization.  _arXiv preprint arXiv:2311.08045_ , 2023b.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Coste et al. (2023) Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger.  Reward model ensembles help mitigate overoptimization.  _arXiv preprint arXiv:2310.02743_ , 2023.
  * Csurka (2017) Gabriela Csurka.  A comprehensive survey on domain adaptation for visual applications.  _Domain adaptation in computer vision applications_ , pp. 1–35, 2017.
  * Davani et al. (2022) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _Transactions of the Association for Computational Linguistics_ , 10:92–110, 2022.
  * David (1963) Herbert Aron David.  _The method of paired comparisons_ , volume 12.  London, 1963.
  * Deng et al. (2020) Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato.  Residual energy-based models for text generation.  _arXiv preprint arXiv:2004.11714_ , 2020.
  * Dong et al. (2023a) Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou.  How abilities in large language models are affected by supervised fine-tuning data composition.  _arXiv preprint arXiv:2310.05492_ , 2023a.
  * Dong et al. (2023b) Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang.  Raft: Reward ranked finetuning for generative foundation model alignment.  _arXiv preprint arXiv:2304.06767_ , 2023b.
  * Eisenstein et al. (2023) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al.  Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.  _arXiv preprint arXiv:2312.09244_ , 2023.
  * Estabrooks et al. (2004) Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz.  A multiple resampling method for learning from imbalanced data sets.  _Computational intelligence_ , 20(1):18–36, 2004.
  * Gunantara (2018) Nyoman Gunantara.  A review of multi-objective optimization: Methods and its applications.  _Cogent Engineering_ , 5(1):1502242, 2018.
  * Jaggi (2013) Martin Jaggi.  Revisiting frank-wolfe: Projection-free sparse convex optimization.  In _Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013_ , volume 28 of _JMLR Workshop and Conference Proceedings_ , pp. 427–435. JMLR.org, 2013.  URL http://proceedings.mlr.press/v28/jaggi13.html.
  * Jang et al. (2023) Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu.  Personalized soups: Personalized large language model alignment via post-hoc parameter merging.  _arXiv preprint arXiv:2310.11564_ , 2023.
  * Jaques et al. (2020) Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard.  Human-centric dialog training via offline reinforcement learning.  _arXiv preprint arXiv:2010.05848_ , 2020.
  * Katharopoulos & Fleuret (2018) Angelos Katharopoulos and François Fleuret.  Not all samples are created equal: Deep learning with importance sampling.  In _International conference on machine learning_ , pp. 2525–2534. PMLR, 2018.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick.  Openassistant conversations - democratizing large language model alignment.  _CoRR_ , abs/2304.07327, 2023.  doi: 10.48550/ARXIV.2304.07327.  URL https://doi.org/10.48550/arXiv.2304.07327.
  * Leike et al. (2018) Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg.  Scalable agent alignment via reward modeling: a research direction.  _arXiv preprint arXiv:1811.07871_ , 2018.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pp. 1273–1282. PMLR, 2017.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 29, 2015.
  * Nakano et al. (2021a) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman.  Webgpt: Browser-assisted question-answering with human feedback.  _CoRR_ , abs/2112.09332, 2021a.  URL https://arxiv.org/abs/2112.09332.
  * Nakano et al. (2021b) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021b.
  * OpenAI (2023) OpenAI.  ChatGPT, Mar 14 version.  https://chat.openai.com/chat, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Pan & Yang (2009) Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ , 22(10):1345–1359, 2009.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sener & Koltun (2018) Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  _Advances in neural information processing systems_ , 31, 2018.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano.  Learning to summarize from human feedback.  In _NeurIPS_ , 2020.
  * Touvron et al. (2023a) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wan et al. (2023) Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang.  Everyone’s voice matters: Quantifying annotation disagreement using demographic information.  _arXiv preprint arXiv:2301.05036_ , 2023.
  * Wang et al. (2023a) Jiashuo Wang, Haozhao Wang, Shichao Sun, and Wenjie Li.  Aligning language models with human preferences via a bayesian approach.  _arXiv preprint arXiv:2310.05782_ , 2023a.
  * Wang et al. (2023b) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.  A comprehensive survey of continual learning: Theory, method and application.  _arXiv preprint arXiv:2302.00487_ , 2023b.
  * Wang et al. (2021) Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh.  Learning to diversify for single domain generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 834–843, 2021.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  Rrhf: Rank responses to align language models with human feedback without tears.  _arXiv preprint arXiv:2304.05302_ , 2023.
  * Zeng et al. (2023) Dun Zeng, Zenglin Xu, Yu Pan, Qifan Wang, and Xiaoying Tang.  Tackling hybrid heterogeneity on federated optimization via gradient diversity maximization.  _arXiv preprint arXiv:2310.02702_ , 2023.
  * Zhao et al. (2023a) Siyan Zhao, John Dang, and Aditya Grover.  Group preference optimization: Few-shot alignment of large language models.  _arXiv preprint arXiv:2310.11523_ , 2023a.
  * Zhao et al. (2023b) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023b.
  * Zhou et al. (2022a) Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022a.
  * Zhou et al. (2022b) Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie Gu, and Wenwu Zhu.  On the convergence of stochastic multi-objective gradient manipulation and beyond.  _Advances in Neural Information Processing Systems_ , 35:38103–38115, 2022b.
  * Zhu et al. (2023) Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao.  On the calibration of large language models and alignment.  _arXiv preprint arXiv:2311.13240_ , 2023.

##  Appendix A Decomposition of ranking loss

We present the detailed decomposing of vanilla ranking loss:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{rank}}(\theta;\mathcal{B})$
$\displaystyle=\sum_{b=1}^{B}\left[-\sigma\left(r_{\theta}(x^{(b)},y_{l}^{(b)})-r_{\theta}(x^{(b)},y_{w}^{(b)})\right)\right]\cdot\left[\nabla_{\theta}r_{\theta}(x^{(b)},y_{w}^{(b)})-\nabla_{\theta}r_{\theta}(x^{(b)},y_{l}^{(b)})\right]$
$\displaystyle=\sum_{b=1}^{B}\left[-\sigma\left(r_{\theta}(x^{(b)},y_{l}^{(b)})-r_{\theta}(x^{(b)},y_{w}^{(b)})\right)\right]\cdot\left[\nabla_{\theta}(r_{\theta}^{*}(x^{(b)},y_{w}^{(b)}-y_{l}^{(b)}))\right]$
$\displaystyle\quad+\sum_{b=1}^{B}\left[-\sigma\left(r_{\theta}(x^{(b)},y_{l}^{(b)})-r_{\theta}(x^{(b)},y_{w}^{(b)})\right)\right]\cdot\left[\nabla_{\theta}\tilde{r}_{\theta_{i}}(x^{(b)},y_{w}^{(b)}-y_{l}^{(b)}))\right]$
$\displaystyle=\sum_{b=1}^{B}\left[-\sigma\left(r_{\theta}(x^{(b)},y_{l}^{(b)})-r_{\theta}(x^{(b)},y_{w}^{(b)})\right)\right]\cdot\left[\nabla_{\theta}(r_{\theta}^{*}(x^{(b)},y_{w}^{(b)}-y_{l}^{(b)})\right]$
$\displaystyle\quad+\sum_{b=1}^{B}\left[-\sigma\left(r_{\theta}(x^{(b)},y_{l}^{(b)})-r_{\theta}(x^{(b)},y_{w}^{(b)})\right)\right]\cdot\left[\sum_{i=1}^{K}\frac{1}{K}\nabla_{\theta}\tilde{r}_{\theta_{i}}(x^{(b)},y_{w}^{(b)}-y_{l}^{(b)}))\right]$
$\displaystyle=\sum_{b=1}^{B}\left[-\sigma\left(r_{\theta}(x^{(b)},y_{l}^{(b)})-r_{\theta}(x^{(b)},y_{w}^{(b)})\right)\right]\cdot\left[\nabla_{\theta}r_{\theta}^{*}(x^{(b)},y_{w}^{(b)}-y_{l}^{(b)})\right]$
$\displaystyle\quad+K\sum_{i=1}^{K}\frac{1}{K}\sum_{(x,y_{w},y_{l})\in\mathcal{B}_{i}}\left[-\sigma\left(r_{\theta}(x,y_{l})-r_{\theta}(x,y_{w})\right)\right]\cdot\left[\nabla_{\theta}\tilde{r}_{\theta}(x,y_{w}-y_{l}))\right],$

where we let
$r_{\theta}(x,y_{w}-y_{l})=r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})$. In
contrast, the gradient
$\nabla_{\theta}\mathcal{L}_{\text{MORE}}(\theta;\mathcal{B})$ replaces the
coefficients $\frac{1}{K}$ with adjustable variable $\lambda$. Therefore, the
vanilla ranking loss is a special care of MORE loss.

##  Appendix B Experiments on Two Diverse Preference Datasets

In our preliminary experiments, we train RMs on two diverse preference
datasets, including Helpful-Harmless and Webgpt-Oasst1. We use the full size
of datasets from the original datasets. In details, For Helpful, Harmless,
Webgpt and Oasst1, we have 43,774, 42,537, 17.106 and 18,165 training samples
correspondingly. The number of training samples and test samples of each group
is balanced. We present the results in the tables below.

Table 3: Performance on Webgpt-Oasst1 datasets.

Webgpt OASST1 Avg. Acc ECE Raw 51.15 49.84 50.50 0.234 Webgpt 67.37 64.00
65.68 0.103 OASST1 66.31 66.61 66.46 0.102 Hybrid 66.43 65.88 66.15 0.101
Sequential 65.51 67.53 66.52 0.113 MORE 67.76 67.22 67.49 0.130

Table 4: Performance on Helpful-Harmless datasets.

Helpful Harmless Avg. Acc ECE Raw 49.78 47.18 48.48 0.244 Helpful 66.75 35.89
51.32 0.300 Harmless 35.67 67.21 51.44 0.312 Hybrid 63.77 63.45 63.61 0.032
Sequential 64.15 63.40 63.75 0.028 MORE 64.58 63.14 63.86 0.051

The results show that MORE implements comparable results with baseline methods
in reward accuracy. However, the MORE is not the best on the ECE metric. This
is because the number of diverse datasets is small (K=2), resulting in less
diverse preference information. Hence, the reward drift mitigation task is
solved inefficiently, making MORE degenerate to the vanilla method with rank
loss as we demonstrated in Section A. Furthermore, we have achieved better
results as shown in the main paper with $K=5$, indicating that MORE is
efficient in handling diverse preferences in reward modeling.

##  Appendix C Diversity of Reward Values

In Figure LABEL:fig:additional-results, we provide additional experimental
results as supplements of Figure 3.