# Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal
Impact on Coherence and Evasiveness in Dialogue Agents

San Kim
GSAI POSTECH
<EMAIL_ADDRESS>
&Gary Geunbae Lee
GSAI POSTECH
CSE POSTECH
<EMAIL_ADDRESS>

###### Abstract

Warning: this paper contains data that may be offensive or upsetting.

Recent advancements in open-domain dialogue systems have been propelled by the
emergence of high-quality large language models (LLMs) and various effective
training methodologies. Nevertheless, the presence of toxicity within these
models presents a significant challenge that can potentially diminish the user
experience. In this study, we introduce an innovative training algorithm, an
improvement upon direct preference optimization (DPO), called adversarial DPO
(ADPO). The ADPO algorithm is designed to train models to assign higher
probability distributions to preferred responses and lower distributions to
unsafe responses, which are self-generated using the toxic control token. We
demonstrate that ADPO enhances the model’s resilience against harmful
conversations while minimizing performance degradation. Furthermore, we
illustrate that ADPO offers a more stable training procedure compared to the
traditional DPO. To the best of our knowledge, this is the first adaptation of
the DPO algorithm that directly incorporates harmful data into the generative
model, thereby reducing the need to artificially create safe dialogue data.

Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal
Impact on Coherence and Evasiveness in Dialogue Agents

San Kim GSAI POSTECH<EMAIL_ADDRESS>Gary Geunbae Lee GSAI POSTECH CSE
POSTECH<EMAIL_ADDRESS>

##  1 Introduction

The enhancement of large language models (LLMs) has significantly improved the
overall performance of major NLP systems (Ousidhoum et al., 2021).
Furthermore, increasing the size of these models not only enhances performance
but also enables new capabilities previously unattainable, such as code
generation (Gao et al., 2023b) and applications in medical science (Moor et
al., 2023). Open-domain dialogue systems have particularly benefited from
advancements in LLMs, with several researchers demonstrating substantial
improvements in human preference gained through reinforcement learning from
human feedback (RLHF) (Ouyang et al., 2022; Stiennon et al., 2020).

To further enhance the performance of LLMs, scaling up the model and pre-
training dataset size is essential. However, this creates a trade-off between
performance and the potential increase in harmful content due to the growth in
the size of toxic data within the collected datasets (Touvron et al., 2023).
Numerous studies have demonstrated that many LLMs possess a non-trivial
propensity to generate toxic responses (Bender et al., 2021; Gehman et al.,
2020; Bommasani et al., 2021; Weidinger et al., 2021), posing significant
risks in downstream tasks, especially in dialogue systems. A direct solution
to mitigate this issue is using filtered datasets (Gehman et al., 2020).
However, this approach incurs considerable computational costs and becomes
increasingly challenging with larger pre-training datasets. An alternative
solution is employing RLHF, which aligns the model with human preferences.
Nonetheless, Ouyang et al. (2022) found that RLHF alone does not effectively
reduce toxicity.

Figure 1: ADPO pipeline with control token and RLAIF method. (Top) Supervised
Fine-Tuning process, additionally using toxic dialogue with "[TOXIC]"
appended. This enables model to generate harmful response which will be used
in ADPO. (Bottom) Labeling generated responses by LLM. By appending "[TOXIC]"
right after human utterance, model generates toxic response and if not
generate ordinary responses (Response1, Response2).

In this research, we introduce an advanced training methodology Adversarial
DPO (ADPO), which builds upon the principles of Direct Preference Optimization
(DPO) as proposed by Rafailov et al. (2023). The primary aim of ADPO is to
mitigate the generation of harmful responses by the model, while preserving
overall performance. This approach is a progression from the conventional DPO,
an algorithm offering stability and competitive performance as an alternative
to RLHF.

The novelty of ADPO lies in its targeted optimization to reduce the generation
of toxic responses. We hypothesize that training the model with potential
toxic responses within its capability range is more effective than using out-
of-scope responses. To achieve this, we fine-tune the model using a dataset of
toxic dialogues derived from the BAD dataset (Xu et al., 2021), augmented with
a toxic control token "[TOXIC]". This process empowers the model to
autonomously generate toxic responses when prompted by the "[TOXIC]" token.
Furthermore, we employ an inner toxic model configuration to demonstrate the
efficacy of ADPO. Our results, benchmarked against the baseline model Llama2
(Touvron et al., 2023), highlight the comparative performance of ADPO against
standard DPO. These findings underscore the potential of ADPO in reducing
undesirable outputs in language models while maintaining robust performance
metrics.

##  2 Related Work

Mitigating toxicity remains a significant challenge in deploying AI for safe
and effective human interaction. One prevalent strategy involves filtering
inappropriate data, which can be achieved through heuristic rule-based methods
or safety detectors such as offensive detection model Dinan et al. (2019).
However, as emphasized by Touvron et al. (2023), this filtration process comes
with a performance trade-off, highlighting the need to balance filtration
levels. Achieving this balance can be challenging and often relies on
empirical determination. An alternative approach is to append instructions to
pre-training data to signal the presence of toxicity (Prabhumoye et al.,
2023). While these methods can be effective, they entail substantial data
processing costs and depend on classifier performance, potentially limiting
optimal outcomes.

Another promising approach involves optimizing the training process, such as
RLHF. RLHF has been successfully implemented in models like InstructGPT
(Ouyang et al., 2022) and Sparrow (Glaese et al., 2022), aiming to optimize
human preferences. This is achieved by replacing actual human rewards with a
reward model and aligning AI with human values, a goal that traditional cross-
entropy loss cannot fully accomplish. However, this approach has limitations,
including the extensive human effort required for labeling model responses and
the instability and sensitivity to initialization inherent in the proximal
policy optimization (PPO) algorithm (Schulman et al., 2017; Casper et al.,
2023). As an advancement or alternative, reinforcement learning from AI
feedback (RLAIF) has reduced costs by replacing human annotators with LLMs
while maintaining competitive performance compared to RLHF (Bai et al., 2022b;
Lee et al., 2023). DPO has recently emerged (Rafailov et al., 2023),
transforming RL optimization into supervised training, significantly enhancing
stability and reducing computational demands. Several LLMs using DPO have
demonstrated impressive results, surpassing some models trained with RLHF. In
this paper, we combine these advancements to address the vulnerabilities of
RLHF and introduce an additional loss function specifically designed to
mitigate inherent toxicity in AI models.

Figure 2: Dialogue examples from reference model, inter toxic model, DPO model
and ADPO model.

##  3 Methodology

###  3.1 Training Pipeline

Our methodology follows an intuitive approach, primarily focusing on
penalizing the generation of undesirable responses. Figure 1 provides an
overview of the training process using ADPO. Before commencing ADPO training,
the model undergoes fine-tuning in a supervised manner. This phase, known as
supervised fine-tuning (SFT), incorporates both normal and toxic dialogues.
Normal dialogues are processed in a standard supervised manner, while toxic
dialogues are postfixed with a toxic control token, following the method
applied by Keskar et al. (2019). This token instructs the model to
intentionally generate harmful responses. We refer to this appending toxic
control token procedure as the inner toxic model, characterized by its ability
to produce toxic responses while maintaining the same parameter set as the
original model. This configuration ensures that toxic responses are generated
within the same distribution as normal responses. In the subsequent step of
creating preference data, we adopt a methodology similar to that described by
(Lee et al., 2023), utilizing a powerful LLM to label the model’s responses as
either "chosen" or "rejected". Additionally, within the same contextual
framework, we generate toxic responses using the inner toxic model. These
chosen, rejected, and toxic responses are then employed in the ADPO phase. The
training is designed to guide the model towards generating responses that
closely align with the chosen label while distancing from those labeled as
rejected or toxic.

###  3.2 ADPO

$\displaystyle\begin{split}D_{\theta}=&\beta\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}(y_{\theta}|x)||\pi_{ref}(y_{\theta}|x)]\\\
D_{t}=&\gamma\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}(y_{t}|x)||\pi_{tox}(y_{t}|x)]\\\
J(\theta)=&\max_{\pi_{\theta}}\mathbb{E}_{(x\sim\mathcal{D},y_{\theta}\sim\pi_{\theta})}[r(x,y_{\theta})-D_{\theta}]\\\
&-\mathbb{E}_{(x\sim\mathcal{D},y_{t}\sim\pi_{tox})}[p(x,y_{t})-D_{t}]\end{split}$
(1)

In our approach, ADPO utilizes an inner toxic model in combination with unsafe
dialogue data. This is accomplished by introducing an additional term into the
traditional RLHF objective function (Rafailov et al., 2023; Ouyang et al.,
2022), as illustrated in Eq. 1. Here, $x$ represents the dialogue history, and
$y$ denotes the response generated by the model $\pi$. The responses
$y_{\theta}$ and $y_{t}$ are produced by $\pi_{\theta}$ and $\pi_{tox}$
respectively. Furthermore, ADPO employs three distinct models: $\pi_{\theta}$,
the dialogue agent we train; $\pi_{ref}$, a reference model identical to
$\pi_{\theta}$ but with fixed parameters; and $\pi_{tox}$, the toxic model,
which is also equivalent to $\pi_{\theta}$ but non-trainable and uses the
toxic control token "[TOXIC]" at the beginning. The reward model $r$ is
designed to assign high rewards to preferred responses, while $p$ imposes
significant penalties for unsafe responses. The additional term in the
objective function encourages the model to simultaneously minimize the penalty
from $p(x,y)$ and maximize $D_{t}$, where $D_{t}$ evaluates the likelihood of
our model $\pi_{\theta}$ generating a response initially produced by the inner
toxic model $\pi_{tox}$. We found that incorporating an extra penalty $p$,
interpreted as providing detailed criteria in conjunction with $r$, enhances
training stability. This is because $p_{t}$ serves as a supplementary element
to $r$, as detailed in Section 5.4.

$\displaystyle\begin{split}R&=r(x,y_{\theta})-p(x,y_{t})\\\
&=\beta\log\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{ref}(y_{\theta}|x)}+\gamma\log\frac{\pi_{tox}(y_{t}|x)}{\pi_{\theta}(y_{t}|x)}\end{split}$
(2)

Drawing from the objective function as outlined in Eq. 2, we combine the
reward component $r$ and the penalty term $p$ to formulate the cumulative
metric $R$. This approach aligns with the methodologies used in Rafailov et
al. (2023). Detailed equations are provided in Appendix A.

$\displaystyle
R_{\beta}=\beta(\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{ref}(y_{w}|x)}-\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{ref}(y_{l}|x)})$
(3) $\displaystyle
R_{\gamma}=\gamma(\log\frac{\pi_{tox}(y_{t}|x)}{\pi_{\theta}(y_{t}|x)}-\log\frac{\pi_{tox}(y_{w}|x)}{\pi_{\theta}(y_{w}|x)})$
(4)
$\displaystyle\mathcal{L}_{\mathrm{ADPO}}=-\mathbb{E}_{(x,y_{w},y_{l},y_{t})\sim\mathcal{D}}[\log\sigma(R_{\beta}+R_{\gamma})]$
(5)

Eq. 5 illustrates our final objective function, where $y_{w}$, $y_{l}$, and
$y_{t}$ represent the chosen, rejected, and toxic responses, respectively.
Note that in Eq. 4 $y_{w}$ works as a "non-toxic" response. The primary goal,
as encapsulated in Eq. 5, is to maximize the sum of $R_{\beta}$ and
$R_{\gamma}$. To amplify $R_{\beta}$ in Eq. 3, considering that $\pi_{ref}$
and $\pi_{tox}$ are non-trainable, it is inevitable for $\pi_{\theta}$ to
learn to generate $y_{w}$ with a higher probability compared to $\pi_{ref}$,
while simultaneously generating $y_{l}$ with a lower probability than
$\pi_{ref}$. Similarly in $R_{\gamma}$, model is encouraged to generate
$y_{t}$ with a lower probability than $\pi_{tox}$, while generating $y_{w}$
with a higher probability. Although Eq. 3 aligns with Rafailov et al. (2023),
our findings suggest that relying solely on $R_{\beta}$ can lead to
instability due to the potential ambiguity in the criteria for chosen and
rejected labels. By incorporating an additional penalty term, we aim to
enhance both stability and performance. This is achieved by explicitly
introducing a criterion inherent in the existing preference data. The
distinctions between employing a penalty term are demonstrated in Figure 2.
This is illustrated through examples wherein the $\pi_{DPO}$ model
occasionally generates dull responses, whereas the $\pi_{ADPO}$ model adeptly
identifies potential hazards in the user’s utterance and responds safely. The
effectiveness of this approach is validated by the results discussed in
Section 5.

##  4 Experimental Details

###  4.1 Datasets

In this section, we present the datasets employed in our experimental setup:

  * •

Helpful and Harmless Human Preference Dataset from Anthropic (Bai et al.,
2022a): This dataset consists of dialogues between humans and an AI assistant.
The data collection process involved interactions between annotators and an AI
model, wherein annotators were presented with two AI-generated responses at
each turn and were tasked with selecting the preferable one. This procedure
enabled the labeling of data as either preferred or non-preferred, with a
specific emphasis on choosing responses that were both helpful and harmless.

  * •

Bot Adversarial Dialogue (BAD) (Xu et al., 2021): The BAD dataset comprises
conversational exchanges between a user and an AI model. Crowd workers were
instructed to engage in natural conversations with the AI while attempting to
elicit harmful responses. The AI’s responses at each turn were subsequently
labeled by the crowd workers as either safe or unsafe.

  * •

Blended Skill Talk (BST) (Smith et al., 2020): This dataset contains dialogues
between two participants. The participants were instructed to demonstrate
knowledge, empathy, or their assigned persona during the conversation when
appropriate. Notably, one of the participants, termed the "guided" speaker,
had the option to utilize responses generated by a dialogue model, thereby
diversifying the conversational context.

Overall all data had no risk of information that can identify specific person.
It is worth noting that our experiments utilized only 10% of the Anthropic
dataset, which contains over 160k dialogues, yet still yielded significant
results, demonstrating the data efficiency of ADPO. From the BAD dataset, we
extracted 8k dialogues that met the following criteria: (1) the last response
was generated by the AI model, and (2) the response was labeled as unsafe. The
incorporation of a harmful dataset for fine-tuning, although different from
the standard practices in DPO, is a distinguishing feature of ADPO. This
strategy allows the model to acquire and integrate additional contextual
information, thereby enhancing its learning process. However, it is important
to acknowledge that this aspect is unique to ADPO, and a direct comparison
between ADPO and DPO methodologies may not be entirely equitable if based on
differently fine-tuned models. To address this, we have conducted an
additional experiment, detailed in Section 5.3, where DPO is also trained on
an SFT model that has been fine-tuned with the toxic control token. This
experiment aims to facilitate a more balanced and fair comparison of the two
methodologies.

###  4.2 Preference Data Generation

For better convergence, instead of using labeled data in Anthropic dataset, we
use model’s generated response from chosen and rejected data, removing each
response and using overlapped dialogue history. In this generation phase, two
variants of responses are created with temperatures set at 1.0 and 1.5,
respectively, along with a toxic response generated at a temperature of 1.5.
Adhering to the procedure outlined in RLAIF (Bai et al., 2022b; Lee et al.,
2023), we employ the Llama2-chat model (Touvron et al., 2023) for the task of
labeling these model-generated responses. While Bai et al. (2022b) emphasizes
the significance of parameter size in such applications, we observed that a
model with 13 billion parameters was sufficiently capable of yielding
meaningful progress in our context. Excluding toxic response, response pairs
are given to Llama2-chat and labeled either chosen or rejected. Note that if
both responses are considered preferred or not preferred, we dropped out
corresponding data. This decision was made to maintain the integrity and
relevance of the data in our study.

###  4.3 Model Training

In our experiments, the base model used was Llama2 with 7 billion parameters,
which is open-source and permitted for research purpose, attached with LoRA
(Hu et al., 2021) adaptor at a rank of 16, and the alpha parameter was set to
32. During the SFT phase, we utilized 40% of the Anthropic dataset, reserving
the remaining 60% for generating preference data in both the DPO and ADPO
training. Notably, the SFT models for DPO and ADPO were trained independently,
referred to as $\mathrm{SFT}$ with non-toxic dataset and $\mathrm{SFT}$ with
toxic dataset, respectively. Every SFT models are trained for 2 epochs. For
ADPO training, we incorporated an additional dataset BAD for the SFT phase
appending a toxic control token to each dialogue. In generating preference
data, we used the unused portion of the Anthropic dataset, excluding the
model’s final response in each dialogue. The details of this phase are
explained in Section 4.2. Subsequently, both DPO and ADPO were trained for
five epochs. The optimal models were found when using $\beta=0.9$ for 2 epochs
in DPO and $\beta=0.3$ and $\gamma=0.2$ for 4 epochs in ADPO. Model was
trained with only single run as it takes plenty of resources to train, with
seed value of 42. With 4 x NVIDIA A100 GPUs, the SFT and DPO or ADPO training
processes collectively required about 17 hours, and an additional 12 hours
were needed for the response annotation phase using the Llama2-13B-chat model.
During each training iteration, the train set was divided into an 8:2 ratio
for the validation set. We used a learning rate of 3e-5 and a lambda learning
rate scheduler for all training purposes.

##  5 Results and Analysis

|  | Bot Adversarial Dialogue (BAD) |  | Blended Skill Talk (BST) |
---|---|---|---|---|---
Method | Dataset | Coherence | Evasiveness | Toxicity |  | Coherence | Evasiveness | Toxicity |
$\mathrm{SFT}$ | original | 80.6% | 47.5% | 3.2% |  | - | - | - |
$\mathrm{SFT}$ | non-toxic | 86.0% | 35.1% | 4.7% |  | 91.3% | 9.4% | 0.2% |
$\mathrm{SFT}$ | toxic | 73.8% | 31.7% | 13.3% |  | 98.5% | 2.2% | 0.1% |
$\mathrm{DPO}$ | non-toxic | 91.5% | 56.0% | 0.1% |  | 81.5% | 23.7% | 0.0% |
$\mathrm{DPO}$ | toxic | 89.8% | 41.5% | 0.2% |  | 87.7% | 10.9% | 0.0% |
$\mathrm{ADPO}$ | toxic | 92.6% | 33.9% | 1.2% |  | 98.0% | 2.7% | 0.1% |
Table 1:  Comparison of response frequency in BAD dataset and BST dataset.
Toxic and non-toxic datasets denote the dataset with self-generated responses,
which contain toxic responses or not, respectively. Note that each DPO and
ADPO are originated from the resulted model by SFT which shares same dataset
(e.g. DPO with non-toxic dataset is trained additionally on the SFT with non-
toxic dataset. DPO with toxic dataset is trained on the SFT with toxic
dataset.). Original dataset denotes the usage of Anthropic dataset without
response sampling. A higher value indicates better coherence, whereas lower
values are preferred for evasiveness and toxicity.

###  5.1 Evaluation

Evaluating natural language generation (NLG) systems remains challenging, as
traditional automatic metrics primarily focus on token-level similarity,
potentially missing semantically equivalent responses. To address this issue,
recent research has suggested using LLMs for NLG evaluation (Fu et al., 2023;
Wang et al., 2023), with significant advancements by Liu et al. (2023) in
improving the correlation between human judgments and LLM evaluations.
Following the methodology established by Liu et al. (2023), which incorporates
the chain-of-thought approach (Wei et al., 2022), we conducted our evaluation
using GPT-4. To validate this approach, we also conducted human evaluations on
300 randomly selected responses from a total of 772 entries in the BAD test
dataset, achieving an F1 score of 0.776 using scikit-learn package (Pedregosa
et al., 2011).
In our evaluation process, each model generated responses on the BAD test
dataset with a temperature setting of 1.2. Other than Toxicity, we also
evaluated coherence and evasiveness, recognizing these as essential yet
potentially vulnerable aspects of generative systems that can lead to
incoherent or uninspiring responses (Ni et al., 2023). Instead of using a
numeric scoring system for evaluation, which can introduce variability, we
opted for a classification approach. This involved categorizing the presence
of specified metrics within each response and calculating the frequency ratio
of these occurrences relative to the total dataset. This methodology provides
a more consistent way to assess model performance.

###  5.2 Evasiveness-Toxicity Trade-off

Our results are presented in Table 1, comparing models trained by three
methods (SFT, DPO, ADPO) across two datasets (BAD, BST). Models trained by
$\mathrm{SFT}$ with toxic and non-toxic datasets serve as "ADPO base model"
and "DPO base model", respectively, as these methods implies additional
training on the model initially trained by SFT (except for model trained by
DPO with toxic dataset since it is trained on ADPO base model). The result of
the BAD dataset is consistent with previous studies utilizing RLHF (Ouyang et
al., 2022; Rafailov et al., 2023; Glaese et al., 2022; Lee et al., 2023), as
both DPO and ADPO methods demonstrate superior performance compared to
$\mathrm{SFT}$. Comparing ADPO and DPO, ADPO significantly reduces its
toxicity, achieving a nearly tenfold decrease from ADPO base model. This
reduction results in all toxic metrics falling below 1%. However, it is
important to acknowledge that these toxicities in ADPO are still marginally
higher than those observed in DPO, which demonstrates near-zero toxicity.
Nonetheless, it is noteworthy that the evasiveness metric increased by more
than 20% in $\mathrm{DPO}$ relative to DPO base model, while it only increased
by 0.02 in $\mathrm{ADPO}$ from ADPO base model. This suggests that in
scenarios involving potentially unsafe user prompts, the DPO model avoids
answering, frequently resorting to expressions like "I don’t know" or "I don’t
understand." This behavior highlights an emerging challenge in the form of
"Evasiveness", where the model opts for avoidance rather than directly
addressing or refuting unsafe prompts.

This issue becomes more apparent in the results obtained from the BST dataset.
Due to the nature of the BST dataset, which does not encompass dialogues
designed to elicit harmful responses, all models exhibited near-zero toxicity.
However, concerning coherence and evasiveness, ADPO significantly outperformed
DPO, demonstrating superior effectiveness. This difference highlights that DPO
tends to train models towards increased evasiveness and reduced coherence,
even in general conversational contexts. This phenomenon aligns with findings
from other studies Casper et al. (2023); Go et al. (2023); Glaese et al.
(2022), suggesting RLHF often leads to mode collapse, which model loses
variety in generation, thereby diminishing the diversity of the model’s
response generation. Despite being trained in a supervised manner, DPO retains
characteristics of reinforcement learning as it not only trains the model to
replicate singularly chosen data but also generates responses simultaneously,
likely in chosen data and unlikely in rejected data compared to its reference
model. The model’s requirement to seek an optimal answer is analogous to the
exploratory behavior of reinforcement learning agents. Consequently, DPO tends
to guide the model towards generating evasive responses. This strategy aims to
secure moderate rewards (or minimize loss) from both selected and non-selected
data rather than generating responses that are distinctly aligned or opposed
to one particular category. This challenge becomes more pronounced when the
presented preference data spans a broad spectrum of human values, resulting in
ambiguous criteria for distinguishing between preferred and non-preferred
responses. In addressing this issue, it is imperative to introduce
supplementary criteria to preserve response diversity. ADPO relies on
generating unsafe responses, employing these as an additional criterion for
penalization. By explicitly defining clear and undesirable values, ADPO not
only facilitates the reduction of unwanted responses, specifically unsafe
responses in this study, but also aids in maintaining response diversity. This
approach effectively circumvents the tendency towards uniform, evasive
responses often observed in models trained solely on preference data.

###  5.3 Unsafe Data Utilization

While ADPO’s effectiveness in reducing toxicity with minimal compromise in
evasiveness is notable, it may gain contextual information from unsafe data,
which is not typically employed in supervised training models like DPO base
model. This section compares the outcomes of both DPO and ADPO when trained on
same ADPO base model, presumed to contain richer contextual insights.
In Table 1, the model trained via DPO from ADPO base model is labeled as
$\mathrm{DPO}$ with toxic dataset. All models exhibit nearly zero toxicity due
to the absence of toxic dialogue in the BST dataset. However, $\mathrm{DPO}$
with toxic dataset demonstrates enhanced contextual understanding,
outperforming $\mathrm{DPO}$ with non-toxic dataset in coherence and
evasiveness. Despite sharing the same SFT model, $\mathrm{DPO}$ with toxic
dataset lags behind in dialogue quality, with $\mathrm{ADPO}$ showing over a
10% higher coherence and a fourfold reduction in evasiveness. This underscores
ADPO’s proficient use of unsafe data to accurately discern harmful content,
establishing clearer and more detailed criteria. The comparison of
$\mathrm{DPO}$ with toxic dataset and $\mathrm{ADPO}$, both originating from
ADPO base model, further reveals that ADPO effectively reduces toxicity while
barely affecting performance metrics (coherence: -0.5%, evasiveness: +0.5%),
unlike $\mathrm{DPO}$ with toxic dataset which significantly compromises
conversational capabilities (coherence: -10.8%, evasiveness: +8.7%). These
findings affirm that ADPO efficiently utilizes unsafe data to reduce toxicity,
enhancing its contextual understanding and maintaining diverse response
generation.

###  5.4 Training Assessment

Optimizing models using RLHF presents challenges due to its sensitivity to
hyperparameters (Christiano et al., 2017; McKinney et al., 2022) and the
difficulty in detecting over-optimization (Casper et al., 2023). To evaluate
our training procedure, we employed KL divergence between $\pi_{\theta}$ and
$\pi_{ref}$, as well as between $\pi_{\theta}$ and $\pi_{tox}$, inspired by
Gao et al. (2023a).

Figure 3: (Top) KL divergence on chosen data between $\mathrm{DPO}$ and
$\mathrm{ADPO}$ training. (Bottom) KL divergence on toxic data and chosen
data. Note that the top and bottom have the same ADPO-Chosen KL but in
different y-axis scales.

As illustrated in Figure 3, we analyze two types of KL divergence: chosen KL
($\mathbf{D}_{KL}(\pi_{\theta}(y_{w}|x)||\pi_{ref}(y_{w}|x))$) on the chosen
data, and toxic KL
($\mathbf{D}_{KL}(\pi_{\theta}(y_{t}|x)||\pi_{tox}(y_{t}|x))$) on the toxic
data. A higher chosen KL is desirable, indicating a greater likelihood of
$\pi_{\theta}$ generating chosen data. However, extremely high values should
be avoided due to potential errors in human-labeled preference data (Pandey et
al., 2022; Saunders et al., 2022) and over-optimization. Optimal chosen KL
values for the best-performing models in our experiment ranged from $[-2,1]$,
with $\mathrm{DPO}$ and $\mathrm{ADPO}$ achieving $-2.0$ and $0.06$
respectively. Notably, $\mathrm{ADPO}$ maintained chosen KL within the optimal
range and showed a steady decrease, while $\mathrm{DPO}$ experienced a rapid
drop, demonstrating sensitivity to the $\beta$.
For toxic KL, lower values are preferable, indicating a reduced likelihood of
generating toxic responses. However, extremely low values may lead to "reward
hacking" (Skalse et al., 2022), where the model produces nonsensical but non-
toxic responses. Interestingly, both chosen KL and toxic KL exhibited similar
trends, suggesting that as training progresses, the model optimizes a balanced
response that aligns with chosen-rejected-toxic data, maximizing rewards from
equations 3 and 4.

##  6 Conclusion

In this paper we have concentrated on training open-domain dialogue models
while mitigating inherent toxicity. Our study introduces ADPO, an advanced
algorithm of the DPO method, which effectively reduces toxicity levels without
compromising dialogue performance. ADPO utilizes an internal toxic model,
using harmful datasets to enhance safety. This approach enables the model to
assimilate both contextual information and safety criteria derived from toxic
data. Moreover, compared to models trained using DPO, ADPO exhibits higher
stability during training across a range of hyperparameters, enhancing
optimization based on human preferences while penalizing the generation of
unsafe responses.

To the best of our knowledge, this research represents the first adaptation of
the DPO algorithm, uniquely employing unsafe data in generative models to
incorporate criteria for harmlessness. In the future, we believe exploring
various methodologies for effectively utilizing unsafe data presents a
promising avenue for research. Although toxic, it contains rich contextual
information and can be instrumental in instructing dialogue agents on
behaviors to avoid. Further advancements in improving both helpfulness and
harmfulness is also encouraging. Helpfulness and harmfulness sometimes
conflict each other (Bai et al., 2022a, a) where aiding user may inadvertently
result in harmful outcomes. This suggests that models should be trained to
discern when to appropriately decline a request based on the context, rather
than being constantly positive.

##  7 Human Annotation

For the validation of GPT-4 evaluation through human annotation, three
English-fluent speakers participated, all of whom are graduate students
specializing in the NLP research field. Annotators are all from Asia, with
using English as their second language. Since the minimum hourly wage is
approximately $7.5, we compensated each annotator with $23, considering the
task does not exceed three hours.

##  8 Ethical Considerations

Our main concern related to ethical considerations lies within the deployment
of the SFT model, particularly when it is trained with a toxic control token.
While users have the capacity to avoid the generation of unsafe responses by
refraining from employing the toxic control token, it is still possible to
inadvertently activate the model’s inherent toxicity. Moreover, the potential
for the model’s exploitation for malicious purposes cannot be overlooked.
Therefore it is highly advised to conduct thorough monitoring of the model’s
possible outputs prior to its deployment and to implement strict measures for
regulating its use.

##  9 Limitations

There are few limitations in our work that needs to be mentioned. First is LLM
utilization. As it is still ongoing research about how LLM works, using LLM
for annotating model responses can be variant and sometimes labels reflect the
harmfulness and bias transferred from LLM (Lee et al., 2023). Additionally,
for evaluation even though we followed Liu et al. (2023) and showed moderate
F1 score with human evaluation, it is still unstable because human annotators
are from same demographic group, which can result in biased annotation.
Another limitation is the amount of data used. 16k of Anthropic preference
data (Bai et al., 2022a) was enough to show ADPO’s improvement from DPO, but
using full 160k data would lead to better result. Same in inner toxic model,
using more and various toxic data can provide model more contextual and
desirable criterion information, which would lead to better model. We hope
future work uses as many data as possible for optimal result and conduct
strict observation about LLM utilization.

## Acknowledgements

This research is supported by the MSIT (Ministry of Science and ICT), Korea,
under the ITRC (Information Technology Research Center) support program
(IITP-2024-2020-0-01789) supervised by the IITP (Institute for Information &
Communications Technology Planning & Evaluation), and by Smart HealthCare
Program (www.kipot.or.kr) funded by the Korean National Police Agency (KNPA,
Korea) [Project Name: Development of an Intelligent Big Data Integrated
Platform for Police Officers’ Personalized Healthcare / Project Number:
220222M01], and by Institute of Information & communications Technology
Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
(No.2019-0-01906, Artificial Intelligence Graduate School Program(POSTECH)).

## References

  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 610–623.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _arXiv preprint arXiv:2307.15217_.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Dinan et al. (2019) Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019.  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4537–4546, Hong Kong, China. Association for Computational Linguistics.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023.  Gptscore: Evaluate as you desire.  _arXiv preprint arXiv:2302.04166_.
  * Gao et al. (2023a) Leo Gao, John Schulman, and Jacob Hilton. 2023a.  Scaling laws for reward model overoptimization.  In _International Conference on Machine Learning_ , pages 10835–10866. PMLR.
  * Gao et al. (2023b) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b.  Pal: Program-aided language models.  In _International Conference on Machine Learning_ , pages 10764–10799. PMLR.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369, Online. Association for Computational Linguistics.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv preprint arXiv:2209.14375_.
  * Go et al. (2023) Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. 2023.  Aligning language models with preferences through f-divergence minimization.  In _International Conference on Machine Learning_.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-eval: NLG evaluation using gpt-4 with better human alignment.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2511–2522, Singapore. Association for Computational Linguistics.
  * McKinney et al. (2022) Lev E McKinney, Yawen Duan, David Krueger, and Adam Gleave. 2022.  On the fragility of learned reward functions.  In _Deep Reinforcement Learning Workshop NeurIPS 2022_.
  * Moor et al. (2023) Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. 2023.  Foundation models for generalist medical artificial intelligence.  _Nature_ , 616(7956):259–265.
  * Ni et al. (2023) Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023.  Recent advances in deep learning based dialogue systems: a systematic survey.  _Artificial Intelligence Review_ , 56(4):3055–3155.
  * Ousidhoum et al. (2021) Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. 2021.  Probing toxic content in large pre-trained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4262–4274, Online. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pandey et al. (2022) Rahul Pandey, Hemant Purohit, Carlos Castillo, and Valerie L Shalin. 2022.  Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning.  _International Journal of Human-Computer Studies_ , 160:102772.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Prabhumoye et al. (2023) Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2023.  Adding instructions during pretraining: Effective way of controlling toxicity in language models.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2636–2651, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022.  Self-critiquing models for assisting human evaluators.  _arXiv preprint arXiv:2206.05802_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Skalse et al. (2022) Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022.  Defining and characterizing reward gaming.  _Advances in Neural Information Processing Systems_ , 35:9460–9471.
  * Smith et al. (2020) Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020.  Can you put it all together: Evaluating conversational agents’ ability to blend skills.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2021–2030, Online. Association for Computational Linguistics.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023.  Is chatgpt a good nlg evaluator? a preliminary study.  _arXiv preprint arXiv:2303.04048_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_.
  * Xu et al. (2021) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021.  Bot-adversarial dialogue for safe conversational agents.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2950–2968, Online. Association for Computational Linguistics.

##  Appendix A ADPO Algorithm

###  A.1 Objective Transformation

In this appendix we show how ADPO algorithm of Eq. is derived from objective
function in RLHF.

$\displaystyle\begin{split}D_{\theta}=&\beta\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}(y_{\theta}|x)||\pi_{ref}(y_{\theta}|x)]\\\
D_{t}=&\gamma\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}(y_{t}|x)||\pi_{tox}(y_{t}|x)]\\\
J(\theta)=&\max_{\pi_{\theta}}\mathbb{E}_{(x\sim\mathcal{D},y_{\theta}\sim\pi_{\theta})}[r_{\theta}(x,y_{\theta})-D_{\theta}]\\\
&-\mathbb{E}_{(x\sim\mathcal{D},y_{t}\sim\pi_{tox})}[p_{t}(x,y_{t})-D_{t}]\end{split}$
(6)

From Eq. 6 we can incorporate two expectation terms and transform maximization
problem to minimization problem.

$\displaystyle\begin{split}J(\theta)=&\max_{\pi_{\theta}}\mathbb{E}_{(x\sim\mathcal{D},y_{\theta}\sim\pi_{\theta})}[r_{\theta}(x,y_{\theta})-D_{\theta}]\\\
&-\mathbb{E}_{(x\sim\mathcal{D},y_{t}\sim\pi_{tox})}[p_{t}(x,y_{t})-D_{t}]\\\
\\\ \end{split}$ (7)

Here, we define $\tau$ and $R$ for comprehensibility.

$\displaystyle\begin{split}\tau=&(x\sim\mathcal{D},y_{\theta}\sim\pi_{\theta},y_{t}\sim\pi_{tox})\\\
R=&r(x,y_{\theta})-p(x,y_{t})\\\ \end{split}$ (8)

With using $\tau$ and $R$, objective function $J(\theta)$ can be described as
follows.

$\displaystyle\begin{split}J(\theta)=&\min_{\pi_{\theta}}\mathbb{E}_{\tau}[D_{\theta}-D_{t}\\\
&-(r(x,y_{\theta})-p(x,y_{t}))]\\\
=&\min_{\pi_{\theta}}\mathbb{E}_{\tau}\Bigg{[}\log\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{ref}(y_{\theta}|x)}\Bigg{.}\\\
&\Bigg{.}-\log\frac{\pi_{\theta}(y_{t}|x)^{\frac{\gamma}{\beta}}}{\pi_{tox}(y_{t}|x)^{\frac{\gamma}{\beta}}}-\frac{1}{\beta}R\Bigg{]}\end{split}$
(9)

Finally, with defining $R_{e}$ we can transform previous objective function
for ADPO.

$\displaystyle\begin{split}R_{e}=&\exp(\frac{1}{\beta}R)\\\
J(\theta)=&\min_{\pi_{\theta}}\mathbb{E}_{\tau}\Bigg{[}\log\frac{\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{\theta}(y_{t}|x)^{\frac{\gamma}{\beta}}}}{\frac{\pi_{ref}(y_{\theta}|x)}{\pi_{tox}(y_{t}|x)^{\frac{\gamma}{\beta}}}R_{e}}\Bigg{]}\end{split}$
(10)

To optimize $J(\theta)$ it is required to make numerator equal to denominator,
which is achieved when we have optimal model $\pi_{\theta}^{*}$.

$\displaystyle\begin{split}\frac{\pi_{\theta}^{*}(y_{\theta}|x)}{\pi_{\theta}^{*}(y_{t}|x)^{\frac{\gamma}{\beta}}}=\frac{\pi_{ref}(y_{\theta}|x)}{\pi_{tox}(y_{t}|x)^{\frac{\gamma}{\beta}}}R_{e}\end{split}$
(11)

Following work in Rafailov et al. (2023), since $\pi^{*}(y|x)\geq 0$ for all
$y$ and $\sum_{y}{\pi^{*}(y|x)=1}$ we can derive following objective from Eq.
10

$\displaystyle\begin{split}J(\theta)=&\min_{\pi_{\theta}}\mathbb{E}_{\tau}\Bigg{[}\log\frac{\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{\theta}(y_{t}|x)^{\frac{\gamma}{\beta}}}}{\frac{\pi_{\theta}^{*}(y_{\theta}|x)}{\pi_{\theta}^{*}(y_{t}|x)^{\frac{\gamma}{\beta}}}}\Bigg{]}\end{split}$
(12)

Eq. 12 can be minimized by

$\displaystyle\begin{split}\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{\theta}(y_{t}|x)^{\frac{\gamma}{\beta}}}=\frac{\pi_{\theta}^{*}(y_{\theta}|x)}{\pi_{\theta}^{*}(y_{t}|x)^{\frac{\gamma}{\beta}}}=\frac{\pi_{ref}(y_{\theta}|x)}{\pi_{tox}(y_{t}|x)^{\frac{\gamma}{\beta}}}R_{e}\end{split}$
(13)

###  A.2 ADPO Objective

To apply Bradley-Terry model (Bradley and Terry, 1952) to our objective, we
can define $R$ from Eq. 13 by following equation.

$\displaystyle\begin{split}R_{e}=&\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{\theta}(y_{t}|x)^{\frac{\gamma}{\beta}}}\frac{\pi_{tox}(y_{t}|x)^{\frac{\gamma}{\beta}}}{\pi_{ref}(y_{\theta}|x)}\\\
R=&\beta\log\left[\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{\theta}(y_{t}|x)^{\frac{\gamma}{\beta}}}\frac{\pi_{tox}(y_{t}|x)^{\frac{\gamma}{\beta}}}{\pi_{ref}(y_{\theta}|x)}\right]\\\
=&\beta\log\frac{\pi_{\theta}(y_{\theta}|x)}{\pi_{ref}(y_{\theta}|x)}+\gamma\log\frac{\pi_{\theta}(y_{t}|x)}{\pi_{tox}(y_{t}|x)}\\\
=&r(x,y_{\theta})-p(x,y_{t})\end{split}$ (14)

Applying Eq. 14 to Bradley-Terry model, we can get final ADPO objective.

$\displaystyle\begin{split}R_{w}=&r(x,y_{w})-p(x,y_{t})\\\
=&\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{ref}(y_{w}|x)}+\gamma\log\frac{\pi_{\theta}(y_{t}|x)}{\pi_{tox}(y_{t}|x)}\\\
R_{l}=&r(x,y_{l})-p(x,y_{w})\\\
=&\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{ref}(y_{l}|x)}+\gamma\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{tox}(y_{w}|x)}\\\
\mathcal{L}_{\mathrm{ADPO}}=&-\mathbb{E}_{(x,y_{w},y_{l},y_{t})\sim\mathcal{D}}[\log\sigma(R_{w}-R_{l})]\end{split}$
(15)

Note that Eq. 15 is equivalent to Eq. 5 if we use $R_{\beta}$, $R_{\gamma}$ in
Eq. 3 and Eq. 4, which we can get re-arranging $R_{w}$ and $R_{l}$ in terms of
$\beta$ and $\gamma$.

##  Appendix B LLM Annotation

To guide the selection or rejection of responses, we follow the prompt format
outlined in Bai et al. (2022b), which provides a Human-Assistant dialogue
alongside instructions to choose between two potential responses, accompanied
by a rationale for the selection. As described in Figure 5, we give 2-shots of
examples initially, followed by instructions to identify the more favorable
response as either "(A)" or "(B)". Should neither response be deemed suitable,
model may answer as "PASS". Figure 4 shows a sample dataset after the
annotation by Llama2-chat, which is used for ADPO training.

##  Appendix C GPT-4 evaluation

Figure 6 illustrates the example prompt utilized for evaluating responses via
GPT-4. As we mentioned in Section 5.1, the prompt is adapted from the work
proposed by Liu et al. (2023) with certain modifications. Initially we give
task introduction and evaluation criteria, which are devised by human.
Providing task and criteria, we ask GPT-4 to generate evaluation steps
required to accomplish the task, which are then consistently applied across
all dialogue assessments. Upon integrating these self-devised evaluation steps
into the prompt, the current dialogue and its corresponding evaluation form
are presented.

##  Appendix D Human Annotation Instruction

In this section we present the guidelines provided to human annotators, as
explicated in Figure 7. Annotators were instructed to assign labels of 0 or 1
according to the presence of specific characteristics within the text. To
mitigate the potential ambiguity inherent in the subjective nature of these
characteristics, additional clarifications were included. Furthermore, we
integrated "Gender bias" and "Social bias" into a single "Toxicity" metric
because of significant correlation between these variables. Table 2 presents
the correlation between human assessments and GPT-4 evaluations, featuring the
average F1 scores of GPT-4 alongside those of three independent annotators.
The data indicate a pronounced correlation in terms of coherence, which can be
attributed to the objective clarity in determining coherence levels. In
contrast, the evasiveness presents a more nuanced challenge with the lowest
score. This complexity arises from the subjective interpretation of responses,
as certain evasive replies may be perceived as appropriate or non-malicious,
thus complicating the assessment process. As mentioned in Section 5.1, we
observed an overall average F1 score of 0.776 in the comparison between human
annotators and GPT-4 assessments. We believe that this decent correlation is
attributed to the binary classification evaluation ("YES" or "NO"), which
inherently reduces variability compared to numeric scoring systems.

Figure 4: Dataset example after LLM annotation. Metric | Coherence | Evasiveness | Toxicity
---|---|---|---
F1-Score | 0.843 | 0.704 | 0.781
Table 2: Average F1 score between human evaluation and GPT-4 assessment.

Figure 5: Prompt example of LLM annotation. Figure 6: Prompt example of GPT-4
evaluation. Figure 7: Instruction for human annotators.