# CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement
Learning

Zeyuan Liu Equal contribution. Kai Yang**footnotemark: * Xiu Li Corresponding
Author. Email<EMAIL_ADDRESS>Tsinghua Shenzhen International
Graduate School, Tsinghua University

###### Abstract

Distribution shift is a major obstacle in offline reinforcement learning,
which necessitates minimizing the discrepancy between the learned policy and
the behavior policy to avoid overestimating rare or unseen actions. Previous
conservative offline RL algorithms struggle to generalize to unseen actions,
despite their success in learning good in-distribution policy. In contrast, we
propose to use the gradient fields of the dataset density generated from a
pre-trained offline RL algorithm to adjust the original actions. We decouple
the conservatism constraints from the policy, thus can benefit wide offline RL
algorithms. As a consequence, we propose the Conservative Denoising Score-
based Algorithm (CDSA) which utilizes the denoising score-based model to model
the gradient of the dataset density, rather than the dataset density itself,
and facilitates a more accurate and efficient method to adjust the action
generated by the pre-trained policy in a deterministic and continuous MDP
environment. In experiments, we show that our approach significantly improves
the performance of baseline algorithms in D4RL datasets, and demonstrate the
generalizability and plug-and-play capability of our model across different
pre-trained offline RL policy in different tasks. We also validate that the
agent exhibits greater risk aversion after employing our method while
showcasing its ability to generalize effectively across diverse tasks.

123

##  1 Introduction

Reinforcement learning (RL) algorithms have been demonstrated to be successful
on a range of challenging tasks, from games [29, 35] to robotic control [34].
As one of its branches, offline RL only uses a fixed training dataset during
training, which aims at finding effective policies while avoiding online
interactions with the real environment and a range of related issues [20, 9].
However, due to the data-dependent nature of offline RL algorithms, this
pattern makes distribution shift a major obstacle to the effectiveness of the
algorithm [9, 25]. Offline RL algorithms are prone to producing inaccurate
predictions and catastrophic action commands when queried outside of the
distribution of the training data, which leads to catastrophic outcomes. To
strike a suitable trade-off between learning an improved policy and minimizing
the divergence from the behavior policy, aiming to avoid errors due to
distribution shift, previous work has provided various perspectives, including
constraining the system in the training dataset distribution [18, 9, 26], or
developing a distributional critic to leverage risk-averse measures [27, 41].

However, most previous conservative offline RL algorithms failed to fully
disentangle the knowledge related to conservatism from the algorithm’s
training process. This knowledge is typically incorporated into functions such
as the final policy or critics, rendering it inseparable from other
components. Consequently, even if the training dataset remains unchanged,
various algorithms are unable to directly exchange their conservatism-related
knowledge, particularly if this knowledge is deemed solely dependent on the
distribution of the training dataset. To tackle this issue, we explore the
possibility of learning conservatism-related knowledge exclusively from the
training dataset to obtain a plug-and-play decision adjuster. One intuitive
approach is to leverage the density distribution of each dataset to guide the
agent towards states located in areas of high density as much as possible.
This can be achieved by adjusting the actions within the dataset to steer
transitions towards states with higher density. Essentially, this modification
makes the executed actions safer and more conservative. This approach enables
us to utilize any offline RL algorithm as the baseline algorithm. By ensuring
the algorithm uses the same training dataset, we can effectively harness the
acquired plug-and-play conservatism-related knowledge to enhance the
algorithm’s performance.

To obtain the distribution of the dataset, various methods commonly utilize
the approach of reconstructing the density within the training dataset [18, 9,
32, 10, 28]. Previous studies in this domain have utilized density models to
restrict or regularize the controller, preventing the agent from selecting
actions or exploring states with low likelihood in the dataset. In contrast,
our approach does not impose constraints but rather guides the agent.
Specifically, during each step of the decision-making process, we generate
multiple action components to supplement the original action produced by a
pre-trained offline RL algorithm. This encourages the agent to select actions
that are more prevalent in the dataset and transition to states with higher
likelihoods, mitigating inaccuracies caused by distribution shifts.
Additionally, if the agent encounters a low-likelihood state, our method
provides guidance to navigate away from regions lacking sufficient support in
the training data.

Figure 1: The Conservative Denoising Score-based Algorithm (CDSA) leverages
conservatism-related knowledge to enhance the performance of offline RL
algorithms. As depicted in (a), the original RL algorithm generates actions
based on the current state to interact with the environment. To address the
distribution shift problem, we propose to generate auxiliary actions based on
the current action-state pair, guiding the entire trajectory towards high-
density regions of the training dataset. This is illustrated in (b), where
CDSA generates two action components, utilizing conservatism-related knowledge
acquired from the training dataset, to be added to the action generated from a
pre-trained policy $\pi$.

Combining these altogether, we propose the Conservative Denoising Score-based
Algorithm (CDSA), which does not intervene in the training process of the
original offline RL algorithm. CDSA adjusts the generated actions during
testing while avoiding excessive interference with the original decisions. It
also proactively guides the trajectory into regions of higher density in the
training dataset as shown in Figure 1. Our idea is similar to the approach
used in Lyapunov Density Models (LDM) [15], which employs strong theoretical
constraints and model-based planning methods to keep the agent confined to
high-density regions of the training dataset. However, in the LDM approach,
the density of early termination points must be manually labeled to maintain
low density for specific experiments (as outlined in section D.2 of [15]).
This introduces subjective human knowledge and imposes stringent constraints
that restrict the algorithm’s versatility. In contrast, our methodology
operates without such limitations. Moreover, while LDM exclusively deals with
scenarios within the training data distribution, our approach also
incorporates guiding the agent outside the training data distribution.
Furthermore, it is worth noting that while LDM trains a Dynamics Model and
performs MPC, recurrent use of the dynamics model may result in accumulating
inference errors. To address the adverse effects stemming from network
uncertainty, we adopt a strategy where, in each step, we employ the inverse
dynamics model for inference only once. This practice effectively minimizes
errors.

Our experiments demonstrate that CDSA directly enhances the performance of
Offline RL baseline algorithms and can be applied to different algorithms
without any fine-tuning or new conservatism-related knowledge learning. The
baseline algorithms, IQL [17] and POR [49], exhibit improvements of 12.7% and
5.2% respectively when enhanced with CDSA on d4rl dataset. We also conducted
supplementary experiments, including the ablation study on the effect of
auxiliary actions in Appendix C.

##  2 Related Work

###  2.1 Offline RL

Offline (or batch) RL aims to learn a policy using a fixed dataset collected
by some unknown behavior policies [20, 21]. The critical challenge in offline
RL is the distribution shift [9, 18], where the agent overestimates and
prefers the out-of-distribution (OOD) actions, with the result that it
performs poorly. Currently, there are various solutions to this problem
involving constraining the learned policy to be closer to the original
behavior policy [8, 18, 45], regularizing the critic learning to be more
pessimistic with OOD actions [17, 19, 50], adopting model-based approaches
[51, 6, 24, 52], leveraging uncertainty measurement [3, 48, 2], importance
sampling [39, 23, 10], etc. In particular, some works learn the density model
of the training data to help constrain the agent in distribution [32, 28, 9,
18]. We mainly draw lessons from these works and propose to learn a gradient
field, which can solve the bootstrapping error problem more flexibly.

###  2.2 Risk-averse RL

In risk-averse reinforcement learning, the goal is to optimize some risk
measure of the returns. The most common risk-averse measures are the Value-at-
Risk (VaR) and Conditional Value-at-Risk (CVaR) [33], which use quantiles to
parameterize the policy return distribution. There are also other measures
such as the Wang measure [44], the mean-variance criteria [30], and the
cumulative probability weighting (CPW) metric [40]. However, our goal is not
to optimize these criteria but to use them as an evaluation indicator of the
conservatism of algorithms. Our work mainly focuses on VaR and uses it as a
measure to evaluate the risk-averse ability of our algorithm.

###  2.3 Scored-based model

Recently, score-based generative models [36, 37, 42] have received much
attention. The main idea of the score-based model is that the probability
distribution of real data is represented by score [22], a vector field, which
points to the direction where the data is most likely to increase. Leveraging
the learned score function as a prior, we can perform the Langevin Markov
chain Monte Carlo sampling [46] to generate the desired data from random noise
[43]. The score-based model has been successful at generating data from
various modalities such as images [13, 37], audio [16] and graphs [31]. Some
methods introduce score-based models into RL [47], which try to use the score-
based model to solve the problem of object rearrangement. It only considers
how to improve the probability of the states in the original distribution and
must participate in the training process of the baseline algorithm. In our
work, we use the loss from denoising score matching [43], which is more
concise than the original score-based method.

##  3 Preliminaries

We consider the Markov Decision Process (MDP) setting [38] with continuous
states $s\in\mathcal{S}$, continuous actions $a\in\mathcal{A}$, transition
probability distribution function $P(\cdot|s,a)$, reward function $r(s,a)$,
initial state distribution $\rho$ and discount factor $\gamma$. The aim of RL
is to learn a policy $\pi(a|s)$ that maximizes the cumulative discounted
returns

$\displaystyle\pi^{*}=\underset{\pi}{\arg\max\
}\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r\left(s_{t},a_{t}\right)\right].$
(1)

We consider the offline settings in our work. In the offline setting, we only
have access to a fixed dataset $D=\\{(s,a,r,s^{\prime})\\}$ consisting of
trajectories collected by different policies. The agent cannot directly
interact with the real environment and will encounter extrapolation errors
when visiting OOD states or taking OOD actions.

The score [22] of probability density $p_{\text{data}}(\mathbf{x})$ is
commonly defined as $\nabla_{\mathbf{x}}\log p_{\text{data}}(\mathbf{x})$.
Score matching [14] is able to estimate $\nabla_{\mathbf{x}}\log
p_{\text{data}}(\mathbf{x})$ without training a model to estimate
$p_{\text{data}}(\mathbf{x})$. The objective of score matching model minimizes

$\displaystyle\frac{1}{2}\mathbb{E}_{p_{\text{data}}(\mathbf{x})}\left[\left\|s_{\theta}({\mathbf{x}})-\nabla_{\mathbf{x}}\log
p_{\text{data}}(\mathbf{x})\right\|_{2}^{2}\right].$ (2)

Once the score function is known, the approximate samples for
$p_{\text{data}}(\mathbf{x})$ can be generated by Langevin dynamics. The
Langevin method recursively computes the following

$\displaystyle\mathbf{x}_{t}\leftarrow\mathbf{x}_{t-1}+\alpha\nabla_{\mathbf{x}}\log
p_{\text{data}}(\mathbf{x}_{t-1})+\sqrt{2\alpha}z_{t},$ (3)

where $z_{t}\sim N(0,I)$ and $\alpha$ is a constant. The noise added to the
equation is to prevent multiple data points from mapping to the same location
within the distribution. When $\alpha$ is small enough and the number of
iterations is large enough, the distribution of $\mathbf{x}_{t}$ is considered
to be the same as $p_{\text{data}}(\mathbf{x})$. The $\nabla_{\mathbf{x}}\log
p_{\text{data}}(\mathbf{x})$ in this formula is substituted by $s_{\theta}(x)$
since the score network is a good estimation of $\nabla_{\mathbf{x}}\log
p_{\text{data}}(\mathbf{x})$.

Suppose states and actions obey an unknown probability data distribution
$p_{\text{data}}(s,a)$ in the environment. The dataset consists of i.i.d.
samples $\\{(s,a)_{i}\ s\in S,a\in A\\}_{i=1}^{N}$ from
$p_{\text{data}}(s,a)$. Suppose we have a pre-trained offline policy
$\pi(a|s)$. Our goal is to make the action more conservative, in other words,
to increase the probabilities of $p_{\text{data}}(s,a)$.

##  4 Methodology

We use offline datasets that consist of trajectories sampled from any unknown
policy to generate gradient fields for action correction. We modify the
original action $a_{o}$ by introducing two action correction terms associated
with the action and the state respectively: $a\leftarrow
a_{o}+K_{1}*a_{1}+K_{2}*a_{2}$, where $a_{o}$ is the original action sampled
from the policy $\pi$, $a_{1}$ and $a_{2}$ encourages the agent to favor high
likelihood regions, $K_{1}$ and $K2$ are hyperparameters. The challenge lies
in designing $a_{1}$ and $a_{2}$ given the unknown grounded distribution
$p_{\text{data}}(s,a)$. We draw inspiration from the score matching method to
tackle this issue and propose our solution. We do not learn an extra critic or
actor network but learn gradient fields to make the agent in or close to
distribution.

###  4.1 Learning the Density Gradient Fields from Data

We consider the situation where the offline dataset contains a large fraction
of trajectories, and we refer to the probability distribution of state-action
pairs in this dataset as $p_{\text{data}}(s,a)$. Our goal is to get our
trajectory as close to the distribution $p_{\text{data}}(s,a)$ of the dataset
as possible so that the agent can make better decisions. To make the agent in
distribution, we need to find directions to increase the log-likelihood of the
density for both state and action, and the fastest direction is
$\nabla_{(s,a)}\log p_{\text{data}}(s,a)$. What we want to do is to generate
auxiliary actions based on the current action-state pair to increase the
probability of the entire trajectory within the distribution, which is shown
in Figure 1. To find an approach to achieve this goal, we refer to the
denoising score-matching model [13] and use networks to find these directions.

We adapt the approach of approximating the gradient of points from the
denoising score-matching model to offline RL. While this model can easily
learn the gradient of points since the coordinates are independent, obtaining
the gradient of state-action pairs in RL settings is challenging due to the
dependency between states and actions. To address this, we utilize a score-
matching model to learn the gradients of actions and states separately.

Firstly, for the gradient of the action, we consider learning the gradient by
using a network $g_{\theta}({s,a})$ to fit $\nabla_{a}\log p_{\text{data
}}(s,a)$ where $\theta$ is the parameters of this network. To train this
network, we adopt the denoising score-matching objective [43], which
guarantees a reasonable estimation of the score. For simplicity, we define
$\mathbf{x}=(s,a)$ as current state-action pair and define
$\mathbf{\tilde{x}}=(\tilde{s},\tilde{a})$ as the noised state-action pair.
The pre-specified noise distribution
$q_{\sigma}(\mathbf{\tilde{x}|\mathbf{x}})$ is used to perturb $\mathbf{x}$
and the target of the network is to learn the score of the perturbed target
distribution. The loss of the network is

$\displaystyle\mathcal{L}_{\theta}=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|g_{\theta}(\tilde{\mathbf{x}})-\nabla_{\tilde{a}}\log
q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})\right\|_{2}^{2}\right].$ (4)

The optimal network satisfies $g_{\theta^{*}}(\mathbf{x})=\nabla_{a}\log
q_{\sigma}(\mathbf{x})$ and $\nabla_{a}\log
q_{\sigma}(\mathbf{x})\approx\nabla_{a}\log p_{\text{data }}(\mathbf{x})$.
When the loss is small enough, it can be thought that
$g_{\theta^{*}}(\mathbf{x})\approx\nabla_{a}\log p_{\text{data
}}(\mathbf{x})$. When we choose the pre-specified noise distribution as the
normal distribution, the relationship between original data $\mathbf{x}=(s,a)$
and perturbed data $\mathbf{x}=(\tilde{s},\tilde{a})$ is

$\displaystyle\tilde{s}=s,\tilde{a}\sim N(a,\sigma I).$ (5)

Figure 2: An example comparing CDSA control and common control. CDSA controls
more closely to areas of distribution for action decision making.

Directly optimizing Eq. (4) is difficult because we do not have the direct
access to the $\nabla_{\tilde{a}}\log
q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})$. Thanks to the help of diffusion
score-based model, we show in the following lemma that the loss can be
rewritten to a simpler form:

######  Lemma 1.

The loss $\mathcal{L}_{\theta}$ in Eq. (4) is equivalent to the following
loss:

$\displaystyle\mathcal{L}_{\theta}=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|g_{\theta}(\mathbf{x}+\sigma\mathbf{z})+\frac{z}{\sigma}\right\|_{2}^{2}\right],$
(6)

where $\mathbf{z}=(0,z)$ and $z\sim N(0,I)$.

The proof of the Lemma 1 can refer to Appendix A.1. When the loss converges,
$g_{\theta}(\mathbf{x})\approx\nabla_{a}\log p_{\text{data }}(\mathbf{x})$ can
be used to modify the action to make it more conservative.

Secondly, for the gradient of the state, we consider learning the gradient by
using a network $h_{\varphi}(\mathbf{x})$ to approximate $\nabla_{s}\log
p_{\text{data }}(\mathbf{x})$. The approach employed for learning the gradient
of states is analogous to that of learning the gradient of actions, wherein
the perturbed data can be expressed as:

$\displaystyle\tilde{s}\sim N(s,\sigma I),\tilde{a}=a.$ (7)

The loss of this network can be expressed as

$\displaystyle\mathcal{L}_{\varphi}=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|h_{{\varphi}}(\mathbf{x}+\sigma\mathbf{z^{\prime}})+\frac{z^{\prime}}{\sigma}\right\|_{2}^{2}\right],$
(8)

where $\mathbf{z^{\prime}}=(0,z^{\prime})$ and $z^{\prime}\sim N(0,I)$.
$h_{\varphi}(\mathbf{x})$ has a reasonable estimation of $\nabla_{s}\log
p_{\text{data }}(\mathbf{x})$ when the loss function converges.

As mentioned earlier, the quickest way to increase $p_{\text{data }}(s,a)$ is
through $\nabla_{(s,a)}\log p_{\text{data}}(s,a)$. However, since the current
state remains fixed, we cannot change it while we can change the action. Thus,
we initially planned to use a forward model $F(s^{\prime}|s,a)$ to predict the
next state based on the current state and action. This simple model-based
trick could help us measure the density of the next state in the training
dataset and calculate the gradient of the density function. However, the
prediction accuracy of this forward model outside the support of the training
dataset cannot be guaranteed. Therefore, it becomes challenging for the
resulting gradient to effectively guide the changes in action.

To address these issues, we devised a method leveraging an inverse dynamic
model. Our aim is to increase the density $p_{\text{data}}$ of agent states in
the dataset. An intuitive approach is to utilize gradient descent to identify
the direction of increasing $p_{\text{data}}$, thus necessitating the use of
gradients of $p_{\text{data}}$ with respect to states $s$ and actions $a$,
i.e., $\Delta s=\nabla_{s}\log p_{\text{data}}(x),\Delta a=\nabla_{a}\log
p_{\text{data}}(x)$. In this way, we can adjust the sampeld action $a$ and
encourage the agent to come closer to $s+\Delta s$. In our experiments, we
found that employing $\Delta{s}=\nabla_{s}\log p_{\text{data}}(\mathbf{x})$
and integrating an inverse dynamic model $I_{\phi}(s,\tilde{s})$ to generate
an action as an auxiliary action component can yield more stable and effective
results. This method demonstrates greater reliability compared to using
$\Delta{s^{\prime}}=\nabla_{s^{\prime}}\log
p_{\text{data}}(F(\mathbf{x}),\cdot)$, as it relies on only one model-based
prediction network, as opposed to the original concept. The efficacy of these
auxiliary action components is substantiated through experiments.

The loss of the inverse dynamic network to learn this knowledge, which is
learned by imitation learning [12, 1], is:

$\displaystyle\mathcal{L}_{\phi}=\mathbb{E}_{s,a,\tilde{s}\sim\mathcal{D}}\left\|I_{\phi}(s,\tilde{s})-a\right\|_{2}^{2},$
(9)

where $I_{\phi}(s,\tilde{s})$ is the inverse dynamic network that learns
action from state $s$ to state $\tilde{s}$. When the inverse dynamic network
is well-trained, we can input the original state and noised state into this
network and get the action with good estimation to change $\Delta{s}$.

Algorithm 1 Training CDSA from offline data

1: Input: Dataset $D$, iterations $T$, learning rate
$\eta_{\theta},\eta_{\varphi},\eta_{\phi}$

2: Initialize parameters $\theta,\varphi,\phi$

3: for $t=1,2,..T$ do

4: Sample $\left(s_{t},a,s_{t+1}\right)\sim D$

5: Sample $z,z^{\prime}\sim N(0,I)$

6: Calculate $\mathcal{L}_{\theta}$ by Eq. (6); update
$\theta\leftarrow\theta-\eta_{\theta}\nabla\mathcal{L}_{\theta}$

7: Calculate $\mathcal{L}_{\varphi}$ by Eq. (8); update
$\varphi\leftarrow\varphi-\eta_{\varphi}\nabla\mathcal{L}_{\varphi}$

8: Calculate $\mathcal{L}_{\phi}$ by Eq. (9); update
$\phi\leftarrow\phi-\eta_{\phi}\nabla\mathcal{L}_{\phi}$

9: end for

10: Output: networks
$g_{\theta}(s,a),h_{\varphi}(s,a),F_{\phi}\left(s,s^{\prime}\right)$

###  4.2  Control with CDSA

With the essential knowledge covered, we can now utilize the gradient fields.
During the sampling process, we obtain the current state $s$ and an original
action $a_{\text{o}}$ from a baseline algorithm such as CQL, IQL, and others.
From the gradient field, we extract $\Delta{s}$ and $\Delta{a}$, allowing us
to compute $a_{1}=\Delta{a}\approx g_{\theta}(s,a_{o})$ and
$a_{2}=I_{\phi}(s,s+\Delta{s})\approx I_{\phi}(s,s+h_{\varphi}(s,a_{o}))$.
Here, $\Delta{a}$ and $\Delta{s}$ are obtained from the score model,
representing the directions in action and state space that maximize dataset
density at the current state-action pair. We directly set $a_{1}$ as
$\Delta{a}$ and then feed $\Delta{s}$ into the Inverse dynamics model
$I_{\phi}(\cdot,\cdot)$ to obtain $a_{2}$. $a_{1}$ and $a_{2}$ represent the
action correction components obtained from both action and state perspectives.
$a_{1}$ We add these two items to the original action $a_{o}$ linearly, which
can be written as

$\displaystyle a=a_{\text{o}}+\delta a,$ (10)

where $\delta a=K_{1}g_{\theta}(s,a_{o})+K_{2}I_{\phi}(s,s+h_{\varphi})$ and
$K_{1},K_{2}$ are two hyperparameters. However, this auxiliary action can only
temporarily and quickly increase the probability of the current state-action
pair within the distribution, without considering future situations.
Therefore, drawing on the idea of generative model, After using the model to
obtain the corrected action $a$, we repeatedly put $(s,a)$ into the model to
generate a new $a$, and constantly improve the probability of $(s,a)$ in the
distribution.

(a) environment

(b) search

(c) CQL

Figure 3: Experiments in the Risky PointMass environment are depicted in (a),
where the red circle represents the risky zone, leading to negative rewards if
occupied. The agent begins at the blue point, targeting the purple circle. In
(b) and (c), we employ simple shortest path finding and CQL as baseline
algorithms, demonstrating the corrective impact of our method. CDSA learns
from an offline dataset generated by a pretrained CODAC agent, following the
identical training procedure outlined in its official repository [27]. Maroon
trajectories illustrate baseline algorithm results, while black trajectories
depict agents equipped with CDSA. Our method displays two sets of direction
arrows: green indicating baseline algorithm directions, and blue indicating
conservative auxiliary action directions from CDSA. The arrow length signifies
action magnitude. With CDSA modifications, the agent effectively avoids the
risky region. Algorithm 2 Control with CDSA

1: Input: Environment $E$, pre-trained policy $\pi(a\mid s)$, networks
$g_{\theta}(s,a),h_{\varphi}(s,a),I_{\phi}\left(s,s^{\prime}\right)$,
hyperparameters $\quad K_{1},K_{2}$

2: Get initial State $s$ from $E$, set $d$ as False

3: while not $d$ do

4: Sample original action $a_{o}$ from $\pi(a\mid s)$

5: Get safety action component $a_{1}=g_{\theta}(s,a)$

6: Get safety action component $a_{2}=I_{\phi}\left(s,s+h_{\varphi}(s)\right)$

7: $a\leftarrow a_{o}+K_{1}*a_{1}+K_{2}*a_{2}$

8: Roll out $a$ and get $\left(s^{\prime},r,d\right)$

9: end while

10: Set $s\leftarrow s^{\prime}$

The complete algorithm is presented in Algorithm 1 and 2. Algorithm 1 trains
CDSA using the input dataset and generates three networks. One network serves
as the inverse dynamic model, while the other two networks capture information
about the gradient fields. In Algorithm 2, these networks are utilized to
adjust the agent’s actions in the given environment. The pre-trained policy
used in Algorithm 2 can be obtained from any RL baseline algorithm, and
$K_{1}$ and $K_{2}$ are hyperparameters that control the scopes of auxiliary
action components.

Table 1: Average normalized scores of algorithms. We chose 7 popular offline
RL algorithms to evaluate the effectiveness of our algorithm. The scores are
taken over the final 20 evaluations for MuJoCo and 100 evaluations for
AntMaze. CDSA (IQL) and CDSA (POR) achieved the highest scores in 12 out of 15
tasks. The abbreviations in the table correspond to the following meanings: r
= random, m = medium, e = expert, u = umaze, l = large, p = play, d = diverse.

Dataset | One-step | 10%BC | TD3+BC | CQL | CODAC | IQL | POR | CDSA (IQL) | CDSA (POR)
---|---|---|---|---|---|---|---|---|---
hopper-r | 5.2 | 4.2 | 8.5 | 7.9 | 11.0 | 10.8 | 12.5 |  30.9$\pm$0.19 |  31.9$\pm$0.14
hopper-m | 59.6 | 56.9 | 59.3 | 53.0 | 70.8 | 62.1 | 89.4 |  65.4 $\pm$5.38 |  90.6$\pm$10.47
hopper-m-e | 103.3 | 110.9 | 98.0 | 105.6 | 112.0 | 109.5 | 104.0 |  112.0$\pm$1.54 |  106.5$\pm$3.21
halfcheetah-r | 3.7 | 5.4 | 11.0 | 17.5 | 34.6 | 16.8 | 17.2 |  17.0$\pm$0.21 |  17.5$\pm$0.91
halfcheetah-m | 48.4 | 42.5 | 48.3 | 47.0 | 46.3 | 48.5 | 48.1 |  49.1 $\pm$0.38 |  48.1$\pm$1.13
halfcheetah-m-e | 93.4 | 92.9 | 90.7 | 75.6 | 70.4 | 79.0 | 81.6 |  81.1$\pm$1.86 |  85.0$\pm$1.20
walker2d-r | 5.6 | 6.7 | 1.6 | 5.1 | 18.7 | 5.9 | 7.6 |  7.4$\pm$0.20 |  7.9$\pm$0.17
walker2d-m | 81.8 | 75.0 | 83.7 | 73.3 | 82.0 | 79.6 | 82.1 |  80.2$\pm$1.61 |  83.8$\pm$4.12
walker2d-m-e | 113.0 | 109.0 | 110.1 | 113.8 | 106.0 | 107.2 | 111.6 |  113.9$\pm$0.29 |  114.4$\pm$1.63
Mujoco Average | 57.1 | 55.9 | 56.8 | 55.4 | 61.3 | 57.7 | 61.6 |  61.9$\pm$1.3 |  65.1$\pm$2.6
antmaze-u | 64.3 | 62.8 | 78.6 | 74.0 | 52.8 | 76.4 | 88.4 |  89.4$\pm$5.55 |  93.4$\pm$2.07
antmaze-u-d | 60.7 | 50.2 | 71.4 | 84.0 | 38.4 | 63.2 | 80.8 |  86.6$\pm$4.03 |  83.0$\pm$11.85
antmaze-m-p | 0.3 | 5.4 | 10.6 | 61.2 | 0.0 | 65.4 | 88.2 |  72.2$\pm$17.05 |  93.8$\pm$2.05
antmaze-m-d | 0.0 | 9.8 | 3.0 | 53.7 | 0.0 | 61.0 | 88.0 |  72.4 $\pm$17.51 |  89.2$\pm$4.76
antmaze-l-p | 0.0 | 0.0 | 0.2 | 15.8 | 1.4 | 38.0 | 64.6 |  53.4$\pm$6.69 |  69.6$\pm$12.70
antmaze-l-d | 0.0 | 0.0 | 0.0 | 14.9 | 3.8 | 35.4 | 67.4 |  37.0$\pm$4.06 |  70.4$\pm$7.23
AntMaze Average | 20.9 | 21.4 | 27.3 | 50.6 | 16.1 | 56.6 | 79.6 |  68.5$\pm$9.1 |  83.23$\pm$6.8

##  5 Experiments

We present empirical evaluations of CDSA in this section. We first conducted
experiments on environment Risky PointMass and provided visualized results. We
then demonstrate the effectiveness of CDSA in offline D4RL [7] MuJoCo and
AntMaze datasets. We also verify the generalizability of CDSA with different
tasks in the same environment. Due to space constraints, we place the ablation
study in the appendix.

###  5.1 Risky PointMass

Consider an ant robot with the purpose of fast travelling from a random
beginning condition to a purple circle. Using red circles, significant
expenses are triggered with a low chance, increasing hazards. To train the
agent, we use an offline dataset, which is the replay buffer from a CODAC
agent. The result is shown in Figure 3. Because the baseline algorithm
incorrectly estimates unfamiliar scenes, it chooses to cross dangerous areas
to reach the end point. Using CDSA, we can successfully get the agent out of
dangerous areas and into familiar situations to make decisions.

Figure 4: Results of VaR(the $n^{th}$ percentile of cumulative sorted reward).
Here only shows the results of CDSA (IQL) (the blue lines) and IQL (the green
lines) in the D4RL benchmark. The results of the POR and CDSA (POR) algorithms
are shown in Appendix B.1

###  5.2 D4RL offline tasks

To verify the effectiveness of CDSA in offline scenarios, we evaluate our
approach on D4RL MuJoCo and AntMaze datasets. The D4RL MuJoCo benchmark
consists of datasets collected by SAC agents that have different performances
(random, medium, and expert, etc.) in Hopper, HalfCheetah, and Walker2d
environments. The AntMaze environment requires the agent to manipulate a
quadruped robot to find the target point in a maze. There are datasets (umaze,
medium, and large) divided by the size of the maze. For all datasets we use
the "v2" version.

In prospect, CDSA should be able to increase the performance of the given
policy, hence increasing the final score and VaR (value at risk) in these
datasets.

#### Baselines.

We choose the well-known offline algorithms IQL [17] and POR [49] as our
baseline algorithm. In our experiment, we train models using the IQL and POR
methods, each with 5 seeds, and then combine these models with CDSA. We
compare the normalized scores with popular RL algorithms, including One-step
[4], 10%BC [5], TD3+BC [8], CQL [19], CODAC [27], IQL and POR. The
experimental details are included in Appendix B.1.

Figure 5: The results of risky transportation experiment. (a) shows the map of
the environment.  is the start point and  is the target point, the
trajectories of the agent are represented by several lines that increase
saturation with the number of steps.  is river,  is mountain, and  is ice,
which are risky regions. (b) shows the states in the offline dataset. The
black color represents $\sum_{a}p_{\text{data}}(s,a)=0$ and the white color
represents $\sum_{a}p_{\text{data}}(s,a)$ is non-zero in the dataset. (c) is
the gradient field of states learned from CDSA. We only show the gradient
field of states since the gradient field of actions is hard to present. (d) is
the results of SAC and CDSA (SAC). After employing CDSA to maintain the agent
within the known region, the agent exhibits a higher degree of risk aversion.
(e) is the task that requires the agent to bring goods to the target point,
is the region where the goods are placed. (f) shows the results after adding
the airport to this environment,  is the airport area. In these two tasks, we
use the CDSA models learned from the path finding task without any fine-
tuning. The agent avoids all risky regions after using CDSA.

#### Implementation details.

We run the baseline algorithms IQL and POR with their official codes for 1M
gradient steps. For CDSA, we train two score models and the inverse dynamic
model on each dataset for 10,000 gradient steps. Due to space limit, we defer
the detailed hyperparameter setup as well as the setup of action correction
coefficients $K_{1},K_{2}$ on each dataset to table 3 and table 2 in the
appendix. All algorithms are trained with random seeds 0-4. We report the
average performance of each method post-training.

#### Results.

The outcomes for all datasets are summarized in Table 1. Significantly, CDSA
(POR) and CDSA (IQL) exhibit superior performance across 12 tasks compared to
the baseline algorithms, underscoring the efficacy of CDSA in enhancing
performance across most datasets. In MuJoCO datasets, CDSA performs
exceptionally well in the hopper random dataset (186.1% for IQL and 155.2% for
POR), with marginal improvements in most datasets (ranging from 0% to 25.4%,
averaging 6.45%). We attribute the particular advantage of CDSA in the hopper
random dataset to its potentially poor data quality and the relatively simple
nature of the hopper controlled by the agents. CDSA’s ability to incorporate
safety action components aids in stabilizing the robot, thereby yielding
higher healthy rewards (reward obtained for maintaining the hopper healthy at
each time step). In expert datasets, CDSA shows only limited enhancement,
possibly due to the already conservative and high-quality nature of expert
datasets, where further conservatism provided by CDSA might not yield
substantial improvements. In antmaze datasets, CDSA demonstrates significant
improvements across all datasets (ranging from 1.4% to 40.5%, averaging
11.4%), indicating its efficacy in environments with simpler dynamics. We
posit that CDSA’s success across these datasets stems from the increased
likelihood of action-state pairs within the distribution.

#### Risky-averse Evaluation.

To verify that CDSA can reduce the risk of entering hazardous zones, we
visualize the VaR of IQL and CDSA (IQL) as shown in Figure 4. Compared to IQL,
CDSA (IQL) improves VaR significantly in almost every percentile. We can also
see that VaR increases more when the quantile is smaller since the
trajectories of high reward are safe enough and hardly cross risky regions.
Overall, the auxiliary action components help the agent take safer action,
lowering the probability of risky situations and increasing the cumulative
reward in almost every trajectory.

###  5.3 Risky transportation

We use CDSA in the Risky transportation environment. The task of the agent is
to find a path from the start point to the target point. We call this task
path finding task. There are risky regions such as mountains, rivers, and ice
roads, where the agent is at risk of accidents. We give a large negative
reward with a small probability in these areas to indicate that the agent has
an accident. Before reaching the target point, the agent receives a negative
reward proportional to the distance between the agent and the target point.

We use CDSA to learn safe gradient fields of a safe dataset whose trajectory
does not contain any risk regions in this environment. We use SAC [11] as our
baseline algorithms. As depicted in Figure 5, most of the trajectories pass
through risky regions to reach the target point quickly. After adding
auxiliary safety action components of CDSA in each step, the agent can avoid
all risky regions to reach the destination.

To verify that CDSA is effective of generalization, we design two more tasks.
The first task is to bring goods to the target point. The agent must reach the
point of goods and then go to the target point. In the second task, we add an
airport to this environment. Using an airplane, the agent can land near the
target point. We only employ the CDSA models trained for the path finding task
and do not train any new models for these additional tasks. Figure 5 shows
that in the first task, the agent finds the goods on the shortest path and
travels to the target point despite the presence of risky regions. The agent
tends to go to the airport and take a plane to the location near the target
point and then goes to the target point in the shortest path in the second
task. After adding auxiliary actions generated from CDSA, the trajectories are
much safer, and the agent can completely avoid entering risky regions. It is
clear that the CDSA model trained in the same environment can be applied to
other tasks, which demonstrates the generalization ability of CDSA.

##  6 Discussion

Our work introduces the CDSA algorithm, which learns gradient fields from data
and utilizes them to acquire auxiliary actions. These auxiliary actions guide
state-action pairs towards high-density regions within the dataset
distribution, mitigating exposure to unfamiliar states. Since CDSA focuses
solely on learning gradient fields from data, independent of RL baseline
algorithms, it seamlessly integrates with various algorithms such as CQL, IQL,
and POR. Our experiments in offline settings demonstrate that our method
effectively navigates away from hazardous areas and makes decisions within
familiar scenarios within the dataset distribution. Combining baseline
algorithms with CDSA leads to improved performance on D4RL datasets across
various qualities. Notably, CDSA (IQL) and CDSA (POR) exhibit superior
performance in 12 tasks. Furthermore, employing CDSA significantly enhances
the Value at Risk (VaR) of baseline algorithms, underscoring our method’s
risk-averse capability. In the Risky Transportation environment, we visualize
and validate the generalizability of CDSA across different tasks within the
same environment.

While our method shows promising results, it is essential to acknowledge its
limitations. Specifically, CDSA’s effectiveness is confined to scenarios with
continuous action spaces, limiting its applicability in discrete action
spaces. Additionally, accurately determining hyperparameters $K_{1}$ and
$K_{2}$ poses a challenge, requiring careful balancing to achieve optimal
performance. However, setting these hyperparameters to the same value often
yields satisfactory results, reducing the need for extensive tuning efforts.
An area for future exploration involves developing an automated mechanism for
adjusting $K_{1}$ and $K_{2}$.

By using the ack environment to insert your (optional) acknowledgements, you
can ensure that the text is suppressed whenever you use the doubleblind
option. In the final version, acknowledgements may be included on the extra
page intended for references.

## References

  * Abbeel and Ng [2004] P. Abbeel and A. Y. Ng.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the twenty-first international conference on Machine learning_ , page 1, 2004.
  * An et al. [2021] G. An, S. Moon, J.-H. Kim, and H. O. Song.  Uncertainty-based offline reinforcement learning with diversified q-ensemble.  _Advances in neural information processing systems_ , 34:7436–7447, 2021.
  * Bai et al. [2022] C. Bai, L. Wang, Z. Yang, Z. Deng, A. Garg, P. Liu, and Z. Wang.  Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning.  _arXiv preprint arXiv:2202.11566_ , 2022.
  * Brandfonbrener et al. [2021] D. Brandfonbrener, W. Whitney, R. Ranganath, and J. Bruna.  Offline rl without off-policy evaluation.  _Advances in Neural Information Processing Systems_ , 34:4933–4946, 2021.
  * Chen et al. [2021] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Diehl et al. [2021] C. Diehl, T. Sievernich, M. Krüger, F. Hoffmann, and T. Bertran.  Umbrella: Uncertainty-aware model-based offline reinforcement learning leveraging planning.  _arXiv preprint arXiv:2111.11097_ , 2021.
  * Fu et al. [2020] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Fujimoto and Gu [2021] S. Fujimoto and S. S. Gu.  A minimalist approach to offline reinforcement learning.  _Advances in neural information processing systems_ , 34:20132–20145, 2021.
  * Fujimoto et al. [2019] S. Fujimoto, D. Meger, and D. Precup.  Off-policy deep reinforcement learning without exploration.  In _International conference on machine learning_ , pages 2052–2062. PMLR, 2019.
  * Gelada and Bellemare [2019] C. Gelada and M. G. Bellemare.  Off-policy deep reinforcement learning by bootstrapping the covariate shift.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3647–3655, 2019.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pages 1861–1870. PMLR, 2018.
  * Ho and Ermon [2016] J. Ho and S. Ermon.  Generative adversarial imitation learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Hyvärinen and Dayan [2005] A. Hyvärinen and P. Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Kang et al. [2022] K. Kang, P. Gradu, J. Choi, M. Janner, C. Tomlin, and S. Levine.  Lyapunov density models: Constraining distribution shift in learning-based control, 2022.
  * Kong et al. [2020] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ , 2020.
  * Kostrikov et al. [2021] I. Kostrikov, A. Nair, and S. Levine.  Offline reinforcement learning with implicit q-learning.  _arXiv preprint arXiv:2110.06169_ , 2021.
  * Kumar et al. [2019] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine.  Stabilizing off-policy q-learning via bootstrapping error reduction.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Kumar et al. [2020] A. Kumar, A. Zhou, G. Tucker, and S. Levine.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Lange et al. [2012] S. Lange, T. Gabel, and M. Riedmiller.  Batch reinforcement learning.  In _Reinforcement learning_ , pages 45–73. Springer, 2012.
  * Levine et al. [2020] S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Liu et al. [2016] Q. Liu, J. Lee, and M. Jordan.  A kernelized stein discrepancy for goodness-of-fit tests.  In _International conference on machine learning_ , pages 276–284. PMLR, 2016.
  * Liu et al. [2019] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill.  Off-policy policy gradient with state distribution correction.  _arXiv preprint arXiv:1904.08473_ , 2019.
  * Lyu et al. [2022a] J. Lyu, X. Li, and Z. Lu.  Double check your state before trusting it: Confidence-aware bidirectional offline model-based imagination.  In _Thirty-sixth Conference on Neural Information Processing Systems_ , 2022a.
  * Lyu et al. [2022b] J. Lyu, X. Ma, X. Li, and Z. Lu.  Mildly conservative q-learning for offline reinforcement learning.  In _Thirty-sixth Conference on Neural Information Processing Systems_ , 2022b.
  * Lyu et al. [2023] J. Lyu, A. Gong, L. Wan, Z. Lu, and X. Li.  State advantage weighting for offline RL.  In _International Conference on Learning Representation tiny paper_ , 2023.  URL https://openreview.net/forum?id=PjypHLTo29v.
  * Ma et al. [2021] Y. Ma, D. Jayaraman, and O. Bastani.  Conservative offline distributional reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34:19235–19247, 2021.
  * McAllister et al. [2019] R. McAllister, G. Kahn, J. Clune, and S. Levine.  Robustness to out-of-distribution inputs via task-aware generative uncertainty.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 2083–2089. IEEE, 2019.
  * Mnih et al. [2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Namkoong and Duchi [2017] H. Namkoong and J. C. Duchi.  Variance-based regularization with convex objectives.  _Advances in neural information processing systems_ , 30, 2017.
  * Niu et al. [2020] C. Niu, Y. Song, J. Song, S. Zhao, A. Grover, and S. Ermon.  Permutation invariant graph generation via score-based generative modeling.  In _International Conference on Artificial Intelligence and Statistics_ , pages 4474–4484. PMLR, 2020.
  * Richter and Roy [2017] C. Richter and N. Roy.  Safe visual navigation via deep learning and novelty detection.  2017\.
  * Rockafellar and Uryasev [2002] R. T. Rockafellar and S. Uryasev.  Conditional value-at-risk for general loss distributions.  _Journal of banking & finance_, 26(7):1443–1471, 2002.
  * Schulman et al. [2015] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz.  Trust region policy optimization.  In _International conference on machine learning_ , pages 1889–1897. PMLR, 2015.
  * Silver et al. [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Song and Ermon [2019] Y. Song and S. Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Song et al. [2020] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Sutton et al. [1998] R. S. Sutton, A. G. Barto, et al.  Introduction to reinforcement learning.  1998\.
  * Sutton et al. [2016] R. S. Sutton, A. R. Mahmood, and M. White.  An emphatic approach to the problem of off-policy temporal-difference learning.  _The Journal of Machine Learning Research_ , 17(1):2603–2631, 2016.
  * Tversky and Kahneman [1992] A. Tversky and D. Kahneman.  Advances in prospect theory: Cumulative representation of uncertainty.  _Journal of Risk and uncertainty_ , 5(4):297–323, 1992.
  * Urpí et al. [2021] N. A. Urpí, S. Curi, and A. Krause.  Risk-averse offline reinforcement learning.  _arXiv preprint arXiv:2102.05371_ , 2021.
  * Vahdat et al. [2021] A. Vahdat, K. Kreis, and J. Kautz.  Score-based generative modeling in latent space.  _Advances in Neural Information Processing Systems_ , 34:11287–11302, 2021.
  * Vincent [2011] P. Vincent.  A connection between score matching and denoising autoencoders.  _Neural computation_ , 23(7):1661–1674, 2011\.
  * Wang [1996] S. Wang.  Premium calculation by transforming the layer premium density.  _ASTIN Bulletin: The Journal of the IAA_ , 26(1):71–92, 1996.
  * Wang et al. [2022] Z. Wang, J. J. Hunt, and M. Zhou.  Diffusion policies as an expressive policy class for offline reinforcement learning.  _arXiv preprint arXiv:2208.06193_ , 2022.
  * Welling and Teh [2011] M. Welling and Y. W. Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pages 681–688, 2011.
  * Wu et al. [2022] M. Wu, F. Zhong, Y. Xia, and H. Dong.  Targf: Learning target gradient field for object rearrangement.  _arXiv preprint arXiv:2209.00853_ , 2022.
  * Wu et al. [2021] Y. Wu, S. Zhai, N. Srivastava, J. Susskind, J. Zhang, R. Salakhutdinov, and H. Goh.  Uncertainty weighted actor-critic for offline reinforcement learning.  _arXiv preprint arXiv:2105.08140_ , 2021.
  * Xu et al. [2022] H. Xu, L. Jiang, J. Li, and X. Zhan.  A policy-guided imitation approach for offline reinforcement learning.  _arXiv preprint arXiv:2210.08323_ , 2022.
  * Yang et al. [2024] K. Yang, J. Tao, J. Lyu, and X. Li.  Exploration and anti-exploration with distributional random network distillation.  _arXiv preprint arXiv:2401.09750_ , 2024.
  * Yu et al. [2020] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma.  Mopo: Model-based offline policy optimization.  _Advances in Neural Information Processing Systems_ , 33:14129–14142, 2020.
  * Zhang et al. [2023] J. Zhang, J. Lyu, X. Ma, J. Yan, J. Yang, L. Wan, and X. Li.  Uncertainty-driven trajectory truncation for model-based offline reinforcement learning.  _ArXiv_ , abs/2304.04660, 2023.

##  Appendix A Proof

###  A.1 Proof of equivalent loss form

The loss of network $g_{\theta}(\tilde{x})$ is

$\mathcal{L}_{\theta}=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|g_{\theta}(\tilde{\mathbf{x}})-\nabla_{\tilde{a}}\log
q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})\right\|_{2}^{2}\right].$

When the noise distribution $q_{\sigma}(\tilde{\mathbf{x}})$ obey
$N(\mathbf{x},\sigma I)$, the partial derivative of $\log
q_{\sigma}(\tilde{\mathbf{x}})$ with respect to $a$ is

$\displaystyle\nabla_{\tilde{a}}\log
q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})$
$\displaystyle=\nabla_{\tilde{a}}\log\left(\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(\tilde{\mathbf{x}}-{x})^{2}}{2\sigma^{2}}}\right)$
$\displaystyle=\nabla_{\tilde{a}}\left(C-\frac{(\tilde{\mathbf{x}}-\mathbf{x})^{2}}{2\sigma^{2}}\right)$
$\displaystyle=-\frac{\partial\tilde{\mathbf{x}}}{\partial\tilde{a}}\frac{(\tilde{\mathbf{x}}-\mathbf{x})}{\sigma^{2}}$
$\displaystyle=-(0,1)\cdot\left(\frac{(\tilde{\mathbf{s}}-\mathbf{s})}{\sigma^{2}},\frac{(\tilde{\mathbf{a}}-\mathbf{a})}{\sigma^{2}}\right)$
$\displaystyle=-\frac{(\tilde{\mathbf{a}}-\mathbf{a})}{\sigma^{2}}.$

Using reparameterization trick, $\mathbf{\tilde{a}}$ can be expressed as
$\mathbf{\tilde{a}}=\mathbf{a}+\sigma z$ where $z$ follows the standard normal
distribution. $\mathbf{\tilde{x}}$ can be also represent as
$\mathbf{\tilde{x}}=\mathbf{x}+(0,\sigma z)$. The loss formula can be
converted into

$\displaystyle\mathcal{L}_{\theta}$
$\displaystyle=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|g_{\theta}(\tilde{\mathbf{x}})-\nabla_{\tilde{a}}\log
q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})\right\|_{2}^{2}\right]$
$\displaystyle=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|g_{\theta}(\tilde{\mathbf{x}})+\frac{(\tilde{\mathbf{a}}-\mathbf{a})}{\sigma^{2}}\right\|_{2}^{2}\right]$
$\displaystyle=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|g_{\theta}({\mathbf{x}+(0,\sigma
z)})+\frac{z}{\sigma}\right\|_{2}^{2}\right]$
$\displaystyle=\frac{1}{2}\mathbb{E}_{q_{\sigma}(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data
}}(\mathbf{x})}\left[\left\|g_{\theta}(\mathbf{x}+\sigma\mathbf{z})+\frac{z}{\sigma}\right\|_{2}^{2}\right].$

where $\mathbf{z}=(0,z)$ and $z\sim N(0,I)$.

##  Appendix B Experiment details

Our experiments were performed by using the following hardware and software:

  * •

GPUs: NVIDIA GeForce RTX 3090

  * •

Python 3.10.8

  * •

Numpy 1.23.4

  * •

Pytorch 1.13.0

  * •

Mujoco-py 2.1.2.14

  * •

Mujoco 2.3.1

  * •

D4RL 1.1

###  B.1 D4RL experiments

The VaR of POR and CDSA(POR) are shown in Figure 6. The blue lines represent
the result of the baseline algorithm with CDSA and the green lines show the
result of the baseline algorithm. We can see that CDSA can improve the VaR in
almost every dataset, especially when the percentile is small, verifying the
risk-averse effect of CDSA.

Figure 6: Results of CDSA (POR) and POR. Table 2: Hyperparameters of each dataset | IQL | POR
---|---|---
| K1 | K2 | K1 | K2
hopper-r | 0.1 | 1 | 0.1 | 0.5
hopper-m | 0.001 | 0.001 | 0.001 | 0.001
hopper-m-e | 0.001 | 0.001 | 0.001 | 0.001
halfcheetah-r | 0.001 | 0.005 | 0.003 | 0.03
halfcheetah-m | 0.005 | 0.05 | 0.01 | 0.01
halfcheetah-m-e | 0.01 | 0.01 | 0.003 | 0.03
walker2d-r | 0.03 | 0.3 | 0.00 | 0.01
walker2d-m | 0.03 | 0.003 | 0.005 | 0.01
walker2d-m-e | 0.1 | 0.01 | 0.00 | 0.01
antmaze-u | 0.5 | 0.1 | 0.1 | 0.1
antmaze-u-d | 0.1 | 0.5 | 0.05 | 0.05
antmaze-m-p | 0.1 | 0.3 | 0.1 | 0.1
antmaze-m-d | 0.3 | 0.3 | 0.05 | 0.05
antmaze-l-p | 0.3 | 0.3 | 0.1 | 0.1
antmaze-l-d | 0.1 | 0.1 | 0.03 | 0.03
Table 3: Hyperparameters of CDSA

| Name | Value
---|---|---
Architecture | Hidden layers of $g_{\theta}$ | 3
Hidden dim of $g_{\theta}$ | 32,128,32
Activation function of $g_{\theta}$ | LeakyReLU(0.1)
Hidden layers of $h_{\varphi}$ | 3
Hidden dims of $h_{\varphi}$ | 32,128,32
Activation function of $h_{\varphi}$ | LeakyReLU(0.1)
Hidden layers of $I_{\phi}$ | 3
Hidden dims of $I_{\phi}$ | 128
Activation function of $I_{\phi}$ | LeakyReLU(0.2)
Hyperparameters | Optimizer | Adam
Learning rate of $g_{\theta}$ | 3e-4
Learning rate of $h_{\varphi}$ | 3e-4
Learning rate of $I_{\phi}$ | 1e-3
Batch size | 256
Iteration number | 10000

##  Appendix C Ablation Study

We present an ablation study to verify the effect of auxiliary actions $a_{1}$
and $a_{2}$. We denote "CDSA w/o $a_{1}$" only uses action $a_{2}$ to correct
action $a_{o}$ and "CDSA w/o $a_{2}$" only uses auxiliary action $a_{1}$. The
results are shown in Figure 7.

We can see that the results of "CDSA w/o $a_{1}$" and "CDSA w/o $a_{2}$" are
better than the baseline algorithm, which proves the effectiveness of using
auxiliary action $a_{1}$ and $a_{2}$ solely. However, the performance of CDSA
is better than these two algorithms and the baseline algorithm in almost every
dataset, verifying it is better to use both together.

Figure 7: Results of ablation study.


# LIDA: Lightweight Interactive Dialogue Annotator

Edward Collins
Wluper Ltd.
London, United Kingdom
<EMAIL_ADDRESS>
Nikolai Rozanov
Wluper Ltd.
London, United Kingdom
<EMAIL_ADDRESS>
Bingbing Zhang
Wluper Ltd.
London, United Kingdom
<EMAIL_ADDRESS>

###### Abstract

Dialogue systems have the potential to change how people interact with
machines but are highly dependent on the quality of the data used to train
them. It is therefore important to develop good dialogue annotation tools
which can improve the speed and quality of dialogue data annotation. With this
in mind, we introduce LIDA, an annotation tool designed specifically for
conversation data. As far as we know, LIDA is the first dialogue annotation
system that handles the entire dialogue annotation pipeline from raw text, as
may be the output of transcription services, to structured conversation data.
Furthermore it supports the integration of arbitrary machine learning models
as annotation recommenders and also has a dedicated interface to resolve
inter-annotator disagreements such as after crowdsourcing annotations for a
dataset. LIDA is fully open source, documented and publicly available
111https://github.com/Wluper/lida.

Annotation Tool |  Turn/Dialogue Segmentation |  Classification Labels |  Edit Dialogues/Turns |  Recommenders |  Inter-Annotator Disagreement Resolution |  Language
---|---|---|---|---|---|---
LIDA |  YES |  YES |  YES |  YES |  YES |  PYTHON
INCEpTion Klie et al. (2018) |  NO |  YES |  NO |  YES |  YES/NO44footnotemark: 4 |  JAVA
GATE Cunningham (2002) |  NO |  YES |  NO |  NO |  YES/NO 55footnotemark: 5 |  JAVA
TWIST Pluss (2012) |  YES |  NO |  YES |  NO |  NO |  -
BRAT Stenetorp et al. (2012) |  NO |  YES |  NO |  YES |  NO |  PYTHON
DOCCANO33footnotemark: 3 |  NO |  YES |  NO |  NO |  NO |  PYTHON
DialogueView Heeman et al. (2002) |  YES |  YES |  YES |  NO |  NO |  TcK/TK

Table 1: Annotator Tool Comparison Table

Turn/Dialogue Segmentation: segment raw text into turns and dialogues.
Classification Labels: label classification data. Edit Dialogues/Turns: allow
users to add/edit/delete new turns or dialogues. Recommenders: ML models to
suggest annotations. Inter-Annotator Disagreement Resolution: whether the
system has an interface to resolve disagreements between different annotators.
Language: what programming language the system uses

##  1 Introduction

Of all the milestones on the road to creating artificial general intelligence
perhaps one of the most significant is giving machines the ability to converse
with humans. Dialogue systems are becoming one of the most active research
areas in Natural Language Processing (NLP) and Machine Learning (ML). New,
large dialogue datasets such as MultiWOZ Budzianowski et al. (2018) have
allowed data-hungry deep learning algorithms to be applied to dialogue
systems, and challenges such as the Dialogue State Tracking Challenge (DSTC)
Henderson et al. (2014) and Amazon’s Alexa Prize Khatri et al. (2018)
encourage competition among teams to produce the best systems.

The quality of a dialogue system dependents on the quality of the data used to
train the system. Creating a high-quality dialogue dataset incurs a large
annotation cost, which makes good dialogue annotation tools essential to
ensure the highest possible quality. Many annotation tools exist for a range
of NLP tasks but none are designed specifically for dialogue with modern
usability principles in mind - in collecting MultiWOZ, for example,
Budzianowski et al. (2018) had to create a bespoke annotation interface.

In this paper, we introduce LIDA, a web application designed to make dialogue
dataset creation and annotation as easy and fast as possible. In addition to
following modern principles of usability, LIDA integrates best practices from
other state-of-the-art annotation tools such as INCEpTION Klie et al. (2018),
most importantly by allowing arbitrary ML models to be integrated as
annotation recommenders to suggest annotations for data. Any system with the
correct API can be integrated into LIDA’s back end, meaning LIDA can be used
as a front end for researchers to interact with their dialogue systems and
correct their responses, then save the interaction as a future test case.

When data is crowdsourced, it is good practice to have multiple annotators
label each piece of data to reduce noise and mislabelling Deng et al. (2009).
Once you have multiple annotations, it is important to be able to resolve
conflicts by highlighting where annotators disagreed so that an arbiter can
decide on the correct annotation. To this end, LIDA provides a dedicated
interface which automatically finds where annotators have disagreed and
displays the labels alongside a percentage of how many annotators selected
each label, with the majority annotated labels selected by default.

###  1.1 Main Contributions

Our main contributions with this tool are:

  * •

A modern annotation tool designed specifically for task-oriented conversation
data

  * •

The first dialogue annotator capable of handling the full dialogue annotation
pipeline from turn and dialogue segmentation through to labelling structured
conversation data

  * •

Easy integration of dialogue systems and recommenders to provide annotation
suggestions

  * •

A dedicated interface to resolve inter-annotator disagreements for dialogue
data

##  2 Related Work

Various annotation tools have been developed for NLP tasks in recent years.
Table 1 compares LIDA with other recent annotation tools. TWIST Pluss (2012)
is a dialogue annotation tool which consists of two stages: turn segmentation
and content feature annotation. Turn segmentation allows users to highlight
and create new turn segments from raw text. After this, users can annotate
sections of text in a segment by highlighting them and selecting from a
predefined feature list. However, this tool doesn’t allow users to specify
custom annotations or labels and doesn’t support classification or slot-value
annotation. This is not compatible with modern dialogue datasets which require
such annotations Budzianowski et al. (2018).

INCEpTION Klie et al. (2018) is a semantic annotation platform for interactive
tasks that require semantic resources like entity linking. It provides machine
learning models to suggest annotations and allows users to collect and model
knowledge directly in the tool. GATE Cunningham (2002) is an open source tool
that provides predefined solutions for many text processing tasks. It is
powerful because it allows annotators to enhance the provided annotation tools
with their own Java code, making it easily extensible and provides an enormous
number of predefined features. However, GATE is a large and complicated tool
with a significant setup cost - its instruction manual alone is over 600 pages
long222https://gate.ac.uk/sale/tao/tao.pdf. Despite their large feature sets,
INCEpTION and GATE are not designed for annotating dialogue and cannot display
data as turns, an important feature for dialogue datasets.

BRAT Stenetorp et al. (2012) and Doccano333https://github.com/chakki-
works/doccano are web-based annotation tools for tasks such as text
classification and sequence labeling. They have intuitive and user-friendly
interfaces which aim to make the creation of certain types of dataset such as
classification or sequence labelling datasets as fast as possible. BRAT also
supports annotation suggestions by integrating ML models. However, like
INCEpTION444Getting the scores is available as a plugin:
https://dkpro.github.io/dkpro-statistics/dkpro-agreement-poster.pdf -
resolving the issues seems to be not supported and GATE555Again inter-
annotator score calculation capabilities are available as separate plug-in
https://gate.ac.uk/releases/gate-5.1-beta1-build3397-ALL/doc/tao/splitch10.html
- however support for resolutions is not apparent, they are not designed for
annotating dialogues and do not support generation of formatted conversational
data from a raw text file such as may be output by a transcription service.
LIDA aims to fill these gaps by providing a lightweight, easy-to-setup
annotation tool which displays data as a series of dialogues, supports
integration of arbitrary ML models as recommenders and supports segmentation
of raw text into dialogues and turns.

DialogueView Heeman et al. (2002) is a tool for dialogue annotation. However,
the main use-cases are not focused on building dialogue systems, rather it is
focused on segmenting recorded conversations. It supports annotating audio
files as well as discourse segmentation - hence, granular labelling of the
dialogue, recommenders, inter-annotator agreement, and slot-value labelling is
not possible with DialogueView.

##  3 System Overview

LIDA is built according to a client-server architecture with the front end
written in standard web languages (HTML/CSS/JavaScript) that will run on any
browser. The back end written in Python using the
Flask666http://flask.pocoo.org/ web framework as a RESTful API.

The main screen which lists all available dialogues is shown in Figure 3, in
the Appendix. The buttons below this list allow a user to add a blank or
formatted dialogue file. Users can also drag and drop files in this screen to
upload them. The user is then able to add, delete or edit any particular
dialogue. There is also a button to download the whole dataset as a JSON file
on this page. Clicking on a dialogue will take users to the individual
dialogue annotation screen shown in Figure 1.

LIDA uses the concept of a “turn” to organise how a dialogue is displayed and
recorded. A turn consists of a query by the user followed by a response from
the system, with an unlimited number of labels allowed for each user query.
The user query and system response are displayed in the large area on the left
of the interface, while the labels for each turn are shown in the scrollable
box on the right. There are two forms that these labels can currently take
which are particularly relevant for dialogue: multilabel classification and
slot-value pair.

An example of multilabel classification is whether the user was informing the
system or requesting a piece of information. An example of a slot-value pair
is whether the user mentioned the type of restaurant they’d like to eat at
(slot: restaurant-type) and if so what it was (value: italian, for example).
The front-end code is written in a modular form so that it is easy for
researchers using LIDA to add custom types of labels and annotations, such as
sequence classification, to LIDA.

Once annotation is complete, users can resolve inter-annotator disagreements
on the resolution screen. Here, each dialogue is listed along with the number
of different people who have annotated it. More annotations can be added by
dragging and dropping dialogue files into this screen. When the user clicks on
one of these dialogues, they are taken to the resolution screen shown in
Figure 2. Here, all of the disagreements in a dialogue are listed and the
label which annotators disagreed on is also shown. The label most frequently
selected by annotators is assigned as correct by default, and the arbiter can
accept this annotation simply by pressing “Enter” or else re-label the data
item. Once the arbiter has checked an annotation, it is displayed as
“Accepted” in the error list and the dialogue file automatically saved.

Figure 1: Turn List : A list of turns for one specific dialogue, users can add
new turns, delete turns, edit utterances and annotate labels here.

###  3.1 Use Cases

####  3.1.1 Experimenting with Dialogue Systems

While generic evaluation metrics are important for understanding the
performance of a dialogue system, another important method of evaluation is to
talk to the dialogue system and see if it gives subjectively satisfying
results. This gives the researcher insight into which part of the system most
urgently needs improvement faster than performing more complex error analysis.
However, if the user talks to their dialogue system through a terminal
interface, they have no way of correcting the system when it answers
incorrectly. The researcher should be able to record every interaction they
have with their system and correct the predictions of the system easily and
quickly. That way, the researcher will be able to use each previous recorded
interaction as a test case for future versions of their system.

LIDA is designed with this in mind - a dialogue system can be integrated in
the back end so that it will run whenever the user enters a new query in the
front end. The user will then be able to evaluate whether the system gave the
correct answer and correct the labels it gets wrong using the front end. LIDA
will record these corrections and allow the user to download the interaction
with their dialogue system with the corrected labels so that it can be used as
a test case in future versions of the system.

####  3.1.2 Creating a New Dialogue Dataset

Users can create a blank dialogue on LIDA’s home screen, then enter queries in
the box shown at the bottom of Figure 1. Along with whole dialogue systems,
arbitrary ML models can be added as recommenders in the back end. Once the
user hits “Enter”, the query is run through the recommender models in the back
end and the suggested annotations displayed for the label. If no recommender
is specified in the back end, the label will be left blank. Users can delete
turns and navigate between them using “Enter” or the arrow keys. The name of
the dialogue being annotated can be seen next to the “Back” button at the top
left of the screen and can be edited by clicking on it.

Figure 2: Screenshot of the inter-annotator disagreement resolution screen.

####  3.1.3 Annotating An Existing Dataset

Datasets can be uploaded via drag-and-drop to the home screen of the system,
or paths can be specified in the back end if the system were being used for
crowdsourcing. Datasets can be in one of two forms, either a “.txt” file such
as may be produced by a transcription service, or a formatted “.json” file, a
common format for dialogue data Budzianowski et al. (2018); Henderson et al.
(2014). Once the user has uploaded their data, their dialogue(s) will appear
on the home screen. The user can click on each dialogue and will be taken to
the single dialogue annotation screen shown in Figure 1 to annotate it. If the
user uploaded a text file, they will be taken to a dialogue and turn
segmentation screen. Following the same constraints imposed in MultiWOZ
Budzianowski et al. (2018) and DSTC Henderson et al. (2014), this turn
segmenter assumes that there are only two participants in the dialogue: the
user and the system, and that the user asks the first query. The user
separates each utterance in the dialogue by a blank line, and separates
dialogues with a triple equals sign (“===”). Once the user clicks “Done”, the
text file will automatically be parsed into the correct JSON format and each
query run through the recommenders in the back-end to obtain annotation
suggestions.

####  3.1.4 Resolving Annotator Disagreement

Researchers could use LIDA’s main interface to crowdsource annotations for a
dialogue dataset. Once they have several annotations for each dialogue, they
can upload these to the inter-annotator resolution interface of LIDA. The
disagreements between annotators will be detected, with a percentage shown
beside each label to show how many annotators selected it. The label with the
highest percentage of selections is checked by default. The arbiter can accept
the majority label simply by pressing “Enter” and can change errors with the
arrow keys to facilitate fast resolution. This interface also displays an
averaged (over turns) version of Cohen’s Kappa Cohen (1960), the total number
of annotations, total number of errors and averaged (over turns) accuracy.

###  3.2 Features777We refer the reader to visit the public repository for a
full documentation https://github.com/Wluper/lida.

##### Specifying Custom Labels

LIDA’s configuration is controlled by a single script in the back end. This
script defines which labels will be displayed in the UI and is easy to extend.
Users can define their own labels by altering this configuration script. If a
user wishes to add a new label, all they need to do is specify the label’s
name, its type (classification or slot-value pair, currently) and the possible
values the classification can take. Alongside the label specification, they
can also specify a recommender to use for the label values. The label will
then automatically be displayed in the front end. Note that labels in uploaded
datasets will only be displayed if the label has an entry in the configuration
file.

##### Custom Recommenders

When creating a dialogue dataset from scratch, LIDA is most powerful when used
in conjunction with recommenders which can suggest annotations for user
queries to be corrected by the annotator. State-of-the-art tools such as
INCEpTION Klie et al. (2018) emphasise the importance of being able to use
recommenders in annotation systems. Users can specify arbitrary ML models to
use for each label in LIDA’s back end. The back end is written in Python, the
de facto language for machine learning, so researchers can directly integrate
models written in Python to the back end. This is in contrast to tools such as
INCEpTION Klie et al. (2018) and GATE Cunningham (2002) which are written in
Java and so require extra steps to integrate a Python-based model. To
integrate a recommender, the user simply provides an instantiated Python
object in the configuration file that has a method called “transform” that
takes a single string and returns a predicted label.

##### Dialogue and Turn Segmentation from Raw Data

When uploading a .txt file, users can segment each utterance and each dialogue
with a simple interface. This means that raw dialogue data with no labels,
such as obtained from a transcription service, can be uploaded and processed
into a labelled dialogue. Segmented dialogues and turns are automatically run
through every recommender to give suggested labels for each utterance.

##  4 Evaluation

Table 1 shows a comparison of LIDA to other annotation tools. To our
knowledge, LIDA is the only annotation tool designed specifically for dialogue
systems which supports the full pipeline of dialogue annotation from raw text
to labelled dialogue to inter-annotator resolution and can also be used to
test the subjective performance of a dialogue system.

To test LIDA’s capabilities, we designed a simple experiment: we took a
bespoke dataset of 154 dialogues with an average of 3.5 turns per dialogue and
a standard deviation of 1.55. The task was to assign three classification
labels to each user utterance in each dialogue. Each annotator was given a
time limit of 1 hour and told to annotate as many dialogues as they could in
that time. We had six annotators perform this task, three of whom were
familiar with the system and three of whom had never seen it before.

These annotators annotated an average of 79 dialogues in one hour with a
standard deviation of 30, which corresponds to an average of 816.5 individual
annotations. The annotators who had never seen the system before annotated an
average of 60 dialogues corresponding to an average of 617 individual
annotations.

Once we had these six annotations, we performed a second experiment whereby a
single arbiter resolved inter-annotator disagreements. In one hour, the
arbiter resolved 350 disagreements and noted that resolution was slowest when
resolving queries with a high degree of disagreement.

These results show that LIDA provides fast annotation sufficient for
collecting large scale data. At the recorded pace of the annotators who had
not seen the system before, 100 workers could create a dialogue dataset of
6000 dialogues with approximately $6000*3.5=21000$ turns with three
annotations per turn in one hour. In large scale collections, such as MultiWOZ
Budzianowski et al. (2018) where 1249 workers were used, much larger datasets
with richer annotations could be created. Clearly annotation quantity will
depend on the difficulty of the task, length of dialogue and number of labels
to be assigned to each utterance but our results suggest that a high speed is
achievable.

##  5 Conclusion

We present LIDA, an open source, web-based annotation system designed
specifically for dialogue data. LIDA implements state-of-the-art annotation
techniques including recommenders, fully customisable labels and inter-
annotator disagreement resolution. LIDA is the only dialogue annotation tool
which can handle the full pipeline of dialogue dataset creation from turn and
dialogue segmentation to structured conversation data to inter-annotator
disagreement resolution.

Future work will look at adding new label types to LIDA, adding the
possibility to have more than two actors in the conversation, a centralised
admin page, additional labelling (e.g. co-reference resolution) and in general
enhancing usability as users provide feedback. Our hope is that this work will
find applications and usability beyond what we have outlined and developed so
far and that with a community effort a modern and highly accessible tool will
become widely available.

## References

  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46.
  * Cunningham (2002) Hamish Cunningham. 2002.  Gate, a general architecture for text engineering.  _Computers and the Humanities_ , 36(2):223–254.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee.
  * Heeman et al. (2002) Peter A. Heeman, Fan Yang, and Susan E. Strayer. 2002.  DialogueView - an annotation tool for dialogue.  In _Proceedings of the Third SIGdial Workshop on Discourse and Dialogue_ , pages 50–59, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014.  The second dialog state tracking challenge.  In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_ , pages 263–272.
  * Khatri et al. (2018) Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, et al. 2018.  Advancing the state of the art in open domain dialog systems through the alexa prize.  _arXiv preprint arXiv:1812.10757_.
  * Klie et al. (2018) Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018.  The inception platform: Machine-assisted and knowledge-oriented interactive annotation.  In _Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations_ , pages 5–9.
  * Pluss (2012) Brian Pluss. 2012.  Twist dialogue annotation tool.  http://mcs.open.ac.uk/nlg/non-cooperation/resources/user-guide.pdf.
  * Stenetorp et al. (2012) Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012.  Brat: a web-based tool for nlp-assisted text annotation.  In _Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 102–107. Association for Computational Linguistics.

##  Appendix A Additional System Screenshots & Scree Cast link:

Screen Cast Video:
https://vimeo.com/329824847

Figure 3: Dialogue List: A list of dialogues are shown on the home page. Users
are able to add, delete and edit any dialogue data. The button below allows
you to upload a raw text file which leads you to Figure 4 Figure 4: Screenshot
of dialogue/turn segmentation: after uploading the raw text, all text is
presented in a text area, which allows users to segment them into different
dialogues and turns. Figure 5: Screenshot of the inter-annotator agreement
main dialogue list: clicking on any of the items takes the user to the error
resolution screen. Figure 6: Screenshot of the inter-annotator agreement
scores shown by clicking on the “Inter-annotator Agreement” button in the list
screen for inter-annotator disagreement resolution (Figure 5).


9(1), 86-89.

  * •

Troster, V., Tiwari, A. K., Shahbaz, M., & Macedo, D. N. (2019). Bitcoin
returns and risk: A general GARCH and GAS analysis. Finance Research Letters,
30, 187-193.

  * •

Uhlig, H. (2022). A Luna-tic Stablecoin Crash (No. w30256). National Bureau of
Economic Research.

  * •

Van Dijk, M. A. (2011). Is size dead? A review of the size effect in equity
returns. Journal of Banking & Finance, 35(12), 3263-3274.

  * •

Veldkamp, L. L. (2005). Slow boom, sudden crash. Journal of Economic theory,
124(2), 230-257.

  * •

Wang, S., Yuan, Y., Wang, X., Li, J., Qin, R., & Wang, F. Y. (2018). An
overview of smart contract: architecture, applications, and future trends. In
2018 IEEE Intelligent Vehicles Symposium (IV) (pp. 108-113). IEEE.

  * •

Wang, H. (2019). VIX and volatility forecasting: A new insight. Physica A:
Statistical Mechanics and its Applications, 533, 121951.

  * •

Werner, S. M., Perez, D., Gudgeon, L., Klages-Mundt, A., Harz, D., &
Knottenbelt, W. J. (2021). Sok: Decentralized finance (defi). arXiv preprint
arXiv:2101.08778.

  * •

Westall, F., & Brack, A. (2018). The importance of water for life. Space
Science Reviews, 214(2), 1-23.

  * •

Wray, L. R. (2015). Modern money theory: A primer on macroeconomics for
sovereign monetary systems. Springer.

  * •

Xi, D., O’Brien, T. I., & Irannezhad, E. (2020). Investigating the investment
behaviors in cryptocurrency. The Journal of Alternative Investments, 23(2),
141-160.

  * •

Xu, M., Chen, X., & Kou, G. (2019). A systematic review of blockchain.
Financial Innovation, 5(1), 1-14.

  * •

Xu, J., Paruch, K., Cousaert, S., & Feng, Y. (2021). Sok: Decentralized
exchanges (dex) with automated market maker (AMM) protocols. arXiv preprint
arXiv:2103.12732.

  * •

Xu, J., & Feng, Y. (2022). Reap the Harvest on Blockchain: A Survey of Yield
Farming Protocols. IEEE Transactions on Network and Service Management.

  * •

Yadav, S. P., Agrawal, K. K., Bhati, B. S., Al-Turjman, F., & Mostarda, L.
(2020). Blockchain-based cryptocurrency regulation: An overview. Computational
Economics, 1-17.

  * •

Yli-Huumo, J., Ko, D., Choi, S., Park, S., & Smolander, K. (2016). Where is
current research on blockchain technology?—a systematic review. PloS one,
11(10), e0163477.

  * •

Zeng, X., Hao, N., Zheng, J., & Xu, X. (2019). A consortium blockchain
paradigm on hyperledger-based peer-to-peer lending system. China
Communications, 16(8), 38-50.

  * •

Zetzsche, D. A., Arner, D. W., & Buckley, R. P. (2020). Decentralized finance.
Journal of Financial Regulation, 6(2), 172-203.

  * •

Zheng, Z., Xie, S., Dai, H. N., Chen, W., Chen, X., Weng, J., & Imran, M.
(2020). An overview on smart contracts: Challenges, advances and platforms.
Future Generation Computer Systems, 105, 475-491.

  * •

Zou, W., Lo, D., Kochhar, P. S., Le, X. B. D., Xia, X., Feng, Y., … & Xu, B.
(2019). Smart contract development: Challenges and opportunities. IEEE
Transactions on Software Engineering, 47(10), 2084-2106.