# EmoDM: Empathetic Response Generation with Emotion-aware
Dialogue Management

Yuhan Liu Jun Gao Jiachen Du Lanjun Zhou Ruifeng Xu
Harbin Institute of Technology (Shenzhen)
{yhanliu<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Corresponding author

###### Abstract

A good empathetic dialogue system should first track and understand a user’s
emotion and then reply with an appropriate emotion. However, current
approaches to this task either focus on improving the understanding of users’
emotion or on proposing better responding strategies, and very few works
consider both at the same time. Our work attempts to fill this vacancy.
Inspired by task-oriented dialogue systems, we propose a novel empathetic
response generation model with emotion-aware dialogue management. The emotion-
aware dialogue management contains two parts: (1) Emotion state tracking
maintains the current emotion state of the user and (2) Empathetic dialogue
policy selection predicts a target emotion and a user’s intent based on the
results of the emotion state tracking. The predicted information is then used
to guide the generation of responses. Experimental results show that
dynamically managing different information can help the model generate more
empathetic responses compared with several baselines under both automatic and
human evaluations.

EmoDM: Empathetic Response Generation with Emotion-aware
Dialogue Management

Yuhan Liu Jun Gao Jiachen Du Lanjun Zhou Ruifeng Xu††thanks:  Corresponding
author Harbin Institute of Technology (Shenzhen) {yhanliu,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Empathy plays a critical role in daily communication. Many researches have
shown that endowing dialogue systems with empathy can improve the human-
computer interaction experience Mercer and Reynolds (2002); Partala and
Surakka (2004). In this work, we focus on empathetic response generation
Rashkin et al. (2019), whose goal is to enhance the ability of dialogue
systems to perceive and express emotions in conversations. As shown with an
example in Fig. 1, given a dialog context, the empathetic dialogue system (the
listener) is required to track and understand the speaker’s emotion state in a
multi-turn dialogue scenario and reply with an appropriate emotion.

Figure 1: An empathetic conversation instance with the corresponding listener
states

Existing approaches to empathetic response generation can be divided into two
categories: the first improves understanding of dialogue emotion by tracking
the user’s emotion Rashkin et al. (2018); Lin et al. (2019a); Majumder et al.
(2020); Gao et al. (2021b). For example, Majumder et al. (2020) propose to
generate empathetic responses by mimicking the emotion of the speaker to a
varying degree, depending on its affective charge (positive or negative) and
the dialogue context. Gao et al. (2021b) prove that recognizing the emotion
cause is helpful for the empathetic response generation model to better
understand human emotions. The second improves empathetic response generation
by using response emotion signals as feedback Li et al. (2020a) or
relationships between the dialogue history and the gold responses Li et al.
(2020b). While promising results have been achieved by both kinds of
approaches, they ignore the interaction between the use’s emotion and the
responding strategy, i.e. they either focus on improving the understanding of
users’ emotion or on proposing better responding strategies. Very few works
consider both at the same time, and our work attempts to fill this vacancy.

Our work is mainly inspired by task-oriented dialogue systems, in which the
dialogue management component is designed for tracking the dialogue state and
selecting an executable action based on the results of the dialogue state
tracing. Similarly, a good empathetic dialogue system should first track and
understand the user’s emotion and then reply with an appropriate emotion based
on the user’s emotion. To achieve this, we propose a novel empathetic response
generation model with emotion-aware dialogue management. It contains two
components: (1) Emotion State Tracker that maintains the current emotion state
of the user and (2) Empathetic Dialogue Policy Predictor that predicts the
response emotion and the intent of the response to be generated, based on the
results of the emotion state tracker. The predicted information is then used
to guide the generation of responses. As shown in Figure 1, in the first turn,
the listener shows a surprise feeling in consideration of the speaker’s joyful
emotion, and the listener expresses his empathy with two kinds of intents. The
speaker’s emotion, the listener’s emotion, and the listener’s intent construct
a triple. We define such triple as an emotion state, and the response
generation can be regarded as an emotion state shift process.

To model interaction between the use’s emotion and the responding strategy, we
also introduce emotion shift patterns. Intuitively, there exists emotion shift
patterns among Empathetic Dialogue Rashkin et al. (2019). We make a
statistical experiment and have the following findings: probability that
surprise shifts to joy is 32.2% and probability that neutral shifts to
surprise is 30.3%, and human observation to the subset of the dialogue data
proves that responses are influenced by the cooperation of speaker’s emotion
and listener’s emotions Zhou et al. (2018) rather than the other one. Besides,
Welivita and Pu (2020) provide a dataset annotated with intents, and they find
strong evidence that there are shift patterns between the speaker’s emotions
and response intents over multiple turns, which can provide a solution to
detect goals, relationships and empathy.

Overall, we make the following contributions:

  * •

Inspired by advance in dialogue management from task-oriented dialogue
systems, we propose an emotion-aware dialogue management method for empathetic
response generation. It contains two components: the Emotion State Tracker and
the Empathetic Dialogue Policy Predictor.

  * •

To model interaction between the use’s emotion and the responding strategy, we
devise emotion shift patterns. Our preliminary experiments show that there
exists emotion shift patterns among Empathetic Dialogue.

  * •

Experimental results show that our model outperforms other state-of-the-art
baselines in both automatic and human evaluations, and the proposed emotion-
aware dialogue management can effectively improve empathetic response
generation.

##  2 Related Work

Recently, open domain neural dialogue systems (Gao et al., 2019; Lowe et al.,
2017; Wolf et al., 2019; Zhang et al., 2020b) have get promoted in a
tremendous speed. Specifically, emotionalizing (Ghosh et al., 2017; Zhou et
al., 2018; Zhou and Wang, 2018; Song et al., 2019; Shen and Feng, 2020) and
personalizing (Kottur et al., 2017; Chu et al., 2018) the chatbots have become
a tendency.

With the application of deep learning technique, emotion-aware response
generation gets promoted dramatically. Zhou et al. (2018) uses internal and
external memory modules to store emotion features for explicitly generating
different emotional responses. Huang et al. (2018) utilizes several approaches
to infuse emotions into encoders and decoders to generate emotional responses.
Colombo et al. (2019) proposes word-level and sequence-level emotion
regularizers and emotion-oriented sampling method. Song et al. (2019) adopts
lexicon-based attention and sequence-level emotion classifier to increase
emotion intensity during generation.

Empathetic dialogue generation can be seen as a sub-task of emotion-aware
response generation, whose goal is to generate responses with proper emotions.
Rashkin et al. (2019) proposes a standard benchmark that contains large-scale
multi-round empathetic conversations. Then MoEL (Lin et al., 2019a) is
proposed, which uses several dedicated emotion decoders to generate softly-
combined outputs. Shin et al. (2020) applies reinforcement learning to
introduce user feeling towards the generated response during training. Li et
al. (2020b) leverages multi-type external knowledge and emotional signal
distilling to enhance information flow. EmpDG (Li et al., 2020a) focuses on
the potential of user feedback (Li et al., 2020c) and introduces an
adversarial learning framework to capture the nuances of user emotion. MIME
(Majumder et al., 2020) assumes that empathetic responses should mimic
speakers’ emotion and adds stochasticity during response generation. Gao et
al. (2021a) utilizes emotion cause information within utterances to help
dialog systems better understand human emotions for empathically responding.
Lin et al. (2019b) builds an online empathetic chatbot demo with pre-trained
language model GPT2 (Radford et al., 2019).

##  3 Methodology

Figure 2: Architecture of the proposed model. It contains two parts:
(1)Emotion-aware Dialogue Management and (2) Response Generator. Given an
input dialogue history, the emotion-aware dialogue management first tracks the
speaker’s emotion, and then predict listener’s emotion and intent based on the
result of emotion state tracking module and our proposed emotion shift
patterns. The predicted information is then used by the response generator to
guide the generation of responses.

###  3.1 Preliminary

Given a dialogue history $\\{s_{0},l_{0},s_{1},l_{1},\cdots,s_{M}\\}$ where
$s_{*}$ and $l_{*}$ represent utterances from speakers and listeners, the task
goal is to generate proper empathetic response $y=\\{y_{1},\cdots,y_{m}\\}$.
Following previous works Lin et al. (2019a); Rashkin et al. (2019), we flatten
the dialogue history into a sequence $x=\\{x_{1},\cdots,x_{n}\\}$. We assume
that the whole generation process is controlled by speaker emotion
$\bm{v}_{\varepsilon_{s}}$, listener emotion $\bm{v}_{\varepsilon_{l}}$, and
listener intent $\bm{v}_{\tau}$. Therefore, the problem can be optimized
through maximizing the likelihood
$\textstyle\prod_{i=1}^{m}\mathcal{P}(y_{i}|y_{<i};x;\bm{v})$.

Since the original dataset Rashkin et al. (2019) does not contain emotion
labels and intent labels for each utterance, we leverage several external
datasets. Specifically, we first obtain an emotion prediction model by fine-
tuning the BERT model Devlin et al. (2019) on a fine-grained emotion111the 7
emotion categories are: anger, disgust, fear, joy, neutral, sadness and
surprise. classification dataset, goEmotion Demszky et al. (2020). The emotion
prediction model is then used to annotate an emotion label for each utterance.
The emotion prediction model achieved an accuracy of 70% on test set,
indicating that it is reliable for emotion classification. The intent222The 9
intent labels are: questioning, acknowledging, agreeing, consoling,
encouraging, sympathizing, wishing, suggesting and neutral label of each
utterance is obtained using the data proposed by Welivita and Pu (2020), where
each utterance is annotated with intent information.

As shown in Fig 2, our proposed model contains two parts: the Emotion-aware
Dialogue Management (EmoDM) is used to predict the overall states and extract
their features and a Response Generator (RespG) is used to generate responses
conditioned on dialogue context and the states.

###  3.2 Emotion-aware Dialogue Management

We build the Emotion-aware Dialogue Management (EmoDM) on a transformer
Vaswani et al. (2017) encoder. Formally, given a dialogue history sequence
$x=\\{[\mathrm{CLS}],x_{1},\cdots,x_{n}\\}$, the input representation for each
word is the sum of its token embedding and position embedding, then they are
fed into the transformer encoder to get a sequence of hidden outputs
$\\{\bm{h_{[\mathrm{CLS}]},h_{x_{1}},\cdots,h_{x_{n}}}\\}$.

Emotion State Tracker. We use the representation of [CLS] token as the context
representation of the dialogue history. The representation is then fed into a
linear layer to obtain the speaker emotion probability distribution
$\mathcal{P}(\varepsilon_{s}|\bm{x})=\mathrm{softmax}(\bm{h_{[\mathrm{CLS}]}}\bm{V}_{\varepsilon}^{T})$.
Specifically, the layer weights are shared with emotion state embedding
$\bm{V}_{\varepsilon}\in\mathbb{R}^{N_{\varepsilon}\times d}$. Here
$N_{\varepsilon}$ is the number of emotion and $d$ is the vector dimension.

Dialogue Policy Predictor. There is strong evidence that emotion-emotion (emo-
emo) shift pattern exists among conversations. We can use this prior knowledge
to predict emotion as accurately as possible with only dialogue history
context. Through normalizing the frequency that emotion $i$ shifts to $j$, we
can compute the shift probabilities to build emo-emo prior matrix
$\bm{\mathcal{M}}_{\varepsilon}=[a_{i,j}]\in\mathbb{R}^{N_{\varepsilon}\times
N_{\varepsilon}}$ based on training and validation datasets. We can use the
predicted speaker emotion to infer some clues about possible shift
probabilities. According to the predicted label index $k$, we can get the
whole possible emotion shift probabilities
$\bm{m}_{sft}=[a_{k,1},\cdots,a_{k,N_{\varepsilon}}]$. We multiply this prior
distribution with the emotion state embedding $\bm{V}_{\varepsilon}$ to get
the representation $\bm{v}_{sft}=\bm{V}_{\varepsilon}^{T}\bm{m}_{sft}$. Then
we can obtain future emotion probability distribution

$\mathcal{P}(\varepsilon_{l}|\bm{x},\varepsilon_{s},\bm{\mathcal{M}}_{\varepsilon})={\rm
softmax}(\bm{W_{1}}[\bm{h_{[\mathrm{CLS}]}};\bm{v}_{sft}]+\bm{b_{1}})$ (1)

Here $\bm{W_{1}}$ and $\bm{b_{1}}$ are trainable parameters.

To predict the listener’s intent, it is also necessary to use the emotion-
intent (emo-itt) shift pattern as the prior knowledge. Similar to listener
emotion prediction, we can build the emot-itt prior matrix
$\bm{\mathcal{M}}_{\tau}=[b_{i,j}]\in\mathbb{R}^{N_{\varepsilon}\times
N_{\tau}}$. Here $N_{\tau}$ is the number of intent. After that, we can get
emotion-intent shift probabilities
$\bm{m}_{sft}=[b_{k,1},...,b_{k,N_{\tau}}]$. We use $\bm{m}_{sft}$ an intent
embedding $\bm{V}_{\tau}\in\mathbb{R}^{N_{\tau}\times d}$ to get the prior
intent $\bm{v}_{sft}=\bm{V}_{\tau}^{T}\bm{m}_{sft}$ , which is similar to
listener emotion prediction. Then we can predict intent probability
distribution

$\mathcal{P}(\tau|\bm{x},\varepsilon_{s},\bm{\mathcal{M}}_{\tau})=\sigma(\bm{W_{2}}[\bm{h_{[\mathrm{CLS}]}};\bm{v}_{sft}]+\bm{b_{2}})$
(2)

Here we use sigmoid instead of softmax since there exist multiple intent
labels in one response. $\bm{W_{2}}$ and $\bm{b_{2}}$ are trainable
parameters.

Optimization. The EmoDM can be optimized through a negative log likelihood
(NLL) loss, where $\alpha$ and $\beta$ are hyper-parameters:

$\begin{split}\mathcal{L}_{U}&=\alpha\log\mathcal{P}(\varepsilon_{s}|\bm{x})+(1-\alpha)\log\mathcal{P}(\varepsilon_{l}|\bm{x},\varepsilon_{s},\bm{\mathcal{M}}_{\varepsilon})\\\
&+\beta\log\mathcal{P}(\tau|\bm{x},\varepsilon_{s},\bm{\mathcal{M}}_{\tau})\end{split}$
(3)

###  3.3 Response Generator

Our Response Generator (RespG) is based on a transformer decoder, which is
used to generate empathetic responses given a dialogue context and the
predicted states. We concatenate the special token$\mathrm{[CLS]}$, dialogue
context $\\{x_{1},x_{2},\cdots,x_{n}\\}$, and ground-truth response
$\\{y_{1},y_{2},\cdots,y_{m}\\}$ together and obtain their hidden outputs
$\\{\bm{h_{[\mathrm{CLS}]},h_{x_{1}},\cdots,h_{y_{m}}}\\}$ through the
transformer decoder. In our RespG, we modify the original attention mask in
the original transformer decoder to let $\mathrm{[CLS]}$ attend to the
rightward history context for later usage.

During decoding, we fuse the hidden output $\bm{h}_{y_{i}}$ of $i$-th step
with the state feature predicted by EmoDM to control the generation process.
Specifically, we first obtain speaker emotion state $\bm{v}_{\varepsilon_{s}}$
and listener emotion state $\bm{v}_{\varepsilon_{l}}$ from
$\bm{V}_{\varepsilon}$ according to predicted emotion labels where we can get
them from $\mathcal{P}(\varepsilon_{s}|\bm{x})$ and
$\mathcal{P}(\varepsilon_{l}|\bm{x},\varepsilon_{s},\bm{\mathcal{M}}_{\varepsilon})$
by argmax operation. Then we use the following approach to inject emotion
state bias, where $\bm{W_{3}}$ and $\bm{W_{4}}$ are trainable parameters:

$\bm{h}_{y_{i}}^{\varepsilon}=\bm{W_{3}}(\bm{h}_{y_{i}}+\bm{v}_{\varepsilon_{s}})+\tanh(\bm{W_{4}}\bm{h}_{y_{i}})\bm{v}_{\varepsilon_{l}}$
(4)

Intent prediction is a multi-label classification task, which means the
response may have multiple intent labels. To fuse multiple intents, we first
obtain the intent $\bm{v}_{\tau}$ from $\bm{V}_{\tau}$ with an average
operation as in Eq.5 and then inject intent features into token hidden outputs
to provide intent information:

$\displaystyle\bm{v}_{\tau}$
$\displaystyle=\hat{I}_{\tau}\bm{V}_{\tau}/{\textstyle\sum_{i=1}^{N_{\tau}}\hat{I}_{\tau}^{(i)}}$
(5) $\displaystyle\bm{h}_{y_{i}}^{\tau}$
$\displaystyle=\bm{v}_{\tau}\odot\bm{h}_{y_{i}}+\bm{v}_{\tau}$ (6)

Where $\odot$ denotes element-wise multiplication and $\hat{I}_{\tau}$ is the
predicted multi-label vector. We can get it through applying argmax operation
to each row of the intent probability distribution
$\mathcal{P}(\tau|\bm{x},\varepsilon_{s},\bm{\mathcal{M}}_{\tau})$.

Now that two different kinds of state fused representations have been
obtained, they are projected into vocabulary logit space. To effectively merge
two distributions, we design a gate control layer to monitor the information
flow. We pass $\mathrm{[CLS]}$ hidden output through a linear layer with
sigmoid activation to obtain the control value
$\gamma=\sigma(\bm{W_{5}}\bm{h}_{\mathrm{[CLS]}}+\bm{b_{5}})$. After that, we
apply softmax to the merged logits to get final probabilities:

$\mathcal{P}(y_{i}|y_{<i};x;\bm{v})=\mathrm{softmax}(\bm{E}_{w}^{T}(\gamma\bm{h}_{y_{i}}^{\varepsilon}+(1-\gamma)\bm{h}_{y_{i}}^{\tau}))$
(7)

Where $\bm{W_{5}}$ and $\bm{b_{5}}$ are trainable parameters and
$\bm{E}_{w}\in\mathbb{R}^{V\times d}$ is the token embedding lookup table. For
optimizing the RespG, we also adopt NLL loss for it.

|  $P_{\rm BERT}$(%) |  $R_{\rm BERT}$ (%) |  $F_{\rm BERT}$(%) | BLEU-4 | DIST-1 | DIST-2
---|---|---|---|---|---|---
MoEL | 14.5$\pm$0.9 | 12.1$\pm$0.4 | 13.3$\pm$0.3 | 1.592$\pm$0.090 | 0.008$\pm$0.001 | 0.076$\pm$0.005
MIME | 14.9$\pm$0.6 | 12.5$\pm$0.6 | 13.8$\pm$0.4 | 1.658$\pm$0.064 | 0.008$\pm$0.001 | 0.055$\pm$0.004
EmpDG | 11.3$\pm$0.4 | 10.3$\pm$1.4 | 10.9$\pm$0.8 | 1.318$\pm$0.021 | 0.010$\pm$0.002 | 0.072$\pm$0.018
MultiGPT |  18.9$\pm$0.2 |  15.5$\pm$0.2 |  17.0$\pm$0.1 | 1.448$\pm$0.012 |  0.034$\pm$0.001 |  0.210$\pm$0.002
Ours(Trs) | 15.3$\pm$0.2 | 12.7$\pm$0.1 | 14.6$\pm$0.1 |  1.730$\pm$0.141 | 0.017$\pm$0.002 | 0.107$\pm$0.008
Ours(LM) |  19.2$\pm$0.2 |  15.8$\pm$0.3 |  17.6$\textsuperscript{\textdagger}\pm$0.2 |  1.666$\pm$0.048 |  0.028$\pm$0.001 |  0.193$\pm$0.005
Table 1: Automatic metric results. we repeat 5 runs with different seeds and
average the results for each method. Standard deviations are given in the
small text. †means the results are statistically significant at $p<0.05$. The
bold numbers denote the best results and the underlined numbers denote the
second best results.

##  4 Experiments

###  4.1 Experiment Setup

We conduct experiments on Empathetic Dialogue dataset (Rashkin et al., 2019),
which consists of 25k one-to-one open-domain conversations with 8:1:1
train/validation/test split.

We explore our models with two different initialization methods: the vanilla
transformer and the pretrained language model, denoted as Ours(Trs) and
Ours(LM) respectively. We first use the dataset to fine-tune the RespG
backbone to learn some semantic features. Then we alternatively train the
EmoDM and the RespG so that RespG can gradually adapt to the influence of the
state feature. For Ours(Trs), we train the model for 20 epochs, where the
batch size and learning rate are set to 16 and 2e-4. For Ours(LM), we use
BERT-base as EmoDM backbone and GPT2-small as ResG backbone. We train the
model for 10 epochs, where the batch size and learning rate are set to 16 and
5e-5. The value for $\alpha$ and $\beta$ are 0.6 and 0.5. We use AdamW
(Loshchilov and Hutter, 2019) as the optimizer. and apply the schedule
sampling (Bengio et al., 2015) so that RespG can access true state feature
with some probability. We choose the model where RespG performs best in the
validation set for final evaluation. And top-k sampling (Holtzman et al.,
2020) is adopted during inference.

###  4.2 Evaluation Metrics

Automatic Metrics. Three kinds of automatic metrics are chosen: (1) BLEU:
Following (Li et al., 2020a; Majumder et al., 2020; Lin et al., 2019a), we use
BLEU-4 (Papineni et al., 2002) to calculate n-gram overlap ratio. 2) BERTscore
(Zhang et al., 2020a) shows stronger system-level and segment-level
correlations with human judgments. It uses contextual embeddings of reference
and the generated sentence to compute a weighted cosine similarity. We use
matching precision, recall and F1 score ($R_{\textup{BERT}}$,
$P_{\textup{BERT}}$ and $F_{\textup{BERT}}$) to measure the similarity. 3)
Diversity metrics (Li et al., 2016) aims at computing the proportion of
different grams in the text to obtain text diversity. We use Dist-{1,2} to get
diversity score.

Human Judgements. Two types of human evaluations are carried out to fully
verify advantages and shortcomings of different models: human rating and human
A/B test. Fleiss-Kappa is calculated after annotation to measure the
agreement. For Human rating, we randomly sample 100 dialogues and
corresponding generated responses of different models. 5 research students are
required to give each response a rating score from Relevance aspect, Fluency
aspect, and Empathy aspect (Rel., Flu., and Emp.). The score is a 5-point
scale (1: not at all, 3: somewhat, 5: very much).

For the human A/B test, we rearrange the samples in an A-vs-B format, where A
is our model and B is another baseline. Another 3 research students are asked
to choose the better response for each instance. they can also choose a Tie if
both are good or bad. In order to keep the anonymization of compared methods,
the response order in each sample is totally shuffled and each group of A/B
test uses a distinct dialogue context. Annotators are not allowed to discuss
with others during evaluation.

###  4.3 Baselines

We compare our models with the following baselines333To ensure rationality and
fairness during comparison, all the baselines use top-k sampling strategy.:
(1) MoEL (Lin et al., 2019a): the transformer-based seq2seq (Sutskever et al.,
2014) model which softly combines the different decoder outputs according to
emotion distribution. (2) MIME (Majumder et al., 2020): An extension of the
transformer model, which introduces emotion grouping and sampling
stochastically to generate responses. (3) EmpDG (Li et al., 2020a): an
adversarial model which adopts multi-resolution emotion perception and two
discriminators for interacting with the user feedback. (4) MultiGPT: similar
to (Lin et al., 2019b; Zandie and Mahoor, 2020), we re-implement a multi-task
framework with GPT2 as backbone, which is responsible for language modelling,
speaker emotion prediction, and listener emotion prediction simultaneously.

Ablation study is also conducted to thoroughly analyze the impact of different
components in our proposed method. The results and analysis can be found in
Sec. 4.6.

###  4.4 Automatic Evaluation Results

Table 2 illustrates the whole automatic metric results of different models.
Comparing to other baselines, Ours(LM) and Ours(Trs) reaches the highest
results in most metrics, indicating that our proposed methods can effectively
learn expression patterns in the corpus. For Ours(LM), although Dist-1 of ours
are slightly worse than MultiGPT, all three BERTscore of ours are higher than
other baselines. This indicates that the generated responses of our model have
better relevance to history contexts and MultiGPT may generate some noise and
unrelated words, which verifies that incorporating with state features is
helpful to capture the semantic characteristics accurately. Besides, For
Ours(Trs), we achieve the best performances in all three metrics among vanilla
transformer backbone based methods. Performance of both Ours(Trs) and Ours(LM)
illustrate that our proposed method can generate more informative and
empathetic responses.

###  4.5 Human Evaluation Results

The results of human ratings and A/B test are illustrated in Table 2 and Table
3. The Fleiss-Kappa of each three rating aspects and A/B test are 0.29, 0.14,
0.21, and 0.35, which indicate that the multiple annotators almost reach a
fair agreement. The results indicate that Ours(LM) obtains the highest rating
score in all three rating aspects. In particular, our model exceeds MultiGPT
about 0.5 points in both Relevance and Empathy, which shows the effective
control from the overall states. For Ours(Trs), although the model performance
is slightly worse than MultiGPT, we still achieve the best results comparing
to other transformer based models. The human A/B test results also validate
the conclusions mentioned above. Due to limited semantic mapping, Ours(Trs)
model is not as good as Ours(LM) model. And the high winning percentage of
Ours(LM) correlates with high human rating scores. Overall, the two
evaluations together manifest that responses from our proposed model are more
preferable and of better quality.

| Rel. | Flu. | Emp.
---|---|---|---
MoEL | 2.70 | 3.81 | 2.62
MIME | 2.65 | 3.89 | 2.73
EmpDG | 2.74 | 3.71 | 2.65
MultiGPT | 3.20 | 4.14 | 3.04
Ours(Trs) | 2.87 | 3.82 | 2.94
Ours(LM) | 3.70 | 4.37 | 3.67
Table 2: Results on human judgement. | Win(%) | Loss(%) | Tie(%)
---|---|---|---
Ours(Trs) vs EmpDG | 45.2 | 29.2 | 25.6
Ours(Trs) vs MoEL | 42.4 | 35.2 | 22.4
Ours(Trs) vs MIME | 43.6 | 34.0 | 22.4
Ours(Trs) vs MultiGPT | 25.9 | 50.0 | 24.1
Ours(LM) vs EmpDG | 72.2 | 24.1 | 3.7
Ours(LM) vs MoEL | 77.7 | 11.9 | 10.4
Ours(LM) vs MIME | 61.1 | 25.9 | 13.0
Ours(LM) vs MultiGPT | 51.8 | 35.1 | 13.1
Ours(LM) vs Ours(Trs) | 65.7 | 23.5 | 13.8
Table 3: Results on human A/B test. |  $P_{\rm BERT}$(%) |  $R_{\rm BERT}$(%) |  $F_{\rm BERT}$(%) | BLEU-4 | DIST-1 | DIST-2
---|---|---|---|---|---|---
Ours(Trs) |  15.3$\pm$0.2 | 12.7$\pm$0.1 |  14.6$\pm$0.1 |  1.730$\pm$0.141 | 0.017$\pm$0.002 | 0.107$\pm$0.008
vs. Naive | 15.3$\pm$0.2 |  12.6$\pm$0.3 | 14.2$\pm$0.2 | 1.521$\pm$0.074 | 0.021$\pm$0.002 | 0.114$\pm$0.009
w/o itt | 14.6$\pm$0.3 | 12.3$\pm$0.3 | 13.7$\pm$0.3 | 1.486$\pm$0.090 |  0.028$\pm$0.005 |  0.138$\pm$0.024
w/o emo | 14.5$\pm$0.7 | 12.1$\pm$0.6 | 12.4$\pm$0.3 | 1.415$\pm$0.001 | 0.018$\pm$0.003 | 0.101$\pm$0.011
Ours(LM) |  19.2$\pm$0.2 |  15.8$\pm$0.3 |  17.5$\pm$0.2 |  1.666$\pm$0.048 | 0.028$\pm$0.001 | 0.193$\pm$0.005
vs. Naive | 16.7$\pm$0.5 | 14.7$\pm$0.3 | 15.8$\pm$0.3 | 1.149$\pm$0.181 |  0.305$\pm$0.001 |  0.226$\pm$0.005
w/o itt | 19.0$\pm$0.1 | 15.8$\pm$0.3 |  17.5$\pm$0.1 | 1.620$\pm$0.070 | 0.264$\pm$0.002 | 0.182$\pm$0.010
w/o emo | 18.3$\pm$0.2 | 14.8$\pm$0.3 | 16.6$\pm$0.2 | 1.335$\pm$0.117 | 0.018$\pm$0.002 | 0.140$\pm$0.012
Table 4: Results on Ablation Study. We thoroughly analyze the impact of
different components in our method: (1) w/o emo: only the intent is predicted
and fused. (2) w/o itt: only emotion state is predicted and fused. (3) Naive:
only speaker emotion is predicted, and simply utilize prior shift matrix to
get the other states for generation.

###  4.6 Ablation Study

Ablation study results are presented in Table 4. For Ours(LM), performance of
w/o-itt (only utilizing emotion state) is better than that of w/o-emo (only
utilizing intent information). Its DIST and $F_{\mathrm{BERT}}$ are higher
than those of w/o-emo. which may indicate that emotion states can lead to
generating more diverse and more semantic-related words. The emotion state may
play a more crucial role in managing generation. Nevertheless, when we combine
two states through gate control, the full model achieves the best metric
results. This shows that both state feature can provide useful information
from different views. For Ours(Trs), such conclusion can also be inferred.

Besides, there are some slightly different findings when observing the
performance of Naive model. For Ours(Trs)-Naive, its BERTscore is better than
other two sub-models. For Ours(LM)-Naive, its BERTscore and BLEU is worse than
other two sub-models, while it gets the highest DIST score. It illustrates
that for an empathetic response model where its parameters are randomly
initialized, the introduction of two state features can better help the model
to generate empathetic and semantically relevant responses even if its state
prediction is not accurate enough. But for the model where its parameters are
initialized by pretrained language model, it generates some unrelated words
because of inaccurate state prediction.

###  4.7 Further Analysis

| speaker | listener
---|---|---
| acc(%) | wF1(%) | acc(%) | wF1(%)
MultiGPT | 87.7$\pm$ 0.5 | 87.6$\pm$0.5 | 52.1$\pm$0.4 | 47.1$\pm$1.4
Ours(Trs) | 76.5$\pm$0.2 | 76.2$\pm$0.2 | 50.4$\pm$0.2 | 42.6$\pm$0.3
vs. Naive | 76.3$\pm$0.3 | 76.0$\pm$0.4 | 43.2$\pm$0.3 | 29.4$\pm$0.1
w/o itt | 77.0$\pm$0.1 | 76.8$\pm$0.1 | 50.7$\pm$0.2 | 43.1$\pm$0.4
w/o emo | - | - | - | -
Ours(LM) |  88.5$\pm$0.4 |  88.4$\pm$0.4 |  52.6$\pm$0.4 | 45.5$\pm$1.2
vs. Naive | 87.1$\pm$1.2 | 87.0$\pm$1.2 | 42.4$\pm$0.3 | 29.7$\pm$0.1
w/o itt | 88.1$\pm$0.3 | 88.0$\pm$0.3 | 51.7$\pm$0.7 |  48.0$\pm$0.6
w/o emo | - | - | - | -
Table 5: Emotion state prediction results of the EmoDM.  | HL(%) | miF1(%) | AP(%)
---|---|---|---
Ours(Trs) | 15.4$\pm$0.2 | 44.2$\pm$0.1 | 54.9$\pm$0.1
vs. Naive | 15.9$\pm$0.1 | 42.4$\pm$0.1 | 52.6$\pm$0.3
w/o itt | - | - | -
w/o emo | 15.3$\pm$0.1 | 44.1$\pm$0.2 | 54.8$\pm$0.2
Ours(LM) |  15.2$\pm$0.1 |  44.8$\pm$0.1 |  55.3$\pm$0.1
vs. Naive | 15.8$\pm$0.1 | 42.4$\pm$0.1 | 52.6$\pm$0.1
w/o itt | - | - | -
w/o emo | 15.8$\pm$0.1 | 42.8$\pm$0.4 | 52.8$\pm$0.3
Table 6: Intent prediction results of the EmoDM.

State Prediction Comparison. Next, we evaluate the state prediction
performance of the EmoDM. The results on state prediction are presented in
Table 5 and Table 6. For emotion state prediction performance, we report
accuracy (acc) and weighted F1 (wF1). For intent prediction performance, we
report hamming loss (HL), average precision (AP) and micro F1 (miF1).

As can be seen in the table, Ours(LM) model achieves the highest current
emotion wF1, future emotion acc, intent AP and intent miF1. Specifically, for
speaker emotion prediction and listener emotion prediction, our proposed model
achieves 88.32% and 52.67% accuracy (acc). Comparing to MultiGPT which get
87.61% and 51.81% acc, we get better results than it. We also achieve 55.60%
averaged precision (AP) in intent prediction, which means our model has a good
capability of predicting intents. This demonstrates that removing either
intent prediction component or emotion prediction component will hurt the
EmoDM performance. And all the metric scores of Ours(LM) are higher than
MultiGPT. This shows that the EmoDM in our proposed methods can get better
results than adopting the multi-task framework in this task.

Besides, for Naive method, we also find that the results of Naive method is
not as good as our proposed methods. The speaker and listener emotion
prediction acc are 87.67% and 42.69%, and the intent AP is 52.65%. This shows
that simply using state shift statistics for prediction is not accurate, and
our proposed EmoDM is more effective.

State | sadness | sadness | questioning
---|---
Context | my wife has been really sad here recently, i have had to hold her while she cries herself to sleep at night.
reference | that is very sad. did something recently happen to her?
MoEL | that is a great thing to have a little bit of the pain .
MIME |  i know that feeling . how long ago was it for you ?
EmpDG |  oh no ! how did she do that ?
MultiGPT |  | You can always talk to her about it though! It’s important to remember that when you’re alone,
---
it takes care of themselves!
Ours(Trs) | i am so sorry to hear that . i hope she is doing better now
Ours(LM) | I’m so sorry. I know that feeling. Is she ok?
State | joy | surprise | acknowledging,questioning
Context | I have recently become vegan and I am very impressed with the food
reference | that is interesting . do you miss meat or dairy at all ?
MoEL | i love my food . i do not think i do not like to have those things like that !
MIME | that is great ! i love my family , but it was nice to have a great time
EmpDG | thats awesome ! what kind of i was it
MultiGPT | What type of foods are you getting?
Ours(Trs) | That is awesome. I have never considered vegan, so it is awesome.
Ours(LM) | that is awesome ! what kind of food is it ?
Table 7: Example responses generated by our proposed model and other
baselines. In State row, $*|*|*$ denotes speaker’s emotion, listener’s
emotion, and listener’s intent.

Gate Control. Fig.3 visualizes changing tendencies of $\lambda$ for our two
models during training. In the test set, for Ours(LM), gate control value
$\lambda$ ranges from 0.8536 to 0.8951. And for Ours(Trs), $\lambda$ ranges
from 0.2122 to 0.2588. For Ours(LM), it seems that more information is
provided by emotion states, whereas intents are also indispensable. For
Ours(Trs), such a conclusion is opposite. We analyze that maybe intent belongs
to elementary mental state and is easier to learn, but emotion belongs to
advanced mental state (Pettinelli, 2012) and plays a bigger role in
controlling generation. If the RespG of the model has already got a deep
semantic understanding, it will pay more attention to emotion states.

Figure 3: Impact of $\lambda$ value on model training.

###  4.8 Case Study

As illustrated in Table 7, we list two detailed cases generated by our
proposed models and other baselines. In the first case, nearly all the
generated responses correctly match the history situation, but responses from
our proposed models are more informative and easier to resonate with the
speaker. Our models not only convey a strong sympathetic feeling, but also
illustrate obvious empathy by using acknowledging and questioning propositions
like "I know that feeling, is it ok?". In the second case, although the
general response emotions of several baselines are approximately correct, our
models can also generate some keywords like "vegan" and "food". Besides,
responses of our models have fewer logical errors and grammar mistakes under
state constraints than those of other models.

##  5 Conclusion

In this paper, we propose a novel empathetic response generation model with
emotion-aware dialogue management. The emotion-aware dialogue management
consists of two modules: (1) the Emotion State Tracker maintains the current
emotion state of the user, and (2) the Empathetic Dialogue Policy Predictor
predicts the response emotion and the intent of the response. To model
interaction between the use’s emotion and the responding strategy, we also
introduce emotion shift patterns. Experimental results show that our model
outperforms other state-of-the-art baselines in both automatic and human
evaluations, and the proposed emotion-aware dialogue management can
effectively improve empathetic response generation.

In the future, it is possible to use some more powerful dialogue management
strategies such as reinforcement learning and adding empathy intensity
measurement as feedback to further improve the model performance.

## References

  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  _Advances in neural information processing systems_ , 28:1171–1179.
  * Chu et al. (2018) Eric Chu, Prashanth Vijayaraghavan, and Deb Roy. 2018.  Learning personas from dialogue with attentive memory networks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2638–2646.
  * Colombo et al. (2019) Pierre Colombo, Wojciech Witon, Ashutosh Modi, James Kennedy, and Mubbasir Kapadia. 2019.  Affect-driven dialog generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3734–3743, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, et al. 2020.  GoEmotions: A dataset of fine-grained emotions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4040–4054.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186.
  * Gao et al. (2019) Jun Gao, Wei Bi, Xiaojiang Liu, et al. 2019.  Generating multiple diverse responses for short-text conversation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):6383–6390.
  * Gao et al. (2021a) Jun Gao, Yuhan Liu, Haolin Deng, Wei Wang, Yu Cao, Jiachen Du, and Ruifeng Xu. 2021a.  Improving empathetic response generation by recognizing emotion cause in conversations.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 807–819, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gao et al. (2021b) Junfeng Gao, Yuhan Liu, Haolin Deng, Wei Wang, Yunlv Cao, Jiachen Du, and Ruifeng Xu. 2021b.  Improving empathetic response generation by recognizing emotion cause in conversations.  In _EMNLP_.
  * Ghosh et al. (2017) Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017.  Affect-LM: A neural language model for customizable affective text generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , pages 634–642.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, et al. 2020.  The curious case of neural text degeneration.  In _Proceedings of International Conference on Learning Representations_.
  * Huang et al. (2018) Chenyang Huang, Osmar Zaïane, Amine Trabelsi, and Nouha Dziri. 2018.  Automatic dialogue generation with expressed emotions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 49–54.
  * Kottur et al. (2017) Satwik Kottur, Xiaoyu Wang, and Vitor Carvalho. 2017.  Exploring personalized neural conversational models.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , pages 3728–3734.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, et al. 2016.  A diversity-promoting objective function for neural conversation models.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 110–119.
  * Li et al. (2020a) Qintong Li, Hongshen Chen, Zhaochun Ren, et al. 2020a.  EmpDG: Multi-resolution interactive empathetic dialogue generation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4454–4466.
  * Li et al. (2020b) Qintong Li, Piji Li, Zhumin Chen, and Zhaochun Ren. 2020b.  Towards empathetic dialogue generation over multi-type knowledge.  _arXiv preprint arXiv:2009.09708_.
  * Li et al. (2020c) Shifeng Li, Shi Feng, Daling Wang, Kaisong Song, Yifei Zhang, and Weichao Wang. 2020c.  Emoelicitor: An open domain response generation model with user emotional reaction awareness.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3637–3643.  Main track.
  * Lin et al. (2019a) Zhaojiang Lin, Andrea Madotto, Jamin Shin, et al. 2019a.  MoEL: Mixture of empathetic listeners.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 121–132.
  * Lin et al. (2019b) Zhaojiang Lin, Peng Xu, Genta Indra Winata, et al. 2019b.  Caire: An end-to-end empathetic chatbot.  _CoRR_ , abs/1907.12108.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _Proceedings of International Conference on Learning Representations_.
  * Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017.  Towards an automatic Turing test: Learning to evaluate dialogue responses.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , pages 1116–1126.
  * Majumder et al. (2020) Navonil Majumder, Pengfei Hong, Shanshan Peng, et al. 2020.  MIME: MIMicking emotions for empathetic response generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8968–8979.
  * Mercer and Reynolds (2002) Stewart W. Mercer and William J. Reynolds. 2002.  Empathy and quality of care.  _The British journal of general practice : the journal of the Royal College of General Practitioners_ , 52 Suppl:S9–12.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, et al. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Partala and Surakka (2004) Timo Partala and Veikko Surakka. 2004.  The effects of affective interventions in human–computer interaction.  _Interacting with computers_ , 16(2):295–309.
  * Pettinelli (2012) Mark Pettinelli. 2012.  _The psychology of emotions, feelings and thoughts_.  Lightning Source.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, et al. 2019.  Language models are unsupervised multitask learners.
  * Rashkin et al. (2018) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018.  I know the feeling: Learning to converse with empathy.  _ArXiv_ , abs/1811.00207.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, et al. 2019.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5370–5381.
  * Shen and Feng (2020) Lei Shen and Yang Feng. 2020.  CDL: Curriculum dual learning for emotion-controllable response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 556–566.
  * Shin et al. (2020) Jamin Shin, Peng Xu, Andrea Madotto, et al. 2020.  Generating empathetic responses by looking ahead the user’s sentiment.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7989–7993.
  * Song et al. (2019) Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, and Xuanjing Huang. 2019.  Generating responses with a specific emotion in dialog.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3685–3695.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems_ , volume 27, pages 3104–3112.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. 2017.  Attention is all you need.  In _Proceeding of Advances in neural information processing systems_ , pages 5998–6008.
  * Welivita and Pu (2020) Anuradha Welivita and Pearl Pu. 2020.  A taxonomy of empathetic response intents in human social conversations.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4886–4899.
  * Wolf et al. (2019) Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019.  Transfertransfo: A transfer learning approach for neural network based conversational agents.  _CoRR_ , abs/1901.08149.
  * Zandie and Mahoor (2020) Rohola Zandie and Mohammad H Mahoor. 2020.  Emptransfo: A multi-head transformer architecture for creating empathetic dialog systems.  In _Proceedings of The Thirty-Third International Flairs Conference_.
  * Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, et al. 2020a.  Bertscore: Evaluating text generation with bert.  In _Proceedings of International Conference on Learning Representations_.
  * Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 270–278.
  * Zhou et al. (2018) Hao Zhou, Minlie Huang, Tianyang Zhang, et al. 2018.  Emotional chatting machine: Emotional conversation generation with internal and external memory.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Zhou and Wang (2018) Xianda Zhou and William Yang Wang. 2018.  MojiTalk: Generating emotional responses at scale.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , pages 1128–1137.