11institutetext: Department of Computing Science, University of Alberta,
Alberta, Canada 11email<EMAIL_ADDRESS>
22institutetext: Cognitive Computing Research, Intel Labs, Santa Clara,
California, USA 22email<EMAIL_ADDRESS>
33institutetext: Computer Science and Engineering, Indian Institute of
Technology Patna, Patna, Bihar, India
33email<EMAIL_ADDRESS>
# A Unified Framework for Slot based Response Generation in a Multimodal
Dialogue System

Mauajama Firdaus denotes equal contribution by both authors11 Avinash Madasu0
22 Asif Ekbal 33

###### Abstract

Natural Language Understanding (NLU) and Natural Language Generation (NLG) are
the two critical components of every conversational system that handles the
task of understanding the user by capturing the necessary information in the
form of slots and generating an appropriate response in accordance with the
extracted information. Recently, dialogue systems integrated with
complementary information such as images, audio, or video have gained immense
popularity. In this work, we propose an end-to-end framework with the
capability to extract necessary slot values from the utterance and generate a
coherent response, thereby assisting the user to achieve their desired goals
in a multimodal dialogue system having both textual and visual information.
The task of extracting the necessary information is dependent not only on the
text but also on the visual cues present in the dialogue. Similarly, for the
generation, the previous dialog context comprising multimodal information is
significant for providing coherent and informative responses. We employ a
multimodal hierarchical encoder using pre-trained DialoGPT and also exploit
the knowledge base (Kb) to provide a stronger context for both the tasks.
Finally, we design a slot attention mechanism to focus on the necessary
information in a given utterance. Lastly, a decoder generates the
corresponding response for the given dialogue context and the extracted slot
values. Experimental results on the Multimodal Dialogue Dataset (MMD) show
that the proposed framework outperforms the baselines approaches in both the
tasks. The code is available at https://github.com/avinashsai/slot-gpt.

###### Keywords:

Conversational AI, Multimodal Dialogue System, Response Generation, DialoGPT

##  1 Introduction

Advancement in Artificial Intelligence (AI) has opened up new frontiers in
conversational agents. Human-machine interaction is an essential application
of AI helping humans in their day-to-day lives. Progress in AI has led to the
creation of personal assistants like Apple’s Siri, Amazon’s Alexa, and
Microsoft’s Cortana which assist humans in their everyday work. The machines’
capability to comprehend and complete the user’s goals has empowered
researchers to build advanced dialogue systems. Dialogue systems with the
ability to help users solve critical tasks such as selecting the appropriate
restaurants, and hotels, or booking movie tickets have gained immense
popularity in the field of artificial intelligence (AI). Through a well-
designed conversation system as an efficient personal assistant, users can
easily achieve everyday tasks through natural language interactions.

With the growth in AI, the latest progress in deep learning has encouraged
many neural conversational systems [60, 76, 29]. A typical goal-oriented
dialogue system comprises several key modules such as: (i) Natural language
understanding (NLU) component that helps in identifying the domain, intent and
extract slot information from the user utterance [77, 82, 8, 47]; (ii). a
dialogue state tracker (DST) that predicts the current dialogue state [90,
56]; (iii). a dialogue policy that regulates the next system action given the
current state [74, 49]; (iv). a natural language generator (NLG) module that
outputs a response given the semantic frame [62, 61, 71, 38]. These modules
occur in a pipeline in every robust dialogue system. Therefore, it is slightly
time-consuming and computationally expensive.

With the progress in AI, the integration of information from different
modalities, such as text, image, audio, and video, has been known to provide
complete information for building effective end-to-end dialogue systems [58,
27, 39, 37] by bringing the different areas of computer vision (CV) and
natural language processing (NLP) together. Hence, a multimodal dialogue
system bridges the gap between vision and language, ensuring interdisciplinary
research.

Multimodal conversational systems provide completeness to the existing
dialogue systems by providing necessary information that lacks in unimodal
systems as the visual (in the case of images and videos) and audio information
help build robust systems. In [58] the authors proposed a multimodal dialogue
dataset having textual and image information for the fashion domain. From the
dataset, it is clear that image information is necessary for selecting the
right clothes and accessories for different individuals.

Figure 1: NLU and NLG modules in a Dialogue System

###  1.1 Motivation and Contribution

As demonstrated in Figure 1, the primary goal of every NLU component is to
extract necessary information in the form of slots from the user utterance,
while the ultimate goal of the NLG module is to respond to the user based on
the extracted semantic information. Both these tasks are complementary; hence
information extracted from the NLU is significant for generating the correct
response by the NLG unit. Instead of performing these tasks separately in a
pipeline manner, recently, researchers have focused on performing these tasks
simultaneously to improve the performance of both tasks [68, 18].

We take a step forward in our current work by proposing an end-to-end system
that can concurrently extract the necessary slot information from the user
utterance and provide the corresponding system response in a multimodal
dialogue setting. This is more challenging as the slot information is not
entirely dependent on the textual utterance but also on the visual
information. Hence, for a better generation of responses, extraction of the
correct semantic information from the current dialogue context is crucial.

Slots are crucial as it provides the key semantic information for a better
understanding of the user utterance. To provide informative responses to the
user it is important to capture the semantic information in the form of slots.
Based on the slot values, the response generation module can provide responses
that are informative and engaging. The proposed end-to-end framework first
captures the slot information and then uses this slot information captured
from both text and image as input for the generation module.

The key contributions of our current work are three-fold:

  * •

We propose the task of simultaneously performing two critical components of
every conversational system, i.e. NLU and NLG, in a multimodal dialogue system
employing information from both text and images.

  * •

We design a slot attention-based hierarchical generation system using pre-
trained DialoGPT.

  * •

Our proposed system achieves the best performance compared to the existing and
baseline approaches in both tasks.

The rest of the paper is structured as follows. In Section 2, we present a
brief review of the existing literature. We provide the details of the
baseline and the proposed methodology in Sections 3 and 4 respectively. In
Section 5, we provide the details of the dataset used and its statistics
followed by implementation details and evaluation metrics. Experimental
results are presented in Section 6 along with a detailed analysis, including
error analysis. Finally, in Section 7 we conclude with future directions of
research.

##  2 Related Work

In any dialogue framework, Natural Language Generation (NLG) is a classic
problem. With the fast growth of Artificial Intelligence (AI), there has been
a trend in recent times to develop multimodal dialogue systems by combining
text with images, audio and video modalities. A brief description of some of
the works carried out in unimodal chatbots, accompanied by multimodal dialogue
systems for both the tasks of slot filling and response generation, is
provided below.

###  2.1 Slot Filling

Several deep learning architectures have also been employed for extracting
essential information in the form of slots from a given utterance. The authors
in [16] investigated deep belief networks (DBN) for slot filling on the ATIS
dataset. In [42], the authors investigated Elman and Jordan-type RNNs for slot
filling. In [41], several hybrid variants of RNN were proposed due to the
stronger ability of RNNs to capture dependencies compared to traditional
models, such as Conditional Random Field (CRF). In [86] lexical, syntactic and
word-class features were used as input to an RNN for the SLU task of slot
filling.

The authors in [85] used the transition features to improve RNNs and the
sequence level criteria for optimisation of CRF to capture the dependencies of
the output label explicitly. The authors in [84] used deep LSTMs along with
regression models to obtain the output-label dependency for slot filling. The
usage of kernel deep convex networks (K-DCN) was investigated in [15] for slot
filling. In [92], a focus mechanism for an encoder-decoder framework was
proposed for slot filling on the ATIS dataset. The authors in [89] introduced
a generative network based on the sequence-to-sequence model along with a
pointer network for slot filling. In [63], an attention-based encoder-decoder
framework has been employed for slot filling.

In [52], a pre-trained language model was employed in an RNN framework for the
slot-filling task. Attention-based RNN framework was proposed in [81] along
with pre-trained word embeddings for identifying the slots on ATIS and MEDIA
datasets. On the ATIS dataset in [26], an adversarial multi-task model
combining a bi-directional language model with a slot tagging model was used
for identifying the slots in a given user utterance.

The adversarial framework was used in [34] for learning common representation
across multiple domains for slot-filling tasks. In [93], the authors proposed
the concept of transfer learning for the task of slot filling as it is an
essential task of language understanding. Authors in [77] encoded lexicon
information as features for use in a Long-short term memory neural network for
slot-filling tasks. With advancements in AI, multimodality has been
incorporated into conversational systems to make them more robust and
complete. Recently, authors in [88] used an adaptive attention mechanism to
extract the necessary slot values in a multimodal dialogue system.

###  2.2 Response Generation

####  2.2.1 Unimodal Dialogue System:

The effectiveness of deep learning clearly shows significant improvements in
dialogue generation. Deep neural models are very effective in modelling the
dialogues, as seen in [73, 62]. In [66], a context-sensitive neural language
approach was presented where, given the textual conversational background, the
model chooses the most likely answer. To capture the context of the previous
queries by the users, the authors in [65] proposed a hierarchical framework
capable of preserving past information. Sequence-to-sequence (seq2seq) neural
models often generate incomplete and boring responses, such as “I don’t know”,
“Okay”, “Yes”, “No”, etc. Hence, bringing diversity in responses is an
extremely challenging and interesting research problem for every
conversational agent.

Similarly, to preserve the dependencies among the utterances, a hierarchical
encoder-decoder framework was investigated in [60, 61]. The authors in [83]
extended the hierarchical encoder-decoder framework by adding a latent
variable for understanding the intentions of the conversations in a task-
oriented dialog system. Lately, memory networks [40] have been intensely
investigated for capturing the contextual information in dialogues for the
generation of responses infusing pointer networks.

Hierarchical pointer networks [54] has also been employed for response
generation in task-oriented dialogues. The authors in [80] incorporated a
global encoder and a local decoder to share external knowledge in a task-
oriented dialogue setup. The ability to infuse knowledge in responses was
achieved by using a Bag-of-sequence memory unit [55] for generating coherent
responses in goal-oriented dialogue systems. The authors in [57] proposed a
multi-level memory framework for task-oriented dialogues. A memory-augmented
framework with the ability to extract meaningful information during training
for better response generation has been explored in [71]. With the release of
MultiWoz [6], a task-oriented dialogue dataset, several works have focused on
multi-domain dialogue generation.

The authors in [5] used a pre-trained language model for dialogue generation.
A hierarchical graph framework employing the dialogue acts of the utterances
was investigated for dialogue generation in [9]. The meta-learning approach
[43, 50] has been applied to different datasets to increase the domain
adaptability for generating the responses. To increase the ability to memorize
the dialogue context, the authors in [79] used a memory-to-sequence framework
and the pointer generator for response generation. A multi-task framework to
enhance the performance of natural language generation was investigated in
[91].

In [10], working memory was employed for dialogue generation. The working
memory interacts with two long-term memories that capture the dialogue history
and the knowledge base tuples for the informative response generation.
Recently, a heterogeneous memory network [33] has been explored for response
generation having the capability to simultaneously use the dialogue context,
user utterance, and the knowledge base for response generation. Dynamic fusion
technique has been employed in [51] to share the features across different
domains for a better generation.

####  2.2.2 Multimodal Dialogue System:

Research in the dialogue system has recently shifted towards incorporating
different sources of information, such as images, audio, video, and text in
order to make a robust system. The research reported in [13, 45, 14, 23, 19]
has been useful in narrowing the gap between vision and language. In [45], an
Image Grounded Conversations (IGC) task was proposed, where conversations are
natural and focused upon a shared image. Similarly, the authors in [13]
introduced the task of visual dialogue, which requires an AI agent to hold a
meaningful dialogue with humans in natural, conversational language about the
visual content.

Recently, video and textual modalities were investigated with the release of
the DSTC7 dataset in [28] that used a multimodal transformer network to encode
videos and incorporate information from the different modalities. Similarly in
[27, 4, 32], the DSTC7 dataset has been used for generation by incorporating
audio and visual features. The release of the Multimodal dialogue (MMD)
dataset [58], having conversations on the fashion domain in both text and
images, has facilitated response generation in multimodal setup.

Several works on the MMD dataset reported in [2, 1, 30] used the hierarchical
encoder-decoder model to generate responses by capturing information from
text, images, and the knowledge base. Recently, [7] proposed attribute-aware
and position-aware attention for generating textual responses. The authors in
[12] used a hierarchical attention mechanism for generating responses on the
MMD dataset. In [20], the authors proposed a stochastic method for generating
diverse responses in a multimodal dialogue setup. Multi-domain multi-modal
aspect controlled response generation task was introduced in [21].

Lately, the authors have focused on jointly addressing NLU and NLG tasks in a
unimodal framework [72, 68, 18] for improving the performance of both tasks.
Author’s in [72] proposed a generative model which couples NLU and NLG through
a shared latent variable. Similarly, in [68] a new learning framework was
designed for language understanding and generation on top of dual supervised
learning, providing a way to exploit the duality.

Our current work differs from the existing NLU and NLG works as we intend to
build a comprehensive framework that extracts the necessary slot information
and generates the appropriate response adhering to the elicited slot
information in a multimodal framework. The task becomes more complex as visual
cues in the form of images are also crucial for providing the complete context
for both the tasks along with the textual information.

##  3 Methodology

In this section, we discuss the problem statement followed by the baseline and
the proposed methodology.

###  3.1 Problem Definition

In this paper, we address the task of extracting the slot values from the user
utterance and generating informative and relevant textual responses according
to the conversational history in a multimodal dialogue setting. The dialogues
consist of textual utterances along with multiple images. More precisely,
given a user utterance $U_{p}=u_{p,1},u_{p,2},...,u_{p,j}$, a set of images
$I_{p}=img_{p,1},img_{p,2},...,img_{p,j^{\prime}}$, with the dialogue history
$H_{p}=(U_{1},I_{1}),(U_{2},I_{2}),...,(U_{p-1},I_{p-1})$, we focus on
extracting the slot information from $U_{p}$ and $I_{p}$ and simultaneously
generate interesting, informative, context-aware response
$Y_{p}=(y_{p,1},y_{p,2},\ldots,y_{p,k}$) instead of template like generic and
monotonous responses, such as I don’t know, Yes, No, Similar to…, etc. This
will enhance human-machine conversations by keeping the users engaged in the
conversation. Here, $p$ is the $p^{th}$ turn of a given dialogue, while $j$ is
the number of words in a given textual utterance and $j^{\prime}$ is the
number of images in a given utterance. Note that in every turn, the number of
images $j^{\prime}$ $\leq$ 5, so in the case of only text, vectors of zeros
are considered in place of image representation.

Figure 2: Architecture of the proposed model

###  3.2 Multimodal Hierarchical Encoder Decoder:

We construct a generative model for response generation, an extension of the
recently introduced Hierarchical Encoder-Decoder (HRED) architecture [61, 60].
As opposed to a standard sequence-to-sequence model [69], the dialogue context
among the utterances is captured by adding utterance-level RNN (Recurrent
Neural Network) over the word-level RNN, increasing the efficacy of the
encoder to capture the hierarchy in dialogue. The multimodal HRED (MHRED) is
built upon the HRED to include text and image information in a single
framework. The critical components of MHRED are the utterance encoder, image
encoder, context encoder, and decoder.

####  3.2.1 Utterance Encoder:

Given an utterance $U_{p}$, we use bidirectional Gated Recurrent Units (BiGRU)
[11] to encode each word $n_{p,i}$, where $i\in(1,2,3,.....k)$ having
$d$-dimensional embedding vectors into the hidden representation $h_{U,p,i}$
as follows:

$\begin{split}\overrightarrow{h_{U,p,i}}=GRU_{U,f}({n_{p,i}},\overrightarrow{h_{U,p,i-1}})\\\
\overleftarrow{h_{U,p,i}}=GRU_{U,b}({n_{p,i}},\overleftarrow{h_{U,p,i-1}})\\\
h_{U,p,k}^{txt}=[\overrightarrow{h_{U,p,i}},\overleftarrow{h_{U,p,i}}]\end{split}$
(1)

here $\overrightarrow{h_{U,p,i}}$ represent the utterance representation in
the forward direction while $\overleftarrow{h_{U,p,i}}$ represents in the
backward direction. The overall representation of the utterance is given by
$h_{U,p,k}^{txt}$.

####  3.2.2 Image Encoder:

A pre-trained VGG-16 [64] having a 16-layer deep convolutional neural network
(CNN) trained on more than one million images present in the ImageNet dataset
is used for encoding the images. It can classify images into 1000 object
categories, such as dresses, shoes, animals, keyboards, mouse, etc. As a
result, the network can learn rich features from a wide range of images. Here,
it is also used to extract the “local” image representation for all the images
in the dialogue turns and concatenates them together. The concatenated image
vector is passed through the linear layer to form the global image context
representation as given below:

$\begin{split}T_{p,i}=VGG(I_{p,i})\\\
T_{p}=Concat(T_{p,1},T_{p,2},\ldots,T_{p,j’})\\\
h_{I,p}^{img}=ReLU(W_{I}T_{p}+b_{I})\end{split}$ (2)

where $W_{I}$ and $b_{I}$ are the trainable weight matrix and biases. In every
turn, the number of images $i\leq$ 5, so in the case of only text, vectors of
zeros are considered in place of image representation.

####  3.2.3 Context Encoder:

The final hidden representations from both image and text encoders are
concatenated for each turn and are given as input to the context-level GRU. A
hierarchical encoder is built to model the conversational history on top of
the image and text encoder. The decoder GRU is initialized by the final hidden
state of the context encoder.

$h_{w,p}^{ctx}=GRU_{w}([h_{U,p,k}^{txt};h_{I,p}^{img}],h_{w,p-1})$ (3)

where $h_{w,p}^{ctx}$ is the final hidden representation of the context for a
given turn.

####  3.2.4 Decoder:

In the decoding section, we build another GRU for generating the words
sequentially based on the hidden state of the context GRU and the previously
decoded words. We use input feeding decoding and the attention [36] mechanism
for enhancing the performance of the model. Using the decoder state
$h_{d,t}^{dec}$ as the query vector, the attention layer is applied to the
hidden state of the context encoder. The context vector and the decoder state
are concatenated and used to calculate a final distribution of probability
over the output tokens.

$\begin{split}h_{q,t}^{dec}=GRU_{d}(y_{p,t-1},h_{q,{t-1}});\\\
\alpha_{t,m}=softmax({{h_{w,p}^{ctx}}^{T}}{W_{f}}{h_{q,t}})\\\
c_{t}=\sum_{m=1}^{k}{\alpha_{t,m}}{h_{w,p}^{ctx}},\\\
\tilde{h}_{t}=tanh(W_{\tilde{h}}{[{h_{q,t}};c_{t}]});\\\
P(y_{t}/y_{<t})=softmax({W_{S}}{\tilde{h}_{t}})\end{split}$ (4)

where, $W_{f}$, $W_{S}$ and $W_{\tilde{h}}$ are trainable weight matrices.

###  3.3 Proposed Approach:

To further improve the MHRED model’s performance, we propose to apply slot
attention to the utterances. The goal is to focus on the slot values in a user
utterance crucial in generating an appropriate system response. If the model
fails to attend to vital slot information, then it generates an inappropriate
system response. For example, for the given user utterance “Will neck-tie
having 25 cm size be paired well with any of these?”, the slot values neck-tie
and 25 cm are crucial to understanding user utterance.

Furthermore, in a Multimodal Dialogue system, the user often refers to the
images generated in the previous system responses. The model should account
for this subtle but very essential information. For example, in the following
user utterance, “Show me more in style as in the 4th image”, the model must
understand that the user is referring to the 4th image. Any failure in doing
so will generate an inapt system response. Hence, in our proposed approach, we
employ mechanisms to improve the performance of slot identification. The
architecture of the proposed model is shown in Figure 2.

####  3.3.1 Slot Attention:

We employ self-attention on the output from Utterance Encoder
$h_{U,p,k}^{txt}$ as in the final Equation 1. We refer to Slot Attention as SA
in the rest of the paper. Let $h_{U,p,k}^{txt}$ be the Key (K), Query (Q) and
Value (V).

$SA(K,Q,V)=softmax(\frac{QK^{\rm T}}{\sqrt{d_{k}}})V.$ (5)

where $d_{k}$ is the hidden dimension size of $h_{U,p,k}^{txt}$. The output
from Slot Attention is concatenated with the Image Encoder’s output. The
concatenated output is sent as an input to the Context Encoder.

####  3.3.2 Knowledge Base (KB):

The knowledge base encoder used in our framework is the same as [2]. The
knowledge base of the MMD dataset contains information about contextual
queries and celebrities endorsing various products and brands. Hence, to
provide this additional information to our proposed model, we employ self-
attention on KB input to achieve more focused information as follows:

$\begin{split}h_{s}^{query}=n_{p}^{query}(h_{n-1}^{query},k_{l,t})\\\
h_{f}^{ent}=n_{p}^{ent}(h_{n-1}^{ent},d_{l,t})\\\
h_{net}^{kb}=[h_{s}^{query},h_{f}^{ent}]\end{split}$ (6)

Let $h_{net}^{kb}$ be the Key (K), Query (Q) and Value (V).

$SA(K,Q,V)=softmax(\frac{QK^{\rm T}}{\sqrt{d_{k}}})V.$ (7)

where $d_{k}$ is the hidden dimension size of $h_{net}^{kb}$. We use the
attended KB output and the decoder input as the combined input at each time
step of the decoder.

As the knowledge base (KB) input remains intact for a particular dialogue
context, we concatenate the KB input with the decoder input in a similar
manner as [2].

####  3.3.3 Pretraining DialoGPT (P-GPT):

In a dialogue system, understanding contextual information is crucial to
performance enhancement. Pre-trained language models have achieved state-of-
the-art results on several Natural Language Understanding (NLU) tasks [53].
Furthermore, pre-training dialogue systems significantly improved the
performance of generation [25]. Therefore, we pre-train Multi-modal Dialog
(MMD) dataset using DialoGPT [87]. The input to the DialoGPT is a combination
of the previous system response, the current user utterance, and the current
system response. This helps the model to learn long-range contextual
information effectively. We use DialoGPT-small for a context size of 2. The
pre-trained contextual embeddings are passed as input to the Text Encoder and
KB Encoder.

####  3.3.4 Slot Prediction:

For slot prediction, we take the scores obtained by applying the softmax layer
on the output of the dot product between Query (Q) and Key (K) in the self-
attention for a given user utterance to find the distribution on the slot
values for a given user utterance.

####  3.3.5 Training and Inference:

The generation model is trained using teacher-forced cross entropy [78] at
every decoding step to minimize the negative log likelihood on the model
distribution. We define
$\hat{y}={\hat{y}_{1},\hat{y}_{2},\hat{y}_{3},\ldots,\hat{y}_{m}}$ as the
ground truth of the given input sequence.

$J(\theta)=-\sum_{s=1}^{n}logp({\hat{y}_{t}}|y_{1},y_{2},\ldots,y_{m-1})$ (8)

here $y_{1}$, $y_{2}$ $\ldots$ and $y_{m-1}$ is the generated utterance.

##  4 Comparison Methods

In this section, we provide a comparison with the other existing techniques
comprising both state-of-the-art methods and other baselines.

###  4.1 State-of-the-Art Models:

Seq2Seq: It is an encoder-decoder framework with attention which is a standard
baseline in Machine Translation, Generation [69]. The input to the encoder is
dialogue history and the decoder output is the next round-generated dialogue.

HRED: It is the first hierarchical encoder-decoder architecture proposed for
text-based dialogue systems and is a standard baseline for Unimodal system
[59]. It also follows a similar input-output format as the Seq2Seq model.

MHRED: Multimodal hierarchical encoder-decoder is the first model proposed for
Multimodal dialogue systems. Along with text, image is also served as input
[58]. The input to the encoder is a concatenated input of image and text
features and the decoder output is the generated dialogue.

UMD: A user-guided attention model is proposed to consider hierarchical
product taxonomy and users’ attention to products. It is based upon MHRED
architecture [12]. The attention model focuses on user preferences for the
products and generates responses based on them.

OAM: In this paper, a novel position and attribute-aware attention mechanism
are proposed to learn the enhanced image representation conditioned on the
user utterance. The proposed model can generate appropriate responses while
preserving the position and attribute information [7].

MAGIC: Multimodal diAloG system with adaptIve deCoders (MAGIC) first judges
response through understanding user intention. It then applies an adaptive
decoder for generating apposite responses [46].

MATE: It is based on the standard transformer architecture. In the encoding
stage, the transformer encoder is used to encode information from multimodal
input. Generation is a two-stage process based on the transformer decoder. In
the first stage, the focus is more on the encoded information. In the second
stage, responses are refined by incorporating domain knowledge into the output
of the first stage [24].

LXMERT: In LXMERT [70], the authors build a large-scale Transformer model that
consists of three encoders: an object relationship encoder, a language
encoder, and a cross-modality encoder. We concatenate all the images and feed
them as input to the visual encoder (i.e., object-relationship encoder) and
the language encoder is used for utterance representation. Finally, the cross-
modality encoder is used to capture the final utterance representation for
both text and images.

###  4.2 Baseline Models:

To show the effectiveness of the proposed components, we implement the models
without these components in the architecture.
Unimodal Baselines: In order to prove that multimodal architectures perform
better, we compare these with unimodal architectures. In unimodal
architectures, only text served as the input to the models.
Without Kb: We experiment with the models without using the Knowledge base as
inputs at the decoder. It is used as a baseline to compare with the models
using the Knowledge base as input.
Without Slot Attention: In these models, Slot Attention (SA) is not applied.
It is used as a comparison to demonstrate the efficiency of the Slot Attention
component.
Without Pretrained Dialog-GPT (P-GPT) representations: To prove our hypothesis
that Pretrained Dialog-GPT improves performance, we perform experiments
without Pretrained Dialog-GPT representations as input. It is to show the
performance differentiation between the models using P-GPT and without using
P-GPT.

##  5 Dataset and Experiments

In this section, we provide the details of the datasets used for experiments,
implementation details, evaluation metrics and the results obtained.

###  5.1 Dataset Description:

Our research is based on the Multi-modal Dialog (MMD) dataset [58]
111https://amritasaha1812.github.io/MMD/download/ consisting of 150k chat
sessions between the customer and sales representative. During the sequence of
customer-agent interactions, domain-specific information in the fashion domain
was collected. The dialogues easily integrate text and image knowledge into a
conversation that brings together different modalities to create a
sophisticated dialogue system.

The dataset presents new challenges for multi-modal, goal-oriented dialogue
systems having complex user sentences. The detailed information of the MMD
dataset is presented in Table 1. The authors [58], for experimentation
“unroll” the different images to incorporate only one image for a single
utterance. Though computationally learns, the method eventually lacks the goal
of capturing multi-modality over the context of multiple images and text.
Therefore, in our study, we use a different version of the dataset as outlined
in [1, 2] to capture a large number of images as the concatenated context
vector for each turn of a dialogue. The motivation behind this is that
multiple images are required to provide the correct responses to the users.

Dataset Statistics Train Valid Test Number of dialogues 105,439 22,595 22,595
Avg. turns per Dialogue 40 40 40 No. of Utterances with Text Response 1.54M
331K 330K Avg. words in Text Response 14 14 14 No. of Utterances with Image
Response 904K 194K 193K

Table 1: Dataset statistics of Multi-modal Dialogue (MMD) dataset

###  5.2 Implementation Details

####  5.2.1 DialoGPT Pretraining

We used the DialoGPT-small model for pre-training. Previous system response,
current user utterances and current system response are concatenated together
into a single sentence separated by a special token. The goal is to capture
long range contextual information. Adam is used as the optimizer with a
learning rate of 5e-5 and a batch size of 16. The pretraining is performed for
3 epochs. The maximum value of the gradient norm is 1.0. The model’s weights
are initialized with the already pre-trained weights in DialoGPT paper [87].
The loss function used is the same as the one used in the DialoGPT paper.

####  5.2.2 Model Training

All the implementations are done using the PyTorch222https://pytorch.org/
framework. The input embedding dimensions are 512 for randomly initialized
word embeddings and 768 for pre-trained contextual embeddings. The hidden size
for all the layers is 512. A dropout [67] of 0.3 is applied on the Slot
Attention for all T-HRED models and 0.5 for all M-HRED models. All the models
are trained for 15 epochs with a batch size of 256. AdamW [35] is used as the
optimizer with a learning rate of 0.0001 for all the models. For image
representation, a 4096-dimensional FC6 layer from VGG-19 network [64] is used,
which is trained on ImageNet.

###  5.3 Automatic Evaluation Metric:

To evaluate the model at the relevance and grammatical level, we report the
results using standard metrics like Rouge-L [31] and BLEU-1,2,3 and 4 [48].
For comparison with the existing approaches, we report NIST metric [17] in a
similar manner as [24]. To evaluate the slot value extraction performance, we
use the traditional metrics such as F1 score and Accuracy similarly as [77,
88]. For accuracy and F1 scores, the percentage values in the range ¿=70% and
¡80%- represent fair results; while $>=$80% and $<$90% represents good models;
and $>=$90% represents very good/excellent models.

###  5.4 Human Evaluation Metrics

We recruit six annotators (in a similar manner as [62, 71]) from a third-party
company, having high-level language skills. We sampled 500 responses per model
for evaluation with the utterance and the conversational history provided for
a generation. First, we evaluate the quality of the response on two
conventional criteria: Fluency and Relevance. We also compute slot consistency
for our proposed task which determines whether the generated response is
consistent with the predicted slot information. These are rated on a five-
scale, where 1, 3, and 5 indicate unacceptable, moderate, and excellent
performance, respectively, while 2 and 4 are used for unsure. We compute
Fleiss’ kappa [22] to measure inter-rater consistency. The Fleiss’ kappa for
Fluency and Relevance is 0.53 and 0.49, indicating moderate agreement. For
Slot Consistency, we obtain 0.65 as the kappa score indicating substantial
agreement.

Model Description Accuracy F1 Score Unimodal Baselines P-GPT HRED + SA 57.8
57.4 P-GPT HRED + Kb + SA 59.4 58.6 Multimodal Baselines P-GPT MHRED + SA 60.5
59.2 Proposed Approach P-GPT MHRED + SA + Kb 62.4 61.3 P-GPT MTrans + SA + Kb
65.1 63.8 P-GPT Mul-Trans + SA + Kb 66.3 64.7

Table 2: Evaluation results on Slot information. Here, P-GPT is the Pre-
trained DialoGPT

(a) Example 1

(b) Example 2

(c) Example 3

(d) Example 4

Figure 3: Attention visualization

##  6 Result and Analysis

This section presents the experimental results for both the tasks and the
necessary analysis of the baseline models generated responses and the proposed
methodology.

###  6.1 Slot Prediction Results

The results of the slot values are provided in Table 2. This shows that the
proposed approach outperforms all the existing baseline models (both unimodal
and multimodal), and these improvements are statistically significant. With
the addition of external knowledge, the slot performance improves
significantly, thereby assisting the framework in capturing the correct values
from the given user utterance. Besides, the pre-trained DialoGPT embeddings
help provide a stronger context for the task with a gain of approximately 3%
from the baseline MHRED + Kb model. This justifies that both the pre-trained
embeddings and external knowledge, along with slot attention, are crucial in
providing the slot information for correct extraction and finally assist in a
generation.

In Figure 3, we provide a few attention visualizations of the user utterances
to show that the slot attention designed to extract the correct values is
efficient in capturing the correct information. The slot attention in Example
1 has been able to focus on the colour brown and material leather, providing
the right slot values. Similarly, in Example 2 the proposed framework has
correctly attended the colour information purple grey and position of the
image $1^{st}$ in the given user utterance.

By utilizing transformers as an encoder we see a boost in performance with an
improvement of 2% in F1 score and around 3% in accuracy for the slot
prediction task. The obvious reason behind the improvement is the ability of
the transformer network to better capture the contextual information of a
given utterance in comparison to the MHRED model which uses GRU as the basic
cell for encoding the utterance.

The proposed approach achieves a performance gain of more than 6% and 11% in
accuracy with respect to the unimodal baselines having the knowledge
information and without knowledge respectively. Similarly, by adding the
multimodal information using the transformer networks there is an increase of
3% compared to P-GPT MHRED + SA + Kb framework. This proves that the
transformer approach captures better semantic knowledge as opposed to the
encoder-decoder method.

Model Description SA P-GPT BLEU-1 BLEU-2 BLEU-3 BLEU-4 Rouge-L Unimodal
Baselines HRED - - 0.624 0.535 0.475 0.425 0.666 $\surd$ - 0.635 0.544 0.483
0.433 0.668 $\surd$ $\surd$ 0.638 0.547 0.486 0.436 0.671 HRED + Kb - - 0.646
0.560 0.503 0.456 0.685 $\surd$ - 0.657 0.565 0.505 0.460 0.682 $\surd$
$\surd$ 0.659 0.571 0.507 0.460 0.683 Multimodal Baselines MHRED - - 0.630
0.541 0.478 0.43 0.669 $\surd$ - 0.636 0.545 0.484 0.434 0.668 $\surd$ $\surd$
0.638 0.547 0.487 0.437 0.672 Proposed Approaches MHRED + Kb - - 0.649 0.563
0.503 0.455 0.685 $\surd$ - 0.659 0.571 0.512 0.463 0.688 $\surd$ $\surd$
0.662 0.574 0.514 0.465 0.690 MTrans + Kb - - 0.665 0.569 0.515 0.462 0.693
$\surd$ - 0.673 0.579 0.527 0.470 0.711 $\surd$ $\surd$ 0.684 0.587 0.535
0.479 0.724 Mul-Trans + Kb - - 0.671 0.575 0.524 0.473 0.704 $\surd$ - 0.679
0.582 0.531 0.481 0.713 $\surd$ $\surd$ 0.690 0.591 0.544 0.493 0.733

Table 3: Results using automatic evaluation metrics. Here, P-GPT is the pre-
trained DialoGPT; HRED, MHRED, HRED + Kb, and MHRED + Kb framework without SA
and P-GPT is similar to [1, 2] respectively.

###  6.2 Response Generation Results

####  6.2.1 Results on Automatic Evaluation:

In Table 3333we perform statistical significance t-test [75], and it is
conducted at 5% (0.05) significance level, we present the results of automatic
evaluation. As already stated, we report BLEU 1,2,3 and 4 as BLEU measures the
n-grams overlap between the generated response and the gold response that
would help measure if the extracted slot values improve the performance of
generation. From the results shown in the table, it is evident that our
proposed framework performs significantly better in comparison to the baseline
models. As the primary objective was to ensure that slot extraction helps in a
better generation, the results justify that by capturing the correct values by
the slot attention mechanism, we achieve better performance in the case of all
the metrics. There is a definite improvement of 1% in the case of baseline and
the proposed framework by incorporating slot attention. This proves that slot
knowledge makes the generated response more informative by adding the correct
slot values in the responses.

Multimodality in the form of images plays a crucial role in building robust
systems. In our framework, visual information in the form of images has been
incorporated, and from the table, it is obvious that there is a slight
improvement in the performance as opposed to the unimodal frameworks having
only textual information. Finally, we employ pre-trained DialoGPT embeddings
to enhance the performance of the overall generation process. From the table,
it is visible that by utilizing the pre-trained embeddings, there is a gain in
performance in the proposed approach and the baselines. In particular, there
is approximately 1 point improvement in the final model with DialoGPT
embeddings compared to the framework without any pre-trained embedding. This
ensures that pre-training is beneficial in capturing better context, thereby
providing stronger dialogue information to generate informative and coherent
responses. The multimodality information provided more complementary
information that is not presented by the textual modality thereby improving
the overall performance.

By employing transformer along with a knowledge base without having the slot
information and pre-trained embeddings, we see that it performs better than
the MHRED + Kb model in terms of Rouge-L and BLEU scores. By adding slot
information and pre-trained embedding, the performance of the model improves
significantly. In Mul-Trans + Kb framework, we use multimodal transformers in
the sense that for utterance representation we use transformers while the
image representation achieved from VGG-19 is fed as input to a transformer
network in a similar manner as [3]. Here, the utterance information from the
transformer along with the output of the image representation from the
transformers is concatenated to get the context of the entire utterance
representation having both textual and visual knowledge. The representation
achieved from both transformers is then used for generating the response.

As evident from Table 3, the model with transformer representations for both
textual and visual representation achieves performance improvement over the
MTrans + Kb framework that uses a transformer network only for utterance
representation. We also compare our framework with the LXMERT [70], model and
we see that it performs better compared to all the baselines still our
proposed network outperforms LXMERT. This is due to the fact that LXMERT
captures object-oriented features accompanied by captions for a single image.
But in our case, we don’t have captions and also the images are multiple in
number compared to the LXMERT framework. Also, certain utterances do not have
visual information in most of the dialogues.

Evidently, the performance of the Mul-Trans + Kb model is significantly better
as opposed to the RNN networks due to the capability and efficacy of the
transformers in capturing better-contextualized representations using multi-
head attention and feed-forward networks. The Rouge-L score and BLEU scores
are the highest compared to all the baselines for the proposed Mul-Trans + Kb
model.

Model BLEU NIST 1 2 3 4 Unimodal Baselines Seq2Seq [69] 35.39 28.15 23.81
20.65 3.3261 HRED [59] 35.44 26.09 20.81 17.27 3.1007 Multimodal Baselines
MHRED [58] 32.60 25.14 23.21 20.52 3.0901 UMD [12] 44.97 35.06 29.22 25.03
3.9831 OAM [7] 48.30 38.24 32.03 27.42 4.3236 MAGIC [46] 50.71 39.57 33.15
28.57 4.2135 MATE [24] 56.55 47.89 42.48 38.06 6.0604 LXMERT [70] 64.32 51.33
45.33 42.76 7.3855 Proposed Approach P-GPT + MHRED + SA + (Joint Training)
66.20 57.40 51.40 46.50 6.3164 P-GPT + MTrans + SA + (Joint Training) 68.40
58.70 53.50 47.90 8.1629 P-GPT + Mul-Trans + SA + (Joint Training) 69.00 59.10
54.40 49.30 8.5371

Table 4: Results of our proposed approach and the existing baselines

####  6.2.2 Comparison to the Existing Approaches:

In Table 4, we present the evaluation results of our proposed framework in
comparison to the existing approaches. From the table, it is clearly evident
that the use of the slot values improves the generation performance compared
to the existing approaches that do not employ slot information for generation.
The BLEU-4 score shows an improvement of more than 20 points compared to the
unimodal baselines, such as Seq2Seq and HRED networks.

By using the images, we see that the existing approaches have shown a notable
gain in performance as opposed to the unimodal baselines. By using the pre-
trained GPT embeddings and slot information, we outperform the best-performing
framework [24], with a BLEU score of 8%. From this, it can be concluded that
slot information assists in correctly responding to user demands and providing
interesting and informative responses.

Model SA P-GPT F R SC Unimodal Baselines HRED - - 3.43 3.27 2.91 $\surd$ -
3.51 3.36 3.03 $\surd$ $\surd$ 3.57 3.43 3.12 HRED + Kb - - 3.61 3.49 3.17
$\surd$ - 3.69 3.55 3.23 $\surd$ $\surd$ 3.78 3.63 3.38 Multimodal Baselines
MHRED - - 3.85 3.72 3.49 $\surd$ - 3.91 3.80 3.57 $\surd$ $\surd$ 3.99 3.85
3.68 Proposed Approaches MHRED + Kb - - 4.08 3.87 3.71 $\surd$ - 4.15 3.92
3.78 $\surd$ $\surd$ 4.16 4.02 3.82 MTrans + Kb - - 4.29 4.02 3.87 $\surd$ -
4.36 4.18 4.05 $\surd$ $\surd$ 4.42 4.20 4.13 Mul-Trans + Kb - - 4.33 4.14
3.96 $\surd$ - 4.42 4.25 4.12 $\surd$ $\surd$ 4.53 4.37 4.22

Table 5: Evaluation results using human evaluation metrics. Here, SA: Slot
Attention, F: Fluency, R: Relevance, SC: Slot Consistency

The MHRED baseline [58] merely concatenates the textual and visual information
for generating responses which have lower BLEU scores in comparison to the
proposed framework. In UMD [12], the authors used attention guided
hierarchical recurrent encoder-decoder framework for generating responses.
Also, enhanced visual representation achieved with the help of a taxonomy
attribute tree was used for correct response generation.

It is visible that by explicitly using the slot information in the MHRED
network, it outperforms the UMD framework, giving a boost of more than 20% in
BLEU-4 scores. The improvement is mainly due to the usage of pre-trained
DialoGPT embeddings and slot attention that provide enhanced contextual
information compared to the recurrent encoders. The MTrans framework yields
superior performance, proving the efficacy of Transformers as opposed to the
recurrent networks.

The OAM [7] network focuses upon the attributes and position of the images,
and employs the MFB fusion technique to obtain the non-linear interaction
between the modalities for generating coherent responses with a NIST score of
4.3236. Our proposed transformer-based approach attains around 4% gain in the
NIST score in contrast to the OAM framework. Though MATE [24] exploits the
transformer as encoder-decoder, our proposed approach still performs well in
comparison. This is primarily because of the slot attention mechanism that
correctly focuses on the correct attributes of the product, and makes the
responses coherent, informative and interactive.

Models SIMCC-Furniture SIMCC-Fashion LSTM 0.022 0.022 HAE 0.075 0.059 HRE
0.075 0.079 MN 0.084 0.065 T-HAE 0.044 0.051 Mul-Trans (our) 0.086 0.080

Table 6: Results of BLEU score on SIMCC dataset

In Table 6, we present the evaluation results of different frameworks on the
SIMCC dataset [44] along with our proposed model. As shown in the table, our
proposed framework performs slightly better than the best performing MN
(memory network) for SIMCC-Furniture data and HRE (Hierarchical Recurrent
Encoder) for SIMCC-Fashion data, respectively. One of the main reasons is that
we use slot-based attention that helps in focusing on the attributes and the
transformer framework which is more robust than the RNN framework. The
effectiveness of our proposed framework is evidenced through another
multimodal dataset, viz. SIMCC, which ensures that it can be used for similar
other datasets.

####  6.2.3 Results of Human evaluation:

Along with the automatic evaluation, we also report the results of the manual
evaluation in Table 5. The results of the manual evaluation are in consonance
with the automatic evaluation results. The fluency of the proposed framework
is highest in comparison to all the baselines. By providing pre-trained
embedding information, slot attention, and the external knowledge base, the
responses are complete, thereby being grammatically fluent.

The proposed framework’s relevance score is maximum, ensuring that the
responses are coherent with the given dialogue context. As can be seen from
the table, the relevance scores of the multimodal frameworks are higher than
the unimodal networks as image information helps provide the full context for
the generation of coherent responses. Also, the inclusion of the knowledge
base improves the score in all the baselines and the proposed network. As our
current work’s primary objective is to generate more informative responses in
accordance with the extracted slot information, we see that the scores of the
slot consistency metric increase with the incorporation of slot attention.

Also, the models having the knowledge base information outperform the other
frameworks mainly because the external knowledge constitutes the important
slot information for a particular dialogue, hence boosting the generation of
responses. Utilizing transformers instead of GRU has been shown to improve the
quality of responses as is evident from the results depicted in Table 5. The
fluency of the responses takes a jump from 4.16 score to 4.42 proving the
efficacy of Transformers to generate better responses. Similarly, the
responses are relevant to the conversational history making the responses
consistent with the ongoing dialogue.

As the primary goal of our work is to make the responses interactive by using
the correct slot information, hence on evaluating we see that compared to the
MHRED network, Mtrans shows a performance gain in the slot consistency metric
as well. Mul-Trans model in consonance with the automatic evaluation results
shows better performance compared to all the baselines in terms of manual
evaluation as well. Visual representation from transformers has achieved
providing better responses that are fluent and contextually coherent as well.

Previously, the author in [68] employed a dual learning mechanism to jointly
address the NLU and NLG tasks in a dialogue system. Similarly in [72], the
author investigates a generative RNN framework for both tasks. Direct
comparison to the existing frameworks has not been shown in our current work,
firstly because these networks are trained solely on textual data. Secondly,
in our work, we focus on extracting the slot information from the user
utterance to facilitate the network in enhancing the next textual system
response generation. In contrast, in the existing frameworks, NLU and NLG are
performed on a single utterance only (i.e. the extracted slot is used to
generate the same utterance while we generate the next response in the
dialogue). Therefore, our work is novel as we exploit multimodal sources for
both NLU and NLG tasks in a user-system dialogue setting.

Figure 4: Examples of generated responses by baseline and proposed framework

###  6.3 Case Studies and Error Analysis

In Figure 4, we provide a few examples generated by our proposed framework and
the baselines. In Example 1 from the Figure, the proposed approach generates a
more informative and relevant response by including the correct slot values
such as block heels as desired by the user. Similarly, in the second example,
the ability to generate the correct brand Nike and type t-shirts of the
product makes the response diverse and interactive as opposed to the responses
generated by the baseline networks.

The baseline networks without the slot information tend to generate safer
responses that lack specific patterns, and brand information in them. While
the examples generated by the proposed approach it is obvious that the correct
slot information assists in generating better responses and providing
specified and desired products to the user thereby increasing customer
satisfaction and increasing customer retention.

We closely analyze the outputs of the generated response to be aware of the
errors made by the proposed dialogue generation framework. The common errors
made by the model are:

  * •

Erroneous image selection: The model is sometimes incompetent in selecting the
images having contextual information of more than 5 turns, thereby generating
incorrect responses in some cases. There are also cases, where due to the
discussion of multiple images in the conversational history, wrong images get
selected, making the responses incorrect.

  * •

Additional Information: The model sometimes generates additional/extra
information in the case of attributes for a few products. For example, Gold:
The material of the trousers is cotton., Predicted: The trousers have cotton
polyester material with check patterns. This is mainly due to the fact that
the conversation history has this additional information which also gets
incorporated into the responses.

  * •

Repetition: Sometimes the baseline and proposed frameworks generate words that
are repeated throughout the response. Also, unknown words due to fewer
instances in the training data get generated as ¡unk¿ tokens in the responses.
For example, Gold: The 3rd image belongs to the Fossil brand with a blue
dial., Predicted: The 3rd image has ¡unk¿ ¡unk¿ ¡unk¿.

  * •

Slot mismatch: Sometimes due to multiple slot information in the ground truth
response and the conversational history, the proposed framework at times gets
confused and generates responses that have incorrect slot information. For
example, Gold: The red synthetic material top with bell sleeves will look good
with the trousers., Predicted: The trousers have red material with bell
patterns.

The above-mentioned errors could be minimized by including better image
encoders to capture visual representations. In addition, fusion techniques
that could capture non-linear interactions between the modalities could help
with the errors of type slot mismatch. For repetition errors we plan to
investigate better pre-trained language models that could also handle the
additional information issue.

##  7 Conclusions and Future Work

With the progress in artificial intelligence, dialogue systems have reached
new paradigms. Narrowing the gap between vision and language, multimodal
conversational systems have gained immense popularity. Complementary
information in the form of images, audio, or videos to the unimodal (text)
systems has helped build robust systems. Task-oriented dialogue systems focus
on assisting humans by assisting them to achieve their desired goals. Response
generation is a crucial component in every dialogue system.

Our current work emphasizes the task of generating responses in a multimodal
dialogue system. In this paper, we have proposed an end-to-end framework
capable of extracting slot values from user utterances and generating a
suitable response. For improving the performance of slot extraction, we apply
a self-attention mechanism on the utterances so that the appropriate slot
values get focused. In addition to this, we also employ self-attention on
Knowledge Base (KB) and use this attended representation to assist in
generation. Furthermore, we pre-train DialoGPT Language Model onto a Multi-
modal Dialog dataset.

This effectively learns the previous dialogue context along with the current
user utterance and system response. Contextual embeddings trained using
DialoGPT are passed as the input to the model. We evaluated our proposed
approach on the Multi-modal Dialog dataset and have shown significant
performance improvement. Our proposed approach focused on essential slot
information in qualitative and quantitative metrics, thereby improving the
generation.

Our current approaches focus most on the text and little on the image. In the
future, we wish to enhance the performance by proposing methodologies that
utilize information from the images. These methods include fusion techniques
capable of incorporating information from both image and text, thereby
improving Multimodal dialogue systems’ performance.

## References

  * [1] Agarwal, S., Dušek, O., Konstas, I., Rieser, V.: Improving context modelling in multimodal dialogue generation. In: Proceedings of the 11th International Conference on Natural Language Generation. pp. 129–134 (2018)
  * [2] Agarwal, S., Dušek, O., Konstas, I., Rieser, V.: A knowledge-grounded multimodal search-based conversational agent. In: Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI. pp. 59–66 (2018)
  * [3] Akbari, H., Yuan, L., Qian, R., Chuang, W.H., Chang, S.F., Cui, Y., Gong, B.: Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104.11178 (2021)
  * [4] Alamri, H., Hori, C., Marks, T.K., Batr, D., Parikh, D.: Audio visual scene-aware dialog (avsd) track for natural language generation in dstc7. In: DSTC7 at AAAI2019 Workshop. vol. 2 (2018)
  * [5] Budzianowski, P., Vulić, I.: Hello, it’s gpt-2–how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:1907.05774 (2019)
  * [6] Budzianowski, P., Wen, T.H., Tseng, B.H., Casanueva, I., Ultes, S., Ramadan, O., Gašić, M.: Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 pp. 5016–5026 (2018)
  * [7] Chauhan, H., Firdaus, M., Ekbal, A., Bhattacharyya, P.: Ordinal and attribute aware response generation in a multimodal dialogue system. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 5437–5447 (2019)
  * [8] Chen, M., Zeng, J., Lou, J.: A self-attention joint model for spoken language understanding in situational dialog applications. arXiv preprint arXiv:1905.11393 (2019)
  * [9] Chen, W., Chen, J., Qin, P., Yan, X., Wang, W.Y.: Semantically conditioned dialog response generation via hierarchical disentangled self-attention. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers pp. 3696–3709 (2019)
  * [10] Chen, X., Xu, J., Xu, B.: A working memory model for task-oriented dialog response generation. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. pp. 2687–2693 (2019)
  * [11] Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL pp. 1724–1734 (2014)
  * [12] Cui, C., Wang, W., Song, X., Huang, M., Xu, X.S., Nie, L.: User attention-guided multimodal dialog systems. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 445–454 (2019)
  * [13] Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J.M., Parikh, D., Batra, D.: Visual dialog. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 326–335 (2017)
  * [14] De Vries, H., Strub, F., Chandar, S., Pietquin, O., Larochelle, H., Courville, A.: Guesswhat?! visual object discovery through multi-modal dialogue. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26. pp. 4466–4475 (2017)
  * [15] Deng, L., Tur, G., He, X., Hakkani-Tur, D.: Use of kernel deep convex networks and end-to-end learning for spoken language understanding. In: Spoken Language Technology Workshop (SLT), 2012 IEEE. pp. 210–215. IEEE (2012)
  * [16] Deoras, A., Sarikaya, R.: Deep belief network based semantic taggers for spoken language understanding. In: INTERSPEECH 2013, 14th Annual Conference of the International Speech Communication Association, Lyon, France, August 25-29, 2013\. pp. 2713–2717 (2013)
  * [17] Doddington, G.: Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: Proceedings of the second international conference on Human Language Technology Research. pp. 138–145 (2002)
  * [18] Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., Hon, H.W.: Unified language model pre-training for natural language understanding and generation. In: Advances in Neural Information Processing Systems. pp. 13063–13075 (2019)
  * [19] Firdaus, M., Chauhan, H., Ekbal, A., Bhattacharyya, P.: Emosen: Generating sentiment and emotion controlled responses in a multimodal dialogue system. IEEE Transactions on Affective Computing (2020)
  * [20] Firdaus, M., Pratap Shandeelya, A., Ekbal, A.: More to diverse: Generating diversified responses in a task oriented multimodal dialog system. Plos one 15(11), e0241271 (2020)
  * [21] Firdaus, M., Thakur, N., Ekbal, A.: Multidm-gcn: Aspect-guided response generation in multi-domain multi-modal dialogue system using graph convolution network. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. pp. 2318–2328 (2020)
  * [22] Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological bulletin 76(5), 378 (1971)
  * [23] Gan, Z., Cheng, Y., Kholy, A.E., Li, L., Liu, J., Gao, J.: Multi-step reasoning via recurrent dual attention for visual dialog. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers pp. 6463–6474 (2019)
  * [24] He, W., Li, Z., Lu, D., Chen, E., Xu, T., Huai, B., Yuan, J.: Multimodal dialogue systems via capturing context-aware dependencies of semantic elements. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 2755–2764 (2020)
  * [25] Ju, D., Shuster, K., Boureau, Y.L., Weston, J.: All-in-one image-grounded conversational agents. arXiv preprint arXiv:1912.12394 (2019)
  * [26] Lan, O., Zhu, S., Yu, K.: Semi-supervised training using adversarial multi-task learning for spoken language understanding. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6049–6053. IEEE (2018)
  * [27] Le, H., Hoi, S., Sahoo, D., Chen, N.: End-to-end multimodal dialog systems with hierarchical multimodal attention on video features. In: DSTC7 at AAAI2019 workshop (2019)
  * [28] Le, H., Sahoo, D., Chen, N., Hoi, S.: Multimodal transformer networks for end-to-end video-grounded dialogue systems. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 5612–5623 (2019)
  * [29] Li, X., Chen, Y.N., Li, L., Gao, J., Celikyilmaz, A.: End-to-end task-completion neural dialogue systems. Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017 - Volume 1: Long Papers pp. 733–743 (2017)
  * [30] Liao, L., Ma, Y., He, X., Hong, R., Chua, T.s.: Knowledge-aware multimodal dialogue systems. In: 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26. pp. 801–809. ACM (2018)
  * [31] Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: Workshop on Text Summarization Branches Out, Post-Conference Workshop of ACL 2004, Barcelona, Spain (July 2004)
  * [32] Lin, K.Y., Hsu, C.C., Chen, Y.N., Ku, L.W.: Entropy-enhanced multimodal attention model for scene-aware dialogue generation. In: DSTC7 at AAAI2019 workshop (2019)
  * [33] Lin, Z., Huang, X., Ji, F., Chen, H., Zhang, Y.: Task-oriented conversation generation using heterogeneous memory networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7 pp. 4557–4566 (2019)
  * [34] Liu, B., Lane, I.: Multi-domain adversarial learning for slot filling in spoken language understanding. arXiv preprint arXiv:1711.11310 (2017)
  * [35] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [36] Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21 pp. 1412–1421 (2015)
  * [37] Madasu, A., Aflalo, E., Ben Melech Stan, G., Tseng, S.Y., Bertasius, G., Lal, V.: Improving video retrieval using multilingual knowledge transfer. In: Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, Proceedings, Part I. pp. 669–684. Springer (2023)
  * [38] Madasu, A., Firdaus, M., Eqbal, A.: A unified framework for emotion identification and generation in dialogues. arXiv preprint arXiv:2205.15513 (2022)
  * [39] Madasu, A., Oliva, J., Bertasius, G.: Learning to retrieve videos by asking questions. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 356–365 (2022)
  * [40] Madotto, A., Wu, C.S., Fung, P.: Mem2seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers pp. 1468–1478 (2018)
  * [41] Mesnil, G., Dauphin, Y., Yao, K., Bengio, Y., Deng, L., Hakkani-Tur, D., He, X., Heck, L., Tur, G., Yu, D., et al.: Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing 23(3), 530–539 (2015)
  * [42] Mesnil, G., He, X., Deng, L., Bengio, Y.: Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In: Interspeech. pp. 3771–3775 (2013)
  * [43] Mi, F., Huang, M., Zhang, J., Faltings, B.: Meta-learning for low-resource natural language generation in task-oriented dialogue systems. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16 pp. 3151–3157 (2019)
  * [44] Moon, S., Kottur, S., Crook, P.A., De, A., Poddar, S., Levin, T., Whitney, D., Difranco, D., Beirami, A., Cho, E., et al.: Situated and interactive multimodal conversations. arXiv preprint arXiv:2006.01460 (2020)
  * [45] Mostafazadeh, N., Brockett, C., Dolan, B., Galley, M., Gao, J., Spithourakis, G.P., Vanderwende, L.: Image-grounded conversations: Multimodal context for natural question and response generation. Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017 - Volume 1: Long Papers pp. 462–472 (2017)
  * [46] Nie, L., Wang, W., Hong, R., Wang, M., Tian, Q.: Multimodal dialog system: Generating responses via adaptive decoders. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 1098–1106 (2019)
  * [47] Niu, P., Chen, Z., Song, M., et al.: A novel bi-directional interrelated model for joint intent detection and slot filling. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers pp. 5467–5471 (2019)
  * [48] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA. pp. 311–318. Association for Computational Linguistics (2002)
  * [49] Peng, B., Li, X., Gao, J., Liu, J., Wong, K.F., Su, S.Y.: Deep dyna-q: Integrating planning for task-completion dialogue policy learning. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers pp. 2182–2192 (2018)
  * [50] Qian, K., Yu, Z.: Domain adaptive dialog generation via meta learning. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers pp. 2639–2649 (2019)
  * [51] Qin, L., Xu, X., Che, W., Zhang, Y., Liu, T.: Dynamic fusion network for multi-domain end-to-end task-oriented dialog. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 pp. 6344–6354 (2020)
  * [52] Qiu, L., Ding, Y., He, L.: Recurrent neural networks with pre-trained language model embedding for slot filling task. arXiv preprint arXiv:1812.05199 (2018)
  * [53] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
  * [54] Raghu, D., Gupta, N.: Hierarchical-pointer generator memory network for task oriented dialog
  * [55] Raghu, D., Gupta, N., et al.: Disentangling language and knowledge in task-oriented dialogs. arXiv preprint arXiv:1805.01216 (2018)
  * [56] Rastogi, A., Hakkani-Tür, D., Heck, L.: Scalable multi-domain dialogue state tracking. In: 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 561–568. IEEE (2017)
  * [57] Reddy, R., Contractor, D., Raghu, D., Joshi, S.: Multi-level memory for task oriented dialogs. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) pp. 3744–3754 (2019)
  * [58] Saha, A., Khapra, M.M., Sankaranarayanan, K.: Towards building large scale multimodal domain-aware conversation systems. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7. pp. 696–704 (2018)
  * [59] Serban, I.V., Sordoni, A., Bengio, Y., Courville, A., Pineau, J.: Hierarchical neural network generative models for movie dialogues. arXiv preprint arXiv:1507.04808 7(8), 434–441 (2015)
  * [60] Serban, I.V., Sordoni, A., Bengio, Y., Courville, A., Pineau, J.: Building end-to-end dialogue systems using generative hierarchical neural network models. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, Phoenix, Arizona, USA. pp. 3776–3784 (2016)
  * [61] Serban, I.V., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A.C., Bengio, Y.: A hierarchical latent variable encoder-decoder model for generating dialogues. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA. pp. 3295–3301 (2017)
  * [62] Shang, L., Lu, Z., Li, H.: Neural responding machine for short-text conversation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). vol. 1, pp. 1577–1586 (2015)
  * [63] Shin, Y., Yoo, K.M., Lee, S.g.: Slot filling with delexicalized sentence generation. In: INTERSPEECH. pp. 2082–2086 (2018)
  * [64] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)
  * [65] Sordoni, A., Bengio, Y., Vahabi, H., Lioma, C., Grue Simonsen, J., Nie, J.Y.: A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. pp. 553–562. ACM (2015)
  * [66] Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.Y., Gao, J., Dolan, B.: A neural network approach to context-sensitive generation of conversational responses. NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015 pp. 196–205 (2015)
  * [67] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1), 1929–1958 (2014)
  * [68] Su, S.Y., Huang, C.W., Chen, Y.N.: Dual supervised learning for natural language understanding and generation. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers pp. 5472–5477 (2019)
  * [69] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada. pp. 3104–3112 (2014)
  * [70] Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations from transformers. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 5100–5111 (2019)
  * [71] Tian, Z., Bi, W., Li, X., Zhang, N.L.: Learning to abstract for memory-augmented conversational response generation. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. pp. 3816–3825 (2019)
  * [72] Tseng, B.H., Cheng, J., Fang, Y., Vandyke, D.: A generative model for joint natural language understanding and generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 pp. 1795–1807 (2020)
  * [73] Vinyals, O., Le, Q.: A neural conversational model. arXiv preprint arXiv:1506.05869 (2015)
  * [74] Wang, J., Zhang, Y., Kim, T.K., Gu, Y.: Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. arXiv preprint arXiv:2006.06814 (2020)
  * [75] Welch, B.L.: The generalization ofstudent’s’ problem when several different population variances are involved. Biometrika 34(1/2), 28–35 (1947)
  * [76] Wen, T.H., Vandyke, D., Mrksic, N., Gasic, M., Rojas-Barahona, L.M., Su, P.H., Ultes, S., Young, S.: A network-based end-to-end trainable task-oriented dialogue system. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers pp. 438–449 (2017)
  * [77] Williams, K.: Neural lexicons for slot tagging in spoken language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers). pp. 83–89 (2019)
  * [78] Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural computation 1(2), 270–280 (1989)
  * [79] Wu, C.S.: Learning to memorize in neural task-oriented dialogue systems. arXiv preprint arXiv:1905.07687 (2019)
  * [80] Wu, C.S., Socher, R., Xiong, C.: Global-to-local memory pointer networks for task-oriented dialogue. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9 (2019)
  * [81] Wu, J., Banchs, R.E., D’Haro, L.F., Krishnaswamy, P., Chen, N.: Attention-based semantic priming for slot-filling. In: Proceedings of the Seventh Named Entities Workshop. pp. 22–26 (2018)
  * [82] Xia, C., Zhang, C., Yan, X., Chang, Y., Yu, P.S.: Zero-shot user intent detection via capsule neural networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 pp. 3090–3099 (2018)
  * [83] Xu, H., Peng, H., Xie, H., Cambria, E., Zhou, L., Zheng, W.: End-to-end latent-variable task-oriented dialogue system with exact log-likelihood optimization. World Wide Web pp. 1–14 (2019)
  * [84] Yao, K., Peng, B., Zhang, Y., Yu, D., Zweig, G., Shi, Y.: Spoken language understanding using long short-term memory neural networks. In: Spoken Language Technology Workshop (SLT), 2014 IEEE. pp. 189–194. IEEE (2014)
  * [85] Yao, K., Peng, B., Zweig, G., Yu, D., Li, X., Gao, F.: Recurrent conditional random field for language understanding. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4077–4081. IEEE (2014)
  * [86] Yao, K., Zweig, G., Hwang, M.Y., Shi, Y., Yu, D.: Recurrent neural networks for language understanding. In: Interspeech. pp. 2524–2528 (2013)
  * [87] Zhang, Y., Sun, S., Galley, M., Chen, Y.C., Brockett, C., Gao, X., Gao, J., Liu, J., Dolan, B.: Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536 (2019)
  * [88] Zhang, Z., Liao, L., Huang, M., Zhu, X., Chua, T.S.: Neural multimodal belief tracker with adaptive attention for dialogue systems. In: The World Wide Web Conference. pp. 2401–2412 (2019)
  * [89] Zhao, L., Feng, Z.: Improving slot filling in spoken language understanding with joint pointer and attention. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 426–431 (2018)
  * [90] Zhong, V., Xiong, C., Socher, R.: Global-locally self-attentive dialogue state tracker. arXiv preprint arXiv:1805.09655 (2018)
  * [91] Zhu, C., Zeng, M., Huang, X.: Multi-task learning for natural language generation in task-oriented dialogue. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7. pp. 1261–1266 (2019)
  * [92] Zhu, S., Yu, K.: Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5675–5679. IEEE (2017)
  * [93] Zhu, S., Yu, K.: Concept transfer learning for adaptive language understanding. Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, Australia, July 12-14, 2018 pp. 391–399 (2018)