\begin{table}[t!] 
\centering
\resizebox{0.5\textwidth}{!}{
\begin{tabular}{lccc}
\toprule

Statistics  & Train & Validation & Test \\ \midrule
\#Dialogues           & 1,059,117    & 10,000   &  10,000 \\
\#Images        & 1,509,288    & 23,812   & 23,766 \\
\#Turns         & 4,825,054   & 45,382   & 45,798 \\
Avg. \#Turns per Dialogue  & 4.56    & 4.54     & 4.58  \\
Avg. \#Images per Dialogue & 2.59      & 2.58      & 2.61  \\
Avg. \#Tokens per Turn & 15.64     & 15.71      & 15.57 \\ \midrule
Avg. \#(Neg. Images) per Dialogue & - & 999 & 999 \\
Avg. \#(Neg. Utterances) per Dialogue & - & 999 & 999 \\
\bottomrule
\end{tabular}
}
\caption{Statistics of our train, validation, and test sets.}
\label{tab:statistics_split}
\end{table}
\begin{abstract}
Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent. In this paper, we introduce the MMDialog~dataset to facilitate multi-modal conversation better. MMDialog~is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog~has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 8x. Second, it contains massive topics to generalize the open domain. To build an engaging dialogue system with this dataset, we propose and normalize two response prediction tasks based on retrieval and generative scenarios. In addition, we build two baselines for the above tasks with state-of-the-art techniques and report their experimental performance. We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses. Our dataset and scripts are available in \url{https://github.com/victorsungo/MMDialog}.
\end{abstract}

Furthermore, we define the multi-modal response generation and retrieval tasks based on MMDialog~that are essential for building a more engaging multi-modal dialogue agent. We build baseline models and conduct several analyses of their performance. For the generative task, we follow~\citet{sun-etal-2022-multimodal} and implement the models for multi-modal response generation. For the retrieval task, we also propose a CLIP-based dual-encoder for retrieval tasks inspired by~\citet{zang-etal-2021-photochat}. Since in our multi-modal response prediction settings, the modality orders of generated responses may not be aligned with the ground-truth responses. Thus, it is non-trivial to conduct evaluation on cross-modal response elements. To tackle the above challenges, we propose a novel evaluation metric named MM-Relevance, which performs visual-language matching based on the large-scale pre-trained multi-modal CLIP model~\cite{radford2021learning}. Evaluation results on MMDialog~demonstrate that our designed baselines can achieve considerable performance on generation and retrieval tasks of both modalities.

\subsection{Multi-Modal Dialogue Modeling}
Based on the aforementioned multi-modal dialogue datasets, many advanced works have been proposed. Several modeling works~\citep{qi2020two,niu2019recursive,gan2019multi} investigate how to escalate the performance of conversational agents in image-grounded dialogue. Afterward, researchers~\citep{yang2021open,liang2021maria} explore enriching textual expressions of generated dialogue responses through associative vision scenes.~\citet{zang-etal-2021-photochat} proposes two tasks, including photo-sharing intent prediction to predict whether model should intend to share the photo in the next dialogue turn and a dialogue-based image retrieval task to retrieve the most proper photo given the dialogue context. They also propose a dual-encoder model that uses object labels to encode image features, which achieves the best performance among all the models w/o cross-attention mechanisms. However, the authors do not conduct textual response retrieval tasks.~\citet{zheng-etal-2022-mmchat} proposes a multi-modal dialogue generation model based on Seq2Seq architecture, which was proved to be superior to textual Seq2Seq model. However, this model can only generate plain textual responses, which is not in line with the open domain multi-modal response generation scenario. Recently,~\citet{sun-etal-2022-multimodal} make the first attempt to build a multi-modal dialogue response generation model named Divter that can effectively understand multi-modal dialogue context and generate informative text and high-resolution image responses. As advanced works on dialogue systems include retrieval-based methods~\citep{wu2017sequential,zhou2018multi,whang2020effective,li2021small} and generative methods~\citep{li2015diversity,serban2016building,zhang2020dialogpt}. Therefore, we adapt Divter~\cite{sun-etal-2022-multimodal} to our multi-modal response generation settings and extend the dual-encoder~\cite{zang-etal-2021-photochat} to the retrieval-based scenarios as baselines.

\paragraph{Task-2: Multi-modal Response Retrieval}
As for the retrieval-based models, each dialogue example $(U, R)$ additionally provides a series of negative multi-modal elements as distractions. We compose the ground-truth elements $\{ r^m_l\}_{l=1}^{L}$ in $R$ and the including distracting elements into an element candidate set $C = \{ r^{m}_z\}_{z=1}^{Z}$ for retrieval where $n_z$ is the total number of candidate elements including ground-truth and distracting ones. Meanwhile, we also assign corresponding element labels $\{l^{m}_z\}_{z=1}^{Z} \in \{0,1\}$ with $l^{m}_z = 1$ indicating $r^{m}_z$ a proper element of $m$ modal type for $U$ and otherwise $l^{m}_z = 0$. Thus, the goal of a response retrieval model is to extract an element from a given element candidate set $C$ step by step while predicting each element $r^m_z$ in $R$ . Through such an element retrieval process in an auto-regressive style, we can obtain a multi-modal response consisting of retrieved elements, including textual utterances or visual images, or both of them.

To tackle the above issue, we propose a novel evaluation metric, named MM-Relevance, which performs visual-language matching based on the large-scale pre-trained multi-modal CLIP model~\cite{radford2021learning} for multi-modal dialogue response generation task. CLIP is trained on a vast corpus of image-caption pairs from the Web. It learns to bring the embeddings of both modalities (visual and textual) together via a contrastive objective. Therefore, we utilize this model to assess the relevance between the generated responses and the ground-truth responses to mitigate modal misalignment issues. In specific, suppose we obtain a generated multi-modal response $\Tilde{R}=\{ \Tilde{r}^m_k\}_{k=1}^{n_{\Tilde{r}}}$ where $n_{\Tilde{r}}$ is the number of elements in $\Tilde{R}$, and the corresponding ground-truth response $R = \{ r^m_k\}_{k=1}^{n_r}$. We first align the two sequences from the left. Then, the representation vector of each element is obtained by encoding the textual response or visual image through text encoder or image encoder pre-trained by CLIP respectively. We denote the encoded vectors of two responses as: $E=\{ e^m_k\}_{k=1}^{n_r}$ and $\Tilde{E}=\{ \Tilde{e}^m_k\}_{k=1}^{n_{\Tilde{r}}}$. Then, we compute the CLIP scores of the two elements position by position until they cannot be aligned:

\begin{equation}
\begin{aligned}
    \text{MM}_{\text{Rel}}(R, \Tilde{R}) &= \sum_{k=1}^{K}(e^m_k)^T \cdot \Tilde{e}^m_k \\
    K & = \min\{n_r, n_{\Tilde{r}} \}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \text{P}_{\text{MM}} &= \frac{\text{MM}_{\text{Rel}}(R, \Tilde{R})}{n_{\Tilde{r}}} \\ 
    \text{R}_{\text{MM}} &= \frac{\text{MM}_{\text{Rel}}(R, \Tilde{R})}{n_r} \\
    \text{F1}_{\text{MM}} &= \frac{2\text{P}_{\text{MM}}\text{R}_{\text{MM}}}{\text{P}_{\text{MM}}+\text{R}_{\text{MM}}} \\
\end{aligned} 
\end{equation}

\subsection{Experimental Setup}
We first sample 10K and 10K dialogues sessions for validation and testing respectively. The detailed statistics are presented in Table~\ref{tab:statistics_split}. For retrieval tasks, we randomly sample 999 negative textual utterances and 999 negative visual images from the same split set for each dialogue, maintaining the total number of candidate elements at 1K. While in training phase, the negative ones are in-batch sampled similar to~\citet{radford2021learning}. For the textual dialogue response generator, we fine-tune DialoGPT~\cite{zhang2020dialogpt} with \textit{transformers} library provided by huggingface\footnote{\url{https://github.com/huggingface/transformers}} using the version ``DialoGPT-medium'' consistent with~\citet{sun-etal-2022-multimodal}. For the description-to-image translator, we implement DALL-E~\cite{ramesh2021zero} using the code of ``mega'' version in \url{https://github.com/borisdayma/dalle-mini}, which also has the same model settings with~\citet{sun-etal-2022-multimodal}. We fine-tune DALL-E mega for one epoch with initial learning rate 1e-7 and mini-batch size of 64. We process all images into 256 $\times$ 256 RGB format for DALL-E. To obtain the description of images in MMDialog, we adopt OFA-huge~\cite{wang2022ofa} using the code \url{https://github.com/OFA-Sys/OFA/tree/feature/add_transformers} for image captioning. All version of CLIP models we leveraged in this paper are ``openai/clip-vit-base-patch32'' in \url{https://huggingface.co/openai/clip-vit-base-patch32}. When implementing Divter, we follow the same experimental configuration. As for the retrieval baseline, the representation vectors for both modality are obtained by CLIP model and fixed during training. The transformers used in retrieval tasks consist of 4 Transformer layers with a hidden size of 512 and 8 heads. We train the retrieval models with an initial learning rate of 5e-7 and mini-batch size of 512. For all baselines, early stopping on the validation set is adopted as a regularization strategy and the best model is selected based on the validation performance. The training of both tasks is conducted on 8 Nvidia Tesla A100 80G GPU cards.
{"value": "1,059,117", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# dialogues", "number of dialogues"], "sub-set/group name": "train", "dataset features": {"xx": "yy"}} {"value": "1,509,288", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# images", "number of images"], "sub-set/group name": "train", "dataset features": {"xx": "yy"}} {"value": "4,825,054", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# turns", "number of turns"], "sub-set/group name": "train", "dataset features": {"xx": "yy"}} {"value": "4.56", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # turns per dialogue", "average number of turns per dialogue"], "sub-set/group name": "train", "dataset features": {"xx": "yy"}} {"value": "2.59", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # images per dialogue", "average number of images per dialogue"], "sub-set/group name": "train", "dataset features": {"xx": "yy"}} {"value": "15.64", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # tokens per turn", "average number of tokens per turn"], "sub-set/group name": "train", "dataset features": {"xx": "yy"}} {"value": "10,000", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# dialogues", "number of dialogues"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "23,812", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# images", "number of images"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "45,382", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# turns", "number of turns"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "4.54", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # turns per dialogue", "average number of turns per dialogue"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "2.58", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # images per dialogue", "average number of images per dialogue"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "15.71", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # tokens per turn", "average number of tokens per turn"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "999", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # (neg. images) per dialogue", "average number of negative images per dialogue"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "999", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # (neg. utterances) per dialogue", "average number of negative utterances per dialogue"], "sub-set/group name": "validation", "dataset features": {"xx": "yy"}} {"value": "10,000", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# dialogues", "number of dialogues"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}} {"value": "23,766", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# images", "number of images"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}} {"value": "45,798", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# turns", "number of turns"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}} {"value": "4.58", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # turns per dialogue", "average number of turns per dialogue"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}} {"value": "2.61", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # images per dialogue", "average number of images per dialogue"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}} {"value": "15.57", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # tokens per turn", "average number of tokens per turn"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}} {"value": "999", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # (neg. images) per dialogue", "average number of negative images per dialogue"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}} {"value": "999", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # (neg. utterances) per dialogue", "average number of negative utterances per dialogue"], "sub-set/group name": "test", "dataset features": {"xx": "yy"}}