\begin{table}[t!] 
\centering
\resizebox{0.5\textwidth}{!}{
\begin{tabular}{lccc}
\toprule

Statistics  & PhotoChat & MMChat & MMDialog \\ \midrule
\#Language           & English    & Chinese   & English \\
\#Open-domain           &   \ding{56}  & \ding{52}   & \ding{52} \\ \midrule
\#Dialogues           & 12.29K    & 120.84K   & 1.08M \\
\#Images        & 10.92K    & 204.32K   & 1.53M \\
\#Turns         & 156.10K   & 314.13K   & 4.92M \\
Avg.  \#Turns per Dialogue  & 12.71     & 2.59      & 4.56  \\
Avg. \#Images per Dialogue & 0.89      & 2.91      & 2.59  \\
Avg.  \#Tokens per Turn & 6.33      & 8.52      & 15.64 \\
\bottomrule
\end{tabular}
}
\caption{Statistics of MMDialog~and previous multi-modal dialogue datasets.}
\label{tab:statistics}
\end{table}
\begin{abstract}
Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent. In this paper, we introduce the MMDialog~dataset to facilitate multi-modal conversation better. MMDialog~is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog~has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 8x. Second, it contains massive topics to generalize the open domain. To build an engaging dialogue system with this dataset, we propose and normalize two response prediction tasks based on retrieval and generative scenarios. In addition, we build two baselines for the above tasks with state-of-the-art techniques and report their experimental performance. We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses. Our dataset and scripts are available in \url{https://github.com/victorsungo/MMDialog}.
\end{abstract}

Existing approaches to building multi-modal dialogue systems are primarily data-driven, requiring the collection of a large-scale dataset first. To facilitate this line of research, the community emerges a few dialogue datasets incorporating visual information~\cite{meng2020openvidial,wang2021openvidial,zang-etal-2021-photochat,zheng-etal-2022-mmchat}. For example, Visual Dialog~\cite{das2017visual} is set up for visual question answering involving image inputs. IGC~\cite{mostafazadeh-etal-2017-image} and Image-Chat~\cite{shuster-etal-2020-image} are constructed in a crowd-sourcing method in which annotators are employed to chat about given images. PhotoChat~\cite{zang-etal-2021-photochat} is also built via crowd-sourcing but contains sharing photos in conversations. MMChat~\cite{zheng-etal-2022-mmchat} is collected from real conversations on Chinese social media.

Despite the diversity of multi-modal dialogue corpora, these datasets still have limitations. Firstly, several corpora, including Visual Dialog, IGC and Image-Chat, are derived from crowd-sourcing dialogues talking about given images. The topics of human utterances in a dialogue session are often triggered and grounded by these images, which is inconsistent with our daily communications, where the utterances are not always image-related~\cite{zheng-etal-2022-mmchat}. Secondly, other groups of datasets, such as OpenViDial 1.0/2.0~\cite{meng2020openvidial,wang2021openvidial} and dialogues collected by \citet{lee-etal-2021-constructing}, are not originated from a real multi-modal conversation scenario. The former directly extracts dialogues and their visual contexts from movies and TV series, and the latter replaces some utterances with retrieved relevant images. Both methods artificially construct images from the multi-turn conversation to simulate multi-modal dialogues. Finally, some recently proposed multi-modal dialogue data like PhotoChat and MMChat introduce real human-human conversations. They are still limited by their small scale or lack of domain diversity, impeding the further explorations on multi-modal dialogue modeling.  
To address the aforementioned issue, we present MMDialog, a large-scale multi-turn dialogue dataset containing multi-modal open-domain conversations derived from real human-human chat content in social media. MMDialog~contains 1.08M dialogue sessions and 1.53M associated images. We elaborately design a series of data filtering processes during the data collection phase. On average, one dialogue session has 2.59 images, which can be located anywhere at any conversation turn.  {Figure~\ref{fig:intro_case} depicts an example of human conversations in our MMDialog~dataset.} To the best of our knowledge, this is the first million-scale open-domain multi-modal dialogue corpora. We hope the large amount of dialogues and images can shed light on this line of research.

\begin{itemize}
\setlength{\itemsep}{0pt}
    \item We construct a novel multi-turn dialogue dataset \textbf{MMDialog}~that contains 1.08M multi-modal open-domain conversations and 1.53M associated images derived from social media and conduct data filtering and post-processing elaborately. To the best of our knowledge, this is the first million-scale multi-turn open-domain multi-modal dialogue corpus.
    \item We propose two benchmark tasks on MMDialog~that are essential for building more engaging multi-modal dialogue systems.
    \item We propose a novel evaluation metric \textbf{MM-Relevance} measuring the relevance between generated multi-modal response and ground-truth response. It builds upon the large-scale pre-trained multi-modal CLIP model, which can specifically mitigate the modal misalignment issues.
    \item We design two baselines for corresponding tasks to promote future research on this dataset and achieve considerable performance on generation and retrieval tasks of both modalities. We also give comprehensive analysis to provide more insights into multi-modal dialogue modeling.
\end{itemize}

\section{Related Works}
\subsection{Multi-Modal Dialogue Datasets}
Recently has witnessed a rapid development of visual-language modeling and emerged several multi-modal datasets. The multi-modal datasets include MSCOCO Image Caption dataset~\cite{chen2015microsoft} for image captioning and image generation task; VQAv2 dataset~\cite{goyal2017making} for visual question answering task; SNLI-VE dataset~\cite{xie2019visual} for visual entailment task; RefCOCO~\cite{yu2016modeling}, RefCOCO+~\cite{yu2016modeling} and RefCOCOg~\cite{mao2016generation} for referring expression comprehension task.

Concurrent with the above works, several dialogue-related tasks have also been explored.~\citet{das2017visual} introduced the task of Visual Dialog, which requires an AI agent to hold a meaningful dialogue with humans in natural, conversational language about visual content.~\citet{mostafazadeh-etal-2017-image} proposed IGC, which contains 4K dialogues where each includes an image with a textual description, along with the questions and responses around the image. However, IGC is usually used for evaluation due to its small scale.~\citet{shuster-etal-2020-image} released Image-Chat that is larger than IGC and consists of 202K image-grounded dialogues. However, the above three datasets were created by asking the crowd workers to talk about a shared image to generate the conversation. Therefore, the utterances are often triggered and grounded by these images. In contrast, human daily communication utterances are not always image-related~\cite{zheng-etal-2022-mmchat}, which retain gaps with open-domain multi-modal conversation scenarios. Then, other groups of works proposed to derive the images from the multi-turn conversations:~\citet{meng2020openvidial,wang2021openvidial} constructed OpenViDial 1.0/2.0 by directly extracting dialogues and their visual contexts from movies and TV series.~\citet{lee-etal-2021-constructing} also built a multi-modal dialogue dataset by replacing the selected utterances with retrieved relevant images. However, although these corpora were constructed from open-domain conversations with images, they did not originate from a real multi-modal conversation scenario. Therefore, recently some researchers begin to introduce real human-human conversations.~\citet{zang-etal-2021-photochat} created the first human-human dialogue dataset with photo-sharing acts via crowd-sourcing.~\citet{zheng-etal-2022-mmchat} collected multi-modal dialogues from real conversations on social media. Nevertheless, they were still limited by their small scale or lack of domain diversity, which may hinder further explorations on multi-modal dialogue modeling. To address the aforementioned issue, we make the first attempt to construct a million-scale multi-turn dialogue dataset, namely \textbf{MMDialog}, derived from social media and conduct data filtering and post-processing elaborately.

\subsection{Multi-Modal Dialogue Modeling}
Based on the aforementioned multi-modal dialogue datasets, many advanced works have been proposed. Several modeling works~\citep{qi2020two,niu2019recursive,gan2019multi} investigate how to escalate the performance of conversational agents in image-grounded dialogue. Afterward, researchers~\citep{yang2021open,liang2021maria} explore enriching textual expressions of generated dialogue responses through associative vision scenes.~\citet{zang-etal-2021-photochat} proposes two tasks, including photo-sharing intent prediction to predict whether model should intend to share the photo in the next dialogue turn and a dialogue-based image retrieval task to retrieve the most proper photo given the dialogue context. They also propose a dual-encoder model that uses object labels to encode image features, which achieves the best performance among all the models w/o cross-attention mechanisms. However, the authors do not conduct textual response retrieval tasks.~\citet{zheng-etal-2022-mmchat} proposes a multi-modal dialogue generation model based on Seq2Seq architecture, which was proved to be superior to textual Seq2Seq model. However, this model can only generate plain textual responses, which is not in line with the open domain multi-modal response generation scenario. Recently,~\citet{sun-etal-2022-multimodal} make the first attempt to build a multi-modal dialogue response generation model named Divter that can effectively understand multi-modal dialogue context and generate informative text and high-resolution image responses. As advanced works on dialogue systems include retrieval-based methods~\citep{wu2017sequential,zhou2018multi,whang2020effective,li2021small} and generative methods~\citep{li2015diversity,serban2016building,zhang2020dialogpt}. Therefore, we adapt Divter~\cite{sun-etal-2022-multimodal} to our multi-modal response generation settings and extend the dual-encoder~\cite{zang-etal-2021-photochat} to the retrieval-based scenarios as baselines.

\section{Corpus Statistics}
MMDialog~consists of 1,079,117 unique dialogues and 1,531,339 images. The statistics of several multi-modal open-domain dialogue corpora are shown in Table~\ref{tab:statistics}. On average one dialogue session has 2.59 images and 4.56 turns, and the images can be located anywhere in any turns of the conversation. We believe that in daily life, people are free to choose any modalities of conversational expressions at any stage of the conversation, and our dialogue data reflects this organizing style. Compared to the recently released multi-modal open domain dialogue dataset PhotoChat~\cite{zang-etal-2021-photochat} and MMChat~\cite{zheng-etal-2022-mmchat}, MMDialog~enjoys a significantly larger scale of dialogue data and more visual objects, especially that the volume of dialogue sessions has reached million-level. Since conversations originate from a wide range of hashtags presenting broad domains, the dialogues in MMDialog~are open-domain and cover diverse topics, which can shed light on research of multi-modal dialogue modeling. Besides, on average each dialogue turn in MMDialog~contains more text tokens than PhotoChat and MMChat, demonstrating that our proposed data may convey more semantic information in textual utterances.
{"value": "12.29K", "type": "Data Stat.", "dataset": "PhotoChat", "attribute name": ["# dialogues", "number of dialogues"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "false", "Language": "English"}} {"value": "120.84K", "type": "Data Stat.", "dataset": "MMChat", "attribute name": ["# dialogues", "number of dialogues"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "Chinese"}} {"value": "1.08M", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# dialogues", "number of dialogues"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "English"}} {"value": "10.92K", "type": "Data Stat.", "dataset": "PhotoChat", "attribute name": ["# images", "number of images"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "false", "Language": "English"}} {"value": "204.32K", "type": "Data Stat.", "dataset": "MMChat", "attribute name": ["# images", "number of images"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "Chinese"}} {"value": "1.53M", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# images", "number of images"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "English"}} {"value": "156.10K", "type": "Data Stat.", "dataset": "PhotoChat", "attribute name": ["# turns", "number of turns"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "false", "Language": "English"}} {"value": "314.13K", "type": "Data Stat.", "dataset": "MMChat", "attribute name": ["# turns", "number of turns"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "Chinese"}} {"value": "4.92M", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["# turns", "number of turns"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "English"}} {"value": "12.71", "type": "Data Stat.", "dataset": "PhotoChat", "attribute name": ["avg. # turns per dialogue", "average number of turns per dialogue"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "false", "Language": "English"}} {"value": "2.59", "type": "Data Stat.", "dataset": "MMChat", "attribute name": ["avg. # turns per dialogue", "average number of turns per dialogue"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "Chinese"}} {"value": "4.56", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # turns per dialogue", "average number of turns per dialogue"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "English"}} {"value": "0.89", "type": "Data Stat.", "dataset": "PhotoChat", "attribute name": ["avg. # images per dialogue", "average number of images per dialogue"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "false", "Language": "English"}} {"value": "2.91", "type": "Data Stat.", "dataset": "MMChat", "attribute name": ["avg. # images per dialogue", "average number of images per dialogue"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "Chinese"}} {"value": "2.59", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # images per dialogue", "average number of images per dialogue"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "English"}} {"value": "6.33", "type": "Data Stat.", "dataset": "PhotoChat", "attribute name": ["avg. # tokens per turn", "average number of tokens per turn"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "false", "Language": "English"}} {"value": "8.52", "type": "Data Stat.", "dataset": "MMChat", "attribute name": ["avg. # tokens per turn", "average number of tokens per turn"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "Chinese"}} {"value": "15.64", "type": "Data Stat.", "dataset": "MMDialog", "attribute name": ["avg. # tokens per turn", "average number of tokens per turn"], "sub-set/group name": "xx", "dataset features": {"Open-domain": "true", "Language": "English"}}