\begin{table*}[ht!] 
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{lccccccccccc}
\toprule
\multirow{2}{*}{Models} &  \multicolumn{1}{c}{Intent}  & \multicolumn{1}{c}{Image Generation} &  \multicolumn{3}{c}{Textual Response Generation} & \multicolumn{1}{c}{Multi-Modal Generation} \\ 
\cmidrule(lr){2-2}  \cmidrule(lr){3-3} \cmidrule(lr){4-6} \cmidrule(lr){7-7} 
                            &  F1      & IS$\uparrow$  & BLEU-1 & BLEU-2 & ROUGE-L &  MM-Relevance$\uparrow$   \\ \midrule
Divter~\cite{sun-etal-2022-multimodal} &  75.56     &  21.84 $\pm$ 0.81  &   8.78    & 6.86 &   11.57   &  68.40   \\ 
\bottomrule
\end{tabular}
}
\caption{Automatic evaluation results of the generative baseline on the test set of MMDialog. All numbers except ``IS'' and ``MM-Relevance'' are in percentage.}
\label{tab:main_gen}
\end{table*}
Furthermore, we define the multi-modal response generation and retrieval tasks based on MMDialog~that are essential for building a more engaging multi-modal dialogue agent. We build baseline models and conduct several analyses of their performance. For the generative task, we follow~\citet{sun-etal-2022-multimodal} and implement the models for multi-modal response generation. For the retrieval task, we also propose a CLIP-based dual-encoder for retrieval tasks inspired by~\citet{zang-etal-2021-photochat}. Since in our multi-modal response prediction settings, the modality orders of generated responses may not be aligned with the ground-truth responses. Thus, it is non-trivial to conduct evaluation on cross-modal response elements. To tackle the above challenges, we propose a novel evaluation metric named MM-Relevance, which performs visual-language matching based on the large-scale pre-trained multi-modal CLIP model~\cite{radford2021learning}. Evaluation results on MMDialog~demonstrate that our designed baselines can achieve considerable performance on generation and retrieval tasks of both modalities.

\begin{itemize}
\setlength{\itemsep}{0pt}
    \item We construct a novel multi-turn dialogue dataset \textbf{MMDialog}~that contains 1.08M multi-modal open-domain conversations and 1.53M associated images derived from social media and conduct data filtering and post-processing elaborately. To the best of our knowledge, this is the first million-scale multi-turn open-domain multi-modal dialogue corpus.
    \item We propose two benchmark tasks on MMDialog~that are essential for building more engaging multi-modal dialogue systems.
    \item We propose a novel evaluation metric \textbf{MM-Relevance} measuring the relevance between generated multi-modal response and ground-truth response. It builds upon the large-scale pre-trained multi-modal CLIP model, which can specifically mitigate the modal misalignment issues.
    \item We design two baselines for corresponding tasks to promote future research on this dataset and achieve considerable performance on generation and retrieval tasks of both modalities. We also give comprehensive analysis to provide more insights into multi-modal dialogue modeling.
\end{itemize}

\subsection{Multi-Modal Dialogue Modeling}
Based on the aforementioned multi-modal dialogue datasets, many advanced works have been proposed. Several modeling works~\citep{qi2020two,niu2019recursive,gan2019multi} investigate how to escalate the performance of conversational agents in image-grounded dialogue. Afterward, researchers~\citep{yang2021open,liang2021maria} explore enriching textual expressions of generated dialogue responses through associative vision scenes.~\citet{zang-etal-2021-photochat} proposes two tasks, including photo-sharing intent prediction to predict whether model should intend to share the photo in the next dialogue turn and a dialogue-based image retrieval task to retrieve the most proper photo given the dialogue context. They also propose a dual-encoder model that uses object labels to encode image features, which achieves the best performance among all the models w/o cross-attention mechanisms. However, the authors do not conduct textual response retrieval tasks.~\citet{zheng-etal-2022-mmchat} proposes a multi-modal dialogue generation model based on Seq2Seq architecture, which was proved to be superior to textual Seq2Seq model. However, this model can only generate plain textual responses, which is not in line with the open domain multi-modal response generation scenario. Recently,~\citet{sun-etal-2022-multimodal} make the first attempt to build a multi-modal dialogue response generation model named Divter that can effectively understand multi-modal dialogue context and generate informative text and high-resolution image responses. As advanced works on dialogue systems include retrieval-based methods~\citep{wu2017sequential,zhou2018multi,whang2020effective,li2021small} and generative methods~\citep{li2015diversity,serban2016building,zhang2020dialogpt}. Therefore, we adapt Divter~\cite{sun-etal-2022-multimodal} to our multi-modal response generation settings and extend the dual-encoder~\cite{zang-etal-2021-photochat} to the retrieval-based scenarios as baselines.

To tackle the above issue, we propose a novel evaluation metric, named MM-Relevance, which performs visual-language matching based on the large-scale pre-trained multi-modal CLIP model~\cite{radford2021learning} for multi-modal dialogue response generation task. CLIP is trained on a vast corpus of image-caption pairs from the Web. It learns to bring the embeddings of both modalities (visual and textual) together via a contrastive objective. Therefore, we utilize this model to assess the relevance between the generated responses and the ground-truth responses to mitigate modal misalignment issues. In specific, suppose we obtain a generated multi-modal response $\Tilde{R}=\{ \Tilde{r}^m_k\}_{k=1}^{n_{\Tilde{r}}}$ where $n_{\Tilde{r}}$ is the number of elements in $\Tilde{R}$, and the corresponding ground-truth response $R = \{ r^m_k\}_{k=1}^{n_r}$. We first align the two sequences from the left. Then, the representation vector of each element is obtained by encoding the textual response or visual image through text encoder or image encoder pre-trained by CLIP respectively. We denote the encoded vectors of two responses as: $E=\{ e^m_k\}_{k=1}^{n_r}$ and $\Tilde{E}=\{ \Tilde{e}^m_k\}_{k=1}^{n_{\Tilde{r}}}$. Then, we compute the CLIP scores of the two elements position by position until they cannot be aligned:

\begin{equation}
\begin{aligned}
    \text{MM}_{\text{Rel}}(R, \Tilde{R}) &= \sum_{k=1}^{K}(e^m_k)^T \cdot \Tilde{e}^m_k \\
    K & = \min\{n_r, n_{\Tilde{r}} \}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \text{P}_{\text{MM}} &= \frac{\text{MM}_{\text{Rel}}(R, \Tilde{R})}{n_{\Tilde{r}}} \\ 
    \text{R}_{\text{MM}} &= \frac{\text{MM}_{\text{Rel}}(R, \Tilde{R})}{n_r} \\
    \text{F1}_{\text{MM}} &= \frac{2\text{P}_{\text{MM}}\text{R}_{\text{MM}}}{\text{P}_{\text{MM}}+\text{R}_{\text{MM}}} \\
\end{aligned} 
\end{equation}

\subsection{Multi-modal Response Generation Model}
We consider to implement the state-of-the-art multi-modal dialogue response generation model Divter (Figure \ref{fig:model}a) proposed by~\citet{sun-etal-2022-multimodal}, which consists of two components: a textual dialogue response generator $\mathcal{G}$ and a  description-to-image translator $\mathcal{F}$. 

\subsection{Results of Multi-modal Response Generation Model}
Table~\ref{tab:main_gen} reports the evaluation results of multi-modal response generation baseline. Follow~\citet{sun-etal-2022-multimodal}, we evaluate the textual response generation, image generation and intent prediction tasks. Firstly, we can find that the state-of-the-art model Divter achieves relatively low textual response generation performance (8.78 on BLEU-1 and 11.57 on ROUGE-L) on our proposed MMDialog, which validates the difficulty of multi-modal response generation tasks and also demonstrates the necessity of constructing a large-scale multi-modal dialogue dataset for building data-driven models. Secondly, compared with the results on text generation, we are surprised to find that the model achieves better performance on the image generation task and reaches 21.84 on IS. Thirdly, we observe that the baseline achieve a 75.56 F1 score on intent prediction task, indicating that the model has a considerable ability to determine whether to generate text or images during the conversation. Finally, we also leverage the proposed MM-Relevance to evaluate the overall relevance degree between the generated multi-modal dialogue responses and ground-truth ones and our baseline achieves a score of 68.40.
{"value": "75.56", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Intent Prediction", "metric": "F1", "experimental settings": {"xx": "yy"}, "model": "Divter", "model settings": {"xx": "yy"}} {"value": "21.84", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Image Generation", "metric": "IS", "experimental settings": {"xx": "yy"}, "model": "Divter", "model settings": {"xx": "yy"}} {"value": "8.78", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Textual Response Generation", "metric": "BLEU-1", "experimental settings": {"xx": "yy"}, "model": "Divter", "model settings": {"xx": "yy"}} {"value": "6.86", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Textual Response Generation", "metric": "BLEU-2", "experimental settings": {"xx": "yy"}, "model": "Divter", "model settings": {"xx": "yy"}} {"value": "11.57", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Textual Response Generation", "metric": "ROUGE-L", "experimental settings": {"xx": "yy"}, "model": "Divter", "model settings": {"xx": "yy"}} {"value": "68.40", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Multi-Modal Generation", "metric": "MM-Relevance", "experimental settings": {"xx": "yy"}, "model": "Divter", "model settings": {"xx": "yy"}}