\begin{table*}[ht!]
\centering
\resizebox{0.69\textwidth}{!}{
\begin{tabular}{lccccccc}
\toprule
\multirow{2}{*}{Models} &  \multicolumn{1}{c}{Intent} & \multicolumn{4}{c}{Image Retrieval}  
\\ \cmidrule(lr){2-2} \cmidrule(lr){3-6}

      &  F1  & R@1   & R@5   & R@10 & Sum(R@1,5,10)  \\ \midrule 
DE++~\cite{zang-etal-2021-photochat}     & 59.04  & 29.55     &  45.14   & 53.61    &  128.30  \\ 

\bottomrule
\end{tabular}
}
\caption{Automatic evaluation results of the retrieval baselines on the test set of MMDialog. All numbers are in percentage.}
\label{tab:main_ret}
\end{table*}
\subsection{Multi-Modal Dialogue Modeling}
Based on the aforementioned multi-modal dialogue datasets, many advanced works have been proposed. Several modeling works~\citep{qi2020two,niu2019recursive,gan2019multi} investigate how to escalate the performance of conversational agents in image-grounded dialogue. Afterward, researchers~\citep{yang2021open,liang2021maria} explore enriching textual expressions of generated dialogue responses through associative vision scenes.~\citet{zang-etal-2021-photochat} proposes two tasks, including photo-sharing intent prediction to predict whether model should intend to share the photo in the next dialogue turn and a dialogue-based image retrieval task to retrieve the most proper photo given the dialogue context. They also propose a dual-encoder model that uses object labels to encode image features, which achieves the best performance among all the models w/o cross-attention mechanisms. However, the authors do not conduct textual response retrieval tasks.~\citet{zheng-etal-2022-mmchat} proposes a multi-modal dialogue generation model based on Seq2Seq architecture, which was proved to be superior to textual Seq2Seq model. However, this model can only generate plain textual responses, which is not in line with the open domain multi-modal response generation scenario. Recently,~\citet{sun-etal-2022-multimodal} make the first attempt to build a multi-modal dialogue response generation model named Divter that can effectively understand multi-modal dialogue context and generate informative text and high-resolution image responses. As advanced works on dialogue systems include retrieval-based methods~\citep{wu2017sequential,zhou2018multi,whang2020effective,li2021small} and generative methods~\citep{li2015diversity,serban2016building,zhang2020dialogpt}. Therefore, we adapt Divter~\cite{sun-etal-2022-multimodal} to our multi-modal response generation settings and extend the dual-encoder~\cite{zang-etal-2021-photochat} to the retrieval-based scenarios as baselines.

\begin{equation}
\begin{aligned}
    \text{MM}_{\text{Rel}}(R, \Tilde{R}) &= \sum_{k=1}^{K}(e^m_k)^T \cdot \Tilde{e}^m_k \\
    K & = \min\{n_r, n_{\Tilde{r}} \}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \text{P}_{\text{MM}} &= \frac{\text{MM}_{\text{Rel}}(R, \Tilde{R})}{n_{\Tilde{r}}} \\ 
    \text{R}_{\text{MM}} &= \frac{\text{MM}_{\text{Rel}}(R, \Tilde{R})}{n_r} \\
    \text{F1}_{\text{MM}} &= \frac{2\text{P}_{\text{MM}}\text{R}_{\text{MM}}}{\text{P}_{\text{MM}}+\text{R}_{\text{MM}}} \\
\end{aligned} 
\end{equation}

As for the retrieval tasks, we also include ranking-based evaluation metrics recall $n$ at $k$ including R@1, R@5 and R@10 consistent with previous works~\cite{zang-etal-2021-photochat}, measuring if the ground-truth textual or visual outputs can be ranked in top $k \in \{1, 5, 10\}$ positions among $n$ element candidates. With regard to intent prediction, we follow~\citet{zang-etal-2021-photochat} and adopt F1 score as the evaluation metrics that measures the accuracy of the model's prediction of the modality order for a dialogue turn. 

\subsection{Multi-modal Response Retrieval Model}
Inspired by~\citet{parekh-etal-2021-crisscrossed} and \citet{zang-etal-2021-photochat}, we also build a retrieval model $\mathcal{R}$ named DE++ which consists of a modality intent prediction module $\mathcal{R}_\alpha$ and a ranking module $\mathcal{R}_\beta$. As shown in Figure \ref{fig:model}b, before each ranking action, $\mathcal{R}_\alpha$ firstly takes the dialogue context $U$ and previous retrieved response elements $\Tilde{R}_{<z}$ before $z$-th step as inputs and predicts i) the response is completed and model should stop retrieving new elements. or ii) the modality of next elements. If i), the $\mathcal{R}_\alpha$ will take $U$, $\Tilde{R}_{<z}$ as input to predicts the intention $\mathcal{I} ([U,\Tilde{R}_{<z}])$; if ii), $\mathcal{R}_\beta$ will calculate the relevance score  $\mathcal{S} ([U,\Tilde{R}_{<z}], r)$. In the same light,  $\mathcal{R}_\beta$ measures all candidates in  $\{r^{m}_z\}_{z=1}^{Z}$ and selects the one with highest relevance score as the final response.

\subsection{Experimental Setup}
We first sample 10K and 10K dialogues sessions for validation and testing respectively. The detailed statistics are presented in Table~\ref{tab:statistics_split}. For retrieval tasks, we randomly sample 999 negative textual utterances and 999 negative visual images from the same split set for each dialogue, maintaining the total number of candidate elements at 1K. While in training phase, the negative ones are in-batch sampled similar to~\citet{radford2021learning}. For the textual dialogue response generator, we fine-tune DialoGPT~\cite{zhang2020dialogpt} with \textit{transformers} library provided by huggingface\footnote{\url{https://github.com/huggingface/transformers}} using the version ``DialoGPT-medium'' consistent with~\citet{sun-etal-2022-multimodal}. For the description-to-image translator, we implement DALL-E~\cite{ramesh2021zero} using the code of ``mega'' version in \url{https://github.com/borisdayma/dalle-mini}, which also has the same model settings with~\citet{sun-etal-2022-multimodal}. We fine-tune DALL-E mega for one epoch with initial learning rate 1e-7 and mini-batch size of 64. We process all images into 256 $\times$ 256 RGB format for DALL-E. To obtain the description of images in MMDialog, we adopt OFA-huge~\cite{wang2022ofa} using the code \url{https://github.com/OFA-Sys/OFA/tree/feature/add_transformers} for image captioning. All version of CLIP models we leveraged in this paper are ``openai/clip-vit-base-patch32'' in \url{https://huggingface.co/openai/clip-vit-base-patch32}. When implementing Divter, we follow the same experimental configuration. As for the retrieval baseline, the representation vectors for both modality are obtained by CLIP model and fixed during training. The transformers used in retrieval tasks consist of 4 Transformer layers with a hidden size of 512 and 8 heads. We train the retrieval models with an initial learning rate of 5e-7 and mini-batch size of 512. For all baselines, early stopping on the validation set is adopted as a regularization strategy and the best model is selected based on the validation performance. The training of both tasks is conducted on 8 Nvidia Tesla A100 80G GPU cards.

\subsection{Results of Multi-modal Response Generation Model}
Table~\ref{tab:main_gen} reports the evaluation results of multi-modal response generation baseline. Follow~\citet{sun-etal-2022-multimodal}, we evaluate the textual response generation, image generation and intent prediction tasks. Firstly, we can find that the state-of-the-art model Divter achieves relatively low textual response generation performance (8.78 on BLEU-1 and 11.57 on ROUGE-L) on our proposed MMDialog, which validates the difficulty of multi-modal response generation tasks and also demonstrates the necessity of constructing a large-scale multi-modal dialogue dataset for building data-driven models. Secondly, compared with the results on text generation, we are surprised to find that the model achieves better performance on the image generation task and reaches 21.84 on IS. Thirdly, we observe that the baseline achieve a 75.56 F1 score on intent prediction task, indicating that the model has a considerable ability to determine whether to generate text or images during the conversation. Finally, we also leverage the proposed MM-Relevance to evaluate the overall relevance degree between the generated multi-modal dialogue responses and ground-truth ones and our baseline achieves a score of 68.40.

\subsection{Results of Multi-modal Response Retrieval Model}
We also conduct the retrieval baselines and show the results in Table~\ref{tab:main_ret}. Our proposed baseline DE++ achieves the image retrieval performance with 29.55\% R@1, 45.14\% R@5 and 53.61\% R@10, which demonstrating the capacity of multi-modal retrieval model and the effectiveness of CLIP representation. As for the intent prediction, the F1 score is 59.04 which is inferior to the counterpart in generative baseline Divter. This may be due to the fact that 24 layers of transformer (i.e., DialoGPT-medium) is used to encode the context in Divter but only 4 layers are used in DE++.
{"value": "59.04", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Intent Prediction", "metric": "F1", "experimental settings": {"xx": "yy"}, "model": "DE++", "model settings": {"xx": "yy"}} {"value": "29.55", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Image Retrieval", "metric": "R@1", "experimental settings": {"xx": "yy"}, "model": "DE++", "model settings": {"xx": "yy"}} {"value": "45.14", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Image Retrieval", "metric": "R@5", "experimental settings": {"xx": "yy"}, "model": "DE++", "model settings": {"xx": "yy"}} {"value": "53.61", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Image Retrieval", "metric": "R@10", "experimental settings": {"xx": "yy"}, "model": "DE++", "model settings": {"xx": "yy"}} {"value": "128.30", "type": "Result", "training data/set": "MMDialog", "test data/set": "MMDialog", "task": "Image Retrieval", "metric": "Sum(R@1,5,10)", "experimental settings": {"xx": "yy"}, "model": "DE++", "model settings": {"xx": "yy"}}