Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities. However, existing work learns joint representations using multiple modalities as input and may be sensitive to noisy or missing modalities at test time. With the recent success of sequence to sequence models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time. In this paper, we propose a method to learn robust joint representations by translating between modalities. Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input. We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities. Once our translation model is trained with paired multimodal data, we only need data from the source modality at test-time for prediction. This ensures that our model remains robust from perturbations or missing target modalities. We train our model with a coupled translation-prediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICT-MMMO, and YouTube. Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to perturbations of all other modalities. Sentiment analysis, which involves identifying a speaker's opinion, is a core research problem in machine learning and natural language processing. However, language-based sentiment analysis through words, phrases, and their compositionality was found to be insufficient for inferring affective content from spoken opinions BID33 , which contain rich nonverbal behaviors in addition to verbal text. As a result, there has been a recent push towards using machine learning methods to learn joint representations from additional behavioral cues present in the visual and acoustic modalities. This research field has become known as multimodal sentiment analysis and extends the conventional textbased definition of sentiment analysis to a multimodal setup. For example, BID21 explore the additional acoustic modality while BID61 use the language, visual, and acoustic modalities present in monologue videos to predict sentiment. The abundance of multimodal data has led to the creation of multimodal datasets, such as CMU-MOSI BID66 and ICT-MMMO BID61 , as well as deep multimodal models that are highly effective at learning discriminative joint multimodal representations BID29 BID57 BID7 . Existing work learns joint representations using multiple modalities as input with neural networks BID28 , graphical models BID33 or geometric classifiers BID66 . However, this results in joint representations that are sensitive to noisy or missing modalities at test time. To address this problem, we draw inspirations from the recent success of sequence to sequence models for unsupervised representation learning BID55 . We propose the Multimodal Cyclic Translation Network model (MCTN) to learn robust joint multimodal representations by translating between modalities. FIG0 illustrates these translations between the language, visual and acoustic modalities. Our method is based on the key insight that translation from a source modality S to a target modality T results in an intermediate representation that captures joint information between modalities S and T . MCTN extends this insight using a cyclic translation loss involving both forward translations from source to target, and backward translations from the predicted target back to the source modality. Together, we call these multimodal cyclic translations to ensure that the learned joint representations capture maximal information from both modalities. We also propose a hierarchical MCTN to learn joint representations between a source modality and multiple target modalities. MCTN is trainable end-to-end with a coupled translation-prediction loss which consists of (1) the cyclic translation loss, and (2) a prediction loss to ensure that the learned joint representations are task-specific. Another advantage of MCTN is that once trained with paired multimodal data (S, T ), we only need data from the source modality S at test time to infer the joint representation and sentiment prediction. As a result, MCTN is completely robust to test-time perturbations on target modality T and missing modalities.Even though translation and generation of videos, audios, and text are difficult BID27 , our experiments show that the learned joint representations can help for discriminative tasks: MCTN achieves new state-of-the-art results on multimodal sentiment analysis using the CMU-MOSI, ICT-MMMO, and YouTube public datasets. Additional experiments show that MCTN learns increasingly discriminative joint representations with more input modalities while maintaining robustness to all target modalities. This section discusses several research questions and presents our experimental results. To conclude, this paper investigated learning joint representations via cyclic modality translations from source to target modalities. During testing, we only need the source modality for prediction which ensures that our model remains robust from noisy or missing target modalities. We demonstrate that cyclic translations and seq2seq models are especially useful for learning joint multimodal representations. In addition to achieving state-of-the-art results on three datasets, our model learns increasingly discriminative representations with more input modalities while maintaining robustness to all target modalities. Our approach presents several exciting areas for future work, such as: 1) combining our approach with the transformer architecture BID59 for modality translations, 2) exploring pretrained deep language models BID12 BID42 for translations, as well as 3) extending our translation model to work other multimodal tasks involving language and raw speech signals (prosody), videos with multiple speakers (diarization), and combinations of static and temporal data (i.e. image captioning).
We present a model that learns robust joint representations by performing hierarchical cyclic translations between multiple modalities. This paper presents the Multimodal Cyclic Translation Network (MCTN) and evaluates it for multimodal sentiment analysis.