# Analyzing Modality Robustness in Multimodal Sentiment Analysis

Devamanyu
Hazarika${}^{\includegraphics[height=8.67424pt]{images/nus_logo.png}}$∗,
Yingting Liα ${}^{\includegraphics[height=8.67424pt]{images/declarelogo.pdf}}$
,
Bo Chengα, Shuai Zhaoα, Roger
Zimmermann${}^{\includegraphics[height=8.67424pt]{images/nus_logo.png}}$,
Soujanya
Poria${}^{\includegraphics[height=8.67424pt]{images/declarelogo.pdf}}$
${}^{\includegraphics[height=8.67424pt]{images/nus_logo.png}}$ National
University of Singapore, Singapore
α Beijing University of Posts and Telecommunications
${}^{\includegraphics[height=8.67424pt]{images/declarelogo.pdf}}$ DeCLaRe Lab,
Singapore University of Technology and Design, Singapore
{hazarika<EMAIL_ADDRESS>
{cindyyting, chengbo<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Equal Contribution.

###### Abstract

Building robust multimodal models are crucial for achieving reliable
deployment in the wild. Despite its importance, less attention has been paid
to identifying and improving the robustness of Multimodal Sentiment Analysis
(MSA) models. In this work, we hope to address that by ($i$) Proposing simple
diagnostic checks for modality robustness in a trained multimodal model. Using
these checks, we find MSA models to be highly sensitive to a single modality,
which creates issues in their robustness; ($ii$) We analyze well-known robust
training strategies to alleviate the issues. Critically, we observe that
robustness can be achieved without compromising on the original performance.
We hope our extensive study–performed across five models and two benchmark
datasets–and proposed procedures would make robustness an integral component
in MSA research. Our diagnostic checks and robust training solutions are
simple to implement and available at https://github.com/declare-lab/MSA-
Robustness

##  1 Introduction

Multimodal Sentiment Analysis (MSA) is a burgeoning field of research that has
seen accelerated developments in recent years. Numerous models have been
proposed that utilize multiple modalities such as audio, visual, and language
signals to predict sentiments, emotions, and other forms of affect. While
progress in MSA has been driven mainly by improvements in multimodal
performance, we call for attention towards an equally important aspect in
multimodal systems – multimodal robustness. Robustness is crucial when models
are deployed in the wild, where it is common to encounter inadvertent errors
in the source modalities due to data loss, data corruption, jitter, privacy
issues, amongst others.

A well-known fact in the MSA research is that language modality tends to be
the most effective, which has prompted models to utilize language as its core
modality Wu et al. (2021); Han et al. (2021a); Zeng et al. (2021). In this
work, we focus on skewed dependence on language and try to understand how it
affects the robustness of MSA models. Specifically, we ask,

Figure 1: Removing modalities one at a time from the testing set of CMU-MOSI
Zadeh et al. (2016) on a trained MISA Hazarika et al. (2020).

RQ1: Are models in MSA over-reliant on a subset of modalities, particularly
language?

RQ2: If yes, what implications does it have on modality robustness?

To answer RQ1, we look at Fig. 1. The figure illustrates a setup where we
fully remove one modality during testing on the MISA model Hazarika et al.
(2020). Here, we observe a sharp drop in performance when language modality is
removed but do not see statistically significant drops when audio or visual
modalities are removed. This observation aligns with recent findings in the
MSA literature highlighting the dominance of language.

This brings us to RQ2 where we try to understand the robustness implications
over this dominance. We design an elaborate study in § 3—over five state-of-
the-art (SOTA) MSA models and across two benchmark datasets—where we propose
diagnostic checks to understand modality robustness, i.e., how robust are
models against modality errors such as missing or noisy modalities.

Based on our findings, we then proceed to ask,

RQ3: How can we improve the robustness of these models?

RQ4: Does robust training lead to a performance trade-off?

For RQ3, we study well-known robust training methods, that act as a pre-
emptive strategy to reduce the performance drops. Critically, our training is
model-agnostic and can be easily included in any existing multimodal model (§
4). For RQ4, we observe that our method to improve robustness does not trade-
off with the final performance on the clean testing set, thus achieving
similar performance as the original model.

##  2 Related Works

While MSA has received increased attention in recent times, the topic of
robustness has not taken center stage. Fortunately, few works have started
changing this trend. Gat et al. (2020) reveals how multimodal classifiers
often utilize a subset of modalities, which they addressed by inducing uniform
contribution from all input modalities. In MSA, multiple works over-rely on
language modality to improve the performance. Wu et al. (2021) constructs a
text-centered shared-private framework for multimodal fusion, and Han et al.
(2021a) obtains two text-related modal pairs and iteratively push the
interaction between modalities to supplement information for better
performance. While this has enabled performance boosts, our goal is to explore
the double-edged nature of this feature and how it impacts robustness. Our
motivation for diagnostics is similar to Frank et al. (2021), but unlike them,
we do not perturb the raw data (such as image patches). Instead, we intervene
on modality representations, which is easier to integrate with existing models
and do not require prior knowledge of the modality structure.

To address robustness in MSA, Tsai et al. (2018) proposes a factored model
that can accommodate modality drops. Also, Ma et al. (2021) introduces
modality drops during training and testing and uses meta-learning to make
models robust. However, our work comprises some crucial distinctions: $i)$
Unlike these works, our diagnostics and robust training do not require
sophisticated architecture and can be easily integrated into existing models.
$ii)$ We perform an exhaustive analysis of robustness across multiple models,
which is previously not done in the MSA literature.

##  3 Testing Robustness via Diagnostic Checks

In this section we perform an elaborate study on modality robustness by
simulating potential issues with modality signals during testing (or
deployment) of MSA models.

Figure 2: Diagnostic checks (missing modality) for modality robustness in MOSI
and MOSEI datasets. Results are averaged over three independent runs. Each
modality error is applied to 30% of testing data.

###  3.1 Experiment Setup

#### Models.

In order to fully verify the universality of our experiments, we select a
series of diverse SOTA models, ranging from RNN-based to Transformer-based
architectures. These models work across different granularities from word-
level to sentence-level variants:

$(i)$ MISA Hazarika et al. (2020) is a popular model that generates modality-
invariant and -specific features of multimodal data, to learn both shared and
unique characteristics of each modality. $(ii)$ BBFN Han et al. (2021a) in a
similar vein performs fusion and separation to increase cross-modal relevances
and differences. This work acknowledges the dominance of text modality in MSA
and proposes two text-centric bi-modal transformers to increase performance.
$(iii)$ Self-MM Yu et al. (2021) focuses on the relationship between multi-
and uni-modal predictions by multi-tasking consistencies and differences
between them. $(iv)$ MMIM Han et al. (2021b) incorporates mutual information
(MI) into MSA by maximizing MI at the input and fusion level. $(v)$ MulT Tsai
et al. (2019) merges multimodal time series through multiple sets of
directional pairwise cross-modal transformers. It accounts for long-range
dependencies across modality elements to create a strong baseline (see
Appendix B).

#### Datasets.

We consider two benchmark datasets widely used in the field of multimodal
sentiment analysis, CMU-MOSI Zadeh et al. (2016), which is a popular dataset
for studying the intensity of multimodal sentiment in the MSA field and CMU-
MOSEI Bagher Zadeh et al. (2018) which is a larger counterpart of MOSI with
richer annotations and more diverse samples. Both these datasets contain short
utterance videos and provide language, audio, and visual modality features.

###  3.2 Proposed Diagnostic Checks

We propose two diagnostic checks that introduce $i)$ Missing Modalities, which
drops (or nullifies) a modality from the input and $ii)$ Noisy Modalities
which include random changes to the modality representations, introduced via
white Gaussian noise to the respective modality representations111While
missing and noisy errors are predominant in the wild, we leave other potential
forms of errors, such as affine transformations to the representations for
future work.. To simulate a realistic scenario, we apply these checks to 30%
of the testing data set222We set 30% arbitrarily to simulate modality errors
to a proportion of the input signals.. Given the increased dependence on
language modality in MSA models, we limit our study to errors introduced only
in language modality without loss of generality.

Figure 3: Diagnostic checks (noisy modality) for modality robustness in MOSI
and MOSEI datasets. Results are averaged over three independent runs. Each
modality error is applied to 30% of testing data.

#### Procedure.

We aim to intervene on modality representations to simulate modality errors.
For the language modality $l$, all models map the sequence of tokens $U_{l}\in
R^{T_{l}}$ to its low-level embedding $\mathbf{U}_{l}\in R^{T_{l}\times
d_{l}}$ with $T_{l}$ tokens and $d_{l}$ embedding dimension. This low-level
sequence is then encoded into hidden representations using an encoder of
choice, such as BERT Devlin et al. (2019), to achieve the language
representation vector
$\mathbf{u}_{l}=enc_{\theta_{l}}(U_{l})\in\mathbb{R}^{d}$. We intervene on
this representation and apply our diagnostics as follows.

We sample 30% of $u_{l}$ from the testing set and modify them as
$\mathbf{\hat{u}}_{l}=f(\mathbf{u}_{l})$, where $f(\mathbf{x})$ is defined as
either $f(\mathbf{x})=\mathbf{x}\odot\mathbf{0}$ for modality dropping
(nulling the vector to 0s by element-wise multiplication) or
$f(\mathbf{x})=\mathbf{x}+\mathcal{N}(\mathbf{0},\mathbf{1})$ to add white
noise. The modified $\mathbf{\hat{u}}_{l}$ is then fed to the rest of the
network as usual.

In the selected models, we apply diagnostics at different network locations.
These include the representations before the hidden projection, such as in
MISA, or fusion operation, such as in Self-MM. For MulT and BBFN, we apply the
interventions right after the word embeddings. Detailed discussion on the
location of interventions is provided in Appendix A.

#### Observations.

Figs. 2 and 3 presents the results, where across both MOSI and MOSEI datasets,
we find that language modality is highly sensitive to modality errors in the
language source (across all models). This trend is observed for both missing
and noisy modality checks, thus highlighting the concerns over robustness of
these SOTA models. These diagnostic checks are easy to analyze, and we hope
they will become an integrated part of the model-development pipeline in MSA.

##  4 Robust Training

In this section, we explore how to reduce the sensitivity of the models to the
dominant modality, i.e., language. One of the popular ways to alleviate such
issues is to teach the model such scenarios during training. We dub this
approach as modality-perturbation, which is conceptually similar to removing
or masking modalities in Ma et al. (2021); Georgiou et al. (2021) or adding
noise in Miyato et al. (2018). It simulates the modality errors during
training so that the model learns to expect such events during
testing/deployment. The procedure is as follows,

  1. 1.

Training:

    1. (a)

For a particular batch of data, sample a proportion of the data to be
perturbed.

    2. (b)

Similar to the diagnostic checks in § 3, perturb the dominant modality (in our
case, language) of half of this data with missing and the other half with
noisy perturbation. Repeat both these steps for the next batch.

  2. 2.

Testing: Apply the diagnostic checks as in § 3.

This simple approach can be interpreted as regularization akin to dropouts or
noising strategies used in de-noising auto-encoders.

###  4.1 Results

#### Robustness.

Table 1 presents the results, where we perform balanced perturbation between
missing and noisy modalities. For the 30% perturbable data in training, we
drop the language modality on 15% and for the other 15%, we add noise. This
setting improves the diagnostics in both kinds of errors. Appendix C presents
results on other proportions of the training data.

With balanced perturbation, (BBFN-MOSI) reduces the relative drop on missing
language diagnostic by 31% (in F1) and 98% on noise. Also, missing drop
reduces by 11% in Corr and by 99% for noisy diagnostic. (Self-MM, MOSI)
increases the relative drop in Corr slightly on missing diagnostic, but in all
other cases, it is significantly reduced. For example, the F1 drop on MOSI for
noisy diagnostic reduces significantly by 93%. Table 1 also shows that our
method performs well on both RNN-based and Transformer-based models,
demonstrating the wide applicability of our method.

#### Performance Trade-off.

While alleviating robustness via regularization is well-known in the
literature, there is often a trade-off with absolute performance in the
original testing setup. Most approaches that achieve robustness take a hit at
their best performance on clean input Zhang et al. (2022) Nakkiran (2019) Su
et al. (2018) Tsipras et al. (2019). This raises the question of whether
introducing modality-perturbation reduces the performance of the model on the
original testing set.

We find the answer to this is No. Surprisingly, our robust training procedure
does not degrade in its original performance and can perform similar to the
original model variants. This is highly ideal as we achieve robustness without
compromising on performance in clean data.

|  Diagnostic (30%) |  Robust Training | MOSI | MOSEI
---|---|---|---|---
| Corr | F1 | Corr | F1
MISA |  | - | 0.737 | 82.40 | 0.765 | 85.76
Yes | 0.736 | 81.42 | 0.767 | 85.97
missing | - |  $\downarrow$ 0.122 | $\downarrow$ 11.53 | $\downarrow$ 0.186 | $\downarrow$ 8.45
Yes | $\downarrow$ 0.210 |  $\downarrow$ 9.96 |  $\downarrow$ 0.147 |  $\downarrow$ 8.26
noise | - |  $\downarrow$ 0.122 | $\downarrow$ 11.67 | $\downarrow$ 0.136 | $\downarrow$ 8.36
Yes | $\downarrow$ 0.163 |  $\downarrow$ 10.10 |  $\downarrow$ 0.002 |  $\downarrow$ 0.19
BBFN |  | - | 0.754 | 83.12 | 0.764 | 85.70
Yes | 0.754 | 83.28 | 0.763 | 85.43
missing | - | $\downarrow$ 0.127 | $\downarrow$ 10.55 | $\downarrow$ 0.139 | $\downarrow$ 10.57
Yes |  $\downarrow$ 0.119 |  $\downarrow$ 7.28 |  $\downarrow$ 0.124 |  $\downarrow$ 7.88
noise | - | $\downarrow$ 0.232 | $\downarrow$ 8.58 | $\downarrow$ 0.308 | $\downarrow$ 9.62
Yes |  $\downarrow$ 0.046 |  $\downarrow$ 0.16 |  $\downarrow$ 0.003 |  $\downarrow$ 0.23
Self-MM |  | - | 0.794 | 85.61 | 0.759 | 84.62
Yes | 0.790 | 84.73 | 0.754 | 84.67
missing | - |  $\downarrow$ 0.099 | $\downarrow$ 11.74 | $\downarrow$ 0.126 | $\downarrow$ 9.04
Yes | $\downarrow$ 0.120 |  $\downarrow$ 9.66 |  $\downarrow$ 0.122 |  $\downarrow$ 6.86
noise | - | $\downarrow$ 0.154 | $\downarrow$ 8.35 | $\downarrow$ 0.172 | $\downarrow$ 9.48
Yes |  $\downarrow$ 0.041 |  $\downarrow$ 0.58 |  $\downarrow$ 0.051 |  $\downarrow$ 1.02
MMIM |  | - | 0.796 | 86.02 | 0.758 | 84.89
Yes | 0.784 | 84.67 | 0.751 | 83.15
missing | - |  $\downarrow$ 0.117 |  $\downarrow$ 9.37 | $\downarrow$ 0.122 | $\downarrow$ 8.15
Yes | $\downarrow$ 0.146 | $\downarrow$ 10.48 |  $\downarrow$ 0.115 |  $\downarrow$ 6.62
noise | - | $\downarrow$ 0.197 | $\downarrow$ 9.55 | $\downarrow$ 0.191 | $\downarrow$ 9.18
Yes |  $\downarrow$ 0.180 |  $\downarrow$ 8.88 |  $\downarrow$ 0.096 |  $\downarrow$ 4.41
Mult |  | - | 0.747 | 82.25 | 0.738 | 83.37
Yes | 0.744 | 82.21 | 0.745 | 83.95
missing | - |  $\downarrow$ 0.113 | $\downarrow$ 12.17 |  $\downarrow$ 0.109 |  $\downarrow$ 6.60
Yes | $\downarrow$ 0.117 |  $\downarrow$ 9.63 | $\downarrow$ 0.113 | $\downarrow$ 7.03
noise | - | $\downarrow$ 0.295 | $\downarrow$ 8.99 | $\downarrow$ 0.263 | $\downarrow$ 7.95
Yes |  $\downarrow$ 0.068 |  $\downarrow$ 6.13 |  $\downarrow$ 0.001 |  $\uparrow$ 0.02

Table 1: Robust Training is performed with 15% missing and 15% noise
perturbation. Results are averaged over 3 random runs. More perturbation
results are provided in Appendix C. Higher drops between Non-robust and robust
training (consecutive rows) are highlighted.

##  5 Conclusion

In this work, we performed a systematic study that demonstrate the double-
edged nature of dominant modality in SOTA MSA models. Our analysis using
diagnostic checks reveal high susceptibility to performance drops when
presented with unwanted errors in their representations.

To alleviate the issues, we also study robust training methods that uses
modality perturbations. Critically, we find that robustness and performance
can co-exist without an explicit trade-off. These improvements demonstrate a
positive nudge in the effort to achieve robustness and we believe there
remains significant room for improvement. With this work, by proposing simple
and easy-to-integrate diagnostic checks and training methods, we hope to
permeate discussions on robustness into mainstream MSA research.

## Acknowledgements

This research is supported by $(i)$ T2MOE2008 awarded by Singapore’s MoE under
its Tier-2 grant scheme, and $(ii)$ SUTD SRG grant #T1SRIS19149. We also thank
the anonymous reviewers for their valuable comments.

## References

  * Bagher Zadeh et al. (2018) AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018.  Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2236–2246, Melbourne, Australia. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Frank et al. (2021) Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021.  Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9847–9857.
  * Gat et al. (2020) Itai Gat, Idan Schwartz, Alexander G. Schwing, and Tamir Hazan. 2020.  Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Georgiou et al. (2021) Efthymios Georgiou, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021.  M3: Multimodal masking applied to sentiment analysis.  _Proc. Interspeech 2021_ , pages 2876–2880.
  * Han et al. (2021a) Wei Han, Hui Chen, Alexander F. Gelbukh, Amir Zadeh, Louis-Philippe Morency, and Soujanya Poria. 2021a.  Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis.  In _ICMI ’21: International Conference on Multimodal Interaction, Montréal, QC, Canada, October 18-22, 2021_ , pages 6–15. ACM.
  * Han et al. (2021b) Wei Han, Hui Chen, and Soujanya Poria. 2021b.  Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 9180–9192. Association for Computational Linguistics.
  * Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020.  MISA: modality-invariant and -specific representations for multimodal sentiment analysis.  In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_ , pages 1122–1131. ACM.
  * Ma et al. (2021) Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. 2021.  Smil: Multimodal learning with severely missing modality.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 2302–2310.
  * Miyato et al. (2018) Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8):1979–1993.
  * Nakkiran (2019) Preetum Nakkiran. 2019.  Adversarial robustness may be at odds with simplicity.  _CoRR_ , abs/1901.00532.
  * Su et al. (2018) Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yu Chen, and Yupeng Gao. 2018\.  Is robustness the cost of accuracy? - A comprehensive study on the robustness of 18 deep image classification models.  In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XII_ , volume 11216 of _Lecture Notes in Computer Science_ , pages 644–661. Springer.
  * Tsai et al. (2019) Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 6558–6569. Association for Computational Linguistics.
  * Tsai et al. (2018) Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018.  Learning factorized multimodal representations.  In _International Conference on Learning Representations_.
  * Tsipras et al. (2019) Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019.  Robustness may be at odds with accuracy.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wu et al. (2021) Yang Wu, Zijie Lin, Yanyan Zhao, Bing Qin, and Li-Nan Zhu. 2021.  A text-centered shared-private framework via cross-modal prediction for multimodal sentiment analysis.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4730–4738, Online. Association for Computational Linguistics.
  * Yu et al. (2021) Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021.  Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 10790–10797. AAAI Press.
  * Zadeh et al. (2016) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ , 31(6):82–88.
  * Zeng et al. (2021) Ying Zeng, Sijie Mai, and Haifeng Hu. 2021.  Which is making the contribution: Modulating unimodal and cross-modal dynamics for multimodal sentiment analysis.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1262–1274.
  * Zhang et al. (2022) Yuxin Zhang, Jindong Wang, Yiqiang Chen, Han Yu, and Tao Qin. 2022.  Adaptive memory networks with self-supervised learning for unsupervised anomaly detection.  _CoRR_ , abs/2201.00464.

##  Appendix A Model Details

#### MISA:

We get the MISA model from its official repository
333https://github.com/declare-
lab/MISA/tree/ec42faddde0d210cf7368aebf2118fe9570e7102. In this model, we
apply the interventions at the following encoded language representation from
the original paper:

$\displaystyle\mathbf{u}_{l}=\operatorname{Bert}\left(\mathbf{U}_{l};\theta_{l}^{bert}\right)$
(1) $\displaystyle\mathbf{\hat{u}}_{l}=f(\mathbf{\hat{u}}_{l})$ (2)
$\displaystyle\mathbf{h}_{l}^{c}=E_{c}\left(\mathbf{\hat{u}}_{l};\theta^{c}\right),\quad\mathbf{h}_{l}^{p}=E_{p}\left(\mathbf{\hat{u}}_{l};\theta_{l}^{p}\right)$
(3)

That is, the interventions are applied before the language representation is
projected to its shared and private subspaces.

#### BBFN:

We get the BBFN model from its official repository
444https://github.com/declare-
lab/BBFN/tree/be15f947ed7539b3c54381e453f09439466ed915. In this model, we
execute the interventions after the following language embedding from the
original paper.

$\displaystyle\mathbf{M}_{l}=\left(m_{0},m_{1},\ldots,m_{n+1}\right)$ (4)
$\displaystyle\mathbf{\hat{M}}_{l}=f(\mathbf{M}_{l})$ (5)

#### Self-MM:

We get the Self-MM model from its official
repository555https://github.com/thuiar/Self-MM. In this model, we set the
interventions after the language features encoded below from the original
paper.

$\displaystyle\mathbf{F}_{l}=BERT\left(I_{l};\theta_{l}^{bert}\right)\in
R^{d_{l}}$ (6) $\displaystyle\mathbf{\hat{F}}_{l}=f(\mathbf{F}_{l})$ (7)

#### MMIM:

We get the MMIM model from its official repository
666https://github.com/declare-lab/Multimodal-Infomax. For this model, we
perform the interventions after the following encoded language representation
from the original paper.

$\displaystyle\mathbf{x}_{l}=\operatorname{BERT}\left(X_{l};\theta_{l}^{BERT}\right)$
(8) $\displaystyle\mathbf{\hat{x}}_{l}=f(\mathbf{x}_{l})$ (9)

#### MulT:

We get the MulT model from its official repository
777https://github.com/yaohungt/Multimodal-Transformer. We intervene in this
model after the following encoded language representation:.

$\displaystyle\mathbf{x}_{l}=\operatorname{Conv1D}(X_{l},k_{l})\in\mathbb{R}^{T_{l}\times
d}$ (10) $\displaystyle\mathbf{\hat{x}}_{l}=f(\mathbf{x}_{l})$ (11)

##  Appendix B Reproducing Results

Models | Item | CMU-MOSI | CMU-MOSEI
---|---|---|---
MISA | Learning rate | 1e-5 | 4e-5
Optimizer | RMSprop | Adam
Activation | hardtanh | relu
BBFN | Learning rate | 1e-4 | 5e-5
MMIM | Batch size | 32 | 64
learning rate $\eta_{lld}$ | 4e-3 | 1e-3
learning rate $\eta_{main}$ | 1e-3 | 5e-4
$\alpha$ | 0.3 | 0.1
$\beta$ | 0.1 | 0.05
V-LSTM hidden dim | 32 | 64
A-LSTM hidden dim | 32 | 16
MulT | Batch size | 128 | 16
Learning rate | 1e-4 | 1e-4
Optimizer | Adam | Adam
Transformers Hidden Unit Size d | 30 | 30
No. of Crossmodal Attention Heads | 10 | 10
No. of Crossmodal a Blocks D | 4 | 4
Textual Embedding Dropout | 0.3 | 0.3
Crossmodal Attention Block Dropout | 0.2 | 0.1
Output Dropout | 0.2 | 0.1
Gradient Clip | 0.8 | 1.0
No. of Epochs | 100 | 20
Use Bert | Yes | Yes

Table 2: Hyper-parameter config used to train the models.

For each model we train the models to achieve performances close to reported
in the respective papers. Table 2 presents the hyper-parameters we used to
reproduce their results.

##  Appendix C Additional Results on Modality-Perturbation

We also analyze with varying proportions of perturbations in the training and
testing phase, respectively. As seen in Table 3, as the noise gradually
increases from 5% to 15%, the drop of Corr in MOSI is gradually reduced ,
which shows the robustness is getting better, until it reaches the optimum at
30% perturbation (15% missing + 15% noise). In other models, 30% perturbation
is also advantageous. For example, in Table 7, (Mult, MOSEI) reduces Corr drop
while improving F1 performance in 30% perturbation. Although it is only a
small improvement at present, we believe that there will be more meaningful
improvements in the future.

|  Robust Training | Diagnostic | MOSI | MOSEI
---|---|---|---|---
| Corr | F1 | Corr | F1
MISA | - |  | 0.737 | 82.40 | 0.765 | 85.76
missing 5% |  $\downarrow$ 0.011 |  $\downarrow$ 1.18 |  $\downarrow$ 0.042 |  $\downarrow$ 1.57
noise 5% |  $\downarrow$ 0.022 |  $\downarrow$ 0.75 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.031 |  $\downarrow$ 2.19 |  $\downarrow$ 0.073 |  $\downarrow$ 2.41
noise 10% |  $\downarrow$ 0.047 |  $\downarrow$ 2.56 |  $\downarrow$ 0.031 |  $\downarrow$ 1.75
missing 15% |  $\downarrow$ 0.050 |  $\downarrow$ 3.99 |  $\downarrow$ 0.126 |  $\downarrow$ 4.10
noise 15% |  $\downarrow$ 0.076 |  $\downarrow$ 4.05 |  $\downarrow$ 0.068 |  $\downarrow$ 3.88
missing 30% |  $\downarrow$ 0.122 |  $\downarrow$ 11.53 |  $\downarrow$ 0.186 |  $\downarrow$ 8.45
noise 30% |  $\downarrow$ 0.210 |  $\downarrow$ 9.96 |  $\downarrow$ 0.147 |  $\downarrow$ 8.26
5% missing 5% noise |  | 0.714 | 80.99 | 0.765 | 85.68
missing 5% |  $\downarrow$ 0.010 |  $\downarrow$ 1.38 |  $\downarrow$ 0.042 |  $\downarrow$ 1.57
noise 5% |  $\downarrow$ 0.032 |  $\downarrow$ 1.37 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.030 |  $\downarrow$ 2.17 |  $\downarrow$ 0.074 |  $\downarrow$ 2.41
noise 10% |  $\downarrow$ 0.055 |  $\downarrow$ 2.44 |  $\downarrow$ 0.041 |  $\downarrow$ 2.22
missing 15% |  $\downarrow$ 0.044 |  $\downarrow$ 4.52 |  $\downarrow$ 0.126 |  $\downarrow$ 0.41
noise 15% |  $\downarrow$ 0.094 |  $\downarrow$ 3.65 |  $\downarrow$ 0.058 |  $\downarrow$ 3.53
missing 30% |  $\downarrow$ 0.121 |  $\downarrow$ 11.08 |  $\downarrow$ 0.186 |  $\downarrow$ 8.37
noise 30% |  $\downarrow$ 0.201 |  $\downarrow$ 8.71 |  $\downarrow$ 0.150 |  $\downarrow$ 8.12
10% missing 10% noise |  | 0.734 | 81.40 | 0.765 | 85.68
missing 5% |  $\downarrow$ 0.011 |  $\downarrow$ 1.04 |  $\downarrow$ 0.022 |  $\downarrow$ 1.58
noise 5% |  $\downarrow$ 0.028 |  $\downarrow$ 1.20 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.030 |  $\downarrow$ 2.05 |  $\downarrow$ 0.042 |  $\downarrow$ 2.54
noise 10% |  $\downarrow$ 0.045 |  $\downarrow$ 2.27 |  $\downarrow$ 0.001 |  $\downarrow$ 0.02
missing 15% |  $\downarrow$ 0.049 |  $\downarrow$ 3.54 |  $\downarrow$ 0.071 |  $\downarrow$ 4.04
noise 15% |  $\downarrow$ 0.073 |  $\downarrow$ 3.42 |  $\downarrow$ 0.001 |  $\downarrow$ 0.43
missing 35% |  $\downarrow$ 0.118 |  $\downarrow$ 11.11 |  $\downarrow$ 0.135 |  $\downarrow$ 8.17
noise 30% |  $\downarrow$ 0.181 |  $\downarrow$ 9.55 |  $\downarrow$ 0.035 |  $\downarrow$ 0.54
15% missing 15% noise |  | 0.736 | 81.42 | 0.767 | 85.97
missing 5% |  $\downarrow$ 0.012 |  $\downarrow$ 1.50 |  $\downarrow$ 0.021 |  $\downarrow$ 1.46
noise 5% |  $\downarrow$ 0.027 |  $\downarrow$ 1.50 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.033 |  $\downarrow$ 2.71 |  $\downarrow$ 0.041 |  $\downarrow$ 2.43
noise 10% |  $\downarrow$ 0.032 |  $\downarrow$ 2.12 |  $\downarrow$ 0.001 |  $\downarrow$ 0.03
missing 15% |  $\downarrow$ 0.052 |  $\downarrow$ 4.40 |  $\downarrow$ 0.021 |  $\downarrow$ 4.11
noise 15% |  $\downarrow$ 0.073 |  $\downarrow$ 4.99 |  $\downarrow$ 0.001 |  $\downarrow$ 0.12
missing 30% |  $\downarrow$ 0.122 |  $\downarrow$ 11.67 |  $\downarrow$ 0.136 |  $\downarrow$ 8.36
noise 30% |  $\downarrow$ 0.163 |  $\downarrow$ 10.10 |  $\downarrow$ 0.002 |  $\downarrow$ 0.19

Table 3: MISA Robust Training. Results are averaged over three random runs.

|  Robust Training | Diagnostic | MOSI | MOSEI
---|---|---|---|---
| Corr | F1 | Corr | F1
BBFN | - |  | 0.754 | 83.12 | 0.764 | 85.70
missing 5% |  $\downarrow$ 0.013 |  $\downarrow$ 1.51 |  $\downarrow$ 0.020 |  $\downarrow$ 1.97
noise 5% |  $\downarrow$ 0.034 |  $\downarrow$ 0.75 |  $\downarrow$ 0.050 |  $\downarrow$ 1.21
missing 10% |  $\downarrow$ 0.028 |  $\downarrow$ 2.59 |  $\downarrow$ 0.038 |  $\downarrow$ 3.34
noise 10% |  $\downarrow$ 0.093 |  $\downarrow$ 2.90 |  $\downarrow$ 0.121 |  $\downarrow$ 3.54
missing 15% |  $\downarrow$ 0.032 |  $\downarrow$ 3.79 |  $\downarrow$ 0.055 |  $\downarrow$ 5.45
noise 15% |  $\downarrow$ 0.080 |  $\downarrow$ 2.28 |  $\downarrow$ 0.154 |  $\downarrow$ 4.18
missing 30% |  $\downarrow$ 0.127 |  $\downarrow$ 10.55 |  $\downarrow$ 0.139 |  $\downarrow$ 10.57
noise 30% |  $\downarrow$ 0.232 |  $\downarrow$ 8.58 |  $\downarrow$ 0.308 |  $\downarrow$ 9.62
5% missing 5% noise |  | 0.743 | 82.39 | 0.765 | 85.48
missing 5% |  $\downarrow$ 0.020 |  $\downarrow$ 0.94 |  $\downarrow$ 0.017 |  $\downarrow$ 1.00
noise 5% |  $\downarrow$ 0.002 |  $\rightarrow$ 0 |  $\rightarrow$ 0 |  $\downarrow$ 0.08
missing 10% |  $\downarrow$ 0.039 |  $\downarrow$ 2.17 |  $\downarrow$ 0.032 |  $\downarrow$ 1.95
noise 10% |  $\rightarrow$ 0 |  $\rightarrow$ 0 |  $\rightarrow$ 0 |  $\downarrow$ 0.06
missing 15% |  $\downarrow$ 0.045 |  $\downarrow$ 2.18 |  $\downarrow$ 0.050 |  $\downarrow$ 3.57
noise 15% |  $\downarrow$ 0.002 |  $\downarrow$ 0.15 |  $\downarrow$ 0.001 |  $\rightarrow$ 0
missing 30% |  $\downarrow$ 0.049 |  $\downarrow$ 1.92 |  $\downarrow$ 0.122 |  $\downarrow$ 7.60
noise 30% |  $\rightarrow$ 0 |  $\downarrow$ 0.20 |  $\downarrow$ 0.001 |  $\downarrow$ 0.04
10% missing 10% noise |  | 0.742 | 81.66 | 0752 | 85.15
missing 5% |  $\downarrow$ 0.018 |  $\downarrow$ 1.66 |  $\downarrow$ 0.018 |  $\downarrow$ 1.62
noise 5% |  $\downarrow$ 0.001 |  $\rightarrow$ 0 |  $\downarrow$ 0.022 |  $\downarrow$ 0.05
missing 10% |  $\downarrow$ 0.034 |  $\downarrow$ 2.87 |  $\downarrow$ 0.035 |  $\downarrow$ 2.87
noise 10% |  $\downarrow$ 0.001 |  $\rightarrow$ 0 |  $\downarrow$ 0.003 |  $\downarrow$ 0.09
missing 15% |  $\downarrow$ 0.036 |  $\downarrow$ 3.63 |  $\downarrow$ 0.050 |  $\downarrow$ 4.47
noise 15% |  $\downarrow$ 0.001 |  $\downarrow$ 0.01 |  $\downarrow$ 0.004 |  $\downarrow$ 0.07
missing 30% |  $\downarrow$ 0.126 |  $\downarrow$ 9.75 |  $\downarrow$ 0.125 |  $\downarrow$ 9.40
noise 30% |  $\downarrow$ 0.003 |  $\downarrow$ 0.46 |  $\downarrow$ 0.079 |  $\downarrow$ 0.31
15% missing 15% noise |  | 0.754 | 83.28 | 0.763 | 85.43
missing 5% |  $\downarrow$ 0.020 |  $\downarrow$ 1.50 |  $\downarrow$ 0.018 |  $\downarrow$ 0.96
noise 5% |  $\downarrow$ 0.001 |  $\rightarrow$ 0 |  $\downarrow$ 0.001 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.031 |  $\downarrow$ 2.26 |  $\downarrow$ 0.036 |  $\downarrow$ 2.08
noise 10% |  $\downarrow$ 0.002 |  $\rightarrow$ 0 |  $\rightarrow$ 0 |  $\downarrow$ 0.05
missing 15% |  $\downarrow$ 0.035 |  $\downarrow$ 2.71 |  $\downarrow$ 0.057 |  $\downarrow$ 3.62
noise 15% |  $\downarrow$ 0.003 |  $\downarrow$ 0.14 |  $\downarrow$ 0.001 |  $\rightarrow$ 0
missing 30% |  $\downarrow$ 0.119 |  $\downarrow$ 7.28 |  $\downarrow$ 0.124 |  $\downarrow$ 7.88
noise 30% |  $\downarrow$ 0.046 |  $\downarrow$ 0.16 |  $\downarrow$ 0.003 |  $\downarrow$ 0.23

Table 4: BBFN Robust Training. Results are averaged over three random runs.

|  Robust Training | Diagnostic | MOSI | MOSEI
---|---|---|---|---
| Corr | F1 | Corr | F1
Self-MM | - |  | 0.794 | 85.61 | 0.759 | 84.62
missing 5% |  $\downarrow$ 0.023 |  $\downarrow$ 1.93 |  $\downarrow$ 0.018 |  $\downarrow$ 1.99
noise 5% |  $\downarrow$ 0.009 |  $\downarrow$ 0.91 |  $\downarrow$ 0.022 |  $\downarrow$ 1.32
missing 10% |  $\downarrow$ 0.046 |  $\downarrow$ 3.57 |  $\downarrow$ 0.040 |  $\downarrow$ 3.51
noise 10% |  $\downarrow$ 0.039 |  $\downarrow$ 2.43 |  $\downarrow$ 0.058 |  $\downarrow$ 2.91
missing 15% |  $\downarrow$ 0.051 |  $\downarrow$ 4.77 |  $\downarrow$ 0.056 |  $\downarrow$ 4.51
noise 15% |  $\downarrow$ 0.050 |  $\downarrow$ 2.71 |  $\downarrow$ 0.069 |  $\downarrow$ 3.65
missing 30% |  $\downarrow$ 0.099 |  $\downarrow$ 11.74 |  $\downarrow$ 0.126 |  $\downarrow$ 9.04
noise 30% |  $\downarrow$ 0.154 |  $\downarrow$ 8.35 |  $\downarrow$ 0.172 |  $\downarrow$ 9.48
5% missing 5% noise |  | 0.798 | 83.97 | 0.789 | 0.837
missing 5% |  $\downarrow$ 0.022 |  $\downarrow$ 1.73 |  $\downarrow$ 0.021 |  $\downarrow$ 1.83
noise 5% |  $\downarrow$ 0.002 |  $\rightarrow$ 0 |  $\downarrow$ 0.002 |  $\downarrow$ 0.05
missing 10% |  $\downarrow$ 0.046 |  $\downarrow$ 3.32 |  $\downarrow$ 0.046 |  $\downarrow$ 3.28
noise 10% |  $\downarrow$ 0.012 |  $\downarrow$ 0.14 |  $\downarrow$ 0.011 |  $\downarrow$ 0.44
missing 15% |  $\downarrow$ 0.050 |  $\downarrow$ 4.63 |  $\downarrow$ 0.053 |  $\downarrow$ 4.60
noise 15% |  $\downarrow$ 0.018 |  $\downarrow$ 0.09 |  $\downarrow$ 0.018 |  $\downarrow$ 0.51
missing 30% |  $\downarrow$ 0.119 |  $\downarrow$ 9.57 |  $\downarrow$ 0.124 |  $\downarrow$ 9.77
noise 30% |  $\downarrow$ 0.046 |  $\downarrow$ 0.33 |  $\downarrow$ 0.043 |  $\downarrow$ 1.04
10% missing 10% noise |  | 0.789 | 83.67 | 0.764 | 0.849
missing 5% |  $\downarrow$ 0.021 |  $\downarrow$ 1.83 |  $\downarrow$ 0.017 |  $\downarrow$ 0.63
noise 5% |  $\downarrow$ 0.002 |  $\downarrow$ 0.05 |  $\downarrow$ 0.005 |  $\downarrow$ 0.03
missing 10% |  $\downarrow$ 0.046 |  $\downarrow$ 3.28 |  $\downarrow$ 0.038 |  $\downarrow$ 1.93
noise 10% |  $\downarrow$ 0.011 |  $\downarrow$ 0.44 |  $\downarrow$ 0.011 |  $\downarrow$ 0.15
missing 15% |  $\downarrow$ 0.053 |  $\downarrow$ 4.60 |  $\downarrow$ 0.057 |  $\downarrow$ 3.16
noise 15% |  $\downarrow$ 0.018 |  $\downarrow$ 0.51 |  $\downarrow$ 0.019 |  $\downarrow$ 0.13
missing 30% |  $\downarrow$ 0.124 |  $\downarrow$ 9.77 |  $\downarrow$ 0.126 |  $\downarrow$ 7.42
noise 30% |  $\downarrow$ 0.043 |  $\downarrow$ 1.04 |  $\downarrow$ 0.056 |  $\downarrow$ 1.01
15% missing 15% noise |  | 0.790 | 84.73 | 0.754 | 84.67
missing 5% |  $\downarrow$ 0.022 |  $\downarrow$ 1.91 |  $\downarrow$ 0.017 |  $\downarrow$ 0.60
noise 5% |  $\downarrow$ 0.003 |  $\rightarrow$ 0 |  $\downarrow$ 0.004 |  $\downarrow$ 0.01
missing 10% |  $\downarrow$ 0.046 |  $\downarrow$ 3.39 |  $\downarrow$ 0.038 |  $\downarrow$ 1.91
noise 10% |  $\downarrow$ 0.010 |  $\downarrow$ 0.44 |  $\downarrow$ 0.011 |  $\downarrow$ 0.14
missing 15% |  $\downarrow$ 0.049 |  $\downarrow$ 4.58 |  $\downarrow$ 0.055 |  $\downarrow$ 3.08
noise 15% |  $\downarrow$ 0.018 |  $\downarrow$ 0.28 |  $\downarrow$ 0.016 |  $\downarrow$ 0.16
missing 30% |  $\downarrow$ 0.120 |  $\downarrow$ 9.66 |  $\downarrow$ 0.122 |  $\downarrow$ 6.86
noise 30% |  $\downarrow$ 0.041 |  $\downarrow$ 0.58 |  $\downarrow$ 0.051 |  $\downarrow$ 1.02

Table 5: Self-MM Robust Training. Results are averaged over three random runs.

|  Robust Training | Diagnostic | MOSI | MOSEI
---|---|---|---|---
| Corr | F1 | Corr | F1
MMIM | - |  | 0.796 | 86.02 | 0.758 | 84.89
missing 5% |  $\downarrow$ 0.028 |  $\downarrow$ 1.34 |  $\downarrow$ 0.016 |  $\downarrow$ 0.93
noise 5% |  $\downarrow$ 0.035 |  $\downarrow$ 1.52 |  $\downarrow$ 0.031 |  $\downarrow$ 1.71
missing 10% |  $\downarrow$ 0.067 |  $\downarrow$ 4.16 |  $\downarrow$ 0.034 |  $\downarrow$ 2.43
noise 10% |  $\downarrow$ 0.058 |  $\downarrow$ 2.10 |  $\downarrow$ 0.070 |  $\downarrow$ 2.64
missing 15% |  $\downarrow$ 0.056 |  $\downarrow$ 2.78 |  $\downarrow$ 0.056 |  $\downarrow$ 4.13
noise 15% |  $\downarrow$ 0.078 |  $\downarrow$ 4.65 |  $\downarrow$ 0.094 |  $\downarrow$ 4.67
missing 30% |  $\downarrow$ 0.117 |  $\downarrow$ 9.37 |  $\downarrow$ 0.122 |  $\downarrow$ 8.15
noise 30% |  $\downarrow$ 0.197 |  $\downarrow$ 9.55 |  $\downarrow$ 0.191 |  $\downarrow$ 9.18
5% missing 5% noise |  | 0.797 | 85.13 | 0.755 | 0.836
missing 5% |  $\downarrow$ 0.057 |  $\downarrow$ 1.19 |  $\downarrow$ 0.016 |  $\downarrow$ 0.62
noise 5% |  $\downarrow$ 0.021 |  $\downarrow$ 1.20 |  $\downarrow$ 0.025 |  $\downarrow$ 1.07
missing 10% |  $\downarrow$ 0.035 |  $\downarrow$ 1.80 |  $\downarrow$ 0.046 |  $\downarrow$ 2.29
noise 10% |  $\downarrow$ 0.075 |  $\downarrow$ 2.25 |  $\downarrow$ 0.056 |  $\downarrow$ 2.43
missing 15% |  $\downarrow$ 0.057 |  $\downarrow$ 4.37 |  $\downarrow$ 0.168 |  $\downarrow$ 2.26
noise 15% |  $\downarrow$ 0.072 |  $\downarrow$ 3.96 |  $\downarrow$ 0.063 |  $\downarrow$ 2.75
missing 30% |  $\downarrow$ 0.117 |  $\downarrow$ 8.24 |  $\downarrow$ 0.120 |  $\downarrow$ 6.52
noise 30% |  $\downarrow$ 18.75 |  $\downarrow$ 9.31 |  $\downarrow$ 0.178 |  $\downarrow$ 7.34
10% missing 10% noise |  | 0.794 | 84.76 | 0.758 | 84.81
missing 5% |  $\downarrow$ 0.020 |  $\downarrow$ 1.50 |  $\downarrow$ 0.016 |  $\downarrow$ 1.07
noise 5% |  $\downarrow$ 0.006 |  $\downarrow$ 0.46 |  $\downarrow$ 0.024 |  $\downarrow$ 1.28
missing 10% |  $\downarrow$ 0.032 |  $\downarrow$ 2.09 |  $\downarrow$ 0.043 |  $\downarrow$ 2.79
noise 10% |  $\downarrow$ 0.060 |  $\downarrow$ 2.57 |  $\downarrow$ 0.045 |  $\downarrow$ 2.58
missing 15% |  $\downarrow$ 0.060 |  $\downarrow$ 3.27 |  $\downarrow$ 0.058 |  $\downarrow$ 3.84
noise 15% |  $\downarrow$ 0.062 |  $\downarrow$ 3.65 |  $\downarrow$ 0.063 |  $\downarrow$ 3.42
missing 30% |  $\downarrow$ 0.110 |  $\downarrow$ 8.40 |  $\downarrow$ 0.119 |  $\downarrow$ 7.50
noise 30% |  $\downarrow$ 0.163 |  $\downarrow$ 9.18 |  $\downarrow$ 0.177 |  $\downarrow$ 8.16
15% missing 15% noise |  | 0.784 | 84.67 | 0.751 | 83.15
missing 5% |  $\downarrow$ 0.014 |  $\downarrow$ 1.79 |  $\downarrow$ 0.020 |  $\downarrow$ 0.89
noise 5% |  $\downarrow$ 0.028 |  $\downarrow$ 1.04 |  $\downarrow$ 0.016 |  $\downarrow$ 0.69
missing 10% |  $\downarrow$ 0.038 |  $\downarrow$ 2.18 |  $\downarrow$ 0.030 |  $\downarrow$ 2.24
noise 10% |  $\downarrow$ 0.050 |  $\downarrow$ 1.30 |  $\downarrow$ 0.028 |  $\downarrow$ 0.62
missing 15% |  $\downarrow$ 0.061 |  $\downarrow$ 4.91 |  $\downarrow$ 0.059 |  $\downarrow$ 2.77
noise 15% |  $\downarrow$ 0.071 |  $\downarrow$ 4.08 |  $\downarrow$ 0.040 |  $\downarrow$ 1.58
missing 30% |  $\downarrow$ 0.146 |  $\downarrow$ 10.48 |  $\downarrow$ 0.115 |  $\downarrow$ 6.62
noise 30% |  $\downarrow$ 0.180 |  $\downarrow$ 8.88 |  $\downarrow$ 0.096 |  $\downarrow$ 4.41

Table 6: MMIM Robust Training. Results are averaged over three random runs.

|  Robust Training | Diagnostic | MOSI | MOSEI
---|---|---|---|---
| Corr | F1 | Corr | F1
MulT | - |  | 0.747 | 82.25 | 0.738 | 83.37
missing 5% |  $\downarrow$ 0.032 |  $\downarrow$ 3.76 |  $\downarrow$ 0.019 |  $\downarrow$ 0.84
noise 5% |  $\downarrow$ 0.053 |  $\downarrow$ 1.67 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.046 |  $\downarrow$ 3.92 |  $\downarrow$ 0.031 |  $\downarrow$ 1.85
noise 10% |  $\downarrow$ 0.069 |  $\downarrow$ 3.03 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 15% |  $\downarrow$ 0.052 |  $\downarrow$ 4.84 |  $\downarrow$ 0.051 |  $\downarrow$ 3.29
noise 15% |  $\downarrow$ 0.172 |  $\downarrow$ 5.00 |  $\downarrow$ 0.152 |  $\downarrow$ 4.50
missing 30% |  $\downarrow$ 0.113 |  $\downarrow$ 12.17 |  $\downarrow$ 0.109 |  $\downarrow$ 6.60
noise 30% |  $\downarrow$ 0.295 |  $\downarrow$ 8.99 |  $\downarrow$ 0.263 |  $\downarrow$ 7.95
5% missing 5% noise |  | 0.748 | 81.90 | 0.748 | 84.59
missing 5% |  $\downarrow$ 0.031 |  $\downarrow$ 1.50 |  $\downarrow$ 0.018 |  $\downarrow$ 0.93
noise 5% |  $\downarrow$ 0.021 |  $\downarrow$ 1.67 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.043 |  $\downarrow$ 2.43 |  $\downarrow$ 0.030 |  $\downarrow$ 1.99
noise 10% |  $\downarrow$ 0.021 |  $\downarrow$ 1.96 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 15% |  $\downarrow$ 0.050 |  $\downarrow$ 3.36 |  $\downarrow$ 0.050 |  $\downarrow$ 3.51
noise 15% |  $\downarrow$ 0.067 |  $\downarrow$ 3.30 |  $\downarrow$ 0.173 |  $\downarrow$ 3.97
missing 30% |  $\downarrow$ 0.077 |  $\downarrow$ 5.12 |  $\downarrow$ 0.074 |  $\downarrow$ 4.90
noise 30% |  $\downarrow$ 0.094 |  $\downarrow$ 5.17 |  $\downarrow$ 0.198 |  $\downarrow$ 5.50
10% missing 10% noise |  | 0.741 | 81.31 | 0.746 | 84.13
missing 5% |  $\downarrow$ 0.034 |  $\downarrow$ 2.57 |  $\downarrow$ 0.019 |  $\downarrow$ 0.96
noise 5% |  $\downarrow$ 0.012 |  $\downarrow$ 0.09 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.046 |  $\downarrow$ 3.06 |  $\downarrow$ 0.030 |  $\downarrow$ 0.019
noise 10% |  $\downarrow$ 0.031 |  $\downarrow$ 2.42 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 15% |  $\downarrow$ 0.053 |  $\downarrow$ 5.52 |  $\downarrow$ 0.048 |  $\downarrow$ 3.19
noise 15% |  $\downarrow$ 0.045 |  $\downarrow$ 4.08 |  $\downarrow$ 0.153 |  $\downarrow$ 4.08
missing 30% |  $\downarrow$ 0.077 |  $\downarrow$ 6.57 |  $\downarrow$ 0.072 |  $\downarrow$ 4.62
noise 30% |  $\downarrow$ 0.056 |  $\downarrow$ 4.57 |  $\downarrow$ 0.202 |  $\downarrow$ 5.58
15% missing 15% noise |  | 0.744 | 82.21 | 0.745 | 83.95
missing 5% |  $\downarrow$ 0.034 |  $\downarrow$ 2.56 |  $\downarrow$ 0.018 |  $\downarrow$ 0.86
noise 5% |  $\downarrow$ 0.023 |  $\downarrow$ 1.52 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 10% |  $\downarrow$ 0.047 |  $\downarrow$ 3.93 |  $\downarrow$ 0.032 |  $\downarrow$ 1.86
noise 10% |  $\downarrow$ 0.039 |  $\downarrow$ 3.62 |  $\rightarrow$ 0 |  $\rightarrow$ 0
missing 15% |  $\downarrow$ 0.052 |  $\downarrow$ 5.15 |  $\downarrow$ 0.051 |  $\downarrow$ 3.27
noise 15% |  $\downarrow$ 0.046 |  $\downarrow$ 3.62 |  $\rightarrow$ 0 |  $\downarrow$ 0.08
missing 30% |  $\downarrow$ 0.117 |  $\downarrow$ 9.63 |  $\downarrow$ 0.113 |  $\downarrow$ 7.03
noise 30% |  $\downarrow$ 0.068 |  $\downarrow$ 6.13 |  $\downarrow$ 0.001 |  $\uparrow$ 0.02

Table 7: MulT Robust Training. Results are averaged over three random runs.