# Does a Technique for Building Multimodal Representation Matter? –
Comparative Analysis

Maciej Pawłowski Faculty of Mathematics and Information Science, Warsaw
University of Technology, Warsaw, Poland<EMAIL_ADDRESS><EMAIL_ADDRESS>Anna Wróblewska Faculty of Mathematics and
Information Science, Warsaw University of Technology, Warsaw, Poland
<EMAIL_ADDRESS><EMAIL_ADDRESS>Sylwia Sysko-Romańczuk
Faculty of Management, Warsaw University of Technology, Warsaw, Poland
<EMAIL_ADDRESS>
###### Abstract

Creating a meaningful representation by fusing single modalities (e.g., text,
images, or audio) is the core concept of multimodal learning. Although several
techniques for building multimodal representations have been proven
successful, they have not been compared yet. Therefore it has been ambiguous
which technique can be expected to yield the best results in a given scenario
and what factors should be considered while choosing such a technique. This
paper explores the most common techniques for building multimodal data
representations – the late fusion, the early fusion, and the sketch, and
compares them in classification tasks. Experiments are conducted on three
datasets: Amazon Reviews, MovieLens25M, and MovieLens1M datasets. In general,
our results confirm that multimodal representations are able to boost the
performance of unimodal models from $0.919$ to $0.969$ of accuracy on Amazon
Reviews and $0.907$ to $0.918$ of AUC on MovieLens25M. However, experiments on
both MovieLens datasets indicate the importance of the meaningful input data
to the given task. In this article, we show that the choice of the technique
for building multimodal representation is crucial to obtain the highest
possible model’s performance, that comes with the proper modalities
combination. Such choice relies on: the influence that each modality has on
the analyzed machine learning (ML) problem; the type of the ML task; the
memory constraints while training and predicting phase.

_Keywords_ Multimodal Representation, Multimodal Learning, Data Fusion,
Comparative Analysis

##  1 Introduction

Multimodal learning involves working with data that contains multiple
modalities such as text, images, audio, numerical or behavioral data. The
interest in multimodal learning started in the 1980s and has gained popularity
since then. In one of the first works on the subject (Yuhas et al. (1989)),
the authors demonstrated that acoustic and visual data could be successfully
combined in the speech recognition system. They showed that integrating these
modalities outperforms the audio data model.

Research shows that modalities are complementary and multimodal learning
should be explored comprehensively, as it can significantly enhance models’
performance in any machine learning task (Baltrusaitis et al. (2019); Cao et
al. (2020); Gao et al. (2020)). So far, a multimodal approach has been proven
effective in numerous fields, i.e., in information retrieval (Cao et al.
(2020)), medicine (El-Sappagh et al. (2020)) and human behavior analysis
(Jaiswal et al. (2020)).

The central concept of multimodal learning is to develop a data fusion
technique that can be applied to numerous machine learning tasks with a few
chosen or all kinds of modalities (universal technique). According to Gao et
al. (2020), such a universal approach has not been established so far. In
their work, the authors noticed that all current state-of-the-art data fusion
models are suffering from some issues. Either they are task-specific or too
complicated, in the sense that they lack interpretability and flexibility.

We believe that in order to develop such a universal technique, it is
necessary to understand the differences between the existing methods. To the
best of our knowledge, we are the first to investigate the behaviors of
multimodal data fusion techniques in several scenarios. In this paper, we show
that the effective multimodal representation depends on the type of technique
used to build this representation and other criteria/requirements. We
distinguish three main criteria that define this dependency: the influence
that each modality has on the analyzed machine learning (ML) problem; the type
of the ML task; the memory constraints while training and predicting phase.

This work contributes to examining the most common techniques of building
multimodal data representations: the late fusion, the early fusion, and the
sketch. In the first approach – the late fusion – all the modalities are
learned independently and are combined right before the model makes a
decision. The second technique – the early fusion – first combines all
modalities and then the model is trained. The last one – the sketch – is
similar to the early fusion; however, modalities are transformed into a common
space instead of being concatenated. All of these techniques have been proven
effective. In this paper, we perform this comparison in classification tasks.
Finally, we define criteria and recommendations on how to choose the
multimodal technique. The criteria comprise: the influence that each modality
has on the analyzed machine learning (ML) problem, the ML task type, the
memory constraints while the training and predicting phase. In our work we use
three datasets – Amazon Reviews (He and McAuley (2016)), MovieLens25M and
MovieLens1M (Harper and Konstan (2016)). They encapsulate three modalities –
textual, visual, and graph data; each can bring unique information to the ML
model.

Thus, in the following, in Section 2 we state the problems of multimodality,
and present employed datasets (Section 3). Finally, we experiment with the
chosen multimodal approaches (Section 4 and 5), and provide a few conclusions
in Section 6.

##  2 Related Work

###  2.1 Types of Modalities

This section introduces the types of modalities that can be encountered while
working on any machine learning problem. Figure 1 depicts the full spectrum of
existing modalities. Accordingly, to Varshney (2022), we identify 4 four main
groups of modalities:

  * •

tabular data – observations are stored as rows, and their features as columns;

  * •

graphs – observations are vertices, and their features are in the form of
edges between individual vertices;

  * •

signals – observations are files of appropriate extension (images - .jpeg,
audio - .wav, etc.), their features are the numerical data provided within
files;

  * •

sequences – observations are in the form of characters/words/documents, where
the type of characters/words corresponds to features.

Figure 1: Types of modalities that can be encountered. The division comes from
Varshney (2022).

Our study includes modalities coming from each of the identified groups:
labels are an example of tabular data, reviews, titles, descriptions represent
textual data (sequences), images of products, and movie posters are examples
of visual data, and relations between movies (movies seen by one user) are
graphs. Table 1 introduces some other examples of works on various
combinations of modalities. Additionally, Zhang et al. (2021) concentrate on
the image segmentation task to learn the optimal joint representation from
rich and complementary features of the same scene based on images retrieved
from different sensors. All these studies show that there are no restrictions
on modalities that can be studied together to solve a specific problem.
Moreover, they prove that the combination of modalities boosts performance,
that is, the multimodal model achieves better results than unimodal models.

Furthermore, these works encapsulate all multimodal fusion techniques, that we
examine in our research, the early fusion, the late fusion, and the sketch.
All of them are proven effective but have not been compared until our work.

Modalities |  Article |  Overview
---|---|---
images; time series; tabular |  El-Sappagh et al. (2020) |  Prediction of Alzheimer’s disease based on magnetic resonance imaging, positron emission tomography (images) that are performed multiple time on one patient within specified periods of time (time series). Patient demographics, and genetic data are also taken into account (tabular).
audio; video; event streams |  Jaiswal et al. (2020) |  Behavioural analysis - emotion, and stress prediction. Analysed data consist of 45-minutes recordings of students during final exam period. They are recorded with the use of cameras (video), thermal physiological measurements of heart and breathing rates (event streams), lapel microphones (audio).
text; images |  Singh et al. (2019) |  Question answering based on images containing some textual data.
images; text; graphs |  Rychalska et al. (2020) Laenen and Moens (2020) Salah et al. (2020) |  Outfit / Movie recommender systems. Movies are recommended based on plot (text), poster (image), liked and disliked movies, cast (graphs). Outfits are chosen based on product features in images and text descriptions.
Table 1: Examples of multimodal tasks

###  2.2 Multimodal Representation

Learning to represent is an unsupervised task, and there is no single manner
to describe what a good representation is. However, several works have already
identified the main features demanded while deriving any numerical
representation of a given modality.

The problem of unimodal representation has been already solved with modality-
dedicated models, such as BERT (Devlin et al. (2019)) for textual data, ResNet
(He et al. (2016)) for images, etc. However, there has not been established
(Gao et al. (2020)) a universal method that could be applied to any machine
learning task when it comes to multimodal data.

Bengio et al. (2013) characterize several features that an appropriate vector
representation should possess, including:

  * •

smoothness – transformation should preserve objects similarities,
mathematically $x\approx y\Rightarrow f(x)\approx f(y)$. For instance, the
words "book" and "read" are expected to have similar embeddings;

  * •

manifolds – probability mass is concentrated within regions of lower
dimensionality than the original space, e.g., we can expect the words
"Poland", "USA", "France" to have embeddings within a certain region, and the
words "jump", "ride," "run" in another, distinct region;

  * •

natural clustering – categorical values could be assigned to observations
within the same manifold, e.g., a region with the words "Poland", "USA",
"France" can be described as "countries";

  * •

sparsity – given an observation, only a subset of its numerical representation
features should be relevant. Otherwise, we end up with complicated embeddings
whose highly correlated features may lead to numerous ambiguities.

For multimodal representation, Srivastava and Salakhutdinov (2014) identify
more factors that should be taken into account: (1) similarity between
individual modalities should be preserved in their joint representation; (2)
robustness to the absence of some modalities – it should be always possible to
create multimodal embedding.

###  2.3 Multimodal Data Fusion

Multimodal data fusion is an approach for combining single modalities to
derive multimodal representation. A few issues should be taken into account
(Gao et al. (2020)) when it comes to fusing several modalities:

  * •

intermodality – the combination of several modalities, which leads to better
and more robust model predictions (Baltrusaitis et al. (2019))

  * •

cross-modality – this aspect assumes inseparable interactions between
modalities. No reasonable conclusions can be drawn from data unless all
modalities are joined (Frank et al. (2021))

  * •

missing data – for some cases, particular modalities might not be available.
An ideal multimodal data fusion algorithm is robust to missing modalities and
uses others to compensate for the information loss, widespread in recommender
systems.

Classically, the existing multimodal representation techniques are divided in
two categories (Gallo et al. (2017), Baltrusaitis et al. (2019)): early
(feature) and late (decision) fusion. In the early feature approach, all
modalities are combined. This is usually achieved by concatenating their
vector representations at an initial stage, and then one model is trained
(Kiela et al. (2018)). In the case of late fusion, several independent models
concerning each modality are trained. Then their outputs are connected. The
connection can be made arbitrarily. One can average the outputs, pick the most
frequent one (in classification tasks) or concatenate them and build a model
on top of that to obtain a final output (Kiela et al. (2018)). Neither of
these data fusion approaches can be described as the best one (Gallo et al.
(2017)); both have been proven to yield promising results in various
scenarios.

####  2.3.1 Deep Learning Models

The most popular multimodal fusion techniques are based on deep learning
solutions. Gao et al. (2020) describe such architecture ideas, along with
cases of their most representative. Four prominent approaches are: Deep Belief
Nets, Stacked Autoencoders, Convolution Networks, and Recurrent Networks.
However, despite their promising results in the field of multimodal data
fusion, deep learning models suffer from two main issues (Gao et al. (2020)).
Firstly, deep learning models contain an enormous amount of free weights,
especially parameters associated with a modality that brings little
information. That results in high resource requirements – an undesirable
feature in a production scenario. Secondly, multimodal data usually comes from
very dynamic environments. Therefore, there is a need for a flexible model
that can easily adapt to all data changes.

Enormous computational requirements and low flexibility suggest the
possibility to explore other ideas applied to any task, despite its type of
modalities. Furthermore, these ideas can combine with deep learning techniques
and existing multimodal models to obtain state-of-the-art solutions, which
would be applicable in every field, and robust to all data imperfections
(missing modalities, data distribution changes over time in a production case,
etc.). One of the possible solutions is the usage of hashing methods. The
following section discusses the strengths and weaknesses of such algorithms.

####  2.3.2 Hashing Ideas

Another promising approach in multimodal data fusion is associated with
hashing models. They work by identifying manifolds in origin space and then
transforming data to lower-dimensional spaces while preserving observations’
similarities. Such algorithms are capable of constructing multimodal
representation on-the-fly and have been proven effective in information
retrieval problems (Cao et al. (2020)), recommendation systems (Dabrowski et
al. (2020)), and object detection cases. (Ghayoumi et al. (2018)). The main
advantages (Cao et al. (2020)) of hashing methods are that they: (1) are cost-
effective in terms of memory usage, (2) detect and work within manifolds, (3)
preserve semantic similarities between points, (4) are usually data-
independent, (5) are suitable for production cases as they are robust to any
data changes.

Unfortunately, hashing methods struggle with one issue. The mapping of high
dimensional data to much simpler representations can result in the loss of
certain information about specific observations (Cao et al. (2020)).
Therefore, it has to be verified if hashing ideas can be applied to other
fields, apart from similarity search tasks. Perhaps their ability to combine
multiple modalities while maintaining low costs and robustness to data changes
recompenses the lost information.

####  2.3.3 Sketch Representation

The sketch representation has already been proven effective if fed with
visual, behavioral, and textual data for the recommendation and similarity
search tasks (Dabrowski et al. (2020)). The idea of this representation comes
from combining two algorithms: Locality Sensitive Hashing and Count-Min
Sketch. All modalities are transformed into mutual space with the use of hash
functions. Generally, a sketch is a one-hot sparse matrix containing all
combined modalities. Hash functions make the representation modality-
independent, robust to missing modalities, and easily interpreted.
Furthermore, modalities can be added to the sketch in an on-the-fly manner,
which is extremely important in a production scenario.

In our research, we slightly modify this sketch representation to the
binarized form. Instead of representing an observation with the subspace id,
it can be represented as a set of binary features. Then $0$ and $1$ represent
where the point lies concerning a single hyperplane. Such a sketch should
preserve more information about a single observation.

Figure 2: The idea of binarizing the sketch. Instead of representing an
observation with the subspace id, it can be represented as a set of binary
features. Then $0$ and $1$ represent where the point lies concerning a single
hyperplane. Such a sketch consumes much less memory and perhaps preserves more
information about a single observation.

###  2.4 Multimodal Model Evaluation

The problem of evaluating the multimodal data fusion algorithm is not
straightforward, and there does not exist any universal metric that would
measure the aspect of captured inter and cross-modalities (Frank et al.
(2021)). However, we can assess whether learning from multiple data types at
once enhances task performance.

The most popular way of verifying the quality of the multimodal fusion model
(Gallo et al. (2017)) is to compare its performance scores (precision, AUC,
etc.) to ones achieved by models considering single modalities. With such an
approach, we can state whether and to what extent combining modalities brings
new information. We also aim to preserve all similarities between
observations, i.e., similar observations should be comparable in their
multimodal representations. Therefore several works (Kiela et al. (2018),
Dabrowski et al. (2020)), compare their multimodal models to NN algorithms,
which serve as a good baseline. Lastly, multimodal models should be tested on
the time they need to adjust additional modalities. In certain cases (Kiela et
al. (2018)), adding new modalities slightly improves the results, while the
training time increases dramatically. As a result, the model, despite its
excellent performance, might be unfeasible in the production scenario.
Therefore not only should we focus on the scores the model achieves, but also
take into account its flexibility and simplicity.

In our work, we analyze three techniques for building the multimodal
representation – the late fusion, the early fusion, and the sketch. The main
ideas of all of them are described in this section. The next part introduces
datasets that are used for analyses.

##  3 Datasets

In order to evaluate different multimodal representations, we compare them in
classification problems, so we can easily assess the amount of information
that each modality brings. Furthermore, we analyze scenarios, where the
multimodality is expected to benefit the model, i.e. model based on several
modalities should obtain better performance than the unimodal model.

Table 2 presents the overall characteristics of three used datasets (Amazon
Reviews, MovieLens25M, and MovieLens1M) and the employed machine learning
tasks. In our experiments, Amazon Reviews and Movielens25M are split at
$0.6/0.2/0.2$, the training, validation, and test set. With such division, the
results on test sets are representative (12,000 samples), while maintaining
enough observations for training. The split of $0.8/0.2$ – the training and
test set – is used in MovieLens1M. Instead of one predefined validation set,
MovieLens1M uses the k-fold Cross-Validation (on the training set). We opt for
Cross-Validation cause this dataset is much smaller than the other two.

| Dataset
---
name
Size | Modalities | Task
| Amazon
---
Reviews
60,000 |  | \- textual (product description
---
and title)
\- visual (product image)
| product classification
---
(multiclass)
| MovieLens
---
25M
60,763 |  | \- textual (movie plot)
---
\- visual (movie poster)
\- graph (movies seen by one user)
| genres classification
---
(multilabel)
| MovieLens
---
1M
6,040 |  | \- textual (movie plot)
---
\- visual (movie poster)
| gender classification
---
(binary)
Table 2: Dataset characteristics

Amazon Reviews dataset consists of products from Amazon. The data used in
experiments are only a small subset of the original
dataset111jmcauley.ucsd.edu/data/amazon/ offers – "Clothing, Shoes, and
Jewelry" category. Intuitively, products within this category should be
divergent enough. Furthermore, products within the fashion field can be
expected to contain meaningful descriptions, titles, and, most importantly,
images. The dataset extracted from Amazon Reviews is characterized by missing
modalities. Around $25\%$ of descriptions are lost, and $3\%$ of titles are
empty; however, all the records contain images.

MovieLens25M dataset is the largest of the MovieLens
family.222https://grouplens.org/datasets/movielens/ It consists of 25 million
ratings applied to $62\,000$ movies by $162\,000$ users. Each movie is
associated with genres and with one IMDb identifier. The metadata are
collected based on such an identifier – movie’s poster and plot, the visual
and textual embedding, respectively. Some movies are subscribed to incorrect
IMDb identifiers and therefore removed. After preprocessing, the dataset in
our experiment consists of $60\,763$ unique movies. The existing modalities
are: movie poster (image), movie plot (text), also viewed (graph).

There are some missing modalities: every movie has a plot, $63$ movies lack
the poster, and $3\,231$ movies are not rated by users in MovieLens25M.
Missing modalities are proportional to class sizes, meaning the most
considerable number of missing modalities is in the Drama class. Here, classes
are highly imbalanced. Around 40% of all observations are dramas, 27% are
comedies, whereas film-noir and IMAX movies are associated with less than 1%
of observations. There is also a high number of movies that are not subscribed
to any specific category – the (no genres listed) class.

MovieLens1M is a smaller version of the previously described dataset and
consists of $6\,040$ users that have rated approximately $4\,000$ movies.
However, apart from users’ ratings, their demographic data are available. It
allows for the investigation of dependencies between watched content and some
user characteristics such as gender, age, or occupation. After preprocessing
for our experiment, only two modalities are utilized: textual (the plot) and
visual (the poster). Again the data are downloaded based on IMDb identifiers
associated with movies. There are no missing modalities. Classes are
imbalanced – around $71.7\%$ of users are males. This imbalance is taken into
account during the evaluation, by using the MCC metric.

##  4 Research Design and Experiments

We carry out three experiments with three classification tasks: multiclass,
multilabel, binary (see Table 2). The type of task is determined by the
characteristics of the dataset.

###  4.1 Multiclass Classification – Amazon Reviews Dataset

Analyses on the Amazon Reviews Dataset consider the multiclass classification
task based on two modalities – textual and visual. Classes are evenly
distributed, and therefore the accuracy metric is used. The primary purpose of
experiments on this dataset is to compare multimodal fusion techniques based
on neural networks:

  * •

the late fusion, where modalities are treated independently. Here models’
architectures are based on Wirojwatanakul and Wangperawong (2019).

  * •

early fusion models, where embeddings of each modality are concatenated at the
input level.

  * •

the sketch, including the classical and binarized version.

Their overall architectures are visible in Figure 3. All of these approaches
use Adam as the optimizer and categorical cross-entropy loss function.
Everywhere the batch size is set to $32$. All models are trained for 10
epochs, except for the binarized sketch models which are trained for 20
epochs. The learning rate differs per approach and equals
$10^{-4},10^{-3},10^{-5},10^{-4}$ in the late fusion, the early fusion, the
classical, and the binarized sketch approach, respectively.

Figure 3: Here we present the architectures that are used in the Amazon
Reviews Dataset. They are depicted in their trimodal forms. Bimodal and
unimodal architectures look the same. The only difference is that unimodal
models do not have a model on top of their outputs in the late fusion
approach. Above the output means the probabilities of observations belonging
to respective classes.

####  4.1.1 Late Fusion Models

While deploying ideas from Wirojwatanakul and Wangperawong (2019), slight
modifications are required to transfer from multilabel to multiclass
classification task: a different loss function or the number of used neurons
within layers. Both description and title are represented with GloVe. Image
embeddings are created with ResNet50. All multimodal models are identical. The
outputs from unimodal models are fed into a dense layer, with 20 units and
ReLU as the activation function. Finally, a 12-unit dense layer for product
classification is added. Therefore, the only difference between bimodal and
trimodal models is the input size, equal to 24 and 32.

####  4.1.2 Early Fusion

Here each modality is firstly transformed into a numerical vector with BERT
(bert-base-uncased) for both titles and descriptions, resulting in
$768$-dimensional vectors. For image embeddings, ResNet50 is used. The last
layer is removed, which results in $2048$-dimensional vectors. In this
approach, models have the same architecture, whether they are uni or
multimodal. Therefore the only difference is that in the multimodal case,
appropriate embeddings are concatenated, resulting in $1\,536$ (textual +
textual), $2\,816$ (textual + image), $3\,584$ (trimodal) input embedding
sizes. The model architecture consists of three blocks: dense-dropout layers,
with 64 units, ReLU as the activation function, and the dropout layer – rate
is set to $0.1$. Subsequently, one dense layer is added, also with 64 units
and ReLU. However, in this case, the dropout is not applied. Finally, there is
the 12-unit dense layer with softmax, responsible for classification.

####  4.1.3 Sketch Representation

Similarly to the early fusion approach, BERT is used for the textual and
ResNet50 for the visual data. Firstly, embeddings are transformed into
sketches. The sketch’s depth equals $128$, and the width equals $512$. Then
sketches are flattened and concatenated in multimodal cases. The architecture
is universal, despite the number of modalities. The model starts with two
identical blocks consisting of a dense layer, a dropout layer, and batch
normalization. The dense layer in the first block consists of $1\,024$ neurons
and the second of $512$ neurons. After the second block, there is a
$128$-dimensional dense layer followed by a dropout layer. All of these dense
layers use ReLU, and the dropout rate is set to $0.2$. Finally, there is a
$12$-dimensional dense layer with softmax, responsible for classification. To
provide a fair comparison with the classical sketch, the binarized version
uses the same architecture.

###  4.2 Multilabel Classification – MovieLens25M

Movielens25M dataset considers multilabel genre classification; its classes
are highly imbalanced and correlated. Therefore, threshold-independent metrics
are used for evaluation: Area Under Curve (AUC) and mean Average Precision
(mAP). Such an approach makes analyzed solutions independent from class
threshold selection at testing time. Furthermore, to take the class imbalance
into account, micro versions of these metrics are used: micro-AUC and micro-
mAP.

Here only the sketch and the early fusion representations are considered.
Figure 4 illustrates both of the approaches.

Figure 4: Architectures used in MovieLens25M dataset.

Movielens25M encapsulates three kinds of modalities: graph, text, and images.
The first one is encoded with Cleora (Rychalska et al. (2021)). Cleora’s
hyperparameters are set as follows: the embedding size - $1\,024$, the number
of iterations - 1. Such parameters are optimal in terms of preserving strong
similarities between entities. Text data are represented with BERT (bert-base-
cased) and visual data with ResNet50. Both sketch’s hyperparameters (depth,
width) are set to $128$.

Models consist of four dense layers. The first three use ReLU and have
$1\,024$, $512$, $128$ neurons, respectively. The last 20-dimensional layer is
used for multilabel classification and therefore uses sigmoid. Models use
Binary Cross-Entropy loss function, the Adam optimizer with a learning rate
equal to $1^{-5}$, and its other parameters are set as default. Each model is
trained for 20 epochs, apart from multimodal early fusion models and unimodal
graph models that are trained for $30$ epochs. The batch size is set to 64.
Every model is trained $5$ times using train and validation data and evaluated
on the test data to obtain robust results.

###  4.3 Binary Classification – MovieLens1M

The task on MovieLens1M is a binary classification – predicting people’s
gender based on their behavioral data (here, movies they watched).

Here, in the sketch representation, every movie is represented with the sketch
and the user as a sum of such sketches, see Figure 5. However, unlike the
recommender scenario, the output is a number between 0 and 1, representing the
probability that the user is a male or a female.

Figure 5: The architecture of the gender classification. The user is
represented as a sum of sketches. Then the width-wise L2 normalization is
performed. Finally the sketch is flattened and fed into the classifier.

Every user has rated (watched) at least 20 movies. Movies are represented with
the plot and the poster, the same way as in MovieLens25M. Then, users are
represented with the content they watch. Once a user sketch is created, it is
flattened and fed into Logistic Regression with regularization parameter $C$
set to $0.1$. The sketch’s width equals 512, while the depth is adjusted to
the modality type – it equals 210 for the movie plot, and 128 for the movie
poster. In the bimodal model, the flattened sketches are concatenated. Because
of the class imbalance, the Matthew Correlation Coefficient (MCC) is used for
the evaluation. Movie plots are embedded with BERT (bert-base-cased), and
posters are embedded with ResNet50.

##  5 Results

Tables 3, 4, and 5 shows our results for different tasks and datasets. We can
see that only in one dataset – Amazon Reviews – do all the used modalities
improve the final results. Other tests on MovieLens datasets show consequently
that posters reduce the results, and intuitively we can imagine that the
posters do not have much substantial information about movies.

Modality | Late fusion | Early fusion | Sketch | Sketch binarized
---|---|---|---|---
title | 0.919$\ \pm\ $0.002 | 0.857$\ \pm\ $0.002 | 0.780$\ \pm\ $0.002 | 0.822$\ \pm\ $0.004
description | 0.706$\ \pm\ $0.003 | 0.662$\ \pm\ $0.001 | 0.613$\ \pm\ $0.003 | 0.636$\ \pm\ $0.004
image | 0.836$\ \pm\ $0.005 | 0.851$\ \pm\ $0.002 | 0.770$\ \pm\ $0.002 | 0.812$\ \pm\ $0.003
title & description | 0.962$\ \pm\ $0.001 | 0.910$\ \pm\ $0.002 | 0.851$\ \pm\ $0.002 | 0.868$\ \pm\ $0.006
image & description | 0.927$\ \pm\ $0.001 | 0.898$\ \pm\ $0.003 | 0.847$\ \pm\ $0.002 | 0.859$\ \pm\ $0.003
image & title | 0.962$\ \pm\ $0.001 | 0.930 $\ \pm\ $0.002 | 0.885$\ \pm\ $0.002 | 0.903$\ \pm\ $0.004
image & title & description | 0.969$\ \pm\ $0.001 | 0.940$\ \pm\ $0.002 | 0.912$\ \pm\ $0.002 | 0.914$\ \pm\ $0.002
Table 3: Amazon Reviews Dataset – the accuracy of each approach, for each modality combinations. Results are averaged over 10 runs. | Early fusion approach | Sketch approach
---|---|---
Modality | AUC | mAP | AUC | mAP
text | 0.907 $\pm$ 0.001 | 0.549 $\pm$ 0.001 | 0.877 $\pm$ 0.003 | 0.465 $\pm$ 0.003
image | 0.815 $\pm$ 0.002 | 0.350 $\pm$ 0.001 | 0.778 $\pm$ 0.003 | 0.297 $\pm$ 0.003
graph | 0.824 $\pm$ 0.001 | 0.382 $\pm$ 0.002 | 0.797 $\pm$ 0.003 | 0.323 $\pm$ 0.003
text & graph | 0.918 $\pm$ 0.001 | 0.592 $\pm$ 0.002 | 0.883 $\pm$ 0.002 | 0.484 $\pm$ 0.002
text & graph & images | 0.915 $\pm$ 0.001 | 0.583 $\pm$ 0.002 | 0.880 $\pm$ 0.002 | 0.482 $\pm$ 0.003
Table 4: Results of the early fusion and the sketch on MovieLens25M. Results
are averaged over 5 runs.

Since the impact that each modality has on the model’s performance is not so
clear on Amazon Reviews Datasets, we decided to explore them thoroughly with
respect to each category, see Figure 6. The analyzed results come from the
late fusion approach, which achieved the best results.

The bimodal and trimodal models outperform unimodal models in every category
(accuracy metric). The most significant benefit from exploiting multiple
modalities can be noticed within classes, where unimodal models yield close
results, for instance, the Wallet class. On the other hand, categories, where
one of the unimodal approaches is undeniably better, do not gain much from
multimodality – the Watch class.

Surprisingly, despite the absence of around 25% of the data concerning the
Sweater category, the description model significantly outperforms the image
model within this class. This example shows that each modality, even the one
with missing data, can yield unique information and be helpful until proven
otherwise.

In most cases, the trimodal model does not significantly outperform the
bimodal, based on title and image. Furthermore, training the trimodal model
takes twice the time the bimodal model needs. In the real-world scenario, the
trimodal model could be too expensive for the slight improvement it provides.
Then, only titles and images could be considered valuable.

Figure 6: The late fusion models results with respect to categories.

In the case of gender classification, the highest MCC – $0.543$ – indicates
that the proposed architecture might be successfully used to solve demographic
classification problems. It could be a powerful tool, for instance, for
e-commerce or user-generated platforms. By modeling users’ characteristics,
companies cold personalize their offers to maximize their profits. On the
other hand, users will benefit from personalized content. However, more
experiments on larger datasets should be performed to confirm the usefulness
in the production scenario.

Modality | Matthew Corellation
---|---
Movie poster | 0.518
Movie plot | 0.543
Movie plot & poster | 0.532
Table 5: Results of the gender classification on test set – Logistic
Regression.

To the best of the authors’ knowledge, our work, for the first time,
classifies users based on the content of their interests (see experiment on
MovieLens1M for gender classification). So far, the existing approaches (De
Cnudde et al. (2020)) have considered only the behavioral data, which in the
case of MovieLens1M are present as ratings that users gave to movies. Our
experiment shows that it is possible to model user characteristics with the
content in which the users are interested. Though only simple classifiers were
tested, it was proven that our approach is reasonable.

All the tests show that combining several modalities boosts model performance;
however, the data must be meaningful for the given task. If possible,
consideration of additional modalities is suggested, as they may provide new
insight into the analyzed data. Our research concludes that multiple data
modalities should be utilized cautiously, as some might yield little
information (here, movie posters are redundant for genres and gender
classification).

The main conclusions that can be drawn from tests are summarized in Table 6.
These conclusions are made upon the following criteria: the influence that
each modality has on the analyzed machine learning (ML) problem; the type of
the ML task; the memory constraints while training and predicting phase.

Technique |  Notes and recommendations:
---|---
Late fusion |  • use when one modality is dominant (its unimodal model gives significantly better results than others)
• use when every unimodal model achieves high performance
• preferred in classification tasks, in which probabilities from unimodal
models can be easily combined, such as in Wirojwatanakul and Wangperawong
(2019)
Early fusion |  • use when modalities are dependent, i.e. no reasonable conclusions can be made unless modalities are studied together (e.g. questions to pictures Singh et al. (2019))
• use when all unimodal models yield similar results
• easy to use pre-trained models; the model input is their early fusion
Sketch |  • memory efficient (sketches can be very short vectors, can be stored and used efficiently, with low memory usage)
• suggested for information filtering problems, such as recommender systems
(Dabrowski et al. (2020))
• easy to use pre-trained models as inputs to sketching technique
Table 6: The recommendations are constructed with respect to three criteria:
the influence that each modality has on the analysed machine learning (ML)
problem; the type of the ML task; the memory constraints while training and
predicting phase

Tests conducted on Amazon Reviews prove that the best one is the late fusion
in nearly all scenarios. Here, the reason is probably the type of textual
data. Many descriptions and titles contain words that are sufficient on their
own for a good prediction. Therefore in situations where one of the modalities
is dominant, we believe that the best approach would be the late fusion. The
model efficiently exploits the information from the most informative modality
and can expand it using additional modalities.

On the other hand, the early fusion approach can reveal information from the
combination of the modalities (interaction between modalities), which can be
interdependent. Therefore in scenarios, where several modalities affect the
model equally, their interactions might reveal hidden information, and hence
the early fusion should be chosen.

Our experiments show that the sketch performs worse in typical classification
problems. However, this representation can be the least consuming in storing
the data representations. Furthermore, when the number of used modalities
increases, the performance of sketch-based model almost always increases.
Possibly, this improvement could be maintained until reaching the state-of-
the-art performance. If such behavior is confirmed, the sketch representation
could offer ground-breaking perspectives in the world of Big Data – highly
accurate models with optimized storage consumption.

##  6 Conclusions

In this research, we compared three techniques for multimodal fusion, the late
fusion, the early fusion, and the sketch, and identified their strengths and
weaknesses.

This work does not entirely exploit the problem of choosing the proper
technique for building the multimodal representation. Having such a technique,
one could successfully utilize the information that comes with all available
modalities, hence, obtaining the best possible performance in the considered
task.

However, we confirmed that such a choice matters, and outlined the issues that
come with it. Namely, we distinguished three main selection criteria for
choosing a technique appropriate to the ML task (see Table 6) that should be
taken into account while building multimodal representation: the impact each
modality has on the task; the type of the task; the amount of used memory.
Furthermore, using these criteria we summarized the three explored techniques,
the late fusion, the early fusion, and the sketch, deriving recommendations
that can be used by all scientists that will be working on any multimodal
task.

We believe that the next step for comparative analysis is to create a
benchmark for multimodal datasets, that would confirm all conclusions made so
far in this, previous, and future works on the problem of multimodal
representation.

## Acknowledgements

The research was funded by the Centre for Priority Research Area Artificial
Intelligence and Robotics of Warsaw University of Technology within the
Excellence Initiative: Research University (IDUB) programme (grant no
1820/27/Z01/POB2/2021).

## References

  * Yuhas et al. [1989] B.P. Yuhas, M.H. Goldstein, and T.J. Sejnowski.  Integration of acoustic and visual speech signals using neural networks.  _IEEE Communications Magazine_ , 27(11):65–71, November 1989.  ISSN 0163-6804.  doi:10.1109/35.41402.
  * Baltrusaitis et al. [2019] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal Machine Learning: A Survey and Taxonomy.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41(2):423–443, February 2019.  ISSN 0162-8828, 2160-9292, 1939-3539.  doi:10.1109/TPAMI.2018.2798607.
  * Cao et al. [2020] Wenming Cao, Wenshuo Feng, Qiubin Lin, Guitao Cao, and Zhihai He.  A Review of Hashing Methods for Multimodal Retrieval.  _IEEE Access_ , 8:15377–15391, 2020.  ISSN 2169-3536.  doi:10.1109/ACCESS.2020.2968154.
  * Gao et al. [2020] Jing Gao, Peng Li, Zhikui Chen, and Jianing Zhang.  A Survey on Deep Learning for Multimodal Data Fusion.  _Neural Computation_ , 32(5):829–864, May 2020\.  ISSN 0899-7667, 1530-888X.  doi:10.1162/neco_a_01273.
  * El-Sappagh et al. [2020] Shaker El-Sappagh, Tamer Abuhmed, S.M. Riazul Islam, and Kyung Sup Kwak.  Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series data.  _Neurocomputing_ , 412:197–215, October 2020.  ISSN 09252312.
  * Jaiswal et al. [2020] Mimansa Jaiswal, Cristian-Paul Bara, Yuanhang Luo, Mihai Burzo, Rada Mihalcea, and Emily Mower Provost.  MuSE: a Multimodal Dataset of Stressed Emotion.  In _LREC_ , pages 1499–1510, Marseille, France, May 2020. European Language Resources Association.  ISBN 9791095546344.
  * He and McAuley [2016] Ruining He and Julian McAuley.  Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering.  In _Proceedings of the 25th International Conference on World Wide Web_ , pages 507–517, Montréal Québec Canada, April 2016. International World Wide Web Conferences Steering Committee.  ISBN 9781450341431.
  * Harper and Konstan [2016] F. Maxwell Harper and Joseph A. Konstan.  The MovieLens Datasets: History and Context.  _ACM Transactions on Interactive Intelligent Systems_ , 5(4):1–19, January 2016.  ISSN 2160-6455, 2160-6463.  doi:10.1145/2827872.
  * Varshney [2022] Kush R. Varshney.  _Trustworthy Machine Learning_.  Independently published, February 2022.  ISBN 9798411903959.
  * Zhang et al. [2021] Yifei Zhang, Désiré Sidibé, Olivier Morel, and Fabrice Mériaudeau.  Deep multimodal fusion for semantic image segmentation: A survey.  _Image and Vision Computing_ , 105:104042, 2021.  ISSN 0262-8856.  doi:https://doi.org/10.1016/j.imavis.2020.104042.  URL https://www.sciencedirect.com/science/article/pii/S0262885620301748.
  * Singh et al. [2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards VQA Models That Can Read.  In _CVPR_ , pages 8309–8318, Long Beach, CA, USA, June 2019. IEEE.  ISBN 9781728132938.
  * Rychalska et al. [2020] Barbara Rychalska, Dominika Basaj Basaj, Jacek Dabrowski, and Michal Daniluk.  I know why you like this movie: Interpretable Efficient Multimodal Recommender.  _arXiv:2006.09979 [cs]_ , June 2020.
  * Laenen and Moens [2020] Katrien Laenen and Marie-Francine Moens.  A comparative study of outfit recommendation methods with a focus on attention-based fusion.  _Information Processing and Management_ , 57(6):102316, 2020.  ISSN 0306-4573.  doi:https://doi.org/10.1016/j.ipm.2020.102316.  URL https://www.sciencedirect.com/science/article/pii/S0306457320308116.
  * Salah et al. [2020] Aghiles Salah, Quoc-Tuan Truong, and Hady W. Lauw.  Cornac: A comparative framework for multimodal recommender systems.  _Journal of Machine Learning Research_ , 21(95):1–5, 2020.  URL http://jmlr.org/papers/v21/19-805.html.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi:10.18653/v1/N19-1423.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _CVPR_ , pages 770–778, Las Vegas, NV, USA, June 2016. IEEE.  ISBN 9781467388511.  doi:10.1109/CVPR.2016.90.
  * Bengio et al. [2013] Y. Bengio, A. Courville, and P. Vincent.  Representation Learning: A Review and New Perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(8):1798–1828, August 2013.  ISSN 0162-8828, 2160-9292.  doi:10.1109/TPAMI.2013.50.
  * Srivastava and Salakhutdinov [2014] Nitish Srivastava and Ruslan Salakhutdinov.  Multimodal learning with deep Boltzmann machines.  _The Journal of Machine Learning Research_ , 15(1):2949–2980, January 2014.  ISSN 1532-4435.
  * Frank et al. [2021] Stella Frank, Emanuele Bugliarello, and Desmond Elliott.  Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers.  _arXiv:2109.04448 [cs]_ , September 2021.  URL http://arxiv.org/abs/2109.04448.  arXiv: 2109.04448.
  * Gallo et al. [2017] Ignazio Gallo, Alessandro Calefati, and Shah Nawaz.  Multimodal Classification Fusion in Real-World Scenarios.  In _ICDAR_ , pages 36–41, Kyoto, November 2017. IEEE.  ISBN 9781538635865.  doi:10.1109/ICDAR.2017.326.
  * Kiela et al. [2018] D. Kiela, E. Grave, A. Joulin, and T. Mikolov.  Efficient Large-Scale Multi-Modal Classification.  _arXiv:1802.02892 [cs]_ , February 2018.
  * Dabrowski et al. [2020] Jacek Dabrowski, Barbara Rychalska, Michal Daniluk, Dominika Basaj, Konrad Goluchowski, Piotr Babel, Andrzej Michalowski, and Adam Jakubowski.  An efficient manifold density estimator for all recommendation systems.  _arXiv:2006.01894 [cs, stat]_ , June 2020.
  * Ghayoumi et al. [2018] Mehdi Ghayoumi, Miguel Gomez, Kate E. Baumstein, Narindra Persaud, and Andrew J. Perlowin.  Local Sensitive Hashing (LSH) and Convolutional Neural Networks (CNNs) for Object Recognition.  In _ICMLA_ , pages 1197–1199, Orlando, FL, December 2018. IEEE.  ISBN 9781538668054.  doi:10.1109/ICMLA.2018.00193.
  * Wirojwatanakul and Wangperawong [2019] Pasawee Wirojwatanakul and Artit Wangperawong.  Multi-Label Product Categorization Using Multi-Modal Fusion Models.  _CoRR_ , abs/1907.00420, 2019.
  * Rychalska et al. [2021] Barbara Rychalska, Dominika Basaj, Jacek Dabrowski, and Michal Daniluk.  Cleora: A Simple, Strong and Scalable Graph Embedding Scheme.  _arXiv:2102.02302 [cs]_ , February 2021.
  * De Cnudde et al. [2020] Sofie De Cnudde, David Martens, Theodoros Evgeniou, and Foster Provost.  A benchmarking study of classification techniques for behavioral data.  _International Journal of Data Science and Analytics_ , 9(2):131–173, March 2020.  ISSN 2364-415X, 2364-4168.  doi:10.1007/s41060-019-00185-1.