# Multimodal Data Integration for Oncology in the Era of Deep Neural Networks:
A Review

Asim Waqas111Corresponding author<EMAIL_ADDRESS>contribution
Department of Machine Learning, Moffitt Cancer Center, Tampa, Florida, USA.
Aakash Tripathi† Department of Machine Learning, Moffitt Cancer Center, Tampa,
Florida, USA.  Ravi P. Ramachandran  Department of Electrical and Computer
Engineering, Rowan University, Glassboro, New Jersey, USA.  Paul Stewart
Department of Machine Learning, Moffitt Cancer Center, Tampa, Florida, USA.
Ghulam Rasool  Department of Machine Learning, Moffitt Cancer Center, Tampa,
Florida, USA.

(January 2024)

###### Abstract

Cancer has relational information residing at varying scales, modalities, and
resolutions of the acquired data, such as radiology, pathology, genomics,
proteomics, and clinical records. Integrating diverse data types can improve
the accuracy and reliability of cancer diagnosis and treatment. Traditional
methods typically focus on partial or unimodal information about biological
systems at individual scales and fail to encapsulate the complete spectrum of
the heterogeneous nature of data. Deep neural networks have facilitated the
development of sophisticated multimodal data fusion approaches that can
extract and integrate relevant information from multiple sources. Recent deep
learning frameworks such as Graph Neural Networks (GNNs) and Transformers have
shown remarkable success in multimodal learning. This review article provides
an in-depth analysis of the state-of-the-art in GNNs and Transformers for
multimodal data fusion in oncology settings, highlighting notable research
studies and their findings. We also discuss the foundations of multimodal
learning, inherent challenges, and opportunities for integrative learning in
oncology. By examining the current state and potential future developments of
multimodal data integration in oncology, we aim to demonstrate the promising
role that multimodal neural networks can play in cancer prevention, early
detection, and treatment through informed oncology practices in personalized
settings.

##  1 Introduction

Cancer is a disease marked by a disordered growth of abnormal cells that may
lead to death if not treated. Around 1.9 million people in the US are expected
to be diagnosed with cancer in 2023. Cancer is the second most common reason
for death in the US, and it is expected to cause 1,670 deaths per day in the
US in 2023 [1]. However, with advances in oncology research, it is estimated
that nearly 42$\%$ of newly diagnosed cases can be potentially avoided. Being
a complex disease, the development and growth of cancer involve multiple
microscopic and macroscopic changes in the cell morphology, which are not yet
fully understood. In recent years, increasing interest has been in using
machine learning techniques, such as deep neural networks (DNNs), to assist
with cancer diagnosis and treatment.

Deep learning (DL), a subset of machine learning (ML), uses neural networks
with multiple layers to analyze and process large amounts of data. This
technique has been used to develop computer-aided diagnostic (CAD) systems
that automatically detect and classify cancerous regions in medical images,
such as mammograms and computerized tomography (CT) scans. DL has also been
used to analyze molecular data to predict a patient’s treatment response and
identify new biomarkers for cancer diagnosis. With the growing amount of data
generated in oncology, DL is becoming an increasingly important tool for
understanding cancer, making diagnoses, predicting clinical outcomes,
assessing the effectiveness of different therapies, and developing new
treatments [2].

Figure 1: Number of publications involving deep learning, graph neural
networks (GNNs), GNNs in the medical domain, overall multimodal and multimodal
in biomedical and clinical sciences in the period 2014-2023 [3].

###  1.1 Multimodal Learning (MML)

One particularly promising approach is multimodal learning (MML), which
combines information from multiple sources (or modalities) to improve the
accuracy and reliability of a given task [4]. MML has gained notable
popularity in recent years, as evidenced by the increasing number of
publications related to MML (see Fig. 1). With the increasing availability of
multimodal data, MML has become an important tool for advancing understanding
of complex diseases like cancer. Multimodal data fusion has the potential to
enable the integration of diverse data types, such as medical images,
molecular data, and electronic health records (EHR), to gain a more
comprehensive understanding of a given problem. Combining information from
multiple sources makes it possible to extract and integrate relevant features
that may be missed when each data modality is analyzed in isolation.

Recent approaches to multimodal data fusion, facilitated by DNNs, can learn
from computer vision (CV) and natural language processing (NLP) data
simultaneously. Contrastive Language-Image Pre-training (CLIP) and Generative
Pre-training Transformer (GPT-4) introduced by OpenAI are multimodal
foundation models [5, 6]. Another example of MML is the Foundational Language
And Vision Alignment Model (FLAVA) [7], which combined vision and language
representation learning for multimodal reasoning. MML has also been applied in
oncology. Recently proposed RadGenNets model predicts gene mutations in Non-
small cell lung cancer (NSCLC) patients by integrating clinical and genomics
data, Positron Emission Tomography (PET) scans, and gene mutation data using a
fusion of Convolutional Neural Networks (CNNs) and Dense Neural Networks [8].

###  1.2 Prior Work

Several survey papers have been published on the topic of multimodal data
learning [9, 10, 11, 12], which provide an overview of the current state of
the field and have identified key trends and challenges. In oncology,
researchers have traditionally pursued multimodal integration through
classical ML approaches. Recently, multimodal integration using DNNs such as
graph neural networks (GNNs) and Transformers has gained significant traction.
GNNs and Transformers have been proposed for various tasks in oncology,
including tumor classification [13], prognosis [14], and treatment response
assessment [15].

###  1.3 Contributions

The success of multimodal data fusion models in traditional ML areas, such as
CV and NLP, shows their potential for improving cancer care [9]. While many
recent surveys cover the topic of MML, there is a lack of surveys specifically
focused on using multimodal DL models (such as GNNs and Transformers) in
oncology settings. In the following, we provide the contributions of this
article:

  1. 1.

Identifying large-scale MML approaches in oncology. We provide an overview of
the current state-of-the-art in using DNNs, specifically GNNs and
Transformers, for multimodal data fusion in oncology.

  2. 2.

Highlighting the challenges and limitations of multimodal oncology data
fusion. We discuss the challenges and limitations of implementing multimodal
data fusion models in oncology, including the need for large datasets, the
complexity of integrating diverse data types, data alignment, missing data
modalities and samples.

  3. 3.

Providing a taxonomy for describing multimodal architectures. We present a
comprehensive taxonomy for describing multimodal architectures, including both
traditional and DL methods, to facilitate future research in this area.

  4. 4.

Identifying key contributions and future directions for multimodal data fusion
in oncology. Finally, we explore the various ways of integrating multimodal
oncology data using neural networks. We identify GNNs and Transformers as a
potential solution for across-the-spectrum multimodal integration and present
the associated challenges.

##  2 Fundamentals of Multimodal Learning

Figure 2: From population to single cell, cancer data modalities at scales.

The word _modality_ represents the expression of an entity or a particular
form of sensory perception, such as the visual actions of the characters,
sounds of the dialogues being spoken, or the background music [16]. A
collective notion of these modalities is called _multi-modality_ [11].

###  2.1 Data Modalities in Oncology

In oncology, data is inherently multimodal, as it includes information from
multiple sources or modalities that are related in various ways. Fig. 2
provides a view of multiple modalities of cancer at various scales, from the
population level to the single-cell analysis. Oncology data can be broadly
classified into three categories: clinical, molecular, and imaging, where each
category provides complementary information about the patient’s disease. Fig.
3 highlights different modalities within clinical, molecular, and imaging
categories.

Traditional methods to study cancer are targeted toward individual data
modalities (e.g., EHR [17], radiology [18], pathology [19], and molecular,
including genomics [20], transcriptomics [21], proteomics [22], etc.). Multi-
scale or multi-resolution studies aim to analyze the disease across different
scales that may comprise cellular, tissue, and organismic levels [23] or
identify the infection outcome of disease across tissues and organs scales
[24]. Multimodal analysis endeavors to gain holistic insights into the disease
through multiple measurement methods. For the confines of this review, we will
collectively call all these multi-categories as _multimodal_ data.

####  2.1.1 Molecular Data

Molecular modalities provide information about the underlying genetic changes
and alterations in the cancer cells [25]. Efforts toward integrating molecular
data resulted in the _multi-omics_ research field. Two main areas of molecular
analysis in oncology are proteomics and genomics. Proteomics involves the
study of proteins and their changes in response to cancer, and it provides
information about the biological processes taking place in cancer cells.
Genomics involves the study of the entire genome of cancer cells, including
changes in DNA sequence, gene expression, and structural variations [9]. Other
molecular modalities include transcriptomics, pathomics, radiomics, and their
combinations, radiogenomics and proteogenomics. Many publicly available
datasets provide access to molecular data, including the Proteomics Data
Commons (PDC) for proteomics data and the Genome Data Commons (GDC) for
genetic data [26, 27].

Figure 3: A landscape of oncology data (sub)modalities acquired for cancer
care.

####  2.1.2 Imaging Data

Imaging modalities play a crucial role in diagnosing and monitoring cancer.
The imaging category can be divided into two main categories: radiological and
histopathological. Radiological imaging encompasses various techniques such as
X-rays, CT scans, and Magnetic resonance imaging (MRI), which provide
information about the location and extent of cancer within the body. These
images can be used to determine the size and shape of a tumor, monitor its
growth, and assess the effectiveness of treatments. Histopathological imaging
involves the examination of tissue samples obtained through biopsy or surgery
[28]. These images (referred to as WSI) provide detailed information about the
micro-structural changes in the cancer cells and can be used to diagnose
cancer and determine its subtype.

####  2.1.3 Clinical Data

Clinical data provides information about the patient’s medical history,
physical examination, and laboratory results. Clinical data can be divided
into time-series data, electronic health records (EHR), and clinician notes.
Time-series data refers to the collection of clinical data over time, such as
repeated blood tests. This data informs the changes in the patient’s condition
and monitor the disease progression [29]. EHR consists of digital records of a
patient’s health information stored in a centralized database. They provide a
comprehensive view of a patient’s medical history, past diagnoses, treatments,
and laboratory test results, which help in understanding the disease [30].
Clinical notes are part of EHR and refer to the written records of a patient’s
medical history and examinations, as well as the observations and
recommendations of the healthcare provider[29].

###  2.2 Taxonomy of Multimodal Learning

We follow the taxonomy proposed by William _et al._ [16] (see Fig. 4), which
defines five main stages of multimodal classification - preprocessing, feature
extraction, data fusion, primary learner, and final classifier, as given
below:

####  2.2.1 Pre-processing

Pre-processing involves modifying the input data to a suitable format before
being fed into the model for training. It includes data cleaning,
normalization, class balancing, and augmentation. Data cleaning removes
unwanted noise or bias, errors, and missing data points [31]. Normalization
scales the input data within a specific range to ensure that each modality
contributes equally to the training [32]. Class balancing is done in cases
where one class may have a significantly larger number of data points than
another, resulting in model bias towards the dominant class. Data augmentation
artificially increases the size of the dataset by generating new samples based
on the existing data to improve model’s robustness and generalizability [31].

####  2.2.2 Feature Extraction

The different modalities may have different features, and extracting relevant
features can improve the model’s learning. Several manual and automated
feature engineering techniques generate representations (or _embeddings_) for
each data modality. Feature engineering involves designing features relevant
to the task and extracting them from the input data. This can be time-
consuming but may allow the model to incorporate prior knowledge about the
problem. Text encoding techniques, such as bag-of-words, word embeddings,
topic models [33, 34], transform textual data into a numerical representation,
which can be used as input to an ML model [35]. ML model-generated filters are
a feature extraction technique particularly useful for image and video data
[36]. These filters are learned automatically by DNNs during the training
process and can capture relevant features such as edges, textures, or shapes
[36].

####  2.2.3 Data Fusion

Data fusion combines raw features, extracted features, or class prediction
vectors from multiple modalities to create a single data representation.
Fusion enables the model to use the complementary information provided by each
modality and improve its learning. Data fusion can be done using early, late,
or intermediate fusion. Section 2.3 presents a detailed discussion of these
fusion stages. The choice of fusion technique depends on the characteristics
of the data and the specific problem being addressed [37].

####  2.2.4 Primary Learner

The primary learner stage involves training the model on the pre-processed,
feature-extracted data. Depending on the specific problem and data, the
primary learner can be implemented using various ML techniques. DNNs are a
popular choice for primary learners in MML because they can automatically
learn high-level representations from the input data and have demonstrated
state-of-the-art performance in many applications. CNNs are often used for
image and video data, while recurrent neural networks (RNNs) and Transformers
are commonly used for text and sequential data. The primary learner can be
implemented independently for each modality or shared between modalities,
depending on the specific problem and data.

####  2.2.5 Final Classifier

The final stage of MML is the classifier, which produces category labels or
class scores and can be trained on the output of the primary learner or the
fused data. This stage can include anything from a simple shallow neural
network, decision tree, or an ensemble model [16]. Ensemble methods, such as
stacking or boosting, are often used to improve the performance of the final
classifier. Stacking involves training multiple models and then combining
their predictions at the output stage, while boosting involves iteratively
training weak learners and adjusting their weights based on the errors made by
previous learners [38].

###  2.3 Data Fusion Stages

Multimodal data fusion can be performed at different levels, including early
(feature level), intermediate (model level), and late (decision level), as
illustrated in Fig. 4. Each fusion stage has its own advantages and
challenges, and the choice of fusion stage depends on the characteristics of
the data and the problem at hand. The following sections will explore each
fusion stage and discuss its applications and limitations.

Figure 4: Taxonomy, stages, and techniques of multimodal data fusion presented
in this survey. _Early_ , _late_ , _cross-modality_ fusion methods integrate
individual modality features _before_ , _after_ , or _at_ the primary learning
step, respectively.

The early fusion approach involves merging the features extracted from
different modalities before the model is trained and combining them into a
single feature vector. In this method, the feature vectors of the different
modalities are combined into a single vector, which is used as the input to
the ML model [16]. This approach can be used when the modalities have
complementary information and can be easily aligned, such as combining visual
and audio features in a video analysis application. The main challenge with
early fusion is ensuring that the features extracted from different modalities
are compatible and provide complementary information.

The intermediate fusion approach involves training separate models on each
modality and then combining the outputs of these models to make a final
prediction [16]. This approach is suitable when the data modalities are
independent of each other and cannot be easily combined at the feature level.
The main challenge with intermediate fusion is selecting an appropriate method
for combining the output of the separate models. This can be done by averaging
the separate models’ output or using a weighted sum.

In late fusion, the output of each model is used to make a separate decision,
which is then combined to make a final decision. This approach is suitable
when the modalities provide complementary information but are not necessarily
independent of each other. The main challenge with late fusion is selecting an
appropriate method for combining individual predictions. This can be done
using majority voting, weighted voting, or employing other ML models.

###  2.4 Neural Network Architectures on Oncology Data

ML and DL techniques are pervasive and effective in oncology research [39].
Cahall _et al._ [40] used Inception modules on U-Net for brain tumor
segmentation. Syed _et al._ [41] used a Random Forest classifier to fuse
radiology image representations learned from the singular value decomposition
method with the textual annotation representation learned from the fastText
algorithm for prostate and lung cancer patients. Liu _et al._ [42] proposed a
hybrid DL framework for combining breast cancer patients’ genomic and
pathology data using fully-connected (FC) network for genomic data, CNN for
radiology data, and Simulated Annealing algorithm for late fusion. Multiview
multimodal network (MVMM-Net) [43] combined two different modalities (low-
energy and dual-energy subtracted) from contrast-enhanced spectral mammography
images, each learned through CNN and late-fusion through FC network in breast
cancer detection task. Yap _et al._ [44] used a late-fusion method to fuse
image representations from ResNet50 and clinical representations from random
forest for a multimodal skin lesion classification task. An award-winning work
[45] on brain tumor grade classification followed a late-fusion
(concatenation) from CNNs on radiology and pathology images separately and
classified using logistic regression algorithm.

The single-cell unimodal data alignment is one technique in MML. Jansen _et
al._ devised an approach (SOMatic) to combine ATAC-seq regions with RNA-seq
genes using self-organizing maps [46]. Single-Cell data Integration via
Matching (SCIM) [47] matched cells in multiple datasets in low-dimensional
latent space using autoencoder. Graph-linked unified embedding (GLUE) [48]
model learned regulatory interactions across omics layers and aligned the
cells using variational autoencoders. These aforementioned methods cannot
incorporate high-order interactions among cells or different modalities.
Single-cell data integration using multiple modalities is mostly based on
auto-encoders (scDART [49], Cross-modal Autoencoders [50], Mutual Information
Learning for Integration of Single Cell Omics Data (SMILE) [51]).

##  3 GNNs in Multimodal Learning

Graphs are commonly used to represent the relational connectivity of any
system that has interacting entities[52]. Graphs have been used in various
fields, such as to study brain networks [53], driving maps [54], and the
structure of DNNs themselves [55]. Graph neural networks (GNNs) are a type of
neural network specifically designed to process data represented as a graph
[56], which makes them well-suited for analyzing multimodal oncology data
[12]. In this section, we first introduce the graph data with its types and
tasks, then we discuss the various graph learning algorithms, followed by a
detailed review of the graph learning methods used on unimodal and multimodal
oncology data tasks.

###  3.1 The Graph Data

A graph is represented as $G$=$(V,E)$ having node-set
$V$=$\\{v_{1},v_{2},...,v_{n}\\}$, where node $v$ has feature vector xv, and
edge set $E$=$\\{(v_{i},v_{j})\mid v_{i},v_{j}\in V\\}$. The neighborhood of
node $v$ is defined as $N(v)$=$\\{u\mid(u,v)\in E\\}$.

####  3.1.1 Graph Types

As illustrated in Fig. 5(a), the common types of graphs include undirected,
directed, homogeneous, heterogeneous, static, dynamic, unattributed, and
attributed. Undirected graphs comprise undirected edges, that is, the
direction of relation is not important between any ordered pair of nodes. In
the directed graphs, the nodes have a directional relationship(s). Homogeneous
graphs have the same type of nodes, whereas heterogeneous graphs have
different types of nodes within a single graph [57]. Static graphs do not
change over time with respect to the existence of edges and nodes, whereas
dynamic graphs change over time resulting in changes to structure, attributes,
and node relationships. Unattributed graphs have unweighted edges indicating
that the weighted value for all edges in a graph is the same, i.e., $1$ if
present, $0$ if absent. Attributed graphs have different edge weights that
capture the strength of relational importance [56].

Figure 5: (a) The commonly occurring graph types: undirected/ directed,
homogeneous/ heterogeneous, dynamic/ static, attributed (edges)/ Unattributed.
(b) Three tasks on graph data: node-level, link-level, and graph-level tasks.
(c) Various categories of representation learning for graphs are presented.

####  3.1.2 Tasks on Graphs

In Fig. 5(b), we present three major types of tasks defined on graphs,
including:

  * •

Node-level tasks. These may include node classification, regression,
clustering, attributions, and generation.

  * •

Edge-level task. Edge classification and prediction (presence or absence) are
two common edge-level tasks.

  * •

Graph-level tasks. These tasks involve predictions on the graph level, such as
graph classification and generation.

###  3.2 Learning on Graphs

Representing data as graphs can enable capturing and encoding the
relationships among entities of the samples [58]. Based on the way the nodes
are encoded, the data representation learning on graphs can be categorized
into the traditional (or shallow) and deep neural network-based methods, as
illustrated in Fig. 5(c). The reader is referred to [59, 58] for details on
these methods.

####  3.2.1 Traditional (Shallow) Methods

These methods usually employ classical ML methods, and their two categories
commonly found in the literature are _graph embedding_ and _probabilistic
methods_. Graph embedding methods represent a graph with low-dimensional
vectors (graph embedding and node embedding), preserving the structural
properties of the graph. The learning tasks in graph embedding usually involve
dimensionality reduction through linear (principal component or discriminant
analysis), kernel (nonlinear mapping), or tensor (higher-order structures)
methods [59]. Probabilistic graphical methods use graph data to represent
probability distribution, where nodes are considered random variables, and
edges depict the probability relations among nodes [59]. Bayesian networks,
Markov’s networks, variational inference, variable elimination, and others are
used in probabilistic methods [59].

####  3.2.2 Deep Methods - GNNs

GNNs are gaining popularity in the ML community, as evident from Fig. 1. GNNs
are DL techniques defined for graphs where information aggregation from the
neighborhood is fused into a node’s representation. Traditional DL methods
such as CNNs and their variants have shown remarkable success in processing
the data in Euclidean space; however, they fail to perform well when faced
with non-Euclidean or relational datasets. Compared to CNNs, where the
locality of the nodes in the input is fixed, GNNs have no canonical ordering
of the neighborhood of a node and can learn the given task for any permutation
of the input data, as depicted in Fig. 6. GNNs often employ a message-passing
mechanism in which a node’s representation is derived from its neighbors’
representations via a recursive computation. The message passing for a GNN is
given as follows:

$\displaystyle\mathbf{h}_{v}^{(l+1)}=$
$\displaystyle\sigma\left(W_{l}\sum_{u\in
N(v)}\frac{\mathbf{h}_{u}^{(l)}}{|N(v)|}+B_{l}\mathbf{h}_{v}^{(l)}\right)$ (1)

Figure 6: Image vs. graph convolutions. The canonical order of the input is
important in CNNs, whereas in GNNs, the order of the input nodes is not
important. This essentially makes CNNs a subset of GNNs [60].

where $h_{v}^{(l+1)}$ is the updated embedding of node $v$ after $l$+$1$
layer, $\sigma$ is the non-linear function (such as rectified linear unit or
ReLU), $h_{u}^{(l)}$ and $h_{v}^{(l)}$ represent the embeddings of nodes $u$
and $v$ at layer $l$. $W_{l}$ and $B_{l}$ are the trainable weight matrices
for neighborhood aggregation and (self)hidden vector transformation,
respectively. The message passing can encode high-order structural information
in node embedding through multiple aggregation layers. GNNs smooth the
features by aggregating neighbors’ embedding and filter eigenvalues of graph
Laplacian, which provides an extra denoising mechanism [61]. GNNs comprise
multiple permutation equivariant and invariant functions, and they can handle
heterogeneous data [62]. As described earlier, traditional ML models deal with
Euclidean data. In oncology data, the correlations may not exist in Euclidean
space; instead, its features may be highly correlated in the non-Euclidean
space [63]. Based on the information fusion and aggregation methodology, GNNs-
based deep methods are classified into the following:

  * •

_Recurrent GNNs_ : RecGNNs are built on top of the standard Recurrent Neural
Network (RNN) by combining with GNN. RecGNNs can operate on graphs having
variable sizes and topologies. The recurrent component of the RecGNN captures
temporal dependencies and learns latent states over time, whereas the GNN
component captures the local structure. The information fusion process is
repeated a fixed number of times until an equilibrium or the desired state is
achieved [64]. RecGNNs employ the model given by:

$\mathbf{h}_{v}^{(l+1)}=\text{RecNN}\left(\mathbf{h}_{u}^{(l)},\mathbf{Msg}_{N(v)}^{(l)}\right)$
(2)

where, RecNN is any RNN, and $Msg_{N(v)}^{(l)}$ is the neighborhood message-
passing at layer $l$.

  * •

_Convolutional GNNs_ : ConvGNNs undertake the convolution operation on graphs
by aggregating neighboring nodes’ embeddings through a stack of multiple
layers. ConvGNNs use the symmetric and normalized summation of the
neighborhood and self-loops for updating the node embeddings given by:

$\mathbf{h}_{v}^{(l+1)}=\sigma\left(W_{l}\sum_{u\in N(v)\cup
v}\frac{\mathbf{h}_{v}}{\sqrt{|N(v)||N(u)|}}\right)$ (3)

The ConvGNN can be spatial or spectral, depending on the type of convolution
they implement. Convolution in spatial ConvGNNs involves taking a weighted
average of the neighboring vertices. Examples of spatial ConvGNNs include
GraphSAGE[64], Message Passing Neural Network (MPNN)[65], and Graph Attention
Network (GAT) [66]. The spectral ConvGNNs operate in the spectral domain by
using the eigendecomposition of the graph Laplacian matrix. The convolution
operation is performed on the eigenvalues, which can be high-dimensional.
Popular spectral ConvGNNs are ChebNet [67] and Graph Convolutional Network
(GCN)[68]. An interesting aspect of these approaches is representational
containment, which is defined as:

$convolution\subseteq attention\subseteq message\\_passing.$

  * •

_Graph Auto-Encoder Networks_ : GAEs are unsupervised graph learning networks
for dimensionality reduction, anomaly detection, and graph generation. They
are built on top of the standard auto-encoders to work with graph data. The
encoder component of the GAE maps the input graph to a low-dimensional latent
space, while the decoder component maps the latent space back to the original
graph [69].

  * •

_Graph Adversarial Networks (GraphANs)_ : Based on Generative Adversarial
Networks, GraphANs are designed to work with graph-structured data and can
learn to generate new graphs with similar properties to the input data. The
generator component of the GraphAN maps a random noise vector to a new graph,
while the discriminator component tries to distinguish between the generated
vs. the actual input. The generator generates graphs to fool the
discriminator, while the discriminator tries to classify the given graph as
real or generated.

  * •

_Other GNNs_ : Other categories of GNNs may include scalable GNNs [70],
dynamic GNNs [71], hypergraph GNNs [72], heterogeneous GNNs [73], and many
others [74].

Figure 7: (a) Data processing pipeline for histopathology images using GNNs
[75]. (b) Graph processing pipeline on radiology data. Adapted from [76].

###  3.3 Unimodal Oncology Learning using GNNs

Here, we explore how GNNs have been used to analyze single-modality oncology
data. Single modality refers to using one resolution or type of cancer data,
such as pathology, radiology, or molecular. A detailed survey on the use of
GNNs for both oncology and non-oncology data is presented in [77].

####  3.3.1 Pathology

Traditionally, CNN-based models are predominantly used to learn features from
digital pathology data. However, unlike GNNs, CNNs fail to capture the global
contextual information which is important in the tissue phenotypical and
structural micro and macro environment [78]. For using histology images in
GNNs, the cells, tissue regions, or image patches are depicted as nodes. The
relations and interactions among these nodes are represented as (un)weighted
edges. Usually, a graph of the patient histology slide is used along with a
patient-level label for training a GNN, as illustrated in Fig. 7(a). Here we
review a few GNNs-based pathology publications representative of a trove of
works in this field. Histographs [79] used breast cancer histology data to
distinguish cancerous and non-cancerous images. Pre-trained VGG-UNet was used
for nuclei detection, micro-features of the nuclei were used as node features,
and Euclidean distance among nuclei was incorporated as edge features. The
resulting cell-graphs were used to train the GCN-based robust spatial
filtering (RSF) model and showed superior performance compared to the CNN-
based classification. Wang _et al._ [80] analyzed grade classification in
tissue micro-arrays of prostate cancer using the weakly-supervised technique
on a variant of GraphSAGE with self-attention pooling (SAGPool). Cell-Graph
Signature ($CG_{signature}$) [81] predicted patient survival in gastric cancer
using cell-graphs of multiplexed immunohistochemistry images processed through
two types of GNNs (GCNs and GINs) with two types of pooling (SAGPool,
TopKPool). Besides above mentioned cell-graphs, there is an elaborate review
of GNN-based tissue-graphs or patch-graphs methods implemented on unimodal
pathology cancer data given in [78]. Instead of individual cell- and tissue-
graphs, a combination of the multilevel information in histology slides can
help understand the intrinsic features of the disease.

####  3.3.2 Radiology

GNNs have been used in radiology-based cancer data for segmentation,
classification, and prediction tasks, especially on X-rays, mammograms, MRI,
PET, and CT scans. Fig. 7(b) illustrates a general pipeline of using
radiology-based data to train GNNs. Here we give a non-exhaustive review of
GNNs-based works on radiological oncology data as a single modality input. Mo
_et al._ [82] proposed a framework that improved the liver cancer lesion
segmentation in the MRI-T1WI scans through guided learning of MRI-T2WI
modality priors. Learned embeddings from fully convolutional networks on
separate MRI modalities are projected into the graph domain for learning by
GCNs through the co-attention mechanism and finally to get the refined
segmentation by re-projection. Radiologists usually review radiology images by
zooming into the region of interest (ROIs) on high-resolution monitors. Du _et
al._ [83] used a hierarchical GNN framework to automatically zoom into the
abnormal lesion region of the mammograms and classify breast cancer. The pre-
trained CNN model extracts image features, whereas a GAT model is used to
classify the nodes for deciding whether to zoom in or not based on whether it
is benign or malignant. Based on the established knowledge that lymph nodes
(LNs) have connected lymphatic system and LNs cancer cells spread on certain
pathways, Chao _et al._ [84] proposed a lymph node gross tumor volume learning
framework. The framework was able to delineate the LN appearance as well as
the inter-LN relationship. The end-to-end learning framework was superior to
the state-of-the-art on esophageal cancer radiotherapy dataset. Tian _et al._
[85] suggested interactive segmentation of MRI scans of prostate cancer
patients through a combination of CNN and two GCNs. CNN model outputs a
segmentation feature map of MRI, and the GCNs predict the prostate contour
from this feature map. Saueressig et al. [86] used GNNs to segment brain
tumors in 3D MRI images, formed by stacking different modalities of MRI (T1,
T2, T1-CE, FLAIR) and representing them as supervoxel graph. The authors
reported that GraphSAGE-pool was best for segmenting brain tumors. Besides
radiology, a parallel field of radiomics has recently gained attraction.
Radiomics is the automated extraction of quantitative features from radiology
scans. A survey of radiomics and radiogenomic analysis on brain tumors is
presented in[76].

####  3.3.3 Molecular Data

Graphs are a natural choice for representing molecular data such as omic-
centric (DNA, RNA, or proteins) or single-cell centric. Individual modalities
are processed separately to generate graph representations that are then
processed through GNNs followed by the classifier to predict the downstream
task, as illustrated in Fig. 8. One method of representing proteins as graphs
is to depict the amino acid residue in the protein as the node and the
relationship between residues denoted by edge [87]. The residue information is
depicted as node embedding, whereas the relational information between two
residues is represented as the edge feature vector. Fout _et al._ [87] used
spatial ConvGNNs to predict interfaces between proteins which is important in
drug discovery problems. Deep predictor of drug-drug interactions (DPDDI)
predicted the drug-drug interactions using GCN followed by a 5-layer classical
neural network [88]. Molecular pre-training graph net (MPG) [89] is a powerful
framework based on GNN and BERT to learn drug-drug and drug-target
interactions. Graph-based Attention Model (GRAM) [90] handled the data
inefficiency by supplementing EHRs with hierarchical knowledge in the medical
ontology. A few recent works have applied GNNs to single-cell data. scGCN [91]
is a knowledge transfer framework in single-cell omics data such as mRNA or
DNA. scGNN [92] processed cell-cell relations through GNNs for the task of
missing-data imputation and cell clustering on single-cell RNA sequencing
(scRNA-seq) data.

###  3.4 Multimodal Pre-learning Fusion

The first and most primitive form of MML is the pre-learning fusion (see Fig.
4), where features extracted from individual modalities of data are merged,
and the fused representations are then used for training the multimodal
primary learner model. In the context of GNNs being the primary learning
model, the extraction step of individual modality representations can be hand-
engineered (e.g., dimensionality reduction) or learned by DL models (e.g.,
CNNs, Transformers). Cui _et al._ [93] proposed a GNN-based early fusion
framework to learn latent representations from radiological and clinical
modalities for Lymph node metastasis (LNM) prediction in esophageal squamous
cell carcinoma (ESCC). The extracted features from the two modalities using
UNet and CNN-based encoders were fused together with category-wise attention
as node representation. The message passing from conventional GAT and
correlation-based GAT learned the neighborhood weights. The attention
attributes were used to update the final node features before classification
by a 3-layer fully connected network. For Autism spectrum disorder,
Alzheimer’s disease, and ocular diseases, a multimodal learning framework
called Edge-Variational GCN (EV-GCN) [94] fuses the radiology features
extracted from fMRI images with clinical feature vectors for each patient. An
MLP-based pairwise association encoder is used to fuse the input feature
vectors and to generate the edge weights of the population graph. The
partially labeled population graph is then processed through GCN layers to
generate the diagnostic graph of patients.

Figure 8: Graph processing pipeline on non-imagery data. Adapted from [95].

###  3.5 Multimodal Cross-learning Fusion

Cross-MML involves intermediate fusion and/or cross-links among the models
being trained on individual modalities (see Fig. 4). For this survey, we
consider the GNN-based hierarchical learning mechanisms as the cross-MML
methods. Hierarchical frameworks involve learning for one modality and using
the learned latent embeddings in tandem with other data modalities
sequentially to get the final desired low-dimensional representations. Lian
_et al._ [96] used a sequential learning framework where tumor features
learned from CT images using the ViT model were used as node features of the
patient population graph for subsequent processing by the GraphSAGE model. The
hierarchical learning from radiological and clinical data using Transformer-
GNN outperformed the ResNet-Graph framework in survival prediction of early-
stage NSCLC. scMoGNN [97] is the first method to apply GNNs in multimodal
single-cell data integration using a cross-learning fusion-based GNN
framework. Officially winning first place in modality prediction task at the
NeurIPS 2021 competition, scMoGNN showed superior performance on various tasks
by using paired data to generate cell-feature graphs. Hierarchical cell-to-
tissue-graph network (HACT-Net) combined the low-level cell-graph features
with the high-level tissue-graph features through two hierarchical GINs on
breast cancer multi-class prediction [98]. Data imputation, a method of
populating the missing values or false zero counts in single-cell data mostly
done using DL autoencoders (AE) architecture, has recently been accomplished
using GNNs. scGNN [92] used imputation AE and graph AE in an iterative manner
for imputation, and GraphSCI [99] used GCN with AE to impute the single-cell
RNA-seq data using the cross-learning fusion between the GCN and the AE
networks. Clustering is a method of characterizing cell types within a tissue
sample. Graph-SCC [100] clustered cells based on scRNA-seq data through self-
supervised cross-learning between GCN and a denoising AE network. Recently a
multilayer GNN framework, Explainable Multilayer GNN (EMGNN), has been
proposed for cancer gene prediction task using multi-omics data from 16
different cancer types [101].

###  3.6 Multimodal Post-learning

Because of their superior performance, the most sought-after multimodal fusion
technique in the literature is the post-learning fusion method, where the
individual data views are first processed by the primary learner and later
fused for the downstream predictive task [102]. In the post-learning fusion
paradigm, the hand-crafted features perform better than the deep features when
the dimensionality of input data is low, and vice versa [102]. Many
interesting GNN-based works involving the post-learning fusion mechanism have
recently been published. Decagon [103] used a multimodal approach on GCNs
using proteins and drugs interactions to predict exact side effects as a
multi-relational link prediction task. Drug–target affinity (DTA) [104]
experimented with four different flavors of GNNs (GCN, GAT, GIN, GAT-GCN)
along with a CNN to fuse together molecular embeddings and proteins sequences
for predicting drug-target affinity. PathomicFusion [75] combined the
morphological features extracted from image patches (using CNNs), cell-graph
features from cell-graphs of histology images (GraphSAGE-based GCNs), and
genomic features (using feed-forward network) for survival prediction on
glioma and clear cell renal cell carcinoma. Shi _et al._ [105] proposed a
late-fusion technique to study screening of cervical cancer at early stages,
by using CNNs to extract features from histology images, followed by K-means
clustering to generate graphs which are processed through two-layer GCN. BDR-
CNN-GCN (batch normalized, dropout, rank-based pooling) [106] used the same
mammographic images to extract image-level features using CNN and relation-
aware features using GCN. The two feature sets are fused using dot product
followed by a trainable linear projection for the breast cancer
classification. Under the umbrella of multi-omics data, many GNN-based
frameworks have been proposed recently. Molecular omics network(MOOMIN)[107],
a multi-modal heterogeneous GNN to predict oncology drug combinations,
processed molecular structure, protein features, and cell lines through GCN-
based encoders, followed by late-fusion using a bipartite drug-protein
interaction graph. Multi-omics graph convolutional networks (MOGONET) [95]
used a GCN-GAN late fusion technique for the classification of four different
diseases, including three cancer types, breast, kidney, and glioma. Leng _et
al._ [108] extended MOGONET to benchmark three multi-omics datasets on two
different tasks using sixteen DL networks and concluded that GAT-based GNN had
the best classification performance. Multi-Omics Graph Contrastive
Learner(MOGCL) [109] used graph structure and contrastive learning information
to generate representations for improved downstream classification tasks on
the breast cancer multi-omics dataset using late-fusion. Similar to MOGCL,
Park _et al._ [110] developed a GNN-based multi-omics model that integrated
mRNA expression, DNA methylation, and DNA sequencing data for NSCLC diagnosis.

##  4 Transformers in Multi-modal Learning

Transformers can be viewed as a type of GNN [10]. Transformers are attention-
based DNN models that originally gained attention in NLP [111]. Transformers
implement scaled dot-product of the input with itself and can process various
types of data in parallel [111]. Transformers can handle sequential data and
learn long-range dependencies, making them well-suited for tasks such as
language translation, language modeling, question answering, and many more
[112]. Unlike Recurrent Neural Networks (RNNs) and CNNs, Transformers use
self-attention operations to weigh the importance of different input tokens
(or embeddings) at each time step. This allows them to handle sequences of
arbitrary length and to capture dependencies between input tokens that are far
apart in the sequence [111]. Transformer models have also been applied to
other data types, such as images [113], audio [114], and time-series analysis
[115], resulting in a new wave of multi-modal applications. Transformers can
handle input sequences of different modalities in a unified way, using the
same self-attention mechanism which processes the inputs as a fully connected
graph [10]. This allows Transformers to capture complex dependencies between
different modalities, such as visual and textual information in visual
question-answering (VQA) tasks [116]. Pre-training Transformers on large
amounts of data, using unsupervised or self-supervised learning, and then
fine-tuning for specific downstream tasks, has led to the development of
foundation models, such as BERT [33], GPT [117], RoBERTa [34], CLIP [5], T5
[118], BART [119], BLOOM [120], ALIGN [121], CoCa [122] and more.

Multimodal Transformers are a recent development in the field of MML, which
extends the capabilities of traditional Transformers to handle multiple data
modalities. The inter-modality dependencies are captured by the cross-
attention mechanism in multimodal Transformers, allowing the model to jointly
reason and extract rich data representations. There are various types of
multimodal Transformers, such as Unified Transformer (UniT) [123], Multi-way
Multimodal Transformer (MMT) [124], CLIP [5], Flamingo [125], CoCa [122],
Perceiver IO [126], and GPT-4[6].

Figure 9: The original Transformer architecture [111]. A Transformer has
multiple encoders and decoders blocks based on application and data type.

###  4.1 The Transformer Architecture

The original Transformer (Fig. 9) was composed of encoder/ decoder blocks,
each made up of several layers of self-attention and feed-forward neural
networks. The encoder takes the input sequence and generates hidden
representations, which are then fed to the decoder. The decoder generates the
output sequence by attending to the encoder’s hidden representations and the
previous tokens (i.e., auto-regressive). The self-attention operation (or
scaled dot-product) is a crucial component of Transformer. It determines the
significance of each element in the input sequence with respect to the whole
input. Self-attention operates by computing a weighted sum of the input
sequence’s hidden representations, where the weights are determined by the dot
product between the _query_ vector and the _key_ vector, followed by a scaling
operation to stabilize the gradients. The resulting weighted sum is multiplied
by a _value_ vector to obtain the output of the self-attention operation.
There has been a tremendous amount of work on various facets of Transformer
architecture. The readers are referred to relevant review papers [112, 10,
127, 128].

###  4.2 Multimodal Transformers

The self-attention allows a Transformer model to process each input as a fully
connected graph and attend to (or equivalently learn from) the global patterns
present in the input. This makes Transformers compatible with various data
modalities by treating each token (or its embedding) as a node in the graph.
To use Transformers for a data modality, we need to tokenize the input and
select an embedding space for the tokens. Tokenization and embedding
selections are flexible and can be done at multiple granularity levels, such
as using raw features, ML-extracted features, patches from the input image, or
graph nodes. Table 1 summarizes some common practices related to these aspects
in oncology.

Table 1: Oncology data modalities and their respective tokenization and embeddings selection techniques Modalities | Tokenization | Token Embeddings
---|---|---
Pathology | Patch | CNNs [129]
Radiology | Patch | CNNs [130]
Electronic Health Record | ICD Code |  | GNNs [131],
---
ML models [132]
-Omics |  | Graphs
---
K-mers
| GNNs [133]
---
ML model [134]
Clinical notes | Word |  | BERT [33],
---
RoBERTa [34],
BioBERT [135]

Handling inter-modality interactions is the main challenge in developing
multimodal Transformer models. Usually, it is done through one of these fusion
methods; _early fusion_ of data modalities, _cross-attention_ , _hierarchical
attention_ , and _late fusion_ , as illustrated in Fig. 10. In the following,
we present and compare data processing steps for these four methods using two
data modalities as an example. The same analysis can be extended to multiple
modalities.

####  4.2.1 Early Fusion

Early fusion is the simplest way to combine data from multiple modalities. The
data from different modalities are concatenated to a single input before
feeding it to the Transformer model which processes input as a single entity.
Mathematically, the concatenation operation is: $x_{cat}$=$[x_{1},x_{2}]$,
where $x_{1}$ and $x_{2}$ are the inputs from two data modalities, and
$x_{cat}$ is the concatenated input to the model. Early fusion is simple and
efficient, however, it assumes that all modalities are equally important and
relevant for the task at hand [136], which may not always be practically true
[137].

####  4.2.2 Cross-Attention Fusion

Cross-attention is a relatively more flexible approach to model the
interactions between data modalities and learn their joint representations.
The Transformer layers attend to different modalities at different stages of
data processing. Cross-attention allows the model to selectively attend to
different modalities based on their relevance to the task [138], and capture
complex interactions between the modalities [139].

####  4.2.3 Hierarchical Fusion

Hierarchical fusion is a complex approach to combine multiple modalities. For
instance, DFTR (Depth-supervised Fusion Transformer for Salient Object
Detection) employs hierarchical feature extraction to improve salient object
detection performance by fusing low-level spatial features and high-level
semantic features from different scales [140]. Yang _et al._[141], introduced
a hierarchical approach to fine-grained classification using a fusion
transformer. Furthermore, the Hierarchical Multimodal Transformer (HMT) for
video summarization can capture global dependencies and multi-hop
relationships among video frames using a transformer model [142].

####  4.2.4 Late Fusion

In late fusion, each data modality is processed independently by its
Transformer model, and the branch outputs are concatenated and passed through
the final classifier. Late fusion allows the model to capture the unique
features of each modality while still learning their joint representation. Sun
_et al._ [143] proposed a multi-modal adaptive late fusion transformer network
for estimating the levels of depression. Their model independently extracts
long-term temporal information from audio and visual data and then fuses
weights at the end to learn joint representation.

Figure 10: Four different strategies of fusing information from various data
modalities in multimodal Transformers.

###  4.3 Transformers for Oncology Data Processing

Transformers have been successfully applied to various tasks in oncology,
including cancer screening, diagnosis, prognosis, treatment selection, and
prediction of clinical variables [9, 144, 145, 96, 129]. For instance, a
Transformer-based model was used to predict the presence and grade of breast
cancer using a combination of imaging and genomics data [9]. Transmil [144]
was a transformer model proposed to process histopathology images using self-
attention to learn and classify the correlation between multiple instances.
Recently, a Transformer and convolution parallel network, TransConv [145], was
proposed for automatic brain tumor segmentation using MRI data. Transformers
and GNNs have also been combined in MML for early-stage NSCLC prognostic
prediction using the patient’s clinical and pathological features and by
modeling the patient’s physiological network [96]. Similarly, a multimodal co-
attention Transformer was proposed for survival prediction using WSIs[129] by
learning from the image and genomic sequences and using the co-attention
mechanism to learn the interactions between the two data modalities [129].

##  5 Challenges in Multimodal Learning (MML)

Learning from multimodal oncology data is a complex and rapidly growing field
that presents both challenges and opportunities. While MML has shown
significant promise, there are many challenges owing to the inductive biases
of the ML models [146]. In this context, we present major challenges of MML in
oncology settings that, if addressed, could unlock the full potential of this
emerging field.

###  5.1 Availability of Large Amounts of High-quality Data

DL models are traditionally trained on large datasets with enough samples for
training, validation, and testing, such as JFT-300M[147] and YFCC100M[148],
which are not available in the cancer domain. For example, the largest
genomics data repository, Gene Expression Omnibus (GEO) database, has
approximately 1.1 million samples with the keyword ‘cancer’ compared to 3
billion images in JFT-300M [149]. Annotating medical and oncology data is a
time-consuming and manual process that requires significant expertise in many
different areas of medical sciences. Factors like heterogeneity of the
disease, noise in data recording, background and training of medical
professionals leading to inter- and intra-operator variability, cause lack of
reproducibility and inconsistent clinical outcomes [150].

###  5.2 Data Registration and Alignment

Data alignment and registration refer to the process of combining and aligning
data from different modalities in a useful manner [151]. In multimodal
oncology data, this process involves aligning data from multiple modalities
such as CT, MRI, PET, WSIs images, as well as genomics, transcriptomics, and
clinical records. Data registration involves aligning the data modalities to a
common reference frame and may involve identifying common landmarks or
fiducial markers. If the data is not registered or aligned correctly, it may
be difficult to fuse the information from different modalities accurately
[152].

###  5.3 Pan-Cancer Generalization and Transference

Transference in MML aims to transfer knowledge between modalities and their
representations to improve the performance of a model trained on a primary
modality [152]. Because of the unique characteristics of each cancer type and
site, it is challenging to develop models that are able to generalize across
different cancer sites. Furthermore, the models trained on a specific
modality, such as radiology images, will not perform well with other imaging
modalities, such as histopathology slides. Fine-tuning the model on a
secondary modality, multimodal co-learning, and model induction are a few
techniques to achieve transference and generalization [153]. To overcome this
challenge, there is a need to devise mechanisms for improved universality of
ML models.

###  5.4 Missing Data Samples and Modalities

The unavailability of one or more modalities or absence of samples in a
modality affects the model learning, as most of the existing DL models cannot
process the “missing information”. This requirement, in turn, constrains the
already insufficient size of datasets in oncology. Almost all publicly
available oncology datasets have missing data for a large number of samples
[149]. Various approaches for handling missing data samples and modalities are
gradually gaining traction, however, this is still an open challenge in
oncology [154].

###  5.5 Imbalanced Data

Class imbalance refers to the situation where one class (e.g., cancer
negative/positive) is significantly more frequent in the data than another
class. Class imbalance is a common issue in oncology data [154]. DL models
struggle to accurately classify underrepresented classes. Techniques such as
data augmentation, ensemble, continual learning, and transfer learning are
usually used to counter the class imbalance challenge [154].

###  5.6 Explainability and Trustworthiness of DL Models

The explainability in DL, e.g., how GNNs and Transformers make a specific
decision, is still an area of active research [155]. GNNExplainer [156], PGM-
Explainer [157], and SubgraphX [158] are some attempts to explain the
decision-making process of GNNs. The explainability methods for Transformers
have been analyzed in [159]. Existing efforts and a roadmap to improve the
trustworthiness of GNNs have been presented in the latest survey [160].
However, explainability and trustworthiness of multimodal GNNs and
Transformers is an under-research challenge.

###  5.7 Oversmoothing in GNNs

One particular challenge in using GNNs is oversmoothing, which occurs when the
GNN is trained for too many iterations, causing the node representations to
become almost similar [58]. This leads to a loss of information, decrease in
the model’s performance, and generalization [161]. Regularization techniques
such as dropout, weight decay, skip-connection, and incorporation of higher-
order structures, such as motifs and graphlets, have been proposed, but
building deep architectures that can scale and adapt to varying structural
patterns of graphs is still an open challenge.

###  5.8 Modality Collapse

Modality collapse is a phenomenon that occurs in MML, where a model trained on
multiple modalities may become over-reliant on a single modality, to the point
where it ignores or neglects the other modalities [162]. Recent work explored
the reasons and theoretical understanding of modality collapse [163], but
counter-actions need active investigation by the ML community.

###  5.9 Dynamic and Temporal Data

Dynamic and temporal data refers to the data that changes over time [58].
Tumor surveillance is a well-known technique to study longitudinal cancer
growth over multiple data modalities [18]. Spatio-temporal methods such as
multiple instance learning, GNNs, and hybrid of multiple models can capture
complex change in the data relationships over time, but learning on multimodal
dynamic data is an active area of research in oncology [164].

###  5.10 Data Privacy and Federated Learning

With the increased concern for the privacy of data, especially in medical
settings, MML techniques need to adapt to local data processing and remote
federation. Federated learning can help train large multimodal models across
various sites without sharing data [165].

###  5.11 Other Challenges

MML requires extensive computational resources and time to train models on a
variety of datasets and tasks. Robustness and failure detection [166] are
critical aspects of MML, particularly in applications such as oncology.
Uncertainty quantification techniques, such as Bayesian neural networks [167],
are still under-explored avenues in the MML. By addressing these challenges,
it is possible to develop MML models that are able to surpass performance
offered by single-modality models.

##  6 Multimodal Oncology Data Sources

Building central archives to unify the different collections of oncology data
requires a dedicated effort. We have compiled a non-exhaustive list of
datasets from data portals shared by the National Institute of Health (NIH)
and others. The purpose of this compilation is to provide ML oncology
researchers with a unified source of data. This periodically updated
collection is available at _https://lab-rasool.github.io/pan-cancer-dataset-
sources/_.

##  7 Conclusion

Research in integrating data across a few modalities has already shown
encouraging results. There is no unified framework available for scaling
across all possible modalities of cancer. The convergence of individual
methodologies and data across varying scales may hold vital clues in creating
a unified view of the disease that is more prognostic, predictive, and
insightful than the individual view or modality. In this survey, we reviewed
the multimodal learning task on oncology data. The future resides in
developing a deployment-ready, scalable deep learning framework with inherent
uncertainty quantification, interpretability, and generalizability to
integrate oncology data across multiple scales, modalities, and resolutions.

##  8 Acknowledgments

This work was partly supported by the National Science Foundation awards
ECCS-1903466, OAC-2008690 and OAC-2234836.

## References

  * [1] R. L. Siegel, K. D. Miller, N. S. Wagle, and A. Jemal, “Cancer Statistics, 2023,” _CA: A Cancer Journal for Clinicians_ , vol. 73, no. 1, pp. 17–48, 2023.
  * [2] H.-P. Chan, L. M. Hadjiiski, and R. K. Samala, “Computer-aided diagnosis in the era of deep learning,” _Medical Physics_ , vol. 47, no. 5, pp. e218–e227, 2020.
  * [3] D. W. Hook, S. J. Porter, and C. Herzog, “Dimensions: Building Context for Search and Evaluation,” _Frontiers in Research Metrics and Analytics_ , vol. 3, p. 23, 2018.
  * [4] Y. Huang, C. Du, Z. Xue, X. Chen, H. Zhao, and L. Huang, “What makes multi-modal learning better than single (provably),” in _Advances in Neural Information Processing Systems_ , A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.
  * [5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” 2021.
  * [6] OpenAI, “GPT-4,” 2023, available at: https://openai.com/research/gpt-4. Last accessed on April 5, 2023.
  * [7] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, “FLAVA: A Foundational Language And Vision Alignment Model,” 2021.
  * [8] S. Tripathi, E. J. Moyer, A. I. Augustin, A. Zavalny, S. Dheer, R. Sukumaran, D. Schwartz, B. Gorski, F. Dako, and E. Kim, “RadGenNets: Deep learning-based radiogenomics model for gene mutation prediction in lung cancer,” _Informatics in Medicine Unlocked_ , vol. 33, p. 101062, 2022.
  * [9] K. M. Boehm, P. Khosravi, R. Vanguri, J. Gao, and S. P. Shah, “Harnessing multimodal data integration to advance precision oncology,” _Nature Reviews Cancer_ , pp. 1–13, 2021.
  * [10] P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” 2022.
  * [11] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 2, pp. 423–443, 2018.
  * [12] Y. Ektefaie, G. Dasoulas, A. Noori, M. Farhat, and M. Zitnik, “Multimodal learning with graphs,” _Nature Machine Intelligence_ , pp. 1–11, 2023.
  * [13] M. Khan, I. Ashraf, M. Alhaisoni, R. Damaševičius, R. Scherer, A. Rehman, and S. Bukhari, “Multimodal brain tumor classification using deep learning and robust feature selection: A machine learning application for radiologists,” _Diagnostics (Basel)_ , vol. 10, no. 8, p. 565, 2020.
  * [14] S. Schulz, A.-C. Woerl, F. Jungmann, C. Glasner, P. Stenzel, S. Strobl, A. Fernandez, D.-C. Wagner, A. Haferkamp, P. Mildenberger, W. Roth, and S. Foersch, “Multimodal deep learning for prognosis prediction in renal cancer,” _Frontiers in Oncology_ , vol. 11, 2021.
  * [15] S. Joo, E. Ko, S. Kwon, E. Jeon, H. Jung, J. Kim, M. Chung, and Y. Im, “Multimodal deep learning models for the prediction of pathologic response to neoadjuvant chemotherapy in breast cancer,” _Sci Rep_ , vol. 11, no. 1, p. 18800, 2021.
  * [16] W. C. Sleeman IV, R. Kapoor, and P. Ghosh, “Multimodal classification: Current landscape, taxonomy and future directions,” _ACM Computing Surveys_ , vol. 55, no. 7, pp. 1–31, 2022.
  * [17] R. Miotto, L. Li, B. A. Kidd, and J. T. Dudley, “Deep patient: an unsupervised representation to predict the future of patients from the electronic health records,” _Scientific reports_ , vol. 6, no. 1, pp. 1–10, 2016.
  * [18] A. Waqas, D. Dera, G. Rasool, N. C. Bouaynaya, and H. M. Fathallah-Shaykh, “Brain tumor segmentation and surveillance with deep artificial neural networks,” _Deep Learning for Biomedical Data Analysis_ , pp. 311–350, 2021.
  * [19] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical image analysis_ , vol. 42, pp. 60–88, 2017.
  * [20] C. Angermueller, H. J. Lee, W. Reik, and O. Stegle, “DeepCpG: accurate prediction of single-cell DNA methylation states using deep learning,” _Genome biology_ , vol. 18, no. 1, pp. 1–13, 2017.
  * [21] S. Yousefi, F. Amrollahi, M. Amgad, C. Dong, J. E. Lewis, C. Song, D. A. Gutman, S. H. Halani, J. E. Velazquez Vega, D. J. Brat _et al._ , “Predicting clinical outcomes from large scale cancer genomic profiles with deep survival models,” _Scientific reports_ , vol. 7, no. 1, 2017.
  * [22] S. Wang, S. Sun, Z. Li, R. Zhang, and J. Xu, “Accurate de novo prediction of protein contact map by ultra-deep learning model,” _PLoS computational biology_ , vol. 13, no. 1, p. e1005324, 2017.
  * [23] D. Paul and N. L. Komarova, “Multi-scale network targeting: A holistic systems-biology approach to cancer treatment,” _Progress in biophysics and molecular biology_ , vol. 165, pp. 72–79, 2021.
  * [24] L. Talman, E. Agmon, S. M. Peirce, and M. W. Covert, “Multiscale models of infection,” _Current Opinion in Biomedical Engineering_ , vol. 11, pp. 102–108, 2019.
  * [25] J. Liu, P. Pandya, and S. Afshar, “Therapeutic advances in oncology,” _International Journal of Molecular Sciences_ , vol. 22, no. 4, 2021.
  * [26] [Online]. Available: https://pdc.cancer.gov/
  * [27] R. L. Grossman, A. P. Heath, V. Ferretti, H. E. Varmus, D. R. Lowy, W. A. Kibbe, and L. M. Staudt, “Toward a shared vision for cancer genomic data,” _New England Journal of Medicine_ , vol. 375, no. 12, p. 1109–1112, 2016.
  * [28] S. P. Rowe and M. G. Pomper, “Molecular imaging in oncology: Current impact and future directions,” _CA: A Cancer Journal for Clinicians_ , vol. 72, no. 4, pp. 333–352, 2022.
  * [29] M. Quinn, J. Forman, M. Harrod, S. Winter, K. E. Fowler, S. L. Krein, A. Gupta, S. Saint, H. Singh, and V. Chopra, “Electronic health records, communication, and data sharing: challenges and opportunities for improving the diagnostic process,” _Diagnosis_ , vol. 6, no. 3, pp. 241–248, 2019.
  * [30] O. Asan, A. B. Nattinger, A. P. Gurses, J. T. Tyszka, and T. W. Yen, “Oncologists’ Views Regarding the Role of Electronic Health Records in Care Coordination,” _JCO Clinical Cancer Informatics_ , no. 2, pp. 1–12, 2018, pMID: 30652555.
  * [31] K. K. Al-jabery, T. Obafemi-Ajayi, G. R. Olbricht, and D. C. Wunsch II, “Data preprocessing,” in _Computational Learning Approaches to Data Analytics in Biomedical Applications_ , K. K. Al-jabery, T. Obafemi-Ajayi, G. R. Olbricht, and D. C. Wunsch II, Eds. Academic Press, 2020, pp. 7–27.
  * [32] C. V. Gonzalez Zelaya, “Towards explaining the effects of data preprocessing on machine learning,” in _IEEE 35th International Conference on Data Engineering (ICDE)_ , 2019, pp. 2086–2090.
  * [33] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” 2018.
  * [34] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692
  * [35] D. Wang, J. Su, and H. Yu, “Feature extraction and analysis of natural language processing for deep learning english language,” _IEEE Access_ , vol. 8, pp. 46 335–46 345, 2020.
  * [36] S. Dara and P. Tumma, “Feature extraction by using deep learning: A survey,” in _2018 Second International Conference on Electronics, Communication and Aerospace Technology (ICECA)_ , 2018.
  * [37] P. Jiang, S. Sinha, K. Aldape, S. Hannenhalli, C. Sahinalp, and E. Ruppin, “Big Data in basic and translational cancer research,” _Nature Reviews Cancer_ , vol. 22, no. 11, p. 625–639, 2022.
  * [38] V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci, “Deep Neural Networks and Tabular Data: A Survey,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–21, 2022.
  * [39] R. Correa, N. Alonso-Pupo, and E. Rodríguez, “Multi-omics data integration approaches for precision oncology,” _Molecular Omics_ , 2022.
  * [40] D. E. Cahall, G. Rasool, N. C. Bouaynaya, and H. M. Fathallah-Shaykh, “Inception modules enhance brain tumor segmentation,” _Frontiers in computational neuroscience_ , vol. 13, p. 44, 2019.
  * [41] K. Syed, W. C. Sleeman IV, M. Hagan, J. Palta, R. Kapoor, and P. Ghosh, “Multi-view data integration methods for radiotherapy structure name standardization,” _Cancers_ , vol. 13, no. 8, p. 1796, 2021.
  * [42] T. Liu, J. Huang, T. Liao, R. Pu, S. Liu, and Y. Peng, “A hybrid deep learning model for predicting molecular subtypes of human breast cancer using multimodal data,” _Irbm_ , vol. 43, no. 1, pp. 62–74, 2022.
  * [43] J. Song, Y. Zheng, M. Zakir Ullah, J. Wang, Y. Jiang, C. Xu, Z. Zou, and G. Ding, “Multiview multimodal network for breast cancer diagnosis in contrast-enhanced spectral mammography images,” _International Journal of Computer Assisted Radiology and Surgery_ , vol. 16, no. 6, pp. 979–988, 2021.
  * [44] J. Yap, W. Yolland, and P. Tschandl, “Multimodal skin lesion classification using deep learning,” _Experimental dermatology_ , vol. 27, no. 11, pp. 1261–1267, 2018.
  * [45] X. Ma and F. Jia, “Brain tumor classification with multimodal MR and pathology images,” in _Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 5th International Workshop, BrainLes 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Revised Selected Papers, Part II 5_. Springer, 2020, pp. 343–352.
  * [46] C. Jansen, R. N. Ramirez, N. C. El-Ali, D. Gomez-Cabrero, J. Tegner, M. Merkenschlager, A. Conesa, and A. Mortazavi, “Building gene regulatory networks from scATAC-seq and scRNA-seq using linked self organizing maps,” _PLoS computational biology_ , vol. 15, no. 11, p. e1006555, 2019.
  * [47] S. G. Stark, J. Ficek, F. Locatello, X. Bonilla, S. Chevrier, F. Singer, G. Rätsch, and K.-V. Lehmann, “SCIM: universal single-cell matching with unpaired feature sets,” _Bioinformatics_ , vol. 36, no. Supplement_2, pp. i919–i927, 2020.
  * [48] Z.-J. Cao and G. Gao, “Multi-omics single-cell data integration and regulatory inference with graph-linked embedding,” _Nature Biotechnology_ , vol. 40, no. 10, pp. 1458–1466, 2022.
  * [49] Z. Zhang, C. Yang, and X. Zhang, “scDART: integrating unmatched scRNA-seq and scATAC-seq data and learning cross-modality relationship simultaneously,” _Genome Biology_ , vol. 23, no. 1, p. 139, 2022.
  * [50] K. D. Yang, A. Belyaeva, S. Venkatachalapathy, K. Damodaran, A. Katcoff, A. Radhakrishnan, G. Shivashankar, and C. Uhler, “Multi-domain translation between single-cell imaging and sequencing data using autoencoders,” _Nature communications_ , vol. 12, no. 1, p. 31, 2021.
  * [51] Y. Xu, P. Das, and R. P. McCord, “SMILE: mutual information learning for integration of single-cell omics data,” _Bioinformatics_ , vol. 38, no. 2, pp. 476–486, 2022.
  * [52] M. M. Li, K. Huang, and M. Zitnik, “Graph representation learning in biomedicine and healthcare,” _Nature Biomedical Engineering_ , pp. 1–17, 2022.
  * [53] H. Farooq, Y. Chen, T. T. Georgiou, A. Tannenbaum, and C. Lenglet, “Network curvature as a hallmark of brain structural connectivity,” _Nature communications_ , vol. 10, no. 1, pp. 1–11, 2019.
  * [54] A. Derrow-Pinion, J. She, D. Wong, O. Lange, T. Hester, L. Perez, M. Nunkesser, S. Lee, X. Guo, B. Wiltshire _et al._ , “ETA prediction with graph neural networks in google maps,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 3767–3776.
  * [55] A. Waqas, H. Farooq, N. C. Bouaynaya, and G. Rasool, “Exploring robust architectures for deep artificial neural networks,” _Communications Engineering_ , vol. 1, no. 1, p. 46, 2022.
  * [56] L. Waikhom and R. Patgiri, “A survey of graph neural networks in various learning paradigms: methods, applications, and challenges,” _Artificial Intelligence Review_ , pp. 1–70, 2022.
  * [57] T. Yang, L. Hu, C. Shi, H. Ji, X. Li, and L. Nie, “HGAT: Heterogeneous graph attention networks for semi-supervised short text classification,” _ACM Transactions on Information Systems (TOIS)_ , vol. 39, no. 3, pp. 1–29, 2021.
  * [58] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A Comprehensive Survey on Graph Neural Networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, pp. 4–24, 2020.
  * [59] L. Jiao, J. Chen, F. Liu, S. Yang, C. You, X. Liu, L. Li, and B. Hou, “Graph representation learning meets computer vision: A survey,” _IEEE Transactions on Artificial Intelligence_ , 2022.
  * [60] W. L. Hamilton, “Graph representation learning,” _Synthesis Lectures on Artifical Intelligence and Machine Learning_ , vol. 14, no. 3, 2020.
  * [61] Y. Ma, X. Liu, T. Zhao, Y. Liu, J. Tang, and N. Shah, “A unified view on graph neural networks as graph signal denoising,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 1202–1211.
  * [62] D. Jin, C. Huo, J. Dang, P. Zhu, W. Zhang, W. Pedrycz, and L. Wu, “Heterogeneous graph neural networks using self-supervised reciprocally contrastive learning,” _arXiv preprint arXiv:2205.00256_ , 2022.
  * [63] H.-C. Yi, Z.-H. You, D.-S. Huang, and C. K. Kwoh, “Graph representation learning in bioinformatics: trends, methods and applications,” _Briefings in Bioinformatics_ , vol. 23, no. 1, p. bbab340, 2022.
  * [64] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [65] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in _International conference on machine learning_. PMLR, 2017, pp. 1263–1272.
  * [66] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” _arXiv preprint arXiv:1710.10903_ , 2017.
  * [67] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [68] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [69] J. Park, J. Cho, H. J. Chang, and J. Y. Choi, “Unsupervised hyperbolic representation learning via message passing auto-encoders,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5516–5526.
  * [70] L. Ma, Z. Yang, Y. Miao, J. Xue, M. Wu, L. Zhou, and Y. Dai, “NeuGraph: Parallel Deep Neural Network Computation on Large Graphs.” in _USENIX Annual Technical Conference_ , 2019, pp. 443–458.
  * [71] A. Sankar, Y. Wu, L. Gou, W. Zhang, and H. Yang, “Dynamic graph representation learning via self-attention networks,” _arXiv preprint arXiv:1812.09430_ , 2018.
  * [72] S. Bai, F. Zhang, and P. H. Torr, “Hypergraph convolution and hypergraph attention,” _Pattern Recognition_ , vol. 110, p. 107637, 2021.
  * [73] Y. Wei, X. Wang, L. Nie, X. He, R. Hong, and T.-S. Chua, “MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video,” in _Proceedings of the 27th ACM international conference on multimedia_ , 2019, pp. 1437–1445.
  * [74] Y. Ma and J. Tang, _Deep learning on graphs_. Cambridge University Press, 2021.
  * [75] R. J. Chen, M. Y. Lu, J. Wang, D. F. Williamson, S. J. Rodig, N. I. Lindeman, and F. Mahmood, “Pathomic Fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 4, pp. 757–770, 2020.
  * [76] G. Singh, S. Manjila, N. Sakla, A. True, A. H. Wardeh, N. Beig, A. Vaysberg, J. Matthews, P. Prasanna, and V. Spektor, “Radiomics and radiogenomics in gliomas: a contemporary update,” _British Journal of Cancer_ , vol. 125, no. 5, pp. 641–657, 2021.
  * [77] D. Ahmedt-Aristizabal, M. A. Armin, S. Denman, C. Fookes, and L. Petersson, “Graph-based deep learning for medical diagnosis and analysis: past, present and future,” _Sensors_ , vol. 21, p. 4758, 2021.
  * [78] ——, “A survey on graph-based deep learning for computational histopathology,” _Computerized Medical Imaging and Graphics_ , vol. 95, p. 102027, 2022.
  * [79] D. Anand, S. Gadiya, and A. Sethi, “Histographs: graphs in histopathology,” in _Medical Imaging 2020: Digital Pathology_ , vol. 11320. SPIE, 2020, pp. 150–155.
  * [80] J. Wang, R. J. Chen, M. Y. Lu, A. Baras, and F. Mahmood, “Weakly supervised prostate TMA classification via graph convolutional networks,” in _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2020, pp. 239–243.
  * [81] Y. Wang, Y. G. Wang, C. Hu, M. Li, Y. Fan, N. Otter, I. Sam, H. Gou, Y. Hu, T. Kwok _et al._ , “Cell graph neural networks enable the precise prediction of patient survival in gastric cancer,” _NPJ precision oncology_ , vol. 6, no. 1, p. 45, 2022.
  * [82] S. Mo, M. Cai, L. Lin, R. Tong, Q. Chen, F. Wang, H. Hu, Y. Iwamoto, X.-H. Han, and Y.-W. Chen, “Multimodal priors guided segmentation of liver lesions in MRI using mutual information based graph co-attention networks,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part IV 23_. Springer, 2020, pp. 429–438.
  * [83] H. Du, J. Feng, and M. Feng, “Zoom in to where it matters: a hierarchical graph based model for mammogram analysis,” _arXiv preprint arXiv:1912.07517_ , 2019.
  * [84] C.-H. Chao, Z. Zhu, D. Guo, K. Yan, T.-Y. Ho, J. Cai, A. P. Harrison, X. Ye, J. Xiao, A. Yuille _et al._ , “Lymph node gross tumor volume detection in oncology imaging via relationship learning using graph neural network,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VII 23_. Springer, 2020, pp. 772–782.
  * [85] Z. Tian, X. Li, Y. Zheng, Z. Chen, Z. Shi, L. Liu, and B. Fei, “Graph-convolutional-network-based interactive prostate segmentation in MR images,” _Medical physics_ , vol. 47, no. 9, pp. 4164–4176, 2020.
  * [86] C. Saueressig, A. Berkley, E. Kang, R. Munbodh, and R. Singh, “Exploring graph-based neural networks for automatic brain tumor segmentation,” in _From Data to Models and Back: 9th International Symposium, DataMod 2020, Virtual Event, October 20, 2020, Revised Selected Papers 9_. Springer, 2021, pp. 18–37.
  * [87] A. Fout, J. Byrd, B. Shariat, and A. Ben-Hur, “Protein interface prediction using graph convolutional networks,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [88] Y.-H. Feng, S.-W. Zhang, and J.-Y. Shi, “DPDDI: a deep predictor for drug-drug interactions,” _BMC bioinformatics_ , vol. 21, pp. 1–15, 2020.
  * [89] P. Li, J. Wang, Y. Qiao, H. Chen, Y. Yu, X. Yao, P. Gao, G. Xie, and S. Song, “An effective self-supervised framework for learning expressive molecular global representations to drug discovery,” _Briefings in Bioinformatics_ , vol. 22, no. 6, p. bbab109, 2021.
  * [90] E. Choi, M. T. Bahadori, L. Song, W. F. Stewart, and J. Sun, “GRAM: graph-based attention model for healthcare representation learning,” in _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2017, pp. 787–795.
  * [91] Q. Song, J. Su, and W. Zhang, “scGCN is a graph convolutional networks algorithm for knowledge transfer in single cell omics,” _Nature communications_ , vol. 12, no. 1, p. 3826, 2021.
  * [92] J. Wang, A. Ma, Y. Chang, J. Gong, Y. Jiang, R. Qi, C. Wang, H. Fu, Q. Ma, and D. Xu, “scGNN is a novel graph neural network framework for single-cell RNA-Seq analyses,” _Nature communications_ , vol. 12, no. 1, p. 1882, 2021.
  * [93] H. Cui, P. Xuan, Q. Jin, M. Ding, B. Li, B. Zou, Y. Xu, B. Fan, W. Li, J. Yu _et al._ , “Co-graph attention reasoning based imaging and clinical features integration for lymph node metastasis prediction,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24_. Springer, 2021, pp. 657–666.
  * [94] Y. Huang and A. C. Chung, “Edge-variational graph convolutional networks for uncertainty-aware disease prediction,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VII 23_. Springer, 2020, pp. 562–572.
  * [95] T. Wang, W. Shao, Z. Huang, H. Tang, J. Zhang, Z. Ding, and K. Huang, “MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification,” _Nature communications_ , vol. 12, no. 1, p. 3445, 2021.
  * [96] J. Lian, J. Deng, E. S. Hui, M. Koohi-Moghadam, Y. She, C. Chen, and V. Vardhanabhuti, “Early stage NSCLS patients’ prognostic prediction with multi-information using transformer and graph neural network model,” _eLife_ , vol. 11, p. e80547, oct 2022.
  * [97] H. Wen, J. Ding, W. Jin, Y. Wang, Y. Xie, and J. Tang, “Graph neural networks for multimodal single-cell data integration,” in _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 4153–4163.
  * [98] P. Pati, G. Jaume, L. A. Fernandes, A. Foncubierta-Rodríguez, F. Feroce, A. M. Anniciello, G. Scognamiglio, N. Brancati, D. Riccio, M. Di Bonito _et al._ , “HACT-Net: A Hierarchical Cell-to-Tissue Graph Neural Network for Histopathological Image Classification,” in _Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis: Second International Workshop, UNSURE 2020, and Third International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 8, 2020, Proceedings 2_. Springer, 2020, pp. 208–219.
  * [99] J. Rao, X. Zhou, Y. Lu, H. Zhao, and Y. Yang, “Imputing single-cell RNA-seq data by combining graph convolution and autoencoder neural networks,” _Iscience_ , vol. 24, no. 5, p. 102393, 2021.
  * [100] Y. Zeng, X. Zhou, J. Rao, Y. Lu, and Y. Yang, “Accurately clustering single-cell RNA-seq data by capturing structural relations between cells through graph convolutional network,” in _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2020.
  * [101] M. Chatzianastasis, M. Vazirgiannis, and Z. Zhang, “Explainable multilayer graph neural network for cancer gene prediction,” _arXiv preprint arXiv:2301.08831_ , 2023.
  * [102] M. Tortora, E. Cordelli, R. Sicilia, L. Nibid, E. Ippolito, G. Perrone, S. Ramella, and P. Soda, “Radiopathomics: Multimodal learning in non-small cell lung cancer for adaptive radiotherapy,” _IEEE Access_ , 2023.
  * [103] M. Zitnik, M. Agrawal, and J. Leskovec, “Modeling polypharmacy side effects with graph convolutional networks,” _Bioinformatics_ , vol. 34, no. 13, pp. i457–i466, 2018.
  * [104] T. Nguyen, H. Le, T. P. Quinn, T. Nguyen, T. D. Le, and S. Venkatesh, “GraphDTA: predicting drug-target binding affinity with graph neural networks,” _Bioinformatics_ , vol. 37, no. 8, pp. 1140–1147, 2021.
  * [105] J. Shi, R. Wang, Y. Zheng, Z. Jiang, and L. Yu, “Graph convolutional networks for cervical cell classification,” in _MICCAI 2019 Computational Pathology Workshop COMPAY_ , 2019.
  * [106] Y.-D. Zhang, S. C. Satapathy, D. S. Guttery, J. M. Górriz, and S.-H. Wang, “Improved breast cancer classification through combining graph convolutional network and convolutional neural network,” _Information Processing & Management_, vol. 58, no. 2, p. 102439, 2021.
  * [107] B. Rozemberczki, A. Gogleva, S. Nilsson, G. Edwards, A. Nikolov, and E. Papa, “MOOMIN: Deep Molecular Omics Network for Anti-Cancer Drug Combination Therapy,” in _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, 2022, pp. 3472–3483.
  * [108] D. Leng, L. Zheng, Y. Wen, Y. Zhang, L. Wu, J. Wang, M. Wang, Z. Zhang, S. He, and X. Bo, “A benchmark study of deep learning-based multi-omics data fusion methods for cancer,” _Genome Biology_ , vol. 23, no. 1, pp. 1–32, 2022.
  * [109] N. Rajadhyaksha and A. Chitkara, “Graph contrastive learning for multi-omics data,” _arXiv preprint arXiv:2301.02242_ , 2023.
  * [110] M.-K. Park, J.-M. Lim, J. Jeong, Y. Jang, J.-W. Lee, J.-C. Lee, H. Kim, E. Koh, S.-J. Hwang, H.-G. Kim _et al._ , “Deep-Learning Algorithm and Concomitant Biomarker Identification for NSCLC Prediction Using Multi-Omics Data Integration,” _Biomolecules_ , vol. 12, p. 1839, 2022.
  * [111] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [112] D. W. Otter, J. R. Medina, and J. K. Kalita, “A survey of the usages of deep learning for natural language processing,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, pp. 604–624, 2021.
  * [113] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” 2020.
  * [114] N. Zhang, “Learning adversarial transformer for symbolic music generation,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–10, 2020.
  * [115] S. Ahmed, I. E. Nielsen, A. Tripathi, S. Siddiqui, G. Rasool, and R. P. Ramachandran, “Transformers in time-series analysis: A tutorial,” _arXiv preprint arXiv:2205.01138_ , 2022.
  * [116] J. Ma, J. Liu, Q. Lin, B. Wu, Y. Wang, and Y. You, “Multitask learning for visual question answering,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–15, 2021.
  * [117] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [118] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, jan 2020.
  * [119] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” in _Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * [120] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé _et al._ , “BLOOM: A 176B-Parameter Open-Access Multilingual Language Model,” _arXiv preprint arXiv:2211.05100_ , 2022.
  * [121] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 4904–4916.
  * [122] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, “Coca: Contrastive captioners are image-text foundation models,” _Transactions on Machine Learning Research_ , 2022. [Online]. Available: https://openreview.net/forum?id=Ee277P3AYC
  * [123] R. Hu and A. Singh, “UniT: Multimodal Multitask Learning With a Unified Transformer,” 2021.
  * [124] J. Tang, K. Li, M. Hou, X. Jin, W. Kong, Y. Ding, and Q. Zhao, “MMT: Multi-way Multi-modal Transformer for Multimodal Learning,” in _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_. International Joint Conferences on Artificial Intelligence Organization, 2022, pp. 3458–3465.
  * [125] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: a Visual Language Model for Few-Shot Learning,” 2022.
  * [126] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira, “Perceiver IO: A General Architecture for Structured Inputs & Outputs,” 2021.
  * [127] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, Z. Yang, Y. Zhang, and D. Tao, “A Survey on Vision Transformer,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 1, pp. 87–110, 2023.
  * [128] A. Galassi, M. Lippi, and P. Torroni, “Attention in Natural Language Processing,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 10, pp. 4291–4308, 2021.
  * [129] R. J. Chen, M. Y. Lu, W.-H. Weng, T. Y. Chen, D. F. Williamson, T. Manz, M. Shady, and F. Mahmood, “Multimodal co-attention transformer for survival prediction in gigapixel whole slide images,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 3995–4005.
  * [130] Y. Xie, J. Zhang, C. Shen, and Y. Xia, “CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation,” 2021.
  * [131] J. Shang, T. Ma, C. Xiao, and J. Sun, “Pre-training of graph augmented transformers for medication recommendation,” 2019.
  * [132] L. Rasmy, Y. Xiang, Z. Xie, C. Tao, and D. Zhi, “Med-BERT: Pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction,” _npj Digital Medicine_ , vol. 4, no. 1, 2021.
  * [133] E. Kaczmarek, A. Jamzad, T. Imtiaz, J. Nanayakkara, N. Renwick, and P. Mousavi, “Multi-omic graph transformers for cancer classification and interpretation,” in _PACIFIC SYMPOSIUM ON BIOCOMPUTING 2022_. World Scientific, 2021, pp. 373–384.
  * [134] Y. Ji, Z. Zhou, H. Liu, and R. V. Davuluri, “DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome,” _bioRxiv_ , 2020.
  * [135] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “BioBERT: a pre-trained biomedical language representation model for biomedical text mining,” _Bioinformatics_ , vol. 36, no. 4, pp. 1234–1240, sep 2019.
  * [136] M. E. Kalfaoglu, S. Kalkan, and A. A. Alatan, “Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition,” in _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_. Springer, 2020, pp. 731–747.
  * [137] Z. Zhong, D. Schneider, M. Voit, R. Stiefelhagen, and J. Beyerer, “Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation,” in _2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2023, pp. 6057–6066.
  * [138] P. Li, J. Gu, J. Kuen, V. I. Morariu, H. Zhao, R. Jain, V. Manjunatha, and H. Liu, “SelfDoc: Self-Supervised Document Representation Learning,” 2021.
  * [139] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” 2021.
  * [140] H. Zhu, X. Sun, Y. Li, K. Ma, S. K. Zhou, and Y. Zheng, “DFTR: Depth-supervised Fusion Transformer for Salient Object Detection,” 2022.
  * [141] L. Yang, T. L. J. Ng, B. Smyth, and R. Dong, “HTML: Hierarchical Transformer-Based Multi-Task Learning for Volatility Prediction,” in _Proceedings of The Web Conference 2020_ , ser. WWW ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 441–451.
  * [142] B. Zhao, M. Gong, and X. Li, “Hierarchical multimodal transformer to summarize videos,” _Neurocomputing_ , vol. 468, pp. 360–369, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231221015253
  * [143] H. Sun, J. Liu, S. Chai, Z. Qiu, L. Lin, X. Huang, and Y. Chen, “Multi-Modal Adaptive Fusion Transformer Network for the estimation of depression level,” _Sensors_ , vol. 21, no. 14, p. 4764, 2021.
  * [144] Z. Shao, H. Bian, Y. Chen, Y. Wang, J. Zhang, X. Ji _et al._ , “TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [145] J. Liang, C. Yang, M. Zeng, and X. Wang, “TransConver: transformer and convolution parallel network for developing automatic brain tumor segmentation in MRI images,” _Quantitative Imaging in Medicine and Surgery_ , vol. 12, no. 4, 2022.
  * [146] Y. Ektefaie, G. Dasoulas, A. Noori, M. Farhat, and M. Zitnik, “Geometric multimodal representation learning,” _arXiv preprint arXiv:2209.03299_ , 2023.
  * [147] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 843–852.
  * [148] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “YFCC100M: The new data in multimedia research,” _Communications of the ACM_ , vol. 59, pp. 64–73, 2016.
  * [149] P. Jiang, S. Sinha, K. Aldape, S. Hannenhalli, C. Sahinalp, and E. Ruppin, “Big data in basic and translational cancer research,” _Nature Reviews Cancer_ , vol. 22, no. 11, pp. 625–639, 2022.
  * [150] J. Lipkova, R. J. Chen, B. Chen, M. Y. Lu, M. Barbieri, D. Shao, A. J. Vaidya, C. Chen, L. Zhuang, D. F. Williamson, M. Shaban, T. Y. Chen, and F. Mahmood, “Artificial intelligence for multimodal data integration in oncology,” _Cancer Cell_ , vol. 40, pp. 1095–1110, 2022.
  * [151] M. Zhao, X. Huang, J. Jiang, L. Mou, D.-M. Yan, and L. Ma, “Accurate Registration of Cross-Modality Geometry via Consistent Clustering,” _IEEE Transactions on Visualization and Computer Graphics_ , 2023.
  * [152] P. P. Liang, A. Zadeh, and L.-P. Morency, “Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions,” _arXiv preprint arXiv:2209.03430_ , 2022.
  * [153] H. Wei, L. Feng, X. Chen, and B. An, “Combating noisy labels by agreement: A joint training method with co-regularization,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 13 726–13 735.
  * [154] B. Mirza, W. Wang, J. Wang, H. Choi, N. C. Chung, and P. Ping, “Machine learning and integrative analysis of biomedical big data,” _Genes_ , vol. 10, no. 2, p. 87, 2019.
  * [155] P. Li, Y. Yang, M. Pagnucco, and Y. Song, “Explainability in graph neural networks: An experimental survey,” _arXiv preprint arXiv:2203.09258_ , 2022.
  * [156] Z. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec, “GNNExplainer: Generating Explanations for Graph Neural Networks,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [157] M. Vu and M. T. Thai, “PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks,” _Advances in neural information processing systems_ , vol. 33, pp. 12 225–12 235, 2020.
  * [158] H. Yuan, H. Yu, J. Wang, K. Li, and S. Ji, “On explainability of graph neural networks via subgraph explorations,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 12 241–12 252.
  * [159] E. Remmer, “Explainability methods for transformer-based artificial neural networks: a comparative analysis,” 2022.
  * [160] H. Zhang, B. Wu, X. Yuan, S. Pan, H. Tong, and J. Pei, “Trustworthy graph neural networks: Aspects, methods and trends,” _arXiv preprint arXiv:2205.07424_ , 2022.
  * [161] D. Valsesia, G. Fracastoro, and E. Magli, “RAN-GNNs: Breaking the Capacity Limits of Graph Neural Networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [162] A. Javaloy, M. Meghdadi, and I. Valera, “Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 9938–9964.
  * [163] Y. Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, “Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably),” in _International Conference on Machine Learning_. PMLR, 2022, pp. 9226–9259.
  * [164] C. Fritz, E. Dorigatti, and D. Rügamer, “Combining graph neural networks and spatio-temporal disease models to improve the prediction of weekly covid-19 cases in germany,” _Scientific Reports_ , vol. 12, no. 1, p. 3930, 2022.
  * [165] S. Pati, U. Baid, B. Edwards, M. Sheller, S.-H. Wang, G. A. Reina, P. Foley, A. Gruzdev, D. Karkada, C. Davatzikos _et al._ , “Federated learning enables big data for rare cancer boundary detection,” _Nature communications_ , vol. 13, no. 1, p. 7346, 2022.
  * [166] S. Ahmed, D. Dera, S. U. Hassan, N. Bouaynaya, and G. Rasool, “Failure detection in deep neural networks for medical imaging,” _Frontiers in Medical Technology_ , vol. 4, 2022.
  * [167] D. Dera, N. C. Bouaynaya, G. Rasool, R. Shterenberg, and H. M. Fathallah-Shaykh, “PremiUm-CNN: Propagating Uncertainty Towards Robust Convolutional Neural Networks,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 4669–4684, 2021.