11institutetext: University of Pittsburgh, Pittsburgh, PA 15260, USA
22institutetext: Arizona State University, Tempe, AZ 85281, USA
33institutetext: University of California San Francisco, San Francisco, CA
94143, USA 44institutetext: University of Illinois Chicago, Chicago, IL 60607,
USA 55institutetext: University of Southern California, Los Angeles, CA 90007,
USA

# Functional2Structural: Cross-Modality Brain Networks Representation Learning

Haoteng Tang 11 Xiyao Fu 11 Lei Guo 11 Yalin Wang 22 Scott Mackin 33 Olusola
Ajilore 44 Alex Leow 44 Paul Thompson 55 Heng Huang 11 Liang Zhan 11

###### Abstract

MRI-based modeling of brain networks has been widely used to understand
functional and structural interactions and connections among brain regions,
and factors that affect them, such as brain development and disease. Graph
mining on brain networks may facilitate the discovery of novel biomarkers for
clinical phenotypes and neurodegenerative diseases. Since brain networks
derived from functional and structural MRI describe the brain topology from
different perspectives, exploring a representation that combines these cross-
modality brain networks is non-trivial. Most current studies aim to extract a
fused representation of the two types of brain network by projecting the
structural network to the functional counterpart. Since the functional network
is dynamic and the structural network is static, mapping a static object to a
dynamic object is suboptimal. However, mapping in the opposite direction is
not feasible due to the non-negativity requirement of current graph learning
techniques. Here, we propose a novel graph learning framework, known as Deep
Signed Brain Networks (DSBN), with a signed graph encoder that, from an
opposite perspective, learns the cross-modality representations by projecting
the functional network to the structural counterpart. We validate our
framework on clinical phenotype and neurodegenerative disease prediction tasks
using two independent, publicly available datasets (HCP and OASIS). The
experimental results clearly demonstrate the advantages of our model compared
to several state-of-the-art methods.

###### Keywords:

Signed Graph Learning Multimodal Structural Network Functional Network
Reconstruction Prediction

##  1 Introduction

Recent years have witnessed great progress in applying graph theory to study
MRI-derived brain networks, to discover novel biomarkers for clinical
phenotypes or neurodegenerative diseases (e.g., Alzheimer’s disease or AD)
[12, 26, 4]. Different MRI techniques can be used to reconstruct brain
networks corresponding to different aspects of the brain organization or
dynamics. For example, functional MRI-derived functional networks provide
measures of BOLD relationships over time between brain regions but not the
existence of a physical direct link between these regions. By contrast, the
diffusion MRI-derived structural network can describe the connectivity of
white matter tracts between brain regions, yet does not inform us about
whether this tract, or the regions it connects, are “activated” or “not
activated” in a specific state. Therefore, these brain networks provide
distinct but complementary information, and separately analyzing each of these
networks will always be suboptimal.

Graph neural networks (GNNs) [17] have gained enormous attention recently.
Although some progress [16, 19] has been made by applying GNNs to brain
networks, most existing techniques have been developed based on single-
modality brain networks. It is well known that a high-level dependency, based
on network communications, exists between brain structural and functional
networks [24, 3], which has motivated many studies [31, 14, 10] to integrate
multimodal brain networks by constructing a projection between them using GNNs
(e.g., [31]). Most existing studies [31, 14, 10] aim to reconstruct functional
networks from brain structural counterparts. However, mapping a static object
(structural network) to a dynamic one (functional network) is not optimal,
even though the structural network may be better suited to serve as the
template. Very few studies aim to construct the opposite mapping (i.e., from
functional to structural networks); this may be because current GNNs can only
encode unsigned graphs (i.e., structural networks in which all edge weights
have non-negative values). The brain functional network is a signed graph by
definition (including entries that denote negative correlations); as such,
network edge weights can have negative values which will destroy the
information aggregation mechanism in current GNNs.

To tackle this, we propose a new framework to encode signed graphs, and we
apply this framework to project and encode brain functional networks to the
corresponding structural networks. Our results demonstrate that the extracted
latent network representations from this mapping can be used for clinical
tasks (regression or classification) with better performance, compared with
baseline methods. Our contributions are three-fold: (1) We propose an end-to-
end network representation framework to model brain structural networks based
on functional ones; this is the first work on this topic to the best of our
knowledge. (2) We propose a signed graph encoder to embed the functional brain
networks. (3) We draw graph saliency maps for clinical tasks, to enable
phenotypic and disease-related biomarker detection and aid in interpretation.

##  2 Preliminaries of Signed Brain Networks

A brain network is an attributed and weighted graph $G=\\{V,E\\}=(A,X)$ with
$N$ nodes, where $V=\\{v_{i}\\}_{i=1}^{N}$ is the set of graph nodes
representing brain regions, and $E=\\{e_{i,j}\\}$ is the edge set.
$X\in\mathcal{R}^{N\times c}$ is the node feature matrix where
$x_{i}\in\mathbb{R}^{1\times c}$ is the $i-$th row of $X$ representing the
node feature of $v_{i}$. $A\in\mathbb{R}^{N\times N}$ is the adjacency matrix
where $a_{i,j}\in\mathbb{R}$ represents the weights of the edge between
$v_{i}$ and $v_{j}$. In signed brain networks (i.e., functional brain
networks), $a_{i,j}\in(-\infty,+\infty)$, while in unsigned brain networks
(i.e., structural brain networks), $a_{i,j}\in[0,+\infty)$. We use
$G^{F}=(A^{F},X^{F})$ and $G^{S}=(A^{S},X^{S})$ to represent the functional
and structural brain networks, respectively. Given a specific node $v_{i}$ in
a signed network, we define their positive and negative neighbors set as
$\mathcal{N}^{+}_{i}$ and $\mathcal{N}^{-}_{i}$, respectively. Following the
balance theory [7, 8, 13, 22], any node $v_{j}$ belongs to the balanced set
(denoted as $\mathbf{\Gamma}$) of $v_{i}$ if the path between $v_{i}$ and
$v_{i}$ contains even number of negative edges. Otherwise, $v_{j}$ belongs to
the unbalanced set (denoted as $\mathbf{\Upsilon}$) of $v_{i}$.

##  3 Methodology

We first introduce the proposed multi-head signed graph encoder (SGE). Then,
we present an end-to-end framework with the proposed encoder to reconstruct
structural networks from functional networks and perform downstream tasks.

Figure 1: Pipeline of the DSBN framework, including a signed graph encoder (in
red dash blox) scheme with two branches (BUE and PNE encoder heads in yellow
and blue box) for functional network embedding, an inner-product decoder for
structural network reconstruction and a downstream task branch for
classification and regression.

###  3.1 Signed Graph Encoder (SGE)

Our SGE consists of a balanced-unbalanced encoder (BUE) head and a positive-
negative encoder (PNE) head. Motivated by the balance theory in Section 2, the
BUE head encodes the graph node to the latent features with balanced and
unbalanced components. Meanwhile, we split the signed graph into a positive
sub-graph and a negative counterpart. The PNE head encodes these two sub-
graphs to generate the node latent features with positive and negative
components. The yellow and blue boxes in Fig. 1 illustrate the BUE and PNE,
respectively.

Balanced-Unbalanced Encoder (BUE). We use $T$ to denote the encoder layer
number and the $T-th$ layer of the encoder focuses on aggregating the $T-th$
hop neighbors of node $v_{i}$. To improve the generalization of local
information aggregation in brain networks, a dynamic and fluctuating
adjustment of aggregation weights (i.e., brain network edge weights) is
arguably more reasonable and has been advocated during graph learning [31]. To
this end, our BUE is designed based on the concept of graph attention [28].

Initial Layer ($T=1$) We initialize balanced and unbalanced node latent
feature components by:

$\displaystyle
x_{i}^{\mathbf{\Gamma}(1)}=\sigma(\sum_{j\in\mathcal{N}_{i}^{+}}\alpha_{i,j}^{\mathbf{\Gamma}(1)}x^{(0)}_{j}W^{\mathbf{\Gamma}(1)}),\quad
x_{i}^{\mathbf{\Upsilon}(1)}=\sigma(\sum_{j\in\mathcal{N}_{i}^{-}}\alpha_{i,j}^{\mathbf{\Upsilon}(1)}x^{(0)}_{j}W^{\mathbf{\Upsilon}(1)}),$
(1)

where $\sigma$ is a nonlinear activation function. $x^{(0)}=x$ is input node
features. $W^{\mathbf{\Gamma}(1)}$ and $W^{\mathbf{\Upsilon}(1)}$ are
trainable weights to generate two latent feature components.
$\alpha_{i,j}^{\mathbf{\Gamma}(1)}$ and $\alpha_{i,j}^{\mathbf{\Upsilon}(1)}$
are attention scores of brain nodes from the balanced and unbalanced set
respectively. Following the work in [28], we first compute the attention
coefficients by:

$\displaystyle
e_{i,j}^{\mathbf{\Gamma}(1)}=a[x^{(0)}_{i}W^{\mathbf{\Gamma}(1)},x^{(0)}_{j}W^{\mathbf{\Gamma}(1)}],\quad
e_{i,j}^{\mathbf{\Upsilon}(1)}=a[x^{(0)}_{i}W^{\mathbf{\Upsilon}(1)},x^{(0)}_{j}W^{\mathbf{\Upsilon}(1)}]$
(2)

where $a$ is a trainable attentional weight vector and $[,]$ is a
concatenation operator. Based on the attention coefficients, we derive the
attention scores by:

$\displaystyle\alpha_{i,j}^{\mathbf{\Gamma}(1)}=\frac{exp(e_{i,j}^{\mathbf{\Gamma}(1)})}{\sum_{n\in\mathcal{N}_{i}^{+}}exp(e_{i,n}^{\mathbf{\Gamma}(1)})},\quad\alpha_{i,j}^{\mathbf{\Upsilon}(1)}=\frac{exp(e_{i,j}^{\mathbf{\Upsilon}(1)})}{\sum_{n\in\mathcal{N}_{i}^{-}}exp(e_{i,n}^{\mathbf{\Upsilon}(1)})}$
(3)

Following Layers ($T>1$) In the subsequent layers ($T>1$),
$x_{i}^{\mathbf{\Gamma}(T)}$ and $x_{i}^{\mathbf{\Upsilon}(T)}$ is generated
by:

$\displaystyle x_{i}^{\mathbf{\Gamma}(T)}$ $\displaystyle=$
$\displaystyle\sigma(\sum_{j\in\mathcal{N}_{i}^{+},k\in\mathcal{N}_{i}^{-}}\alpha_{i,j}^{\mathbf{\Gamma}(T)}x^{\mathbf{\Gamma}(T-1)}_{j}W^{\mathbf{\Gamma}(T)}+\alpha_{i,k}^{\mathbf{\Gamma}(T-1)}x^{\mathbf{\Upsilon}(T-1)}_{k}W^{\mathbf{\Gamma}(T)})$
$\displaystyle x_{i}^{\mathbf{\Upsilon}(T)}$ $\displaystyle=$
$\displaystyle\sigma(\sum_{j\in\mathcal{N}_{i}^{+},k\in\mathcal{N}_{i}^{-}}\alpha_{i,j}^{\mathbf{\Upsilon}(T)}x^{\mathbf{\Upsilon}(T-1)}_{j}W^{\mathbf{\Upsilon}(T)}+\alpha_{i,k}^{\mathbf{\Upsilon}(T-1)}x^{\mathbf{\Gamma}(T-1)}_{k}W^{\mathbf{\Upsilon}(T)})$
(4)

Similarly, we compute $4$ attention coefficients as

$\displaystyle e_{i,j}^{\mathbf{\Gamma}(T)}$ $\displaystyle=$ $\displaystyle
a[x^{\mathbf{\Gamma}(T-1)}_{i}W^{\mathbf{\Gamma}(T)},x^{\mathbf{\Gamma}(T-1)}_{j}W^{\mathbf{\Gamma}(T)}]$
$\displaystyle e_{i,k}^{\mathbf{\Gamma}(T)}$ $\displaystyle=$ $\displaystyle
a[x^{\mathbf{\Upsilon}(T-1)}_{i}W^{\mathbf{\Gamma}(T)},x^{\mathbf{\Upsilon}(T-1)}_{k}W^{\mathbf{\Gamma}(T)}]$
$\displaystyle e_{i,j}^{\mathbf{\Upsilon}(T)}$ $\displaystyle=$ $\displaystyle
a[x^{\mathbf{\Upsilon}(T-1)}_{i}W^{\mathbf{\Upsilon}(T)},x^{\mathbf{\Upsilon}(T-1)}_{j}W^{\mathbf{\Upsilon}(T)}]$
$\displaystyle e_{i,k}^{\mathbf{\Upsilon}(T)}$ $\displaystyle=$ $\displaystyle
a[x^{\mathbf{\Gamma}(T-1)}_{i}W^{\mathbf{\Upsilon}(T)},x^{\mathbf{\Gamma}(T-1)}_{k}W^{\mathbf{\Upsilon}(T)}]$
(5)

Then the $4$ attention scores,
$\alpha_{i,j}^{\mathbf{\Gamma}(T)}=exp(e_{i,j}^{\mathbf{\Gamma}(T)})/\sum_{\mathbf{\Gamma}(T)}$,
$\alpha_{i,k}^{\mathbf{\Gamma}(T)}=exp(e_{i,k}^{\mathbf{\Gamma}(T)})\\\
/\sum_{\mathbf{\Gamma}(T)}$,
$\alpha_{i,j}^{\mathbf{\Upsilon}(T)}=exp(e_{i,j}^{\mathbf{\Upsilon}(T)})/\sum_{\mathbf{\Upsilon}(T)}$
and
$\alpha_{i,k}^{\mathbf{\Upsilon}(T)}=exp(e_{i,k}^{\mathbf{\Upsilon}(T)})/\sum_{\mathbf{\Upsilon}(T)}$,
where
$\sum_{\mathbf{\Gamma}(T)}=\sum_{n\in\mathcal{N}_{i}}exp(e_{i,n}^{\mathbf{\Gamma}(T)})$
and
$\sum_{\mathbf{\Upsilon}(T)}=\sum_{n\in\mathcal{N}_{i}}exp(e_{i,n}^{\mathbf{\Upsilon}(T)})$.

After we obtain the two latent components, we concatenate them as the latent
features generated by the BUE head by:
$x_{i}^{BUE}=[x_{i}^{\mathbf{\Gamma}},x_{i}^{\mathbf{\Upsilon}}]$.

####  3.1.1 Positive-Negative Encoder (PNE)

We split the adjacency matrix of the functional network into positive and
negative sub-network pairs (i.e., $G^{+F}=(A^{+F},X)$ and
$G^{-F}=(A^{-F},X)$). For each sub-network, we forward it into $T$ graph
attention layers [28] to generate the positive and negative latent feature
components (i.e., $x_{i}^{+}$ and $x_{i}^{-}$). As a result, the latent
feature generated by the PNE head can be computed by
$x_{i}^{PNE}=[x_{i}^{+},x_{i}^{-}]$. The final fused latent features generated
by our SGE can be computed by: $x_{i}=[x_{i}^{BUE}W,x_{i}^{PNE}W]$, where $W$
is a set of trainable parameters to control feature dimensions.

###  3.2 Deep Signed Brain Networks (DSBN)

Our DSBN framework is illustrated in Fig. 1, which includes (1) a signed graph
encoder (SGE) to generate the latent node features from brain functional
networks, (2) a decoder to reconstruct brain structural networks from latent
node features, and (3) Multilayer Perceptrons (MLP) for downstream tasks.

####  3.2.1 Structural Network Reconstruction and Downstream Tasks

The structural network edges can be reconstructed by an inner-product
decoder[18] as: $\hat{a}_{i,j}^{S}=sigmoid(x_{i}^{\top}\cdot x_{j})$, where
the $\cdot$ is an inner-product operator, $\top$ is vector transpose.

We use a sum function as a global readout operator to obtain the whole graph
representation (i.e., $X_{G}=\sum_{i=1}^{N}x_{i}$). Then, MLP are used to
generate the final classification or regression output (i.e.,
$\hat{y}=MLP(X_{G})$). Moreover, we use parameters in the last MLP layer and
node latent features $x_{i}$ to generate the brain network saliency map using
the Class Activation Mapping (CAM) approach [23, 2] for the classification
task. The brain saliency map identifies the top-K most important brain regions
associated with each class.

Loss Functions We summarize the loss functions here for our framework.
Reconstruction Loss The ground-truth brain structural network is sparse while
the reconstructed structural network is fully connected. To facilitate network
reconstruction, in the training stage, we add a small perturbation value
($\delta$) to the edge weights of the ground-truth structural networks (i.e.,
$\bar{a}^{S}_{i,j}=a^{S}_{i,j}+\delta$) and build up the reconstruction loss
as
$\mathcal{L}_{recon}=\frac{1}{|E|}\sum_{i,j}(\hat{a}_{i,j}^{S}-\bar{a}_{i,j}^{S})^{2},$
where $|E|$ is the number of edges.

Supervised Loss We deploy our framework on both regression and classification
tasks. For the classification task, we use the negative log likelihood loss
where $\mathcal{L}_{super}=NLLLoss(\hat{y},y)$. For the regression task, we
use $L_{1}$ loss where $\mathcal{L}_{super}=L_{1}Loss(\hat{y},y)$. In summary,
the overall loss function of the DSBN is
$\mathcal{L}_{all}=\eta_{1}\mathcal{L}_{recon}+\eta_{2}\mathcal{L}_{super},$
where $\eta_{1}$, $\eta_{2}$ are loss weights.

##  4 Experiments

Figure 2: Cross-modality learning results on the OASIS data. (A) is the
averaged functional network, (B) and (C) are the mean reconstructed and
ground-truth structural networks. (D) is to show that edge weights in the
predicted structural network are significantly correlated with the ground
truth data and $r=0.917$ with $p=0.0129$

###  4.1 Data Description and Preprocessing

Two publicly available datasets were used to evaluate our framework. The first
includes data from $1206$ young healthy subjects (mean age $28.19\pm 7.15$,
$657$ women) from the Human Connectome Project [27] (HCP). The second includes
1326 subjects (mean age $=70.42\pm 8.95$, $738$ women) from the Open Access
Series of Imaging Studies (OASIS) dataset [20]. Details of each dataset may be
found on their official websites. CONN [29] and FSL [15] were used to
reconstruct the functional and structural networks, respectively. For the HCP
data, both networks have a dimension of $82\times 82$ based on 82 ROIs defined
using FreeSurfer (V6.0) [11]. For the OASIS data, both networks have a
dimension of $132\times 132$ based on the Harvard-Oxford Atlas and AAL Atlas.
We deliberately chose different network resolutions for HCP and OASIS, to
evaluate whether the performance of our new framework is affected by the
network dimension or atlas.

###  4.2 Implementation Details

The edge weights of the functional networks and structural networks were first
normalized to the range of $[-1,1]$ and $[0,1]$, respectively. The node
features are initialized as the min, $25\%$, median, $75\%$, max values of the
fMRI bold signal. We randomly split each dataset into $5$ disjoint sets for
$5$-fold cross-validations in the following experiments. The model is trained
using the Adam optimizer with a batch size of $128$. The initial learning rate
is set to $0.001$ and decayed by
$(1-\frac{current\,epoch}{max\,epoch})^{0.9}$. We also regularize the training
with an $L_{2}$ weight decay of $1e^{-5}$. We stop the training if the
validation loss does not improve for $100$ epochs in an epoch termination
condition with a maximum of $500$ epochs, as was done in [21, 25]. The
experiments are deployed on one NVIDIA TITAN RTX GPU.

###  4.3 Brain Structural Network Reconstruction using DSBN

To show the performance of our DSBN on the reconstruction of brain structural
networks, we train the model in a task-free manner where no task-specific
supervised loss is involved. Since the $\delta$ value is set to $0.05$ when we
train the model, we set the reconstructed edge weights to $0$ if the predicted
weights are less than $0.05$. The mean absolute error (MAE) values between the
edge weights in the ground-truth and reconstructed networks are $0.074\pm
0.016$ and $0.039\pm 0.058$ under $5$-fold cross-validation on the OASIS and
HCP data, respectively. The reconstruction results on OASIS data are
visualized in Fig. 2.

Table 1: Classification accuracy, precision, and F1-scores, with their
standard deviation values under 5-fold cross-validation. The best results are
highlighted in bold font.

Method | OASIS (Disease) | HCP (Gender)
---|---|---
| Acc. | Prec. | F1 | Acc. | Prec. | F1
t-BNE | $57.84\pm 2.14$ | $52.26\pm 2.39$ | $55.17\pm 3.15$ | $59.84\pm 2.05$ | $57.77\pm 2.21$ | $55.89\pm 1.96$
MK-SVM | $54.62\pm 3.33$ | $49.86\pm 3.11$ | $55.27\pm 4.15$ | $58.21\pm 3.75$ | $51.51\pm 2.84$ | $59.19\pm 2.40$
DIFFPOOL | $66.29\pm 2.60$ | $63.87\pm 2.14$ | $69.91\pm 2.37$ | $72.12\pm 1.81$ | $68.84\pm 1.97$ | $74.01\pm 2.16$
SAGPOOL | $64.53\pm 2.06$ | $61.76\pm 2.99$ | $68.97\pm 3.07$ | $69.29\pm 1.84$ | $70.18\pm 1.49$ | $67.38\pm 1.12$
BrainChey | $69.26\pm 1.47$ | $71.22\pm 1.95$ | $70.46\pm 2.81$ | $74.11\pm 2.05$ | $76.26\pm 1.77$ | $75.08\pm 2.60$
BrainNet-CNN | $73.37\pm 2.14$ | $74.06\pm 1.59$ | $73.27\pm 1.85$ | $71.88\pm 1.69$ | $70.18\pm 2.21$ | $70.29\pm 2.10$
DSBN w/o BUE | $73.04\pm 3.03$ | $74.74\pm 1.96$ | $73.53\pm 2.23$ | $78.15\pm 2.96$ | $79.01\pm 1.68$ | $80.47\pm 2.02$
DSBN w/o PNE | $75.38\pm 2.52$ | $76.26\pm 2.96$ | $78.68\pm 3.02$ | $80.78\pm 2.44$ | $81.16\pm 1.74$ | $82.98\pm 2.01$
DSBN w/o Recon. | $75.94\pm 2.73$ | $77.09\pm 2.52$ | $75.46\pm 2.17$ | $79.12\pm 2.06$ | $80.55\pm 2.18$ | $81.19\pm 1.96$
DSBN | $78.92\pm 1.38$ | $79.81\pm 1.41$ | $80.22\pm 2.25$ | $82.19\pm 2.01$ | $85.35\pm 1.99$ | $84.71\pm 2.37$

Table 2: Regression Mean Absolute Error (MAE) $\pm$ SD (standard deviation) under 5-fold cross-validation. The best results are highlighted in bold font. | OASIS (MMSE) | HCP (MMSE)
---|---|---
tBNE | $2.39\pm 0.74$ | $2.17\pm 0.48$
SAGPOOL | $1.86\pm 0.27$ | $1.55\pm 0.33$
DIFFPOOL | $1.69\pm 0.36$ | $1.63\pm 0.14$
BrainNet-CNN | $1.40\pm 0.20$ | $1.29\pm 0.06$
BrainChey | $1.12\pm 0.19$ | $1.14\pm 0.25$
DSBN w/o BUE | $1.17\pm 0.22$ | $1.11\pm 0.17$
DSBN w/o PNE | $1.06\pm 0.24$ | $0.86\pm 0.34$
DSBN w/o Recon. | $0.97\pm 0.11$ | $1.03\pm 0.19$
DSBN | $0.87\pm 0.18$ | $0.69\pm 0.21$

###  4.4 Disease and Sex Classification Tasks

Experimental Setup. $6$ baselines were used for comparison, including $2$
traditional graph embedding models (t-BNE [5] and MK-SVM [9]), $2$ deep graph
convolution models designed for brain network embedding (BrainChey [19] and
BrainNet-CNN [16]), and $2$ hierarchical graph neural networks with graph
pooling strategies (DIFFPOOL [30] and SAGPOOL [21]) As mentioned above, the
baseline methods can only embed unsigned graphs. Therefore, we convert the
functional networks to unsigned graphs by using the absolute values of the
edge weights. Meanwhile, $3$ variant models of our DSBN, including DSBN w/o
BUE encoder, DSBN w/o PNE encoder and DSBN w/o reconstruction decoder, are
evaluated for ablation studies. The number of BUE and PNE encoder layers is
set to $3$. We search the loss weights (details in Supplementary) $\eta_{1}$
and $\eta_{2}$ in the range of $[0.01,0.1,0.5,1]$ and $[0.1,1,5]$
respectively, and determine the loss weights as $\eta_{1}=0.1$, $\eta_{2}=1$
for AD classification, $\eta_{1}=0.1$, $\eta_{2}=1$ for sex classification.
The results are reported in terms of classification accuracy, precision and
F1-scores, with their _std_.

Results Classification results for AD on OASIS and sex on HCP are presented in
Table 1, which shows that our model achieves the best accuracy for both tasks,
among all methods. For example, in the AD classification, our model
outperforms the baselines with at least $7.6\%$, $7.8\%$ and $9.4\%$ increases
in accuracy, precision and F1 scores, respectively. In general, the deep graph
models perform better than the traditional graph embedding methods (_i.e._ ,
t-BNE and MK-SVM). When we abandon the cross-modality learning (i.e., DSBN w/o
Recon.), the performance, though comparable to baselines, decreases
significantly. This shows the effectiveness of the cross-modality learning.
The performance will also decrease when we remove BUE encoder or PNE encoder,
which indicates that both balance and polarity properties are important for
embedding functional networks. The brain saliency map is shown in
Supplementary where we identify $10$ key brain regions associated with AD
(from OASIS) and with each sex (in HCP), respectively. The salient regions for
females are concentrated in frontal regions of the brain while males have the
opposite trend, consistent with the finding [6] that women are typically less
aggressive than men, and, on average, less physically strong. For AD, most of
the salient regions are located in subcortical structures, as well as the
bilateral intracalcarine region, the caudate, and planum polare, which have
been implicated as potential AD biomarkers [1].

###  4.5 MMSE Regression

Experimental Setup Mini-Mental State Exam (MMSE) is a quantitative measure of
cognitive status in adults. In the MMSE regression task, the selected
baselines for comparison (except for MK-SVM which is designed only for
classification problems) and the structure of our DSBN remain unchanged. The
loss weights are set to $\eta_{1}=0.5$ and $\eta_{2}=1$ for both datasets.
Details of hyperparameter analysis are provided in the Supplementary. The
regression results are reported as average Mean Absolute Errors (MAE) with
their _std_.

Results The MMSE regression results on the HCP and OASIS datasetsare
summarized in Table 2, which also demonstrates that our model outperforms all
baselines. The regression results also indicate the superiority of the cross-
modality learning and the importance of both BUE and PNE encoders.

##  5 Conclusion

We propose a novel multimodal brain network representation learning framework
with a signed functional network encoder. The cross-modality network embedding
is generated by mapping a functional brain network to its structural
counterpart. The embedded network representations contribute to important
clinical prediction tasks and the brain saliency map may assist with disease-
related biomarker identification. We will explore the bijection between these
two networks in the future.

## References

  * [1] Amoroso, N., La Rocca, M., Bruno, S., Maggipinto, T., Monaco, A., Bellotti, R., Tangaro, S.: Brain structural connectivity atrophy in alzheimer’s disease. arXiv preprint arXiv:1709.02369 (2017)
  * [2] Arslan, S., Ktena, S.I., Glocker, B., Rueckert, D.: Graph saliency maps through spectral convolutional networks: Application to sex classification with brain connectivity. In: Graphs in biomedical image analysis and integrating medical imaging and non-imaging modalities, pp. 3–13. Springer (2018)
  * [3] Bullmore, E., Sporns, O.: The economy of brain network organization. Nature reviews neuroscience 13(5), 336–349 (2012)
  * [4] Calhoun, V.D., Sui, J.: Multimodal fusion of brain imaging data: a key to finding the missing link (s) in complex mental illness. Biological psychiatry: cognitive neuroscience and neuroimaging 1(3), 230–244 (2016)
  * [5] Cao, B., He, L., Wei, X., Xing, M., Yu, P.S., Klumpp, H., Leow, A.D.: t-bne: Tensor-based brain network embedding. In: Proceedings of the 2017 SIAM International Conference on Data Mining. pp. 189–197. SIAM (2017)
  * [6] Carlo, G., Raffaelli, M., Laible, D.J., Meyer, K.A.: Why are girls less physically aggressive than boys? personality and parenting mediators of physical aggression. Sex Roles 40(9), 711–729 (1999)
  * [7] Cartwright, D., Harary, F.: Structural balance: a generalization of heider’s theory. Psychological review 63(5), 277 (1956)
  * [8] Derr, T., Ma, Y., Tang, J.: Signed graph convolutional networks. In: 2018 IEEE International Conference on Data Mining (ICDM). pp. 929–934. IEEE (2018)
  * [9] Dyrba, M., Grothe, M., Kirste, T., Teipel, S.J.: Multimodal analysis of functional and structural disconnection in a lzheimer’s disease using multiple kernel svm. Human brain mapping 36(6), 2118–2131 (2015)
  * [10] Finger, H., Bönstrup, M., Cheng, B., Messé, A., Hilgetag, C., Thomalla, G., Gerloff, C., König, P.: Modeling of large-scale functional brain networks based on structural connectivity from dti: comparison with eeg derived phase coupling networks and evaluation of alternative methods along the modeling path. PLoS computational biology 12(8), e1005025 (2016)
  * [11] Fischl, B.: Freesurfer. Neuroimage 62(2), 774–781 (2012)
  * [12] Hao, X., Xu, D., Bansal, R., Dong, Z., Liu, J., Wang, Z., Kangarlu, A., Liu, F., Duan, Y., Shova, S., et al.: Multimodal magnetic resonance imaging: The coordinated use of multiple, mutually informative probes to understand brain structure and function. Human brain mapping 34(2), 253–271 (2013)
  * [13] Heider, F.: Attitudes and cognitive organization. The Journal of psychology 21(1), 107–112 (1946)
  * [14] Huang, H., Ding, M.: Linking functional connectivity and structural connectivity quantitatively: a comparison of methods. Brain connectivity 6(2), 99–108 (2016)
  * [15] Jenkinson, M., Beckmann, C.F., Behrens, T.E., Woolrich, M.W., Smith, S.M.: Fsl. Neuroimage 62(2), 782–790 (2012)
  * [16] Kawahara, J., Brown, C.J., Miller, S.P., Booth, B.G., Chau, V., Grunau, R.E., Zwicker, J.G., Hamarneh, G.: Brainnetcnn: Convolutional neural networks for brain networks; towards predicting neurodevelopment. NeuroImage 146, 1038–1049 (2017)
  * [17] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
  * [18] Kipf, T.N., Welling, M.: Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016)
  * [19] Ktena, S.I., Parisot, S., Ferrante, E., Rajchl, M., Lee, M., Glocker, B., Rueckert, D.: Metric learning with spectral graph convolutions on brain connectivity networks. NeuroImage 169, 431–442 (2018)
  * [20] LaMontagne, P.J., Benzinger, T.L., Morris, J.C., Keefe, S., Hornbeck, R., Xiong, C., Grant, E., Hassenstab, J., Moulder, K., Vlassenko, A.G., et al.: Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. MedRxiv (2019)
  * [21] Lee, J., Lee, I., Kang, J.: Self-attention graph pooling. In: International conference on machine learning. pp. 3734–3743. PMLR (2019)
  * [22] Li, Y., Tian, Y., Zhang, J., Chang, Y.: Learning signed network embedding via graph attention. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 4772–4779 (2020)
  * [23] Pope, P.E., Kolouri, S., Rostami, M., Martin, C.E., Hoffmann, H.: Explainability methods for graph convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10772–10781 (2019)
  * [24] Rusinek, H., De Santi, S., Frid, D., Tsui, W.H., Tarshish, C.Y., Convit, A., de Leon, M.J.: Regional brain atrophy rate predicts future cognitive decline: 6-year longitudinal mr imaging study of normal aging. Radiology 229(3), 691–696 (2003)
  * [25] Shchur, O., Mumme, M., Bojchevski, A., Günnemann, S.: Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 (2018)
  * [26] Uludağ, K., Roebroeck, A.: General overview on the merits of multimodal neuroimaging data fusion. Neuroimage 102, 3–10 (2014)
  * [27] Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K., Consortium, W.M.H., et al.: The wu-minn human connectome project: an overview. Neuroimage 80, 62–79 (2013)
  * [28] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
  * [29] Whitfield-Gabrieli, S., Nieto-Castanon, A.: Conn: a functional connectivity toolbox for correlated and anticorrelated brain networks. Brain connectivity 2(3), 125–141 (2012)
  * [30] Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., Leskovec, J.: Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems 31 (2018)
  * [31] Zhang, W., Zhan, L., Thompson, P., Wang, Y.: Deep representation learning for multimodal brain networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 613–624. Springer (2020)