# Graph Multi-Similarity Learning for Molecular Property Prediction

Hao Xu  Zhengyang Zhou  Pengyu Hong

###### Abstract

Enhancing accurate molecular property prediction relies on effective and
proficient representation learning. It is crucial to incorporate diverse
molecular relationships characterized by multi-similarity (self-similarity and
relative similarities) (Wang et al., 2019) between molecules. However, current
molecular representation learning methods fall short in exploring multi-
similarity and often underestimate the complexity of relationships between
molecules. Additionally, previous multi-similarity approaches require the
specification of positive and negative pairs to attribute distinct pre-defined
weights to different relative similarities, which can introduce potential
bias. In this work, we introduce Graph Multi-Similarity Learning for Molecular
Property Prediction (GraphMSL) framework, along with a novel approach to
formulate a generalized multi-similarity metric without the need to define
positive and negative pairs. In each of the chemical modality spaces (e.g.,
molecular depiction image, fingerprint, NMR, and SMILES) under consideration,
we first define a self-similarity metric (i.e., similarity between an anchor
molecule and another molecule), and then transform it into a generalized
multi-similarity metric for the anchor through a pair weighting function.
GraphMSL validates the efficacy of the multi-similarity metric across
MoleculeNet datasets. Furthermore, these metrics of all modalities are
integrated into a multimodal multi-similarity metric, which showcases the
potential to improve the performance. Moreover, the focus of the model can be
redirected or customized by altering the fusion function. Last but not least,
GraphMSL proves effective in drug discovery evaluations through post-hoc
analyses of the learnt representations.

Machine Learning, ICML

##  1 Introduction

Graph Neural Networks (GNNs) have emerged as a prominent approach for
molecular representation learning, addressing drug-related challenges (Wieder
et al., 2020; Zhang et al., 2022; Fang et al., 2022; Wang et al., 2023).
However, the generation of task-specific labels for training molecular GNNs is
hindered by the resource-intensive and time-consuming nature of chemical
synthesis and biological testing experiments. To address this challenge,
current research prioritizes self-supervised learning approaches for
pretraining molecular GNNs, with a prevalent trend of adopting contrastive
learning approaches (Wang et al., 2021, 2022b; Liu et al., 2022a). (See an
introduction of pre-training approaches in Section 2)

Contrastive Learning (CL) is a discriminative representation learning approach
by bringing similar instances into close proximity within the latent
representation space and pushing apart dissimilar instances (Schroff et al.,
2015). A fundamental prerequisite of CL is to define positive pairs denoting
similarity, and negative pairs representing dissimilarity (Jaiswal et al.,
2020). In the CL-based pre-training of molecular GNNs, positive pairs are
often established through either data augmentation (Sun et al., 2021; You et
al., 2020a), such as node deletion, edge perturbation, subgraph extraction,
attribute masking, and subgraph substitution, or domain knowledge, exemplified
by reactant-product pairing (Wang et al., 2022a) or conformer grouping (Moon
et al., 2023). However, such a binary characterization of the relationships
among molecules, by designating them as either positive or negative pairs,
oversimplifies the complex nature of these connections. Moreover, these CL
approaches fail to notice relationships among multiple instances
simultaneously by adapting simple contrastive loss, thereby hindering the
effectiveness and generalizability of representation learning (Wang et al.,
2019; Mu et al., 2023; Zhang et al., 2023). (See an illustration of similarity
types in CL in Figure Appendix A.1)

Figure 1: Graph Similarity Learning for Molecular Property Prediction
(GraphMSL). $\mathcal{\mathrm{missing}}t_{i,j}^{g}$,
$\mathcal{\mathrm{missing}}t_{l,m}^{n}$ represent graph-level and node-level
target similarity, respectively. $\mathcal{\mathrm{missing}}d_{i,j}^{g}$,
$\mathcal{\mathrm{missing}}d_{l,m}^{n}$ represent the similarity for graph-
level and node-level embeddings, respectively. Here, $i$ and $j$ are the
indices of molecular graphs in the graph pool, and $l$ and $m$ are the indices
of atomic nodes in the node pool. Unlike the general contrastive learning
framework shown in Appendix Figure A.2, GraphMSL doesn’t need to define
positive or negative pairs and is capable of learning continuous ordering from
target similarity.

Unlike CL, which adapts a binary similarity metric, Similarity Learning (SL)
employs a continuous similarity metric for representation learning. It
measures the similarity between two instances in the given space (Balcan &
Blum, 2006; Wen et al., 2023). This pairwise similarity provides a localized
perspective on the relations between two instances in a given instance pool,
known as self-similarity (Wang et al., 2019). However, the global relations
between two instances can be influenced by their belonging to the instance
pool. Therefore, self-similarity alone is inadequate for capturing
relationships among multiple instances simultaneously, known as relative
similarity (Wang et al., 2019). To address this challenge, Multi-Similarity
Learning (MSL) has emerged as a solution, expanding its focus from self-
similarity to a global view of relations and encapsulating both self-
similarity and relative similarity (Wang et al., 2019; Zhang et al., 2021; Mu
et al., 2023; Zhang et al., 2023).

When formulating a similarity metric, a common guiding principle is that an
effective similarity function or metric should align tightly with the
objectives of specific tasks (Balcan & Blum, 2006; Müller et al., 2018; Yang &
Jin, 2006). To target the drug discovery tasks, a robust similarity metric
must be adept at discerning molecules based on key properties regarding drug
development. Insights drawn from prior research (Xu et al., 2023a) indicate
that distinct chemical modalities, such as chemical languages, molecular
depiction images, and chemical spectroscopic spectra, possess unique expertise
in expressing specific molecular properties. More importantly, a continuous
similarity metric for molecules can be projected from each chemical modality
space. Thus, a more promising multimodal similarity metric can be formulated
by integrating these individual metrics.

In response to the challenges and opportunities in molecular graph
representation learning, we propose the Graph Multi-Similarity Learning for
Molecular Property Prediction (GraphMSL) framework. This approach aims to
advance graph contrastive learning to graph multi-similarity learning by
incorporating a continuous multi-similarity metric. A self-similarity metric
can be derived from each heterogeneous chemical modality, such as chemical
languages, molecular depiction images, and chemical spectroscopic spectra,
through representation learning. Then, this self-similarity metric can be
transformed into a multi-similarity metric through a pair weighting function.
Besides, various unimodal multi-similarity metrics can be fused into a
multimodal similarity metric. In addition, GraphMSL framework can be
customized for singular or multi-view perspective.

In summary, our contribution comprises three major aspects: Conceptually: We
introduce a generalized multi-similarity metric for graph representation
learning, capturing both self-similarity and relative similarity. Our approach
doesn’t rely on pre-defined negative or positive pairs, and it satisfies the
requirement of convergent similarity learning as shown in Section 3.1. To the
best of our knowledge, this is the first work to demonstrate such generalized
multi-similarity for graph representation learning. Methodologically: We
extract a self-similarity metric from a chemical modality, and transition it
into a generalized multi-similarity metric through a pair weighting function,
and each modality contributes to a unique multi-similarity metric.
Furthermore, we integrate these metrics into a fused multimodal form that has
the potential to improve the performance. Empirically: GraphMSL excels in
various downstream tasks, with performance enhancements achieved through
multi-level graph learning. Last but not least, we demonstrate the
explainability of the learned representations through two post-hoc analysis.
Notably, we explore minimum positive subgraphs and maximum common subgraphs to
gain insights for further drug molecule design.

##  2 Preliminaries

Directed Message Passing Neural Network (DMPNN). The Message Passing Neural
Network (MPNN) (Gilmer et al., 2017) is a GNN model that processes an
undirected graph $G$ with node (atom) features $x_{v}$ and edge (chemical
bond) features $e_{vw}$. It operates through two distinct phases: a message
passing phase, facilitating information transmission across the molecule to
construct a neural representation, and a readout phase, utilizing the final
representation to make predictions regarding properties of interest. The
primary distinction between DMPNN and a generic MPNN lies in the message
passing phase. While MPNN uses messages associated with nodes, DMPNN crucially
differs by employing messages associated with directed edges (Yang et al.,
2019). This design choice is motivated by the necessity to prevent totters
(Mahé et al., 2004), eliminating messages passed along paths of the form
$v_{1}v_{2}\dots v_{n}$, where $v_{i}=v_{i+2}$ for some $i$, thereby
eliminating unnecessary loops in the message passing trajectory.

Pre-Training for Molecular GNNs. There are two levels of pre-training tasks:
node-level (atomic level) and graph-level (molecular level), which enhance the
generalization capabilities of molecular GNNs across diverse downstream tasks
(Hu et al., 2019; Xia et al., 2022; Fang et al., 2022). Node-level tasks aim
to capture local context, often involving the random masking of nodes and
subsequent prediction of their properties based on node representations. In
contrast, graph-level tasks focus on extracting global information, such as
predicting graph properties based on the graph representation.Self-supervised
learning (SSL) is a paradigm in which a model is trained on a task by
leveraging the data itself to generate supervisory signals, eliminating the
need for external annotations (Liu et al., 2021b, 2022c). SSL finds active
applications in the pre-training tasks of molecular GNNs by formulating label-
free pretext tasks, such as graph reconstructions (Hu et al., 2020; You et
al., 2020b; Liu et al., 2021a), context predictions (Hu et al., 2019; Peng et
al., 2020), and adopting contrastive learning approaches (Wang et al., 2021,
2022b; Liu et al., 2022a).

Similarity Learning. Given object $i$ and object $j$, the optimization of
object similarity $d_{i,j}$ in the latent representation space is directed by
the target similarity $t_{i,j}$ in a given space, a process commonly
recognized as similarity learning (Moutafis et al., 2016; Suárez-Díaz et al.,
2018). Contrastive learning constrains its similarity metric $t_{i,j}$ to a
binary setting, taking on values of either 1 or 0. However, similarity
learning allows its similarity metric $t_{i,j}$ to be continuous values. Two
distinct types of similarities can be identified, as illustrated in Appendix
Figure A.1: self-similarity (the pairwise similarity between two objects,
typically defined through cosine similarity), relative similarity
(distinctions in self-similarity with other pairs) (Wang et al., 2019).
Contrastive learning, implemented with a binary similarity metric,
concentrates on self-similarity and fails to explore the complete
relationships between samples (Oh Song et al., 2016; Wang et al., 2019).

##  3 Methods

We begin by presenting the theorem of convergent similarity learning, followed
by introducing the generalized multi-similarity metric and the multimodal
multi-similarity metric.

###  3.1 Convergent Similarity learning

Let $\mathcal{S}$ be a set of instances with size of $|\mathcal{S}|$, and let
$\mathcal{P}$ represent the learnable latent representations of instances in
$\mathcal{S}$ such that $|\mathcal{P}|=|\mathcal{S}|$. For any two instances
$i,j\in\mathcal{S}$, their respective latent representations are denoted by
$\mathcal{P}_{i}$ and $\mathcal{P}_{j}$. Let $t_{i,j}$ represent the target
similarity between instances $i$ and $j$ in a given domain, and let $d_{i,j}$
be the similarity between $\mathcal{P}_{i}$ and $\mathcal{P}_{j}$ in the
latent space.

######  Theorem 3.1 (Theorem of Convergent Similarity learning).

If $t_{i,j}$ is non-negative and $\\{t_{i,j}\\}$ satisfies the constraint
$\sum_{j=1}^{|\mathcal{S}|}t_{i,j}=1$, consider the loss function for an
instance $i$ defined as follows:

$L(i)=-\sum_{j=1}^{|\mathcal{S}|}t_{i,j}\log\left(\frac{e^{d_{i,j}}}{\sum_{k=1}^{|\mathcal{S}|}e^{d_{i,k}}}\right)$
(1)

then when it reaches ideal optimum, the relationship between $t_{i,j}$ and
$d_{i,j}$ satisfies:

$\text{softmax}(d_{i,j})=t_{i,j}$ (2)

For detailed proof, please refer to Appendix Section B.1.

###  3.2 Generalized Multi-Similarity

We formulate a generalized multi-similarity metric from self-similarity by
adapting the softmax function as a pair weighting function. The formula for
the generalized multi-similarity, denoted as $t_{i,j}^{R}$ between the
$i^{th}$ and $j^{th}$ instances under a given space $R$, is provided below:

${t}_{i,j}^{R}=softmax({S}_{i,j}^{R})=\frac{e^{S_{i,j}^{R}}}{\sum_{k=1}^{|\mathcal{S}|}e^{{S}_{i,k}^{R}}}$
(3)

where $S_{i,j}^{R}$ represents self-similarity, and $|\mathcal{S}|$ is the
size of the instance set. Defined in this manner, the generalized multi-
similarity metric incorporates both self-similarity and relative similarities.
Notably, unlike other multi-similarity learning approaches (Wang et al., 2019;
Zhang et al., 2021), our method does not rely on the categorization of
negative and positive pairs for the pair weighting function. Additionally, the
use of the softmax function ensures that the generalized target similarity
$t_{i,j}$ adheres to the principles of Convergent Similarity Learning (refer
to Section 3.1).

###  3.3 Multimodal Multi-Similarity

With a set of generalized multi-similarities $\\{t^{R}\\}$ from various
modality spaces, we can transform generalized multi-similarities from
respective unimodality space to multimodal space through a fusion function.
There are numerous potential designs of the fusion function. For simplicity,
we take linear combination as a demonstration. The multimodal generalized
multi-similarity $t_{i,j}^{M}$ between $i^{th}$ and $j^{th}$ objects can be
defined as follows:

$\displaystyle t_{i,j}^{M}$ $\displaystyle=fusion(\\{t^{R}\\})$ (4)
$\displaystyle=\sum w_{R}\cdot t_{i,j}^{R}$ (5)

where $t_{i,j}^{R}$ represents the target similarity between $i^{th}$ and
$j^{th}$ instance in unimodal space $R$, $w_{R}$ is the pre-defined weights
for the corresponding modal, and $\sum w_{R}=1$. Such that, it still satisfy
the requirement of Convergent Similarity Learning (See proof in Appendix
SectionB.2).

##  4 Experiments

In this section, we begin by presenting the design of the multi-similarity
metric and the GraphMSL framework. Subsequently, we showcase the results
obtained from GraphMSL. Finally, we demonstrate the explainability of the
learned molecular representations. (Please refer to the experimental details
of pre-training and fine-tuning in the Appendix Section D.)

###  4.1 The Design of Multi-Similarity Metric

####  4.1.1 Graph-Level Multi-Similarity Metric

Self-Similarity. By employing representation learning, various chemical
modalities, such as NMR spectra, depiction images, and SMILES, can be encoded
into latent representation vectors. The cosine similarity between two vectors
serves as self-similarity. The unimodal self-similarity for 13C NMR spectrum,
denoted as $S_{i,j}^{C}$, can be defined as follows:

$S_{i,j}^{C}=Cos(\mathcal{V}_{i},\mathcal{V}_{j})=\frac{\mathcal{V}_{i}\cdot\mathcal{V}_{j}^{T}}{\|\mathcal{V}_{i}\|\cdot\|\mathcal{V}_{j}\|}$
(6)

where $\mathcal{V}_{i},\mathcal{V}_{j}$ represents the embedding of NMR
spectra for two given molecules. Similarly, the uni-modality similarity
similarity for depiction images and SMILES can be obtained. The self-
similarity of fingerprints adapts a well-established similarity function for
molecules, namely the Tanimoto similarity (Bajusz et al., 2015). (See more
details in Appendix Section C.2)

Generalized Multi-Similarity Metric. Transitioning from self-similarity to a
generalized similarity, we apply the softmax function as our pair weighting
mechanism. The formula of generalized similarity is provided below,
illustrated with an example of 13C NMR similarity:

$t^{C}_{i,j}=softmax(S_{i,j}^{C})_{i}=\frac{e^{S_{i,j}^{C}}}{\sum_{k=1}^{|\mathcal{G}|}e^{s_{i,k}^{C}}}$
(7)

where $t^{C}_{i,j}$ represents generalized similarity, $S_{i,j}^{C}$ denotes
the self-similarity, and $|\mathcal{G}|$ is the size of molecular graph pool.
For the expressions for SMILES and Image modality, please refer to Appendix
Section C.2

Multimodal Multi-Similarity Metric. We employ a simple linear combination to
formulate the multimodal multi-similarity $t_{i,j}^{M}$ between the $i^{th}$
and $j^{th}$ molecules, represented as a graph-level similarity $t_{i,j}^{g}$,
as follows:

$\displaystyle t_{i,j}^{g}$ $\displaystyle=t_{i,j}^{M}$ (8)
$\displaystyle=w_{SM}\cdot t^{SM}_{i,j}+w_{{C}}\cdot t^{C}_{i,j}+w_{{I}}\cdot
t^{I}_{i,j}+w_{{F}}\cdot t^{F}_{i,j}$ (9)

where $t^{SM}_{i,j}$ denotes the similarity based on SMILES, $t^{C}_{i,j}$
denotes the similarity with respect to 13C NMR spectrum, $t^{I}_{i,j}$ denotes
the similarity regarding images, and $f$ denotes the similarity based on
fingerprints, $w_{SM}$, $w_{C}$, $w_{I}$, and $w_{F}$ are the pre-defined
weights for their respective similarity, and
$w_{SM}+w_{{C}}+w_{{I}}+w_{{F}}=1$.

####  4.1.2 Node-Level Multi-Similarity Metric

Self-Similarity. The self-similarity among nodes (atoms) is derived from the
positions of their signal peaks on 13C NMR spectra, measured in parts per
million (ppm). The ppm values are continuous, typically ranging from 0 to 200
(see more introduction of ppm in Appendix C.3). The self-similarity of NMR
peaks $S^{P}_{l,m}$ can be defined as following:

$S^{P}_{l,m}=\frac{\tau_{2}}{|ppm_{l}-ppm_{m}|+\tau_{1}}$ (10)

where $ppm_{l}$ and $ppm_{m}$ are the positions of NMR peaks for the $l^{th}$,
$m^{th}$ Carbon atom, $\tau_{1}$ and $\tau_{2}$ are temperature hyper-
parameter.

Generalized Similarity Metric. A generalized multi-similarity $t^{P}_{l,m}$,
as a node-level similarity $t_{l,m}^{n}$, can be formulated with softmax
function, as shown below

$t_{l,m}^{n}=t^{P}_{l,m}=softmax(S^{P}_{l,m})_{l}=\frac{e^{S^{P}_{l,m}}}{\sum_{q=1}^{|\mathcal{N}|}e^{S^{P}_{l,q}}}$
(11)

where $S^{P}_{l,m}$ represents self-similarity of NMR peaks, $|\mathcal{N}|$
is the size of atomic node pool.

###  4.2 The Design of GraphMSL Framework

The GraphMSL framework is versatile, allowing customization for singular or
multiple views. In this context, we showcase the customization options for
graph-level, node-level, or bi-level GraphMSL. The graph-level GraphMSL,
denoted as $\text{GraphMSL}_{graph}$, incorporates multi-similarity metrics
derived from molecular level chemical semantics. The node-level GraphMSL,
denoted as $\text{GraphMSL}_{node}$, incorporates multi-similarity metrics
derived from atomic level chemical semantics. The bi-level GraphMSL, denoted
as $\text{GraphMSL}_{bi-level}$, incorporates multi-similarity metrics derived
from both molecular and atomic level chemical semantics. In these
implementations, the graph encoder adopts DMPNN (Yang et al., 2019)
architecture, an interactive message passing scheme considering the
interactions. Additionally, neither of these two modules requires additional
projection layers.

The loss function of $\text{GraphMSL}_{graph}$ model, noted as $L_{graph}$,
can be expressed as:

$L_{graph}=-\frac{1}{|\mathcal{G}|}\sum_{\begin{subarray}{c}1\leq
j\leq|\mathcal{G}|\end{subarray}}\mathcal{\mathrm{missing}}t_{i,j}^{g}\log\frac{e^{\mathcal{\mathrm{missing}}{d}_{i,j}^{g}}}{\sum_{\begin{subarray}{c}1\leq
k\leq|\mathcal{G}|\end{subarray}}e^{\mathcal{\mathrm{missing}}{d}_{i,k}^{g}}}$
(12)

where $t_{i,j}^{g}$ represents graph-level similarity metrics, $d_{i,j}^{g}$
represents graph-level latent space similarity metrics, $i,j,k$ represent the
indices of molecular graphs within a graph pool of size $|\mathcal{G}|$ per
batch.

The loss function of $\text{GraphMSL}_{node}$ model, noted as $L_{node}$, can
be expressed as:

$L_{node}=-\frac{1}{|\mathcal{N}|}\sum_{\begin{subarray}{c}1\leq
m\leq|\mathcal{N}|\end{subarray}}\mathcal{\mathrm{missing}}t_{l,m}^{n}\log\frac{e^{\mathcal{\mathrm{missing}}{d}_{l,m}^{n}}}{\sum_{\begin{subarray}{c}1\leq
q\leq|\mathcal{N}|\end{subarray}}e^{\mathcal{\mathrm{missing}}{d}_{lq}^{n}}}$
(13)

where $t_{l,m}^{n}$ represents node-level similarity metrics, $d_{l,m}^{g}$
represents node-level latent space similarity metrics, $l,m,q$ represent the
indices of nodes within a node pool of size $|\mathcal{N}|$.

The loss function of $\text{GraphMSL}_{bi-level}$ model, noted as $L_{bi-
level}$, can be expressed as:

$L_{bi-level}=L_{graph}+L_{node}$ (14)

###  4.3 Results

####  4.3.1 Overall performance of $\text{GraphMSL}_{{graph}}$

The performance of $\text{GraphMSL}_{graph}$ is evaluated through a
comparative analysis with a range of baselines, the specifics of which are
described in Appendix Section D.2.2. We report the performance metrics of
$\text{GraphMSL}_{graph}$ across 8 classification under ROC-AUC and 3
regression tasks under RMSE from the MoleculeNet benchmark (Wu et al., 2018a),
as shown in Tables 1. Within these tables, the best results are denoted in
bold, and the second-best are indicated with underlining. From the comparative
evaluation, we find that:

1) $\text{GraphMSL}_{graph}$ outperforms the baselines in seven of the eight
evaluated classification tasks, including BBBP, BACE, SIDER, HIV, MUV, Tox21
and ToxCast.

2) In the regression tasks, $\text{GraphMSL}_{graph}$ also achieves the
highest performance across all evaluated benchmarks, which include ESOL,
FreeSolv, and Lipophilicity.

3) While $\text{GraphMSL}_{graph}$ doesn’t outperform baselines on the Clintox
task, it attains average performance.

In short, these findings underscore the proficiency of
$\text{GraphMSL}_{graph}$ in learning molecular representations that are
effective and impactful, as evidenced by its commendable performance across a
suite of diverse tasks.

Table 1: Overall performances (ROC-AUC) on classification downstream tasks. The best results are denoted in bold, and the second-best are indicated with underlining. (Note: N-Gram is highly time-consuming on ToxCast.)  Data Set | BBBP | bace | Sider | Clintox | HIV | MUV | Tox21 | ToxCast
---|---|---|---|---|---|---|---|---
AttentiveFP |  64.3$\pm$1.8 |  78.4$\pm$2.2 |  60.6$\pm$3.2 |  84.7$\pm$0.3 |  75.7$\pm$1.4 |  76.6$\pm$1.5 |  76.1$\pm$0.5 |  63.7$\pm$0.2
DMPNN |  91.9$\pm$3.0 |  85.2$\pm$0.6 |  57.0$\pm$0.7 |  90.6$\pm$0.6 |  77.1$\pm$0.5 |  78.6$\pm$1.4 |  75.9$\pm$0.7 |  63.7$\pm$0.2
N-Gram |  91.2$\pm$0.3 |  79.1$\pm$1.3 |  63.2$\pm$0.5 |  87.5$\pm$2.7 |  78.7$\pm$0.4 |  76.9$\pm$0.7 |  76.9$\pm$2.7 | -
GEM |  72.4$\pm$0.4 |  85.6$\pm$1.1 |  67.2$\pm$0.4 |  90.1$\pm$1.3 |  80.6$\pm$0.9 |  81.7$\pm$0.5 |  78.1$\pm$0.1 |  69.2$\pm$0.4
Uni-Mol |  72.9$\pm$0.6 |  85.7$\pm$0.2 |  65.9$\pm$1.3 | 91.9$\pm$1.8 |  80.8$\pm$0.3 | 82.1$\pm$1.3 |  79.6$\pm$0.5 |  69.6$\pm$0.1
GROVER |  86.8$\pm$2.2 |  82.4$\pm$3.6 |  61.2$\pm$2.5 |  70.3$\pm$13.7 |  68.2$\pm$1.1 |  67.3$\pm$1.8 |  80.3$\pm$2.0 |  56.8$\pm$3.4
InfoGraph |  69.2$\pm$0.8 |  73.9$\pm$2.5 |  59.2$\pm$0.2 |  75.1$\pm$5.0 |  74.5$\pm$1.8 |  74.0$\pm$1.5 |  73.0$\pm$0.7 |  62.0$\pm$0.3
GraphCL |  67.5$\pm$3.3 |  68.7$\pm$7.8 |  60.1$\pm$1.3 |  78.9$\pm$4.2 |  75.0$\pm$0.4 |  77.1$\pm$1.0 |  75.0$\pm$0.3 |  62.8$\pm$0.2
MolCLR |  73.3$\pm$1.0 |  82.8$\pm$0.7 |  61.2$\pm$3.6 |  89.8$\pm$2.7 |  77.4$\pm$0.6 |  78.9$\pm$2.3 |  74.1$\pm$5.3 |  65.9$\pm$2.1
$\text{MolCLR}_{\text{cmpnn}}$ |  72.4$\pm$0.7 |  85.0$\pm$2.4 |  59.7$\pm$3.4 |  88.0$\pm$4.0 |  77.8$\pm$5.5 |  74.5$\pm$2.1 |  78.4$\pm$2.6 |  69.1$\pm$1.2
GraphMVP |  72.4$\pm$1.6 |  81.2$\pm$9.0 |  63.9$\pm$1.2 |  79.1$\pm$2.8 |  77.0$\pm$1.2 |  77.7$\pm$6.0 |  75.9$\pm$5.0 |  63.1$\pm$0.4
$\text{GraphMSL}_{graph}$ |  93.2$\pm$0.8 | 93.6$\pm$2.7 | 68.1$\pm$1.5 |  88.8$\pm$4.6 | 83.3$\pm$1.1 | 84.5$\pm$2.9 | 86.1$\pm$0.6 | 71.4$\pm$0.2
$\text{GraphMSL}_{node}$ | 93.4$\pm$2.7 |  89.3$\pm$1.7 |  62.8$\pm$2.1 |  86.1$\pm$5.4 |  82.1$\pm$0.4 |  75.4$\pm$5.2 |  84.9$\pm$1.0 |  70.6$\pm$0.8
$\text{GraphMSL}_{bi-level}$ | 94.3$\pm$0.8 | 94.5$\pm$0.7 | 67.3$\pm$0.6 | 93.8$\pm$0.8 | 83.0$\pm$0.7 |  81.5$\pm$3.7 | 86.1$\pm$0.8 | 71.2$\pm$1.1
Table 2: Overall performances (RMSE) on regression downstream tasks. The best results are denoted in bold, and the second-best are indicated with underlining. Data Set | ESOL | FreeSolv | Lipo
---|---|---|---
AttentiveFP |  0.877$\pm$0.029 |  2.073$\pm$0.183 |  0.721$\pm$0.001
DMPNN |  1.050$\pm$0.008 |  2.082$\pm$0.082 |  0.683$\pm$0.016
$\text{N-Gram}_{\text{RF}}$ |  1.074$\pm$0.107 |  2.688$\pm$0.085 |  0.812$\pm$0.028
$\text{N-Gram}_{\text{XGB}}$ |  1.083$\pm$0.082 |  5.061$\pm$0.744 |  2.072$\pm$0.030
GEM |  0.798$\pm$0.029 |  1.877$\pm$0.094 |  0.660$\pm$0.008
Uni-Mol | 0.788$\pm$0.029 |  1.620$\pm$0.035 |  0.660$\pm$0.008
GROVER |  1.423$\pm$0.288 |  2.977$\pm$0.615 |  0.823$\pm$0.010
MolCLR |  1.113$\pm$0.023 |  2.301$\pm$0.247 |  0.789$\pm$0.009
$\text{MolCLR}_{\text{CMPNN}}$ |  0.911$\pm$0.082 |  2.021$\pm$0.133 |  0.875$\pm$0.003
$\text{GraphMSL}_{graph}$ | 0.746$\pm$0.060 | 1.437$\pm$ 0.134 | 0.537$\pm$0.005
$\text{GraphMSL}_{node}$ |  0.924$\pm$0.083 |  1.707$\pm$0.126 |  0.587$\pm$0.021
$\text{GraphMSL}_{bi-level}$ |  0.843$\pm$0.094 | 1.601$\pm$0.057 | 0.562$\pm$0.005

####  4.3.2 Ablation Study-Various Mutil,modal Similarity Metrics

We evaluate the performance of $\text{GraphMSL}_{graph}$ using a diverse set
of similarity metrics, as outlined in Table 3. Each uni-modal similarity
metric demonstrates unique strengths across various tasks. For instance, the
model guided by Image similarity metrics exhibits outstanding performance in
ESOL compared to other uni-modality metrics. Solubility is closely tied to the
polarity of molecules. High polar atoms possess the ability to form hydrogen
bonds with water, thereby enhancing solubility. In molecular depiction images,
nonpolar carbon atoms (C) are typically represented as dots. Conversely,
highly polar atoms like oxygen (O), nitrogen (N), and fluorine (F) are
depicted more prominently, occupying a significant portion of the image.
Consequently, image representations place considerable emphasis on this
information, distinguishing between nonpolar and polar atoms by varying pixel
density.

Notably, the true strength lies in the flexibility of multimodal similarity
metrics achieved through the fusion of multiple unimodal metrics. There are
five variations of multimodal similarity metrics, denoted as
$\text{Fusion}_{\text{{Smiles}}}$, $\text{Fusion}_{\text{{NMR}}}$,
$\text{Fusion}_{\text{{Image}}}$, $\text{Fusion}_{\text{{Fingerprint}}}$, and
$\text{Fusion}_{\text{{Average}}}$ (Please refer to their configurations in
Appendix Section C.4). Through the comparisons, it becomes evident that a
well-designed multimodal similarity metric can significantly enhance model
performance compared to unimodal metrics. For example,
$\text{Fusion}_{\text{Smiles}}$ boosts performance in Tox21 tasks,
$\text{Fusion}_{\text{NMR}}$ enhances results in MUV task, and
$\text{Fusion}_{\text{Fingerprint}}$ contributes to improved outcomes in BBBP
and BACE tasks.

####  4.3.3 Ablation Study-Mutil-View Learning

While the $\text{GraphMSL}_{{graph}}$ model demonstrates its efficacy across a
variety of downstream tasks, we further investigate the potential of GraphMSL
framework by including both graph (molecule) and node (atom) levels to assess
possible performance improvements. Our findings indicate that:

1) For Clintox, $\text{GraphMSL}_{{bi-level}}$ exhibits superior performance,
compared with the baselines and $\text{GraphMSL}_{{graph}}$. Notably,
$\text{GraphMSL}_{{bi-level}}$ achieves a 5% enhancement in ROC-AUC on the
Clintox dataset over $\text{GraphMSL}_{{graph}}$.

2) For tasks such as BBBP, BACE, HIV, MUV, and TOX21, $\text{GraphMSL}_{{bi-
level}}$ outperforms all compared models, though $\text{GraphMSL}_{{bi-
level}}$ is better than $\text{GraphMSL}_{{graph}}$ by a marginal degree.

3) Conversely, $\text{GraphMSL}_{{bi-level}}$ underperforms relative to
$\text{GraphMSL}_{{graph}}$ in Sider, ToxCast, ESOL, FreeSolv, and Lipo
datasets. This discrepancy in performance may stem from small portion of
unresolved interactions or slight discord between the graph-level and node-
level similarities under the scenarios of these tasks.

###  4.4 Explainability of Learnt Representations

To demonstrate the interpretability of learnt representations, we present
post-hoc analysis for two tasks, ESOL and BACE, as demonstration. The results
showcase learnt representations can capture task-specific patterns and offer
valuable insights for molecular design.

ESOL. We apply t-SNE to reduce molecule embeddings from 300 to 2, generating a
heatmap correlating Log solubility (Figure 3.a). The heatmap visually depicts
a smooth transition from high solubility (depicted in red) to low solubility
(depicted in blue). The embeddings adeptly capture essential structural
information and solubility-related patterns, organizing molecules with
analogous solubility in close proximity within the embedding space. Through
our investigation, we discovered that molecules with comparable solubility
share common graph features, such as recurring motifs or an increased
frequency of specific nodes (atoms). For example, in the region of lowest
solubility (enclosed by blue dashed line), the molecular graphs all contain
biphenyl groups. Biphenyls are nonpolar molecules, and as a consequence, they
have limited interaction with water, leading to low solubility in aqueous
environments. In the region of highest solubility (enclosed by red solid
line), molecules exhibit high polarity, characterized by the prevalence of
nitrogen (N) and oxygen (O) atoms. This specific graph configuration
facilitates the formation of hydrogen bonds with water molecules, thereby
enhancing solubility.

Table 3: Ablation study on the performances of $\text{GraphMSL}_{graph}$. The best results are denoted in boldf, and the second-best are indicated with underlining. The first 8 tasks are for classification under evaluation of ROC-AUC, while the last three are for regression with evaluation of RMSE. See the detailed performance of $\text{GraphMSL}_{bi-level}$ upon different similarity metrics in Appendix Table E.1.  Data Set | BBBP | bace | Sider | Clintox | HIV | MUV | Tox21 | ToxCast | ESOL | FreeSolv | Lipo
---|---|---|---|---|---|---|---|---|---|---|---
SMILES |  92.9$\pm$1.5 |  90.9$\pm$3.3 |  64.9$\pm$0.3 |  78.2$\pm$1.9 | 83.3$\pm$1.1 |  80.1$\pm$2.5 |  85.7$\pm$1.2 |  70.5$\pm$2.5 |  0.811$\pm$ 0.109 |  1.623$\pm$ 0.168 |  0.539$\pm$ 0.017
NMR |  91.0$\pm$2.0 |  93.2$\pm$2.7 | 68.1$\pm$1.5 |  87.7$\pm$6.5 |  80.9$\pm$5.0 |  80.9$\pm$5.0 |  85.1$\pm$0.4 |  71.1$\pm$0.8 |  0.844$\pm$ 0.123 |  2.417$\pm$ 0.495 |  0.609$\pm$ 0.031
Image |  93.1$\pm$2.4 |  92.9$\pm$1.8 |  65.3$\pm$1.5 |  86.2$\pm$6.5 |  82.3$\pm$0.6 |  78.7$\pm$1.7 | 86.0$\pm$1.0 |  71.0$\pm$1.6 | 0.761$\pm$ 0.068 |  1.648$\pm$ 0.045 | 0.537$\pm$ 0.005
Fingerprint |  92.9$\pm$2.3 |  91.7$\pm$3.6 |  65.6$\pm$0.7 |  87.5$\pm$6.0 |  81.2$\pm$2.5 | 82.9$\pm$3.1 |  85.3$\pm$1.3 |  70.0$\pm$1.4 |  0.808$\pm$ 0.071 | 1.437$\pm$ 0.134 |  0.565$\pm$ 0.017
$\text{Fusion}_{\text{Smiles}}$ | 93.1$\pm$1.4 |  91.4$\pm$3.9 |  66.1$\pm$1.0 |  86.6$\pm$6.6 |  82.7$\pm$1.1 |  82.2$\pm$4.1 | 86.1$\pm$0.6 | 71.3$\pm$1.3 |  0.800$\pm$0.068 |  1.505$\pm$0.177 | 0.537$\pm$0.145
$\text{Fusion}_{\text{NMR}}$ |  93.0$\pm$1.6 |  93.0$\pm$2.4 |  64.3$\pm$1.9 |  83.5$\pm$10.6 |  81.4$\pm$3.1 | 84.5$\pm$2.9 |  85.8$\pm$1.1 |  70.9$\pm$1.1 |  0.783$\pm$0.105 | 1.472$\pm$0.072 |  0.552$\pm$0.029
$\text{Fusion}_{\text{Image}}$ |  92.9$\pm$3.4 |  92.9$\pm$2.4 |  64.3$\pm$1.6 | 88.6$\pm$4.6 | 83.0$\pm$0.9 |  81.6$\pm$4.8 |  85.8$\pm$3.8 |  70.6$\pm$1.7 | 0.746$\pm$0.060 |  1.587 $\pm$0.143 |  0.549$\pm$0.025
$\text{Fusion}_{\text{Fingerprint}}$ | 93.2$\pm$0.8 | 93.6$\pm$2.7 |  65.8$\pm$0.7 |  85.4$\pm$9.4 |  82.4$\pm$3.1 |  81.6$\pm$2.5 |  85.3$\pm$1.1 |  71.1$\pm$1.1 |  0.818$\pm$0.054 |  1.535$\pm$0.080 |  0.573$\pm$0.040
$\text{Fusion}_{\text{Average}}$ |  90.2$\pm$6.8 | 93.4$\pm$2.7 | 67.0$\pm$0.6 | 88.8$\pm$4.6 |  80.8$\pm$2.2 |  79.2$\pm$5.4 |  85.4$\pm$0.8 | 71.4$\pm$0.2 |  0.781$\pm$0.082 |  1.528$\pm$0.180 |  0.559$\pm$0.018
Figure 2: T-SNE visualization depicting the ESOL molecule embeddings alongside
molecules within the highlighted region. Each point in the heatmap corresponds
to the embeddings of respective molecules in ESOL, with color indicating
solubility levels. Red denotes higher solubility, while blue indicates lower
solubility. Figure 3: T-SNE Visualization of BACE embedding and clustering
based on minimum positive subgraph (MPS). MPS represents minimum positive
subgraph of a positive molecule; MCS represents maximum common subgraph of
several positive molecules, sharing the same MPS. Pink nodes represent MPS,
blue nodes depict molecules, and edge colors indicate binding potential
differences. Red edges denote successful designs (original higher than MPS),
while blue indicates less efficient designs (original lower than MPS).

BACE. We explore the binding potential of positive inhibitor molecules
targeting BACE and their associated key functional substructures, referred to
as minimum positive subgraphs (MPS). To identify MPS, we employ a Monte Carlo
Tree Search (MCTS) approach integrated into our BACE classification model, as
implemented in RationalRL (Jin et al., 2020). MCTS, being an iterative
process, allows us to evaluate each candidate substructure for its binding
potential with our model. Following the determination of MPSs, we categorize
the original positive BACE molecules based on their respective MPSs and
subsequently compute the maximum common subgraph (MCS) for each group. In
Figure 3.b, we present the top 10 most frequently occurring MPSs, accompanied
by their respective groups. As expected, molecular graph embeddings within the
same group exhibit proximity after T-SNE reduction. This proximity is a result
of shared identical motifs among graphs within each group.

In the development of new inhibitors, it is common to introduce additional
functional groups or complex motifs to have more interactions with enzyme,
thereby increasing the complexity of the molecules and making synthesis more
challenging. However, such modifications do not guarantee a positive
contribution or reward to inhibitor design. It is evident by the results of
extending molecules from MPS to original molecules, which demonstrate both
positive and negative contributions to binding potential. Negative
Contribution: The complex molecules exhibits lower binding potentials than
simple MPS 4. This disparity can be illustrated by the distinction between MPS
4 and MCS 4a: in MPS 4, one of the NH groups, as a secondary amine,
establishes a strong binding interaction with the enzyme, but this NH group
transforms into an amide group in MCS 4a, resulting in a significant decrease
in binding capability. Therefore, these designs do not receive commensurate
rewards considering the increased difficulty in synthesis. Positive
Contribution: the complex molecules demonstrates higher binding potentials
than MPS 9, indicating successful design. This outcome can be attributed to
MCS 9a, which features more nitrogen and fluorine atoms capable of forming
strong bindings with the enzyme. In summary, this experiment can serve as
valuable guidelines for advancing inhibitor development.

##  5 Related work

Contrastive Learning on Molecular Graphs. The primary focus within the domain
of contrastive learning applied to molecular graphs centers on 2D-2D graphs
comparisons. Noteworthy representative examples: InfoGraph (Sun et al., 2019)
maximizes the mutual information between the representations of the graph and
its substructures to guide the molecular representation learning; GraphCL (You
et al., 2020a), MoCL (Sun et al., 2021), and MolCLR (Wang et al., 2022b)
employs graph augmentation techniques to construct positive pairs; MoLR (Wang
et al., 2022a) establishes positive pairs with reactant-product relationships.
In addition to 2D-2D graph contrastive learning, there are also noteworthy
efforts exploring 2D-3D and 3D-3D contrastive learning in the field. 3DGCL
(Moon et al., 2023) is 3D-3D contrastive learning model, establishing positive
pairs with conformers from the same molecules. GraphMVP (Liu et al., 2022b),
GeomGCL (Li et al., 2022), and 3D Informax (Stärk et al., 2022) proposes 2D–3D
view contrastive learning approaches. To conclude, 2D-2D and 3D-3D comparisons
are intra-modality contratsive leraning, as only one graph encoder is employed
in these studies. And these approaches often focus on the motif and graph
levels, leaving atom-level contrastive learning less explored.

Multi-Similarity Learning. Instance-wise discrimination, a crucial facet of
similarity learning, involves evaluating the similarity between instances
directly based on their latent representations or features (Wu et al., 2018b).
Naive instance-wise discrimination relies on self-similarity, leading to the
development of contrastive loss (Hadsell et al., 2006). Although there are
improved loss functions such as triplet loss (Hoffer & Ailon, 2015),
quadruplet loss (Law et al., 2013), lifted structure loss (Oh Song et al.,
2016), N-pairs loss (Sohn, 2016), and angular loss (Wang et al., 2017), these
methods still fall short in thoroughly capturing relative similarities (Wang
et al., 2019). To address this limitation, a joint multi-similarity loss has
been proposed, incorporating pair weighting for each pair to enhance instance-
wise discrimination (Wang et al., 2019; Zhang et al., 2021). Notably, it is
crucial to emphasize that employing these pair weightings requires the manual
categorization of negative and positive pairs, as distinct weights are
assigned to losses based on their categories.

##  6 Discussion

In summary, unlike other multi-similarity learning approaches that require
explicit categorization of negative and positive pairs, our method enables a
straightforward generalization of similarity measures, encapsulating both
self-similarities and relative-similarities. Meanwhile, the generalized multi-
similarity metrics satisfy the requirement of convergent similarity learning.
Notably, our model adeptly integrates chemical semantics from diverse
modalities, enhancing its performance across various downstream tasks.
Additionally, our framework bridges machine learning and chemical domain
knowledge through post-hoc experiment by identifying easily synthesizable and
functional substructures, which can be refined into appropriate configurations
by experts. Despite these accomplishments, further exploration is needed to
achieve more effective integration of graph- and node-level similarities.
Looking ahead, we are enthusiastic about the prospect of applying our model to
additional fields, such as social science, thereby broadening its
applicability and impact.

## Accessibility

The code and dataset will be made available upon the date of publication.

## References

  * Bajusz et al. (2015) Bajusz, D., Rácz, A., and Héberger, K.  Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations?  _Journal of cheminformatics_ , 7(1):1–13, 2015.
  * Balcan & Blum (2006) Balcan, M.-F. and Blum, A.  On a theory of learning with similarity functions.  In _Proceedings of the 23rd international conference on Machine learning_ , pp. 73–80, 2006.
  * Clevert et al. (2021) Clevert, D.-A., Le, T., Winter, R., and Montanari, F.  Img2mol–accurate smiles recognition from molecular graphical depictions.  _Chemical science_ , 12(42):14174–14181, 2021.
  * Costanti et al. (2023) Costanti, F., Kola, A., Scarselli, F., Valensin, D., and Bianchini, M.  A deep learning approach to analyze nmr spectra of sh-sy5y cells for alzheimer’s disease diagnosis.  _Mathematics_ , 11(12):2664, 2023.
  * Fang et al. (2022) Fang, X., Liu, L., Lei, J., He, D., Zhang, S., Zhou, J., Wang, F., Wu, H., and Wang, H.  Geometry-enhanced molecular representation learning for property prediction.  _Nature Machine Intelligence_ , 4(2):127–134, 2022.
  * Fang et al. (2023) Fang, Y., Zhang, Q., Zhang, N., Chen, Z., Zhuang, X., Shao, X., Fan, X., and Chen, H.  Knowledge graph-enhanced molecular contrastive learning with functional prompt.  _Nature Machine Intelligence_ , pp. 1–12, 2023.
  * Gerothanassis et al. (2002) Gerothanassis, I. P., Troganis, A., Exarchou, V., and Barbarossou, K.  Nuclear magnetic resonance (nmr) spectroscopy: basic principles and phenomena, and their applications to chemistry, biology and medicine.  _Chemistry Education Research and Practice_ , 3(2):229–252, 2002.
  * Gilmer et al. (2017) Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E.  Neural message passing for quantum chemistry.  In _International conference on machine learning_ , pp. 1263–1272. PMLR, 2017.
  * Hadsell et al. (2006) Hadsell, R., Chopra, S., and LeCun, Y.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06)_ , volume 2, pp. 1735–1742. IEEE, 2006.
  * Halgren (1996) Halgren, T. A.  Merck molecular force field. i. basis, form, scope, parameterization, and performance of mmff94.  _Journal of computational chemistry_ , 17(5-6):490–519, 1996.
  * Heid et al. (2023) Heid, E., Greenman, K. P., Chung, Y., Li, S.-C., Graff, D. E., Vermeire, F. H., Wu, H., Green, W. H., and McGill, C. J.  Chemprop: A machine learning package for chemical property prediction.  _Journal of Chemical Information and Modeling_ , 2023.
  * Hoffer & Ailon (2015) Hoffer, E. and Ailon, N.  Deep metric learning using triplet network.  In _Similarity-Based Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3_ , pp. 84–92. Springer, 2015.
  * Hu et al. (2019) Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J.  Strategies for pre-training graph neural networks.  _arXiv preprint arXiv:1905.12265_ , 2019.
  * Hu et al. (2020) Hu, Z., Dong, Y., Wang, K., Chang, K.-W., and Sun, Y.  Gpt-gnn: Generative pre-training of graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1857–1867, 2020.
  * Jaiswal et al. (2020) Jaiswal, A., Babu, A. R., Zadeh, M. Z., Banerjee, D., and Makedon, F.  A survey on contrastive self-supervised learning.  _Technologies_ , 9(1):2, 2020.
  * Jin et al. (2020) Jin, W., Barzilay, R., and Jaakkola, T.  Multi-objective molecule generation using interpretable substructures, 2020.
  * Lambert et al. (2019) Lambert, J. B., Mazzola, E. P., and Ridge, C. D.  _Nuclear magnetic resonance spectroscopy: an introduction to principles, applications, and experimental methods_.  John Wiley & Sons, 2019.
  * Landrum (2006) Landrum, G.  Rdkit: Open-source cheminformatics. 2006.  _Google Scholar_ , 2006.
  * Law et al. (2013) Law, M. T., Thome, N., and Cord, M.  Quadruplet-wise image similarity learning.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 249–256, 2013.
  * Li et al. (2022) Li, S., Zhou, J., Xu, T., Dou, D., and Xiong, H.  Geomgcl: Geometric graph contrastive learning for molecular property prediction.  In _Proceedings of the Thirty-Six AAAI Conference on Artificial Intelligence_ , pp. 4541–4549, 2022.
  * Liu et al. (2022a) Liu, H., Huang, Y., Liu, X., and Deng, L.  Attention-wise masked graph contrastive learning for predicting molecular property.  _Briefings in bioinformatics_ , 23(5):bbac303, 2022a.
  * Liu et al. (2019) Liu, S., Demirel, M. F., and Liang, Y.  N-gram graph: Simple unsupervised representation for graphs, with applications to molecules.  _Advances in neural information processing systems_ , 32, 2019.
  * Liu et al. (2021a) Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J.  Pre-training molecular graph representation with 3d geometry.  _arXiv preprint arXiv:2110.07728_ , 2021a.
  * Liu et al. (2022b) Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J.  Pre-training molecular graph representation with 3d geometry.  In _International Conference on Learning Representations_ , 2022b.  URL https://openreview.net/forum?id=xQUe1pOKPam.
  * Liu et al. (2021b) Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., and Tang, J.  Self-supervised learning: Generative or contrastive.  _IEEE transactions on knowledge and data engineering_ , 35(1):857–876, 2021b.
  * Liu et al. (2022c) Liu, Y., Jin, M., Pan, S., Zhou, C., Zheng, Y., Xia, F., and Philip, S. Y.  Graph self-supervised learning: A survey.  _IEEE Transactions on Knowledge and Data Engineering_ , 35(6):5879–5900, 2022c.
  * Mahé et al. (2004) Mahé, P., Ueda, N., Akutsu, T., Perret, J.-L., and Vert, J.-P.  Extensions of marginalized graph kernels.  In _Proceedings of the twenty-first international conference on Machine learning_ , pp. 70, 2004.
  * Moon et al. (2023) Moon, K., Im, H.-J., and Kwon, S.  3d graph contrastive learning for molecular property prediction.  _Bioinformatics_ , 39(6):btad371, 2023.
  * Moutafis et al. (2016) Moutafis, P., Leng, M., and Kakadiaris, I. A.  An overview and empirical comparison of distance metric learning methods.  _IEEE transactions on cybernetics_ , 47(3):612–625, 2016.
  * Mu et al. (2023) Mu, E., Guttag, J., and Makar, M.  Multi-similarity contrastive learning.  _arXiv preprint arXiv:2307.02712_ , 2023.
  * Müller et al. (2018) Müller, K.-R., Mika, S., Tsuda, K., and Schölkopf, K.  An introduction to kernel-based learning algorithms.  In _Handbook of neural network signal processing_ , pp. 4–1. CRC Press, 2018.
  * Oh Song et al. (2016) Oh Song, H., Xiang, Y., Jegelka, S., and Savarese, S.  Deep metric learning via lifted structured feature embedding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4004–4012, 2016.
  * Peng et al. (2020) Peng, Z., Dong, Y., Luo, M., Wu, X.-M., and Zheng, Q.  Self-supervised graph representation learning via global context prediction.  _arXiv preprint arXiv:2003.01604_ , 2020.
  * Rong et al. (2020) Rong, Y., Bian, Y., Xu, T., Xie, W., Wei, Y., Huang, W., and Huang, J.  Self-supervised graph transformer on large-scale molecular data.  _Advances in Neural Information Processing Systems_ , 33:12559–12571, 2020.
  * Schroff et al. (2015) Schroff, F., Kalenichenko, D., and Philbin, J.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 815–823, 2015.
  * Sohn (2016) Sohn, K.  Improved deep metric learning with multi-class n-pair loss objective.  _Advances in neural information processing systems_ , 29, 2016.
  * Stärk et al. (2022) Stärk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., Günnemann, S., and Liò, P.  3d infomax improves gnns for molecular property prediction.  In _International Conference on Machine Learning_ , pp. 20479–20502. PMLR, 2022.
  * Steinbeck et al. (2003) Steinbeck, C., Krause, S., and Kuhn, S.  Nmrshiftdb constructing a free chemical information system with open-source components.  _Journal of chemical information and computer sciences_ , 43(6):1733–1739, 2003.
  * Suárez-Díaz et al. (2018) Suárez-Díaz, J. L., García, S., and Herrera, F.  A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges (with appendices on mathematical background and detailed algorithms explanation).  _arXiv preprint arXiv:1812.05944_ , 2018.
  * Sun et al. (2019) Sun, F.-Y., Hoffmann, J., Verma, V., and Tang, J.  Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization.  _arXiv preprint arXiv:1908.01000_ , 2019.
  * Sun et al. (2021) Sun, M., Xing, J., Wang, H., Chen, B., and Zhou, J.  Mocl: data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pp. 3585–3594, 2021.
  * Wang et al. (2022a) Wang, H., Li, W., Jin, X., Cho, K., Ji, H., Han, J., and Burke, M. D.  Chemical-reaction-aware molecule representation learning.  In _International Conference on Learning Representations_ , 2022a.  URL https://openreview.net/forum?id=6sh3pIzKS-.
  * Wang et al. (2017) Wang, J., Zhou, F., Wen, S., Liu, X., and Lin, Y.  Deep metric learning with angular loss.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2593–2601, 2017.
  * Wang et al. (2019) Wang, X., Han, X., Huang, W., Dong, D., and Scott, M. R.  Multi-similarity loss with general pair weighting for deep metric learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 5022–5030, 2019.
  * Wang et al. (2021) Wang, Y., Min, Y., Shao, E., and Wu, J.  Molecular graph contrastive learning with parameterized explainable augmentations.  In _2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pp. 1558–1563. IEEE, 2021.
  * Wang et al. (2022b) Wang, Y., Wang, J., Cao, Z., and Barati Farimani, A.  Molecular contrastive learning of representations via graph neural networks.  _Nature Machine Intelligence_ , 4(3):279–287, 2022b.
  * Wang et al. (2023) Wang, Y., Chen, S., Chen, G., Shurberg, E., Liu, H., and Hong, P.  Motif-based graph representation learning with application to chemical molecules.  In _Informatics_ , volume 10, pp. 8. MDPI, 2023.
  * Wen et al. (2023) Wen, Y., Liu, W., Feng, Y., Raj, B., Singh, R., Weller, A., Black, M. J., and Schölkopf, B.  Pairwise similarity learning is simple.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 5308–5318, 2023.
  * Wieder et al. (2020) Wieder, O., Kohlbacher, S., Kuenemann, M., Garon, A., Ducrot, P., Seidel, T., and Langer, T.  A compact review of molecular property prediction with graph neural networks.  _Drug Discovery Today: Technologies_ , 37:1–12, 2020.
  * Wu et al. (2018a) Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V.  Moleculenet: a benchmark for molecular machine learning.  _Chemical science_ , 9(2):513–530, 2018a.
  * Wu et al. (2018b) Wu, Z., Xiong, Y., Yu, S. X., and Lin, D.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3733–3742, 2018b.
  * Xia et al. (2022) Xia, J., Zhao, C., Hu, B., Gao, Z., Tan, C., Liu, Y., Li, S., and Li, S. Z.  Mole-bert: Rethinking pre-training graph neural networks for molecules.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Xiong et al. (2019) Xiong, Z., Wang, D., Liu, X., Zhong, F., Wan, X., Li, X., Li, Z., Luo, X., Chen, K., Jiang, H., et al.  Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism.  _Journal of medicinal chemistry_ , 63(16):8749–8760, 2019.
  * Xu et al. (2023a) Xu, H., Wang, Y., Li, Y., and Hong, P.  Asymmetric contrastive multimodal learning for advancing chemical understanding.  _arXiv preprint arXiv:2311.06456_ , 2023a.
  * Xu et al. (2023b) Xu, H., Zhou, Z., and Hong, P.  Molecular identification and peak assignment: Leveraging multi-level multimodal alignment on nmr.  _arXiv preprint arXiv:2311.13817_ , 2023b.
  * Yang et al. (2019) Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al.  Analyzing learned molecular representations for property prediction.  _Journal of chemical information and modeling_ , 59(8):3370–3388, 2019.
  * Yang & Jin (2006) Yang, L. and Jin, R.  Distance metric learning: A comprehensive survey.  _Michigan State Universiy_ , 2(2):4, 2006.
  * Yang et al. (2021) Yang, Z., Song, J., Yang, M., Yao, L., Zhang, J., Shi, H., Ji, X., Deng, Y., and Wang, X.  Cross-modal retrieval between 13c nmr spectra and structures for compound identification using deep contrastive learning.  _Analytical Chemistry_ , 93(50):16947–16955, 2021.
  * You et al. (2020a) You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y.  Graph contrastive learning with augmentations.  _Advances in neural information processing systems_ , 33:5812–5823, 2020a.
  * You et al. (2020b) You, Y., Chen, T., Wang, Z., and Shen, Y.  When does self-supervision help graph convolutional networks?  In _international conference on machine learning_ , pp. 10871–10880. PMLR, 2020b.
  * Zhang et al. (2023) Zhang, C., Luo, L., and Gu, B.  Denoising multi-similarity formulation: a self-paced curriculum-driven approach for robust metric learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pp. 11183–11191, 2023.
  * Zhang et al. (2021) Zhang, L., Shen, S., Li, L., Wang, H., Li, X., and Lang, J.  Jointly multi-similarity loss for deep metric learning.  In _2021 IEEE International Conference on Data Mining (ICDM)_ , pp. 1469–1474. IEEE, 2021.
  * Zhang et al. (2022) Zhang, Z., Chen, L., Zhong, F., Wang, D., Jiang, J., Zhang, S., Jiang, H., Zheng, M., and Li, X.  Graph neural network approaches for drug-target interactions.  _Current Opinion in Structural Biology_ , 73:102327, 2022.
  * Zhou et al. (2023) Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., Zhang, L., and Ke, G.  Uni-mol: A universal 3d molecular representation learning framework.  In _International Conference on Learning Representations_ , 2023.  URL https://api.semanticscholar.org/CorpusID:259298651.

Appendix

##  Appendix A Multi-Similarity & Contrastive Learning

###  A.1 Multi-Similarities in Contrastive Learning

Two distinct types of similarities, as illustrated in Appendix Figure A.1, can
be identified: self-similarity (the pairwise similarity between two objects,
typically defined through cosine similarity) and relative similarity
(distinctions in self-similarity with other pairs) (Wang et al., 2019).

Figure A.1: Illustration of Different Types of Similarities.

###  A.2 Current Molecular Graph Contrastive Learning Approaches

In current molecular graph contrastive learning approaches, positive pairs are
commonly formed through either data augmentation (Sun et al., 2021; You et
al., 2020a), employing techniques such as node deletion, edge perturbation,
subgraph extraction, attribute masking, and subgraph substitution, or domain
knowledge, as demonstrated by reactant-product pairing (Wang et al., 2022a) or
conformer grouping (Moon et al., 2023).

Figure A.2: General framework of Intra-Modality Graph Contrastive Learning. It
relies on definition of positive and negative pairs.

##  Appendix B Supplementary Proof

###  B.1 Revisiting Theorem of Convergent Similarity Learning

Let $\mathcal{S}$ be a set of instances with size $|\mathcal{S}|$, and let
$\mathcal{P}$ represent the tunable latent representations of instances in
$\mathcal{S}$ such that $|\mathcal{P}|=|\mathcal{S}|$. For any two instances
$i,j\in\mathcal{S}$, their latent representations are denoted by
$\mathcal{P}_{i}$ and $\mathcal{P}_{j}$, respectively. Let $t_{i,j}$ represent
the target similarity between instances $i$ and $j$ in a given domain, and
$d_{i,j}$ be the similarity between $\mathcal{P}_{i}$ and $\mathcal{P}_{j}$ in
the latent space.

######  Theorem B.1 (Theorem of Convergent Similarity learning).

Given $t_{i,j}$ is non-negative and $\\{t_{i,j}\\}$ satisfies the constraint
$\sum_{j=1}^{|\mathcal{S}|}t_{i,j}=1$, consider the loss function for an
instance $i$ defined as follows:

$L(i)=-\sum_{j=1}^{|\mathcal{S}|}t_{i,j}\log\left(\frac{e^{d_{i,j}}}{\sum_{k=1}^{|\mathcal{S}|}e^{d_{i,k}}}\right)$
(B.1)

then when it reaches ideal optimum, the relationship between $t_{i,j}$ and
$d_{i,j}$ satisfies:

$\text{softmax}(d_{i,j})=t_{i,j}$ (B.2)

###### Proof.

In order to optimize the loss $L(i)$, we need to set the following partial
derivative to be 0 for each $d_{i,j}$ with $1\leq j\leq|\mathcal{M}|$. Here
are the detailed steps:

$\displaystyle\frac{\partial L(i)}{\partial{d_{i,j}}}$
$\displaystyle=\frac{\partial}{\partial
d_{i,j}}\underbrace{\left(-t_{i,j}\log\frac{e^{d_{i,j}}}{e^{d_{i,j}}+\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}e^{d_{i,k}}}\right)}_{\text{When the numerator includes
}e^{d_{i,j}}}+\frac{\partial}{\partial
d_{i,j}}\underbrace{\left(\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}-t_{i,k}\log\frac{e^{d_{i,k}}}{e^{d_{i,j}}+\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}e^{d_{i,k}}}\right)}_{\text{When the numerator does not
include }e^{d_{i,j}}}$
$\displaystyle=-(t_{i,j}-t_{i,j}\cdot\text{softmax}(d_{i,j}))-\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}t_{i,k}\cdot\text{softmax}(d_{i,j})$
$\displaystyle=-\left(t_{i,j}-\left(t_{i,j}+\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}t_{i,k}\right)\cdot\text{softmax}(d_{i,j})\right)$

Since $\sum_{l=1}^{|\mathcal{M}|}t_{i,l}=1$, we can further simplify it as

$\displaystyle\frac{\partial
L(i)}{\partial{d_{i,j}}}=-(t_{i,j}-\text{softmax}(d_{i,j}))$

In order to optimize, we need to see the above partial derivative to be 0:

$\displaystyle\frac{\partial
L(i)}{\partial{d_{i,j}}}=-(t_{i,j}-\text{softmax}(d_{i,j}))=0$

In addition, the corresponding second partial derivative denoted as
$\frac{\partial L(i)}{\partial{d^{2}_{i,j}}}$ manifests as follows:

$\displaystyle\frac{\partial
L(i)}{\partial{d^{2}_{i,j}}}=\text{softmax}(d_{i,j})(1-\text{softmax}(d_{i,j}))$

As $\text{softmax}(d_{i,j})$ takes values within the open interval (0,1), it
follows that $\frac{\partial L(i)}{\partial{d^{2}_{i,j}}}$ is always positive.
Consequently, the global optimum is global minimum.
Furthermore, when it comes to optimum:

$\displaystyle t_{i,j}$ $\displaystyle=\text{softmax}(d_{i,j})$ $\displaystyle
d_{i,j}$ $\displaystyle=\log(t_{i,j})+\log\left(\sum_{\begin{subarray}{c}1\leq
l\leq|\mathcal{M}|\end{subarray}}e^{d_{i,j}}\right)$

It is easy to show that when it reaches optimum, $d_{i,j}$ is consistent with
target similarity metric $t_{i,j}$. Without loss of generosity, suppose
$t_{i,j}>t_{i,j^{\prime}}$ :

$\displaystyle d_{i,j}-d_{i,j^{\prime}}$
$\displaystyle=\log(t_{i,j})+\log\left(\sum_{\begin{subarray}{c}1\leq
l\leq|\mathcal{M}|\end{subarray}}e^{d_{il}}\right)-\left(\log(t_{i,j^{\prime}})+\log\left(\sum_{\begin{subarray}{c}1\leq
l\leq|\mathcal{M}|\end{subarray}}e^{d_{il}}\right)\right)$
$\displaystyle=\log(t_{i,j})-\log(t_{i,j^{\prime}})$
$\displaystyle=\log\left(\frac{t_{i,j}}{t_{i,j^{\prime}}}\right)>0$

∎

###  B.2 Guarantee of Sum of Fused Multimodal Similarity

Given sets of uni-modal generalized similarity $\\{t^{R}\\}$ and $\sum
w_{t^{R}}=1$, the sum of fused multimodal similarity also equals 1, as
demonstrated below:

$\displaystyle\sum(t_{i,j}^{R})$ $\displaystyle=\sum\sum(w_{R}\cdot
t^{R}_{i,j})$ $\displaystyle=\sum(w_{R}\sum t^{R}_{i,j})$ $\displaystyle=\sum
w_{R}\cdot 1=1$

##  Appendix C Revisiting Multi-Similarity Settings

###  C.1 Encoders & Packages

To derive the self-similarities, we need to reply on pre-trained encoders or
well-defined packages as follows:

Table C.1: Encoders and packages used to produce self-similarities Unimodal | Representation | Encoder/Package | Pre-trained Source
---|---|---|---
Image | 2D image | CNN |  Img2mol (Clevert et al., 2021)
SMILES | Sequence | Transformer |  CReSS (Yang et al., 2021)
13CNMR Spectrum | Sequence | 1D CNN |  AutoEncoder (Costanti et al., 2023)
13CNMR peak | Scalar |  NMRShiftDB2 (Steinbeck et al., 2003) | N/A
Fingerprint | Sequence |  RDKit (Landrum, 2006) | N/A

###  C.2 Self-Similarity at Graph Level

Fingerprint. The mathematical formula of fingerprint similarity, denoted as
$S_{i,j}^{F}$, can be viewed as follows:

$\displaystyle S_{i,j}^{F}$ $\displaystyle=Tanimoto(A,B)=\frac{|A\cap
B|}{|A\cup B|}$ (C.1)

where $A$ and $B$ are sets of molecular fragments for molecule $i$ and $j$,
and $|A\cap B|$ and $|A\cup B|$ denote the size of their intersection and
union, respectively.

Image. The self-similarity for Image, denoted as $S_{i,j}^{I}$, can be defined
as follows:

$S_{i,j}^{I}=Cos(\mathcal{V}_{i},\mathcal{V}_{j})=\frac{\mathcal{V}_{i}\cdot\mathcal{V}_{j}^{T}}{\|\mathcal{V}_{i}\|\cdot\|\mathcal{V}_{j}\|}$
(C.2)

where $\mathcal{V}_{i},\mathcal{V}_{j}$ represents the embedding of Image for
two given molecules.

NMR Spectrum. The self-similarity for NMR spectrum, denoted as $S_{i,j}^{C}$,
can be defined as follows:

$S_{i,j}^{C}=Cos(\mathcal{V}_{i},\mathcal{V}_{j})=\frac{\mathcal{V}_{i}\cdot\mathcal{V}_{j}^{T}}{\|\mathcal{V}_{i}\|\cdot\|\mathcal{V}_{j}\|}$
(C.3)

where $\mathcal{V}_{i},\mathcal{V}_{j}$ represents the embedding of NMR
spectra for two given molecules.

Smiles. The self-similarity for Smiles, denoted as $S_{i,j}^{S}$, can be
defined as follows:

$S_{i,j}^{S}=Cos(\mathcal{V}_{i},\mathcal{V}_{j})=\frac{\mathcal{V}_{i}\cdot\mathcal{V}_{j}^{T}}{\|\mathcal{V}_{i}\|\cdot\|\mathcal{V}_{j}\|}$
(C.4)

where $\mathcal{V}_{i},\mathcal{V}_{j}$ represents the embedding of Smiles for
two given molecules.

###  C.3 A Brief Introduction to PPM

In chemistry, 13C NMR stands out as a common technique for structural analysis
by revealing molecular structures by elucidating the chemical environments of
carbon atoms and their magnetic responses to external fields (Gerothanassis et
al., 2002; Lambert et al., 2019). It quantifies these features in parts per
million (ppm) relative to a reference compound, such as tetramethylsilane
(TMS), thereby simplifying comparisons across experiments. As a result, the
continuous peak positions, measured in parts per million (ppm), offer a robust
knowledge span—a natural ordering metric that can be employed to derive
measures of similarity (Xu et al., 2023b).

###  C.4 Configuration of Fused Multimodal Generalized Similarity Metric

A simple linear combination is used to formulate the multimodal multi-
similarity $t_{i,j}^{M}$ between the $i^{th}$ and $j^{th}$ molecules,
represented as a graph-level similarity $t_{i,j}^{g}$, as follows:

$\displaystyle t_{i,j}^{g}$ $\displaystyle=t_{i,j}^{M}=w_{SM}\cdot
t^{SM}_{i,j}+w_{{C}}\cdot t^{C}_{i,j}+w_{{I}}\cdot t^{I}_{i,j}+w_{{F}}\cdot
t^{F}_{i,j}$ (C.5)

where $t^{SM}_{i,j}$ denotes the similarity based on SMILES, $t^{C}_{i,j}$
denotes the similarity with respect to 13C NMR spectrum, $t^{I}_{i,j}$ denotes
the similarity regarding images, and $f$ denotes the similarity based on
fingerprints, $w_{SM}$, $w_{C}$, $w_{I}$, and $w_{F}$ are the pre-defined
weights for their respective similarity, and
$w_{SM}+w_{{C}}+w_{{I}}+w_{{F}}=1$.

For pre-defined weights, denoted as $w_{SM}$, $w_{C}$, $w_{I}$, and $w_{F}$,
we configure them with various settings for ablation studies, as shown in the
following table:

Table C.2: The configuration of weight for each unimodality similarity in different Fusion. In particular $w_{S}$, $w_{N}$, $w_{M}$ and $w_{F}$ represents the weight of Smiles, NMR, Image and Fingerprint, respectively. Fused multimodal | $w_{SM}$ | $w_{N}$ | $w_{M}$ | $w_{F}$
---|---|---|---|---
Smiles | 1.00 | 0.00 | 0.00 | 0.00
NMR | 0.00 | 1.00 | 0.00 | 0.00
Image | 0.00 | 0.00 | 1.00 | 0.00
Fingperprint | 0.00 | 0.00 | 0.00 | 1.00
$\text{Fusion}_{\text{Smiles}}$ | 0.70 | 0.10 | 0.10 | 0.10
$\text{Fusion}_{\text{NMR}}$ | 0.10 | 0.70 | 0.10 | 0.10
$\text{Fusion}_{\text{Image}}$ | 0.10 | 0.10 | 0.70 | 0.10
$\text{Fusion}_{\text{Fingerprint}}$ | 0.10 | 0.10 | 0.10 | 0.70
$\text{Fusion}_{\text{Average}}$ | 0.25 | 0.25 | 0.25 | 0.25

##  Appendix D Experimental Settings

###  D.1 Pre-Training Setting

During pretraining, we utilized an Adam optimizer with a learning rate set to
0.001, spanning 200 epochs and employing a batch size of 256. The model was
trained on 30,000 data points. The NMR data were experimental data, extracted
from NMRShiftDB2 (Steinbeck et al., 2003). Other chemical modalities, such as
images, fingerprints and graphs, were produced from SMILES by RDKit (Landrum,
2006).

###  D.2 Fine-Tuning Setting

####  D.2.1 Datasets

For fine-tuning, our model was trained on 11 drug discovery-related benchmarks
sourced from MoleculeNet (Wu et al., 2018a). Eight of these benchmarks were
designated for classification downstream tasks, including BBBP, BACE, SIDER,
CLINTOX, HIV, MUV, TOX21, and ToxCast, while three were allocated for
regression tasks, namely ESOL, Freesolv, and Lipo. The datasets were divided
into train/validation/test sets using a ratio of 80%:10%:10%, accomplished
through the scaffold splitter (Halgren, 1996; Landrum, 2006) from Chemprop
(Yang et al., 2019; Heid et al., 2023), like previous works. The scaffold
splitter categorizes molecular data based on substructures, ensuring diverse
structures in each set. Molecules are partitioned into bins, with those
exceeding half of the test set size assigned to training, promoting scaffold
diversity in validation and test sets. Remaining bins are randomly allocated
until reaching the desired set sizes, creating multiple scaffold splits for
comprehensive evaluation.

####  D.2.2 Baselines

We systematically compared GraphMSL’s performance with various state-of-the-
art baseline models across different categories. In the realm of supervised
models, AttentiveFP (Xiong et al., 2019) and DMPNN (Yang et al., 2019) stand
out by leveraging graph attention networks and node-edge interactive message
passing, respectively. The unsupervised learning method N-Gram (Liu et al.,
2019) employs graph embeddings and short walks for graph representation.
Predictive self-supervised learning methods, such as GEM (Fang et al., 2022)
and Uni-Mol (Zhou et al., 2023), are specifically designed for predicting
molecular geometric information. GROVER (Rong et al., 2020) integrates Message
Passing Networks into a Transformer-style architecture, creating a class of
more expressive encoders for molecules. Moreover, our evaluation encompasses a
range of contrastive learning methods, namely InfoGraph (Sun et al., 2019),
GraphCL (You et al., 2020a), MolCLR (Wang et al., 2022b), and GraphMVP (Liu et
al., 2022b), all serving as essential baselines. The baseline results are
collected from recent works (Fang et al., 2022; Zhou et al., 2023; Moon et
al., 2023; Fang et al., 2023).

####  D.2.3 Evaluation

To assess the effectiveness of our fine-tuned model, we measure the ROC-AUC
for classification downstream tasks, and the root mean squared error (RMSE)
metric for regression tasks. In order to ensure a fair and robust comparisons,
we conduct three independent runs using three different random seeds for
scaffold splitting across all datasets. The reported performance metrics are
then averaged across these runs, and the standard deviation is computed as
prior works.

##  Appendix E More Ablation Study

$\text{GraphMSL}_{\text{bi-level}}$ demonstrates top performance in BBBP,
BACE, CLINTOX, HIV, MUV, and Tox21. Additionally, within the GraphMSLbi-level
framework, FusionSmiles enhances performance in Sider, CLINTOX, and MUV.
FusionNMR improves results in BBBP and HIV, while
$\text{Fusion}_{\text{Image}}$ contributes to better outcomes in the Tox21 and
ToxCast tasks.

Table E.1: Ablation study on the performances of GraphMSL with bi-level (Graph + Node) and solo-node-level. The best results are denoted in boldf, and the second-best are indicated with underlining. Data Set | BBBP | bace | Sider | Clintox | HIV | MUV | Tox21 | ToxCast | ESOL | FreeSolv | Lipo
---|---|---|---|---|---|---|---|---|---|---|---
SMILES + Node |  93.7$\pm$1.3 |  93.2$\pm$3.7 |  65.9$\pm$1.7 |  87.7$\pm$7.8 |  82.3$\pm$1.8 |  80.9$\pm$5.5 |  84.7$\pm$1.7 |  70.4$\pm$1.5 |  0.873$\pm$ 0.085 |  1.658$\pm$ 0.243 |  0.594$\pm$ 0.031
NMR + Node |  91.9$\pm$3.1 |  92.8$\pm$1.6 |  65.7$\pm$1.6 |  89.5$\pm$3.4 |  81.2$\pm$1.2 |  80.6$\pm$5.5 |  85.1$\pm$0.2 |  69.3$\pm$0.1 |  1.052$\pm$ 0.105 |  2.391$\pm$ 0.175 |  0.654$\pm$ 0.025
Image + Node |  94.1$\pm$1.7 |  90.8$\pm$0.7 |  63.8$\pm$2.8 |  86.5$\pm$8.0 |  80.3$\pm$1.5 |  76.7$\pm$2.7 |  85.3$\pm$1.1 |  70.8$\pm$1.5 | 0.843$\pm$ 0.094 | 1.601$\pm$ 0.057 | 0.562$\pm$ 0.005
Fingerprint + Node |  90.2$\pm$8.4 | 94.5$\pm$0.7 |  64.3$\pm$2.9 |  91.0$\pm$1.5 |  82.0$\pm$2.4 |  79.2$\pm$5.9 |  85.7$\pm$0.5 |  69.7$\pm$1.3 |  1.170$\pm$ 0.174 |  2.801$\pm$ 0.276 |  0.607$\pm$ 0.034
$\text{Fusion}_{\text{Smiles}}$ \+ Node |  91.7$\pm$5.3 |  91.5$\pm$1.7 | 67.3$\pm$0.6 | 93.8$\pm$0.8 |  82.1$\pm$1.7 | 81.5$\pm$3.7 |  85.1$\pm$0.1 |  70.4$\pm$1.3 |  0.965$\pm$0.085 |  2.859$\pm$0.281 |  0.647$\pm$0.029
$\text{Fusion}_{\text{NMR}}$ \+ Node | 94.3$\pm$0.8 |  93.6$\pm$1.9 |  66.8$\pm$1.4 |  90.4$\pm$3.1 | 83.0$\pm$0.7 |  80.1$\pm$3.5 |  85.5$\pm$0.6 |  70.6$\pm$1.8 |  1.009$\pm$0.160 |  2.224$\pm$0.368 |  0.589$\pm$0.038
$\text{Fusion}_{\text{Image}}$ \+ Node |  94.2$\pm$1.2 |  93.1$\pm$2.5 |  66.4$\pm$1.6 |  90.7$\pm$3.5 |  82.0$\pm$2.4 |  80.8$\pm$3.8 | 86.1$\pm$0.8 | 71.2$\pm$1.1 |  0.898$\pm$0.098 |  1.691$\pm$0.386 |  0.579$\pm$0.018
$\text{Fusion}_{\text{Fingerprint}}$ \+ Node |  91.6$\pm$5.0 |  94.3$\pm$2.4 |  66.4$\pm$1.9 |  85.3$\pm$6.8 |  82.0$\pm$2.4 |  80.6$\pm$3.2 |  85.2$\pm$0.2 |  69.8$\pm$1.1 |  1.037$\pm$0.170 |  2.093$\pm$0.090 |  0.607$\pm$0.034
$\text{Fusion}_{\text{Average}}$ \+ Node |  92.7$\pm$1.5 |  92.6$\pm$2.1 |  65.6$\pm$0.7 |  89.3$\pm$4.0 |  81.8$\pm$1.7 |  81.0$\pm$5.0 |  85.4$\pm$1.3 | 71.2$\pm$1.9 |  1.019$\pm$0.118 |  1.733$\pm$0.267 |  0.593$\pm$0.004
Node |  93.4$\pm$2.7 |  89.3$\pm$1.7 |  62.8$\pm$2.1 |  86.1$\pm$5.4 |  82.1$\pm$0.4 |  75.4$\pm$5.2 |  84.9$\pm$1.0 |  70.6$\pm$0.8 |  0.924$\pm$0.083 |  1.707$\pm$0.126 |  0.587$\pm$0.021