# 3D-Mol: A Novel Contrastive Learning Framework for Molecular Property
Prediction with 3D Information

Taojie Kuang
Peng Cheng Laboratory
South China University of Technology
<EMAIL_ADDRESS>
Yiming Ren
Peng Cheng Laboratory
<EMAIL_ADDRESS>
Zhixiang Ren
Peng Cheng Laboratory
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Molecular property prediction offers an effective and efficient approach for
early screening and optimization of drug candidates. Although deep learning
based methods have made notable progress, most existing works still do not
fully utilize 3D spatial information. This can lead to a single molecular
representation representing multiple actual molecules. To address these
issues, we propose a novel 3D structure-based molecular modeling method named
3D-Mol. In order to accurately represent complete spatial structure, we design
a novel encoder to extract 3D features by deconstructing the molecules into
three geometric graphs. In addition, we use 20M unlabeled data to pretrain our
model by contrastive learning. We consider conformations with the same
topological structure as positive pairs and the opposites as negative pairs,
while the weight is determined by the dissimilarity between the conformations.
We compare 3D-Mol with various state-of-the-art baselines on 7 benchmarks and
demonstrate our outstanding performance in 5 benchmarks.

_K_ eywords Deep Learning $\cdot$ Molecular representation $\cdot$ Molecular
property prediction $\cdot$ Graph neural network $\cdot$ Self-supervised
learning $\cdot$ Contrastive learning

##  1 Introduction

Molecular property prediction accelerates drug candidate identification,
saving time and resources. It helps researchers prioritize promising
compounds, streamlining drug development and increasing success rates.
Moreover, it aids in understanding structure-activity relationships, revealing
how specific features impact properties, interactions, and biological effects.
Although deep learning has achieved success in molecular property prediction,
its potential is significantly constrained by the scarcity of labeled data
because labeled data for molecular properties usually requires expensive and
time-consuming experiments[1].
Self-supervised learning uses large amounts of unlabeled data to pretrain
models to leverage unlabeled data to learn rich feature representations. Many
works[2, 3, 4] improve their model performance through self-supervised
learning. Early deep learning methods for molecular property prediction[5, 6,
7, 8, 9] effectively utilized NLP-based self-supervised learning methods to
handle data represented by the SMILES molecular formula[10]. However, SMILES
can’t fully reflect the topological relationships of a molecule. Therefore,
many self-supervised methods based on molecular graphs have emerged, such as
PretrainGNN[11], N-Gram-graph[12], MolCLR[13], GROVER[14]. They designed
unique pretraining methods based on molecular graph. Besides, lots of work use
graph neural networks to capture the topological information of molecules,
such as MPNN[15], AttentiveFP[16] and D-MPNN[17]. However, these methods have
not dealt with the 3D spatial information of molecules, which is critical for
predicting molecular properties. As shown in Figure 1, Thalidomide is divided
into two forms, R-Thalidomide and S-Thalidomide, due to different 3D
structures. The former can be used to treat skin diseases, while the latter
has been implicated in teratogenesis. Recently, some integrating geometric
information into graph structures has attracted research attention in some
molecular property estimation tasks[18, 19, 20, 21, 22, 23, 24, 25, 22], they
either fail to fully extract the 3D information of molecules or only use
masking methods for data augmentation in self-supervised learning, and only
use the stablest conformation but ignore the others.

Figure 1: Thalidomide exists in two distinct 3D stereoisomeric forms, known as
R-Thalidomide and S-Thalidomide. The former is recognized for its therapeutic
properties in the treatment of various skin conditions, but the latter has
been implicated in teratogenesis. It shows that despite having identical 2D
molecular topology, the properties of two molecules vary significantly due to
their distinct 3D structures.

To address these issues, we propose a novel method, 3D-Mol, for molecular
property prediction. Firstly, we use atom-bond graph, bond-angle graph and
plane-angle graph to represent the spatial structural information of
molecules, extracting the 3D spatial structure representation of molecules
through the information transfer within these three graphs and their
interactions. In pretraining stage, we design a unique weighted contrastive
learning method, which uses the different 3D conformations of the same SMILES
as weighted positive pair SMILES, and the weight is dependent on the
difference between those conformations. We also employ the geometry
pretraining task following GearNet[26]. We learn 3D structural feature of
molecular representations from a large volume of unlabeled data, and then
finetune the well-pretrained model according to downstream tasks and data to
predict molecular properties. We compared our approach with several state-of-
the-art baselines on 7 molecular property prediction benchmarks[27], where our
method achieved the best results on 5 benchmarks. In summary, our main
contributions are as follows:
$\bullet$ We propose a novel molecular representation method and design a
corresponding model to fully extract the 3D spatial structural features of
molecules.
$\bullet$ We design a unique weighted contrastive learning task, using the
different 3D conformations from the same SMILES as weighted positive pair, and
the weight is dependent on the difference between the conformations, thereby
deeply learning the microscopic characteristics of molecular 3D space.
$\bullet$ We have conducted thorough evaluations of the 3D-Mol model on
various molecular property prediction datasets. Experimental results show that
3D-Mol significantly outperforms existing competitive models in multiple
benchmark tests.

##  2 Related Work

In general, there are two strategies to improve molecular property prediction.
One is to design a novel molecular encoder based on molecular information for
effective latent vector extraction, the other is to design a novel pretraining
method to pretrain the molecular encoder by using a large amount of unlabeled
data. The following are the related works of each.

###  2.1 Molecular Representation and Encoder

Proposing a novel molecular representation and encoder method is usually the
first option for researchers to improve the accuracy of molecular property
prediction. Some early works learn representation from chemical fingerprints,
such as ECFP[21] and MACCS[28], frequently used in early machine learning[21,
29, 30]. The other learns representation from molecular descriptors, such as
SMILES[10]. Inspired by mature NLP models, SMILES-BERT[31] applies the BERT[2]
strategy to pretrain on SMILES to extract molecular representations. However,
These methods depend on feature engineering, failing to capture the complete
topological structure of molecules.
Recently, many works use molecular graph as molecular representation because
the natural representation of molecule is molecular graph and it represents
the topology information. GG-NN[15], DMPNN[17], and DeepAtomicCharge[32]
employed a message passing scheme for molecular property prediction.
AttentiveFP[16] uses a graph attention network to aggregate and update node
information. The MP-GNN[33] merges specific-scale Graph Neural Networks and
element-specific Graph Neural Networks, capturing various atomic interactions
of multiphysical representations at different scales. MGCN[34] designed a GCN
to capture multilevel quantum interactions from the conformation and spatial
information of molecules. But these works focuses on 2D molecular
representation, and extracting only 2D topological information from molecules
is insufficient.
Many works[35] show the necessity of using 3D spatial information of
molecules. Recently, some research has also begun modeling 3D molecules to
address this issue. SGCN[25] applies different weights according to atomic
distances during the GCN-based message passing process. SchNet[18] models
complex atomic interactions using Gaussian radial basis functions for
potential energy surface predictionor to accelerate the exploration of
chemical space. DimeNet[22] proposes directional message passing to fully
utilize directional information within molecules. [23] develops a novel
geometrically-enhanced molecular representation learning method, and employs a
specifically designed geometric-based graph neural network structure. However,
these methods do not fully exploit the 3D structural information of molecules
and lack the ability to learn the representations of 3D conformations with the
same molecular topology.

###  2.2 Self-supervised Learning on Molecules

Self-supervised learning has achieved enormous success in BERT[2] and GPT[36].
Inspired by these, numerous works for molecular property prediction use this
approch to effectively utilize large amounts of unlabeled data for
pretraining.
For one-dimensional data, SMILES is frequently used to extract molecular
feature in pretraining stage. ChemBERTa[9] followed RoBERTa[37] by employing
masked language modeling (MLM) as a pretraining task, predicting masked tokens
to restore the original sentence, aiding pretraining models to understand
sequence semantics. SMILES Transformer[38] used a SMILES string as inputb to
produce a temporary embedding, which is then restored to the original input
SMILES by a decoder.
As the topological information of molecular graphs is gaining more attention,
many pretraining methods aimed at graph data have been proposed. Shengchao Liu
et al[12] used the n-gram method in NLP to extract and represent features of
molecules. PretrainGNN[11] proposed a new pretraining strategy, include node-
level and graph-level self-supervised pretraining tasks. GraphCL[39], MOCL[40]
and MolCLR[13] performed molecular contrastive learning via graph neural
networks, proposed new molecule augmentation methods. MPG[41] and GROVER[14]
focused on node level and graph level representation and corresponding
pretraining tasks for node level and graph level. iMolCLR[42], Sugar[43] and
ReLMole[44] focused on the substructure of molecule, and designed the
substructure pretraining task by using substructure information. However, the
aforementioned pretraining strategies are only targeted at learning the
topology information of the molecule.
With the 3D information of molecules proven to aid in the prediction of
molecular representations, recent works have focused on pretraining task for
the 3D structure information of molecules. 3DGCN[45] introduced a relative
position matrix that includes 3D positions between atoms to ensure
translational invariance during convolution. GraphMVP[46] proposed a SSL
method involving contrastive learning and generative learning between 3D and
2D molecular views. [23] proposed a self-supervised framework using molecular
geometric information. They constructed a new bond angle graph, where the
chemical bonds within a molecule are considered as nodes instead of edges, and
the angle formed between two bonds is considered as the edge between them.
Uni-Mol[47] employed the transformer to extract molecular representation by
predicting atom distance. Although these works have used the spatial
information of molecules, but they have not fully utilized the spatial
information of molecules, nor have they enabled the model to learn the
representation of geometric isomers. To address these issues, we use RDKit to
generate multiple conformations from same topology structure as positive pairs
and design a weighted contrastive learning task for self-supervised training.

Figure 2: The overview of the 3D-Mol model framework. a) In the pretraining
stage, we employ weighted contrastive learning to effectively pretrain our
model. In addition to using the mask strategy for graph data augmentation, we
consider conformations stemming from the same topological structure as
positive pairs, with their weight determined by the dissimilarity between the
conformations. Conversely, distinct topological structures are treated as
negative pairs, and we further utilize fingerprint differences to compute the
weight of negative pairs. b) In the finetuning stage, we refine the well-
pretrained encoder using diverse downstream datasets, followed by supervised
learning.

##  3 Method

###  3.1 Molecular Representation

We deconstruct molecular conformation into three graph, denoted as
$Mol=\\{G_{a-b},G_{b-a},G_{p-a}\\}$. In most databases, molecular raw data is
represented by SMILES. To extract topological and spatial structure
information from molecules, we need to use RDKit to transform the SMILES
representation into molecular conformations. In our method, we decompose
molecular conformation into three graphs. The first graph, named atom-bond
graph, is the commonly used as 2D molecular graph, and is represented as
$G_{a-b}=\\{V,E,P_{atom},P_{bond}\\}$, where $V$ is the set of atoms and $E$
is the set of bonds. $P_{atom}\in R^{|V|*d_{atom}}$, and are the attributes of
atoms, and $d_{atom}$ is the number of atom attributes. $P_{bond}\in
R^{|E|*d_{bond}}$, and is the attributes of bonds, and $d_{bond}$ is the
number of bond attributes. The second graph, named bond-angle graph, is
represented as $G_{b-a}=\\{E,P,Ang_{\theta}\\}$, where $P$ is a set of the
plane that is comprised with 3 connected atoms. $Ang_{\theta}$ is the set of
corresponding bond angle $\theta$. The third graph, named the plane-angle
graph, is represented as $G_{p-a}=\\{P,C,Ang_{\phi}\\}$. $C$ represents the
set of two connected planes, which connect with a bond. $Ang_{\phi}$
represents the corresponding dihedral angle $\phi$. The first graph allows the
model to learn the topological information of molecules, and the second and
third graphs allow the model to learn the spatial structure information of
molecules.

###  3.2 Attribute Embedding Layer

The 3D information of the molecule, such as the length of bonds and the angle
between bonds, carries key chemical information. Firstly, we convert float
numbers, like angle and bond length, to latent vectors. Referring to the
previous work (Shui and Karypis 2020)[24], we employed several RBF layers to
encode different geometric factors:

$F^{k}_{l}=exp(-\beta^{k}_{l}(exp(-l)-\mu^{k}_{l})^{2})*W^{k}_{l}$ (1)

where $F^{k}_{l}$ is the k-dimensional feature of bond length $l$, and
$\mu^{k}_{l}$ and $\beta^{k}_{l}$ are the center and width of $l$
respectively. $\mu^{k}_{l}$ is 0.1$k$ and $\beta^{k}_{l}$ is 10. Similarly,
the k-dimensional feature of $F^{k}_{\theta}$ and $F^{k}_{\phi}$ of x is
computed as:

$F^{k}_{\theta}=exp(-\beta^{k}_{\theta}(-\theta-\mu^{k}_{\theta})^{2})*W^{k}_{\theta}$
(2)
$F^{k}_{\phi}=exp(\beta^{k}_{\phi}(-\phi-\mu^{k}_{\phi})^{2})*W^{k}_{\phi}$
(3)

where $\mu^{k}_{\theta}$ and $\beta^{k}_{\theta}$ are the center and width of
$\theta$, and $\mu^{k}_{\phi}$ and $\beta^{k}_{\phi}$ are the center and width
of $\phi$. Centers of bond angle and dihedral angle are represented as
$\mu^{k}_{\phi}$ and $\mu^{k}_{\theta}$ respectively, and the numerical value
of them are $\pi$/K, where K is the number of feature dimensions.
For the other property of atom and bond, we represent them with $P_{atom}$ and
$P_{bond}$ respectively. Inspired by NLP, we embed them with the word
embedding function. The initial features of atom and bond are represented as
$F^{0}_{atom}$ and $F^{0}_{bond}$ respectively.

###  3.3 3D-Mol Layer

Inspired by GEM[23], we use message passing strategy to let node send and
receive messages with edge in $\\{G^{i}_{a-b},G^{i}_{b-a},G^{i}_{p-a}\\}$. For
the $i_{th}$ layer in 3D-Mol, the information of
$\\{G^{i}_{a-b},G^{i}_{b-a},G^{i}_{p-a}\\}$ will be updated by message passing
neural network orderly. The message passing of the latter graph need the
information of the former graph. The overview is shown in figure 3, and the
details are as follow:
First, we use $GNN^{i}_{a-b}$ to aggregate the message and update the atom and
bond latent vectors in $G^{i}_{a-b}$. Given an atom v, its representation
vector $F^{i}_{v}$ is formalized by:

$a^{i,a-b}_{v}=Agg^{(i)}_{a-b}({F^{i-1}_{v},F^{i-1}_{u},F^{i-1}_{uv}|u\in
N(v)})$ (4) $F^{i}_{v}=Comb^{(k)}_{a-b,n}(F^{i-1}_{v},a^{i}_{v})$ (5)
$F^{i,temp}_{uv}=Comb^{(k)}_{a-b,e}(F^{i-1}_{uv},F^{i-1}_{u},F^{i-1}_{v})$ (6)

where $N(v)$ is the set of neighbors of atom v in $G^{i}_{a-b}$, and
$Agg^{(i)}_{a-b}$ is the aggregation function for aggregating messages from an
atom’s neighborhood in $G^{i}_{a-b}$. $Comb^{(k)}_{a-b,n}$ is the update
function for updating the atom latent vectors in $G^{i}_{a-b}$, and
$Comb^{(k)}_{a-b,e}$ is the update function for updating the bond latent
vectors in $G^{i}_{a-b}$. $a^{i,a-b}_{v}$ is the information from the
neighboring atom and the corresponding bond after be aggregated in
$G^{i}_{a-b}$. $F^{i,temp}_{uv}$ is the temporary bond latent vectors of bond
$uv$ in $i_{th}$ layer. Processing $G^{i}_{a-b}$ with $GNN^{i}_{a-b}$ make
features of atom and bond can be updated by information from their neighbor,
and the model will learn the topology information of the molecule.
$F^{i,temp}_{uv}$ is part of bond feature in $G^{i}_{b-a}$.

Figure 3: Overview of the 3D-Mol encoder layer. The 3D-Mol encoder layer
comprises three steps. Firstly, employing a message passing strategy, nodes in
each graph exchange messages with their connected edges, leading to the
updating of edge and node latent vectors. Secondly, the edge latent vector
from the lower-level graph is transmitted to the higher-level graph as part of
the node latent vector. Finally, the iteration is performed n times to derive
the $n_{th}$ node latent vector, from which we extract the molecular latent
vectors.

Then, we use $GNN^{i}_{b-a}$ to aggregate the message and update the bond and
plane vector in $G^{i}_{b-a}$. Given a bond $uv$, its latent vector
$F^{i}_{uv}$ is formalized by:

$a^{i,b-a}_{uv}=Agg^{(i)}_{b-a}(\\{F^{i-1}_{uv},F^{i-1}_{vw},F^{i-1}_{uvw}|u\in
N(v)\cap\in N(v)\cap u\neq w\\})$ (7)
$F^{i}_{uv}=Comb^{(k)}_{b-a,n}(F^{i-1}_{uv},F^{i,temp}_{uv},a^{i}_{uv})$ (8)
$F^{i-1,temp}_{uvw}=Comb^{(k)}_{b-a,e}(F^{i-1}_{uvw},F^{i-1}_{uv},F^{i-1}_{vw})$
(9)

where $Agg^{(i)}_{b-a}$ is the aggregation function for aggregating messages
from a bond’s neighborhood in $G^{i}_{b-a}$, and $Comb^{(k)}_{b-a,n}$ is the
update function for updating the bond latent vector in $G^{i}_{b-a}$, and
$Comb^{(k)}_{b-a,e}$ is the update function for updating the plane latent
vector in $G^{i}_{b-a}$. $a^{i,b-a}_{uv}$ is the information from the
neighboring bond and the corresponding bond angle after being aggregated.
Processing $G^{i}_{b-a}$ with $GNN^{i}_{b-a}$ make feature of bond and bond
angle can be update based on information from their neighbor, and the model
will learn the 3D information of the molecular. $F^{i-1,temp}_{uvw}$ is part
of plane feature in $G^{i}_{p-a}$.
After process the $G^{i}_{b-a}$, we use $GNN^{i}_{p-a}$ to aggregate the
message and update the plane latent vector in $G^{i}_{p-a}$. Given a plane
which is constructed by node u, v, w and bond $uv$, $vw$, its latent vector
$F^{i}_{uvw}$ is formalized by:

$a^{i,p-a}_{uvw}=Agg^{(i)}_{p-a}(\\{F^{i-1}_{uvw},F^{i-1}_{vwh},F^{i-1}_{uvwh}|u\in
N(v)\cap v\in N(w)\cap w\in N(h)\cap u\neq v\neq w\neq h\\})$ (10)
$F^{i}_{uvw}=Comb^{(k)}_{p-a,n}(F^{i-1}_{uv},F^{i,temp}_{uv},a^{i}_{uv})$ (11)
$F^{i-1}_{uvw}=Comb^{(k)}_{b-a,e}(F^{i-1}_{uvw},F^{i-1}_{uv},F^{i-1}_{vw})$
(12)

where $agg^{(i)}_{p-a}$ is the aggregation function for aggregating messages
from a plane’s neighborhood in $G^{i}_{p-a}$, and $Comb^{(k)}_{p-a,n}$ is the
update function for updating the plane latent vector in $G^{i}_{p-a}$, and
$Comb^{(k)}_{p-a,e}$ is the update function for updating the dihedral angle
latent vector in $G^{i}_{p-a}$. $a^{i}_{uv}$ is the information from the
neighboring plane and the corresponding dihedral angle after be aggregated.
Processing $G^{i}_{p-a}$ with $GNN^{i}_{p-a}$ makes feature of plane and
dihedral angle can be updated by information from their neighbor, and the
model will learn the 3D information of the molecular, and model the bond
interaction.
The representation vectors of the atoms at the final iteration are integrated
to gain the molecular representation vector $F_{mol}$ by the Readout function,
which is formalized as:

$F_{mol}=Readout(F^{n}_{u}|u\in N(v))$ (13)

where $F^{n}$ is the last 3D-Mol layer output. The molecular latent vector
$F_{mol}$ is used to predict molecular properties. We extract the $F_{mol}$
from all atom latent vector in the last layer.

###  3.4 Pretrain Strategy

To improve the performance of 3D-Mol encoder, we employ the constractive
learning as pretraining method, using different conformations of the same
topological structure as positive pair. We also combine our pretraining method
with geometry task in [26] to pretrain 3D-Mol with a large amount of unlabeled
data. The overview of our pretraining method is shown in figure 3, and the
following are the details of our pretraining method.

####  3.4.1 Weighted contrastive learning task

Our objective is to facilitate the learning of the consistency and difference
between the most stable molecular conformation, denoted as $Mconf_{i}$, and
another randomly selected conformation, denoted as $Mconf_{j}$. To accomplish
this, we employ weighted contrastive learning using a batch of molecular
representations, with the loss function defined as follows:

$L_{i,j}^{conf}=-log\frac{exp(w^{conf}_{i,j}sim(F_{i},F^{mk}_{j})/\tau)}{\Sigma_{k=1}^{2N}1{\\{k\neq
i\\}}exp(w^{fp}_{i,k}sim(F_{i},F_{k})/\tau)}$ (14)
$w^{conf}_{i,j}=1-\lambda_{conf}*ConfSim(Mconf_{i},Mconf_{j})$ (15)
$w^{fp}_{i,k}=1-\lambda_{fp}*FPSim(Mconf_{i},Mconf_{k})$ (16)

where $F_{i}$ is the latent vector extracted from $Mconf_{i}$, and
$sim(F_{i},F_{j})$ is the similarity between two latent vectors $(F_{i}$,
$F_{j})$, and penalized by a weight coefficient $w^{conf}_{i,j}$.
$w^{conf}_{i,j}$ is computed by $ConfSim(Mconf_{i},Mconf_{j})$, the difference
between $Mconf_{i}$ and $Mconf_{j}$, which can be computed by RDKit.
$\lambda_{conf}\in[0,1]$ is the hyperparameter that determines the scale of
penalty for the difference between two conformation. Except using different
conformations as the positive pair, we also use node and subgraph masking as
the molecular data augmentation strategy. We mask $Mconf_{i}$ 15$\%$ nodes and
corresponding edges, and the mask latent vector conformation is denoted as
$F^{mk}_{j}$. Following iMolCLR[42], the similarity measurement between two
latent vectors $F_{i}$, $F_{k}$ from a negative molecule pair
($Mconf_{i},Mconf_{j}$) is penalized by a weight coefficient $w^{fp}_{i,k}$,
which computed by the Fingerprint similarity between $Mconf_{i}$ and
$Mconf_{k}$. $FPSim(Mconf_{i},Mconf_{j})$ evaluates the fingerprint similarity
of the given two molecules $Mconf_{i},Mconf_{j}$, and $\lambda_{fp}\in[0,1]$
is the hyperparameter that determines the scale of penalty for faulty
negatives.

Table 1: Statistics information of datasets Dataset |  $\\#$ Tasks | Task Type |  $\\#$ Molecules
---|---|---|---
BACE | 1 | Classifcation | 1513
Sider | 27 | Classifcation | 1427
Tox21 | 12 | Classifcation | 7831
ToxCast | 617 | Classifcation | 8597
ESOL | 1 | Regression | 1128
FreeSolv | 1 | Regression | 643
Lipophilicity | 1 | Regression | 4200

####  3.4.2 Geometry task

3D information have been shown to be important features[22], so we employ
geometry task as pretraining method. For bond angle and dihedral angle
prediction, we sample adjacent atoms to better capture local structural
information. Since angular values are more sensitive to errors in protein
structures than distances, we use discretized values for prediction.

$L_{i,j}^{dist}=(f_{dist}(Fn^{mk}_{n,i},Fn^{mk}_{n,j})-dist_{i,j})^{2}$ (17)
$L_{i,j}^{l}=(f_{l}(Fn^{mk}_{n,i},Fn^{mk}_{n,j})-l_{i,j})^{2}$ (18)
$L_{i,j,k}^{\theta}=CE(f_{\theta}(Fn^{mk}_{n,i},Fn^{mk}_{n,j},Fn^{mk}_{n,k}),bin({\theta}_{i,j,k}))$
(19)
$L_{i,j,k,p}^{\phi}=CE(f_{\phi}(Fn^{mk}_{n,i},Fn^{mk}_{n,j},Fn^{mk}_{n,k},Fn^{mk}_{n,p}),bin({\phi}_{i,j,k,p}))$
(20)

where $f_{\phi}(.)$, $f_{\theta}(.)$, $f_{dist}(.)$ and $f_{l}$ are the MLPs
for each task, and $L_{i,j}^{dist}$, $L_{i,j}^{l}$, $L_{i,j,k}^{\theta}$,
$L_{i,j,k,p}^{\phi}$ and $L_{i}^{FP}$ are the loss functions for each task.
$CE(.)$ is cross entropy loss, and $bin(·)$ is used to discretize the bond
angle and dihedral angle. $Fn^{mk}_{n,i}$ is the latent vector of node i after
masking the corresponding sampled items in each task.
In addition to the aforementioned pretraining tasks to capture global
molecular information, we leverage masked molecular latent vectors for
fingerprint (FP) prediction, effectively incorporating latent representations
to enrich the predictive capability.

$L_{i}^{FP}=BCE(f_{FP}(F^{mk}),FP_{i})$ (21)

where $f_{FP}$ is the MLPs for global geometric task, and $L_{i}^{FP}$ is the
loss function. $BCE(.)$ is binary cross entropy loss. $F^{mk}$ is the latent
vector of the masking molecule.

##  4 Experiment

In this section, we conduct experiments on 7 benchmark datasets in MoleculeNet
to demonstrate the effectiveness of 3D-Mol for molecular property prediction.
Firstly we use a large amount of unlabeled data and our pretraining method to
pretrain the 3D-Mol model, then we use the downstream task to finetune well-
pretrained model and predict the molecular property. We compared it with a
variety of state-of-the-art methods. Also we conduct several ablation studies
to confirm the 3D-Mol model and our pretraining method is useful.

###  4.1 Datasets and Setup

####  4.1.1 Pretraining stage

We use 20 million unlabeled molecules to pretrain 3D-Mol. The unlabeled data
is extracted from ZINC20 and PubChem, both of which are publicly accessible
databases containing drug-like compounds. To ensure consistency with prior
research[23], we randomly selected 90$\%$ of these molecules for training
purposes, while the remaining $\%$ was set aside for evaluation.The raw data
obtained from ZINC20 and PubChem was provided in the SMILES format. In order
to convert the SMILES representations into molecular conformations, we
employed RDKit and applied the ETKDG method. For our model, we use Adam
optimizer with a learning rate of 1e-3. The batch size is set to 256 for
pretraining and 32 for finetuning. The hidden size of all models is
unspecified. The geometric embedding dimension K is 64, and the number of
angle domains is 8. The hyperparameters $\lambda_{conf}$ and $\lambda_{fp}$
are both set to 0.5.

Table 2: Comparison of performance on the 7 molecular property prediction tasks and the methods below are pretraining method. We mark the best results in bold and underline the second best. model | Classification (ROC-AUC $\%$, higher is better ↑ ) | Regression (RMSE, lower is better ↓)
---|---|---
| BACE | SIDER | Tox21 | ToxCast | ESOL | FreeSolv | Lipophilicity
$\rm N$-$\rm Gram_{RF}$ | $\rm 0.779_{0.015}$ | $\rm 0.668_{0.007}$ | $\rm 0.743_{0.004}$ | $-$ | $\rm 1.074_{0.107}$ | $\rm 2.688_{0.085}$ | $\rm 0.812_{0.028}$
$\rm N$-$\rm Gram_{XGB}$ | $\rm 0.791_{0.013}$ | $\rm 0.655_{0.007}$ | $\rm 0.758_{0.009}$ | $-$ | $\rm 1.083_{0.107}$ | $\rm 5.061_{0.744}$ | $\rm 2.072_{0.030}$
$\rm PretrainGNN$ | $\rm 0.845_{0.007}$ | $\rm 0.627_{0.008}$ | $\rm 0.781_{0.006}$ | $\rm 0.657_{0.006}$ | $\rm 1.100_{0.006}$ | $\rm 2.764_{0.002}$ | $\rm 0.739_{0.003}$
$\rm GROVER_{base}$ | $\rm 0.826_{0.007}$ | $\rm 0.648_{0.006}$ | $\rm 0.743_{0.001}$ | $\rm 0.654_{0.004}$ | $\rm 0.983_{0.090}$ | $\rm 2.176_{0.052}$ | $\rm 0.817_{0.008}$
$\rm GROVER_{large}$ | $\rm 0.810_{0.014}$ | $\rm 0.654_{0.001}$ | $\rm 0.735_{0.001}$ | $\rm 0.653_{0.005}$ | $\rm 0.895_{0.017}$ | $\rm 2.272_{0.051}$ | $\rm 0.823_{0.010}$
$\rm MolCLR$ | $\rm 0.824_{0.009}$ | $\rm 0.589_{0.014}$ | $\rm 0.750_{0.002}$ | $-$ | $\rm 1.271_{0.040}$ | $\rm 2.594_{0.249}$ | $\rm 0.691_{0.004}$
$\rm 3DInfomax$ | $\rm 0.797_{0.015}$ | $\rm 0.606_{0.008}$ | $\rm 0.644_{0.011}$ | $\rm 0.745_{0.007}$ | $\rm 0.894_{0.028}$ | $\rm 2.337_{0.107}$ | $\rm 0.695_{0.012}$
$\rm GraphMVP$ | $\rm 0.812_{0.009}$ | $\rm 0.639_{0.012}$ | $\rm 0.759_{0.005}$ | $\rm 0.631_{0.004}$ | $\rm 1.029_{0.033}$ | $-$ | $\rm 0.681_{0.010}$
$\rm GEM$ | $\rm 0.856_{0.011}$ | $\rm\textbf{0.672}_{0.004}$ | $\rm 0.781_{0.005}$ | $\rm 0.692_{0.004}$ | $\rm 0.798_{0.029}$ | $\rm 1.877_{0.094}$ | $\rm 0.660_{0.008}$
$\rm Uni$-$\rm Mol$ | $\rm\underline{0.857}_{0.005}$ | $\rm\underline{0.659}_{0.013}$ | $\rm\textbf{0.796}_{0.006}$ | $\rm\underline{0.696}_{0.001}$ | $\rm\underline{0.788}_{0.029}$ | $\rm\underline{1.620}_{0.035}$ | $\rm\underline{0.603}_{0.010}$
$\rm 3D$-$\rm Mol_{wcl}$ | $\rm\textbf{0.875}_{0.004}$ | $\rm 0.656_{0.002}$ | $\rm\underline{0.786}_{0.003}$ | $\rm\textbf{0.697}_{0.003}$ | $\rm\textbf{0.783}_{0.009}$ | $\rm\textbf{1.617}_{0.050}$ | $\rm\textbf{0.598}_{0.018}$

####  4.1.2 Finetuning stage

We use 7 molecular datasets obtained from MoleculeNet to demonstrate the
effectiveness of 3D-Mol. These datasets encompass a range of biophysics
datasets such as BACE, physical chemistry datasets like ESOL, and physiology
datasets like Tox21. In the fine-tuning stage, we employed nine molecular
datasets obtained from MoleculeNet. These datasets encompass a range of
biophysics datasets such as BACE, physical chemistry datasets like ESOL, and
physiology datasets like Tox21. Table 1 provides a summary of the statistical
information for these datasets, while the remaining details are outlined
below:
• BACE. The BACE dataset provides both quantitative (IC50) and qualitative
(binary label) binding results for a set of inhibitors targeting human
$\beta$-secretase 1 (BACE-1).
• Tox21. The Tox21 initiative aims to advance toxicology practices in the 21st
century and has created a public database containing qualitative toxicity
measurements for 12 biological targets, including nuclear receptors and stress
response pathways.
• Toxcast. ToxCast, an initiative related to Tox21, offers a comprehensive
collection of toxicology data obtained through in vitro high-throughput
screening. It includes information from over 600 experiments and covers a
large library of compounds.
• SIDER. The SIDER database is a compilation of marketed drugs and their
associated adverse drug reactions (ADRs), categorized into 27 system organ
classes.
• ESOL. The ESOL dataset is a smaller collection of water solubility data,
specifically providing information on the log solubility in mols per liter for
common organic small molecules.
• FreeSolv. The FreeSolv database offers experimental and calculated
hydration-free energy values for small molecules dissolved in water.
• Lipo. Lipophilicity is a crucial characteristic of drug molecules that
affects their membrane permeability and solubility. The Lipo dataset contains
experimental data on the octanol/water distribution coefficient (logD at pH
7.4).
Following the previous works[23], We partitioned the datasets into
train/validation/test sets in an 80/10/10 ratio for downstream tasks, and We
use scaffold splitting and report the mean and standard deviation by the
results of 3 random seeds.

###  4.2 Metric

Consistent with prior studies, we adopt the average ROC-AUC as the evaluation
metric for the classification datasets (BACE, SIDER, Tox21 and ToxCast), which
is a widely used metric for assessing the performance of binary classification
tasks. For the regression datasets (ESOL, FreeSolv and Lipophilicity), we
utilize the RMSE as the evaluation metric.

###  4.3 Result

a) To validate the efficacy of our proposed method, we compare it with several
baseline methods. The baseline methods are as follows: N-Gram[12] generates a
graph representation by constructing node embeddings based on short walks.
PretrainGNN[11] implements several types of self-supervised learning tasks. 3D
Infomax[48] maximizes the mutual information between learned 3D summary
vectors and the representations of a graph neural network. MolCLR[13] is a
2D-2D view contrastive learning model that involves atom masking, bond
deletion, and subgraph removal. GraphMVP[46] introduces 2D-3D view contrastive
learning approaches. GROVER[14] focused on node level and graph level
representation and corresponding pretraining tasks for node level and graph
level. GEM[23] employs predictive geometry self-supervised learning schemes
that leverage 3D molecular information. Uni-Mol[47] enlarge the application
scope and representation ability of molecular representation learning by using
transformer. As result in Table 2, Our method gets the best result in 5
datasets and gets the second best result in 1 dataset. Furthermore, our method
achieves overwhelming performance on BACE by a large margin. That shows that
our method is better at extracting the molecular information. To do the
ablation study, We compare the results of 3D-Mol and 3D-Mol without
pretraining. It shows that the former achieve overwhelming performance and our
pretraining method can improve the 3D-Mol model performance.

Table 3: Comparison of performance on the 9 molecular property prediction tasks, and the methods below are no pretraining. We mark the best results in bold and underline the second best. model | Classification (ROC-AUC $\%$, higher is better ↑ ) | Regression (RMSE, lower is better ↓)
---|---|---
| BACE | SIDER | Tox21 | ToxCast | ESOL | FreeSolv | Lipophilicity
$\rm DMPNN$ | $\rm 0.809_{0.006}$ | $\rm 0.570_{0.007}$ | $\rm 0.759_{0.007}$ | $\rm 0.655_{0.3}$ | $\rm 1.050_{0.008}$ | $\rm 2.082_{0.082}$ | $\rm 0.683_{0.016}$
$\rm AttentiveFP$ | $\rm 0.784_{0.000}$ | $\rm 0.606_{0.032}$ | $\rm 0.761_{0.005}$ | $\rm 0.637_{0.002}$ | $\rm 0.877_{0.029}$ | $\rm 2.073_{0.183}$ | $\rm 0.721_{0.001}$
$\rm MGCN$ | $\rm 0.734_{0.008}$ | $\rm 0.587_{0.019}$ | $\rm 0.741_{0.006}$ | $-$ | $-$ | $-$ | $-$
$\rm SGCN$ | $-$ | $\rm 0.559_{0.005}$ | $\rm 0.766_{0.002}$ | $\rm 0.657_{0.003}$ | $\rm 1.629_{0.001}$ | $\rm 2.363_{0.050}$ | $\rm 1.021_{0.013}$
$\rm HMGNN$ | $-$ | $\rm\underline{0.615}_{0.005}$ | $\rm 0.768_{0.002}$ | $\rm 0.672_{0.001}$ | $\rm 1.390_{0.073}$ | $\rm 2.123_{0.179}$ | $\rm 2.116_{0.473}$
$\rm DimeNet$ | $-$ | $\rm 0.612_{0.004}$ | $\rm\underline{0.774}_{0.006}$ | $\rm 0.637_{0.004}$ | $\rm 0.878_{0.023}$ | $\rm 2.094_{0.118}$ | $\rm 0.727_{0.019}$
$\rm GEM$ | $\underline{0.828}_{0.012}$ | $\rm 0.606_{0.010}$ | $\rm{0.773}_{0.007}$ | $\rm\underline{0.675}_{0.005}$ | $\rm\underline{0.832}_{0.010}$ | $\rm\underline{1.857}_{0.071}$ | $\rm\underline{0.666}_{0.015}$
$\rm 3D$-$\rm Mol_{w.o\ pre}$ | $\rm\textbf{0.832}_{0.005}$ | $\rm\textbf{0.624}_{0.013}$ | $\rm\textbf{0.780}_{0.004}$ | $\rm\textbf{0.682}_{0.007}$ | $\rm\textbf{0.794}_{0.027}$ | $\rm\textbf{1.769}_{0.039}$ | $\rm\textbf{0.674}_{0.007}$
Table 4: Ablation study. We compare the performance of no pretraining 3DGNN, pretraining by geometrical tasks and pretraining by geometrical tasks and weighted contrastive loss, and mark the best results in bold and underline the second best. model | Classification (ROC-AUC $\%$, higher is better ↑ ) | Regression (RMSE, lower is better ↓)
---|---|---
| BACE | SIDER | Tox21 | ToxCast | ESOL | FreeSolv | Lipophilicity
$\rm 3DGNN$ | $\rm\textbf{0.875}_{0.004}$ | $\rm\underline{0.656}_{0.002}$ | $\rm\underline{0.786}_{0.003}$ | $\rm\textbf{0.697}_{0.003}$ | $\rm\textbf{0.783}_{0.009}$ | $\rm\underline{1.617}_{0.050}$ | $\rm\textbf{0.598}_{0.018}$
$\rm 3DGNN_{wo.pre}$ | $\rm 0.832_{0.005}$ | $\rm 0.624_{0.013}$ | $\rm 0.780_{0.004}$ | $\rm 0.682_{0.007}$ | $\rm\underline{0.794}_{0.027}$ | $\rm 1.769_{0.039}$ | $\rm 0.674_{0.007}$
$\rm 3DGNN_{wo.cl_{weighted}}$ | $\rm\underline{0.874}_{0.006}$ | $\rm\textbf{0.661}_{0.005}$ | $\rm\textbf{0.790}_{0.003}$ | $\rm\underline{0.693}_{0.005}$ | $\rm 0.795_{0.014}$ | $\rm\textbf{1.557}_{0.003}$ | $\rm\underline{0.607}_{0.006}$

b) To validate the efficacy of our proposed model 3D-Mol encoder, we compare
it with several baseline molecular encoder without pretraining. The baseline
molecular encoders are as follows: DMPNN[17] employed a message passing scheme
for molecular property prediction. AttentiveFP[16] is an attention-based GNN
that incorporates graph-level information. MGCN[34] designed hierarchical
graph neural network to directly extract features from the conformation and
spatial information followed by the multilevel interactions. HMGNN[24]
leverages global molecular representations through an attention mechanism.
SGCN[25] applies different weights according to atomic distances during the
GCN-based message passing process. DimeNet[22] proposes directional message
passing to fully utilize directional information within molecules. GEM[23]
employs message passing strategy to extract 3D molecular information. We
present the experimental result to show the efficiency of our 3D-Mol model, as
can be seen in Table 3. From Table 4, 3D-Mol encoder significantly outperforms
all the baselines on both types of tasks and improves the performance over the
best baselines with 2$\%$ and 11$\%$ for classification and regression tasks
respectively, since 3D-Mol incorporates geometrical parameters.
c) To validate the efficacy of our proposed pretraining task, we compare the
performance of no pretraining 3DGNN, pretraining by geometrical tasks and
pretraining by geometrical tasks and weighted contrastive loss, and the result
shows in Table 4. The result shows that geometry tasks signifcantly improve
the performance of 3DGNN. Compared with the pretraining method combined with
weighted contrastive learning. In general, The combined pretraining method is
better to improve the 3DGNN performance.

##  5 Conclusion

In this paper, we innovatively propose a novel 3D molecular model framework,
3D-Mol, to extract 3D molecular features. Furthermore, to effectively utilize
a large number of unlabeled molecules and molecular conformations for feature
extraction, we have designed a new self-supervised pretraining strategy. Our
approach has been validated through numerous experiments and compared with
multiple competitive benchmarks, demonstrating superior performance over other
methods across various benchmarks.

## Acknowledgment

The research was supported by the PengCheng Laboratory and by PengCheng
Laboratory Cloud-Brain.

## References

  * [1] W. Patrick Walters and Regina Barzilay.  Applications of Deep Learning in Molecule Generation and Molecular Property Prediction.  Accounts of Chemical Research, 54(2):263–270, January 2021.
  * [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  2018\.
  * [3] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [5] Garrett B. Goh, Nathan O. Hodas, Charles Siegel, and Abhinav Vishnu.  SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties.  2017\.
  * [6] Kexin Huang, Tianfan Fu, Lucas M Glass, Marinka Zitnik, Cao Xiao, and Jimeng Sun.  Deeppurpose: a deep learning library for drug–target interaction prediction.  Bioinformatics, 36(22-23):5545–5547, 2020.
  * [7] Stanisław Jastrzebski, Damian Leśniak, and Wojciech Marian Czarnecki.  Learning to SMILE(S).  2016\.
  * [8] Zheng Xu, Sheng Wang, Feiyun Zhu, and Junzhou Huang.  Seq2seq Fingerprint: An Unsupervised Deep Molecular Embedding for Drug Discovery.  In Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology,and Health Informatics, pages 285–294, Boston Massachusetts USA, August 2017. ACM.
  * [9] Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar.  ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction.  2020\.
  * [10] David Weininger.  Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.  Journal of chemical information and computer sciences, 28(1):31–36, 1988.
  * [11] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec.  Strategies for Pre-training Graph Neural Networks.  2019\.
  * [12] Shengchao Liu, Mehmet F Demirel, and Yingyu Liang.  N-gram graph: Simple unsupervised representation for graphs, with applications to molecules.  Advances in neural information processing systems, 32, 2019.
  * [13] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani.  Molecular contrastive learning of representations via graph neural networks.  Nature Machine Intelligence, 4(3):279–287, March 2022.
  * [14] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying WEI, Wenbing Huang, and Junzhou Huang.  Self-supervised graph transformer on large-scale molecular data.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12559–12571. Curran Associates, Inc., 2020.
  * [15] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl.  Neural message passing for quantum chemistry.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1263–1272. PMLR, 06–11 Aug 2017\.
  * [16] Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al.  Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism.  Journal of medicinal chemistry, 63(16):8749–8760, 2019.
  * [17] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al.  Analyzing learned molecular representations for property prediction.  Journal of chemical information and modeling, 59(8):3370–3388, 2019\.
  * [18] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller.  Schnet: A continuous-filter convolutional neural network for modeling quantum interactions.  Advances in neural information processing systems, 30, 2017.
  * [19] Jintang Li, Kun Xu, Liang Chen, Zibin Zheng, and Xiao Liu.  Graphgallery: A platform for fast benchmarking and easy development of graph neural networks based intelligent software.  In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 13–16. IEEE, 2021\.
  * [20] Łukasz Maziarka, Tomasz Danel, Sławomir Mucha, Krzysztof Rataj, Jacek Tabor, and Stanisław Jastrzebski.  Molecule attention transformer.  arXiv preprint arXiv:2002.08264, 2020.
  * [21] David Rogers and Mathew Hahn.  Extended-connectivity fingerprints.  Journal of chemical information and modeling, 50(5):742–754, 2010\.
  * [22] Johannes Gasteiger, Janek Groß, and Stephan Günnemann.  Directional message passing for molecular graphs.  arXiv preprint arXiv:2003.03123, 2020.
  * [23] Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang.  Geometry-enhanced molecular representation learning for property prediction.  Nature Machine Intelligence, 4(2):127–134, 2022.
  * [24] Zeren Shui and George Karypis.  Heterogeneous molecular graph neural networks for predicting molecule properties.  In 2020 IEEE International Conference on Data Mining (ICDM), pages 492–500. IEEE, 2020.
  * [25] Tomasz Danel, Przemysław Spurek, Jacek Tabor, Marek Śmieja, Łukasz Struski, Agnieszka Słowik, and Łukasz Maziarka.  Spatial graph convolutional networks.  In Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18–22, 2020, Proceedings, Part V, pages 668–675. Springer, 2020.
  * [26] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang.  Protein representation learning by geometric structure pretraining.  arXiv preprint arXiv:2203.06125, 2022.
  * [27] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande.  Moleculenet: a benchmark for molecular machine learning.  Chemical science, 9(2):513–530, 2018.
  * [28] Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse.  Reoptimization of mdl keys for use in drug discovery.  Journal of chemical information and computer sciences, 42(6):1273–1280, 2002.
  * [29] Adrià Cereto-Massagué, María José Ojeda, Cristina Valls, Miquel Mulero, Santiago Garcia-Vallvé, and Gerard Pujadas.  Molecular fingerprint similarity search in virtual screening.  Methods, 71:58–63, 2015.
  * [30] Connor W Coley, Regina Barzilay, William H Green, Tommi S Jaakkola, and Klavs F Jensen.  Convolutional embedding of attributed molecular graphs for physical property prediction.  Journal of chemical information and modeling, 57(8):1757–1772, 2017\.
  * [31] Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang.  Smiles-bert: large scale unsupervised pre-training for molecular property prediction.  In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pages 429–436, 2019.
  * [32] Jike Wang, Dongsheng Cao, Cunchen Tang, Lei Xu, Qiaojun He, Bo Yang, Xi Chen, Huiyong Sun, and Tingjun Hou.  Deepatomiccharge: a new graph convolutional network-based architecture for accurate prediction of atomic charges.  Briefings in bioinformatics, 22(3):bbaa183, 2021.
  * [33] Xiao-Shuang Li, Xiang Liu, Le Lu, Xian-Sheng Hua, Ying Chi, and Kelin Xia.  Multiphysical graph neural network (mp-gnn) for covid-19 drug design.  Briefings in Bioinformatics, 23(4), 2022.
  * [34] Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He.  Molecular property prediction: A multilevel quantum interactions modeling perspective.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1052–1060, 2019.
  * [35] Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R Manby, and Thomas F Miller.  Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features.  The Journal of chemical physics, 153(12), 2020.
  * [36] Luciano Floridi and Massimo Chiriatti.  Gpt-3: Its nature, scope, limits, and consequences.  Minds and Machines, 30:681–694, 2020.
  * [37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [38] Shion Honda, Shoi Shi, and Hiroki R Ueda.  Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery.  arXiv preprint arXiv:1911.04738, 2019.
  * [39] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen.  Graph contrastive learning with augmentations.  Advances in neural information processing systems, 33:5812–5823, 2020.
  * [40] Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou.  Mocl: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3585–3594, 2021.
  * [41] Pengyong Li, Jun Wang, Yixuan Qiao, Hao Chen, Yihuan Yu, Xiaojun Yao, Peng Gao, Guotong Xie, and Sen Song.  An effective self-supervised framework for learning expressive molecular global representations to drug discovery.  Briefings in Bioinformatics, 22(6):bbab109, 2021.
  * [42] Yuyang Wang, Rishikesh Magar, Chen Liang, and Amir Barati Farimani.  Improving molecular contrastive learning via faulty negative mitigation and decomposed fragment contrast.  Journal of Chemical Information and Modeling, 62(11):2713–2725, 2022.
  * [43] Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Yuanxing Ning, Philip S Yu, and Lifang He.  Sugar: Subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism.  In Proceedings of the Web Conference 2021, pages 2081–2091, 2021\.
  * [44] Zewei Ji, Runhan Shi, Jiarui Lu, Fang Li, and Yang Yang.  Relmole: Molecular representation learning based on two-level graph similarities.  Journal of Chemical Information and Modeling, 62(22):5361–5372, 2022.
  * [45] Hyeoncheol Cho and Insung S Choi.  Enhanced deep-learning prediction of molecular properties via augmentation of bond topology.  ChemMedChem, 14(17):1604–1609, 2019.
  * [46] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang.  Pre-training molecular graph representation with 3d geometry.  arXiv preprint arXiv:2110.07728, 2021.
  * [47] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke.  Uni-mol: a universal 3d molecular representation learning framework.  2023\.
  * [48] Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò.  3d infomax improves gnns for molecular property prediction.  In International Conference on Machine Learning, pages 20479–20502. PMLR, 2022.