#  Evaluating Self-Supervised Learning for
Molecular Graph Embeddings

Hanchen Wang1* Jean Kaddour2* Shengchao Liu3,4 Jian Tang3,5,6
Matt Kusner2 Joan Lasenby1 Qi Liu7 * Equal Contribution
1Cambridge 2UCL 3MILA 4UdeM 5HEC 6CIFAR 7HKU

###### Abstract

Graph Self-Supervised Learning (GSSL) paves the way for learning graph
embeddings without expert annotation, which is particularly impactful for
molecular graphs since the number of possible molecules is enormous and labels
are expensive to obtain. However, by design, GSSL methods are not trained to
perform well on one particular downstream task but aim for transferability to
many, making evaluating them less straightforward. As a step toward obtaining
profiles of molecular graph embeddings with diverse and interpretable
attributes, we introduce _Molecular Graph Representation Evaluation_
(MolGraphEval), a suite of probe tasks, categorised into (i) generic graph,
(ii) molecular substructure, and (iii) embedding space properties. By
benchmarking existing GSSL methods on both existing downstream datasets and
MolGraphEval, we discover surprising discrepancies between conclusions drawn
from existing datasets alone versus more fine-grained probing, suggesting that
current evaluation protocols do not provide the whole picture. Our modular,
automated end-to-end GSSL pipeline code will be released upon acceptance,
including standardised graph loading, experiment management, and embedding
evaluation.

##  1 Introduction

Figure 1: Overview of _MolGraphEval_. Given molecular graphs, we train GNNs to
predict SSL proxy objectives. We then extract embeddings of (possibly unseen)
graphs using pre-trained models, which form the inputs for probe models,
trained and evaluated on the designed metrics.

Learning neural embeddings of molecular graphs has become of paramount
importance in the field of computer-aided drug discovery [38]. For example, a
molecular property prediction model can help speed up the process and reduce
the cost of designing molecules by reducing the number of molecules that must
be synthesised and measured. Thereby, such models can be immensely useful in
the hit-to-lead and early lead optimisation phase of a drug discovery project
[47]. However, obtaining labels of molecule properties is expensive and time-
consuming, especially since the size of potential pharmocologically active
molecules is estimated to be in the order of $10^{60}$ [3, 56].

Graph Self-Supervised Learning (GSSL) paves the way for learning molecular
graph embeddings without human annotations by learning embeddings that
transfer to a variety of downstream datasets. Unfortunately, the evaluation of
such general-purpose embeddings is fundamentally difficult. Different proxy
objectives will place different demands on them, and no single downstream
dataset can be definitive. Moreover, many of the previously proposed GSSL
works are disconnected in terms of the tasks they target as well as the
datasets they use for evaluation, making direct comparison difficult.

Contributions. Our goal is to evaluate molecular graph embeddings obtained by
popular GSSL methods both on existing downstream tasks as well as a new suite
of probe tasks. We first conduct a rigorous and systematic evaluation of
existing GSSL methods on molecular property prediction (MPP) tasks, using
controlled and consistent training and testing protocols across all methods.
This empirical study in Sec. 4 allows us later to contrast the findings
yielded by typical benchmark datasets with analyses based on probe tasks. In
Sec. 5, we then propose _Molecular Graph Representation Evaluation_
(MolGraphEval, in Fig. 1), a suite of probe tasks to evaluate the produced
molecular graph embeddings on various properties. We categorise the tasks into
(i) generic graph, (ii) molecular substructure, and (iii) embedding space
properties. Finally, in Sec. 6, we evaluate the GSSL methods on MolGraphEval
and discover discrepancies between their performance on MPP and our probe
tasks. Hence, we believe that this benchmark is well-positioned to explore
more nuanced insights into GSSL-produced embeddings, and to serve as a testbed
for novel methodology.

We summarise key empirical findings, which we hope will benefit the GSSL
community.

  1. 1)

On MPP tasks, we observe a significant difference in the ranking of GSSL
methods depending on whether we fine-tune the pre-trained network on the
downstream dataset or not, see Finding 1.

  2. 2)

GSSL generates hierarchical features in a layer-localisable way, see Finding
3.

  3. 3)

There are a number of discrepancies between MPP tasks and MolGraphEval that
demonstrate how the latter complements GSSL evaluation with non-obvious
insights fostering future work:

     * •

While embeddings from a randomly initialised GNN (Random) perform poorly on
MPP tasks, they sometimes outperform GSSL methods on topological properties
(which can be relevant in molecular settings), indicating that GSSL methods
are not universally better, see Finding 5.

     * •

While contrastive GSSL methods perform better than most other methods on
molecular substructure detection, they are not as superior on MPP tasks, see
Finding 10.

     * •

In contrast to previous work’s claims [55], we find that feature distribution
uniformity does not always “strongly agree” with MPP downstream performance.
For instance, maximising the mutual information between multi-scale graph
representations (InfoGraph [48]) results in the most uniform feature
distributions, yet, it ranks 6th among the 10 GSSL methods on the downstream
MPP tasks, see Finding 14.

##  2 Related work

Graph SSL (GSSL) can be divided into contrastive and generative methods [35,
62, 60, 34]. Contrastive GSSL [48, 22, 66] construct multiple views of the
same graph via augmentations and then learn embeddings by contrasting positive
samples against negative ones. Generative GSSL [17, 32, 22, 24] yields
embeddings by reconstructing input graphs. In Appendix A, we explain the
methods considered in this work. Zhu et al. [68] conduct an empirical analysis
of contrastive GSSL methods and their components. In contrast, we investigate
both generative and contrastive GSSL methods, and propose a novel suite of
tasks to probe the produced embeddings’ attributes.

Probing embeddings. In non-graph domains, a popular way to evaluate attributes
of learned embeddings is to use _probe models_ [1]. These models are trained
to perform _probe tasks_ by taking only the extracted embedding vectors as
input (_e.g._ , from a pre-trained model). If the probe models achieve good
performances that way, we interpret the embeddings being of high quality [9].
While we are not aware of previous work using probe tasks in the domain of
GSSL, there are plenty of works providing support for probe models in natural
language processing [9, 31, 21, 49, 26, 52, 28, 20], computer vision [1, 5,
19, 8, 30, 7, 41, 53], and biomedical science [40, 42, 50, 11, 13].

##  3 Preliminaries

Graph. A graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ consists of a set of
nodes $\mathcal{V}$ and edges $\mathcal{E}$. In molecular graphs, nodes are
atoms and edges are bonds. We use ${\bm{x}}_{u}$ and ${\bm{x}}_{uv}$ to denote
the feature of node $u$ and of the bond feature between nodes $[u,v]$,
respectively. For notation simplicity, we use an adjacency matrix
${\mathbf{A}}\in\mathbb{R}^{|\mathcal{V}|\times|\mathcal{V}|}$ to represent
the graph, where ${\mathbf{A}}[u,v]\neq 0$ if the nodes $(u,v)$ are connected.

GNN. Graph neural networks (GNNs) give rise to learn molecular graphs
embeddings [12, 14, 32, 64, 10]. A prototypical GNN relies on messaging
passing [14], which updates atom-level embeddings based on their
neighbourhoods. Given an input atom ${\bm{h}}_{u}^{0}={\bm{x}}_{u}$, we
compute its embedding by:

${\bm{m}}_{u}^{t+1}=\sum_{v:{\mathbf{A}}[u,v]\neq
0}M_{t}({\bm{h}}_{u}^{t},{\bm{h}}_{v}^{t},{\bm{x}}_{uv}),\qquad{\bm{h}}_{u}^{t+1}=U_{t}({\bm{h}}_{u}^{t},{\bm{m}}_{u}^{t+1})$
(1)

where $M_{t}$ and $U_{t}$ are the _message_ functions and _vertex update_
functions, respectively. By repeating message passing for $T$ steps, we encode
the information of the $T$-hop neighbourhood for each atom. We use a readout
function $R$ to pool node-level embeddings for graph-level prediction:
$\hat{y}=R(\\{{\bm{h}}_{u}^{\top}\mid u\in\mathcal{V}\\}).$ Following previous
GSSL methods on molecular graphs [22, 66, 33], we adopt the Graph Isomorphism
Network (GIN) [63] as the backbone model. Further, we incorporate edge
features during message passing following Hu et al. [22].

Pre-Training. We concentrate on ten popular methods for Graph SSL: EdgePred
[17], InfoGraph [48], GPT-GNN [24], AttrMask [22], ContextPred [22],
G-{Contextual, Motif} [43], GraphCL [66], JOAO-{$\cdot$,v2} [67]. We describe
these methods in details in the Appendix A. We follow the same experimental
settings as originally reported. We use the same set of $50$k qualified
molecules to pre-train the same GIN model with the various pre-training
protocols for a fair comparison. The pre-training data is randomly sampled
from the GEOM dataset [2]. After pre-training and for each probe task, we
infer embeddings using the pre-trained weights.

Probe. We use probe models [31] to study whether self-supervised learned
embeddings encode helpful structural information about graphs. Concretely, we
extract embeddings from a pre-trained GNN, and train an MLP model to predict
the probe tasks with node and graph embeddings as inputs. Details on this
network are in Appendix B. We use scaffold splitting to split data into
80%/10%/10% for the training/validation/testing set. The training procedure
runs for 100 epochs with a fixed learning rate of $1e^{-3}$. For model
selection, we select the best model based on the validation set. To account
for statistical significance, we average all experimental results over three
independent runs.

##  4 Benchmarking GSSL methods on MPP

To contrast conclusions drawn from existing evaluation procedures with
MolGraphEval, we first conduct a rigorous empirical study of the pre-training
methods predicting molecular biochemical properties. Following previous GSSL
work [22, 66], we consider eight molecular datasets consisting of 678 binary
property prediction tasks [61, 23]. We extract the node/graph embeddings from
the last GNN layer unless explicitly stated otherwise. We devise two settings:
(i) without fine-tuning (“w/o FT”), where we train the probe models with fixed
embeddings; (ii) with fine-tuning (“w FT”), where we train both the pre-
trained GNNs and the probe models.

Table 1:  Evaluating GSSL methods on molecular property prediction tasks: For
each downstream task, we report the mean (and standard deviation) ROC-AUC of
three seeds with scaffold splitting. The best and second best results are
marked bold and bold, respectively. For fair comparison, we also train and
report the average scores w/o fine-tuning (Avg/Avg(FT)) on downstream tasks.

Dataset | BBBP $\uparrow$ | Tox21 $\uparrow$ | ToxCast $\uparrow$ | Sider $\uparrow$ | ClinTox $\uparrow$ | MUV $\uparrow$ | HIV $\uparrow$ | Bace $\uparrow$ | Avg $\uparrow$ | Avg(FT) $\uparrow$
---|---|---|---|---|---|---|---|---|---|---
# Molecules | 2,039 | 7,831 | 8,575 | 1,427 | 1,478 | 93,087 | 41,127 | 1,513 | – | –
# Tasks | 1 | 12 | 617 | 27 | 2 | 17 | 1 | 1 | – | –
Random | 51.9±0.2 | 65.6±0.4 | 53.3±0.2 | 57.6±0.1 | 52.2±0.0 | 53.7±0.4 | 67.6±0.4 | 69.1±0.2 | 58.85 | 67.21
EdgePred | 57.9±0.1 | 53.2±0.6 | 52.9±0.0 | 48.6±0.4 | 51.5±0.0 |  53.9±0.2 | 69.1±0.0 | 67.6±0.3 | 56.85 | 65.64
AttrMask | 60.4±0.8 | 66.7±0.7 | 56.5±0.6 | 60.0±0.4 | 61.4±0.8 | 46.2±0.7 | 65.4±0.6 | 80.8±0.4 | 62.18 | 70.16
GPT-GNN | 64.1±0.0 | 62.1±0.1 | 53.2±0.8 | 58.4±0.1 | 64.3±0.0 | 46.5±0.1 | 69.0±0.3 | 73.8±0.3 | 61.43 | 68.27
InfoGraph | 66.3±0.6 | 68.1±0.6 | 58.4±0.6 | 57.1±0.8 |  66.3±0.6 | 44.3±0.6 | 70.2±0.6 | 64.8±0.8 | 61.94 | 70.10
Cont.Pred | 54.6±0.2 | 65.0±0.8 | 56.5±0.1 | 59.8±0.2 | 52.1±0.2 | 50.6±0.5 | 65.3±0.1 | 72.7±0.9 | 59.58 | 70.89
G-Cont. | 61.2±0.2 |  69.7±2.3 | 58.1±0.2 |  61.7±1.9 | 70.3±0.2 | 44.1±1.6 | 71.0±0.2 |  80.9±1.9 | 64.63 | 69.21
G-Motif | 66.1±1.0 | 65.6±0.7 |  58.7±1.2 | 58.7±1.1 | 65.0±1.0 | 39.2±0.7 |  71.4±1.2 | 75.9±1.1 | 62.59 | 70.14
GraphCL | 61.6±1.1 | 67.5±0.8 | 58.5±0.9 | 57.6±0.8 | 74.1±1.1 | 44.0±0.8 | 67.7±0.9 | 73.1±0.8 | 63.00 | 70.64
JOAO | 63.8±0.7 | 67.6±0.7 | 57.1±0.7 | 57.1±0.7 | 59.2±0.7 | 42.9±0.7 | 69.4±0.7 | 70.8±0.7 | 60.99 | 69.57
JOAOv2 |  66.4±0.9 | 68.2±0.8 | 57.0±0.5 | 59.1±0.7 | 64.5±0.9 | 47.4±0.8 | 68.4±0.5 | 67.4±0.7 | 62.31 | 70.21
GSSL Worse | 10% | 20% | 10% | 30% | 20% | 90% | 20% | 30% | 10% | 10%

Findings. Table 1 entails the results, and we summarise the following
findings.

  1. 1)

Rankings differ among fixed and fine-tuned embeddings. For example, embeddings
pre-trained with ContextPred achieve the second-lowest score with fixed
scenarios but perform the best after end-to-end fine-tuning. The reason is
unclear and should be investigated by future work.

  2. 2)

Random sometimes beats GSSL methods. For example, on MUV [51], a dataset
designed for validating virtual screening techniques (used in drug discovery
to find molecules that bind to a drug target), only EdgePred performs slightly
better. Yet, EdgePred performs worse than Random in 5 out of 8 cases. There
are 2 and 3 distinct datasets in which Random performs better than 3 and 2
GSSL methods, respectively. We consider this to be a troubling observation.

##  5 Molecular graph representation evaluation

The goal of GSSL for molecular graphs is to obtain embeddings that capture
generic information about the molecule and its properties. However, _there is
no free lunch_ [59]: different training objectives optimise for different
properties and evaluating the extracted embeddings on only a handful of
downstream datasets does not provide the whole picture (as we confirm
empirically in Sec. 6).

To this end, we propose MolGraphEval, which encompasses a variety of
carefully-selected probe tasks, categorised into three classes: (i) generic
graph properties, (ii) molecular substructure properties and (iii) embedding
space properties. In the upcoming subsections, we explain the tasks in more
detail and why they are important for molecular graph embeddings.

###  5.1 Generic graph properties

Topological property statistics are often used as features in machine learning
pipelines on graphs that do not rely on neural networks [16]. Based on their
scale, they can be divided into {node-, pair-, and graph-} level statistics.
For molecular graphs, different scales serve different purposes.

Node-level statistics accompany each node with a local topological measure,
which could be used as features in node classification [16]. For molecules,
node-level information reflects the reaction centres [39]; thus it can
facilitate discovering chemical reactions [4].

  * •

Node degree ($d_{u}$) counts the number of edges incident to node $u$:
$d_{u}=\sum_{v\in V}{\mathbf{A}}[u,v]$.

  * •

Node centrality ($e_{u}$) represents a node’s importance, it is defined as a
recurrence relation that is proportional to the average centrality of its
neighbours: $e_{u}=\left(\sum_{v\in
V}{\mathbf{A}}[u,v]e_{v}\right)/\lambda,\quad\forall u\in\mathcal{V}$.

  * •

Clustering coefficient ($c_{u}$) measures how tightly clustered a node’s
neighbourhood is:
$c_{u}=\left({\left|\left(v_{1},v_{2}\right)\in\mathcal{E}:v_{1},v_{2}\in\mathcal{N}(u)\right|}\right)/{d_{u}^{2}}$,
_i.e._ , the proportion of closed triangles in neighbourhood [58].

Graph-level statistics summarise global topology information and are helpful
for graph classification tasks. For molecules, graph-level statistics can be
used, _e.g._ , to classify a molecule’s solubility [46]. We briefly describe
their intuitions; formal definitions can be found, _e.g._ , in [16].

  * •

Diameter is the maximum distance between the pair of vertices.

  * •

Cycle basis is a set of simple cycles that forms a basis of the graph cycle
space. It is a minimal set that allows every even-degree subgraph to be
expressed as a symmetric difference of basis cycles.

  * •

Connectivity is the minimum number of elements (nodes or edges) that need to
be removed to separate the remaining nodes into two or more isolated
subgraphs.

  * •

Assortativity measures the similarity of connections in the graph with respect
to the node degree. It is essentially the Pearson correlation coefficient of
degrees between pairs of linked nodes.

Pair-level statistics quantify the relationships between nodes (atoms), which
is vital in molecular modelling. For example, molecular docking techniques aim
to predict the best matching binding mode of a ligand to a macro-molecular
partner [45]. For predicting such a binding compatibility, connectivity and
distance awareness (how close a pair of atoms can be) are important.

  * •

Link prediction tests whether two nodes are connected or not, given their
embeddings and inner products. Based on the principle of homophily, it is
expected that embeddings of connected nodes are more similar compared to
disconnected pairs:
$\mathbf{S}_{\mathrm{Link}}[u,v,\mathbf{x}_{u}^{T}\mathbf{x}_{v}]=\mathbbm{1}_{\mathcal{N}(u)}(v)$.

  * •

Jaccard coefficient seeks to quantify the overlap between neighbourhoods while
minimising the biases induced by node degrees [36]: $\mathbf{S}_{\text{Jaccard
}}[u,v]={|\mathcal{N}(u)\cap\mathcal{N}(v)|}/{|\mathcal{N}(u)\cup\mathcal{N}(v)|}$

  * •

Katz index is a global overlap statistic, defined by the number of paths of
all lengths between a pair of nodes:
$\mathbf{S}_{\mathrm{Katz}}[u,v]=\sum_{i=1}^{\infty}\beta^{i}{\mathbf{A}}^{i}[u,v]$,
where $\beta\in\mathbb{R}^{+}$ is a pre-defined parameter controlling how much
weight is given to short vs long paths. A small value ($\beta<1$) down-weights
the importance of long paths. Here we set $\beta=1$, giving the paths of all
lengths equal importance.

###  5.2 Molecular substructure properties

Figure 2: Embedding space property.

(a) Alignment [55]

(b) Uniformity [55]

(c) Dimensional collapse [27]

Molecular substructures can be highly predictive of biochemical properties
[44, 54, 29, 18, 65]. For instance, molecules containing benzene rings usually
have similar physical (_e.g._ , solubility) and chemical (_e.g._ ,
aromaticity) properties [37].

How predictive are substructures? To demonstrate that substructures are very
predictive of molecular properties, we simply feed counts of substructures
within a molecule’s graph into classic ML methods (linear regression, random
forest, and XGBoost) to predict the molecule’s properties on eight downstream
datasets. Table 2 shows the results. For ease of comparison, we add the
performance of Random and GraphCL (best method), as previously discussed in
Sec. 4.

Substructures. We investigate 24 substructures which can be divided into three
groups: rings (Benzene, Beta lactams, Epoxdie, Furan, Imidazole, Morpholine,
Oxazole, Piperdine, Piperdine, Pyridine, Tetrazole, Thiazole, Thiophene);
functional groups (Amides, Amidine, Azo, Ether Guanidine, Halogens,
Hydroxylamine, Imide, Oxygens (including phenoxy), Urea); and redox Active
Sites (Allylic, excluding steroid dienone)).

Table 2: ROC-AUC scores of classifiers predicting molecular properties (like in Sec. 4). Linear Regression | Random Forest | XGBoost | Random | GraphCL
---|---|---|---|---
59.91 | 61.95 | 62.31 | 58.85 | 63.00

###  5.3 Embedding space properties

In addition to the properties of molecules, we evaluate domain-agnostic
embeddings, which have shown to be positively correlated with downstream
generalisation [55]. Using these properties as a proxy for embedding
degeneration can be helpful in data-scarce settings without labels.

Alignment quantifies how similar produced embeddings are for similar samples
[55]. Ideally, two samples with the same (or very similar) semantics should be
mapped to nearby features, and thus be mostly invariant to unneeded noise
factors. To examine alignment, we construct positive and negative molecule
pairs. Positive pairs in a dataset are those that share identical molecular
properties, whereas negative pairs are those that differ in their properties.

Uniformity measures how much information is preserved in the embedding space,
_i.e._ , how uniformly they are distributed on the unit hypersphere [55].

Dimensional collapse refers to the problem of embedding vectors spanning a
lower-dimensional subspace instead of the entire available embedding space
[27, 25]. Following Jing et al. [27], one simple way to test the occurrence of
dimension collapse is to inspect the number of non-zero singular values of a
matrix stacking the embedding vectors
${\mathbf{Z}}=\left\\{\mathbf{z}_{i}\right\\}_{i=1}^{N}$.

##  6 Results

###  6.1 Generic graph properties

Table 3: Benchmarking topological properties. We report the mean square error or the cross entropy on eight datasets. We bold the best and underline the worst performance of each metric. We denote the percentage of GSSL that do not outperform Random (“SSL Worse”). We also report the average ranks of each model on these metrics (“R”).  Metrics | Node | Pair | Graph
---|---|---|---
Degree | Cent. | Cluster | R | Link | Jaccord | Katz | R | Diameter | Conn. | Cycle | Assort. | R
Random | 0.001 | 1.199 | 0.297 | 1.7 | 31.05 | 1.879 | 2.828 | 5 | 222.6 | 0.276 | 9.351 | 0.198 | 10
EdgePred | 0.993 | 1.712 | 0.513 | 9.3 | 71.49 | 2.169 | 2.772 | 9.3 | 250.6 | 0.287 | 19.95 | 0.206 | 11
AttrMask | 0.015 | 1.307 | 0.424 | 5 | 32.23 | 2.029 | 2.634 | 4.3 | 164.7 | 0.178 | 6.075 | 0.102 | 3.8
GPT-GNN | 3.032 | 1.380 | 0.505 | 9 | 41.44 | 2.541 | 2.374 | 8.3 | 178.8 | 0.247 | 9.222 | 0.166 | 8.3
InfoGraph | 1.298 | 1.242 | 0.296 | 4 | 41.15 | 2.273 | 2.238 | 5.7 | 83.24 | 0.204 | 6.169 | 0.159 | 4.3
Cont.Pred | 5.498 | 1.626 | 0.316 | 8 | 37.78 | 2.286 | 2.413 | 7.3 | 183.0 | 0.194 | 8.691 | 0.108 | 5.8
G-Motif | 3.085 | 1.372 | 0.531 | 9.7 | 51.83 | 2.363 | 2.758 | 9.7 | 98.21 | 0.268 | 7.333 | 0.182 | 7.5
G-Cont. | 0.036 | 1.242 | 0.403 | 3.3 | 33.55 | 1.773 | 2.660 | 4.3 | 113.6 | 0.170 | 5.330 | 0.045 | 2.3
GraphCL | 0.854 | 1.110 | 0.461 | 4.7 | 34.97 | 1.863 | 2.271 | 3.3 | 89.79 | 0.226 | 6.191 | 0.152 | 5.8
JOAO | 0.637 | 1.268 | 0.412 | 5 | 33.67 | 2.084 | 2.307 | 4.7 | 89.38 | 0.214 | 5.960 | 0.142 | 3.5
JOAOv2 | 0.591 | 1.272 | 0.463 | 6 | 32.81 | 2.054 | 2.340 | 4 | 88.27 | 0.217 | 5.964 | 0.148 | 4.0
GSSL Worse | 100% | 90% | 90% | 100% | 100% | 80% | 0% | 50% | 10% | 10% | 10% | 10% | 10%
Table 4: Performance on the topological metrics prediction with embeddings extracted from different layers. We report the mean square error or the cross entropy loss (the smaller the better). We bold the best and underline the worst performance on node degree (ND) and graph diameter (GD).  | Random | AttrMask | GraphCL
---|---|---|---
| 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5
ND | 0.031 | 0.013 | 0.007 | 0.001 | 0.001 | 0.009 | 0.024 | 0.019 | 0.013 | 0.015 | 0.016 | 0.361 | 0.426 | 0.583 | 0.854
GD | 184.0 | 236.2 | 167.4 | 184.0 | 222.5 | 198.9 | 181.9 | 184.6 | 171.7 | 164.7 | 93.34 | 131.5 | 97.43 | 95.19 | 89.79

We benchmark GSSL methods on all the molecular graphs from each dataset for
node- and graph-level topological properties. For pair-level metrics, we
bootstrap 10k node pairs from each dataset. The reported test scores are
averaged over test splits across three independent runs. We plot the empirical
distribution of these metrics for the molecular graphs of the downstream
datasets in Appendix G.

Figure 3: Random generates more interpretable latent space: T-SNE
visualisation of node embeddings produced by Random, AttrMask, and GraphCL on
BBBP. Each dot represents an atom from a molecular graph from BBBP dataset,
coloured by node degrees. We plot the distributions of these topological
properties as plotted in Appendix G. We notice that Random embeddings form
more coherent clusters, consistent with their performance on node-level tasks.

Findings. Table 3 shows the results, and we summarise the following findings.

  1. 3)

GSSL learns hierarchical features in a localisable way. This observation
results from the following analysis: in Table 4, we extract embeddings from
different layers of the models and evaluate how they perform on predicting the
structural metrics.

     * •

The first layer of pre-trained embeddings performs the best on local
structural metrics. Such local information is gradually lost as more GNN
layers are added.

     * •

The last layer of pre-trained embeddings performs the best on global
structural metrics. This confirms that global structural information emerges
from GSSL methods (opposed to noise).

  2. 4)

No single method is omnipotent. The best methods for node-, pair-, and graph-
level metrics are Random, GraphCL, and G-Contextual, respectively.

  3. 5)

Random outperforms many GSSL methods on node- and pair-level tasks. (in Fig.
3)

  4. 6)

Random falls short of predicting graph-level metrics. Besides the negatively-
transferred method EdgePred, the remaining GSSL methods outperform Random.

  5. 7)

JOAO methods underperform GraphCL on node- and pair-level tasks, despite being
a follow-up extension of GraphCL.

  6. 8)

The Contextual proxy objective of G-Contextual is helpful for Jaccard
coefficient prediction because of its resemblance with the coefficient’s
neighbourhood overlap.

Table 5: Cramér’s V between molecular substructure counts and binary properties, averaged over 678 binary property prediction tasks or eight downstream datasets. | allylic | benzene | amide | ether | halogen
---|---|---|---|---|---
Cramér’s V (task) | 0.1144 | 0.1630 | 0.0881 | 0.1034 | 0.1721
Cramér’s V (dataset) | 0.1024 | 0.1227 | 0.1336 | 0.1083 | 0.1086
Table 6: Benchmarking substructure properties.  | allylic | amide | benzene | ether | halogen
---|---|---|---|---|---
Random | 3.516 | 18.948 | 3.964 | 6.071 | 3.652
EdgePred | 5.854 | 25.126 | 9.804 | 12.838 | 5.214
AttrMask | 3.371 | 12.932 | 2.860 | 4.958 | 1.192
GPT-GNN | 2.808 | 15.736 | 2.938 | 5.932 | 2.912
InfoGraph | 2.577 | 5.535 | 1.959 | 3.657 | 2.819
Cont.Pred | 4.386 | 18.251 | 3.583 | 7.045 | 2.908
G-Motif | 2.452 | 4.015 | 2.116 | 3.507 | 1.125
G-Cont. | 2.196 | 5.938 | 1.926 | 2.900 | 0.759
GraphCL | 2.088 | 3.922 | 1.722 | 3.766 | 0.798
JOAO | 2.385 | 4.030 | 1.746 | 3.376 | 0.694
JOAOv2 | 2.122 | 3.865 | 1.773 | 3.388 | 0.695
GSSL Worse | 20% | 10% | 10% | 20% | 10%
Rank Corr. | 0.709 | 0.576 | 0.468 | 0.552 | 0.515

###  6.2 Molecular substructure properties

We use Cramér’s V to identify the five substructures that are mostly
associated with downstream biochemical properties, as reported in Table 5,
detailed in Table 14. Then, we predict these five substructure counts from
embeddings. We provide results and plot the distributions of all substructures
in Appendix E.

Findings. Table 6 shows the results, where we bold the best and underline the
worst score of each substructure. We report the percentage of Random
outperform GSSL methods, and the Spearman rank correlation between the
substructure property and downstream performance. We highlight the following
findings.

  1. 9)

Substructure detection performance correlates well with MPP performance. Pre-
trained embeddings outperform Random in both detecting substructures as well
as for MPP tasks. This demonstrates that substructure information emerges from
GSSL.

  2. 10)

However, contrastive GSSL performs better than in MPP experiments:
GraphCL,JOAO, and JOAOv2 perform consistently better than most other GSSL
methods on substructure detection, which is not the case for MPP tasks in Sec.
4.

  3. 11)

Random falls short of predicting substructure properties. Besides the
negatively transferred method EdgePred, other nine GSSL methods outperform
Random.

###  6.3 Embedding space properties

Figure 4: Evaluating alignment in the embedding space.

For alignment, we randomly select 10k positive and negative pairs of molecular
graphs from BBBP and Tox21 datasets, calculate the cosine distance and plot
the histogram in Fig. 4. We choose AttrMask and GraphCL to represent
generative and contrastive GSSL methods, respectively. Table 7 presents
uniformity values as defined in [55]; Fig. 5 plots the magnitude of the
singular values in the logarithm scale provided in Table 8 (in the logarithm
scale).

Findings. Based on above results, we summarise the following findings.

  1. 12)

GSSL methods give rise to better alignment than Random. GSSL embeddings form
distinguishable distributions for positive/negative pairs, while the Random
embeddings do not (a phenomenon often referred to as _over-smoothing_ [6]).

  2. 13)

GSSL methods promote more uniform features than Random: Relatively speaking,
except for EdgePred, all GSSL methods achieve uniformity values that are many
times less than Random’s. Nonetheless, absolutely speaking, there is still
room for improvement, as the range of uniformity is $[-4,0]$ (the lower, the
more uniform) [55].

  3. 14)

Uniformity rankings do not correlate perfectly with MPP rankings, despite the
results shown by Wang and Isola [55]. For example, InfoGraph has achieved the
best score in uniformity yet ranked 6th among the ten GSSL methods on the
downstream MPP tasks.

  4. 15)

Most GSSL methods “lift” the spectrum compared to Random. The singular values
of stacked GSSL embedding vectors are larger than Random’s by multiple
magnitudes.

  5. 16)

Spectrum analysis reveals negative transfer. Previously, in Sec. 4, we noticed
that EdgePred performed worse on MPP tasks than Random both with fixed
embeddings as well as fine-tuned ones (a pathology usually referred to as
“negative transfer” [57]). Now, the spectrum analysis reveals that EdgePred
indeed suffered from dimensional collapse (as described in Sec. 5.3).

Figure 5: Dimensional collapse analysis. We plot the spectrum of the embedding space on BBBP. Left: node; Right: graph. The singular values of EdgePred embeddings drop to zero rapidly. Table 7:  Evaluating GSSL methods on uniformity Dataset | BBBP $\downarrow$ | Tox21 $\downarrow$ | ToxCast $\downarrow$ | Sider $\downarrow$ | ClinTox $\downarrow$ | MUV $\downarrow$ | HIV $\downarrow$ | Bace $\downarrow$
---|---|---|---|---|---|---|---|---
Random | -0.001 | -0.002 | -0.002 | -0.002 | -0.001 | -0.001 | -0.001 | -0.000
EdgePred | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | -0.000 | -0.000 | 0.000
AttrMask | -0.006 | -0.011 | -0.011 | -0.009 | -0.008 | -0.005 | -0.008 | -0.004
GPT-GNN | -0.017 | -0.020 | -0.020 | -0.017 | -0.020 | -0.011 | -0.014 | -0.010
InfoGraph | -0.210 | -0.256 | -0.256 | -0.222 | -0.233 | -0.183 | -0.208 | -0.110
Cont.Pred | -0.010 | -0.016 | -0.016 | -0.015 | -0.013 | -0.005 | -0.010 | -0.007
G-Cont. | -0.012 | -0.015 | -0.015 | -0.013 | -0.013 | -0.010 | -0.013 | -0.006
G-Motif | -0.023 | -0.023 | -0.024 | -0.023 | -0.023 | -0.021 | -0.023 | -0.020
GraphCL | -0.091 | -0.118 | -0.117 | -0.093 | -0.099 | -0.082 | -0.092 | -0.062
JOAO | -0.071 | -0.091 | -0.091 | -0.071 | -0.080 | -0.063 | -0.069 | -0.047
JOAOv2 | -0.053 | -0.067 | -0.066 | -0.053 | -0.062 | -0.045 | -0.047 | -0.033
GSSL Worse | 10% | 10% | 10% | 10% | 10% | 10% | 10% | 10%
Rank Corr. | -0.509 | -0.436 | -0.436 | -0.436 | -0.489 | -0.531 | -0.509 | -0.408

##  7 Conclusion

In this work, we challenged common practices in evaluating graph self-
supervised learned embeddings of molecular graphs. First, we systematically
evaluated 10 popular GSSL methods on common molecular property prediction
tasks (MoleculeNet) in an unbiased and controlled manner.

Table 8: Benchmarking the magnitude of the singular values in the spectrum.  No. Dim | 1 | 50 | 100
---|---|---|---
Random | -2.95 | -14.9 | -17.0
EdgePred | -32.3 | -49.8 | -69.1
AttrMask | 1.16 | -10.3 | -12.9
GPT-GNN | 2.51 | -8.61 | -10.6
InfoGraph | 3.08 | -2.15 | -4.91
Cont.Pred | 1.88 | -11.9 | -14.8
G-Motif | 4.70 | -0.91 | -4.48
G-Cont. | 3.15 | -4.14 | -7.09
GraphCL | 3.69 | -4.38 | -7.66
JOAO | 3.59 | -4.28 | -7.44
JOAOv2 | 3.85 | -4.98 | -7.88
GSSL smaller | 10% | 10% | 10%
Rank Corr. | 0.809 | 0.755 | 0.755

Next, we presented MolGraphEval, which is a diverse collection of probe tasks
divided into three categories: (i) topological properties, (ii) substructure
properties, and (iii) embedding space properties. Then we evaluated GSSL
methods on MolGraphEval and found surprising insights that were not revealed
by the evaluation of MPP tasks alone.

The purpose of this benchmark is to complement current evaluation practices
with probe tasks and metrics that reveal novel insights. Our empirical
findings suggest that there are many open questions on how to learn robust
molecular graph embeddings without labels and a better understanding of these,
along with a new methodology for solving some of the issues mentioned earlier
(_e.g._ , dimensional collapse), are yet to come. Nevertheless, we are
optimistic that the tasks proposed in this paper will benefit the GSSL
research community to tackle these challenges and applied scientists in fields
like drug discovery to yield additional insights that can help their problem.

## References

  * Alain and Bengio [2017] Guillaume Alain and Yoshua Bengio.  Understanding intermediate layers using linear classifier probes.  In _ICLR Workshop_ , 2017.
  * Axelrod and Gomez-Bombarelli [2020] Simon Axelrod and Rafael Gomez-Bombarelli.  Geom: Energy-annotated molecular conformations for property prediction and molecular generation.  _arXiv:2006.05531_ , 2020.
  * Bohacek et al. [1996] RS Bohacek, C McMartin, and WC Guida.  The art and practice of structure-based drug design: a molecular modeling perspective.  _Medicinal research reviews_ , 16(1):3—50, 1996\.
  * Bort et al. [2021] William Bort, Igor I Baskin, Timur Gimadiev, Artem Mukanov, Ramil Nugmanov, Pavel Sidorov, Gilles Marcou, Dragos Horvath, Olga Klimchuk, Timur Madzhidov, et al.  Discovery of novel chemical reactions by deep generative recurrent neural network.  _Scientific reports_ , 11(1):1–15, 2021.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Chen et al. [2020a] Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun.  Measuring and relieving the over-smoothing problem for graph neural networks from the topological view.  In _AAAI_ , 2020a.
  * Chen et al. [2020b] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _ICML_ , 2020b.
  * Chen et al. [2021] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  _arXiv:2104.02057_ , 2021.
  * Conneau and Kiela [2018] Alexis Conneau and Douwe Kiela.  Senteval: An evaluation toolkit for universal sentence representations.  In _LREC_ , 2018.
  * Corso et al. [2020] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković.  Principal neighbourhood aggregation for graph nets.  In _NeurIPS_ , 2020.
  * Dohan et al. [2021] David Dohan, Andreea Gane, Maxwell L Bileschi, David Belanger, and Lucy Colwell.  Improving protein function annotation via unsupervised pre-training: Robustness, efficiency, and insights.  In _KDD_ , 2021.
  * Duvenaud et al. [2015] David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams.  Convolutional networks on graphs for learning molecular fingerprints.  In _NeurIPS_ , 2015.
  * Elnaggar et al. [2021] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al.  Prottrans: towards cracking the language of life’s code through self-supervised deep learning and high performance computing.  _IEEE PAMI_ , 2021.
  * Gilmer et al. [2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In _ICML_ , 2017.
  * Glorot and Bengio [2010] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * Hamilton [2020] William L. Hamilton.  Graph representation learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , 14(3):1–159, 2020.
  * Hamilton et al. [2017] William L Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In _NeurIPS_ , 2017.
  * Hataya et al. [2021] Ryuichiro Hataya, Hideki Nakayama, and Kazuki Yoshizoe.  Graph energy-based model for substructure preserving molecular design.  _arXiv preprint arXiv:2102.04600_ , 2021.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick.  Masked autoencoders are scalable vision learners.  In _CVPR_ , 2022.
  * Hendricks et al. [2021] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, and Aida Nematzadeh.  Decoupling the role of data, attention, and losses in multimodal transformers.  _TACL_ , 2021.
  * Hewitt and Manning [2019] John Hewitt and Christopher D. Manning.  A structural probe for finding syntax in word representations.  In _NAACL_ , 2019.
  * Hu et al. [2020a] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec.  Strategies for pre-training graph neural networks.  In _ICLR_ , 2020a.
  * Hu et al. [2021] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  In _NeurIPS_ , 2021.
  * Hu et al. [2020b] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun.  Gpt-gnn: Generative pre-training of graph neural networks.  In _KDD_ , 2020b.
  * Hua et al. [2021] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao.  On feature decorrelation in self-supervised learning.  In _ICCV_ , 2021.
  * Jawahar et al. [2019] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah.  What does BERT learn about the structure of language?  In _ACL_ , 2019.
  * Jing et al. [2022] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  In _ICLR_ , 2022.
  * Kassner and Schütze [2020] Nora Kassner and Hinrich Schütze.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _ACL_ , 2020.
  * Kwon et al. [2020] Youngchun Kwon, Dongseon Lee, Youn-Suk Choi, Kyoham Shin, and Seokho Kang.  Compressed graph representation for scalable molecular graph generation.  _Journal of Cheminformatics_ , 12(1):1–8, 2020\.
  * Li et al. [2021] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao.  Efficient self-supervised vision transformers for representation learning.  _arXiv preprint arXiv:2106.09785_ , 2021.
  * Liu et al. [2019] Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith.  Linguistic knowledge and transferability of contextual representations.  In _NAACL_ , 2019.
  * Liu et al. [2018] Shengchao Liu, Mehmet Furkan Demirel, and Yingyu Liang.  N-gram graph: Simple unsupervised representation for graphs, with applications to molecules.  In _NeurIPS_ , 2018.
  * Liu et al. [2022] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang.  Pre-training molecular graph representation with 3d geometry.  In _ICLR_ , 2022.
  * Liu et al. [2021a] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang.  Self-supervised learning: Generative or contrastive.  _IEEE TKEE_ , 2021a.
  * Liu et al. [2021b] Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S Yu.  Graph self-supervised learning: A survey.  _arXiv:2103.00111_ , 2021b.
  * Lü and Zhou [2011] Linyuan Lü and Tao Zhou.  Link prediction in complex networks: A survey.  _Physica A: statistical mechanics and its applications_ , 2011.
  * McMurry [2014] John E McMurry.  _Organic chemistry with biological applications_.  Cengage Learning, 2014.
  * Muratov et al. [2020] Eugene N Muratov, Jürgen Bajorath, Robert P Sheridan, Igor V Tetko, Dmitry Filimonov, Vladimir Poroikov, Tudor I Oprea, Igor I Baskin, Alexandre Varnek, Adrian Roitberg, et al.  Qsar without borders.  _Chemical Society Reviews_ , 49(11):3525–3564, 2020.
  * Nugmanov et al. [2019] Ramil I Nugmanov, Ravil N Mukhametgaleev, Tagir Akhmetshin, Timur R Gimadiev, Valentina A Afonina, Timur I Madzhidov, and Alexandre Varnek.  Cgrtools: Python library for molecule, reaction, and condensed graph of reaction processing.  _Journal of chemical information and modeling_ , 59(6):2516–2521, 2019.
  * Rao et al. [2019] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S Song.  Evaluating protein transfer learning with tape.  In _NeurIPS_ , 2019.
  * Resnick et al. [2019] Cinjon Resnick, Zeping Zhan, and Joan Bruna.  Probing the state of the art: A critical look at visual representation evaluation.  _arXiv:1912.00215_ , 2019.
  * Rives et al. [2021] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al.  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.  _PNAS_ , 2021.
  * Rong et al. [2020] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang.  Self-supervised graph transformer on large-scale molecular data.  In _NeurIPS_ , 2020.
  * Rücker and Rücker [2001] Gerta Rücker and Christoph Rücker.  Substructure, subgraph, and walk counts as measures of the complexity of graphs and molecules.  _Journal of chemical information and computer sciences_ , 41(6):1457–1462, 2001.
  * Salmaso and Moro [2018] Veronica Salmaso and Stefano Moro.  Bridging molecular docking to molecular dynamics in exploring ligand-protein recognition process: An overview.  _Frontiers in pharmacology_ , page 923, 2018.
  * Sanchez-Lengeling et al. [2019] Benjamin Sanchez-Lengeling, Loïc M Roch, José Darío Perea, Stefan Langner, Christoph J Brabec, and Alán Aspuru-Guzik.  A bayesian approach to predict solubility parameters.  _Advanced Theory and Simulations_ , 2(1):1800069, 2019.
  * Stanley et al. [2021] Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Marc Brockschmidt.  FS-mol: A few-shot learning dataset of molecules.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Sun et al. [2020] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang.  Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization.  In _ICLR_ , 2020.
  * Tenney et al. [2019] Ian Tenney, Dipanjan Das, and Ellie Pavlick.  BERT rediscovers the classical NLP pipeline.  In _ACL_ , 2019.
  * Villegas-Morcillo et al. [2021] Amelia Villegas-Morcillo, Stavros Makrodimitris, Roeland CHJ van Ham, Angel M Gomez, Victoria Sanchez, and Marcel JT Reinders.  Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function.  _Bioinformatics_ , 2021.
  * Vogt et al. [2010] Martin Vogt, Dagmar Stumpfe, Hanna Geppert, and Jürgen Bajorath.  Scaffold hopping using two-dimensional fingerprints: True potential, black magic, or a hopeless endeavor? guidelines for virtual screening.  _Journal of Medicinal Chemistry_ , 53(15):5707–5715, 2010.
  * Wang et al. [2019a] Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman.  Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling.  In _ACL_ , 2019a.
  * Wang et al. [2021] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matthew J. Kusner.  Unsupervised point cloud pre-training via occlusion completion.  In _ICCV_ , 2021.
  * Wang et al. [2020] Shuang Wang, Zhen Li, Shugang Zhang, Mingjian Jiang, Xiaofeng Wang, and Zhiqiang Wei.  Molecular property prediction based on a multichannel substructure graph.  _IEEE Access_ , 8:18601–18614, 2020.
  * Wang and Isola [2020] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pages 9929–9939. PMLR, 2020.
  * Wang et al. [2022] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani.  Molecular contrastive learning of representations via graph neural networks.  _Nature Machine Intelligence_ , 4(3):279–287, 2022.
  * Wang et al. [2019b] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell.  Characterizing and avoiding negative transfer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11293–11302, 2019b.
  * Watts and Strogatz [1998] Duncan J Watts and Steven H Strogatz.  Collective dynamics of ‘small-world’ networks.  _Nature_ , 1998.
  * Wolpert and Macready [1997] David H Wolpert and William G Macready.  No free lunch theorems for optimization.  _IEEE transactions on evolutionary computation_ , 1(1):67–82, 1997.
  * Wu et al. [2021] Lirong Wu, Haitao Lin, Zhangyang Gao, Cheng Tan, Stan Li, et al.  Self-supervised on graphs: Contrastive, generative, or predictive.  _arXiv:2105.07342_ , 2021.
  * Wu et al. [2018] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande.  Moleculenet: a benchmark for molecular machine learning.  _Chemical Science_ , 2018.
  * Xie et al. [2021] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji.  Self-supervised learning of graph neural networks: A unified review.  _arXiv:2102.10757_ , 2021.
  * Xu et al. [2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  In _ICLR_ , 2019.
  * Yang et al. [2019] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al.  Analyzing learned molecular representations for property prediction.  _Journal of chemical information and modeling_ , 2019.
  * Ye et al. [2022] Xian-bin Ye, Quanlong Guan, Weiqi Luo, Liangda Fang, Zhao-Rong Lai, and Jun Wang.  Molecular substructure graph attention network for molecular property identification in drug discovery.  _Pattern Recognition_ , 128:108659, 2022.
  * You et al. [2020] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen.  Graph contrastive learning with augmentations.  In _NeurIPS_ , 2020.
  * You et al. [2021] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang.  Graph contrastive learning automated.  In _ICML_ , 2021.
  * Zhu et al. [2021] Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu.  An empirical study of graph contrastive learning.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.

##  Appendix A GSSL methods

We refer to the Appendix A & B of GraphMVP [33] and the original literature
for the descriptions of the GSSL methods. The main focus of this work is the
GSSL methods on 2D molecular graphs.

##  Appendix B Probe models - on the choice of architectures

In principle, we want the probe models to be neither simple nor powerful. If
the probe is too simple, then it won’t be able to capture the information
stored in the representations; on the other hand, a powerful probe itself will
directly learn to precisely predict the properties since randomised embeddings
can already provide helpful information (especially node-level metrics);
therefore the predictive performance won’t be an accurate reflection on the
information contained in the representations. On such a basis, we must choose
the probe models’ architecture carefully.

We first narrow down the search space to mono- and multi-layer perceptron, as
they are widely used in previous studies [1, 49, 31, 9, 40]. It then becomes
simple to quantify the capacity/expressiveness of the probe models by varying
the depth and width. To testify the capability, we use embeddings pre-trained
via AttrMask and GraphCL to predict the two metrics, node degree and graph
diameter, with various depth and width. We vary the number of hidden layers
from 0 to 3, and the dimensions from 100 to 1200. We find that: a linear model
is too simple to capture helpful information while three layers of MLPs will
learn to predict by themselves. Neither of them provided much information and
insights on the learned embeddings.

##  Appendix C Pre-Training

###  C.1 In-domain generalisation

We pre-train and probe the generic graph properties (“Node Degree”) on the
same downstream datasets to testify the in-domain generalisation. Table 9
shows the results of Random, AttrMask and GraphCL embeddings. We notice that
pre-train and probe on the same dataset does not yield the best performance.
We then study how the quantity of pre-training data affect the embedding
quality in capturing these generic graph properties.

Table 9: In-domain generalisation, on “Node Degree”.  | BBBP | Tox21 | ToxCast | Sider | ClinTox | MUV | HIV | Bace | Avg
---|---|---|---|---|---|---|---|---|---
Random | 0.001 | 0.002 | 0.002 | 0.004 | 0.001 | 0.001 | 0.003 | 0.000 | 0.002
Pre-trained with AttrMask |  |  |
BBBP | 0.240 | 0.444 | 0.455 | 0.162 | 0.140 | 1.301 | 0.987 | 0.199 | 0.491
Tox21 | 1.795 | 2.220 | 2.815 | 1.139 | 1.247 | 8.288 | 6.513 | 1.179 | 3.149
ToxCast | 2.337 | 1.020 | 1.375 | 0.728 | 0.561 | 1.760 | 1.669 | 0.340 | 1.224
Sider | 0.162 | 0.353 | 0.325 | 0.124 | 0.122 | 0.650 | 0.663 | 0.085 | 0.311
ClinTox | 0.195 | 0.291 | 0.298 | 0.239 | 0.131 | 0.180 | 0.492 | 0.058 | 0.235
MUV | 0.971 | 2.402 | 3.128 | 1.448 | 0.643 | 9.894 | 6.837 | 0.782 | 3.263
HIV | 0.900 | 2.363 | 2.875 | 1.051 | 0.571 | 6.598 | 3.838 | 0.688 | 2.361
BACE | 0.107 | 0.268 | 0.408 | 0.182 | 0.116 | 0.468 | 0.617 | 0.100 | 0.283
Pre-trained with GraphCL |  |  |
bbbp | 0.267 | 0.221 | 0.290 | 0.542 | 0.363 | 0.097 | 0.088 | 0.128 | 0.250
tox21 | 0.420 | 0.622 | 0.456 | 0.835 | 0.141 | 0.190 | 0.193 | 1.166 | 0.503
toxcast | 1.190 | 1.277 | 0.231 | 0.260 | 0.143 | 4.476 | 4.406 | 2.294 | 1.785
sider | 0.138 | 0.111 | 0.049 | 0.058 | 0.312 | 0.368 | 0.225 | 0.071 | 0.166
clintox | 0.071 | 0.059 | 1.669 | 1.204 | 1.316 | 0.084 | 0.030 | 6.563 | 1.374
muv | 25.438 | 23.241 | 24.361 | 26.303 | 5.507 | 6.575 | 7.352 | 95.403 | 26.773
hiv | 86.420 | 104.899 | 4.126 | 4.860 | 0.272 | 1.112 | 1.147 | 1.288 | 25.515
bace | 0.247 | 0.212 | 0.257 | 0.266 | 6.807 | 4.660 | 4.478 | 0.199 | 2.141

###  C.2 On the quantity of the pre-training data

Intuitively, increasing the quantity of (carefully curated) pre-training data
usually would increase the performance of the pre-trained models. While being
validated in vision, language and biomedicine (protein sequence) data, we
observe that such claim is not correct for the case of molecular graphs.

Table 10: Performance on the topological metrics predictions. We report the mean square or the cross entropy loss (_i.e._ , the smaller the better), over all 8 downstream datasets. We vary the amount of pre-training data (_i.e._ , GEOM) from 5k/10k/50k/100k/200k/330k(all) molecules. We use “$\uparrow$” to represent the performance increases as the data quantity increases, “$\downarrow$” to represent the increase of the pre-train data hurdle the performance, “$-$” to represent there is no rules observed.  Metrics | Node | Pair | Graph
---|---|---|---
Pre-training | Degree | Cent. | Cluster | Link | Jaccord | Katz | Diameter | Con. | Cycle | Assort.
AttrMask | $\uparrow$ | - | $\uparrow$ | - | $\downarrow$ | $\uparrow$ | - | $\downarrow$ | - | $\downarrow$
GPT-GNN | $\uparrow$ | - | $\uparrow$ | - | - | $\uparrow$ | - | $\uparrow$ | - | $\downarrow$
InfoGraph | $\downarrow$ | - | $\uparrow$ | - | $\downarrow$ | $\uparrow$ | - | - | - | -
Cont.Pred | $\uparrow$ | $\uparrow$ | - | $\downarrow$ | - | $\uparrow$ | - | $\uparrow$ | - | $\downarrow$
G-Motif | $\uparrow$ | - | $\uparrow$ | - | $\downarrow$ | $\uparrow$ | - | - | - | $\downarrow$
G-Cont. | $\uparrow$ | - | - | - | $\downarrow$ | $\uparrow$ | $\uparrow$ | - | - | -
GraphCL | - | - | $\uparrow$ | $\downarrow$ | - | $\uparrow$ | $\uparrow$ | - | $\downarrow$ | $\uparrow$
JOAO | $\uparrow$ | - | $\downarrow$ | - | $\downarrow$ | $\uparrow$ | - | $\uparrow$ | - | $\downarrow$
JOAOv2 | $\uparrow$ | - | $\downarrow$ | - | $\downarrow$ | $\uparrow$ | - | $\downarrow$ | - | $\downarrow$

Table 10 shows the results. We observe that:

  * •

There is no GSSL methods whose performance consistently improves as the pre-
train data increase.

  * •

Besides “Katz Index”, increasing pre-training data does not always help/hurdle
predict generic graph properties across all the GSSL methods.

##  Appendix D Randomised embeddings

#### How GNN models are initialised.

We first analyse how weights in the GNNs are initialised (PyTorch and PyTorch-
Geometric (PyG)).

  * •

Edge Embedding Layers uses ‘xavier uniform’, essentially samples from uniform
distribution

  * •

GNN Layers in fact only have MLP weights (see PyG Doc), same initialisation as
Linear layers.

  * •

Linear Layers samples from uniform distribution for both weight and bias
(PyTorch Doc)

Table 11: Probe measures after re-initialising parts of weights of the pre-trained GNNs. We bold the best and underline the worst performance of each (pre-trained) embeddings on each structural metric. The digits in the “Re-Init” row represent the index/location of re-initialised GIN convolutional layers.  | AttrMask | GraphCL
---|---|---
| None | Embed | 1 | 2 | 3 | 4 | 5 | None | Embed | 1 | 2 | 3 | 4 | 5
ND | 0.015 | 0.263 | 0.062 | 0.076 | 0.021 | 0.063 | 0.041 | 0.854 | 1.429 | 1.439 | 1.086 | 1.406 | 1.111 | 1.399
GD | 164.7 | 178.4 | 188.5 | 184.6 | 191.1 | 225.1 | 229.7 | 89.79 | 90.14 | 102.6 | 111.5 | 136.5 | 139.5 | 163.9

#### Re-initialise parts of pre-trained models.

We then investigate how the extracted embeddings perform after re-initialising
parts of the pre-trained GNN weights. The GIN architecture [63, 22] used in
the Graph SSL research line consists of a node/edge feature embedding layer,
and five subsequent GINConv layers and an output/readout layer. We use Glorot
uniform initialisation for all network modules [15]. We re-initialise the
parts of pre-trained GNN weights with Glorot uniform initialisation [15], and
extract the embeddings from the output layer, train probe models of these
embeddings on the node degree (“ND”) and graph diameter (“GD”) metrics in
Table 11. We observe:

  * •

Re-initialising _any_ layer might bring downside to the stored local topology
info (node degree);

  * •

Perturbing the late-stage (_i.e._ , 4/5-th) layers would harm the learned
hierarchical info (diameter).

Compared with random initialised networks, the later stage of the pre-trained
message passing modules can better capture the global information. The
embeddings extracted from the early stage contain more information on the
local neighbourhood.

##  Appendix E Substructure

###  E.1 Detailed performance

Descriptions of the chemical subgraphs can be referred to document of
rdkit.Chem.Fragments. Table 12 and Table 13 report the detailed performance of
substructure property prediction.

Table 12:  Substructure Detection, MSE Loss, Part I  Pre-training | allylic | amide | amidine | Azo | benzene | epoxide | ether | furan | guanido | halogen | imidazole | imide
---|---|---|---|---|---|---|---|---|---|---|---|---
– | 3.516 | 18.948 | 0.370 | 0.216 | 3.964 | 0.146 | 6.071 | 0.536 | 0.217 | 3.652 | 0.531 | 0.417
EdgePred | 5.854 | 25.126 | 0.417 | 0.251 | 9.804 | 0.154 | 12.838 | 0.705 | 0.249 | 5.214 | 0.597 | 0.578
AttrMask | 3.371 | 12.932 | 0.364 | 0.174 | 2.860 | 0.123 | 4.958 | 0.242 | 0.213 | 1.192 | 0.459 | 0.319
GPT-GNN | 2.808 | 15.736 | 0.360 | 0.156 | 2.938 | 0.141 | 5.932 | 0.407 | 0.220 | 2.912 | 0.483 | 0.474
InfoGraph | 2.577 | 5.535 | 0.327 | 0.143 | 1.959 | 0.116 | 3.657 | 0.116 | 0.207 | 2.819 | 0.302 | 0.326
Cont.Pred | 4.386 | 18.251 | 0.356 | 0.198 | 3.583 | 0.137 | 7.045 | 0.507 | 0.203 | 2.908 | 0.520 | 0.488
G-Motif | 2.452 | 4.015 | 0.194 | 0.082 | 2.116 | 0.134 | 3.507 | 0.068 | 0.115 | 1.125 | 0.348 | 0.150
G-Cont. | 2.196 | 5.938 | 0.172 | 0.067 | 1.926 | 0.114 | 2.900 | 0.084 | 0.121 | 0.759 | 0.249 | 0.160
GraphCL | 2.088 | 3.922 | 0.262 | 0.124 | 1.722 | 0.127 | 3.766 | 0.074 | 0.167 | 0.798 | 0.274 | 0.192
JOAO | 2.385 | 4.030 | 0.280 | 0.113 | 1.746 | 0.115 | 3.376 | 0.061 | 0.168 | 0.694 | 0.237 | 0.192
JOAOv2 | 2.122 | 3.865 | 0.262 | 0.112 | 1.773 | 0.123 | 3.388 | 0.068 | 0.125 | 0.695 | 0.244 | 0.197
SSL Worse | 20% | 10% | 10% | 10% | 10% | 10% | 20% | 10% | 20% | 10% | 10% | 20%
Table 13:  Substructure Detection, MSE Loss, Part II  Pre-training | lactam | morpholine | NO | oxazole | piperdine | piperzine | pyridine | tetrazole | thiazole | thiophene | urea
---|---|---|---|---|---|---|---|---|---|---|---
– | 0.049 | 0.439 | 0.214 | 0.121 | 1.094 | 0.410 | 1.528 | 0.141 | 0.402 | 0.550 | 0.458
EdgePred | 0.055 | 0.547 | 0.218 | 0.126 | 1.332 | 0.592 | 1.674 | 0.195 | 0.470 | 0.782 | 0.782
AttrMask | 0.043 | 0.196 | 0.212 | 0.107 | 0.917 | 0.250 | 1.111 | 0.089 | 0.339 | 0.313 | 0.222
GPT-GNN | 0.041 | 0.195 | 0.232 | 0.098 | 0.878 | 0.293 | 1.416 | 0.137 | 0.382 | 0.515 | 0.443
InfoGraph | 0.041 | 0.121 | 0.241 | 0.054 | 0.813 | 0.206 | 0.655 | 0.075 | 0.170 | 0.180 | 0.174
Cont.Pred | 0.043 | 0.298 | 0.223 | 0.120 | 1.097 | 0.387 | 1.334 | 0.146 | 0.379 | 0.538 | 0.538
G-Motif | 0.048 | 0.153 | 0.212 | 0.031 | 0.768 | 0.219 | 0.670 | 0.034 | 0.081 | 0.092 | 0.092
G-Cont. | 0.041 | 0.096 | 0.126 | 0.050 | 0.653 | 0.149 | 0.668 | 0.032 | 0.129 | 0.104 | 0.103
GraphCL | 0.042 | 0.113 | 0.199 | 0.048 | 0.706 | 0.159 | 0.460 | 0.037 | 0.129 | 0.088 | 0.087
JOAO | 0.043 | 0.096 | 0.194 | 0.040 | 0.738 | 0.149 | 0.434 | 0.032 | 0.121 | 0.077 | 0.076
JOAOv2 | 0.042 | 0.123 | 0.204 | 0.041 | 0.706 | 0.146 | 0.446 | 0.037 | 0.136 | 0.083 | 0.083
SSL Worse | 10% | 10% | 40% | 10% | 20% | 10% | 10% | 20% | 10% | 10% | 20%

###  E.2 Cramer’s V

#### Cramér’s V

quantifies the strength of the association between the molecular substructure
counts (_i.e._ ,, chemical fragments) and their biochemical properties. It is
defined as:

$V=\sqrt{\chi^{2}/\left(n\cdot\min(k-1,r-1)\right)}=\sqrt{\chi^{2}/n}\quad(r\equiv
2)$ (2)

where $n$ is the sample size, $k$ and $r$ are the total number of substructure
counts and property categories (binary), respectively. The Chi-squared
statistics $\chi^{2}$ is then calculated as:

$\chi^{2}=\sum_{i,j}\left(n_{(i,j)}-n_{(i,\cdot)}\cdot
n_{(\cdot,j)}/n\right)^{2}\Big{/}\left(n_{(i,\cdot)}\cdot
n_{(\cdot,j)}/n\right)$ (3)

where $n_{(i,j)}$ is the total occurrence for the pair of $(i,j)$. Here $i$ is
the specific count of a certain substructure, and $j$ represents the certain
outcome of a molecular biochemical property. Cramér’s V value ranges from 0 to
1, representing the associated strength between two categorical variables.

Table 14:  Cramér’s V between molecular substructure counts (categorical) and downstream properties (binary)  Pre-training | BBBP | Tox21 | ToxCast | Sider | ClinTox | MUV | HIV | Bace | Avg (Task) | Avg (Data)
---|---|---|---|---|---|---|---|---|---|---
# Molecules | 2,039 | 7,831 | 8,575 | 1,427 | 1,478 | 93,087 | 41,127 | 1,513 | – | –
# Tasks | 1 | 12 | 617 | 27 | 2 | 17 | 1 | 1 | – | –
allylic | 0.1602 | 0.1345 | 0.1156 | 0.1276 | 0.0935 | 0.0413 | 0.0280 | 0.1186 | 0.1144 | 0.1024
amide | 0.2692 | 0.0490 | 0.0858 | 0.1841 | 0.1326 | 0.0235 | 0.0689 | 0.2556 | 0.0881 | 0.1336
amidine | 0.0360 | 0.0291 | 0.0412 | 0.0323 | 0.0158 | 0.0117 | 0.0396 | 0.1328 | 0.0399 | 0.0423
azo | 0.0400 | 0.0399 | 0.0393 | 0.0277 | 0.0123 | 0.0007 | 0.2082 | - | 0.0384 | 0.0526
benzene | 0.1476 | 0.1632 | 0.1691 | 0.1149 | 0.1112 | 0.0289 | 0.1374 | 0.1091 | 0.1630 | 0.1227
epoxide | 0.0273 | 0.0481 | 0.0449 | 0.0300 | 0.0049 | 0.0005 | 0.0086 | - | 0.0437 | 0.0235
ether | 0.2314 | 0.0694 | 0.1060 | 0.1069 | 0.1023 | 0.0185 | 0.0498 | 0.1821 | 0.1034 | 0.1083
furan | 0.0635 | 0.0257 | 0.0387 | 0.0227 | 0.0061 | 0.0311 | 0.0148 | 0.0135 | 0.0375 | 0.0270
guanido | 0.0765 | 0.0201 | 0.0509 | 0.0715 | 0.0286 | 0.0057 | 0.0094 | 0.1088 | 0.0499 | 0.0464
halogen | 0.1488 | 0.0849 | 0.1827 | 0.0773 | 0.0908 | 0.0143 | 0.0347 | 0.2353 | 0.1721 | 0.1086
imidazole | 0.0601 | 0.0427 | 0.0492 | 0.0460 | 0.1212 | 0.0102 | 0.0398 | 0.1280 | 0.0483 | 0.0622
imide | 0.0951 | 0.0246 | 0.0401 | 0.0428 | 0.0518 | 0.0094 | 0.0188 | - | 0.0392 | 0.0404
lactam | 0.4263 | 0.0184 | 0.0116 | 0.0646 | 0.0543 | 0.0006 | 0.0048 | - | 0.0182 | 0.0830
morpholine | 0.0512 | 0.0126 | 0.0343 | 0.0268 | 0.0425 | 0.0068 | 0.0101 | 0.0668 | 0.0329 | 0.0314
N_O | 0.0438 | 0.0195 | 0.0467 | 0.0391 | 0.0709 | 0.0195 | 0.0144 | 0.0537 | 0.0452 | 0.0385
oxazole | 0.0126 | 0.0184 | 0.0321 | 0.0359 | 0.0123 | 0.0079 | 0.0080 | 0.0364 | 0.0312 | 0.0205
piperdine | 0.1450 | 0.0305 | 0.0844 | 0.0575 | 0.0418 | 0.0079 | 0.0226 | 0.0935 | 0.0803 | 0.0604
piperzine | 0.0509 | 0.0214 | 0.0421 | 0.0776 | 0.0648 | 0.0111 | 0.0192 | 0.0063 | 0.0424 | 0.0367
pyridine | 0.0598 | 0.0402 | 0.0549 | 0.0338 | 0.0833 | 0.0129 | 0.0300 | 0.1747 | 0.0529 | 0.0612
tetrazole | 0.1161 | 0.0158 | 0.0251 | 0.0300 | 0.0286 | 0.0083 | 0.0123 | 0.0334 | 0.0247 | 0.0337
thiazole | 0.1389 | 0.0521 | 0.0345 | 0.0445 | 0.0183 | 0.0118 | 0.0173 | 0.0539 | 0.0348 | 0.0464
thiophene | 0.0356 | 0.0467 | 0.0472 | 0.0315 | 0.0113 | 0.0166 | 0.0081 | 0.0438 | 0.0456 | 0.0301
urea | 0.0790 | 0.0236 | 0.0506 | 0.0471 | 0.0268 | 0.0079 | 0.0329 | 0.0516 | 0.0489 | 0.0399

###  E.3 Distribution of substructures

#### Allylic Oxide

.

#### Amide

.

#### Amidine

.

#### AZO

.

#### Benzene

.

#### Epoxide

.

#### Ether

.

#### Furan

.

#### Guanido

.

#### Halogen

.

#### Imidazole

.

#### Imide

.

#### Lactam

.

#### Morpholine

.

#### N_O

.

#### Oxazole

.

#### Piperdine

.

#### Piperzine

.

#### Pyridine

.

#### Tetrazole

.

#### Thiazole

.

#### Thiophene

.

#### Urea

.

##  Appendix F Spectrum

We provide more examples on the spectrum analyses of the embedding space from
different datasets. We omit EdgePred (dimensional collapse).

Figure 6: Spectrum of the GSSL embedding space on Bace dataset, Left: Node;
Right: Graph. Figure 7: Spectrum of the GSSL embedding space on Clintox
dataset, Left: Node; Right: Graph. Figure 8: Spectrum of the GSSL embedding
space on HIV dataset, Left: Node; Right: Graph. Figure 9: Spectrum of the GSSL
embedding space on HIV dataset, Left: Node; Right: Graph. Figure 10: Spectrum
of the GSSL embedding space on MUV dataset, Left: Node; Right: Graph. Figure
11: Spectrum of the GSSL embedding space on Sider dataset, Left: Node; Right:
Graph. Figure 12: Spectrum of the GSSL embedding space on Tox21 dataset, Left:
Node; Right: Graph. Figure 13: Spectrum of the GSSL embedding space on Toxcast
dataset, Left: Node; Right: Graph.

##  Appendix G Topological Metrics

We first describe the graph structural metrics and motif substructures used in
Sec. 5.2, we then visualise the histograms of the structural metrics and the
substructure w.r.t. the downstream datasets. Note that the vertical axes
sometimes are in the logarithm scale.

###  G.1 Distribution of Structural Metrics

#### Node Degree

.

#### Node Centrality

.

#### Node Clustering Coefficient

.

#### Link Prediction

.

#### Jaccard Coefficient

.

#### Katz Index

.

#### Cycle Basis

.

#### Graph Diameter

.

#### Aassortativity Coefficient

.

#### Average Clustering Coefficient

.