Will More Expressive Graph Neural Networks do Better on Generative
  Tasks?
Graph generation poses a significant challenge as it involves predicting a
complete graph with multiple nodes and edges based on simply a given label.
This task also carries fundamental importance to numerous real-world
applications, including de-novo drug and molecular design. In recent years,
several successful methods have emerged in the field of graph generation.
However, these approaches suffer from two significant shortcomings: (1) the
underlying Graph Neural Network (GNN) architectures used in these methods are
often underexplored; and (2) these methods are often evaluated on only a
limited number of metrics. To fill this gap, we investigate the expressiveness
of GNNs under the context of the molecular graph generation task, by replacing
the underlying GNNs of graph generative models with more expressive GNNs.
Specifically, we analyse the performance of six GNNs in two different
generative frameworks -- autoregressive generation models, such as GCPN and
GraphAF, and one-shot generation models, such as GraphEBM -- on six different
molecular generative objectives on the ZINC-250k dataset. Through our extensive
experiments, we demonstrate that advanced GNNs can indeed improve the
performance of GCPN, GraphAF, and GraphEBM on molecular generation tasks, but
GNN expressiveness is not a necessary condition for a good GNN-based generative
model. Moreover, we show that GCPN and GraphAF with advanced GNNs can achieve
state-of-the-art results across 17 other non-GNN-based graph generative
approaches, such as variational autoencoders and Bayesian optimisation models,
on the proposed molecular generative objectives (DRD2, Median1, Median2), which
are important metrics for de-novo molecular design.
Xiandong Zou, Xiangyu Zhao, Pietro Liò, Yiren Zhao
# Will More Expressive Graph Neural Networks do Better on Generative Tasks?

###### Abstract

Graph generation poses a significant challenge as it involves predicting a complete graph with multiple nodes and edges based on simply a given label. This task also carries fundamental importance to numerous real-world applications, including de-novo drug and molecular design. In recent years, several successful methods have emerged in the field of graph generation. However, these approaches suffer from two significant shortcomings: (1) the underlying Graph Neural Network (GNN) architectures used in these methods are often underexplored; and (2) these methods are often evaluated on only a limited number of metrics. To fill this gap, we investigate the expressiveness of GNNs under the context of the molecular graph generation task, by replacing the underlying GNNs of graph generative models with more expressive GNNs. Specifically, we analyse the performance of six GNNs in two different generative frameworks (GCPN and GraphAF), on six different molecular generative objectives on the ZINC-250k dataset. Through our extensive experiments, we demonstrate that advanced GNNs can indeed improve the performance of GCPN and GraphAF on molecular generation tasks, but GNN expressiveness is not a necessary condition for a good GNN-based generative model. Moreover, we show that GCPN and GraphAF with advanced GNNs can achieve state-of-the-art results across 17 other non-GNN-based graph generative approaches, such as variational autoencoders and Bayesian optimisation models, on the proposed molecular generative objectives (DRD2, Median1, Median2), which are important metrics for de-novo molecular design.

## 1 Introduction

Graph generation has always been viewed as a challenging task as it involves the prediction of a complete graph comprising multiple nodes and edges based on a given label. This task, however, holds paramount importance in a wide array of real-world applications, such as de-novo molecule design [1]. In de-novo molecule generation, the chemical space is discrete by nature, and the entire search space is huge, which is estimated to be between \(10^{23}\) and \(10^{60}\)[2]. The generation of novel and valid molecular graphs with desired physical, chemical and biological property objectives should also be considered, these together with the large search space makes it a difficult task, since _these property objectives are highly complex and non-differentiable_[3].

Recently, there has been significant progress in molecular graph generation with graph neural network (GNN)-based deep generative models, such as Graph Convolutional Policy Network (GCPN) [3] and Flow-based Autoregressive Model (GraphAF) [4], and they both use the Relational Graph Convolutional Network (R-GCN) [5] as their inner graph representation model. Meanwhile, several researchers have focused on improving GNN expressiveness by introducing architectural changes [6; 7; 8], leading to the discovery of diverse forms of GNNs that excel in graph classification and regression
tasks. Naturally, it is worth considering _will these more expressive GNNs help in molecular graph generation and will they perform better than the de-facto network used in GCPN and GraphAF?_

In addition, nowadays, there are many metrics used in de-novo molecule design (_eg. molecule's bioactivity against its corresponding disease targets: DRD2 and JNK3_) [9]. However, GCPN and GraphAF only consider two molecular generative objectives related to drug design: the Penalized logP and the QED. This brings two major drawbacks. First, there is a lack of consideration of a broader set of generation metrics beyond those related to drug design. Secondly, both Penalized logP and the QED are incapable of effectively distinguishing between various generation models, rendering them disabled for such purpose [9].

In this work, we replace R-GCN in GCPN and GraphAF with more expressive GNNs. Then, we evaluate our proposed models: GCPN variants and GraphAF variants on a wide array of molecular generative objectives (e.g. DRD2, Median1, Median2), which are important metrics for de-novo molecular design, to derive more statistically significant results. We make the following contributions:

* Although GNN expressiveness defined by the Weisfeiler-Lehman (1-WL) graph isomorphism test works well on graph classification and regression tasks, it is not a necessary condition for a good GNN-based generative model. We observe empirically that the expressiveness of GNNs does not correlate well with their performance on GNN-based generative models, and GNNs incorporating edge feature extraction can improve GNN-based generative models.
* Although Penalised logP and QED are widely used in evaluating goal-directed graph generative models, they are not effective metrics to differentiate generative models. Other metrics, such as DRD2, Median1 and Median2, can better evaluate the ability of a graph generative model.
* Our findings reveal that while there is no direct correlation between expressiveness and performance in graph generation, substituting the inner GNNs of GCPN and GraphAF with advanced GNNs like GearNet yields better performance (e.g. \(102.51\%\) better in DRD2 and \(48.96\%\) better in Median2). By doing so, these models surpass or reach comparable performance to state-of-the-art non-graph-based generative methods for de-novo molecule generation.

## 2 Related Work

A variety of deep generative models have been proposed for molecular graph generation recently [3; 4; 10; 11; 12; 13; 14; 15; 16; 17]. In this paper, we confine our scope to single-objective approaches for molecular generation. Specifically, our focus centres on the generation of organic molecules that possess a desired scalar metric, encompassing key physical, chemical, and biological properties.

### GNN-based Graph Generative Models

Recently, there has been significant progress in molecular graph generation with GNN-based deep generative models, such as GCPN [3] and GraphAF [4]. Both of them use R-GCN [5], the state-of-the-art GNN at that time, as their inner graph representation model. Nevertheless, in recent years, the landscape has witnessed the emergence of increasingly expressive GNNs such as GATv2 [6], GSN [7] and GearNet [8]. These advanced GNN models have showcased superior performance in various tasks, including graph classification and regression, surpassing the capabilities of R-GCN. Furthermore, the methods GCPN and GraphAF only evaluate their performance in goal-directed molecule generation tasks using two commonly employed metrics in drug design: quantitative estimate of drug-likeness score (QED) and penalised octanol-water partition coefficient (Penalized logP). However, it is worth noting that many advanced graph generative approaches, as discussed in Section 2.2, can achieve state-of-the-art results on benchmarks for QED and Penalized logP [9; 15]. _QED is likely to have a global maximum of 0.948 and even random sampling could reach that value. Penalized logP is unbounded and the relationship between Penalized logP values and molecular structures is fairly simple: adding carbons monotonically increases the estimated Penalized logP value [3; 9; 18]._ This is presented in Figure 3 in Appendix E. Consequently, one could argue that these scores have reached a saturation point, making them less meaningful as evaluation metrics. Only using these two metrics relevant to drug design on goal-directed graph generation tasks to assess the graph generative models is not convincing enough and cannot provide insights for distinguishing different algorithms' de-novo molecule generation ability [9].
### Non-GNN-based Graph Generative Models

There are other non-GNN-based graph generative models. Genetic algorithms (GA) are generation approaches by relying on biologically inspired operators such as mutation, crossover and selection. Bayesian Optimization (BO) [19] is an approach that uses a sequential optimization technique that leverages probabilistic models to search for the optimal solution. Variational Autoencoders (VAEs) [20] is a type of generative model in machine learning that combines elements of both autoencoders and probabilistic latent variable models to learn and generate data by mapping it to a latent space with continuous distributions. Monte-Carlo Tree Search (MCTS) constructs a search tree by iteratively selecting actions, simulating possible outcomes, and propagating the results to inform future decisions, ultimately aiming to find the optimal solution. Hill Climbing (HC) is an iterative optimization algorithm that starts with an arbitrary solution to a problem, and then attempts to find a better solution by making an incremental change to the solution. Reinforcement Learning (RL) learns how intelligent agents take actions in an environment to maximize the cumulative reward by transitioning through different states. Details about non-GNN-based graph generative models we used in Section 4 can be found in Appendix A.

## 3 Method

In this section, we provide the theoretical background for graph generative models. A GNN-based graph generative model consists of a GNN model and a graph generative framework. The GNN learns the hidden representations of a graph, such as the node feature and the graph feature. The goal of the graph generation framework is to generate realistic molecular graph structures based on a given generative objective. The detail of GCPN is presented in Figure 1, and an illustration of GraphAF is displayed in Figure 2 in Appendix B.

### Preliminaries

Let \(G=(V,\mathcal{E})\) define a graph, where \(V\) denotes the set of nodes, and \(\mathcal{E}\subseteq V\times V\) denotes the set of edges. A relational graph can be expressed as \(G=(V,\mathcal{E},\mathcal{R})\) where \(\mathcal{R}\) denotes the set of edge relations or edge types. For example, \((i,j,r)\) means the edge from node \(i\) to node \(j\) with edge type \(r\). A molecular graph can be represented by a tuple of features \((\mathbf{A},\mathbf{H},\mathbf{E},\mathbf{R})\), where

* \(\mathbf{A}\in\mathbb{R}^{|V|\times|V|}\) is the adjacency matrix, with each entry \(a_{ij}\) representing an edge (if any) between nodes \(i\) and \(j\); note that this is different from the conventional \(0,1^{|V|\times|V|}\) adjacency matrix format, since there are different types of bonds (_i.e._, single, double, triple, aromatic).
* \(\mathbf{H}\in\mathbb{R}^{|V|\times d}\) is the feature matrix, \(\mathbf{h}_{i}\in\mathbb{R}^{d}\) is the \(d\)-dimensional features of node \(i\).
* \(\mathbf{E}\in\mathbb{R}^{|\mathcal{E}|\times d_{e}}\) is the edge feature matrix, \(\mathbf{e}_{ij}\in\mathbb{R}^{d_{e}}\) is the \(d_{e}\)-dimensional features of edge \((i,j)\).
* \(\mathbf{R}\in\mathbb{N}^{|\mathcal{E}|}\) is a vector containing the edge types of each edge \((i,j)\in\mathcal{E}\) and \(r_{(i,j)}\in\mathcal{R}\).

Figure 1: An overview of the GCPN model: this is an example of iterative graph generation from an intermediate graph \(G_{t}\) to an intermediate graph \(G_{t+1}\). Part 1 is the illustration of the graph representation learning process based on a GNN. Part 2 is the illustration of the graph generative procedure based on a reinforcement learning (RL) agent. New nodes or edges are marked in red.
The degree matrix \(\mathbf{D}\in\mathbb{R}^{|V|\times|V|}\) of a graph \(G=(V,\mathcal{E})\) is a diagonal matrix with each diagonal entry \(d_{ii}=\text{deg}(v_{i})\), where the \(\text{deg}(v_{i})\) of a vertex counts the number of times an edge terminates at that vertex in an undirected graph.

#### 3.1.1 Subgraph & Isomorphism

A graph \(G^{\prime}=(V^{\prime},\mathcal{E}^{\prime})\) is a _subgraph_ of a graph \(G=(V,\mathcal{E})\) (denoted \(G^{\prime}\subseteq G\)) if and only if \(V^{\prime}\subseteq V\) and \(\mathcal{E}^{\prime}\subseteq\mathcal{E}\). Two graphs \(G=(V,\mathcal{E})\) and \(G^{\prime}=(V^{\prime},\mathcal{E}^{\prime})\) are _isomorphic_ (denoted \(G\cong G^{\prime}\)) if and only if there exists an adjacency-preserving bijective mapping \(f:V\to V^{\prime}\), i.e.,

\[\forall i,j\in V,(i,j)\in\mathcal{E}\Longleftrightarrow(f(i),f(j))\in \mathcal{E}^{\prime} \tag{1}\]

An _automorphism_ of a graph \(G=(V,\mathcal{E})\) is an isomorphism that maps \(G\) onto itself.

### Graph Neural Networks

All the Graph Neural Networks (GNNs) investigated in this paper can be abstracted as Message Passing Neural Networks (MPNNs). A general MPNN operation iteratively updates the node features \(\mathbf{h}_{i}^{(l)}\in\mathbb{R}^{d}\) from layer \(l\) to layer \(l+1\) via propagating messages through neighbouring nodes \(j\in\mathcal{N}_{i}\), which can be formalised by the following equation:

\[\mathbf{h}_{i}^{(l+1)}=\text{UPDATE}\left(\mathbf{h}_{i}^{(l)},\bigoplus_{j \in\mathcal{N}_{i}}\text{MESSAGE}\left(\mathbf{h}_{i}^{(l)},\mathbf{h}_{j}^{(l )},\mathbf{e}_{ij}\right)\right) \tag{2}\]

where MESSAGE and UPDATE are learnable functions, such as Multi-Layer Perceptrons (MLPs), \(\mathcal{N}_{i}=\{j|(i,j)\in\mathcal{E}\}\) is the (1-hop) neighbourhood of node \(i\), and \(\bigoplus\) is a permutation-invariant local neighbourhood aggregation function, such as sum, mean or max. After \(k\) iterations of aggregation, node \(i\)'s representation \(\mathbf{h}_{i}^{(k)}\) can capture the structural information within its \(k\)-hop graph neighbourhood. Then, the graph embedding \(\mathbf{h}_{G}\in\mathbb{R}^{d}\) can be obtained via a READOUT function:

\[\mathbf{h}_{G}=\text{READOUT}_{i\in V}\left(\mathbf{h}_{i}^{(k)}\right) \tag{3}\]

which aggregates the node features to obtain the entire graph's representation \(\mathbf{h}_{G}\).

Ideally, a maximally powerful GNN could distinguish different graph structures by mapping them to different representations in the embedding space. This ability to map any two different graphs to different embeddings, however, implies solving the challenging graph isomorphism problem. That is, we want isomorphic graphs to be mapped to the same representation and non-isomorphic ones to different representations. Thus, the expressiveness of a GNN is defined as the ability to distinguish non-isomorphic graphs, and can be analysed by comparing to the 1-WL graph isomorphism test, which are summarised in Appendix C.

### Graph Generative Frameworks

In this paper, Graph Convolutional Policy Network (GCPN) [3] and Flow-Based Autoregressive Model (GraphAF) [4] are used as graph generative frameworks for molecular graph generation tasks. They formalize the problem of goal-directed graph generation as a sequential decision process through RL, i.e. the decisions are generated from the generation policy network.

In GCPN, the iterative graph generation process is formulated as a general decision process: \(M=(S,\mathcal{A},P,R,\gamma)\), where \(S=\{s_{i}\}\) is the set of states that consists of all possible intermediate and final graphs, \(\mathcal{A}=\{a_{i}\}\) is the set of actions that describe the modification made to the current graph at each time step determined by the Graph Convolutional Policy Network (a group of MLPs predicting a distribution of actions), \(P\) is the Markov transition dynamics that specifies the possible outcomes of carrying out an action, \(p(s_{t+1}|s_{t},\cdots,s_{0},a_{t})=p(s_{t+1}|s_{t},a_{t})\). \(R(s_{t})\) is a reward function that specifies the reward after reaching state \(s_{t}\), and \(\gamma\) is the discount factor. GCPN takes the intermediate graph \(G_{t}\) and the collection of proposed scaffold subgraphs \(C\) as inputs, and outputs the action \(a_{t}\), which predicts a new link to be added.
GraphAF defines an invertible transformation from a base distribution (e.g. multivariate Gaussian) to a molecular graph structure \(G=(V,\mathcal{E})\) as the generation policy network. Starting from an empty graph \(G_{0}\), in each step a new node \(v_{i}\) is generated based on the current sub-graph structure \(G_{i}\), \(i.e.\), \(p(v_{i}|G_{i})\) by the policy network. Next, the edges between this new node and existing nodes are sequentially generated according to the current graph structure, i.e., \(p(\mathcal{E}_{ij}|G_{i},v_{i},\mathcal{E}_{i,1:j-1},\mathcal{E}_{1:j-1,i})\). This process is repeated until all the nodes and edges are generated.

## 4 Evaluation

### Experimental Setup

Dataset.We use the ZINC-250k [21] dataset for both pre-training and fine-tuning proposed models for its pharmaceutical relevance, moderate size, and popularity. All molecules are presented in the kekulised form with hydrogen removed. ZINC-250k contains around 250,000 drug-like molecules with 9 atom types and 3 edge types, and with a maximum graph size of 38 sampled from the ZINC database [21]. The molecules in ZINC are readily synthesisable molecules: _It contains over 120 million purchasable "drug-like" compounds--effectively all organic molecules that are for sale--a quarter of which are available for immediate delivery._[21]. Other datasets, such as QM9 [22], which is a subset of GDB9 [23], contain, at least to some degree, virtual molecules that are likely to be synthesisable but have not been made yet, including many molecules with complex annulated ring systems [15]. In addition, the original works of GCPN, GraphAF and the benchmark [9] we used to compare graph generative models have also been trained on ZINC-250k. Thus, ZINC-250k is well-suited for models to learn representations of drug-like and synthesisable molecules in a de-novo molecule generation task.

Implementation details.We use the open-source platform TorchDrug [24] in dataset preparation and graph generative models training. Advanced GNNs are implemented in PyTorch [25] in the MPNN framework aligned with TorchDrug [24]. The basic architectures of GCPN and GraphAF are implemented in TorchDrug. Note that the original GraphAF cannot allow GNN module to aggregate edge features in the message-passing mechanism, since the intermediate molecular graph generated by the autoregressive flow doesn't contain edge features. Therefore, we propose an improved version of GraphAF considering edge features, named GraphAF+e. To conduct a fairly analogous evaluation on GCPN, we considered GCPN without considering edge features, named GCPN-e. A detailed description of how to incorporate edge features is in Appendix D.

In the experiments, we replace R-GCN in GCPN (both with and without edge features) and GraphAF (both with and without edge features) with six more expressive GNNs: GIN [26], GAT [27], GATv2 [6], PNA [28], GSN [7] and GearNet [8]. We set all GNNs in the experiments to have 3 hidden layers with batch normalisation [29] and ReLU [30] activation applied after each layer. We find little improvement when further adding GCN layers. Each GNN uses a 1-layer MLP to transform the edge features in the graph to a hidden embedding and concatenates with the node features for message passing. In addition, they use summation as the READOUT function for graph representations. For PNA and GSN, the MESSAGE and UPDATE functions are parameterised by single-layer perceptrons. For GSN, we set the graph substructure set to contain cycle graphs of sizes between 3 and 8 (both inclusive), which are some of the most important substructures in molecules [7]. For GAT and GATv2, we use multi-head attention with \(k=3\) after a manual grid search for attention heads.

The GCPN and GraphAF models, with all different GNNs, are pre-trained on the ZINC-250k dataset for 1 epoch, as we do not observe too much performance gain from increasing pre-training epochs. They are then fine-tuned towards the target properties with RL, using the proximal policy optimisation (PPO) algorithm [31]. The models are fine-tuned for 5 epochs for all goal-directed generation tasks with early stopping if all generation results in one batch collapse to singleton molecules. We set the agent update interval for GCPN and GraphAF models to update their RL agents every 3 and 5 batches respectively, as considering the cost of computing resources. During generating process, we allow our RL agents in all models to make 20 resamplings on the intermediate generated graphs if they cannot generate chemically valid molecular graphs. The max graph size is set as 48 empirically. For GCPN, the collection of proposed scaffold subgraphs \(C\) for GCPN are sampled from non-overlapping scaffolds in the ZINC-250k dataset. For GraphAF, we define multivariate Gaussian as our base distribution and use node MLPs and edge MLPs which have two fully-connected layers equipped
with tanh non-linearity to generate the nodes and edges respectively. We notice limited improvements in performance when increasing the number of MLP layers. Adam [32] is used as the optimiser for both pre-training and fine-tuning tasks. The experiments were run with a mix of NVIDIA A100 GPU with 40GB memory and NVIDIA V100 (Volta) GPU with 16GB. The total amount of training time for all GCPN and GraphAF variants under all metrics is around 1200 GPU hours. Reward temperature for each metric, the number of neurons in each hidden layer and the learning rate for fine-tuning) for each model on each task through grid search by Optuna [33] independently. All details are provided in Appendix D.

Baselines.We compare our proposed models based on the original GCPN (both with and without edge features) and GraphAF (both with and without edge features) on six goal-directed molecule generation tasks: Penalised logP [34], QED [35], synthetic accessibility (SA) [36], DRD2 [1], Median1 and Median2. All generation metrics are taken from the Therapeutic Data Commons (TDC) [37]. Due the practicality of de-novo molecule design, we only consider the single generation objective for all generation tasks. Specifically, SA stands for how hard or how easy it is to synthesize a given molecule. Penalized logP is a logP score that also accounts for ring size and SA, while QED is an indicator of drug-likeness. DRD2 is derived from a support vector machine (SVM) classifier with a Gaussian kernel fitting experimental data to predict the bioactivities against their corresponding disease targets. Median1 [37] measures the average score of the molecule's Tanimoto similarity [38] to Camphor and Menthol. Median2 [37] measures the average score of the molecule's Tanimoto similarity to Tadalafil and Sildenafil. Penalized logP has an unbounded range, while QED, DRD2, Median1, Median2 and SA have a range of \([0,1]\) by definition. Higher scores in Penalized logP, QED, DRD2, Median1 and Median2 and a lower score in SA are desired. Note that all scores are calculated from empirical prediction models.

We choose to use DRD2, Median1 and Median2 to evaluate generative models, since they are mature and representative generative metrics [9, 15]. In addition, some other metrics are data-missing or inappropriate and thus cannot reflect the ability of generative models properly. For example, GSK3 cannot evaluate all the generated molecules; multi-property objectives (MPO) measure the geometric means of several scores, which will be 0 if one of the scores is 0; Valsartan Samarts is implemented incorrectly way in TDC: it computes the geometric means of several scores instead of arithmetic means of several scores, which lead to incorrect results in the benchmark [9].

We compare the best GCPN variant and GraphAF variant with eight state-of-the-art approaches for molecule generation [9] on 6 generation metrics: Penalised logP, SA, DRD2, Median1, Median2 and QED. All results of baselines are taken from original papers unless stated.

### Results

#### 4.2.1 Improving GNN-based Graph Generative Methods

De-novo molecule design with more expressive GNNs.As we re-evaluate the Penalised logP and QED scores of the top-3 molecules found by GCPN, we note that our results turn out to be higher than the results reported in the original GCPN paper. We hypothesise this to be due to our more

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{3}{c}{Penalised logP} & \multicolumn{3}{c}{QED} & \multicolumn{3}{c}{SA} \\ \cline{2-10}  & **1st** & **2nd** & **3rd** & **1st** & **2nd** & **3rd** & **1st** & **2nd** & **3rd** \\ \hline ZINC & 4.52 & 4.30 & 4.23 & 0.948 & 0.948 & 0.948 & 1.0 & 1.0 & 1.0 \\ \hline R-GCN (baseline) & 7.98 & 7.85 & 7.80 & 0.948 & 0.947 & 0.946 & - & - & - \\ R-GCN (ours) & 8.67 & 8.67 & 8.67 & 0.948 & 0.948 & 0.948 & **1.0** & **1.0** & **1.0** \\ GIN & **11.19** & **11.19** & **11.19** & 0.942 & 0.926 & 0.923 & 1.2 & 1.2 & 1.2 \\ GAT & 7.70 & 7.47 & 7.44 & 0.926 & 0.911 & 0.911 & **1.0** & **1.0** & **1.0** \\ GATv2 & 8.08 & 7.48 & 7.34 & 0.945 & 0.908 & 0.907 & **1.0** & **1.0** & **1.0** \\ PNA & 8.66 & 8.61 & 8.22 & 0.833 & 0.825 & 0.754 & **1.0** & **1.0** & **1.0** \\ GSN & **11.19** & **11.19** & **11.19** & 0.804 & 0.783 & 0.783 & **1.0** & **1.0** & **1.0** \\ GearNet & **11.19** & **11.19** & **11.19** & **0.948** & **0.948** & **0.948** & **1.0** & **1.0** & **1.0** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of the top-3 Penalised logP, QED and SA scores of the generated molecules by GCPN variants, with the top-3 property scores of molecules in the ZINC dataset for reference.
extensive hyperparameter searching. In Table 1, we explore a set of simpler generation metrics based on the GCPN framework, such as Penalised logP, QED and SA, which have been widely used as objectives in previous work on GNN-based graph generative models.

As summarised in Table 1, after replacing the inner R-GCN in GCPN with more expressive GNNs, we can observe a significant improvement of GCPN in Penalised logP: GCPN with GIN, GearNet and GSN can achieve the saturated \(11.19\) in Penalised logP. Moreover, with GearNet, the performance of the GCPN variants can outperform the original GCPN on all three metrics.

However, we find that _these benchmarks are not suited to differentiate between different models_, since we see many GCPN variants with different GNNs achieve similar and saturated results on those metrics. In addition, these metrics are not representative enough to obtain meaningful conclusions, as discussed in Section 2.1. Therefore, _there is a need of better graph generation objectives_.

De-novo molecule design with better graph generation objectives.From Table 1, we notice that both Penalised logP and SA are saturating on advanced GNNs, making them inappropriate metrics for distinguishing the capability of different GNN models, and we need better de-novo molecule generation metrics. Therefore, we introduce three more representative metrics: DRD2, Median1 and Median2, as described in Section 4.1, and report the top-3 property scores of molecules generated by each model trained on those three metrics in Table 3 (GCPN) and Table 6 (GraphAF) in Appendix B. The results show that GCPN and GraphAF with more expressive GNNs, such as GearNet, can outperform the original GCPN and GraphAF with R-GCN on all generation tasks by a significant margin. Specifically, on metrics such as DRD2 and Meidan2, GCPN and GraphAF with more expressive GNNs can vastly improve the original performance. This observation further indicates that by combining with more expressive GNNs, GCPN and GraphAF can successfully capture the distribution of desired molecules. Therefore, we suggest that DRD2, Median1, Median2 and QED are better graph generation metrics for differentiating different GNNs.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{3}{c}{DRD2} & \multicolumn{3}{c}{Median1} & \multicolumn{3}{c}{Median2} \\ \cline{2-10}  & **1st** & **2nd** & **3rd** & **1st** & **2nd** & **3rd** & **1st** & **2nd** & **3rd** \\ \hline ZINC & 0.9872 & 0.9815 & 0.9773 & 0.3243 & 0.3096 & 0.3096 & 0.2913 & 0.2765 & 0.2749 \\ \hline R-GCN & 0.8315 & 0.7576 & 0.7551 & 0.3152 & 0.3152 & 0.3001 & 0.1932 & 0.1613 & 0.1592 \\ GIN & 0.2791 & 0.1980 & 0.1752 & 0.3152 & 0.3152 & 0.3152 & 0.1140 & 0.1113 & 0.1069 \\ GAT & 0.1980 & 0.1580 & 0.1580 & 0.3243 & 0.3243 & 0.3175 & 0.1196 & 0.1100 & 0.1098 \\ GATv2 & 0.2992 & 0.2992 & 0.2992 & 0.3281 & 0.3281 & 0.3281 & 0.1042 & 0.1009 & 0.0911 \\ PNA & 0.4448 & 0.4448 & 0.4448 & 0.3243 & 0.3202 & 0.3175 & 0.0911 & 0.0764 & 0.0716 \\ GSN & 0.4790 & 0.4790 & 0.4448 & 0.3175 & 0.3175 & 0.3015 & 0.0982 & 0.0978 & 0.0897 \\ GearNet & **0.9990** & **0.9705** & **0.9574** & **0.3482** & **0.3482** & **0.3482** & **0.2084** & **0.2043** & **0.2037** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of the top-3 DRD2, Median1 and Median2 scores of the generated molecules by GCPN–e variants, with the top-3 property scores of molecules in the ZINC dataset for reference.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{3}{c}{DRD2} & \multicolumn{3}{c}{Median1} & \multicolumn{3}{c}{Median2} \\ \cline{2-10}  & **1st** & **2nd** & **3rd** & **1st** & **2nd** & **3rd** & **1st** & **2nd** & **3rd** \\ \hline ZINC & 0.9872 & 0.9815 & 0.9773 & 0.3243 & 0.3096 & 0.3096 & 0.2913 & 0.2765 & 0.2749 \\ \hline R-GCN & 0.4790 & 0.4790 & 0.4790 & 0.3367 & 0.3367 & 0.3242 & 0.1921 & 0.1891 & 0.1891 \\ GIN & 0.3460 & 0.3094 & 0.3094 & 0.3243 & 0.3243 & 0.3235 & 0.1770 & 0.1766 & 0.1730 \\ GAT & 0.4946 & 0.4946 & 0.4946 & 0.3367 & 0.3367 & 0.3328 & 0.1648 & 0.1640 & 0.1637 \\ GATv2 & 0.5101 & 0.4946 & 0.4946 & 0.3367 & 0.3367 & 0.3331 & 0.1759 & 0.1720 & 0.1697 \\ PNA & 0.5828 & 0.4448 & 0.4448 & 0.3472 & 0.3254 & 0.3254 & 0.1629 & 0.1619 & 0.1605 \\ GSN & 0.5363 & 0.4946 & 0.4946 & 0.3243 & 0.3243 & 0.3235 & 0.1770 & 0.1766 & 0.1730 \\ GearNet & **0.9696** & **0.9684** & **0.9404** & **0.3367** & **0.3367** & **0.3367** & **0.2862** & **0.2794** & **0.2794** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of the top-3 DRD2, Median1 and Median2 scores of the generated molecules by GCPN variants, with the top-3 property scores of molecules in the ZINC dataset for reference.
Comparison with non-GNN-based graph generative methods.We report the top-1 DRD2, Median1, Median2 and QED scores found by all the GNN-based and non-GNN-based graph generative models in Table 4. As displayed in Table 4, original GCPN and GraphAF are not competitive among graph generative models on all the goal-directed molecule generation tasks. However, after modifying their inner GNNs with more advanced GNNs, such as GearNet, they can outperform or match state-of-the-art results across other generative approaches on the de-novo molecule generation task. Specifically, Replacing R-GCN with GearNet can achieve an average of \(50.49\%\) improvement on GCPN and an average of \(12.51\%\) improvement on GraphAF in de-novo molecule design, with the proposed generation metrics.

Visualisations of the generated molecules with desired generative metrics by GCPN and GraphAF variants are presented in Figure 4 in Appendix E. In addition, we only report eight selected graph generative methods in the benchmark [9] in Table 4, and GCPN with GearNet can achieve comparable results across 17 other non-GNN-based graph generative methods on the proposed metrics, which are fully reported in Table 8 in Appendix B.

#### 4.2.2 Correlation Between GNN Expressiveness and Graph Generation

In the pre-training phase, the GNNs are used to predict all node types and edge types in all masked graphs in the training data. We report the graph classification results for all GNNs in Table 5. In the GCPN framework, we can see GSN perform the best with \(93.08\%\) accuracy on the graph classification task. In the GraphAF framework, we can see GearNet perform the best with the lowest edge and node average negative log likelihood: \(-0.9275\) and \(-2.8981\) respectively. Both GSN and GearNet are more expressive GNNs than R-GCN according to the 1-WL test demonstrated in Appendix C. It is not surprising that expressive GNNs can outperform other GNNs on the graph classification task, since the expressivity of a GNN is defined as the ability to distinguish non-isomorphic graphs.

However, by comparing Table 5 with Tables 2, 3, 6, 7, it is worth noting that _more expressive GNNs cannot ensure better performance of GNN-based graph generative models in molecular generation

\begin{table}
\begin{tabular}{l l l l l} \hline \hline Model & DRD2 & Median1 & Median2 & QED \\ \hline GCPN (RGCN) & 0.479 & 0.337 & 0.192 & 0.948 \\ GCPN (GearNet) & 0.970 (+**102.51**\(\%\)) & 0.337 (+**0.00**\(\%\)) & 0.286 (+**48.96**\(\%\)) & 0.948 (+**0.00**\(\%\)) \\ \hline GraphAF (RGCN) & 0.928 & 0.281 & 0.143 & 0.946 \\ GraphAF (GearNet) & 0.987 (+**6.36**\(\%\)) & 0.290 (+**3.20**\(\%\)) & 0.183 (+**27.97**\(\%\)) & 0.947 (+**0.11**\(\%\)) \\ \hline LSTM HC (SMILES) [15] & 0.999 & 0.388 & 0.339 & 0.948 \\ DoG-Gen [16] & 0.999 & 0.322 & 0.297 & 0.948 \\ GP BO [12] & 0.999 & 0.345 & 0.337 & 0.947 \\ SynNet [10] & 0.999 & 0.244 & 0.259 & 0.948 \\ GA+D (11) & 0.836 & 0.219 & 0.161 & 0.945 \\ VAE BO (SMILES) [13] & 0.940 & 0.231 & 0.206 & 0.947 \\ Graph MCTS [14] & 0.586 & 0.242 & 0.148 & 0.928 \\ MolDQN [17] & 0.049 & 0.188 & 0.108 & 0.871 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of the top-1 DRD2, Median1, Median2 and QED scores with the selected non-GNN-based generative models. The full table can be found in Table 8 in Appendix B.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{GCPN} & \multicolumn{2}{c}{GraphAF} \\ \cline{2-5}  & \(NLL_{all}\) & \(Acc\) & \(NLL_{e}\) & \(NLL_{n}\) \\ \hline RGCN & 2.1427 \(\pm\) 0.1326 & 0.8656 \(\pm\) 0.0088 & -1.0674 \(\pm\) 0.0204 & -3.3312 \(\pm\) 0.1557 \\ GIN & 2.1500 \(\pm\) 0.3158 & 0.8680 \(\pm\) 0.0220 & -1.1691 \(\pm\) 0.1167 & -3.6067 \(\pm\) 0.2589 \\ GAT & 2.8705 \(\pm\) 0.2354 & 0.7957 \(\pm\) 0.0211 & -1.2737 \(\pm\) 0.0340 & -3.1541 \(\pm\) 0.0844 \\ GATv2 & 2.8285 \(\pm\) 0.2878 & 0.8013 \(\pm\) 0.0244 & -1.2426 \(\pm\) 0.0370 & -3.1325 \(\pm\) 0.0681 \\ PNA & 2.1172 \(\pm\) 0.0036 & 0.8716 \(\pm\) 0.0003 & -1.1043 \(\pm\) 0.1148 & -3.2840 \(\pm\) 0.2154 \\ GSN & **1.4219 \(\pm\) 0.0026** & **0.9308 \(\pm\) 0.0002** & -1.1319 \(\pm\) 0.0678 & -3.1280 \(\pm\) 0.0566 \\ GearNet & 2.1265 \(\pm\) 0.0017 & 0.8677 \(\pm\) 0.0003 & **-0.9275 \(\pm\) 0.0048** & **-2.8981 \(\pm\) 0.0398** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Graph classification metrics among GCPN and GraphAF variants. \(NLL_{all}\) means the average negative log likelihood loss for all node and edge classification tasks. \(Acc\) means the average accuracy for all node and edge classification tasks. \(NLL_{e}\) and \(NLL_{n}\) mean the average edge negative log likelihood evaluated on classification tasks for edge and node respectively.
tasks_. For example, PNA and GSN perform better than R-GCN on graph classification tasks (Table 5), but GCPN with PNA or GSN cannot surpass the original GCPN on all generation metrics (e.g. Median2 in Table 3). Therefore, we conclude that the graph generation task is a different task from graph classification and requires other abilities of GNNs, such as edge feature extraction.

De-novo molecule design with GNNs incorporating edge features.We investigate the performance of GCPN and GraphAF using GNNs with and without edge features. It is worth mentioning that the original GCPN considers edge features but GraphAF does not. The results of the GCPN without edge features (GCPN-e) are summarised in Table 2, and those of the original GCPN are in Table 3. Both sets of results demonstrate that including edge features can significantly improve the top-3 scores on all three metrics for most GNNs. For instance, with the help of edge feature extraction, GCPN with GIN, GAT, GATv2, PNA and GSN can increase by \(24.0\%\), \(149.8\%\), \(70.5\%\), \(31.0\%\) and \(12.0\%\) on the top-1 score on the metric DRD2, compared with GCPN-e.

The original GraphAF comes without edge feature extraction and we thus improved it by incorporating the edge features. Table 6 (the orginal GraphAF) and Table 7 (GraphAF+e) in Appendix B summarised the results of both implementations accordingly. The results align with the GCPN case, where GraphAF+e significantly improves the top-3 scores on all three metrics for most GNNs than the original GraphAF. More specifically, with the help of edge feature extraction, GraphAF+e with GIN, GAT, GATv2, PNA and GSN can be improved by \(43.5\%\), \(60.0\%\), \(20.0\%\), \(230.4\%\) and \(312.4\%\) on the top-1 score on the metric DRD2.

The results above indicate _the importance of aggregating edge features for GNN-based generative models_. By harnessing the power to extract knowledge from edge information, the potential for generating more refined and accurate graphs is enhanced. We also notice that, when the GNNs used are either R-GCN or GearNet, we find GCPN and GraphAF perform well with and without considering edge information, so we hypothesise the reason is that they absorb the number of edge relations as prior information in the model.

In summary, we conclude the following results:

1. More expressive GNN can lead to better results in the graph classification task. However, GNN expressiveness defined by the 1-WL test is not a necessary condition for a good GNN-based generative model. Generation tasks require other abilities of GNNs, such as edge feature extraction and edge relation detection.
2. Although Penalised logP and QED are widely used for generative metrics in evaluating goal-directed graph generative models, they are not effective metrics to differentiate different generative models. Other metrics, such as DRD2, Median1 and Median2, can better evaluate the ability of a graph generative model. Under those metrics, we can see the performance of GCPN and GraphAF can be enhanced by using more robust GNNs.
3. After applying advanced GNN to current GNN-based graph generative methods, such as GCPN and GraphAF, they can outperform or match state-of-the-art results across 17 other generative approaches in the de-novo molecule generation task.

## 5 Limitation and Conclusion

Due to computation cost, we acknowledge several limitations of the current study: we cannot exhaustively explore every method, such as other Relational GNNs, and thoroughly tune every hyperparameter; we cannot evaluate all generative models on other complicated datasets besides Zinc 250k, such as ChEMBL [39], and other generation metrics [9]. However, our efforts have still provided valuable insights into investigating the expressiveness of GNN on the graph generation task, because of our focus on many different generative models and diverse generation objectives.

After exploring (1) unexplored underlying GNNs and (2) non-trivial generative objectives, we would like to conclude that expressiveness is not a necessary condition for a good GNN-based generative model. By evaluating GCPN variants and GraphAF variants on effective metrics, we demonstrate that GCPN and GraphAF can be improved by using more robust GNNs (e.g. strong edge features extraction and edge relation detection). With more robust GNNs, GNN-based graph generative models can outperform or match state-of-the-art results across 17 other generative approaches on de-novo molecule design tasks [9]. In the future, we plan to explore the necessary conditions for GNN to enhance the performance of GNN-based graph generative models.