An Exploration of Conditioning Methods in Graph Neural Networks
The flexibility and effectiveness of message passing based graph neural
networks (GNNs) induced considerable advances in deep learning on
graph-structured data. In such approaches, GNNs recursively update node
representations based on their neighbors and they gain expressivity through the
use of node and edge attribute vectors. E.g., in computational tasks such as
physics and chemistry usage of edge attributes such as relative position or
distance proved to be essential. In this work, we address not what kind of
attributes to use, but how to condition on this information to improve model
performance. We consider three types of conditioning; weak, strong, and pure,
which respectively relate to concatenation-based conditioning, gating, and
transformations that are causally dependent on the attributes. This
categorization provides a unifying viewpoint on different classes of GNNs, from
separable convolutions to various forms of message passing networks. We provide
an empirical study on the effect of conditioning methods in several tasks in
computational chemistry.
Yeskendir Koishekenov, Erik J. Bekkers
# An Exploration of Conditioning Methods in Graph Neural Networks

###### Abstract

The flexibility and effectiveness of message passing based graph neural networks (GNNs) induced considerable advances in deep learning on graph-structured data. In such approaches, GNNs recursively update node representations based on their neighbors and they gain expressivity through the use of node and edge attribute vectors. E.g., in computational tasks such as physics and chemistry usage of edge attributes such as relative position or distance proved to be essential. In this work, we address not what kind of attributes to use, but _how to condition_ on this information to improve model performance. We consider three types of conditioning; weak, strong, and pure, which respectively relate to concatenation-based conditioning, gating, and transformations that are causally dependent on the attributes. This categorization provides a unifying viewpoint on different classes of GNNs, from separable convolutions to various forms of message passing networks. We provide an empirical study on the effect of conditioning methods in several tasks in computational chemistry.

## 1 Introduction

Graph neural networks (GNNs) are a family of neural networks that can learn from graph-structured data. Starting with the success of GCN (Kipf and Welling, 2016) in achieving state-of-the-art performance on semi-supervised classification, several variants of GNNs have been developed for this task, including Graph-SAGE (Hamilton et al., 2017), GAT(Velickovic et al., 2017), GATv2 (Brody et al., 2021), EGNN (Satorras et al., 2021) to name a few most recent ones.

Most of the models based on the message-passing framework utilize conditional linear layers. We define _"conditioning"_ as using additional information together with feature vectors from its neighbors' nodes. For example, EGNN (Satorras et al., 2021) conditions message vectors on the distance between two nodes or DimeNet (Gasteiger et al., 2020) additionally utilizes angle information. Many neural network models use conditioning in their layers without exploring their different variants. Therefore, improving upon the type of conditioning could still improve most state-of-the-art models. We believe that this is the first work that analyzes different conditioning methods in GNNs.

In this paper, we categorize three conditioning methods: weak, strong, and pure. They differ in the level of dependency on a given quantity, such as edge attributes, and differ in complexity. Message passing neural network (MPNN) using _weak conditioning_ method _concatenates_ attributes with node features. In this scenario, linear layers effectively gain an attribute-dependent bias, which we consider a weak type of conditioning as this does not guarantee that the attribute is actually utilized, i.e., it could be ignored. On the other hand, we have _pure conditioning_ method which forces the model to always use the attributes by letting them causally parametrize transformation matrices. However, from a practical perspective pure conditioning is computationally expensive and it can be simplified to a _strong conditioning_ method, which corresponds to an attribute-dependent _gating_ of the outputs of linear layers. We experiment with these three conditioning methods in variations of the EGNN model (Satorras et al., 2021) on computational chemistry datasets QM9 (Ramakrishnan et al., 2014) and MD17 (Chmiela et al., 2017) and show the advantage of strong conditioning over weak conditioning in performance, and over pure conditioning in training time.

The main contributions of this paper are:
1. A unifying analysis of geometric message passing by formulating conditional transformations in terms of various forms of conditional linear layers.
2. An intuitive exposition of different conditioning methods in the context of convolutional message passing.
3. Empirical studies that show the benefit of _strong conditioning_ methods, as well as the benefit of _deep conditioning_ in multi-layer perceptron-based message functions.

In this work, we address not what kind of attributes to use, but _how to condition_ on this information to improve model performance. As such, we focus on an intuitive analysis, and in the experimental section, we do not intend to achieve the best performance but focus on ablation studies in order to obtain general take-home messages.

## 2 Preliminaries

In this section, we introduce the relevant materials on graph neural networks on top of which we will later complement our analysis and definitions of conditioning methods.

### Graph Neural Network

In this work, we consider the graph regression task as an example. A graph is represented by \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) with nodes \(v_{i}\in\mathcal{V}\) and edges \(e_{ij}\in\mathcal{E}\). A typical message passing layer (Gilmer et al., 2017) is defined as:

\[\mathbf{m}_{ij}=\phi_{e}(\mathbf{h}_{i}^{l},\mathbf{h}_{j}^{l},\mathbf{a}_{ij}) \tag{1}\]

\[\mathbf{m}_{i}=\sum_{j\in N(i)}\mathbf{m}_{ij} \tag{2}\]

\[\mathbf{h}_{i}^{l+1}=\phi_{h}(\mathbf{h}_{i}^{l},\mathbf{m}_{i}) \tag{3}\]

Where \(\mathbf{h}_{i}^{l}\) is the embedding of node \(v_{i}\) at layer \(l\), \(\mathbf{a}_{ij}\) is the edge attribute of nodes \(v_{i}\) and \(v_{j}\), and \(N(i)\) is the set of neighbors of the node \(v_{i}\). Finally, \(\phi_{e}\) and \(\phi_{h}\) are the message (edge) and update (node) functions respectively which are commonly parametrized by Multilayer Perceptrons (MLPs).

### Geometric Graph Neural Networks

When the graphs have an embedding in Euclidean space, i.e., each node \(v_{i}\) has an associated position \(\mathbf{x}_{i}\in\mathbb{R}^{n}\), we want to leverage this geometric information whilst preserving stability/invariance to rigid-body transformations. That is, many tasks are invariant to Euclidean distance preserving transformations in \(E(n)\). E.g., the prediction of energy of a system of atoms is invariant to its global position and orientation in space. Several works have shown how to build equivariant message passing based graph neural networks for such geometric graphs.

Central in those works is the conditioning of the message and update function on invariant geometric attributes, such as the pairwise distance \(\mathbf{a}_{ij}=\|\mathbf{x}_{j}-\mathbf{x}_{i}\|\), as popularized in (Satorras et al., 2021), or covariant spherical/circular harmonic embeddings of relative position \(\mathbf{a}_{ij}=Y(\mathbf{x}_{j}-\mathbf{x}_{i})\) as is common steerable group convolution-based graph NNs (Brandstetter et al., 2021). Here we consider attributes that transform predictably via representations of \(E(n)\) as _covariants_, and those that remain invariant as _invariants_. Such covariants typically contain more (directional) information but require specialized operations such as the Clebsch-Gordan tensor product (Thomas et al., 2018; Anderson et al., 2019) in order to preserve equivariance of the graph NNs. Satorras et al. (2021) show that with a simple recipe based on invariant attributes, one can often obtain equally powerful graph NNs. As such, we focus this paper on the use of \(\mathbf{a}_{ij}=\|\mathbf{x}_{j}-\mathbf{x}_{i}\|\) as a sufficiently expressive attribute, and model the message and update functions \(\phi_{e}\) and \(\phi_{h}\) as regular MLPs.

Our objective then is to understand what is the most effective way of utilizing attributes in geometric graph NNs. To make this notion of conditioning explicit, we will denote the message and update function as

\[\phi_{e}(\mathbf{h}_{i}^{l},\mathbf{h}_{j}^{l}\,|\,\mathbf{a}_{ij})\quad\text{ and}\quad\phi_{h}(\mathbf{h}_{i}^{l}\,|\,\mathbf{a}_{i})\,,\]

where we note that, although uncommon, it is possible to define invariant or covariant geometric node attributes \(\mathbf{a}_{i}\)(Brandstetter et al., 2021).
## 3 Analysis of Conditioning Methods

In the subsequent, we unify several conditioning methods used in literature through the notion of conditional linear layers, and by discussing them in relation to the prevalent convolution layer and its variations. As a starting point, we use the fact that convolution is a simple form of message passing with linear message functions conditioned on relative position, i.e.,

\[\mathbf{m}_{ij}=\phi_{e}(\mathbf{f}_{j}\mid\mathbf{x}_{j}-\mathbf{x}_{i})= \mathbf{W}(\mathbf{x}_{j}-\mathbf{x}_{i})\mathbf{f}_{j}\:, \tag{4}\]

and the following update function typically is the application of a point-wise activation function \(\sigma\), i.e., \(\phi_{h}(\mathbf{f}_{i})=\sigma(\mathbf{f}_{i})\), possibly with a skip connection as in ResNets (He et al., 2016). In general, geometric graph NNs do not just _linearly_ transform node features, but generally do this _non-linearly_ via message/update functions parametrized by MLPs, leading to a notion of non-linear convolutions when the attributes are invariant/covariant quantities (Brandstetter et al., 2021).

Importantly, these MLPs themselves are parametrized by linear layers, intertwined with non-linear activation functions, and nothing prevents from conditioning each of these linear layers on the attributes. It is the purpose of this paper to categorize several options for conditioning and understand what effect this has on performance.

### Conditional linear layers

We propose the following modifications of the linear layer, as to make them conditional on attributes

\[\mathbf{W}_{\mathbf{a}}\mathbf{h} :=\mathbf{W}\:\mathbf{h}\] _no_ (5) \[\mathbf{W}_{\mathbf{a}}\mathbf{h} :=\mathbf{W}\left(\mathbf{h}\oplus\mathbf{a}\right)\] _weak_ (6) \[\mathbf{W}_{\mathbf{a}}\mathbf{h} :=(\mathbf{W}^{a}\mathbf{a})\odot(\mathbf{W}^{h}\:\mathbf{h})\] _strong_ (7) \[\mathbf{W}_{\mathbf{a}}\mathbf{h} :=\mathbf{W}(\mathbf{a})\mathbf{h}\] _pure_ (8)

where to keep the similarity to the common notation for linear transformations, we use the notation \(\mathbf{W}_{\mathbf{a}}\) to denote that the linear transformation is conditioned on \(\mathbf{a}\). We further use to notation \(\mathbf{h}\oplus\mathbf{a}\) to denote the concatenation of vectors \(\mathbf{h}\) and \(\mathbf{a}\), and use \(\odot\) to denote element-wise multiplication. To distinguish matrices applied to different attributes, we use italic labels, e.g. \(\mathbf{W}^{h}\).

We stress the hierarchy in terms of the dependence of the transformation on \(\mathbf{a}\). The most direct dependence is in the _pure_ method, in which the transformation is causally parametrized by \(\mathbf{a}\), followed by _strong_ conditioning in which a standard unconditional transformation \(\mathbf{W}^{h}\:\mathbf{h}\) is _gated_ by a vector \(\mathbf{W}^{a}\:\mathbf{a}\). Both the pure and strong methods are by construction forced to utilize the attribute, as, in particular in the strong case, the transformation would not exists without attribute \(\mathbf{a}\). We refer to equation 6 as _weak_ conditioning, as in principle the transformation would still exist in the absence of the attribute, and moreover, if the dimensionality of \(\mathbf{h}\) is much larger than that of \(\mathbf{a}\), the transformation of the attribute only contributes to a small extent to the output of this layer. We hypothesize that this hierarchy correlates with performance and experimentally test this hypothesis in Sec. 5. What follows is a brief analysis of these types of conditioning.

### Pure conditioning corresponds to bi-linear layers

As common for implementations of convolutions, one typically expands the convolution kernel \(\mathbf{W}(\mathbf{x}_{j}-\mathbf{x}_{i})\in\mathbb{R}^{d_{o}\times d_{i}}\), i.e., a transformation matrix with elements \(W_{oi}(\mathbf{x}_{j}-\mathbf{x}_{i})\) that depends on relative position, in a basis \(\{\phi_{b}:\mathbb{R}^{n}\rightarrow\mathbb{R}\}_{b=1}^{d_{b}}\) via

\[W_{oi}(\mathbf{x}_{j}-\mathbf{x}_{i})=\sum_{b=1}^{d_{b}}W_{boi}\:\phi_{b}( \mathbf{x}_{j}-\mathbf{x}_{i})\:. \tag{9}\]

The basis could be the usual \(3\times 3\) pixel basis, or it could be a continuous basis for when the continuous structure of the data is to be respected, such as circular or spherical harmonics (Worrall et al., 2017; Weiler and Cesa, 2019; Thomas et al., 2018; Anderson et al., 2019), B-splines (Bekkers, 2020; Fey et al., 2018), or hermite polynomials (Sosnovik et al., 2020). In recent works on the parametrization of continuous functions as Neural Fields (Xie et al., 2022) or in transformer-based methods (Vaswani et al., 2017), such basis functions are often referred to as _coordinate embeddings_ or _position encodings_.
An important observation is that, given such a parametrization through basis functions, the pure conditioning layer corresponds to a bi-linear layer

\[h_{o}^{l+1}=\sum_{b}\sum_{i}\phi_{b}(\mathbf{a})\,W_{boi}\,h_{i}^{l}\quad \Longleftrightarrow\quad\mathbf{h}^{l+1}=\boldsymbol{\phi}(\mathbf{a})^{bilinear} \mathbf{W}^{l}\,\mathbf{h}^{l}\,, \tag{10}\]

where we use \(i\) to index the input feature vector, \(o\) the output feature vector, and \(b\) the basis functions.

Such bilinear layers are often implicitly used in convolutional architectures, where the expansion in the basis function is typically hard-coded or pre-computed. On continuous data, such as point cloud methods, the bilinear layer is ubiquitous. Notably, in the context of steerable group equivariant convolutions, the transformations happen via bilinear operators called the Clebsch-Gordan tensor product, in combination with spherical harmonic embeddings of relative position (Brandstetter et al., 2021). Outside of the (group) convolution literature, bilinear layers are often used to explicitly model conditioning on geometric attributes, for which a relevant example to our current work is DimeNet (Gasteiger et al., 2020). DimeNet uses an advanced message passing framework in which messages and updates are conditioned on geometric quantities (embedded as spherical harmonics and radial basis functions) via combinations of weak, strong, and pure conditioning.

### Strong conditioning corresponds to depth-wise separable convolutions

In later works, DimeNet was improved by DimeNet++ (Gasteiger et al., 2020), both in performance and speed, by replacing the compute-heavy bilinear layers (pure conditioning) with the more efficient gating type conditioning (strong conditioning). The computational bottleneck of pure conditioning motives the use of strong conditioning, a route that has proven successful in convolutional architectures computer vision as well, via the use of so-called depth-wise separable convolutions (Sifre and Mallat, 2014; Chollet, 2017).

Chollet (2017) shows huge efficiency gains, both in terms of computing and performance, when factorizing convolution kernels in two parts. One part does the channel mixing, which does not depend on the relative position, while another part depends on the relative position which scales/gates the output. That is if the kernel is given by

\[W_{oi}(\mathbf{x}_{j}-\mathbf{x}_{i})=W_{o}^{a}(\mathbf{x}_{j}-\mathbf{x}_{i} )W_{oi}^{h}\,, \tag{11}\]

the convolution boils down to message passing with conditional linear layers of the _strong_ type, as we can write

\[\mathbf{h}^{l+1}=\left(W^{a}(\mathbf{x}_{j}-\mathbf{x}_{i})\right)\odot( \mathbf{W}^{h}\mathbf{h}^{l})\,, \tag{12}\]

where we can define \(\mathbf{W}^{a}(\mathbf{x}_{j}-\mathbf{x}_{i})=\mathbf{W}^{a}\mathbf{a}_{ij}\) as the linear transformation of a coordinate embedding \(\mathbf{a}_{ij}=\boldsymbol{\phi}(\mathbf{x}_{j}-\mathbf{x}_{i})\) if we want the connection to equation 7 explicit. The fact that the transformation overall is linear and that it splits into a part that does and a part that doesn't depend on pair-wise attributes allows for very efficient implementations that first perform a group-wise convolution, followed by channel mixing. This principle is at the core of the recently popularized ConvNeXt architecture (Liu et al., 2022) and plays an essential role in equivariant graph NNs on (molecular) point clouds in order to be able to scale up (Thomas et al., 2018). Separability recently also proved necessary in order for equivariant convolutional NNs to scale up to large groups, such as the scale-rotation-translation group (Knigge et al., 2022). In the context of conditional NN to parametrize _neural fields_(Xie et al., 2022), strong conditioning commonly appears in the form of so-called _film layers_(Perez et al., 2018).

### Weak conditioning corresponds to linear layers with a conditional bias

Weak conditioning (Eq. 6) corresponds to a standard linear layer with an adaptive bias:

\[\mathbf{W}_{\mathbf{a}}\,\mathbf{h}=\boldsymbol{W}\left(\mathbf{h}\oplus \mathbf{a}\right)=\underbrace{\boldsymbol{W}^{{}^{\prime}}\mathbf{h}}_{\text{ no condition}}+\underbrace{\boldsymbol{W}^{{}^{\prime\prime}}\mathbf{g}}_{\text{ conditional bias}}=\mathbf{W}^{{}^{\prime}}\,\mathbf{h}+\mathbf{b}(\mathbf{a})\,, \tag{13}\]

where \(\mathbf{W}^{{}^{\prime}}\) are the first \(d_{h}\) rows of \(\mathbf{W}\) that are applied to the \(\mathbf{h}\in\mathbb{R}^{d_{h}}\) of the concatenated vector, and \(\mathbf{W}^{{}^{\prime\prime}}\) are the last \(d_{a}\) rows of \(\mathbf{W}\) that are applied to \(\mathbf{a}\in\mathbb{R}^{d_{a}}\). This simple form of conditioning is most used in literature to condition any type of NN on the conditioning vector \(\mathbf{a}\), from message passing methods (Gilmer et al., 2017; Satorras et al., 2021), to conditional neural fields, e.g. as a simple but effective form of modulation in sirens (Dupont et al., 2022), to conditional variational auto-encoders (Sohn et al., 2015).
### Conditional MLPs

With the various forms of conditional linear layers given in equations 6, 7, and 8 we can build conditional MLPs, by simply replacing the usual linear layers with the conditional ones1. Such MLPs could e.g. be denoted with \(\operatorname{MLP}(\mathbf{h}\,|\,\mathbf{a})\). Usually, only the first layer of such a conditional MLP is conditional and usually of the weak type, as with EGNN (Satorras et al., 2021). In the experiments we show that this simple choice is sub-optimal in the context of geometric message passing a la EGNN, and show that improvements can be made by either by conditioning more layers, or switching to strong conditioning.

Footnote 1: Code is available at [https://github.com/YeskendirK/conditioning-GNNs](https://github.com/YeskendirK/conditioning-GNNs)

## 4 Related work: Geometric Message Passing for Computational Chemistry

In the previous section, we discussed several options for conditioning message/update functions for us in message passing graph neural networks, as well as their use in other fields of deep learning. In our experiments we benchmark the three conditioning methods (equations 6, 7, 8) in the context geometric message passing for computational chemistry. Recent works in this category, and the types of conditioning used in those works, are as follows.

EGNN (Satorras et al., 2021) is a message passing neural network (MPNN) that uses a _weak_ conditioning method to utilize the distance between nodes.

DimeNet (Gasteiger et al., 2020) is the type of MPNN, where message embeddings interact based on the distance between atoms and the angle directions. _Pure_ conditioning is adapted to utilize angle directions in message update and aggregation. Tensor Field Network (Thomas et al., 2018) is a neural network for 3D point clouds. Each point in TFN is associated with a vector in a representation of \(SO(3)\). To condition one representation on another, a tensor product of representations is used which in its general form corresponds to _pure_ conditioning. However, many works of the steerable message passing kind, including TFN and NequIP (Batzner et al., 2022), implement a _separable_ variation (_strong_ conditioning) for the sake of computational efficiency. The exception is steerable EGNN (Brandstetter et al., 2021), which uses steerable MLPs with _pure_ conditioning in each layer, i.e., not only the first layer of the message function as in EGNN.

DimeNet++ (Klicpera et al., 2020) is an extension of DimeNet which changed conditioning method from pure to strong to increase efficiency. Klicpera et al. (2020) showed that changing the conditioning method decreased the runtime time of the original DimeNet by a factor of 5. SchNet (Schutt et al., 2017) is another example of MPNN that utilizes atom locations using a strong conditioning method. PaiNN (Schutt et al., 2021) further extends SchNet by projecting the interatomic distances via radial basis functions and iteratively updating the vectors along with the scalar features but also using a strong conditioning method.

## 5 Experiments

In this section, we design experiments to evaluate the effectiveness of three conditioning methods shown in equation 6, 7, and 8 on two real-world datasets: QM9 (Ramakrishnan et al., 2014) and MD17 (Chmiela et al., 2017). We want to demonstrate the effect of the choice of the conditioning method and present the results of some other models on benchmarks to provide context. In our experiments, we report Mean Absolute Error (MAE) between model predictions and ground truth.

Implementation detailsAs a baseline architecture, we use the EGNN model that consists of 7 layers, 128 features per hidden layer, and a 2-layer message (edge) function \(\phi_{e}=\operatorname{MLP}(\mathbf{h}_{i}\oplus\mathbf{h}_{j}\,|\,\|\mathbf{ x}_{j}-\mathbf{x}_{i}\|^{2})\) with only weak conditioning via equation 6 in the first layer. We will test modifications of this model by changing the weak conditioning, to strong or pure, and explore the effect of conditioning multiple layers in the edge function MLP.
We found it useful to pass geometric distances through two-layer MLP, as a form of vectorization/coordinate embedding of the pairwise distance, for the QM9 dataset and Random Fourier Feature layer (Rahimi & Recht, 2007) for the MD17 dataset.

We use the same values for hyperparameters from the EGNN paper (Satorras et al., 2021) for both datasets: we trained each model on QM9/MD17 dataset for a total of 1.000/500 epochs, used Adam optimizer, batch size 96 (64 for pure-EGNN), weight decay 1e-16, and cosine decay for the learning rate starting at 5e-4.

Qm9The QM9 (Ramakrishnan et al., 2014) dataset consists of small molecules represented as a set of atoms (up to 29 atoms per molecule), each atom having a 3D position associated and a five-dimensional one-hot node embedding that describes the atom type (H, C, N, O, F). The QM9 dataset has 3D coordinate locations of each atom and we use the distance between two atoms (nodes) as an edge attribute. The dataset comprises 12 quantum properties for each of the molecules. We use 100k molecules for training, 18k for validation, and 13k for testing.

Table 1 shows the Mean Absolute Error for the prediction of 12 molecular properties for weak and strong conditioning methods. We can clearly see that using _strong conditioning shows lower MAE than weak conditioning for all properties by an average of 10%_. Due to a large number of data points and our limited computational resources, we limit our experiments to weak and strong conditioning methods. Training one epoch of pure-EGNN was on average 267x longer than training strong-EGNN, which made it intractable to validate pure conditioning on this dataset.

The advantage in the performance of strong conditioning over weak conditioning is clearly shown in Table 1. We also hypothesize that the advantage of the strong conditioning method will be more evident in constrained settings of shallow networks. In order to test whether strong conditioning provide a more efficient parametrization then weak conditioning, we repeat experiments from Table 1 on three molecular properties, but with a more shallow network. Table 2a shows the MAE for the prediction of three molecular properties for weak and strong conditioning methods for models with 7 and 3 layers. On average improvement of strong conditioning over weak conditioning was more highlighted for the shallow network than for the deep network (10.8% vs 7.7%). It demonstrates the importance of conditioning methods in models with less capacity.

We define conditioning depth as the number of conditional layers in the message function. In our experiments, the conditioning depth was 2 as we conditioned all two layers in the message function. We test the effect of conditioning depth for different conditioning methods. Table 2b shows the MAE for the prediction of five molecular properties for weak and strong conditioning methods for models with conditioning depths 1 and 2. The weak-EGNN shows better performance with a single conditional layer while strong-EGNN benefits more from two conditional layers.

Md17MD17 (Chmiela et al., 2017) is a dataset of eight small organic molecules containing up to 17 total atoms composed of the atoms H, C, N, O, F. For each molecule, an ab-initio molecular dynamics simulation was run using DFT to calculate the ground state energy and forces. At intermittent timesteps, the energy, forces, and configuration (positions of each atom) were recorded. We uniformly sample 50k molecules for training, 10k for validation, and 10k for testing.

Table 3 shows the Mean Absolute Error for the prediction of energies for 8 molecules for three conditioning methods. From Table 3 we can see that strong conditioning improved the performance of 5 molecules by an average of 14.5%. It did not show improvement in molecules that initially had high MAE. In experiments with pure conditioning, to decrease computational cost we used half of the model: \(2\to 1\) conditional layers, hidden embedding size of \(128\to 64\), and \(7\to 3\) layers. With

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c|c|c|c} \hline Task & \(\alpha\) & \(\Delta\epsilon\) & \(\epsilon_{\textit{HOMO}}\) & \(\mu_{\textit{LUMO}}\) & \(\mu\) & \(C_{v}\) & \(G\) & \(H\) & \(R^{2}\) & \(U\) & \(U_{0}\) & \(ZPVE\) \\ Units & \(bolr^{3}\) & meV & meV & meV & D & cal/mol K & meV & meV & \(bolr^{3}\) & meV & meV & meV & meV \\ \hline \hline SchNet &.235 & 69 & 43 & 38 &.030 &.040 & 19 & 17 &.180 & 20 & 20 & 1.50 \\ DimNet++ &.044 & 33 & 25 & 20 &.030 &.023 & 8 & 7 &.331 & 6 & 6 & 1.21 \\ EGNN (baseline) &.071 & 48 & 29 & 25 &.029 &.031 & 12 & 12 & 106 & 12 & 11 & 1.55 \\ \hline \hline weak-EGNN (ours) &.067 & 50.52 & 28.11 & 25.74 &.032 &.031 & 9.81 & 10.51 &.138 & 11.14 & 9.94 & 1.455 \\ strong-EGNN (ours) &.061 & 44.52 & 27.48 & 23.83 &.023 &.029 & 9.79 & 10.29 &.088 & 9.39 & 9.88 & 1.45 \\ \hline \end{tabular}
\end{table}
Table 1: Mean Absolute Error for the molecular property prediction benchmark in QM9 dataset.
these changes, a pure-EGNN model was 14.3x and 16x times slower than strong-EGNN and weak-EGNN respectively. The negative impact of decreasing the number of layers on the performance of strong-EGNN on the QM9 dataset can be seen in Table (a)a. Considering that we reduced by half the number of layers, hidden embedding size, and conditioning depth, pure-EGNN shows competitive performance with weak/strong-EGNN on 5 molecules and outperforms it on some of them. This potential gain in performance might, however, not outweigh the computation costs of pure over strong or weak conditioning.

## 6 Conclusion

In this work, we explore how graph neural networks can recursively update node embeddings with edge attribute information such as geometric distance. We provide a unifying analysis of several works in literature that utilize attributes, through a notion of conditional linear layers. We present three conditioning methods to this end: weak, strong, and pure. Weak conditioning method concatenates edge attributes to node features, strong conditioning method gates node features, and in pure conditioning, edge attributes causally parametrize transformation matrices. We explain the intuition of each method and apply them to the EGNN model to empirically show their difference in performance and computational cost on QM9 and MD17 datasets.

Our conclusion is that strong conditioning (gating) generally beats weak conditioning (concatenation) in the message passing framework. We also conclude that pure conditioning is computationally prohibitive in geometric message passing, whilst it can achieve competitive performance with weak and strong conditioning methods with a smaller network. This confirms the impact observed by other works on separable convolutions, s.a. in Depth-wise separable convolutions (Chollet, 2017) and ConvNeXt (Liu et al., 2022), and justifies the use of separable (group) convolutions of steerable tensor-based methods such as TFN (Thomas et al., 2018). While these methods can be formulated as linear message passing methods of the convolutional form, we show that performance gains can be achieved through multi-layer conditional message functions, as a form of non-linear convolution (Brandstetter et al., 2021). In this setting, we show that it can be beneficial to condition all layers in the conditional MLP, rather than to only condition the first layer as is the convention.

\begin{table}

\end{table}
Table 2: Mean Absolute Error (MAE) for the molecular property prediction benchmark in the QM9 dataset for different numbers of layers and conditioning depths. \(\Delta\) is the percentage difference of MAE, lower \(\Delta\) is better.

\begin{table}

\end{table}
Table 3: Mean Absolute Error (MAE) for the conformational energies (meV) prediction benchmark on MD17 dataset.
We believe that our categorization of conditioning methods, combined with our empirical findings can be used as a guideline in designing the next generation of geometric graph neural network architectures.

**Acknowledgements:** This work is part of the research programme VENI with project "context-aware AI" with number 17290, which is (partly) financed by the Dutch Research Council (NWO).
