Attention-based Graph Convolution Fusing Latent Structures and Multiple
  Features for Graph Neural Networks
We present an attention-based spatial graph convolution (AGC) for graph
neural networks (GNNs). Existing AGCs focus on only using node-wise features
and utilizing one type of attention function when calculating attention
weights. Instead, we propose two methods to improve the representational power
of AGCs by utilizing 1) structural information in a high-dimensional space and
2) multiple attention functions when calculating their weights. The first
method computes a local structure representation of a graph in a
high-dimensional space. The second method utilizes multiple attention functions
simultaneously in one AGC. Both approaches can be combined. We also propose a
GNN for the classification of point clouds and that for the prediction of point
labels in a point cloud based on the proposed AGC. According to experiments,
the proposed GNNs perform better than existing methods. Our codes open at
https://github.com/liyang-tuat/SFAGC.
Yang Li, Yuichi Tanaka
Attention-based Graph Convolution Fusing Latent Structures and Multiple Features for Graph Neural Networks

###### Abstract

We present an attention-based spatial graph convolution (AGC) for graph neural networks (GNNs). Existing AGCs focus on only using node-wise features and utilizing one type of attention function when calculating attention weights. Instead, we propose two methods to improve the representational power of AGCs by utilizing 1) structural information in a high-dimensional space and 2) multiple attention functions when calculating their weights. The first method computes a local structure representation of a graph in a high-dimensional space. The second method utilizes multiple attention functions simultaneously in one AGC. Both approaches can be combined. We also propose a GNN for the classification of point clouds and that for the prediction of point labels in a point cloud based on the proposed AGC. According to experiments, the proposed GNNs perform better than existing methods. Our codes open at [https://github.com/liyang-tuat/SFAGC](https://github.com/liyang-tuat/SFAGC).

Attention-based graph convolution, graph neural network, 3D point cloud, deep learning.

## 1 Introduction

We often encounter irregularly structured data (signals) in the real world where they do not have a fixed spatial sampling frequency. Such data include opinions on social networks, the number of passengers on traffic networks, coordinates of 3D point clouds, and so on.

Deep neural networks have been widely used in recent years to detect, segment, and recognize regular structured data [1, 2, 3]. However, classical deep learning methods can not directly process the irregularly structured data mentioned above. They can be mathematically represented as data associated with a _graph_. An example of graph-structured data, that is, _graph signals_, is shown in Fig. 1. Deep neural networks for graph signals are called graph neural networks (GNNs): They have received a lot of attention [4, 5, 6, 7].

GNNs typically contain multiple graph convolution (GC) layers. The primary mechanism of GCs is to iteratively aggregate (i.e., filter) features from neighbors before integrating the aggregated information with that of the target node [4, 5, 8, 9]. In many existing GC methods, the node-wise features are typically utilized [10, 11, 12, 13]. Furthermore, it is observed that GCs are a special form of Laplacian smoothing [14]. This low-pass filtering effect often results in over-smoothing [4, 5, 14]. Over-smoothing means that the node-wise feature values become indistinguishable across nodes. Intuitively, representational power of GCs refers to the ability to distinguish different nodes [15]. Therefore, over-smoothing may negatively affect the performance of GNNs.

To improve the representational power of GCs, attention-based spatial graph convolutions (AGCs) such as graph attention networks (GATs) [16] have been proposed. AGCs are believed to have high representational power than the direct spatial methods because they can use features on neighboring nodes through the attention weights. However, there exist two major limitations in existing AGCs: 1) They may lose the _structural information_ of the surrounding neighboring nodes, especially in a high-dimensional space. 2) When calculating attention weights for each neighboring node, only one type of attention function is used, e.g., dot-production, subtraction, or concatenation. Different types of attention functions will
lead to different attention weights, which affect the representational power of AGCs.

In this paper, we propose a new AGC to overcome the above-mentioned limitations. First, we propose a _local structure projection aggregation_. This operation aggregates the structural information of neighboring nodes of a target node. Second, we also propose an AGC that utilizes multiple-type attention functions. We can simultaneously utilize these two methods to present the attention-based graph convolution fusing latent structures and multiple features (SFAGC). Our contributions are summarized as follows:

1. By using local structure projection aggregation, we can obtain a representation of the local structural information of the graph in high-dimensional space in an AGC. This allows the convolved nodes to contain richer information than existing methods.
2. By using multiple-type attention functions simultaneously, we can obtain better attention weights than the single attention function.
3. We construct GNNs for graph and node classifications based on the proposed AGC with 3D point clouds. We demonstrate that GNNs using our method present higher classification accuracies than existing methods through experiments on ModelNet [17] for graph classification and ShapeNet [18] for node classification.

_Notation:_ An undirected and unweighted graph is defined as \(\mathcal{G}:=(\mathcal{V},\mathcal{E})\), where \(\mathcal{V}\) is a set of nodes, and \(\mathcal{E}\) is a set of edges. The adjacency matrix of \(\mathcal{G}\) is denoted as \(A\). \(\widetilde{D}\) is the diagonal degree matrix. The graph Laplacian is defined as \(L:=\widetilde{D}-A\). \(I_{n}\) is the matrix whose elements in the diagonal are 1.

Here, \(h_{v}:=[\mathrm{h}_{v1},\ldots,\mathrm{h}_{vi},\ldots,\mathrm{h}_{vD}]^{ \mathsf{T}}\in\mathbb{R}^{D}\) represents a feature vector on the node \(v\in\mathcal{V}\), and \(D\) is the number of features in \(h_{v}\). \(co_{v}:=[\mathrm{co}_{v1},\ldots,\mathrm{co}_{vj},\ldots,\mathrm{co}_{vC}]^{ \mathsf{T}}\in\mathbb{R}^{C}\) represents a coordinate of the node \(v\in\mathcal{V}\), and \(C\) is the dimension of coordinate in \(co_{v}\).

The non-linearity function is denoted as \(\sigma(\cdot)\). The set of neighboring nodes is \(N(\cdot)\) in which its cardinality is denoted as \(|N(\cdot)|\). A multilayer perceptron (MLP) layer is represented as MLP(\(\cdot\)). A channel-wise avg-pooling is denoted as AvgPool(\(\cdot\)). A channel-wise max-pooling is denoted as MaxPool(\(\cdot\)). The vector concatenation operation is denoted as cat(\(\cdot\)). The SoftMax operation is represented as SoftMax(\(\cdot\)).

## II Preliminary

In this section, we present related work for the proposed GC.

### _Graph convolutions_

The mechanism of GCs is based on message passing. Message passing involves iteratively aggregating information from neighboring nodes and then integrating the aggregated information with that of the target node [4, 5].

Typically, a GC has two parts, i.e., an aggregator and an updater. The aggregator is to collect and aggregate the node-wise features of neighboring nodes. The updater merges the aggregated features into the target node to update the node-wise features of the target node. These two parts are illustrated in Fig. 2.

Existing GCs can be classified into spectral and spatial methods. Furthermore, spatial methods can be classified into direct and attention-based methods. We briefly introduce them in the following.

#### Ii-A1 Spectral methods

In the spectral methods, the aggregation operation is carried out in the graph Fourier domain. Eigenvectors of graph Laplacian are known as the graph Fourier bases [1, 8, 19]. In order to reduce the computational complexity for large graphs, polynomial approximations of graph filters are often utilized [10, 11, 20].

GCN [12] further reduces computational complexity through the first-order graph filters. It can be formulated as follows:

\[H_{\text{GCN}}=\sigma(\widetilde{D}^{-1/2}\widetilde{A}\widetilde{D}^{-1/2}HW), \tag{1}\]

where \(H:=\{h_{v}\}_{v\in\mathcal{V}}\) is the set of node-wise features, \(\widetilde{A}=A+I_{n}\) is the adjacency matrix with self-loops, and \(W\) is a learnable weight matrix.

#### Ii-A2 Spatial methods

Spatial methods are a counterpart of spectral methods. As we described above, spatial methods can be classified into direct and attention-based methods. Direct spatial methods directly use the node-wise features and the aggregation operation is carried out spatially. A representative method for direct spatial methods is GraphSAGE [13], which treats each neighbor equally with mean aggregation.

Later, attention-based spatial methods are proposed. Instead of treating all neighboring nodes equally, attention-based methods calculate an attention weight for each neighboring node. Then, they use the weighted sum to aggregate features of neighboring nodes. GAT [16] is a representative method for attention-based spatial methods. It is composed

Figure 1: An example of graph-structured data.

Figure 2: A GC has two parts, i.e., an aggregator and an updater. The aggregator is to collects and aggregates the node-wise features of neighboring nodes. The updater merges the aggregated features into the target node to update the node-wise features of the target node.
of three steps. In the first step, the learnable weights are multiplied by the node-wise features, i.e., \(h^{\prime}_{\nu}=\{W\cdot h_{v}\}_{v\in V}\), where \(W\) is the learnable weights. The second step computes attention weights as follows:

\[a_{uv}=\text{SoftMax}(\sigma(W_{a}\cdot(\text{cat}(h^{\prime}_{v},h^{\prime}_{u} )))),u\in N(v) \tag{2}\]

where \(W_{a}\) is learnable weights. In the third step, the node-wise features of a target node \(v\) is updated as follows:

\[h^{\prime\prime}_{v}=\sigma\left(\sum_{u\in N(v)}a_{uv}\cdot(h^{\prime}_{u}) \right). \tag{3}\]

However, as mentioned earlier, existing methods ignore the structural information of neighboring nodes in the high-dimensional space, and use only one type of attention function when calculating an attention weight for each neighboring node.

In contrast to the existing methods, we first focus on computing a representation of the structure of neighboring nodes in high-dimensional feature space and installing them into a spatial GC. Second, we use multiple types of attention functions simultaneously in an AGC. We use both of these two methods simultaneously to achieve SFAGC.

### Structural Features

In previous works [21, 22], three structural features, i.e., feature angle, feature distance, and relational embedding, are proposed to describe the structural information between the target node and neighboring nodes. Below, we briefly introduce them since they are also used in our attention-based GCs.

#### 2.2.1 Feature angle

_Feature angle_ describes the local structure of neighboring nodes. First, a set of structure vectors pointing from target node \(v\) to the neighboring nodes are calculated as \(\text{SV}_{N(v)}=\{\text{s}\text{v}_{uv}:=h_{u}-h_{v}\}_{u\in N(v)}\). Then, a base structure vector \(\text{sv}_{b}\) is learned from \(\text{SV}_{N(v)}\) as follows:

\[\text{sv}_{b}=\text{AvgPool}(\{\sigma(W_{b}\cdot\text{s}\text{v}_{uv})\}_{ \text{sv}_{uv}\in\text{SV}_{N(v)}}) \tag{4}\]

where \(W_{b}\) is learnable weights. An example of a base structure vector \(\text{sv}_{b}\) is shown in Fig. 3

Finally, the cosine of the angle between \(\text{sv}_{uv}\) and \(\text{sv}_{b}\) is calculated to obtain the feature angle \(\text{fa}_{uv}\) as follows:

\[\text{fa}_{uv}=\cos(\theta_{u})=\frac{\text{s}\text{s}\text{v}_{uv}\cdot\text{ s}\text{v}_{b}^{\text{T}}}{\|\text{s}\text{v}_{uv}\|\cdot\|\text{s}\text{v}_{b}\|}, \text{sv}_{uv}\in\text{SV}_{N(v)} \tag{5}\]

An example is shown in Fig. 4 (a).

#### 2.2.2 Feature distance

The second structural feature is _feature distance_. It is the absolute difference between the node-wise features of \(h_{u}\) and \(h_{v}\) represented as follows:

\[\text{fd}_{uv}=[|\text{h}_{u1}-\text{h}_{v1}|,...,|\text{h}_{uD}-\text{h}_{vD} |]|^{\text{T}}. \tag{6}\]

An example is shown in Fig. 4 (b).

#### 2.2.3 Relational embedding

The third structural feature is _relational embedding_. It can be learned from \(\{\text{s}\text{v}_{uv}\}\) as follows:

\[\text{re}_{uv}=\sigma(W_{\text{re}}\cdot\text{s}\text{v}_{uv}),u\in N(v). \tag{7}\]

where \(W_{\text{re}}\) is learnable weights. An example of it is shown in Fig. 4 (c).

Figure 4: Example of our structural features. (a) is our feature angle; (b) is our feature distance, \(\text{h}_{v1}\) is the element of \(h_{v}\), \(D\) is the dimensions of node-wise features; (c) is our relational embedding.

Figure 5: An example of a graph in high-dimensional space. A node consists of a coordinate and the node-wise features, i.e., \(\{v:=((c_{o},h_{v}))_{v\in V}\). For example, the coordinate of a node in a graph of a 3D color point cloud is \((x,y,z)\), and the node-wise features are the values of RGB. We use the coordinates to calculate structural features.

Figure 3: An example of a base structure vector. The number in the black circle are the node indices.
## III Sfaoc

In this section, we introduce SFAGC. As mentioned above, we have two goals: Utilizing 1) the structural information of neighboring nodes in a high-dimensional feature space during a single step GC and 2) multiple-type attention functions simultaneously when calculating the attention weights.

Fig. 5 illustrates an example of a graph with feature vectors in the spatial domain. Spatially distributed nodes often have their coordinates and associated node-wise features, i.e., \(\left\{v:=(co_{v},h_{v})\right\}_{v\in\mathcal{V}}\), where \(co_{v}\) is the coordinate of the node \(v\). For example, a 3D color point cloud equips a 3-D \((x,y,z)\) coordinate and its node-wise features as RGB values. In the previous structure-aware GCs [21, 22], the node-wise features are simply used as their coordinates. In contrast, the proposed GC, SFAGC, simultaneously considers the coordinates and node-wise features.

To achieve our goals, the SFAGC has four parts:

\begin{tabular}{l l} \(A\). & Local structure projection aggregation and fusing \\ \(B\). & Position embedding \\ \(C\). & Weighted sum aggregation and update \\ \(D\). & Coordinates processing \\ \end{tabular}

Figure 6: The details of our SFAGC.
SFAGC is illustrated in Fig. 6 and the algorithm of SFAGC is summarized in Algorithm 1. We sequentially introduce these parts.

### Local structure projection aggregation and Fusing

We propose a projection aggregation operation to obtain the structure representation in the feature space. We then fuse this with the node-wise features of the target node.

#### -A1 Local structure projection aggregation

The inputs of this step are the feature angle, feature distance and relational embedding, i.e., \(\text{fa}_{uv}\), \(\text{fd}_{uv}\) and \(\text{re}_{uv}\), introduced in Section II-B. We first compute structure vectors as follows:

\[\text{s}_{uv}=\text{cat}(\text{fd}_{uv},\text{re}_{uv},\sigma(\left.W_{\text{ se}}(\text{cat}(\text{fd}_{uv},\text{re}_{uv})))\right),u\in N(v), \tag{8}\]

where \(W_{\text{se}}\) is a learnable weight matrix.

Then, we project each \(\text{s}_{uv}\) as follows:

\[\hat{\text{se}}_{uv}=\text{fa}_{uv}\cdot\text{s}_{uv},u\in N(v). \tag{9}\]

Finally, we calculate the summation of the projected structure vectors as follows:

\[\text{af}_{v}=\sum_{u\in N(v)}\hat{\text{se}}_{uv}. \tag{10}\]

Fig. 7 illustrates an example of the local structure projection aggregation.

#### -A2 Fusing structure information with node-wise features

In this step, we fuse the \(\text{af}_{v}\) with the \(h_{v}\) as follows:

\[h_{v}^{\prime}=\sigma(W_{s}(\text{cat}(h_{v},\text{af}_{v}))),v\in\mathcal{V}. \tag{11}\]

where \(W_{s}\) is learnable weights.

### Position Embedding

Position encoding is crucial for a self-attention mechanism because it enables the aggregation operation to adapt to local data structure [23, 24]. Our method directly learns a position embedding, while existing methods use cosine function-based position encoding [24].

We embed the difference of the coordinates between the neighboring node \(u\) and target node \(v\) in the feature space. Our position embedding \(p_{u}\) for the node \(u\) is represented as follows:

\[p_{u}=W_{\text{P2}}\cdot(\sigma(W_{\text{P1}}\cdot(co_{u}-co_{v}))),u\in N(v) \tag{12}\]

where \(W_{\text{P1}}\) and \(W_{\text{P2}}\) are learnable weights.

### Weighted Sum Aggregation and Update

In this part, we update the node-wise features of the target node \(v\).

First, we introduce the calculation steps of the attention weights. Then, we present the weighted sum aggregation and node-wise features update step used in SFAGC.

#### -A1 Attention weights

As we described above, we simultaneously use multiple-type attention functions to calculate attention weights. In existing methods [23, 24, 25], the subtraction or dot-production attention function is often utilized to calculate attention weights. Instead of the single attention function, we use these attention functions simultaneously. The subtraction attention function is defined as

\[\text{a}_{1vu}:=W_{q1}\cdot h_{v}^{\prime}-W_{k1}\cdot h_{u}^{\prime},u\in N( v), \tag{13}\]

where \(W_{q1}\) and \(W_{k1}\) are learnable weights. The dot-production attention function is represented as

\[\text{a}_{2vu}:=(W_{q2}\cdot h_{v}^{\prime})\cdot(W_{k2}\cdot h_{u}^{\prime}) ^{T},u\in N(v) \tag{14}\]

where \(W_{q2}\) and \(W_{k2}\) are learnable weights.

Then, the two types of attention functions are added with the position embedding \(p_{u}\) as follows:

\[\text{qk}_{vu}=\text{a}_{1vu}+W_{c}\cdot\text{a}_{2vu}+p_{u},u\in N(v) \tag{15}\]

where \(W_{c}\) is learnable weights that also converts \(\text{a}_{2vu}\) into the same dimensions as \(\text{a}_{1vu}\).

Finally, \(\text{qk}_{vu}\) is input into a small network to calculate the attention weights between the target node \(v\) and the neighboring node \(u\) as follows:

\[\text{at}_{eu}=\text{SoftMax}\left(\frac{W_{a2}\cdot\sigma(W_{a1}\cdot\text{ qk}_{vu})}{\sqrt{d_{out}}}\right),u\in N(v), \tag{16}\]

where \(W_{a1}\in\mathbb{R}^{d_{eu}\times d_{out}}\) and \(W_{a2}\in\mathbb{R}^{d_{out}\times 1}\) are learnable weights, \(d_{in}\) and \(d_{out}\) are the dimensions of the \(W_{a1}\).

#### -A2 Weighted-sum aggregation

For \(h_{v}^{\prime}\) and \(h_{u}^{\prime},u\in N(v)\), weighted sum aggregation is calculated as follows:

\[\widetilde{h_{v}}=\sum_{u\in N(v)}\text{at}_{eu}\cdot(W_{v}\cdot h_{u}^{ \prime}+p_{u}), \tag{17}\]

where \(W_{v}\) is a learnable matrix.

Figure 7: An example of the local structure projection aggregation. The \(\text{s}_{uv}\) is the base structure vector defined in (4).
3 Node-wise features update \(h_{v}\), \(co_{v}\) and \(\widetilde{h_{v}}\) are integrated as follows:

\[h_{v}^{\prime\prime}=\sigma(W\cdot\text{cat}(h_{v},co_{v},\widetilde{h_{v}})). \tag{18}\]

where \(W\) is learnable weights.

### Coordinate update

Finally, we update the coordinate of the target node \(v\) as follows:

\[co_{v}^{\prime}=\text{MLP}(co_{v}),v\in\mathcal{V}. \tag{19}\]

Hereafter, we represent the set of these operations as \(\{h_{\mathcal{V}}^{\prime\prime},co_{\mathcal{V}}^{\prime}\}:=\text{SFAGC}(h_ {\mathcal{V}},co_{\mathcal{V}},A)\).

## IV Implementation

In this section, we construct classification and segmentation networks for 3D point clouds based on SFAGC. Their architectures are illustrated in Fig. 8. In the following, first, we introduce the components shared by the two GNNs. Then, specific implementations for each of the GNNs are introduced.

Here, we suppose that the input point cloud is given by \(\mathcal{X}=\{x_{i}\}_{i=1}^{N}\) where \(N\) is the number of points.

### Preprocessing

To alleviate effects on rotations of point clouds, we use the same transformation module as PointNet [26] for preprocessing.

### SFAGC module

The inputs to this module are a set of features obtained from the previous layer and a set of node coordinates in the feature space. Suppose that the set of features output from the previous layer is \(\mathcal{H}=\{h_{i}\}_{i=1}^{M}\) where \(M\) is the number of features of the input. For the first module, \(\mathcal{H}=CO=\mathcal{X}\) where \(CO=\{co_{j}\}_{j=1}^{M}\) is the set of node coordinates.

First, we construct \(\mathcal{G}\) with the \(k\)-NN graph from \(CO\) where the \(i\)th node \(v_{i}\) in \(\mathcal{G}\) corresponds to the \(i\)th feature \(h_{i}\) and the \(i\)th coordinate \(co_{i}\).

Recent studies [27, 28] have shown that dynamic graph convolution, i.e., allowing the graph structure to change at each layer, can perform better than that with a fixed graph structure. Therefore, we update coordinates of nodes at each SFAGC module, and we construct different graphs for different SFAGC modules.

### Design of 3d point cloud classification network

Fig. 8 (a) illustrates the architecture of 3D point cloud classification network based on SFAGC. In the following, we describe the details of the building blocks that are specifically designed for point cloud classification.

#### Iv-1 Multi-resolution point clouds

We use a layer of PointNet++ [29] to generate a low-resolution point cloud. Both global and local information of the point cloud can be obtained through the multi-resolution structure.

#### Iv-2 Score-based graph pooling

Effective graph pooling methods are a hot topic in GNNs and graph signal processing [30, 31]. Early work has been done by global pooling of all node-wise features or by using graph coarsening algorithms.

Recently, trainable graph pooling operations DiffPool [32], GraphU-net [33], and AttPool [34] have been proposed. Inspired by the ResNeXt [35], we extend the score-based graph pooling module proposed in SAMGC [22] by introducing the multi-branch architecture. The score-based graph pooling module has three branches: score-based sampling, integration, and SFAGC branches. The architecture of the score-based graph pooling is shown in Fig. 9. In the following, we introduce their details.

In the score-based sampling branch, we propose a score-based sampling to find the indices of the best \(t\) nodes according to their scores. The score associated with each node is first computed as follows:

\[score_{v}=\text{SoftMax}(W_{1}\cdot h_{v}),v\in\mathcal{V}, \tag{20}\]

where \(W_{1}\) is learnable weights. We then sort the nodes in descending order according to their scores. We find the indices of the top \(t\) nodes as follows:

\[\text{idx}_{select}=\text{rank}(\{score_{v}\}_{\mathcal{V}},t), \tag{21}\]

where \(\text{rank}(\cdot)\) is the ranking operation, and it finds the indices of the \(t\) highest scores.

In the integration branch, node-wise features are multiplied by the node scores as follows:

\[\hat{h}_{\mathcal{V}}=\{score_{v}\cdot h_{v}\}_{v\in\mathcal{V}}. \tag{22}\]

In the SFAGC branch, the input graph is processed using SFAGC as follows:

\[\{h_{\mathcal{V}}^{\prime},co_{\mathcal{V}}^{\prime}\}=\text{SFAGC}(h_{ \mathcal{V}},co_{\mathcal{V}}). \tag{23}\]

Finally, the subset of \(\hat{h}_{\mathcal{V}}\) and the subset of \(\{h_{\mathcal{V}}^{\prime},co_{\mathcal{V}}^{\prime}\}\) are found using \(\text{idx}_{select}\) as follows:

\[\hat{h}_{\mathcal{V}_{sub}} =\hat{h}_{\mathcal{V}}[\text{idx}_{select}] \tag{24}\] \[\{h_{\mathcal{V}_{sub}}^{\prime},co_{\mathcal{V}_{sub}}^{\prime}\} =\{h_{\mathcal{V}}^{\prime},co_{\mathcal{V}}^{\prime}\}[\text{idx}_{select}]\]

\(\hat{h}_{\mathcal{V}_{sub}}\) and \(h_{\mathcal{V}_{sub}}^{\prime}\) are merged using learnable weights as follows:

\[\hat{h}_{\mathcal{V}_{sub}}^{\prime}=\sigma(W_{pl}\cdot\text{cat}(\hat{h}_{ \mathcal{V}_{sub}},h_{\mathcal{V}_{sub}}^{\prime})), \tag{25}\]

where \(W_{pl}\) is learnable weights. The score-based graph pooling can be summarized as follows:

\[\{co_{\mathcal{V}_{sub}}^{\prime},\hat{h}_{\mathcal{V}_{sub}}^{\prime}\}= \text{GraphPool}_{s}(co_{\mathcal{V}},h_{\mathcal{V}}). \tag{26}\]
#### 3.3.3 Hierarchical prediction architecture

Here, we also use the intermediate supervision technique [36] and the hierarchical prediction architecture (Fig. 8 (a)), as in SAMGC [22]. The advantage of this architecture is that, by combining the outcomes of the different phases, more reliable and robust predictions can be produced [22]. The details are presented below.

We use two SFAGC modules in phase 1 to phase 5. One PointNet++ [29] layer is used in phase 6. Each phase connects with a max pooling layer and an average pooling layer. The outputs are then concatenated and input into a fully connected layer. We calculate the prediction and the classification loss for each phase. The total classification loss is obtained by adding the losses of several phases. Meanwhile, the total prediction is also increased by the predictions of several phases. The following is a representation of this processing:

\[prediction =\sum_{i=1}^{P}prediction_{i}, \tag{27}\] \[loss =\sum_{i=1}^{P}loss_{i}, \tag{28}\]

where \(prediction_{i}\) is the prediction of the \(i\)th phase, \(loss_{i}\) is the cross-entropy loss of the \(i\)th phase. \(prediction\) and \(loss\) are the total prediction and classification loss, respectively. \(P\) is the number of phases.

Figure 8: The architectures of our 3D point cloud classification network and our 3D point cloud segmentation network. (a) is the architecture of 3D point cloud classification network; (b) is the architecture of 3D point cloud segmentation network, L_p_points is the set of outputs of the \(j\)th phase. \(N_{\rm Lip}\) is the number of nodes of the L_p_points.

Figure 9: The details of our graph pooling operation. The score-based graph pooling is used in 3D point cloud classification network. The FPS-based graph pooling is used in 3D point cloud segmentation network. Here, we set \(t=3\).
### Design of 3D point cloud segmentation network

Fig. 8 (b) illustrates the architecture of the 3D point cloud segmentation network based on SFAGC. In the following, we describe the details of the building blocks that are specifically designed for point cloud segmentation.

#### 1.4.1 Farthest point sampling-based graph pooling

For point cloud segmentation, we use the graph pooling module to reduce the overall computation cost. Here, we propose the farthest point sampling-based graph pooling (FPS-based graph pooling) by modifying the score-based graph pooling. The architecture of the FPS-based graph pooling is illustrated in Fig. 9. In the following, we introduce its details.

The FPS-based graph pooling has a multi-branch architecture like the score-based graph pooling in Section 4.2.2. In contrast to the score-based method, the FPS-based graph pooling has two branches, i.e., the FPS and SFAGC branches.

In the FPS branch, we perform FPS on nodes to obtain indices of the best \(t\) nodes according to their coordinates. FPS algorithm is widely utilized in 3D point cloud processing [29, 37]. The mechanism of FPS is to iteratively select the node that is farthest from the existing sampled nodes. Therefore, the sampled nodes with the FPS-based sampling are expected to be more evenly distributed than those with the score-based sampling. This branch can be summarized as follows:

\[\text{idx}_{select}=\text{FPS}(\{co_{v}\}_{\mathcal{V}},t). \tag{29}\]

where \(\text{idx}_{select}\) is the indices of the \(t\) selected nodes, \(\text{FPS}(\cdot)\) is the farthest point sampling algorithm.

The SFAGC branch is the same as the one that is used in the score-based graph pooling represented as

\[\{h^{\prime}_{\mathcal{V}},co^{\prime}_{\mathcal{V}}\}=\text{ SFAGC}(h_{\mathcal{V}},co_{\mathcal{V}}). \tag{30}\]

Finally, the subset of \(\{h^{\prime}_{\mathcal{V}},co^{\prime}_{\mathcal{V}}\}\) are extracted using \(\text{idx}_{select}\) as follows:

\[\{co^{\prime}_{\mathcal{V}_{sub}},h^{\prime}_{\mathcal{V}_{sub}}\}=\{co_{ \mathcal{V}},h_{\mathcal{V}}\}[\text{idx}_{select}]. \tag{31}\]

The FPS-based graph pooling is represented as follows:

\[\{co^{\prime}_{\mathcal{V}_{sub}},h^{\prime}_{\mathcal{V}_{sub}}\}=\text{ GraphPool}_{f}(co_{\mathcal{V}},h_{\mathcal{V}}). \tag{32}\]

#### 1.4.2 Upsampling operation

Graph pooling can be regarded as a downsampling operation. For point cloud segmentation, the network also needs to perform upsampling in order to maitain the number of points. Therefore, the feature propagation used in PointNet++ [29] is also used in our network as the upsampling operation.

## 5 Experiments

In this section, we conduct experiments on 3D point cloud classification and segmentation to validate the proposed GC.

### 3D point cloud classification

Here, we present the 3D point cloud classification experiment using the 3D point cloud classification network introduced in Section 4.

#### 5.1.1 Dataset

The ModelNet dataset [17] is used in our point cloud classification experiment. 12,308 computer-aided design (CAD) models in 40 categories are included in ModelNet40. In addition, 9,840 CAD models are utilized for training and

\begin{table}
\begin{tabular}{c|l|l|l|l} \hline \hline \multicolumn{1}{c|}{Epoch} & \multicolumn{1}{c}{200} \\ \hline \multicolumn{1}{c|}{Batch size} & \multicolumn{1}{c}{16} \\ \hline \multicolumn{1}{c|}{Learning rate} & \multicolumn{1}{c}{0.001} \\ \hline \multicolumn{1}{c|}{Drop out} & \multicolumn{1}{c}{0.3} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \hline \multicolumn{1}{c|}{Phase} & \multicolumn{1}{c}{Graph convolution layer} & \multicolumn{1}{c}{Coordinates update} & \multicolumn{1}{c}{\(k\)} & \multicolumn{1}{c}{[\(\text{co}_{\text{fa}},\text{co}_{\text{out}},\text{f}_{\text{out}}\)]} \\ \hline \multicolumn{1}{c|}{Phase1} & \multicolumn{1}{c}{SFAGC(\(co_{v}=x_{v}\))} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 20 & [3,3,26,4] \\ \multicolumn{1}{c|}{} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=co_{v}\) & 20 & [32,64,-64] \\ \hline \multicolumn{1}{c|}{Phase2} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 20 & [32,64,64,64] \\ \multicolumn{1}{c|}{} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=co_{v}\) & 20 & [64,64,-128] \\ \hline \multicolumn{1}{c|}{Phase3} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 20 & [64,128,128,256] \\ \hline \multicolumn{1}{c|}{Phase4} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 20 & [64,128,64,128] \\ \multicolumn{1}{c|}{} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=\text{co}_{v}\) & 20 & [64,128,-128] \\ \hline \multicolumn{1}{c|}{Phase5} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 20 & [128,128,128,256] \\ \multicolumn{1}{c|}{} & \multicolumn{1}{c}{SFAGC} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 20 & [128,128,256] \\ \hline \multicolumn{1}{c|}{\begin{tabular}{c} Graph pooling layername & Graph convolution layer \\ \end{tabular} } & \multicolumn{1}{c}{Coordinates update} & \multicolumn{1}{c}{\(k\)} & \multicolumn{1}{c}{[\(\text{co}_{\text{fa}},\text{co}_{\text{out}},\text{f}_{\text{out}}\)]} \\ \hline \multicolumn{1}{c|}{\begin{tabular}{c} GraphPool\_s 1 \\ GraphPool\_2 \\ GraphPool\_3 \\ GraphPool\_3 \\ \end{tabular} } & \multicolumn{1}{c}{SFAGC(\(co_{v}=h_{v}\))} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 36 & 512 & [131,131,32,64] \\ \multicolumn{1}{c|}{\begin{tabular}{c} GraphPool\_3 \\ \end{tabular} } & \multicolumn{1}{c}{SFAGC(\(co_{v}=h_{v}\))} & \(co^{\prime}_{\mathcal{V}}=\text{MLP}(co_{v})\) & 64 & 128 & [320,320,64,128] \\ \hline \multicolumn{1}{c|}{
\begin{tabular}{c} Pointnet++ layername & \(\mathcal{S}\) \\ \end{tabular} } & \multicolumn{1}{c}{\(r\)} & \multicolumn{1}{c}{\(D\)} & \multicolumn{1}{c}{[input channels, output channels]} \\ \hline \multicolumn{1}{c|}{Pointnet++} & \multicolumn{1}{c}{512} & \multicolumn{1}{c}{0.2} & \multicolumn{1}{c}{32} & \multicolumn{1}{c}{[3,64]} \\ \hline \hline \end{tabular}
\end{table}
Table 1: The hyperparameters of the point cloud classification network. \(k\) is the value of \(k\)-NN, \(t\) is the number of selected nodes. For Pointnet++ layer, \(S\) is the number of sampled points, \(r\) is the radius of each group, \(D\) is the number of points of each group, \(\text{co}_{\text{fa}}\) is the number of channels of the input coordinates. \(\text{fa}_{\text{fa}}\) is the number of channels of the input node-wise features. \(\text{co}_{\text{fa}}\) is the number of channels of the output coordinates. \(\text{fa}_{\text{fa}}\) is the number of channels of the output node-wise features. The symbol \(\mathcal{V}^{-}\) indicates that the parameters are not available. The input 3D point cloud is \(\mathcal{X}=\{x_{v}\}_{v=1}^{N}\).
2,468 CAD models are used for testing. 4,899 CAD models are included in ModelNet10. They are divided into 3,991 for training and 908 for testing from ten categories. For each CAD models, the CAD mesh faces were evenly sampled with 1,024 points. Initially, all point clouds were normalized to be in a unit sphere.

#### 3.2.2 Settings and evaluation

The settings of hyperparameters are summarized in Table 1. We use the average accuracy of all test instances (OA) and the average accuracy of all shape classes (mAcc) to evaluate the performance of our network.

#### 3.2.3 Results and discussion

Table 2 summarizes the results for point cloud classification. The results of existing methods are taken from their original papers. In terms of both OA and mAcc, our method performs better than the others.

In the following, we focus on the comparison of our method and graph-based methods. The GCs utilized in DPAM [45] are GCN [12]. Therefore, it only uses node-wise features of one-hop neighboring nodes. RGCNN [46], 3DTI-Net [47], PointGCN [48], and LocalSpaceGCN [49] are the spectral methods. Although they can design the global spectral response, they may neglect the local spatial information. In comparison with the direct spatial methods [27], [28],

\begin{table}
\begin{tabular}{c|l|c c|c c} \hline \hline \multicolumn{1}{c|}{Type} & Method & \multicolumn{2}{c|}{ModelNet40} & \multicolumn{2}{c}{ModelNet10} \\ \cline{3-6} \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & OA & mAcc & OA & mAcc \\ \hline Pointwise MLP & PointNet [26] & 89.2\% & 86.2\% & - & - \\ Methods & PointNet++ [29] & 90.7\% & - & - & - \\  & SRN-PointNet++ [38] & 91.5\% & - & - & - \\ \hline Transformer-based & PointASNL [37] & 93.2\% & - & 95.9\% & - \\ Methods & PCT [39] & 93.2\% & - & - & - \\  & PointTransformer [25] & 93.7\% & 90.6 \% & - & - \\ \hline Convolution-based & PointConv [40] & 92.5\% & - & - & - \\ Methods & A-CNN [41] & 92.6\% & 90.3\% & 95.5\% & 95.3\% \\  & SFCNN [42] & 92.3\% & - & - & - \\  & InterpCNN [43] & 93.0\% & - & - & - \\  & ConvPoint [44] & 91.8\% & 88.5\% & - & - \\ \hline Graph-based & Spectral & DPAM [45] & 91.9\% & 89.9\% & 94.6\% & 94.3\% \\ Methods & Methods & RGCNN [46] & 90.5\% & 87.3\% & - & - \\  & & 3DTI-Net [47] & 91.7\% & - & - & - \\  & PointGCN [48] & 89.5\% & 86.1\% & 91.9\% & 91.6\% \\  & LocalSpecGCN [49] & 92.1\% & - & - & - \\ \hline Spatial & ECC [28] & 87.4\% & 83.2\% & 90.8\% & 90.0\% \\ Methods & KCNet [50] & 91.0\% & - & 94.4\% & - \\  & DGCNN [27] & 92.2\% & 90.2\% & - & - \\  & LDGCNN [51] & 92.9\% & 90.3\% & - & - \\  & Hassan et al. [52] & 89.1\% & - & - & - \\  & ClusterNet [53] & 87.1\% & - & - & - \\  & Grid-GCN [54] & 93.1\% & 91.3\% & 97.5\% & 97.4\% \\  & SAGConv [21] & 93.5\% & 91.3\% & 98.3\% & 97.7\% \\  & SAMGC [22] & 93.6\% & 91.4\% & 98.3\% & 97.7\% \\ \cline{2-6}  & **SFAGC** & **94.0\%** & **91.6\%** & **98.6\%** & **97.8\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison results of the 3D shape classification on the ModelNet benchmark. OA indicates the average accuracy of all test instances, and mAcc indicates the average accuracy of all shape categories. The symbol ‘-’ indicates that the results are not available from the references.

\begin{table}
\begin{tabular}{c|l|l|l|l} \hline \hline \multicolumn{1}{c|}{Epoch} & \multicolumn{1}{c|}{251} \\ \hline \multicolumn{1}{c|}{Batch size} & \multicolumn{1}{c|}{16} \\ \hline \multicolumn{1}{c|}{Learning rate} & \multicolumn{1}{c|}{0.001} \\ \hline \multicolumn{1}{c|}{Drop out} & \multicolumn{1}{c|}{0.4} \\ \hline \multicolumn{1}{c|}{Phase} & Graph convolution layer & \multicolumn{1}{c|}{Coordinates update} & \(k\) & \([\text{co}_{n}\text{-}\text{f}_{n}\text{co}_{n}\text{-f}_{out}]\) \\ \hline \multicolumn{1}{c|}{Phase1} & SFAGC(\(cov_{n}=x_{v}\)) & \(cov_{n}^{\prime}=\text{MLP}(cov_{n})\) & 20 & \([3,3,32,64]\) \\ \hline \multicolumn{1}{c|}{Phase2} & SFAGC & \(cov_{n}^{\prime}=\text{MLP}(cov_{n})\) & 20 & \([3,64,32,64]\) \\ \multicolumn{1}{c|}{} & SFAGC & \(cov_{n}^{\prime}=\text{co}_{n}\) & 20 & \([32,64,-128]\) \\ \hline \multicolumn{1}{c|}{Phase3} & SFAGC & \(cov_{n}^{\prime}=\text{MLP}(cov_{n})\) & 20 & \([3,128,128,256]\) \\ \hline \multicolumn{1}{c|}{Graph pooling layername} & Graph convolution layer & Coordinates update & \(k\) & \(t\) & \([\text{co}_{n}\text{-f}_{n}\text{co}_{n}\text{-f}_{out}]\) \\ \hline \multicolumn{1}{c|}{GraphPool\_1} & SFAGC(\(cov_{n}=x_{v}\)) & \(cov_{n}^{\prime}=cov_{n}\) & 36 & 512 & \([3,131,36,4]\) \\ \multicolumn{1}{c|}{GraphPool\_2} & \multicolumn{1}{c|}{SFAGC(\(cov_{n}=x_{v}\))} & \(cov_{n}^{\prime}=cov_{n}\) & 64 & 128 & \([3,320,3,128]\) \\ \hline \multicolumn{1}{c|}{Feature propagation layer name} & \multicolumn{1}{c|}{[input channels, output channels]} \\ \hline \multicolumn{1}{c|}{Feature propagation1} & \multicolumn{1}{c|}{[256+256+128+128,256]} \\ \multicolumn{1}{c|}{Feature propagation2} & \multicolumn{1}{c|}{[256+64+64,128]} \\ \hline \hline \end{tabular}
\end{table}
Table 3: The hyperparameters of the point cloud segmentation network. \(k\) is the value of \(k\)-NN, \(t\) is the number of selected nodes. \(\text{co}_{n}\) is the number of channels of the input coordinates. \(\text{f}_{n}\) is the number of channels of the input node-wise features. \(\text{co}_{n}\) is the number of channels of the output coordinate. \(\text{f}_{out}\) is the number of channels of the output node-wise features. The symbol ‘-’ indicates that the parameters are not available. The input 3D point cloud is \(\mathcal{X}=\{\chi_{i}\}_{i=1}^{k}\).
[50, 51, 52, 53, 54], our method can obtain the local structural information of the graph in the feature space, and the information of neighboring nodes can be utilized efficiently using attention-based aggregation. In comparison with the direct spatial methods, i.e., SAGConv [21] and SAMGC [22], the proposed method can better aggregate the structural information of the neighboring nodes using the local structure projection aggregation. Furthermore, the information of neighboring nodes can be utilized efficiently using attention-based aggregation. These are possible reasons for the performance improvement of the proposed method.

### 3D Point Cloud Segmentation

In this subsection, we also perform a 3D point cloud segmentation experiment.

#### 3.c.1 Dataset

The ShapeNet dataset [18] is used in the experiment. It contains 16,846 computer-aided design (CAD) models in 40 categories and each point cloud has 2,048 points. 2,874 point clouds are used for testing, and 13,998 of them are taken for training.

#### 3.c.2 Settings and evaluation

Table 3 shows the hyperparameter settings. For this experiment, we re-perform existing methods with their corresponding codes available online. The hyperparameters (epoch, batch size, learning rate, and drop out) used in the existing methods experiments are the same with those shown in Table 3.

We evaluated the average mIoU of all test instances with other neural networks designed specifically for point cloud segmentation.

#### 3.c.3 Results and discussion

Experimental results for point cloud segmentation are summarized in Table 4. It is observed that our method has higher mIoU than the existing methods.

Here, we discuss the possible reasons of the improvement. We first focus on the comparison between our method and DGCNN [27]. DGCNN corresponds to a graph-based direct spatial method. In contrast to the direct method, our method can utilize the local structural information in the feature space, and it also collects the information of neighboring nodes through attention-based aggregation.

We also compare our method with transformer-based methods PointASNL [37] and PCT [39]. While the transformer basically has a large number of parameters, they are restricted to use only node-wise features and the single dot-production attention function to calculate attention weights. In contrast, SFAGC utilizes the local structural information in the feature space and the multiple-type attention functions.

### Ablation Study

We also perform extra 3D point cloud segmentation experiments to validate the effectiveness of the components in the SFAGC.

The hyperparameters are the same as those shown in Table 3. Here, we use some AGCs with different settings as follows:

1. **SFAGC**. This is the full SFAGC described in Section 3.
2. **SFAGC-nS**. To validate the effectiveness of the local structure projection aggregation and fusing part proposed in the Section 3.A, we discard this part from the SFAGC.
3. **SFAGC-nP**. To confirm the effectiveness of the position embedding proposed in Section 3.B, we bypass the position embedding part from the SFAGC.
4. **SFAGC-ndot and SFAGC-nsub**. To validate the effectiveness of the multi-type attention function proposed in Section 3.C, SFAGC-ndot is set as the SFAGC without the dot-production attention function, and SFAGC-nsub is set as the SFAGC without the subtraction attention function.

The architecture of the network and hyperparameters are the same as the previous experiment.

The results are summarized in Fig. 10. By comparing SFAGC with SFAGC-nS, use of the local structures in feature space increases 0.3 mIoU. In SFAGC vs. SFAGC-nP, the position-embedding phase increases 0.4 mIoU. In SFAGC vs. SFAGC-ndot, the multi-type attention function increases 0.2 mIoU. In SFAGC vs. SFAGC-nsub, the multi-type attention

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c} \hline \hline Method & mIoU & air- & bag & cap & car & chair & car- & guitar & knife & lamp & laptop & motor & mug & pistol & rocket & skate- & table \\  & & plane & & & & phone & & & top & & & & & board & \\ \hline PointNet [26] & 83.0 & 81.5 & 64.8 & 77.2 & 73.5 & 88.6 & 68.3 & 90.4 & 84.1 & 80.0 & 95.1 & 59.2 & 91.8 & 79.7 & 52.1 & 72.4 & 81.6 \\ Point- & 84.7 & 81.4 & 73.4 & 80.6 & **77.8** & 89.9 & 74.5 & 90.6 & 85.8 & 83.5 & 95.1 & **69.3** & **94.0** & 81.0 & 58.5 & **74.3** & 82.0 \\ DGCNN [27] & 85.0 & 82.6 & 79.8 & **85.3** & 76.9 & 90.4 & 77.1 & **91.0** & 86.9 & 84.0 & **95.6** & 61.5 & 93.0 & 79.9 & 58.2 & 73.7 & **83.2** \\ Point- & 84.6 & 82.4 & 80.3 & 83.2 & 76.8 & 89.9 & **80.6** & 90.8 & 86.7 & 83.2 & 95.3 & 60.1 & 93.5 & **81.6** & 59.1 & 73.7 & 82.3 \\ ASNL [37] & 84.7 & 83.6 & 67.6 & 83.6 & 75.4 & 90.1 & 74.5 & 90.8 & 85.8 & 82.1 & 95.4 & 64.0 & 92.1 & 81.1 & 56.2 & 72.5 & **83.2** \\ \hline
**SFAGC** & **85.5** & **83.7** & **80.9** & 83.5 & 77.4 & **90.5** & 76.2 & **91.0** & **88.7** & **84.2** & 95.5 & 67.4 & 93.7 & 81.1 & **59.4** & 74.1 & **83.2** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison results of the 3D point cloud segmentation on the ShapeNet. mIoU indicates the average mIoU of all test instances. The mIoU of each class is also shown. The results are obtained by experimenting with the same hyper-parameters.
function increases 0.3 mIoU. The effectiveness of the SFAGC modules was demonstrated in this study.

## VI Conclusion

In this paper, we propose a new attention-based graph convolution named SFAGC. It can better aggregate the structural information of the neighboring nodes in high-dimensional feature space by using local structure projection aggregation. It also can computes better attention weights by using multi-type attention functions simultaneously. Through experiments on point cloud classification and segmentation, our method outperforms existing methods.
