TransGNN: Harnessing the Collaborative Power of Transformers and Graph
  Neural Networks for Recommender Systems
Graph Neural Networks (GNNs) have emerged as promising solutions for
collaborative filtering (CF) through the modeling of user-item interaction
graphs. The nucleus of existing GNN-based recommender systems involves
recursive message passing along user-item interaction edges to refine encoded
embeddings. Despite their demonstrated effectiveness, current GNN-based methods
encounter challenges of limited receptive fields and the presence of noisy
"interest-irrelevant" connections. In contrast, Transformer-based methods excel
in aggregating information adaptively and globally. Nevertheless, their
application to large-scale interaction graphs is hindered by inherent
complexities and challenges in capturing intricate, entangled structural
information. In this paper, we propose TransGNN, a novel model that integrates
Transformer and GNN layers in an alternating fashion to mutually enhance their
capabilities. Specifically, TransGNN leverages Transformer layers to broaden
the receptive field and disentangle information aggregation from edges, which
aggregates information from more relevant nodes, thereby enhancing the message
passing of GNNs. Additionally, to capture graph structure information
effectively, positional encoding is meticulously designed and integrated into
GNN layers to encode such structural knowledge into node attributes, thus
enhancing the Transformer's performance on graphs. Efficiency considerations
are also alleviated by proposing the sampling of the most relevant nodes for
the Transformer, along with two efficient sample update strategies to reduce
complexity. Furthermore, theoretical analysis demonstrates that TransGNN offers
increased expressiveness compared to GNNs, with only a marginal increase in
linear complexity. Extensive experiments on five public datasets validate the
effectiveness and efficiency of TransGNN.
Peiyan Zhang, Yuchen Yan, Xi Zhang, Chaozhuo Li, Senzhang Wang, Feiran Huang, Sunghun Kim
# Can Transformer and GNN Help Each Other?

###### Abstract.

Although Transformer has achieved great success in natural language process and computer vision, it has difficulty generalizing to medium and large scale graph data for two important reasons: (i) High complexity. (ii) Failing to capture the complex and entangled structure information. In graph representation learning, Graph Neural Networks(GNNs) can fuse the graph structure and node attributes but have limited receptive fields. Therefore, we question that can we combine Transformer and GNNs to help each other? In this paper, we propose a new model named **TransGNN** where the Transformer layer and GNN layer are used alternately to improve each other. Specifically, to expand the receptive field and disentangle the information aggregation from edges, we propose using Transformer to aggregate more relevant nodes' information to improve the message passing of GNNs. Besides, to capture the graph structure information, we utilize positional encoding and make use of the GNN layer to fuse the structure into node attributes, which improves the Transformer in graph data. We also propose to sample the most relevant nodes for Transformer and two efficient samples update strategies to lower the complexity. At last, we theoretically prove that TransGNN is more expressive than GNNs only with extra linear complexity. The experiments on eight datasets corroborate the effectiveness of TransGNN on node and graph classification tasks.

Graph data, Graph Neural Networks, Transformer +
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: thanks: Both authors contributed equally to this research.
+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.

+
Footnote †: Both authors contributed equally to this research.