Multi-Behavior Graph Neural Networks for Recommender System
Recommender systems have been demonstrated to be effective to meet user's
personalized interests for many online services (e.g., E-commerce and online
advertising platforms). Recent years have witnessed the emerging success of
many deep learning-based recommendation models for augmenting collaborative
filtering architectures with various neural network architectures, such as
multi-layer perceptron and autoencoder. However, the majority of them model the
user-item relationship with single type of interaction, while overlooking the
diversity of user behaviors on interacting with items, which can be click,
add-to-cart, tag-as-favorite and purchase. Such various types of interaction
behaviors have great potential in providing rich information for understanding
the user preferences. In this paper, we pay special attention on user-item
relationships with the exploration of multi-typed user behaviors. Technically,
we contribute a new multi-behavior graph neural network (MBRec), which
specially accounts for diverse interaction patterns as well as the underlying
cross-type behavior inter-dependencies. In the MBRec framework, we develop a
graph-structured learning framework to perform expressive modeling of
high-order connectivity in behavior-aware user-item interaction graph. After
that, a mutual relation encoder is proposed to adaptively uncover complex
relational structures and make aggregations across layer-specific behavior
representations. Through comprehensive evaluation on real-world datasets, the
advantages of our MBRec method have been validated under different experimental
settings. Further analysis verifies the positive effects of incorporating the
multi-behavioral context into the recommendation paradigm. Additionally, the
conducted case studies offer insights into the interpretability of user
multi-behavior representations.
Lianghao Xia, Chao Huang, Yong Xu, Peng Dai, Liefeng Bo
# Multi-Behavior Graph Neural Networks for Recommender System

###### Abstract

Recommender systems have been demonstrated to be effective to meet user's personalized interests for many online services (_e.g._, E-commerce and online advertising platforms). Recent years have witnessed the emerging success of many deep learning-based recommendation models for augmenting collaborative filtering architectures with various neural network architectures, such as multi-layer perceptron and autoencoder. However, the majority of them model the user-item relationship with single type of interaction, while overlooking the diversity of user behaviors on interacting with items, which can be click, add-to-cart, tag-as-favorite and purchase. Such various types of interaction behaviors have great potential in providing rich information for understanding the user preferences. In this paper, we pay special attention on user-item relationships with the exploration of multi-typed user behaviors. Technically, we contribute a new multi-behavior graph neural network (MBRec), which specially accounts for diverse interaction patterns as well as the underlying cross-type behavior inter-dependencies. In the MBRec framework, we develop a graph-structured learning framework to perform expressive modeling of high-order connectivity in behavior-aware user-item interaction graph. After that, a mutual relation encoder is proposed to adaptively uncover complex relational structures and make aggregations across layer-specific behavior representations. Through comprehensive evaluation on real-world datasets, the advantages of our MBRec method have been validated under different experimental settings. Further analysis verifies the positive effects of incorporating the multi-behavioral context into the recommendation paradigm. Additionally, the conducted case studies offer insights into the interpretability of user multi-behavior representations. We release our model implementation at [https://github.com/akakih/MBRec](https://github.com/akakih/MBRec).

 Graph Neural Network, Recommender System, Collaborative Filtering, Multi-Behavior Recommendation

## I Introduction

With the growth of Internet services and mobile applications, recommender systems have played an increasingly critical role in addressing the information overload for many online platforms [56, 10, 49]. For example, the benefits of recommendation systems could lie in providing personalized recommendations in e-commerce sites (_e.g._, Amazon and Taobao), or satisfying users' interest in online music streaming services (_e.g._, Pandora and Spotify). Currently, collaborative filtering techniques serve as one of the most important paradigms to accurately understand the preferences of users, based on their interaction behaviors [57, 28].

With the remarkable success of deep learning, there exist renewed interests in modeling user-item interactions with various neural network architectures, such as multi-layer perceptron [11, 26], autoencoder network [25] and neural autoregressive models [58]. Built on the recent strength of graph neural networks, several studies seek to aggregate feature information from the graph-structured relational data generated by the observed user behaviors [34, 53]. These neural network models generally focus on single type of user interaction behaviors over items, during the vectorized representation procedure of users and items. However, in real-life applications, items are often interacted by users with diverse ways [7, 54, 51]. For example, users can view, tag-as-favourite and purchase different products in E-commerce platforms. In such real-life scenarios, effectively modeling of multi-typed user-item interactions can provide auxiliary knowledge to characterize the diverse user behavior semantics for interest representation in recommender systems [9, 36]. For simplifying the model design, the embedding functions in most existing recommendation models nearly ignore the explicit encoding of multi-behavior collaborative signals, which are insufficient to yield satisfactory latent representations for both users and items. In this paper, we tackle the multi-behavior recommendation by enhancing user preference learning with the exploration of multi-typed user behavior data.

Although it is desirable to consider the behavior diversity in user interest representation learning for accurate recommendations, it is not a trivial task to capture the complex multi-behavioral collaborative relations. In particular, each type of user behaviors has its own interaction contexts and there exist complex dependencies across various types of interactions. Different behavior views usually provide complementary information for encoding user's interests. Therefore, in order to learn meaningful behavior representations from multi-typed user-item interactions, an effective cross-type behavior dependency modeling is a necessity in solving the multi-behavior recommendation. In addition, in the view of user-item interaction graph, the effectiveness of exploring subgraph structures has been shown in recently emerged graph-based methods (_e.g._, PinSage [50] and NGCF [34]), with the consideration of high-hop neighbors. However, to design effective embedding function in the recommendation architecture for representation aggregations, it is crucial to expressively model the high-order multi-behavior patterns of users over the interaction graph structure from different propagation layers. We show the multi
behavior recommendation scenario with the illustrated examples in Figure 1. We can observe that user can interact with items with different behavior types (differentiated with weight lines), _e.g_., click, add-to-cart and purchase. In such cases, we generate a multi-behavior interaction graph to represent the diverse collaborative relations among users and items with the high-order multiplex connectivity information.

**Present Work.** Motivated by the aforementioned challenges, this work proposes a new recommendation framework: Multi-Behavior Graph Neural Network (MBRec) that explicitly incorporates multi-typed behavior context into the encoding of diverse user preference. In the proposed MBRec framework, we can capture the heterogeneous relationships across different types of user-item interactions. Specifically, to cope with the cross-type behavior dependencies, we propose an attention-enhanced graph neural network to preserve the high-order interaction patterns over the multiplex graph structures. Through this way, MBRec is able to preserve the fine-grained semantics of user-item relations and facilitate the modeling of diverse user interest. To differentiate the influences of different types of behaviors, a gated aggregation mechanism is developed to help fuse the contextual signals from different types of behaviors for a better embedding learning. In addition, we endow our MBRec with the capability of aggregating high-order behavior representations in an adaptive way. To achieve this goal, a mutual relation encoder is proposed to learn summarized representations across different graph layers. This component allows our model to better capture and interpret the global property of user-item interaction graph.

Lastly, it is worth mentioning that although the multi-behavior information has been considered in recent studies [7, 8], these work only consider the multi-behavior dependencies with the predefined correlations, which can hardly comprehensively capture the complex cross-type behavior dependencies in real-world recommendation scenarios. In addition, another study of multi-behavior recommendation model proposes to consider the dependencies between different interactions based on multi-channel graph convolutional network [16]. Different from these approaches, we contribute a new recommendation framework to explicitly exploit the high-order collaborative signals in the form of multi-behavior patterns. To enhance the global multi-behavior relation learning, we design a new graph aggregation scheme with order-wise mutual dependency modeling, which automatically differentiates the importance of behavior-aware representations from different graph hops during the message passing process. Lack of considering the complex global multi-behavior patterns could easily lead to suboptimal representations for user preference modeling.

In summary, we highlight our contributions as follows:

* We introduce a new multi-behavior recommendation framework, which explores the high-order and cross-type behavior inter-dependencies in a hierarchical manner.
* We propose a new recommendation framework that inherits the merits of graph-enhanced collaborative filtering paradigm and designs a multi-behavior propagation strategy for heterogeneous user-item interactions.
* Then, a graph-structured mutual relation encoder is developed to explore high-order user preference and promote the collaboration of different layer-specific patterns for robust multi-behavior representations. Furthermore, a graph sampling algorithm is developed to improve the scalability of MBRec for dealing with large-scale graph data.
* Experimental results on three real-world datasets show the superiority of our MBRec model over a variety of baselines. We further perform the model ablation study to better understand the effect of our designed sub-modules.

In this paper, we propose to advance our previous work [43] from the following aspects: i) Different from our previous method which overlooks the cross-layer implicit dependency between behavior representations, we propose a new recommendation framework to explicitly promote the cooperation of behavior patterns across different graph layers, for more accurate user and item representations (Section III). ii) We provide a comprehensive complexity analysis and computational cost evaluation of our model to show that our proposed model could achieve competitive time efficiency as compared to most state-of-the-art techniques (Section III-E and Section IV-J). iii) We further evaluate the model performance with respect to different sparsity levels of user-item interaction data, to justify the robustness of our multi-behavior recommendation method in capturing user's preference under different data sparsity degrees (Section IV-H). iv) We add three new recently developed baselines (_i.e_., MBGCN, MATN and NGCF+M) in our performance evaluation to show the superiority of our method (Section IV). Additionally, we present more hyperparameter study results with respect to the hidden state dimensionality and behavior embedding channels (Section IV-I). v) We perform case studies to show the interpretation capability of our approach in capturing the behavior relationships as well as the layer-wise representation dependence (Section IV-K). vi) We adopt two new datasets (BeiBei and IJCAI-contest) collected from real-world e-commerce platforms for performance evaluation across different experimental settings. The user-item interactions in online retailing systems are multiplex in nature and can well reflect the relation heterogeneity between users and items. vii) Finally, we present detailed discussion about the related work from three research lines: neural collaborative filtering techniques, recommendation with multi-behavior modeling and graph neural networks (Section V).

Figure 1: Illustration of the user-item multi-behavior interactions and the corresponding multi-behavior high-order connectivity. Best viewed in color.
## II Preliminaries

We begin by describing the multi-behavior recommendation and introducing key notations. Suppose we have a recommendation scenario with a set of users \(U\) (\(u_{i}\in U\)) and a set of items \(V\) (\(v_{j}\in V\)). Here, the index of user and item is denoted by \(i\) (\(i\in[1,...,I]\)) and \(j\) (\(j\in[1,...,J]\)), respectively. Different from most existing recommender systems which associate users with their interacted items based on singular type of user-item relations, this work explores the inter-dependent relations across different types of user-item behaviors (_e.g_., click, tag-as-favorite, review, like, or purchase).

Definition 1. **Multi-Behavior Interaction Tensor X**. To represent the multi-typed interactions between users and items, we define a three-way multi-behavior interaction tensor \(\textbf{X}\in\mathbb{R}^{I\times J\times K}\), where \(K\) (indexed by \(k\)) denotes the number of types of user-item interactions. Given the \(k\)-th type of interactions, the corresponding element \(x_{i,j}^{k}\in\textbf{X}\) is set to 1 if the item \(v_{j}\) has been adopted by user \(u_{i}\). Otherwise, \(x_{i,j}^{k}=0\).

**Multi-Behavior Recommendation**. In the recommendation scenario with multi-type interaction behaviors, we first define the target behavior type (_e.g_., \(k\)-th) as our predictive objective and consider other types (\(k^{\prime}\in[1,...,K]\&k^{\prime}\neq k\)) of interaction as auxiliary behaviors. In real-world recommendation scenarios, the target behavior type can be set according to different task-specific requirements. For example, some e-commerce systems may be more interested in ultimate purchase transactions [37], while forecasting click-interactive behavior is also very important for online advertising platforms [24]. We formally present the multi-behavior recommendation as:

**Input**: multi-behavior interaction tensor \(\textbf{X}\in\mathbb{R}^{I\times J\times K}\) which jointly includes source and target behavior of users in \(U\).

**Output**: A predictive framework which effectively forecasts the unknown target type of user-item interactions.

## III Methodology

In this section, we first elaborate the technical details of our proposed MBRec framework. Its key idea is to explore the complex inter-dependencies across different types of users' interactive behavior, to parameterize weight matrices for the relation heterogeneity aggregation, high-order message passing and propagation modules. Such process can be decomposed into two key components: (i) Multi-Behavior Graph-Structured Dependency Modeling: it jointly preserves the type-specific behavior semantics and type-wise behavior inter-dependencies within a graph-structured learning architecture. (ii) Cross-Layer Mutual Relation Learning: it captures the mutual relationships between the aggregated multi-hop feature representations of neighbors at different hops.

### _Multi-Behavior Graph Dependency Modeling_

Figure 2 presents the model flow of our multi-behavior graph dependency modeling. With the consideration of different types of user-item behavioral relations, we first construct the multi-behavior user-item interaction graph.

Definition 2. **Multi-Behavior Graph**\(G\). Based on the input multi-behavior interaction tensor **X**, we generate a graph \(G=\{U,V,\mathcal{E}\}\) where user node \(u_{i}\in U\) and item node \(v_{j}\in V\) is connected with the edge \(e_{i,j,k}\in\mathcal{E}\), if \(u_{i}\) interacts with \(v_{j}\) under the \(k\)-th behavior type (_i.e_., \(x_{i,j}^{k}=1\)). Each edge \(e_{i,j,k}\in\mathcal{E}\) is associated with a specific behavior type of \(k\). Due to the interaction heterogeneity property, there exist multiplex edges between the same user-item pair given \(u_{i}\) interacts with \(v_{j}\) under multiple behavior types.

#### Iii-A1 **Behavior-aware Message Construction**

Based on the multi-behavior graph \(G\), we first generate the propagated information for the user(\(u_{i}\))-item(\(v_{j}\)) pair with the following behavior-aware message passing paradigm:

\[\textbf{H}_{i\leftarrow}^{k,(l+1)}=\lambda(\{\textbf{E}_{j}^{(l)}: x_{i,j}^{k}=1\})\] \[\textbf{H}_{j\leftarrow}^{k,(l+1)}=\lambda(\{\textbf{E}_{i}^{(l)}: x_{i,j}^{k}=1\}) \tag{1}\]

where \(\lambda(\cdot)\) denotes the behavior-aware encoding function for preserving the semantics of individual type of interactive behavior for \(u_{i}\) and \(v_{j}\). Here, \(\textbf{H}_{i\leftarrow}^{k,(l+1)}\in\mathbb{R}^{d}\) and \(\textbf{H}_{j\leftarrow}^{k,(l+1)}\in\mathbb{R}^{d}\) with dimensionality of \(d\) (output from \(\lambda(\cdot)\) function) are the \((l+1)\)-th layer representation which preserves the characteristics of the behavior type of \(k\). In the first-layer propagation, we generate the input feature vector \(\textbf{E}_{i}^{0}\) and \(\textbf{E}_{j}^{0}\) of user \(u_{i}\) and item \(v_{j}\) with the Autoencoder-based pre-training [25] over multi-behavior interaction tensor **X**, to project different types of high-dimensional behavior embeddings into low-dimensional latent space. The reasons for using Autoencoder

\begin{table}
\begin{tabular}{c|c} \hline Notations & Description \\ \hline \(U\) (\(u_{i}\in U\)) & Set of users \\ \(V\) (\(v_{i}\in V\)) & Set of items \\ \(K\) (indexed by \(k\)) & Number of behavior types \\ \(\textbf{X}\in\mathbb{R}^{I\times J\times K}\) & Multi-behavior interaction tensor \\ \(\textbf{H}_{i\leftarrow}^{k,(l)}\), \(\textbf{H}_{j\leftarrow}^{k,(l)}\) & Behavior-aware representations of users/items \\  & Behavior-aware encoding function \\ \(\hat{\textbf{H}}_{i\leftarrow}^{k,(l)}\) & Recharlted type-specific behavior embedding \\ \(\hat{\textbf{\beta}}_{k}\) & Attention value for type-specific behavior \\ \(\psi^{(l)}\) & Cross-behavior message aggregation function \\ \(G=\{U,V,\mathcal{E}\}\) & Multi-behavior high-order graph \\ \(\hat{\textbf{E}}_{i}^{(l)}\), \(\hat{\textbf{E}}_{j}^{(l)}\) & Cross-layer fused embedding \\ \(\Gamma_{i,j}\) & Fused representation for prediction \\ \hline \end{tabular}
\end{table} TABLE I: Summary of Key Notations

Fig. 2: Multi-behavior graph dependency modeling.
is to generate more informative initial embeddings for users and items, through the auto-encoding training paradigm.

In the multi-behavior recommendation, various types of users' behaviors reflect the preference from behavior-specific characteristics. For example, view-interactive behavior happens more frequently than add-to-cart and add-to-favorite activities [9]. Additionally, add-to-favorite interactive behavior could provide rich information to characterize users' implicit interests over items, although the purchase behavior may be postponed and does not happen right away [2]. Hence, we design our semantic encoding function \(\lambda(\cdot)\), to capture the type-specific behavior contextual information in the message construction process.

**Multi-Channel Behavior Embedding Layer**. Inspired by recent advancement of memory-augmented neural network models in multi-dimensional context learning [33], we build our behavior semantic encoder upon the multi-channel neural framework to learn customized representations for individual type of user-item interactions. Our multi-channel embedding layer utilizes the external representation units, in which each unit corresponds to a certain dimensional of behavior semantics (_e.g_., behavior data distributions with respect to different categories of items). Specifically, we integrate the channel-based projection layer with the behavior-aware attention mechanism to fuse the learned semantic information across different channels. For each behavior type \(k\), this module is firstly equipped with a contextual transformation layer to update the input user/item embeddings \(\mathbf{E}_{j}^{(l)}\) and \(\mathbf{E}_{j}^{(l)}\) in the \(l\)-th graph layer. Without loss of generality, we formally present the behavior semantic encoder with \(M\) (indexed by \(m\)) channels for the message of user \(u_{i}\) from his/her connected item nodes \(\{v_{j}|x_{i,j}^{k}=1\}\) under the behavior type of \(k\) as below:

\[\mathbf{H}_{i\leftarrow}^{k,(l+1)} =\sum_{m=1}^{M}\omega_{m}^{k}\mathbf{U}_{m}\sum_{x_{i,j}^{k}=1} \mathbf{E}_{j}^{(l)}\] \[\omega_{m}^{k} =\delta(\mathbf{K}\cdot\sum_{x_{i,j}^{k}=1}\mathbf{E}_{j}^{(l)} +\mathbf{b})(m) \tag{2}\]

where \(\mathbf{U}_{m}\in\mathbb{R}^{d\times d}\) is the \(m\)-th channel transformation, \(\omega_{m}^{k}\) is the \(m\)-th weight calculated from the neighboring nodes \(\{v_{j}|x_{i,j}^{k}=1\}\) of \(u_{i}\) under behavior type of \(k\). \(\sum_{m=1}^{M}\omega_{m}^{k}\mathbf{U}_{m}\) represents the learned behavior-type-specific contextual transformation. Additionally, \(\mathbf{K}\in\mathbb{R}^{M\times d}\) and \(\mathbf{b}\in\mathbb{R}^{M}\) are transformation and bias parameters to calculate the weights, and \(\delta(\cdot)\) denotes ReLU activation. Similar semantic encoder can be applied to learn embedding \(\mathbf{H}_{j\leftarrow}^{k,(l+1)}\) between item \(v_{j}\) and its connected users \(\{u_{i}|x_{i,j}^{k}=1\}\) with the \(k\)-th behavior type. By performing the dimension-wise relation learning for each type of behavior patterns, the underlying semantic information can be smoothly captured with multi-channel behavior embedding in our framework.

#### Iii-B2 **Behavior Inter-dependency Modeling**

In addition to encoding the type-specific behavior semantics, another key aspect of multi-behavior relation learning lies in exploring the inter-dependencies across different types of user-item interactive behavior. For instance, add-to-cart and tag-as-favorite activities are good indicators for purchase of users. Hence, after the message propagating process with the learned representations \(\mathbf{H}_{i\leftarrow}^{k,(l+1)}\) and \(\mathbf{H}_{j\leftarrow}^{k,(l+1)}\), it is crucial to consider the inter-dependencies among different types of behaviors (_i.e_., \(k\in[1,...,K]\)), and further refine the type-specific behavior embeddings propagated from other behavior types.

Motivated by the recent success of transformer networks in distilling inter-correlations between entities [52], we develop a self-attentive network for multi-behavior inter-dependency modeling. Based on the paradigm of self-attention layer with scaled dot-product attention, we create three transformation matrices to project input representation into three latent dimensions, _i.e_., \(\mathbf{Q}^{h}\in\mathbb{R}^{\frac{d}{6}\times d}\), \(\mathbf{V}^{h}\in\mathbb{R}^{\frac{d}{6}\times d}\) and \(\mathbf{K}^{h}\in\mathbb{R}^{\frac{d}{6}\times d}\) as the query, value and key transformations for each of the \(C\) (indexed by \(c\)) attention heads. Then, the dependency scores between the \(k\)-th and the \(k^{\prime}\)-th behavior message is calculated in the dot-product manner as follows:

\[\alpha_{k,k^{\prime}}^{c} =\frac{(\mathbf{Q}^{c}\mathbf{H}_{i\leftarrow}^{k,(l+1)})^{\top} \cdot(\mathbf{K}^{c}\mathbf{H}_{i\leftarrow}^{k^{\prime},(l+1)})}{\sqrt{\frac {d}{C}}}\] \[\hat{\alpha}_{k,k^{\prime}}^{c} =\frac{\exp\alpha_{k,k^{\prime}}^{c}}{\sum_{k^{\prime}=1}^{k,(l+ 1)}\exp\alpha_{k,k^{\prime}}^{c}} \tag{3}\]

where \(\hat{\alpha}_{k,k^{\prime}}^{c}\) is the learned quantitative correlation weight between the behavior type \(k\) on \(k^{\prime}\), which is calculated from the intermediate score \(\alpha_{k,k^{\prime}}^{c}\) by the softmax function. To expand the model ability in capturing cross-behavior dependency from different hidden dimensions (_e.g_., co-occurrence frequencies and regularties), we augment the self-attention layer with the multi-head mechanism to endow the attentive relation learning under multiple representation subspaces. In such cases, we associate each head-specific attention component with the a set of Query (\(\mathbf{Q}^{c}\)), Key (\(\mathbf{K}^{c}\)) and Value (\(\mathbf{V}^{c}\)) weight matrices. Based on the head-specific correlation scores, we then recalibrate the type-specific behavior message by concatenating the multiple attention heads:

\[\tilde{\mathbf{H}}_{i\leftarrow}^{k,(l+1)} =\text{MH-Att}(\mathbf{H}_{i\leftarrow}^{k,(l+1)})\] \[=\left|\left|\sum_{c=1}^{K}\hat{\alpha}_{k^{\prime}=1}^{h} \mathbf{V}^{c}\cdot\mathbf{H}_{i\leftarrow}^{k^{\prime},(l+1)}\right. \tag{4}\]

where \(\left|\left|\right.\right.\) denotes the vector concatenation, and \(\tilde{\mathbf{H}}_{i}^{k,(l+1)}\) is the recalibrated message propagated to the target node \(u_{i}\) under behavior type \(k\). To preserve the original type-specific behavior message and prevent the gradient vanishing issue, we perform the element-wise addition between the original and recalibrated type-specific message with a residual connection scheme. That is, \(\tilde{\mathbf{H}}_{i\leftarrow}^{k,(l+1)}=\tilde{\mathbf{H}}_{i\leftarrow}^{k,(l+1)}+\mathbf{H}_{i\leftarrow}^{k,(l+1)}\) is the refined type-specific behavior representation which preserves the cross-type behavior inter-dependent information.

#### Iii-B3 **Personalized Multi-Behavior Aggregation**

With the incorporation of both the type-specific behavior semantics and type-wise inter-correlations into our multi-behavior dependency modeling module, we introduce an aggregation layer to fuse interactive patterns across different types of behaviors. Formally, we define the message aggregation function as:

\[\mathbf{E}_{i}^{(l+1)}=\psi(\{\tilde{\mathbf{H}}_{i\leftarrow}^{k,(l+1)}:k=[1,2,...,K]\}) \tag{5}\]
In real-life online platforms (_e.g._, online retailing or review sites), item interactive patterns may vary by users due to their different behavior preferences. For example, some people prefer to add their interested items into the favorite list but sporadically make purchases, while the add-to-favorite action is more likely to be followed by the purchase behavior. Therefore, to aggregate message from different types of behavior embeddings and obtain expressive representations on the local user-item multi-behavior interaction graph, it is essential to identify the contribution of different types of behavior in assisting the final prediction on the target type of user behavior in a customized manner.

Towards this end, with the recalibrated type-specific behavior message \(\hat{\mathbf{H}}_{i\leftarrow}^{k,(i+1)}\), the multi-behavior graph encoder is further equiped with an attention network to differentiate the influences of different behavior types for user \(u_{i}\). Specifically, the message aggregation module adopts a two-layer feed-forward network for weight estimation:

\[\beta_{k} =\mathbf{w}_{2}^{\top}\cdot\delta(\mathbf{W}_{1}\hat{\mathbf{H}} _{i\leftarrow}^{k,(l+1)}+\mathbf{b}_{1})+b_{2}\] \[\hat{\beta}_{k} =\frac{\exp\beta_{k}}{\sum_{k^{\prime}=1}^{K}\exp\beta_{k^{\prime}}} \tag{6}\]

where \(\mathbf{W}_{1}\in\mathbb{R}^{d^{\prime}\times d}\), \(\mathbf{b}_{1}\in\mathbb{R}^{d^{\prime}}\), \(\mathbf{w}_{2}\in\mathbb{R}^{d^{\prime}}\), and \(b_{2}\in\mathbb{R}\) are transformations and bias vectors for weights calculation, and \(d^{\prime}\) is the dimensionality of the intermediate hidden layer. \(\delta\) is the ReLU activation function. \(\beta_{k}\) is the intermediate attention value and \(\hat{\beta}_{k}\) is the final influence weight of interaction type \(k\) calculated using softmax function. With the calculated weights, the message aggregation layer then applies perform the weighted summation over type-specific behavior embeddings \(\mathbf{H}_{i\leftarrow}^{k,(l+1)}\), so as to acquire the encoded node embeddings \(\mathbf{E}_{i}^{(l+1)}=\sum_{k=1}^{K}\hat{\beta}_{k}\hat{\mathbf{H}}_{i \leftarrow}^{k,(l+1)}\) corresponding to the \(l\)-th layer graph encoder. With the developed multi-behavior modeling component, we endow the MBRec with the capability of modeling of type-specific behavioral semantics and cross-type dependencies.

### _High-order Mutual Relation Learning_

#### Iii-B1 **High-order Multi-Behavior Pattern Propagation**

With the multi-behavior pattern representation performed by first-order dependency learning, we can enable the high-order multi-behavior relational structures on the global graph \(G=\{U,V,\mathcal{E}\}\) by stacking more embedding propagation layers (as shown in Figure 3). Based on our developed multi-behavior graph encoder, in the high-order information propagation process, MBRec is capable of capturing the high-order collaborative signals across different types of user-item interactive relations. After performing the multi-behavior graph encoder \(l\) times, each node (_i.e._, \(u_{i}\) or \(v_{j}\)) could receive the messages propagated from its neighbors with \(l\)-th hop distance, which is presented as follows:

\[\mathbf{E}_{u}^{(l+1)} =\psi(\text{MH-Att}(\lambda(\{\mathbf{E}_{j}^{(l)}:x_{i,j}^{k}=1 \})))\] \[=\sum_{k=1}^{K}\hat{\beta}_{k}\cdot\text{MH-Att}(\mathbf{X}^{k} \mathbf{E}_{v}^{(l)}\sum_{m=1}^{M}\omega_{m}^{k}\mathbf{U}_{m}^{\top}) \tag{7}\]

where \(\mathbf{X}^{k}\in\mathbb{R}^{I\times J}\) is the adjacent matrix under the \(k\)-th behavior type, \(\mathbf{E}_{u}^{(l+1)}\in\mathbb{R}^{I\times d}\) and \(\mathbf{E}_{v}^{(l)}\in\mathbb{R}^{J\times d}\) refer to the embedding matrix of users and items, respectively.

While the high-order connectivity is exploited in the multi-behavior graph \(G\), the message propagation process with the rigid order hinders the representation power in learning the global graph-structured behavior dependencies. In specific, the current graph-based message passing architecture maps the input multi-behavior user-item interactions into a fixed length embedding with the highest \(L\)-th layer, which results in the information loss of previous layer-specific aggregated patterns. For example, a user can interact with different items with direct (one-hop) or indirect (multi-hop) connected relationships. In such case, it is crucial to identify the most relevant signals across both low- and high-order connections, for characterizing the preference of this target user. To overcome this limitation and free the message passing graph neural architecture from fixed-length internal representation, we introduce a cross-layer mutual relation learning module, to endow our graph neural architecture MBRec with the capability to focus on certain parts of graph-structured behavioral patterns across different-order connectivity.

#### Iii-B2 **Cross-Layer Mutual Relation Modeling**

To remedy the existing shortcomings, we propose to model the importance between users' and items' multi-order embeddings with a multi-head self-attention network. For user \(u_{i}\)'s multi-order embeddings \(\mathbf{E}_{i}^{(0)},...,\mathbf{E}_{(1)}^{(1)},...,\mathbf{E}_{(L)}^{(L)}\) (\(\mathbf{E}_{1}^{(1)}\in\mathbb{R}^{d}\)) and item \(v_{j}\)'s embeddings \(\mathbf{E}_{j}^{(0)},...,\mathbf{E}_{j}^{(1)},...,\mathbf{E}_{j}^{(L)}\) (\(\mathbf{E}_{j}^{(1)}\in\mathbb{R}^{d}\)) learned by the stacked multi-behavioral graph encoders, we first normalize the multi-order embeddings for cross-layer embedding fusion with the following operation:

\[\hat{\mathbf{E}}_{i}^{(l)}=\frac{\mathbf{E}_{i}^{(l)}}{\sqrt{\|\mathbf{E}_{i}^ {(l)}\|_{2}^{2}}};\ \ \ \hat{\mathbf{E}}_{j}^{(l)}=\frac{\mathbf{E}_{j}^{(l)}}{\sqrt{\|\mathbf{E}_{j}^{(l )}\|_{2}^{2}}} \tag{8}\]

Based on the block of multi-head self-attention mechanism, we calculate \(C\) (corresponds to \(C\) heads) importance matrices \(\phi^{c}\in\mathbb{R}^{(L+1)\times(L+1)}\) for adaptive multi-order combination.

Fig. 3: High-order propagation with layer-wise mutual relation learning for multi-behavior representations.
With the design of 2-D matrix \(\phi^{c}\), we can capture the pairwise mutual relation across different layer-specific multi-behavior patterns. We present this process as follows:

\[\phi^{c}_{l,l^{\prime}}=\delta((\textbf{P}^{c}\hat{\textbf{E}}^{l}_{i})^{\top} \cdot(\textbf{P}^{c}\hat{\textbf{E}}^{l^{\prime}}_{j})) \tag{9}\]

where \(\phi^{c}_{l,l^{\prime}}\) is the \(c\)-th importance score for the combination between the \(l\)-th layer user embedding and \(l^{\prime}\)-th layer item embeddings. \(\textbf{P}^{c}\in\mathbb{R}^{\frac{d}{\mathcal{N}}\times d}\) is the transformation to acquire key vectors, and \(\delta(\cdot)\) is the ReLU activation function to learn non-linearities of feature interactions. Based on the learned importance scores, we then generate the fused representation for prediction as follows:

\[\boldsymbol{\Gamma}_{i,j}=\Big{|}\Big{|}\sum_{c=1}^{C}\sum_{l=0}^{L}\sum_{l^{ \prime}=0}^{L}\phi^{h}_{l,l^{\prime}}(\textbf{T}^{c}\hat{\textbf{E}}^{(l)}_{i })\circ(\textbf{T}^{c}\hat{\textbf{E}}^{(l^{\prime})}_{j}) \tag{10}\]

where \(\circ\) denotes the element-wise production, and \(\textbf{T}^{c}\in\mathbb{R}^{\frac{d}{\mathcal{N}}\times d}\) is the value transformation. The \(C\) head-specific representations are concatenated to generate the fused representation \(\boldsymbol{\Gamma}_{i,j}\in\mathbb{R}^{d}\), which is fed into a feed-forward network to make forecasting on the unknown user-item interaction with the target behavior type \(k\):

\[\textbf{Pr}_{i,j}=\textbf{w}_{4}^{\top}(\delta(\textbf{W}_{3}\boldsymbol{ \Gamma}_{i,j}+\textbf{b}_{3})+\boldsymbol{\Gamma}_{i,j}) \tag{11}\]

where \(\textbf{W}_{3}\in\mathbb{R}^{d\times d}\), \(\textbf{w}_{4}\in\mathbb{R}^{d}\) and \(\textbf{b}_{3}\in\mathbb{R}^{d}\) are network parameters, \(\delta(\cdot)\) is the ReLU activation. Note that a residual connection is employed for better gradients propagation.

``` Input: seed users \(\mathbb{U}\), seed items \(\mathbb{V}\), adjacent tensor \(\textbf{X}\in\mathbb{R}^{l\times J\times K}\), sampling depth \(D\), sampling number per step \(N\) Output: sampled users \(\mathbb{U}\), sampled items \(\hat{\mathbb{V}}\), adjacent matrix of the sampled sub-graph \(\hat{\textbf{X}}\)
1 Initialize the normalized adjacent matrix \(\bar{\textbf{X}}\in\mathbb{R}^{l\times J}\) with \(\bar{\textbf{X}}_{i,j}=\frac{\|\textbf{X}_{i,j}\|_{1}}{\sqrt{\|\textbf{X}_{i }\|_{1}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2} \|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2} \|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2}\|_{2} \|_{2}
and the cost for model training and inference. As described in Algorithm 1, the major cost of the former process is \(O(D\times N\times(I+J))\) for updating the sampling probability \(P_{u}\) and \(P_{v}\), where \(D\) is the number of sampling steps and \(N\) is the number of sampled nodes per step. In the model running phase, MBRec takes \(O(L\times|\mathbf{X}|\times d)\) (\(|\mathbf{X}|\) denotes the number of non-zero elements in \(\mathbf{X}\)) to encode the type-specific message, in which \(O(L\times(I+J)\times K\times d^{2})\) is needed by the attention module. The complexity of the type-wise inter-dependency modeling and the aggregation layer is analogously \(O(L\times(I+J)\times K\times d^{2})\), in which the primary contributor is the matrix-multiplications. The complexity of the cross-order mutual relation learning is \(O(|\mathbf{X}|\times L^{2}\times d^{2})\) which comes from the order-wise representation fusion, and this term dominates the complexity of the model running process. Empirically, by sharing the sampled sub-graph among training/testing steps, the sub-graph sampling costs much less time compared to the entire computational cost.

```
Input: multi-behavior interaction tensor \(\mathbf{X}\in\mathbb{R}^{I\times J\times K}\), initial node embeddings \(\mathbf{E}^{(0)}\), the number of graph layer \(L\), the number samples for training \(S\), weight \(\lambda\) for regularization, the number of epochs \(E\) Output: trained model parameters \(\mathbf{\Theta}\)
1 hyperparameter Initializations \(\mathbf{\Theta}\)
2for\(e=1\) to \(E\)do
3 Seed node sampling \(\mathbb{U}\), \(\mathbb{V}\) sub-graph generation \((\tilde{\mathbb{U}},\tilde{\mathbb{V}},\tilde{\mathbf{X}})\) using the seeds according to Algorithm 1
4 Get \(\mathbf{E}^{(0)}\) from \(\tilde{\mathbf{E}}^{(0)}\) for \(u_{i}\) in \(\mathbb{U}\) and \(v_{j}\) in \(\mathbb{V}\)
5for\(l=1\) to \(L\)do
6foreach\(u_{i}\) in \(\hat{\mathbb{U}}\), \(v_{j}\) in \(\hat{\mathbb{V}}\) and \(k=1\) to \(K\)do
7 Type-specific behavior message generation \(\mathbf{H}^{k}\)
8 Message refinement \(\mathbf{\hat{H}}^{k}\)
9 Embedding aggregation \(\mathbf{E}^{(l)}\)
10
11 end for
12
13 end for
14\(\mathcal{L}=\lambda\|\mathbf{\Theta}_{\mathbf{\Theta}}^{2}\|\)
15for each\(u_{i}\) in \(\hat{\mathbb{U}}\)do
16 Positive and negative items are sampled from \(\hat{\mathbb{V}}\)
17for each\(v_{p_{s}}\) and \(v_{n_{s}}\)do
18 Interaction probability inference \(\text{Pr}_{i,j}\).
19\(\mathcal{L}+=\max(1-\text{Pr}_{t,p_{s}}+\text{Pr}_{t,n_{s}})\)
20 end for
21
22 end for
23 Model training with the optimized objective. \(\mathcal{L}\)
24 end for
25return\(\mathbf{\Theta}\)
```

**Algorithm 2**Model Optimization of MBRec

**Space Complexity.** Due to sampling larger sub-graphs for computing efficiency and data integrity, MBRec takes more memory than some GNN model in sub-graph sampling. But the memory cost is fully acceptable for common devices. For the model memory cost, the space complexity of MBRec is \(O(L\times(I+J)\times K\times d)\), which is mainly for the intermediate hidden states, the same as a common graph neural networks (_e.g_. GCN and GraphSAGE) for modeling multi-behavior data.

## IV Evaluation

We evaluate MBRec to answer the research questions as:

* **RQ1**: How does our _MBRec_ perform compared with various recommendation baselines on different datasets?
* **RQ2**: How does each model design (_e.g_., multi-channel behavior embedding layer and cross-layer mutual relation learning module) affect the model performance?
* **RQ3**: What is the impact of incorporating different types of behaviour context in our graph neural multi-behavior recommender system?
* **RQ4**: How do different interaction sparsity degrees influence the recommendation performance?
* **RQ5**: How do the key hyperparameters impact the performance of _MBRec_ neural architecture?
* **RQ6**: How is the model efficiency of _MBRec_ when competing with various types of recommendation techniques?
* **RQ7**: How does the user multi-behavior dependency study benefit the interpretation ability for recommendation?
* **RQ8**: What is the effect of the graph sampling algorithm on the model performance of _MBRec_?

### _Data Description_

Our evaluations are performed on three real-world datasets: Tmall, BeiBei and IJCAI-Competition. We summarize the detailed statistical information of those datasets in Table II.

* **Tmall**. This is a public recommendation dataset from the Tmall e-commerce platform by including four types of user behaviors: click, add-to-cart, tag-as-favorite and purchase. This data contains 47,894 users and 99,037 items.
* **BeiBei**. This is another e-commerce dataset for item recommendation from one of the largest infant product retail site in China. There are 21,716 users and 7,977 items in this dataset with three types of user-item interactions, namely, click, add-to-cart and purchase.
* **IJCAI-Competition**. This data comes from the released repository of IJCAI competition to provide researchers with user online behavior modeling. It involves four types of interaction behavior between user and item, _i.e_., click, add-to-cart, tag-as-favorite and purchase. 423,423 users and 874,328 items are included in this data source.

To be consistent with the settings in [44, 16], the target predicted behaviors in our recommendation scenario are user purchases and other types of behaviors (_e.g_., click, add-to-cart) are regarded as the auxiliary behaviours.

### _Evaluation Metrics_

In our experiments, the models are evaluated on the top-\(N\) item recommendation task with the metrics of Hit Ratio (HR)@\(N\) and NDCG@\(N\). In our evaluation protocol, we use the leave one item out strategy [55] to consider the last

\begin{table}
\begin{tabular}{l l l l} \hline \hline Dataset & User \# & Item \# & Interaction \# & Interactive Behavior Type \\ \hline Tmall & 147894 & 99037 & 7658926 & \{Page View, Favorite, Cart, Purchase\} \\ BeiBei & 21716 & 7977 & 3333086 & \{Page View, Cart, Purchase\} \\ DeCI & 423423 & 874328 & 36203512 & \{Page View, Favorite, Cart, Purchase\} \\ \hline \hline \end{tabular}
\end{table}
Table II: Statistics of our evaluation datasets.
interaction with the target behavior of each user as the testing set. In particular, following the similar settings in [17, 29], for individual user, we sample 99 items as negative instances from the set of all non-interacted items. Items in the test set are regarded as the positive instances.

### _Baseline Models_

To demonstrate the effectiveness of our _MBRec_ framework, we compare our method with the following state-of-the-art methods, which involves different categories:

**Conventional Matrix Factorization Method**:

* **BiasMF**[19]: this model attempts to incorporate user and item bias information into the matrix factorization, so as to learn latent embeddings of users/items.

**Neural Collaborative Filtering**:

* **NCF**[11]: it augments the embedding paradigm in collaborative filtering with the multilayer perceptron to enable the non-linear feature interactions.
* **DMF**[47]: this is another neural collaborative filtering technique, to learn a common low dimensional space for users and items with non-linear transformations.

**Autoencoder-based Recommendation Models**:

* **AutoRec**[25]: this recommendation model stacks multiple autoencoder layers to project user-item interaction inputs into the latent representations for data reconstruction.
* **CDAE**[41]: It is a model-based CF recommender with the denoising auto-encoder technique to learn user correlations.

**Neural Auto-regressive Recommender Systems**:

* **NADE**[58]: it designs a neural autoregressive architecture for recommendation task with the parameter sharing between different ratings.
* **CF-UiCA**[6]: this is an user-item co-autoregressive framework with a new stochastic learning strategy to encode correlations between users and items.

**Graph Neural Network-based Recommendation Methods**:

* **ST-GCN**[53]: this graph-based method is built over an encoder-decoder framework to perform the convolution-based embedding propagation between user and item nodes.
* **NGCF**[34]: it is a state-of-the-art GNN-based collaborative filtering model which exploits the high-order user-item interaction structures.

**Multi-Behavior Recommender Systems**:

* **NMTR**[7]: This method relies on the defined cascaded behavior relationships for encoding the multi-behavior semantics with a multi-task learning scheme.
* **DIPN**[9]: this deep intent prediction network aims to integrate the browsing and buying preferences of users with a new type touch-interactive behavior patterns.
* **NGCF+M**[34]: we generate a new multi-behavior recommendation variant of NGCF by injecting the multi-behavior context into the message passing scheme.
* **MATN**[44]: this recommendation model considers the influences among different types of interactions with attentive weights for pattern aggregation.
* **GNMR**[43]: this is the previous version of our _MBRec_ which captures the pairwise dependencies between different types of behaviors with the integration of the multi-channel behavior representation layer and self-attention network for relation aggregation. However, it ignores the layer-wise embedding dependency during the representation integration.
* **MBGCN**[16]: this multi-behavior recommender system leverages the graph convolutional network to capture the multi-behaviour patterns over the interaction graph.

### **Parameter Settings**

Our _MBRec_ model is implemented with TensorFlow. The parameter inference is conducted with the Adam optimizer and the training phase is performed with the learning rate of \(1e^{-3}\) and batch size of 32. For the model hyperparameters, the dimensionality of hidden state \(d\) is set as 16 in our representation space. The number of channels for behavior embedding layer is set as 8. We use 2 attention-based representation heads in our behavior inter-dependency modeling component. To alleviate the overfitting issue, the regularization strategy with the weight decay parameter sampled from {0.05, 0.01, 0.005, 0.001}.

### _Performance Comparison (RQ1)_

The evaluation results (measured by HR@10 and NDCG@10) of all compared methods on three datasets are shown in Table III. In all cases, we could observe that _MBRec_

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{\(\overline{o_{5}}\)} & \multicolumn{2}{c|}{\(\overline{o_{10}}\)} & \multicolumn{2}{c|}{\(\overline{o_{20}}\)} & \multicolumn{2}{c|}{\(\overline{o_{50}}\)} \\ \cline{2-10}  & HR & NDCG & HR & NDCG & HR & NDCG & HR & NDCG & HR & NDCG \\ \hline \hline BiashMF & 0.453 & 0.287 & 0.588 & 0.333 & 0.678 & 0.357 & 0.807 & 0.379 \\ \hline NCF & 0.447 & 0.283 & 0.601 & 0.336 & 0.698 & 0.359 & 0.819 & 0.383 \\ \hline NCF & 0.471 & 0.496 & 0.337 & 0.634 & 0.372 & 0.743 & 0.381 & 0.872 & 0.407 \\ \hline NDCG & 0.498 & 0.337 & 0.642 & 0.376 & 0.740 & 0.398 & 0.902 & 0.429 \\ \hline AutoRec & 0.456 & 0.291 & 0.607 & 0.341 & 0.707 & 0.366 & 0.826 & 0.391 \\ \hline MATN & 0.467 & 0.330 & 0.625 & 0.385 & 0.667 & 0.342 & 0.833 & 0.396 \\ \hline _MBRec_ & **0.527** & **0.389** & **0.670** & **0.402** & **0.788** & **0.433** & **0.927** & **0.461** \\ \hline \end{tabular}
\end{table} TABLE IV: Recommendation accuracy with different Top-\(N\) values in terms of _HR@N_ and _NDCG@N_ on BeiBei dataset.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{Data} & Metric & BiasMF & DMF & NCF & AutoRec & CDAE & NADE & CF-UiCA & ST-GCN & NCF & NMTR & DIPN & NCF+M & MBGCN & MATN & GNMR & _MBRec_ \\ \hline \multirow{4}{*}{Bei} & HR & 0.588 & 0.597 & 0.959 & 0.607 & 0.608 & 0.608 & 0.610 & 0.609 & 0.611 & 0.613 & 0.631 & 0.634 & 0.642 & 0.626 & 0.631 & **0.670** \\ \cline{2-16}  & Imprw & 13.957 & 12.529 & 12.614 & 10.386 & 10.209 & 0.1026 & 0.296 & 0.606 & 3.906 & 0.184 & 5.686 & 4.366 & 7.039 & 6.188 & - \\ \cline{2-16}  & NDCG & 0.333 & 0.336 & 0.332 & 0.341 & 0.341 & 0.343 & 0.346 & 0.343 & 0.375 & 0.349 & 0.384 & 0.372 & 0.376 & 0.385 & 0.380 & **0.402** \\ \cline{2-16}  & Imprw & 20.725 & 19.642 & 12.087 & 17.897 & 17.896 & 17.209 & 16.188 & 17.709 & 7.200 & 15.199 & 4.699 & 8.066 & 6.915 & 4.425 & 5.799 & - \\ \cline{2-16}  & HR & 0.262 & 0.305 & 0.319 & 0.313 & 0.329 & 0.317 & 0.332 & 0.347 & 0.302 & 0.332 & 0.317 & 0.354 & 0.369 & 0.354 & 0.424 & **0.444** \\ \hline \multirow{4}{*}{Transl} & Imprw & 69.474 & 35.746 & 39.184 & 41.85\% & 34.95\% & 40.06\% & 33.73\% & 27.95\% & 47.02\% & 33.73\% & 40.06\% & 18.72\% & 20.33\% & 25.425 & 4.726 & - \\ \cline{2-16}  & NDCG & 0.153 & 0.1189 & 0.191 & 0.190 & 0.191 & 0.198 & 0.206 & 0.188 & 0.179 & 0.178 & 0.221 & 0.222 & 0.229 & 0.399 & **0.292** \\ \cline{2-16}  & Imprw & 17.24\% & 38.62\% & 37.17\% & 3.89\% & 33.67\% & 37.17\% & 32.32\% & 27.18\% & 41.62\% & 46.37\% & 47.19\% & 18.55\% & 18.00\% & 25.36\% & 25.26 & - \\ \hline \multirow{4}{*}{10CAI} & HR & 0.285 & 0.392 & 0.449 & 0.448 & 0.455 & 0.469 & 0.429 & 0.452 & 0.461 & 0.481 & 0.475 & 0.481 & 0.463 & 0.489 & 0.519 & **0.554** \\ \cline{2-16}  & Imprw & 94.99\% & 41.39\% & 23.98\% & 23.66\% & 21.70\% & 18.12\% & 29.14\% & 22.57\% & 20.17\% & 18.18\% & 18.168\% & 19.65\% & 13.58\% & 13.58\% & 9.643 & 13.29\% & 6.74\% & - \\ \cline{2-16}  & NDCG & 0.183 & 0.284 & 0.287 & 0.288 & 0.304 & 0.260 & 0.285 & 0.292 & 0.04 & 0.
consistently outperforms baseline methods from various research lines by a significant margin. We attribute such performance improvement to the joint learning of multi-behavior inter-dependencies as well as the cross-layer collaborative signals under graph neural network. For example, _MBRec_ makes over 34% and 33% relatively improvement with respect to HR@10 and NDCG@10 respectively, as compared to autoencoder-based recommendation models (_i_.\(e\)., AutoRec & CDAE) on Tmall data. Additionally, for the results in terms of HR@10 on IJCAI-Competition data, the constant gain achieved by the developed _MBRec_ is around 20-22% over graph neural network-based CF models (ST-GCN and NGCF), and 18-29% over neural auto-regressive recommendation methods (NADE and CF-UIcA).

The proposed _MBRec_ also outperforms all other baseline methods with the modeling of multi-behavior data with respect to all metrics. Results show that our _MBRec_ allows the graph neural architecture to capture the multi-behavior interaction patterns, and successfully distinguish the layer-wise representations. While MBGCN and NGCF+M are built over the graph neural network to model behavior correlations, they fall short in encoding the latent type-specific characteristics and cross-type behavior inter-dependencies simultaneously. The performance of NMTR and MATN are limited to the failure for considering the high-order collaborative effects over the multi-behavior interaction graph. Furthermore, our new version model _MBRec_ always achieves better recommendation accuracy than the simplified version GNMR, which also confirms the effectiveness of our designed component for cross-layer mutual relation modeling. We further evaluate the performance of our _MBRec_ and several representative baselines with different top-\(N\) positions. The results are reported in Table IV. The best performance is achieved by our framework under different settings. To further evaluate the performance of our MBRec framework, we make the performance comparison by varying the number of sample negative instances. The evaluation results are shown in Table V. We can observe that our MBRec method consistently outperforms other alternative methods under different settings of negative samples in the range of {400, 800, 1600, 3200, 6400}. This observation validates the superiority of our MBRec in advancing the recommendation performance with the effective modeling of high-order heterogeneous collaborative relationships.

### _Ablation Study (RQ2)_

In this section, we would like to answer the question that if the designed individual component could help improve the recommendation accuracy. Specifically, we generate four types of model variants of our _MBRec_ corresponding to different aspects of our model design:

\(\bullet\)**Impact of Multi-Channel Behavior Embedding**. To evaluate the effect of our multi-channel behavior embedding layer, we compare the proposed method with the variant (w/o-MCE). This variant discards the behavior semantic modeling with multi-channel representation spaces. As shown in Table VI about the evaluation on three datasets, we can observe that the results of _MBRec_ are better than that of the variant (w/o-MCE). It demonstrates that the encoding of type-specific behavior characteristic could facilitate the multi-behavior dependency modeling.

\(\bullet\)**Impact of Behavior Inter-dependency Modeling**. To investigate the rationality of our behavior inter-dependency modeling, our _MBRec_ is compared with another model implementation (w/o-BIM) by removing the multi-behavior attention network. From the results in Table VI, _MBRec_ outperforms w/o-BIM in all cases, which benefits from the user/item representation enhanced by the exploration of pairwise behavior relational structures.

\(\bullet\)**Impact of Behavior Pattern Fusion**. We generate another simplified implementation of our recommendation architecture: (w/o-BFu) that does not consider the aggregation layer for pattern aggregation across various types of behavior representations. Instead, the type-aware behavior representations are directly combined through the element-wise mean pooling. As expected, _MBRec_ achieves better recommendation accuracy as compared to the variant (w/o-BFu). It verifies the necessity of our embedding fusion scheme during our multi-behavior dependency modeling.

\(\bullet\)**Impact of High-order Mutual Relation Learning**. To evaluate the effect of augmenting the graph neural model by capturing the cross-layer collaborative relations, we generate another variant (w/o-HMR) by only generating the output from the highest graph order after the information propagation process. From the evaluation results, we can observe the efficacy of the designed mutual relation encoder in learning the contributions of order-specific embeddings for the final prediction result.

### _Analysis on Individual Behavior Context (RQ3)_

This section conducts ablation studies on the influence of type-specific behavior context for the recommendation performance. The compared model variants are generated with the rubric as: First, "+" behavior type means that merely considering the target behaviors into the system to make

\begin{table}
\begin{tabular}{|l|l|l|l|l|l|l|l|l|} \hline \# samples & 400 & 800 & 1600 & 3200 & 6400 \\ \hline Model & HR & NDCG & HR & NDCG & HR & NDCG & HR & NDCG \\ \hline Biask & 0.285 & 0.140 & 0.149 & 0.078 & 0.091 & 0.049 & 0.055 & 0.032 & 0.036 & 0.022 \\ \hline NCF & 0.295 & 0.144 & 0.153 & 0.080 & 0.090 & 0.047 & 0.053 & 0.029 & 0.037 & 0.019 \\ \hline AuibRec & 0.245 & 0.122 & 0.134 & 0.074 & 0.0761 & 0.047 & 0.0500 & 0.033 & 0.036 & 0.025 \\ \hline ST-GCN & 0.311 & 0.183 & 0.184 & 0.093 & 0.104 & 0.053 & 0.058 & 0.031 & 0.036 & 0.019 \\ \hline MBGCN & 0.353 & 0.181 & 0.213 & 0.099 & 0.122 & 0.063 & 0.073 & 0.035 & 0.039 & 0.018 \\ \hline MATN & 0.339 & 0.165 & 0.192 & 0.093 & 0.113 & 0.056 & 0.058 & 0.031 & 0.037 & 0.020 \\ \hline GNMR & 0.345 & 0.176 & 0.218 & 0.100 & 0.103 & 0.055 & 0.063 & 0.040 & 0.024 \\ \hline _MBRec_ & **0.361** & **0.1848** & **0.129** & **0.199** & **0.124** & **0.064** & **0.075** & **0.040** & **0.044** & **0.025** \\ \hline \end{tabular}
\end{table}
Table V: Performance comparison with different number of negative samples in terms of _HR@10_ and _NDCG@10_

\begin{table}
\begin{tabular}{c|c c|c c|c c} \hline Data & Beibei Data & Tmall Data & \multicolumn{2}{c|}{IJCAI Data} \\ \hline Metrics & HR & NDCG & HR & NDCG & HR & NDCG \\ \hline \hline w/o-MCE & 0.6549 & 0.3876 & 0.4399 & 0.2580 & 0.5420 & 0.3289 \\ w/o-BIM & 0.6696 & 0.4000 & 0.4391 & 0.2554 & 0.5358 & 0.3228 \\ w/o-BFu & 0.6572 & 0.3907 & 0.4238 & 0.2487 & 0.5494 & 0.3321 \\ w/o-HMR & 0.6169 & 0.3470 & 0.3856 & 0.2240 & 0.3445 & 0.1760 \\ \hline _MBRec_ & **0.6701** & **0.4021** & **0.4435** & **0.2624** & **0.5535** & **0.3376** \\ \hline \end{tabular}
\end{table}
Table VI: Ablation study on key components of MBRec.
predictions (_i.e_., +buy). Second, "-" behavior type indicates the removing of this certain type of user behaviors (_e.g_., -pv, -cart) from the recommendation architecture. For instance, -pv indicates that we do not include the page view behaviors into the interaction inter-dependency modeling. We present the evaluation results in terms of NDCG@N and HR@N when \(N=10\) on three real-world datasets in Figure 4. As shown in Table II, the number of behavior types is 3 (_i.e_., page view, add-to-cart, buy) on BeiBei data and 4 (_i.e_., page view, add-to-cart, tag-as-favorite, buy) on Tamil, IJCAI-Competition data. From the results, we can observe that each type of interaction behavior individually contributes to improve the user preference learning, and integrate multi-behavior behavior patterns for performance improvement.

### _Performance Under Different Sparsity (RQ4)_

In our experiments, we also evaluate the recommendation performance of different models under different interaction sparsity. Following the similar settings in [34, 38], we first partition users into five groups based on the number of interactions. For example, "\(<\)36" and "\(<\)52" indicate that users belong to this group have the number of interactions ranging from 1 to 35, and 36 to 51, respectively. We keep the same number of users in each group and select the corresponding ranges as shown in x-axis of Figure 5. The total number of users contained in each group and the recommendation accuracy with respect to HR (Figure 5 (a)) and NDCG (Figure 5 (b)) are shown in the left side and right side of y-axis in Figure 5. From evaluation results, we can notice the superiority of our _MBRec_ with different sparsity levels. It suggests that the incorporation of multi-typed behaviour patterns into the user preference learning could reach performance improvement as compared with other baselines. In addition, we can observe that the overall performance of all compared methods share similar increase trend as users have more interactions. This may indicate that more user behavior data may help characterize user preference with more accurate latent representations.

### _Analysis on Hyperparameters (RQ5)_

We study the impact of different hyperparameter settings on the model performance in our joint learning framework.

* **Comparison with Different Hidden Dimensionality**. Our model results with different dimension size of hidden states are shown in Figure 6. We observe that larger embedding size does not always bring the positive effect for improving model performance, especially for sparse experimented datasets. The larger size of hidden state dimensionality may lead to the overfitting issue. We set the hidden dimensionality \(d=16\) as the default value in our _MBRec_.
* **Comparison with Different Graph Model Depth**. To investigate the performance of our MBGNN method by stacking multiple graph neural layers, we conduct experiments by varying the number of graph-based embedding propagation layers. As shown in Table VII, we can observe that _MBRec_-2 and _MBRec_-3 obtain consistent improvement over _MBRec_-1 which merely considers the first-order neighbors for message passing. We attribute the performance improvement to the encoding of collaborative relations based on our considered second- and third-order neighboring node dependency. With the further increase of model depth from two to three graph layers, the performance slight degrades with the configuration of deep graph neural architecture. The reason may lie in that deep graph neural framework tends to be overfitting and involve the over-smoothing issue in the generated user/item representations. According to the statistical information from our experimented Tmall data, with the consideration of three-hop connections, a large percentage of user-item pairs may be connected, which unavoidably leads to the over-smoothing issue of making user embeddings indistinguishable.
* **Comparison with Different Number of Channels**. We vary the number of embedding channels in our multi-channel behavior embedding layer. The results in terms of HR@10 and NGCD@10 are presented in Table VIII, from which we notice that the performance of _MBRec_ is

Figure 4: Impact study of diverse behavior types. There are three types of behaviors for BeiBei data, and four types of behaviors for Tmall, IJCAI data.

Figure 5: Performance comparison of MBRec and baseline methods _w.r.t_ different data sparsity levels on Tmall data.

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c} \hline Data & \multicolumn{2}{c|}{BeiBei Data} & \multicolumn{2}{c|}{Tmall Data} & \multicolumn{2}{c}{IJCAI Data} \\ \hline Metrics & HR & NDCG & HR & NDCG & HR & NDCG \\ \hline \hline _MBRec_-1 & 0.662 & 0.394 & 0.383 & 0.226 & 0.533 & 0.313 \\ _MBRec_-2 & **0.670** & **0.402** & **0.444** & **0.262** & **0.554** & **0.338** \\ _MBRec_-3 & 0.664 & 0.398 & 0.408 & 0.237 & 0.543 & 0.332 \\ \hline \end{tabular}
\end{table}
Table VII: Effect of embedding propagation layers.
improved at first, with the increase of behavior representation channels. But we can observe that the recommendation performance degrades with the further increase of channel numbers, due to the overfitting. Hence, behavior embedding channels with the dimension of 16 is enough for encoding interaction semantics.

### _Computational Cost Analysis (RQ6)_

Our evaluation also includes the computational cost investigation of our _MBRec_ model and several representative methods in terms of their implementation time on different datasets. We evaluate the computational cost of all compared methods on the machine of NVIDIA TITAN RTX GPU with the configurations of Intel Xeon W2133 CPU 3.6G Hz and 64GB RAM. For fair comparison, we apply the same setting of hidden state dimensionality for all methods. For graph-based methods, the number of embedding propagation layers is set as 2. From the reported evaluation results in Table IX, we can observe that our _MBRec_ model can achieve comparable model efficiency compared with other baselines in terms of the implementation time. In particular, when competing with multi-behavior recommendation baselines (NGCF+M, MBGCN), our _MBRec_ requires less implementation time, which indicates the efficiency of our multi-behavior graph neural framework. Additionally, compare with the autoregressive collaborative filtering model-CF-UcA, our _MBRec_ can still achieve competitive model efficiency with the incorporation of multi-typed behaviour context. In summary, the above observations justify the scalability of our proposed _MBRec_ in dealing with large-scale user behavior data for recommendation.

### _Model Interpretation with User Study (RQ7)_

To analyze the multi-behavior dependency interpretation of our proposed MBRec framework, we conduct user studies with identified real user examples. We show the study results in Figure 7. In this figure, the cross-type behavior dependencies between user (\(u_{116}\), \(u_{1621}\)) and item (\(v_{13844}\), \(v_{64224}\)) are shown with the learned quantitative dependency weights. Specifically, \(E_{i}^{(1)}\) and \(E_{i}^{(2)}\) denote our produced user representation encoded from the \(1^{st}\) and \(2^{nd}\) graph-based embedding propagation layer, respectively. Similarly, \(E_{j}^{(1)}\) and \(E_{j}^{(2)}\) represents the encoded item embeddings from the \(1^{st}\) and \(2^{nd}\) message passing layers, respectively. From the study results, we summarize the key observations as follows:

* **Encoded Behavior Inter-Correlations**. In this user study, we present the learned behavior inter-correlation matrix with the dimension of \(\mathbb{R}^{4\times 4}\) to reflect the pairwise correlations between different types of user behaviors, \(i\)._e_., page view, add-to-cart, tag-as-favorite and purchase.
* **Type-specific Behavior Pattern Fusion**. In our MBRec recommendation framework, we design the multi-behavior pattern aggregation module with the aim of integrating type-specific behaviour patterns for making final recommendation. In particular, each user is associated with a learned attention-based behavior importance vector with the dimension of \(\mathbb{R}^{1\times 4}\) (as shown in Figure 7). For example, we can observe that users who view item \(v_{13844}\) are more likely to purchase it compared with item \(v_{64224}\).
* **Cross-layer Mutual Relation Encoding**. In our multi-layer graph neural framework, we introduce a mutual relation encoding component to explicitly aggregate representations from different hops in the multi-behavior interaction graph. \(E_{i}^{(1)}\) and \(E_{i}^{(2)}\) represents the encoded embeddings of user \(u_{i}\) from his/her first- and second-order neighboring nodes. In Figure 7, the correlations among hop-aware user/item representations are shown with different connection lines. From the visualization results, We can observe that cross-layer user/item embeddings (_e_.\(g\)., \(E_{i}^{(0)}\) and \(E_{j}^{(1)}\)) are often highly correlated with each other compared with the embeddings of the same layer (_e_.\(g\)., \(E_{i}^{(2)}\) and \(E_{j}^{(2)}\)).

### _Effect of Graph Sampling_

In this section, we investigate the effect of our graph sampling algorithm on the model performance by testing the prediction accuracy of MBRec with different number of training and testing sub-graphs. In specific, MBRec is trained with sub-graphs containing 5000, 10000, 20000, 40000 nodes, and is tested using input sub-graphs containing 5000, 10000,

\begin{table}
\begin{tabular}{l c c c} \hline \hline Models & BeiBei & Tmall & IJCAI \\ \hline NADE & 4.1s & 26.9s & 60.4s \\ CF-UcA & 11.5s & 61.7s & 139.1s \\ ST-GCN & 12.6s & 58.5s & 94.8s \\ NGCF+M & 15.8s & 74.6s & 152.3s \\ NMTR & 14.0s & 37.3s & 118.0s \\ MBGCN & 17.4s & 85.3s & 186.5s \\ MATN & 11.5s & 74.7s & 196.5s \\ DIPN & 53.2s & 172.6s & 284.6s \\ \hline _MBRec_ & 14.3s & 58.6s & 101.1s \\ \hline \hline \end{tabular}
\end{table} TABLE IX: Computational cost (seconds) study.

\begin{table}
\begin{tabular}{c|c c|c c|c c} \hline Data & BeiBei & Data & Tmall & Data & IJCAI Data \\ \hline Metrics & HR & NDCG & HR & NDCG & HR & NDCG \\ \hline \hline _MBRec_-2 & 0.655 & 0.390 & 0.438 & 0.260 & 0.518 & 0.308 \\ _MBRec_-4 & 0.656 & 0.401 & 0.443 & 0.260 & 0.553 & 0.328 \\ _MBRec_-8 & **0.670** & **0.402** & **0.444** & **0.262** & **0.554** & **0.338** \\ _MBRec_-16 & 0.646 & 0.387 & 0.419 & 0.241 & 0.558 & 0.336 \\ \hline \end{tabular}
\end{table} TABLE VIII: Effect of behavior embedding channels.

Fig. 6: Impact of hidden state dimensionality of MBRecframework on BeiBei, Tmall, IJCAI datasets.
20000, 40000, 60000 nodes. The results are shown in Table X, from which we can conclude that testing on larger sub-graphs always yields better performance, while training on larger sub-graphs does not always result in better performance. This is because training with smaller sub-graphs may serve as regularization operation for predictions.

## V Related Work

### _Neural Network Collaborative Filtering Models_

Collaborative Filtering (CF) has become one of the most important paradigms for personalized recommender systems in real-life platforms [32, 15]. The general idea of CF models is that users may share similar preference if they interact with similar items [12]. In recent years, many efforts have been made to augment the CF techniques with deep neural network models [27]. These methods apply different neural mechanisms (_e.g._, autoencoder, attention mechanism, graph neural network) in the matching function to parameterize users and items into latent representation space. The learned representations of users and items can be used to estimate the likelihood of unobserved interactions.

Some of studies follow this research line to enable the non-linear feature interactions with the multi-Layer feed-forward network, such as NCF [11] and DMF [47]. To consider item relational data into the CF model, the relational collaborative filtering (RCF [46]) framework designs neural two-stage attention mechanism to enhance the item embedding process. Another recent research line of recommendation models is to explore the user-item interaction graph to capture the collaborative filtering signals. For example, NGCF [34] is developed based on the high-hop information propagation framework to guide the user/item representation procedure. ST-GCN [53] is another graph learning model to encode user-item interaction patterns with an encoder-decoder framework. In addition, to bridge the logical reasoning and representation learning in recommender systems, a neural collaborative reasoning approach (NLR) [4] is proposed to incorporate the logic priors into the neural architecture.

### _Recommendation with Multi-Behavior Modeling_

There exist some research works aiming at enhancing recommendation models by considering multi-typed behavior of users [31]. In those methods, the implicit user-item feedback from auxiliary behaviors (_e.g._, click, add-to-cart) are considered as behavior contextual signals to predict target user behaviors (_e.g._, purchase) [16, 45]. For example, multi-task learning frameworks are developed to perform the joint training among the prediction tasks of different behavior types [7]. However, those methods reply on the predefined dependent relationships between different types of user behaviors, and can hardly be reflective of the complex multi-behaviour context in practical scenarios.

To capture the correlations between different types of behaviors, MATN [44] utilizes the attention network for multi-behavior information aggregation. Both browsing and buying behaviors of users are considered in DIPN [9] with an attention-based RNN model. However, the high-order behavior dependent structures have been overlooked in them. In this work, the proposed MBRec framework aims to encode the high-order collaborative signals in the embedding function. Additionally, graph-based methods have been designed to tackle the multi-behavior recommendation problem. Specifically, Zhang _et al_. [54] employs the multiplex network embedding technique to generate behavior-aware embeddings. MBGCN _et al_. [16] is built on the graph convolutional network to propagate the behavior embeddings over the interaction graph. Our new MBRec differs from those graph-based models from two perspectives: i) we discriminate the influence between various behaviour patterns through a dual-stage relation learning scheme. The designed new message passing paradigm endows the multi-behavior graph neural network with the capability of encoding behavior-aware characteristics and dependencies simultaneously. ii) The high-order collaborative signals are aggregated across different graph layers explicitly, under the cross-layer message passing architecture.

### _Graph Neural Networks for Recommendation_

In view of the effectiveness of Graph Neural Networks (GNNs), GNNs have been widely used to perform the representation learning over the graph-structured data [35, 42, 22, 5, 21, 30, 18]. Recent research works apply the graph neural network to model user-item interactions in recommender systems: PinSage [50] is a graph convolutional network to propagate embeddings over the pin-board bipartite

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|} \hline \multicolumn{1}{|c|}{\multirow{2}{*}{Train. \(N\)}} & \multicolumn{1}{c|}{\multirow{2}{*}{5,J000}} & \multicolumn{1}{c|}{\multirow{2}{*}{10,000}} & \multicolumn{1}{c|}{\multirow{2}{*}{20,000}} & \multicolumn{1}{c|}{\multirow{2}{*}{40,000}} & \multicolumn{1}{c|}{\multirow{2}{*}{60,000}} \\ \cline{3-10} \cline{6-10} \multicolumn{1}{|c|}{} & HR & NDCG & HR & NDCG & HR & NDCG & HR & NDCG & HR & NDCG \\ \hline \hline
5,000 & 0.365 & 0.201 & 0.049 & 0.230 & 0.463 & 0.266 & 0.527 & 0.310 & 0.552 & 0.388 \\ \hline
10,000 & 0.359 & 0.198 & 0.407 & 0.229 & 0.464 & 0.266 & 0.529 & 0.307 & 0.552 & 0.336 \\ \hline
20,000 & 0.357 & 0.196 & 0.407 & 0.228 & 0.466 & 0.270 & 0.537 & 0.326 & 0.554 & 0.338 \\ \hline
40,000 & 0.322 & 0.168 & 0.367 & 0.197 & 0.424 & 0.236 & 0.500 & 0.292 & 0.548 & 0.330 \\ \hline \end{tabular}
\end{table}
Table X: Influence of the sub-graph sampling scale.

Figure 7: Interpretation study of multi-behavior inter-dependency in our _MBRec w.r.t_ behavior inter-dependency modeling, behavior pattern aggregation and cross-layer mutual relation learning. Dark color indicates higher relevance. Best viewed in color.
graph structure. Additionally, modeling the dynamic user-item interactions has attracted much attention for recommender systems [3, 48]. To encode the sequential patterns, graph neural networks have been utilized to consider the transitions between items of session sequences in SRGNN [40] and MTD [14], or user interaction sequence in H2SeqRec [20]. In addition, the graph diffusion network [38] and graph attention mechanism [39] have been utilized to capture the influence among users, so as to incorporate the social relations into the recommendation and alleviate the data sparsity issue.

## VI Conclusion

In this work, we contribute a new end-to-end framework (MBRec) for multi-behavior recommendation via the modeling of cross-behavior inter-dependencies under a high-order graph learning architecture. In our MBRec model, we first learn the dependent relationships among various types of user interactions with a behavior-aware message passing mechanism. Additionally, a designed high-order mutual relation learning scheme is integrated with the graph neural architecture, so as to encode the implicit dependencies between layer-specific behavior representations. When evaluated on three real-world datasets, our framework achieves significantly better recommendation performance as compared to various baselines. Further studies on model ablation show the rationality of designed key components in our proposed recommendation framework. In future, we would like to integrate the causal effect analysis [1] with our designed multi-behavior graph neural paradigm, in order to infer the causal relations from observed user behaviors and identify the implicit factors which influence user preference.

## Acknowledgments

We thank the reviewers for their valuable feedback and comments. This research work is supported by the research grants from the Department of Computer Science & Muske-teres Foundation Institute of Data Science at the University of Hong Kong (HKU). The research is also partially supported by National Nature Science Foundation of China (62072188), Major Project of National Social Science Foundation of China (18ZDA062), Science and Technology Program of Guangdong Province (2019A050510010).
