11institutetext:  The Institute of Computing Technology of the Chinese Academy
of Sciences, China
11email<EMAIL_ADDRESS>22institutetext: Beijing Kuaishou Technology
Co., Ltd., China
22email: {changchao, chenzhenyang, huguoyong,
<EMAIL_ADDRESS>Alibaba Group, China
33email<EMAIL_ADDRESS>
# Graph Attention Collaborative Similarity Embedding for Recommender System

Jinbo Song 11 Chao Chang 22 Fei Sun 33 Zhenyang Chen 22 Guoyong Hu 22 Peng
Jiang 22

###### Abstract

We present Graph Attention Collaborative Similarity Embedding (GACSE), a new
recommendation framework that exploits collaborative information in the user-
item bipartite graph for representation learning. Our framework consists of
two parts: the first part is to learn explicit graph collaborative filtering
information such as user-item association through embedding propagation with
attention mechanism, and the second part is to learn implicit graph
collaborative information such as user-user similarities and item-item
similarities through auxiliary loss. We design a new loss function that
combines BPR loss with adaptive margin and similarity loss for the
similarities learning. Extensive experiments on three benchmarks show that our
model is consistently better than the latest state-of-the-art models.

###### Keywords:

Recommendation systems Collaborative Filtering Graph Neural Networks.

##  1 Introduction

Personalized recommendation plays a pivotal role in many internet scenarios,
such as e-commerce, short video recommendations and advertising. Its core
method is to analyze the user’s potential preferences based on the user’s
historical behavior, to measure the possibility of the user to select a
certain item and to tailor the recommendation results for the user.

One of the major topics to be investigated in the personalized recommendation
is Collaborative Filtering (CF) which generates recommendations by taking
advantage of the collective wisdom from all users. Matrix factorization (MF)
[12] is one of the most popular CF model, which decomposes the interaction
matrix between the user and item into discrete vectors and then calculates the
inner product to predict the connected edges between the user and item. Neural
Collaborative Filtering (NCF) [19] predict the future behavior of users by
learning the historical interactions between users and items. It employs
neural network instead of traditional matrix factorization to enhance the non-
linearity of the model. In general, there are two key components in learnable
CF models—1) embeddings that represent users and items by vectors, and 2)
interaction modeling, which formulates historical interactions upon the
embeddings.

Despite their prevalence and effectiveness, we argue that these models are not
sufficient to learn optimal embdeddings. The major limitation is that the
embdeddings does not explicitly encode collaborative information propagated in
user-item interaction graph. Following the idea of representation learning in
graph embedding, Graph Neural Networks (GNN) are proposed to collect aggregate
information from graph structure. Methods based on GraphSAGE [8] or GAT [23]
have been applied to recommender systems. For example, NGCF [17] generates
user and item embeddings based on the information propagation in the user-item
bipartite graph; KGAT [24] adds a knowledge graph and entities attention based
on the bipartite graph and uses entity information to more effectively model
users and items. Inspired by the success of GNN in recommendation, we build a
embedding propagation and aggregating architecture based on attention
mechanism to learn the variable weight of each neighbor. The attention weight
explicitly represents the relevance of interaction between user and item in
bipartite graph.

Another limitation is that many existing model-based CF algorithms leverage
only the user-item associations available in user-item bipartite graph. The
effectiveness of these algorithms depends on the sparsity of the available
user-item associations. Therefore, other types of collaborative relations,
such as user-user similarity and item-item similarity, can also be considered
for embedding learning. Some works [27, 25] exploit higher-order proximity
among users and items by taking random walks on the graph. A recent work [26]
presents collaborative similarity embedding (CSE) to model direct and in-
direct edges of user-item interactions. The effectiveness of these methods
lies in sampling auxiliary information from graph to augment the data for
representation learning.

Based on the above limitation and inspiration, in this paper, we propose a
unified representation learning framework, called Graph Attention
Collaborative Similarity Embedding (GACSE). In the framework, the embedding is
learned from direct, user-item association through embedding propagation with
attention mechanism, and indirect, user-user similarities and item-item
similarities through auxiliary loss, user-item similarities in bipartite
graph. Meanwhile, we combine adaptive margin in BPR loss [2] and similarity
loss to optimize GACSE.

The contributions of this work are as follows:

  * •

We propose GACSE, a graph based recommendation framework that combines both
attention propagation & aggregation in graph and similarity embedding learning
process.

  * •

To optimize GACSE, we introduce a new loss function, which, to the best of our
knowledge, is the first time to combine both BPR loss with adaptive margin and
similarity loss for similarity embedding learning.

  * •

We compare our model with state-of-the-art methods and demonstrate the
effectiveness of our model through quantitative analysis on three benchmark
datasets.

  * •

We conduct a comprehensive ablation study to analyze the contributions of key
components in our proposed model.

##  2 Related Work

In this section, we will briefly review several lines of works closely related
to ours, including general recommendation and graph embedding-based
recommendation.

###  2.1 General Recommendation

Recommender systems typically use Collaborative Filtering (CF) to model users’
preferences based on their interaction histories [3, 1]. Among the various CF
methods, item-based neighborhood methods [5] estimate a user’s preference on
an item via measuring its similarities with the items in her/his interaction
history using a item-to-item similarity matrix. User-based neighborhood
methods find similar users to the current user using a user-to-user similarity
matrix, following by recommending the items in her/his similar users’
interaction history. Matrix Factorization (MF) [12, 13] is another most
popular one, which projects users and items into a shared vector space and
estimate a user’s preference on an item by the inner product between user’s
and items’ vectors. BPR-MF [2] optimizes the matrix factorization with
implicit feedback using a pairwise ranking loss. Recently, deep learning
techniques has been revolutionizing the recommender systems dramatically. One
line of deep learning based model seeks to take the place of conventional
matrix factorization [7, 11, 19]. For example, Neural Collaborative Filtering
(NCF) estimates user preferences via Multi-Layer Perceptions (MLP) instead of
inner product.

###  2.2 Graph based Recommendation

Another line is to integrate the distributed representations learning from
user-item interaction graph. GC-MC [10] employs a graph convolution auto-
encoder on user-item graph to solve the matrix completion task. HOP-Rec [15]
employs label propagation and random walks on interaction graph to compute
similarity scores for user-item pairs. NGCF [19] explicitly encodes the
collaborative information of high-order relations by embedding propagation in
user-item interaction graph. PinSage [16] utilizes efficient random walks and
graph convolutions to generate embeddings which incorporate both graph
structure as well as node feature information. Multi-GCCF [9] constructs two
separate user-user and item-item graphs. It employs a multi-graph encoding
layer to integrate the information provided by the user-item, user-user and
item-item graphs.

##  3 Our Model

In this section, we introduce our proposed model called GACSE. The overall
framework is illustrated in Figure 1. There are three components in the model:
(1) an embedding layer that can map users and items from one hot vector to
initial embeddings; (2) an embedding propagation layer, which consists of two
sub-layers: a warm-up layer that propagates and aggregates graph embeddings
with equal weight, and an attention layer that uses attention mechanism to
perform non-equal weight aggregation on the embedding of neighboring nodes;
and (3) a prediction layer that concatenates embeddings from embedding layer
and attention layer, then outputs affinity score between user and item. The
following descriptions take user as central node, if there is no special
instruction, it is also applicable to item as centre node.

Figure 1: An illustration of GACSE model architecture. The flow of embedding
is presented by the arrowed lines. FC1 and FC2 are shared parameters on user
embedding side and item embedding side. An illustration of attention
aggregation is shown on the left. FC-a transforms the concatenated vector into
a new vector. FC-b transforms vector into attention score.

###  3.1 Embedding Layer

Embedding layer aims at mapping the ids of user $u$ and item $i$ into
embedding vectors $\mathbf{e}_{u}^{(0)}\in\mathbb{R}^{d}$ and
$\mathbf{e}_{i}^{(0)}\in\mathbb{R}^{d}$, where $d$ denotes the embedding dim.
We use a trainable embedding lookup table to build our embedding layer for
embedding propagation:

$\mathbf{E}=[\overbrace{\mathbf{e}_{u_{1}}^{(0)},\cdots,\mathbf{e}_{u_{N}}^{(0)}}^{\text{users}\,\text{embeddings}}\;,\;\overbrace{\mathbf{e}_{i_{1}}^{(0)},\cdots,\mathbf{e}_{i_{M}}^{(0)}}^{\text{items}\,\text{embeddings}}]$
(1)

where $N$ is number of users and $M$ is number of items.

###  3.2 Embedding Propagation Layer

In order to establish the embedding propagation architecture for collaborative
information in graph, we define a embedding propagation layer. Our embedding
propagation layer consists of two parts: (1) a warm-up layer and (2) an
attention layer. Both layers have two steps: embedding propagation and
aggregation.

####  3.2.1 Warm-up Layer

To make the model expand the receptive field and grasp the structure of the
user-item bipartite graph, we set up a warm-up layer based on the GNN [8, 18]
embedding propagation architecture.

##### Warm-up Propagation

In the warm-up layer, we set all the embeddings to have equal weight. We
define warm-up embedding propagation function as:

$\left\\{\begin{aligned} &\mathbf{m}_{i\rightarrow
u}^{(0)}=\pi^{(0)}_{(u,i)}\mathbf{W}_{0}\mathbf{e}_{i}^{(0)}\\\
&\mathbf{m}_{u\rightarrow
u}^{(0)}=\mathbf{W}_{0}\mathbf{e}_{u}^{(0)}\end{aligned}\right.$ (2)

where $\mathbf{W}_{0}\in\mathbb{R}^{d_{1}\times d_{0}}$ is the trainable
weight matrix to distill important information in embedding propagation.
$d_{0}$ is the dimension of $\mathbf{e}^{(0)}$, and $d_{1}$ is the dimension
of transformation. $\mathbf{m}_{i\rightarrow u}^{(0)}$ is the embedding from
item $i$ to user $u$. $\mathbf{m}_{u\rightarrow u}^{(0)}$ is self-connection
of $u$. $\pi_{(u,i)}^{(0)}$ is the weight of the embedding that $i$ passes to
$u$.

In embedding propagation of warm-up layer, we set the propagation weight
$\pi^{0}_{(u,i)}$ to be equal. Inspired by GCN [8], we define weight of each
user’s interacted item as:

$\pi^{(0)}_{(u,i)}=\frac{1}{\sqrt{|\mathcal{N}_{u}\|\mathcal{N}_{i}|}}$ (3)

where $\mathcal{N}_{u}$ and $\mathcal{N}_{i}$ denote the first hop neighboring
nodes of user u and item i.

##### Warm-up Aggregation

After receiving the embeddings from neighbor nodes, we need to aggregate these
embeddings. We define embeddings aggregation function in warm-up layer as:

$\mathbf{e}^{(1)}_{u}=\sigma\bigl{(}\mathbf{m}_{u\rightarrow
u}^{(0)}+\sum_{i\in\mathcal{N}_{u}}\mathbf{m}_{i\rightarrow u}^{(0)}\bigr{)}$
(4)

where $\sigma$ is nonlinear function such as LeakyReLU. Analogously, we can
obtain item $i$’s embedding $\mathbf{e}^{(1)}_{i}$.

####  3.2.2 Attention Layer

Next, in order to further encode the variable weight of neighbors, we build an
embedding propagation and aggregation architecture based on the attention
mechanism. The attention mechanism explicitly captures the relevance of
interaction between user and item in bipartite graph.

##### Attention Propagation

Intuitively, the importance of each item that interacts with the user should
be different. We introduce attention mechanism into embedding passing
function:

$\left\\{\begin{aligned} &\mathbf{m}_{i\rightarrow
u}^{(1)}=\pi^{(1)}_{(u,i)}\mathbf{e}_{i}^{(1)}\\\ &\mathbf{m}_{u\rightarrow
u}^{(1)}=\mathbf{e}_{u}^{(1)}\end{aligned}\right.$ (5)

where $\pi^{(1)}_{(u,i)}$ is attention weight.

Inspired by several kinds of attention score functions, we define score
function of our model:

$\text{score}(\mathbf{e}_{u}^{(1)},\mathbf{e}_{i}^{(1)})=\mathbf{V}^{\top}\text{tanh}\Bigl{(}\mathbf{P}\left[\mathbf{e}_{u}^{(1)}\|\mathbf{e}_{i}^{(1)}\right]\Bigr{)}$
(6)

where $\mathbf{V}\in\mathbb{R}^{d_{2}\times 1}$ and
$\mathbf{P}\in\mathbb{R}^{d_{2}\times 2d_{1}}$ are trainable parameters.
$d_{2}$ is the dimension of attention transformation. $\|$ denotes concatenate
operation. After calculating attention score, we normalize it to get the
attention weight via softmax function:

$\pi^{(1)}_{(u,i)}=\frac{\text{exp}(\text{score}(\mathbf{e}^{(1)}_{u},\mathbf{e}^{(1)}_{i}))}{\sum_{j\in\mathcal{S}_{u}}\text{exp}(\text{score}(\mathbf{e}^{(1)}_{u},\mathbf{e}^{(1)}_{j}))}$
(7)

where $\mathcal{S}_{u}$ is a set of user $u$’s one hop neighboring items
sampled in this mini-batch.

##### Attention Aggregation

After attention massage passing, the attention aggregation function is defined
as:

$\displaystyle\mathbf{e}^{(2)}_{u}=$
$\displaystyle\sigma(\mathbf{W}_{1}(\mathbf{m}^{(1)}_{u\rightarrow
u}+\sum_{i\in\mathcal{S}_{u}}\mathbf{m}^{(1)}_{i\rightarrow u}))+$ (8)
$\displaystyle\sigma(\mathbf{W}_{2}(\mathbf{m}^{(1)}_{u\rightarrow
u}\odot\sum_{i\in\mathcal{S}_{u}}\mathbf{m}^{(1)}_{i\rightarrow u}))$

where $\sigma$ is LeakyReLU non-linear function. $\mathbf{W}_{1}$,
$\mathbf{W}_{2}\in\mathbb{R}^{d_{3}\times d_{1}}$ are trainable parameters.
$d_{3}$ is the dimension of attention aggregation. $\odot$ denotes element-
wise product. Similar to NGCF, the aggregated embedding $\mathbf{e}^{(2)}_{u}$
does not only related to $\mathbf{e}^{(1)}_{i}$, but also encodes the
interaction between $\mathbf{e}^{(1)}_{u}$ and $\mathbf{e}^{(1)}_{i}$. The
interaction information can be represented by the element-wise product between
$\mathbf{m}_{u\rightarrow u}$ and
$\sum_{i\in\mathcal{S}_{u}}\mathbf{m}_{i\rightarrow u}$. Analogously, item
$i$’s attention layer embedding $\mathbf{e}^{(2)}_{i}$ can be obtained. We
incorporate the attention mechanism to learn variable weight
$\pi^{(1)}_{(u,i)}$ for each neighbor’s propagated embedding
$\mathbf{m}^{(1)}_{i\rightarrow u}$.

###  3.3 Model Prediction

After embedding passing and aggregation with attention mechanism, we obtained
two different representations $\mathbf{e}_{u}^{(0)}$ and
$\mathbf{e}_{u}^{(2)}$ of user node $u$; also analogous to item node $i$, we
obtained $\mathbf{e}_{i}^{(0)}$ and $\mathbf{e}_{i}^{(2)}$. We choose to
concatenate the two embeddings as follows:

$\displaystyle\mathbf{e}_{u}^{*}=\mathbf{e}_{u}^{(0)}\|\,\mathbf{e}_{u}^{(2)},\quad\mathbf{e}_{i}^{*}=\mathbf{e}_{i}^{(0)}\|\,\mathbf{e}_{i}^{(2)}$
(9)

where $\|$ denotes the concatenate operation. In this way, we could predict
the matching score between user and item by inner product:

$y_{ui}={\mathbf{e}_{u}^{*}}^{\top}\mathbf{e}_{i}^{*}$ (10)

More broadly, we can define the matching score between any two nodes a and b:

$y_{ab}={\mathbf{e}_{a}^{*}}^{\top}\mathbf{e}_{b}^{*}$ (11)

##  4 Optimization

To optimize the GACSE model, we carefully designed our loss function. Our loss
function consists of two basic parts: BPR loss with adaptive margin and
similarity loss.

###  4.1 BPR loss with adaptive margin

We employ BRP loss for optimization, which considers the relative order
between observed and unobserved interactions. In order to improve the model’s
discrimination of similar positive and negative samples, we define BPR loss
with adaptive margin as:

$\displaystyle\mathcal{L}_{\text{BPR}}=\frac{1}{|\mathcal{B}|}\sum_{(u,i,j)\in\mathcal{B}}-\sigma(y_{ui}-y_{uj}-\max(0,y_{ij}))$
(12)

where
$\mathcal{B}\subseteq\\{(u,i,j)|(u,i)\in\mathcal{R}^{+},(u,j)\in\mathcal{R}^{-}\\}$
denotes the sampled data of mini-batch. $\mathcal{R}^{+}$ denotes observed
interactions, and $\mathcal{R}^{-}$ is unobserved interactions. $\sigma$ is
softplus function. $\max(0,y_{ij})$ indicates that the more similar the
positive and negative samples of a node are, the larger the margin of the loss
function is.

###  4.2 Similarity loss

Other types of collaborative relations, such as user-user similarity and item-
item similarity in graph, can also be considered for embedding learning. The
introduce of similarity loss for both user-user and item-item pair can reduce
the sparsity problem by augmenting the data for representation learning. In
this paper, the 2-order neighborhood proximity of a pair of users (or items)
is defined as the similarity.

In order to avoid similarity loss affecting the embedding in the embedding
propagation, we only calculate between $\mathbf{E}$ and context mapping
embedding matrices $\mathbf{E}^{\text{UC}}$ and $\mathbf{E}^{\text{IC}}$ for
users and items, respectively. Context mapping embedding matrices are defined:

$\displaystyle\mathbf{E}^{\text{UC}}$
$\displaystyle=[\mathbf{e}^{\text{UC}}_{u_{1}},\cdots,\mathbf{e}^{\text{UC}}_{u_{N}}]$
(13) $\displaystyle\mathbf{E}^{\text{IC}}$
$\displaystyle=[\mathbf{e}^{\text{IC}}_{i_{1}},\cdots,\mathbf{e}^{\text{IC}}_{i_{M}}]$

It should be noted that the dimensions of embeddings in $\mathbf{E}$,
$\mathbf{E}^{\text{UC}}$ and $\mathbf{E}^{\text{IC}}$ are equal. The
similarity loss for $\mathbf{e}^{(0)}$ with context embeddings
$\mathbf{e}^{\text{UC}}\in\mathbf{E}^{\text{UC}}$ and
$\mathbf{e}^{\text{IC}}\in\mathbf{E}^{\text{IC}}$ is defined as:

$\displaystyle\mathcal{L}_{\text{similarity}}=$
$\displaystyle-\sum\text{log}(\sigma(\mathbf{e}^{(0)\text{T}}_{u}\mathbf{e}^{\text{UC}}_{\text{u-pos}}))+\sum\text{log}(\sigma(\mathbf{e}^{(0)\text{T}}_{u}\mathbf{e}^{\text{UC}}_{\text{u-neg}}))$
(14)
$\displaystyle-\sum\text{log}(\sigma(\mathbf{e}^{(0)\text{T}}_{i}\mathbf{e}^{\text{IC}}_{\text{i-pos}}))+\sum\text{log}(\sigma(\mathbf{e}^{(0)\text{T}}_{i}\mathbf{e}^{\text{IC}}_{\text{i-neg}}))$

where $\sigma$ is sigmoid function. $\mathbf{e}^{\text{UC}}_{\text{u-pos}}$
and $\mathbf{e}^{\text{UC}}_{\text{u-neg}}$ are a positive and negative
samples of user $u$, respectively. $\mathbf{e}^{\text{IC}}_{\text{i-pos}}$ and
$\mathbf{e}^{\text{IC}}_{\text{i-neg}}$ are a positive and negative samples of
item $i$, respectively. We employ random walk and negative sampling to
construct positive and negative sample pairs for similarity loss.

###  4.3 Overall Loss Function

Finally, we get the overall loss function:

$\mathcal{L}=\mathcal{L}_{\text{BPR}}+\lambda_{1}\mathcal{L}_{\text{similarity}}+\lambda_{2}\|\Theta\|_{2}^{2}$
(15)

where
$\Theta=\\{\mathbf{E},\mathbf{E}^{\text{UC}},\mathbf{E}^{\text{IC}},\mathbf{W}_{0},\mathbf{W}_{1},\mathbf{W}_{2},\mathbf{V},\mathbf{P}\\}$.
$\lambda_{1}$ controls the strength of BPR loss and $\lambda_{2}$ controls the
$L_{2}$ regularization strength to prevent overfitting. We use mini-batch Adam
[28] to optimize the model and update the parameters of model.

##  5 EXPERIMENTS

###  5.1 Datasets

We evaluate the proposed model on three real-world representative datasets:
Gowalla111 https://snap.stanford.edu/data/loc-gowalla.htm,
Yelp2018222https://www.yelp.com/dataset/challeng and Amazon-
book333http://jmcauley.ucsd.edu/data/amazon/. These datasets vary
significantly in domains and sparsity. The statistics of the datasets are
summarized in Table 1.

For each dataset, the training set is constructed by $80\%$ of the historical
interactions of each user, and the remaining as the test set. We randomly
select $10\%$ of interactions as a validation set from the training set to
tune hyper-parameters. We employ negative sampling strategy to produce one
negative item that the user did not act before and treat observed user-item
interaction as a positive instance. To ensure the quality of the datasets, we
use the 10-core setting, i.e., retaining users and items with at least ten
interactions.

| Gowalla | Yelp2018 | Amazon-Book
---|---|---|---
#Users | 29,858 | 45,919 | 52,643
#Items | 40,981 | 45,538 | 91,599
#Interactions | 1.027m | 1.185m | 2.984m
Density | 0.084% | 0.056% | 0.062%
Table 1: Statistics of the datasets

###  5.2 Experimental Settings

To evaluate the effectiveness of top-K task in recommender system, we adopted
Recall@K and NDCG@K, which has been widely used in [19, 17]. In this paper, 1)
we set K = 20; 2) all items that the user has not interacted with are the
negative items; 3) all items is scored by each method in descend order except
the positive ones used in the training set. Average metrics for all users in
the test set is used for evaluation.

To verify the effectiveness of our approach, we compare it with the following
baselines:

  * •

BPR-MF [2] optimizes the matrix factorization with implicit feedback using a
pairwise ranking loss.

  * •

NCF [19] learns user’s and item’s embeddings from user–item interactions in a
matrix factorization, which by a MLP instead of the inner product .

  * •

PinSage [16] combines efficient random walks and graph convolutions to
generate embeddings of nodes that incorporate both graph structure as well as
node feature information.

  * •

GC-MC [10] is a graph auto-encoder framework based on differentiable embedding
passing on the bipartite interaction graph. The auto-encoder produces latent
user and item representations, and they are used to reconstruct the rating
links through a bilinear decoder.

  * •

NGCF [17] explicitly encodes the collaborative signal of high-order relations
by embedding propagation in user-item inter-action graph.

  * •

Multi-GCCF [9] constructs two separate user-user and item-item graphs. It
employs a multi-graph encoding layer to integrate the information provided by
the user-item, user-user and item-item graphs.

We implement GACSE444For reproducibility, we share the source code of GACSE
online: https://github.com/GACSE/GACSE.git with TensorFlow. The embedding size
is fixed to 64 for all models. All models are optimized with the Adam
optimizer, where the batch size is fixed at 1024. The learning rate of our
model was set to 0.0001; $\lambda_{1}$ was set to $1\times 10^{-4}$;
$\lambda_{2}$ was set to $1\times 10^{-5}$; Number of sampling neighbors was
set to 64. Number of positive and negative samples for similarity loss was set
to 5. All hyper-parameters of the above baselines are either followed the
suggestion from the methods’ author or turned on the validation sets. We
report the results of each baseline under its optimal hyper-parameter
settings.

###  5.3 Performance Comparison

| Gowalla | Yelp2018 | Amazon-Book
---|---|---|---
| Recall | NDCG | Recall | NDCG | Recall | NDCG
BPR-MF | 0.1291 | 0.1878 | 0.0494 | 0.0662 | 0.0250 | 0.0518
NCF | 0.1326 | 0.1985 | 0.0513 | 0.0719 | 0.0253 | 0.0535
PinSage | 0.1380 | 0.1947 | 0.0612 | 0.0750 | 0.0283 | 0.0545
GC-MC | 0.1395 | 0.1960 | 0.0597 | 0.0741 | 0.0288 | 0.0551
NGCF | 0.1547 | 0.2237 | 0.0581 | 0.0719 | 0.0344 | 0.0630
Multi-GCCF | 0.1595 | 0.2126 | 0.0667 | 0.0810 | 0.0363 | 0.0656
GACSE | 0.1654 | 0.2328 | 0.0672 | 0.0836 | 0.0386 | 0.0703
%Improv. | 3.70% | 4.06% | 0.75% | 3.21% | 6.34% | 7.16%
Table 2: Overview performance comparison. Bold scores are the best in each
column, while underlined scores are the second best. Improvements are
statistically significant.

####  5.3.1 Overall Comparison

Table 2 summarized the best results of all models on three benchmark dataset.
The last row is the improvements of GACSE relative to the best baseline.

BPR-MF method gives the worst performance on all datasets since the inner
product cannot capture complex collaborative signals. NCF outperforms BPR-MF
on all datasets consistently. Compared with BPR-MF, the main improvement of
NCF is that MLP can model the nonlinear feature interactions between user and
item embeddings.

Among all the baseline methods, graph based methods (e.g., PinStage, GC-MC,
NGCF, Multi-GCCF) consistently outperform general methods (e.g., BPR-MF, NCF)
on all datasets. The main improvement is that graph based model explicitly
models the graph structure in embedding learning.

Multi-GCCF are the strongest baseline. It outperforms other baselines on all
datasets except NDCG on Gowalla. NGCF gives the best performance of NDCG on
Gowalla. They all employ embedding propagation to obtain neighbor’s
information and stack multiple embedding propagation layers to explore the
high-order connectivity. This verifies the importance of capturing
collaborative signal in the embedding function. Moreover, Multi-GCCF compared
three different multi-grained representations fusion methods.

According to the results, GACSE preforms best among all baselines on three
datasets in terms of all evaluation metrics. It improves over the best
baseline method by $3.70\%$, $0.75\%$, $6.34\%$ in terms of Recall on Gowalla,
Yelp2018 and Amazon-book. It gains $4.06\%$, $3.21\%$, $7.16\%$ NDCG
improvements against the best baseline on Gowalla, Yelp2018 and Amazon-book
respectively. Compared with Multi-GCCF and NGCF, GACSE builds an embedding
propagation and aggregation architecture based on the attention mechanism. The
attention mechanism enable GACSE to learn variable weights of embedding
propagation for neighbors explicitly. Meanwhile it obtains high-order implicit
collaborative information between user-user and item-item through similarity
loss.

####  5.3.2 Ablation Analysis

| Gowalla | Yelp2018
---|---|---
| Recall | NDCG | Recall | NDCG
GACSE | 0.1654 | 0.2328 | 0.0672 | 0.0836
GACSE-sl | 0.1632 | 0.2334 | 0.0641 | 0.0805
(-1.33%) | (+0.26%) | (-4.61%) | (-3.71%)
GACSE-am | 0.1468 | 0.2149 | 0.0571 | 0.0728
(-11.25%) | (-7.68%) | (-15.03%) | (-12.92%)
Table 3: Ablation studies of GACSE. GACSE-sl means GACSE without similarity
loss. GACSE-am means GACSE without adaptive margin.

Table 3 reports the influences of similarity loss and adaptive margin of GACSE
on Gowalla and Yelp2018 datasets. As expected, the performance degrades
greatly after removing adaptive margin and similarity loss. This confirms the
importance of adaptive margin and similarity loss for embedding learning.
Adaptive margin can improve model’s discrimination for positive and negative
sample with similar embeddings. Similarity loss for both user-user and item-
item pair can reduce the sparsity problem by augmenting the data for
representation learning. Similarity loss and adaptive margin can enhance the
effectiveness of attention mechanism for embedding propagation and
aggregation.

####  5.3.3 Test Performance w.r.t. Epoch

Figure 2: Recall@20 on Gowalla

Figure 3: NDCG@20 on Gowalla

Figure 4: Recall@20 on Yelp

Figure 5: NDCG@20 on Yelp

Figure 7 shows the test performance w.r.t. recall of each epoch of MF and
NGCF. Due to the space limitation, we omit the performance w.r.t. ndcg which
has the similar trend. We can see that, NGCF exhibits fast convergence than MF
on three datasets. It is reasonable since indirectly connected users and items
are involved when optimizing the interaction pairs in mini-batch. Such an
observation demonstrates the better model capacity of NGCF and the
effectiveness of performing embedding propagation in the embedding space.

##  6 CONCLUSION AND FUTURE WORK

In this work, we explicitly incorporated collaborative signal and indirect
similarities into the embedding function. We proposed a unified representation
learning framework GACSE, in which the embedding is learned from direct user-
item interaction through attention propagation, and indirect user-user
similarities and item-item similarities through auxiliary loss, user-item
similarities in bipartite graph. In addition, we combine adaptive margin in
BPR loss and similarity loss to optimize GACSE. Extensive experimental results
on three real-world datasets show that our model outperforms state-of-the-art
baselines.

Several directions remain to be explored. A valuable direction is to
incorporate rich side information into GACSE instead of just modeling user &
item ids. Another interesting direction for the future work would be exploring
multi-task & multi-object embedding learning on heterogeneous graph for
recommender system.

## References

  * [1] Koren, Yehuda, and Robert Bell. ”Advances in collaborative filtering.” Recommender systems handbook. Springer, Boston, MA, 2015. 77-118.
  * [2] Rendle, Steffen, et al. ”BPR: Bayesian personalized ranking from implicit feedback.” arXiv preprint arXiv:1205.2618 (2012).
  * [3] Su, Xiaoyuan, and Taghi M. Khoshgoftaar. ”A survey of collaborative filtering techniques.” Advances in artificial intelligence 2009 (2009).
  * [4] Koren, Yehuda. ”Factorization meets the neighborhood: a multifaceted collaborative filtering model.” Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 2008.
  * [5] Linden, Greg, Brent Smith, and Jeremy York. ”Amazon. com recommendations: Item-to-item collaborative filtering.” IEEE Internet computing 7.1 (2003): 76-80.
  * [6] Kim, Donghyun, et al. ”Convolutional matrix factorization for document context-aware recommendation.” Proceedings of the 10th ACM conference on recommender systems. 2016.
  * [7] Sedhain, Suvash, et al. ”Autorec: Autoencoders meet collaborative filtering.” Proceedings of the 24th international conference on World Wide Web. 2015.
  * [8] Kipf, Thomas N., and Max Welling. ”Semi-supervised classification with graph convolutional networks.” arXiv preprint arXiv:1609.02907 (2016).
  * [9] Sun, Jianing, et al. ”Multi-Graph Convolution Collaborative Filtering.” 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 2019.
  * [10] Berg, Rianne van den, Thomas N. Kipf, and Max Welling. ”Graph convolutional matrix completion.” arXiv preprint arXiv:1706.02263 (2017).
  * [11] Wu, Yao, et al. ”Collaborative denoising auto-encoders for top-n recommender systems.” Proceedings of the Ninth ACM International Conference on Web Search and Data Mining. 2016.
  * [12] Koren, Yehuda, Robert Bell, and Chris Volinsky. ”Matrix factorization techniques for recommender systems.” Computer 42.8 (2009): 30-37.
  * [13] Mnih, Andriy, and Russ R. Salakhutdinov. ”Probabilistic matrix factorization.” Advances in neural information processing systems. 2008.
  * [14] Ebesu, Travis, Bin Shen, and Yi Fang. ”Collaborative memory network for recommendation systems.” The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018.
  * [15] Yang, Jheng-Hong, et al. ”HOP-rec: high-order proximity for implicit recommendation.” Proceedings of the 12th ACM Conference on Recommender Systems. 2018.
  * [16] Ying, Rex, et al. ”Graph convolutional neural networks for web-scale recommender systems.” Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018.
  * [17] Wang, Xiang, et al. ”Neural graph collaborative filtering.” Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval. 2019.
  * [18] Hamilton, Will, Zhitao Ying, and Jure Leskovec. ”Inductive representation learning on large graphs.” Advances in neural information processing systems. 2017.
  * [19] He, Xiangnan, et al. ”Neural collaborative filtering.” Proceedings of the 26th international conference on world wide web. 2017.
  * [20] Christakopoulou, Evangelia, and George Karypis. ”Local item-item models for top-n recommendation.” Proceedings of the 10th ACM Conference on Recommender Systems. 2016.
  * [21] Barkan, Oren, and Noam Koenigstein. ”Item2vec: neural item embedding for collaborative filtering.” 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2016.
  * [22] Hu, Yifan, Yehuda Koren, and Chris Volinsky. ”Collaborative filtering for implicit feedback datasets.” 2008 Eighth IEEE International Conference on Data Mining. Ieee, 2008.
  * [23] Veličković, Petar, et al. ”Graph attention networks.” arXiv preprint arXiv:1710.10903 (2017).
  * [24] Wang, Xiang, et al. ”Kgat: Knowledge graph attention network for recommendation.” Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.
  * [25] Grover, Aditya, and Jure Leskovec. ”node2vec: Scalable feature learning for networks.” Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. 2016.
  * [26] Chen, Chih-Ming, et al. ”Collaborative similarity embedding for recommender systems.” The World Wide Web Conference. 2019.
  * [27] Liang, Dawen, et al. ”Factorization meets the item embedding: Regularizing matrix factorization with item co-occurrence.” Proceedings of the 10th ACM conference on recommender systems. 2016.
  * [28] Kingma, Diederik P., and Jimmy Ba. ”Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014).