# Model-Agnostic Decentralized Collaborative Learning for On-Device POI
Recommendation

Jing Long<EMAIL_ADDRESS>1234-5678-9012 The University of
QueenslandBrisbaneQLD4072Australia ,  Tong Chen<EMAIL_ADDRESS>The
University of QueenslandBrisbaneQLD4072Australia ,  Nguyen Quoc Viet Hung
<EMAIL_ADDRESS>Griffith UniversityGold CoastQLD4222Australia ,
Guandong Xu<EMAIL_ADDRESS>University of Technology
SydneySydneyNSW2007Australia ,  Kai Zheng<EMAIL_ADDRESS>School of
Computer Science and Engineering and Shenzhen Institute for Advanced Study,
University of Electronic Science and Engineering of
ChinaChengduSichuan610000China and  Hongzhi Yin<EMAIL_ADDRESS>The
University of QueenslandBrisbaneQLD4072Australia

(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

As an indispensable personalized service in Location-based Social Networks
(LBSNs), the next Point-of-Interest (POI) recommendation aims to help people
discover attractive and interesting places. Currently, most POI recommenders
are based on the conventional centralized paradigm that heavily relies on the
cloud to train the recommendation models with large volumes of collected
users’ sensitive check-in data. Although a few recent works have explored on-
device frameworks for resilient and privacy-preserving POI recommendations,
they invariably hold the assumption of model homogeneity for
parameters/gradients aggregation and collaboration. However, users’ mobile
devices in the real world have various hardware configurations (e.g., compute
resources), leading to heterogeneous on-device models with different
architectures and sizes. In light of this, We propose a novel on-device POI
recommendation framework, namely Model-Agnostic Collaborative learning for on-
device POI recommendation (MAC), allowing users to customize their own model
structures (e.g., dimension & number of hidden layers). To counteract the
sparsity of on-device user data, we propose to pre-select neighbors for
collaboration based on physical distances, category-level preferences, and
social networks. To assimilate knowledge from the above-selected neighbors in
an efficient and secure way, we adopt the knowledge distillation framework
with mutual information maximization. Instead of sharing sensitive
models/gradients, clients in MAC only share their soft decisions on a
preloaded reference dataset. To filter out low-quality neighbors, we propose
two sampling strategies, performance-triggered sampling and similarity-based
sampling, to speed up the training process and obtain optimal recommenders. In
addition, we design two novel approaches to generate more effective reference
datasets while protecting users’ privacy. Extensive experiments on two
datasets have shown the superiority of MAC over advanced baselines.

Point-of-Interest Recommendation; Decentralized Collaborative Learning; Model-
Agnostic

*Corresponding author

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information systems Recommender
systems

##  1\. Introduction

Recently, the rapid growth of Location-based Social Networks (e.g., Weeplace
and Foursquare) has boosted the popularity of next Point-of-Interest (POI)
recommendation, which facilitates diverse applications such as mobility
prediction, route planning, and location-based advertising (Li et al., 2018).
In personalized POI recommendation, various state-of-the-art methods based on
attentive neural networks (Lian et al., 2020; Luo et al., 2021; Yin et al.,
2017) and graph networks (Li et al., 2021a; Wang et al., 2021b) have recently
achieved quality recommendation performance given large volumes of historical
check-in data. To support large-scale recommendation services, a powerful
cloud server is commonly required to host all users’ data, and to perform
training and inference of the recommender, making the maintenance of such
services monetarily and ecologically expensive (Long et al., 2022). In
addition, viewing privacy as a priority, users are increasingly reluctant to
share their personal check-in trajectories with the service provider, impeding
recommendation quality. Besides, in this paradigm, users’ devices only act as
a terminal for transmitting user data and displaying server-generated
recommendations. As a result, could-based POI recommendation is overly reliant
on the server’s capacity and internet connectivity (Wang et al., 2020; Yin et
al., 2016), weakening its resilience.

As such, recent research resorts to on-device POI recommendations (Long et
al., 2022; Wang et al., 2020; Guo et al., 2021) to counter the shortfalls of
centralized paradigms. The core objective of this new paradigm is to deploy an
accurate yet relatively lightweight recommendation model on the user side,
such that recommendations can be generated in a fully on-device fashion with
locally acquired user data. To fulfill this goal, a straightforward solution
in the literature is based on model compression (Bengio et al., 2013; Chen et
al., 2015; Hinton et al., 2015). In the context of on-device POI
recommendation, model compression methods aim to condense a sophisticated POI
recommender into a compact model to support on-device inference. For instance,
a teacher-student distillation framework is utilized in (Wang et al., 2020) to
transfer knowledge from the powerful teacher model to the compact student
model. The well-trained student model is then deployed on mobile devices for
all users. However, the framework is still resource-intensive due to the
server’s full engagement in training both the teacher and student models.
Furthermore, all users are assigned the same model and there are no update
mechanisms to account for the dynamics of spatial activities and diversity of
user interests, leading to suboptimal performance.

Consequently, recent research is more inclined towards collaborative learning
(CL) frameworks, which allow models/gradients sharing. Thus, CL is less
demanding on complex model designs and more suited to the dynamic nature of
POI recommendation tasks. In CL, federated learning-based POI recommenders
(e.g. (Guo et al., 2021)) are a highly representative solution, where users
train their models locally while retaining all private data on-device. To
avoid sparsity of local data, a central server is to iteratively collect and
aggregate these trained local models, and then redistribute the aggregated
model to all users. However, aggregating all local models into a single global
model helps the recommender to generalize, but amplifies bias towards active
users’ preference (Wang et al., 2021a). So, further remedies are proposed in
federated POI recommenders (Xw et al., 2020; Rao et al., 2021; Imran et al.,
2023), which group similar users and perform group-wise aggregation to allow
for more personalization of learned on-device models.

Despite the improved performance and flexibility over model compression
approaches, CL-based methods still bear deficiencies for on-device POI
recommendations in the real world. Firstly, a central server is always
involved throughout the learning process of federated POI recommenders. This
entanglement only gets stronger with the need for repetitively identifying
user groups via model parameter comparison or clustering on the server. Given
that, the semi-decentralized learning paradigm proposed by (Long et al., 2022)
can be viewed as one step above federated recommenders that can significantly
lower the dependency on a central server. In short, the server only needs to
provide pretrained model parameters and assign similar users to the same
group, which is a one-off engagement in the early stage of training.
Afterward, users’ on-device models are optimized by alternating local training
and inter-device communication within the same user group. Unfortunately, as
the intra-group collaborations require exchanging raw model parameters between
users (Long et al., 2022), it breaks the purposes of privacy and communication
efficiency in a decentralized POI recommender. Secondly, there has been a
strong assumption that user-specific knowledge can be aggregated via model
parameters or gradients. However, the model homophily assumption significantly
harms the practicality of such decentralized POI recommendation paradigms, as
in practice, each on-device model should have a customized structure to meet
the specific device capacity (Chen et al., 2021). With an exponentially
increasing diversity of user devices capable of delivering POI
recommendations, this assumption has to be relaxed to allow a host of
structurally heterogeneous on-device models to be jointly optimized. As such,
there still lacks a capable CL-based POI recommendation paradigm that can
accommodate heterogeneous on-device recommenders, minimize dependency on the
central server, and ensure all on-device recommenders’ expressiveness without
incurring privacy breaches or a heavy communication overhead.

To this end, we propose a novel on-device POI recommendation framework, namely
Model-Agnostic Collaborative learning for on-device POI recommendation (MAC).
In MAC, users are allowed to tailor the model configurations to suit their
device capacities. As the POI embeddings are the major source of memory
consumption in the recommender (Wang et al., 2020; Li et al., 2021b), we
propose to store embeddings only for POIs with high relevance to the current
location context on the user device. The core solution to model-agnostic CL in
MAC is a novel knowledge distillation (KD) scheme that facilitates knowledge
exchange between heterogeneous recommendation models. Instead of sharing
sensitive model parameters/gradients for equidimensional aggregations in
homogeneous settings, models in MAC only share their soft decisions on a
reference dataset shared among all users. In the KD process, we let models
mutually incorporate knowledge from their neighbors with mutual information
maximization w.r.t. the soft decisions exchanged. On the one hand, using soft
decisions on the reference data bypasses the equidimensional constraint for
learning decentralized models, thus scaling CL-based POI recommendation to a
larger user base with heterogeneous devices and on-device models. On the other
hand, the soft decisions generated from the desensitized reference data are
immune to breaching user-sensitive information, which also brings a
substantially small communication footprint compared with model aggregation
(Long et al., 2022) approaches. Apparently, for each user, letting her model
collaborate with all other models is computationally prohibitive and prone to
lack of personalization. Given this, we propose identifying neighbors for
communication regarding physical distances, category preferences, and social
network distance. However, simple preliminary screening fails to filter out
worthless clients and we further propose two neighbor sampling strategies,
namely performance-triggered sampling and similarity-based sampling, to speed
up the training process and obtain optimal recommenders. Another concern of
the normal knowledge distillation is that previous works fail to consider the
source of the reference datasets, and they just split the whole dataset to get
the reference dataset (Bistritz et al., 2020; Ye et al., 2022). However,
simply treating users’ historical data as reference datasets has privacy
concerns, and it will fail in the cold start scenario. Considering this, we
design two novel approaches, namely transformative generating and
probabilistic generating, to build more practical and effective reference
datasets while protecting users’ privacy. The primary contributions of this
study are as follows:

  * •

We propose a novel on-device POI recommendation framework, namely Model-
Agnostic Collaborative learning for on-device POI recommendation (MAC), where
users are allowed to tailor the model configurations and store embeddings only
for POIs with high relevance to the current context. The core solution to
model-agnostic collaborative learning (CL) is a novel knowledge distillation
(KD) scheme that facilitates knowledge exchange between heterogeneous
recommendation models.

  * •

To filter out low-quality neighbors, we design two sampling strategies, namely
performance-triggered and similarity-based sampling, to speed up the training
process and obtain optimal user-specific recommenders. In addition, we propose
two novel approaches, namely transformative generating and probabilistic
generating, which can build more practical reference datasets while protecting
users’ privacy.

  * •

We evaluate MAC with two real-world datasets. The experimental results show
the effectiveness and efficiency of the proposed model in terms of privacy
protection, recommendation accuracy, and the utilization efficiency of on-
device computing resources.

Figure 1. The overview of our proposed MAC.

##  2\. PRELIMINARIES

In this section, we first introduce important notations used in this paper and
then formulate our core task.

We denote the sets of users $u$, POIs $p$ and categories $c$ as $\mathcal{U}$,
$\mathcal{P}$, $\mathcal{C}$, respectively. Each POI $p\in\mathcal{P}$ is
associated with a category tag (e.g., entertainment or restaurant)
$c_{p}\in\mathcal{C}$ and coordinates $(lon_{p},lat_{p})$.

Definition 1: Check-in Sequence. A check-in activity of a user indicates a
user $u\in\mathcal{U}$ has visited POI $p\in\mathcal{P}$ at timestamp $t$. By
sorting a user’s check-ins chronologically, a check-in sequence contains
$M_{i}$ consecutive POIs visited by a user $u_{i}$, denoted by
$\mathcal{X}(u_{i})=\\{p_{1},p_{2},...,p_{M_{i}}\\}$.

Definition 2: Category Sequence. A category sequence substitutes all POIs in
the check-in sequence $\mathcal{X}(u_{i})$ with their associated category
tags, denoted by
$\mathcal{X}^{c}(u_{i})=\\{c_{p_{1}},c_{p_{2}},...,c_{p_{M_{i}}}\\}$.

Definition 3: Region. A region $r$ is essentially a geographical segment that
can provide additional information about the POIs within it. Without any
assumptions on predefined city districts/suburbs, we obtain a set of regions
$\mathcal{R}$ by applying $k$-means clustering (Macqueen, 1967) on all POIs’
coordinates in our paper.

Definition 4: Geographical Reference Dataset. A geographical reference dataset
$\mathcal{D}^{g}(r)=\\{\mathcal{X}_{v}\\}_{v=1}^{V_{r}}$ for region $r$
contains $V_{r}$ anonymous check-in sequences. Each region has its dedicated,
unique reference dataset, of which the check-in activities only cover POIs in
region $r$. Each user $u$ only holds region-specific reference datasets of
regions she is currently or previously at.

Definition 5: Semantic Reference Dataset. A semantic reference dataset
$\mathcal{D}^{s}=\\{\mathcal{X}^{c}_{z}\\}_{z=1}^{Z}$ contains $Z$ anonymous
categorical sequences covering all POI categories $\mathcal{C}$. The semantic
reference dataset is universal and shared among all users. We will detail our
strategies for generating both $\mathcal{D}^{s}$ and $\mathcal{D}^{g}$ in
Section 3.

Task 1: Decentralized Next POI Recommendation. In MAC, the roles of
devices/users and the central server are defined below:

  * •

User/Device: Each user $u_{i}$ holds her data $\mathcal{X}(u_{i})$,
$\mathcal{X}^{c}(u_{i})$ and a structurally personalized model
$\phi_{i}(\cdot)$ that jointly trained with the local data and enhanced by
collaborating with other users. To save storage, we assume that the model
$\phi_{i}(\cdot)$ only stores the embeddings of POIs from the regions she is
currently or previously at, denoted by $r\in\mathcal{R}(u_{i})$.

  * •

Server: The server is responsible for identifying neighbor sets for all users
with the low-sensitivity data collected, as well as gathering reference
datasets and transmitting this information back to users.

Then, by enabling model-agnostic collaborations between users, we aim to learn
a performant local model for each user, which estimates a ranked list of
possible POIs for her next movement.

##  3\. The Framework

This section introduces our design of the MAC framework, where an overview is
depicted in Figure 1. The main components include: (1) A local objective
function that guides the optimization of each user’s on-device model. (2) A
neighbor identification process for securely identifying cohesive user groups,
operated under the cooperation between local devices and the central server.
(3) Two data generation methods for constructing quality yet anonymous
reference datasets. (4) A knowledge distillation-based collaborative learning
scheme with neighbor sampling that allows refined knowledge sharing within
model-agnostic user groups.

###  3.1. Local Objective Function

The main objective of MAC is to enhance personalized POI recommenders that are
locally trained with users’ own check-in histories. To obtain locally trained
recommenders at the first place, we define the following local objective
function:

(1)
$L_{loc}(u_{i})=l\left(\phi_{i}\left(\mathcal{X}(u_{i})\right),\mathcal{Y}(u_{i})\right),$

where $\phi_{i}\left(\mathcal{X}(u_{i})\right)$ is the prediction made by the
recommender $\phi_{i}(\cdot)$ given $\mathcal{X}(u_{i})$. In the POI
recommendation setting, the predictions are made successively on historical
POI sequences $\\{p_{1}\\},\\{p_{1},p_{2}\\},...,\\{p_{1},$
$p_{2},...,p_{M_{i}-1}\\}$, and
$\mathcal{Y}(u_{i})=\\{p_{2},p_{3},...,p_{M_{i}}\\}$ is the set of
corresponding ground truth POIs. $l$ is the loss function (i.e., cross-entropy
in our case) to quantify the prediction error. It is worth noting that, MAC is
compatible with most of the deep neural network POI recommenders as its base
model, where the key innovation of our work lies in the distributed
collaborative learning (CL) paradigm that coordinates with individual POI
recommenders deployed on-device.

###  3.2. Neighbor Identification

To alleviate data scarcity compared with learning individual on-device
recommenders in silos and reduce possible noise and overhead compared with
communicating with all user devices, each user device $u_{i}$ in MAC is
allowed to exchange knowledge with its neighbors, which are a subset of users
having high affinity to $u_{i}$. For a robust CL process, we present two
parallel strategies to identify quality neighbors for $u_{i}$ below.

Geographical Neighbors. Given the location-sensitive nature of POI
recommendation tasks, users have high geographical affinity if they frequently
visit venues in the same region $r\in\mathcal{R}$, thus making geographically
similar users’ information mutually beneficial for future movement prediction.
We term such users as geographical neighbors of $u_{i}$, denoted by
$\mathcal{G}(u_{i})$. As mentioned above, with a moderate granularity of
$\mathcal{R}$, we can identify a relatively dense neighbor set without risking
the exposure of user privacy. Formally, we use
$\mathcal{R}(u_{i})=\\{r^{i}_{0},r^{i}_{1},r^{i}_{2},...,r^{i}_{R_{i}}\\}$ to
represent all regions visited by $u_{i}$, where $r^{i}_{0}$ denotes the user’s
current region (that corresponds to $p_{M_{i}}$) while all others are regions
the user has been to. In our definition, for two users $u_{i}$ and $u_{j}$,
only if $u_{j}$ has visited $u_{i}$’s current region, i.e.,
$r^{i}_{0}\in\mathcal{R}(u_{j})$, $u_{j}$ is regarded as the geographical
neighbor of $u_{i}$. Note that, this relationship is directional, and we only
identify $u_{i}$’s neighbors via $r^{i}_{0}$ instead of all historical
regions, because the prediction on $u_{i}$’s next POI is mainly based on her
most recent check-in $p_{M_{i}}$.

Semantic Neighbors. Meanwhile, users’ relevance is manifested in not only
being physically close to each other, but also being similar at the semantic
level. That is, even if being geographically distant, users can be highly
relevant if their visited POIs fall into the same categories (Li et al.,
2021a), and we term such users as semantic neighbors of $u_{i}$, denoted by
$\mathcal{S}(u_{i})$. On this basis, we leverage category-level user
preferences to quantify two users’ semantic similarity. Formally, we use
$CP(u_{i})=\\{\mathbb{P}(c_{1}),\mathbb{P}(c_{2}),...\mathbb{P}(c_{|\mathcal{C}|})\\}$
to denote each user’s distribution over all $|\mathcal{C}|$ POI categories,
which can be easily observed from $\mathcal{X}^{c}(u_{i})$. Then, we adopt
Kullback-Leibler (KL) divergence (Giantomassi et al., 2014) to quantify the
distance between two users’ categorical preferences:

(2) $d_{cat}(u_{i},u_{j})=KL\left(CP(u_{i})\left|\right|CP(u_{j})\right).$

For user $u_{i}$, we regard $h$ users with the smallest distances
$d_{cat}(u_{i},u_{j})$ as semantic neighbors. we also account for the explicit
social connections by adding users who are actual friends with $u_{i}$ into
$\mathcal{S}(u_{i})$.

Notably, in our decentralized setting, with their own check-in data, users
cannot get the information to decide their geographical/semantic neighbors. As
such, a central server is maintained to collect related regions
$\mathcal{R}(u_{i})$ and categorical preferences $CP(u_{i})$ computed from
$\mathcal{X}^{c}(u_{i})$, and thus, the server can inform all users of their
neighbor IDs. Compared with transmitting raw trajectories or models, the
privacy risk of uploading related regions and categorical preferences is
substantially lower (Long et al., 2022).

###  3.3. Generating Anonymous Reference Data

Given the geographical neighbors $\mathcal{G}(u_{i})$ and semantic neighbors
$\mathcal{S}(u_{i})$, the new challenge is how to extract their knowledge to
enhance each user’s personalized model. Although model aggregation has been
proven effective in previous works (Guo et al., 2021; Long et al., 2022), it
requires all participants to hold homogeneous models, and the recommendation
utility is restricted by the device with the least computational resources.
Thus, parameter sharing is impractical with the increasing diversity of
personal devices and their varying computational capacity. As a solution, we
propose a novel knowledge distillation-based CL method. Rather than
transmitting gradients or weights, clients only need to share their soft
decisions on desensitized reference datasets with their neighbors.
Essentially, with the same reference dataset, those generated soft decisions
are a compact characterization of each on-device model. Besides, compared with
exchanging gradients/models, the use of soft decisions can substantially lower
the communication cost and is far less sensitive.

Previous CL frameworks (Bistritz et al., 2020; Ye et al., 2022) have tried
assigning the same reference dataset across all users to facilitate such
model-agnostic knowledge exchange, but they are subject to strong limitations
in POI recommendation tasks. Firstly, the reference dataset is supposed to
cover all regions’ POIs, allowing all users to produce probability
distributions that are mutually comparable. Apparently, this will harm the
quality of soft decisions, since each user’s local model is learned with
geographically constrained data, and can hardly make reliable predictions on
POIs outside familiar regions. Secondly, storing and processing check-in
sequences covering a full set of POIs is a heavy burden for user devices,
marking down the practicality of MAC. Thirdly, despite the important role of
the reference datasets in CL, a common practice to obtain them (Bistritz et
al., 2020; Ye et al., 2022) is to draw a subset of real samples from the user
data, which negatively affects user privacy and thus hurts real-world
usability.

Geographical and Semantic Reference Datasets. For knowledge sharing among
geographical neighbors, to guarantee each local model’s prediction quality
over the reference dataset, we prepare one reference dataset for each region,
where each region-specific reference dataset
$\mathcal{D}^{g}(r)=\\{\mathcal{X}_{v}\\}_{v=1}^{V}$ contains $V$ anonymous
check-in sequences in region $r$. Given this, user $u_{i}$ can communicate
with her geographical neighbors by comparing their soft decisions w.r.t. each
check-in sequence in $\mathcal{D}^{g}(r^{i}_{0})$ (recall that $r^{i}_{0}$ is
$u_{i}$’s current region). Also, as now each $\mathcal{D}^{g}(r)$ only
contains region-specific POIs, the amount of POI information needed to be
stored is substantially reduced. Unfortunately, this strategy is inapplicable
to CL with semantic neighbors, as $u_{i}$ and her semantic neighbors might be
geographically distant from each other, and their locally trained models may
exhibit divergent behaviors on the same region-specific reference dataset.
Hence, instead of predicting the actual POIs, we let them predict high-level
categorical transitions on a semantic reference dataset
$\mathcal{D}^{s}=\\{\mathcal{X}^{c}_{z}\\}_{z=1}^{Z}$ with $Z$ anonymous
categorical sequences, defined in Section 2. Analogously, knowledge sharing
between $u_{i}$ and her semantic neighbors is enabled by computing soft
decisions upon $\mathcal{D}^{s}$. Given that only categorical transitions are
modeled with this dataset and the number of POI categories is relatively
small, sharing one universal semantic reference dataset will meet both quality
and memory efficiency demands.

Meanwhile, the challenge from privacy aspect still needs further attention. To
maintain all users’ privacy when building reference datasets, we resort to
pseudo data as a natural solution. Specifically, we put forward two data
generation strategies, namely transformative generation and probabilistic
generation, which allow MAC to produce sensible yet anonymous trajectories. We
introduce both strategies below, which are adopted alternatively in MAC.

Transformative Generation. Instead of directly publishing users’ check-in
sequences, we propose to obfuscate them by a series of transformations, which
are widely used for augmenting time series data (Guennec et al., 2016; Fawaz
et al., 2018; Flores et al., 2021). In a nutshell, we mix and match two users’
check-in/category sequences via a decompose-recompose pipeline, where the
revamped sequences differ from but imitate original ones. As the raw check-ins
are exchanged in this process, we only allow this to happen between two users
$u_{i}$ and $u_{m}$ if (1) they are mutual friends; and (2) they have visited
at least one same POI, which can be judged once $u_{m}$ sends her check-in
sequence to $u_{i}$. Specifically, with two check-in sequences
$\mathcal{X}(u_{i})$, $\mathcal{X}(u_{j})$, we decompose each of them into two
subsequences by slicing from a common POI
$p^{ij}\in\mathcal{X}(u_{i})\cap\mathcal{X}(u_{j})$, i.e.,
$\mathcal{X}^{\prime}(u_{i})=\\{p^{i}_{1},...,p^{ij}\\}$ and
$\mathcal{X}^{\prime\prime}(u_{i})=\\{p^{ij},...,p^{i}_{M_{i}}\\}$ for
$\mathcal{X}(u_{i})$, and
$\mathcal{X}^{\prime}(u_{j})=\\{p^{j}_{1},...,p^{ij}\\}$ and
$\mathcal{X}^{\prime\prime}(u_{j})=\\{p^{ij},...,p^{j}_{M_{j}}\\}$ for
$\mathcal{X}(u_{j})$. Then, we join $\mathcal{X}^{\prime}(u_{i})$ with
$\mathcal{X}^{\prime\prime}(u_{j})$, and $\mathcal{X}^{\prime}(u_{j})$ with
$\mathcal{X}^{\prime\prime}(u_{i})$ to form two new sequences
$\mathcal{X}^{g}_{1}$ and $\mathcal{X}^{g}_{2}$, where a repetitive $p^{ij}$
is deleted from both sequences. For both newly generated sequences, we specify
their region(s) $r$ as the most frequently visited one, and discard POIs out
of region $r$ to guarantee all generated POI sequences are region-specific.
Analogously, the universal semantic reference dataset $\mathcal{D}^{s}$ can be
constructed via transformative generation on users’ category sequences, while
the additional step of region-specific processing is not required.

Probabilistic Generation. Though only perturbed data is used in transformative
generation, it originates from users’ raw historical data. To meet stricter
privacy restrictions and deal with regions with limited check-ins, we propose
an alternative that only leverages the statistics from users’ insensitive
category sequences for data generation, which we call probabilistic
generation. Firstly, we calculate the conditional probabilities of all
categories based on the category sequences submitted by all users. Formally,
we use
$P(c_{n})=\\{\mathbb{P}(c_{n}|c_{1}),\mathbb{P}(c_{n}|c_{2}),...,\mathbb{P}(c_{n}|c_{m}),...,\mathbb{P}(c_{n}|c_{|C|})\\}$
to represent all conditional probabilities for $c_{n}$, and each
$\mathbb{P}(c_{n}|c_{m})$ is calculated as:

(3)
$\mathbb{P}(c_{n}|c_{m})=\frac{\mu(c_{n}|c_{m})}{\sum_{m^{\prime}=1}^{|C|}\mu(c_{n}|c_{m^{\prime}})},$

where $\mu(c_{n}|c_{m})$ denotes the number of occurrences of $c_{m}$ right
after $c_{n}$. Then, a pseudo category sequence $\mathcal{X}^{s}$ can be
generated by: (1) selecting a category randomly as the starting point; and
then (2) repeatedly deciding subsequent categories according to the
conditional probabilities of its previous category. By iteratively generating
multiple $\mathcal{X}^{c}$, we can create a semantic reference dataset
$\mathcal{D}^{s}$ that cover all categories. The next step is to generate POI
sequences based on the semantic reference dataset. Within each region, we
randomly sample a sequence of POIs that correspond to the category sequence
$\mathcal{X}^{c}\in\mathcal{D}^{s}$, where we further restrict the distance
between any two consecutive POIs to be less than 5km.

###  3.4. Collaboration via Knowledge Distillation

With the reference dataset ready, we design a novel knowledge distillation-
based CL protocol to extract knowledge from soft decisions received from
geographical neighbors $\mathcal{G}(u_{i})$ and semantic neighbors
$\mathcal{S}(u_{i})$.

Collaborative Learning with Geographical Neighbors. Given user $u_{i}$ and the
geographical reference dataset of her current region
$\mathcal{D}^{g}(r^{i}_{0})$, the CL with her geographical neighbors
$\mathcal{G}(u_{i})$ is facilitated by minimizing their disagreement of soft
decisions on $\mathcal{D}^{g}(r^{i}_{0})$, which is quantified as:

(4)
$L_{geo}=\frac{1}{|\mathcal{G}(u_{i})|}\sum_{u_{j}\in\mathcal{G}(u_{i})}\bigg{(}\sum_{\mathcal{X}\in\mathcal{D}^{g}(r^{i}_{0})}\left|\left|\phi_{i}\left(\mathcal{X}\right)-\phi_{j}(\mathcal{X})\right|\right|^{2}_{2}\bigg{)},$

where $\phi_{i}(\cdot)$ and $\phi_{j}(\cdot)$ respectively denote the local
recommenders possessed by $u_{i}$ and her neighbors.

Collaborative Learning with Semantic Neighbors. Similarly, we can also align
the soft decisions on the semantic reference dataset $\mathcal{D}^{s}$ between
$u_{i}$ and $\mathcal{S}(u_{i})$. However, as only category tags are recorded
in the reference dataset and hence predictions can only be made on categorical
preferences, we cannot directly update the POI embeddings in $\phi_{i}(\cdot)$
by comparing users’ soft decisions. Instead, we design a two-step strategy to
acquire knowledge from semantic neighbors. Firstly, we update the category
embeddings in $\phi_{i}(\cdot)$ via the following:

(5)
$L_{cat}=\frac{1}{|\mathcal{S}(u_{i})|}\sum_{u_{j}\in\mathcal{S}(u_{i})}\bigg{(}\sum\limits_{\mathcal{X}^{c}\in\mathcal{D}^{s}}\left|\left|\psi_{i}\left(\mathcal{X}^{c}\right)-\psi_{j}(\mathcal{X}^{c})\right|\right|^{2}_{2}\bigg{)},$

where $\psi_{i}(\cdot)$ is a lightweight predictor that infers a probability
distribution over all categories from a historical category trajectory
$X^{c}\in\mathcal{D}^{s}$. Hereby, since the embeddings of a POI $p$ and its
associated category $c_{p}$ can be treated as two views of the POI, we further
update POI embeddings by maximizing the mutual information (MI) between these
two views:

(6) $L_{MI}=-\sum_{\forall p}\log\frac{\exp({f(p,c_{p}))}}{\sum_{\forall
c^{\prime}\neq c_{p}}\exp(f(p,c^{\prime}))},$

where $f(\cdot)$ is a function of the pairwise similarity between the POI and
the category. In MAC it is implemented by a bilinear network that takes the
POI and category embeddings as its input:

(7) $f(p,c)=\sigma(\textbf{e}_{p}^{\top}\textbf{W}\textbf{e}_{c}),$

where $\textbf{e}_{p}\in\mathbb{R}^{d}$ and $\textbf{e}_{c}\in\mathbb{R}^{d}$
are POI and category embeddings, and $\textbf{W}\in\mathbb{R}^{d\times d}$ is
a learnable weight matrix. By optimizing the loss function, we can maximize
the mutual information between positive pairs while minimizing that between
negative pairs. Then, the final loss for collaborative learning with semantic
neighbors is defined as the ensemble of both parts:

(8) $L_{sem}=L_{cat}+L_{MI}.$

Algorithm 1 Optimizing MAC. All processes are implemented on user’s side
unless specified.

1:/*Server-side engagement starts*/

2:Server receives $R(u_{i})$, $CP(u_{i})$ from all $u_{i}\in U$;

3:Generate $\mathcal{G}(u_{i})$ and $\mathcal{S}(u_{i})$ w.r.t. $R(u_{i})$ and
$CP(u_{i})$ for all $u_{i}\in\mathcal{U}$;

4:Compose $\mathcal{D}^{g}(r)$ for all $r\in\mathcal{R}$ and
$\mathcal{D}^{sem}$ via transformative or probabilistic generation;

5:Send $\mathcal{G}(u_{i})$, $\mathcal{S}(u_{i})$,
$\mathcal{D}^{g}(r^{i}_{0})$, $\mathcal{D}^{s}$ to each $u_{i}\in U$;

6:/*Server engagement ends*/

7:for $u_{i}\in\mathcal{U}$ do in parallel

8: Impute $\mathcal{S}(u_{i})$ with $u_{i}$’s social friends;

9: Initialize $\phi_{i}(\cdot)$, $\mathcal{G}^{\prime}(u_{i})$,
$\mathcal{S}^{\prime}(u_{i})$ for all $u_{i}\in\mathcal{U}$;

10: repeat

11: Receive soft decisions on $\mathcal{D}^{g}(r^{i}_{0})$, $\mathcal{D}^{s}$
respectively

12: from $\mathcal{G}^{\prime}(u_{i})$, $\mathcal{S}^{\prime}(u_{i})$;

13: Receive soft decisions on $\mathcal{D}^{c}$ from $N^{s}_{cat}(u_{i})$;

14: Take a gradient step w.r.t. $L_{loc}+\gamma(\mu L_{geo}+(1-\mu)L_{sem})$

15: to update $\phi_{i}(\cdot)$;

16: Update $\mathcal{G}^{\prime}(u_{i})$, $\mathcal{S}^{\prime}(u_{i})$ with
performance-triggered or

17: similarity-based sampling;

18: until convergence

19:end for

###  3.5. Learning Local Recommenders with Dynamic Neighbor Sampling

The optimization workflow of MAC is presented in Algorithm 1. At the very
beginning, we perform neighbor identification and prepare all reference
datasets on the server side (lines 1-3). Then, the server sends the
corresponding reference datasets and neighbor information back to all users
(line 4), where its engagement halts. On the device side (lines 5-14), each
user will iteratively update its local recommender based on the synergic loss
$L_{loc}+\gamma(\mu L_{geo}+(1-\mu)L_{sem})$, where $\gamma$ and $\mu$ jointly
controls the strength of different knowledge distillation components.

Meanwhile, while the neighbor identification process provides $u_{i}$ with
abundant users that are likely to contribute to learning $\phi_{i}(\cdot)$,
not all users in $\mathcal{G}(u_{i})$/$\mathcal{S}(u_{i})$ are necessarily
valuable and might conversely dilute the local model’s quality. On top of the
influence of noise, it is also inefficient to communicate with all predefined
neighbors in every optimization step. Hence, we further propose two neighbor
sampling strategies to draw a small subset of $\alpha$ users
$\mathcal{G}^{\prime}(u_{i})\in\mathcal{G}(u_{i})$ and
$\mathcal{S}^{\prime}(u_{i})\in\mathcal{S}(u_{i})$ for CL. By swapping the
full neighbor sets $\mathcal{G}(u_{i})$ and $\mathcal{S}(u_{i})$ in Eq.(4) and
Eq.(5) to $\mathcal{G}^{\prime}(u_{i})$ and $\mathcal{S}^{\prime}(u_{i})$, we
are able to speed up the training process and obtain optimal recommenders.

Performance-triggered Sampling. Intuitively, picking neighbors that provide
high-quality knowledge is beneficial, where the fluctuations in the local loss
function $L_{loc}(u_{i})$ is a strong indicator on whether or not the
knowledge distilled from $u_{i}$’s neighbors is sufficiently informative. So,
in every CL epoch $o$, if the change in $\Delta L_{loc}^{o}(u_{i})$ falls
below a predefined threshold $\tau$, the previously sampled neighbors
$\mathcal{G}^{\prime}(u_{i})$/$\mathcal{S}^{\prime}(u_{i})$ will be redrawn
from $\mathcal{G}(u_{i})$/$\alpha$. In our work, we define $\Delta
L_{loc}(u_{i})$ in the following ratio format:

(9) $\Delta
L_{loc}^{o}(u_{i})=\frac{|L^{o}_{loc}(u_{i})-L^{o-1}_{loc}(u_{i})|}{L^{o-1}_{loc}(u_{i})}\times
100\%,$

where we can adjust $\tau$ to actively control the sensitivity to performance
change, and we empirically adopt $\tau=1\%$ for both neighbor types in our
work. Note that we draw samples uniformly at random with replacements to avoid
repetition.

Similarity-based Sampling. Naturally, another feasible solution is to narrow
down the neighbors to communicate in $\mathcal{G}(u_{i})$/$\mathcal{S}(u_{i})$
on the go by similarity measures. To achieve this, we utilize the soft
decisions w.r.t. the reference datasets by comparing them in every CL
iteration via KL divergence (Giantomassi et al., 2014):

(10) $d_{soft}(u_{i},u_{j})=\sum_{\forall
X\in\mathcal{D}^{ref}}KL\left(\phi_{i}\left(X\right)\left|\right|\phi_{j}\left(X\right)\right),$

where $\mathcal{D}^{ref}=\mathcal{D}^{g}(r^{i}_{0})$ if
$u_{j}\in\mathcal{G}^{\prime}(u_{i})$, and $\mathcal{D}^{ref}=\mathcal{D}^{s}$
if $u_{j}\in\mathcal{S}^{\prime}(u_{i})$. Then, for user $u_{i}$, $\beta$
users with the smallest $d_{soft}(u_{i},u_{j})$ will be respectively selected
from $\mathcal{G}(u_{i})$ and $\mathcal{S}(u_{i})$.

In MAC, we employ either similarity-based or performance-triggered sampling as
they substitute each other. Also, as all models are changing over time, the
sampling process dynamically adjusts $\mathcal{G}^{\prime}(u_{i})$ and
$\mathcal{S}^{\prime}(u_{i})$, as shown in line 12 of Algorithm 1.

##  4\. Experiments

In this section, we conduct extensive experiments on two real-world datasets
to evaluate the effectiveness and efficiency of MAC.

Table 1. Dataset statistics. | Foursquare | Weeplace
---|---|---
#users | 7,507 | 4,560
#POIs | 80,962 | 44,194
#categories | 436 | 625
#check-ins | 1,214,631 | 923,600
#check-ins per user | 161.80 | 202.54
Table 2. Recommendation performance comparison with baselines. | Foursquare | Weeplace
---|---|---
| HR@5 | NDCG@5 | HR@10 | NDCG@10 | HR@5 | NDCG@5 | HR@10 | NDCG@10
MF | 0.0835 | 0.0598 | 0.0962 | 0.0669 | 0.1045 | 0.0612 | 0.1293 | 0.0872
LSTM | 0.1920 | 0.1227 | 0.2812 | 0.1648 | 0.2215 | 0.1356 | 0.3301 | 0.1578
STAN | 0.2914 | 0.1756 | 0.4157 | 0.2524 | 0.3205 | 0.1846 | 0.4503 | 0.2783
LLRec | 0.2771 | 0.1488 | 0.3406 | 0.1908 | 0.2912 | 0.1781 | 0.3613 | 0.2255
PREFER | 0.2895 | 0.1696 | 0.3607 | 0.2183 | 0.3055 | 0.1874 | 0.3707 | 0.2359
DCLR | 0.3078 | 0.1797 | 0.4297 | 0.2653 | 0.3304 | 0.1912 | 0.4664 | 0.2838
D-Dist | 0.2257 | 0.1193 | 0.2915 | 0.1621 | 0.2474 | 0.1343 | 0.3581 | 0.1933
SQMD | 0.2818 | 0.1471 | 0.4054 | 0.2357 | 0.2953 | 0.1531 | 0.4342 | 0.2360
MAC-P | 0.3138 | 0.1865 | 0.4319 | 0.2712 | 0.3414 | 0.2032 | 0.4843 | 0.2937
MAC-S | 0.3055 | 0.1843 | 0.4257 | 0.2637 | 0.3365 | 0.1988 | 0.4734 | 0.2845

###  4.1. Datasets and Evaluation Protocols

We adopt two real-world datasets to evaluate our proposed MAC, namely
Foursquare (Chen et al., 2020a) and Weeplace (Liu et al., 2013). Both datasets
include users’ check-in histories in the cities of New York, Los Angeles, and
Chicago. Following (Li et al., 2018; Chang et al., 2018), users and POIs with
less than 10 interactions are removed. Table 1 summarizes the statistics of
the two datasets. Among those datasets, 10% of the check-in sequences are
randomly selected as the reference datasets with the constraint of covering
all POIs. The two strategies to generate reference datasets are also processed
based on those selected check-in activities. Then, inspired by (Wang et al.,
2018, 2019), we employ the leave-one-out protocol for evaluation.
Specifically, for each of the remaining check-in sequences, the last check-in
POI is for testing, the second last POI is for validation, and all others are
for training. In addition, the maximum sequence length is set to 200. For each
ground truth, instead of ranking all e-commerce products (Krichene and Rendle,
2020), we only pair it with 200 unvisited and nearest POIs within the same
region of the sequence as the candidates for ranking. The rationale is,
different from e-commerce products (Krichene and Rendle, 2020), in the
scenario of POI recommendations that are location-sensitive, users seldom
travel between two POIs consecutively that are far away from each other (Long
et al., 2022; Li et al., 2021a). On this basis, we leverage two ranking
metrics, namely Hit Ratio at Rank $k$ (HR@$k$) and Normalized Discounted
Cumulative Gain at Rank $k$ (NDCG@$k$) (Weimer et al., 2007) where HR@$k$ only
measures the times that the ground truth is present on the top-$k$ list, while
NDCG@$k$ cares whether the ground truth can be ranked as highly as possible.

###  4.2. Baselines and Experimental Setting

We compare MAC with both the centralized and on-device POI recommenders:

  * •

MF (Lian et al., 2014): It is a classic centralized POI recommendation system
based on user-item matrix factorization.

  * •

LSTM (Hochreiter and Schmidhuber, 1997): This recurrent neural network can
capture short-term and long-term dependencies in sequential data.

  * •

STAN (Luo et al., 2021): It learns explicit spatiotemporal correlations of
check-in trajectories via a bi-attention approach.

  * •

LLRec (Wang et al., 2020): It utilizes the teacher-student training strategy
to obtain the compressed model that can be deployed locally.

  * •

PREFER (Guo et al., 2021): This federated POI recommendation paradigm allows
the server to collect and aggregate locally trained models, as well as
redistribute the federated model.

  * •

DCLR (Long et al., 2022): This decentralized collaborative learning framework
allows locally trained models to share knowledge between homogeneous neighbors
by model aggregation.

  * •

D-Dist (Bistritz et al., 2020): It aims at letting locally trained models with
random heterogeneous neighbors via comparing their soft decisions on a public
reference dataset.

  * •

SQMD (Ye et al., 2022): It is also a decentralized distillation framework
where neighbors are defined by comparing their responses on the shared
reference dataset.

###  4.3. Experimental Settings

As mentioned above, this work is compatible with almost any centralized POI
recommendation model. To achieve advanced accuracy, we exploit STAN (Luo et
al., 2021) as the base model. Besides, since this work supports heterogeneous
structures, we randomly assign the latent dimension $d\in\\{8,16,32,64,128\\}$
to users and each one makes up 20%. This setting is also applied to other
heterogeneous frameworks (i.e., D-Dist and SQMD). For fairness, all other
baselines are evaluated with the above dimensions separately and the final
results are averaged. Additionally, in this work, each city is divided into 5
regions by applying k-means clustering which is discussed in Section 2. It is
worth noting that in MAC, we implement both the performance-triggered and
similarity-based approaches for neighbor sampling, which are respectively
marked as MAC-P and MAC-S.

For hyperparameters, we set $\alpha$ to $5$, $\beta$ to $10$, $\gamma$ to
$0.5$ and $\mu$ to $0.7$. The impacts of the above four hyperparameters will
be further discussed in Section 4.7. Apart from this, we set $h$ to 50, and
adopt a learning rate of $0.002$, dropout of $0.2$ on all deep layers, batch
size of $16$, and the maximum training epoch is set to $50$.

Table 3. Model size (kb) and accuracy among decentralized POI recommenders. | d = 8 | d = 16 | d = 32 | d = 64 | d = 128
---|---|---|---|---|---
| Size | HR@10 | Size | HR@10 | Size | HR@10 | Size | HR@10 | Size | HR@10
DCLR | 1130 | 0.3674 | 2252 | 0.4413 | 4502 | 0.5417 | 9020 | 0.5165 | 18127 | 0.4651
D-Dist | 1057 | 0.2863 | 1963 | 0.3309 | 3852 | 0.3514 | 7968 | 0.3925 | 15564 | 0.3293
SQMD | 1157 | 0.3637 | 2054 | 0.4194 | 3648 | 0.4845 | 8245 | 0.4585 | 16362 | 0.4449
MAC-P | 232 | 0.4057 | 462 | 0.4678 | 694 | 0.5404 | 1316 | 0.5114 | 3186 | 0.4963
MAC-S | 226 | 0.4026 | 479 | 0.4341 | 742 | 0.5416 | 1276 | 0.5010 | 2987 | 0.4877

Table 4. Traning time (in hours). | Foursquare | Weeplace
---|---|---
DCLR | 76 | 57
D-Dist | 24 | 19
SQMD | 51 | 39
MAC-P | 40 | 32
MAC-S | 29 | 21

###  4.4. Recommendation Effectiveness

The performance comparison among all the POI recommenders is summarized in
Table 2, where we observe the following findings.

Among the centralized POI recommenders, LSTM is superior to MF on both
datasets due to the effective use of short-term and long-term dependencies of
sequential check-in activities. Besides, thanks to spatiotemporal correlations
of consecutive and non-consecutive check-in activities, STAN has higher
accuracy than LSTM. Compared with the advanced baseline STAN, our method still
yields highly competitive results. This is because the centralized STAN model
is trained with check-ins across multiple cities where knowledge learned from
one city might be noisy for the recommendation tasks of other cities, leading
to inferior performance of STAN. Instead, our work can achieve better
personalization, and learn more expressive models with the collaborative
learning architecture.

In the meantime, MAC outperforms all on-device POI recommenders on both
datasets in terms of all metrics. Specifically, LLRec has the worst
performance since all personal data is not included in the training process,
ignoring individual preferences, while our method can trade negligible privacy
risks for large improvement in accuracy. Although PREFER has a noticeable
improvement in accuracy, it still needs to collect and aggregate users’
personalized models, exposing user privacy to potential breaches. Thus, MAC is
more capable of providing both more accurate recommendations and stronger
privacy protection with less reliance on the cloud.

###  4.5. Efficiency Analysis

Amid all homogeneous on-device recommenders, DCLR achieves the best
performance due to its effective collaborative learning strategies. However,
it suffers from the constraint that all participants must hold homogeneous
models for model aggregation. In contrast, MAC can not only support users
having varied dimensions like D-Dist and SQMD, but also allows user devices
effectively utilize limited storage resources by storing partial POI
embeddings. To prove the above view, with respect to all heterogeneous on-
device recommenders, we record the averaged model size (kb) and recommendation
accuracy (HR@10 on Weeplace) for the latent dimensions
$d\in\\{8,16,32,64,128\\}$. The results are shown in Table 4, where we can
observe that the average model size of MAC is far less than that of the other
three on-device recommenders among all dimensions. Additionally, MAC still
outperforms DCLR and other heterogeneous models in terms of recommendation
accuracy at all dimensions, showing that MAC can make more efficient use of
on-device resources to provide more performant recommendations.

Besides, our proposed strategies for neighbor identification and neighbor
sampling can speed up the training process. We record the training time till
model convergence, where the results are shown in Table 4. We can observe that
D-Dist has the least training time since each device in the network will
communicate with a static group of peers at each iteration. In exchange,
D-Dist has the worst recommendation accuracy. Beyond that, MAC converges
faster than all other models while providing more accurate recommendations.
Interestingly, compared to MAC-P, MAC-S achieves higher recommendation
accuracy, but it needs more training time as it tends to change the neighbor
set composition more frequently. Instead, MAC-S only communicates with a
relatively stable neighbor set, thus converging faster but being prone to
local optima.

###  4.6. Analysis on Reference Data Generation

To validate the efficacy of two proposed strategies for generating reference
datasets described in Section 3.3, we evaluate the recommendation accuracy of
both MAC-P and MAC-S on original, transformative generated, and probabilistic
generated reference datasets. The results are shown in table 5. Apparently,
the performance on the original reference datasets is similar to the
transformative generated reference datasets. This proves the efficacy of the
transformative strategy considering the strong privacy protection. Meanwhile,
although the performance on the probabilistic reference datasets lightly
drops, it is still capable for the region without enough historical check-in
activities. Please note all other experiments are based on transformative
generated reference datasets.

Table 5. Effect of reference dataset sources on recommendation accuracy (HR@10 is demonstrated). | Foursquare | Weeplace
---|---|---
| MAC-P | MAC-S | MAC-P | MAC-S
Original | 0.4357 | 0.4232 | 0.4901 | 0.4698
Transformative | 0.4319 | 0.4257 | 0.4843 | 0.4734
Probabilistic | 0.4173 | 0.4129 | 0.4715 | 0.4581

###  4.7. Hyperparameter Sensitivity

Figure 2. Hyperparameter sensitivity. The horizontal lines indicate MAC
variants without any sampling strategy.

In this section, we illustrate the effect of four hyperparameters on the
performance of MAC. Specifically, we evaluate the sample size in performance-
triggered sampling $\alpha$ on the accuracy and training time of MAC-P, the
sample size in similarity-based sampling $\beta$ on the accuracy and training
time of MAC-S, the weight $\gamma$ that controls the knowledge from neighbors
on the accuracy of MAC-P and MAC-S, and the trade-off $\mu$ between
geographical and semantic neighbors for communication on the accuracy of MAC-P
and MAC-S. The results are shown in Figure 2.

Impact of $\alpha$. We experiment on a series of
$\alpha\in\\{1,5,10,20,40\\}$. In general, a larger $\alpha$ harms the
performance of MAC-P on both datasets, as low-quality neighbors are included,
but it requires less training time with lower communication frequencies. In
the extreme case of communicating with all predefined neighbors (dotted line),
we record the lowest accuracy, which proves the effectiveness of performance-
triggered sampling.

Impact of $\beta$. We study the impact of $\beta\in\\{1,5,10,20,40\\}$. As
$\beta$ increases from $1$ to $10$, there is a generally upward trend in
MAC-S’s accuracy with more valuable knowledge. However, it starts to decrease
when $\beta$ exceeds $10$ as low-quality neighbors are included. Meanwhile,
the training time keeps rising since more neighbors are processed. Compared
with the extreme case (dotted line), MAC-S achieves higher accuracy with less
training time, showing the effectiveness of similarity-based sampling.

Impact of $\gamma$. $\gamma$ is evaluated in $\\{0,0.1,0.3,0.5,0.7,0.9,1\\}$.
Apparently, the lowest accuracy is obtained if users fail to share knowledge
with neighbors ($\gamma=0$), showing the significance of the collaborative
learning framework. However, the performance will decline if knowledge from
neighbors has an excessive proportion ($\gamma>0.5$).

Impact of $\mu$. $\mu$ is examined in $\\{0,0.1,0.3,0.5,0.7,0.9,1\\}$. The
best performance is observed when $\mu=0.7$ in most cases, showing more
importance of geographical factors compared to semantic factors.

##  5\. Related Work

This section reviews recent literature on related areas including centralized
models for POI recommendation, on-device frameworks for POI recommendation,
and distributed knowledge distillation.

###  5.1. Next POI Recommendation

To help people discover attractive places by analyzing user-POI interactions,
early models mainly focused on matrix factorization (Lian et al., 2014) and
Markov chains (Cheng et al., 2013; Zheng et al., 2016). Recently, models based
on recurrent neural networks (RNN) have been proven effective in capturing
spatiotemporal dependencies among POI sequences (Yin et al., 2013; Chen et
al., 2020b; Jie et al., 2018; Yin and Cui, 2016; Chen et al., 2020c).
Meanwhile, SGRec (Li et al., 2021a) constructs graph-augmented POI sequences
to fully capture the collaborative signals from semantically correlated POIs
and mine sequential properties, which achieves better accuracy than RNN-based
models. In addition, attentive neural networks (Luo et al., 2021; Yang et al.,
2022; Chen et al., 2019; Yin et al., 2015) utilize self-attention layers to
capture relative spatiotemporal information of all check-in activities along
the sequence. It is worth noting that all the above models are cloud-based,
leading to undeniable problems of privacy issues and high demand for powerful
cloud. Instead, MAC lies in the distributed learning paradigm to provide
secure and stable services.

###  5.2. On-device POI Recommendation

On-device frameworks have been proven effective in addressing most
shortcomings of cloud-based learning for POI recommendations. Intuitively,
Wang et al. (Wang et al., 2020) deployed compressed models on mobile devices
for secure and stable POI recommendations. To maintain the robustness of the
whole on-device framework, local compressed models inherit the knowledge from
the teacher model which is trained with public data. Then, Guo et al. (Guo et
al., 2021) proposed a federated learning framework for POI recommendations,
allowing edge servers to collect and aggregate locally trained models, and
send the aggregated model back to all users. Apparently, the above models are
still highly dependent on the cloud server. On this basis, Jing et al. (Long
et al., 2022) proposed a semi-decentralized learning paradigm with device
collaboration that allows user devices to learn and combine knowledge from two
types of neighbors. However, all aforementioned collaborative learning-based
POI recommenders hold a strong assumption that all on-device models must share
an identical design, allowing user-specific knowledge sharing via
parameters/gradients aggregation. In contrast, MAC supports heterogeneity in
model structures.

###  5.3. Distributed Knowledge Distillation

Knowledge distillation was initially proposed in model compression (Phuong and
Lampert, 2019), aimed at extracting knowledge from a powerful teacher model
trained with mass data, to improve the performance of the light student model.
The information is transferred by minimizing the disagreement between teacher
and student models on an unlabeled reference dataset. However, such a way of
knowledge transmission is not restricted to the teacher-student pattern, where
(Guo et al., 2020) and (Zhang et al., 2021) have proven the effectiveness of
exploiting knowledge distillation to jointly train student models without any
teacher model. This is also applicable in knowledge sharing between
heterogeneous device networks where user devices are allowed to have different
model architectures (Bistritz et al., 2020; Ye et al., 2022). Our work is the
first one to exploit knowledge distillation for knowledge sharing between
heterogeneous POI recommendation models.

##  6\. Conclusion

To support heterogeneous architecture in decentralized POI recommendations
where users are allowed to have a customized structure to meet the specific
device capacity, we incorporate mutual information maximization into knowledge
distillation to exchange information between similarity-based groups decided
by physical distances, category distributions, and social networks. To remove
low-quality neighbors, during the training process, we design two strategies
to further sample neighbors according to their performance or similarity to
the target user, reducing training time and obtaining optimal recommenders.
Apart from this, we propose two novel approaches to build more practical
reference datasets while providing strong privacy protection. Experimental
results have rigorously demonstrated the efficacy of MAC, revealing its
ability to provide excellent services in decentralized POI recommendations.

###### Acknowledgements.

This work is supported by the Australian Research Council under the streams of
Future Fellowship (Grant No. FT210100624), Discovery Project (Grant No.
DP190101985), Discovery Early Career Research Award (Grant No. DE200101465 and
No. DE230101033), and Industrial Transformation Training Centre (No.
IC200100022). It is also supported by NSFC (No. 61972069, 61836007 and
61832017), Shenzhen Municipal Science and Technology R&D Funding Basic
Research Program (ICYJ 20210324133607021), and Municipal Government of Quzhou
under Grant No. 2022D037.

## References

  * (1)
  * Bengio et al. (2013) Y. Bengio, Nicholas Leonard, and A. Courville. 2013\.  Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation.  _Computer Science_ (2013).
  * Bistritz et al. (2020) I. Bistritz, A. Mann, and N. Bambos. 2020.  Distributed Distillation for On-Device Learning. In _Neural Information Processing Systems_.
  * Chang et al. (2018) B. Chang, Y. Park, D. Park, S. Kim, and J. Kang. 2018.  Content-Aware Hierarchical Point-of-Interest Embedding Model for Successive POI Recommendation. In _Twenty-Seventh International Joint Conference on Artificial Intelligence IJCAI-18_.
  * Chen et al. (2019) Tong Chen, Hongzhi Yin, Hongxu Chen, Rui Yan, Quoc Viet Hung Nguyen, and Xue Li. 2019\.  Air: Attentional intention-aware recommender systems. In _2019 IEEE 35th International Conference on Data Engineering (ICDE)_. IEEE, 304–315.
  * Chen et al. (2020b) Tong Chen, Hongzhi Yin, Quoc Viet Hung Nguyen, Wen-Chih Peng, Xue Li, and Xiaofang Zhou. 2020b.  Sequence-aware factorization machines for temporal predictive analytics. In _2020 IEEE 36th International Conference on Data Engineering (ICDE)_. IEEE, 1405–1416.
  * Chen et al. (2020c) Tong Chen, Hongzhi Yin, Guanhua Ye, Zi Huang, Yang Wang, and Meng Wang. 2020c.  Try this instead: Personalized and interpretable substitute recommendation. In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_. 891–900.
  * Chen et al. (2021) T. Chen, H. Yin, Y. Zheng, Z. Huang, Y. Wang, and M. Wang. 2021.  Learning Elastic Embeddings for Customizing On-Device Recommenders.
  * Chen et al. (2015) W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen. 2015.  Compressing Neural Networks with the Hashing Trick.
  * Chen et al. (2020a) Z. Chen, H. Cao, H. Wang, F. Xu, and Y. Li. 2020a.  Will You Come Back / Check-in Again? Understanding Characteristics Leading to Urban Revisitation and Re-check-in.  _Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies_ (2020).
  * Cheng et al. (2013) C. Cheng, H. Yang, M. R. Lyu, and I. King. 2013\.  Where You Like to Go Next: Successive Point-of-Interest Recommendation.. In _International Joint Conference on Artificial Intelligence_.
  * Fawaz et al. (2018) H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P. A. Muller. 2018.  Data augmentation using synthetic data for time series classification with deep residual networks.  (2018).
  * Flores et al. (2021) Anibal Flores, Hugo Tito-Chura, and Honorio Apaza-Alanoca. 2021\.  Data Augmentation for Short-Term Time Series Prediction with Deep Learning.  (2021).
  * Giantomassi et al. (2014) A. Giantomassi, F. Ferracuti, S. Iarlori, G. Ippoliti, and S. Longhi. 2014.  Electric Motor Fault Detection and Diagnosis by Kernel Density Estimation and Kullback-Leibler Divergence based on Stator Current Measurements.  _IEEE Transactions on Industrial Electronics_ (2014).
  * Guennec et al. (2016) A. L. Guennec, S. Malinowski, and R. Tavenard. 2016\.  Data Augmentation for Time Series Classification using Convolutional Neural Networks. In _European Conference on Principles of Data Mining and Knowledge Discovery_.
  * Guo et al. (2020) Q. Guo, X. Wang, Y. Wu, Z. Yu, and P. Luo. 2020.  Online Knowledge Distillation via Collaborative Learning. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Guo et al. (2021) Y. Guo, F. Liu, Z. Cai, H. Zeng, and N. Xiao. 2021.  PREFER: Point-of-interest REcommendation with efficiency and privacy-preservation via Federated Edge leaRning.  _Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies_ 5, 1 (2021), 1–25.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the Knowledge in a Neural Network.  _Computer Science_ 14, 7 (2015), 38–39.
  * Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.
  * Imran et al. (2023) Mubashir Imran, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Alexander Zhou, and Kai Zheng. 2023\.  ReFRS: Resource-efficient federated recommender system for dynamic and diversified user preferences.  _ACM Transactions on Information Systems_ 41, 3 (2023), 1–30.
  * Jie et al. (2018) F. Jie, L. Yong, Z. Chao, F. Sun, and D. Jin. 2018.  DeepMove: Predicting Human Mobility with Attentional Recurrent Networks. In _the 2018 World Wide Web Conference_.
  * Krichene and Rendle (2020) W. Krichene and S. Rendle. 2020.  On Sampled Metrics for Item Recommendation. In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_.
  * Li et al. (2018) Ranzhen Li, Yanyan Shen, and Yanmin Zhu. 2018.  Next Point-of-Interest Recommendation with Temporal and Multi-level Context Attention.  _2018 IEEE International Conference on Data Mining (ICDM)_ (2018), 1110–1115.
  * Li et al. (2021a) Yang Li, Tong Chen, Yadan Luo, Hongzhi Yin, and Zi Huang. 2021a.  Discovering collaborative signals for next POI recommendation with iterative Seq2Graph augmentation. In _IJCAI_. 1491–1497.
  * Li et al. (2021b) Y. Li, T. Chen, P. F. Zhang, and H. Yin. 2021b.  Lightweight Self-Attentive Sequential Recommendation.  (2021).
  * Lian et al. (2020) D. Lian, Y. Wu, Y. Ge, X. Xie, and E. Chen. 2020.  Geography-Aware Sequential Location Recommendation. In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_.
  * Lian et al. (2014) D. Lian, C. Zhao, X. Xie, G. Sun, E. Chen, and Y. Rui. 2014.  _GeoMF: joint geographical modeling and matrix factorization for point-of-interest recommendation_.  ACM.
  * Liu et al. (2013) X. Liu, Y. Liu, K. Aberer, and C. Miao. 2013\.  Personalized point-of-interest recommendation by mining users’ preference transition. In _Acm International Conference on Conference on Information & Knowledge Management_.
  * Long et al. (2022) J. Long, T. Chen, Nq Viet Hung, and H. Yin. 2022\.  Decentralized Collaborative Learning Framework for Next POI Recommendation.  _TOIS_ (2022).
  * Luo et al. (2021) Y. Luo, Q. Liu, and Z. Liu. 2021.  STAN: Spatio-Temporal Attention Network for Next Location Recommendation.
  * Macqueen (1967) J. Macqueen. 1967\.  Some methods for classification and analysis of multivariate observations.  _Proc. Symp. Math. Statist. and Probability, 5th_ 1 (1967).
  * Phuong and Lampert (2019) M. Phuong and C. H. Lampert. 2019.  Towards understanding knowledge distillation. In _International Conference on Machine Learning_.
  * Rao et al. (2021) J. Rao, S. Gao, M. Li, and Q. Huang. 2021\.  A privacy-preserving framework for location recommendation using decentralized collaborative machine learning.  _Transactions in GIS_ (2021).
  * Wang et al. (2020) Q. Wang, H. Yin, T. Chen, Z. Huang, and Nqv Hung. 2020.  Next Point-of-Interest Recommendation on Resource-Constrained Mobile Devices. In _WWW ’20: The Web Conference 2020_.
  * Wang et al. (2021a) Qinyong Wang, Hongzhi Yin, Tong Chen, Junliang Yu, Alexander Zhou, and Xiangliang Zhang. 2021a.  Fast-adapting and Privacy-preserving Federated Recommender System.  (2021).
  * Wang et al. (2018) Q. Wang, H. Yin, Z. Hu, D. Lian, H. Wang, and Z. Huang. 2018.  Neural Memory Streaming Recommender Networks with Adversarial Training.  (2018), 2467–2475.
  * Wang et al. (2019) Q. Wang, H. Yin, H. Wang, Qvh Nguyen, and L. Cui. 2019.  Enhancing Collaborative Filtering with Generative Augmentation. In _the 25th ACM SIGKDD International Conference_.
  * Wang et al. (2021b) Zhaobo Wang, Yanmin Zhu, Qiaomei Zhang, Haobing Liu, Chunyang Wang, and Tong Liu. 2021b.  Graph-enhanced Spatial-temporal Network for Next POI Recommendation.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ (2021).
  * Weimer et al. (2007) M. Weimer, A. Karatzoglou, Q. V. Le, and A. J. Smola. 2007.  CoFiRank - Maximum Margin Matrix Factorization for Collaborative Ranking. In _Neural Information Processing Systems_.
  * Xw et al. (2020) A Xw, A Mn, A Jc, B Lc, and C Kl. 2020.  A group preference-based privacy-preserving POI recommender system.  _ICT Express_ 6, 3 (2020), 204–208.
  * Yang et al. (2022) Song Yang, Jiamou Liu, and Kaiqi Zhao. 2022.  GETNext: Trajectory Flow Map Enhanced Transformer for Next POI Recommendation. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Madrid, Spain) _(SIGIR ’22)_. Association for Computing Machinery, New York, NY, USA, 1144–1153.  https://doi.org/10.1145/3477495.3531983
  * Ye et al. (2022) Guanhua Ye, Hongzhi Yin, and Tong Chen. 2022.  A Decentralized Collaborative Learning Framework Across Heterogeneous Devices for Personalized Predictive Analytics.  (2022).
  * Yin and Cui (2016) Hongzhi Yin and Bin Cui. 2016.  _Spatio-temporal recommendation in social media_.  Springer.
  * Yin et al. (2015) Hongzhi Yin, Bin Cui, Zi Huang, Weiqing Wang, Xian Wu, and Xiaofang Zhou. 2015\.  Joint modeling of users’ interests and mobility patterns for point-of-interest recommendation. In _Proceedings of the 23rd ACM international conference on Multimedia_. 819–822.
  * Yin et al. (2013) Hongzhi Yin, Yizhou Sun, Bin Cui, Zhiting Hu, and Ling Chen. 2013.  Lcars: a location-content-aware recommender system. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_. 221–229.
  * Yin et al. (2017) Hongzhi Yin, Weiqing Wang, Hao Wang, Ling Chen, and Xiaofang Zhou. 2017.  Spatial-aware hierarchical collaborative deep learning for POI recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ 29, 11 (2017), 2537–2551.
  * Yin et al. (2016) Hongzhi Yin, Xiaofang Zhou, Bin Cui, Hao Wang, Kai Zheng, and Quoc Viet Hung Nguyen. 2016\.  Adapting to user interest drift for poi recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ 28, 10 (2016), 2566–2581.
  * Zhang et al. (2021) H. Zhang, Z. Hu, W. Qin, M. Xu, and M. Wang. 2021.  Adversarial co-distillation learning for image recognition.  _Pattern Recognition_ 111 (2021), 107659.
  * Zheng et al. (2016) Bolong Zheng, Kai Zheng, Xiaokui Xiao, Han Su, Hongzhi Yin, Xiaofang Zhou, and Guohui Li. 2016.  Keyword-aware continuous knn query on road networks. In _2016 IEEE 32Nd international conference on data engineering (ICDE)_. IEEE, 871–882.