# Space MIMO: Direct Unmodified Handheld to Multi-Satellite Communication

Yasaman Omid, Zohre Mashayekh Bakhsh, Farbod Kayhan, Yi Ma, Rahim Tafazolli
All authors are with Faculty of Electronics and Physical Sciences, University
of Surrey, UK. e-mail: (y.omid, z.mashayekhbakhsh, f.kayhan, y.ma,
<EMAIL_ADDRESS>
###### Abstract

This paper examines the uplink transmission of a single-antenna handsheld user
to a cluster of satellites, with a focus on utilizing the inter-satellite
links to enable cooperative signal detection. Two cases are studied: one with
full CSI and the other with partial CSI between satellites. The two cases are
compared in terms of capacity, overhead, and bit error rate. Additionally, the
impact of channel estimation error is analyzed in both designs, and robust
detection techniques are proposed to handle channel uncertainty up to a
certain level. The performance of each case is demonstrated, and a comparison
is made with conventional satellite communication schemes where only one
satellite can connect to a user. The results of our study reveal that the
proposed constellation with a total of $3168$ satellites in orbit can enable a
capacity of $800$ Mbits/sec through cooperation of $12$ satellites with and
occupied bandwidth of 500 MHz. In contrast, conventional satellite
communication approaches with the same system parameters yield a significantly
lower capacity of less than $150$ Mbits/sec for the nearest satellite.

###### Index Terms:

Capacity, LEO Satellite, Mega Constellations, Multi-Satellite Cooperative
Communication.

##  I Introduction

Satellite networks, particularly those comprised of low earth orbit (LEO)
satellites, are widely considered as a promising technology for achieving
global connectivity. These networks have the potential to connect remote and
under-served areas, consequently providing critical access to communication
services. For instance, Globalstar and Iridium offer voice and data
connectivity to satellite phones and messenger devices but do not provide
broadband internet services. Alternatively, OneWeb and SpaceX provide
broadband internet access with very small-aperture terminals (VSATs). A high-
throughput connection between satellites and unmodified handheld (UH), such as
cellphones, has the potential to enable seamless global connectivity. Despite
the benefits of UH-satellite communication, the achievable capacity is limited
by the user terminal’s (UT) antenna size and gain and the power constraints at
both ends. One potential solution to overcome the limitations of UH antennas
is to employ large antennas on satellites, e.g. the AST Bluewalker 3 with its
$64\ \text{m}^{2}$ antenna. However, this technology is still in the
experimental phase and has several drawbacks. These include problems related
to space debris, high implementation and launch costs, and probable
interference with astronomical observations. Another possible solution is to
enhance data transfer rates through cooperative satellite operations. By doing
so, the requirement for larger antennas could be relaxed, and the data
throughput within the same constellation could be enhanced. This technique,
which involves collaboration for joint detection/precoding, has only been
examined in a limited number of recent publications, such as [1], and from now
on is referred to as space multi-input multi-output (Space MIMO). To the best
of our knowledge, Space MIMO communication has not yet been thoroughly studied
in the literature, particularly in the uplink scenario. Thus, focusing on the
uplink, in this paper, we study the possibility of a high throughput link for
direct UH to a cluster of satellites that cooperatively detect the transmitted
signal.

The MIMO technology is a promising technique which enables increased data
rates, improved reliability, and better coverage in challenging environments.
The use of MIMO technology in satellite communication was reviewed in [2]
including both fixed and mobile systems. While several techniques have been
proposed to enhance the capacity of MIMO satellite systems, there is currently
no research that explores the use of MIMO for direct communication between a
mobile user and multiple satellites.

The majority of the literature on using MIMO in satellite communication
focuses on enhancing capacity through other means, rather than considering
satellite cooperation. In [3], a distributed satellite MIMO setup was proposed
to improve the spectral efficiency without increasing the transmit power by
using a uniform linear array (ULA) at the ground side. The satellites were
positioned far enough apart to allow for a distributed MIMO setup. The study
focused only on the downlink scenario for geostationary earth orbit (GEO)
satellites. The authors in [4] investigated a comparable system scenario by
analyzing the return link (from UT to anchor station), where each satellite
payload is a non regenerative amplify and forward relay. A Rician channel
model was examined for MIMO uplink, whereas in downlink two SISO channel was
employed, considering only LOS propagation channel model. The authors of [5]
also investigated a similar system considering the effect of the positions of
the ground antenna and the satellite antenna, as well as the perturbations of
GEO satellites, in both uplink and downlink scenarios using a flat fading
model for MIMO channels. In [6], the UT movement was examined in a downlink
scenario, where each channel coefficient was modelled by a four-state Markov
transition process with LOO distribution. This study compared the performance
of different transmission schemes, such as dual satellite dual polarization
(DSDP), dual satellite single polarization (DSSP), single satellite single
polarization (SSSP), and single input single output (SISO). It was revealed
that DSDP provides the best bit error rate (BER) performance among all tested
schemes. In [7], a mobile MIMO ultra-high frequency (UHF) channel was examined
by considering two GEO satellites as transmitters and a uniform circular array
(UCA) as a receiver. The MIMO bandwidth efficiency was analysed by rotating
the antenna array. The measurement results revealed that a four-element UCA
maintains high bandwidth efficiency, regardless of the receiver’s spatial
orientation, making it the preferable option. In [8], a cooperative
beamforming transmission technique between UTs and GEO satellites was
proposed. The study derived a ground user selection criterion for user
cooperation, and their method resulted in significant improvements in
achievable rates for both cooperative and non-cooperative systems. Compared to
a single high-power satellite, the cooperative system achieved twice the
maximum rate, and the non-cooperative system attained a rate 1.7 times higher.

Some recent studies have explored the advantages of satellite collaboration
through inter-satellite links. One such study is [9], which examined a
downlink scenario where single antenna LEO satellites communicate with a land
terminal with a UCA antenna. Through synchronization and transmission rate
adjustments, the satellites cooperate to improve the data rate. The study
demonstrated that the achievable data rate through joint transmission is
considerably higher than the achievable rate using a single satellite, even
when the latter used twice the transmit power. In [1], a novel architecture
for LEO satellite networks utilizing distributed massive MIMO (DM-MIMO)
technology was proposed to improve service time for users. The satellite-UT
channels were modelled by the Rician distribution, and instantaneous CSI was
estimated using pilot-based techniques in each satellite. A cooperative
precoding technique was proposed where all satellites in a cluster jointly
transmit signals to the UTs to minimize interference. The authors aimed to
maximize the UT’s aggregate downlink data rate by optimizing power allocation
and maximizing the UT’s service time. Their proposed approach outperformed
single satellite approaches in terms of average service time and spectral
efficiency.

Motivated by the above, in this paper we design a space MIMO system for the
direct UH to satellite transmission, where the satellites cooperate with each
other to detect the transmitted signal. Taking the mobility of the UH into
account and following 3GPP standard [10] we consider the land mobile satellite
(LMS) channel model in this model. We investigate the problem of mean square
error (MSE) minimization, and we consider satellite cooperation in two cases
of full CSI and partial CSI, which are compared with each other in terms of
capacity, overhead and BER. Our numerical results compare the capacity of
these two cases with a conventional satellite communication scheme where only
one satellite is connected to the UH. Our study considers a constellation of
$3168$ satellites and demonstrates that by incorporating satellite
cooperation, we can achieve a remarkable capacity of over $800$ Mbit/sec and a
BER of under $0.1$ for uncoded BPSK when only $12$ satellites cooperate on
signal processing. Notice that conventional communication which lacks
satellite collaboration can only achieve a capacity of under $150$ Mbit/sec
and a BER of over $0.3$ for the nearest satellite.

The rest of this paper is organized as follows. The system model and channel
model is presented in Section II. Section III is dedicated to the joint
processing techniques in both cases of full and partial CSI. Section IV
presents the simulation results and section V concludes the paper.

##  II System Model

The objective of this study is to examine the uplink transmission of a single-
antenna UH user to multiple LEO satellites. The satellites under consideration
are at different orbits and altitudes, causing the distance between users and
each satellite to vary over time. At any given time, we denote by $L_{max}$
the number of satellites that are within a $30^{\degree}$ elevation angle of
the users. Notice that $L_{max}$ depends on the constellation and is a
function of time. All satellites have both line-of-sight (LOS) and non line-
of-sight (NLOS) links with all users. To enhance system performance and
increase capacity, a cluster of satellites is formed by utilizing the inter-
satellite links, which jointly detect the received signal from the user. We
order the satellites depending on their distances from the user, i.e. the
nearest satellite to the user is considered as satellite 1. In this paper we
select $L$ satellites from the $L_{max}$ visible satellites that have the
least distances from the user to form the cluster at each time $t$. Each user
has one omni-directional antenna and transmits the signal towards all
satellites simultaneously. The inter-satellite links are assumed to be
perfectly known by all satellites, and their effect on the shared information
can be eliminate. The receiver noise at the inter-satellite link is considered
negligible. The following subsection presents the channel model of user-
satellites links.

Figure 1: LEO satellite communication system model

###  II-A Channel Model

Our flat fading channel employs the two-state model introduced in [10]. This
model encompasses both the shadow fading and clutter loss effects. The
calculation of the Free Space Path Loss (FSPL) is then modified by including
these effects, and can be expressed as ${\text{FSPL}}_{m}=\frac{4\pi
d_{m}}{\lambda}$, where $d_{m}$ represents the distance between the user and
satellite $m$, $\lambda$ is the wavelength of the signal which is given by
$\lambda=c/f_{c}$, and $f_{c}$ denotes the carrier frequency. Doppler shift
effect can be compensated at the transmitter [11]. The two-state model, as
defined, categorizes the channel states as either good, with LOS or slightly
shadowed conditions, or bad, with severe shadowed conditions. To describe the
duration of each state, a semi-Markov model is used, while the LOO
distribution characterizes the fading of the channel within each state. This
distribution combines both the LOS and non-LOS (NLOS) received signals. The
channel coefficient between the user and the $m$th satellite is obtained using
the following equation:

$\displaystyle
h_{m}=\frac{1}{{\text{FSPL}}_{m}}(\sqrt{\frac{K}{K+1}}h_{{\text{LOS}}}+\sqrt{\frac{1}{K+1}}h_{\text{NLOS}})$
(1)

where $K$ is the ratio of the LOS signal power to the mean power of the NLOS
signal. The LOS and NLOS channels are denoted as $h_{\text{LOS}}$ and
$h_{\text{NLOS}}$ respectively, where
$h_{\text{LOS}}=|h_{\text{LOS}}|e^{j\phi_{\text{LOS}}}$ and
$h_{\text{NLOS}}=|h_{\text{NLOS}}|e^{j\phi_{\text{NLOS}}}$. The magnitude of
$h_{\text{LOS}}$ is log-normally distributed with a mean $M_{A}$ and standard
deviation $\Sigma_{A}$, and the magnitude of $h_{\text{NLOS}}$ is Rayleigh
distributed with an average power $MP$. The phase angles $\phi_{\text{LOS}}$
and $\phi_{\text{NLOS}}$ are uniformly distributed between 0 and $2\pi$.

##  III Joint Processing

In this section, we aim to present a joint processing technique among the
satellites to minimize the MSE. Here, we assume that each satellite can
estimate its own channel, and the CSI information and the received signals are
shared among all satellites in the cluster. In the absence of inter-satellite
link noise, the aggregate capacity of the cluster where each satellite shares
its data with all other satellites would be the same as the capacity that
would result if all satellites transmitted their data to a single satellite.
This is because all satellites would have access to the identical information,
and any one of them can perform the detection. Thus, we consider that all
satellites share their data with only one satellite which performs the signal
processing. We investigate two cases of full CSI and partial CSI between the
satellites. The following subsections elaborate on each of these scenarios.

###  III-A Case 1: full CSI

In this scenario, we consider that all satellites share their data and
instantaneous CSI with only one satellite which performs the signal
processing. The overhead of such a system for data sharing is in the order of
$\mathcal{O}(L)$ per symbol and for the CSI sharing is in the order of
$\mathcal{O}(L)$ per coherence interval.In more detail, each satellite $m$,
$m\in\\{1,...,L\\}$, shares its estimated channel $\hat{h}_{m}$, and its
received signal $y_{m}=\sqrt{p}h_{m}s+n_{m}$ with a satellite in the cluster
which performs the detection. Note that $p=P_{T}G_{T}G_{R}$, where $P_{T}$ is
the transmit power, $G_{T}$ is the gain of the transmit antenna and $G_{R}$ is
the gain of the receive antenna at the satellite. The additive noise at the
receiver is modeled by $n_{m}\sim\mathcal{CN}(0,\sigma^{2})$, where the noise
power is calculated by $\sigma^{2}=k_{B}\times T\times BW$. Here,
$k_{B}=1.380649\times 10^{-23}$ is the Boltzmann constant, $T=290$ is the
noise temperature, and $BW$ is the Bandwidth in Hz. We assume that the channel
estimation is imperfect, i.e. $\hat{h}_{m}=h_{m}+\tilde{h}_{m}$ with
$\hat{h}_{m}$ being the estimation and $\tilde{h}_{m}$ being the estimation
error, where

$\displaystyle\tilde{h}_{m}\sim\mathcal{CN}(0,\sigma_{\tilde{h}_{m}}^{2}).$
(2)

Now, assuming the joint detection vector is represented by
$\mathbf{v}\in\mathbb{C}^{1\times L}$, the detected signal can be represented
by

$\hat{s}=\mathbf{v}\mathbf{y}=\sqrt{p}s\mathbf{v}\mathbf{h}+\mathbf{v}\mathbf{n}=\sqrt{p}s\mathbf{v}(\mathbf{\hat{h}}-\mathbf{\tilde{h}})+\mathbf{v}\mathbf{n}$
(3)

where $\mathbf{h}=[h_{1},...,h_{L}]^{T}$,
$\mathbf{\hat{h}}=[\hat{h}_{1},...,\hat{h}_{L}]^{T}$,
$\mathbf{\tilde{h}}=[\tilde{h}_{1},...,\tilde{h}_{L}]^{T}$ and
$\mathbf{n}=[n_{1},...,n_{L}]^{T}$.

The objective is to find the detection vector that minimizes the MSE. This can
be achieved by solving the following optimization problem

$\begin{array}[]{cl}\min\limits_{\mathbf{v}}&\mathbb{E}\\{|\hat{s}-s|^{2}\\},\end{array}$
(4)

where the expectation is taken over the additive noise, channel estimation
error, and the transmitted symbol. The first-order optimality condition can be
used to calculate the optimal solution for this optimization problem.
Specifically, the gradient of the objective function with respect to the
detection vector should be zero at the optimal solution. Therefore, we can set
the gradient of the objective function equal to zero and solve the problem for
the detection vector. The optimal detection vector can be given by

$\mathbf{v}=\sqrt{p}\hat{\mathbf{h}}^{H}\left(p\hat{\mathbf{h}}\hat{\mathbf{h}}^{H}+\sigma^{2}\mathbf{I}+p\Sigma_{h}\mathbf{I}\right)^{-1},$
(5)

where
$\Sigma_{h}=diag[\sigma_{\tilde{h}_{1}}^{2},...,\sigma_{\tilde{h}_{L}}^{2}]$.
For further details on (5), please refer to appendix A.

The capacity in this case is given by the mutual information between $s$ and
$\hat{s}$, as

$I(s;\hat{s})=H(\hat{s})-H(\hat{s}|s),$ (6)

where we have $H(\hat{s})\leq\log(\pi
e(p\mathbf{v}\mathbf{h}\mathbf{h}^{H}\mathbf{v}^{H}+\sigma^{2}\mathbf{v}\mathbf{v}^{H}))$,
and $H(\hat{s}|s)=\log(\pi e\sigma^{2}\mathbf{v}\mathbf{v}^{H})$. Assuming $s$
has a Gaussian distribution, then $\hat{s}$ would have a Gaussian distribution
as well and the capacity for the full CSI scenario is given by

$R_{\text{FC}}=\log\left(1+\frac{p\mathbf{v}\mathbf{h}\mathbf{h}^{H}\mathbf{v}^{H}}{\sigma^{2}\mathbf{v}\mathbf{v}^{H}}\right).$
(7)

The proof is given in Appendix A.

###  III-B Case 2: partial CSI

In this scenario, given that satellite $1$ has the smallest path loss and thus
has the strongest channel, we take this satellite as our network controller
(NC), and assume that all satellites share their data only with the NC. Note
that unlike case 1, the instantaneous estimated CSI of each satellite is not
shared with the NC. In this case, a given satellite $m$, $m\in\\{1,...,L\\}$
estimates its own channel imperfectly, and it can perform the following local
processing to cancel the effects of channel attenuation

$y^{{}^{\prime}}_{m}=\frac{y_{m}}{\hat{h}_{m}}=\frac{\sqrt{p}h_{m}s+n_{m}}{\hat{h}_{m}}=\sqrt{p}s-\sqrt{p}s\frac{\tilde{h}_{m}}{\hat{h}_{m}}+\frac{n_{m}}{\hat{h}_{m}}.$
(8)

Then, the satellites $m\in\\{2,...,L\\}$ send their processed signal
$y^{{}^{\prime}}_{m}$ to the NC, along with the following long-term
information, i) the average power of the estimated channel
$\mathbb{E}\\{|\hat{h}_{m}|^{2}\\}$, ii) the average power of estimation error
$\mathbb{E}\\{|\tilde{h}_{m}|^{2}\\}$, and iii) the average power of inverse
estimated channel $\mathbb{E}\\{|\frac{1}{\hat{h}_{m}}|^{2}\\}$. Since these
are long-term information, sharing them with the NC does not add to the system
overhead. Thus, in this case, the system overhead would be in the order of
$\mathcal{O}(L)$. After the local processing is performed in each satellite,
the processed data is sent to the NC for central processing. The central
processing vector is shown by $\mathbf{w}\in\mathbb{C}^{1\times L}$, and the
final detected signal at the NC is presented by
$\hat{s}=\mathbf{w}\mathbf{y}^{{}^{\prime}}$, where
$\mathbf{y}^{{}^{\prime}}=[y^{{}^{\prime}}_{1},...,y^{{}^{\prime}}_{L}]^{T}$.
Similar to case 1, the following optimization problem is solved

$\begin{array}[]{cl}\min\limits_{\mathbf{w}}&\mathbb{E}\\{|\hat{s}-s|^{2}\\},\end{array}$
(9)

where the expectation is taken over the additive noise, channel estimation
error, the transmitted symbol, and the estimated channels for the satellites
$2$ to $L$. Given the distribution of the channel estimation error in (2), the
objective function in (9) can be rewritten as

$\displaystyle\mathbb{E}\\{|\hat{s}-s|^{2}\\}=$ $\displaystyle
p\mathbf{w}\mathbf{1}\mathbf{1}^{T}\mathbf{w}^{H}-\sqrt{p}\mathbf{w}\mathbf{1}-\sqrt{p}\mathbf{1}^{T}\mathbf{w}^{H}$
$\displaystyle+\mathbf{w}\mathbf{B}\mathbf{w}^{H}+p\mathbf{w}\mathbf{S}\mathbf{w}^{H}+1,$
(10)

where $\mathbf{1}$ represents an all-one $L\times 1$ vector, and the matrices
$\mathbf{B}$ and $\mathbf{S}$ are respectively

$\mathbf{B}=\left[\begin{array}[]{cccc}\frac{\sigma^{2}}{|\hat{h}_{1}|^{2}}&0&...&0\\\
0&\sigma^{2}\mathbb{E}|\frac{1}{\hat{h}_{2}}|^{2}&...&0\\\
\vdots&\vdots&\ddots&\vdots\\\
0&0&...&\sigma^{2}\mathbb{E}|\frac{1}{\hat{h}_{L}}|^{2}\end{array}\right],$
(11)

and

$\displaystyle\mathbf{S}=\operatorname{diag}\Bigg{[}$
$\displaystyle\mathbb{E}\left|{\tilde{h}_{1}}\right|^{2}\frac{1}{|\hat{h}_{1}|^{2}},\mathbb{E}\left|{\tilde{h}_{2}}\right|^{2}\mathbb{E}\left|\frac{1}{\hat{h}_{2}}\right|^{2},$
$\displaystyle...,\mathbb{E}\left|{\tilde{h}_{L}}\right|^{2}\mathbb{E}\left|\frac{1}{\hat{h}_{L}}\right|^{2}\Bigg{]},$
(12)

Using the first-order optimality condition, the optimal solution for the
optimization problem in (9) is given by

$\mathbf{w}=\sqrt{p}\mathbf{1}^{T}\left(p\mathbf{1}\mathbf{1}^{T}+p\mathbf{S}+\mathbf{B}\right)^{-1}.$
(13)

The capacity in this scenario is given by the mutual information between the
transmitted signal $s$ and the final centrally-processed signal $\hat{s}$, as
in (6), where $H(\hat{s})\leq\log(\pi
e(p\mathbf{w}\mathbf{1}\mathbf{1}^{T}\mathbf{w}^{H}+\mathbf{w}\mathbf{B}\mathbf{w}^{H}))$
and $H(\hat{s}|s)=\log(\pi e\mathbf{w}\mathbf{B}\mathbf{w}^{H})$. Thus, the
achievable rate for the partial CSI case can be written as

$R_{\text{PC}}=\log\left(1+\frac{p\mathbf{w}\mathbf{1}\mathbf{1}^{T}\mathbf{w}^{H}}{\mathbf{w}\mathbf{B}\mathbf{w}^{H}}\right).$
(14)

Note that this capacity is maximized when perfect CSI is available, i.e.,
$\mathbf{\hat{h}}=\mathbf{h}$. For estimated values of CSI, the given rate is
reduced.

##  IV Numerical Results

In this section, we present the results for both cases in section III. For the
simulations we adopt the Starlink constellation with two different layers
corresponding to two groups in Starlink launching plans [12]. However, we only
use the coordinates of the constellation and not the specifics of the Starlink
satellites. Each group has $22$ orbital planes spaced $16.4^{\degree}$ apart
with $72$ satellites per plane. The first group has $1584$ satellites orbiting
at an altitude of $550$ km with a $53$-degree inclination and the second group
has $1584$ satellites orbiting at an altitude of $540$ km with an inclination
of $53.2^{\degree}$. The true anomaly phasing is $1.1364^{\degree}$, the RAAN
spread is $360^{\degree}$, and the minimum elevation angle for satellite
visibility is $30^{\degree}$. Figure 2 depicts the constellation where there
are a total of $3168$ satellites in orbit.

Figure 2: Constellation

The uplink communication is performed using the C-band, with a carrier
frequency of $f_{c}=6$ GHz and a bandwidth of $BW=500$ MHz. The transmitter
device is assumed to be a mobile UH with a transmit power of $P_{T}=-2$ dBW
and a transmit antenna gain of $G_{T}=5$ dB. The minimum receiver antenna gain
is assumed to be $G_{R}=35$ dB. The simulation is conducted using the
approximate user location of London with a latitude of $51.4880572$ degrees
and a longitude of $-0.0762838$ degrees. In this constellation, at least $28$
satellites are within the visibility region of the user in the given location.
The channel estimation error distribution in (2) is assumed to be
$\sigma_{\tilde{h}_{m}}^{2}=\epsilon^{2}\text{var}({h}_{m})$, with a parameter
$\epsilon=3$. Our simulations evaluate the system’s performance using uncoded
BPSK modulation.

Figure 3 presents the capacity of the system over a duration of $6000$ sec,
for both cases. The results indicate that employing both groups of satellites
in the constellation, with $L=28$ satellites per cluster, leads to a capacity
gain exceeding $1000$ Mbit/sec. Furthermore, by choosing only $L=14$ nearest
satellites, a capacity of over $800$ Mbit/sec can be attained. When only one
group of satellites in the constellation is used, with $L=L_{max}=14$, the
capacity is approximately $700$ Mbit/sec. This capacity reduction can be
attributed to the constellation’s density. Also, this figure demonstrates that
having more satellites cooperating improves system performance at the expense
of system overhead. As expected, Case 2 with partial CSI shared in a cluster
exhibits slightly lower capacity than Case 1 with full CSI. The capacity of a
single satellite without handover during this time period is also depicted
here, which peaks for approximately one minute while the satellite is within
the visibility region for nearly five minutes. This suggests that, handovers
would be needed every minute in non-cooperative systems to achieve the
capacity of the nearest satellite. Thus, satellite cooperation can
significantly enhance system performance compared to conventional satellite
communication schemes.

Figure 4, depicts the average capacity of the system as a function of $L$, in
both cases, for different channel uncertainty distributions, $\epsilon=3\
\text{and}\ 5$. As expected, increasing the number of satellites engaged in
cooperative detection enhances the system capacity. However, the capacity does
not increase linearly due to the selection of satellites in a cluster based on
their proximity to the user. Incorporating distant satellites may not yield
significant performance improvements and could lead to high overhead. For
instance, assuming perfect CSI, Case 1 achieves a capacity of $800$ Mbit/sec
with $L=12$ satellites. However, doubling the number of satellites to $L=24$
results in a capacity increase to $1.04$ Gbit/sec, representing a mere $25\%$
improvement despite the doubled satellite count. Hence, selecting an optimal
number of satellites in a cluster is crucial for maximizing system capacity
while minimizing overhead. The optimization of constellation parameters is
essential to strike the right balance between capacity and cost, making it an
intriguing area for future research.

Figure 5, demonstrates the BER of the system as a function of $L$, in both
cases, for different receiver gains, $G_{R}=35\ \text{and}\ 20\ \text{dB}$.
Note that the BER results are achieved without using any coding; thus, the
actual BER when coding gain is applied would be much lower. As expected,
higher values of $G_{R}$ can compensate for the path loss and enhance system
performance. It is noteworthy to mention that the antenna gains affect the
slope of the BER reduction.

In Figure 6, the average capacity of the system is presented in different
carrier frequencies and bandwidths. Note that the chosen values for frequency
and bandwidth are selected from the 3GPP standard for 5G and the predefined
bands for satellite uplink.111The frequency band $7.9-8.4$ GHz in X-band is a
NATO band type 1 for satellite uplink, the frequency band $14-14.5$ GHz in ku-
band is designated for fixed satellite service uplink in North America, and
the frequency band $27.5-31$ GHz in ka-band is used for uplink satellite
communications.  The second case is analyzed in this figure in terms of
average capacity under a scenario of imperfect CSI, with a value of
$\epsilon=3$ and $G_{R}=35$ dB. By using both groups of satellites in the
constellation, we compare the system when $L=28$, $L=14$ and $L=1$. It is a
well-known fact that increasing the bandwidth or decreasing the frequency
leads to an increase in capacity. However, it is difficult to predict the
behavior of capacity when both parameters are changed simultaneously and thus
requires evaluation. This figure demonstrates that even in ka-band, utilizing
the cooperative satellite technique enables achieving a capacity of
approximately $200$ Mbit/sec. In contrast, conventional satellite
communication yields a lower capacity of less than $20$ Mbit/sec. In lower
frequency bands on the other hand, the difference between the capacity of
single-satellite scenario where $L=1$ and the cooperative satellite scenarios
where $L=14$ and $28$ is much smaller. This is due to the fact that in higher
frequencies where path-loss is high, in the formulas for capacity presented in
both (7) and (14), the second term in logarithm becomes much smaller than 1
and thus the capacities can be approximated by
$R_{\text{FC}}\approx\frac{p\mathbf{v}\mathbf{h}\mathbf{h}^{H}\mathbf{v}^{H}}{\sigma^{2}\mathbf{v}\mathbf{v}^{H}}$
and
$R_{\text{PC}}\approx\frac{p\mathbf{w}\mathbf{1}\mathbf{1}^{T}\mathbf{w}^{H}}{\mathbf{w}\mathbf{B}\mathbf{w}^{H}}$,
respectively. This means that in higher frequencies the capacity increases
with $L$ linearly, while in lower frequencies, the increase in capacity is
logarithmic. Additionally, it is evident that at lower frequencies, the
highest capacity is obtained when the bandwidth is wide. It is important to
note that these findings may vary depending on the value of the parameter
$G_{R}$.

Figure 3: Average capacity in a 100-minute duration of time.  Figure 4:
Average capacity vs. number of satellites in a cluster. Figure 5: BER vs.
number of satellites in a cluster.  Figure 6: Average Capacity in different
$f_{c}$ and $BW$.

##  V conclusion

This paper explores the uplink transmission of a single-antenna mobile user to
a cluster of satellites that collaborate to detect the received signal. Two
cases were considered: one with full CSI and the other with partial CSI shared
among the satellites. The latter imposed less overhead on the system, while
the performance loss was found to be negligible compared to full CSI sharing.
Numerical results demonstrated that our cooperative designs outperformed
conventional satellite communication schemes, where only one satellite was
able to connect to a user. The study also highlighted that the performance of
the system is affected not only by the number of satellites in a cluster but
also by constellation parameters such as satellite density. Furthermore, the
results underscored the significance of frequency and bandwidth evaluation in
system design and emphasized the importance of satellite cooperation in higher
frequency bands.

## Acknowledgment

This work is produced in the framework of the Surrey-Huawei joint-lab project.
We would also like to acknowledge the support of the University of Surrey 5GIC
& 6GIC.

## Appendix A

Here we provide a proof for (5). Given $\mathbb{E}|s|^{2}=1$, we have

$\displaystyle\text{MSE}=$
$\displaystyle\mathbb{E}\\{|\hat{s}-s|^{2}\\}=\mathbb{E}\\{(\mathbf{v}\mathbf{y}-s)(\mathbf{y}^{H}\mathbf{v}^{H}-s^{*})\\}$
$\displaystyle=p\mathbf{v}\mathbf{\hat{h}}\mathbf{\hat{h}}^{H}\mathbf{v}^{H}-2\operatorname{Re}\\{\sqrt{p}\mathbf{\hat{h}}^{H}\mathbf{v}^{H}\\}$
$\displaystyle\quad+p\mathbf{v}\mathbb{E}\\{\mathbf{\tilde{h}}\mathbf{\tilde{h}}^{H}\\}\mathbf{v}^{H}+\mathbf{v}\mathbb{E}\\{\mathbf{n}\mathbf{n}^{H}\\}\mathbf{v}^{H}+1.$
(15)

Now we calculate the gradient, w.r.t $\mathbf{v}^{H}$ as

$\displaystyle\frac{\partial\text{MSE}}{\partial\mathbf{v}^{H}}=p\mathbf{v}\mathbf{\hat{h}}\mathbf{\hat{h}}^{H}-\sqrt{p}\mathbf{\hat{h}}^{H}+p\mathbf{v}\mathbb{E}\\{\mathbf{\tilde{h}}\mathbf{\tilde{h}}^{H}\\}+\mathbf{v}\mathbb{E}\\{\mathbf{n}\mathbf{n}^{H}\\}.$
(16)

We have $\Sigma_{h}=\mathbb{E}\\{\mathbf{\tilde{h}}\mathbf{\tilde{h}}^{H}\\}$
and $\mathbb{E}\\{\mathbf{n}\mathbf{n}^{H}\\}=\sigma^{2}\mathbf{I}$. Thus by
setting the gradient equal to zero, the proof is completed.

Now, similarly, a proof for (7) is provided. We have

$\displaystyle H(\hat{s})\leq\log(\pi eR_{\hat{s}}),$ (17)

where $R_{\hat{s}}=\mathbb{E}\\{\hat{s}\hat{s}^{*}\\}$. Similar to proof of
(5), we have

$\displaystyle R_{\hat{s}}$
$\displaystyle=\mathbb{E}\\{\mathbf{v}\mathbf{y}\mathbf{y}^{H}\mathbf{v}^{H}\\}=p\mathbf{v}\mathbf{\hat{h}}\mathbf{\hat{h}}^{H}\mathbf{v}^{H}+\mathbf{v}\mathbb{E}\\{\mathbf{n}\mathbf{n}^{H}\\}\mathbf{v}^{H}$
$\displaystyle=p\mathbf{v}\mathbf{\hat{h}}\mathbf{\hat{h}}^{H}\mathbf{v}^{H}+\sigma^{2}\mathbf{v}\mathbf{v}^{H}.$
(18)

The expression $H(\hat{s}|s)=\log(\pi e\sigma^{2}\mathbf{v}\mathbf{v}^{H})$
can be proved in the same manner. Thus, the proof is completed.

## References

  * [1] M. Y. Abdelsadek, G. K. Kurt, and H. Yanikomeroglu, “Distributed massive mimo for leo satellite networks,” _IEEE Open Journal of the Communications Society_ , vol. 3, pp. 2162–2177, 2022.
  * [2] P.-D. Arapoglou, K. Liolis, M. Bertinelli, A. Panagopoulos, P. Cottis, and R. De Gaudenzi, “Mimo over satellite: A review,” _IEEE communications surveys & tutorials_, vol. 13, no. 1, pp. 27–51, 2010.
  * [3] C. Li, H. Zhu, G. Li, Y. Zhang, and G. Li, “Analysis of ula spacing threshold and capacity reduction in distributed satellite mimo scenarios,” _IEEE Communications Letters_ , vol. 25, no. 11, pp. 3669–3673, 2021.
  * [4] B. Ramamurthy, W. G. Cowley, and G. Bolding, “Mimo applicability to uhf satcom,” in _2016 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2016, pp. 1–7.
  * [5] R. T. Schwarz and A. Knopp, “Mimo capacity of co-located satellites in longitude separation,” in _ICC 2019-2019 IEEE International Conference on Communications (ICC)_. IEEE, 2019, pp. 1–7.
  * [6] H. Peng, C. Qi, Y. Chen, Q. Zhang, D. Chen, and R. Ding, “Channel modeling and signal transmission for land mobile satellite mimo,” in _2021 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2021, pp. 1–6.
  * [7] C. A. Hofmann, R. T. Schwarz, and A. Knopp, “Multisatellite uhf mimo channel measurements,” _IEEE Antennas and Wireless Propagation Letters_ , vol. 16, pp. 2481–2484, 2017.
  * [8] S. Dou, L. Bai, J. Xie, and Z. Xiao, “Cooperative transmission for geostationary orbiting satellite collocation system,” in _2014 IEEE Global Communications Conference_. IEEE, 2014, pp. 2880–2885.
  * [9] R. Richter, I. Bergel, Y. Noam, and E. Zehavi, “Downlink cooperative mimo in leo satellites,” _IEEE Access_ , vol. 8, pp. 213 866–213 881, 2020.
  * [10] “3rd generation partnership project; technical specification group radio access network; study on new radio (NR) to support non terrestrial networks (release 15),” _Tech. Rep. 3GPP TR 38.811 V15.0.0_ , Jun. 2018.
  * [11] E. G. Peters and C. R. Benson, “A doppler correcting software defined radio receiver design for satellite communications,” _IEEE Aerospace and Electronic Systems Magazine_ , vol. 35, no. 2, pp. 38–48, 2020.
  * [12] J. Liang, A. U. Chaudhry, and H. Yanikomeroglu, “Phasing parameter analysis for satellite collision avoidance in starlink and kuiper constellations,” in _2021 IEEE 4th 5G World Forum (5GWF)_. IEEE, 2021, pp. 493–498.


# M2GNN: Metapath and Multi-interest Aggregated Graph Neural Network for Tag-
based Cross-domain Recommendation

Zepeng Huai  School of Artificial Intelligence, University of Chinese Academy
of SciencesBeijingChina<EMAIL_ADDRESS>,  Yuji Yang
MeituanBeijingChina<EMAIL_ADDRESS>,  Mengdi Zhang  MeituanBeijingChina
<EMAIL_ADDRESS>,  Zhongyi Zhang  MeituanBeijingChina
<EMAIL_ADDRESS>,  Yichun Li  MeituanBeijingChina
<EMAIL_ADDRESS>and  Wei Wu  MeituanBeijingChina<EMAIL_ADDRESS>
(2023)

###### Abstract.

Cross-domain recommendation (CDR) is an effective way to alleviate the data
sparsity problem. Content-based CDR is one of the most promising branches
since most kinds of products can be described by a piece of text, especially
when cold-start users or items have few interactions. However, two vital
issues are still under-explored: (1) From the content modeling perspective,
sufficient long-text descriptions are usually scarce in a real recommender
system, more often the light-weight textual features, such as a few keywords
or tags, are more accessible, which is improperly modeled by existing methods.
(2) From the CDR perspective, not all inter-domain interests are helpful to
infer intra-domain interests. Caused by domain-specific features, there are
part of signals benefiting for recommendation in the source domain but harmful
for that in the target domain. Therefore, how to distill useful interests is
crucial. To tackle the above two problems, we propose a metapath and multi-
interest aggregated graph neural network (M2GNN). Specifically, to model the
tag-based contents, we construct a heterogeneous information network to hold
the semantic relatedness between users, items, and tags in all domains. The
metapath schema is predefined according to domain-specific knowledge, with one
metapath for one domain. User representations are learned by GNN with a
hierarchical aggregation framework, where the intra-metapath aggregation
firstly filters out trivial tags and the inter-metapath aggregation further
filters out useless interests. Offline experiments and online A/B tests
demonstrate that M2GNN achieves significant improvements over the state-of-
the-art methods and current industrial recommender system in Dianping,
respectively. Further analysis shows that M2GNN offers an interpretable
recommendation.

Cross-domain Recommendation, Graph Neural Network, Tag-based Recommendation

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
46th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 23–27, 2023; Taipei, Taiwan††booktitle:
Proceedings of the 46th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’23), July 23–27, 2023, Taipei,
Taiwan††price: 15.00††doi: 10.1145/3539618.3591720††isbn:
978-1-4503-9408-6/23/07††ccs: Information systems Recommender systems

##  1\. Introduction

Figure 1. Dianping review feed recommendaiton and some examples to explain how
tags are associated with the review, search, and consumption domain. Best
viewed in color.

Dianping review feed recommendaiton.

Dianping is one of the most popular online platforms in China111Monthly,
hundreds of million UV and ten million user-generated reviews are accumulated.
for discovering local life services, including dining, shopping,
entertainment, hotel, etc.. With the purpose of seeking trusted local business
information, users are eager to search and consume in Dianping and write their
reviews to share experiences. To increase user retention rates, Dianping
establishes a review feed recommender system on the homepage as shown on the
left side of Figure 1, which faces a severe data sparsity problem in the
collection log222Here, our goal is to improve the collection rate rather than
the click-through rate, because the former is the most important indicator for
profit growth in our business data analysis.. Take one-month data (Nov. 2022)
as an example, Figure 2(a) shows the distribution of collection numbers across
the users. Cold-start users account for about 96% of all users, while active
users (with more than ten collect histories) less than 0.2%.

To solve this dilemma, we introduce the more accessible light-weight textual
side information called tags. Tag is a word or phrase, which refers to an
object or activity, like hot pot and skiing. Take Figure 1 as an example, we
input beijing barbecue in the search bar (blue box) and the associated tags
are beijing and barbecue. Obviously, user tag-based interests extracted from
other domains are useful for suggesting reviews on the homepage. For example,
a user might get interested in a bar review because he searches for wines. We
count tags associated with three domains (review, search, and consumption) via
the same one-month data as shown in Figure 1. The results are in Figure 2(b)
and show that users have rich behaviors in the latter two domains, especially
for cold-start and inactive users. Therefore, (Q1) how to leverage tag-based
contents to enhance recommendations in the target domain is a key issue.

Furthermore, we analyze whether all associated tags of search and consumption
are advantageous for review feed recommendation, that is, the proportion of
useful tags. We gather tags associated with the above three domains in Nov.
2022, called training tags, and then collect tags just associated with reviews
in the following week (from Dec. 1, 2022 to Dec. 7, 2022), called testing
tags. All tags are embedded by SBERT (Reimers and Gurevych, 2019), which is
pretrained on Dianping corpus specifically. For testing tags, we mine the
relevant training tags by setting a threshold of cosine similarity between two
embeddings. Figure 3 visualizes tag embeddings of a randomly selected user and
shows the percentage of useful tags by averaging all users at the bottom
right, which reveals a reasonable phenomenon that only a small part of tags
(less than 10% when the threshold is 0.7) can represent user interests towards
reviews. Therefore, (Q2) another pivotal problem is to distill a fraction of
useful tags representing intra-domain interests from a large quantity, most of
which are noises.

(a) Data sparsity of collection log.

(b) Tag distribution across the users in three domains.

Figure 2. User cross-domain behavior analysis.

Cross-domain recommendation (CDR) is an intuitive branch to solve the above
problems. Early research (Singh and Gordon, 2008; Yan et al., 2019) focus on
different mapping strategies for user cross-domain preferences. However, all
of them don’t introduce content information. To achieve content enhanced
recommendation, some single-domain workings realize great success (Seo et al.,
2017; Wu et al., 2019; Zheng et al., 2017; Xia et al., 2019). Based on the
previous efforts, content enhanced CDR methods firstly extract content-aware
interests and then transfer them across domains. CATN (Zhao et al., 2020)
adopts text convolution to learn aspect-level features, which are matched
between the source and target domain via a correlation matrix. MTNet (Huang
and Stokes, 2016) attentively distills useful contents via a memory network
and selectively feeds them into a transfer network. However, these approaches
neglect how to model tag-based contents. In recent years, graph neural network
(GNN) based CDR has attracted considerable research attention for its
superiority in modeling high-order connections, like PPGN (Zhao et al., 2019)
and HecRec (Yin et al., 2019). However, the aggregation via the GNN layer in
the above methods overlooks noisy neighbor nodes, which are indeed harmful
when the unuseful nodes account for the most.

Figure 3. The proportion of useful tags and an example.

The proportion of useful tags and an example.

In this paper, we propose a metapath and multi-interest aggregated graph
neural network for tag-based cross-domain recommendation, called M2GNN. To
solve Q1, we leverage the ability of graph to recognize the pattern of non-
euclidean data (Zhou et al., 2020) including tag-based contents. Technically,
a tag-associated heterogeneous graph is established to model the multi-homed
behaviors, where each domain is represented by a metapath. Then a GNN-based
method is designed to explore high-order tag-based cross-domain interests for
enhancing recommendation in the target domain. To address Q2, we introduce a
novel two-step aggregation mechanism employing the dynamic routing network and
self-attention network. The former filters out the trivial tags and learns
multiple high-level user interests in each domain. The latter furthermore
transfers more significant interests into the target domain. Moreover, we
introduce the skip-gram regularization to model the semantic correlation
between tags. To summarize, the main contributions of this work are as
follows:

  * •

We propose a novel CDR problem whose side information is tags and design a
GNN-based method to solve it.

  * •

We present a hierarchical aggregation schema to learn user preferences when
noisy neighbor nodes are in the majority.

  * •

We conduct offline experiments on a newly proposed industrial dataset and a
public dataset, which proves the effectiveness and interpretability of M2GNN.
Online A/B tests demonstrate the improvement to the current industrial
recommender system.

##  2\. Preliminaries

###  2.1. Problem Setup

Tag-based cross-domain recommendation (TCDR). We have a target domain (e.g.
review collection) and some source domains (e.g. search and consumption),
denoted as $D^{t}$ and $D^{s}$, where $S=\\{s_{1},s_{2},...\\}$ represents
there are $|S|$ source domains. Each domain includes its users $u$, items $i$,
and interactions $\hat{y}_{ui}$. Note that users are full overlap, also known
as multi-homed users, while items are non-overlap. We identify the item in
different domains by its superscript, such as $i^{t}$, $i^{s_{1}}$, and
$i^{s_{2}}$. According to interaction numbers in the target domain, users are
divided into three groups, which are cold-start, inactive and active users.
Let $U^{cs}$, $U^{ia}$ and $U^{a}$ represent them, respectively. In this
paper, the bound to distinguish the latter two groups is set to be 10 times.
Cold-start users have no interactions with items in $D^{t}$ , but with items
in $D^{s}$. Each item in all domains has the corresponding content information
denoted by $D_{i}$. Different from the document with a common structure of the
linguistic sequence, $D_{i}$ is a set of tags. Note that all tags come from
the same vocabulary333Dianping constructs a large-scale tag database extracted
from user-generated contents via a customized tokenizer and provides a unified
API to associate tags with contents., which is shared within all domains.

The goal of TCDR is to improve the recommendation performance in the target
domain by exploring latent tag-based interests behind cross-domain behaviors,
which is especially needed for $U^{cs}$ and $U^{ia}$. In this work, we target
at the top-K recommendation task444Dianping review feed recommender system is
a complex industrial system with a tandem structure. After the top-K rating,
also known as the matching or recall process, a customized CTR module will
rate all candidate items by taking into account a cluster of additional
features, like user age, occupation, and whether paid for advertising.
Therefore, this work just aims to improve the recall performance..

###  2.2. Graph Construction

Definition 2.1. Tag-associated Heterogenous Graph. Figure 4 illustrates three
kinds of interactions from the review feed, search, and consumption domain in
Dianping APP. There are six kinds of entities in this TCDR task, which are
user, review feed, search query, POI, review of POI, and tag. We denote them
as
$\mathcal{U}=\\{u\\}$,$\mathcal{R}=\\{r\\}$,$\mathcal{Q}=\\{q\\}$,$\mathcal{P}=\\{p\\}$,
$\mathcal{R}_{\mathcal{P}}=\\{r_{p}\\}$ and $\mathcal{T}=\\{t\\}$,
respectively. Obviously, $i^{t}$ is identical to $r$,
$i^{s}=\\{i^{s_{1}},i^{s_{2}}\\}=\\{q,p\\}$ and $D_{i}=\\{t\\}$. Eight
relations are used to connect the above nodes, which are
$\\{u-r,u-q,u-p,p-r_{p},r-t,q-t,r_{p}-t,t-t\\}$. The first seven edges are
easy to understand and we explain the last relation. $t-t$ represents two tags
that have related semantics, like Japanese barbecue and Japanese coffee. We
use Elasticsearch (ES) to store all tags and utilize its search engine to
construct $t-t$ by regarding each tag as a query. Then a heterogeneous graph
$\mathcal{G}$ is defined as $\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$ with
nodes $\mathcal{V}$ and edges $\mathcal{E}$. In Dianping dataset,
$\mathcal{V}=\\{\mathcal{U},\mathcal{R},\mathcal{Q},\mathcal{P},\mathcal{R}_{\mathcal{P}},\mathcal{T}\\}$
and $\mathcal{E}=\\{u-r,u-q,u-p,p-r_{p},r-t,q-t,r_{p}-t,t-t\\}$. Note that
only $t-t$ is a symmetric relation and the others are directed.

Definition 2.2. Domain-specific Metapath. A metapath $\rho$ is a path defined
on the graph $\mathcal{G}$, and is denoted in the form of ordered
nodes$A_{1}\rightarrow A_{2}\rightarrow...\rightarrow A_{L}$. $d_{\rho}$ is
the number of metapaths. In Dianping dataset, three are three metapaths
($d_{\rho}=3$) to represent domain-specific knowledge. $u-r-t$ represents user
interested tags from the collected review feed. $u-q-t$ contains user
potential preference inferred from his search log. $u-p-r_{p}-t$ describes
what content of a local business can develop users’ willingness to consume.

Definition 2.3. Metapath-based neighbor. Given a metapath $\rho$, the
metapath-based neighbors $\mathcal{N}^{\rho}_{v}$ of a node $v$ is defined as
the set of nodes that connect with $v$ using $\rho$.

Figure 4. The tag-associated heterogenous graph constructed by Dianping data.
Best viewed in color.

The heterogenous graph constructed by Dianping data.

##  3\. The M2GNN framework

Figure 5. The framework of M2GNN. Best viewed in color.

The framework of M2GNN.

###  3.1. Motivation

As aforementioned in Q1, the content side information in this TCDR task is
tags. Different from contents with the linguistic sequence, tags have no
natural sequential distribution, leading to the recent works focusing on
sequential feature extraction like CATN, which will be compared in Section 4,
is less applicable. What’s more, tags have an important feature, that is
global shareability, which means any items in all domains are described by the
same tag dict. This inspires us to recall the review feed including the same
or similar tags appeared in other domains. For example,
$user\xrightarrow{search}korean\,barbecue\xrightarrow{associated\,with}barbecue\ext@arrow
9999{\arrowfill@\leftarrow\relbar\rightarrow}{}{has\,related\,semantics}janpanese\,barbecue\xleftarrow{associated\,with}review\,feed$
can help to provide a review about janpanese barbecue for a user who searches
korean barbecue. GNN, an intuitive tool to explore such a high-order reasoning
chain, motivates us to build a tag-based graph and corresponding neural
networks.

With determining to employ the GNN framework, we face severe noisy information
in aggregation as shown in Q2. This is an inevitable problem when the number
of attention members is large, which also appears in GraphAir (Hu et al.,
2021). Hard attention (Xu et al., 2015), also known as nondeterministic
attention, is a close solution to focus on exactly one or a few members, which
will be compared in Section 4. However, the tag is a fine-grained object,
which means a cluster of related tags indicates one entire high-level user
preference. Therefore, discarding most of tags directly adopted in hard
attention will lose content side information and degrade performance. Inspired
by the success of multi-interest recommender systems (RSs), we find extracting
multiple interests from each domain can distill intra-domain interests in a
corpus where noisy contents account for the most with two reasons: (1)
learning different aspects of user preference can preserve main but filter out
trivial tags; (2) attention mechanism works better in transferring high-level
interests into target domain than transferring tags directly because the
number of interests is much less than that of tags.

###  3.2. Overview

A general framework that can generalize most GNN-based RSs (e.g. NGCF (Wang et
al., 2019b), LightGCN (He et al., 2020), KGAT (Wang et al., 2019a), KGIN (Wang
et al., 2021), GC-SAN (Xu et al., 2019) and SURGE (Chang et al., 2021)) adopts
the following scheme: aggregate feature (neighbor) nodes to user or item
(center) nodes and update $\rightarrow$ pool node representations after
multiple layers $\rightarrow$ train the model according to downstream tasks
like top-K recommendation or CTR prediction.

Inspired by the above straightforward but effective idea, M2GNN aggregates tag
nodes into user nodes by domain-specific metapaths without self-connection,
learns item embeddings by their corresponding tags, and enhances
representations of tag nodes by their homogeneous neighbors with related
semantics. Since the number of tags associated with an item is small, adopting
complex aggregation methods is unnecessary and here we use average pooling.
The inherence of the tag-to-tag update is to bring the vectors of two tags
with a similar meaning closer in latent semantic space, we also utilize
average pooling and leave other aggregation functions like attention as a
future work. Then sum operation is conducted to pool multi-layer embeddings,
which is widely used in GNN for the recommendation task to fuse multi-hop
collaborative filtering signals. Finally, the inner product is adopted to
predict top-K items. Formally, the above message passing and recommendation
mechanism are defined as follows,

(1) $\displaystyle\mathbf{e}_{t}^{l+1}\leftarrow\mathop{\bm{{\rm
Mean}}}\limits_{t^{\prime}\in\mathcal{N}^{t-t}_{t}}(\mathbf{e}_{{t^{\prime}}}^{l})$
(2) $\displaystyle\mathbf{e}_{r}^{l+1}\leftarrow\mathop{\bm{{\rm
Mean}}}\limits_{t\in\mathcal{N}^{r-t}_{r}}(\mathbf{e}_{{t}}^{l})$ (3)
$\displaystyle\mathbf{e}_{u}^{l+1}\leftarrow\mathop{\bm{{\rm
Aggregate}}}\limits_{t\in\mathcal{N}^{\rho}_{u}}(\mathbf{e}_{t}^{l})$ (4)
$\displaystyle\mathbf{e}^{*}_{v}=\sum_{l=0}^{L}\mathbf{e}^{l}_{v}$ (5)
$\displaystyle y(u,r)={\mathbf{e}^{*}_{u}}^{\top}\mathbf{e}^{*}_{r}$

where $v$ can be either a user or an item node in target domain $r$,
$\mathbf{e}_{v}^{l}\in\mathbb{R}^{d\times 1}$ denotes the representation of
node $v$ in the $l$-th layer, superscript $*$ indicates the final
representation for predicting, $y(u,r)$ is the matching score of a pair of
user and item.

Obviously, the key component is equation 3. As mentioned in Section 3.1, we
first learn multiple interests in each domain (cf. Section 3.3) and transfer
them into the target domain (cf. Section 3.4).

###  3.3. Intra-domain Aggregation

We utilize the dynamic routing network from (Sabour et al., 2017) as multi-
interest extractor. For each domain (metapath), tag embeddings and user
preferences are regarded as low and high-level capsules, respectively. Given a
set of metapath-based neighbor tag nodes $t\in\mathcal{N}^{\rho}_{u}$, we
learn interest capsules as

(6)
$\displaystyle\mathbf{c}_{u,j}^{\rho,l}=\sum_{t\in\mathcal{N}^{\rho}_{u}}w_{tj}\mathbf{S}\mathbf{e}_{t}^{l}$

where $\mathbf{c}_{u,j}^{\rho,l}$ denotes $j$-th interest capsule of user $u$
using metapath $\rho$ in $l$-th layer, $w_{tj}$ represents the contribution
degree of low-level capsule $t$ to high-level capsule $j$,
$\mathbf{S}\in\mathbb{R}^{d\times d}$ is transformation matrix and is shared
across all interest capsules and metapaths.

The superiority of dynamic routing is to update $w_{tj}$ via a new parameter
called routing logit $b_{tj}$ in an iterative way. We initialize $b_{tj}$
randomly in the first iteration. $w_{tj}$ is calculated by performing softmax
on $b_{tj}$ as

(7) $\displaystyle w_{tj}=\frac{{\rm exp}\ b_{tj}}{\sum_{z=1}^{K}{\rm exp}\
b_{tz}}$

where $K$ is the number of interest capsules and is adaptively calculated by
the number of interacted tags as

(8) $\displaystyle K={\rm max}(1,{\rm min}(K_{\rm
max},\lceil\log_{2}(|\mathcal{N}^{\rho}_{u}|)\rceil))$

where $K_{\rm max}$ is the maximum number of interest capsules and $\lceil
x\rceil$ refers to the smallest integer that is more than or equal to $x$.

To normalize the length of the output vector of a capsule between 0 and 1,
which benefits updating $b_{tj}$ in next iteration, a non-linear squashing
function is utilized as

(9) $\displaystyle\mathbf{z}_{u,j}^{\rho,l}={\rm
squash}(\mathbf{c}_{u,j}^{\rho,l})=\frac{\left\|\mathbf{c}_{u,j}^{\rho,l}\right\|^{2}}{1+\left\|\mathbf{c}_{u,j}^{\rho,l}\right\|^{2}}\frac{\mathbf{c}_{u,j}^{\rho,l}}{\left\|\mathbf{c}_{u,j}^{\rho,l}\right\|}$

Then $b_{tj}$ is updated by

(10) $\displaystyle
b_{tj}=b_{tj}+\mathbf{z}_{u,j}^{\rho,l}\mathbf{S}\mathbf{e}_{t}^{l}$

Finally, the above routing process is repeated multiple times, which is set to
be three, to obtain the converged output of interest capsules.

###  3.4. Inter-domain Aggregation

We collect all interest embeddings as a matrix
$\mathbf{Z}_{u}^{l}=[\mathbf{z}_{u,1}^{\rho_{1},l},...,\mathbf{z}_{u,K_{\rm
max}}^{\rho_{1},l},\mathbf{z}_{u,1}^{\rho_{2},l},...\mathbf{z}_{u,K_{\rm
max}}^{\rho_{d_{\rho}},l}]\in\mathbb{R}^{d\times(d_{\rho}\cdot K_{\rm max})}$.
For faster and unified tensor calculation, we pad zero vectors when $K<K_{\rm
max}$. Now each user has an interest matrix ($\mathbf{Z}_{u}^{l}$) consisting
of many interest embeddings ($\mathbf{z}_{u,j}^{\rho,l}$). To transfer useful
preferences to the target domain, we assign a weight $\alpha_{u,j}^{\rho,l}$
to each interest embedding and output an intra-domain interest embedding by
pooling $\mathbf{Z}_{u}^{l}$ attentively.

Since sparse data in the target domain is undesirable to train accurate
weights, a reasonable alternative solution is to utilize user similarity. For
example, if two users have the same search and consumption log, it is
reasonable to recommend a review feed collected by one user to the other.
Technologically, we aim to assign the same weight to a pair of close interest
embeddings from two users with similar portraits (a.k.a.
$\mathbf{Z}_{u}^{l}$). Therefore, a global shared mechanism is required. To
achieve this, a self-attention module is designed as

(11)
$\displaystyle\mathbf{H}_{u}^{l}=\mathbf{S}_{2}^{\top}\tanh\left(\mathbf{S}_{1}\mathbf{Z}_{u}^{l}\right)$

where $\mathbf{H}_{u}^{l}\in\mathbb{R}^{1\times(d_{\rho}\cdot K_{\rm max})}$
is the attention vector, $\mathbf{S}_{1}\in\mathbb{R}^{d\times d}$ and
$\mathbf{S}_{2}\in\mathbb{R}^{d\times 1}$ are trainable matrixes, which are
shared across all users.

Q2 reveals that only a fraction of interests contributes to the target domain,
therefore we introduce the element-wise exponentiation operator before a
softmax function to calculate weights as:

(12) $\displaystyle\alpha_{u,j}^{\rho,l}=\frac{{\rm exp}({\rm
pow}(h_{u,j}^{l},\gamma))}{\sum_{z=1}^{d_{\rho}\cdot K_{\rm max}}{\rm
exp}({\rm pow}(h_{u,z}^{l},\gamma))}$

where $h_{u,j}^{l}$ is the $j-$th element in $\mathbf{H}_{u}^{l}$ and $\gamma$
is a hyperparameter to control the attention distribution. As $\gamma$
increases, more significant interest embeddings will receive more attention
and bigger weights, and vice versa. Consider the limit case, $\gamma=0$ is
identical to mean pooling and $\gamma=\infty$ means the hard attention which
chooses only one candidate as output. Therefore, $\gamma$ is set to be bigger
than 1, which can alleviate the problem in Q2.

Finally, the user representation is inferred to represent his intra-domain
interests as

(13)
$\displaystyle\mathbf{e}_{u}^{l+1}=\mathbf{Z}_{u}^{l}{\mathbf{A}_{u}^{l}}^{\top}$

where $\mathbf{A}_{u}^{l}=[\alpha_{u,1}^{\rho,l},...,\alpha_{u,d_{\rho}\cdot
K_{\rm max}}^{\rho,l}]\in\mathbb{R}^{1\times(d_{\rho}\cdot K_{\rm max})}$ is
the weight matrix of user $u$ in $l$ layer.

###  3.5. Skip-gram Regularization

It is clear that the quality of tag embeddings deeply affects the clustering
performance in Section 3.3 and weight computing in Section 3.4. To further
draw related tags closer and unconcerned tags removed in latent semantic
space, we adopt skip-gram idea to regularize the training process.
Specifically, a tag node is regarded as the center word, its homogeneous
neighbor nodes are set to be nearby words. The principle is to maximize the
co-occurrence probability of related tags, and vice versa. We randomly sample
a negative node for each positive pair of tags and define skip-gram objective
loss as

(14)
$\displaystyle\mathcal{L}_{1}=-\frac{1}{|\mathcal{E}^{t-t}|}\sum_{t\in\mathcal{T}}\sum_{t^{\prime}\in\mathcal{N}^{t-t}_{t},t^{\prime\prime}\notin\mathcal{N}^{t-t}_{t}}(p(t^{\prime}\mid
t)-p(t^{\prime\prime}\mid t))$

where $|\mathcal{E}^{t-t}|$ denotes the number of all edges belonging to type
$t-t$ in graph $\mathcal{G}$.

The basic formulation to model probability $p$ is the global softmax performed
on all tags, which is impractical for the huge cost of time and calculation.
Inspired by hierarchical softmax, an auxiliary set of trainable embeddings
$\mathcal{V}\in\mathbb{R}^{d\times|\mathcal{T}|}$ is introduced to define $p$
as

(15) $\displaystyle p(t^{\prime}\mid
t)=\operatorname{log}\sigma({\mathbf{v}_{t^{\prime}}}^{\top}\mathbf{e}_{t}^{0})$

where $\mathbf{v}_{t^{\prime}}$ is the embedding of tag $t^{\prime}$ from
$\mathcal{V}$, $\mathbf{e}_{t}^{0}$ is the initial embedding of tag $t$.

###  3.6. Model Optimization

With the user representation $\mathbf{e}^{*}_{u}$ and the review
representation $\mathbf{e}^{*}_{r}$ ready, equation 5 is adopted to predict
top-K items. We employ the BPR loss (Rendle et al., 2012) as

(16)
$\displaystyle\mathcal{L}_{2}=-\frac{1}{|\mathcal{O}|}\sum_{(u,r^{+},r^{-})\in\mathcal{O}}\ln\sigma\bigg{(}y(u,r^{+})-y(u,r^{-})\bigg{)}+\lambda\|\Theta\|_{2}^{2}$

where
$\mathcal{O}=\\{(u,r^{+},r^{-})|(u,r^{+})\in\hat{y}_{ui^{t}},(u,r^{-})\notin\hat{y}_{ui^{t}}\\}$
denotes the training set, where $r^{+}$ is the observed interaction and
$r^{-}$ is a negative counterpart which is sampled from the unobserved
interactions randomly, $\sigma(\cdot)$ is the sigmoid function.
$\Theta=\\{\mathbf{e}_{x}^{0},\mathcal{S},\mathcal{S}_{1},\mathcal{S}_{2},\mathbf{v}_{t}|x\in\\{\mathcal{U},\mathcal{R},\mathcal{T}\\},t\in\mathcal{T}\\}$
is the model parameter set. $L_{2}$ regularization parameterized by $\lambda$
on $\Theta$ is conducted to prevent overfitting. Finally, the joint loss is

(17) $\displaystyle\mathcal{L}=\mathcal{L}_{1}+\mathcal{L}_{2}$

##  4\. experiments

We evaluate our proposed M2GNN method on two benchmark datasets to answer the
following research questions:

  * •

RQ1: Does our proposed M2GNN outperform the state-of-the-art recommendation
methods?

  * •

RQ2: How do different components (i.e., two-step aggregation module and skip-
gram regularization) and hyperparameters (i.e. capsule numbers, GNN layer
numbers, and exponent before softmax) affect M2GNN?

  * •

RQ3: Does M2GNN achieve great improvement towards cold-start and inactive
users?

  * •

RQ4: Is there an online improvement of review feed recommendation in Dianping
homepage by M2GNN?

  * •

RQ5: Can M2GNN provide potential explanations about tag-based preferences of
users?

###  4.1. Experimental Settings

Dataset Description. We propose a novel industrial dataset, called DPBJ
(Dianping Beijing). The dataset is encrypted and desensitized, and does not
contain any personally identifiable information. To avoid data leakage, we
collect the multi-site data in one month (from Nov. 1, 2022, to Nov. 30, 2022)
as the training and validation set. Then the latter one-week data (from Dec.
1, 2022, to Dec. 7, 2022) is the testing set. We also use a common dataset,
Amazon review dataset555http://jmcauley.ucsd.edu/data/amazon/, where the book
and movie are chosen as the source and target domain. To ensure data quality,
we remove the interaction records without review text and some low-quality
items following earlier studies (He et al., 2017; Kang et al., 2019). We split
the review text into individual words and filter out stopwords by word
tokenization tool like nltk666https://github.com/nltk/nltk. The above all
words comprise the tag dict. The detailed statistics of these two datasets are
reported in Table 1.

Table 1. Statistics of two datasets. $m$ and $b$ denote movie and book, respectively. ’int.’ indicates the interaction data of each domain. |  | DPBJ | Amazon
---|---|---|---
# $u$ | 6,156,837 | 41,465
# $t$ | 94,029 | 20,000
# $t$-$t$ | 1,679,832 | 184,776
$D^{t}$ | item | # $r$ | 2,036,665 | # $m$ | 7,379
int. | # $u$-$r$ | 297,658 | # $u$-$m$ | 109,864
# $r$-$t$ | 2,917,242 | # $m$-$t$ | 49,127,984
$D^{s_{1}}$ | item | # $q$ | 3,436,125 | # $b$ | 34,396
int. | # $u$-$q$ | 54,527,624 | # $u$-$b$ | 1,425,569
# $q$-$t$ | 5,936,250 | # $b$-$t$ | 169,399,312
$D^{s_{2}}$ | item | # $p$ | 207,766 |
# $r_{p}$ | 2,103,670
int. | # $u$-$p$ | 15,473,732
# $p$-$r_{p}$ | 160,263
# $r_{p}$-$t$ | 3,646,553
metapath | # $u$-$r$-$t$ | 497,532 | # $u$-$m$-$t$ | 64,498,631
# $u$-$q$-$t$ | 73,904,447 | # $u$-$b$-$t$ | 354,326,927
# $u$-$p$-$r_{p}$-$t$ | 184,203,329 |
Table 2. Performance comparison. $\blacktriangle$% denotes the relative improvement of M2GNN over the best SOTA algorithm. | DPBJ | Amazon
---|---|---
Recall@50 | Recall@100 | Hit@50 | Hit@100 | Recall@50 | Recall@100 | Hit@50 | Hit@100
single-domain | MF | 0.0265 | 0.0421 | 0.0529 | 0.0796 | 0.0612 | 0.0989 | 0.3874 | 0.5218
LightGCN | 0.0388 | 0.0595 | 0.0697 | 0.1045 | 0.1044 | 0.1599 | 0.5232 | 0.6475
content-free cross-domain | EMCDR | 0.0425 | 0.0699 | 0.0752 | 0.1137 | 0.0670 | 0.1056 | 0.4271 | 0.5790
PTUPCDR | 0.0499 | 0.0731 | 0.0825 | 0.1331 | 0.0727 | 0.1341 | 0.4901 | 0.6515
content-based cross-domain | CATN | 0.0304 | 0.0495 | 0.0560 | 0.0829 | 0.0815 | 0.1354 | 0.5323 | 0.6811
MTNet | 0.0531 | 0.0822 | 0.0877 | 0.1362 | 0.0707 | 0.1199 | 0.4761 | 0.6377
GNN-based cross-domain | HeroGRAPH | 0.0437 | 0.0713 | 0.0884 | 0.1381 | 0.0884 | 0.1523 | 0.5514 | 0.7065
HCDIR | 0.0320 | 0.0562 | 0.0653 | 0.1099 | 0.1005 | 0.1680 | 0.5785 | 0.7194
GA-MTCDR-P | 0.0341 | 0.0577 | 0.0690 | 0.1153 | 0.1028 | 0.1703 | 0.5977 | 0.7318
M2GNN | 0.0629 | 0.0953 | 0.0967 | 0.1464 | 0.1115 | 0.1783 | 0.6471 | 0.7768
$\blacktriangle$% | 18.46 | 15.94 | 9.39 | 6.01 | 6.80 | 4.70 | 8.27 | 6.15

Baselines. To demonstrate the effectiveness, we compare M2GNN with single-
domain (MF, lightGCN), content-free cross-domain (EMCDR, PTUPCDR), content-
based cross-domain (CATN, MTNet), and GNN-based cross-domain (HeroGRAPH,
HCDIR, GA) methods.

  * •

MF (Rendle et al., 2012) is a matrix factorization method optimized by the
Bayesian personalized ranking (BPR) loss.

  * •

lightGCN (He et al., 2020) simplifies NGCF (Wang et al., 2019b) and provides
simple but effective model for user-item bipartite graph-based RS.

  * •

EMCDR (Man et al., 2017) first pretrains embeddings in each domain and then
learns a mapping function to transfer interests across domains.

  * •

PTUPCDR (Zhu et al., 2022) designs a meta network fed with users’
characteristic embeddings to generate personalized bridge functions to achieve
personalized transfer of user preferences.

  * •

CATN (Wang et al., 2019a) extracts multiple aspects from the documents and
learns aspect correlations across domains attentively.

  * •

MTNet (Hu et al., 2018) designs a memory component to find the words highly
relevant to the user preferences, which are then fused with source domain-
specific knowledge to improve final prediction.

  * •

HeroGRAPH (Cui et al., 2020) constructs a heterogeneous graph to fuse all
interactions as a whole and learn additional global embeddings to enhance
recommendation in all domains.

  * •

HCDIR (Bi et al., 2020) separately learns user embeddings in each domain via
an insurance information network and then employs MLP to perform latent space
matching across domains.

  * •

GA (Zhu et al., 2021) proposes a unified framework, which constructs separate
heterogeneous graphs to generate more representative embeddings and uses an
element-wise attention mechanism to effectively combine them of common
entities.

Parameter Settings. We implement our M2GNN model in Pytorch and Deep Graph
Library (DGL)777https://github.com/dmlc/dgl, which is a Python package for
deep learning on graphs. Since DPBJ has million-scale nodes and billion-scale
edges, the API of distributed data parallel (DDP) in Pytorch is used to train
M2GNN and the data loading package in DGL is adopted to generate subgraph
samples for each batch. We released all implementations (code, datasets,
parameter settings, and training logs) to facilitate
reproducibility888https://github.com/huaizepeng2020/M2GNN_git. The embedding
size of all models is set to be 64 for a fair comparison. We adopt Adam
(Kingma and Ba, 2014) as the optimizer and the batch size is fixed at 1024 for
all methods.

Evaluation Metrics. As mentioned in Section 2.1, Recall@$K$ and Hit@$K$999K is
set to be 50 rather than a commonly used value like 20 or 100. The reason is
that the subsequent CTR module only ranks the top 50 items from the recall
phase. are used in the task of top-K recommendation.

###  4.2. Performance Comparison (RQ1)

We report the empirical results in Table 2. The observations are as follows:

  * •

M2GNN consistently achieves the best performance on two datasets in terms of
all measures. Specifically, it achieves significant improvements over the
strongest baselines w.r.t. Recall@50 by 18.46%, 6.80% in DPBJ and Amazon,
respectively. The results prove the effectiveness of M2GNN.

  * •

Despite some particular cases, the average performances of content-aware
(content-based&GNN-based) cross-domain, content-free cross-domain, and single-
domain models are in decreasing order. A clear reason is whether to introduce
the auxiliary information, like contents or cross-domain interactions.

  * •

MTNet outperforms CATN in DPBJ while underperforms CATN in Amazon. The reason
is the difference between content extractors in these two models. MTNet adopts
an attention network to model unstructured text without regarding the
sequential feature, thus performing better in DPBJ dataset. CATN utilizes text
convolution and has superiority in modeling sequential signals, which is more
suitable for the Amazon dataset.

  * •

HeroGRAPH outperforms GA-MTCDR-P (HCDIR) in DPBJ while underperforms GA-
MTCDR-P (HCDIR) in Amazon. DPBJ has a large number of noisy tags, leading to a
big gap between intra and cross-domain interests. Therefore, using multiple
embeddings to represent domain-specific interests separately (HeroGRAPH)
performs better than using one representation (GA-MTCDR-P/HCDIR). However,
Amazon dataset has higher consistency in user preferences toward books and
movies, thus aggregating cross-domain interests is more applicable.

Table 3. Impact of two-step aggregation and skip-gram regularization. | DPBJ | Amazon
---|---|---
Recall@50 | Hit@50 | Recall@50 | Hit@50
M2GNN-mean | 0.0511 | 0.0762 | 0.0864 | 0.5264
M2GNN-softmax | 0.0524 | 0.0791 | 0.0837 | 0.4879
M2GNN-HAN | 0.0544 | 0.0809 | 0.0898 | 0.5631
M2GNN-hard | 0.0586 | 0.0873 | 0.0951 | 0.6062
M2GNN-w/o$\mathcal{L}_{1}$ | 0.0526 | 0.0822 | 0.0903 | 0.5400

###  4.3. Study of M2GNN (RQ2)

Figure 6. Effect of hyperparameters. Best viewed in color.

The impact of hyperparameters.

Figure 7. Performance comparison over the sparsity distribution of user
groups. The background histograms indicate the density of each user group and
‱ means permyriad.

The impact of hyperparameters.

Impact of two-step aggregation module. To demonstrate the superiority of the
two-step aggregation module, we compare the performance of M2GNN with the
following four variants which adopt different methods of aggregation.

  * •

M2GNN-mean uses average pooling in both two steps.

  * •

M2GNN-softmax adopts the standard schema of attention, proposed in Transformer
(Vaswani et al., 2017), in both two steps. $K$ is user embedding, while $Q$
and $V$ are tag embeddings.

  * •

M2GNN-HAN regards intra-domain and inter-domain aggregation as node-level and
semantic-level attention in HAN (Wang et al., 2019c).

  * •

M2GNN-hard utilizes local alignment attention in (Luong et al., 2015), which
extends original hard attention (Xu et al., 2015) to select a subset of input
members via implementing a softmax distribution.

The results are shown in the upper part of Table 3 and the major findings are
as follows:

  * •

M2GNN-mean and M2GNN-softmax achieve the worst performances on the two
datasets. The reason is both of them don’t track the noisy content problem.
The former regards all tags as equally important, while the latter receives a
large number of attention members, leading to the weights of significant tags
are not trained as expected.

  * •

M2GNN-HAN slightly outperforms the first two variants. A possible reason is
that the different mechanism to calculate attention scores using the type-
specific transformation matrix and the weight coefficient helps to increase
the weights of useful tags.

  * •

M2GNN-hard performs the best among the four variants. A clear reason is that
only a subset of significant tags are aggregated into user representations,
which avoids using all tags to infer final embeddings. However, as
aforementioned in Section 3.1, it is inevitable to discard some useful tags in
hard attention, which causes that M2GNN-hard underperforms M2GNN.

Impact of skip-gram regularization. To demonstrate the superiority of the
skip-gram regularization, we discard $\mathcal{L}_{1}$ and only leverage
collaborative filtering signals to train the model, termed
M2GNN-w/o$\mathcal{L}_{1}$. The lower part of Table 3 shows that
M2GNN-w/o$\mathcal{L}_{1}$ sharply underperforms M2GNN without regularizing
tag embeddings via $\mathcal{L}_{1}$ , which proves the necessity of the skip-
gram regularization.

Impact of hyperparameters. We investigate the influence of three
hyperparameters: layer numbers, capsule numbers, and exponent before softmax,
all of which greatly affect multi-interest extraction and transfer. $L$,
$K_{{\rm max}}$ and $\gamma$ are searched in $\\{1,2,3\\}$, $\\{2,4,6,8\\}$
and $\\{2,4,6,8\\}$, respectively. Figure 6 shows the results in the DPBJ
dataset and it has a similar conclusion in the Amazon dataset, which is
omitted. From the observations above, we make several conclusions. Firstly,
$L=2$ results in the best performance, which means 2-order connections
integrate more similar tags into users and stacking more layers continually
brings in noisy tags, which is harmful to the recommendation. Secondly, the
increase of interest capsules indeed improves the recall list since local
businesses contain multiple aspects, while there is a saturation limit which
has a similar conclusion in (Li et al., 2019). Thirdly, the optimal number of
$\gamma$ is 6~8. The reason is that a proper exponent helps the softmax
function to identify more important interests and assign bigger scores.

Figure 8. Case study. Each tag is represented by a color block. The original
Chinese text is in brackets and we translate it into English. Best viewed in
color.

Case study.

###  4.4. Cold-start and Inactive Users Analysis (RQ3)

One motivation for exploiting CDR is to alleviate the severe sparsity problem
in Dianping, where an overwhelming majority of users are cold-start in Figure
2. Therefore, we investigate whether introducing CDR helps alleviate this
issue. Towards this end, we divide the testing set into four groups based on
the interaction number per user, which has the same criterion in Section 1.
Metrics are calculated for each group individually. EMCDR and HCDIR have
similiar performances with PTUPCDR and GA-MTCDR-P respectively, therefore they
are omitted. Figure 7 illustrates the results and we have two conclusions: (1)
All models perform better for active users since they have richer interactions
in the target domain. (2) There is a marked decrease caused by the data
sparsity problem for inactive and cold-start users in baselines, while M2GNN
has the least impact. This proves M2GNN has stronger robustness to serve the
inactive and cold-start users.

###  4.5. Online A/B Test (RQ4)

We report a 2-week online A/B test experiment of M2GNN in Beijing, from Dec.
1, 2022, to Dec. 15, 2022. Specifically, the current recommender system on the
Dianping homepage101010This is not disclosed for confidentiality reasons. is
the base model. M2GNN is integrated into it as an additional recall method and
directly offers 200 reviews for the later rank stage, which is regarded as the
tested model. 3.4 million users participated in this A/B test. Three kinds of
metrics are chosen. ‘Collection number’ (CN) indicates the number of all
collected reviews, which is an absolute amount. ‘Collection number per user’
(UV-CN) means the number of collected reviews per user, which eliminates users
with no collection behaviors. ’Collection rate’ (CR) is the ratio of collected
reviews to all exposed reviews. We just give the increase rate since the true
value of each metric is sensitive. As in Table 4, M2GNN achieves great
improvements relative to the base model. What’s more, the improvement of
inactive and cold-start users (1.1986% CR) is more significant than that of
active users (0.4663% CR), which demonstrates the superiority of M2GNN in
alleviating data sparsity problems by transferring cross-domain knowledge.

Table 4. Online metrics in a week. Metrics | CN | UV-CN | CR
---|---|---|---
active users | +0.4715% | +0.3749% | +0.4663%
inactive and cold-start users | +1.0957% | +1.3447% | +1.1986%

###  4.6. Case Study (RQ5)

To prove the interpretability of M2GNN, we visualize the two-step aggregation
process to show how to distill intra-domain interests from tags, most of which
are noises. Figure 8 takes a specific user as an example. The bottom lists all
tags from three domains, and obviously he is a cold-start user because of no
interactions in the target domain. In the pink block, there are four interest
capsules in each domain according to equation 8. The number $ⓧ$ near to a
capsule $\mathbf{z}$ represents that $x$-th tag has the largest routing logit
towards $\mathbf{z}$ among four capsules, which is also reflected by
highlighting tags with the same color. The results indicate that tags are
clustered via their semantics and the interest capsule represents the high-
level preference of each domain. For example, in the consumption domain, the
first capsule consists of tags about cakes (e.g. Dessert, Cream and Beijing
cake). What’s more, the influence of trivial tags (e.g. Mashed tar) is
weakened since a capsule can only represent one aspect of semantics. Then
eight weights are calculated by the self-attention module as shown in the blue
block. Three capsules, which are concerned with barbecue (0.213), cake
(0.413), and popular shop (0.203) respectively, account for the major portion
of final intra-domain interest. Finally, the top-100 candidate reviews contain
all ground-truth items at the right of the figure, which are highly relevant
to the above three interest capsules and confirm the reasonability and
interpretability of M2GNN.

##  5\. Related work

Our work is highly related to two subareas of recommendation, which are
content-based and GNN-based CDR.

Content-based CDR. The underlying assumption of this method is that the
interests extracted from descriptive information in the content can be shared
in different domains because of the consistency of user preferences. According
to the method for modeling text features, there are two kinds of content-based
CDR. The first way is directly projecting documents into the common low-
dimensional space (Elkahky et al., 2015; Fu et al., 2019; Hu et al., 2018).
RC-DFM (Fu et al., 2019) learns latent vectors of reviews by applying a max
pooling operation over pretrained word embeddings and utilizes the output of
hidden layer under SDAE (Vincent et al., 2010) framework to recommend items.
The second category of solutions fulfills high-level semantics extraction,
also known as aspects (Song et al., 2017; Zhao et al., 2020). CATN (Zhao et
al., 2020) leverages the aspect gate as a soft on-off switch to control which
latent feature derived from the text convolution layer is relevant to the
aspect, and then performs pair-wise aspect matchings as the final rating
prediction. Despite their success in processing text content, solutions in
both categories don’t specially take into consideration the corpus where the
majority of tokens are noise. To our best knowledge, M2GNN is the first
attempt to filter out noisy content and distill intra-domain interests
simultaneously.

GNN-based CDR. This category of algorithms aims to leverage the high-order
connectivity of GNN by transforming domain-specific knowledge into graphs.
There are two common graph data topologies: (1) user-item bipartite graph (Cui
et al., 2020; Wang et al., 2017; Li et al., 2020; Liu et al., 2020). BiTGCF
(Liu et al., 2020) incorporates graph neural network to a bi-direction dual
transfer learning approach, where each domain has a unique user-item bipartite
graph and the output of each GNN layer is integrated across domains in the
propagation process. (2) heterogeneous graph (Yin et al., 2019; Bi et al.,
2020). HCDIR (Bi et al., 2020) transforms four types of insurance products
with their properties and relations into an insurance graph and utilizes
relational neighbor aggregation to learn embeddings in each domain, which is
fed into an MLP to transfer knowledge. However, all of the above models fail
to incorporate review information into the graph explicitly and just use
documents to generate initial representations. To our best knowledge, M2GNN is
the first attempt to regard the words (tags) as one type of node in GNN-based
CDR.

##  6\. conclusion

In this paper, we aim to improve the review feed recommender system in the
Dianping app. The proposed M2GNN adopts the GNN-based cross-domain framework
to extract content-based interests via tags from other sites like search and
consumption logs to enhance target recommendation. A key contribution is a
novel two-step aggregation mechanism, which can distill intra-domain interests
from a large number of tags, most of which are noises. Both offline and online
experiments demonstrate the effectiveness and explainability of M2GNN. For
future work, we plan to investigate how to filter out unuseful tags via node
sampling technology in GNN.

## References

  * (1)
  * Bi et al. (2020) Ye Bi, Liqiang Song, Mengqiu Yao, Zhenyu Wu, Jianming Wang, and Jing Xiao. 2020\.  A heterogeneous information network based cross domain insurance recommendation system for cold start users. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2211–2220.
  * Chang et al. (2021) Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2021\.  Sequential Recommendation with Graph Neural Networks. In _SIGIR_. 378–387.
  * Cui et al. (2020) Qiang Cui, Tao Wei, Yafeng Zhang, and Qing Zhang. 2020\.  HeroGRAPH: A Heterogeneous Graph Framework for Multi-Target Cross-Domain Recommendation.. In _ORSUM@ RecSys_.
  * Elkahky et al. (2015) Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. 2015.  A multi-view deep learning approach for cross domain user modeling in recommendation systems. In _Proceedings of the 24th international conference on world wide web_. 278–288.
  * Fu et al. (2019) Wenjing Fu, Zhaohui Peng, Senzhang Wang, Yang Xu, and Jin Li. 2019.  Deeply fusing reviews and contents for cold start users in cross-domain recommendation systems. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 94–101.
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020\.  Lightgcn: Simplifying and powering graph convolution network for recommendation. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 639–648.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _Proceedings of the 26th international conference on world wide web_. 173–182.
  * Hu et al. (2021) Fenyu Hu, Yanqiao Zhu, Shu Wu, Weiran Huang, Liang Wang, and Tieniu Tan. 2021\.  Graphair: Graph representation learning with neighborhood aggregation and interaction.  _Pattern Recognition_ 112 (2021), 107745.
  * Hu et al. (2018) Guangneng Hu, Yu Zhang, and Qiang Yang. 2018.  MTNet: a neural approach for cross-domain recommendation with unstructured text.  _KDD Deep Learning Day_ (2018), 1–10.
  * Huang and Stokes (2016) Wenyi Huang and Jack W Stokes. 2016.  MtNet: a multi-task neural network for dynamic malware classification. In _International conference on detection of intrusions and malware, and vulnerability assessment_. Springer, 399–418.
  * Kang et al. (2019) SeongKu Kang, Junyoung Hwang, Dongha Lee, and Hwanjo Yu. 2019\.  Semi-supervised learning for cross-domain recommendation to cold-start users. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 1563–1572.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Li et al. (2019) Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. 2019\.  Multi-interest network with dynamic routing for recommendation at Tmall. In _KDD_. 2615–2623.
  * Li et al. (2020) Jin Li, Zhaohui Peng, Senzhang Wang, Xiaokang Xu, Philip S Yu, and Zhenyun Hao. 2020\.  Heterogeneous Graph Embedding for Cross-Domain Recommendation Through Adversarial Learning. In _International Conference on Database Systems for Advanced Applications_. Springer, 507–522.
  * Liu et al. (2020) Meng Liu, Jianjun Li, Guohui Li, and Peng Pan. 2020\.  Cross domain recommendation via bi-directional transfer graph collaborative filtering networks. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 885–894.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015\.  Effective approaches to attention-based neural machine translation.  _EMNLP_ (2015).
  * Man et al. (2017) Tong Man, Huawei Shen, Xiaolong Jin, and Xueqi Cheng. 2017\.  Cross-domain recommendation: An embedding and mapping approach.. In _IJCAI_ , Vol. 17. 2464–2470.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback.  _arXiv preprint arXiv:1205.2618_ (2012).
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017\.  Dynamic routing between capsules.  _Advances in neural information processing systems_ 30 (2017).
  * Seo et al. (2017) Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu. 2017\.  Interpretable convolutional neural networks with dual local and global attention for review rating prediction. In _Proceedings of the eleventh ACM conference on recommender systems_. 297–305.
  * Singh and Gordon (2008) Ajit P Singh and Geoffrey J Gordon. 2008.  Relational learning via collective matrix factorization. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. 650–658.
  * Song et al. (2017) Tianhang Song, Zhaohui Peng, Senzhang Wang, Wenjing Fu, Xiaoguang Hong, and Philip S Yu. 2017\.  Based cross-domain recommendation through joint tensor factorization. In _International conference on database systems for advanced applications_. Springer, 525–540.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vincent et al. (2010) Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. 2010.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _Journal of machine learning research_ 11, 12 (2010).
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019a.  Kgat: Knowledge graph attention network for recommendation. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 950–958.
  * Wang et al. (2017) Xiang Wang, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2017\.  Item silk road: Recommending items from information domains to social users. In _Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval_. 185–194.
  * Wang et al. (2019b) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019b.  Neural graph collaborative filtering. In _Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval_. 165–174.
  * Wang et al. (2021) Xiang Wang, Tinglin Huang, Dingxian Wang, Yancheng Yuan, Zhenguang Liu, Xiangnan He, and Tat-Seng Chua. 2021.  Learning Intents behind Interactions with Knowledge Graph for Recommendation. In _Proceedings of the Web Conference 2021_. 878–887.
  * Wang et al. (2019c) Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019c.  Heterogeneous graph attention network. In _WWW_. 2022–2032.
  * Wu et al. (2019) Libing Wu, Cong Quan, Chenliang Li, Qian Wang, Bolong Zheng, and Xiangyang Luo. 2019\.  A context-aware user-item representation learning for item recommendation.  _ACM Transactions on Information Systems (TOIS)_ 37, 2 (2019), 1–29.
  * Xia et al. (2019) Haifeng Xia, Zengmao Wang, Bo Du, Lefei Zhang, Shuai Chen, and Gang Chun. 2019\.  Leveraging ratings and reviews with gating mechanism for recommendation. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 1573–1582.
  * Xu et al. (2019) Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. 2019\.  Graph Contextualized Self-Attention Network for Session-based Recommendation.. In _IJCAI_ , Vol. 19. 3940–3946.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015\.  Show, attend and tell: Neural image caption generation with visual attention. In _International conference on machine learning_. PMLR, 2048–2057.
  * Yan et al. (2019) Huan Yan, Xiangning Chen, Chen Gao, Yong Li, and Depeng Jin. 2019.  Deepapf: Deep attentive probabilistic factorization for multi-site video recommendation.  _IJCAI 2019_ (2019).
  * Yin et al. (2019) Jiangyi Yin, Yuchun Guo, and Yishuai Chen. 2019.  Heterogenous Information Network Embedding Based Cross-Domain Recommendation System. In _2019 International Conference on Data Mining Workshops (ICDMW)_. IEEE, 362–369.
  * Zhao et al. (2019) Cheng Zhao, Chenliang Li, and Cong Fu. 2019.  Cross-domain recommendation via preference propagation graphnet. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 2165–2168.
  * Zhao et al. (2020) Cheng Zhao, Chenliang Li, Rong Xiao, Hongbo Deng, and Aixin Sun. 2020.  CATN: Cross-domain recommendation for cold-start users via aspect transfer network. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 229–238.
  * Zheng et al. (2017) Lei Zheng, Vahid Noroozi, and Philip S Yu. 2017.  Joint deep modeling of users and items using reviews for recommendation. In _Proceedings of the tenth ACM international conference on web search and data mining_. 425–434.
  * Zhou et al. (2020) Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2020.  Graph neural networks: A review of methods and applications.  _AI Open_ 1 (2020), 57–81.
  * Zhu et al. (2021) Feng Zhu, Yan Wang, Jun Zhou, Chaochao Chen, Longfei Li, and Guanfeng Liu. 2021\.  A unified framework for cross-domain and cross-system recommendations.  _IEEE Transactions on Knowledge and Data Engineering_ (2021).
  * Zhu et al. (2022) Yongchun Zhu, Zhenwei Tang, Yudan Liu, Fuzhen Zhuang, Ruobing Xie, Xu Zhang, Leyu Lin, and Qing He. 2022\.  Personalized transfer of user preferences for cross-domain recommendation. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_. 1507–1515.