# Multi-Sparse-Domain Collaborative Recommendation via Enhanced Comprehensive
Aspect Preference Learning

Xiaoyun Zhao  School of Computer ScienceSichuan University, China
<EMAIL_ADDRESS>,  Ning Yang  School of Computer ScienceSichuan
University, China<EMAIL_ADDRESS>and  Philip S. Yu  Department of
Computer ScienceUniversity of Illinois at Chicago, USA<EMAIL_ADDRESS>
(2022)

###### Abstract.

Cross-domain recommendation (CDR) has been attracting increasing attention of
researchers for its ability to alleviate the data sparsity problem in
recommender systems. However, the existing single-target or dual-target CDR
methods often suffer from two drawbacks, the assumption of at least one rich
domain and the heavy dependence on domain-invariant preference, which are
impractical in real world where sparsity is ubiquitous and might degrade the
user preference learning. To overcome these issues, we propose a Multi-Sparse-
Domain Collaborative Recommendation (MSDCR) model for multi-target cross-
domain recommendation. Unlike traditional CDR methods, MSDCR treats the
multiple relevant domains as all sparse and can simultaneously improve the
recommendation performance in each domain. We propose a Multi-Domain
Separation Network (MDSN) and a Gated Aspect Preference Enhancement (GAPE)
module for MSDCR to enhance a user’s domain-specific aspect preferences in a
domain by transferring the complementary aspect preferences in other domains,
during which the uniqueness of the domain-specific preference can be preserved
through the adversarial training offered by MDSN and the complementarity can
be adaptively determined by GAPE. Meanwhile, we propose a Multi-Domain
Adaptation Network (MDAN) for MSDCR to capture a user’s domain-invariant
aspect preference. With the integration of the enhanced domain-specific aspect
preference and the domain-invariant aspect preference, MSDCR can reach a
comprehensive understanding of a user’s preference in each sparse domain. At
last, the extensive experiments conducted on real datasets demonstrate the
remarkable superiority of MSDCR over the state-of-the-art single-domain
recommendation models and CDR models.

Cross-Domain Recommendation, Dual-Target CDR, Multi-Target CDR, Transfer
Learning

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
Fifteenth ACM International Conference on Web Search and Data Mining; February
21–25, 2022; Tempe, AZ, USA††booktitle: Proceedings of the Fifteenth ACM
International Conference on Web Search and Data Mining (WSDM ’22), February
21–25, 2022, Tempe, AZ, USA††price: 15.00††doi: 10.1145/3488560.3498381††isbn:
978-1-4503-9132-0/22/02††ccs: Information systems Recommender systems

##  1\. Introduction

Figure 1. Illustration of Multi-Sparse-Domain Collaborative Recommendation.

Cross-domain recommendation (CDR) has been attracting increasing attention of
researchers for its ability to alleviate the data sparsity problem in
recommender systems (Zhu et al., 2021; Wang et al., 2019). The common idea of
most existing CDR methods is to improve the recommendation performance on a
sparse target domain by transferring the information of a relevant source
domain with rich data (Fu et al., 2019; Elkahky et al., 2015; Yuan et al.,
2019; Zhao et al., 2020; Yu et al., 2020), which is called single-target CDR
(Zhu et al., 2019). In real world, however, almost every domain suffers from
data sparsity problem due to its ubiquity, which may cause the existing
single-target CDR methods to fail in finding a dense auxiliary domain to help
the sparse target domain. Recently, dual-target CDR has been proposed to
improve recommendation performances on both domains simultaneously, with dual
transfer learning of common knowledge (i.e., domain-invariant preference)
shared across two domains (Singh and Gordon, 2008; Long et al., 2012; Wang et
al., 2019; Zhu et al., 2019; Zhao et al., 2019; Zhu et al., 2020a; Li and
Tuzhilin, 2020). However, dual-target CDR also assumes that at least one
domain has rich information, which still tends to fail under the situations
where all domains are sparse.

To tackle this dilemma, we propose a novel recommendation model called Multi-
Sparse-Domain Collaborative Recommendation (MSDCR) for multi-target CDR. In
sharp contrast with traditional CDR methods, MSDCR assumes the multiple
relevant domains are all sparse and makes them collaborate with each other so
that their recommendation performance can be improved simultaneously. Unlike
the existing CDR methods which heavily rely on the learning of the domain-
invariant preferences, the main idea of MSDCR is to build a comprehensive
understanding of a user’s preference to aspects (rather than items) in each
domain, by capturing not only the user’s domain-invariant aspect preferences
but also her enhanced domain-specific aspect preferences. Although the sparse
interactions observed in a single domain involves only a small collection of
items and can only reveal partial preferences of a user in that domain, we can
still enhance a user’s domain-specific preference in a domain by transferring
the user’s complementary aspect preferences in other domains. Different from
domain-invariant preferences, the complementary preferences refers to those
revealed by a user’s interactions that are observed only in some domains, but
beneficial to the inference of the user’s potential interest in other domains
where they are not observed. For example, Figure 1 shows a user likes science
fiction movies, fantasy movies and novels, suspense movies and novels, and
martial arts novels. ”Fantasy” and ”Suspense” are the user’s domain-invariant
preferences to type (genre) aspect shared across the book domain and movie
domain, while ”Martial arts” and ”Science fiction, Tom Hanks” are the user’s
aspect preferences specific to the book domain and movie domain, respectively.
Although in the book domain we do not observe the user reading science fiction
novels, we can reasonably infer her potential interest to them because of her
observed preference to science fiction movies in the movie domain. By the same
token, we can also guess the user may like martial arts movies based on her
interactions with martial arts novels in the book domain. However, learning
comprehensive preference for each domain is not easy due to the following
challenges:

  * •

Uniqueness and Commonness A comprehensive view of a user’s preference in each
domain is an integration of two ingredients, the domain-specific preference
embeddings and the domain-invariant preference embeddings. The domain-specific
preference embeddings need to be distinguishable enough to capture the user’s
unique interest specific to each domain for a meaningful complement between
domains. On the contrary, the domain-invariant preference embeddings are
supposed to be as indistinguishable as possible, so that the common
preferences of a user shared across domains can be revealed for the
comprehensive preference learning.

  * •

Different Complementarity The complementarity of different aspect preferences
is different. On the one hand, the preferences to different aspects in the
same domain may have different contributions to the preference learning in
another domain. On the other hand, the aspect preferences in other different
domains may also have unequal importances to the preference learning for the
same domain.

To address the above challenges, (1) we propose a Multi-Domain Separation
Network (MDSN) to enforce the uniqueness of the domain-specific aspect
preferences, which uses an auxiliary domain separation discriminator to
identify the domain label of a domain-specific aspect preference embedding.
The objective of MDSN is to make the domain-specific aspect preference
embeddings learned from different domains to be as separate from each other as
possible, via an adversarial training between the domain-specific aspect
preference embedding and the domain separation discriminator. (2) For the
enhanced domain-specific aspect preference learning, we propose a Gated Aspect
Preference Enhancement (GAPE) module to implicitly model the complementarity
of a user’s latent aspect preferences in different domains. GAPE can generate
gate control vectors to regulate the transfer of the user’s complementary
aspect preferences, by which MSDCR can enhance a user’s domain-specific aspect
preference in each domain. (3) To obtain a user’s comprehensive preference in
a domain, we also propose a Multi-Domain Adaptation Network (MDAN) for the
domain-invariant aspect preference learning, which applies domain adaptation
(Yu et al., 2020) to capture a user’s preferences common to different domains.
(4) Finally, we introduce a multi-task framework to MSDCR, by which MSDCR can
be trained under the collaborative supervisions of multiple relevant domains
and make multi-target recommendations for a user based on the user’s learned
comprehensive preference in each domain. The main contributions of this paper
can be summarized as follows:

  * •

We propose a novel recommendation model called Multi-Sparse-Domain
Collaborative Recommendation (MSDCR). Different from traditional CDR methods,
MSDCR treats the multiple relevant domains as all sparse and can
simultaneously improve the recommendation performance in each domain with
comprehensive aspect preference learning which considers both the domain-
specific preference and the domain-invariant preference of a user.

  * •

We propose a Multi-Domain Separation Network (MDSN) and a Gated Aspect
Preference Enhancement (GAPE) module for MSDCR to enhance a user’s domain-
specific aspect preferences in a domain by transferring the complementary
aspect preferences in other domains, during which the uniqueness of the
domain-specific preference can be preserved through the adversarial training
offered by MDSN and the complementarity can be adaptively determined by GAPE.

  * •

We propose a Multi-Domain Adaptation Network (MDAN) for MSDCR to capture the
domain-invariant aspect preferences, which together with the enhanced domain-
specific aspect preferences form a comprehensive view of a user’s preference
in each domain.

  * •

We conduct extensive experiments on real datasets from three relevant domains.
The experimental results verify the feasibility and the effectiveness of
MSDCR.

Figure 2. The Architecture of MSDCR.

##  2\. Problem Definition

We consider $S$ relevant domains, and let $\mathcal{D}^{(s)}$ denote the $s$th
domain ($1\leq s\leq S$). Let $x^{(s)}$ represent an item in
$\mathcal{D}^{(s)}$, associated with a raw feature vector
$\mathbf{x}^{(s)}\in\mathbb{R}^{d_{s}}$, where $d_{s}$ is the dimensionality
of the domain-specific features of $\mathcal{D}^{(s)}$. We use a one-hot
vector $\mathbf{u}\in\mathbb{R}^{N}$ to represent a user $u$ ($1\leq u\leq
N$), where $N$ is the number of users shared across domains. Let
$\mathcal{X}_{u}^{(s)}=\\{\mathbf{x}_{u,1}^{(s)},\cdots,\mathbf{x}_{u,n_{u}^{(s)}}^{(s)}\\}$
be user $u$’s collection of $n_{u}^{(s)}$ interactions with items in domain
$\mathcal{D}^{(s)}$, where $\mathbf{x}_{u,i}^{(s)}\in\mathcal{D}^{(s)}$
($1\leq i\leq n_{u}^{(s)}$) represents $u$’s $i$th interaction (item) in
$\mathcal{D}^{(s)}$. Then the target problem of this paper is to learn a
function $f^{(s)}$ for each domain $\mathcal{D}^{(s)}$ that can predict the
probability $\hat{r}_{u,x^{(s)}}$ of user $u$ will interact with an item
$x^{(s)}\in\mathcal{D}^{(s)}\setminus\mathcal{X}_{u}^{(s)}$, given $u$’s
historical interaction collections in $S$ relevant domains
$\mathcal{X}_{u}^{(1)},\dots,\mathcal{X}_{u}^{(S)}$.

##  3\. The Proposed Model

###  3.1. Overview

Figure 2 shows the architecture of MSDCR, which comprises an embedding layer,
a domain aspect preference learning layer, and a prediction layer. At the item
embedding layer, MSDCR first transforms the interactions
$\\{\mathbf{x}_{u,i}^{(s)}$, $1\leq i\leq n_{u}^{(s)}\\}$ in each domain
$\mathcal{D}^{(s)}$ ($1\leq s\leq S$) to their corresponding latent
representations $\\{\mathbf{z}_{u,i}^{(s)}\in\mathbb{R}^{d}\\}$, using a
domain-specific item embedding module for $\mathcal{D}^{(s)}$, where $d$ is
the embedding dimensionality. To capture the nonlinear relations between the
interactions in a domain, MSDCR will further apply a domain-specific self-
attention mechanism to $\\{\mathbf{z}_{u,i}^{(s)}\\}$ to generate the
attentional item embeddings $\\{\mathbf{h}_{u,i}^{(s)}\in\mathbb{R}^{d}\\}$.
At the same time, MSDCR will also generate the user’s embedding
$\mathbf{p}_{u}\in\mathbb{R}^{d}$ at the embedding layer.

At the domain aspect preference learning layer, with the attentional item
embeddings and user embedding as input, MSDCR applies the Domain-Specific
Aspect Preference Encoder (DSAPE) to generate a domain-specific latent aspect
preference embedding matrix
${\mathbf{A}}_{u}^{(s)}=[{\mathbf{a}}_{u,m}^{(s)}]_{m=1}^{M}$
$\in\mathbb{R}^{d\times M}$ for each domain, where $M$ is the number of latent
aspects and each column vector ${\mathbf{a}}_{u,m}^{(s)}\in\mathbb{R}^{d}$
represents $u$’s domain-specific preference embedding of $m$th latent aspect
of domain $\mathcal{D}^{(s)}$. Each $\mathbf{a}_{u,m}^{(s)}$ represents an
enhanced view of the user $u$’s latent aspect preference specific to domain
$\mathcal{D}^{(s)}$, which fuses with the complementary information provided
by $u$’s aspect preferences in other domains via the GAPE and keeps unique via
the MDSN. At the same time, by using the MDAN, the Domain-Invariant Aspect
Preference Encoder (DIAPE) will generate a domain-invariant latent aspect
preference embedding matrix
$\mathbf{C}_{u}^{(s)}=[\mathbf{c}_{u,m}^{(s)}]_{m=1}^{M}\in\mathbb{R}^{d\times
M}$ for each domain $\mathcal{D}^{(s)}$, where the column vector
$\mathbf{c}_{u,m}^{(s)}\in\mathbb{R}^{d}$ represents $u$’s domain-invariant
preference embedding of $m$th latent aspect in domain $\mathcal{D}^{(s)}$.
$\mathbf{A}_{u}^{(s)}$, together with ${\mathbf{C}}_{u}^{(s)}$, forms the
comprehensive view of user $u$’s preference in domain $\mathcal{D}^{(s)}$.

At last, MSDCR will employs a multi-task framework to make the prediction of
the probability $\hat{r}_{u,x^{(s)}}$ that $u$ will interact with an item
$x^{(s)}\in\mathcal{D}^{(s)}\setminus\mathcal{X}_{u}^{(s)}$, by feeding the
item embedding $\mathbf{x}^{(s)}$, the domain-specific aspect preference
embeddings ${\mathbf{A}}_{u}^{(s)}$, and the domain-invariant aspect
preference embeddings ${\mathbf{C}}_{u}^{(s)}$ into a Multi-Layer Perceptron
(MLP) $f^{(s)}$ which is specific to domain $\mathcal{D}^{(s)}$.

###  3.2. Item and User Emebedding

####  3.2.1. Item Embedding

Given an item represented by original feature vector
$\mathbf{x}_{u,i}^{(s)}\in\mathbb{R}^{d_{s}}$, we first transform it to an
embedding $\mathbf{z}_{u,i}^{(s)}\in\mathbb{R}^{d}$ with a domain-specific
mapping, $\mathbf{z}_{u,i}^{(s)}=\mathbf{W}^{(s)}\mathbf{x}_{u,i}^{(s)}$,
where $\mathbf{W}^{(s)}\in\mathbb{R}^{d_{s}\times d}$ is the learnable mapping
matrix for domain $\mathcal{D}^{(s)}$ ($1\leq s\leq S$). To capture the
nonlinear relationships between the interactions, next we apply a domain-
specific self-attention mechanism to the item embeddings to generate
attentional item embeddings. To simplify the expression, we horizontally
assemble the item embeddings of user $u$ in domain $\mathcal{D}^{(s)}$ into
the embedding matrix
$\mathbf{Z}_{u}^{(s)}=[\mathbf{z}_{u,i}^{(s)}]_{i=1}^{n_{u}^{(s)}}\in\mathbb{R}^{d\times
n_{u}^{(s)}}$, where $\mathbf{z}_{u,i}^{(s)}$ is the $i$th column and
$n_{u}^{(s)}$ is the number of items the user interacted with in
$\mathcal{D}^{(s)}$. Similarly, let
$\mathbf{H}_{u}^{(s)}=[\mathbf{h}_{u,i}^{(s)}]_{i=1}^{n_{u}^{(s)}}\in\mathbb{R}^{d\times
n_{u}^{(s)}}$ be the attentional item embedding matrix, where $i$th column
$\mathbf{h}_{u,i}^{(s)}$ is the attentional embedding of
$\mathbf{z}_{u,i}^{(s)}$. For each item embedding, we first generate its query
vector $\mathbf{q}_{u,i}^{(s)}\in\mathbb{R}^{d}$, key vector
$\mathbf{k}_{u,i}^{(s)}\in\mathbb{R}^{d}$, and value vector
$\mathbf{v}_{u,i}^{(s)}\in\mathbb{R}^{d}$, with the following transformations:

(1)
$\mathbf{Q}_{u}^{(s)}=\mathbf{W}_{\text{q}}^{(s)}\mathbf{Z}_{u}^{(s)},\text{
}\mathbf{K}_{u}^{(s)}=\mathbf{W}_{\text{k}}^{(s)}\mathbf{Z}_{u}^{(s)},\text{
}\mathbf{V}_{u}^{(s)}=\mathbf{W}_{\text{v}}^{(s)}\mathbf{Z}_{u}^{(s)},$

where $\mathbf{Q}_{u}^{(s)}=[\mathbf{q}_{u,i}^{(s)}]_{i=1}^{n_{u}^{(s)}}$,
$\mathbf{K}_{u}^{(s)}=[\mathbf{k}_{u,i}^{(s)}]_{i=1}^{n_{u}^{(s)}}$,
$\mathbf{V}_{u}^{(s)}=[\mathbf{v}_{u,i}^{(s)}]_{i=1}^{n_{u}^{(s)}}\in\mathbb{R}^{d\times
n_{u}^{(s)}}$, and $\mathbf{W}_{\text{q}}^{(s)}$,
$\mathbf{W}_{\text{k}}^{(s)}$,
$\mathbf{W}_{\text{v}}^{(s)}\in\mathbb{R}^{d\times d}$ are learnable
projection matrices. Then $\mathbf{H}_{u}^{(s)}$ can be obtained by

(2)
$\mathbf{H}_{u}^{(s)}=\text{softmax}\bigg{(}\frac{\mathbf{Q}_{u}^{(s)}{\mathbf{K}_{u}^{(s)}}^{T}}{\sqrt{d}}\bigg{)}\mathbf{V}_{u}^{(s)}.$

####  3.2.2. User Embedding

A user $u$’s embedding $\mathbf{p}_{u}\in\mathbb{R}^{d}$ will be obtained with
a lookup over a learnable embedding matrix
$\mathbf{W}_{\text{u}}\in\mathbb{R}^{d\times N}$, i.e.,
$\mathbf{p}_{u}=\mathbf{W}_{\text{u}}\mathbf{u}$, where $N$ is the number of
common users and $\mathbf{u}\in\mathbb{R}^{N}$ is a one-hot vector
representing user $u$.

###  3.3. Domain-Specific Aspect Preference Encoder (DSAPE)

DSAPE is the crucial component of MSDCR, which generates the enhanced domain-
specific aspect preference embedding matrix $\mathbf{A}_{u}^{(s)}$ for each
domain $\mathcal{D}^{(s)}$ with the help of the novel MDSN and GAPE.

####  3.3.1. Domain-Specific Latent Aspect Preference Embedding

First, based on the attentional item embedding $\mathbf{h}^{(s)}_{u,i}$, we
generate the $m$th latent aspect embedding of user $u$’s $i$th interaction
item in domain $\mathcal{D}^{(s)}$,
$\tilde{\mathbf{e}}_{u,i,m}^{(s)}\in\mathbb{R}^{d}$, as follow

(3)
$\tilde{\mathbf{e}}_{u,i,m}^{(s)}=\widetilde{\mathbf{W}}_{\text{a}}^{(m)}(\mathbf{h}_{u,i}^{(s)}\oplus\mathbf{p}_{u})+\tilde{\mathbf{b}}_{\text{a}}^{(m)},$

where $\sigma$ is sigmoid function, $\oplus$ represents concatenation,
$\widetilde{\mathbf{W}}_{\text{a}}^{(m)}\in\mathbb{R}^{d\times 2d}$ and
$\tilde{\mathbf{b}}_{\text{a}}^{(m)}\in\mathbb{R}^{d}$ are learnable
parameters. It is noteworthy that the latent aspect embeddings of items in
different domains lie in the same latent aspect space, as the projection
matrices $\\{\widetilde{\mathbf{W}}_{\text{a}}^{(m)}\\}$ and the bias terms
$\\{\tilde{\mathbf{b}}_{\text{a}}^{(m)}\\}$ ($1\leq m\leq M$) are shared
across domains.

Intuitively, a user’s preference to a latent aspect of domain
$\mathcal{D}^{(s)}$ is revealed by the latent aspect embeddings of the
$n_{u}^{(s)}$ items she interacts with in $\mathcal{D}^{(s)}$. Therefore, we
can regard the user $u$’s $m$th aspect preference embedding in domain
$\mathcal{D}^{(s)}$, $\tilde{\mathbf{a}}^{(s)}_{u,m}\in\mathbb{R}^{d}$, as a
composition of all the $m$th latent aspect embeddings
$\\{\tilde{\mathbf{e}}_{u,i,m}^{(s)}\\}$ ($1\leq i\leq n_{u}^{(s)}$). In the
light of this idea, $\tilde{\mathbf{a}}^{(s)}_{u,m}$ can be generated with the
following attention mechanism,

(4)
$\tilde{\mathbf{a}}^{(s)}_{u,m}=\sum_{i=1}^{n_{u}^{(s)}}\tilde{\alpha}_{u,i,m}^{(s)}\tilde{\mathbf{e}}_{u,i,m}^{(s)},\
\tilde{\alpha}_{u,i,m}^{(s)}=\frac{\exp\big{(}(\tilde{\mathbf{e}}_{u,i,m}^{(s)})^{T}\tilde{\gamma}^{(m)}\big{)}}{\sum_{j=1}^{n_{u}^{(s)}}\exp\big{(}(\tilde{\mathbf{e}}_{u,j,m}^{(s)})^{T}\tilde{\gamma}^{(m)}\big{)}},$

where $\tilde{\gamma}^{(m)}\in\mathbb{R}^{d}$ is the learnable query vector
for $m$th latent aspect. By horizontally assembling the user latent aspect
embeddings as columns, we can obtain user $u$’s domain-specific aspect
preference embedding matrix
$\widetilde{\mathbf{A}}_{u}^{(s)}=[\tilde{\mathbf{a}}^{(s)}_{u,m}]_{m=1}^{M}\in\mathbb{R}^{d\times
M}$.

####  3.3.2. Multi-Domain Separation Network (MDSN)

In order to avoid the homogenization of a user’s domain-specific preferences,
which impedes the extraction of complementary information in different
domains, the objective of DSAPE is to learn the user’s aspect preferences that
are unique to each domain. For this purpose, inspired by the idea in (Tsai and
Chien, 2017), we introduce a domain separation discriminator $\psi$ together
with DSAPE to form a multi-domain separation network. The domain separation
discriminator is implemented as an MLP taking
$\widetilde{\mathbf{A}}_{u}^{(s)}$ as input and using a softmax layer to
output an $S$-dimensional vector
$\hat{\mathbf{y}}_{u,s}=\psi(\tilde{\mathbf{a}}_{u}^{(s)})$
($\tilde{\mathbf{a}}_{u}^{(s)}=\tilde{\mathbf{a}}^{(s)}_{u,1}\oplus\cdots\oplus\tilde{\mathbf{a}}^{(s)}_{u,M}$),
of which the $i$th component $\hat{y}_{u,s}^{(i)}\in[0,1]$ represents the
predicted probability that $\widetilde{\mathbf{A}}_{u}^{(s)}$ comes from
domain $\mathcal{D}^{(i)}$. Let $\boldsymbol{\Theta}_{\text{ds}}$ and
$\boldsymbol{\Theta}_{\uppsi}$ represent the learnable parameters of DSAPE and
the domain separation discriminator, respectively, and the domain label
prediction loss can be defined as

(5)
$\mathcal{L}_{\text{ds}}(\boldsymbol{\Theta}_{\text{ds}},\boldsymbol{\Theta}_{\uppsi})=-\sum_{u=1}^{N}\sum_{s=1}^{S}\sum_{i=1}^{S}y_{u,s}^{(i)}\log\hat{y}_{u,s}^{(i)},$

where the ground-truth $y_{u,s}^{(i)}=1$ if $i=s$, otherwise
$y_{u,s}^{(i)}=0$. Then we define the following minimax optimization objective
for multi-domain separation:

(6)
$\min_{\boldsymbol{\Theta}_{\text{DS}}}\max_{\boldsymbol{\Theta}_{\uppsi}}\mathcal{L}_{\text{ds}}.$

Note that the optimization objective of the discriminator $\psi$ is to weaken
its discriminating ability by maximizing the prediction loss
($\mathcal{L}_{\text{ds}}$). Here the insight is that by adversarially
training the DSAPE with the objective of minimizing the prediction loss, the
parameters of DSAPE can be adjusted so that the generated domain-specific
preference matrices $\\{\widetilde{\mathbf{A}}_{u}^{(s)}\\}$ ($1\leq s\leq S$)
are distinguishable enough even for a poor discriminator $\psi$.

####  3.3.3. Gated Aspect Preference Enhancement (GAPE)

The domain-specific aspect preference embedding matrices we have learned so
far can only uncover partial truth about a user’s preference in a domain due
to the sparse observations. To overcome this issue, the main novelty of the
proposed method is to build an enhanced view of the user’s preferences in a
domain $\mathcal{D}^{(s)}$, such that we can transfer the complementary aspect
preferences in other domains $\\{\mathcal{D}^{(s^{\prime})}\\}$
($s^{\prime}\neq s$). For this purpose, we first introduce an enhancement gate
control matrix
$\mathbf{G}_{u}^{(s,s^{\prime})}=[\mathbf{g}_{u,m}^{(s,s^{\prime})}]_{m=1}^{M}\in\mathbb{R}^{d\times
M}$ to regulate the transfer from $\mathcal{D}^{(s^{\prime})}$ to
$\mathcal{D}^{(s)}$, where the $m$th column
$\mathbf{g}_{u,m}^{(s,s^{\prime})}\in\mathbb{R}^{d}$ is the gate control
vector representing the complementarity of the $m$th latent aspect preference
in $\mathcal{D}^{(s^{\prime})}$ to $\mathcal{D}^{(s)}$. Inspired by the idea
in (Zhao et al., 2020), we compute the gate control matrix as

(7)
$\mathbf{G}_{u}^{(s,s^{\prime})}=\sigma\bigg{(}\mathbf{W}_{\text{g}}^{(s^{\prime})}\big{[}(\widetilde{\mathbf{A}}_{u}^{(s)}\odot\widetilde{\mathbf{A}}_{u}^{(s^{\prime})})\oplus(\widetilde{\mathbf{A}}_{u}^{(s)}-\widetilde{\mathbf{A}}_{u}^{(s^{\prime})})\big{]}+\mathbf{B}_{\text{g}}^{(s^{\prime})}\bigg{)},$

where $\mathbf{W}_{\text{g}}^{(s^{\prime})}\in\mathbb{R}^{d\times 2d}$ and
$\mathbf{B}_{\text{g}}^{(s^{\prime})}\in\mathbb{R}^{d\times M}$ are learnable
projection matrix and bias matrix, respectively, and $\odot$ represents
element-wise product.

What does $\mathbf{G}_{u}^{(s,s^{\prime})}$ do for us? Basically,
$\mathbf{G}_{u}^{(s,s^{\prime})}$ implicitly models the complementarity of
user aspect preferences from two angles, the feasibility and the
meaningfulness. At first, we can see that the larger the element-wise product
term
$\widetilde{\mathbf{A}}_{u}^{(s)}\odot\widetilde{\mathbf{A}}_{u}^{(s^{\prime})}$
and the difference term
$\widetilde{\mathbf{A}}_{u}^{(s)}-\widetilde{\mathbf{A}}_{u}^{(s^{\prime})}$,
the larger the gate control matrix $\mathbf{G}_{u}^{(s,s^{\prime})}$ and the
more the complementarity of a user $u$’s latent aspect preferences in
$\mathcal{D}^{(s^{\prime})}$ to those in $\mathcal{D}^{(s)}$. The insight here
is that a large
$\widetilde{\mathbf{A}}_{u}^{(s)}\odot\widetilde{\mathbf{A}}_{u}^{(s^{\prime})}$
implies both $\tilde{\mathbf{a}}_{u,m}^{(s)}$ and
$\tilde{\mathbf{a}}_{u,m}^{(s^{\prime})}$ are far away from zero vector, i.e.,
the user $u$’s $m$th latent aspect preferences in $\mathcal{D}^{(s)}$ and
$\mathcal{D}^{(s^{\prime})}$ are both valid (e.g., $u$ cares about the
category aspect in both book and movie domains), which makes the complementing
to be feasible. At the same time, a large difference
$\widetilde{\mathbf{A}}_{u}^{(s)}-\widetilde{\mathbf{A}}_{u}^{(s^{\prime})}$
means the user preferences to the same aspect in the two domains are different
(e.g., $u$ likes category ”martial arts” in book domain but category ”science
fiction” in movie domain), which makes the complementing to be meaningful.

At last, the enhanced aspect preference embedding matrix
$\mathbf{A}_{u}^{(s)}=[\mathbf{a}^{(s)}_{u,m}]_{m=1}^{M}\in\mathbb{R}^{d\times
M}$ specific to domain $\mathcal{D}^{(s)}$ can be obtained with the following
gated fusion of the aspect preferences in other domains:

(8)
$\displaystyle\mathbf{A}_{u}^{(s)}=\sigma\bigg{(}\mathbf{W}_{\text{e}}^{(s)}\big{[}(\mathbf{G}_{u}^{(s,1)}\odot\widetilde{\mathbf{A}}_{u}^{(1)})\oplus\cdots\oplus(\mathbf{G}_{u}^{(s,s-1)}\odot\widetilde{\mathbf{A}}_{u}^{(s-1)})$
$\displaystyle\oplus\widetilde{\mathbf{A}}_{u}^{(s)}\oplus(\mathbf{G}_{u}^{(s,s+1)}\odot\widetilde{\mathbf{A}}_{u}^{(s+1)})\oplus\cdots\oplus(\mathbf{G}_{u}^{(s,S)}\odot\widetilde{\mathbf{A}}_{u}^{(S)})\big{]}+\mathbf{B}_{\text{e}}^{(s)}\bigg{)},$

where $\mathbf{a}^{(s)}_{u,m}\in\mathbb{R}^{d}$ is the user $u$’s enhanced
$m$th aspect preference embedding in domain $\mathcal{D}^{(s)}$,
$\mathbf{W}_{\text{e}}^{(s)}\in\mathbb{R}^{d\times Sd}$ and
$\mathbf{B}_{\text{e}}^{(s)}\in\mathbb{R}^{d\times M}$ are learnable
projection matrix and bias matrix, respectively.

###  3.4. Domain-Invariant Aspect Preference Encoder (DIAPE)

The challenge for the domain-invariant preference learning is that a user’s
common preferences revealed by different domains are similar but still
slightly different. For example, a user may like suspense novel and movie with
different intension. Therefore it may be inappropriate to generate the common
preference by manually summing over the domain-specific ones as does the
existing method (Liu et al., 2020). To address this issue, we propose a Multi-
Domain Adaptation Network (MDAN) for DIAPE to learn a domain-invariant
preference embedding matrix for each domain.

####  3.4.1. Domain-Invariant Latent Aspect Preference Embedding

The goal of DIAPE is to extract a user’s domain-invariant latent aspect
preference from each domain. Similar to DSAPE, we first generate the item
latent aspect embeddings $\bar{\mathbf{e}}_{u,i,m}^{(s)}\in\mathbb{R}^{d}$ for
domain-invariant preference learning,

(9) $\bar{\mathbf{e}}_{u,i,m}^{(s)}=\hbox{\vbox{\hrule height=0.5pt\kern
2.15277pt\hbox{\kern-1.00006pt$\mathbf{W}$\kern-1.00006pt}}}_{\text{a}}^{(m)}(\mathbf{h}_{u,i}^{(s)}\oplus\mathbf{p}_{u})+\bar{\mathbf{b}}_{\text{a}}^{(m)},$

where $\hbox{\vbox{\hrule height=0.5pt\kern
2.15277pt\hbox{\kern-1.00006pt$\mathbf{W}$\kern-1.00006pt}}}_{\text{a}}^{(m)}$$\in\mathbb{R}^{d\times
2d}$ and $\bar{\mathbf{b}}_{\text{a}}^{(m)}$$\in\mathbb{R}^{d}$ are learnable
parameters, and $1\leq u\leq N$, $1\leq i\leq n_{u}^{(s)}$, $1\leq m\leq M$.
Then we extract the domain-invariant latent aspect preference embedding
$\mathbf{c}^{(s)}_{u,m}\in\mathbb{R}^{d}$ with the similar attention mechanism
used in DSAPE (Equation (4)). At last, by horizontal assembly of the
embeddings, we can obtain the domain preference matrix
$\mathbf{C}_{u}^{(s)}=[\mathbf{c}^{(s)}_{u,m}]_{m=1}^{M}\in\mathbb{R}^{d\times
M}$.

####  3.4.2. Multi-Domain Adaptation Network (MDAN)

To ensure $\mathbf{C}_{u}^{(s)}$ to be domain-invariant, we also introduce an
auxiliary domain adaptation discriminator $\phi$, which together with the
domain-invariant latent aspect preference embedding forms a domain adaptation
network. The loss of the domain adaptation network is defined as

(10)
$\mathcal{L}_{\text{da}}(\boldsymbol{\Theta}_{\text{da}},\boldsymbol{\Theta}_{\upphi})=-\sum_{u=1}^{N}\sum_{s=1}^{S}\sum_{i=1}^{S}z_{u,s}^{(i)}\log\hat{z}_{u,s}^{(i)},$

where $\boldsymbol{\Theta}_{\text{da}}$ and $\boldsymbol{\Theta}_{\upphi}$ are
the parameters of DIAPE and the domain adaptation discriminator, respectively,
$\hat{z}_{u,s}^{(i)}$ is the $i$th component of the softmax output
$\phi(\mathbf{c}_{u}^{(s)})\in\mathbb{R}^{S}$
($\mathbf{c}_{u}^{(s)}=\mathbf{c}^{(s)}_{u,1}\oplus\cdots\oplus\mathbf{c}^{(s)}_{u,M}$),
and $z_{u,s}^{(i)}$ is the ground-truth such that $z_{u,s}^{(i)}=1$ if $i=s$,
otherwise $z_{u,s}^{(i)}=0$. Note that the role of the domain adaptation
discriminator $\phi$ is to help DIAPE generate indistinguishable preference
matrices $\\{\mathbf{C}_{u}^{(s)}\\}$. Therefore, the optimization objective
of the multi-domain adaptation network is defined as the following minimax
game,

(11)
$\min_{\boldsymbol{\Theta}_{\text{DI}}}\max_{\boldsymbol{\Theta}_{\upphi}}-\mathcal{L}_{\text{da}}.$

It is noteworthy that the adversarial optimization objective of the domain
adaptation discriminator is different from that of the domain separation
discriminator. In Equation (11), we want to train the domain adaptation
discriminator to maximize the negative cross entropy
(-$\mathcal{L}_{\text{da}}$), which equivalently minimizes the cross entropy
to yield a strong discriminator, while in Equation (6), we want to maximize
the cross entropy ($\mathcal{L}_{\text{ds}}$). Therefore, the adversarial
optimization of Equation (11) results in a good DIAPE able to generate
$\\{\mathbf{C}_{u}^{(s)}\\}$ that are indistinguishable enough to fool a
strong domain adaptation discriminator.

###  3.5. Interaction Prediction

For each domain $\mathcal{D}^{(s)}$, we use an MLP $f^{(s)}$ with a sigmoid
function as output to predict the probability that a user $u$ will interact
with an item $x^{(s)}$,

(12)
$\hat{r}_{u,x^{(s)}}=f^{(s)}(\mathbf{x}^{(s)},\mathbf{a}_{u}^{(s)},\mathbf{c}_{u}^{(s)};\boldsymbol{\Theta_{\text{f}}^{(s)}}),$

where $\boldsymbol{\Theta_{\text{f}}^{(s)}}$ represents the learnable
parameters, $\mathbf{x}^{(s)}$ is the original feature vector of the item,
$\mathbf{a}_{u}^{(s)}=\mathbf{a}^{(s)}_{u,1}\oplus\cdots\oplus\mathbf{a}^{(s)}_{u,M}$
represents $u$’s enhanced domain-specific preference in domain
$\mathcal{D}^{(s)}$, and
$\mathbf{c}_{u}^{(s)}=\mathbf{c}^{(s)}_{u,1}\oplus\cdots\oplus\mathbf{c}^{(s)}_{u,M}$
represents $u$’s domain-invariant preference revealed by domain
$\mathcal{D}^{(s)}$.

###  3.6. Model Training

For each domain $\mathcal{D}^{(s)}$, we build a training set
$\mathcal{T}^{(s)}=\\{(u,x_{+},x_{-})\\}$, where $x_{+}\in\mathcal{D}^{(s)}$
and $x_{-}\in\mathcal{D}^{(s)}$ represent a positive sample and a negative
sample of a user $u$, respectively. By applying the popular pair-wise ranking
loss BPR (Rendle et al., 2012), we define the following objective function for
a single domain $\mathcal{D}^{(s)}$,

(13)
$\mathcal{L}_{\text{f}}^{(s)}(\boldsymbol{\Theta_{\text{f}}^{(s)}})=\frac{1}{|\mathcal{T}^{(s)}|}\sum_{(u,x_{+},x_{-})\in\mathcal{T}^{(s)}}\log(\hat{r}_{u,x_{+}}-\hat{r}_{u,x_{-}}).$

Then the total prediction loss over $S$ domains is

(14)
$\mathcal{L}_{\text{f}}(\boldsymbol{\Theta_{\text{f}}})=\frac{1}{S}\sum_{s=1}^{S}\mathcal{L}_{\text{f}}^{(s)},$

where
$\boldsymbol{\Theta_{\text{f}}}=\\{\boldsymbol{\Theta_{\text{f}}^{(1)}},\cdots,\boldsymbol{\Theta_{\text{f}}^{(S)}}\\}$.
At last, by combining $\mathcal{L}_{\text{ds}}$, $\mathcal{L}_{\text{da}}$,
and $\mathcal{L}_{\text{f}}$, we can get the following overall optimization
objective

(15)
$\min_{\boldsymbol{\Theta_{\text{ds}}},\boldsymbol{\Theta_{\text{da}}},\boldsymbol{\Theta_{\text{f}}}}\max_{\boldsymbol{\Theta_{\uppsi}},\boldsymbol{\Theta_{\upphi}}}\mathcal{L}_{\text{ds}}+\mathcal{L}_{\text{da}}+\mathcal{L}_{\text{f}}+\lambda\lVert\boldsymbol{\Theta}\rVert_{2},$

where $\boldsymbol{\Theta}$ represents all trainable parameters and $\lambda$
is a factor to control the contribution of the regularization term. We will
apply Adam as the optimizer for the iterative training of MSDCR, where the
parameters will be updated alternately. It is noteworthy that MSDCR will be
trained under the multi-task framework, so that the training will benefit from
the collaboration of the supervision signals from multiple relevant domains.

Domain | Movie | Book | Music
---|---|---|---
# Common Users | 800
# Items | 154,886 | 165,461 | 166,447
# Interactions | 93,074 | 29,781 | 30,487
Sparsity | 0.075% | 0.022% | 0.023%
Table 1. Statistics of datasets Domain | Feature | Dimensionality | Type
---|---|---|---
| Director | $\sim 10^{3}$ | one-hot
| Writer | $\sim 10^{4}$ | multi-hot
| Actor | $\sim 10^{4}$ | multi-hot
Movie | Type | 20 | one-hot
| Country | $\sim 10^{2}$ | one-hot
| Language | $\sim 10^{2}$ | one-hot
| Level | 4 | one-hot
| Writer | $\sim 10^{4}$ | one-hot
Book | Publish | $\sim 10^{3}$ | one-hot
| Translator | $\sim 10^{3}$ | one-hot
| Level | 4 | one-hot
| Player | $\sim 10^{4}$ | one-hot
| Genre | $\sim 10^{2}$ | one-hot
Music | Album | $\sim 10^{4}$ | one-hot
| Media | 4 | one-hot
| Publisher | $\sim 10^{3}$ | one-hot
| Level | 4 | one-hot
Table 2. Statistics of features of each domain

##  4\. Experiments

###  4.1. Experimental Setting

####  4.1.1. Datasets

We collect three datasets corresponding to the Movie, Book, and Music domains
of Douban website 111https://www.douban.com. Table 1 shows the statistics of
the three domains, where sparsity is defined as the ratio of the observed
interactions over all possible interactions. By different combinations of the
domains, we build three dual-domain scenarios, Movie-Book, Movie-Music, and
Book-Music, and one triple-domain scenario Movie-Book-Music. In each scenario,
we will check the recommendation performance for the overlapped users in each
domain. The features of each domain are summarized in Table 2. In each
scenario, we randomly select 70% of the data as training set, 10% as
validation set, and the remaining 20% as testing set. We will repeat such
procedure five times and report the average results.

####  4.1.2. Baseline Methods

We compare our MSDCR with eight state-of-the-art recommendation methods,
including two single-domain models (NCF, AMCF), two single-target CDR models
(MV-DNN, CCCFNet), and four dual-target CDR models (CoNet, DDTCDR, GA-DTCDR,
and BiTGCF), which are briefly described as follows:

  * •

NCF (He et al., 2017) NCF is a deep learning based model for singe-domain
recommendation, which employs an MLP to model nonlinear interactions between
user and item embeddings.

  * •

AMCF (Pan et al., 2020) AMCF is another single-domain recommendation model,
which decomposes item features into multiple aspect embeddings to capture
fine-grained user preference.

  * •

MV-DNN (Elkahky et al., 2015) MV-DNN is a single-target CDR model that can
learn domain-invariant user preference by multi-view learning over two
domains.

  * •

CCCFNet (Lian et al., 2017) CCCFNet is a content-based single-target CDR
model, which combines matrix factorization and collaborative filtering for
domain-invariant representation learning.

  * •

CoNet (Hu et al., 2018) CoNet can transfer dual knowledge across two domains
via the cross connections between them, which improves the recommendation
performance on both domains.

  * •

GA-DTCDR (Zhu et al., 2020b) GA-DTCDR is a dual-target model that leverages
the data of dual domains, transfers the preference of common users across
domains, and makes recommendations on both domains.

  * •

DDTCDR (Li and Tuzhilin, 2020) DDTCDR is also a dual-target CDR model, which
realizes the dual knowledge transfer between two domains, based on a cross-
domain preference mapping with orthogonal constraints.

  * •

BiTGCF (Liu et al., 2020) BiTGCF is a GCN based dual-target CDR model, which
can improve the recommendation performance of both domains simultaneously
through a bidirectional knowledge transfer between the two domains.

####  4.1.3. Metrics and Hyper-parameter Setting

In this paper, we use Hit Ratio (HR) and Normalized Discounted Cumulative Gain
(NDCG) as the metrics, which are widely adopted by the baseline methods. The
HR@$k$ is the ratio of the ranking lists where the testing item is ranked in
the first $k$ positions, while the NDCG@$k$ further accounts for the position
of the hit by assigning higher weight to hits at higher positions. We also
adopt the popular leave-one-out testing strategy (He et al., 2017; Li and
Tuzhilin, 2020), which ranks a positive item of a testing user among her 99
negative items which are randomly sampled.

The hyper-parameters are tuned on validation sets. We set the embedding
dimensionality and the number of latent aspects $(d,M)$ as $(128,5)$,
$(32,7)$, and $(16,8)$ for scenarios Movie-Music, Movie-Book, and Book-Music,
respectively. The MLP used for the prediction function $f^{(s)}$ consists of 2
hidden layers, with 256 and 128 neurons, respectively. The MLPs used for the
domain separation discriminator and the domain adaptation discriminator both
have 1 hidden layer with 128 neurons. For fairness, the hyper-parameters of
the baseline methods are set to their optimal configuration tuned on
validation sets.

Metric | NDCG@5 | NDCG@10 | HR@5 | HR@10
---|---|---|---|---
Domain | Movie | Music | Movie | Music | Movie | Music | Movie | Music
NCF | 0.1501 | 0.1172 | 0.2230 | 0.1708 | 0.2457 | 0.1958 | 0.3727 | 0.3620
AMCF | 0.1719 | 0.1285 | 0.2143 | 0.1958 | 0.2540 | 0.2146 | 0.3862 | 0.4250
MV-DNN | 0.1105 | 0.1265 | 0.1570 | 0.1689 | 0.2015 | 0.2264 | 0.2893 | 0.2769
CCCFNet | 0.1469 | 0.1363 | 0.1778 | 0.2058 | 0.2103 | 0.2393 | 0.3169 | 0.2707
CoNet | 0.0645 | 0.1258 | 0.1192 | 0.1558 | 0.1251 | 0.2360 | 0.2585 | 0.2656
GA-DTCDR | 0.0378 | 0.0418 | 0.0602 | 0.0596 | 0.0687 | 0.0734 | 0.1217 | 0.1132
DDTCDR | 0.1937 | 0.1871 | 0.2216 | 0.2308 | 0.2889 | 0.2405 | 0.3504 | 0.3019
BiTGCF | 0.1542 | 0.1361 | 0.1997 | 0.2006 | 0.2671 | 0.2333 | 0.3334 | 0.3152
MSDCR | 0.2541 | 0.2258 | 0.2815 | 0.3027 | 0.3105 | 0.3547 | 0.4456 | 0.4596
Table 3. Performance comparison in Movie-Music

###  4.2. Performance Comparison

Tables 3, 4, and 5 show the performances of dual-target recommendation over
the three cross-domain scenarios, Movie-Book, Movie-Music, and Book-Music,
respectively. For the single-target CDR methods MV-DNN and CCCFNet, the two
domains in each scenario will be the target domain by turn.

First, we can see that in each scenario, MSDCR significantly improves the
recommendation performances on both domains. This result verifies the
advantage of MSDCR offered by the gated enhancement of domain-specific aspect
preference and the comprehensive aspect preference learning. Second, it is
also noteworthy that in most cases the single-domain baseline methods (NCF and
AMCF) interestingly perform better than the CDR baseline methods. For example,
in each scenario, the performances of the single-target CDR methods on the
target domain (i.e., the movie domain) are worse than the sing-domain methods.
We argue that this is because in the situations where both domains are sparse,
the traditional CDR methods cannot accurately capture user preference in a
single domain because of the negative transfer between two sparse domains. At
last, we can also note that DDTCDR and BiTGCF perform better than the other
dual-target CDR methods CoNet and GA-DTCDR, partly due to their ability to
simultaneously capture domain-specific features and cross-domain features,
which is similar to MSDCR. However, different from MSDCR, DDTCDR and BiTGCF
still require at least one rich domain, which results in their inferior
learning of user preference in multiple sparse domains.

Metric | NDCG@5 | NDCG@10 | HR@5 | HR@10
---|---|---|---|---
Domain | Movie | Book | Movie | Book | Movie | Book | Movie | Book
NCF | 0.1515 | 0.1647 | 0.2077 | 0.1882 | 0.2543 | 0.2676 | 0.4424 | 0.4631
AMCF | 0.2233 | 0.1774 | 0.2106 | 0.2154 | 0.2179 | 0.2670 | 0.4336 | 0.4482
MV-DNN | 0.1309 | 0.1269 | 0.1958 | 0.1772 | 0.1833 | 0.1732 | 0.3043 | 0.3009
CCCFNet | 0.1220 | 0.1335 | 0.1705 | 0.1827 | 0.2012 | 0.1991 | 0.2909 | 0.2986
CoNet | 0.0753 | 0.1133 | 0.1252 | 0.1564 | 0.1472 | 0.1993 | 0.2689 | 0.3020
GA-DTCDR | 0.0398 | 0.0407 | 0.0556 | 0.0787 | 0.0666 | 0.0803 | 0.1015 | 0.1104
DDTCDR | 0.1906 | 0.1713 | 0.2408 | 0.2213 | 0.2478 | 0.2802 | 0.3603 | 0.3446
BiTGCF | 0.0890 | 0.1012 | 0.1357 | 0.1508 | 0.1396 | 0.1756 | 0.3324 | 0.2872
MSDCR | 0.2497 | 0.2086 | 0.3065 | 0.2304 | 0.2893 | 0.3051 | 0.5035 | 0.4371
Table 4. Performance comparison in Movie-Book Metric | NDCG@5 | NDCG@10 | HR@5 | HR@10
---|---|---|---|---
Domain | Book | Music | Book | Music | Book | Music | Book | Music
NCF | 0.1068 | 0.1083 | 0.1602 | 0.2818 | 0.2381 | 0.2387 | 0.4006 | 0.3641
AMCF | 0.1347 | 0.1246 | 0.1727 | 0.2656 | 0.2244 | 0.2140 | 0.3425 | 0.3415
MV-DNN | 0.1025 | 0.1257 | 0.1631 | 0.1803 | 0.1985 | 0.2149 | 0.3074 | 0.2953
CCCFNet | 0.1153 | 0.1496 | 0.1596 | 0.1703 | 0.2185 | 0.2046 | 0.2977 | 0.3034
CoNet | 0.0657 | 0.1152 | 0.1408 | 0.1569 | 0.1663 | 0.2107 | 0.2009 | 0.2563
GA-DTCDR | 0.0378 | 0.0362 | 0.0509 | 0.0548 | 0.0621 | 0.0603 | 0.1001 | 0.1207
DDTCDR | 0.1538 | 0.1781 | 0.1883 | 0.2049 | 0.2608 | 0.2777 | 0.3912 | 0.3796
BiTGCF | 0.1380 | 0.1408 | 0.1812 | 0.2005 | 0.2462 | 0.2702 | 0.4003 | 0.3809
MSDCR | 0.1996 | 0.2162 | 0.2325 | 0.2581 | 0.3172 | 0.3114 | 0.4339 | 0.4402
Table 5. Performance comparison in Book-Music Metric | NDCG@5 | NDCG@10 | HR@5 | HR@10
---|---|---|---|---
Domain | Movie | Book | Music | Movie | Book | Music | Movie | Book | Music | Movie | Book | Music
MSDCR-w/o-DSAPE | 0.1928 | 0.0927 | 0.2715 | 0.2574 | 0.1594 | 0.2989 | 0.2558 | 0.1668 | 0.3385 | 0.4571 | 0.3750 | 0.4238
MSDCR-w/o-DIAPE | 0.2156 | 0.2110 | 0.2061 | 0.2734 | 0.2509 | 0.2450 | 0.2743 | 0.3124 | 0.2956 | 0.4555 | 0.4350 | 0.4146
MSDCR-w/o-Sep | 0.2417 | 0.1291 | 0.1911 | 0.2955 | 0.1870 | 0.2300 | 0.3293 | 0.2113 | 0.2833 | 0.4951 | 0.3904 | 0.4023
MSDCR-w/o-Enhan | 0.1798 | 0.1340 | 0.1712 | 0.2454 | 0.1968 | 0.2194 | 0.2733 | 0.2019 | 0.2565 | 0.4750 | 0.3972 | 0.4039
MSDCR-Single-Target | 0.2748 | 0.2280 | 0.1509 | 0.3221 | 0.2723 | 0.2091 | 0.3581 | 0.2981 | 0.2174 | 0.5037 | 0.4348 | 0.3983
MSDCR | 0.2967 | 0.2897 | 0.2965 | 0.3422 | 0.3305 | 0.3578 | 0.3711 | 0.3552 | 0.3483 | 0.5219 | 0.4821 | 0.4652
Table 6. Performance comparison in Movie-Book-Music

(a) Movie-Music

(b) Movie-Book

(c) Book-Music

(d) Movie-Book-Music

Figure 3.  Performance against sparsity.

###  4.3. Ablation Study

We use the triple-domain scenario Movie-Book-Music to check the effectiveness
of the components of MSDCR by comparing it with its five variants:
MSDCR-w/o-DIAPE (the variant without DIAPE), MSDCR-w/o-DSAPE (the variant
without DSAPE), MSDCR-w/o-Sep (the variant where the domain separation is
removed from DSAPE), MSDCR-w/o-Enhan (the variant where the gated preference
enhancement is removed from DSAPE), and MSDCR-Single-Target (the variant where
the multi-task framework is removed and MSDCR is trained separately for each
single target domain).

The results are shown in Table 6. First, it can be observed that MSDCR
performs remarkably better than MSDCR-w/o-DIAPE and MSDCR-w/o-DSAPE in all
domains, which shows that a user’s domain-specific preference and domain-
invariant preference both play an importance role for the learning of a user’s
comprehensive preference in each domain. Second, we can see that MSDCR also
outperforms MSDCR-w/o-Sep and MSDCR-w/o-Enhan in each domain. This verifies
that the complementary aspect preference transfer via the gated aspect
preference enhancement, together with the preserving of the uniqueness via the
domain separation, can significantly improve the recommendation performance in
each sparse domain. Third, we can note that MSDCR outperforms MSDCR-Single-
Target, which shows the multi-domain collaboration is of benefit to MSDCR in
multi-target CDR. At last, it is worthy note that for each domain, MSDCR
performs better in triple-domain scenario (Table 6) than in all dual-domain
scenarios (Tables 3 \- 5). This result verifies the superiority of MSDCR that
the more sparse domains considered, the better the domains strengthen each
other.

###  4.4. Performance Against Sparsity

Figures 3(a) \- 3(d) show the average performances of MSDCR, the baseline
methods and the variants in the three dual-domain scenarios and the triple-
domain scenario, respectively, where the horizontal axes represent the
proportion of the data that are randomly removed from the training set. We can
observe that the performances of all methods degrade as the sparsity level
increases (less training data). However, at most sparsity levels, MSDCR
consistently performs better than all baseline methods, and the higher
sparsity level, the bigger the performance gap between MSDCR and the baseline
methods. The results demonstrate the superiority of MSDCR in the CDR scenarios
where domains are all sparse, which is brought by its ability to build the
comprehensive view of a user’s preferences in each domain by transferring
complementary preferences between the multiple domains.

##  5\. Related Works

Single-target CDR aims at improving the recommendation performance in a sparse
target domain with the help of a richer source domain. Early methods for
single-target CDR jointly factorize the rating matrices in relevant domains to
generate the representations that can capture the common preferences of shared
users (Singh and Gordon, 2008; Li et al., 2009; Hu et al., 2013; Lian et al.,
2017). Recently, many DNN based methods have been proposed for better
preference capturing across domains (Elkahky et al., 2015; Man et al., 2017;
Hu et al., 2018; Yuan et al., 2019; Fu et al., 2019; Li and Tuzhilin, 2020;
Zhao et al., 2020; Yu et al., 2020; Du et al., 2020; Gao et al., 2021; Jin et
al., 2021), which often apply transfer learning techniques like domain
adaptation (Duan et al., 2009; Sun et al., 2015; Yu et al., 2020) to transfer
domain-invariant preferences from a source domain to a target domain.

In contrast to single-target CDR, dual-target CDR pursues better performance
simultaneously on both two relevant domains (Zhu et al., 2019). The existing
methods for dual-target CDR often apply bidirectional transfer learning to
jointly model a user’s preferences in two relevant domains (Hu et al., 2018;
Ma et al., 2019; Zhu et al., 2019; Li and Tuzhilin, 2020; Zhang et al., 2021).
Recently, researchers have also proposed a few graph-based methods for CDR,
which apply GNN to capture the high-order relations between features of
different domains (Zhao et al., 2019; Zhu et al., 2020b; Liu et al., 2020).
PPGN (Zhao et al., 2019) leverages GCN to capture the high-order information
propagation over the joint user-item interaction graph across different
domains. GA-DTCDR (Zhu et al., 2020b) models the complex relations between
users and items with a heterogeneous graph in each domain, and applies an
element-wise graph attention network to fuse the embeddings of common users
learned from both domains. BiTGCF (Liu et al., 2020) captures the high-order
connectivity in user-item graph of single domain through GCN, and realizes a
two-way transfer between two domains with the common user as anchors.

Basically, our MSCDR differs from the existing CDR methods in two aspects.
First, the existing CDR methods, whether for single-target or for dual-target,
often assume at least one domain has richer data, which might be impractical
since sparsity is a ubiquitous problem. In contrast, our MSDCR treats the
multiple domains as all sparse and can simultaneously improve recommendation
performance for each domain. Second, the existing CDR methods heavily depend
on the domain-invariant preferences, which might degrade the preference
learning for multiple sparse domains. Different from the existing methods,
MSCDR can build a comprehensive understanding of the user’s preferences in
each domain by considering both the domain-invariant preference and the
enhanced domain-specific preference which are learned separately.

##  6\. Conclusion

In this paper, we propose a Multi-Sparse-Domain Collaborative Recommendation
(MSDCR) model for cross-domain recommendation. Unlike traditional CDR methods,
MSDCR treats multiple relevant domains as all sparse and can simultaneously
improve recommendation performance in each domain via comprehensive aspect
preference learning. In particular, we propose a Multi-Domain Separation
Network (MDSN) and a Gated Aspect Preference Enhancement (GAPE) module, which
enables MSDCR to learn a user’s unique enhanced domain-specific aspect
preferences by adaptively transferring the complementary aspect preferences
between domains. We also propose a Multi-Domain Adaptation Network (MDAN) for
MSDCR to capture the domain-invariant aspect preferences, which together with
the enhanced domain-specific aspect preferences form a comprehensive view of a
user’s preference in each domain. At last, remarkable performance improvements
on dual-target and multi-target CDR demonstrate the effectiveness of our
MSDCR.

###### Acknowledgements.

This work is supported by National Natural Science Foundation of China under
grant 61972270, and NSF under grants III-1763325, III-1909323, III-2106758,
and SaTC-1930941.

## References

  * (1)
  * Du et al. (2020) Honghui Du, Leandro L Minku, and Huiyu Zhou. 2020.  MARLINE: Multi-Source Mapping Transfer Learning for Non-Stationary Environments. In _ICDM_. 122–131.
  * Duan et al. (2009) Lixin Duan, Ivor W Tsang, Dong Xu, and Tat-Seng Chua. 2009\.  Domain adaptation from multiple sources via auxiliary classifiers. In _ICML_.
  * Elkahky et al. (2015) Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. 2015.  A multi-view deep learning approach for cross domain user modeling in recommendation systems. In _WWW_.
  * Fu et al. (2019) Wenjing Fu, Zhaohui Peng, Senzhang Wang, Yang Xu, and Jin Li. 2019.  Deeply fusing reviews and contents for cold start users in cross-domain recommendation systems. In _AAAI_.
  * Gao et al. (2021) Chen Gao, Yong Li, Fuli Feng, Xiangning Chen, Kai Zhao, Xiangnan He, and Depeng Jin. 2021.  Cross-domain Recommendation with Bridge-Item Embeddings.  _TKDD_ (2021).
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _WWW_.
  * Hu et al. (2018) Guangneng Hu, Yu Zhang, and Qiang Yang. 2018.  Conet: Collaborative cross networks for cross-domain recommendation. In _CIKM_.
  * Hu et al. (2013) Liang Hu, Jian Cao, Guandong Xu, Longbing Cao, Zhiping Gu, and Can Zhu. 2013\.  Personalized recommendation via cross-domain triadic factorization. In _WWW_.
  * Jin et al. (2021) Di Jin, Cuiying Huo, Chundong Liang, and Liang Yang. 2021\.  Heterogeneous Graph Neural Network via Attribute Completion. In _WWW_.
  * Li et al. (2009) Bin Li, Qiang Yang, and Xiangyang Xue. 2009.  Can movies and books collaborate? cross-domain collaborative filtering for sparsity reduction. In _IJCAI_.
  * Li and Tuzhilin (2020) Pan Li and Alexander Tuzhilin. 2020.  DDTCDR: Deep dual transfer cross domain recommendation. In _WSDM_.
  * Lian et al. (2017) Jianxun Lian, Fuzheng Zhang, Xing Xie, and Guangzhong Sun. 2017\.  CCCFNet: a content-boosted collaborative filtering neural network for cross domain recommender systems. In _WWW_.
  * Liu et al. (2020) Meng Liu, Jianjun Li, Guohui Li, and Peng Pan. 2020\.  Cross Domain Recommendation via Bi-directional Transfer Graph Collaborative Filtering Networks. In _CIKM_.
  * Long et al. (2012) Mingsheng Long, Jianmin Wang, Guiguang Ding, Wei Cheng, Xiang Zhang, and Wei Wang. 2012\.  Dual transfer learning. In _SDM_.
  * Ma et al. (2019) Muyang Ma, Pengjie Ren, Yujie Lin, Zhumin Chen, Jun Ma, and Maarten de Rijke. 2019\.  $\pi$-Net: A parallel information-sharing network for shared-account cross-domain sequential recommendations. In _SIGIR_.
  * Man et al. (2017) Tong Man, Huawei Shen, Xiaolong Jin, and Xueqi Cheng. 2017\.  Cross-Domain Recommendation: An Embedding and Mapping Approach.. In _IJCAI_.
  * Pan et al. (2020) Deng Pan, Xiangrui Li, Xin Li, and Dongxiao Zhu. 2020\.  Explainable recommendation via interpretable feature mapping and evaluation of explainability. In _IJCAI_.
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback. In _arXiv preprint arXiv:1205.2618_.
  * Singh and Gordon (2008) Ajit P Singh and Geoffrey J Gordon. 2008.  Relational learning via collective matrix factorization. In _KDD_.
  * Sun et al. (2015) Shiliang Sun, Honglei Shi, and Yuanbin Wu. 2015.  A survey of multi-source domain adaptation.  _Information Fusion_ (2015).
  * Tsai and Chien (2017) Jen-Chieh Tsai and Jen-Tzung Chien. 2017.  Adversarial domain separation and adaptation. In _IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP)_.
  * Wang et al. (2019) Yaqing Wang, Chunyan Feng, Caili Guo, Yunfei Chu, and Jenq-Neng Hwang. 2019.  Solving the sparsity problem in recommendations via cross-domain item embedding based on co-clustering. In _WSDM_.
  * Yu et al. (2020) Wenhui Yu, Xiao Lin, Junfeng Ge, Wenwu Ou, and Zheng Qin. 2020.  Semi-supervised Collaborative Filtering by Text-enhanced Domain Adaptation. In _KDD_.
  * Yuan et al. (2019) Feng Yuan, Lina Yao, and Boualem Benatallah. 2019.  DARec: Deep domain adaptation for cross-domain recommendation via transferring rating patterns. In _IJCAI_.
  * Zhang et al. (2021) Yin Zhang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi, Lichan Hong, and Ed H Chi. 2021\.  A Model of Two Tales: Dual Transfer Learning Framework for Improved Long-tail Item Recommendation. In _WWW_.
  * Zhao et al. (2019) Cheng Zhao, Chenliang Li, and Cong Fu. 2019.  Cross-domain recommendation via preference propagation GraphNet. In _CIKM_.
  * Zhao et al. (2020) Cheng Zhao, Chenliang Li, Rong Xiao, Hongbo Deng, and Aixin Sun. 2020.  CATN: Cross-domain recommendation for cold-start users via aspect transfer network. In _SIGIR_.
  * Zhu et al. (2019) Feng Zhu, Chaochao Chen, Yan Wang, Guanfeng Liu, and Xiaolin Zheng. 2019.  DTCDR: A framework for dual-target cross-domain recommendation. In _CIKM_.
  * Zhu et al. (2020a) Feng Zhu, Yan Wang, Chaochao Chen, Guanfeng Liu, and Xiaolin Zheng. 2020a.  A Graphical and Attentional Framework for Dual-Target Cross-Domain Recommendation. In _IJCAI_.
  * Zhu et al. (2020b) Feng Zhu, Yan Wang, Chaochao Chen, Guanfeng Liu, and Xiaolin Zheng. 2020b.  A Graphical and Attentional Framework for Dual-Target Cross-Domain Recommendation. In _IJCAI_.
  * Zhu et al. (2021) Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. 2021\.  Cross-Domain Recommendation: Challenges, Progress, and Prospects. In _arXiv preprint arXiv:2103.01696_.


# CLINS: Continuous-Time Trajectory Estimation for
LiDAR-Inertial System

Jiajun Lv1,†, Kewei Hu1,†, Jinhong Xu1, Yong Liu1, Xiushui Ma2, Xingxing Zuo1
1 April Lab, Zhejiang University, Hangzhou, China. (Yong Liu is the
corresponding author, email: yongliu@iipc.zju.edu.cn)2 NingboTech University,
Ningbo, China.† Jiajun Lv and Kewei Hu contribute equally to this work.

###### Abstract

In this paper, we propose a highly accurate continuous-time trajectory
estimation framework dedicated to SLAM (Simultaneous Localization and Mapping)
applications, which enables fuse high-frequency and asynchronous sensor data
effectively. We apply the proposed framework in a 3D LiDAR-inertial system for
evaluations. The proposed method adopts a non-rigid registration method for
continuous-time trajectory estimation and simultaneously removing the motion
distortion in LiDAR scans. Additionally, we propose a two-state continuous-
time trajectory correction method to efficiently and efficiently tackle the
computationally-intractable global optimization problem when loop closure
happens. We examine the accuracy of the proposed approach on several publicly
available datasets and the data we collected. The experimental results
indicate that the proposed method outperforms the discrete-time methods
regarding accuracy especially when aggressive motion occurs. Furthermore, we
open source our code at https://github.com/APRIL-ZJU/clins to benefit research
community.

##  I Introduction

Multi-sensor fusion plays an essential role in simultaneous localization and
mapping (SLAM) algorithms with its complementarity and robustness, and it has
been widely deployed in autonomous navigation, scene reconstruction, mixed
reality, etc. In this paper, we study the high-accuracy continuous-time
trajectory estimation with a fusion of LiDAR and IMU measurements. Most
existing methods process LiDAR and IMU measurements in a discrete-time
fashion. The LiDAR scan collecting points when the laser heads rotate around a
mechanical axis, thus motion distortion is unavoidable when the LiDAR does not
keep static in the data collecting process. Discrete-time based methods
undistort the LiDAR points into the start time instant of the LiDAR scan by
interpolations. IMU measurements are interpolated and integrated to formulate
relative poses constraints between discrete LiDAR scans. Discrete-time based
methods have several inherent limitations that hurt the estimation. Firstly,
in practice, different sensors do not get measurements at the same frequency,
let alone the same time instants. Interpolations has to be employed to fuse
measurements from different sensors, which introduces non-negligible errors.
Secondly, it is unlikely to leverage the raw measurements directly, i.e. the
raw LiDAR points and raw IMU measurements. Raw LiDAR points are undistorted
into the specific time instants to constitute LiDAR scans, while IMU
measurements are assembled to get integrated relative pose measurements. These
phenomenons result from the intractable super-high-frequency raw sensor
measurements, which requires huge amount of pose variable to be estimated if
they are utilized in a direct way. The above-mentioned difficulties can be
summarized as the discrete pose representation fails to meet the system’s
demand of high temporal resolution. Recently, the continuous-time based
method, which models the trajectory as a function of time and supports
querying poses at any timestamp that naturally solves the problem of
integrating asynchronous and high-frequency data. With those sound properties,
continuous-time based method has been applied to many areas, such as visual-
inertial navigation system [1, 2], event camera [3], rolling-shutter camera
[1], actuated LiDAR [4], intrinsic and extrinsic calibration between sensors
[5, 6].

Figure 1: Dense 3D reconstruction of Kaist-Urban-07 dataset by simply
assembling 2D LiDAR scans from SICK LMS-511 with the estimated continuous-time
trajectory from CLINS. The global trajectory is estimated with 3D LiDAR,
Velodyne VLP-16 and Xsens IMU, MTi-300. 2D LiDAR scans is accumulated for
reconstruction due to its high density.

This paper proposes a complete continuous-time trajectory estimation framework
for LiDAR-inertial system. We summarize the contributions as follows:

  * •

We propose a continuous-time trajectory estimator, which now supports the
fusion of 3D LiDAR points and inertial data, and it is easy to expand and fuse
data from other asynchronous sensors at arbitrary frequencies.

  * •

We propose a two-stage continuous-time trajectory correction method to
efficiently and effectively tackle loop closures.

  * •

The proposed approach is extensively evaluated on several publicly available
datasets and our collected datasets, and compared to several state-of-the-art
methods. We further make the code open-sourced. To the best of our knowledge,
this is the first open-sourced continuous-time LiDAR-inertial trajectory
estimator.

##  II Related Works

Continuous-time state estimation for solving SLAM problem is firstly
systematically derived in [5] by Furgale et al. They firstly employ the
continuous-time batch optimization on calibrating the rigid transformation
between camera and IMU in visual-inertial system, and successively explore to
calibrate temporal offset between the camera and IMU [7], shutter timings for
rolling-shutter camera [8], spatio-temporal extrinsics between a LiDAR and a
stereo-visual-inertial system [9]. Rehder et al. summarize their works and
develop a general framework [10] for general spatio-temporal calibration
between diverse sensors and evaluate on several combinations of different
sensors in support of the generality claim. In addition to the calibration
applications of continuous-time state estimation, Furgale et al. also present
the details of theoretical derivations and effective implementation using
B-splines in the well-known Kalibr calibration toolbox [11]. Sommer et al.
further investigate the initialization and analytical Jacobians of B-splines
on Lie group [11]. This series of works has made a significant contribution to
the continuous-time state estimation.

Recently, continuous-time trajectory method has been employed in LiDAR
odometry. In [12, 13], researchers propose a continuous SLAM solution with a
spinning 2D laser scanner which is relatively dense and friendly for surfel-
based registration. Alismail et al. propose continuous-ICP [4] that explicitly
accounts for sensor motion during registration which improves the accuracy
compared to rigid registration. Since global batch optimization of continuous-
time trajectory is highly time-consuming, Droeschel et al. [14] present a
hierarchical refinement structure that optimizes a single firing sequence
based sub-graph firstly and incorporate sub-graph when optimizing the
allocentric pose graph. Park et al. [15] adopt a map-centric method which
introduces map deformation to remove the need for global trajectory
optimization. Instead of adjusting the global continuous trajectory directly
when loop closure happens, we propose a two-state based continuous trajectory
correction method. We firstly perform a discrete-time pose graph optimization
involved with key-scan poses; then the continuous-time trajectory is aligned
with the optimized key-scan poses while maintaining local shape via original
local velocities.

##  III Representation of Continuous-time Trajectory

First of all, we introduce notations used in this paper. We denote the 6-DoF
rigid transformation by
${}^{B}_{A}\bm{\mathbf{T}}\in\mathit{SE}(3)\in\mathbb{R}^{4\times 4}$, which
transforms the point ${}^{A}\bm{\mathbf{p}}\in\mathbb{R}^{3}$ in the frame
$\\{A\\}$ into frame $\\{B\\}$.
${}^{B}_{A}\bm{\mathbf{T}}=\begin{bmatrix}{}^{B}_{A}\bm{\mathbf{R}}&{}^{B}\bm{\mathbf{p}}_{A}\\\
\mathbf{0}&1\end{bmatrix}$ consists of rotational part
${}^{B}_{A}\bm{\mathbf{R}}\in\mathit{SO}(3)$ and translational part
${}^{B}\bm{\mathbf{p}}_{A}\in\mathbb{R}^{3}$. For simplicity, we omit the
homogeneous conversion in the rigid transformation by
${}^{B}\bm{\mathbf{p}}={}^{B}_{A}\bm{\mathbf{T}}{}^{A}\bm{\mathbf{p}}$.
$\operatorname*{Exp}(\cdot)$ maps tangent vector in $\mathbb{R}^{3}$ to Lie
group $\mathit{SO}(3)$, and $\operatorname*{Log}(\cdot)$ is its inverse
operation. $(\cdot)_{\vee}$ is used to map the elements in the Lie algebra to
tangent vectors.

Uniform B-Spline Basics
---
Order | $k$
Knot distance | $\Delta t$
Coeff. Vec. $(^{1})$ | $\bm{\mathbf{\phi}}(u)=\bm{\mathbf{{M}}}^{(k)}\bm{\mathbf{u}}$
Cumul. Coeff. Vec.$(^{1,2})$ | $\bm{\mathbf{\lambda}}(u)=\bm{\mathbf{\widetilde{M}}}^{(k)}\bm{\mathbf{u}}$
Position in Cumulative Form
Control point | $\bm{\mathbf{\Phi}}_{p}=\\{\bm{\mathbf{p}}_{i}\\}\in\mathbb{R}^{3},i\in[0,n]$
Distance | $\bm{\mathbf{d}}_{j}^{i}=\bm{\mathbf{p}}_{i+j}-\bm{\mathbf{p}}_{i+j-1}\in\mathbb{R}^{3}$
Position | $\bm{\mathbf{p}}(u)=\bm{\mathbf{p}}_{i}+\sum_{j=1}^{k-1}{\lambda_{j}(u)\cdot\bm{\mathbf{d}}^{i}_{j}}$
Velocity | $\bm{\mathbf{v}}(u)=\sum_{j=1}^{k-1}{\dot{\lambda}_{j}(u)\cdot\bm{\mathbf{d}}^{i}_{j}}$
Acceleration | $\bm{\mathbf{a}}(u)=\sum_{j=1}^{k-1}{\ddot{\lambda}_{j}(u)\cdot\bm{\mathbf{d}}^{i}_{j}}$
Orientation in Cumulative Form
Control point | $\bm{\mathbf{\Phi}}_{R}=\\{\bm{\mathbf{R}}_{i}\\}\in\mathit{SO}(3),i\in[0,n]$
Distance | $\bm{\mathbf{d}}^{i}_{j}=\operatorname*{Log}\left(\bm{\mathbf{R}}_{i+j-1}^{-1}\bm{\mathbf{R}}_{i+j}\right)\in\mathbb{R}^{3}$
Position | $\bm{\mathbf{R}}(u)=\bm{\mathbf{R}}_{i}\cdot\prod_{j=1}^{k-1}{\operatorname*{Exp}\left(\lambda_{j}(u)\cdot\bm{\mathbf{d}}^{i}_{j}\right)}$
Velocity | $\bm{\mathbf{\omega}}(u)=(\bm{\mathbf{R}}^{\top}\dot{\bm{\mathbf{R}}})_{\vee}$
$(^{1})$ $t\in[t_{i},t_{i+1})$, $u(t)=s(t)-i,\quad s(t):=(t-t_{0})/\Delta t$.
$(^{2})$ Note that $\lambda_{0}(u)\equiv 1$.

TABLE I: Summarization of uniform B-Spline based continuous-time trajectory
representation.

B-spline is smooth ($C^{2}$ continuity in case of cubic spline) and has local
continuity. Most importantly, it has closed-form analytic derivatives which is
easy to match against IMU measurements. To this end, we adopt two separate
groups of B-splines to parameterize the 3D translation and 3D rotation,
$\bm{\mathbf{p}}(t)\in\mathbb{R}^{3}$ and
$\bm{\mathbf{R}}(t)\in\mathit{SO}(3)$. This formulation is termed as split
representation of continuous-time trajectory in [16, 2]. A comprehensive
overview of the B-spline can be found in [17]. Here we also summarize the
definition of continuous trajectory representation by uniform B-splines in
Tab. I. Specifically, spline matrix $\bm{\mathbf{{M}}}^{(k)}$ and cumulative
matrix $\bm{\mathbf{\widetilde{M}}}^{(k)}$ are constant for uniform B-spline
[17]. For translational trajectory representation in $\mathbb{R}^{n}$ , the
basic form and the cumulative form of the B-spline representation are
equivalent and can be converted to each other. However, this is not the case
in the non-Euclidean space $\mathit{SO}(3)$, The orientational trajectory
representation is feasibly represented in the cumulative form. Tab. I lists
cumulative form trajectory in $\mathbb{R}^{3}$ and $\mathit{SO}(3)$ that are
adopted in this paper. Taking derivative of splines with respect to time, we
can get velocity and acceleration. As shown in Tab. I, linear velocity
$\bm{\mathbf{v}}(t)$ and linear acceleration $\bm{\mathbf{a}}(t)$ are in
global frame while angular velocity $\bm{\mathbf{\omega}}(t)$ is in local
frame.

The continuous-time trajectory of IMU in global frame $\\{G\\}$ is denoted as

$\displaystyle{}^{G}_{I}\bm{\mathbf{T}}(t)=\begin{bmatrix}\bm{\mathbf{R}}(t)&\bm{\mathbf{p}}(t)\\\
\mathbf{0}&1\end{bmatrix}\text{,}$ (1)

where we omit the superscripts and subscripts for simplicity. With the known
pre-calibrated extrinsic transformation ${}^{I}_{L}\bm{\mathbf{T}}$ between
IMU and LiDAR using toolbox LI-Calib [6], the trajectory of LiDAR could be
calculated as

$\displaystyle{}^{G}_{L}\bm{\mathbf{T}}(t)={}^{G}_{I}\bm{\mathbf{T}}(t){}^{I}_{L}\bm{\mathbf{T}}\text{.}$
(2)

Due to the locality of the B-spline basis, for $t\in[t_{i},t_{i+1})$,
${}^{G}_{I}\bm{\mathbf{T}}(t)$ is only controlled by knots at
$\\{t_{i},t_{i+1},\cdots,t_{i+k-1}\\}$ with their corresponding control points
set $\bm{\mathbf{\Phi}}(t_{i},t_{i+1})$:

$\begin{split}\bm{\mathbf{\Phi}}(t_{i},t_{i+1})&=\bm{\mathbf{\Phi}}_{R}(t_{i},t_{i+1})\cup\bm{\mathbf{\Phi}}_{p}(t_{i},t_{i+1})\\\
&=\\{\bm{\mathbf{R}}_{i},\cdots,\bm{\mathbf{R}}_{i+k-1}\\}\cup\\{\bm{\mathbf{p}}_{i},\cdots,\bm{\mathbf{p}}_{i+k-1}\\}\text{.}\end{split}$
(3)

##  IV Methodology

Figure 2: An illustration of the proposed continuous-time trajectory for
LiDAR-inertial system based on cubic B-splines.

Fig. 2 illustrates the definitions involved in the proposed continuous-time
trajectory estimator of LiDAR-inertial system. We define the active segments
of time based on the data-collection time interval of current LiDAR scan. The
control point sequence $\bm{\mathbf{\Phi}}_{\text{active}}$ corresponding to
the active segments are denoted as active control points (blue dots and red
dots). The subset of basis functions corresponding to
$\bm{\mathbf{\Phi}}_{\text{active}}$ are denoted as active basis functions
(blue curves and red curves). The active trajectory (in blue and red) is
determined by the active control points in a local region. Furthermore, except
the active segments of time, the time segments involved with the active basis
functions are notated as static segments. Accordingly, static control points
$\bm{\mathbf{\Phi}}_{\text{static}}$ (green dots) and static basis functions
(green curves) are defined.

In the proposed system, after the coming of a new LiDAR scan, new control
points (red dots) are added into the state vector to parameterize the extended
trajectory. We utilize the integrated IMU poses to initialize these new
control points (see Sec. IV-A). Subsequently, we extract LOAM features [18]
including the edge points and planar points from current scan to make data
association with local submap, which is composed of plenty of selected key-
scans based on spatio-temporal distance. Finally, given the raw IMU
measurements in the local window and the associated LiDAR features, we
estimate the continuous-time trajectory in a batch optimization fashion. The
state estimation problem could be formulated as a maximum a posteriori (MAP)
problem. With the assumption of independent Gaussian noise corruption on the
raw sensor measurements, we can estimate the continuous-time trajectory by
solving a non-linear least squares (NLLS) problem.

###  IV-A Initialization

When new LiDAR scan arrives, new control points are added to extend the
existing trajectory. We integrate discrete IMU data to obtain high-frequency
rotation, position and linear velocity estimations, which are denoted as
$\bm{\mathbf{R}}_{I_{m}},\bm{\mathbf{p}}_{I_{m}},\bm{\mathbf{v}}_{I_{m}}$ at
time $t_{m}$, respectively. We can minimize the following cost function to
initialize the newly added control points $\bm{\mathbf{\Phi}}_{\text{new}}$

$\begin{split}\arg\min_{\bm{\mathbf{\Phi}}_{\text{new}}}\sum\Big{(}{\left\|\operatorname*{Log}(\bm{\mathbf{R}}^{\top}_{I_{m}}\bm{\mathbf{R}}(t_{m}))\right\|}+{\left\|\bm{\mathbf{p}}(t_{m})-\bm{\mathbf{p}}_{I_{m}}\right\|}+\\\
{\left\|\bm{\mathbf{v}}(t_{m})-\bm{\mathbf{v}}_{I_{m}}\right\|}\Big{)}\text{.}\end{split}$
(4)

###  IV-B Non-rigid Registration in Local Window

For new-coming scan $\mathcal{S}_{k}$, measuring during time interval
$[t_{k},t_{k}+\Delta T]$, where $t_{k}$ is the timestamp of the first point in
scan $\mathcal{S}_{k}$ and $\Delta T$ is the period of completing a LiDAR
scan, it is essentially a non-rigid point cloud in $\mathcal{S}_{k}$ if there
is external motion while sensor is scanning. Therefore traditional
registration algorithms, which compute the relative transformation between two
rigid point clouds, are not applicable or have degraded performance. To tackle
this problem, we propose a non-rigid registration method which estimates the
continuous-time trajectory in current scan.

Specifically, each point in $\mathcal{S}_{k}$ is transformed to a unified
frame $\\{L_{k}\\}$

$\displaystyle{}^{L_{k}}\bm{\mathbf{x}}_{kj}={}^{G}_{L}\bm{\mathbf{T}}(t_{k})^{\top}{}^{G}_{L}\bm{\mathbf{T}}(t_{k}+\tau_{j}){}^{L_{kj}}\bm{\mathbf{x}}_{kj}$
(5)

where $\tau_{j}$ is relative timestamp of point
${}^{L_{kj}}\bm{\mathbf{x}}_{kj}$ with respect to the start time of current
scan. Similar to [18], in order to keep the computation tractable, we extract
the planar and edge features from the raw point cloud by computing local
curvature. These extracted features are used in the following registration.

Local submap consists of some selected key-scans that are distributed in time
or space. Note that when the changes of system’ pose are over a certain
extent, a key-scan is selected and all the points in that key-scan are
undistorted into the start of the LiDAR scan by using the non-active
trajectory. To find current features’ correspondences in the local submap, we
transform current features into the frame of local submap and make
correspondences based on closet neighbors [18].

Considering LiDAR data degenerate in some structureless environments while the
high-frequency IMU data are not affected, we tightly couple IMU measurements
with LiDAR features to constrain the trajectory. Thus the non-rigid
registration problem can be defined as: given the associated LiDAR features in
current scan and inertial measurements in active segments and static segments,
estimate the active control points $\bm{\mathbf{\Phi}}(t_{k},t_{k}+\Delta T)$
of the trajectory and the biases of IMU. Specifically, we can solve this
problem by minimizing the following objective function

$\displaystyle\arg\min_{\mathcal{X}}\sum{\left\|\bm{\mathbf{r}}_{\mathcal{L}}\right\|}_{\bm{\mathbf{\Sigma}}_{\mathcal{L}}}+{\left\|\bm{\mathbf{r}}_{a}\right\|}_{\bm{\mathbf{\Sigma}}_{a}}+{\left\|\bm{\mathbf{r}}_{w}\right\|}_{\bm{\mathbf{\Sigma}}_{w}}$
(6)

where $\mathcal{X}=\\{\bm{\mathbf{\Phi}}(t_{k},t_{k}+\Delta
T),\bm{\mathbf{b}}_{a},\bm{\mathbf{b}}_{g}\\}$, and $\bm{\mathbf{b}}_{a}$ ,
$\bm{\mathbf{b}}_{w}$ are the bias of accelerator and gyroscope, respectively.
$\bm{\mathbf{r}}_{\mathcal{L}},\bm{\mathbf{r}}_{a},\bm{\mathbf{r}}_{w}$ are
residual errors associated to LiDAR features and IMU measurements,
respectively.
$\bm{\mathbf{\Sigma}}_{\mathcal{L}},\bm{\mathbf{\Sigma}}_{a},\bm{\mathbf{\Sigma}}_{w}$
are the corresponding covariance matrices. The residuals are defined as

$\displaystyle\bm{\mathbf{r}}_{\mathcal{L}}$
$\displaystyle=\pi\left({}^{G}_{L}\bm{\mathbf{T}}(t_{k}+\tau_{j}){}^{L_{kj}}\bm{\mathbf{x}}_{kj}\right)$
(7) $\displaystyle\bm{\mathbf{r}}_{a}$
$\displaystyle={}^{G}_{I}\bm{\mathbf{R}}^{\top}(t_{m})\left(\bm{\mathbf{a}}(t_{m})-{}^{G}\bm{\mathbf{g}}\right)-\bm{\mathbf{a}}_{m}+\bm{\mathbf{b}}_{a}$
(8) $\displaystyle\bm{\mathbf{r}}_{\omega}$
$\displaystyle=\bm{\mathbf{\omega}}(t_{m})-\bm{\mathbf{\omega}}_{m}+\bm{\mathbf{b}}_{w}$
(9)

where project function $\pi(\cdot)$ is a point-to-plane projection for planar
features and a point-to-line projection for edge features.
$\bm{\mathbf{a}}_{m},\bm{\mathbf{\omega}}_{m}$ are inertial measurements at
time $t_{m}$. Note that static control points are involved in the optimization
process but remain unchanged during the optimization. We use the
Levenberg–Marquardt method implemented in Ceres Solver [19] to solve the above
non-linear problem.

Figure 3: An illustration of the two-stage continuous-time trajectory
correction method for loop closures.

###  IV-C Trajectory Correction

Accumulative estimation drift is unavoidable in odometry systems, we also
correct the estimations when loop closure occurs to mitigate the drift.
Normallly, global optimization is required for smooth and consistent
trajectory correction. Considering the fact that it is really time-consuming
to perform global optimization on the continuous trajectory with abundant of
knots, we propose a two-stage trajectory correction method that is effictive
but computationally friendly. At stage one, when loop closures are detected,
we perform a pose-graph optimization over the involved discrete poses of key-
scans to eliminate cumulative draft. This stage is same with the traditional
descrete-time based method [20]. At stage two, we try to update the control
points with the updated poses of key-scans. This operation can be interpreted
as projecting the continuous trajectory to some anchored poses and preserving
the local shape of the trajectory. In general, the estimation results of the
inertial-aided LiDAR odometry is accurate in local regions, so we compute the
local linear and angular velocities from the original continuous-time
trajectory at certain time instants. Those velocities can be used to locally
constrain the shape of the trajectory. To this end, the potential problem at
state two to be solved could be explicitly formulated as

$\begin{split}\arg\min_{\bm{\mathbf{\Phi}}_{\text{update}}}\sum\left({\left\|\operatorname*{Log}(\hat{\bm{\mathbf{R}}}^{\top}_{k}\bm{\mathbf{R}}(t_{k}))\right\|}+{\left\|\bm{\mathbf{p}}(t_{k})-\hat{\bm{\mathbf{p}}}_{k}\right\|}\right)+\\\
\sum\left({\left\|\bm{\mathbf{R}}(t_{j})^{\top}\bm{\mathbf{v}}(t_{j})-\hat{\bm{\mathbf{v}}}_{j}\right\|}+{\left\|\bm{\mathbf{\omega}}(t_{j})-\hat{\bm{\mathbf{\omega}}}_{j}\right\|}\right)\end{split}$
(10)

where $\hat{\bm{\mathbf{R}}}_{k},\hat{\bm{\mathbf{p}}}_{k}$ are the updated
poses of key-scan at time $t_{k}$ after the pose-graph optimization. And
$\hat{\bm{\mathbf{v}}}_{j},\hat{\bm{\mathbf{\omega}}}_{j}$ are the computed
linear velocity and angular velocity at time $t_{j}$ from the trajectory
before correction, respectively. Fig. 3 illustrates the proposed two-stage
based trajectory correction method.

##  V Experimental Results

TABLE II: RMSE of translational and rotational estimation in the 6 sequences
with motion varies from fast to slow.

Error | Sequence | LOAM | LIO-SAM | LIOM | CLINS
---|---|---|---|---|---
Trans- lation RMSE (m) | fast1 | 0.4469 | 0.1058 | 0.0529 | 0.0436
fast2 | 0.2023 | 0.1557 | 0.0663 | 0.0616
mid1 | 0.1740 | 0.1486 | 0.0576 | 0.0488
mid2 | 0.1010 | 0.0952 | 0.0874 | 0.0731
slow1 | 0.0606 | 0.0727 | 0.0318 | 0.0295
slow2 | 0.0666 | 0.0674 | 0.0435 | 0.0376
Rotation RMSE (rad) | fast1 | 0.1104 | 0.1047 | 0.0537 | 0.0565
fast2 | 0.0763 | 0.1022 | 0.0574 | 0.0538
mid1 | 0.0724 | 0.1020 | 0.0523 | 0.0567
mid2 | 0.0617 | 0.0789 | 0.0567 | 0.0538
slow1 | 0.0558 | 0.0698 | 0.0496 | 0.0438
slow2 | 0.0614 | 0.0715 | 0.0530 | 0.0570

Figure 4: The linear acceleration and angular velocity fitting results on
fast1 sequence. Only the z-axis components are shown. Red is from the
derivatives of the estimated continuous-time trajectory, while blue is from
the raw IMU measurements. Figure 5: The unmanned ground vehicle with self-
assembled sensors rigidly mounted. Sensors with red box are used to collect YQ
sequences in campus.

Sequence | Distance | Duration |  | Average Velocity
---
[km] | [s] | [m/s]
YQ-01 | 3.26 | 2471 | 1.32
YQ-02 | 0.95 | 690 | 1.38
Kaist-Urban-07 | 2.54 | 570 | 4.46
Kaist-Urban-08 | 1.56 | 307 | 5.08

TABLE III: Some specifications of YQ and Kaist-Urban sequences.

Method | YQ-01 | YQ-02 |  | Kaist-
---
Urban-07
| Kaist-
---
Urban-08
LIO-SAM (odom) | 8.857 | 3.215 | 1.288 | 3.524
CLINS(odom) | 5.917 | 3.15 | 1.383 | 3.907
LIOM | 3.931 | 0.881 | 1.515 | 16.277
LIO-SAM | 2.220 | 2.487 | 1.972 | 3.951
CLINS | 2.311 | 1.509 | 0.562 | 1.133

TABLE IV: Summary of RMSE(m) of APE results on YQ and Kaist-Urban sequences.
The best results are shown in bold.

Figure 6: Top: Trajectory comparison with different methods on YQ-01 sequence.
The red star indicates the start position, and the end part of the trajectory
are shown in zoom view. Bottom: Mapping results of CLINS with loop correction
using YQ-01 Sequence. The map is consistent with the Google Earth imagery.
Figure 7: Mapping results of CLINS using the garden, walking and small campus
datasets from left to right, respectively. All are colored with reflective
intensity.

We qualitatively and quantitatively evaluate the proposed method in a series
of publicly available datasets as well as the datasets we collected.
Considering that there is no open-source continuous-time trajectory estimation
method for LiDAR inertial system, we compare the proposed method with LIO-SAM
[21] and LIOM [22] (abbreviation of LIO-mapping). In the following
experiments, LIO-SAM without loop correction is notated as LIO-SAM(odom), and
CLINS(odom) for CLINS without loop correction.

###  V-A Trajectory Estimation in Room-scale Scenes

We evaluate the proposed CLINS on the publicly available datasets111Available
at https://drive.google.com/drive/folders/1dPy667dAnJy9wgXmlnRgQZxQF_ESuve3
provided in [22] with ground truth to examine the representation capability of
our continuous-time trajectory formulation and the estimation accuracy of
positions and orientations. The knot distance $\Delta t$ of B-spline is set as
0.05 second to deal with the highly-dynamic motion. Tab II summarizes the root
mean square error (RMSE) of the estimations from different methods. Note that
the experimental results for LOAM [18] and LIOM in Tab. II are acquired
directly from the paper [22]. It is also important to note that the output
pose estimation of LIOM is around 5 Hz and LIO-SAM in 10 Hz, while for CLINS,
we query and evaluate the estimated pose at 100Hz from the obtained
continuous-time trajectory . From Tab. II we can see that CLINS provides more
accurate translation estimation in all sequences while the accuracy of
rotation is similar between LIOM and CLINS. Fig. 4 shows the fitting results
of the estimated trajectory on fast1 sequence compared to the raw IMU
measurements. in which the angular velocity varies between -4.87 rad and 6.18
rad, while the acceleration reaches a maximum of 5.8 m/s2. The experimental
results on fast1 sequence in Tab. II and Fig. 4 show that our proposed method
can not only achieve the high accuracy in trajectory estimation, but also well
fits the derivatives of the trajectories to the inertial measurements.

###  V-B Trajectory Estimation and Scene Mapping in Large-scale Scenes

Vehicle platform. We further investigate the accuracy of the CLINS with or
without loop correction on outdoor long-distance sequence, YQ-01, YQ-02,
Kaist-Urban-07 and Kaist-Urban-08. The first two sequences are collected on
campus by ourselves using a Velodyne VLP-16 LiDAR, an Xsens-300 IMU and
JingLing-K50 RTK-GPS, and all these sensors are rigidly mounted on a small
vehicle as shown in Fig. 5. The last two sequences are from Kaist Urban
datasets [23], we leverage the IMU measurements and the data from the 3D LiDAR
mounted on the vehicle at a tilt of about 45 degrees. Tab. III lists distances
and durations of the above mentioned four sequences. Due to the slow motion of
the on-board data, we set the knot distance $\Delta t$ as 0.1 second. We
compute absolute pose error (APE) [24] with the provided ground truth to
compare the CLINS, LIO-SAM and LIOM. Tab. IV summarizes RMSE results, and the
proposed CLINS shows promising accuracy in the experiments. Fig. 6 illustrates
the trajectory comparison results with different methods on YQ-01 sequence.
Additionally, the bottom figure in Fig. 6 shows the mapping result of CLINS
using YQ-01 sequence. The map, about 700m$\times$500m in size, is consistent
with the Google Earth imagery.

Handheld device. Considering that the main advantage of CLINS is non-rigid
registration, we conduct qualitative experiments on open source handheld
datasets provided from LIO-SAM222Available at
https://drive.google.com/drive/folders/1gJHwfdHCRdjP7vuT556pv8atqrCJPbUq. Fig.
7 shows the mapping results for garden, walking and small campus sequences
from left to right, respectively. Note that, during collection of walking
sequence, aggressive motion both in translation and rotation are coupled.

###  V-C Application of Global Continuous Trajectory

In this section, we introduce an interesting application of global continuous
trajectory. Generally, a 2D LiDAR-inertial system is challenging to determine
6D pose no matter in the vehicle platform or the handheld platform. With the
assistance of the estimated globally consistent continuous trajectory, 2D
LiDAR can provide highly accurate reconstructions. Fig. 1 shows the 3D
reconstruction result of Kaist-Urban-07 using 2D LiDAR data from SICK LMS-511.
Note that global continuous trajectory is provided by CLINS with 3D LiDAR-
inertial system without the participation of 2D LiDAR.

###  V-D Implementation details

We adopt the flexible least squares solver Ceres to iteratively solve the NLLS
problem and compute derivatives automatically. Typically the non-rigid
registration converges within four or five iterations, consuming about 200ms.
The computation of jacobians takes most of the computation time and real-time
performance can be obtained by using analytical derivatives and exploring the
efficient derivative computation for B-Spline [17].

##  VI Conclusion

In this paper, we propose a continuous-time trajectory estimation approach for
LiDAR inertial system, termed CLINS. To the best of our knowledge, this is the
first open-source continuous time LiDAR-inertial trajectory estimation method.
In addiction, we propose a two-state continuous-time trajectory correction
method to efficiently and effectively cope with the computationally-intensive
loop closure correction. We compare CLINS against several open-source state-
of-the-art discrete-time based algorithms. The experiments indicate that CLINS
outperforms the discrete-time based methods regarding accuracy, which is even
more significant in the case of aggressive motion due to the non-rigid
registration in our method. There are still potential future works to improve
the accuracy and efficiency of the system. For example, it is interesting to
investigate reducing the number of static control points in the local window.
Instead of automatic derivation, using analytical jacobians and taking full
advantage of the recurrence relations [17] in the spline computation to reduce
computational effort is also worth exploring.

##  VII Acknowledgement

This work is supported by the National Natural Science Foundation of China
under Grant 61836015. We like to thank Hangyu Wu for his assistance with
visualizing the results.

## References

  * [1] S. Lovegrove, A. Patron-Perez, and G. Sibley, “Spline fusion: A continuous-time representation for visual-inertial fusion with application to rolling shutter cameras.” in _BMVC_ , vol. 2, no. 5, 2013, p. 8.
  * [2] H. Ovrén and P.-E. Forssén, “Trajectory representation and landmark projection for continuous-time structure from motion,” _The International Journal of Robotics Research_ , vol. 38, no. 6, pp. 686–701, 2019\.
  * [3] E. Mueggler, G. Gallego, H. Rebecq, and D. Scaramuzza, “Continuous-time visual-inertial odometry for event cameras,” _IEEE Transactions on Robotics_ , vol. 34, no. 6, pp. 1425–1440, 2018.
  * [4] H. Alismail, L. D. Baker, and B. Browning, “Continuous trajectory estimation for 3d slam from actuated lidar,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014, pp. 6096–6101.
  * [5] P. Furgale, T. D. Barfoot, and G. Sibley, “Continuous-time batch estimation using temporal basis functions,” in _2012 IEEE International Conference on Robotics and Automation_. IEEE, 2012, pp. 2088–2095.
  * [6] J. Lv, J. Xu, K. Hu, Y. Liu, and X. Zuo, “Targetless calibration of lidar-imu system based on continuous-time batch estimation,” _arXiv preprint arXiv:2007.14759_ , 2020.
  * [7] P. Furgale, J. Rehder, and R. Siegwart, “Unified temporal and spatial calibration for multi-sensor systems,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 1280–1286.
  * [8] L. Oth, P. Furgale, L. Kneip, and R. Siegwart, “Rolling shutter camera calibration,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 1360–1367.
  * [9] J. Rehder, P. Beardsley, R. Siegwart, and P. Furgale, “Spatio-temporal laser to visual/inertial calibration with applications to hand-held, large scale scanning,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 459–465.
  * [10] J. Rehder, R. Siegwart, and P. Furgale, “A general approach to spatiotemporal calibration in multisensor systems,” _IEEE Transactions on Robotics_ , vol. 32, no. 2, pp. 383–398, 2016.
  * [11] P. Furgale, C. H. Tong, T. D. Barfoot, and G. Sibley, “Continuous-time batch trajectory estimation using temporal basis functions,” _The International Journal of Robotics Research_ , vol. 34, no. 14, pp. 1688–1710, 2015\.
  * [12] R. Zlot and M. Bosse, “Efficient large-scale three-dimensional mobile mapping for underground mines,” _Journal of Field Robotics_ , vol. 31, no. 5, pp. 758–779, 2014.
  * [13] L. Kaul, R. Zlot, and M. Bosse, “Continuous-time three-dimensional mapping for micro aerial vehicles with a passively actuated rotating laser scanner,” _Journal of Field Robotics_ , vol. 33, no. 1, pp. 103–132, 2016.
  * [14] D. Droeschel and S. Behnke, “Efficient continuous-time slam for 3d lidar-based online mapping,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 5000–5007.
  * [15] C. Park, P. Moghadam, S. Kim, A. Elfes, C. Fookes, and S. Sridharan, “Elastic lidar fusion: Dense map-centric continuous-time slam,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1206–1213.
  * [16] A. Haarbach, T. Birdal, and S. Ilic, “Survey of higher order rigid body motion interpolation methods for keyframe animation and continuous-time trajectory estimation,” in _2018 International Conference on 3D Vision (3DV)_. IEEE, 2018, pp. 381–389.
  * [17] C. Sommer, V. Usenko, D. Schubert, N. Demmel, and D. Cremers, “Efficient derivative computation for cumulative b-splines on lie groups,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 148–11 156.
  * [18] J. Zhang and S. Singh, “Loam: Lidar odometry and mapping in real-time.” in _Robotics: Science and Systems_ , vol. 2, no. 9, 2014.
  * [19] S. Agarwal, K. Mierle, and Others, “Ceres solver,” http://ceres-solver.org.
  * [20] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “Orb-slam: a versatile and accurate monocular slam system,” _IEEE Transactions on Robotics_ , vol. 31, no. 5, pp. 1147–1163, 2015.
  * [21] T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and D. Rus, “Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping,” _arXiv preprint arXiv:2007.00258_ , 2020.
  * [22] H. Ye, Y. Chen, and M. Liu, “Tightly coupled 3d lidar inertial odometry and mapping,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3144–3150.
  * [23] J. Jeong, Y. Cho, Y.-S. Shin, H. Roh, and A. Kim, “Complex urban dataset with multi-level sensors from highly diverse urban environments,” _The International Journal of Robotics Research_ , vol. 38, no. 6, pp. 642–657, 2019\.
  * [24] M. Grupp, “evo: Python package for the evaluation of odometry and slam.” https://github.com/MichaelGrupp/evo, 2017.