# Manifold-Aware Self-Training for Unsupervised Domain Adaptation on
Regressing 6D Object Pose

Yichen Zhang1 Jiehong Lin1 Ke Chen1,2,111Corresponding authors. Zelin Xu1
Yaowei Wang2&Kui Jia1,2,∗ 1South China University of Technology
2Peng Cheng Laboratory {eezyc, lin.jiehong<EMAIL_ADDRESS>{chenk<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Domain gap between synthetic and real data in visual regression (_e.g._ 6D
pose estimation) is bridged in this paper via global feature alignment and
local refinement on the coarse classification of discretized anchor classes in
target space, which imposes a piece-wise target manifold regularization into
domain-invariant representation learning. Specifically, our method
incorporates an explicit self-supervised manifold regularization, revealing
consistent cumulative target dependency across domains, to a self-training
scheme (_e.g._ the popular Self-Paced Self-Training) to encourage more
discriminative transferable representations of regression tasks. Moreover,
learning unified implicit neural functions to estimate relative direction and
distance of targets to their nearest class bins aims to refine target
classification predictions, which can gain robust performance against
inconsistent feature scaling sensitive to UDA regressors. Experiment results
on three public benchmarks of the challenging 6D pose estimation task can
verify the effectiveness of our method, consistently achieving superior
performance to the state-of-the-art for UDA on 6D pose estimation. Code is
available at https://github.com/Gorilla-Lab-SCUT/MAST.

##  1 Introduction

Figure 1: Illustration of the proposed Manifold-Aware Self Training (MAST) for
UDA on 6D pose estimation. Top: a novel cumulative target correlation (CTC)
regularization on representation learning. Middle: one bin with the highest
confidence (_i.e._ highlighted with the largest red dish) with further local
refinement (_i.e._ the gray lines) are adopted to generate pseudo pose labels
in our MAST. Bottom: the t-SNE visualization of feature distribution w/ and
w/o the proposed CTC, which can verify the effectiveness of introduction of
target manifolds with a more smooth distribution; and comparison with our MAST
and its backbone with an example from the Occluded LineMOD dataset is given.

The problems of visual regression such as estimation of 6D pose of object
instances (_i.e._ their orientation and translation with respect to the camera
optical center) and configuration of human body parts given RGB images are
widely encountered in numerous fields such as robotics Collet et al. (2011);
He et al. (2020), augmented reality Marchand et al. (2015) and autonomous
driving Geiger et al. (2012); Chen et al. (2017), which can be typically
addressed by learning a single or multi-output regression mapping on deep
representations of visual observations. Recent regression algorithms have
gained remarkable success to handle with inconsistent lighting conditions and
heavy occlusions in-between foreground and contextual objects in uncontrolled
and cluttered environment, owing to recent development of representation
learning of visual regression, such as introduction of self-supervised
regularization Lin et al. (2022) and powerful network architectures He et al.
(2021).

In those regression tasks, visual observations, _i.e._ RGB images, can be
easily acquired in practice or directly collected from the Internet, but it is
laborious or even unfeasible for manual noise-free annotation with continuous
targets. As a result, the size of real training data with precise labels is
typically limited and less scalable, _e.g._ eggbox and holepuncher training
samples in the LineMOD Hinterstoisser et al. (2012) for 6D pose estimation,
which increases the difficulty of learning good representations. The synthesis
of images can be a powerful solution to cope with data sparsity, which can be
gained via photorealistic rendering Hodaň et al. (2019) with CAD models.
However, domain discrepancy between synthetic and real data, _e.g._ appearance
difference between CAD models and real objects, scene illumination, and
systematic imaging noises, can lead to collapse of regression performance,
which encourages the practical setting of unsupervised domain adaptation on
visual regression (UDAVR), _i.e._ samples in the source and target domains
cannot satisfy the i.i.d. condition.

Different from the widely-investigated problem of unsupervised domain
adaptation on visual classification (UDAVC) Gopalan et al. (2011); Zou et al.
(2018, 2021), only a few works Chen et al. (2021); Lee et al. (2022) have
explored the vital factors of representation learning of visual regression
that different from classification in the context of UDA. Chen et al. (2021)
revealed and exploited the sensitivity of feature scaling on domain adaptation
regression performance to regularize representation learning, which can
achieve promising results to bridge domain gap. We argue that _cumulative
dependent nature_ and _piece-wise manifolds_ in target space are two key
factors of UDA regression yet missing in the existing algorithms. To this end,
this paper proposes a Manifold-Aware Self-Training (MAST) scheme to decompose
the problem of learning a domain-invariant regression mapping into a
combination of a feature-scaling-robust globally coarse classification of
discretized target anchors via self-training based feature alignment and a
locally regression-based refinement less sensitive to inconsistent feature
scale, as shown in Figure 1.

For exploiting the cumulative dependent nature of regression targets different
from those in classification, the self-training method (_e.g._ the self-paced
self-training Zou et al. (2018)) originally designed for the UDAVC problem is
now adapted to the coarse classification on discretization of continuous
target space, with incorporating a novel piece-wise manifold regularization on
domain-invariant representation learning, namely a self-supervised cumulative
target correlation regularization. Intuitively, appearance ambiguities across
domains in representation learning can be mitigated via leveraging consistent
target correlation under certain distance metrics in target space (_e.g._ the
Euclidean distance in $R(3)$ translation space). Furthermore, considering the
risk of sensitivity to varying feature scaling in the UDAVR problem Chen et
al. (2021), learning unified local regression functions with those shared
features of the classification of discretized target bins (typically having
inconsistent feature scales) can achieve superior robustness against large
scale variations of transferable representations. Extensive experiments on
three popular benchmarks of the challenging UDA on 6D pose estimation can
confirm the effectiveness of our MAST scheme, consistently outperforming the
state-of-the-art.

The novelties of our paper are summarized as follows.

  * •

This paper proposes a novel and generic manifold-aware self-training scheme
for unsupervised domain adaptation on visual regression, which exploits
cumulative correlation and piece-wise manifolds in regression target space for
domain-invariant representation learning.

  * •

Technically, a novel cumulative target correlation regularization is proposed
to regularize the self-training algorithm on coarse classification with latent
dependency across regression targets, while local refinement can be achieved
via learning implicit functions to estimate residual distance to the nearest
anchor within local target manifolds in a unified algorithm.

  * •

Experiment results on multiple public benchmarks of UDA on 6D pose estimation
can verify consistent superior performance of our scheme to the state-of-the-
art UDA pose regressors.

##  2 Related Works

#### 6D Pose Estimation.

The problem of estimating 6D pose of object instances within an RGB image
(optionally with a complementary depth image) is active yet challenging in
robotics and computer vision. With the rise of deep learning, recent methods
for predicting 6D poses can be divided into two main groups – keypoint-based
Peng et al. (2019); Zakharov et al. (2019) and regression based Xiang et al.
(2018); Labbé et al. (2020); Wang et al. (2021b). The former relied on
learning a 2D-to-3D correspondence mapping between object keypoints in 3D
space and their 2D projection on images with the Perspective-n-Point (PnP)
Fischler and Bolles (1981). Such a correspondence can be achieved by either
detecting a limited size of landmarks Tekin et al. (2018); Peng et al. (2019)
or pixel-wise voting from a heatmap Park et al. (2019); Zakharov et al.
(2019). The latter concerned on deep representation learning for direct pose
regression with the point-matching loss for optimizing output pose Xiang et
al. (2018); Labbé et al. (2020) or proposing a differentiable PnP paradigm in
an end-to-end training style Wang et al. (2021b); Chen et al. (2022a).
Alternatively, the problem can also be formulated into ordinal classification
via discretization of $SE(3)$ space into class bins Su et al. (2015); Kehl et
al. (2017). To alleviate representation ambiguities, the estimated 6D pose of
objects can be further refined via either an iterative refinement with
residual learning Li et al. (2018b); Manhardt et al. (2018) or simply the
Iterative Closest Point Xiang et al. (2018), while some work introduced cross-
view fusion based refinement Labbé et al. (2020); Li et al. (2018a). Existing
refinement strategies are typically employed as a post-processing step
following the main module of 6D pose estimation, some of which such as Li et
al. (2018b); Xu et al. (2022) can be designed in an end-to-end learning
cascade to obtain significant performance gain, but they are not designed for
bridging domain gap and therefore cannot ensure good performance under the
UDAVR setting. Alternatively, Li et al. (2018a) introduced a combined scheme
of both coarse classification and local regression-based refinement
simultaneously, which is similar to our MAST method. However, the main
differences lie in the introduction of the cumulative target correlation
regularization in our scheme to encourage domain-invariant pose
representations revealing the dependent nature of regression targets.

#### Unsupervised Domain Adaptation on Visual Regression.

Most of regression methods Xu et al. (2019); Bao et al. (2022) employ
annotated real data for model training, but manual annotations on real data
are usually laboriously expensive or even unfeasible. Lack of sufficient
annotated real data encourages the practical setting of Simulation-to-Reality
(Sim2Real) UDAVR, _i.e._ learning a domain-agnostic representation given
annotated synthetic data as source domain and unlabeled real data as target
domain. A simple yet effective way to narrow Sim2Real domain gap can rely on
domain randomization Kehl et al. (2017); Manhardt et al. (2018), while recent
success of self-supervised learning for UDAVC Zou et al. (2021); Yue et al.
(2021) inspired a number of self-supervised regressors Wang et al. (2021a);
Yang et al. (2021) in the context of Regression. Self6D Wang et al. (2020) and
its extension Self6D++ Wang et al. (2021a) leveraged a differentiable renderer
to conduct self-supervised visual and geometrical alignment on visible and
amodal object mask predictions. Bao _et al_. Bao et al. (2022) introduced a
self-supervised representation learning of relative rotation estimation to
adapt one gaze regressor to the target domain. Zhang _et al_. Zhang et al.
(2021) utilized a Graph Convolutional Network to model domain-invariant
geometry structure among key-points, which is applied to guide training of the
object pose estimator on real images. These mentioned algorithms were designed
for only one specific task and cannot be directly applied to other visual
regression problems. Chen et al. (2021) proposed the representation subspace
distance (RSD) generic to multiple UDAVR problems, but cannot perform well on
the challenging task having severe representation ambiguities, _e.g._ 6D pose
estimation investigated in this paper (see Table 3). In contrast, the proposed
MAST scheme is generic to UDAVR owing to exploiting explicit target
correlation in the style of local manifolds to regularize deep representation
learning agnostic to domains.

#### Self-Training.

Self-training methods utilize a trained model on labeled data to make
predictions of unannotated data as pseudo labels Lee and others (2013) (_i.e._
supervision signals assigned to unlabeled data), which is widely used in semi-
supervised learning Lee and others (2013); Sohn et al. (2020) and UDA Roy et
al. (2019). Sohn et al. (2020) generated pseudo labels from weakly augmented
images, which are adopted as supervision of strongly augmented variants in
semi-supervised learning; similar scripts are shared with the noisy student
training Xie et al. (2020). Chen et al. (2011) proposed the co-training for
domain adaptation that slowly adding to the training set both the target
features and instances in which the current algorithm is the most confident.
Zou et al. (2018) proposed the self-paced self-training (SPST) for
unsupervised domain adaptation classification that can perform a self-paced
learning Tang et al. (2012) with latent variable objective optimization. The
representative SPST has inspired a number of follow-uppers such as Zou et al.
(2021) and Chen et al. (2022b). Nevertheless, all of existing self-training
algorithms were designed for classification or segmentation, while self-
training for the UDAVR remains a promising yet less explored direction.

##  3 Methodology

Given a source domain $\\{\mathcal{I}^{i}_{S},\bm{y}^{i}_{S}\\}_{i=1}^{N_{S}}$
with $N_{S}$ labeled samples and a target domain
$\\{\mathcal{I}^{i}_{T}\\}_{i=1}^{N_{T}}$ with $N_{T}$ unlabeled samples,
tasks of UDAVR aim at learning a domain-invariant regression mapping to a
shared continuous label space $\mathcal{Y}$. In the context of our focused 6D
object pose estimation, the source and target domains are often the synthetic
and real-world data, respectively, while the shared label space between two
domains is the whole learning space of $SE(3)$.

To deal with the problems of UDAVR introduced in Sec. 1, _e.g._ , cumulative
dependent nature and piece-wise manifolds in target space, we propose in this
paper a manifold-aware self-training scheme, which decomposes the learning of
$SE(3)$ space into a global classification on discretized pose anchors and a
local pose refinement for feature scaling robustness, and incorporates a self-
supervised manifold regularization to the self-training.

Figure 2: The pipeline of our manifold-aware self-training scheme.

###  3.1 The Design of Network Architecture

Given a batch of $B$ object-centric RGB images
$\\{\mathcal{I}^{b}\\}_{b=1}^{B}$ as input, the proposed scheme is designed to
predict 6D object poses $\\{\mathcal{T}^{b}\\}_{b=1}^{B}$, with each pose
$\mathcal{T}=[\bm{R}|\bm{t}]$ represented by a 3D rotation $\bm{R}\in SO(3)$
and a 3D translation $\bm{t}\in\mathbb{R}^{3}$. The whole network architecture
is shown in Fig. 2, which consists of three main modules, including a Feature
Extractor, a Coarse Classifier of discretized pose anchors, and a Fine
Regressor of Residual Poses to the nearest anchor.

More specifically, we employ the same feature extractor as Labbé et al. (2020)
to learn the pose-sensitive feature vectors
$\\{\bm{f}^{b}\in\mathbb{R}^{C}\\}_{b=1}^{B}$ from each frame, which are then
fed into the decoupled coarse classifier and fine regressor individually,
whose output are combined together as final pose predictions. The former
learns coarse poses via classification on the discretized pose anchors, while
the latter learns residual poses to refine the coarse ones of pose anchors
locally; both modules share the same input features, achieving superior
robustness against inconsistent feature scaling. We will take a single image
as an example to detail the two pose estimation modules shortly, and thus omit
the superscript $b$ of the notations for simplicity in the following
subsections.

#### Coarse Classification on Discretized Pose Anchors.

Given the pose-sensitive feature $\bm{f}$ of $\mathcal{I}$, the goal of this
module is to globally make coarse predictions of $\bm{R}$ and $\bm{t}$ via
classification on their pre-defined anchors, respectively. For the rotation
$\bm{R}$, we generate $N_{\bm{R}}$ anchors that are uniformly distributed on
the whole $SO(3)$ space as Li et al. (2018a), which are denoted as
$\\{\bm{R}_{a}^{1},\cdots,\bm{R}_{a}^{N_{\bm{R}}}\\}$. For the translation
$\bm{t}$, we factorize it into three individual classification targets,
including the two translation components $v_{x}$ and $v_{y}$ on the image
coordinate system along X-axis and Y-axis, with the remaining component $z$
along Z-axis; for each classification target $t\in\\{v_{x},v_{y},z\\}$, we
discretize the range of $[d_{t}^{min},d_{t}^{max}]$ into $N_{t}$ bins
uniformly, and use the bin centers $\\{t_{a}^{1},\cdots,t_{a}^{N_{t}}\\}$ as
the anchors of $t$. We implement the classifier as four Multilayer Perceptrons
(MLPs) with $N_{\bm{R}},N_{v_{x}},N_{v_{y}},N_{z}$ output neurons, which are
collectively denoted as the probabilities
$\bm{S}_{\bm{R}}\in\mathbb{R}^{N_{\bm{R}}}$,
$\bm{S}_{v_{x}}\in\mathbb{R}^{N_{v_{x}}}$,
$\bm{S}_{v_{y}}\in\mathbb{R}^{N_{v_{y}}}$, and
$\bm{S}_{z}\in\mathbb{R}^{N_{z}}$ of $\bm{R}$, $v_{x}$, $v_{y}$ and $z$,
respectively. Denoting their indexes of maximal probabilities as
$i_{\bm{R}}^{max}$, $i_{v_{x}}^{max}$, $i_{v_{y}}^{max}$ and $i_{z}^{max}$,
the classifier finally gives out their coarse pose predictions as
$\bm{R}_{cls}=\bm{R}^{i_{\bm{R}}^{max}}_{a}$,
$v_{x,cls}=v_{x,a}^{i_{v_{x}}^{max}}$, $v_{y,cls}=v_{y,a}^{i_{v_{y}}^{max}}$
and $z_{cls}=z_{a}^{i_{z}^{max}}$.

#### Fine Regressor of Residual Poses.

This module shares the same input feature $\bm{f}$ as the coarse classifier to
make the learning more robust to feature scale variations, and is implemented
as four MLPs with $N_{\bm{R}}\times 6,N_{v_{x}},N_{v_{y}},N_{z}$ output
neurons to regress the residuals of the pose anchors. We collectively denote
the outputs as $\\{\bm{R}_{reg,6D}^{i}\\}_{i=1}^{N_{\bm{R}}}$,
$\\{v_{x,reg}^{i}\\}_{i=1}^{N_{v_{x}}}$,
$\\{v_{y,reg}^{i}\\}_{i=1}^{N_{v_{y}}}$, and
$\\{z_{reg}^{i}\\}_{i=1}^{N_{z}}$; here we use the continuous 6D
representations of rotation Zhou et al. (2019) as the regression target, which
can be transformed into rotation matrices
$\\{\bm{R}_{reg}^{i}\\}_{i=1}^{N_{\bm{R}}}$. According to probabilities of the
classifier, the fine regressor refines the coarse predictions via the
residuals $\bm{R}_{reg}=\bm{R}^{i_{\bm{R}}^{max}}_{reg}$,
$v_{x,reg}=v_{x,reg}^{i_{v_{x}}^{max}}$,
$v_{y,reg}=v_{y,reg}^{i_{v_{y}}^{max}}$, and $z_{reg}=z_{reg}^{i_{z}^{max}}$.

Combining coarse anchor predictions and their residuals, our proposed network
can generate the final object pose $\mathcal{T}=[\bm{R}|\bm{t}]$, with
$\bm{t}=[x,y,z]$, as follows:

$\left\\{\begin{aligned} \bm{R}&=\bm{R}_{reg}\cdot\bm{R}_{cls}\\\
x&=(v_{x,cls}+v_{x,reg})\cdot z/f_{x}\\\ y&=(v_{y,cls}+v_{y,reg})\cdot
z/f_{y}\\\ z&=z_{cls}+z_{reg}\end{aligned}\right.,$ (1)

where $f_{x}$ and $f_{y}$ are the focal lengths along X-axis and Y-axis,
respectively.

###  3.2 Manifold-Aware Objective

To train our network, we formulate the following manifold-aware objective
$\mathcal{L}$ via combining a coarse-to-fine pose decomposition loss
$\mathcal{L}_{pose}$ with a cumulative target correlation regularization
$\mathcal{L}_{ctc}$:

$\mathcal{L}=\mathcal{L}_{pose}+\mathcal{L}_{ctc},$ (2)

where $\mathcal{L}_{pose}$ favors for domain-invariant representations in 6D
pose estimation across domains, while $\mathcal{L}_{ctc}$ enforces target
manifolds into representation learning.

#### Coarse-to-fine Pose Decomposition Loss.

$\mathcal{L}_{pose}$ consists of two loss terms $\mathcal{L}_{cls}$ and
$\mathcal{L}_{reg}$ for the coarse classifier and the fine regressor,
respectively, as follows:

$\mathcal{L}_{pose}=\frac{1}{B}\sum_{b=1}^{B}\mathcal{L}_{cls}^{b}+\mathcal{L}_{reg}^{b}.$
(3)

For simplicity, we introduce $\mathcal{L}_{pose}$ on single input, and thus
omit the batch index $b$ accordingly.

For the coarse classifier, given the ground truth pose
$\tilde{\mathcal{T}}=[\tilde{\bm{R}}|\tilde{\bm{t}}]$, with
$\tilde{\bm{t}}=[\tilde{x},\tilde{y},\tilde{z}]$ (and
$\tilde{v}_{x},\tilde{v}_{y}$), we first adopt a sparse scoring strategy to
assign the labels for $\bm{S}_{\bm{R}}$, $\bm{S}_{v_{x}}$, $\bm{S}_{v_{y}}$
and $\bm{S}_{z}$, resulting in $\tilde{\bm{S}}_{\bm{R}}$,
$\tilde{\bm{S}}_{v_{x}}$, $\tilde{\bm{S}}_{v_{y}}$ and $\tilde{\bm{S}}_{z}$,
respectively, with each element $\tilde{s}_{t}^{i}$
$(t\in\\{\bm{R},v_{x},v_{y},z\\})$ assigned as follows:

$\tilde{s}_{t}^{i}=\left\\{\begin{array}[]{lll}\theta_{t,1},&&{i\in\text{NN}_{1}(\tilde{t})}\\\
\theta_{t,2},&&{i\in\text{NN}_{k_{t}}(\tilde{t})\backslash\text{NN}_{1}(\tilde{t})}\\\
0,&&{Otherwise}\end{array}\right.,$ (4)

where $\theta_{t,1}\gg\theta_{t,2}$, and
$\theta_{t,1}+(k_{t}-1)\theta_{t,2}=1$. $\text{NN}_{k_{t}}(\tilde{t})$ denotes
the set of indexes of the $k_{t}$ nearest anchors of $\tilde{t}$.222We use the
geodesic distance Gao et al. (2018) to measure the distance of two rotations
$\bm{R}_{1}$ and $\bm{R}_{2}$ as
$\arccos(\frac{trace(\bm{R}_{1}\bm{R}_{2}^{T})-1}{2})$, and use the difference
value to measure that of two scalars. With the assigned labels, we use the
cross-entropy loss $\mathcal{H}$ on top of the classifier as follows:

$\mathcal{L}_{cls}=\sum_{t\in\\{\bm{R},v_{x},v_{y},z\\}}\mathcal{H}(\bm{S}_{t},\tilde{\bm{S}}_{t}).$
(5)

For the fine regressor, we make individual predictions on each anchor of
$t\in\\{\bm{R},(v_{x},v_{y}),z\\}$ by combining the paired classification and
regression results, and supervise the predictions of their top K nearest
anchors of $\tilde{t}$ as follows:

$\displaystyle\mathcal{L}_{reg}=$
$\displaystyle\sum_{i\in\text{NN}_{k_{\bm{R}}}(\tilde{\bm{R}})}\mathcal{D}(\mathcal{T}_{\bm{R}^{i}},\tilde{\mathcal{T}})+\sum_{i\in\text{NN}_{k_{z}}(\tilde{z})}\mathcal{D}(\mathcal{T}_{z^{i}},\tilde{\mathcal{T}})$
(6)
$\displaystyle+\sum_{i\in\text{NN}_{k_{v_{x}v_{y}}}(\tilde{v}_{x}\tilde{v}_{y})}\mathcal{D}(\mathcal{T}_{v_{x}^{i}v_{y}^{i}},\tilde{\mathcal{T}}),$

where $t^{i}$ denotes the prediction of the anchor $i$ of $t$, and
$\mathcal{T}_{t^{i}}$ denotes the object pose computed by $t^{i}$ and other
ground truths
$\\{\tilde{\bm{R}},(\tilde{v}_{x},\tilde{v}_{y}),\tilde{z}\\}\backslash\tilde{t}$.
$\mathcal{D}(\cdot,\cdot)$ is the $L_{1}$ distance between the point sets
transformed by two object poses from the same object point cloud
$\mathcal{O}$, as follows:

$\mathcal{D}(\mathcal{T},\tilde{\mathcal{T}})=\frac{1}{|\mathcal{O}|}\sum_{x\in\mathcal{O}}\|\mathcal{T}x-\tilde{\mathcal{T}}x\|_{1}.$
(7)

Following Labbé et al. (2020), we combine the supervision of $v_{x}$ and
$v_{y}$ for convenience in (6), and also employ the same strategy to handle
object symmetries by finding the closest ground truth rotation to the
predicted one.

#### Cumulative Target Correlation Regularization.

For regression tasks, continuous targets preserve latent cumulative dependency
Chen et al. (2013). When we discretize the continuously changing targets into
discretized labels as classification, the assumption of independence across
targets is adopted, which is invalid in regressing continuous targets. As a
result, each class cannot seek support from samples of correlated class, which
can significantly reduce performance especially for sparse and imbalanced data
distributions. To better cope with this problem, we propose to regularize the
features by an explicit relation in the regression target space.

Given the pose-sensitive feature vectors
$\\{\bm{f}^{b}\in\mathbb{R}^{C}\\}_{b=1}^{B}$ of a mini-batch inputs
$\\{\mathcal{I}^{b}\\}_{b=1}^{B}$, we first build the feature correlation
graph $\mathcal{G}\in\mathbb{R}^{B\times B}$ across the data batch via feature
cosine similarities, with the element $g^{ij}$ indexed by $(i,j)$ computed as
follows:

$g^{ij}=\frac{<\bm{f}^{i},\bm{f}^{j}>}{||\bm{f}^{i}||_{2}\cdot||\bm{f}^{j}||_{2}},$
(8)

where $<\cdot,\cdot>$ denotes inner product. We then build the ground truth
$\tilde{\mathcal{G}}$ based on a pre-computed correlation graph
$\tilde{\mathcal{G}}_{0}\in\mathbb{R}^{N\times N}$ with $N$ pose classes;
assuming the classes of $\mathcal{I}_{i}$ and $\mathcal{I}_{j}$ are $n_{i}$
and $n_{j}$, respectively, we assign the value of
$\tilde{g}^{ij}\in\tilde{\mathcal{G}}$ as that of
$\tilde{g}_{0}^{n_{i}n_{j}}$. Finally, the proposed target correlation
regularizer can be simply written as the squared $L_{2}$ distance between
$\mathcal{G}$ and $\tilde{\mathcal{G}}$:

$\mathcal{L}_{ctc}=\|\mathcal{G}-\tilde{\mathcal{G}}\|_{2}^{2}.$ (9)

There are multiple choices for building the pose-related correlation graph
$\tilde{\mathcal{G}}_{0}$; here we introduce a simple but effective one, which
utilizes the similarity of depth components of translations along Z-axis to
initialize $\tilde{\mathcal{G}}_{0}$, with $N=N_{z}$. Specifically, for the
anchors $\\{z_{a}^{1},\cdots,z_{a}^{N}\\}$ of $z$, we map them linearly to the
angles $\\{\phi^{1},\cdots,\phi^{N}\\}$ as follows:

$\phi^{n}=\frac{z_{a}^{n}}{z^{max}-z^{min}}\cdot\frac{\pi}{2},$ (10)

and the element $\tilde{g}_{0}^{n_{i}n_{j}}$ of $\tilde{\mathcal{G}}_{0}$
indexed by $(n_{i},n_{j})$ can be defined as the cosine of difference between
the angles:

$\tilde{g}_{0}^{n_{i}n_{j}}=\cos(|\phi^{n_{i}}-\phi^{n_{j}}|).$ (11)

When $z_{a}^{n_{i}}$ and $z_{a}^{n_{j}}$ are close, the difference of their
corresponding angles is small, and thus the correlation value of
$\tilde{g}_{0}^{n_{i}n_{j}}$ will be large. The reason for choosing $z$ is
that the learning of this component is very challenging in 6D pose estimation
without depth information. Experimental results in Sec. 4.2 also verify the
effectiveness of our regularization.

###  3.3 Manifold-Aware Self-training

To reduce the Sim2Real domain gap, we design a manifold-aware self-training
scheme for unsupervisedly adapting the pose estimator, which adaptively
incorporates our proposed manifold-aware training objective in (2) with Self-
Paced Self-Training Zou et al. (2018) to select target samples in an easy-to-
hard manner. More specifically, we first train a teacher model
$\mathcal{M}_{T}$ on the labeled synthetic data (source domain) as a pseudo-
label annotator for the unlabeled real-world data (target domain), and select
the training samples from the real data with pseudo labels for the learning of
a student model $\mathcal{M}_{S}$. Both teacher and student models share the
same networks introduced in Sec. 3.1, and are trained by solving the problems
of $\min_{\mathcal{M}_{T}}\mathcal{L}$ and
$\min_{\mathcal{M}_{S}}\mathcal{L}$, respectively.

The core of sample selection on the target domain lies on the qualities of
pseudo labels. For the tasks of visual classification, the categorical
probabilities are usually used as the measurement of qualities, while for
those of visual regression tasks, _e.g._ , object pose estimation in this
paper, direct usage of the typical mean square error (MSE) can be less
effective due to lack of directional constraints for adaptation. In geometric
viewpoint, the surface of a super ball can have the same MSE distance to its
origin, but the optimal regions of object surface for domain adaptation exist,
which can be omitted by the MSE metric. Owing to the decomposition of object
pose estimation into coarse classification and fine regression in our MAST
scheme, we can flexibly exploit the classification scores to indicate the
qualities of pseudo labels, since the coarse classification points out the
overall direction of pose estimation. In practice, we use the probabilities
$\bm{S}_{z}$ as confidence scores because UDA on classification can perform
more stably and robustly, and set a threshold $\tau$ to select the samples
with scores larger than $\tau$ for training $\mathcal{M}_{S}$. Larger score
indicates higher quality of the pseudo label. Following Zou et al. (2018), the
threshold $\tau$ is gradually decreased during training, realizing the
learning in an easy-to-hard manner and making $\mathcal{M}_{S}$ generalized to
harder target samples.

##  4 Experiments

3 Method | Ape | Bvise | Cam | Can | Cat | Drill | Duck | Eggbox | Glue | Holep | Iron | Lamp | Phone | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| Data: syn (w/ GT)
AAE Sundermeyer et al. (2020) | 4.0 | 20.9 | 30.5 | 35.9 | 17.9 | 24.0 | 4.9 | 81.0 | 45.5 | 17.6 | 32.0 | 60.5 | 33.8 | 31.4
DSC-PoseNet Yang et al. (2021) | 23.4 | 75.6 | 11.7 | 40.1 | 26.7 | 53.8 | 14.0 | 73.6 | 26.7 | 19.5 | 56.2 | 39.4 | 20.0 | 37.0
MHP Manhardt et al. (2019) | 11.9 | 66.2 | 22.4 | 59.8 | 26.9 | 44.6 | 8.3 | 55.7 | 54.6 | 15.5 | 60.8 | - | 34.4 | 38.8
DPOD Zakharov et al. (2019) | 35.1 | 59.4 | 15.5 | 48.8 | 28.1 | 59.3 | 25.6 | 51.2 | 34.6 | 17.7 | 84.7 | 45.0 | 20.9 | 40.5
SD-Pose Li et al. (2021) | 54.0 | 76.4 | 50.2 | 81.2 | 71.0 | 64.2 | 54.0 | 93.9 | 92.6 | 24.0 | 77.0 | 82.6 | 53.7 | 67.3
Self6D++ Wang et al. (2021a) | 50.9 | 99.4 | 89.2 | 97.2 | 79.9 | 98.7 | 24.6 | 81.1 | 81.2 | 41.9 | 98.8 | 98.9 | 64.3 | 77.4
MAR (ours) | 68.6 | 97.4 | 79.4 | 98.3 | 87.1 | 94.2 | 61.3 | 82.0 | 87.1 | 56.7 | 94.3 | 92.3 | 68.8 | 82.1
| Data: syn (w/ GT) + real (w/ GT)
DPOD Zakharov et al. (2019) | 53.3 | 95.2 | 90.0 | 94.1 | 60.4 | 97.4 | 66.0 | 99.6 | 93.8 | 64.9 | 99.8 | 88.1 | 71.4 | 82.6
DSC-PoseNet Yang et al. (2021) | 59.2 | 98.1 | 88.0 | 92.1 | 79.4 | 94.5 | 51.7 | 98.5 | 93.9 | 78.4 | 96.2 | 96.3 | 90.0 | 85.9
Self6D++ Wang et al. (2021a) | 85.0 | 99.8 | 96.5 | 99.3 | 93.0 | 100.0 | 65.3 | 99.9 | 98.1 | 73.4 | 86.9 | 99.6 | 86.3 | 91.0
MAR (ours) | 81.4 | 99.9 | 90.7 | 99.6 | 94.6 | 98.1 | 85.5 | 97.6 | 98.5 | 89.2 | 97.1 | 99.7 | 96.0 | 94.5
| Data: syn (w/ GT) + real (w/o GT)
Self6D-RGB Wang et al. (2020) | 0.0 | 10.1 | 3.1 | 0.0 | 0.0 | 7.5 | 0.1 | 33.0 | 0.2 | 0.0 | 5.9 | 20.7 | 2.4 | 6.4
DSC-PoseNet Yang et al. (2021) | 35.9 | 83.1 | 51.5 | 61.0 | 45.0 | 68.0 | 27.6 | 89.2 | 52.5 | 26.4 | 56.3 | 68.7 | 46.3 | 54.7
Zhang _et al_. Zhang et al. (2021) | – | – | – | – | – | – | – | – | – | – | – | – | - | 60.4
Sock _et al_. Sock et al. (2020) | 37.6 | 78.6 | 65.5 | 65.6 | 52.5 | 48.8 | 35.1 | 89.2 | 64.5 | 41.5 | 80.9 | 70.7 | 60.5 | 60.6
Self6D++ Wang et al. (2021a) | 76.0 | 91.6 | 97.1 | 99.8 | 85.6 | 98.8 | 56.5 | 91.0 | 92.2 | 35.4 | 99.5 | 97.4 | 91.8 | 85.6
MAST (ours) | 73.5 | 97.2 | 80.8 | 98.6 | 89.1 | 93.9 | 66.9 | 95.3 | 95.4 | 69.8 | 95.5 | 98.6 | 79.1 | 87.2
3 |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 1: Comparative evaluation on the LineMOD dataset w.r.t. the Average
Recall (%) of ADD(-S). Symmetric object classes are in italic. ‘MAR’
(manifold-aware regression) denotes our method without self-training.

3 Method | Occluded LineMOD | HomebrewedDB
---|---|---
Ape | Can | Cat | Drill | Duck | Eggbox | Glue | Holep | Mean | Bvise | Drill | Phone | Mean
| Data: syn (w/ GT)
DPOD Zakharov et al. (2019) | 2.3 | 4.0 | 1.2 | 7.2 | 10.5 | 4.4 | 12.9 | 7.5 | 6.3 | 52.9 | 37.8 | 7.3 | 32.7
CDPN Li et al. (2019) | 20.0 | 15.1 | 16.4 | 22.2 | 5.0 | 36.1 | 27.9 | 24.0 | 20.8 | – | – | – | –
SD-Pose Li et al. (2021) | 21.5 | 56.7 | 17.0 | 44.4 | 27.6 | 42.8 | 45.2 | 21.6 | 34.6 | – | – | – | –
SSD6D+Ref. Manhardt et al. (2018) | – | – | – | – | – | – | – | – | – | 82.0 | 22.9 | 24.9 | 43.3
Self6D++ Wang et al. (2021a) | 44.0 | 83.9 | 49.1 | 88.5 | 15.0 | 33.9 | 75.0 | 34.0 | 52.9 | 7.1 | 2.2 | 0.1 | 3.1
MAR (ours) | 44.9 | 78.4 | 40.3 | 73.5 | 47.9 | 26.9 | 72.1 | 58.0 | 55.3 | 92.6 | 91.5 | 80.0 | 88.0
| Data: syn (w/ GT) + real (w/o GT)
DSC-PoseNet Yang et al. (2021) | 13.9 | 15.1 | 19.4 | 40.5 | 6.9 | 38.9 | 24.0 | 16.3 | 21.9 | 72.9 | 40.6 | 18.5 | 44.0
Sock _et al_. Sock et al. (2020) | 12.0 | 27.5 | 12.0 | 20.5 | 23.0 | 25.1 | 27.0 | 35.0 | 22.8 | 57.3 | 46.6 | 41.5 | 52.0
Zhang _et al_. Zhang et al. (2021) | – | – | – | – | – | – | – | – | 33.7 | – | – | – | 63.8
Self6D++ Wang et al. (2021a) | 57.7 | 95.0 | 52.6 | 90.5 | 26.7 | 45.0 | 87.1 | 23.5 | 59.8 | 56.1 | 97.7 | 85.1 | 79.6
MAST (ours) | 47.6 | 82.9 | 45.4 | 75.0 | 53.7 | 48.2 | 75.3 | 63.0 | 61.4 | 93.8 | 91.5 | 81.8 | 89.0
3 |  |  |  |  |  |  |  |  |  |  |  |  |

Table 2: Comparative evaluation on the Occluded LineMOD and HomebrewedDB
datasets w.r.t. the Average Recall (%) of the ADD(-S). Symmetric object
classes are in italic. ‘MAR’ (manifold-aware regression) denotes our method
without self-training.

#### Datasets and Settings.

The LineMOD dataset Hinterstoisser et al. (2012) provides individual videos of
13 texture-less objects, which are recorded in cluttered scenes with
challenging lighting variations. For each object, we follow Brachmann et al.
(2014) to use randomly sampled $15\%$ of the sequence as the real-world
training data of the target domain, and the remaining images are set aside for
testing. The Occluded LineMOD dataset Brachmann et al. (2014) is a subset of
the LineMOD with 8 different objects, which is formed by the images with
severe object occlusions and self-occlusions. We follow Wang et al. (2021a) to
split the training and test sets. The HomebrewedDB dataset Kaskman et al.
(2019) provides newly captured test images of three objects in the LineMOD,
including bvise, driller and phone. Following the Self-6D Wang et al. (2020),
the second sequence of HomebrewedDB is used to test our models which are
trained on the LineMOD, to evaluate the robustness of our method on different
variations, _e.g._ , scene layouts and camera intrinsics. In the experiments,
the above three real-world datasets are considered as the target domains, all
of which share the same synthetic source domain. We employ the publicly
available synthetic data provided by BOP challenge Hodaň et al. (2020) as the
source data, which contains 50k images generated by physically-based rendering
(PBR) Hodaň et al. (2019).

#### Evaluation Metrics.

Following Wang et al. (2020), we employ the Average Distance of model points
(ADD) Hinterstoisser et al. (2012) as the evaluation metric of the 6D poses
for asymmetric objects, which measures the average deviation of the model
point set $\mathcal{O}$ transformed by the estimated pose
$\mathcal{T}=[\bm{R}|\bm{t}]$ and that transformed by the ground-truth pose
$\tilde{\mathcal{T}}=[\tilde{\bm{R}}|\tilde{\bm{t}}]$:

$\mathcal{D}_{\text{ADD}}(\mathcal{T},\tilde{\mathcal{T}})=\frac{1}{|\mathcal{O}|}\sum_{\bm{x}\in\mathcal{O}}\|(\bm{R}\bm{x}+\bm{t})-(\tilde{\bm{R}}\bm{x}+\tilde{\bm{t}})\|_{2}.$
(12)

For symmetric objects, we employ the metric of Average Distance of the closest
points (ADD-S) Hodaň et al. (2016):

$\mathcal{D}_{\text{ADD-S}}(\mathcal{T},\tilde{\mathcal{T}})=\frac{1}{|\mathcal{O}|}\sum_{\bm{x}_{1}\in\mathcal{O}}\min_{\bm{x}_{2}\in\mathcal{O}}\|(\bm{R}\bm{x}_{1}+\bm{t})-(\tilde{\bm{R}}\bm{x}_{2}+\tilde{\bm{t}})\|_{2}.$
(13)

Combining (12) and (13), we report the Average Recall ($\%$) of ADD(-S) less
than $10\%$ of the object’s diameter on all the three datasets.

#### Implementation Details.

For object detection, we use Mask R-CNN He et al. (2017) trained purely on
synthetic PBR images to generate the object bounding boxes for the target real
data. For pose estimation, we set the numbers of anchors as
$N_{\bm{R}}=60,N_{v_{x}}=N_{v_{y}}=20,N_{z}=40$, and set the ranges of
$v_{x}$, $v_{y}$ and $z$ as
$[d_{v_{x}}^{min},d_{v_{x}}^{max}]=[d_{v_{y}}^{min},d_{v_{y}}^{max}]=[-200,200]$,
and $[d_{z}^{min},d_{z}^{max}]=[0.0,2.0]$, respectively. To train our network,
we choose $\theta^{\bm{R}}_{1}=0.7,\theta^{\bm{R}}_{2}=0.1$ and $k_{\bm{R}}=4$
for rotation in (4), and also set
$\theta^{v_{x}}_{1}=\theta^{v_{y}}_{1}=\theta^{z}_{1}=0.55,\theta^{v_{x}}_{2}=\theta^{v_{y}}_{2}=\theta^{z}_{2}=0.075$,
and $k_{v_{x}}=k_{v_{y}}=k_{z}=7$ for translation. Following the popular
setting Wang et al. (2021a), we train individual networks for all the objects
with the Adam optimizer Kingma and Ba (2014). The teacher model
$\mathcal{M}_{T}$ is firstly pre-trained on the synthetic images of all
objects, and then fine-tuned on the single object, while the parameters of the
student model $\mathcal{M}_{S}$ is initialized as those of $\mathcal{M}_{T}$;
their initial learning rates are $3\times 10^{-4}$ and $3\times 10^{-5}$,
respectively. The training batch size is set as $B=32$. We also include the
same data augmentation as Labbé et al. (2020) during training.

###  4.1 Comparative Evaluation

We compare our method with the existing ones on three benchmarks for 6D object
pose estimation with RGB images.

On the LineMOD, we conduct experiments under three settings of training data,
including 1) labeled synthetic data, 2) labeled synthetic and real data, and
3) labeled synthetic data and unlabeled real data. Results of the first two
settings are the lower and upper bounds of that of the last setting. We report
qualitative results of comparative methods in Table 4, where our method
outperforms its competitors by large margins under all the settings, _e.g._ ,
with the respective improvements of $4.7\%$, $3.5\%$ and $1.6\%$ over the
state-of-the-art Self6D++ Wang et al. (2021a). On the Occluded LineMOD and the
HomebrewedDB, results are shown in Table 2, where our method consistently
performs better than the existing ones on both datasets, demonstrating
superior robustness of our method against occlusion and the generalization to
new scenes and cameras.

###  4.2 Ablation Studies and Analyses

#### Effects of Decomposing into Coarse Classification and Fine Regression.

We decompose the problem of UDA on estimating object poses into a coarse
classification on discretized anchors and a residual regression. As shown in
Table 3, for the models trained purely on synthetic data, the design of pose
decomposition realizes $4.0\%$ and $5.7\%$ improvements on the LineMOD and the
Occluded LineMOD, respectively, compared to direct regression of object poses,
since the global classification eases the difficulty in learning along with
feature-scaling robustness, and the local regression achieves pose refinement.

3 Pose Estimator | $\mathcal{L}_{ctc}$ | Method of UDA | Dataset
---|---|---|---
LM | LMO
Data: syn (w/ GT)
Reg. | $\times$ | - | 75.3 | 44.0
Cls. + Reg. | $\times$ | - | 79.3 | 49.7
Cls. + Reg. | ✓ | - | 82.1 | 55.3
Data: syn (w/ GT) + real (w/o GT)
Cls. + Reg. | ✓ | RSD Chen et al. (2021) | 83.9 | 55.0
Cls. + Reg. | $\times$ | Self-Training | 85.6 | 60.1
Cls. + Reg. | ✓ | Self-Training | 87.2 | 61.4
3 |  |  |  |

Table 3: Ablation studies on LineMOD (LM) and Occluded LineMOD (LMO) datasets
w.r.t. the Average Recall (%) of ADD(-S). ‘Reg.’ denotes direct regression of
object poses, while ‘Cls. + Reg.’ denotes the combined use of coarse
classification and fine regressor for pose estimation.

#### Effects of Cumulative Target Correlation Regularization.

As shown in Table 3, $\mathcal{L}_{ctc}$ consistently improves the results
under different settings across different datasets, _e.g._ , $5.6\%$
improvement on the Occluded LineMOD for the model trained on synthetic data,
which demonstrates the effectiveness of $\mathcal{L}_{ctc}$ on mining latent
correlation across regression targets. We also visualize the feature
distribution of an example via the t-SNE Van der Maaten and Hinton (2008) in
Fig. 1, where, with $\mathcal{L}_{ctc}$, features assigned to different pose
anchors preserve smooth and continuously changing nature of regression targets
in the feature space.

#### Effects of Manifold-Aware Self-Training on Coarse Classification.

The self-training schemes have been verified their effectiveness on reducing
the Sim2Real domain gap by incorporating the unlabeled real data into training
via pseudo label generation and training sample selection. Taking our network
with $\mathcal{L}_{ctc}$ as example, the results are improved from $55.3\%$ to
$61.4\%$ on the Occluded LineMOD via self-training. Compared to the RSD Chen
et al. (2021) designed for the problem of UDA on regression, our MAST scheme
can significantly beat the competing RSD (see results in Table 3), where the
only difference lies in replacing self-training on coarse classification with
RSD on whole regression. Such an observation can again confirm the superiority
of the proposed MAST scheme, consistently outperforming the state-of-the-art
UDA on regression.

#### On More Visual Regression Tasks.

We conduct more experiments on the dSprites dataset Matthey et al. (2017) for
assessing UDAVR performance. For simplicity, the problem aims to regress the
”scale” variable of a shape from images. Using the same backbone as RSD, under
the UDA setting from the scream (S) domain to the noisy (N) domain, our MAST
can achieve 0.024 in terms of mean absolute error, while the RSD only obtains
0.043.

#### Run-time analysis.

On a server with NVIDIA GeForce RTX 3090 GPU, given a 640 $\times$ 480 image,
the run-time of our network is up to 5.8 ms/object including object detection
and pose estimation when using Mask R-CNN He et al. (2017) as detector. Pose
estimation takes around 5 ms/object.

#### Details of output pose.

We employ a render-and-compare style pose refinement process as Labbé et al.
(2020) to get final object pose. An initial guess pose
$[\bm{R}_{init},x_{init},y_{init},z_{init}]$ is calculated from bounding box
and object CAD model using the same strategy as Labbé et al. (2020). Given the
network output $[\bm{R},x,y,z]$, the estimated object pose
$[\bm{R}_{obj},x_{obj},y_{obj},z_{obj}]$ can be calculated by:

$\left\\{\begin{aligned} \bm{R}_{obj}&=\bm{R}\cdot\bm{R}_{init}\\\
x_{obj}&=x+x_{init}\\\ y_{obj}&=y+y_{init}\\\ z_{obj}&=z\cdot
z_{init}\end{aligned}\right.,$ (14)

#### On selecting samples with pseudo pose labels.

We choose the probability $\bm{S}_{z}$ as confidence scores in practice, Fig.
3 shows the average recall of selected samples with pseudo pose labels via
$\bm{S}_{\bm{R}}$, $\bm{S}_{v_{x}}$, $\bm{S}_{v_{y}}$, $\bm{S}_{z}$, which
tells that as the confidence threshold becomes larger, only red line
($\bm{S}_{z}$) grows in terms of the average recall while others remain
unchanged or decreased.

Figure 3: The effect of selecting samples via $\bm{S}_{\bm{R}}$,
$\bm{S}_{v_{x}}$, $\bm{S}_{v_{y}}$, $\bm{S}_{z}$ confidence w.r.t. the recall
of the ADD(-S) among selected samples on driller object of LineMOD dataset
training set.

##  5 Conclusion

This paper proposes a novel and generic manifold-aware self-training scheme
for UDA on regression, which is applied to the challenging 6D pose estimation
of object instances. We address the UDAVR problem via decomposing it into
coarse classification and fine regression, together with a cumulative target
correlation regularization. Experiment results on three popular benchmarks can
verify the effectiveness of our MAST scheme, outperforming the state-of-the-
art methods with significant margins. It is worth pointing out that our MAST
scheme can readily be applied to any UDA regression tasks, as the UDA on
coarse classification making our method robust against feature scaling while
maintaining latent cumulative correlation underlying in regression target
space.

## Acknowledgments

This work is supported in part by the National Natural Science Foundation of
China (Grant No.: 61902131), the Guangdong Youth Talent Program (Grant No.:
2019QN01X246), the Guangdong Basic and Applied Basic Research Foundation
(Grant No.: 2022A1515011549), the Program for Guangdong Introducing Innovative
and Enterpreneurial Teams (Grant No.: 2017ZT07X183), and the Guangdong
Provincial Key Laboratory of Human Digital Twin (Grant No.: 2022B1212010004).

## References

  * Bao et al. [2022] Yiwei Bao, Yunfei Liu, Haofei Wang, and Feng Lu.  Generalizing gaze estimation with rotation consistency.  In CVPR, 2022.
  * Brachmann et al. [2014] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother.  Learning 6d object pose estimation using 3d object coordinates.  In ECCV, 2014.
  * Chen et al. [2011] Minmin Chen, Kilian Q Weinberger, and John Blitzer.  Co-training for domain adaptation.  NeurIPS, 2011.
  * Chen et al. [2013] Ke Chen, Shaogang Gong, Tao Xiang, and Chen Change Loy.  Cumulative attribute space for age and crowd density estimation.  In CVPR, 2013.
  * Chen et al. [2017] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.  Multi-view 3d object detection network for autonomous driving.  In CVPR, 2017.
  * Chen et al. [2021] Xinyang Chen, Sinan Wang, Jianmin Wang, and Mingsheng Long.  Representation subspace distance for domain adaptation regression.  In ICML, 2021.
  * Chen et al. [2022a] Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li.  Epro-pnp: Generalized end-to-end probabilistic perspective-n-points for monocular object pose estimation.  In CVPR, 2022.
  * Chen et al. [2022b] Yongwei Chen, Zihao Wang, Longkun Zou, Ke Chen, and Kui Jia.  Quasi-balanced self-training on noise-aware synthesis of object point clouds for closing domain gap.  In ECCV, 2022.
  * Collet et al. [2011] Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa.  The moped framework: Object recognition and pose estimation for manipulation.  The international journal of robotics research, 2011.
  * Fischler and Bolles [1981] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 1981.
  * Gao et al. [2018] Ge Gao, Mikko Lauri, Jianwei Zhang, and Simone Frintrop.  Occlusion resistant object rotation regression from point cloud segments.  In ECCV Workshops, 2018.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * Gopalan et al. [2011] Raghuraman Gopalan, Ruonan Li, and Rama Chellappa.  Domain adaptation for object recognition: An unsupervised approach.  In ICCV, 2011.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * He et al. [2020] Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun.  Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation.  In CVPR, 2020.
  * He et al. [2021] Yisheng He, Haibin Huang, Haoqiang Fan, Qifeng Chen, and Jian Sun.  Ffb6d: A full flow bidirectional fusion network for 6d pose estimation.  In CVPR, 2021.
  * Hinterstoisser et al. [2012] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab.  Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes.  In ACCV, 2012.
  * Hodaň et al. [2016] Tomáš Hodaň, Jiří Matas, and Štěpán Obdržálek.  On evaluation of 6d object pose estimation.  In ECCV, 2016.
  * Hodaň et al. [2019] Tomáš Hodaň, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta N Sinha, and Brian Guenter.  Photorealistic image synthesis for object instance detection.  In ICIP, 2019.
  * Hodaň et al. [2020] Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas.  BOP challenge 2020 on 6D object localization.  ECCV Workshops, 2020.
  * Kaskman et al. [2019] Roman Kaskman, Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic.  Homebreweddb: Rgb-d dataset for 6d pose estimation of 3d objects.  In ICCV Workshops, 2019.
  * Kehl et al. [2017] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab.  Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again.  In ICCV, 2017.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * Labbé et al. [2020] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic.  Cosypose: Consistent multi-view multi-object 6d pose estimation.  In ECCV, 2020.
  * Lee and others [2013] Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on challenges in representation learning, ICML, 2013\.
  * Lee et al. [2022] Taeyeop Lee, Byeong-Uk Lee, Inkyu Shin, Jaesung Choe, Ukcheol Shin, In So Kweon, and Kuk-Jin Yoon.  Uda-cope: Unsupervised domain adaptation for category-level object pose estimation.  In CVPR, 2022.
  * Li et al. [2018a] Chi Li, Jin Bai, and Gregory D Hager.  A unified framework for multi-view multi-class object pose estimation.  In ECCV, 2018.
  * Li et al. [2018b] Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox.  Deepim: Deep iterative matching for 6d pose estimation.  In ECCV, 2018.
  * Li et al. [2019] Zhigang Li, Gu Wang, and Xiangyang Ji.  Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation.  In ICCV, 2019.
  * Li et al. [2021] Zhigang Li, Yinlin Hu, Mathieu Salzmann, and Xiangyang Ji.  Sd-pose: Semantic decomposition for cross-domain 6d object pose estimation.  In AAAI, 2021.
  * Lin et al. [2022] Jiehong Lin, Zewei Wei, Changxing Ding, and Kui Jia.  Category-level 6d object pose and size estimation using self-supervised deep prior deformation networks.  In ECCV, 2022.
  * Manhardt et al. [2018] Fabian Manhardt, Wadim Kehl, Nassir Navab, and Federico Tombari.  Deep model-based 6d pose refinement in rgb.  In ECCV, 2018.
  * Manhardt et al. [2019] Fabian Manhardt, Diego Martin Arroyo, Christian Rupprecht, Benjamin Busam, Tolga Birdal, Nassir Navab, and Federico Tombari.  Explaining the ambiguity of object detection and 6d pose from visual data.  In ICCV, 2019.
  * Marchand et al. [2015] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler.  Pose estimation for augmented reality: a hands-on survey.  IEEE transact. on visualization and computer graphics, 2015.
  * Matthey et al. [2017] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner.  dsprites: Disentanglement testing sprites dataset, 2017.
  * Park et al. [2019] Kiru Park, Timothy Patten, and Markus Vincze.  Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation.  In ICCV, 2019.
  * Peng et al. [2019] Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao.  Pvnet: Pixel-wise voting network for 6dof pose estimation.  In CVPR, 2019.
  * Roy et al. [2019] Subhankar Roy, Aliaksandr Siarohin, Enver Sangineto, Samuel Rota Bulo, Nicu Sebe, and Elisa Ricci.  Unsupervised domain adaptation using feature-whitening and consensus loss.  In CVPR, 2019.
  * Sock et al. [2020] Juil Sock, Guillermo Garcia-Hernando, Anil Armagan, and Tae-Kyun Kim.  Introducing pose consistency and warp-alignment for self-supervised 6d object pose estimation in color images.  In 3DV, 2020.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  NeurIPS, 2020.
  * Su et al. [2015] Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas.  Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views.  In ICCV, 2015.
  * Sundermeyer et al. [2020] Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, and Rudolph Triebel.  Augmented autoencoders: Implicit 3d orientation learning for 6d object detection.  IJCV, 2020.
  * Tang et al. [2012] Kevin Tang, Vignesh Ramanathan, Li Fei-Fei, and Daphne Koller.  Shifting weights: Adapting object detectors from image to video.  NeurIPS, 2012.
  * Tekin et al. [2018] Bugra Tekin, Sudipta N Sinha, and Pascal Fua.  Real-time seamless single shot 6d object pose prediction.  In CVPR, 2018.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 2008.
  * Wang et al. [2020] Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, and Federico Tombari.  Self6d: Self-supervised monocular 6d object pose estimation.  In ECCV, 2020.
  * Wang et al. [2021a] Gu Wang, Fabian Manhardt, Xingyu Liu, Xiangyang Ji, and Federico Tombari.  Occlusion-aware self-supervised monocular 6d object pose estimation.  IEEE TPAMI, 2021.
  * Wang et al. [2021b] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji.  Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation.  In CVPR, 2021.
  * Xiang et al. [2018] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  RSS, 2018.
  * Xie et al. [2020] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In CVPR, 2020.
  * Xu et al. [2019] Zelin Xu, Ke Chen, and Kui Jia.  W-posenet: Dense correspondence regularized pixel pair pose regression.  arXiv preprint arXiv:1912.11888, 2019.
  * Xu et al. [2022] Yan Xu, Kwan-Yee Lin, Guofeng Zhang, Xiaogang Wang, and Hongsheng Li.  Rnnpose: Recurrent 6-dof object pose refinement with robust correspondence field estimation and pose optimization.  In CVPR, 2022.
  * Yang et al. [2021] Zongxin Yang, Xin Yu, and Yi Yang.  Dsc-posenet: Learning 6dof object pose estimation via dual-scale consistency.  In CVPR, 2021.
  * Yue et al. [2021] Xiangyu Yue, Zangwei Zheng, Shanghang Zhang, Yang Gao, Trevor Darrell, Kurt Keutzer, and Alberto Sangiovanni Vincentelli.  Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation.  In CVPR, 2021.
  * Zakharov et al. [2019] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic.  Dpod: 6d pose object detector and refiner.  In ICCV, 2019.
  * Zhang et al. [2021] Shaobo Zhang, Wanqing Zhao, Ziyu Guan, Xianlin Peng, and Jinye Peng.  Keypoint-graph-driven learning framework for object pose estimation.  In CVPR, 2021.
  * Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In CVPR, 2019.
  * Zou et al. [2018] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In ECCV, 2018.
  * Zou et al. [2021] Longkun Zou, Hui Tang, Ke Chen, and Kui Jia.  Geometry-aware self-training for unsupervised domain adaptation on object point clouds.  In ICCV, 2021.


11institutetext: Massachusetts Institute of Technology, Cambridge, USA
<EMAIL_ADDRESS>

Authors’ Instructions

# Dynamic Urban Planning: an Agent-Based Model Coupling Mobility Mode and
Housing Choice. Use case Kendall Square.

Mireia Yurrita (✉) Arnaud Grignard  Luis Alonso  Yan Zhang
Cristian Jara-Figueroa  Markus Elkatsha  Kent Larson

###### Abstract

As cities become increasingly populated, urban planning plays a key role in
ensuring the equitable and inclusive development of metropolitan areas. MIT
City Science group created a data-driven tangible platform, CityScope, to help
different stakeholders, such as government representatives, urban planners,
developers, and citizens, collaboratively shape the urban scenario through the
real-time impact analysis of different urban interventions. This paper
presents an agent-based model that characterizes citizens’ behavioural
patterns with respect to housing and mobility choice that will constitute the
first step in the development of a dynamic incentive system for an open
interactive governance process. The realistic identification and
representation of the criteria that affect this decision-making process will
help understand and evaluate the impacts of potential housing incentives that
aim to promote urban characteristics such as equality, diversity, walkability,
and efficiency. The calibration and validation of the model have been
performed in a well-known geographic area for the Group: Kendall Square in
Cambridge, MA.

###### Keywords:

Agent-based modelling $\cdotp$ Housing choice $\cdotp$ Residential Mobility
$\cdotp$ Dynamic Urban Planning $\cdotp$ Pro-social City Development

##  1 Introduction

Even if humans started to cluster around cities 5,500 years ago, it was not
until about 150 years ago that cities became increasingly populated [1].
Nowadays more than 50% of the world’s population lives in urban areas, which
makes urban planning emerge as one of the key elements for citizens’ well-
being [2]. As in the following decades cities may account for 60% of the total
global population, 75% of the global $C0_{2}$ emissions, and up to 80% of all
world energy use [3], traditional systems of centralised planning might be
obsolete [2].

Figure 1: CityScope Volpe: a data-driven interactive simulation tool for urban
design [2]

In view of the need for more participatory decision-making processes, MIT City
Science (CS) Group developed a tangible data-driven platform, CityScope. This
platform brings different stakeholders together in an effort to enable a more
collaborative urban design process [2] [4] –Figure 1–. As part of this
collective decision-making approach, the CS Group is designing a dynamic
incentive system, that aims to promote pro-social urban features through
dynamic incentive policies. By using data-driven agent-based simulations, the
impacts on equality, diversity, walkability, and efficiency of such incentives
are measured. Thanks to this information, different stakeholders, such as
government representatives, urban planners, developers, and citizens, can
collaboratively decide which intervention shapes the most favourable urban
scenario.

Some of the incentives explored in this line of research are focused on the
promotion of affordable housing within a 20-minute walking distance from one’s
workplace. This vision is part of a broader perspective where cities are
encouraged to be comprised of autonomous communities, where citizens’ daily
needs are met within a walking distance from their doorstep. MIT City Science
intends to create the technological tools to facilitate consensus between
community members, local authorities, and design professionals [2], so that
cities can fulfil their ambition of having walkable districts.

This paper arises from the need to forecast citizens’ criteria when choosing
their homes, and the consequent mobility patterns. By doing so the effects of
numerous pro-social incentives that aim to address the deficiency of
affordable housing in city centres can be tested. Although some previous
studies focusing on citizens’ mobility choices had already been held in the
group, the need to combine mobility trends with housing-related aspects was
major. In contrast to the consulted bibliography, where most approaches make
use of statistical tools to respond to this joint choice research question,
Agent-Based Modeling (ABM) has been deployed in this study. People’s
behavioral patterns have been represented based on their income profiles.

This document is organized as follows. Section 2 describes previous work on
the representation of housing and residential mobility choices and highlights
the benefits of using ABM when embracing the complexity of urban dynamics.
Section 3 gives a detailed summary of the agents that constitute the model,
the behaviours of each of them and the dynamics of the housing and mobility
mode choice. Section 4 calibrates the model for the particular use case of
Kendall Square using census and transportation data and section 6 gives some
final thoughts on the effectiveness of the method.

##  2 Background

###  2.1 Related Work

The present research is an agent-based joint representation of housing and
mobility choices that intends to forecast citizens’ behavioural patterns when
choosing their residencies as well as to highlight impacts on mobility. This
study has been built on top of previous work where different approaches to
address this question have been suggested. Some of them merely focused on the
housing choice search, others concentrated on the mobility mode issue and some
merged both points of view.

Among the former methodologies, a two-stage behavioural housing search model
[5] had been created. This model was comprised of a hazard-based “choice set
formation step”, and a “final residential location selection step” using a
multinomial logit formulation. A theoretical housing preference and choice
framework had also been given using the theory of mean-end chain [6]. The
motivations that make people want to move had been studied [7] using a nested
logit model, which paid special attention to how residential decisions are
impacted by transportation. A multi-agent approach had also been used to model
and analyze the residential segregation phenomenon [8]. And an ABM had been
designed to represent the dynamics of the housing market and for the study of
the effects that certain parameters, like school accessibility, would have on
the outcome [9].

As far as mobility choices are concerned, some previous studies had been
carried out within the MIT City Science Group using both statistical tools as
well as agent-based models. The HCl platform deployed for the real-time
prediction of mobility choices through discrete choice models [10] is one of
the examples of how the CS Group had previously addressed the mobility choice
issue. Agent-based models had also been developed in our Group to evaluate new
mobility mode choices and to assess their impact on cities [11].

Among previous joint choice models, it is important to highlight a model based
on the disaggregate theory where the combinations of location, housing,
automobile ownership and the commuting mode had been studied [12].
Additionally, a nested multinomial logit model to estimate the joint choice of
residential mobility, and housing choice [13] had also been used.

This paper presents a novel joint choice model based on agents. Following the
Schelling segregation model [14], the factors related to housing, and those
related to transportation will be deployed to determine if a certain agent is
willing to move or not.

###  2.2 Agent-Based Simulations

Most of the related work mentioned prior in this paper make use of statistical
tools to study the housing and residential mobility choice phenomenon. Our
research, on the contrary, will artificially reproduce the dynamics of the
urban scenario using agent-based simulations. Although several platforms
dedicated to agent-based modelling have been created lately, this model has
been developed using the GAMA platform. GAMA allows modellers to create
complex multi-level spatially explicit simulations where GIS data can be
easily integrated. It also provides a high-level agent-oriented language,
GAML, as well as an integrated development environment [15][16]. The behaviour
of each ‘people agent’ has been developed using this tool and some agent-
specific functions have been designed in order to recreate their decision-
making reasoning while representing the complexity of the surrounding
dynamics.

##  3 Model description

This model aims to represent the criteria adopted by citizens when choosing
their residential location and mobility mode, as well as the importance given
to each of these parameters. The realistic characterization of these
behaviours enables the study of the consequences that certain housing
incentives and urban disruptions might entail.

###  3.1 Entities, state variables, and scales

The environmental variables used to shape the urban scenario and represent its
dynamics include:

  * •

Census block group: polygon representing the combination of census blocks,
smallest geographic areas for which the US Bureau of Census collects data
[17]. Each census block group will be formed by the following attributes:
GEOID: unique geographical ID for each census block group, vacant_spaces:
number of vacant housing options available within the boundaries of this
polygon (previously a web scraping process has been held and the availability
of accommodation and its price identified [18] within the area of interest),
city: broader unit the block group belongs to [17], rent_vacancy: mean rent
for each housing option based on online rent information [18], population: map
indicating the number of citizens of each income profile that actually live in
the area [19], has_T: boolean attribute pointing out whether the block group
has a T station or not [20] and has_bus: boolean attribute that indicates if
bus services are available in the area [20]. Census block groups have been the
environmental variables chosen to represent the broadest granularity in the
model, since applying the same unit as the US Census Bureau enables the
calibration and validation of the model using census data.

  * •

Building: polygon representing a finer granularity for the area where the
urban interventions are considered to take place. The aforementioned census
block groups are the alternative to living in a building within the area of
interest. The attributes of ‘building’ agents include: associated_block_group:
block group in which the building is located [17], vacant_spaces: number of
vacant dwelling units within the building [18] and rent_vacancy: mean rent for
each housing option within the building [18].

  * •

Road: network of roads that agents can use to move around. mobility_allowed is
the only attribute of the road network, which is taken into account when
considering different commuting mobility options and when the resulting time
for each of them is calculated [17].

Figure 2: Model overview, where People, Building, Road and Census Block Group
–areas with different transparency– agents have been illustrated for Kendall
Square in Cambridge, MA.

As far as agents are concerned, ‘people agents’ are defined the following way:

  * •

People: citizens who work in the area of interest and whose objective is to
find the most appropriate housing and mobility option based on their criteria.
Their attributes include: type: type of ‘people agent’ based on their annual
income profile [19], living_place: building or census block group where they
decide to live in each iteration of the searching process, activity_place:
building in the area of interest where each agent works,
possible_mobility_modes: list of mobility modes that are available for each
agent considering if they own some kind of private mobility mode [19] or if in
their living place there is public transportation available [20],
mobility_mode commuting mobility mode chosen in each iteration of the housing
search, time_main_activity: commuting time in minutes, distance_main_activity:
commuting distance depending on the housing option being considered and
commuting_cost: cost based on the distance and the mobility mode chosen.

###  3.2 Process overview

The dynamics of this model are ruled by the behaviour of ‘people agents’. Each
of them is assigned a nonresidential building in the finer grained area as
their workplace. An iterative process will be performed in order to find the
housing option and mobility mode that best meets their needs.

####  3.2.1 Iteration 0:

The initialisation process takes place in three different steps: (i) a random
residential unit is assigned to each agent, (ii) based on the area where this
residential unit is located and the probability of owning private means of
transportation, possible mobility modes are defined (iii) travelling options
will be weighed based on each profile’s criteria and a final score for each
mode will be calculated as a linear combination of the importance given to
each of these factors. The option that maximizes the score will be the chosen
mobility mode. Criteria related to transportation preferences include both
quantitative and qualitative factors such as price, resulting commuting time,
the difficulty of usage and the social pattern [11]. Table 1 displays some
synthetic values of this criteria for three medium income profiles [11]
(modified from working status profiles into income profiles), whereas table 2
shows the features of some of the available transportation modes [11] [20].

| Price [-] | Time [-] | Difficulty [-] | Pattern [-]
---|---|---|---|---
$45,000-$59,999 | -0.8 | -0.8 | -0.7 | 0.7
$60,000-$99,999 | -0.7 | -0.85 | -0.75 | 0.8
$100,000-$124,999 | -0.6 | -0.9 | -0.8 | 0.95
Table 1: Synthetic weighing values given to transportation criteria [11]
according to different income profiles. These values are normalized between -1
and 0 for price, time and difficulty and between 0 and 1 for pattern
importance. They will determine each ‘people agent’s ’ preference towards a
particular mobility choice.

Note that, in order to be able to make use of public transportation, both the
surroundings of the residential unit and the workplace have to offer that
particular commuting option. So as to calculate the time needed to commute for
each mode, both the waiting time (when non-private modes are considered) and
the commute itself –following the topology of the transportation network– have
been taken into account.

| Price/km [$] |  | Mean speed [km/h]
---
Pattern [-]
bike | 0.01 | 5 | 0.5
bus | 0.1 | 20 | 0.4
car | 0.32 | 30 | 1
Table 2: Price, speed and social pattern features of some of the available
mobility modes [11]. Price/km and mean speed values are based on the
Massachusetts Bay Transportation Authority data [20], whereas the pattern
weight is a synthetic normalized weighing value between 0 and 1. A linear
combination between these values and the mobility choice criteria for each
income profile is performed so as to select the most suitable transportation
option.

####  3.2.2 Subsequent iterations:

Once the initialisation process has been held, the procedure will change into:
(i) an alternative housing option is randomly assigned to each agent (ii) this
new housing option is compared to the current unit and the suitability of
moving to the new alternative is evaluated based on certain factors according
to each income profile (iii) the ‘time’ parameter is calculated as the
resulting commuting time required using the most suitable mobility option for
each housing unit as described for iteration 0. ‘People agents’ will decide to
move in iteration ‘i’ if the score of the alternative housing unit is greater
than the current one. Factors considered when assessing housing include
quantitative data such as price or commuting time using the most suitable
available means of transportation and qualitative factors like the zone
preference, importance given to the unit being in this zone and diversity
acceptance [9].

| Price [-] | Diversity acceptance [-] | Zone weight [-]
---|---|---|---
$45,000-$59,999 | -0.8 | 0 | 0.4
$60,000-$99,999 | -0.6 | -0.3 | 0.5
$100,000-$124,999 | -0.5 | -0.5 | 0.6
Table 3: Price, diversity acceptance and zone criteria according to different
income profiles (synthetic data based example). Values are normalized between
-1 and 0 for price, between -1 and 1 for diversity acceptance and between 0
and 1 for zone weight. These criteria, along with the transportation criteria
will point out the housing and mobility choice that maximizes the score
obtained through a linear combination.

Diversity is calculated using the Shannon-Weaver formula following the
diversity calculation methodology deployed on the CityScope platform [2]. This
measurement quantitatively measures the amount of species (different income
profiles in our research) in an ecosystem (the spatial unity being considered
in each case) [2]. This diversity metrics will be applied either to a
building, if the housing unit being considered is within the finer grained
area, or to a census block group should the dwelling be located on the
outskirts. This iterative process will be applied until the number of people
moving in each iteration asymptotically approaches zero. Once this situation
is reached and every ‘people agent’ is assigned the most convenient mobility
and housing option, further mobility studies can be performed departing from
this t=0 situation.

###  3.3 Initialization and input data

The model initialization relies on two main types of files: the ones
containing geographical information and the ones containing information
regarding agents and mobility modes. The first group of files includes (1) a
shapefile that consists of the block groups of the area in question, (2) a
shepefile where the availability of apartments according to each block group
is displayed, (3) a shapefile where the buildings of the finer grained area
are incorporated, (4) a shapefile that is comprised of the public
transportation system and (5) a shapefile with the road network of the region.
The second type of input data consists of .csv files where (1) income profiles
are defined and their characteristics such as the proportion within the
population, the probability of owning a car and a bike are listed [19], (2)
‘people’ profiles define the weight given to each of the criterion when
choosing a house –Table 3–, (3) these same profiles describe the criteria
regarding mobility mode preferences –Table 1– and (4) different mobility modes
are listed and their characteristics detailed –Table 2–, including the price
and time per distance unit, the waiting time, the difficulty, and the social
pattern.

##  4 Model calibration and validation. Use case Kendall Square

The model described above in this paper presents a generic framework where, as
long as the input data is modified, the methodology can be applied to any
city. In order to develop a first version of the model, Kendall Square has
been analysed. Kendall Square is a neighbourhood in Cambridge, Massachusetts,
–Figure 2– that has been transformed from an industrial area into a leading
biotech and IT company hub in the last decades. As the number of companies
clustering around this site has soared, the cost of housing has exploded,
making it increasingly difficult for low and medium income members to live
within a walking distance from their workplace [21]. The commuting patterns
derived from this situation, which include high private vehicle density and
saturation of the public transportation infrastructure, lead to serious
mobility challenges. This section is focused on the description of the
calibration process of ‘people agents’ ’ behavioural criteria, so that their
reactions to potential housing incentives can be characterized and the
consequences of their acts monitored. Input files (2) and (3) gather the
aforementioned criteria and will be, therefore, inferred from this validation
process, whereas file (1) is deduced from census data [19] and file (4) is an
adaptation of the Massachusetts Bay Transportation Authority information [20].

The calibration process is based on the definition of two types of errors: one
related to housing choices and the second one related to mobility mode
patterns. Batch experiments are then deployed so as to find the combination of
criteria that leads to the minimum value of the sum of both of these errors.
There are eight different factors that affect the decision-making process of
‘people agents’ –housing price, diversity acceptance, preferred zone and
weight given to zone [9]; transportation price, time importance, commuting
difficulty and social pattern [11]– which applied to eight different income
profiles, leads to a total of sixty four variables handled by batch
experiments.

The housing error is defined as the difference in percentage of the
distribution of each income profile in each census block group where citizens
working in Kendall have certain presence. The real percentage value inferred
from transportation data –the census block group where different Kendall
workers live can be identified and the income profile of these approximated to
the mean annual income based on census data [19]– is then compared to the
resulting simulated value and the root-mean-square deviation calculated
–Equation 1–.

$RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-\hat{Y}_{i})^{2}}$ (1)

$Y_{i}$ = real percentage of people of type i living in the neighbourhood in
question

$\hat{Y}_{i}$ = percentage of ‘people agents’ of type i living in that same
neighbourhood as a result of the housing selection methodology

As for the mobility mode error, the resulting simulated percentages of
pedestrians and car, bike, bus and T usage are compared to the Parking and
Transportation Demand Management Data in the City of Cambridge [22]. This
error is equally calculated as the root-mean-square deviation, where $Y_{i}$
represents the percentage of people that choose a certain mobility mode i,
whereas $\hat{Y}_{i}$ stands for the percentage of ‘people agents’ that choose
that same mobility mode once the iterative process has been performed.

As far as the exploration method is concerned, the hill climbing algorithm
[23] has been applied and batch experiments have been performed, getting a
reasonably good solution. The most suitable criteria combination has led to a
mean housing error of 3.87% and a mean mobility error of 2.30% –Fig 3–.

Figure 3: Evolution of housing and mobility errors [%] in a hundred of the
calibration process batch experiments.

##  5 Discussion

In the described model citizens’ housing and mobility preferences are scored
taking into account various parameters, which include price (for both housing
and commuting), diversity acceptance, zone preference, time, difficulty of
usage and social pattern. These parameters have been defined based on
bibliography, considering that they should constitute a reduced set of
variables that heavily affect the decision-making process. Should a finer-
grained study be held, it should be noted that the zone preference criterion
encompasses itself numerous possible attributes. The effect of greenery, high
quality education centers or fresh food availability, to mention some, could
also be included if a more detailed study is to be held. For a first version
model, though, it has been assumed that the suggested approach was already
complex enough. This same complexity, along with the metropolitan area scope,
makes it challenging for the impacts resulting from changes in urban
configurations to be calculated-real time, which represents a constraint when
it comes to computational cost. Finally, it should be noted that in order to
calibrate the criteria in question, the exploration of the design space has
been performed using the hill climbing algorithm. The minimization process has
resulted in acceptable housing and mobility errors. However, this algorithm
heavily relies on the selected starting point and, thus, alternative
algorithms could be deployed if this limitation is to be avoided in future
calibration processes.

##  6 Conclusion

This paper presents a generic methodology where citizens’ criteria towards
housing and mobility choices can be tested, as well as the resulting
behavioural patterns monitored. Criteria regarding mobility and residential
preferences have been calibrated and validated for the particular use case of
Kendall Square, where major mobility issues have arisen as a consequence of
the technological development of the surroundings and the explosion of housing
prices. However, variations in input data and research into combinations of
criteria that are particularised for a certain city are possible following the
guidelines given earlier in the document. As part of the CityScope platform,
this model enables the prediction of citizens’ response to housing-related
urban disruptions that aim to promote the pro-social development of cities.
The usage of Agent-Based Modelling opens the door to the monitorisation of
people’s reactions to dynamically reconfigurable zoning policies. However, the
deployment of such a data-driven platform requires of real-time feedback, so
that different stakeholders can benefit from the instant impact analysis of
the suggested actions. Further research regarding the conversion of this model
into a real-time analysis tool is being held. This new approach will need to
embrace the complexity of urban dynamics, while constituting an agile and
dynamic decision-making tool.

## References

  * [1] Sjoberg, G.: The Origin and Evolution of Cities. In: Scientific American, Vol 213, No. 3. Scientific American, a division of Nature America, Inc (1965).
  * [2] Alonso, L., Zhang, Y., Grignard, A., Noyman, A., Sakai, Y., ElKatsha, M., Doorley, R., Larson, K.: CityScope: a data-driven interactive simulation tool for urban design. Use case volpe. In: Morales, A.J., Gershenson, C., Braha, D., Minai, A.A., Bar-Yam, Y. (eds.) ICCS 2018. SPC, pp. 253–261. Springer, Cham (2018).
  * [3] United Nations. Sustainable Development Goals (2020). https://www.un.org/sustainabledevelopment/cities/
  * [4] Grignard, A., Macià, N., Alonso Pastor, L., Noyman, A., Zhang, Y., Larson, K.: Cityscope Andorra: A multi-level interactive and tangible agent-based visualization. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1939-1940 (2018).
  * [5] Rashidi, T., Auld, J., Mohammadian, A.: A behavioral housing search model: Two-stage hazard-based and multinomial logit approach to choice-set formation and location selection. In: Transportation Research Part A 46, pp. 1097-1107 Elsevier Ltd. (2012).
  * [6] Zinas, b., Mohd Jusan, M.: Housing Choice and Preference: Theory and Measurement. In: 1st National Conference on Environment-Behaviour Studies. Faculty of Architecture, Planning & Surveying, Universiti Teknologi MARA, Shah Ala,, Selangor, Maysia, 14-15 November 2009, pp. 282-292 (2012).
  * [7] Kim, J., Pagliara, F., Preston, J.: The Intention to Move and Residential Location Choice Behaviour. In: Urban Studies, Vol. 42, No.9, pp. 1621-1636 (2005).
  * [8] Aguilera, A., Ugalde, E.: A Spatially Extended Model for Residential Segregation. In: Discrete Dynamics in Nature and Society, Volume 2007, Article ID 48589. Hindawi Publishing Corporation (2007).
  * [9] Jordan, R., Birkin, M., Evans, A.: Agent-Based Modelling of Residential Mobility, Housing Choice and Regeneration. In: Heppenstall A., Crooks A., See L., Batty M. (eds) Agent-Based Models of Geographical Systems. Springer, Dordrecht (2012).
  * [10] Doorley, R., Noyman, A., Sakai, Y., Larson, K.: What’s your MoCho? Real-time Mode Choice Prediction Using Discrete Choice Models and a HCl Platform. In: Urbcomp ’19, August 5, 2019, Anchorage, AK (2019).
  * [11] Grignard, A., Alonso, L., Taillandier, P., Gaudou, B., Nguyen-Huu, T., Gruel, W., Larson, K.: The impact of new mobility modes on a city: a generic approach using ABM. In: Morales, A.J., Gershenson, C., Braha, D., Minai, A.A., Bar-Yam, Y. (eds.) ICCS 2018. SPC, pp. 272–280. Springer, Cham (2018).
  * [12] Lerman, S.: Location, Housing, Automobile Ownership, and Mode to Work: A Joint Choice Model. In: Transportation Research Record, pp. 6-11. Transportation Research Board (1977).
  * [13] Clark, W., Onaka, J.: An empirical test of a joint model of residential mobility and housing choice. In: Environment and Planning A: Economy and Space, 17(7), pp. 915-930 (1985).
  * [14] Schelling, T.: Models of Segregation. In: The American Economic Review, Vol 59, No. 2, pp. 488-493. Papers and Proceeding of the Eighty-first Annual Meeting of the American Economic Association (1969).
  * [15] Grignard, A., Taillandier, P., Gaudou, B., Vo, D.A., Huynh, N.Q., Drogoul, A.: GAMA 1.6: advancing the art of complex agent-based modeling and simulation. In: Boella, G., Elkind, E., Svarimuthu, B.T.R., Dignum, F., Purvis, M.K. (eds.) PRIMA 2013. LNCS (LNAI), vol 8291, pp. 117-131. Springer, Heidelberg (2013).
  * [16] Taillandier, P., Gaudou, B., Grignard, A., Huynh, Q. N., Marilleau, N., Caillou, P., Philippon, D., Drogoul, A.: Building, composing and experimenting complex spatial models with the GAMA platform. GeoInformatica, 23(2), pp. 299-322 (2019).
  * [17] United States Government’s Open Data (2020). https://www.data.gov/
  * [18] PadMapper. Apartments for Rent from the Trusted Apartment Finder (2020). https://www.padmapper.com/
  * [19] United States Census Bureau. Census Profiles (2020). https://data.census.gov/cedsci/
  * [20] MBTA. Massachusetts Bay Transportation Authority (2020). https://www.mbta.com/
  * [21] Patrick Sisson. As top innovation hub expands, can straining local infrastructure keep pace? (2018) https://www.curbed.com/2018/11/6/18067326/boston-real-estate-cambridge-mit-biotech-kendall-square
  * [22] Parking and Transportation Demand Management Data in the City of Cambridge (2014). https://www.cambridgema.gov/CDD/Transportation/fordevelopers/ptdm
  * [23] Russell, J., Norvig, P.: Artificial Intelligence. A Modern Approach. Prentice-Hall, USA (1995).