# Generative Models for Multi-Illumination Color Constancy

Partha Das1,2
<EMAIL_ADDRESS>Yang Liu2
<EMAIL_ADDRESS>Sezer Karaoglu1,2
<EMAIL_ADDRESS>Theo Gevers1,2
<EMAIL_ADDRESS>University of Amsterdam, The Netherlands1 3DUniversum,
Amsterdam, The Netherlands2

###### Abstract

In this paper, the aim is multi-illumination color constancy. However, most of
the existing color constancy methods are designed for single light sources.
Furthermore, datasets for learning multiple illumination color constancy are
largely missing. We propose a seed (physics driven) based multi-illumination
color constancy method. GANs are exploited to model the illumination
estimation problem as an image-to-image domain translation problem.
Additionally, a novel multi-illumination data augmentation method is proposed.
Experiments on single and multi-illumination datasets show that our methods
outperform sota methods.

##  1 Introduction

The color of an object is influenced by the illumination color. The process of
recovering the original object color, independent of the illumination color,
is called Color constancy. Previous approaches solve the problem by enforcing
various priors [31, 12, 37]. More recently, with the availability of
relatively large scale datasets [16, 19, 4, 3], learning based approaches are
explored [33, 9]. Various learning based approaches are proposed such as
local-global patch correction [25] and exemplar patch based correction [30],
that show robust performance. However, these methods predict the illumination
vector directly. This limits the learning capability of the network by
limiting all the pixels to be supervised by the illumination vector. Modelling
the problem as an image-to-image translation could provide a solution to this,
allowing finer pixel wise supervision. However, standard encoder-decoder
network works by minimising the Euclidean distance between the prediction and
the ground truth. This may lead to blurry outputs and border artefacts, while
being fully dependent only on the input.

Generative Adversarial Networks [26] (GAN) solve this problem by learning an
image-to-image translation as a domain transformation. In this setting, the
generator learns an expanded pixel-wise supervision, while the discriminator
allows the network to learn domain specific information. To exploit the
unified framework, we propose to use GANs to model color constancy as a domain
transformation, instead of image to illumination prediction.

However, the previous methods are often limited to a single illumination
assumption. For multi-illumination scenes, each pixel in an image is
influenced by different light sources. GAN models like Pix2Pix or CycleGAN
learn a global transformation. Even with pixel wise supervision, Pix2Pix
learns a pixel wise transformation, excluding the spatial context. In
contrast, we propose a hybrid learning and seed based multi-illumination color
constancy. Using physics based (initial) seed points, the network is provided
with illumination regions. The network then diffuses these seed points over
the image for a multi-illumination prediction. To model mixtures of lights at
a pixel, an illumination probability is predicted. This allows the model to
compute per-pixel mixture probability belonging to the different
illuminations. Finally, due to the lack of multi-illumination datasets, a data
augmentation method is proposed to create a new multi-illumination dataset. In
summary, the proposed contributions of this work are as follows:

  * •

We propose to model the problem of color constancy as a domain transformation
with GANs.

  * •

We propose a new method for exploiting physics based initial seed points for
multi-illumination color constancy.

  * •

We propose a novel data augmentation method to create multi-illumination
dataset using off-the-shelf methods and single illumination datasets.

##  2 Related Work

Color constancy methods can be classified into optimization-based methods and
learning based methods.

Optimization based methods: These methods exploit various low-level image
statistics or priors [31, 12, 22, 28, 20, 42]. New features are proposed in
[37] tackling both single and multi-illumination in a learning free manner.
Obviously, these methods may fail when the underlying assumptions/priors are
violated.

Learning based methods: Various learning based methods are proposed such as
[39, 21, 10, 23, 8, 14] using classical learning methods like Bayesian
frameworks, Support Vector Machines, Decision Forests, and Conditional Random
Fields. Exemplar based methods [30] explore multi-illumination settings using
pixel independent features. With the availability of larger datasets, CNNs are
proposed [33, 11, 35, 27]. These methods employ deep learning frameworks in an
end-to-end manner to model the problem as an illumination prediction. While
these previous works focused on a single image, [36] integrates an implicit
illumination constraint by using multiple frames and a deep recurrent network.
[1], on the other hand, uses a two-camera-one-subject setup to estimate the
illumination. Finally, [41, 40] formulated the problem into an image-to-image
transformation. However, the methods suffer from dataset illumination bias.

In contrast to previous work, this paper proposes to approach the problem of
color constancy as a domain transformation. Following our work [17], different
GANs are first studied for the problem of single illumination color constancy.
Their shortcomings for a multi-illumination setting are analyzed. These
observations are then employed to propose an end-to-end hybrid multi-
illumination color correction pipeline.

##  3 Methodology

###  3.1 Color Constancy

The problem of color constancy can be defined as a process of chromatic
adaptation [18]:

$I=W*L\;,$ (1)

where, $W$ is the image under a canonical white illumination and $L$ is the
unknown colored illumination. The $*$ operation denotes the pixel-wise
multiplication between the components. Thus, $L$ acts as a scaling term on $W$
to generate the final $I$. All the components are in linear $RGB$. For a
single light source, L is uniformly constant, while for multi-illumination L
is varied.

The process of color correction can thus be approximated as modifying the
gains of the $RGB$ channels independently. To recover the color corrected
image ($W$), the Von Kries [44] method can be used as follows:

$\displaystyle\begin{pmatrix}R_{c}\\\ G_{c}\\\
B_{c}\end{pmatrix}=\begin{pmatrix}L_{R}&0&0\\\ 0&L_{G}&0\\\
0&0&L_{B}\end{pmatrix}\begin{pmatrix}R_{u}\\\ G_{u}\\\ B_{u}\end{pmatrix}\;,$
(2)

where $R_{u}$, $G_{u}$, $B_{u}$ denote $RGB$ channels of an image under an
unknown colored illumination source, $R_{c}$, $G_{c}$, $B_{c}$ represent $RGB$
channels under a canonical (corrected) illumination, and $L_{R}$, $L_{G}$ and
$L_{B}$ correspond to the unknown illumination color.

Traditional approaches directly recover the illumination color from the input
image. This forces the network to use the same supervision and “copy” it over
all the pixels. In contrast, it is proposed to model the process of color
constancy as domain mapping, i.e. from color biased ($I$) image to color
corrected ($W$) image. This allows the network to be flexible to both single
illumination and multi-illumination cases. GANs are used for this image-to-
image translation, due to their ability to learn effective domain translation.

####  3.1.1 Single illumination Color Constancy

In this section, three GAN models are adapted for the problem of color
constancy under a single unknown illumination. The color biased image is
denoted as $I$, while the color corrected image is denoted as $W$. The entire
domain of color biased images is denoted by $X$ and color corrected image
domain as $Y$, that is $I\in X$ and $W\in Y$.

Pix2Pix [29]: Pix2pix employs a U-net architecture [38] as the generator. Skip
connections [34] allow the net to transfer high frequency details. $PatchGAN$
is introduced as the discriminator that enforces high frequency consistency
through a Markov random field. Standard GAN losses and L1 losses on paired
cross domain images are used to train the network. This allows the network to
learn a pixel-wise transfer between domains while keeping the underlying image
structure consistent.

This formulation is particularly well suited to the problem of color
constancy. The problem is defined as a domain transformation of the image
taken under an unknown illumination ($I$) to the color corrected image ($W$).
The L1 loss makes the network approximate a color correction function, while
the adversarial loss ensures that the transformation is close to the target
domain. This allows the network to predict color corrected images directly,
while removing the unknown illumination color influence. Unfortunately, the L1
loss necessitates dense ground-truth pairs.

CycleGAN [45]: CycleGAN improves on Pix2Pix by introducing a forward and
backward cycle. The network learns a closed cycle through: 1) forward
transformation ($I\rightarrow W$), and 2) backward transformation
($W\rightarrow I$). This allows the network to be trained unsupervised. The
supervision signal is obtained from the outputs of the forward and backward
cycles, called the cycle consistency loss. Given two generators, G (the
forward generator) and F (the backward generator), we have:

$\mathcal{L}_{cycle}(G,F)=E_{x\sim p_{data}(x)}\;||F(G(x))-x||_{1}^{1}\;,$ (3)

The network then learns the cycles $G(X)\rightarrow Y^{\prime}$ and
$F(Y^{\prime})\rightarrow X^{\prime}$. This leads to learning a global
transformation to approximate the color correction. Due to the unsupervised
nature of training, any color biased and color corrected image can be assigned
to the respective domain without the need for a dense ground-truth
correspondence to train the network. Unfortunately, the network can learn only
a single type of illumination correction for the color biased domain.

StarGAN [15]: To handle multi-domain translations simultaneously, StarGAN
introduces an additional conditioning on the domain. The base model
architecture of CycleGAN is used, with a domain vector on top of it. This
vector specifies which target domain the network should map the input to.
Consequently, the discriminator learns to classify the domain, in the form of
an additional output. This allows the network to adversarially learn specific
domain transformations conditioned on the input vector. Both the forward and
backward cycles from CycleGAN are preserved, allowing the network to be
trained in an unsupervised manner.

The ability to learn a transformation between various domains translates well
to the problem of color constancy. This allows the network to learn a common
transformation between different illumination types using a single model.
Furthermore, the common domain defined transformation, yields a network having
a consistency check in the form of whether the model can recover the same
image from different illumination colors. The domains are also learned in an
unsupervised manner, suited to train with limited data.

Limitations: The different GANs provide useful formulations for the problem of
color constancy. However, (1) the networks are merely suited to the problem of
single illumination color constancy, (2) the networks are not tailored to
model the color correction process, (3) the networks implicitly learn the
illumination as a latent state, and (4) the transformations learned are often
global and are lacking spatial context. As a result, these networks needs to
be adapted to multi-illumination color constancy.

####  3.1.2 Multi-Illumination Color Constancy

In this section, a sample-based learning approach to solve the multi-
illumination color constancy is proposed. In this, no assumption on the number
or spatial influence of the illuminants are made. Instead, random initial seed
points are provided to the network as inputs. The ground truth illumination
map is used to randomly sample seed points for different light sources during
training. The network uses these initial points to disentangle the
illumination influence for all pixels. However, this might lead the network to
be biased to the initially selected seeds. To avoid the bias, the color biased
image ($I$) is also provided to the network as an additional input. The color
biased image provides the network with a global cue, while the initial seed
points provide a local cue. Using these cues, the network can learn to diffuse
the initial sample points through the image and in-paint an illumination map.
The input modalities are given in figure 1.

Figure 1: Examples of the dataset augmentation. The 1st-4th samples are the
random samples from the respective illuminants. The color patches below it are
the illumination themselves. The Final Illum Map is the total illumination
maps consisting of the four illuminants. All inputs are of the same spatial
resolution as the color biased image. They have been resized here for
visualisation.

For multi-illumination, every pixel can have varying influence from the
different illuminants. To model this, the network predicts a probability map,
instead of the color corrected image. In this, each pixel is assigned a
probability of belonging to a specific illumination. This transforms the
problem to a per-pixel classification task, rather than a global
transformation. As a result, the network does not have to cluster the
illuminants in the latent space, while having a realistic illumination mixture
for each pixel. The output of the network is a $N$-channel probability map for
$N$ illuminants. Each of the channels represents the probability of a single
pixel belonging to the $N$-th illumination. The predicted illumination map is
reconstructed using the following:

$L_{pred}=\sum_{i=1}^{N}P_{i}\times I_{i}$ (4)

where, $P_{i}$ is the $i$-th channel in the $N$ channel probability map
predicted by the network and $I_{i}$ is the corresponding $i$-th illumination.
$L_{pred}$ is the predicted illumination map obtained from the probability
map.

Finally, to let the network learn a realistic transformation, a PatchGAN
discriminator is used. This helps to keep the network from learning a dataset
bias. An overview of the generator is shown in figure 2.

Figure 2: Overview of the proposed multi-illumination color constancy network.
The samples are a concatenation of the image samples and the corresponding
masks. The cube denotes convolutions. The output of the network is a
probability map, which is processed using equation (4) to obtain the
illumination map. Please refer to the supplementary material for details.

During inference, Grayness Index [37] is used to determine the illumination
clusters and randomly sample from those clusters. The grayness index is a
physics-based, non-learning estimation. Pix2Pix is used as the base model
architecture since it is more suited for local illumination estimation which
is beneficial for multi-illumination color constancy. The direct supervision
in Pix2Pix is also useful for the probability output of the network. In
CycleGAN, the transformation would have to be done from image and sampled seed
to probability, in the forward cycle, and then from probability to image and
sampled seed for the backward cycle. This cycle consistency would be more
difficult for a GAN to learn since the probability map from the forward cycle
would not give enough information for the backward cycle. Additionally,
compared to CycleGAN and StarGAN, Pix2Pix consists of just a single generator
and discriminator, hence it has comparatively lower parameters, making it
quicker and easier to train. The network also provides a local transformation,
which is useful for scenes with multiple light sources since each pixel can be
affected by the combined effect of all the light sources. To integrate the
sample dispersion learning mechanism, an additional shallow network before the
Pix2Pix generator is added. This consists of two layers of convolutions to
condition the incoming features, increasing and then squeezing the feature
channels. For the input illumination samples, the sample mask is also
concatenated. This allows the network to learn a diffusion of the samples to
their correct regions and back-propagate the errors from the seed sample.
Details on the shallow network can be found in the supplementary materials.

Loss Functions: The predicted probability map is constrained by a loss on the
predicted illumination calculated using (4):

$\mathcal{L}_{illum}=\frac{1}{M}\sum_{i=1}^{M}||\hat{L_{i}}-L_{i}||$ (5)

where $M$ are the total number of pixels, $\hat{L_{i}}$ is the predicted
illumination map and $L_{i}$ is the ground truth illumination map.

Furthermore, a loss on the color corrected image is added:

$\mathcal{L}_{rgb}=\frac{1}{M}\sum_{i=1}^{M}||\hat{W_{i}}-W_{i}||$ (6)

where $\hat{W_{i}}$ is the predicted color corrected image. This is obtained
by using (2) with the input color biased image and the predicted illumination
map. The $W_{i}$ is the ground truth color corrected image.

Furthermore, to ensure that the probability map assigns the illuminants to the
proper illumination region, an illumination mask loss is added. The initial
seed points, given as input to the network, are compared to the corresponding
points on the predicted illumination map.

$\mathcal{L}_{masks}=\sum_{j=1}^{N}||\hat{L_{j}}*B-L_{j}*B||$ (7)

where $B$ are the binary masks denoting the locations of seed sampling points.
This loss is calculated for each illumination.

Finally, we use an adversarial training to train the entire network:

$\displaystyle\mathcal{L}_{GAN}$ $\displaystyle=E[logD(I)]\;+$ (8)
$\displaystyle\;\;\;\;E[log(1-D(G(\hat{I})))]\;,$

where, $G(x)$ is the generator and $D(x)$ is the discriminator.

Thus, the total training objective for the network becomes:

$\mathcal{L}_{total}=\mathcal{L}_{GAN}+\lambda(\mathcal{L}_{illum}+\mathcal{L}_{rgb}+\mathcal{L}_{masks})$
(9)

where, $\lambda$ is a hyper-parameter. This was empirically fixed to be $100$.

##  4 Experiments

###  4.1 Datasets

####  4.1.1 Single Illumination Datasets

The GAN models are trained and evaluated on 3 standard single illumination
benchmarks: (1) The SFU Gray Ball [16] dataset, (2) the recalculated version
of [19] called _ColorChecker RECommended_ (RCC) and (3) the Cube dataset [4].
For the experiments, the datasets are randomly split into 80% training and 20%
testing. The reference objects in the images are masked out.

####  4.1.2 Multi Illumination Datasets

Large datasets for multi-illumination color constancy are difficult to obtain.
Therefore, there are only a few datasets publicly available with limited
number of images [8]. To train our network, a multi-illumination augmentation
approach is proposed. Semantic segmentation algorithms are used to obtain
object boundaries. Then, using a mixture model, the images are augmented with
different illuminants. The illuminants are obtained from an existing multi-
illumination dataset like MIMO [8]. The illuminants are then channel shuffled
and combined with color corrected images using (2) to obtain a multi
illumination image. This approximates images taken from multi-illumination
scenes with realistic illumination interaction, textures, and colors.

For the experiments, the Intel TUT [3] dataset, consisting of 5000 single
illumination images, is used. Images are composed of various scenes, including
indoors and outdoors. The dataset provides single illumination ground-truth
for each image, recorded using a color checker object. To augment the dataset
for multi-illumination, a semantic segmentation for each image is obtained,
using Semantic Soft Segmentation [2]. The segmentation is then used to create
the illumination map. For each illumination map obtained, random seed pixels
from each of the illumination sections are selected. Thus, for each image,
with $N$ illuminants, there are $N\times 2+2$ image pairs: $N$ illumination
color images, $N$ seed point masks, the multi-illumination color biased image
and the corresponding color corrected image.

During inference, the grayness index [37] is used to obtain the initial multi-
illumination clusters on the MIMO dataset. These clusters are then used to
obtain the initial seed points, which are then passed on to the network. All
numbers reported, and unless specifically mentioned, are from networks trained
completely on the augmented Intel TUT dataset and tested on the MIMO dataset.

###  4.2 Error Metrics

The angular error between the ground-truth illumination $e$ and the estimated
illumination $\hat{e}$ is reported:

$d_{ang}=arccos(\dfrac{e\cdot\hat{e}}{||e||\times||\hat{e}||})\;,$ (10)

where $||.||$ is the L2 norm. For each image, the angular error is computed,
and the mean, median, trimean, means of the lowest-error 25%, the highest-
error 25%, and maximum angular errors are reported.

For the architectures that do not provide an illumination estimation directly,
like the single illumination GAN experiments, the apparent illumination from
the color corrected image is obtained. This is done by inverting equation (1),
as follows:

$e=I*\hat{W}^{-1}\;,$ (11)

where $I$ is the input image and $\hat{W}$ is the white balanced estimation
from the network. Both $I$ and $\hat{W}$ are converted into the linear $RGB$
space before computing the estimated illumination.

###  4.3 Single Illumination Color Constancy

####  4.3.1 GAN Comparison

Figure 3: Performance of different GAN models on the color constancy task. All
models can recover proper white balanced images. The images are gamma adjusted
for visualization.

In this experiment, the performance of 3 different state-of-the-art GAN models
for color constancy is given: Pix2Pix [29], CycleGAN [45] and StarGAN [15].
All models use $sRGB$ inputs and generate a white balanced image. The color of
the light source is estimated by Equation (11). The results are presented in
Tables 2, 3 and 1 for the different datasets. Figure 3 shows some sample
outputs.

Model | Mean | Med. | Tri. | Best 25% | Worst 25% | Max
---|---|---|---|---|---|---
Pix2Pix [29] | 6.6 | 5.3 | 5.4 | 1.4 | 14.2 | 36.0
StarGAN [15] | 10.3 | 8.9 | 9.0 | 4.0 | 18.9 | 46.0
CycleGAN [45] | 8.4 | 5.9 | 6.4 | 1.5 | 19.6 | 37.8

Table 1: Performance of different GAN models for SFU Gray Ball dataset [16].
Pix2Pix achieves the best performance.

Model | Mean | Med. | Tri. | Best 25% | Worst 25% | Max
---|---|---|---|---|---|---
Pix2Pix [29] | 3.6 | 2.8 | 3.1 | 1.2 | 7.2 | 11.3
StarGAN [15] | 5.3 | 4.2 | 4.6 | 1.5 | 11.0 | 21.8
CycleGAN [45] | 3.4 | 2.6 | 2.8 | 0.7 | 7.3 | 18.0

Table 2: Performance of different GAN models for ColorChecker RECommended
dataset [19]. CycleGAN achieves the best performance.

Model | Mean | Med. | Tri. | Best 25% | Worst 25% | Max
---|---|---|---|---|---|---
Pix2Pix [29] | 1.9 | 1.4 | 1.5 | 0.7 | 4.0 | 8.0
StarGAN [15] | 3.8 | 3.3 | 3.5 | 1.3 | 7.0 | 11.4
CycleGAN [45] | 1.5 | 1.2 | 1.3 | 0.5 | 3.0 | 6.0

Table 3: Performance of different GAN models for Cube dataset [4]. CycleGAN
achieves the best performance.

Tables 2 and 3 show that _CycleGAN_ outperforms _Pix2Pix_ and _StarGAN_. Table
1 shows that Pix2Pix achieves the best performance for the SFU Gray Ball
dataset. Pix2Pix learns a per-pixel mapping of the illumination and is
agnostic about the global illumination or the spatial relations. On the other
hand, the other GAN models learn a global representation of the scene
illumination. Since CycleGAN can learn a more global transformation, it is
able to outperform the other GANs for the color Checker and Cube dataset.
However, for the Gray ball dataset the ground truth estimation is prone to
errors. In conclusion, the global estimate of CycleGAN fits all the light
sources under a single transformation function, while Pix2Pix can work around
this by learning a per-pixel independent transformation.

Figure 4: Visualisations of the StarGAN architecture. This architecture learns
mappings to multiple domains, e.g., Illuminant A and canonical,
simultaneously. The images are gamma adjusted for visualisation.

StarGAN has much lower performance than the other 2 models. This may be due to
the architecture’s latent representation that has to compensate for learning 3
different (input, canonical, Illuminant A) domain transformations. This is not
the case with either Pix2Pix or CycleGAN, where the mapping is one-to-one. The
multiple target illumination domain output of StarGAN can be beneficial, as
shown in Figure 4, for relighting tasks. The accuracy of this internal
estimation defines how close the output is to the target domain. Ideally, this
estimated ground-truth illumination should be the same for all the output
domains. Hence, by checking the performance of the output in estimating the
ground-truth using Equation (11), the consistency of the network can be
checked. This also allows for observing which illumination performs better in
estimating the ground-truth illuminants. Per-illuminant consistency
performance is presented in Table 4. It can be observed from the table that
for the RECommended Color Checker (RCC) dataset _Illuminant A_ estimation
yields better results than the direct canonical estimation. For the Cube
dataset, the canonical performs better. The Cube dataset consists of only
outdoor images. Hence, the input images seldom have an illumination that is
closer to Illuminant A. Conversely, for the RCC dataset, indoor scenes
generally have incandescent lighting, which is closer to Illuminant A.

Dataset + Illumination | Mean | Med. | Tri. | Best 25% | Worst 25% | Max
---|---|---|---|---|---|---
RCC + Canonical | 5.3 | 4.2 | 4.6 | 1.5 | 11.0 | 21.8
RCC + Illuminant A | 2.9 | 2.2 | 2.3 | 0.8 | 6.4 | 13.8
Cube + Canonical | 3.8 | 3.3 | 3.5 | 1.3 | 7.0 | 11.4
Cube + Illuminant A | 4.1 | 3.2 | 3.4 | 1.1 | 8.5 | 20.7
SFU + Canonical | 10.3 | 8.9 | 9.0 | 4.0 | 18.9 | 46.0
SFU + Illuminant A | 12.7 | 11.2 | 11.2 | 3.2 | 24.8 | 61.1

Table 4: Consistency of StarGAN’s illumination estimation.

####  4.3.2 Comparison to the State-of-the-Art

In this section, the GAN models are compared with different benchmarking
algorithms including Grey-World [12], Second-order Grey-Edge [42], Pixel-based
Gamut [5], Intersection-based Gamut [5], Edge-based Gamut [5], Spatial
Correlations[13], Natural Image Statistics [23], High Level Visual Information
[43], Exemplar-Based color constancy [30], Color Tiger [4]. Furthermore,
comparisons to 2 convolutional approaches are given: Deep color constancy [11]
and Fast Fourier color constancy [7]. Some of the results are taken from
related papers, resulting in missing entries for some datasets. Tables 5, 6
and 7 provide quantitative results for 3 different benchmarks.

Method | Mean | Med. | Tri. | Best 25% | Worst 25% | Max
---|---|---|---|---|---|---
Grey-World [12] | 13.0 | 11.0 | 11.5 | 3.1 | 26.0 | 63.0
Edge-based Gamut [5] | 12.8 | 10.9 | 11.4 | 3.6 | 25.0 | 58.3
Spatial Correlations [13] | 12.7 | 10.8 | 11.5 | 2.4 | 26.1 | 41.2
Second-order Grey-Edge [42] | 10.7 | 9.0 | 9.4 | 3.2 | 20.9 | 56.0
Bottom-up & Top-down [43] | 9.7 | 7.7 | 8.2 | 2.3 | 20.6 | 60.0
Natural Image Statistics [23] | 9.9 | 7.7 | 8.3 | 2.4 | 20.8 | 56.1
E. B. color constancy [30] | 8.0 | 6.5 | 6.8 | 2.0 | 16.6 | 53.6
Pix2Pix [29] | 6.6 | 5.3 | 5.4 | 1.4 | 14.2 | 36.0
CycleGAN [45] | 8.4 | 5.9 | 6.4 | 1.5 | 19.6 | 37.8
StarGAN [15] | 11.4 | 9.2 | 10.0 | 3.8 | 21.8 | 41.4

Table 5: Performance on SFU Gray Ball [16].

Method | Mean | Med. | Tri. | Best 25% | Worst 25% | Max
---|---|---|---|---|---|---
Grey-World [12] | 9.7 | 10.0 | 10.0 | 5.0 | 13.7 | 24.8
White-Patch [32] | 9.1 | 6.7 | 7.8 | 2.2 | 18.9 | 43.0
Shades-of-Grey [20] | 7.3 | 6.8 | 6.9 | 2.3 | 12.8 | 22.5
AlexNet + SVR [11] | 7.0 | 5.3 | 5.7 | 2.9 | 14.0 | 29.1
Pixel-based Gamut [5] | 6.0 | 4.4 | 4.9 | 1.7 | 12.9 | 25.3
Intersection-based Gamut [5] | 6.0 | 4.4 | 4.9 | 1.7 | 12.8 | 26.3
Edge-based Gamut [5] | 5.5 | 3.3 | 3.9 | 0.7 | 13.8 | 29.8
Deep Colour Constancy [11] | 4.6 | 3.9 | 4.2 | 2.3 | 7.9 | 14.8
Second-order Grey-Edge [42] | 4.1 | 3.6 | 3.8 | 1.5 | 8.5 | 16.9
First-order Grey-Edge [42] | 4.0 | 3.1 | 3.3 | 1.4 | 8.4 | 20.6
FFCC (model Q) [7] | 2.0 | 1.1 | 1.4 | 0.3 | 4.6 | 25.0
Pix2Pix [29] | 3.6 | 2.8 | 3.1 | 1.2 | 7.2 | 11.3
CycleGAN [45] | 3.4 | 2.6 | 2.8 | 0.7 | 7.3 | 18.0
StarGAN [15] | 5.7 | 4.9 | 5.2 | 1.7 | 10.5 | 19.5

Table 6: Performance on ColorChecker RECommended [19].

Method | Mean | Med. | Tri. | Best 25% | Worst 25%
---|---|---|---|---|---
Grey-World [12] | 3.8 | 2.9 | 3.2 | 0.7 | 8.2
White-Patch [32] | 6.6 | 4.5 | 5.3 | 1.2 | 15.2
Shades-of-Grey [20] | 2.6 | 1.8 | 2.9 | 0.4 | 6.2
General Grey-World [6] | 2.5 | 1.6 | 1.8 | 0.4 | 6.2
Second-order Grey-Edge [42] | 2.5 | 1.6 | 1.8 | 0.5 | 6.0
First-order Grey-Edge [42] | 2.5 | 1.6 | 1.8 | 0.5 | 5.9
Color Tiger [4] | 3.0 | 2.6 | 2.7 | 0.6 | 5.9
Restricted Color Tiger [4] | 1.6 | 0.8 | 1.0 | 0.2 | 4.4
Pix2Pix [29] | 1.9 | 1.4 | 1.5 | 0.7 | 4.0
CycleGAN [45] | 1.5 | 1.2 | 1.3 | 0.5 | 3.0
StarGAN [15] | 4.8 | 3.9 | 4.2 | 1.9 | 8.9

Table 7: Performance on Cube dataset[4].

It is shown in Table 5 (SFU Gray Ball), Pix2Pix [29] achieves the best
performance in all the metrics with 17.5% improvement in mean angular error,
18.4% in median and 20.6% in trimean. For Table 6 (RCC), CycleGAN [45]
framework achieves the second best place in different metrics, only worse than
Fast Fourier color constancy [7]. In Table 7 (Cube), CycleGAN [45] achieves
the best performance in the mean error and worst and comparable results with
all other methods.

####  4.3.3 GAN Performance for Multiple Light Sources

In this section, the performance of GANs in a multi-illumination setting is
studied. The models are trained on 3 datasets and tested on the MIMO dataset.
The results are shown in Tables 8 and 9. The performance of StarGAN is not
provided since, from the experiments, it is observed: 1) to be generally the
worst performing, and 2) CycleGAN is a simplified version of StarGAN with just
one domain transform.

Dataset | Mean | Median
---|---|---
Cube | 18.7 | 18.1
REC-CC | 14.8 | 14.5
SFU | 20.1 | 17.8

Table 8: Performance of Pix2Pix on the MIMO dataset.

Dataset | Mean | Median
---|---|---
Cube | 12.7 | 11.9
REC-CC | 13.3 | 12.9
SFU | 20.1 | 17.4

Table 9: Performance of CycleGAN on the MIMO dataset.

The results show that both the algorithms are unable to cope with images with
multiple light sources. This is because both models are trained to learn a
single illumination transformation. As a result, the models confuse object
colors with illuminants. Pix2Pix shows slightly larger errors because no
contextual constraints are imposed on the pixels. As such, ambiguous points
like edges and boundaries contribute to higher errors. Inclusion of an
explicit illumination context becomes a necessity for the network to be able
to model a multi-illumination setting.

###  4.4 Multi Illumination Color Constancy

In this section, the proposed method for multi-illumination color constancy is
analysed. For all the experiments, four illuminants are chosen, $N=4$.
However, the proposed method is not dependent on the choice of $N$ and can be
arbitrarily scaled. All corresponding hyper-parameters of the model are kept
constant. This is to ensure that the influence of the components being tested
is not influenced by a change in the hyper-parameters. For training, the
entire set of 5000 images from Intel TUT is used, while testing is done
exclusively on the MIMO dataset.

####  4.4.1 Influence of Adversarial Loss

In this experiment, the influence of the adversarial loss is studied. Since
there is a direct supervision from the ground truth, the network is trained in
the traditional manner, where an encoder-decoder architecture is used to
produce the output. Both the generator and the discriminator losses are
removed and the network is trained using only the L1 losses on the predicted
illumination map (5), the color corrected image (6) and the individual seed
consistency (7). The results for the experiment are shown in table 10.

Algorithm | Mean | Median
---|---|---
Without Adv. Loss | 3.7 | 3.5
Proposed | 3.5 | 2.9

Table 10: Influence of the GAN loss.

It is observed that omitting the adversarial loss reduces the performance.
Adding the adversarial loss helps the model to cope with outliers, while also
improving the average performance of the model. This can be explained as
follows. The discriminator in a GAN is actively seeking for unrealistic
outputs, which are often outliers. Furthermore, without the adversarial
losses, the network only minimizes a distance function, which can lead to
blurry outputs or ambiguous edges and borders. Since the proposed method is
pixel-wise, these errors propagate more strongly through the network. In
conclusion, this experiment shows that the inclusion of the adversarial loss
is a necessary component for the proposed formulation.

####  4.4.2 Influence of Illumination Probability Map Prediction

In the previous experiment, we modelled the problem as an image-to-image
translation. In this experiment, we study such a formulation for the proposed
method. For this, instead of predicting a per-pixel probability map, the color
corrected image is directly predicted. For the illumination losses, the
predicted illumination map is obtained by using the predicted color corrected
image and the input color bias image itself. This experiment verifies whether
adding the samples as an extra input to Pix2Pix improves the performance. It
is also shown to what extent Pix2Pix can adapt to a multi-illumination
setting. The results are given in table 11. Figure 5 shows some outputs.

Algorithm | Mean | Median
---|---|---
Without Probability Output | 5.30 | 5.24
Proposed | 3.5 | 2.9

Table 11: Influence of a probability map as the output. Figure 5: Comparison
of the modified direct correction model to the ground truth. Here, the model
directly outputs the color corrected image, instead of producing an
illumination probability map. It is observed that the model is not able to
fully correct for the illumination and as such, has residual illumination
colors (2nd row, the printer.) This can be explained by the lack of the
explicit probability map, which forces the network to instead predict a
uniform illumination.

The results show that regressing the color corrected image directly using the
proposed formulation yields higher errors. Since the model takes initial seed
points and tries to constrain the output to be consistent with the seed
points, the model fails to perform a full correction. This is because with a
probability output, the initial seed points are directly linked to the
illuminants. However, having a color corrected output needs the network to
learn this probability implicitly. This, coupled with the illumination
constraints on the loss functions, causes the network to leak the illumination
over to the image directly. From the figure it can also be observed that the
outputs are a bit more washed out and contain illumination color as residuals.
This is because without the probability map support, the network compensates
by averaging the illumination distribution more uniformly.

####  4.4.3 Comparison to State of the Art

In this experiment, the performance of the proposed method is compared with
the state-of-the art methods. The results are shown in table 12. A visual
comparison is also provided in figure 6. The Grayness Index algorithm is
compared against 4-illumination clustering since the proposed method is tested
on the 4-illumination assumption. A multi-illumination setting, like indoors,
is more likely to have $4$ or less major illuminants. Hence $N$ is set to $4$
for the experiments.

Algorithm | Mean | Median
---|---|---
Doing Nothing | 8.9 | 8.8
[24] | 3.8 | 4.2
CRF ([8]) | 4.1 | 3.3
GI(M=4) ([37] | 4.0 | 3.5
Proposed | 3.5 | 2.9

Table 12: Comparison to State-of-the-Art algorithms.  Figure 6: Comparison of
the proposed method to sota methods. The proposed method can properly recover
color corrected images. Gray Pixel can somewhat handle multi-illumination.
However, it fails for ambiguous cases like the bike (3rd row). Gray Pixel
fails to remove the yellow color cast of the light. The proposed method can
recover such ambiguities.

It is shown that the proposed model can outperform all the baselines by a
comfortable margin. From the figure, it is observed that the proposed method
can properly recover color corrected images. The proposed method can correct
for the illumination on the printer, while Grayness Index classifies that as
part of the object color. Similarly, Grayness Index is unable to color correct
the reflection on the monitor while the proposed method can. The only
requirement for the proposed model is the corresponding seed points for each
illumination, only required for the training phase. This requirement can be
easily satisfied as discussed previously using a single illumination dataset
and augmenting it, without requiring any specialised hardware. For edge cases
of more illuminants in the test image, compared to the $N$ illumination number
during the training, the network will simply collapse them into $N$
illuminants. Explicit handling of such cases is beyond the scope of this work
and can be a future research.

##  5 Conclusion

In this paper, we modelled the problem of color constancy as an image-to-image
translation. GANs are used to learn domain transfers. A hybrid approach has
been proposed. A single-illumination dataset augmentation has also been
proposed to address the problem of the lack of large multi-illumination
datasets to enable learning. Experiments on both single and multi-illumination
datasets show that our methods outperform sota methods.

Supplementary Material

##  6 Network Details

The architectural details are provided. The inputs to the network are first
processed through a shallow convolution head, for feature conditioning. The
image processing and the illumination processing head are detailed separately
in 7 and 8, respectively. The input to the image processing head is the $RGB$
color biased image, while the illumination processing head takes the
concatenation of the sparse illumination map and the mask as input. The output
for the image head and illumination head is a 64 channel and 32 channel
tensor, respectively. All of them have the same spatial dimensions as the
input. Tables 13 and 14 details the configurations.

Figure 7: The architecture for the image processing head is shown. The input
image is passed through two convolution layers, which squeezes and
subsequently expands the channels of the outputs, thus conditioning the
features down to the useful features in an end-to-end fashion. Figure 8: The
architecture for the illumination processing head is shown. The input is the
sparse illumination samples and the corresponding masks, concatenated channel
wise ($3+1$ channels). This is passed through two convolution layers, which
squeezes and subsequently expands the channels of the outputs, thus
conditioning the features down to the useful features in an end-to-end manner.

Name | Layer | Kernel Size, Stride, Padding | Output Size
---|---|---|---
Input | conv1 | 3x3x32, 1, 1 | 256x256x32
Output | conv2 | 3x3x32, 1, 1 | 256x256x64

Table 13: Overview of the image processing head. The input is 3 channel and
the output is a 64 channel tensor with the same spatial dimension.

Name | Layer | Kernel Size, Stride, Padding | Output Size
---|---|---|---
Input | conv1 | 3x3x16, 1, 1 | 256x256x16
Output | conv2 | 3x3x16, 1, 1 | 256x256x32

Table 14: Overview of the illumination processing head. The input is 4 channel
concatenation of the sparsse illumination samples and illumination masks. The
output is a 32 channel tensor with the same spatial dimension.

The Pix2Pix [29] generator architecture is used, with just two changes: 1) The
input channels are changed to match the incoming tensors’ dimension from the
processing heads and, 2) The final activation is changed to Sigmoid instead of
Tanh, to output a probability map instead of an image.

## References

  * [1] Abdelrahman Abdelhamed, Abhijith Punnappurath, and Michael S. Brown.  Leveraging the availability of two cameras for illuminant estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6637–6646, June 2021.
  * [2] Yağız Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys, and Wojciech Matusik.  Semantic soft segmentation.  ACM Transactions on Graphics (Proc. SIGGRAPH), 37(4):72:1–72:13, 2018.
  * [3] Çaglar Aytekin, Jarno Nikkanen, and Moncef Gabbouj.  INTEL-TUT dataset for camera invariant color constancy research.  CoRR, abs/1703.09778, 2017.
  * [4] N. Banic and S. Loncaric.  Unsupervised learning for color constancy.  In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2018.
  * [5] K. Barnard.  Improvements to gamut mapping color constancy algorithms.  In European Conference on Computer Vision, 2000.
  * [6] K. Barnard, L. Martin, A. Coath, and B. Funt.  A comparison of computational color constancy algorithms - part ii: Experiments with image data.  IEEE Transactions on Image Processing, pages 985–996, 2002.
  * [7] J. T. Barron.  Fast fourier color constancy.  In IEEE Conference on Computer Vision and Pattern Recognition, 2016\.
  * [8] Shida Beigpour, C. Riess, Joost van de Weijer, and E. Angelopoulou.  Multi-illuminant estimation with conditional random fields.  IEEE Transactions on Image Processing, 23:83–96, 2014.
  * [9] S. Bianco, G. Ciocca, C. Cusano, and R. Schettini.  Improving color constancy using indoor-outdoor image classification.  IEEE Transactions on Image Processing, pages 1057–7149, 2008.
  * [10] S. Bianco, G. Ciocca, C. Cusano, and R. Schettini.  Automatic color constancy algorithm selection and combination.  Pattern Recognition, pages 695–705, 2010.
  * [11] S. Bianco, C. Cusano, and R. Schettini.  Color constancy using cnns.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015.
  * [12] G. Buchsbaum.  A spatial processor model for object colour perception.  Sources of Color Science, pages 1–26, 1980.
  * [13] A. Chakrabarti, K. Hirakawa, and T. Zickler.  Color constancy with spatio-spectral statistics.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1509–1519, 2012.
  * [14] Dongliang Cheng, Brian Price, Scott Cohen, and Michael S. Brown.  Effective learning-based illuminant estimation using simple features.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1000–1008, 2015.
  * [15] Y. Choi, M. Choi, M. Kim, J. W. Ha, S. Kim, and J. Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [16] F. Ciurea and B. Funt.  A large image database for color constancy research.  In Color and Imaging Conference, 2003.
  * [17] Partha Das, Anil S. Baslamisli, Yang Liu, Sezer Karaoglu, and Theo Gevers.  Color constancy by gans: An experimental survey.  CoRR, abs/1812.03085, 2018.
  * [18] M. D. Fairchild.  Color Appearance Models.  Wiley, Chichester, U.K., 2013.
  * [19] G. D. Finlayson, G. Hemrit, A. Gijsenij, and P. Gehler.  A curious problem with using the colour checker dataset for illuminant estimation.  In Color and Imaging Conference, 2017.
  * [20] G. D. Finlayson and E. Trezzi.  Shades of gray and colour constancy.  In Color and Imaging Conference, 2004.
  * [21] B. Funt and Weihua Xiong.  Estimating illumination chromaticity via support vector regression.  In Color Imaging Conference, 2004.
  * [22] R. Gershon, Allan Jepson, and John Tsotsos.  From [r, g, b] to surface reflectance: Computing color constant descriptors in images.  In International Joint Conferences on Artificial Intelligence, pages 755–758, 01 1987.
  * [23] A. Gijsenij and T. Gevers.  Color constancy using natural image statistics and scene semantics.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 687–698, 2010.
  * [24] A. Gijsenij, T. Gevers, and J. van de Weijer.  Improving color constancy by photometric edge weighting.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5):918–929, May 2012.
  * [25] A. Gijsenij, R. Lu, and T. Gevers.  Color constancy for multiple light sources.  IEEE Transactions on Image Processing, 21(2):697–707, 2012.
  * [26] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, 2014.
  * [27] Y. Hu, B. Wang, and S. Lin.  Fc4: Fully convolutional color constancy with confidence-weighted pooling.  In IEEE Conference on Computer Vision and Pattern Recognition, 2017\.
  * [28] M. A. Hussain and A. S. Akbari.  Color constancy algorithm for mixed-illuminant scene images.  IEEE Access, 6:8964–8976, 2018.
  * [29] P. Isola, J. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial nets.  In IEEE Conference on Computer Vision and Pattern Recognition, 2017\.
  * [30] H. R. V. Joze and M. S. Drew.  Exemplar-based color constancy and multiple illumination.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 860–873, 2014.
  * [31] E. H. Land.  The retinex theory of color vision.  Scientific American, pages 108–129, 1977.
  * [32] E. H. Land and J. J. McCann.  Lightness and retinex theory.  Journal of the Optical Society of America, pages 1–11, 1971.
  * [33] Z. Lou, T. Gevers, N. Hu, and M. P. Lucassen.  Color constancy by deep learning.  In British Machine Vision Conference, 2015.
  * [34] X. Mao, C. Shen, and Y. Yang.  Image restoration using very deep fully convolutional encoder-decoder networks with symmetric skip connections.  In Advances in Neural Information Processing Systems, 2016.
  * [35] S. W. Oh and S. J. Kim.  Approaching the computational color constancy as a classification problem through deep learning.  Pattern Recognition, pages 405–416, 2017.
  * [36] Y. Qian, K. Chen, J. Nikkanen, J. K. Kamarainen, and J. Matas.  Recurrent color constancy.  In IEEE International Conference on Computer Vision, 2017.
  * [37] Yanlin Qian, Jarno Nikkanen, Joni-Kristian Kämäräinen, and Jiri Matas.  On finding gray pixels.  CoRR, abs/1901.03198, 2019.
  * [38] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference On Medical Image Computing & Computer Assisted Intervention, 2015.
  * [39] C. Rosenberg, T. Minka, and A. Ladsariya.  Bayesian color constancy with non-gaussian models.  In Advances in Neural Information Processing Systems, 2004.
  * [40] Oleksii Sidorov.  Conditional gans for multi-illuminant color constancy: Revolution or yet another approach?  CoRR, abs/1811.06604, 2018.
  * [41] S. Tan, F. Peng, H. Tan, S. Lai, and M. Zhang.  Multiple illumination estimation with end-to-end network.  In IEEE International Conference on Image, Vision and Computing, 2018.
  * [42] J. van De Weijer, T. Gevers, and A. Gijsenij.  Edge-based color constancy.  IEEE Transactions on Image Processing, pages 2207–2214, 2007.
  * [43] J. van de Weijer, C. Schmid, and J. Verbeek.  Using high-level visual information for color constancy.  In IEEE International Conference on Computer Vision, 2007.
  * [44] J. von Kries.  Influence of adaptation on the effects produced by luminous stimuli.  Sources of Color Science, pages 109–119, 1970.
  * [45] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE International Conference on Computer Vision, 2017.


# Navigating to Objects in Unseen Environments by Distance Prediction

Minzhao Zhu, Binglei Zhao, Tao Kong Minzhao Zhu and Binglei Zhao contribute
equally. ByteDance AI Lab, Beijing, China. {zhaobinglei, zhuminzhao,
<EMAIL_ADDRESS>
###### Abstract

Object Goal Navigation (ObjectNav) task is to navigate an agent to an object
category in unseen environments without a pre-built map. In this paper, we
solve this task by predicting the distance to the target using semantically-
related objects as cues. Based on the estimated distance to the target object,
our method directly choose optimal mid-term goals that are more likely to have
a shorter path to the target. Specifically, based on the learned knowledge,
our model takes a bird’s-eye view semantic map as input, and estimates the
path length from the frontier map cells to the target object. With the
estimated distance map, the agent could simultaneously explore the environment
and navigate to the target objects based on a simple human-designed strategy.
Empirical results in visually realistic simulation environments show that the
proposed method outperforms a wide range of baselines on success rate and
efficiency. Real-robot experiment also demonstrates that our method
generalizes well to the real world.

##  I INTRODUCTION

Object Goal Navigation (ObjectNav) [1] is one of the fundamental embodied
navigation tasks. In this task, an intelligent agent is required to move to
the location of a target object category in an unseen indoor environment. In
classical navigation tasks, under normal circumstances, the map of the
environment is constructed in advance. Therefore, a goal location can be given
to the agent in the form of coordinates on that map. However, in the ObjectNav
task, a pre-built map is unavailable, and the goal coordinate is not given.
Therefore, the agent has to explore the environment and search for the target
object.

During searching, there are many possible areas to be explored. How to
prioritize these candidate goals in order to improve exploration efficiency?
Obviously, in a new environment, the only information we can use is the
knowledge we have learned in other similar environments, such as the spatial
relations between objects. With this commonsense, humans tend to explore the
object that is usually close to the target object. For example, if our target
is a chair, we should explore around a table first, while temporarily skipping
other regions. This is because we know chairs are often adjacent to tables;
thus, if we move toward the table, it is more likely to find a chair in that
direction than other directions away from that table. If we incorporate this
kind of prior knowledge into a spatial map, we can transform the ObjectNav
task into a classical navigation problem.

Figure 1: In Object Goal Navigation (ObjectNav) task, an agent is required to
navigate to the location of a target object category in an unseen environment
using an RGB-D sensor. We propose to solve this task by predicting the
distance to the target based on spatial relations between objects. The blue
shadow in the figure indicates the agent’s field of view. The red shadow
indicates the area around the target object. In this example, the target
object is "chair." The distance to "chair" is supposed to be estimated based
on the observed object "table." Since the distance from "table" to "chair" is
often smaller than from other objects, navigating toward "table" are more
likely to find "chair" compared with other directions.

Inspired by this, we propose to solve the ObjectNav task by predicting the
distance to the target using semantically-related objects as cues. Inspired by
the concept of Frontier-based Exploration [2], we predict the distance value
(the length of the shortest path to the nearest target object) on the frontier
area, and then select goals on the frontier according to the predicted
distance.

The Target Distance Prediction model, the core module in our system, learns to
predict the distance to the target based on the explored semantic map. The
model is encouraged to capture the spatial relations between different
semantics. For example, if the target object is ’chair,’ the predicted
distance around tables should be small.

Many recent works [3, 7, 4, 6, 5] formulate this task as Reinforcement
Learning (RL) and have achieved good performance. However, RL often suffers
from low sample efficiency [8]. In contrast, we formulate it as a target
distance estimation problem, which could be optimized in a fully supervised
way during training. Our method does not require interacting with the
environment to get a reward during training. There are also recent works [6,
9] training a semantic scene completion model to learn the prior knowledge.
However, it is difficult, even for humans, to predict the exact location of
related objects. For example, although we know chairs may be close to tables,
we cannot predict the chair’s relative pose to the table accurately, since the
chair could be placed anywhere around that table. In contrast, the distance
between the chairs and tables does not vary too much. Our method only needs to
predict this distance, which is relatively easier to learn.

Our main contribution is that we formulate the ObjectNav task as a target
distance prediction problem based on spatial relations between objects. Our
method consists of three parts. First, given the RGB-D image and the agent’s
pose, a bird’s-eye view semantic map is incrementally built. Then, based on
the semantic map, a target distance map is predicted. The cells in this
distance map store the predicted path length to the nearest target object.
Then the distance map is fed to the local policy to get action. Since the
model’s output is the estimated distance to the target, a mid-term goal can be
easily selected in order to get the shortest path to the target. As a result,
it can be easily integrated into a traditional navigation system. In this
paper, we use several simple goal-select strategies and path planning
algorithms to show the effectiveness of the distance map. We perform
experiments in the Matterport3D dataset using the Habitat simulator. Our
method outperforms the baseline [3] method with an improvement of 2.6% success
rate and 0.035 SPL(Success weighted by normalized inverse Path Length) [12].
The Experiment on our robot base also demonstrates that our method generalizes
well to the real world.

Figure 2: System Overview. Our method consists of a semantic mapping module, a
target distance prediction module, and a local policy. Given the RGB-D
observation and the agent’s pose, the Semantic Mapping Module builds a
bird’s-eye view semantic map. Then, the Target Distance Prediction Module
predicts the distance to the target object on the cells around the exploration
frontiers. Based on the distance map, the Local Policy chooses a mid-term goal
and gets action.

##  II RELATED WORK

###  II-A Active SLAM

Given a specific goal location, the classical navigation methods[13] focus on
path planning[17, 14, 15, 16] based on a previously constructed map. However,
Active SLAM[20] aims to explore unknown experiments and build maps
automatically, which is a decision-making problem tightly coupled with
localization, mapping, and planning. Frontier-based Exploration [2] selects
mid-term goals on the frontier according to a designed strategy and extends
its map by moving to new frontiers. Some methods [21] formulate this problem
as a POMDP (Partially Observable Markov Decision Process) [19]. Recently, some
researchers have designed learning-based policies[22, 23, 24, 25] to tackle
this problem. Chaplot et al.[23] propose a novel module that combines a
hierarchical network design and classical path planning, which significantly
improves the sample efficiency and leads to better performance. The main aim
of Active SLAM is to explore the environment efficiently, while the ObjectNav
task requires the agent to search the target object category efficiently.

###  II-B Object Goal Navigation

While Active SLAM is to explore the environment efficiently, Target Driven
Navigation is to navigate to a given target in an unknown environment [26].
The goal could be a point, an object category, a room, etc. Most of the
approaches fall into the following three groups: map-less reactive methods
[26], memory-based methods [27, 28, 29, 30, 5, 4], and explicit map-based
methods [3, 6, 37, 9, 38, 39, 40].

Among the Target Driven Navigation tasks, Object Goal Navigation [1] is the
one that sets an object category as the target. The agent takes RGB-D images
and poses as input, and output actions to search the target object. This task
is similar to active visual search [31] that has been studied for many years.
A feasible solution is to formulate it as a POMDP problem. Some works model
the belief by online learning [38], object-object/object-scene co-occur
probability [32, 33], qualitative spatial relations [34]. In comparison, we
incorporate the knowledge of object relation into a distance prediction model.
The model learns to predict the distance to the target object based on the
explored semantic map, and a goal is selected according to the predicted
distance using a designed strategy or a path planning method.

Recent works [3, 7, 4, 6, 5] formulate this task as Reinforcement Learning
(RL) and have achieved good performance. However, RL often suffers from low
sample efficiency [8]. We replace the RL model in [3] with a distance
prediction model and a simple goal selection strategy. The model can be
trained offline by collecting semantic map samples and does not need to
interact with the environment to get a reward.

Recent works also model scene priors by predicting elements out of sight to
guide the agent to explore the target. Some works [6, 9] use a semantic scene
completion module to complete the unexplored scene. Similar to it, some
methods achieve image-level extrapolation [35] or high-dimension feature space
extrapolation [36]. Accurately predicting the scene out of view is difficult
as the exact relative pose between related objects is hard to determine. We do
not predict the exact semantics and their pose out of sight. Instead, we
estimate the distance from the frontier area to the target, based on observed
objects and spatial relations between objects. The distance between the
related objects is more likely to fall in a reasonable range, which is
relatively easier to learn.

##  III APPROACH

As shown in Fig. 2, our method consists of three modules: a semantic mapping
module, a target distance prediction module, and a local policy. The input of
our system is the RGB-D image and the agent pose; the output is the next
action. At each step, the RGB-D observation and the agent pose are used to
update the bird’s-eye view semantic map. Then, based on the semantic map and
the learned knowledge, the distance prediction model estimates the shortest
path length from the exploration frontier (the map cells between the explored
and unexplored area) to the target. According to the distance map, the local
policy selects a mid-term goal and gets the next action using a path planning
method.

Although the distance prediction may not be correct enough in the beginning,
as the agent moves and receives more observation, the semantic map expands,
the predicted distance map updates and becomes more accurate. With the update
of the distance map, the agent could automatically and implicitly switch from
random exploration to target searching and approaching, and reach the target
object eventually.

In Section III-A we briefly describe the Semantic Mapping module. Section
III-B gives the definition of the target distance map and presents our model.
In Section III-C, we describe the details of the local policy.

Figure 3: Target Distance Ground Truth Map Example. The first row is the
semantic GT map; the second row is the continuous distance map (darker color
corresponds to smaller distance value) and the discrete distance GT map for
goal category ‘TV’. The red dotted circles indicate the location of ‘TV’.

###  III-A Semantic Mapping

We follow SemExp [3] to build a bird’s-eye view semantic map. The semantic map
$L_{S}$ is with $k\times h\times w$, where $h\times w$ denotes the map size,
$k=c_{S}+2$, and $c_{S}$ is the total number of semantic categories. The other
2 channels represent obstacles and explored areas. In practice, we set the
size of each cell in the semantic map to be 5cm $\times$ 5cm.

Given the RGB-D image and the agent’s pose, we use a semantic model, e.g.
Mask-RCNN [41], to predict semantic categories on the RGB image and then get a
3D semantic point cloud in the camera coordinate system using depth
observation and camera intrinsics. The semantic point cloud is transformed to
the global coordinate and projected to the ground to get the semantic map.

###  III-B Target Distance Prediction

Target Distance Map Definition: The target distance map has the same map size
and cell size as the semantic map. It stores the shortest path length the
agent will travel to reach the target object. We define the distance value as
zero for the cells inside the target object and infinite for the cells inside
other obstacles except for those who contain the target (e.g., the table where
a target object is on it).

The input of our target distance prediction model is the local semantic map
$L_{S}$, which is $k\times 240\times 240$. Our model is required to predict a
local target distance map $L_{D}$ based on $L_{S}$ at each time step. In order
to predict the distance to the target based on the explored semantic map, the
model has to learn the spatial relations between different semantics (e.g.,
chairs often near a table). Since estimating the exact distance to the target
is difficult, we formulate this problem as a classification problem instead of
a regression problem. We split the distance into $n_{b}$ discrete bins, so
each bin corresponds to a distance range category. In this paper, we set the
number of distance range categories $n_{b}=5$, and the partition detail is
shown in Fig. 3.

Our model is a fully convolutional neural network with 3 downsample ResBlocks
[42], 3 upsample ResBlocks, and concatenating low-level feature map and
upsampled feature map on each level. The output is the local target distance
map $L_{D}$ for the target. Although one can also choose to predict a $n_{b}$
channel distance map conditioned on the target category, in this paper, we set
the output channel as $n_{b}\times n_{T}$, where $n_{T}$ is the number of
target categories. In this way, every $n_{b}$ channels form a group,
responsible for predicting the distance map of a specific target, so the
distance prediction of all the target categories could be trained
simultaneously.

Training: As shown in Fig. 3, the ground-truth (GT) target distance map is
generated based on the GT semantic map and traversable map. The distance
values for the cells within the traversable areas are calculated and split
into corresponding range categories. The local GT distance map is cropped from
the global GT map according to the agent’s pose. When collecting training
data, the agent performs random exploration and collects the local semantic
map as well as the local GT distance map. Note that there are no restrictions
on how to collect the samples since the model is trained offline.

(a) Integrating with Path Planning

(b) Door-Exploring-First Strategy

Figure 4: Terminology Illustration. The red cells denote the current agent’s
position. The grey cells denote the explored area, and the blue cells
represent the frontier $\mathcal{B}_{exp}$ between the explored and unexplored
area. Black cells indicate obstacles. (a) Integrating Distance Prediction with
Path Planning. The numbers within the frontier $\mathcal{B}_{exp}$ illustrate
the target distance our model has to predict (e.g., the length of the blue
dotted line). Based on the predicted distance values within
$\mathcal{B}_{exp}$, the path planning algorithm finds the optimal path (green
line) to move closer to the target (the red star). (b) In the Door-Exploring-
First strategy, if the agent observes a door or a passage (green cells), the
intersection of a sector with angle $\theta_{d}$ and the frontier
$\mathcal{B}_{exp}$ is defined as $\mathcal{B}_{d}$ (yellow cells).

Besides, since the obstacles and semantics are already known in the explored
area, if the target object is observed, the distance to the target can be
obtained for cells within the explored area. If the target is out of view, the
distance map within the explored area can also be calculated based on the
distance value around the exploration frontier. In this paper, we train our
model using the pixel-wise Cross-Entropy loss only between the predicted
distance range category and the ground-truth label within the area around the
exploration frontier.

###  III-C Local Policy

Our local policy consists of two parts: a mid-term goal selection strategy and
a path planner. At each time step, the goal selection strategy chooses mid-
term goals on the local semantic map $L_{S}$ based on the local target
distance map $L_{Dis}$. Here we design several simple strategies to
demonstrate the effectiveness of the distance map. Following [3], we use the
Fast Marching Method (FMM) [17] algorithm to plan a path based on the obstacle
channel of $L_{S}$. Finally, an action is selected according to the planned
path.

To obtain a mid-term goal, we design three strategies:

####  III-C1 Integrating with Path Planning

With estimated distance to the target for the cells around the exploration
frontiers, we plan a path with the smallest length from the current position
to the target object. Given the distance map $L_{D}$ and the exploration
frontier, we get the position of the mid-term goal

$p_{goal}=\mathop{\arg\min}_{p\in\mathcal{B}_{exp}}\\{d(p_{agent},p)+L_{Dis}(p)\\},$
$None$

where $p_{agent}$ is the current agent’s position on the local map,
$L_{Dis}(p)$ is the predicted distance value on the position $p$,
$d(p_{agent},p)$ is the distance to the current position, which can be
obtained by the path planning algorithm based on the obstacle map,
$\mathcal{B}_{exp}$ is the area around the current exploration frontier (see
Fig. 4). At each time step, the exploration area is expanded as the agent
moves, so a new mid-term goal is selected on the new $\mathcal{B}_{exp}$ based
on the updated $L_{Dis}(p)$. If all the predicted distance values in
$\mathcal{B}_{exp}$ are infinite , the random exploration is adopted. If there
is a target object on the semantic map (meaning that the target object is
found), the area of the target object is selected as the mid-term goal.

####  III-C2 Closest-First Strategy

This strategy makes a little change to the above strategy by simply choosing
the mid-term goal

$p_{goal}=\mathop{\arg\min}_{p\in\mathcal{B}_{exp}}L_{Dis}(p),$ $None$

which means we tend to move to the position where the predicted distance value
is the smallest, regardless of the agent’s current position.

####  III-C3 Door-Exploring-First Strategy

When an agent faces a door or a passage leading to another room, it is more
efficient to explore that room if the distance value in that room is smaller
(which means the agent might see some objects related to the target through
the opened door). Considering that, we design the door-exploring-first
strategy based on the closest-first strategy. The strategy first classifies
whether a door or a passage is in the observed RGB image through a
classification network based on ResNet50 [42]. If true, we obtain the area
that the door (passage) might lead to, that is, the area where the angle
between the current orientation of the agent is less than $\theta_{d}$, as
demonstrated in Fig. 4. Then we obtain the intersection of this area and the
frontier $\mathcal{B}_{exp}$, which is defined as $\mathcal{B}_{d}$. Finally,
We select the mid-term goal

$p_{goal}=\left\\{\begin{array}[]{cl}\mathop{\arg\min}_{p\in\mathcal{B}_{d}}L_{Dis}(p)&{p_{door}\geq
0.5,\ \mathcal{B}_{d}\neq\emptyset}\\\
\mathop{\arg\min}_{p\in\mathcal{B}_{exp}}L_{Dis}(p)&{others}\end{array}\right.,$

where $p_{door}$ means the probability of containing a door (passage). During
training, we use the cross-entropy loss to train the door classifier. In the
experiment, we find that when $\theta_{d}$ is small, the agent may change its
heading direction after reaching the door, leading to walking back and forth
at the same position. Therefore, in practice, we set $\theta_{d}$ as $None$.

After generating a mid-term goal with one of the strategies, we use the FMM
[17] algorithm to get the path, because the distance map used in this
algorithm can be easily obtained based on the predicted distance map
$L_{Dis}$. One can also use other path planning algorithms like $A^{*}$ [18].
Note that if $L_{Dis}$ is accurately predicted, the ObjectNav task will become
a traditional navigation task, since the goal coordinate can be viewed as
known.

TABLE I: Object Goal Navigation Result (Success Rate$\uparrow$ / SPL$\uparrow$). -GT: using local GT semantic maps. -PP: integrating with Path Planning, -CF: Closest-First strategy, -DEF: Door-Exploring-First strategy. † means our reimplementation. Method | Chair | Couch | Plant | Bed | Toilet | TV | Avg.
---|---|---|---|---|---|---|---
RandomExp-GT | 0.859/0.576 | 0.681/0.465 |  0.750/0.489 | 0.579/0.400 | 0.790/0.422 | 0.887/0.579 | 0.762/0.496
FrontierExp-GT [2] | 0.838/0.546 | 0.628/0.453 | 0.741/0.519 | 0.497/0.334 | 0.790/0.448 | 0.877/0.680 | 0.734/0.493
SemExp-GT [3] |  0.888/0.627 | 0.730/0.518 | 0.585/0.409 | 0.597/0.411 | 0.790/0.476 |  0.910/0.666 | 0.755/0.522
SSCExp-GT [6]† | 0.854/0.625 | 0.717/0.526 | 0.623/0.432 | 0.572/0.415 | 0.783/0.484 | 0.899/0.694 | 0.744/0.531
Ours-GT | 0.880/0.687 |  0.735/0.590 | 0.637/0.478 |  0.610/0.393 |  0.841/0.514 | 0.876/0.635 |  0.768/0.566
RandomExp | 0.566/0.257 | 0.398/0.174 | 0.358/0.157 | 0.421/0.246 | 0.305/0.158 | 0.000/0.000 | 0.403/0.193
FrontierExp [2] | 0.543/0.251 | 0.332/0.174 | 0.274/0.128 | 0.371/0.179 | 0.325/0.128 | 0.000/0.000 | 0.364/0.172
SemExp [3] | 0.622/0.289 | 0.385/0.220 | 0.344/0.141 | 0.415/0.213 | 0.280/0.124 | 0.000/0.000 | 0.410/0.197
SSCExp [6]† | 0.552/0.305 | 0.319/0.183 | 0.382/0.193 | 0.289/0.167 | 0.255/0.105 | 0.000/0.000 | 0.363/0.195
Ours-PP |  0.639/0.290 | 0.389/0.179 |  0.387/0.119 |  0.490/0.252 | 0.312/0.109 |  0.011/0.021 |  0.438/0.190
Ours-CF | 0.611/0.353 | 0.412/0.254 |  0.387/0.156 | 0.421/0.238 |  0.338/0.147 |  0.011/0.002 | 0.428/0.231
Ours-DEF | 0.630/0.346 |  0.438/0.258 | 0.368/0.162 | 0.447/0.240 | 0.312/0.154 |  0.011/0.003 | 0.436/0.232

##  IV EXPERIMENTS

###  IV-A Experimental Setup

We perform experiments on the Habitat [10] platform with Matterport3D (MP3D)
[11] dataset. The training set consists of 54 scenes and the test set consists
of 10 scenes. We follow [3] to set $n_{T}=6$ object goal categories: ‘chair’,
‘couch’, ‘plant’, ‘bed’, ‘toilet’, and ‘TV’. Same as [3], the semantic map has
$c_{S}=15$ categories; the global semantic map size is $480\times 480$
($24m\times 24m$). We follow [3] to use a Mask-RCNN [41] pretrained on MS-COCO
[43] as the semantic model.

To collect training data, we randomly initialize the agent’s position and
perform random exploration mentioned in Section IV-A. We collect 1.2 million
samples to train our model. We use pixel-wise Cross-Entropy loss within the
area of 1m distance to the exploration frontiers. The loss weight for the
distance range categories from 1m to infinite is set as from 5 to 1. Adam
optimizer is used with a learning rate of 0.00001. During the evaluation, we
split the scene into several floors according to the scene graph label of
MP3D. For each scene, we first uniformly sample a floor, and then sample the
goal among all the targets categories available on this floor. The agent is
randomly initialized at the position with a distance margin to the target. In
this way, we sample a total of 1200 test episodes. The maximum length for each
episode is 500 steps, and the success threshold is 1m.

We use two metrics to evaluate the performance of ObjectNav:

  * •

Success Rate: The ratio of the episode where the agent successfully reaches
the goal;

  * •

SPL [12]: Success weighted by normalized inverse Path Length, which measures
the efficiency of finding the goal.

We compare our method with the following baselines:

####  IV-A1 Random Exploration (RandomExp)

Instead of random walk, we design a simple strategy to urge the agent to
explore the environment randomly. We set the mid-term goal as one of the
corners of the local map. With the change of local map boundaries due to the
agent’s movement, the mid-term goal also changes with the boundaries. This
goal is periodically switched clockwise among four corners per 100 steps. It
also serves as a supplement to our goal select strategy as mentioned in
Section III-C.

####  IV-A2 Frontier-based Exploration (FrontierExp)

When performing Frontier-based Exploration [2], the agent will select the
nearest traversable and unvisited frontier cell as the mid-term goal.

####  IV-A3 SemExp [3]

SemExp consists of a semantic mapping module, an RL policy deciding mid-term
goals based on the semantic map, and a local path planner based on FMM [17].
The difference between it and our method is the way to select mid-term goals.
Specifically, we utilize the target distance map rather than the RL policy.
The RL model is trained with 10 million steps.

####  IV-A4 Semantic Scene Completion (SSCExp)

Following SSCNav [6], we utilize 4 down-sampling residual blocks [42] and 5
up-sampling residual blocks to build the scene completion and confidence
estimation module. It predicts the full semantic map and confidence map from
the observed map constructed by the semantic mapping module. We add this
semantic scene completion module to SemExp [3]. Then the RL policy generates a
mid-term goal based on the completed maps. The scene completion and confidence
estimation model is trained with 2.7 million samples and the RL model is
trained with 10 million steps.

###  IV-B Main Results

Tab. I shows the result of our method. Ours-PP indicates the strategy
integrating with Path Planning, Ours-CF is the Closest-First strategy, Ours-
DEF corresponds to the Door-Exploring-First strategy. $*$-GT denotes using
local ground-truth semantic maps within the explored area to replace the
constructed map at each time step. Ours-GT uses the Closest-First strategy to
get action. Among the three strategies using distance map, Ours-PP has the
highest success rate but low SPL; this is because the noise of the distance
map makes the agent follow a zigzag route, increasing the total path length.
Ours-DEF has a good success rate and the best efficiency.

Comparison with Baseline Methods. As shown in Tab. I, our method outperforms
the baseline method SemExp[3] (+2.6% in success rate, +3.5% in SPL). The
result demonstrates that the target distance map is able to guide the agent to
the target object more efficiently. Besides, our method only uses 12% data
(1.2 million vs. 10 million) to train the model compared with the RL-based
method SemExp, indicating that our method is more sample efficient. It can be
seen that all methods’ success rates and SPL of target ’TV’ are close to 0.
Comparing with the performance using GT semantic maps, we attribute this
phenomenon to the unsatisfying performance of the semantic model. As mentioned
in [38], the 3D reconstruction quality in some MP3D scenes is not gratifying.
After eliminating the factor of semantic mapping by using GT semantic maps,
the SPL of our method exceeds SemExp [3] by 4.4%. As for the baseline based on
semantic scene completion, the performance of SSCExp is relatively poor in our
experiment. Note that our setting is different from [6] in camera angle,
semantic map size, semantic categories, target categories, etc. We suscept the
reason for the poor performance is that our local map size ($12m\times 12m$)
is much larger than [6] ($6m\times 6m$), causing it difficult to complete the
scene in the local map.

Figure 5: Prediction Example. Our model can guide the agent to the target
object since the predicted directions are correct. From left to right are:
local semantic maps, predicted local distance maps, and local distance GT
maps. The red star denotes target objects. The blue dot corresponds to the
mid-term goal. The red arrow denotes the agent’s pose. The non-shaded area in
the distance map indicates the area of exploration frontiers
$\mathcal{B}_{exp}$. Figure 6: Prediction Failure Cases Example. The left
images of each example are the semantic maps. The images on the right show the
predicted distance map. Upper left: based on the observed object "sink" (blue
dot circle), the direction prediction of the target "toilet" is wrong (blue
dot box). Upper right: it also predicts the target "chair" has a "< 1m"
distance (green dot box) around the other two beds (green dot circle), because
it is uncertain whether there is a chair near a bed unless having explored all
the area around the bed. Bottom row: our model cannot predict the distance due
to the lack of semantic prediction (no object around the agent (bottom left),
or the semantic model can not detect the object (bottom right)).

How does the target distance map work? We test the quantitative performance of
our model during the whole process of navigation. As shown in Tab. II, the
performance is surprisingly low. Nevertheless, our model still guides the
agent to the target object because the predicted directions to the target are
correct (see Fig. 5). If it predicts a relatively small distance in the
direction toward the target compared with other directions, the agent can
still reach the target.

TABLE II: Performance of Distance Prediction Model Distance | <1m | 1-2m | 2-4m | 4-8m | >8m | Avg.
---|---|---|---|---|---|---
Precision | 0.045 | 0.193 | 0.078 | 0.109 | 0.844 | 0.254
Recall | 0.354 | 0.081 | 0.064 | 0.047 | 0.842 | 0.278
TABLE III: Results of Different Representations  Representation | Partition (m) | Success Rate$\uparrow$ | SPL$\uparrow$
---|---|---|---
Discrete | [1,2,4,8,$\infty$] | 0.428 | 0.231
[1,2,4,$\infty$] | 0.416 | 0.226
[1,2,$\infty$] | 0.417 | 0.231
[1,$\infty$] | 0.405 | 0.226
[2,4,8,12,$\infty$] | 0.390 | 0.209
Continuous | - | 0.412 | 0.204

We further studied some wrong predictions. In Fig. 6 upper left, based on the
observed object "sink" (blue dot circle), the direction of the target "toilet"
is wrong (blue dot box). In Fig. 6 upper right, it predicts the target "chair"
has a "< 1m" distance (green dot box) around the other two beds (green dot
circle), but in fact, there are no chairs around them. It demonstrates that
our model has successfully learned the knowledge that "chairs may be close to
beds." Nevertheless, unfortunately, it is still uncertain whether there is a
chair near a bed unless the agent has explored all the area around the bed. On
the contrary, determining a target object is NOT near an unrelated object is
much easier. This may, to some extent, explain why the performance for the
categories "<= 8m" is poor, while the performance of the"> 8m" category in
Tab. II is satisfying. Fig. 6 bottom row shows the cases that our model is not
able to predict the distance due to the lack of semantic prediction. This
often happens when there is no object around the agent (Fig. 6 bottom left),
or the semantic model can not detect the object (Fig. 6 bottom right). This is
also a reason for the low performance in Tab. II.

Fig. 7 illustrates how our method navigates to the target object with the help
of the target distance map. In the beginning, the target distance prediction
is random or invalid (Fig. 7 row 1), because there are few objects on the
semantic map, or because the target is far from the agent. The agent can be
seen as random exploration during this phase. As the agent explores and
receives more observation, the model begins to predict the target distance
distribution more accurately and guide the agent toward the direction of the
potential target (in Fig. 7 row 2-3, the agent is looking for chairs around
beds). If there is no target in the supposed direction, the distance map is
corrected based on the new semantic map, and the agent will head to another
direction with a low target distance. If the distance map is correct, the
agent will reach the target (Fig. 7 row 4-5).

Continuous or Discrete Representation for Target Distance Map. In the target
distance prediction model, we formulate it as the category classification
problem. We also design a regression framework to predict the continuous
distance. Tab. III shows the result of different representations using the
Closest-First Strategy. The best result of discrete representation achieves a
higher Success Rate and SPL than predicting the continuous distance. The
result indicates that although the distance to a target is continuous,
predicting a precise value is not easy. Besides, the result of different
distance partitions indicates that the category of larger distance plays a
less important role than smaller distance.

Figure 7: Simulation episode example. With the help of the target distance
map, the agent first randomly explores, then searches the target around
related objects (bed), and finally reaches the target (chair). From left to
right are RGB observations, semantic map, and predicted distance map. The red
arrow indicates the agent pose. The red line denotes the traversed path.
Figure 8: The real-world environment and the sensor configuration of our robot
base.  Figure 9: Real-world episode example. In a real-world episode with
"plant" as the target object, our method first searches the area around the
bed, then the dining table. Finally, it searches the area around the TV stand
and finds the target.

###  IV-C Real World Experiment

We also deploy our method to a real-world mobile base (see Fig. 8). Our robot
is equipped with a Kinect V2 camera, a 2D LiDAR, and an onboard computer (with
an Intel i5-7500T CPU and an NVIDIA GeForce GTX 1060 GPU). The LiDAR is only
used to perform localization, and no prior map is built. The average time
consumption of each module is shown in Tab. IV. We test our method in a
$120m^{2}$ house with one living room and three bedrooms. The max speed of our
robot is limited to $0.2m/s$. The object goal categories in real-world
experiments are the same as simulation experiments except "toilet."

TABLE IV: Average time consumption of each module  Module |  Mask-RCNN |  Semantic Mapping |  Distance Prediction |  Goal Selection |  Total
---|---|---|---|---|---
Time (ms) |  276.7 |  31.9 |  2.7 |  54.6 |  365.9
TABLE V: Real World Performance |  Chair |  Couch |  Plant |  Bed |  TV |  Avg.
---|---|---|---|---|---|---
Success Rate (%) |  86.7 |  100 |  100 |  76.7 |  60.0 |  84.7
Time (s) |  140.7 |  129.6 |  100.5 |  126.6 |  176.4 |  134.8

The goal "toilet" can not be reached because there are door saddles that are
not traversable for our robot base. Fig. 9 shows one of the episodes with
target "plant." In a total of 150 episodes, we evaluate the success rate and
the average time to reach the goal among success episodes. The quantitative
result in Tab. V demonstrates that our method is transferred well to the real
world. The real-world performance is much higher than that in simulation due
to the better performance of the semantic model in the real world.

###  IV-D Failure Cases

In the experiments, we find that most failure cases are due to low semantic
map accuracy. Sometimes the semantic model can not detect the object, and
sometimes there is wrong detection or wrong projection to the ground due to
the semantic segmentation noise or depth image noise. Besides, since the
camera angle is fixed, and has a certain height from the ground, sometimes it
cannot see the object in a small room, like the object "toilet" in a lavatory.
We also find some cases similar to the failure modes “Goal Bug” and “Void” as
mentioned in [4].

##  V CONCLUSIONS

This paper presents a navigation framework based on predicting the distance to
the target object. In detail, we design a model which takes a bird’s-eye view
semantic map as input, and estimates the path length from frontier cells to
the target. Based on the distance map, the agent could navigate to the target
objects with simple goal selection strategies and a path planning algorithm.
Experimental results on the MP3D dataset demonstrate that our method
outperforms baselines methods on success rate and SPL. The real-robot
experiment also shows that our method can be transferred to the real world.
Future work would focus on predicting the target distance map more accurately,
such as using the room-type prediction as auxiliary tasks. We believe that
with a more powerful target prediction model and RL policy, our method will
achieve much better performance.

## References

  * [1] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans. ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects. In arXiv:2006.13171, 2020.
  * [2] B. Yamauchi. "A frontier-based approach for autonomous exploration." Proceedings of 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation, 1997.
  * [3] D. S. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In Neural Information Processing Systems (NeurIPS), 2020.
  * [4] Ye, Joel, et al. Auxiliary Tasks and Exploration Enable ObjectNav. In arXiv:2104.04112 (2021).
  * [5] Tommaso Campari, Paolo Eccher, Luciano Serafini, and Lamberto Ballan. Exploiting Scene-specific Features for Object Goal Navigation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [6] Yiqing Liang, Boyuan Chen, Shuran Song. SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation. In International Conference on Robotics and Automation(ICRA), 2020.
  * [7] Bar Mayo, Tamir Hazan, Ayellet Tal. Visual Navigation with Spatial Attention. In arXiv:2104.09807, 2021.
  * [8] B. Matthew, S. Ritter, et al. Reinforcement learning, fast and slow. Trends in cognitive sciences 23.5 (2019): 408-422.
  * [9] G. Georgakis, B. Bucher, K. Schmeckpeper, S. Singh, K. Daniilidis. Learning to Map for Active Semantic Goal Navigation. In arXiv:2106.15648, 2021.
  * [10] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research. In International Conference on Computer Vision, 2019.
  * [11] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3D: Learning from RGBD data in indoor environments. In arXiv:1709.06158, 2017.
  * [12] P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir. On evaluation of embodied navigation agents. In ArXiv: 1807.06757, 2018.
  * [13] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics. MIT Press, 2005.
  * [14] P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. In IEEE transactions on Systems Science and Cybernetics, vol. 4, no. 2, pp. 100–107, 1968.
  * [15] S. Karaman and E. Frazzoli.‘ Sampling-based algorithms for optimal motion planning. In International Journal of Robotics Research (IJRR), vol. 30, no. 7, pp. 846–894, 2011.
  * [16] Guo, Hongliang, et al. Safe path planning with Gaussian process regulated risk map. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
  * [17] James A Sethian. A fast marching level set method for monotonically advancing fronts. In Proceedings of the National Academy of Sciences, 93(4):1591–1595, 1996.
  * [18] Hart P E, Nilsson N J, Raphael B. A formal basis for the heuristic determination of minimum cost paths. In IEEE transactions on Systems Science and Cybernetics, 1968, 4(2): 100-107.
  * [19] L. Kaelbling, M. Littman, and A. Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence, vol.101, no. 1-2, pp. 99–134, 1998.
  * [20] A. A. Makarenko, S. B. Williams, F. Bourgault and H. F. Durrant-Whyte. An Experiment in Integrated Exploration. In IEEE International Conference on Intelligent Robots and Systems(IROS), 2002.
  * [21] R. Martinez-Cantin, N. Freitas, E. Brochu, J. Castellanos, and A. Doucet. A bayesian exploration-exploitation approach for optimal online sensing and planning with a visu- ally guided mobile robot. Autonomous Robots, 2009.
  * [22] T. Chen, S. Gupta, A. Gupta. Learning exploration policies for navigation. In International Conference on Learning Representations(ICLR), 2019.
  * [23] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, R. Salakhutdinov. Learning to Explore using Active Neural SLAM. In International Conference on Learning Representations(ICLR), 2020.
  * [24] S. K. Ramakrishnan, Z. Al-Halah, and K. Grauman. Occupancy Anticipation for Efficient Exploration and Navigation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [25] J. A. Placed, J. A. Castellanos. A deep reinforcement learning approach for active slam. Applied Sciences, 10(23), 8386, 2020.
  * [26] Y. Zhu, R. Mottaghi, E. Kolve, J. J Lim, A. Gupta, Li Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In International Conference on Robotics and Automation(ICRA), 2017.
  * [27] A. Mousavian, A. Toshev, M. Fiser, J. Kosecka, and J. Davidson. Visual representations for semantic target driven navigation. arXiv preprint arXiv:1805.06066, 2018.
  * [28] J. Zhang, L. Tai, J. Boedecker, W. Burgard, and . Liu. Neural slam: Learning to explore with external memory. arXiv preprint arXiv:1706.09520, 2017.
  * [29] Wortsman, Mitchell, et al. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  * [30] Kuan Fang, Alexander Toshev, Li Fei-Fei, Silvio Savarese. Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [31] Tsotsos, John K. On the relative complexity of active vs. passive visual search. International journal of computer vision 7.2, pp. 127-141, 1992.
  * [32] Kollar, Thomas, and Nicholas Roy. Utilizing object-object and object-scene context when planning to find things. In 2009 IEEE International Conference on Robotics and Automation, pp. 986-1002, 2009.
  * [33] C. Wang, J. Cheng, J. Wang, , X. Li, M. Meng. Efficient object search with belief road map using mobile robot. IEEE Robotics and Automation Letters, vol 3, no.4, pp. 3081-3088, 2018.
  * [34] Kunze, Lars, Keerthi Kumar Doreswamy, Nick Hawes. Using qualitative spatial relations for indirect object search. 2014 IEEE international conference on robotics and automation (ICRA), 2014.
  * [35] Shuran Song, Andy Zeng, Angel X. Chang, et al. Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [36] S. K. Ramakrishnan, T. Nagarajan, Z. Al-Halah, K. Grauman. Environment predictive coding for embodied agents. In arXiv:2102.02337, 2021.
  * [37] Saurabh Gupta, James Davidson, Sergey Levine, et al. Cognitive Mapping and Planning for Visual Navigation. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2017
  * [38] Francesco Giuliari, Alberto Castellini, Riccardo Berra. POMP++: Pomcp-based Active Visual Search in unknown indoor environments. In IEEE International Conference on Intelligent Robots and Systems(IROS), 2021.
  * [39] Nikolay Savinov, Alexey Dosovitskiy, Vladlen Koltun. Semi-Parametric Topological Memory for Navigation. In International Conference on Learning Representations (ICLR), 2017.
  * [40] D. S. Chaplot, R. Salakhutdinov, A. Gupta, S. Gupta. Neural Topological SLAM for Visual Navigation. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [41] He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969. 2017.
  * [42] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, June 2016.
  * [43] Tsung-Yi Lin, Michael Maire, et al. Microsoft coco: Common objects in context. In ECCV, 2014\.