# Electromagnetic Illusion in Smart Environments

Hamidreza Taghvaee*ref1  Mir Lodroref1  Neekar M Mohammedref2ref3  Sergio
Terranovaref2  Sendy Phangref1  Martin Richterref2  and Gabriele
Gradoniref1ref2  ref1George Green Institute for Electromagnetics Research,
Department of Electrical and Electronics Engineering, University of
Nottingham, Nottingham NG7 2RD, United Kingdom (e-mail:
<EMAIL_ADDRESS>ref2School of Mathematical Sciences,
University of Nottingham, Nottingham NG7 2RD, United Kingdom ref3Department of
Mathematics, College of Science, University of Sulaimani, Sulaymaniyah,
Kurdistan Region, Iraq

###### Abstract

Metasurfaces can be designed to achieve prescribed functionality. Careful
meta-atom design and arrangement achieve homogeneous and inhomogeneous layouts
that can enable exceptional capabilities to manipulate incident waves.
Inherently, the control of scattering waves is crucial in wireless
communications and stealth technologies. Low-profile and light-weight coatings
that offer comprehensive manipulation are highly desirable for applications
including camouflaging, deceptive sensing, radar cognition control, and
defense security. Here, we propose a method that achieves electromagnetic
illusion without altering the object. A proof of principle is proposed and
practiced for one-dimensional media. The idea is to engineer the environment
instead of the object coating. This work paves the way for versatile designs
that will improve electromagnetic security applications with the aid of smart
environments.

##  1 Introduction

Any type of deceiving sense is an illusion. Naturally, there are different
illusions associated with human senses. For instance, an optical illusion is a
misleading visual grasp of a real object [1]. A general extension of this
concept is an electromagnetic (EM) illusion that confuses sensors or receiving
antennas and thus has many potential protection-oriented applications. Since
2006, when Pendry et al [2] and Leonhardt [3] designed invisibility cloaks as
an example of EM illusions, many kinds of illusion devices were designed and
realized [4, 5]. For example, an illusion device can make an object look
larger, appear at another place, or with different shapes and components [6,
7, 8].

Later, metasurfaces are introduced as a powerful platform due to their
complete control over the scattered fields and advanced information encoding
capabilities [9, 10]. These features make them an excellent candidate for
compact illusions devices. Recently, reconfigurable metasurfaces are used for
dynamic illusions by controlling electronic elements [11]. A tunable graphene-
based metasurface is designed by placing graphene ribbons on a dielectric
cavity resonator. The wavefront of the field reflected from a triangular bump,
covered by the metasurface activated by an electric bias, resembles the one
generated by a spherical object [12].

There are a plethora of studies in EM illusions. However, all of them suggest
adopting an object coating to form an illusion. In addition, these studies
have been limited to free space, thus assuming no back-reflection from the
environment. We argue that engineering the object itself may disrupt its
function, and borrow complicated designs as well as manufacturing/installation
procedures. Also, not in all the scenarios we have direct access to the
object. For instance, to conceal the object from imaging/localization
technologies, cloaking within a confined propagation environment, e.g., rooms,
streets, should be achieved. The task of object illusion in complex
environments is challenging and has attracted limited interest in the
literature due to the complicated nature of the wave boundary value problems
associated with it. Finding an appropriate mathematical description of the
object-mirror-environment interaction process remains an open problem.

This scenario is radically different from free-space with regards to the
physics of waves and mathematical modeling. These two aspects are intertwined
and it is thus necessary to solve the boundary value-problem self-consistently
accounting for environment, object and cloak altogether. The solutions present
in the literature can thus be evolved to achieve object cloaking within a
confined environment. From a mathematical point of view, the problem of object
illusion in chaotic environments poses extraordinary challenges. It is a
cascaded scattering process that can be modeled with products and sums of
transfer matrices with arbitrary dimensions.

Here, we considered a 1D case of the problem as a proof of concept. Figure 1,
represents the cartoon of the model under study. The objective of this study
is to engineer a transmissive metasurface before the object or a reflective
metasurface behind the object that creates an EM illusion. The illusion
consists of altering the object’s scattering features to deceive the
observer’s sensing. In the following sections, we describe how to create the
illusion by engineering the susceptibility of the transmissive metasurface or
the surface impedance of the reflective metasurface. As an example, we
consider the illusion of hiding the object.

Figure 1: Illustration of the model under study in which electromagnetic
illusion is created with a smart environment empowered by metasurfaces. Blue
dashed lines represent a transmissive metasurface and black grooved lines
represent a reflective metasurface.

##  2 Methodology

In this section, impedance boundary conditions (IBCs) and generalized sheet
transition conditions (GSTCs) are reviewed. Then scattering coefficients of
one-dimensional interfaces are defined, and related formulas derived from IBCs
and GSTCs are given. The procedure starts by defining the boundary-value
problem associated with object illusion problems. This includes state of the
art wave-scattering modelling within confined complex environments.
Conventional boundary conditions describe the relationship between field
discontinuities across boundary and current sources distributed along
boundary, which can be expressed as

$\hat{n}\times(\vec{H_{2}}-\vec{H_{1}})=\vec{J_{se}}$ (1)
$(\vec{E_{2}}-\vec{E_{1}})\times\hat{n}=\vec{J_{sm}}$ (2)

where ($E_{1}$, $H_{1}$) and ($E_{2}$, $H_{2}$) represent the total fields at
both sides of the interface, while $J_{se}$ and $J_{sm}$ are the surface
electric and magnetic current densities along the boundary. The field jump
across EM surfaces can be characterized by their effective sheet impedances,
based on which IBCs as derived in [13]. The electric sheet impedance $Z_{e}$
is defined as the ratio of average electric fields $E_{av}$ along the boundary
to surface electric current density $J_{se}$, while the magnetic sheet
impedance $Z_{m}$ is the ratio of $J_{sm}$ to average magnetic fields
$H_{av}$. Consequently, the IBCs describe the relationship between average
fields along surfaces and field discontinuities across surfaces

$\hat{n}\times(\vec{H_{2}}-\vec{H_{1}})=Z_{e}^{-1}E_{av}$ (3)
$(\vec{E_{2}}-\vec{E_{1}})\times\hat{n}=Z_{m}H_{av}$ (4)

Note that all field components that appear in IBCs are tangential. With IBCs,
field discontinuities across EM surfaces can be determined once their sheet
impedances $Z_{e}$ and $Z_{m}$ are known. However, they are not characteristic
parameters of EM surfaces and their values for different situations cannot be
readily determined. The characteristic parameters of one-dimensional
interfaces are effective susceptibilities, whose values are independent of
applied fields. Based on susceptibilities, GSTCs are initially derived for
metafilms [14].

$\hat{n}\times(\vec{H_{2}}-\vec{H_{1}})=j\omega\epsilon_{0}\chi_{e}E_{av}-\hat{n}\times\nabla_{t}(\chi_{m}H_{av})$
(5)
$(\vec{H_{2}}-\vec{H_{1}})\times\hat{n}=j\omega\mu_{0}\chi_{m}H_{av}+\hat{n}\times\nabla_{t}(\chi_{e}E_{av})$
(6)

where one-dimensional electric $\chi_{e}$ and magnetic $\chi_{m}$ tensors
represent the effective surface electric and magnetic susceptibilities
respectively. We can calculate the transmission and reflection coefficients
for this interface as [15]

$\tau_{M}=\frac{1-(k_{0}/2)^{2}\chi_{e}\chi_{m}}{1+(k_{0}/2)^{2}\chi_{e}\chi_{m}-jk_{0}/2(\chi_{m}-\chi_{e})}$
(7)
$\rho_{M}=\frac{jk_{0}/2(\chi_{m}+\chi_{e})}{1+(k_{0}/2)^{2}\chi_{e}\chi_{m}-jk_{0}/2(\chi_{m}-\chi_{e})}.$
(8)

Using the IBCs for the fields, we can relate the forward-backward fields on
one side of the interface to those on the other side, expressing the
relationship in terms of a $2\times 2$ transfer matrix. Figure 2 shows both
the interface and the associated shift matrices in terms of reflection $\rho$
and transmission coefficients $\tau$.

Figure 2: A continuous interface can be modeled as a $2\times 2$ shift matrix
(left inset). We can use a $2\times 2$ propagation matrix to model the
propagation within a medium (right inset). $k$ is the wavenumber, $\eta$ is
the medium impedance and $l$ is thickness of the medium.

If there are several interfaces, we can propagate our forward-backward fields
from one interface to the next with the help of a $2\times 2$ propagation
matrix. The combination of a shift and a propagation matrix relating the
fields across different interfaces will be referred to as a transfer or
transition matrix. Multiple interface problems can be handled in a
straightforward way with the help of the transfer matrix. Figure 3 shows the
simulation model for an arbitrary slab in the air that has a discontinuous
interface on one side and a PEC wall on the others.

Figure 3: Simulation model.

##  3 Results

In the first case, we study the possibility of hiding the slab by placing a
metasurface along the PEC wall. The slab is made of FR4 with permittivity
$\epsilon_{d}=3.9-0.08i$ and thickness $D=50mm$ also the air space from
lateral sides is $D=100mm$. A planar impinging wave encounters this slab and
the wave partially transmits and then bounces back and forth inside the slab.
The transmitted part hits the PEC and reflects back. The transfer matrix reads

$\begin{bmatrix}E_{+}^{1}\\\
E_{-}^{1}\end{bmatrix}=\frac{1}{\tau_{1}\tau_{2}\tau_{3}}\times\\\
\begin{bmatrix}Z_{1}^{-1}&\rho_{1}Z_{1}\\\
\rho_{1}Z_{1}^{-1}&Z_{1}\end{bmatrix}\begin{bmatrix}Z_{2}^{-1}&\rho_{2}Z_{2}\\\
\rho_{2}Z_{2}^{-1}&Z_{2}\end{bmatrix}\begin{bmatrix}Z_{3}^{-1}&\rho_{3}Z_{3}\\\
\rho_{3}Z_{3}^{-1}&Z_{3}\end{bmatrix}\begin{bmatrix}E_{+}^{7}\\\
E_{-}^{7}\end{bmatrix}$ (9)

where $Z_{n}=e^{-jk_{n}l_{n}}$ is obtained via the propagation matrix
($Z_{1}=Z_{3}$) and $\rho_{n}$, $\tau_{n}$ are the local reflection and
transmission coefficients defined as

$\rho_{n}=\frac{\eta_{n+1}-\eta_{n}}{\eta_{n+1}+\eta_{n}},\tau_{n}=\frac{2\eta_{n+1}}{\eta_{n+1}+\eta_{n}}.$
(10)

We then calculate the total reflection at $z=0$ as
$\Gamma=E_{-}^{1}/E_{+}^{1}$. Total reflection amplitude and phase are shown
in Fig.!4. Resonances correspond to the stationary waves forming within a
cavity-like structure and generated from the interference of forward and
backward wave fields.

Figure 4: Reflection coefficients from the FR4 slab inside air and background
is a PEC wall.

Now, defining $\Gamma\equiv-Z_{1}^{6}$, we can find the required surface
impedance of the metasurface to conceal the slab scattering. The essential
surface impedance reads

$\eta_{m}=\eta_{0}\frac{A+B}{A-B}$ (11)
$A=-Z_{2}Z_{1}^{9}(\rho_{3}+\rho_{2}Z_{2}^{2})-Z_{1}^{5}Z_{2}(\rho_{3}\rho_{2}+Z_{2}^{2})$
(12)
$B=Z_{2}Z_{1}^{3}(\rho_{2}+\rho_{3}Z_{2}^{2})+Z_{1}^{7}Z_{2}(1+\rho_{3}\rho_{2}Z_{2}^{2})$
(13)

The surface impedance and the corresponding reflection coefficients are shown
in Fig. 5. The negative real part of the surface impedance implies that we
need active elements to hide the slab and reflection amplitudes greater than
one confirm this requirement.

Figure 5: Surface impedance of the engineered metasurface and corresponding
reflection coefficients.

We can also put a metasurface sheet before the slab and by applying GSTCs find
the required susceptibilities to hide the slab from impinging wave. By
replacing $\rho_{1}$ with $\rho_{M}$ in Eq. (8) and only considering the
electric susceptibility ($\chi_{m}=0$)

$\chi_{e}=\frac{a}{j\frac{k_{0}}{2}(a+b)}$ (14)

where

$a=Z_{2}(Z_{1}^{2}-\rho_{3})+\rho_{2}Z_{2}^{-1}(\rho_{3}Z_{1}^{2}-1)-\\\
Z_{1}^{4}(Z_{2}^{-1}+\rho_{3}\rho_{2}Z_{2})+Z_{1}^{6}(\rho_{3}Z_{2}^{-1}+\rho_{2}Z_{2})$
(15)
$b=Z_{2}^{-1}(Z_{1}^{-2}-\rho_{3})+\rho_{2}Z_{2}(\rho_{3}Z_{1}^{-2}-1)-\\\
Z_{1}^{8}(Z_{2}+\rho_{3}\rho_{2}Z_{2}^{-1})+Z_{1}^{6}(\rho_{2}Z_{2}^{-1}+\rho_{3}Z_{2}).$
(16)

The real and imaginary part of the obtained electric susceptibility with
corresponding reflection coefficients are shown in Fig. 6. Here, we also need
active elements to achieve camouflage. While susceptibilities are not showing
high values, their associated reflection amplitudes are high. This means that
reflection coefficients of metasurfaces are more sensitive to susceptibility
compared to surface impedance.

Figure 6: Surface susceptibility of the engineered metasurface and
corresponding reflection coefficients.

Finally, we notice that modeling the metasurface either via a surface
impedance $\eta_{m}$ over the PEC wall or am electric susceptibility
$\chi_{e}$ before the slab, we can conceal the scattering from the slab as
shown in Fig. 7. The amplitude of the reflection coefficient is unity without
resonances and the reflection phase is as a PEC reflection, which shows that
the dielectric slab is hidden via metasurface manipulation of scattered waves.

Figure 7: Reflection coefficients from the slab after placing the metasurfaces
(solid lines). Reflection coefficients from the PEC without any slab in the
medium (symbols).

The proposed strategy is applicable to other illusion effects. This study
shows that environmental engineering is a solid tool for object illusion
without the need for direct access to the object itself.

##  4 Conclusion

A new paradigm of smart environment for dynamic electromagnetic illusion is
introduced. We envision reconfigurable metasurfaces using active elements to
change the interaction waves between object and environment and thus result in
electromagnetic illusion. We proposed a proof of principle within a one-
dimensional medium. In contrast to previous works, we avoided using any object
coating. Also, the object is not necessarily in free space. An FR4 slab is
hidden from radar sensors by placing the metasurface in the background or
before the slab. This work paves the way for 2D or 3D metasurface designs with
powerful illusion abilities.

##  5 Acknowledgements

This work has been supported by the EPSRC under Grant EP/V048937/1 and in part
by the European Commission through the H2020 RISE-6G Project under Grant
101017011.

## References

  * [1] Yun Lai, Jack Ng, HuanYang Chen, DeZhuan Han, JunJun Xiao, Zhao-Qing Zhang, and C. T. Chan, ”Illusion Optics: The Optical Transformation of an Object into Another Object,” _Phys. Rev. Lett._ , 102, 25, Jun 2003, pp. 253902.
  * [2] J. B. Pendry, D. Schurig and D. R. Smith, “Controlling Electromagnetic Fields,” _Science_ , 312, 5781, 2006, pp. 1780–1782.
  * [3] Ulf Leonhardt, “Optical Conformal Mapping,” _Science_ , 312, 5781, 2006, pp. 1777–1780.
  * [4] D. Schurig, J. J. Mock, B. J. Justice, S. A. Cummer, J. B. Pendry, A. F. Starr and D. R. Smith, “Metamaterial Electromagnetic Cloak at Microwave Frequencies,” _Science_ , 314, 5801, 2006, pp. 977–980.
  * [5] R. Liu, C. Ji, J. J. Mock, J. Y. Chin, T. J. Cui and D. R. Smith, “Broadband Ground-Plane Cloak,” _Science_ , 323, 5912, 2009, pp. 366–369.
  * [6] Wei Xiang Jiang and Tie Jun Cui, “Moving targets virtually via composite optical transformation,” _Opt. Express_ 18, 5 2010, pp. 5161–5167.
  * [7] Wei Xiang Jiang, Hui Feng Ma, Qiang Cheng, and Tie Jun Cui, “Illusion media: Generating virtual objects using realizable metamaterials,” _Appl. Phys. Lett._ 96, 12, 2010, pp. 121910.
  * [8] Wei Xiang Jiang, Hui Feng Ma, Qiang Cheng, and Tie Jun Cui, “Virtual conversion from metal object to dielectric object using metamaterials,” _Opt. Express_ , 18, 11, 2010, pp. 11276–11281.
  * [9] C. L. Holloway, E. F. Kuester, J. A. Gordon, J. O’Hara, J. Booth, and D. R. Smith, “An overview of the theory and applications of metasurfaces: The two-dimensional equivalents of metamaterials,” _IEEE Antennas Propag. Mag._ , 54, 2, Apr. 2012, pp. 10–35.
  * [10] H.-T. Chen, A. J. Taylor, and N. Yu, “A review of metasurfaces: Physics and applications,” _Rep. Prog. Phys._ , 79, 7, Jul. 2016, Art. no. 076401.
  * [11] Cheng Huang, Jianing Yang, Xiaoyu Wu, Jiakun Song, Mingbo Pu, Changtao Wang, and Xiangang Luo, “Reconfigurable Metasurface Cloak for Dynamical Electromagnetic Illusions,” _ACS Photonics_ 5, 5, 2018, pp. 1718–1725.
  * [12] Sudipta Romen Biswas, Cristian E. Gutiérrez, Andrei Nemilentsau, In-Ho Lee, Sang-Hyun Oh, Phaedon Avouris, and Tony Low, “Tunable Graphene Metasurface Reflectarray for Cloaking, Illusion, and Focusing,” _Phys. Rev. Applied_ , 9, 3, Mar 2018, pp. 034021.
  * [13] Senior, T.B.A., “Impedance boundary conditions for imperfectly conducting surfaces,” _Appl. sci. Res._ 8, 418, 1960\.
  * [14] Liu, Xiao and Yang, Fan and Li, Maokun and Xu, Shenheng, “Generalized Boundary Conditions in Surface Electromagnetics: Fundamental Theorems and Surface Characterizations,” _Applied Sciences_ , 9, 9, 2019, 1891\.
  * [15] C. L. Holloway, M. A. Mohamed, E. F. Kuester and A. Dienstfrey, “Reflection and transmission properties of a metafilm: with an application to a controllable surface composed of resonant particles,” _IEEE Transactions on Electromagnetic Compatibility_ , 47, 4, Nov. 2005, pp. 853-865.


# ProgressLabeller: Visual Data Stream Annotation for Training Object-Centric
3D Perception

Xiaotong Chen Huijie Zhang Zeren Yu Stanley Lewis Odest Chadwicke Jenkins 1X.
Chen, H. Zhang, Z. Yu, S. Lewis and O. C. Jenkins are with the Department of
Electrical Engineering and Computer Science, and Robotics Institute at the
University of Michigan, Ann Arbor, MI 48109 USA
[cxt|huijiezh|yuzeren|stanlew|ocj] @umich.eduDigital Object Identifier (DOI):
see top of this page.

###### Abstract

Visual perception tasks often require vast amounts of labelled data, including
3D poses and image space segmentation masks. The process of creating such
training data sets can prove difficult or time-intensive to scale up to
efficacy for general use. Consider the task of pose estimation for rigid
objects. Deep neural network based approaches have shown good performance when
trained on large, public datasets. However, adapting these networks for other
novel objects, or fine-tuning existing models for different environments,
requires significant time investment to generate newly labelled instances.
Towards this end, we propose ProgressLabeller as a method for more efficiently
generating large amounts of 6D pose training data from color images sequences
for custom scenes in a scalable manner. ProgressLabeller is intended to also
support transparent or translucent objects, for which the previous methods
based on depth dense reconstruction will fail. We demonstrate the
effectiveness of ProgressLabeller by rapidly create a dataset of over 1M
samples with which we fine-tune a state-of-the-art pose estimation network in
order to markedly improve the downstream robotic grasp success rates.
Progresslabeller is open-source at
https://github.com/huijieZH/ProgressLabeller

##  I Introduction

Visual perception tasks often require vast amounts of labelled data due to
their use of deep neural networks. Such deep neural networks have outperformed
traditional methods in object pose estimation [1, 2, 3] when trained on public
large-scale datasets [4, 5, 6]. However, considering the practice of deploying
such systems in real-world robotics applications, such as semantic grasping
and manipulation, current pose estimation systems can prove the difficulty of
adaptation to different objects and settings without retraining with a
customized large-scale dataset.

Figure 1: The ProgressLabeller offers an interactive GUI for aligning all
kinds of objects in the 3D scene to generate large-scale datasets with ground
truth pose labels. The left image shows the rough 6D pose estimates from one
state-of-the-art RGB-D deep models trained on public YCB dataset, and the
right images shows fine-tuned pose estimates from the same model after
retraining using generated data from ProgressLabeller. The pose estimates are
then used for robotic grasping experiments.

In particular, our need for training data is a result of object labels for
pose estimation being defined to specific 3D object models (both geometry
shape and texture). Learned models cannot be fine-tuned to transfer to similar
object instances without additional training data. Recent work has made
advances in category-level or unseen pose estimation [7, 8]. However, the
objects included only cover a small set and there is no evidence showing the
estimated pose is reliable enough for robotic manipulation. Further, the
estimation results of deep neural networks are often vulnerable to
environmental changes [9], including different lighting conditions, occlusions
and object’s special appearance like transparent or reflective surfaces.
Synthetic data generation with domain randomization and photo-realistic
rendering [10, 11] could improve generalizability, but it is still challenging
to simulate real-world lighting as well as the noise inherent in the sensor
modality. We show in the experiment that the network trained using real data
is still over-performing synthetic data.

To address the problem of adaptation for deep pose estimation systems and
their application to robotic manipulation, we propose ProgressLabeller as a
method and implementation for creating large customized datasets more
efficiently. Inspired by LabelFusion [12], ProgressLabeller collects training
data of objects in situ in a mixed-initiative manner, similar in spirit to
work by Gouravajhala et al. [13]. It takes visual streams of color images that
observe objects in a physical environment as input. Objects in this stream
only need to be labelled once by a human user through visual annotation.
ProgressLabeller builds on recent advances in Structure-from-Motion [14] and
visual SLAM [15] to produce both a 3D reconstruction and camera pose along the
trajectory of the collected visual stream, where the annotated object labels
can be propagated to all frames.

Compared to depth-based fusion methods, the color feature-based pipeline of
ProgressLabeller suffers less noisy or invalid readings than that from depth
sensing. Further, the use of color by ProgressLabeller allows it to include
objects that are transparent and reflective [16] into the pose estimation
process, as long as there exists textures from other objects or background.
From an interface perspective, our implementation of ProgressLabeller aims to
provide a more interactive design geared for users performing labeling tasks.
This interface design enables seamless labelled pose validation in different
views by checking and correcting the discrepancy between the masks of re-
projected object models and original RGB images (an example is shown in Figure
2). The intuition is that, if then the labelled pose is close to ground truth,
the re-projected object mask should align with the object’s true area in RGB
images from multiple views.

In this paper, we introduce ProgressLabeller as a semi-automatic approach to
object 6D pose labelling on RGB(D) image sequence/video frames that works for
transparent objects. Our aim is to release ProgressLabeller as an open-source
tool for more effective dataset generation for object and pose recognition by
robots. We evaluate the labelling accuracy of ProgressLabeller against
LabelFusion on data stream samples from 4 public datasets with respect to
segmentation mask and object pose accuracy. With the proposed system, we
created a dataset of YCB objects [17] (about 1.2M object instance labels)
within 2 days of data collection and labelling. The dataset presents more
challenges than the public YCB-Video dataset [4] with more occlusions and
better coverage of camera view directions, and collected using three different
RGB-D cameras to evaluate the generalization across sensors. We fine-tuned a
state-of-the-art RGB-D deep pose estimator [2] on our dataset and observed a
large improvement on pose estimation accuracy and robotic grasping success
rate, compared with the pretrained model on public dataset, as well as on the
same amount of data from image-synthesis or LabelFusion annotation.

##  II Related Work

With ProgressLabeller, a user can scalably label new datasets with camera
world pose, scene object poses and scene object segmentations. This process is
enabled by fusing streaming RGB (or RGB-D) inputs into a single scene-wide
representation, and then allowing a human user to input relevant 6-DoF
information via 3D modelling interfaces (such as those provided by Blender
[18]). This process demonstrates label stability even over long input video
streams, and due to its functionality with direct RGB inputs, can label even
difficult objects such as transparent cups. We discuss below methods related
to ProgressLabeller.

###  II-A Direct & Human-in-the-loop labelling

The creation of 2D segmentation data is analogous to the object detection,
keypoint detection, or semantic segmentation tasks (depending on desired
output labels). Tools such as LabelMe [19] required users to directly interact
with the underlying data to be labelled. This manual process was improved by
model-assisted approaches such as Deep Extreme Cut [20] which decreases the
amount of user effort necessary to label images. Shared autonomy and mixed-
initiative methods have also been used in this approach, in which the user
provides coarse pose or other estimations which are fine-tuned via a model-
informed approach [21].

###  II-B End-to-End Labellers

Previous tools have been created to enable this style of learning process.
LabelFusion [12] is perhaps the most commonly utilized example. LabelFusion
utilizes streaming RGB-D inputs to create a dense reconstruction of the scene,
which is then labelled semi-manually by aligning 3D object models to the 3D
reconstruction. While this approach is typically robust, it relies on RGB-D
input for reconstruction, and experiences difficulties under certain regimes.
In particular, transparent objects cause problems for commonly employed depth
sensor technologies, and long-running input streams typically result in large
amounts of ’drift’.

Some methods have been introduced to eliminate the need for CAD models in the
labelling process. Singh et al. [22] proposed a method which utilizes user
labelled keypoints and bounding boxes to generate pose and segmentation
labels. This frees the system from dependency on CAD models, but requires user
interaction directly with the images. SALT [23] proposed utilizing GrabCut to
generate 3D bounding boxes and image segmentation labels for relevant scenes.
This allows removing the dependency on object masks while also allows the
labelling of dynamic scenes such as human gait videos.

Other works sought to improve the labelling procedure itself. EasyLabel [24]
allows for semi-automatic labelling of scenes via sequentially added objects.
This process is scalable, and generates high quality labels. However, it
requires tight physical control over the scene to be labelled, which is not
always feasible to obtain. Objectron [25] utilized modern smartphone’s AR
capabilities combined with human-labelled 3D bounding boxes to scalably create
a large scale dataset. This method however is susceptible to label drift
during long-duration input videos. KeyPose [26] specifically sought to
generate labelled datasets for transparent objects. This method utilized
stereoscopic images taken from a robot armature in order to avoid the problems
of typical depth cameras have with transparent objects.

##  III ProgressLabeller

In ProgressLabeller, we provide an interactive GUI for users to label object
poses for a large amount of data in several steps. We incorporate camera pose
estimation systems that reconstructed a 3D scene for aligning objects with,
and calculate pose transforms that can be use to propagate the labelled object
pose to all image frames. We provide interface to switch views among every RGB
images to enable verification of labelled object pose by checking the
alignment of re-projected mask with RGB images in multiple views. In this
work, we assume the 3D mesh models of objects are provided and the objects are
static in the scene during data collection (in special cases like T-LESS
dataset [5], the objects are put onto a turntable and moving altogether while
the cameras are static, the algorithm can still work because no environment
background is captured by the cameras).

###  III-A Camera pose estimation

We incorporated ORB-SLAM3 [27], KinectFusion [28] and COLMAP [14, 29] to do
RGB, depth or RGB-D camera pose estimation as well as reconstruction. ORB-
SLAM3 is selected as the default method for its balanced speed and accuracy.
COLMAP was tested and proved to be more accurate but the time cost of
reconstruction over more than 1000 images is unaffordable.

###  III-B Object alignment by multi-view silhouette matching

Different from methods that align object 3D models with reconstructed point
clouds, our system created a multi-view graphical user interface that overlays
the object model’s projection onto the original RGB images, so that the object
pose errors could be easily detected from areas with misalignment of object
texture, silhouette and boundary, as shown in Figure 2. Compared to depth
reconstructed based methods, the pose label accuracy is improved based on
higher accuracy of RGB than depth sensing. Besides, the system can also be
used to label scenes with unreliable depth from transparent objects and
backgrounds (see Section 8).

Figure 2: Capture of ProgressLabeller user interface for multi-view re-
projection checking. The top-left view shows the aligned object models with
rendered texture at labelled poses. The bottom-left view and top-right view
show the silhouettes and boundaries respectively at the same camera view, and
the bottom-right view shows the boundaries from another view for validation.

###  III-C Semi-automatic labelling pipeline

We build ProgressLabeller as a plugin on Blender [18], which provides a good
multiple view interface of overlaid RGB image, 3D reconstructed scene and
objects for labelling and verifying poses. The overall procedure of labelling
object poses includes several steps within the Blender graphical interface:

  1. 1.

Import RGB(D) images and object 3D models into the 3D interactive workspace.
Set parameters including camera intrinsics, camera pose estimation parameters
and display settings, etc.

  2. 2.

Do camera pose estimation, then the estimated camera poses and a reconstructed
3D point cloud will be added to the workspace. Depth images are used to solve
the scale of reconstruction.

  3. 3.

(optional) When depth input is available, another point cloud fusing depth
input based on estimated camera poses can be generated to provide a denser
view of the entire scene and help find objects’ rough positions.

  4. 4.

(optional) Do plane alignment for a better viewpoint and ease of correcting
object pose

  5. 5.

Align the object so that its re-projection matches the ground truth area in
multiple views of RGB images

  6. 6.

Export labelled object poses and render segmentation masks, bounding box
labels, etc.

In step 2, when there is only RGB image available, the scene scale (3D point
cloud and camera’s trajectory position) is unknown, to solve this scale
problem, we take advantage of the known object sizes by dragging them to align
with the 3D reconstruction and verify that in multiple different views. In
step 3, we implemented a depth fusion module based on the estimated camera
poses from RGB reconstruction, based on the Signed Distance Functions as in
KinectFusion [28]. The fused point cloud could give a rough reference of
object locations. In step 4, Iterative Closest Point (ICP) is used for
aligning RGB feature or depth point cloud to X-Y plane in Blender. In step 5,
the object pose is labelled with visually ensured accuracy, which is the
essential design that enables labelling of objects with only RGB images based
on multi-view geometry. By re-projecting object’s 3D model and check whether
it aligns perfectly with the object’s true textures or silhouettes in the RGB
images from multiple views, users can fine-tune the object’s pose and verify
the error seamlessly until the object matches the images.

The entire labelling pipeline typically takes around 30 minutes for one data
stream about 5K images. Data import and export takes about 20-30% of time,
with a rendering speed around 4-20 images per second depending on number of
objects in the scene. The rendering could be done in parallel on a GPU if a
vast amount of data is required. ORB-SLAM3 camera pose estimation takes about
10-20% of time. Manual labelling and verification take the rest of time.

###  III-D Annotation accuracy estimation from simulation

We verify the accuracy of annotations throughout this multi-view silhouette
matching process by simulating an iterative object pose update process. In
each iteration, given a certain camera frame, we assume the object will be
translated in a plane parallel with its x-y plane, or rotated about z-axis
(for better control), towards a pose that maximizes the Intersection-over-
Union (IoU) between the rendered silhouette at current pose and ground truth.

####  III-D1 Problem Definition

Given set of $N$ images $I^{\\{i\\}}$ with their corresponding camera pose
$T^{\\{i\\}}$ in the world frame, $i\in\\{1,2,3,\ldots,N\\}$. Our goal is to
find ground truth object pose $T^{\text{obj}\\{j\\}}_{gt}$ in the world frame
for all the objects $j\in\\{1,2,3,\ldots,M\\}$ in the scene. We define the
projection operator as
$S^{\\{i,j\\}}=\text{Proj}(T^{\\{i\\}},T^{\text{obj}\\{j\\}})$, which render
object $j$ given its CAD model, camera pose $T^{\\{i\\}}$ and object pose
$T^{\text{obj}\\{j\\}}$ into an object texture/silouette $S^{\\{i,j\\}}$. Also
defined the IoU operator as
$\text{IoU}_{\text{obj}\\{j\\}}(I^{\\{i\\}},\text{Proj}(T^{\\{i\\}},T^{\text{obj}\\{j\\}}))$
to calculate the IoU for pixels in object $j$ between real image $I^{\\{i\\}}$
and synthetic texture/silouette $S^{\\{i,j\\}}$.

The multi-view texture/silhouette matching iterative update is proceeded with
a goal to maximize the IoU. Given the pose for object $j$ in the $k$th
iteration $T^{\text{obj}\\{j\\}}_{(k)}$, in ($k+1$)th iteration:

$T^{\text{obj}\\{j\\}}_{(k+1)}=\operatorname*{argmax}_{f[T^{\text{obj}\\{j\\}}_{(k)}]}\text{IoU}_{\text{obj}\\{j\\}}(I^{\\{i\\}},\text{Proj}(T^{\\{i\\}},f[T^{\text{obj}\\{j\\}}_{(k)}]))$
(1)

where $f[T^{\text{obj}\\{j\\}}_{(k)}]$ describes all possible translation
start from the $T^{\text{obj}\\{j\\}}_{(k)}$ that is within the plane $p$ or
the rotation along the axis $\omega$ as shown in Figure 3. So:

$f[T^{\text{obj}\\{j\\}}_{(k)}]=\exp^{\widehat{\xi}_{1}\theta_{1}}\exp^{\widehat{\xi}_{2}\theta_{2}}T^{\text{obj}\\{j\\}}_{(k)}$
(2)

where ${\xi}_{1}=\begin{bmatrix}-\omega\times v_{o}\\\ \omega\end{bmatrix}$,
${\xi}_{2}=\begin{bmatrix}v\\\ 0\end{bmatrix}$ are the twist coordinates for
twist $\widehat{\xi}_{1},\widehat{\xi}_{2}$.

Figure 3: Diagram for an object shown under a camera. $c$ denote the location
of camera and $c_{x},c_{y},c_{z}$ are its x, y, z axis. $p$ is a plane
parallel to camera plane and passing through object’s center $v_{o}$. $\omega$
the rotation direction parallel to $c_{x}$ and passing through $v_{o}$.
$\theta_{1}$ is the magnitude of rotation radius. $v$ is the translation
direction within the plane $p$ and $\theta_{2}$ is the translation magnitude.
On the right hand side is the projection image, the object in the transparent
color is the object with ground truth pose.

####  III-D2 Simulation results

We generate a CAD model sets with 44 different CAD models. For each run, we
generate $T^{\text{obj}}_{gt}$ with a random rotation matrix and location at
the origin. 40 cameras are created with their z axis pointing towards the
origin and a random location at a sphere around the object. The initial pose
$T^{\text{obj}}_{0}$ is generated by adding a random position noise from
Gaussian distribution with variance of 10cm to origin and with a random 3D
orientation. During each iteration, $v,\theta_{1},\theta_{2}$ in Equation 2
are discretized for simulation. The result shows that it takes around 10.36
iterations for the algorithm to converge within 1mm location error and dot
product larger than 0.99 between ground truth and converged rotation axes.

##  IV Experiments

Figure 4: From left to right, we show the 3D labeller view of LabelFusion,
ProgressLabeller and cropped RGB image with masks from LabelFusion, ground
truth and ProgressLabeller. The top and bottom row show a sample from HOPE and
YCB dataset respectively. We find even ground truth pose labels have easily
observable errors, shown as the spacing between black masks and object
boundaries.

In this section, we report results on several evaluation experiments on the
data generated using ProgressLabeller. The output label quality is firstly
evaluated on public datasets against ground truth. Then we introduce a large
scale dataset with object pose annotations, and report the evaluation results
of fine-tuning a state-of-the-art deep pose estimation on accuracy. The
improved accuracy is further evaluated in robotic grasping experiment.
Finally, we show two potential applications: labelling a complex scene with
transparent objects in front of reflective backgrounds, and training a neural
rendering model on a single object from collected data and do image synthesis.

Dataset |  Method |  Mask IoU $\uparrow$ |  Pos (cm) $\downarrow$ |  Rot (quat) $\downarrow$ |  ADD 1cm (AUC-0.1d) $\uparrow$ |  ADD-S (AUC-0.1d) $\uparrow$ |  Feature (Pixel) $\downarrow$
---|---|---|---|---|---|---|---
HOPE |  LabelFusion∗ |  0.6729 |  1.5825 |  0.2561 |  0.7095 |  0.7101 |  4.1248
|  ProgressLabeller |  0.8600 |  1.0948 |  0.0801 |  0.9566 |  0.9566 |  1.9479
|  Ground Truth |  - |  - |  - |  - |  - |  2.2752
YCB-Video |  LabelFusion |  0.9089 |  0.5165 |  0.0741 |  0.9977 |  0.9989 |  4.7365
|  ProgressLabeller |  0.8849 |  1.7537 |  0.0636 |  0.7642 |  0.7662 |  3.08
|  Ground Truth |  - |  - |  - |  - |  - |  3.3868
NOCS∗ |  LabelFusion |  0.7684 |  2.2825 |  2.9092 |  0.6436 |  0.6438 |  -
|  ProgressLabeller |  0.8785 |  1.4949 |  2.6153 |  0.8674 |  0.8679 |  -
T-LESS∗ |  LabelFusion* |  - |  - |  - |  - |  - |  -
|  ProgressLabeller |  0.8810 |  0.6779 |  0.0856 |  0.6626 |  0.6642 |  -
TABLE I: Results on feature matching distance between rendered object RGB
image and original image. ’Pos (cm)’, ’Rot (quat)’ refers to positional error
in centimeters and rotational error calculated as norm of difference in
quaternions as in [30]. LabelFusion cannot work on T-LESS data streams at all,
and it cannot reconstruct one of HOPE samples scene entirely, so we trim the
sequence to 1/4 length. Object models in T-LESS and NOCS datasets do not have
enough features to measure the feasure-based pixel distances.

###  IV-A Evaluation on label generation

We evaluate the label quality with respect to pose accuracy against ground
truth and time cost on ProgressLabeller versus LabelFusion [12]. Specifically,
we use both tools to label on 64 object instances from 8 sample sequences
among the 4 public pose estimation datasets including YCB-Video [4], T-LESS
[5], NOCS [7], and HOPE [31].

Evaluation metrics. We use mask Intersection-over-Union (IoU) between rendered
mask of object at labelled pose and at ground truth to evaluate the
segmentation, and select 4 distance metrics to compare a labelled pose with
ground truth pose, including 3D positional error $E_{p}$, 3D rotational error
$E_{r}$, average pairwise distance (ADD) $E_{ADD}$ [6] and average pairwise
distance in symmetric case (ADD-S) $E_{ADDS}$ [4], defined as follows:

$\displaystyle E_{p}=\left\lVert t-t^{*}\right\rVert,E_{r}=\min\\{\left\lVert
q-q^{*}\right\rVert,\left\lVert q+q^{*}\right\rVert\\}$ (3) $\displaystyle
E_{ADD}=\dfrac{1}{|\mathit{M}|}\sum_{x\in\mathit{M}}\left\lVert(Rx+t)-(R^{*}x+t^{*})\right\rVert$
(4) $\displaystyle
E_{ADDS}=\dfrac{1}{|\mathit{M}|}\sum_{x_{1}\in\mathit{M}}\min_{x_{2}\in\mathit{M}}\left\lVert(Rx_{1}+t)-(R^{*}x_{2}+t^{*})\right\rVert$
(5)

where $\mathit{M}=\\{x_{i}\in\mathbb{R}^{3}\\}$ is the object 3D point set,
and $R,q,t,R^{*},q^{*},t^{*}$ refers to the 3D rotation matrix, rotation
represented in quaternion, 3D position vector of labelled pose and ground
truth respectively. Specifically for ADD and ADD-S, we report the value of
area under the accuracy-threshold curve obtained by varying the distance
threshold from 0 to 0.1 diameter of objects (AUC-0.1d) following metrics among
pose estimation papers.

Besides, from the observation that even ground truth labels are still not
perfectly correct (re-projected object masks align with true area in RGB
images from multi-view), we propose an RGB feature based evaluation on object
pose without ground truth. Specifically, we render the object at labelled pose
to get an image with only an object in front of black background, and extract
SIFT features from both the rendered image and original RGB image, then
calculate the average of feature matching distance in pixels. In practice, we
assume the labelled pose is close to ground truth, and remove incorrect
matching with a distance threshold of 10 pixels. Figure 4 shows qualitative
examples of labelled poses, where we observe more accurate re-projection
results from ProgressLabeller than LabelFusion output as well as provided
ground truth on masked RGB images. We believe the main error cause in
LabelFusion is the accumulated depth sensing error during reconstruction.
Another drawback of LabelFusion UI design is that, there is no measurement or
display of object poses in the 3D scene, so the users need to switch back-and-
forth between the 3D scene and the final generated images that has image masks
rendered at labelled poses. Also its 3D model as well as rendering result
doesn’t display textures, so it’s hard to deal with symmetric-shape objects in
HOPE and NOCS.

The quantitative results are shown in Table I. From the comparison, we see
ProgressLabeller is more accurate and robust in most streams. LabelFusion has
higher accuracy on YCB-Video dataset, which has a large feature-based pixel
distance. For example, we found the pose annotation of object
006_mustard_bottle was flipped 180∘ in one scene. Besides, the two approaches
took similar annotation time of 10-30 minutes per scene.

###  IV-B Multi-camera dataset creation

From analysis of absolute accuracy result above and to evaluate robustness on
existing deep pose estimation networks, we aim to create a more accurate,
multi-camera, cluttered dataset on YCB objects [17]. We mounted 3 RGB-D
cameras with different sensing technologies, ASUS Xtion Pro Primesense Carmine
1.09 (structured light), Intel RealSense D435i (stereo) and RealSense L515
(LiDAR), on a Fetch robot and collected data streams at $\sim$15Hz when Fetch
is slowly driving around the tabletop object scene with a constant speed. Each
scene contains almost a bit more than full round, with 1K$\sim$3K paired RGB-D
images from each camera. In this way, the collected data is free of motion
blur and covers full round view of objects and occlusions. The dataset
includes 16 scenes with 10 objects placed in both isolation and dense clutter,
and another 4 scenes each with a single object for unit test. Figure 5 gives
an example. It took 2 days collected and labelled the dataset, with about 120K
images and 1.2M labelled object instances.

Figure 5: The top and bottom row shows RGB images in training and testing set respectively. From left to right, the images are taken from Primesense, RealSense D435i and L515. Train set/Test set | Primesense | RealSense L515 | RealSense D435i
---|---|---|---
Metric: AUC (0-10cm) | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S
Pretrained on Asus Xtion Pro Live | 21.63 | 41.16 | 47.43 | 72.90 | 44.67 | 68.00
Fine-tuned on Primesense | 60.02 | 79.21 | 55.32 | 77.85 | 49.75 | 73.35
Fine-tuned on RealSense L515 | 33.45 | 58.16 | 68.84 | 83.69 | 57.67 | 76.26
Fine-tuned on RealSense D435i | 26.39 | 50.38 | 63.82 | 82.52 | 64.64 | 82.36
TABLE II: The pose estimation accuracy of FFB6D cross-validated on datasets
collected using three cameras.

###  IV-C 6D object pose estimation fine-tuning

Using the collected dataset, we fine-tuned one of the state-of-the-art deep
pose estimation neural networks taking RGB-Depth images as input, FFB6D [2]
over its pretrained model on YCB-Video dataset. In particular, we created 3
training sets by downsampling to 1/10 the training set collected by 3 cameras
respectively and got 3 fine-tuned models by training on a RTX 3080 with batch
size 6 for 20 epochs and default settings in other parameters. Then, we cross-
validated the 3 fine-tuned models, along with pretrained model, on 3 test sets
taken from 3 cameras.

####  IV-C1 Evaluation on pose accuracy across different cameras

We report the AUC for ADD and ADD-S metrics between 0 and 10cm, to match the
original result in [2]. The result is shown in Table II, where we find the
fine-tuned dataset recognizes the sensor modality and noise difference between
3 cameras as the test set ADD and ADD-S are mostly the highest on the same
training set.

Figure 6: A synthesized RGB image using Blenderproc is shown on the left. The
original image is shown on the right. The YCB objects are rendered and
overlaid to original image according to labelled poses. Depth synthesized
images are generated in the same way.

####  IV-C2 Evaluation against LabelFusion and synthetic data on pose accuracy
and robotic grasping

We used 3 datasets to train FFB6D with the same setting as above. The first is
the collected Primesense subset labelled by ProgressLabeller, the second is
the same data labelled by LabelFusion, and the third is synthesized images by
rendering YCB objects over both raw RGB and depth images according to the
labelled poses using Blenderproc [11], which mostly maintain the same object
pose distribution and sensor noise as real data, as shown in Figure 6.

Network model | Pretrained | Blenderproc | LabelFusion | ProgressLabeller
---|---|---|---|---
Object name | Grasp tolerance (cm) | ADD | Grasp | ADD | Grasp | ADD | Grasp | ADD | Grasp
002_master_chef_can | 0.36 | 25.16 | 0/5 | 48.67 | 3/5 | 44.78 | 4/5 | 55.42 | 5/5
003_cracker_box | 3.90 | 19.57 | 1/5 | 50.76 | 1/5 | 60.8 | 4/5 | 71.01 | 5/5
005_tomato_soup_can | 3.65 | 27.74 | 1/5 | 51.12 | 2/5 | 65.48 | 4/5 | 66.79 | 5/5
006_mustard_bottle | 6.33 | 31.02 | 1/5 | 58.87 | 5/5 | 70.77 | 5/5 | 73.47 | 5/5
| 2.50 | 31.02 | 0/5 | 58.87 | 5/5 | 70.77 | 4/5 | 73.47 | 5/5
007_tuna_fish_can | 7.09 | 15.14 | 2/5 | 26.41 | 3/5 | 30.02 | 5/5 | 39.87 | 5/5
| 2.04 | 15.14 | 2/5 | 26.41 | 4/5 | 30.02 | 4/5 | 39.87 | 5/5
009_gelatin_box | 7.60 | 13.25 | 2/5 | 32.79 | 4/5 | 51.45 | 5/5 | 55.24 | 5/5
| 3.14 | 13.25 | 2/5 | 32.79 | 3/5 | 51.45 | 5/5 | 55.24 | 5/5
| 1.62 | 13.25 | 2/5 | 32.79 | 4/5 | 51.45 | 5/5 | 55.24 | 4/5
010_potted_meat_can | 4.75 | 19.68 | 5/5 | 39.56 | 3/5 | 47.48 | 4/5 | 50.64 | 5/5
| 2.08 | 19.68 | 1/5 | 39.56 | 4/5 | 47.48 | 4/5 | 50.64 | 4/5
| 0.78 | 19.68 | 2/5 | 39.56 | 0/5 | 47.48 | 5/5 | 50.64 | 5/5
025_mug | 1.24 | 19.10 | 1/5 | 60.70 | 3/5 | 49.31 | 1/5 | 66.53 | 4/5
040_large_marker | 8.67 | 20.10 | 0/5 | 44.75 | 2/5 | 55.89 | 3/5 | 53.73 | 3/5
overall | - | - | 22/75 | - | 46/75 | - | 62/75 | - | 70/75
TABLE III: Grasp record comparison on part of YCB objects using grasp poses
generated based on pose estimates from pretrained and fine-tuned FFB6D on the
collected dataset, with Blenderproc data synthesis, LabelFusion annotations
and ProgressLabeller annotations.

We tested pose estimation accuracy from models trained on the above 3
datasets, with respect to ADD-AUC and pose-based grasping success rate. For
pose-based grasping, we manually defined grasp poses along the symmetrical
axes on the object 3D models, and grouped them by the grasping tolerance,
defined by the distance between two gripper fingers (10.39 cm for Fetch) minus
the object’s diameter along the grasping direction. Obviously, a smaller
grasping tolerance requires more accurate pose estimates for a successful
grasp. For example, 002_master_chef_can is challenging to grasp as its
tolerance is only 0.36 cm. We repeated grasping on every tolerance for 5 times
based on the pose estimates, and the success/failure statistics is shown in
Table III. Overall, the fine-tuning over ProgressLabeller data improves the
grasping success rate the most. Among experiments, we found in some cases the
grasp is still successful given large rotational error, where the object was
aligned to another pose during grasping, such as 002_master_chef_can and
006_mustard_bottle when grasping from the side. Also, sometimes the objects
were grasped along another direction, such as 009_gelatin_box and
010_potted_meat_can, in this case, their actual grasp tolerance might change.
We expect evaluation on object placement accuracy to reveal these errors.
Otherwise, the grasping test results generally matches the pose accuracy
presented in ADD. Figure 7 shows an example of grasping experiment.

Figure 7: The Fetch robot is grasping objects on tabletop based on estimated
poses. The pose estimates are shown as projected point clouds and coordinate
frames in top-right smaller images. The higher and lower frames correspond to
pre-grasp and grasp poses.

###  IV-D Other applications

Figure 8: Comparison of best (left) and worst (right) validation pose
renderings from the NSVF model. The smaller images with black background on
top corners are the ground truth masks, and those on bottom corners are
rendered output from NSVF, and the middle larger images are the renderings
imposed onto the original captured image in greyscale. The best image (left)
had an average pixelwise L2 error of $0.0067$, and the worst (right) had an
average pixelwise L2 error of $0.1867$.

####  IV-D1 Transparent Dataset Labelling

We collected RGB-D videos with transparent cups and labelled their poses using
ProgressLabeller. Figure 9 shows in detail that even there is no 3D points
around the transparent area when we fused raw depth according to the estimated
camera poses, the tool enables accurate pose labelling by matching the
object’s mask with RGB images. We believe a large-scale 3D dataset of
transparent objects can be efficiently created using ProgressLabeller.

Figure 9: The labelling on transparent champagne cups. When there is no
reliable depth (top-left), ProgressLabeller enables checking the pose validity
by re-projection object models (bottom-left) to RGB images from different
views (top-right and bottom-right).

####  IV-D2 Training neural radiance fields on objects

As a brief additional experiment, we provide prima facie data to show the
effectiveness of our labeller in generating datasets for training a Neural
Sparse Voxel Fields (NSVF) model [32]. This model permits rendering objects at
novel poses given multiple input views with corresponding camera poses. We use
ProgressLabeller to label a scene containing a 003_cracker_box, with 3509
images in total. From this dataset, we uniformly random sample 100 images with
object poses for train, test and validation set respectively. We remove the
background from the training images using the segmentation from labeller, and
an NSVF model was trained to 75k total iterations on a dual GPU machine
consisting of an Nvidia RTX 3060 TI and RTX 3070. After training, renderings
were made at each camera pose given in the train, test, and validation sets.
These renderings were evaluated via a pixel averaged L2 error against the
ground truth segmented images produced by ProgressLabeller where each pixel
channel was normalized to the range $[0,1]$. Those results presented in Table
IV. Visual comparisons between the best and worst validation set renderings
along with their associated errors are presented in Figure 8.

Dataset | Train | Validation | Test
---|---|---|---
Per-pixel L2 error | 0.0108 | 0.0460 | 0.0409
TABLE IV: Average pixelwise errors for the train, validation, and test sets
from the NSVF model trained on ProgressLabeller outputs.

##  V Conclusion

In this work, we have presented the design and application ProgressLabeller
for object pose annotation. Through comparison to LabelFusion, we show that
its higher accuracy with similar labelling time, efficiency of creating a
large-scale customized dataset, and potential in fine-tuning pose estimation
deep neural networks for robotic grasping. In future, we can improve
ProgressLabeller by adding online object model generation and support for
dynamic scenes.

Acknowledgement. We thank greatly the support from Dr. Peter Gaskell and Liz
Olson at the University of Michigan, who provided devices for dataset
collection and feedback for user guidance improvement.

## References

  * [1] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic, “Cosypose: Consistent multi-view multi-object 6d pose estimation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 574–591.
  * [2] Y. He, H. Huang, H. Fan, Q. Chen, and J. Sun, “Ffb6d: A full flow bidirectional fusion network for 6d pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3003–3013.
  * [3] T. Hodaň, M. Sundermeyer, B. Drost, Y. Labbé, E. Brachmann, F. Michel, C. Rother, and J. Matas, “Bop challenge 2020 on 6d object localization,” in _European Conference on Computer Vision_. Springer, 2020, pp. 577–594.
  * [4] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” _Robotics Science and Systems (RSS)_ , 2018.
  * [5] T. Hodan, P. Haluza, Š. Obdržálek, J. Matas, M. Lourakis, and X. Zabulis, “T-less: An rgb-d dataset for 6d pose estimation of texture-less objects,” in _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2017, pp. 880–888.
  * [6] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,” in _Asian conference on computer vision_. Springer, 2012, pp. 548–562.
  * [7] X. Li, H. Wang, L. Yi, L. J. Guibas, A. L. Abbott, and S. Song, “Category-level articulated object pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3706–3715.
  * [8] K. Park, A. Mousavian, Y. Xiang, and D. Fox, “Latentfusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 710–10 719.
  * [9] X. Chen, R. Chen, Z. Sui, Z. Ye, Y. Liu, R. I. Bahar, and O. C. Jenkins, “Grip: Generative robust inference and perception for semantic robot manipulation in adversarial environments,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 3988–3995.
  * [10] T. To, J. Tremblay, D. McKay, Y. Yamaguchi, K. Leung, A. Balanon, J. Cheng, W. Hodge, and S. Birchfield, “NDDS: NVIDIA deep learning dataset synthesizer,” 2018, https://github.com/NVIDIA/Dataset_Synthesizer.
  * [11] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam, “Blenderproc,” _arXiv preprint arXiv:1911.01911_ , 2019.
  * [12] P. Marion, P. R. Florence, L. Manuelli, and R. Tedrake, “Label fusion: A pipeline for generating ground truth labels for real rgbd data of cluttered scenes,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 3235–3242.
  * [13] S. R. Gouravajhala, J. Yim, K. Desingh, Y. Huang, O. C. Jenkins, and W. S. Lasecki, “Eureca: Enhanced understanding of real environments via crowd assistance,” in _Sixth AAAI conference on human computation and crowdsourcing_ , 2018.
  * [14] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4104–4113.
  * [15] R. Mur-Artal and J. D. Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” _IEEE transactions on robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [16] X. Liu, S. Iwase, and K. M. Kitani, “Stereobj-1m: Large-scale stereo image dataset for 6d object pose estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 870–10 879.
  * [17] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” in _2015 international conference on advanced robotics (ICAR)_. IEEE, 2015, pp. 510–517.
  * [18] B. O. Community, _Blender - a 3D modelling and rendering package_ , Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. [Online]. Available: http://www.blender.org
  * [19] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman, “Labelme: a database and web-based tool for image annotation,” _International journal of computer vision_ , vol. 77, no. 1-3, pp. 157–173, 2008.
  * [20] K.-K. Maninis, S. Caelles, J. Pont-Tuset, and L. Van Gool, “Deep extreme cut: From extreme points to object segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 616–625.
  * [21] Z. Ye, J. Y. Song, Z. Sui, S. Hart, J. Vilchis, W. S. Lasecki, and O. C. Jenkins, “Human-in-the-loop pose estimation via shared autonomy,” in _26th International Conference on Intelligent User Interfaces_ , 2021, pp. 387–391.
  * [22] R. P. Singh, M. Benallegue, Y. Yoshiyasu, and F. Kanehiro, “Rapid pose label generation through sparse representation of unknown objects,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 10 287–10 293.
  * [23] D. Stumpf, S. Krauß, G. Reis, O. Wasenmüller, and D. Stricker, “Salt: A semi-automatic labeling tool for rgb-d video sequences,” _arXiv preprint arXiv:2102.10820_ , 2021.
  * [24] M. Suchi, T. Patten, D. Fischinger, and M. Vincze, “Easylabel: a semi-automatic pixel-wise object annotation tool for creating robotic rgb-d datasets,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6678–6684.
  * [25] A. Ahmadyan, L. Zhang, A. Ablavatski, J. Wei, and M. Grundmann, “Objectron: A large scale dataset of object-centric videos in the wild with pose annotations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 7822–7831.
  * [26] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige, “Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 602–11 610.
  * [27] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” _IEEE Transactions on Robotics_ , vol. 37, no. 6, pp. 1874–1890, 2021.
  * [28] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: Real-time dense surface mapping and tracking,” in _2011 10th IEEE international symposium on mixed and augmented reality_. IEEE, 2011, pp. 127–136.
  * [29] J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys, “Pixelwise view selection for unstructured multi-view stereo,” in _European Conference on Computer Vision_. Springer, 2016, pp. 501–518.
  * [30] D. Q. Huynh, “Metrics for 3d rotations: Comparison and analysis,” _Journal of Mathematical Imaging and Vision_ , vol. 35, no. 2, pp. 155–164, 2009.
  * [31] “Household objects for pose estimation (hope),” https://github.com/swtyree/hope-dataset, 2021.
  * [32] L. Liu, J. Gu, K. Z. Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel fields,” _Neural Information Processing Systems (NeurIPS)_ , 2020.