# AI-Based Multi-Object Relative State Estimation with
Self-Calibration Capabilities

Thomas Jantos1, Christian Brommer1, Eren Allak1, Stephan Weiss1 and Jan
Steinbrener1 1The authors are with the Control of Networked Systems Group,
University of Klagenfurt, 9020 Klagenfurt am Wörthersee, Austria
<EMAIL_ADDRESS>work was supported by the Federal Ministry for
Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK)
under the grant agreement 881082 (MUKISANO).Pre-print version, accepted
Jan/2023, DOI follows ASAP ©IEEE.

###### Abstract

The capability to extract task specific, semantic information from raw sensory
data is a crucial requirement for many applications of mobile robotics.
Autonomous inspection of critical infrastructure with Unmanned Aerial Vehicles
(UAVs), for example, requires precise navigation relative to the structure
that is to be inspected. Recently, Artificial Intelligence (AI)-based methods
have been shown to excel at extracting semantic information such as 6 degree-
of-freedom (6-DoF) poses of objects from images.

In this paper, we propose a method combining a state-of-the-art AI-based pose
estimator for objects in camera images with data from an inertial measurement
unit (IMU) for 6-DoF multi-object relative state estimation of a mobile robot.
The AI-based pose estimator detects multiple objects of interest in camera
images along with their relative poses. These measurements are fused with IMU
data in a state-of-the-art sensor fusion framework. We illustrate the
feasibility of our proposed method with real world experiments for different
trajectories and number of arbitrarily placed objects. We show that the
results can be reliably reproduced due to the self-calibrating capabilities of
our approach.

##  I INTRODUCTION

Mobile robots, such as unmanned aerial vehicles (UAVs), rely on the
information of their on-board sensors to autonomously navigate the world.
Semantic information, i.e. the higher-level meaning of sensor data, can
improve a robot’s ability to navigate in its surroundings and allows for more
complicated tasks [1]. In semantic navigation, the robot moves depending on
context or task, in many cases with respect to objects of interest in the
scene. Such tasks include infrastructure inspection [2] or object tracking
[3]. While the goal for the latter is to keep the moving object in the field
of view of the camera, infrastructure inspection requires accurate positioning
of the robot with respect to a typically static object of interest. Semantic
information extracted from the robot’s sensor data, namely the detection of
the object of interest and its pose relative to the robot are important
elements to achieving this task. For example, monitoring power pole insulators
for possible damages requires a UAV to fly around the desired insulator and
take high resolution images from specific positions to allow for detection of
damage or changes over time.

Figure 1: Visualization of the coordinate frames in this work. We estimate the
state of a fixed rigid body consisting of IMU $I$ and camera $C$ relative to
up to $N$ different objects $O_{k}$ with respect to a fixed but arbitrary
navigation world $W$. In addition to the core states (red), we also estimate
the calibration between IMU and camera (blue). We also estimate the pose of
the object frames with respect to the world (blue). Our pose sensor consists
of AI-based 6-DoF relative pose measurements between camera and objects
(green).

Current autonomous mission execution is typically based on global navigation
satellite system (GNSS) for localization of the UAV. GNSS always provides a
global position and not a relative position with respect to an object of
interest. Moreover, the accuracy is too low for precise, centimeter-range
navigation and GNSS is prone to signal loss in proximity to large structures.
In this case often other sensor modalities are considered, e.g. visual-
inertial odometry (VIO) [4]. With VIO, a local pose can be estimated by
combining the movement of geometrical features (edges, corners) in monocular
camera images with data from an inertial measurement unit (IMU). Classical,
feature extraction based algorithms are not well suited for semantic
navigation as they rely on raw features that do not provide information about
any objects in the scene and they struggle with fast or slow motion [5].

Recent advancements in artificial intelligence (AI) led to a breakthrough in
the extraction of semantic information from raw sensor measurements like path
detection using camera images [6], object recognition with laser scanners [7],
semantic segmentation for scene understanding [8], and recently, 6 degree-of-
freedom (6-DoF) pose estimation of objects for robotic grasping [9].
Furthermore, the availability of AI capable edge computing devices enables the
usage of such methods on mobile robots.

In this paper, we investigate the suitability of AI-based pose estimation for
full 6-DoF, object relative state estimation for mobile robotics. We consider
a minimal sensor configuration consisting of a single monocular camera and an
IMU in line with size, weight and computational power constraints of mobile
robotic platforms such as UAVs. We utilize an AI-based pose estimator to
detect, classify and estimate the 6-DoF poses of objects of interest contained
in each camera image, and then fuse the information with IMU measurements in a
state-of-the-art sensor fusion framework to infer the 6-DoF object-relative
pose of the robot. A schematic overview of our approach is presented in Fig.
1. Our main contributions can be summarized as follows:

  * •

Extracting semantic information from images with AI and fusing this relative
pose information with IMU data for accurate, 6-DoF, object relative state
estimation.

  * •

Formulating a filter-based method to estimate the state of the mobile robot
and the pose of multiple, different objects based on 6-DoF relative pose
measurements.

  * •

Providing a self-calibrating formulation of the filter that does not require
any assumptions about the global or relative positions of the different
objects in a scene.

  * •

Validating the proposed approach with several real world experiments using
objects of a popular 6-DoF object pose challenge data set to show that our
method works for different trajectories and different number of objects with
reproducible performance.

The remainder of the paper is organized as follows. In Section II, we
summarize the related work. In Section III, we present how we integrate object
relative pose measurements into a state estimation framework. In Section IV,
the experiments and the corresponding results are discussed. Finally, the
paper is concluded in Section V.

##  II RELATED WORK

For state estimation in mobile robotics, typically IMU data and one or more
pose sensors such as GNSS are fused together. GNSS provides global position
information but not 3D orientation information. In the absence of GNSS
signals, VIO, the combination of a monocular camera and IMU data, can estimate
the pose of a robot by triangulating the position of the camera given
geometrical features from an image and estimating the remaining scale factor
with inertial data [4, 10]. Fusing multiple sensors yields a more robust and
reliable estimate of the robot’s state. There exist mainly two different
approaches for sensor fusion: filter-based, recursive and optimization-based
methods. The latter can yield more accurate state estimates but is
computationally more demanding due to optimizing across several sensor
measurements [11]. In comparison, filtering-based methods, such as Extended
Kalman Filters (EKF) [12, 13], are computationally more efficient and thus are
well-suited for mobile robotics.

GNSS and classical VIO do not provide object relative pose measurements and
thus are not suitable for object relative state estimation. However, image-
based 6-DoF relative object pose estimation methods can be utilized as a pose
sensor in state estimation frameworks. There exist classical approaches and AI
methods based on deep learning. Classical approaches are either template-
based, where the object pose is determined by finding a matching template for
the current image [14], or feature-based, where keypoints are extracted from
the image and then matched to the 3D object model [15]. On the other hand,
deep learning-based approaches are mostly end-to-end learned methods, where
the 6-DoF pose is directly estimated from the input image using convolutional
neural networks (CNNs). Deep learning-based methods can be divided even
further depending on the amount of additional information used. [9, 16] take a
single RGB image as input to their network and employ symmetry aware losses
during training that make use of 3D object model information. 3D object models
can also be provided as an additional input to the network [17], utilized for
refining an initial pose estimate [18, 19, 20] or for matching keypoints,
which were regressed by the network [21]. Other forms of additional
information consist of taking multiple images [22, 23] or depth maps [24, 25].
Recently, we have proposed PoET [26] a 6-DoF multi-object pose estimation
framework that achieves state-of-the-art results on benchmark datasets and
only takes a single RGB image as input and does not require any additional
information during training or inference.

An alternative to object relative state estimation is simultaneous
localization and mapping (SLAM) on an object level. [27] uses depth images to
extract 6-Dof object pose information. Similar to us, [28] use an AI-based
pose estimator to predict 6-DoF relative object poses from images. Both
approaches fuse the 6-DoF relative pose information from multiple view points
together and combine it with graph optimization to estimate the pose of the
camera and objects with respect to a map. However, they do not use any other
sensors, such as IMU, in their approach. [29] combines IMU measurements,
geometric features from images, and 6-DoF object poses in a SLAM approach.
Graph optimization still needs to be performed for a graph containing all
object poses. In general, the requirement for a map and optimization in SLAM
results in a higher computational load for the mobile robot. Our approach
still allows for object relative state estimation without this requirement.

Object relative state estimation for mobile robotics has been shown in [30],
where a UAV localizes itself with respect to cylinder shaped infrastructure by
extracting geometrical features from images and assuming a known radius.
Similarly, [31] used a color-based ellipse-detection algorithm to first detect
the object of interest in the image and then used the knowledge about object
size, visual appearance and camera parameters to calculate the relative pose
to the object. Meanwhile, [32] investigated different classical 6-DoF object
pose estimation approaches for object-relative state estimation. They also
investigated the use of machine learning to detect the presence of objects by
training simple classifiers on classical features extracted from images.

In contrast to that, we propose here a fully AI-based method to extract
semantic information from camera images. We do not need to define a geometric
model, keypoints or templates to map object appearances in images to relative
6-DoF poses. Moreover, AI-based models are not limited to specific geometric
object shapes and remove the need for handcrafted features. In our previous
work [26], we have introduced PoET for 6-DoF pose estimation of objects in RGB
images using state-of-the-art AI methods. We mainly focused on the definition,
the training, a thorough ablation study and comparison to other deep learning-
based methods on benchmark datasets for 6-DoF multi-object pose estimation. In
this work, we present a detailed investigation of the suitability of our AI-
based object pose estimator as pose sensor for 6-DoF object-relative state
estimation of a mobile robot using a state-of-the-art sensor fusion framework
with multiple real world experiments.

##  III METHOD

In this section, we present the design of our approach. First, we explain the
notation used for the measurement equations and transformations of coordinate
frames. Second, we reason about the choice of frameworks for 6-DoF pose
estimation and state estimation. Finally, we describe how the estimated 6-DoF
of several known objects can be combined to estimate the 6-DoF pose of the
robot. This includes a detailed description of how our choice of sensor fusion
algorithm is extended to include 6-DoF pose measurements of each individual
object.

###  III-A Notation

Throughout this paper we use the following notation: given three coordinate
frames $A$, $B$ and $C$, the transformation ${}_{{\scriptscriptstyle
A}}{\mathbf{T}}_{\scriptscriptstyle BC}$ defines frame $C$ with respect to
frame $B$ expressed in frame $A$. If the left subscript $A$ is omitted, the
transformation is defined in frame $B$. Furthermore, the transformation
${}_{{\scriptscriptstyle A}}{\mathbf{T}}_{\scriptscriptstyle AB}$ can be split
up into two parts namely ${}_{{\scriptscriptstyle
A}}{\mathbf{p}}_{\scriptscriptstyle AB}$ and ${\mathbf{R}}_{\scriptscriptstyle
AB}$, which describe the translation and rotation respectively. Alternatively,
the rotation can also be expressed by a quaternion
${\mathbf{q}}_{\scriptscriptstyle AB}$. Each quaternion $\mathbf{q}$ can be
represented by
$\mathbf{q}=[\mathbf{q_{v}}~{}q_{w}]^{T}=[q_{x}~{}q_{y}~{}q_{z}~{}q_{w}]^{T}$.
The quaternion multiplication is represented by $\otimes$. $\mathbf{I}_{3}$
and $\mathbf{0}_{3}$ refer to the identity and the null matrix in
$\mathbb{R}^{3x3}$, respectively. $[\omega]_{\times}$ is the skew-symmetric
operator as defined in [33].

###  III-B Pose and State Estimation Frameworks

Mobile robots, in particular UAVs, are subject to payload constraints, which
impose not only limitations on the size and amount of sensors a robot can
carry, but also on the computational power available for data processing.
Hence, the necessity arises for efficient and computationally light
algorithms. Therefore, we chose our object pose estimation framework PoET [26]
as a 6-DoF pose sensor as it only uses RGB images and does not rely on any
depth information and 3D object models, removing the need for additional
hardware components and reducing the computational load by not having to
process 3D models. In a first step, PoET detects all objects it was trained to
detect in an image and also predicts their classes. Afterwards, the predicted
bounding boxes and multi-scale feature maps are fed to a transformer
architecture to predict the relative, up-to scale 6-DoF pose between the
camera and each object. The predicted rotation and translation are unique for
non-symmetric objects. For objects with one or more symmetry axes, the
rotation or translation for some object poses becomes ambiguous with more than
one possible solution. The obvious negative effects of this ambiguity on the
pose estimation of the robot can be minimized by considering multiple objects
in heterogeneous configuration and fusing individual measurements in a proper
sensor fusion framework. This mimics an inspection workflow where typically
several distinct parts of interest of the structure to be inspected are
visible at the same time.

For the sensor fusion framework, we use MaRS [12] for multi-sensor fusion and
state estimation due to being lightweight and computationally efficient as it
was developed specifically with mobile robotics in mind. MaRS was designed for
modularity and separates the propagation of the core state variables based on
inertial data from the state updates based on the measurements of the
individual sensors. It also uses abstract sensor classes that are type
agnostic. This allows for straightforward integration of new sensor modules.
For our method, we define a multi-pose sensor, where a single measurement
consists of a single RGB image. From each image, we then extract the 6-DoF
relative poses of all detected objects with PoET and use them for the EKF
update step as described below.

###  III-C EKF State and Update

As reference frame for the mobile robotic system, we chose the frame of its
IMU ($I$). Thus, our goal is to estimate the pose of the IMU ($I$) with
respect to the world ($W$) by measuring the 6-DoF relative poses between the
camera ($C$) and a set of objects ($O_{k})$. The different coordinate frames
are visualized in Fig. 1. As mentioned earlier, the relative poses of the
objects with respect to the camera are extracted from the image by our AI-
based pose estimator dubbed PoET. PoET will only consider objects that it was
trained for. Given a single RGB image, a 6-DoF pose
${\mathbf{T}}_{\scriptscriptstyle CO_{k}}$ is predicted for each detected
object of interest and the assignment of the predicted pose to an object is
based on the predicted class. For details about the architectural choices in
PoET, we refer the reader to [26]. Depending on the total number of objects
$N$ in a scene, the full state vector $\mathbf{X}$ is then defined as:

$\displaystyle\mathbf{X}=[$ $\displaystyle{\mathbf{p}}_{\scriptscriptstyle
WI}^{T},{\mathbf{v}}_{\scriptscriptstyle
WI}^{T},{\mathbf{q}}_{\scriptscriptstyle
WI}^{T},{\mathbf{b}}_{\scriptscriptstyle\omega}^{T},{\mathbf{b}}_{\scriptscriptstyle
a}^{T},$ (1) $\displaystyle{\mathbf{p}}_{\scriptscriptstyle
IC}^{T},{\mathbf{q}}_{\scriptscriptstyle
IC}^{T},{\mathbf{p}}_{\scriptscriptstyle
O_{0}W}^{T},{\mathbf{q}}_{\scriptscriptstyle
O_{0}W}^{T},\dots,{\mathbf{p}}_{\scriptscriptstyle
O_{N}W}^{T},{\mathbf{q}}_{\scriptscriptstyle O_{N}W}^{T}]^{T}$

The core states necessary for state propagation are the position
${\mathbf{p}}_{\scriptscriptstyle WI}$ of the IMU, its velocity
${\mathbf{v}}_{\scriptscriptstyle WI}$ and its orientation
${\mathbf{q}}_{\scriptscriptstyle WI}$ as well as the gyroscopic bias
${\mathbf{b}}_{\scriptscriptstyle\omega}$ and the accelerometer bias
${\mathbf{b}}_{\scriptscriptstyle a}$. The pose and velocity dynamics are
given as [4]

$\displaystyle{\dot{\mathbf{p}}}_{\scriptscriptstyle WI}$
$\displaystyle={\mathbf{v}}_{\scriptscriptstyle WI}$ (2)
$\displaystyle{\dot{\mathbf{v}}}_{\scriptscriptstyle WI}$
$\displaystyle={\mathbf{R}}_{\scriptscriptstyle
WI}~{}({\mathbf{a}}_{\scriptscriptstyle m}-{\mathbf{b}}_{\scriptscriptstyle
a}-{\mathbf{n}}_{\scriptscriptstyle a})-\mathbf{g}$ (3)
$\displaystyle{\dot{\mathbf{q}}}_{\scriptscriptstyle WI}$
$\displaystyle=\frac{1}{2}\Omega({\mathbf{\omega}}_{\scriptscriptstyle
b}-{\mathbf{b}}_{\scriptscriptstyle\omega}-{\mathbf{n}}_{\scriptscriptstyle\omega})~{}{\mathbf{q}}_{\scriptscriptstyle
WI}$ (4)

where ${\mathbf{a}}_{\scriptscriptstyle m}$ is the measured acceleration in
the IMU frame, ${\mathbf{n}}_{\scriptscriptstyle a}$ is the accelerometer
noise parameter, $\mathbf{g}$ is the gravity vector in $W$,
${\mathbf{\omega}}_{\scriptscriptstyle b}$ is the measured angular velocity in
the IMU frame, ${\mathbf{n}}_{\scriptscriptstyle\omega}$ is the gyroscopic
noise parameter, and $\Omega(\omega)$ is the quaternion multiplication matrix
of $\omega$. The IMU biases are modeled as random walks.

Furthermore, we estimate the calibration between the IMU and the camera given
by ${\mathbf{p}}_{\scriptscriptstyle IC}$ and
${\mathbf{q}}_{\scriptscriptstyle IC}$. Due to implementation reasons we
assume the number of objects in a scene to be known a priori, but neither the
global poses of the objects nor the relative poses between objects is known.
Therefore, we additionally estimate an object-world which describes the
transformation (${\mathbf{p}}_{\scriptscriptstyle
O_{k}W},{\mathbf{q}}_{\scriptscriptstyle O_{k}W}$) between the object frame
and the navigation world. Both the camera-IMU extrinsics
${\mathbf{p}}_{\scriptscriptstyle IC},{\mathbf{q}}_{\scriptscriptstyle IC}$ as
well as the object poses ${\mathbf{p}}_{\scriptscriptstyle
O_{k}W},{\mathbf{q}}_{\scriptscriptstyle O_{k}W}$ are modeled to remain
constant over time.

For each image, every measured relative pose is treated as an independent
measurement with which the pose of the camera can be estimated. To calculate
the required Jacobians for the update step, we consider the inverted relative
pose measurements ${\mathbf{T}}_{\scriptscriptstyle O_{k}C}$, i.e. the camera
pose relative to the object frame. Based on the relationship between the
different coordinate frames and the independent relative position
$z_{\mathbf{p}_{O_{k}}}$ and orientation $z_{\mathbf{q}_{O_{k}}}$
measurements, the residuals for position $\tilde{z}_{\mathbf{p}_{O_{k}}}$ and
orientation $\tilde{z}_{\mathbf{R}_{O_{k}}}$ can be written as:

$\displaystyle\tilde{z}_{\mathbf{p}_{O_{k}}}$
$\displaystyle=z_{\mathbf{p}_{O_{k}}}-\hat{z}_{\mathbf{p}_{O_{k}}}$
$\displaystyle={\mathbf{p}}_{\scriptscriptstyle
O_{k}C}-({\mathbf{p}}_{\scriptscriptstyle
O_{k}W}+{\mathbf{R}}_{\scriptscriptstyle
O_{k}W}({\mathbf{p}}_{\scriptscriptstyle WI}+{\mathbf{R}}_{\scriptscriptstyle
WI}~{}{\mathbf{p}}_{\scriptscriptstyle IC}))$ (5)
$\displaystyle\tilde{z}_{\mathbf{R}_{O_{k}}}$
$\displaystyle=2~{}\frac{\tilde{z}_{\mathbf{q}_{\mathbf{v},O_{k}}}}{\tilde{z}_{q_{w,O_{k}}}}$
(6) $\displaystyle\tilde{z}_{\mathbf{q}_{O_{k}}}$
$\displaystyle=\hat{z}_{\mathbf{q}_{O_{k}}}^{-1}\otimes
z_{\mathbf{q}_{O_{k}}}=({\mathbf{q}}_{\scriptscriptstyle
O_{k}W}\otimes{\mathbf{q}}_{\scriptscriptstyle
WI}\otimes{\mathbf{q}}_{\scriptscriptstyle
IC})^{-1}\otimes{\mathbf{q}}_{\scriptscriptstyle O_{k}C}$ (7)
$\displaystyle\tilde{z}_{O_{k}}$
$\displaystyle=\begin{bmatrix}\tilde{z}_{\mathbf{p}_{O_{k}}}\\\
\tilde{z}_{\mathbf{R}_{O_{k}}}\end{bmatrix}$ (8)

Given these residuals and depending on a single pose measurement from object
$O_{k}$, the Jacobian for the position $H_{\mathbf{p}}$ and orientation
$H_{\mathbf{R}}$ with respect to the states is [33]:

$\displaystyle H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle WI}}$
$\displaystyle={\mathbf{R}}_{\scriptscriptstyle O_{k}W}$ (9) $\displaystyle
H_{\mathbf{p},{\mathbf{R}}_{\scriptscriptstyle WI}}$
$\displaystyle=-{\mathbf{R}}_{\scriptscriptstyle
O_{k}W}{\mathbf{R}}_{\scriptscriptstyle WI}[{\mathbf{p}}_{\scriptscriptstyle
IC}]_{\times}$ (10) $\displaystyle
H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle IC}}$
$\displaystyle={\mathbf{R}}_{\scriptscriptstyle
O_{k}W}{\mathbf{R}}_{\scriptscriptstyle WI}$ (11) $\displaystyle
H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle O_{k}W}}$
$\displaystyle=\mathbf{I}_{3}$ (12) $\displaystyle
H_{\mathbf{p},{\mathbf{R}}_{\scriptscriptstyle O_{k}W}}$
$\displaystyle=-{\mathbf{R}}_{\scriptscriptstyle
O_{k}W}[{\mathbf{p}}_{\scriptscriptstyle
WI}]_{\times}-{\mathbf{R}}_{\scriptscriptstyle
O_{k}W}[{\mathbf{R}}_{\scriptscriptstyle
WI}~{}{\mathbf{p}}_{\scriptscriptstyle IC}]_{\times}$ (13) $\displaystyle
H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle WI}}$
$\displaystyle={\mathbf{R}}_{\scriptscriptstyle IC}^{T}$ (14) $\displaystyle
H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle IC}}$
$\displaystyle=\mathbf{I}_{3}$ (15) $\displaystyle
H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle O_{k}W}}$
$\displaystyle={\mathbf{R}}_{\scriptscriptstyle
IC}^{T}{\mathbf{R}}_{\scriptscriptstyle WI}^{T}$ (16)

where, e.g. $H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle WI}}$ only
considers the part of the residual $\tilde{z}_{\mathbf{p}_{O_{k}}}$ that
depends on the state ${\mathbf{p}}_{\scriptscriptstyle WI}$. The rest of the
Jacobians are equal to $\textbf{0}_{3}$. As relative pose measurements for
different objects are independent of each other, the Jacobians for the other
($i\neq n$) object-world states, i.e.
$H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle O_{i}W}}$,
$H_{\mathbf{p},{\mathbf{R}}_{\scriptscriptstyle O_{i}W}}$,
$H_{\mathbf{R},{\mathbf{p}}_{\scriptscriptstyle O_{i}W}}$,
$H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle O_{i}W}}$, are all equal to
$\mathbf{0}_{3}$. For a single object $O_{k}$, the Jacobian is given by
stacking the individual components:

$\displaystyle H_{\mathbf{p},O_{k}}=[$ $\displaystyle
H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle
WI}}\;H_{\mathbf{p},{\mathbf{v}}_{\scriptscriptstyle
WI}}\;H_{\mathbf{p},{\mathbf{R}}_{\scriptscriptstyle
WI}}\;H_{\mathbf{p},{\mathbf{b}}_{\scriptscriptstyle\omega}}\;H_{\mathbf{p},{\mathbf{b}}_{\scriptscriptstyle
a}}$ (17) $\displaystyle H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle
IC}}\;H_{\mathbf{p},{\mathbf{R}}_{\scriptscriptstyle
IC}}\;H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle
O_{0}W}}\;H_{\mathbf{p},{\mathbf{R}}_{\scriptscriptstyle O_{0}W}}$
$\displaystyle\dots H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle
O_{N}W}}\;H_{\mathbf{p},{\mathbf{R}}_{\scriptscriptstyle O_{N}W}}]$
$\displaystyle H_{\mathbf{R},O_{k}}=[$ $\displaystyle
H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle
WI}}\;H_{\mathbf{R},{\mathbf{v}}_{\scriptscriptstyle
WI}}\;H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle
WI}}\;H_{\mathbf{R},{\mathbf{b}}_{\scriptscriptstyle\omega}}\;H_{\mathbf{R},{\mathbf{b}}_{\scriptscriptstyle
a}}$ (18) $\displaystyle H_{\mathbf{R},{\mathbf{p}}_{\scriptscriptstyle
IC}}\;H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle
IC}}\;H_{\mathbf{R},{\mathbf{p}}_{\scriptscriptstyle
O_{0}W}}\;H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle O_{0}W}}$
$\displaystyle\dots H_{\mathbf{R},{\mathbf{p}}_{\scriptscriptstyle
O_{N}W}}\;H_{\mathbf{R},{\mathbf{R}}_{\scriptscriptstyle O_{N}W}}]$
$H_{O_{k}}=\begin{bmatrix}H_{\mathbf{p},O_{k}}\\\
H_{\mathbf{R},O_{k}}\end{bmatrix}$ (19)

Depending on the current image, the final residual $z$ and observation matrix
$\mathbf{H}$ for the state update is determined by vertically stacking the
residuals and Jacobians, individually, from Eq. 8 and Eq. 19, respectively,
for each object that was detected for the current update step. Similar to
hardware sensors, our AI-based pose sensor might return faulty or inaccurate
measurements. In a similar fashion as described in [34], we conduct a
$\chi^{2}$ test based on the EKF innovation $\mathbf{S}$ and the residual to
detect outlier measurements. This is applied for the measurement of each
object individually. Outlier measurements for object $O_{k}$ are then rejected
and the final residual and Jacobian have to be rebuild by masking the
corresponding rows. The correction is then calculated based on this final
total residual and associated Jacobian. In the update step, measurement
uncertainties for each measurement can be considered. For the present work,
these uncertainties have been fixed to 10 cm and 20 degrees for the position
and orientation measurement of each object, respectively. These numbers were
determined based on the standard deviation of PoET across a video sequence
reported in [26].

The proper initialization of the individual frames is an important aspect to
consider. At the beginning of the recording, we initialize an arbitrary but
fixed navigation world $W$. Without loss of generality, the IMU frame is
initialized at the origin of $W$. The initial extrinsic calibration between
the IMU and camera is determined through visual-inertial calibration [35].
Each object-world is initialized when the corresponding object is seen by the
camera for the first time. The object frame is then placed with respect to the
world frame by taking the currently estimated pose of the camera and the
relative pose measurement:

$\displaystyle{\mathbf{R}}_{\scriptscriptstyle O_{k}W}$
$\displaystyle={\mathbf{R}}_{\scriptscriptstyle
O_{k}C}{\mathbf{R}}_{\scriptscriptstyle
IC}^{T}{\mathbf{R}}_{\scriptscriptstyle WI}^{T}$ (20)
$\displaystyle{\mathbf{p}}_{\scriptscriptstyle O_{k}W}$
$\displaystyle={\mathbf{p}}_{\scriptscriptstyle
O_{k}C}-{\mathbf{R}}_{\scriptscriptstyle
O_{k}W}({\mathbf{R}}_{\scriptscriptstyle
WI}~{}{\mathbf{p}}_{\scriptscriptstyle IC}+{\mathbf{p}}_{\scriptscriptstyle
WI})$ (21)

In our problem formulation, the robot’s pose $I$ is estimated relative to a
set of object-worlds $O_{k}$ through relative pose measurements. As the
robot’s pose is relative to a world frame and the measurements are relative to
the corresponding object frames, the object-worlds can be placed freely with
respect to the world frame, which does cause observability issues. By fixing
the state of one object-world reference frame, the system is rendered
observable. This fixed object, from now on called the main object $O_{m}$,
serves as the anchor point for the object-relative 6-DoF state estimation. The
position of the main object’s frame with respect to the navigation world is
not changed, i.e. the corresponding Jacobians
$H_{\mathbf{p},{\mathbf{p}}_{\scriptscriptstyle O_{m}W}}$ and
$H_{\mathbf{R},{\mathbf{p}}_{\scriptscriptstyle O_{m}W}}$ are set to
$\mathbf{0}_{3}$. Measurements in which the main object is not visible in the
picture are directly rejected. Otherwise, estimating the object-world of
additional objects while the anchor is not visible leads to ambiguous updates.

The propagation step is performed at the rate of the IMU sensor readings,
while the update step happens with the frequency of the camera images.

##  IV EXPERIMENTS & RESULTS

Figure 2: Object configuration that was used for sequence 4 (left) and object
poses as estimated by PoET (right). Note the difference in object package
coloring between real-world objects (left) and YCB-V objects used for training
PoET (right). The left image, as shown here, is directly fed into our pose
estimation framework to get the 6-DoF relative pose measurements.

In this section, we present the experiments conducted and discuss the results
in detail. We trained PoET on the YCB-V dataset [9] as described in [26], a
benchmark dataset for 6-DoF pose estimation, and chose a subset of objects to
serve as objects of interest in our experiments. The images and IMU data were
recorded using an Intel Realsense D435i with an RGB resolution of 1280x720 and
30 FPS. After undistorting the images, they were cropped to a resolution of
640x480, which is the standard resolution of the YCB-V dataset. We record our
own real data by placing the objects in our motion capture room and moving
around the objects with the camera while tracking the body of the camera. This
mimics the inspection of a set of objects of interest with a mobile robotic
platform with 6-DoFs. An example object configuration and image can be found
in Fig. 2. Because we do not record any information regarding the global
position of the objects in the room, the trajectory derived from the object-
relative state estimation has to be aligned with the ground truth trajectory
to calculate the error metrics. It is important to note the differences
between the benchmark data and our own real data. The camera which was used to
record the YCB-V dataset has a different recording resolution, field of view,
and focal point than the camera used during our experiments. In addition, some
real world objects had slightly different appearance (size and coloring) than
the ones used for the data set that PoET was trained with. The results
reported here have been obtained with the YCB-V trained model of PoET. We did
not perform any retraining or fine-tuning of the model to adapt to these
differences.

TABLE I: RMSE for position and orientation across the whole trajectory for
different sequences.

Sequence | #objects | (x, y, z) [m] | (roll, pitch, yaw) [deg]
---|---|---|---
1 | 3 | [0.066, 0.163, 0.034] | [3.00, 6.81, 4.52]
2 | 3 | [0.162, 0.358, 0.233] | [54.38, 19.39, 20.27]
3 | 2 | [0.163, 0.338, 0.136] | [25.34, 16.57, 15.13]
4 | 4 | [0.045, 0.157, 0.108] | [15.38, 5.58, 10.02]
5 | 1 | [0.075, 0.031, 0.054] | [61.12, 9.66, 26.19]
6 | 3 | [0.061, 0.117, 0.029] | [11.00, 5.77, 7.41]
7 | 4 | [0.093, 0.159, 0.099] | [19.38, 12.38, 27.24]
8 | 3 | [0.127, 0.165, 0.104] | [26.19, 11.95, 10.70]
mean | - | [0.099, 0.186, 0.099] | [26.97, 11.0, 15.19]
$\pm$ std |  $\pm$ [0.043, 0.102, 0.062] |  $\pm$ [19.18, 4.75, 8.01]

TABLE II: RMSE for position and orientation across the whole trajectory for
different runs of sequence 4.

Run | (x, y, z) [m] | (roll, pitch, yaw) [deg]
---|---|---
1 | [0.094, 0.107, 0.119] | [4.72, 7.08, 13.16]
2 | [0.077, 0.121, 0.103] | [1.33, 7.24, 8.05]
3 | [0.049, 0.096, 0.092] | [3.14, 1.72, 14.49]
4 | [0.065, 0.119, 0.101] | [7.75, 5.25, 12.90]
5 | [0.069, 0.119, 0.099] | [2.79, 3.35, 11.57]
6 | [0.047, 0.099, 0.099] | [2.53, 3.63, 12.21]
7 | [0.059, 0.090, 0.088] | [1.92, 5.01, 10.89]
8 | [0.057, 0.093, 0.083] | [3.68, 2.92, 11.37]
9 | [0.074, 0.090, 0.096] | [6.85, 4.69, 13.11]
10 | [0.076, 0.101, 0.118] | [7.19, 6.02, 12.10]
mean | [0.067, 0.104, 0.010] | [4.19, 4.69, 11.99]
$\pm$ std |  $\pm$ [0.014, 0.012, 0.011] |  $\pm$[2.20, 1.71, 1.64]

TABLE III: RMSE for position and orientation across the whole trajectory for
different runs of sequence 6.

Run | (x, y, z) [m] | (roll, pitch, yaw) [deg]
---|---|---
1 | [0.061, 0.117, 0.029] | [11.00, 5.77, 7.41]
2 | [0.092, 0.122, 0.052] | [17.17, 16.97, 7.16]
3 | [0.072, 0.112, 0.025] | [13.08, 24.64, 5.93]
4 | [0.083, 0.110, 0.037] | [12.11, 8.24, 4.58]
5 | [0.074, 0.089, 0.023] | [16.02, 5.99, 6.15]
6 | [0.067, 0.079, 0.031] | [12.29, 11.97, 7.19]
7 | [0.064, 0.126, 0.025] | [16.64, 12.26, 7.72]
8 | [0.059, 0.093, 0.036] | [15.86, 6.31, 6.41]
9 | [0.067, 0.092, 0.046] | [12.38, 4.69, 4.49]
10 | [0.057, 0.150, 0.031] | [12.95, 16.12, 6.89]
mean | [0.069, 0.109, 0.034] | [13.95, 11.29, 6.39]
$\pm$ std |  $\pm$ [0.010, 0.020, 0.009] |  $\pm$[2.11, 6.09, 1.07]

Figure 3: Comparison of estimated position and orientation in Euler angles
(mars) and the ground (gt) for run 8 of sequence 4. The components of the
position (x, y, z) and orientation (roll, pitch, yaw) are plotted individually
for the whole sequence. Additionally, we compare the reprojected IMU pose
given the raw PoET estimates for object 3 (obj). The black arrows enclose a
section in which the reprojected IMU pose is out of plotting range. Important
to note, the object was not visible in the camera images between 6.4s and
8.8s.

We conducted two different experiments. First, we investigated the performance
of our approach for 8 different sequences. The sequences varied with respect
to the number of objects present, the constellation of the objects and the
trajectory performed by the camera around the objects. We calculate the root
mean square error (RMSE) for the position and Euler angles by comparing the
estimated trajectory with the respective ground truth trajectory for a single
recording of that sequence. Moreover, we calculate the average RMSE and the
standard deviation (std) over all trajectories. The results are summarized in
Table III. The overall performance across all sequences shows that our method
is able to sufficiently estimate the state given AI-based, relative 6-DoF pose
measurements for a variety of scenarios. For most sequences, the position RMSE
is around 10 cm or less, which is an acceptable error for this rather complex
task. Furthermore, these results indicate that our method is applicable for
object relative state estimation. In some cases, i.e. sequence 2 and 3, the
achieved performance is worse than in others. This is due to outliers in the
predictions of the 6-DoF pose estimator. Objects being partially out of the
image or ambiguous viewpoints as well as motion blur in the images lead to
wrong pose estimates. Especially the latter one results in the predictions for
all objects in a single image to be wrong. While the $\chi^{2}$ test helps to
reject such frames, multiple consecutive images with noisy or wrong
measurements will still affect the state estimation as it can result in phases
with no updates and only IMU propagation leading to deadreckoning. In such
cases, a prediction of uncertainties for each object and measurement would
lead to better results rather than working with the fixed values described
above. Integration of aleatoric and epistemic uncertainties for the
predictions of PoET is subject to future work.

To illustrate the reproducibility of our approach, we chose two out of the 8
sequences (sequence 4 and sequence 6) and repeated each sequence 10 times
resulting in similar but not exactly the same trajectories. For each sequence,
the RMSE across the whole trajectory for each run and the average RMSE and std
across all runs are summarized in Table III and Table III, respectively. For
both sequences, we are able to reproduce the performance of our method across
10 independent runs with a low standard deviation. This shows an AI-based
component can be reliably incorporated into the state estimation of a robot.

Moreover, we compare the estimated and the ground truth position and
orientation across the whole trajectory for an example recording in Fig. 3.
This example shows that our approach reliably estimates the position and
orientation for the whole duration of the recording. Furthermore, the graphs
show that the raw measurements of a single object sometimes lead to a
reprojected IMU pose that does not align with the ground truth trajectory.
However, by fusing IMU information with pose measurements from multiple
objects our method is able to reliably estimate the trajectory, despite
outlier measurements. In addition to that, we show in Fig. 4 an example for
the self-calibration capabilities of our approach with respect to the object-
world states. The object-world is wrongly initialized after it was first
observed due to a possible noisy measurement. Nonetheless, the state converges
after 5 seconds.

##  V CONCLUSIONS

In this paper, we investigated object relative state estimation for mobile
robots with an AI-based method to extract semantic information (object class
and pose) from single RGB images. We defined a minimal sensor configuration,
consisting of an RGB camera and IMU, and an experimental scenario in which
object relative state estimation is required, mimicking the task of inspection
and monitoring. We derived and implemented a filter-based solution for full
state estimation of a mobile robot given 6-DoF relative pose measurements.
Additionally, our method does not require any initial information about the
global and relative poses of the objects. By defining object-world states, the
coordinate frame of each object is estimated concurrently with respect to a
common navigation world by using one of the objects as an anchor point. Our
experiments with own real data showed that our method can be used for state
estimation of the mobile robot in different scenarios and that the results can
be reliably reproduced. Our results show that AI-based, semantic information
from a single sensor is sufficient in combination with IMU data for accurate
state estimation. Future work will consider incorporating aleatoric and
epistemic uncertainties of the AI-based predictions in the sensor fusion
framework for improved outlier rejection as well the integration of our
proposed approach on a real UAV for closed loop experiments.

Figure 4: Visualization of the estimated object-world state
(${\mathbf{p}}_{\scriptscriptstyle O_{k}W},{\mathbf{q}}_{\scriptscriptstyle
O_{k}W}$) and the corresponding state covariance represented by the std for a
non-main object for run 8 of sequence 4. The position is split up into (x, y,
z), while the orientation is represented by the Euler angles. The states are
plotted from the point of time the object is first observed (at about 10s)
until the states converge. At the beginning the object state is wrongly
initialized due to perhaps a noisy measurement. However, after about 5 seconds
the state converges and the uncertainty becomes minimal.

## References

  * [1] J. Crespo, J. C. Castillo, O. M. Mozos, and R. Barber, “Semantic information for robot navigation: A survey,” _Applied Sciences_ , vol. 10, no. 2, p. 497, 2020.
  * [2] S. Jordan, J. Moore, S. Hovet, J. Box, J. Perry, K. Kirsche, D. Lewis, and Z. T. H. Tse, “State-of-the-art technologies for uav inspections,” _IET Radar, Sonar & Navigation_, vol. 12, no. 2, pp. 151–164, 2018.
  * [3] Y. Li, C. Fu, Z. Huang, Y. Zhang, and J. Pan, “Keyfilter-aware real-time uav object tracking,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 193–199.
  * [4] S. Weiss, D. Scaramuzza, and R. Siegwart, “Monocular-slam–based navigation for autonomous micro helicopters in gps-denied environments,” _Journal of Field Robotics_ , vol. 28, no. 6, pp. 854–874, 2011.
  * [5] E. Allak, A. Hardt-Stremayr, and S. Weiss, “Key-frame strategy during fast image-scale changes and zero motion in VIO without persistent features,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, oct 2018.
  * [6] A. Giusti, J. Guzzi, D. C. Cireşan, F.-L. He, J. P. Rodríguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. Di Caro, _et al._ , “A machine learning approach to visual perception of forest trails for mobile robots,” _IEEE Robotics and Automation Letters_ , vol. 1, no. 2, pp. 661–667, 2015.
  * [7] R. Dominguez, E. Onieva, J. Alonso, J. Villagra, and C. Gonzalez, “Lidar based perception solution for autonomous vehicles,” in _2011 11th International Conference on Intelligent Systems Design and Applications_. IEEE, 2011, pp. 790–795.
  * [8] M. Hofmarcher, T. Unterthiner, J. Arjona-Medina, G. Klambauer, S. Hochreiter, and B. Nessler, “Visual scene understanding for autonomous driving using semantic segmentation,” in _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_. Springer, 2019, pp. 285–296.
  * [9] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes,” in _Robotics: Science and Systems (RSS)_ , 2018.
  * [10] M. Bloesch, S. Omari, M. Hutter, and R. Siegwart, “Robust visual inertial odometry using a direct ekf-based approach,” in _2015 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2015, pp. 298–304.
  * [11] J. Solà, J. Vallvé, J. Casals, J. Deray, M. Fourmy, D. Atchuthan, A. Corominas-Murtra, and J. Andrade-Cetto, “Wolf: A modular estimation framework for robotics based on factor graphs,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4710–4717, 2022.
  * [12] C. Brommer, R. Jung, J. Steinbrener, and S. Weiss, “MaRS: A modular and robust sensor-fusion framework,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 359–366, 2020.
  * [13] T. Moore and D. Stouch, “A generalized extended kalman filter implementation for the robot operating system,” in _Intelligent autonomous systems 13_. Springer, 2016, pp. 335–348.
  * [14] Z. Cao, Y. Sheikh, and N. K. Banerjee, “Real-time scalable 6dof pose estimation for textureless objects,” in _2016 IEEE International conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 2441–2448.
  * [15] G. Pavlakos, X. Zhou, A. Chan, K. G. Derpanis, and K. Daniilidis, “6-dof object pose from semantic keypoints,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 2011–2018.
  * [16] A. Amini, A. S. Periyasamy, and S. Behnke, “T6d-direct: Transformers for multi-object 6d pose direct regression,” in _DAGM German Conference on Pattern Recognition_. Springer, 2021, pp. 530–544.
  * [17] G. Billings and M. Johnson-Roberson, “Silhonet: An rgb method for 6d object pose estimation,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3727–3734, 2019.
  * [18] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, “Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1521–1529.
  * [19] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox, “Deepim: Deep iterative matching for 6d pose estimation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 683–698.
  * [20] Z. Li, G. Wang, and X. Ji, “Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7678–7687.
  * [21] G. Wang, F. Manhardt, F. Tombari, and X. Ji, “Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 16 611–16 621.
  * [22] C. Li, J. Bai, and G. D. Hager, “A unified framework for multi-view multi-class object pose estimation,” in _Proceedings of the european conference on computer vision (eccv)_ , 2018, pp. 254–269.
  * [23] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic, “Cosypose: Consistent multi-view multi-object 6d pose estimation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 574–591.
  * [24] A. Krull, E. Brachmann, F. Michel, M. Y. Yang, S. Gumhold, and C. Rother, “Learning analysis-by-synthesis for 6d pose estimation in rgb-d images,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 954–962.
  * [25] F. Michel, A. Kirillov, E. Brachmann, A. Krull, S. Gumhold, B. Savchynskyy, and C. Rother, “Global hypothesis generation for 6d object pose estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 462–471.
  * [26] T. Jantos, M. A. Hamdad, W. Granig, S. Weiss, and J. Steinbrener, “PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation,” in _Proceedings of the 6th Conference on Robot Learning_. PMLR, 2023.
  * [27] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. Kelly, and A. J. Davison, “Slam++: Simultaneous localisation and mapping at the level of objects,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 1352–1359.
  * [28] N. Merrill, Y. Guo, X. Zuo, X. Huang, S. Leutenegger, X. Peng, L. Ren, and G. Huang, “Symmetry and uncertainty-aware object slam for 6dof object pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 14 901–14 910.
  * [29] S. L. Bowman, N. Atanasov, K. Daniilidis, and G. J. Pappas, “Probabilistic data association for semantic slam,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 1722–1729.
  * [30] J. Thomas, G. Loianno, K. Daniilidis, and V. Kumar, “Visual servoing of quadrotors for perching by hanging from cylindrical objects,” _IEEE robotics and automation letters_ , vol. 1, no. 1, pp. 57–64, 2015.
  * [31] G. Loianno, V. Spurny, J. Thomas, T. Baca, D. Thakur, D. Hert, R. Penicka, T. Krajnik, A. Zhou, A. Cho, _et al._ , “Localization, grasping, and transportation of magnetic objects by a team of mavs in challenging desert-like environments,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 1576–1583, 2018.
  * [32] K. Máthé, L. Buşoniu, L. Barabás, C.-I. Iuga, L. Miclea, and J. Braband, “Vision-based control of a quadrotor for an object inspection scenario,” in _2016 International Conference on Unmanned Aircraft Systems (ICUAS)_. IEEE, 2016, pp. 849–857.
  * [33] J. Sola, “Quaternion kinematics for the error-state kalman filter,” _arXiv preprint arXiv:1711.02508_ , 2017.
  * [34] C. Brommer, C. Böhm, J. Steinbrener, R. Brockers, and S. Weiss, “Improved state estimation in distorted magnetic fields,” in _2020 International Conference on Unmanned Aircraft Systems (ICUAS)_. IEEE, 2020, pp. 1007–1013.
  * [35] P. Furgale, J. Rehder, and R. Siegwart, “Unified temporal and spatial calibration for multi-sensor systems,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 1280–1286.


of type $(2)_{2}$ such that $B$ has normal crossings, any section of
$H^{0}(\omega_{X})$ lifts to any first order deformation of $X$. The remaining
argument is identical to the Proof of Theorem 4.1; all the cohomological
criteria have been verified in Proposition 3.9, Corollary 3.10 and in
Proposition 4.6. The proof of assertion 1 (b) and (2) are consequences of
Corollary 3.10, Remark 2.14 and the unobstructedness of $X$.

Using the generalized Atiyah sequence (4.3), and arguing as in the proof of
Theorem 4.1, we obtain

$h^{0}(\mathcal{E}_{\omega_{X}})=1\textrm{ and
}h^{1}(\mathcal{E}_{\omega_{X}})=h^{1}(\mathcal{O}_{X})+h^{1}(T_{X})=30$

thanks to Proposition 4.6 (3). Now, arguing as in the proof of Theorem 4.1, we
get that the dimension of the image of $H^{0}(\mathcal{N}_{\varphi})\to
H^{1}(\mathcal{E}_{\omega_{X}})$ is

$h^{0}(\mathcal{E}_{\omega_{X}})-(h^{0}(\omega_{X}))^{2}+h^{0}(\mathcal{N}_{\varphi})=1-36+65=30$

thanks to Proposition 4.6 (4). Thus, the map
$H^{1}(\mathcal{E}_{\omega_{X}})\to H^{0}(\omega_{X})^{\vee}\otimes
H^{1}(\omega_{X})$ is zero. $\square$

We end this section by asking the following natural questions, concerning the
deformations of surfaces of type $(2)_{m}$, as it is evident that our
technique of showing unobstructedness does not work for them.

######  Question 4.7.

Let $X$ be a smooth surface of type $(2)_{m}$.

  * (1)

Are ${\bf Def}_{\varphi}$ and ${\bf Def}_{X}$ unobstructed if $m\geq 3$? The
problem that we face for these surfaces is that
$H^{1}(\mathcal{N}_{\pi_{2}})\neq 0$. Thus, for these surfaces, we know that
the forgetful map ${\bf Def}_{\pi_{2}/\mathbb{P}^{N}}\to{\bf Def}_{\varphi}$
is smooth; however the smoothness of ${\bf Def}_{\pi_{2}/\mathbb{P}^{N}}$ is
unknown despite knowing the smoothness of ${\bf Def}_{\varphi_{2}}$, since the
forgetful map ${\bf Def}_{\pi_{2}/\mathbb{P}^{N}}\to{\bf Def}_{\varphi_{2}}$
is not smooth.

  * (2)

Can surfaces of type $(2)_{m}$, for $m\geq 3$ be deformed to canonical double
covers over surfaces ruled over smooth curves of genus $m$ ?

###  4.3. Dimensions of the quadruple loci

For $X$ of type $(1)_{m}$, $(1^{\prime})_{m}$, $(2)_{2}$ or $(3)_{m}$, we
define $\mathcal{M}_{[X]}^{\textrm{quad}}$ as the locally closed sublocus, in
the (unique) irreducible moduli component $\mathcal{M}_{[X]}$ of $X$, that
parametrizes surfaces of general type whose canonical morphism is a quadruple
cover of a surface of minimal degree. As pointed out in the introduction,
since $\mathcal{M}_{[X]}^{\textrm{quad}}$ is, except in case $(3)_{1}$, a
proper sublocus, quadruple canonical covers resemble hyperelliptic curves and
the existence of a proper hyperelliptic sublocus in the moduli of curves of
genus bigger than $2$. Thus, it is natural to ask for the dimension of
$\mathcal{M}_{[X]}^{\textrm{quad}}$ and of other related sublocus contained in
$\mathcal{M}_{[X]}^{\textrm{quad}}$, namely, the locally closed sublocus
$\mathcal{M}_{[X]}^{G}$ of surfaces whose canonical morphism is a quadruple
Galois cover with the same Galois group $G$ as $X$ and the locally closed
sublocus $\mathcal{M}_{[X]}^{\textrm{Nat}}(G)$ that contains
$\mathcal{M}_{[X]}^{G}$ and parametrizes the so-called natural deformations of
the cover $\pi:X\longrightarrow Y$ (see [Par91], Definition $5.1$; the
canonical morphism of surfaces of $\mathcal{M}_{[X]}^{\textrm{Nat}}(G)$ is
again four–to–one but not Galois in general). Note that for the moduli of
curves and its hyperelliptic locus the three analogous subloci coincide as
double covers are Galois. In the next remark we compute the dimensions of
$\mathcal{M}_{[X]}^{G}$ and $\mathcal{M}_{[X]}^{\textrm{Nat}}(G)$ (this yields
a lower bound for the dimension of $\mathcal{M}_{[X]}^{\textrm{quad}}$). We
omit the proof as it follows from the description of
$\mathcal{M}_{[X]}^{\textrm{Nat}}(G)$ given in [Par91], Section 5, and general
principles of deformation theory under some vanishing conditions, using
methods same as in the proof of Theorem 2.21.

######  Remark 4.8.

The loci $\mathcal{M}_{[X]}^{G}$ and $\mathcal{M}_{[X]}^{\textrm{Nat}}(G)$ are
uniruled and their dimensions are as stated in the following table, where we
compare them with the dimension, computed in Theorems 4.1 and 4.2 and Remark
4.3, of the irreducible moduli component $\mathcal{M}_{[X]}$ of $X$, that
contains them. We exclude the case $(3)_{1}$ from the table as in this case
${\mathcal{M}_{[X]}^{G}=\mathcal{M}_{[X]}}$.

$X$ is general of type |  Dim. of $\mathcal{M}_{[X]}^{G}$ |  Dim. of $\mathcal{M}_{[X]}^{\textrm{Nat}}(G)$ |  Lower bound for dim. of $\mathcal{M}_{[X]}^{\textrm{quad}}$ |  Dim. of $\mathcal{M}_{[X]}$
---|---|---|---|---
$(1)_{m}$ |  $2m+1$ |  $2m+4$ |  $6m+18$ |  $8m+20$
$(1^{\prime})_{m}$ |  $6m+13$ |  $6m+18$ |  $6m+18$ |  $8m+20$
$(2)_{2}$ |  $25$ |  $25$ |  $25$ |  $28$
$(3)_{m}$ ($m\geq 2$) |  $2m+4$ |  $2m+4$ |  $2m+4$ |  $3m+3$

##  5\. Infinitesimal Torelli theorem

The goal of this section is to prove the infinitesimal Torelli theorem for
some smooth families of quadruple covers. Let $X$ be a smooth algebraic
variety of dimension $n$ with ample canonical bundle
$\omega_{X}=\Omega_{X}^{n}$. Let $p:\mathcal{X}\to U_{X}$ be a semiuniversal
deformation of $X$, and we assume that $S$ is smooth. The infinitesimal
Torelli problem for weight $n$ Hodge structure asks how far the complex
structure of $X$ is determined by the decreasing Hodge filtration

$\left(F^{p}=\bigoplus_{i\geq
p}H^{n-i}(X,\Omega_{X}^{i})\right)_{p\in\mathbb{N}}.$

The Hodge filtrations on the fibres glue together to give a subbundle
$\mathcal{F}^{p}$ of $\mathcal{O}_{U_{X}}\otimes
R^{n}p_{*}\mathbb{Z}_{\mathcal{X}}$ and define the period map
$\Phi_{n}:U_{X}\to D_{n}$ to the space $D_{n}$ parametrizing Hodge filtrations
of weight $n$. The tangent map $T\Phi_{n}$ at the special point is the
composition of an injective map and the sum of the linear maps

$\lambda_{i}:H^{1}(X,T_{X})\to\textrm{Hom}(\mathbb{C};H^{n-i}(X,\Omega_{X}^{i}),H^{k-i+1}(X,\Omega_{X}^{i-1})),\quad
n\geq i\geq 1$

induced by from the contraction maps. Hence $T\Phi_{n}$ is injective if
$\lambda_{i}$ is injective for some $i$, in which case $\Phi_{n}$ is an
immersion. We say that

  * (1)

the infinitesimal Torelli theorem holds (for weight $n$ Hodge structures) for
$X$ if $\Phi_{n}$ is an immersion;

  * (2)

the infinitesimal Torelli theorem for periods of $n$ forms holds for $X$ if
$\lambda_{n}$ is an injection.

The infinitesimal Torelli theorem for periods of $n$ forms holds for $X$ if
and only if the following map is a surjection

$H^{n-1}(X,\Omega_{X})\otimes H^{0}(X,\Omega_{X}^{n})\to
H^{n-1}(\Omega_{X}\otimes\Omega_{X}^{n}).$

Classically, a curve of genus $g\geq 2$ satisfies the infinitesimal Torelli
theorem if and only if $g=2$ or it is non-hyperelliptic. When $n=2$ i.e., when
$X$ is a surface, the infinitesimal Torelli theorem for periods of 2 forms
holds for $X$ if and only if the infinitesimal Torelli theorem for weight 2
Hodge structures holds for $X$. The infinitesimal Torelli problem for abelian
covers was studied by Pardini in [Par98] in a very general setting. We refer
to the article of Catanese (see [Cat84]) for counterexamples of Torelli
problems, the article of Bauer and Catanese (see [BC04]) for counterexamples
of the infinitesimal Torelli theorems with $\omega_{X}$ quasi-very ample.

The following criterion under which the infinitesimal Torelli theorem for
weight $n$ Hodge structures holds for $X$ was developed by Flenner.

######  Theorem 5.1.

([Fle86], Theorem 1.1) Let $X$ be a compact $n$-dimensional Kähler manifold
and assume the existence of a resolution of $\Omega_{X}$ by vector bundles

$0\to\mathcal{G}\to\mathcal{F}\to\Omega_{X}\to 0.$

If both conditions are satisfied:

  * (a)

$H^{j+1}(S^{j}\mathcal{G}\otimes\bigwedge^{n-j-1}\mathcal{F}\otimes\omega_{X}^{-1})=0$
for all $0\leq j\leq n-2$;

  * (b)

The pairing $H^{0}(S^{n-p}\mathcal{G}^{-1}\otimes\omega_{X})\otimes
H^{0}(S^{p-1}\mathcal{G}^{-1}\otimes\omega_{X})\to
H^{0}(S^{n-1}\mathcal{G}^{-1}\otimes\omega_{X}^{\otimes 2})$ is surjective for
a suitable $p\in\\{1,2,\dots,n\\}$

then the canonical map
$\lambda_{p}:H^{1}(X,T_{X})\to\textrm{Hom}(\mathbb{C};H^{n-p}(X,\Omega_{X}^{p}),H^{n+1-p}(X,\Omega_{X}^{p-1}))$
is injective.

Notice that if $X$ is a smooth quadruple Galois canonical cover of a smooth
surface of minimal degree with irregularity one, then $X$ is necessarily of
type $(1^{\prime})_{m}$. In order to prove the theorem, we are going to invoke
the theorem of Flenner i.e., Theorem 5.1.

######  Theorem 5.2.

Let $X$ be a smooth surface with irregularity one. Assume the canonical bundle
$\omega_{X}$ is ample and globally generated, and the canonical morphism
$\varphi$ is a quadruple Galois canonical cover onto a smooth surface of
minimal degree. Then the infinitesimal Torelli theorem holds for $X$.

Proof. We work with the notations of § 3.1. We have the following commutative
diagram

${X}$${Z}$${X_{1}=E\times\mathbb{P}^{1}}$$\scriptstyle{j}$$\scriptstyle{\pi_{1}}$$\scriptstyle{p}$

where $Z=\mathbb{V}(\mathcal{E}_{\pi_{1}})$ is the affine bundle over $X_{1}$
with the natural projection $p$. We have the following short exact sequence
(2.5)

$0\to\pi_{1}^{*}\mathcal{E}_{\pi_{1}}^{\otimes
2}\to\Omega_{Z}|_{X}\to\Omega_{X}\to 0.$

We will use the criterion of Flenner (Theorem 5.1).

(a) Since dim$(X)=2$, we need to check
$H^{1}(\Omega_{Z}|_{X}\otimes\omega_{X}^{-1})=0$. We also have the following
exact sequence (2.6)

(5.1)
$0\to\pi_{1}^{*}\Omega_{X_{1}}\to\Omega_{Z}|_{X}\to\pi_{1}^{*}\mathcal{E}_{\pi_{1}}\to
0.$

Now,
$\Omega_{X_{1}}=(\mathcal{O}_{E}\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-2))\oplus(\mathcal{O}_{E}\boxtimes\mathcal{O}_{\mathbb{P}^{1}})$.
Consequently, we obtain:

$H^{1}(\pi_{1}^{*}\Omega_{X_{1}}\otimes\omega_{X}^{-1})=H^{1}(\pi_{1}^{*}(\Omega_{X_{1}}\otimes(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-1)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-m))))$

Now, the last term by projection formula is just

$\displaystyle
H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-1)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-m-2))\oplus
H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-1)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-m))\oplus$
$\displaystyle
H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-2m-4))\oplus
H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-2m-2)).$

This term is zero by Künneth formula and the projection formula. Notice that
we also have

$H^{1}(\pi_{1}^{*}\mathcal{E}_{\pi_{1}}\otimes\omega_{X}^{-1})=H^{1}(\pi_{1}^{*}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-2m-2)))$

This term is just
$H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-2m-2))\oplus
H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(-3)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(-3m-4))$
which is zero by Künneth formula and the projection formula. Thus, it follows
from (5.1) that $H^{1}(\Omega_{Z}^{1}|_{X}\otimes\omega_{X}^{-1})=0$.

(b) Now we check the surjection corresponding to $p=1$, i.e.,

$H^{0}(\pi_{1}^{*}(\mathcal{E}_{\pi_{1}}^{*})^{\otimes
2}\otimes\omega_{X})\otimes H^{0}(\omega_{X})\to
H^{0}(\pi_{1}^{*}(\mathcal{E}_{\pi_{1}}^{*})^{\otimes
2}\otimes\omega_{X}^{\otimes 2}).$

Thus we need to check the surjection of

$H^{0}(\pi_{1}^{*}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(2m+4))\otimes\omega_{X})\otimes
H^{0}(\omega_{X})\to
H^{0}(\pi_{1}^{*}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(2m+4))\otimes\omega_{X}^{\otimes
2}).$

To check this surjection, we use Castelnuovo-Mumford regularity (see [Mum70]).
By projection formula,

$H^{1}(\pi_{1}^{*}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(2m+4)))=H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(2m+4))\oplus
H^{1}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(1)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(m+2))$

and it is easy to check that both terms are zero by Künneth formula and the
projection formula. Now we compute the following cohomology group

$H^{2}(\pi_{1}^{*}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(2)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(2m+4))\otimes\omega_{X}^{-1})=H^{2}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}(1)\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(m+4))\oplus
H^{2}(\psi^{*}\mathcal{O}_{\mathbb{P}^{1}}\boxtimes\mathcal{O}_{\mathbb{P}^{1}}(2))$

and one checks that both terms are zero. That concludes the proof. $\square$

######  Remark 5.3.

Let $X$ be a surface of type $(3)_{m}$. It is easy to see in these cases that
the infinitesimal Torelli theorem holds only if $X$ is a surface of type
$(3)_{1}$. We give a brief explanation following [BC04] for the sake of
completeness. We resume the notations of § 3.3. Since $\textrm{dim}(X)=2$,
infinitesimal Torelli theorem holds $\iff$ infinitesimal Torelli theorem for
periods of 2 forms holds $\iff$ $H^{1}(\Omega_{X}^{1})\otimes
H^{0}(\Omega_{X}^{2})\to H^{1}(\Omega_{X}^{1}\otimes\Omega_{X}^{2})$ is
surjective.

Using
$\Omega_{X}^{1}=(\Omega_{C_{1}}^{1}\boxtimes\mathcal{O}_{C_{2}})\oplus(\mathcal{O}_{C_{1}}^{1}\boxtimes\Omega_{C_{2}})$,
$\Omega_{X}^{2}=\Omega_{C_{1}}^{1}\boxtimes\Omega_{C_{2}}^{1}$, and Künneth
formula, we have

$H^{1}(\Omega_{X}^{1})=\left(H^{0}(\Omega_{C_{1}}^{1})\otimes
H^{1}(\mathcal{O}_{C_{2}})\right)\oplus\left(H^{1}(\mathcal{O}_{C_{1}})\otimes
H^{0}(\Omega_{C_{2}}^{1})\right),$
$H^{0}(\Omega_{X}^{2})=H^{0}(\Omega_{C_{1}}^{1})\otimes
H^{0}(\Omega_{C_{2}}^{1}),$
$H^{1}(\Omega_{X}^{1}\otimes\Omega_{X}^{2})=H^{1}((\Omega_{C_{1}}^{1})^{\otimes
2}\boxtimes\Omega_{C_{2}}^{1})\oplus
H^{1}(\Omega_{C_{1}}^{1}\boxtimes(\Omega_{C_{2}}^{1})^{\otimes
2})=\left(H^{0}((\Omega_{C_{1}}^{1})^{\otimes 2})\otimes
H^{1}(\Omega_{C_{2}}^{1})\right)\oplus\left(H^{1}(\Omega_{C_{1}}^{1})\times
H^{0}((\Omega_{C_{2}}^{1})^{\otimes 2})\right),$

where the last equality is obtained by using the fact that
$H^{1}((\Omega_{C_{j}}^{1})^{\otimes 2})=0$ for $j=1,2$. Notice also that
$H^{1}(\mathcal{O}_{C_{j}}^{1})\otimes H^{0}(\Omega_{C_{j}}^{1})\to
H^{1}(\Omega_{C_{j}}^{1})$ is surjective for $j=1,2$ by the non-degeneracy of
Serre duality. Thus, $H^{1}(\Omega_{X}^{1})\otimes H^{0}(\Omega_{X}^{2})\to
H^{1}(\Omega_{X}^{1}\otimes\Omega_{X}^{2})$ is surjective if and only if

(5.2) $H^{0}(\Omega_{C_{j}}^{1})\times H^{0}(\Omega_{C_{j}}^{1})\to
H^{0}(\Omega_{C_{j}}^{\otimes 2})$

is surjective for $j=1,2$. Notice that (5.2) is surjective for $j=1$ since
$C_{1}$ is a hyperelliptic curve of genus 2. Since $C_{2}$ is also
hyperelliptic, (5.2) is surjective only when $m=1$. $\blacksquare$

We end this article by asking the natural question regarding the infinitesimal
Torelli theorem for smooth surfaces of type $(2)_{m}$.

######  Question 5.4.

Let $X$ be a smooth surface of type $(2)_{m}$. Does the infinitesimal Torelli
theorem hold for $X$? Let us resume the notations of § 3.2. Let
$Z:=\mathbb{V}(\mathcal{E}_{\pi_{2}})$ be the affine bundle over $X_{2}$. It
is easy to verify that $H^{1}(\Omega^{1}_{Z}|_{X}\otimes\omega_{X}^{-1})=0$.
However, to apply the criterion of Flenner (i.e., Theorem 5.1), we need the
surjectivity of the following multiplication map:

$H^{0}(\pi_{2}^{*}(\mathcal{E}_{\pi_{2}}^{*})^{\otimes
2}\otimes\omega_{X})\otimes H^{0}(\omega_{X})\to
H^{0}(\pi_{2}^{*}(\mathcal{E}_{\pi_{2}}^{*})^{\otimes
2}\otimes\omega_{X}^{\otimes 2}).$

It follows from [GP11], Lemma 2.1 that this map is not surjective.

## References

  * [AC10] Altmann, Klaus; Christophersen, Jan Arthur. _Deforming Stanley-Reisner schemes_. Math. Ann. 348 (2010), no. 3, 513–537.
  * [AK90] Ashikaga, Tadashi; Konno, Kazuhiro. _Algebraic surfaces of general type with $c^{2}_{1}=3p_{g}-7$_. Tohoku Math. J. (2) 42 (1990), no. 4, 517–536.
  * [BC04] Bauer, I. C.; Catanese, F. _Symmetry and variation of Hodge structures_. Asian J. Math. 8 (2004), no. 2, 363–390.
  * [BCG21] Bangere, Purnaprajna; Chen, Jungkai; Gallego, Francisco Javier. _On higher dimensional extremal varieties of general type_. Preprint, https://arxiv.org/abs/2101.03411.
  * [BGG20] Bangere, Purnaprajna; Gallego, Francisco Javier, González, Miguel. _Deformations of hyperelliptic and generalized hyperelliptic polarized varieties_. Preprint, https://arxiv.org/abs/2005.00342.
  * [BGMR21] Bangere, Purnaprajna; Gallego, Francisco Javier; Mukherjee, Jayan; Raychaudhury, Debaditya. _Degree six abelian covers of surfaces of minimal degree_. Preprint.
  * [Bea79] Beauville, Arnaud. _L’application canonique pour les surfaces de type général_. (French) Invent. Math. 55 (1979), no. 2, 121–140.
  * [Bea82] Beauville, Arnaud. _L’inégalité $p_{g}\geq 2q-4$ pour les surfaces de type général_. 1982, appendix to [Deb82].
  * [Bog86] Bogomolov, F. A. _Rationality of the moduli of hyperelliptic curves of arbitrary genus._ Proceedings of the 1984 Vancouver conference in algebraic geometry, 17–37, CMS Conf. Proc., 6, Amer. Math. Soc., Providence, RI, 1986.
  * [BW74] Burns, D. M., Jr.; Wahl, Jonathan M. _Local contributions to global deformations of surfaces_. Invent. Math. 26 (1974), 67–88.
  * [Cat84] Catanese, Fabrizio M. E. _Infinitesimal Torelli theorems and counterexamples to Torelli problems_. Topics in transcendental algebraic geometry (Princeton, N.J., 1981/1982), 143–156, Ann. of Math. Stud., 106, Princeton Univ. Press, Princeton, NJ, 1984.
  * [Deb82] Debarre, O. _Inégalités numériques pour les surfaces de type général. (French) [[Numerical inequalities for surfaces of general type]] With an appendix by A. Beauville_. Bull. Soc. Math. France 110 (1982), no. 3, 319–346.
  * [Fle86] Flenner, Hubert. _The infinitesimal Torelli problem for zero sets of sections of vector bundles_. Math. Z. 193 (1986), no. 2, 307–322.
  * [GGP10] Gallego, Francisco Javier; González, Miguel; Purnaprajna, Bangere P. _Deformation of canonical morphisms and the moduli of surfaces of general type_. Invent. Math. 182 (2010), no. 1, 1–46.
  * [GGP13a] Gallego, Francisco Javier; González, Miguel; Purnaprajna, Bangere P. _Deformations of bicanonical morphisms of genus 2-fibrations_. Commutative algebra and algebraic geometry (CAAG-2010), 77–88, Ramanujan Math. Soc. Lect. Notes Ser., 17, Ramanujan Math. Soc., Mysore, 2013.
  * [GGP13b] Gallego, Francisco Javier; González, Miguel; Purnaprajna, Bangere P. _Canonical double covers of minimal rational surfaces and the non-existence of carpets_. J. Algebra 374 (2013), 231–244.
  * [GGP13c] Gallego, Francisco Javier; González, Miguel; Purnaprajna, Bangere P. _An infinitesimal condition to smooth ropes_. Rev. Mat. Complut. 26 (2013), 253–269.
  * [GGP16a] Gallego, Francisco Javier; González, Miguel; Purnaprajna, Bangere P. _Deformations of canonical double covers_. J. Algebra 463 (2016), 23–32.
  * [GGP16b] Gallego, Francisco Javier; González, Miguel; Purnaprajna, Bangere P. _Deformations of canonical triple covers_. J. Algebra 463 (2016), 1–9.
  * [GP96] Gallego, Francisco Javier; Purnaprajna, B. P. _Normal presentation on elliptic ruled surfaces_. J. Algebra 186 (1996), no. 2, 597–625.
  * [GP04] Gallego, Francisco Javier; Purnaprajna, Bangere P. _On triple canonical covers of varieties of minimal degree_. A tribute to Seshadri (2003), Hindustan Book Agency, 241-270.
  * [GP08] Gallego, Francisco Javier; Purnaprajna, Bangere P. _Classification of quadruple Galois canonical covers. I_. Trans. Amer. Math. Soc. 360 (2008), no. 10, 5489–5507.
  * [GP11] Gallego, Francisco Javier; Purnaprajna, Bangere P. _On the bicanonical morphism of quadruple Galois canonical covers_. Trans. Amer. Math. Soc. 363 (2011), no. 8, 4401–4420.
  * [Gon06] González, Miguel. _Smoothing of ribbons over curves_. J. Reine Angew. Math. 591 (2006), 201–235.
  * [Hor76] Horikawa, Eiji. _Algebraic surfaces of general type with small $C^{2}_{1}.$ I_. Ann. of Math. (2) 104 (1976), no. 2, 357–387.
  * [Hor77] Horikawa, Eiji. _On algebraic surfaces with pencils of curves of genus $2$_. Complex analysis and algebraic geometry, pp. 79–90. Iwanami Shoten, Tokyo, 1977.
  * [Hor82] Horikawa, Eiji. _Algebraic surfaces of general type with small $C^{2}_{1}.$ V_. J. Fac. Sci. Univ. Tokyo Sect. IA Math. 28 (1981), no. 3, 745–755 (1982).
  * [Kaw99] Kawamata, Yujiro. _Deformations of canonical singularities_. J. Amer. Math. Soc. 12 (1999), no. 1, 85–92.
  * [Kat86] P. I. Katsylo _Rationality of the moduli spaces of hyperelliptic curves (Russian)._ Izv. Akad. Nauk SSSR Ser. Mat. 48 (1984), 705–710.
  * [Kon91] Konno, Kazuhiro. _Algebraic surfaces of general type with $c^{2}_{1}=3p_{g}-6$_. Math. Ann. 290 (1991), no. 1, 77–107.
  * [LM05] Fuentes-García, Luis; Pedreira, Manuel. _The projective theory of ruled surfaces_. Note Mat. 24 (2005), no. 1, 25–63.
  * [LP12] Lopes, Margarida Mendes; Pardini, Rita. _The geography of irregular surfaces_. Current developments in algebraic geometry, 349–378, Math. Sci. Res. Inst. Publ., 59, Cambridge Univ. Press, Cambridge, 2012.
  * [Man97] Manetti, Marco. _Iterated double covers and connected components of moduli spaces_. Topology 36 (1997), no. 3, 745–764.
  * [MR20] Mukherjee, Jayan; Raychaudhury, Debaditya. _Smoothing of multiple structures on embedded Enriques manifolds_. Preprint, https://arxiv.org/abs/2002.05846
  * [Mum70] Mumford, David. _Varieties defined by quadratic equations_. 1970 Questions on Algebraic Varieties (C.I.M.E., III Ciclo, Varenna, 1969) pp. 29–100 Edizioni Cremonese, Rome.
  * [Ops05] van Opstall, Michael A. _Moduli of products of curves_. Arch. Math. (Basel) 84 (2005), no. 2, 148–154.
  * [Par91] Pardini, Rita. _Abelian covers of algebraic varieties_. J. Reine Angew. Math. 417 (1991), 191–213.
  * [Par98] Pardini, Rita. _On the period map for abelian covers of projective varieties_. Ann. Scuola Norm. Sup. Pisa Cl. Sci. (4) 26 (1998), no. 4, 719–735.
  * [Ser06] Sernesi, Edoardo. _Deformations of Algebraic Schemes._ Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], 334. Springer-Verlag, Berlin, 2006.
  * [Sei92] Seiler, Wolfgang K. _Deformations of ruled surfaces._ J. Reine Angew. Math. 426 (1992), 203–219.
  * [Sei95] Seiler, Wolfgang K. _Moduli of surfaces of general type with a fibration by genus two curves._ Math. Ann. 301 (1995), no. 4, 771–812.
  * [Weh86] Wehler, Joachim. _Cyclic coverings: deformation and Torelli theorem._ Math. Ann. 274 (1986), no. 3, 443–472.
  * [Xia86] Xiao, Gang. _Algebraic surfaces with high canonical degree._ Math. Ann. 274 (1986), no. 3, 473–483.