# Self-Supervised Monocular Scene Decomposition and Depth Estimation

Sadra Safadoust Fatma Güney
KUIS AI Center, Koç University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Self-supervised monocular depth estimation approaches either ignore
independently moving objects in the scene or need a separate segmentation step
to identify them. We propose MonoDepthSeg to jointly estimate depth and
segment moving objects from monocular video without using any ground-truth
labels. We decompose the scene into a fixed number of components where each
component corresponds to a region on the image with its own transformation
matrix representing its motion. We estimate both the mask and the motion of
each component efficiently with a shared encoder. We evaluate our method on
three driving datasets and show that our model clearly improves depth
estimation while decomposing the scene into separately moving components.

##  1 Introduction

Humans are remarkably good at decomposing a scene into semantically meaningful
components and inferring the 3D structure of the scene. This ability is
crucial for self-driving vehicles while navigating complex environments to
avoid collision. In computer vision, we often address the problems of
inferring 3D structure and semantics of the scene separately. Years of
research in geometric computer vision has perfected matching pixels and using
this as a cue for self-supervision recently in deep learning. Yet, most of
these methods fail to attribute a semantic meaning to discovered geometry with
some notable exceptions [22, 36, 26, 19, 35]. Similarly, semantic segmentation
of the scene often solely relies on RGB images while complicated occlusion
relationships between objects can easily be untangled with the help of 3D
information.

Figure 1: Joint Scene Decomposition and Depth Estimation. Monocular depth
estimation methods assume a static scene by relying on the ego-motion to
explain the scene and fail in foreground regions with independently moving
objects (bottom-left, [16]). By decomposing the scene into a set of
components, we estimate a separate rigid transformation for each component,
representing its motion. This improves the results in regions with moving
objects (bottom-right) while simultaneously recovering a decomposition of the
scene, mostly corresponding to moving regions (top-right).

There has been a great progress in learning of scene structure from a moving
camera without using any labels. Self-supervised models can be trained to
accurately estimate depth and the ego-motion of the vehicle by matching
regularities in the scene across time. By following the progress in self-
supervised monocular depth and ego-motion estimation [43, 16], we use view
synthesis as an objective. In view synthesis, two models are trained to
predict both the scene structure from a reference view and the relative ego-
motion from the source view to the reference view. Then, the reference view
can be synthesized by sampling pixels from the source view according to these
estimations. While monocular methods need to associate semantics of the scene
with structure to estimate depth, semantics are not typically represented
explicitly in the model.

The lack of semantics become apparent when there are independently moving
objects whose motion cannot be explained by the ego-motion alone (Fig. 1). The
previous work either introduces additional models to estimate optical flow
[41, 44, 7] and also motion masks [31, 28] or assumes a given segmentation
mask marking semantic regions [7, 18] or instances of objects [5] on the
image. While separate modelling of residual motion with flow or object motion
improves the performance, it also increases the computational cost. In case of
given segmentation masks, static objects are also segmented and processed
separately. Furthermore, errors in the segmentation can propagate to the scene
structure. In this paper, we propose to efficiently learn a decomposition of
the scene into components with their own motion. As shown in Fig. 1,
components correspond to moving regions by successfully separating moving
objects.

Inspired by the success of SE3-Nets on point clouds [2, 3], we introduce
MonoDepthSeg to segment a scene into a set of components and predict the
motion of each component from monocular image sequences. We fix the maximum
number of components for a model and assume rigid body motion for each
component. Instead of one transformation corresponding to ego-motion, we
estimate a set of SE(3) transformations to account for the motion of
independently moving objects in the scene as well. For this purpose, we modify
the pose network’s decoder to estimate a fixed number of poses, i.e. one per
component. In parallel, a mask decoder learns to segment the image into
regions whose motion is encoded by the corresponding pose. In order to model
the close relationship between the structure and the semantics, mask and pose
estimations are tightly correlated through a shared encoder. This also
provides efficiency in comparison to a separate segmentation network entirely.
We apply a regularization on the masks by encouraging a depth ordering between
them.

Figure 2: MonoDepthSeg. Left: The depth network outputs depth estimate
$\hat{\mathbf{D}}_{t}$ for input $\mathbf{I}_{t}$. Right: Given
$\hat{\mathbf{D}}_{t}$ and two consecutive frames $\mathbf{I}_{s}$ and
$\mathbf{I}_{t}$, the shared encoder maps the input frames and the depth
estimate to a common representation shared by the following two decoders. The
mask decoder produces the same resolution $K$ masks
$\\{\mathbf{M}_{1},\dots,\mathbf{M}_{K}\\}$ with skip connections between the
corresponding layers of the encoder. The pose decoder takes the same encoded
representation and converts it into rigid transformations
$\\{\mathbf{T}_{1},\dots,\mathbf{T}_{K}\\}$ corresponding to the masks. Given
the set of rigid transformations and corresponding masks, we transform a 3D
point as a convex combination of estimated transformations weighted according
to estimated masks. The number of components, $K$, is a hyper-parameter of our
model.

We build on top of the recent self-supervised learning approaches for
monocular depth estimation and show the effectiveness of learning to decompose
the scene on KITTI [14, 13], Cityscapes [8], and DDAD [17] datasets. Without
using an additional network other than the mask decoder, we improve depth
estimation and obtain a decomposition of the scene into moving regions which
is coherent both semantically and structurally. Our extensive experiments
demonstrate that our joint formulation is able to resolve problems of the
baseline Monodepth2 [16], especially in challenging foreground regions with
moving objects on KITTI and significantly outperforms it on more complex
Cityscapes and DDAD due to a large number of independently moving objects. In
addition, our method is able to extract a decomposition of the scene into
moving components which is consistent with the estimated structure (Fig. 1).

In summary, our contributions are as follows: (1) We introduce a framework for
jointly estimating depth and a scene decomposition from monocular video
without any ground-truth labels. (2) We estimate the motion of each
independently moving object in addition to ego-motion to achieve a more
accurate depth estimation.

##  2 Related Work

View synthesis was first proposed by enforcing consistency between two
different views in the stereo setup [12]. By using Spatial Transformer
Networks (STNs) [23], Monodepth [15] proposes a fully differentiable image
reconstruction process by also optimizing for left-right consistency check.
According to the depth estimation from a reference view, the source view can
be geometrically warped in a differentiable way using STNs to synthesize the
reference view. Then, the difference between the synthesized reference view
and the original one can be used as a supervisory signal. Starting with
SfMLearner by Zhou et al. [43], monocular depth estimation approaches
generalize view synthesis to adjacent frames by also estimating the relative
pose from one frame to the next. We generalize [43] to scenes with moving
objects by applying the idea of $K$ motion models to real-world image
sequences that was initially proposed for point clouds in [2].

###  2.1 Self-Supervised Monocular Depth

There is an inherent scale ambiguity in monocular depth estimation and several
approaches address this issue differently. Zhan et al. [42] use stereo pairs
for joint training of single view depth and two-frame visual odometry. Wang et
al. [39] argue the necessity of handling the scale problem for monocular input
and apply an effective normalization on the depth estimation before computing
the loss. Bian et al. [1] enforce scale-consistent predictions with a loss
minimizing normalized differences in predictions over the entire sequence.
Mahjourian et al. [29] propose to go beyond pixel-wise losses by minimizing
the alignment error between the inferred 3D geometries of the scene.

There are a number of approaches that propose to jointly estimate monocular
depth, optical flow and camera motion. The idea behind these approaches is to
handle the camera motion as rigid flow and the remaining non-rigid object
motion as residual flow as proposed in GeoNet [41]. DF-Net [44] proposes to
enforce the consistency between depth and flow estimations geometrically in
order to eliminate the propagation of errors from pose and depth estimation to
flow. GLNet [7] proposes to capture multiple geometric constraints and further
improves the results, especially with the epipolar constraint for flow.

Ranjan et al. [31] learn to segment static parts of the scene where optical
flow is estimated using depth and camera motion. EPC++ [28] proposes a
holistic 3D motion parser to segment moving objects and estimate motion maps
for each as well as the background. These methods explicitly compute optical
flow and motion masks with separate networks at the expense of increased
computational complexity. We assume that the most common objects in outdoor
scenes such as cars move rigidly but differently than the ego-motion. Based on
this, we segment them and model their motion separately, but still in a
unified architecture.

Recent approaches [16, 17] report improved results without using flow or
motion masks. The improvements are due to architectural changes, better loss
functions, and some careful consideration of the main assumption in self-
supervised monocular training, i.e. a moving camera in a static scene. An
auto-masking strategy is used to mask out pixels with the same motion as
camera, which violate this assumption. Monodepth2 [16] proposes to handle
occlusions by computing the photometric loss as the minimum over source views
instead of averaging which leads to blurred results. In addition, while using
multi-scale predictions, photometric error is always computed in the full
input resolution to avoid ambiguities at low resolution in low-texture
regions. PackNet [17] uses 3D convolutions to learn detail-preserving
representations via symmetric packing and unpacking blocks. In this paper, we
follow the improvements proposed by recent approaches, specifically Monodepth2
[16], in terms of architecture and loss design.

###  2.2 Depth with Semantics

Joint training of semantics and depth has been explored before, especially in
the context of indoor data where ground-truth depth is available for training
[24]. For outdoor scenes, Cao et al. [4] propose to predict object motion and
depth from stereo motion sequences by using object proposals as input. The
proposed method can factor the scene into independently moving objects but
require stereo sequences and limited supervision for object bounding boxes. In
this paper, our goal is to generalize their success to monocular setup without
any supervision.

For monocular data, Casser et al. [5] propose to estimate the motion of each
object in addition to the ego-motion based on an initial instance
segmentation. Motion estimation and the following warping are performed in
masked regions according to given segmentation. We follow a similar approach
by estimating the motion of each component but we also estimate the masks
instead of relying on an initial segmentation. This way, we save computation
not only by removing the need to initially segment the instances but also by
focusing on the dynamic instances which require a separate modelling. While
they train separate networks for ego-motion and object motion, our motion
estimation is shared across components, leading to better efficiency.

Another two-stage training process for incorporating semantics is proposed by
Guizilini et al. [18]. A pre-trained semantic segmentation network is used to
guide the depth network via pixel-adaptive convolutions between the decoders.
Semantic-aware features produce better estimations especially for dynamic
objects. This proves the tight relationship between the semantics and the
depth as we also explore in this paper. While [18] uses a pre-trained and
fixed segmentation network for semantic guidance, we train a pose and mask
network to predict corresponding poses and masks, with a shared encoder but
separate decoders. Our approach can produce masks corresponding to separately
moving regions on the image. We also perform a detailed evaluation according
to motion segmentation on KITTI and Cityscapes and semantic classes on DDAD
and show meaningful improvements for moving regions.

##  3 Methodology

Our monocular depth and segmentation network, MonoDepthSeg, jointly estimates
the structure and a decomposition of the scene into moving components from
monocular video sequences without using any labels. An overview of our
approach is presented in Fig. 2. Our solution couples these two related tasks
efficiently within a single framework. Given a source frame $\mathbf{I}_{s}$
and a target frame $\mathbf{I}_{t}$ as input where
$\mathbf{I}_{s},\mathbf{I}_{t}\in\mathbb{R}^{W\times H\times 3}$, we decompose
the scene into $K$ components where each component is represented with a mask
and a rigid body motion:

$\hat{\mathbf{I}}_{s}=\theta(\mathbf{I}_{s},\mathbf{I}_{t})~{}~{}|~{}~{}\theta\coloneqq\\{\mathbf{R}_{i},\mathbf{t}_{i},\mathbf{M}_{i}\\},~{}i=1\dots
K$ (1)

where $\hat{\mathbf{I}}_{s}$ is the warped source image according to estimated
parameters $\theta$. For $K$ components, $\theta$ denotes the set of rigid
body transformations $[\mathbf{R}_{i},\mathbf{t}_{i}]\in\textbf{SE}(3)$ and
corresponding masks $\mathbf{M}_{i}\in[0,1]^{W\times H}$ representing the
component $i$. In our setting, we assume that $K$ is a hyper-parameter which
limits the number of components that can be estimated.

Our framework consists of two parts:  (i) a depth network which estimates per-
pixel depth values  (ii) a pose and mask network  which divides the image into
components and estimates a separate pose for each component. Finally, every
pixel on the target frame is transformed according to estimated parameters,
i.e. inverse warped by sampling from the source frame, and compared to the
original target frame. The two networks are trained jointly and end-to-end.

###  3.1 Scene Decomposition

The depth network estimates the depth $\hat{\mathbf{D}}_{t}$ for every pixel
on the target image $\mathbf{I}_{t}$. The pose network solves two related
problems based on a shared representation, i.e. identifying components that
move together and estimating transformation parameters for each component. We
use the depth network of Monodepth2 [16] as explained in Section 3.4. In this
section, we focus on the pose and mask network starting with the assumptions
in our model.

In this work, we make two assumptions. First, we assume rigidly moving
components whose motion can be explained with a rigid body transformation.
This assumption, as explored before [5], typically holds in outdoor scenes
where most of the moving objects are rigidly moving vehicles. Otherwise, it is
an approximation for deformable objects such as pedestrians. Second, we have a
pre-defined maximum number of components, $K$. Based on the second assumption,
we can estimate the segmentation mask $\mathbf{M}$ by assigning each pixel
into one of the $K$ components. In order to make this process differentiable,
we assign soft weights by allowing pixels to belong to more than one
component.

Formally, $\mathbf{M}_{i}(\mathbf{p})$ denotes the probability of the pixel
$\mathbf{p}$ belonging to the component $i$ where the sum of all possible $K$
values for the pixel $\mathbf{p}$ is equal to 1:

$\mathbf{M}(\mathbf{p})=\\{\mathbf{M}_{1}(\mathbf{p}),\dots,\mathbf{M}_{K}(\mathbf{p})\\}~{}|~{}\sum_{i=1}^{K}\mathbf{M}_{i}(\mathbf{p})=1$
(2)

Similar to [11, 34], we encourage masks to be layered according to a pre-
defined depth order, e.g. the first mask appears first, then the second, and
so on. The ordering helps to account for occlusions by putting the foreground
objects before the background. We assign a scalar number $d_{i}$ to each mask
from $1$ to $K$ denoting its order. Then, we weight the mask logits by this
number before applying softmax across $K$ channels at every pixel:

$\mathbf{M}_{i}(\mathbf{p})=\frac{\mathrm{e}^{d_{i}\mathbf{M}^{{}^{\prime}}_{i}(\mathbf{p})}}{\sum_{j=1}^{K}\mathrm{e}^{d_{j}\mathbf{M}^{{}^{\prime}}_{j}(\mathbf{p})}}$
(3)

where $\mathbf{M}^{{}^{\prime}}$ is the output of the mask decoder. With the
help of this regularization, probabilities over $K$ values are ordered
according to a pre-defined depth ordering. As a baseline to the depth
ordering, we experiment with another regularization to reduce the noise in the
masks. We simply encourage mask values to be locally smooth by using an edge-
aware smoothness term in the loss function.

###  3.2 Pose Estimation

We represent the motion of each component using a 3D rigid transformation
$\mathbf{T}=[\mathbf{R},\mathbf{t}]\in\textbf{SE(3)}$ which is composed of a
rotation matrix $\mathbf{R}\in\textbf{SO(3)}$ and a translation vector
$\mathbf{t}\in\mathbb{R}^{3}$. We first shortly introduce the preliminaries
regarding the back-projection of a pixel into 3D, the application of
transformation $\mathbf{T}$ in 3D, and the projection of the transformed 3D
point into the other view. After introducing the general framework for
establishing correspondences between different viewpoints under a rigid
transformation, we generalize it to our framework with a set of rigid
transformations and corresponding masks.

Preliminaries: Assuming a known intrinsic camera matrix $\mathbf{K}$, for a
pixel $\mathbf{p}$ on the target image $\mathbf{I}_{t}$ and its depth value on
$\hat{\mathbf{D}}_{t}$, the corresponding 3D point $\mathbf{x}$ can be
computed as follows:

$\mathbf{x}=\hat{\mathbf{D}}_{t}(\mathbf{p})~{}\mathbf{K}^{-1}~{}\mathbf{p}$
(4)

Given an estimated rigid transformation $\mathbf{T}=[\mathbf{R},\mathbf{t}]$,
we can transform the 3D point $\mathbf{x}$ and project it to find the
corresponding point $\mathbf{p}^{\prime}$ on the source image $\mathbf{I}_{s}$
as follows:

$\displaystyle\mathbf{x}^{\prime}=\mathbf{T}~{}\mathbf{x}=\mathbf{R}~{}\mathbf{x}+\mathbf{t}$
$\displaystyle\hfill\mathbf{p}^{\prime}=\mathbf{K}~{}\mathbf{x}^{\prime}$ (5)

All operations are performed in homogeneous coordinates which are removed for
simplicity.

Decomposition of Transformation: In this paper, we estimate $K$ rigid
transformations corresponding to the motion of $K$ components. We build a
tight relationship between the region of a component on the image, i.e. the
mask, and the motion of that component through a shared encoder. Based on this
shared representation, there are two separate decoders for predicting the
masks and the transformations.

Given the set of predicted rigid transformations $\\{\mathbf{T}_{i}\\}$ and
corresponding masks $\\{\mathbf{M}_{i}\\}$, we write (5) as a weighted
combination of $K$ predictions in a differentiable manner:

$\mathbf{x}^{\prime}=\sum_{i=1}^{K}\mathbf{M}_{i}(\mathbf{p})~{}\mathbf{T}_{i}~{}\mathbf{x}=\sum_{i=1}^{K}\mathbf{M}_{i}(\mathbf{p})~{}(\mathbf{R}_{i}~{}\mathbf{x}+\mathbf{t}_{i})$
(6)

where $\mathbf{x}^{\prime}$ is the transformed 3D point $\mathbf{x}$
corresponding to the pixel $\mathbf{p}$ on the target image. As stated in [2],
the resulting transformation as a convex combination of the transformed 3D
points is not in SE(3) anymore. However, rigid transformations as frequently
observed in outdoor scenes can be approximated using (6) as shown in our
experiments.

###  3.3 Self-Supervised Training

For a pixel $\mathbf{p}$ on the target image $\mathbf{I}_{t}$, we first obtain
the corresponding 3D point $\mathbf{x}$ as shown in (4). Then, we transform
the 3D point $\mathbf{x}$ to $\mathbf{x}^{\prime}$ according to the set of
transformations and masks predicted by the decoders as shown in (6). Finally,
we project the transformed 3D point $\mathbf{x}^{\prime}$ to the corresponding
pixel $\mathbf{p}^{\prime}$ on the source image. We repeat this process for
all the pixels on the target image.

Following the monocular depth estimation approaches [43], we use a
differentiable inverse warping process to reconstruct the target image
$\mathbf{I}_{t}$ by sampling pixels from the source image $\mathbf{I}_{s}$.
Since the projected pixel coordinates are continuous, we perform bilinear
interpolation by using differentiable bilinear sampling mechanism proposed in
the Spatial Transformer Networks [23]. This way, we can approximate the value
of the warped image $\hat{\mathbf{I}}_{s}$ at location $\mathbf{p}$ in a
differentiable manner. As a result, we obtain the corresponding points on the
target image $\mathbf{I}_{t}$, i.e. the warped source image
$\hat{\mathbf{I}}_{s}$, and the source image $\mathbf{I}_{s}$:

$\mathbf{I}_{t}(\mathbf{p})\approx\hat{\mathbf{I}}_{s}(\mathbf{p})=\mathbf{I}_{s}(\mathbf{p}^{\prime})$
(7)

Our loss function minimizes the difference between the corresponding points
over all pixels for pairs of images from a target view and a couple of source
views, i.e. the previous and the next frame. For the photometric
reconstruction loss, we follow [16] and use per-pixel minimum reprojection
loss, and also incorporate the structural similarity (SSIM) loss [40] by
setting $\alpha$ to $0.85$:

$\begin{split}\mathcal{L}_{\textrm{photo}}(\mathbf{p})=\min_{s}\Bigl{[}&\left(1-\alpha\right)\lvert\mathbf{I}_{t}(\mathbf{p})-\hat{\mathbf{I}}_{s}(\mathbf{p})\rvert\\\
&+\frac{\alpha}{2}\left(1-\textrm{SSIM}(\mathbf{I}_{t},\hat{\mathbf{I}}_{s})(\mathbf{p})\right)\Bigr{]}\end{split}$
(8)

$\mathcal{L}_{\textrm{photo}}$ is minimized over the two source views
considered, i.e. the previous and the next frames.

Similar to [39, 16], we define an edge-aware smoothness loss
$\mathcal{L}_{smooth}$ over the mean-normalized inverse depth values. Our
final loss $\mathcal{L}$ is a weighted sum of the loss functions defined over
all pixels, divided by $N$, the number of pixels:

$\mathcal{L}=\frac{1}{N}\sum_{\mathbf{p}}\mathcal{L}_{photo}(\mathbf{p})+\lambda~{}\mathcal{L}_{smooth}(\mathbf{p})$
(9)

###  3.4 Network Architecture

Depth network is based on the U-Net architecture [32], where we have an
encoder-decoder network with skip connections. The encoder for the depth
network is a ResNet50 architecture [20], which is pre-trained on ImageNet
[33]. We use the same depth decoder proposed in Monodepth2 [16] where the
output layer is a sigmoid converted to depth.

Given two consecutive frames, $\mathbf{I}_{s}$ and $\mathbf{I}_{t}$, and the
depth estimate $\hat{\mathbf{D}}_{t}$ for $\mathbf{I}_{t}$, a second ResNet50
architecture is modified to accept two RGB images and the depth estimate as
input. We concatenate the depth estimate as input to the pose and mask network
similar to [27]. As shown on the right in Fig. 2, the output of this encoder
is shared between the pose and mask decoders [2]. This way, we learn a shared
representation for the structure and the decomposition of the scene in an
efficient way. The pose decoder outputs $K$ 6-DoF relative poses corresponding
to $K$ masks. Following [39, 16], we predict the rotation in axis-angle
representation, and scale the rotation and translation outputs by $0.01$.

Based on the same encoded representation as the pose decoder, the mask decoder
outputs $K$ masks, i.e. a $K$ channel image where each channel represents the
probability of each pixel belonging to that component [2]. We use transposed
convolutions in the mask decoder to compute pixel-wise masks at the input
resolution. We also add skip connections between the encoder and the mask
decoder to better preserve boundaries. For the pose and mask network, we also
experiment by initializing it from a pre-trained segmentation model with
ResNet50 backbone, DeepLabv3+ [6].

##  4 Experiments

Components | Lower Better | Higher Better
---|---|---
($K$) | Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
2 | 0.091 | 0.832 | 4.092 | 0.167 | 0.919 | 0.967 | 0.982
3 | 0.090 | 0.801 | 4.123 | 0.167 | 0.920 | 0.967 | 0.982
5 | 0.084 | 0.719 | 3.943 | 0.163 | 0.925 | 0.967 | 0.983
10 | 0.085 | 0.620 | 3.984 | 0.164 | 0.917 | 0.966 | 0.983
Table 1: Number of Components. This table shows our results according to changing the number of components ($K$). The best in each column is shown in bold and the second best is underlined. Increasing the number of components to $K=5$ performs favorably compared to smaller number of components with $K=2$ and $K=3$ and a large number of components with $K=10$. Components | Depth Ordering | Lower Better | Higher Better
---|---|---|---
($K$) |  | Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
5 | No | 0.089 | 0.832 | 3.983 | 0.164 | 0.924 | 0.967 | 0.982
10 | No | 0.089 | 0.827 | 4.044 | 0.166 | 0.922 | 0.966 | 0.982
5 | Yes | 0.084 | 0.719 | 3.943 | 0.163 | 0.925 | 0.967 | 0.983
10 | Yes | 0.085 | 0.620 | 3.984 | 0.164 | 0.917 | 0.966 | 0.983
Table 2: Depth Ordering. This table shows the results of using depth ordering compared to simply encouraging smooth masks for regularization. Depth ordering improves the performance of our models over all metrics, especially for squared relative error (Sq Rel).  Components | DeepLabv3+ | Lower Better | Higher Better
---|---|---|---
($K$) |  Pre-Training [6] | Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
5 | No | 0.087 | 0.697 | 4.031 | 0.169 | 0.918 | 0.964 | 0.981
10 | No | 0.092 | 0.595 | 4.111 | 0.174 | 0.908 | 0.961 | 0.981
5 | Yes | 0.084 | 0.719 | 3.943 | 0.163 | 0.925 | 0.967 | 0.983
10 | Yes | 0.085 | 0.620 | 3.984 | 0.164 | 0.917 | 0.966 | 0.983
Table 3: Pre-trained Segmentation Network. This table compares the results of
using a pre-trained DeepLabv3+ model for semantic segmentation [6] as the pose
and mask network. We modify it to add the pose decoder in addition to the mask
decoder and train within our framework. For different number of components,
using the modified DeepLabv3+ improves the results.

We evaluate MonoDepthSeg on KITTI [13], Cityscapes [8], and DDAD [17]
datasets. In this section, we provide details of training and present our
results. We first perform some ablation studies to justify our design choices
and to show the effect of different values of hyper-parameters in our
framework. We then present our results both quantitatively and qualitatively.

###  4.1 Datasets

KITTI: We train our model on the Eigen split [10] of KITTI dataset [14, 13].
We follow the pre-processing proposed in [43] and remove the static scenes
which results in 39810 samples for training, 4424 for validation, and 697 for
test set. We also present results using the improved ground-truth obtained
with the technique proposed in [37] to compare our results to the latest
approaches [17]. In the improved ground-truth, Uhrig et al. [37] remove the
outliers in the LiDAR scans and increase the density of the ground-truth,
resulting in high quality 652 depth maps, especially improving the original
ground-truth for dynamic objects. We train only on KITTI dataset, without pre-
training on Cityscapes as done by some previous work, and compare to the other
approaches under the same training conditions.

Cityscapes: We also train and evaluate on the Cityscapes dataset [8]. For
training, we choose 12 of 18 cities containing 30-frame videos from the
training split, which results in 48802 samples. For validation, we choose 2 of
the 3 cities containing 30-frame videos from the validation split, resulting
in 6058 samples. We evaluate on all 1525 samples in the original test set. We
use the provided disparity ground-truths obtained using the SGM stereo method
[21], and convert them to depth for evaluation. Similar to KITTI, we evaluate
depth for distances up to 80 m.

DDAD: We also conduct experiments on the recent DDAD dataset [17] capturing
precise structure across images at longer ranges up to 250 m, making it a more
challenging and realistic benchmark for the task of depth estimation. We use
12350 training and 3850 validation samples from the front camera and evaluate
on 3080 test images using the online evaluation server [9].

###  4.2 Training Details

We use Adam optimizer [25] with parameters
$\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$, with a learning rate of
$10^{-4}$. For KITTI, we fix the input resolution to $640\times 192$ for
efficiency and compare to recent approaches trained on resolutions not higher
than that, and for Cityscapes we resize the images to $512\times 256$. On both
of these datasets, we train for 20 epochs where the learning rate is reduced
to $10^{-5}$ for the last 5 epochs. For DDAD, we use images of size $640\times
384$ (similar to [17]) and train for 30 epochs.

The weighting in the loss, i.e. $\lambda$ is set to $0.001\times\text{scale}$
where $\text{scale}\in\\{1,\frac{1}{2},\frac{1}{4},\frac{1}{8}\\}$. We use
depth ordering for regularization by default, but also experiment with local
smoothing of masks. We use the ResNet50 architecture [20] by default for both
the depth network and the pose and mask network. For the pose and mask
network, we experiment by using the pre-trained DeepLabv3+ architecture for
the mask part [6]. We note that architectural improvements [17] or
improvements due to higher input resolutions as observed in recent work [16,
17] are complementary to our approach and can be used to improve the results.

###  4.3 Ablation Study

In this section, we analyze different parts of our model including the maximum
number of components, depth ordering for regularization, and using a pre-
trained segmentation network for the pose and mask network. We train each
model for 20 epochs on KITTI and present the results on the validation set
using the original ground truth.

Number of Components: In order to evaluate the effect of scene decomposition,
we perform an ablation study by changing the maximum number of components in
the model. Specifically, we compare different values of $K$ ranging over
$2,3,5,\text{and }10$ components. As can be seen from Table 1, the number of
components affects the results. A small number of components can be the
limiting factor for the model as observed for $K=2$ and $K=3$. While
increasing it to $K=5$ improves the results for all the metrics, $K=10$
performs a little worse than $K=5$. The number of components clearly depends
on the complexity of scenes in the dataset. This parameter might need to be
adjusted for other datasets with more foreground objects but on KITTI, $K=5$
components results in the best performance in almost all metrics among the
values compared.

Depth Ordering: We investigate the effect of depth ordering on the performance
of our model by comparing the results with and without depth ordering. When we
remove the depth ordering, we still apply a regularization on the masks with
simple mask smoothing in order to reduce the noise. As can be seen in Table 2,
depth ordering improves the results for both $K=5$ and $K=10$. The difference
between the depth ordering and smoothing masks is local versus global
regularization. Depth ordering relates the masks over the channels
corresponding to components by forcing similar depth-level pixels to cluster
on the same mask while mask smoothing only locally enforces similar values on
each mask independently. For this reason, we use depth ordering for
regularization in the following experiments.

Pre-Trained Segmentation Network: In order to push the limits for the pose and
mask network, we use a successful segmentation model, DeepLabv3+ [6], with the
ResNet50 backbone and first pre-train it on Cityscapes dataset [8]. Then, we
modify the DeepLabv3+ architecture by adding a pose decoder in addition to the
mask decoder. Following our initial design, the encoder is still shared
between the pose and the mask decoders. We train our framework with the
modified DeepLabv3+ for the pose and mask network. Note that we initialize the
encoder and the mask decoder from the DeepLabv3+ model pre-trained in a
supervised manner on Cityscapes but our training is still self-supervised. As
shown in Table 3, using the modified DeepLabv3+ model outperforms the original
ResNet50 model for the pose and mask network. In the following experiments, we
use the modified DeepLabv3+ model.

###  4.4 Results

Data | Method | Abs Rel | RMSE
---|---|---|---
Moving | All | Moving | All
KITTI | Md2 [16] | 0.143 | 0.110 | 5.949 | 4.642
Ours | 0.138 | 0.110 | 5.796 | 4.700
City- scapes | Md2 [16] | 0.158 | 0.170 | 8.043 | 8.155
Ours | 0.143 | 0.142 | 7.649 | 7.361
Table 4: Quantitative Results in Moving Regions and Overall. Comparing our approach to Monodepth2 [16] (denoted as Md2) for moving regions and overall on KITTI and Cityscapes datasets. | Method | Supervision | Lower Better | Higher Better
---|---|---|---|---
| Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
Original [10] | Godard et al. [16] | M | 0.110 | 0.831 | 4.642 | 0.187 | 0.883 | 0.962 | 0.982
Guizilini et al. [17] | M | 0.111 | 0.785 | 4.601 | 0.189 | 0.878 | 0.960 | 0.982
MonoDepthSeg (Ours) | M | 0.110 | 0.792 | 4.700 | 0.189 | 0.881 | 0.960 | 0.982
Casser et al. [5] | S+Ins | 0.141 | 1.025 | 5.290 | 0.215 | 0.816 | 0.945 | 0.979
Chen et al. [7] | S+Sem | 0.118 | 0.905 | 5.096 | 0.211 | 0.839 | 0.945 | 0.977
Guizilini et al. [18] | M+Sem | 0.102 | 0.698 | 4.381 | 0.178 | 0.896 | 0.964 | 0.984
Improved [37] | Luo et al. [28] | M | 0.120 | 0.789 | 4.755 | 0.177 | 0.856 | 0.961 | 0.987
Godard et al. [16] | M | 0.085 | 0.468 | 3.672 | 0.128 | 0.921 | 0.985 | 0.995
Guizilini et al. [17] | M | 0.078 | 0.420 | 3.485 | 0.121 | 0.931 | 0.986 | 0.996
MonoDepthSeg (Ours) | M | 0.085 | 0.458 | 3.779 | 0.131 | 0.919 | 0.985 | 0.996
Table 5: Quantitative Results on KITTI. This table compares our proposed approach MonoDepthSeg to previous approaches on KITTI dataset. The Supervision column shows different types of supervision used by different methods, M stands for monocular sequences, and S for methods that train on stereo pairs. We mark the methods which require semantic (Sem) or instance-level (Ins) information. We show the results for the input resolution $640\times 192$, with pre-trained segmentation network, and by training on KITTI for depth. The best self-supervised method in each column is shown in bold and the second best is underlined. Method | Abs Rel | RMSE | Car | Person
---|---|---|---|---
Guizilini et al. [17] | 0.23 | 17.92 | 0.38 | 0.20
Godard et al. [16] | 0.22 | 17.63 | 0.25 | 0.21
Ours | 0.19 | 16.61 | 0.24 | 0.17
Table 6: Quantitative Results on DDAD. The _Car_ and _Person_ are shown using
_Abs Rel_. Our method achieves the state-of-the-art results without using a
pre-trained segmentation network.

We compare the performance of our method to the state-of-the-art by choosing
the best performing configuration according to our ablation study, i.e. $K=5$
with depth ordering and pre-trained segmentation network. For KITTI, we
evaluate our method both with respect to the original ground-truth [10] and
the improved ground-truth [37]. As shown in Table 5, our method performs
comparably to the state-of-the-art methods PackNet [17] and Monodepth2 [16] in
the monocular setting according to both the original and the improved ground-
truth. Furthermore, we generate a scene decomposition jointly without
conditioning on any segmentation map. Our improvements are particularly
pronounced in foreground regions with moving objects (Fig. 3 and Table 4) but
the overall performance is also improved with respect to the methods which use
stereo pairs and rely on the output of a semantic [7] or instance segmentation
network [5]. Furthermore, our joint scene decomposition formulation can
compute the masks and poses of images in a single pass (9.057 samples per
second), whereas using pre-computed masks as in [5], requires a separate pass
for each object in the image (4.124 samples per second). While the state-of-
the-art approach [17] uses an architecture with a large number of parameters
due to 3D convolutions, we achieve similar results with a simpler architecture
based on ResNet50. The followup work by the same authors improve the
performance further by using supervision for semantic segmentation [18]. Note
that we do not condition on the output of a segmentation network but
automatically discover components which move independently (Fig. 3).

We compare our method with Monodepth2 [16] in moving regions. We extract
moving regions using motion segmentation methods on KITTI [30] and Cityscapes
[38]. Our method achieves significantly better results than Monodepth2 on
Cityscapes in overall, and clearly outperforms it in moving regions on both
KITTI and Cityscapes (Table 4). Please check the supplementary for the full
tables.

Finally, we evaluate the performance of our model on the more complex DDAD
dataset. Note that because the online evaluation server [9] does not allow any
semantic supervision, our model is not initialized by a pre-trained
segmentation network on this dataset. Despite this, our method outperforms the
state-of-the-art method [17] and Monodepth2 [16] in all metrics including car
and person classes, as can be seen in Table 6.

Figure 3: Qualitative Results. Each row shows from top-to-bottom: The input
image (the first column), Monodepth2 [16] results as the baseline (the second
column), our results with decomposition (the last two columns). In the last
column, each channel of the mask is assigned a random color and the brightness
level of the color is adjusted according to our estimation. The moving regions
(circled in green) containing vehicles, bikes, and pedestrians are clearly
segmented out as a separate component and assigned a different color. By
learning a separate transformation for each component, we obtain more accurate
depth estimations especially in moving foreground regions.

##  5 Conclusion and Future Outlook

We proposed a joint scene decomposition and depth estimation model. We showed
that joint formulation improves depth results, especially in moving foreground
regions, compared to Monodepth2 as well as the other recent methods which
condition on the output of a semantic or instance segmentation. Our model can
discover components on the image which correspond to moving regions without
any conditioning. The maximum number of components is fixed, however, outdoor
scenes are not equally complex. For most scenes, ego-motion alone can explain
the structure of the whole scene, as evidenced by the success of previous
monocular depth estimation approaches. As we showed in this paper, their
performance can be improved, especially in moving foreground regions, by
decomposing the scene and allowing other types of motion estimations. We plan
to work on extending our formulation to an adaptive number of components
according to scene complexity in the future.
Acknowledgements. S. Safadoust was supported by KUIS AI Center and F. Güney by
Marie Skłodowska-Curie Individual and TUBITAK 2232 Fellowship programs.

## References

  * [1] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid.  Unsupervised scale-consistent depth and ego-motion learning from monocular video.  In Advances in Neural Information Processing Systems (NeurIPS), pages 35–45, 2019.
  * [2] Arunkumar Byravan and Dieter Fox.  SE3-Nets: Learning rigid body motion using deep neural networks.  In Proc. IEEE International Conf. on Robotics and Automation (ICRA), pages 173–180, 2017.
  * [3] Arunkumar Byravan, Felix Leeb, Franziska Meier, and Dieter Fox.  SE3-Pose-Nets: Structured deep dynamics models for visuomotor control.  In Proc. IEEE International Conf. on Robotics and Automation (ICRA), pages 1–8, 2018.
  * [4] Zhe Cao, Abhishek Kar, Christian Hane, and Jitendra Malik.  Learning independent object motion from unlabelled stereoscopic videos.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 5594–5603, 2019.
  * [5] Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova.  Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos.  In Proc. of the Conf. on Artificial Intelligence (AAAI), pages 8001–8008, 2019.
  * [6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proc. of the European Conf. on Computer Vision (ECCV), 2018.
  * [7] Yuhua Chen, Cordelia Schmid, and Cristian Sminchisescu.  Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 7063–7072, 2019.
  * [8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [9] Dense depth for autonomous driving (DDAD) challenge.  2021\.  https://eval.ai/web/challenges/challenge-page/902/overview.
  * [10] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Advances in Neural Information Processing Systems (NeurIPS), pages 2366–2374, 2014.
  * [11] Matheus Gadelha, Rui Wang, and Subhransu Maji.  Shape reconstruction using differentiable projections and deep priors.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [12] Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid.  Unsupervised CNN for single view depth estimation: Geometry to the rescue.  In Proc. of the European Conf. on Computer Vision (ECCV), pages 740–756, 2016.
  * [13] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The KITTI dataset.  International Journal of Robotics Research (IJRR), 2013.
  * [14] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [15] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 270–279, 2017.
  * [16] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [17] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon.  3D packing for self-supervised monocular depth estimation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [18] Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, and Adrien Gaidon.  Semantically-guided representation learning for self-supervised monocular depth.  In Proc. of the International Conf. on Learning Representations (ICLR), April 2020.
  * [19] Christian Häne, Nikolay Savinov, and Marc Pollefeys.  Class specific 3d object shape priors using surface normals.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [21] Heiko Hirschmuller.  Stereo processing by semiglobal matching and mutual information.  IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 30(2):328–341, 2007.
  * [22] D. Hoiem, A. A. Efros, and M. Hebert.  Geometric context from a single image.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2005.
  * [23] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In Advances in Neural Information Processing Systems (NeurIPS), pages 2017–2025, 2015.
  * [24] Jianbo Jiao, Ying Cao, Yibing Song, and Rynson Lau.  Look deeper into depth: Monocular depth estimation with semantic booster and attention-driven loss.  In Proc. of the European Conf. on Computer Vision (ECCV), pages 53–69, 2018.
  * [25] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv.org, 1412.6980, 2014.
  * [26] Lubor Ladicky, Jianbo Shi, and Marc Pollefeys.  Pulling things out of perspective.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [27] Hanhan Li, Ariel Gordon, Hang Zhao, Vincent Casser, and Anelia Angelova.  Unsupervised monocular depth learning in dynamic scenes.  arXiv.org, 2010.16404, 2020.
  * [28] Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille.  Every pixel counts++: Joint learning of geometry and motion with 3d holistic understanding.  IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 42(10):2624–2641, 2019.
  * [29] Reza Mahjourian, Martin Wicke, and Anelia Angelova.  Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 5667–5675, 2018.
  * [30] Eslam Mohamed, Mahmoud Ewaisha, Mennatullah Siam, Hazem Rashed, Senthil Yogamani, and Ahmad El-Sallab.  Instancemotseg: Real-time instance motion segmentation for autonomous driving.  arXiv preprint arXiv:2008.07008, 2020.
  * [31] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black.  Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 12240–12249, 2019.
  * [32] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234–241, 2015.
  * [33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  ImageNet large scale visual recognition challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [34] Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey E. Hinton, and David J. Fleet.  Unsupervised part representation by flow capsules.  arXiv.org, 2011.13920, 2021.
  * [35] Nikolay Savinov, Christian Haene, Lubor Ladicky, and Marc Pollefeys.  Semantic 3D reconstruction with continuous regularization and ray potentials using a visibility consistency constraint.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [36] Ashutosh Saxena, Jamie Schulte, and Andrew Ng.  Depth estimation using monocular and stereo cues.  In Proc. of the International Joint Conf. on Artificial Intelligence (IJCAI), pages 2197–2203, 2007.
  * [37] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant CNNs.  In Proc. of the International Conf. on 3D Vision (3DV), 2017.
  * [38] Johan Vertens, Abhinav Valada, and Wolfram Burgard.  Smsnet: Semantic motion segmentation using deep convolutional neural networks.  In Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS), Vancouver, Canada, 2017.
  * [39] Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey.  Learning depth from monocular videos using direct methods.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 2022–2030, 2018.
  * [40] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Trans. on Image Processing (TIP), 13(4):600–612, 2004.
  * [41] Zhichao Yin and Jianping Shi.  GeoNet: Unsupervised learning of dense depth, optical flow and camera pose.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1983–1992, 2018.
  * [42] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid.  Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 340–349, 2018.
  * [43] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1851–1858, 2017.
  * [44] Yuliang Zou, Zelun Luo, and Jia-Bin Huang.  DF-Net: Unsupervised joint learning of depth and flow using cross-task consistency.  In Proc. of the European Conf. on Computer Vision (ECCV), pages 36–53, 2018.

Supplementary Material for
Self-Supervised Monocular Scene Decomposition and Depth Estimation

Sadra Safadoust Fatma Güney
---
KUIS AI Center, Koç University
<EMAIL_ADDRESS><EMAIL_ADDRESS>

In this Supplementary document, we first provide the details of the
architecture in Section A. We then provide additional results to the ones
presented in the main paper, both quantitatively (Section B) and qualitatively
(Section C). First, we compare the architecture with two separate encoders for
pose and mask to our shared encoder. Then, we provide experiments by changing
the resolution of input images, with both higher and lower resolutions. At the
end of quantitative results, we include full versions of Table 4 and Table 6
of the main paper which include more results and metrics. Finally in the
qualitative results Section C, we show a visual comparison of different number
of components and provide more qualitative examples from the best model on all
three datasets. Additionally, we provide a video demonstrating the performance
of our model on the three datasets. In the video, similar to Fig. 1 in the
main paper, we show raw image (top-left), Monodepth2’s depth estimation [16]
(bottom-left), our depth estimation (bottom-right), and our scene
decomposition (top-right). To access the video, please visit https://kuis-
ai.github.io/monodepthseg/

##  A Architecture Details

Pose Decoder
---
Layer | k | s | p | Channels | Resolution | Act.
conv0 | 1 | 1 | 0 | 256 | 1/16 | ReLU
conv1 | 3 | 1 | 1 | 256 | 1/16 | ReLU
conv2 | 3 | 1 | 1 | 256 | 1/16 | ReLU
conv3 | 1 | 1 | 0 | $K\times 6$ | 1/16 | -
Table 7: Architecture of the Pose Decoder. The hyper-parameter k is the kernel
size, s the stride, p the padding, Channels the number of output channels,
Resolution is the image size relative to the input image, and Act. is the
activation function used at each layer.

In this section, we describe the architecture of our best performing model.
Our depth network is similar to that of Monodepth2’s depth network [16] but we
use a ResNet50 encoder instead of ResNet18. For the shared encoder of the mask
and pose network, we modify the Deeplabv3+’s encoder to accept two frames as
well as the depth output as input. Therefore, its first layer is a
convolutional layer that takes 7 channels as input. The Deeplabv3+ [6] is
originally pre-trained for semantic segmentation (except for the DDAD [17]
experiment), its last classification layer outputs class probabilities over a
fixed number of classes but we need $K$ numbers for our mask prediction.
Therefore, we modify the last convolutional layer to output $K$ channels and
we initialize its weights randomly. In Table 7, we describe the architecture
of our pose decoder which is similar to the pose decoder used in Monodepth2
except that the number of outputs at the last layer is $K\times 6$ for
predicting $K$ 6-DoF poses for each component.

##  B Additional Quantitative Results

In this section, we provide additional quantitative results on KITTI [13, 14]
in addition to the ones presented in the main paper by comparing the proposed
shared encoder between the pose and mask to separate encoders for each, by
changing the image resolution in the paper to both lower and higher
resolutions, and finally expanding the Cityscapes [8] and DDAD [17] results,
and the results on the moving regions into more detailed tables.

Shared versus Separate Encoders:

Pose & Mask | Ground Truth | Lower Better | Higher Better
---|---|---|---
Encoder | Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
Separate | Original | 0.112 | 0.747 | 4.855 | 0.194 | 0.870 | 0.957 | 0.981
Shared | Original | 0.110 | 0.792 | 4.700 | 0.189 | 0.881 | 0.960 | 0.982
Separate | Improved | 0.089 | 0.481 | 4.100 | 0.140 | 0.906 | 0.980 | 0.995
Shared | Improved | 0.085 | 0.458 | 3.779 | 0.131 | 0.919 | 0.985 | 0.996
Table 8: Comparison of Shared Encoder and Separate Encoders for Pose and Mask
Network on KITTI. Using separate encoders for pose and mask increases the
number of parameters of the model and also reduces the performance according
to both the original and improved ground truth on KITTI. This validates our
design choice to use a shared encoder for the pose and mask.

In Table 8, we evaluate the effect of separating the encoders for pose and
mask in comparison to a shared encoder as we proposed in the main paper. As
can be seen from the results, our shared encoder not only reduces the number
of parameters, leading to faster training, but also improves the performance
by relating the pose and mask predictions. The pose and mask of the components
are interleaved with each other through the shared encoder to produce
corresponding pose and mask predictions for each component via separate
decoders.

Effect of Image Resolution:

Resolution | Ground Truth | Train Time | Lower Better | Higher Better
---|---|---|---|---
(hours) | Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
$416\times 128$ | Original | 11 | 0.119 | 0.827 | 5.053 | 0.201 | 0.856 | 0.952 | 0.979
$640\times 192$ | Original | 22 | 0.110 | 0.792 | 4.700 | 0.189 | 0.881 | 0.960 | 0.982
$1024\times 320$ | Original | 11 + 32 | 0.107 | 0.727 | 4.520 | 0.184 | 0.888 | 0.964 | 0.983
$416\times 128$ | Improved | 11 | 0.094 | 0.553 | 4.328 | 0.149 | 0.896 | 0.974 | 0.993
$640\times 192$ | Improved | 22 | 0.085 | 0.458 | 3.779 | 0.131 | 0.919 | 0.985 | 0.996
$1024\times 320$ | Improved | 11 + 32 | 0.083 | 0.405 | 3.562 | 0.125 | 0.925 | 0.987 | 0.997
Table 9: Effect Of Image Resolution on KITTI. This table compares the
performances and training times for our models using different image
resolutions. The timing for the high resolution model comprises 10 epochs of
of training of the $640\times 192$ model and 10 epochs of training at
$1024\times 320$ resolution.

In the main paper, all of our models on KITTI were trained at $640\times 192$
resolution. In Table 9, we show additional results at higher ($1024\times
320$) and lower ($416\times 128$) resolutions on this dataset to show the
effect of image size on the performance. The low and normal resolution models
were trained for 20 epochs. The high resolution model was initialized using
the weights from the normal resolution model after 10 epochs of training and
was trained for another 10 epochs using a learning rate of $10^{-5}$. We can
see that increasing the resolution improves the performance of the model, but
at the expense of longer training time.

Full Version of Tables: In this section, we provide the full versions of Table
4 and Table 6 of the main paper. Table 10 compares our model to Monodepth2
[16] on the moving regions of KITTI and Cityscapes datasets. Our method
outperforms Monodepth2 [16] on both datasets in all metrics, except for one
metric on the KITTI dataset. In Table 11, we provide the table with all the
metrics to compare our model to Monodepth2 [16] on Cityscapes in all regions.
Our method performs better than Monodepth2 [16] in all metrics. Note that the
full results on KITTI in all regions is already provided in Table 5 of the
main paper.

Table 12 shows the full quantitative results on the DDAD dataset, as reported
by the online evaluation server [9] after the challenge ended on June 19,
2021. In this table, we only include the results for the published methods.
Our method achieves state-of-the-art results in all metrics.

Dataset | Method | Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
---|---|---|---|---|---|---|---|---
KITTI | Godard et al. [16] | 0.143 | 1.826 | 5.949 | 0.229 | 0.823 | 0.913 | 0.951
Ours | 0.138 | 1.709 | 5.796 | 0.223 | 0.837 | 0.912 | 0.953
Cityscapes | Godard et al. [16] | 0.158 | 3.148 | 8.043 | 0.241 | 0.820 | 0.940 | 0.972
Ours | 0.143 | 2.018 | 7.649 | 0.225 | 0.824 | 0.944 | 0.977
Table 10: Full Quantitative Results in Moving Regions on KITTI and Cityscapes. This table is the expanded version of Table 4 in the main paper for the moving regions, comparing our approach to Monodepth2 [16] for moving regions on KITTI and Cityscapes datasets for all metrics. Our method achieves better results than Monodepth2 [16] on both datasets in every metric except for one metric on KITTI dataset. Method | Abs Rel | Sq Rel | RMSE | RMSElog | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$
---|---|---|---|---|---|---|---
Godard et al. [16] | 0.170 | 3.936 | 8.155 | 0.243 | 0.822 | 0.940 | 0.971
Ours | 0.142 | 1.942 | 7.361 | 0.218 | 0.827 | 0.946 | 0.978
Table 11: Full Quantitative Results Overall on Cityscapes. This table is the expanded version of Table 4 in the main paper for Cityscapes overall, comparing our approach to Monodepth2 [16] on Cityscapes in overall for all metrics. Our method achieves better results than Monodepth2 [16] in every metric. Method | Abs Rel | RMSE | SILog | $\delta<1.25$ | Car | Person | Semantic Classes (Average)
---|---|---|---|---|---|---|---
Guizilini et al. [17] | 0.23 | 17.92 | 32.56 | 0.72 | 0.38 | 0.20 | 0.24
Godard et al. [16] | 0.22 | 17.63 | 31.93 | 0.70 | 0.25 | 0.21 | 0.24
Ours | 0.19 | 16.61 | 29.37 | 0.75 | 0.24 | 0.17 | 0.22
Table 12: Full Quantitative Results on DDAD. This table is the expanded
version of Table 6 in the main paper. The _Car_ and _Person_ are shown using
_Abs Rel_. The last column is the average of _Abs Rel_ errors over all of the
semantic classes. For every metric, lower is better, except for $\delta<1.25$,
where higher is better. Our method achieves the state-of-the-art results in
every metric, without being initialized by a pre-trained segmentation network.

##  C Additional Qualitative Results

Figure 4: Effect of $K$ on Masks. Comparing mask visualizations for different
maximum number of components ($K$). Each row shows raw image (first column),
and mask visualizations for $K=2,3,5,10$ in the next columns. Small values for
$K$ cannot separate the moving objects, while larger values of $K$ can result
in a single object being segmented into multiple components.

Figure 5: Effect of $K$ on Depth. Comparing depth visualizations for different
K. Each row shows raw image (first column), and depth visualizations for
$K=2,3,5,10$ in the next columns.

Figure 6: Additional Qualitative Results on KITTI. Each row shows from top-to-
bottom: The input image (the first column), Monodepth2 results as the baseline
(the second column), our results with decomposition (the last two columns). In
the last column, each channel of the mask is assigned a random color and the
brightness level of the color is adjusted according to our estimation. In the
last row it is clear that our model performs better for pedestrians especially
for the person at the right.

Figure 7: Additional Qualitative Results on Cityscapes. Each row shows from
top-to-bottom: The input image (the first column), Monodepth2 results as the
baseline (the second column), our results with decomposition (the last two
columns). In the last column, each channel of the mask is assigned a random
color and the brightness level of the color is adjusted according to our
estimation.

Figure 8: Additional Qualitative Results on DDAD. Each row shows from top-to-
bottom: The input image (the first column), Monodepth2 results as the baseline
(the second column), our results with decomposition (the last two columns). In
the last column, each channel of the mask is assigned a random color and the
brightness level of the color is adjusted according to our estimation. In the
last image, the car on the left is moving while the car on the right is not.

In Figures 6, 7, and 8, we provide additional qualitative results of our best
model on KITTI, Cityscapes and DDAD datasets, respectively. We can see that
our model performs better especially in moving foreground regions, and at the
same time is able to decompose the scene into separately moving components.

Number of Components: To better understand the effect of maximum number of
components $K$ on the performance of the model, we include qualitative results
comparing the performances of our models using $K=2,3,5,10$. In Fig. 4, we
compare the scene decomposition results for the same input for different
values of $K$. We can see that for $K=2$, the model is unable to separate the
moving objects. The scene is divided as the nearby road region and the rest
which is further away. With $K=3$ components, moving objects are better
separated but the scene decomposition is not very good because the third
component is usually allocated to regions which are impossible to match such
as the sky. The setting we used in our experiments which is $K=5$ produces the
best results by separating the nearby road region, moving objects, and parts
of the scene which are further away. As can be seen from the examples, this
also provides flexibility when there is more than one moving object. With
$K=10$, a single object might be decomposed into multiple components, leading
to slightly worse performance than $K=5$. Fig. 5 shows the depth estimations
for the same set of images for different values of $K$.


# Gravitational Wave From Axion-like Particle Inflation

Wei Cheng  School of Science, Chongqing University of Posts and
Telecommunications, Chongqing 400065, P. R. China  State Key Laboratory of
Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of
Sciences, Beijing, 100190, P.R. China  Tao Qian  Department of Physics,
Nanjing University, 22 Hankou Road, Nanjing 210093, P. R. China  Qing Yu
Department of Physics, Chongqing University, Chongqing 401331, People’s
Republic of China  Hua Zhou  Department of Physics, Chongqing University,
Chongqing 401331, People’s Republic of China  Rui-Yu Zhou111Corresponding
author<EMAIL_ADDRESS>School of Science, Chongqing University of Posts
and Telecommunications, Chongqing 400065, P. R. China  Department of Physics,
Chongqing University, Chongqing 401331, People’s Republic of China

###### Abstract

In this paper, we investigate the Axion-like Particle inflation by applying
the multi-nature inflation model, where the end of inflation is achieved
through the phase transition (PT). The events of PT should not be less than
$200$, which results in the free parameter $n\geq 404$. Under the latest CMB
restrictions, we found that the inflation energy is fixed at
$10^{15}\rm{GeV}$. Then, we deeply discussed the corresponding stochastic
background of the primordial gravitational wave (GW) during inflation. We
study the two kinds of $n$ cases, i.e., $n=404,2000$. We observe that the
magnitude of $n$ is negligible for the physical observations, such as $n_{s}$,
$r$, $\Lambda$, and $\Omega_{\rm{GW}}h^{2}$. In the low-frequency regions, the
GW is dominated by the quantum fluctuations, and this GW can be detected by
Decigo at $10^{-1}~{}\rm{Hz}$. However, GW generated by PT dominates the high-
frequency regions, which is expected to be detected by future 3DSR detector.

##  I Introduction

With the first discovery of gravitational wave (GW) from a binary neutron star
(BNS) system by the Laser Interferometer Gravitational Wave Observatory (LIGO)
TheLIGOScientific:2014jea  and by the Virgo detector TheVirgo:2014hva , the
primary GW from inflation is increasingly the focus of theorists and
experimentalists. Inflationary scenario not only explain the problem in the
hot big bang universe gracefully, i.e., the flatness puzzles, the horizon
puzzles, and the monopole puzzles, (refer to Guth:1980zm ; Albrecht:1982wi ;
Linde:1981mu ; revlinde ; liddle ; riotto ; mukhanov ; nojirio ; nojirioo ),
but also predict the nearly scale-invariance which is strongly supported by
the anisotropies of cosmic microwave background (CMB) radiation Akrami:2018odb
.

Typically, in the inflationary models with broken slow-roll conditions
Lerner:2009xg ; Zheng:2014dra ; Munoz:2014eqa ; Aravind:2015xst ;
Cheng:2018ajh ; Cheng:2018axr , the primary GW only comes from the quantum
fluctuations (QF) during inflation Matarrese:1997ay ; Bartolo:2007vp ;
Biagetti:2013kwa ; Biagetti:2014asa , and its intensity is related to the
tensor-to-scalar ratio $r$ of inflation Liu:2015psa ; Boyle:2005se . However,
the $r$ predicted by the theory, in this case, is so small that a weak GW will
be generated Starobinsky:1980te , which brings great challenges for
experimental detection Wang:2018caj .

Alternatively, in the literatures Linde:1990gz ; Adams:1990ds ;
Ashoorioon:2015hya , a new way to end inflation has been proposed, i.e., the
inflationary end of the bubble nucleation method. This method also has the
natural function of reheating the universe through the collision of true
vacuum bubble walls and convert it to radiation. Here are two scalar fields in
this inflationary models, one of the fields is responsible for driving the
inflation of the universe, whereas the other field can indeed percolate the
true vacuum from the meta-stable one, and complete the phase transition (PT)
and end inflation simultaneously. In this case, the source of GW includes
quantum fluctuations during inflation as well as the PT that will produce
considerable GW Caprini:2015zlo ; Binetruy:2012ze ; Ashoorioon:2008nh . This
makes it possible to detect GW during inflation.

In this work, a singlet Axion-like Particle (ALP) is applied to drive cosmic
inflation and to terminate the inflation through the PT simultaneously, and we
then investigate in detail the GW produced by the ALP inflation. The multi-
natural inflation (MNI) Czerny:2014wza ; Takahashi:2019qmh  based on natural
inflation (NI) Adams:1992bn  can increase the feasible parameter space of the
NI that is shrink by the latest CMB data Akrami:2018odb . The inflation of MNI
has a good property, shift symmetry, allows an ALP to act as the inflaton
Daido:2017wwb ; Takahashi:2019qmh . Besides, it also keeps the flatness of
potential energy, which is essential for driving the expansion of the universe
and generating density perturbations. Moreover, MNI can provide multiple false
vacuums due to the multiple cosine functions in the potential of MNI. With the
slow-roll inflation proceeds, the inflation first falls into the false vacuum,
the vacuum bubbles then collide with each other, and the inflation finally
transits into the true vacuum. This provides the possibility for MNI of ALP to
complete cosmic inflation and to end inflation through PT simultaneously. We
calculate the GW for the period of ALP inflation that will consist of two
parts, namely, the QF GW and the PT GW.

The remaining parts of the paper are organized as follows. In Sec. II, we
present the Multi-NI model and study the inflation exit from the PT. In Sec.
III, the GW from both QF and PT are in detail discussed. Sec. IV is a summary
for this work.

##  II Axion-like particle inflation

Figure 1: Schematic diagram of ALP inflation and PT with $n=20$.

The minimal axion-like particle inflation potential derived from a multi-
natural inflation, is defined as follows Czerny:2014wza ; Takahashi:2019qmh :

$\displaystyle
V(\phi)=\Lambda^{4}[\cos(\frac{\phi}{f_{\phi}}+\theta)-\frac{\kappa}{n^{2}}\cos(n\frac{\phi}{f_{\phi}})]+C$
(1)

where $\Lambda$ is related to the inflation scale, $f_{\phi}$ represents the
decay constant. We set the a relative phase $\theta=\pi/2$ for creating false
vacua between the maximum and minimum values of adjacent potential, and take
the coefficient $\kappa=n^{2}$ for equalizing the energy difference between
two adjacent vacua. For the sake of discussion, we require the minimum of the
potential at $\phi=\pi/2$, thus the rational number $n$ is a multiple of $4$.
And $n$ shall determine the tunneling events of the PT which are more than
$200$ with a GUT scale inflation Ashoorioon:2008nh . Thus the rational number
$n$ should be at least $404$. $C=2\Lambda^{4}$ is a constant that shifts the
minimum of the potential to zero.

For the convenience of presentation, we take a small value for $n=20$ and show
the ALP inflation potential in Fig. 1, in which the dashed cyan with arrows
represents the direction of inflation and phase transitions. The inflaton
first rolls to the first false vacuum point B from the highest point A of
potential then reaches the lower vacuum point through a phase transition, and
finally reaches the true vacuum point C.

During the slow-roll inflation, the kinetic energy of inflaton is much less
than that of the potential energy, and the change rate of kinetic energy
should much less than that of the cosmic expansion, which can be characterized
by the so called slow-roll conditions:

$\varepsilon(\phi)\equiv\frac{{M_{pl}}^{2}}{2}\left(\frac{V^{\prime}(\phi)}{V(\phi)}\right)^{2}\ll
1,\\\
\quad\eta(\phi)\equiv{M_{pl}}^{2}\left(\frac{V^{\prime\prime}(\phi)}{V(\phi)}\right)\ll
1,$ (2)

where the prime denotes the derivative with respect to $\phi$.

Generally, the inflation will be finished with any one of the conditions
breaks, i.e. $\varepsilon=1$ or $\mid\eta\mid=1$. However, we need the
inflation exited with the bubble nucleation, which means that the above
conditions are still held at the beginning of PT, i.e.,
$\varepsilon(\phi_{B})\ll 1$ and $\eta(\phi_{B})\ll 1$ with $\phi_{B}$ being
the field value of the first vacuum, which will bring a lower limit for the
decay constant $f_{\phi}$ with a fix $n$. As an example, we will show two
cases as follows:

$\displaystyle f\gg$ $\displaystyle\sqrt{4.39742\times
10^{31}},~{}~{}~{}\rm{for}~{}~{}n=404;$ (3) $\displaystyle f\gg$
$\displaystyle\sqrt{7.10613\times 10^{30}},~{}~{}~{}\rm{for}~{}~{}n=2000.$ (4)

There are two observable quantities, i.e., scalar spectral index
$n_{s}=1-6\varepsilon+2\eta$ and the tensor-to-scalar ratio $r=16\varepsilon$,
which are tightly constrained by the Planck data combined with CMB
observations as Akrami:2018odb :

$\displaystyle n_{s}$ $\displaystyle=0.9649\pm 0.0042,$ (5) $\displaystyle r$
$\displaystyle<0.10~{}~{}~{}(95\%{\rm CL}).$ (6)

For calculating them at the moment of horizon-crossing, we need to know the
value of the field where inflation starts, $\phi_{*}$, which are usually
determined by applying for the e-folding number. Numerical calculation shows
that the contributions of the e-folding number from the highest potential
energy point $\phi_{A}$ to the first vacuum point $\phi_{B}$ are negligible,
thus we’re approximating $\phi_{*}$ around the highest potential energy point
$\phi_{A}$.

There is another important experimental observation, i.e. amplitude of scalar
fluctuations $\Delta_{\mathcal{R}}^{2}$,

$\Delta_{\mathcal{R}}^{2}=\frac{1}{24\pi^{2}M_{\rm
pl}^{4}}\frac{V(\phi_{A})}{\varepsilon(\phi_{A})}.$ (7)

where $\Delta_{\mathcal{R}}^{2}\simeq e^{3.098}\times 10^{-10}\simeq 2.2\times
10^{-9}$ Ade:2015lrj , which will give out the inflation scale $\Lambda$.

Given the above constraints, we fix $n=404,2000$ for the rest of analysis. The
scalar spectral index $n_{s}$ and the tensor-to-scalar ratio $r$ are shown in
Fig. 2. The green and cyan lines (dashed) represent that $\phi_{*}$ are
$602.9\frac{\pi f_{\phi}}{404}$ and $602.8\frac{\pi f_{\phi}}{404}$
($2998.9\frac{\pi f_{\phi}}{2000}$ and $2998.8\frac{\pi f_{\phi}}{2000}$),
respectively. Both $r$ and $n_{s}$ are almost independent of $n$, and the
larger $\phi_{*}$, the smaller $r$. To see the specific values more clearly,
the upper and lower limits of the corresponding other parameters are listed in
Table. 1.

Table 1: Two sets of benchmark points with $n=404,2000$ for $f_{\phi}~{}(\rm{GeV})$ and $10^{-15}\Lambda~{}(\rm{GeV})$, in which the scalar spectral index $n_{s}=0.9649\pm 0.0042$ and $r$ are also presented. | $n=404$ | $n=404$ | $n=2000$ | $n=2000$
---|---|---|---|---
$\phi_{*}$ | $603.9\frac{\pi f_{\phi}}{404}$ | $603.8\frac{\pi f_{\phi}}{404}$ | $2998.9\frac{\pi f_{\phi}}{2000}$ | $2998.8\frac{\pi f_{\phi}}{2000}$
$f_{\phi}$ | $[10^{21.5388},10^{21.5910}]$ | $[10^{21.5373},10^{21.5895}]$ | $[10^{22.2335},10^{22.5889}]$ | $[10^{22.2319},10^{22.2841}]$
$10^{-15}\Lambda$ | $[5.62998,5.30161]$ | $[7.99456,7.52826]$ | $[5.62942,5.30107]$ | $[7.99479,7.52848]$
$r$ | $[0.00385,0.00303]$ | $[0.01508,0.01186]$ | $[0.00385,0.00302]$ | $[0.01509,0.01186]$

Figure 2: ALP inflation predictions in $(n_{s},r)$ planes with $n=404$ and
$n=2000$. Gray, red and blue shadows stand for [TT,TE,EE+lowE+lensing],
[TT,TE,EE+lowE+lensing+BK15] and [TT,TE,EE+lowE+lensing+BK15+BAO] constraints
Akrami:2018odb .

##  III Gravitational waves from inflation

###  III.1 GW from quantum fluctuations

During the ALP-inflation, the quantum fluctuations of inflaton will lead to
the perturbations in energy and momentum, which will transmit to the metric in
terms of Einstein’s equation, in which the tensor perturbations of the metric
will produce the so-called QF GW. Generally, the initial tensor power spectrum
(spectrum at $\tau=0$) $P_{h}(k)$ can be parameterized as a power-law form
Peiris:2003ff , i.e.,

$P_{h}(k)=rk^{2}A_{R}(k_{0})\bigg{(}\frac{k}{k_{0}}\bigg{)}^{n_{t}},$ (8)

where $n_{t}$ is the power index of tensorial spectrum and the value of scalar
power spectrum $A_{R}(k_{0})=2.371\times 10^{-9}$ with
$k_{0}=0.002~{}\rm{Mpc}^{-1}$ Ade:2015xua ; Ade:2015lrj .

Furthermore, the present spectrum of energy density in terms of $P_{h}(k)$ can
be written as Liu:2015psa :

$\Omega_{\rm{GW},\rm{QF}}(k)=\frac{P_{h}(k)}{12H^{2}(\tau_{0})}\times
T^{2}(k,\tau_{0})$ (9)

where the current Hubble parameter $H(\tau_{0})$ is taken as
$67.8\rm{km~{}s^{-1}~{}Mpc^{-1}}$, the current conformal time
$\tau_{0}=1.41\times 10^{4}~{}\rm{Mpc}$, and the last part $T(k,\tau_{0})$ is
the transfer function as followsTurner:1993vb ; Zhang:2005nw ;
Giovannini:2009kg ; Chongchitnan:2006pe ; Kuroyanagi:2009br :

$T(k,\tau_{0})=\frac{3\Omega_{m}j_{1}(k\tau_{0})}{k\tau_{0}}\sqrt{1.0+1.36\frac{k}{k_{eq}}+2.50\bigg{(}\frac{k}{k_{eq}}\bigg{)}^{2}}$
(10)

where matter density $\Omega_{m}=0.308$, the spherical Bessel function of the
first kind $j_{1}(k\tau_{0})\sim\frac{1}{\sqrt{2}k\tau_{0}}$, and
$k_{eq}=0.073\Omega_{m}h^{2}~{}\rm{Mpc}^{-1}$ is the wavenumber corresponding
to the mode that entered the horizon at the equality of matter and radiation.

Figure 3: The spectrum of GW from the quantum fluctuations during the ALP
inflation.

Applying the input parameters in Table. 1 and Formula 9, we show the GW from
the QF during the ALP inflation in Fig. 3, in which the left and right parts
stand for the QF GW with $n=404$ and $n=2000$, respectively. The QF GW first
decreases with the increase of frequency $f$, and then remains almost constant
from frequency $f\sim 10^{-16}~{}\rm{Hz}$. Those two lines with
$\phi_{*}=604.9\pi f/n$ and $\phi_{*}=2998.9\pi f/n$ ($\phi_{*}=604.8\pi f/n$
and $\phi_{*}=2998.8\pi f/n$) almost overlap each other due to the same value
of tensor-to-scalar ratio $r$ that are shown in Table. 1, which also implies
the QF GW are not sensitive to the parameter $n$. When we fix the parameter
$n$, the QF GW will increase with the of decrease $\phi_{*}$.

###  III.2 GW from phase transitions

As we emphasized earlier, in our model, ALP inflation exited through bubble
nucleations rather than the breach of slow-roll conditions. Numerical
calculation of two bubble collisions and more bubble collisions can be
referred to Ref. Kosowsky:1992vn  and Ref. Huber:2007vva , respectively. While
the bubble nucleations will lead to PTs that further produce PT GW. The GW for
a single PT can be expressed as Kosowsky:1992vn :

$\Omega_{m}h^{2}(f)=\left\\{\begin{array}[]{ll}\Omega
h^{2}(f_{m}){\left(\frac{f}{f_{m}}\right)}^{2.8}\quad~{}~{}(f\leq f_{m}),&\\\
\Omega h^{2}(f_{m}){\left(\frac{f}{f_{m}}\right)}^{-1.8}\quad(f\geq
f_{m}),&\end{array}\right.$ (11)

where the peak frequency today $f_{m}=\frac{3\times
10^{-8}}{\chi}e^{-((n-2)/2-m-1)\chi}{\left(\frac{g_{\ast}}{100}\right)}^{1/6}\left(\frac{T_{e}}{1~{}{\rm
GeV}}\right)$ with the temperature of the universe at the end of ALP inflation
being
$T_{e}=\frac{1}{1-e^{-\chi}}{\left(\frac{30\delta\epsilon}{g_{\ast}\pi^{2}}\right)}^{1/4}$.
The total number of relativistic degrees of freedom is taken to be
$g_{\ast}\simeq 100$. And the amplitude $\Omega h^{2}(f_{m})$ at this peak
frequency have the following form:

$\displaystyle\Omega
h^{2}(f_{m})=10^{-6}(1-e^{-\chi})^{4}\chi^{2}e^{-4((n-2)/2-m-1)\chi}{\left(\frac{100}{g_{\ast}}\right)}^{1/3}.$
(12)

Thus, the total phase transition energy density of the gravitational waves is
given by a sum of the multiple tunneling events,

$\Omega_{\rm{GW},\rm{PT}}h^{2}(f)=\sum_{m=1}^{(n-2)/2-1}\Omega_{m}h^{2}(f).$
(13)

We show the PT GW in Fig. 4, in which the left and right parts stand for the
PT GW with $n=404$ and $n=2000$, respectively. The PT GW have almost identical
peaks and shapes for different $n$, which also implies the PT GW is
insensitive to $n$, but the energy spectrum $\Omega_{GW,PT}h^{2}$ sharp is
shifted about five times to the left as $\phi_{*}$ increases. We can see that
the GW produced by PT during inflation is high-frequency GW. Compared to the
previous QF GW, it dominates the low-frequency regions, while PT produces GW
that will dominate the high-frequency regions.

Figure 4: The spectrum of PT GW from the ALP inflation.

###  III.3 Total GW

After acquiring the PT GW and QF GW during the ALP inflation, we can now
analyze the total GW. According to the Eqs. (9) and (13), it can be expressed
as follows:

$\Omega_{\rm{GW},\rm{Tot}}h^{2}=\Omega_{\rm{GW},\rm{PT}}h^{2}+\Omega_{\rm{GW},\rm{QF}}h^{2}.$
(14)

The total GW are shown in Fig. 5, we find that at ultra-low frequency
$f\sim(10^{-19},10^{-17})~{}\rm{Hz}$, all the predictions of the GW meet the
CMB limits Lasky:2015lej ; Sepehri:2016jsu , while near the frequency
$f\sim(10^{-1})~{}\rm{Hz}$, the detector Decigo Yagi:2011wg  can search for
all of the GW scenarios that we predict. For the extremely high-frequency
$f\sim(10^{9},10^{12})~{}\rm{Hz}$, the total GW $\Omega_{\rm{Tot}}h^{2}$ of
our prediction has been reached $\sim 10^{-10}~{}\rm{Hz}$, which is expected
to be searched on the $3\rm{DSR}$ detector proposed by Chongqing University
Li:2003tv ; Tong:2008rz .

Figure 5: The spectrum of total GW from the ALP inflation. The CMB observation
make a limit in the low-frequency Lasky:2015lej ; Sepehri:2016jsu , while the
detector Decigo Yagi:2011wg  can provide a test for the GW in the high-
frequency.

##  IV Summary

In this paper, we investigate the GW generated by the MNI that exits from the
PT. This change makes this model better match the CMB observation data than
the usual slow-roll exit scenario. The GW will consist of two parts, i.e., GW
is generated by QF during inflation and by PT finished the inflation, and it
will bring a sizable GW that may be detected by the future GW detectors.

As the PT events need more than $200$, therefore, the model parameter $n$
should not less than $404$. We investigate two scenarios ($n=404,2000$), and
find that generated GW is very insensitive to $n$, which can be obtained from
all the GW figures. The GW generated by QF dominates the low-frequency part,
while the GW generated by PT dominates the high-frequency part. At $f\sim
10^{-1}\rm{Hz}$, the generated GW can be detected by Decigo Yagi:2011wg ,
while in the extremely high-frequency part, it is expected to be detected by
$3\rm{DSR}$ designed by Chongqing University.

Acknowledgement:

This work has been supported by the National Science Foundation of China under
Grant No.12047564, the Fundamental Research Funds for the Central Universities
under Grant No. 2020CDJQY-Z003, and the China Postdoctoral Science Foundation
under Grant No. 2019TQ0329 and No. 2020M670476.

## References

  * (1) J. Aasi et al. [LIGO Scientific], “Advanced LIGO,” Class. Quant. Grav. 32, 074001 (2015)
  * (2) F. Acernese et al. [VIRGO], “Advanced Virgo: a second-generation interferometric gravitational wave detector,” Class. Quant. Grav. 32, 024001 (2015)
  * (3) A. H. Guth, “The Inflationary Universe: A Possible Solution to the Horizon and Flatness Problems,” Phys. Rev. D 23, 347-356 (1981)
  * (4) A. Albrecht and P. J. Steinhardt, “Cosmology for Grand Unified Theories with Radiatively Induced Symmetry Breaking,” Phys. Rev. Lett. 48, 1220-1223 (1982)
  * (5) A. D. Linde, “A New Inflationary Universe Scenario: A Possible Solution of the Horizon, Flatness, Homogeneity, Isotropy and Primordial Monopole Problems,” Phys. Lett. B 108, 389-393 (1982)
  * (6) A. D. Linde, Particle physics and inflationary cosmology, (Harwood, Chur, Switzerland, 1990) Contemp. Concepts Phys. 5, 1 (1990) [hep-th/0503203].
  * (7) A. R. Liddle and D. H. Lyth, “The Cold dark matter density perturbation,” Phys. Rept. 231, 1-105 (1993)
  * (8) A. Riotto, “Inflation and the theory of cosmological perturbations,” ICTP Lect. Notes Ser. 14, 317-413 (2003)
  * (9) D. H. Lyth and A. Riotto, “Particle physics models of inflation and the cosmological density perturbation,” Phys. Rept. 314, 1-146 (1999)
  * (10) V. Mukhanov, Physical Foundations of Cosmology (Cambridge University Press, 2005).
  * (11) S. Nojiri and S. D. Odintsov, “Unified cosmic history in modified gravity: from F(R) theory to Lorentz non-invariant models,” Phys. Rept. 505, 59-144 (2011)
  * (12) S. Nojiri, S. D. Odintsov and V. K. Oikonomou, “Modified Gravity Theories on a Nutshell: Inflation, Bounce and Late-time Evolution,” Phys. Rept. 692, 1-104 (2017)
  * (13) Y. Akrami et al. [Planck], “Planck 2018 results. X. Constraints on inflation,” Astron. Astrophys. 641, A10 (2020)
  * (14) R. N. Lerner and J. McDonald, “Gauge singlet scalar as inflaton and thermal relic dark matter,” Phys. Rev. D 80, 123507 (2009)
  * (15) S. Zheng, “Dynamical Generation of the Weak Scale and Inflation in High-Scale Supersymmetry,” arXiv:1405.2775 [hep-ph].
  * (16) J. B. Munoz and M. Kamionkowski, “Equation-of-State Parameter for Reheating,” Phys. Rev. D 91, 043521 (2015)
  * (17) A. Aravind, M. Xiao and J. H. Yu, “Higgs Portal to Inflation and Fermionic Dark Matter,” Phys. Rev. D 93, 123513 (2016) [erratum: Phys. Rev. D 96, 069901 (2017)]
  * (18) W. Cheng and L. Bian, “From inflation to cosmological electroweak phase transition with a complex scalar singlet,” Phys. Rev. D 98, 023524 (2018)
  * (19) W. Cheng and L. Bian, “Higgs inflation and cosmological electroweak phase transition with N scalars in the post-Higgs era,” Phys. Rev. D 99, 035038 (2019)
  * (20) S. Matarrese, S. Mollerach and M. Bruni, “Second order perturbations of the Einstein-de Sitter universe,” Phys. Rev. D 58, 043504 (1998)
  * (21) N. Bartolo, S. Matarrese, A. Riotto and A. Vaihkonen, “The Maximal Amount of Gravitational Waves in the Curvaton Scenario,” Phys. Rev. D 76, 061302 (2007)
  * (22) M. Biagetti, M. Fasiello and A. Riotto, “Enhancing Inflationary Tensor Modes through Spectator Fields,” Phys. Rev. D 88, 103518 (2013)
  * (23) M. Biagetti, E. Dimastrogiovanni, M. Fasiello and M. Peloso, “Gravitational Waves and Scalar Perturbations from Spectator Fields,” JCAP 04, 011 (2015)
  * (24) L. A. Boyle and P. J. Steinhardt, “Probing the early universe with inflationary gravitational waves,” Phys. Rev. D 77, 063504 (2008)
  * (25) X. J. Liu, W. Zhao, Y. Zhang and Z. H. Zhu, “Detecting Relic Gravitational Waves by Pulsar Timing Arrays: Effects of Cosmic Phase Transitions and Relativistic Free-Streaming Gases,” Phys. Rev. D 93, 024031 (2016)
  * (26) A. A. Starobinsky, “A New Type of Isotropic Cosmological Models Without Singularity,” Phys. Lett. B 91, 99-102 (1980)
  * (27) Y. T. Wang, Y. Cai and Y. S. Piao, “Phase-transition sound of inflation at gravitational waves detectors,” Phys. Lett. B 789, 191-196 (2019)
  * (28) A. D. Linde, “Eternal extended inflation and graceful exit from old inflation without Jordan-Brans-Dicke,” Phys. Lett. B 249, 18-26 (1990)
  * (29) F. C. Adams and K. Freese, “Double field inflation,” Phys. Rev. D 43, 353-361 (1991)
  * (30) A. Ashoorioon, “Exit from Inflation with a First-Order Phase Transition and a Gravitational Wave Blast,” Phys. Lett. B 747, 446-453 (2015)
  * (31) C. Caprini, M. Hindmarsh, S. Huber, T. Konstandin, J. Kozaczuk, G. Nardini, J. M. No, A. Petiteau, P. Schwaller and G. Servant, et al. “Science with the space-based interferometer eLISA. II: Gravitational waves from cosmological phase transitions,” JCAP 04, 001 (2016)
  * (32) P. Binetruy, A. Bohe, C. Caprini and J. F. Dufaux, “Cosmological Backgrounds of Gravitational Waves and eLISA/NGO: Phase Transitions, Cosmic Strings and Other Sources,” JCAP 06, 027 (2012)
  * (33) A. Ashoorioon and K. Freese, “Gravity Waves from Chain Inflation,”
  * (34) M. Czerny and F. Takahashi, “Multi-Natural Inflation,” Phys. Lett. B 733, 241-246 (2014)
  * (35) F. Takahashi and W. Yin, “ALP inflation and Big Bang on Earth,” JHEP 07, 095 (2019)
  * (36) F. C. Adams, J. Bond, K. Freese, J. A. Frieman and A. V. Olinto, “Natural inflation: Particle physics models, power law spectra for large scale structure, and constraints from COBE,” Phys. Rev. D 47, 426-455 (1993)
  * (37) R. Daido, F. Takahashi and W. Yin, “The ALP miracle: unified inflaton and dark matter,” JCAP 05, 044 (2017)
  * (38) F. Takahashi and W. Yin, “ALP inflation and Big Bang on Earth,” JHEP 07, 095 (2019)
  * (39) P. A. R. Ade et al. [Planck], “Planck 2015 results. XX. Constraints on inflation,” Astron. Astrophys. 594, A20 (2016)
  * (40) P. Ade et al. [Planck], “Planck 2015 results. XX. Constraints on inflation,” Astron. Astrophys. 594, A20 (2016)
  * (41) H. V. Peiris et al. [WMAP], “First year Wilkinson Microwave Anisotropy Probe (WMAP) observations: Implications for inflation,” Astrophys. J. Suppl. 148, 213-231 (2003)
  * (42) P. A. R. Ade et al. [Planck], “Planck 2015 results. XIII. Cosmological parameters,” Astron. Astrophys. 594, A13 (2016)
  * (43) M. S. Turner, M. J. White and J. E. Lidsey, “Tensor perturbations in inflationary models as a probe of cosmology,” Phys. Rev. D 48, 4613-4622 (1993)
  * (44) Y. Zhang, Y. Yuan, W. Zhao and Y. T. Chen, “Relic gravitational waves in the accelerating Universe,” Class. Quant. Grav. 22, 1383-1394 (2005)
  * (45) S. Chongchitnan and G. Efstathiou, “Prospects for direct detection of primordial gravitational waves,” Phys. Rev. D 73, 083511 (2006)
  * (46) M. Giovannini, “Stochastic backgrounds of relic gravitons: a theoretical appraisal,” PMC Phys. A 4, 1 (2010)
  * (47) S. Kuroyanagi, C. Gordon, J. Silk and N. Sugiyama, “Forecast Constraints on Inflation from Combined CMB and Gravitational Wave Direct Detection Experiments,” Phys. Rev. D 81, 083524 (2010) [erratum: Phys. Rev. D 82, 069901 (2010)]
  * (48) A. Kosowsky and M. S. Turner, “Gravitational radiation from colliding vacuum bubbles: envelope approximation to many bubble collisions,” Phys. Rev. D 47, 4372-4391 (1993)
  * (49) S. J. Huber and T. Konstandin, “Production of gravitational waves in the nMSSM,” JCAP 05, 017 (2008)
  * (50) P. D. Lasky, C. M. F. Mingarelli, T. L. Smith, J. T. Giblin, D. J. Reardon, R. Caldwell, M. Bailes, N. D. R. Bhat, S. Burke-Spolaor and W. Coles, et al. “Gravitational-wave cosmology across 29 decades in frequency,” Phys. Rev. X 6, 011035 (2016)
  * (51) A. Sepehri, A. Pradhan and S. Shoorvazi, “$F(R)$ bouncing cosmology with future singularity in brane-anti-brane system,” Astrophys. Space Sci. 361, 58 (2016)
  * (52) K. Yagi and N. Seto, “Detector configuration of DECIGO/BBO and identification of cosmological neutron-star binaries,” Phys. Rev. D 83, 044011 (2011)
  * (53) F. Y. Li, M. X. Tang and D. P. Shi, “Electromagnetic response of a Gaussian beam to high frequency relic gravitational waves in quintessential inflationary models,” Phys. Rev. D 67, 104008 (2003)
  * (54) M. l. Tong, Y. Zhang and F. Y. Li, “Using polarized maser to detect high-frequency relic gravitational waves,” Phys. Rev. D 78, 024041 (2008)