# PanoDepth: A Two-Stage Approach for Monocular Omnidirectional Depth
Estimation

Yuyan Li
University of Missouri
<EMAIL_ADDRESS>Zhixin Yan
BOSCH Research China
<EMAIL_ADDRESS>Ye Duan
University of Missouri
<EMAIL_ADDRESS>Liu Ren
BOSCH Research North America
<EMAIL_ADDRESS>
###### Abstract

Omnidirectional 3D information is essential for a wide range of applications
such as Virtual Reality, Autonomous Driving, Robotics, etc. In this paper, we
propose a novel, model-agnostic, two-stage pipeline for omnidirectional
monocular depth estimation. Our proposed framework PanoDepth takes one 360
image as input, produces one or more synthesized views in the first stage, and
feeds the original image and the synthesized images into the subsequent stereo
matching stage. In the second stage, we propose a differentiable Spherical
Warping Layer to handle omnidirectional stereo geometry efficiently and
effectively. By utilizing the explicit stereo-based geometric constraints in
the stereo matching stage, PanoDepth can generate dense high-quality depth. We
conducted extensive experiments and ablation studies to evaluate PanoDepth
with both the full pipeline as well as the individual modules in each stage.
Our results show that PanoDepth outperforms the state-of-the-art approaches by
a large margin for 360 monocular depth estimation.

##  1 Introduction

Omnidirectional 3D information is essential for a wide range of applications
(e.g. Virtual Reality [2], augmented reality [4], autonomous driving [1], and
robotics [46]). Quick and reliable omnidirectional data acquisition can
facilitate many use cases, such as user interaction with the digital
environment, robot navigation, and object detection for autonomous vehicles.
Another relevant application is remote working/shopping/education [19], which
has become ubiquitous due to the pandemic. To obtain high-quality
omnidirectional 3D information, devices such as omnidirectional LiDARs are
widely used in autonomous driving and indoor 3D scans. However, LiDARs are
either very expensive or can only produce sparse 3D scans. Compared with
LiDARs, cameras are much cheaper and already frequently used for capturing the
visual appearance of the scenes. The cost can be significantly reduced if
high-quality omnidirectional 3D can be generated directly from camera images.

Figure 1: (a) Illustration of our PanoDepth framework. PanoDepth takes one 360
image as input to generate one or more novel views in the first view synthesis
stage. The original and synthesized 360 images are then fed into the second
multi-view stereo matching stage to predict final dense depth map. (b) Two
examples (top and bottom row) of PanoDepth on 360D dataset [69] with 360 image
(left) as input, and output depth (middle) and point cloud (right).

Deep learning techniques, coupled with a growing accessibility of large-scale
datasets, have largely improved the performance of many computer vision tasks
including depth estimation [65]. Depth estimation often uses either a
monocular input or a stereo pair. For the monocular methods, a common practice
is to train a single network to map RGB pixel to real-value depth [18, 21, 30,
33], mostly by learning from various monocular cues such as shape, lighting,
shading, object type, etc. Stereo matching methods [49, 36, 8], on the other
hand, learn the disparity by matching image patches from stereo pairs and
later convert disparity to depth. Despite the significant improvement in
monocular estimation methods, there is still a large gap between monocular and
stereo depth accuracy [50].

To apply stereo matching for monocular depth estimation, Luo et al. [41]
proposed a two-stage pipeline that decomposes monocular depth estimation into
two stages, view synthesis and stereo matching respectively. In their
approach, a second view is first synthesized and fed together with the
original view to the stereo matching stage to compute the disparity. Stereo
matching can leverage more geometric constraints into the network training,
thus reducing the demand for ground truth depth. More recent studies [54, 59]
improved upon this idea and achieved promising performances. These two-stage
methods [41, 54, 59], are mainly designed for perspective images. In 360
domain, most of the recent studies [69, 51, 16, 57, 15] still follow the same
single-stage monocular estimation procedure with adaptation to 360 111We use
the terms 360, omnidirectional, equirectangular, spherical interchangeably in
this paper geometry.

In this paper, we propose a novel, model-agnostic, two-stage pipeline (see
Figure 1) for solving the problem of 360 monocular depth estimation. Our
proposed framework PanoDepth takes one equirectangular projection (ERP) image
as input, produces one or more synthesized views in the first stage, and feeds
the original image and the synthesized images to the subsequent stereo
matching stage to predict the final depth map. In the stereo matching stage,
we propose a novel differentiable Spherical Warping Layer to handle
omnidirectional stereo geometry efficiently and effectively. We conducted
extensive experiments and ablation studies to evaluate PanoDepth with both the
full pipeline and the individual networks in each stage on several public
benchmark datasets. Our results demonstrated that our model-agnostic approach
PanoDepth outperforms the one-stage method by a large margin despite the
combinations of coarse estimation and stereo matching networks. Moreover, by
adjusting these networks, PanoDepth can be adapted to the target computation
constraints and performance requirements.

Our contributions can be summarized as follows:

  * •

We propose a novel, model-agnostic, two-stage framework PanoDepth, including
view synthesis and stereo matching, to fully exploit the synthesized 360 views
and spherical stereo constraints.

  * •

PanoDepth outperforms the state-of-the-art monocular omnidirectional depth
estimation approaches by a large margin.

  * •

We propose a novel differentiable Spherical Warping Layer (SWL) which adapts
regular stereo matching networks to 360 stereo geometry, and enables advanced
features such as multi-view stereo and cascade mechanism for stereo
performance boost.

##  2 Motivation

In this section, we explain the motivation of formulating the 360 monocular
depth estimation problem as two separate stages, namely, a view synthesis
stage based on coarse depth estimation, and a multi-view stereo matching stage
for final depth output.

###  2.1 Why Two-Stage?

One main advantage of monocular depth estimation is its potential in
dramatically reducing the hardware cost for 3D depth acquisition. Motivated by
this, many studies have been proposed to solve this problem. The basic idea of
supervised monocular estimation methods is to train a network that directly
learns the mapping from the input RGB pixels to the real-value output depth in
a single stage. For example, Laina et al. [37] proposed FCRN which uses
ResNet-50 [28] as backbone, followed by multiple up-projection modules. Hu et
al. [30] leveraged SENet-154 [31] as encoder together with multi-scale fusion
module.

On the other hand, deep learning based stereo matching networks [8, 36, 25]
utilize the stereo constraints to improve efficiency and the output depth
quality. These methods simulate the traditional stereo matching process by
learning and optimizing the matching cost across the input image pairs in a
deterministic manner. Unlike monocular depth methods which directly map RGB
into depth by considering all the monocular cues, stereo matching methods
focus on estimating disparity by developing image patch correspondence [18].
Given the predefined baseline and 1D search space along the epipolar line for
image patch matching [27], stereo matching produces more accurate depth maps
in comparison with monocular methods in general [50].

A typical stereo matching network requires at least two images as input, which
is not directly applicable for a monocular input setting. However, if one or
more novel views can be synthesized with high quality, these additional views
can be utilized to train a stereo matching network. Luo et al. [41] first
proposed the two-stage pipeline for perspective images where a novel right
view is synthesized at the first stage and paired with the original view to
the second stereo stage. Later, two-stage approaches [54] mostly followed this
work to generate a coarse disparity/depth, and to synthesize novel views via
image warping or Depth-Image-Based Rendering (DIBR). Taking the original and
the synthesized views as input, the final depth generated from the later
stereo matching stage of these approaches shows a significant improvement over
the one-stage counterparts.

###  2.2 Can we successfully synthesize novel view 360 images?

Recent two-stage approaches [41, 54] have shown promising capabilities in
improving depth quality on perspective images. However, it remains unclear
whether the two-stage approach will be applicable for 360 images, as there are
many fundamental differences between the perspective images and 360 images,
such as camera projection model, image distortion, and field of view (FoV).

The difference in camera projection model, equirectangular projection (see
Figure 3(a)) vs. perspective projection, can be resolved by integrating
spherical geometry into the disparity calculation and cost volume fusion
procedure. In this paper, we propose a novel Spherical Warping Layer
specifically designed for spherical geometry as a solution (Section 4.2).
Moreover, the distortion issue can be addressed by applying distortion-aware
convolutions [16, 14, 52, 51, 13, 66, 9]. Hence, in this section we will
mainly discuss the difference in FoV settings. Comparing with perspective
images, 360 images have much larger FoVs ($360^{\circ}$ horizontally,
$180^{\circ}$ vertically). A 360 image encodes almost every piece of visual
information of the scene except occluded areas, while perspective images
suffer from information loss near the image boundaries in addition to
occlusions. This could be a great advantage for novel view synthesis of 360
images.

To validate this observation, we conducted various experiments regarding the
correlations between image FoV, baseline and synthesized view quality (more
details in the Appendix). Our experiment confirms that i) with greater FoV,
the synthesized views are less sensitive to large baselines, and ii)
synthesized 360 images have the least error and artifacts. According to Gallup
et al. [22], depth error of stereo matching comes from both the disparity
error (proportional) and the baseline (inversely proportional). Thus, with
higher quality synthesized novel views and larger baselines, we expect less
error in the final depth output from stereo matching. This also indicates that
the two-stage pipeline is well-suited for 360 monocular depth estimation.

Figure 2: Illustration of our PanoDepth framework which consists of a view
synthesis stage and a multi-view stereo matching stage. In the view synthesis
stage, a total of $M$ synthesized views are generated. In the multi-view
stereo matching stage, the synthesized 360 views together with the original
input view, are sent to the multi-view stereo matching network to produce the
final depth estimation. To better adapt to 360 stereo geometry, we directly
sample hypothesis plane on the inverse depth, and use a Spherical Warping
Layer (SWL) to warp reference views to the target view (Section 4.2).

##  3 Related Work

###  3.1 Monocular Depth Estimation

Monocular depth estimation [48] has seen significant improvements [37, 30]
since the first adoption of deep learning by Eigen et al. [18]. To further
improve the performance, researchers explored many strategies such as multi-
task learning with normal estimation [47] and semantic segmentation [17, 34]
along with depth estimation, incorporating CRF [6, 42], integrating attention
modules [33, 38], utilizing planar constraints [39, 33], conducting
unsupervised learning using constrainsts such as left-right consistency [24,
5, 44], as well as two-stage approaches where stereo constraints are leveraged
[41, 54, 59].

As 360 cameras become more affordable, researchers start to explore the
possibility to use 360 images for depth estimation [16, 53, 11, 69, 58, 68,
64]. To advance the performance of 360 monocular depth estimation, Eder et al.
[15] proposed joint training of surface normal, boundary, and depth. Zeng et
al. [64] trained a network which combines 3D layout and depth. Jin et al. [35]
took advantage of the correlation between depth map and geometric structure of
360 indoor images. Cheng et al. [11] proposed a low-cost sensing system which
combines an omnidirectional camera with a calibrated projective depth camera.
The 360 image and the limited FoV depth are used together as input to a CNN.
Meanwhile, distortion-aware convolution filters [53, 66, 16, 9] are designed
to handle spherical geometric distortion.

###  3.2 Multi-View Stereo Matching

Besides monocular depth estimation, Multi-View Stereo (MVS) is another group
of methods for predicting depth. Given a set of images with known camera
poses, MVS approaches [32, 63] can produce highly accurate depth estimates
with multi-view geometric constraints. One example is MVSNet [63], in which
the variance-based cost volume is presented to fuse multiple features maps
from source images into one unified cost volume. Stereo matching can be
treated as a special case of MVS. Conventionally, stereo-based depth
estimation methods [49, 29] relied on matching pixels across stereo images.
Many recent stereo matching approaches [36, 8] leveraged CNNs for feature
extraction, cost matching, and aggregation. For example, PSMNet [8]
incorporated spatial pyramid pooling (SPP) module and multi-scale 3D hourglass
modules to further boost the performance. To improve the efficiency and
accelerate training on high-resolution images, Gu et al. [25] presented a
cascade cost volume design to gradually retrieve finer hypothesis plane
ranging over multiple steps.

In 360 stereo domain, SweepNet [62] and OmniMVS [61] estimated depth from
multiple wide-baseline fisheye cameras. Another recent work is 360SD-Net [58]
which predicted disparity/depth from a pair of ERP images that are taken by a
top-bottom camera pair. They [58] incorporated polar angles to solve
distortions and proposed learnable shifting filters to adjust the step size in
disparity cost volume construction. However, the learnable shifting filters
create extra overhead during training. In this paper, we propose a closed-form
solution, Spherical Warping Layer (SWL), that does not require additional
training overhead. Our experiments (Table 2) show that SWL can significantly
improve the performance of 360 stereo matching.

##  4 Approach

We propose an end-to-end framework, PanoDepth, that takes a single ERP image
as input and produces a high-quality omnidirectional depth map. PanoDepth
consists of two stages: i) a view synthesis stage that conducts coarse depth
estimation followed by a differentiable DIBR module for novel view synthesis,
and ii) a stereo matching stage with a customized Spherical Warping Layer for
efficient and high-quality 360 depth estimation. A full framework of PanoDepth
is illustrated in Figure 1.

###  4.1 View Synthesis Stage

To synthesize high-quality novel views, the coarse depth map is usually
estimated first followed by a Depth-Image-Based Rendering (DIBR) module [41,
67]. Based on our empirical observations (see the Appendix for details), such
a procedure also works well for 360 novel view synthesis with different
configurations of coarse depth estimation networks. Considering both
performance and computation cost, in this paper we suggest using a lightweight
network: CoordNet [68] for doing the task. CoordNet utilizes coordinate
convolution [40] to enforce 360 awareness. We append an atrous spatial pyramid
pooling module (ASPP) [10] to the end of the encoder to better aggregate
multi-scale context information. Note that PanoDepth is model-agnostic, thus
any depth estimation network can be used here to fulfill specific requirement.
The estimated coarse depth map and the original ERP image are then used to
render multiple synthesized views of predefined baselines via a differentiable
DIBR operation [55]. In this paper, we choose to use vertical baselines
instead of horizontal ones. The analysis of this choice can be found in the
Appendix.

###  4.2 Stereo Matching Stage

The second stage of our PanoDepth framework is stereo matching. Again, as
PanoDepth is model-agnostic, any stereo matching network can be plugged in
here. Experimental results that show the performance of different stereo
matching network settings is discussed in Section 5.4.

The stereo matching network we used in this paper follows a similar pipeline
as PSMNet [8], with several key modifications. The network consists of five
main modules: feature extraction, spherical warping layer, cost volume
construction, cost aggregation, and depth prediction. Comparing with the
original PSMNet [8], our unique contribution is the Spherical Warping Layer
(SWL) which is specifically designed for the 360 stereo geometry.

Feature Extraction After the view synthesis stage, the input image along with
all the $M$ synthesized novel views will be passed to a weight-sharing neural
network to extract features. We use the same layer setting and keep the SPP
module as the original PSMNet [8].

Spherical Warping Layer (SWL) The extracted feature maps of all views are then
used to build a cost volume at multiple depth hypothesis planes for cost
matching. An essential step of cost volume construction is to determine the
coordinate mapping, which is reflected as disparity, that warps reference view
to the target view. Unlike perspective images where the disparity is
proportional to the inverse depth [45, 8, 56], disparity of 360 stereo pairs
is related to both inverse depth and spherical latitudes. Our SWL performs
direct depth sampling instead of disparity sampling which is commonly used in
perspective image pair stereo matching. Compared with the learnable shifting
filters proposed in [58], our SWL makes the disparity computation adjustable
to the pixel latitudinal value, without introducing additional computation
overhead in training. Moreover, SWL directly samples on the absolute depth
domain, thereby enabling horizontal 360 stereo (which has both horizontal and
vertical disparity, more details in the Appendix), the adaptation of 360
multi-view, and the usage of cascade mechanism.

Figure 3 shows an illustration of the Spherical Warping Layer. We first sample
the inverse depth to cover the whole depth range:

$\frac{1}{d_{j}}=\frac{1}{d_{max}}+(\frac{1}{d_{min}}-\frac{1}{d_{max}})\frac{v\times
j}{D-1},j=0,1,...,D-1$ (1)

where $D$ is the total number of hypothesis planes, $d_{j}$ is the $j^{th}$
depth plane, $d_{min}$ and $d_{max}$ are the minimum and maximum of the depth
image, $v$ is the plane interval. The Spherical Warping Layer then transforms
depth hypothesis $d_{j}$ to displacement in spherical domain $C_{j}$, to map
pixels from the reference synthesized view to the target view. The
displacement $C_{j}$ is defined as:

$C_{x,j}=0,\;C_{y,j}=\frac{cos(\theta)\times b}{d_{j}}\times\frac{H_{f}}{\pi}$
(2)

where $\theta$ refers to the pixel-wise latitudinal values, $b$ represents the
baseline, and $H_{f}$ is the height of the feature map.

Cost Volume Construction The SWL transforms reference view feature maps into
the target view domain at the individual hypothesis plane, and then a total of
$M+1$ feature volumes are generated. The variance-based cost volume formation
method from MVSNet [63] is used for the fusion of these feature volumes into a
compact one. Moreover, we adopt a cascade design from Gu et al. [25] to
further improve the final depth quality. Specifically, at level $l(l>1)$,
$d_{min}$ and $d_{max}$ is recalculated based on the prediction of level
$l-1$, then the new depth range and the new number of planes $D_{l}$ is used
to determine the new intervals. Depth hypothesis for level $l$ is then updated
using equ (13). The corresponding displacements are calculated via the same
spherical coordinate mapping procedure.

Figure 3: Visualization of our spherical warping method. (a) Vertical 360
stereo model. $b$ is the baseline displacement of two cameras. $P$ is a real-
world point. The projection of $P$ on two camera space is represented as
$p(\phi_{1},\theta_{1})$ and $q(\phi_{2},\theta_{2})$. (b) Spherical epipolar
geometry. $P_{i}$ is the points sampled at different depths. (c) Projection of
sampled inverse depth on the ERP image. $p_{i}$ is the projection of $P$ on
the top view at sampled points $P_{i}$. $C_{i}$ is the vertical disparity, it
equals to $C_{y}$ in Equ (15). (d) Projection on the reference image. $q$ is
the projection of $P$ at bottom view.

Cost Aggregation and Depth Prediction After the construction of the cost
volume, multi-scale 3D CNN is used to aggregate different levels of spatial
context information through the hourglass-shape encoding and decoding network.
It has been shown this kind of cost aggregation module helps to regularize
noises in ambiguous regions caused by occlusions, textureless surfaces, and to
improve final prediction quality [36, 8, 26]. Finally, we regress the depth
value at each level $l$:

$\displaystyle\frac{1}{d_{pred,l}}=\frac{1}{d_{min,l}}+(\frac{1}{d_{min,l}}-\frac{1}{d_{max,l}})\frac{k_{l}}{D_{l}-1}$
(3) $\displaystyle k_{l}=\sum^{D_{l}-1}_{j=0}\sigma(p_{j})\times(v_{l}\times
j)$ (4)

where $k_{l}$ is the sum of each plane level weighted by its normalized
probability, $\sigma(\cdot)$ represents softmax function, $p_{j}$ denotes the
probability of $j^{th}$ plane value, $v_{l}$ is the interval for level $l$.

###  4.3 Loss Function

PanoDepth is trained in an end-to-end fashion, supervision is applied on both
stages. The final loss function is defined as follows,

$\mathcal{L}_{total}=\omega_{1}\mathcal{L}_{coarse}+\omega_{2}\mathcal{L}_{stereo}$
(5)

where $\omega_{1}$ and $\omega_{2}$ are the weights of coarse depth estimation
loss and stereo matching loss respectively. For the optimization on the first
stage coarse estimation, we use inverse Huber (berHu) loss as proposed in
[37]:

$\mathcal{L}_{coarse}=\frac{1}{\Omega}\sum_{i\in\Omega}\mathcal{L}_{berHu}(d_{i},\;\;d_{i}^{*})\\\
$ (6)

where $\Omega$ is a binary mask that is used to mask out missing regions
(pixels that have depth values smaller than $d_{min}$ or greater than
$d_{max}$), $d_{i}$ and $d_{i}^{*}$ are the ground truth and the predicted
depth value of a valid pixel $i$ respectively. For stereo matching, we
calculate berHu loss [37] on all outputs from each level $l$ and then compute
the weighted summation. The stereo matching loss is defined as:

$\mathcal{L}_{stereo}=\frac{1}{\Omega}\sum_{i\in\Omega}\sum_{l=1}^{N}\lambda_{l}\mathcal{L}_{berHu}(d_{i},\;\;d_{i}^{*})$
(7)

where $\lambda_{l}$ is the level $l$ stereo loss weight.

##  5 Experiments

###  5.1 Datasets

We train and evaluate our network on three panorama RGBD benchmark datasets
including, Stanford2D3D [3], 360D [69] and the omnidirectional stereo dataset
[68].

Stanford2D3D Stanford2D3D [3] dataset consists of 1413 real-world panorama
images from six large-scale indoor areas. We follow the official train-test
split which uses the fifth area for testing, and other areas for training. We
resize the images to $256\times 512$ to reduce computation time.

360D 360D [69] is a RGBD panorama benchmark provided by Zioulis et al. [69].
It is composed of two other synthetic datasets (SunCG and SceneNet), and two
real-world datasets (Stanford2D3D and Matterport3D). There are 35,977 panorama
RGBD images in the 360D that are rendered from the aforementioned four
datasets. We again follow the default train-test splits.

Omnidirectional Stereo Dataset. The omnidirectional stereo dataset [68]
consists of 7964 stereo pairs of panorama RGBD images rendered from two real-
world datasets, Matterport3D [7] and Stanford2D3D [3]. We use the train-test
split that removes 3 complete buildings from Matterport3D [7] and 1 complete
area from Stanford2D3D [3] for test. Each set of data consists of left-down,
right, and up view 360 RGBD images in a triangular fashion with size
$256\times 512$. We only use images from the left-down view in our single view
depth estimation experiments. Up-down stereo pairs are only used for the
ablation study of the stereo matching network.

To investigate the impact of baselines and FoV on the quality of view
synthesis, we create a new dataset that is rendered from the mesh of
Stanford2D3D [3]. More details of this new dataset as well as the experiments
with various baseline configurations are included in the Appendix.

###  5.2 Implementation Details and Metrics

For parameter settings, we use a default of $N=2$ levels, with $D_{1}=48$ and
$D_{2}=24$ hypothesis planes respectively. The minimum and maximum depth
$d_{min}$ and $d_{max}$ for the first level is set to be $0.2m$ and $8m$. We
use a default of $M=3$ synthesized views rendered at vertical baseline
placements $-0.24m,+0.24m,+0.4m$. The loss weights $\omega_{1}$ and
$\omega_{2}$ are set to $1$ and $0.02$. We train our framework from scratch
using Adam optimizer ($\beta_{1}=0.9,\beta_{2}=0.999$) with a batch size of 8.
Initial learning rates for the first and the second stage are set to 0.0002
and 0.0005. We separately train the coarse network for 10 epochs and then
train the entire framework end-to-end for 200 epochs. Both of the learning
rates decay by a factor of 0.5 every 30 epochs. Performances are evaluated
based on commonly used depth quality measures [18]: absolute relative error
(Abs Rel), square relative error (Sq Rel), linear root mean square error(RMSE)
and its natural log scale (RMSE log) and inlier ratios ($\delta_{i}<1.25^{i}$,
$i\in\\{1,2,3\\}$).

###  5.3 Overall Performance Comparison with the State-of-the-art Algorithms

Table 1 lists quantitative comparison between PanoDepth and other state-of-
the-art omnidirectional monocular depth estimation methods [37, 69, 57, 11] on
both Stanford2D3D [3] and 360D [69] datasets. As shown in Table 1, our method
is able to reduce Abs Rel error by 19.60% on Stanford2D3D and 25.85% on 360D
compare to the current leading 360 monocular depth estimation approach BiFuse
[57]. Note that BiFuse [57] uses a network architecture with more than 200M
parameters and has a large computation complexity for sharing information
between CubeMap [43] and ERP formats. The framework with distortion-aware
module proposed in [9] outperforms ours but it has more than 60M parameters.
Our framework has only around 16M parameters with a smaller computation
overhead. Comparing the performance with ODE-CNN [11] on 360D, our approach
achieves comparable results while ODE-CNN requires additional depth sensor
input. Figure 4 shows the qualitative comparison with the state-of-the-art
approaches. As we can see, our method generates high-quality depth with a
detailed surface, sharp edges, and precise range.

Datasets |  Methods |  Abs Rel$\downarrow$ |  RMSE$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---
Stanford2D3D [3] |  FCRN [37] |  0.1837 |  0.5774 |  0.7230 |  0.9207 |  0.9731
RectNet [69] |  0.1409 |  0.4568 |  0.8326 |  0.9518 |  0.9822
BiFuse with fusion [57] |  0.1209 |  0.4142 |  0.8660 |  0.9580 |  0.9860
Joint wth layout and semantics [64] |  0.0680 |  0.2640 |  0.9540 |  0.9920 |  0.9980
PanoDepth(Ours) |  0.0972 |  0.3747 |  0.9001 |  0.9701 |  0.9900
360D [69] |  FCRN [37] |  0.0699 |  0.2833 |  0.9532 |  0.9905 |  0.9966
RectNet [69] |  0.0702 |  0.2911 |  0.9574 |  0.9933 |  0.9979
Mapped Convolution [16] |  0.0965 |  0.2966 |  0.9068 |  0.9854 |  0.9967
Distortion-aware [9] |  0.0406 |  0.1769 |  0.9865 |  0.9966 |  0.9987
BiFuse with fusion [57] |  0.0615 |  0.2440 |  0.9699 |  0.9927 |  0.9969
ODE-CNN [11] |  0.0467 |  0.1728 |  0.9814 |  0.9967 |  0.9989
PanoDepth(Ours) |  0.0456 |  0.1955 |  0.9830 |  0.9957 |  0.9984
Table 1: A quantitative comparison with the state-of-the-art approaches on
Stanford2D3D [3] dataset and 360D [69] dataset ($\downarrow$ represents lower
the better, $\uparrow$ represents higher the better). We report the results
based on the original papers [69, 57, 11] using the same evaluation metrics.
Note that ODE-CNN [11] requires additional depth sensor input besides the 360
image used by other methods listed in the table. Additional supervision
signals including layout and semantics are used in [64]. For our PanoDepth, we
use the default stereo network setting with three synthesized views and a two-
level cascade design.

###  5.4 Ablation Studies

Methods |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  RMSElog$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---
(1) PSMNet [8], sample on disparity, $D=64$ |  0.0433 |  0.0252 |  0.2541 |  0.1340 |  0.9722 |  0.9833 |  0.9900
(2) 360SD-Net [58], sample on disparity, $D=64$ |  0.0387 |  0.0198 |  0.2286 |  0.0955 |  0.9776 |  0.9900 |  0.9940
(3) PanoDepth (ours), one-level, $D=32$ |  0.0253 |  0.0222 |  0.2268 |  0.0686 |  0.9756 |  0.9874 |  0.9976
(4) PanoDepth (ours), one-level, $D=64$ |  0.0229 |  0.0087 |  0.1731 |  0.0606 |  0.9900 |  0.9969 |  0.9987
(5) PanoDepth (ours), two-level, $D_{1}=48,D_{2}=24$ |  0.0178 |  0.0064 |  0.1415 |  0.0519 |  0.9928 |  0.9976 |  0.9990
Table 2: A quantitative comparison between the PanoDepth stereo matching and existing stereo matching networks on the Omnidirectional stereo dataset [68] where up-down stereo pairs are used as input and output the depth of bottom view. Our proposed stereo matching module (3,4,5) outperforms both (1) PSMNet [8] and (2) 360SD-Net [58]. The two cascade level setting achieves the best performance. |  1st Stage |  2nd Stage w/ 1 synthesize view |  $\\#$params |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  RMSElog$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---|---|---|---
(1) |  CoordNet |  N/A |  6.1M |  0.1264 |  0.0888 |  0.4456 |  0.2084 |  0.8533 |  0.9588 |  0.9813
(2) |  RectNet |  N/A |  10.8M |  0.1409 |  0.0859 |  0.4568 |  0.2124 |  0.8326 |  0.9518 |  0.9822
(3) |  CoordNet |  PSMNet [8], D=32 |  13.0M |  0.1206 |  0.0833 |  0.4293 |  0.2150 |  0.8671 |  0.9548 |  0.9790
(4) |  CoordNet |  w/ SWL, one-level, D=32 |  13.0M |  0.1132 |  0.0686 |  0.4077 |  0.1869 |  0.8757 |  0.9652 |  0.9863
(5) |  RectNet |  w/ SWL, one-level, D=32 |  17.6M |  0.1192 |  0.0775 |  0.4202 |  0.1960 |  0.8655 |  0.9607 |  0.9846
(6) |  CoordNet |  w/ SWL, two-level, $D_{1}$=32, $D_{2}$=16 |  16.6M |  0.1040 |  0.0645 |  0.3918 |  0.1827 |  0.8865 |  0.9676 |  0.9875
(7) |  RectNet |  w/ SWL, two-level, $D_{1}$=32, $D_{2}$=16 |  21.3M |  0.1138 |  0.0761 |  0.4274 |  0.1961 |  0.8711 |  0.9577 |  0.9837
Table 3: An ablation study of the impact of various combinations of coarse
estimation network and stereo matching network on the final performance. The
experiments are trained on Stanford2D3D [3]. We use two types of coarse
estimation networks, (1) CoordNet, and (2) RectNet [69]. We can see that even
with one synthesize view, our proposed two-stage PanoDepth pipeline
(3,4,5,6,7) is able to outperform the one-stage-only methods (1,2). Adding
Spherical Warping Layer (SWL) (4,5,6,7) and two cascade levels (6,7) further
improves the performance. The experimental results indicate that our two-stage
pipeline is model-agnostic under various network settings.  Figure 4: A
qualitative comparison between RectNet [69] (3rd column), BiFuse [57] (4th
column), and our method (5th column) on 360D [69]. We highlight and zoom in
some areas that distinguish the performance of three methods. We can see that
our PanoDepth is able to produce sharp edges, predict depth range accurately,
and recover surface detail. Figure 5: A qualitative comparison to show the
effectiveness of SWL in the PanoDepth stereo matching module. We compare
between one-level stereo matching method without SWL(4th column), one-level
with SWL (5th column), and two-level with SWL (last column). The experiments
are trained on Omnidirectional Stereo Dataset [68]. Our stereo matching module
with SWL recovers clear details and shows fewer artifacts than the one without
SWL (see highlighted areas).

Spherical Warping Layer In order to evaluate the performance of the PanoDepth
stereo matching module with the novel Spherical Warping Layer (SWL), we
compare it to the state-of-the-art stereo matching approaches, PSMNet [8] and
360-SD Net [58], with ground truth up-down 360 stereo pair as input on the
Omnidirectional Stereo Dataset [68]. In the experiment, we use the officially
released code of both approaches, and convert the output disparity into depth
for evaluation. We set the number of depth hypothesis planes to 64 to ensure
fair and consistent comparison. We can see from Table 2 that our proposed
stereo matching method with SWL outperforms PSMNet [8] and 360-SD Net [58],
even with one-level setting. Moreover, by adding SWL, our one-level setting
outperforms the one without SWL (identical to PSMNet[8]) by $47\%$ in terms of
Abs Rel. with the same 64 sampling planes. Qualitative illustrations of the
effectiveness of SWL is shown in Figure 5.

Model-agnostic Evaluations In Table 3, we further test the performance of the
full PanoDepth pipeline given different variations of stereo matching networks
on Stanford2D3D dataset [3]. Comparing with the single-stage coarse depth
estimation, all two-stage configurations show better performances. By adding a
light-weight one-level stereo matching network with 32 depth plane in the
second stage, PanoDepth can already reach comparable performance to BiFuse
[57]. The performance can be further improved by introducing SWL, adding more
cascade levels, and using more sophisticated coarse depth estimation.

In addition, by comparing the performance of two-stage approaches with two
different backbones, CoordNet and RectNet, we can also postulate that coarse
depth, which has an impact on the synthetic image quality, is positively
correlated with the final depth prediction.

More ablation studies regarding the number of synthesized views, the number of
hypothesis depth planes, and the comparison between one-stage alternatives
(e.g., multi-tasking and adding depth refinement) and our two-stage method can
be found in the Appendix.

##  6 Conclusion and Future Work

In this paper, we demonstrate a technique that leverages view synthesis and
stereo constraints to advance monocular depth estimation performance that can
be applied on 360 images. We propose a novel model agnostic two-stage
framework PanoDepth for generating dense high-quality depth from a monocular
360 input. Extensive experiments show that PanoDepth outperforms state-of-the-
art approaches by a large margin. Our stereo matching sub-network in the later
stage adapts to the 360 geometry and achieves top-ranking performance in 360
stereo matching. We believe the good performance of PanoDepth could draw more
interests from both the industry and academia to 360 images for its still
under-explored capability in tasks such as depth estimation. We hope our work
can motivate more research and applications in 360 images. There are several
research venues we would like to further explore in the future, such as
alternative view synthesis methods like [20, 60], and 360 depth estimation in
outdoor scenarios for applications like autonomous driving.

## Acknowledgments

The research of Yuyan Li and Ye Duan were partially supported by the National
Science Foundation under award CNS-2018850, National Institute of Health under
awards NIBIB-R03-EB028427 and NIBIB-R01-EB02943, and U.S. Army Research
Laboratory W911NF2120275. Any opinions, findings, and conclusions or
recommendations expressed in this publication are those of the authors and do
not necessarily reflect the views of the U. S. Government or agency thereof.

## References

  * [1] Naveen Appiah and Nitin Bandaru.  Obstacle detection using stereo vision for self-driving cars.  In IEEE Intelligent Vehicles Symposium (IV), pages 926–932, 2011\.
  * [2] Lemonia Argyriou, Daphne Economou, and Vassiliki Bouki.  Design methodology for 360 immersive video applications: the case study of a cultural heritage virtual tour.  Personal and Ubiquitous Computing, pages 1–17, 2020.
  * [3] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese.  Joint 2d-3d-semantic data for indoor scene understanding.  arXiv preprint arXiv:1702.01105, 2017.
  * [4] Matthias Berning, Takuro Yonezawa, Till Riedel, Jin Nakazawa, Michael Beigl, and Hide Tokuda.  parnorama: 360 degree interactive video for augmented reality prototyping.  In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication, pages 1471–1474, 2013.
  * [5] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid.  Unsupervised scale-consistent depth and ego-motion learning from monocular video.  In Advances in Neural Information Processing Systems, pages 35–45, 2019.
  * [6] Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen.  Estimating depth from monocular images as classification using deep fully convolutional residual networks.  IEEE Transactions on Circuits and Systems for Video Technology, 28(11):3174–3182, 2017.
  * [7] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  International Conference on 3D Vision (3DV), 2017.
  * [8] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid stereo matching network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5410–5418, 2018.
  * [9] Hong-Xiang Chen, Kunhong Li, Zhiheng Fu, Mengyi Liu, Zonghao Chen, and Yulan Guo.  Distortion-aware monocular depth estimation for omnidirectional images.  IEEE Signal Processing Letters, 28:334–338, 2021.
  * [10] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [11] Xinjing Cheng, Peng Wang, Yanqi Zhou, Chenye Guan, and Ruigang Yang.  Omnidirectional depth extension networks.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 589–595. IEEE, 2020.
  * [12] Blender Online Community.  Blender - a 3D modelling and rendering package.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  * [13] Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger.  Spherenet: Learning spherical representations for detection and classification in omnidirectional images.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 518–533, 2018.
  * [14] Marc Eder and Jan-Michael Frahm.  Convolutions on spherical images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1–5, 2019.
  * [15] Marc Eder, Pierre Moulon, and Li Guan.  Pano popups: Indoor 3d reconstruction with a plane-aware network.  In 2019 International Conference on 3D Vision (3DV), pages 76–84. IEEE, 2019.
  * [16] Marc Eder, True Price, Thanh Vu, Akash Bapat, and Jan-Michael Frahm.  Mapped convolutions.  arXiv preprint arXiv:1906.11096, 2019.
  * [17] David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In Proceedings of the IEEE international conference on computer vision, pages 2650–2658, 2015.
  * [18] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Advances in neural information processing systems, pages 2366–2374, 2014.
  * [19] Michael S. Feurstein.  Towards an integration of 360-degree video in higher education.  In DeLFI Workshops, 2018.
  * [20] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2367–2376, 2019.
  * [21] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2002–2011, 2018.
  * [22] David Gallup, Jan-Michael Frahm, Philippos Mordohai, and Marc Pollefeys.  Variable baseline/resolution stereo.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [23] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde.  Learning to predict indoor illumination from a single image.  In ACM Transactions on Graphics (SIGGRAPH Asia), 2017.
  * [24] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 270–279, 2017.
  * [25] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  arXiv preprint arXiv:1912.06378, 2019.
  * [26] Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, and Hongsheng Li.  Group-wise correlation stereo network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3273–3282, 2019.
  * [27] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [29] Heiko Hirschmuller.  Accurate and efficient stereo processing by semi-global matching and mutual information.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 807–814. IEEE, 2005.
  * [30] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani.  Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1043–1051. IEEE, 2019.
  * [31] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  2018\.
  * [32] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang.  Deepmvs: Learning multi-view stereopsis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2821–2830, 2018.
  * [33] Lam Huynh, Phong Nguyen-Ha, Jiri Matas, Esa Rahtu, and Janne Heikkila.  Guiding monocular depth estimation using depth-attention volume.  arXiv preprint arXiv:2004.02760, 2020.
  * [34] Jianbo Jiao, Ying Cao, Yibing Song, and Rynson Lau.  Look deeper into depth: Monocular depth estimation with semantic booster and attention-driven loss.  In Proceedings of the European conference on computer vision (ECCV), pages 53–69, 2018.
  * [35] Lei Jin, Yanyu Xu, Jia Zheng, Junfei Zhang, Rui Tang, Shugong Xu, Jingyi Yu, and Shenghua Gao.  Geometric structure based and regularized depth estimation from 360 indoor imagery.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [36] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.  End-to-end learning of geometry and context for deep stereo regression.  In Proceedings of the IEEE International Conference on Computer Vision, pages 66–75, 2017.
  * [37] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In 2016 Fourth international conference on 3D vision (3DV), pages 239–248. IEEE, 2016.
  * [38] Ruibo Li, Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, and Lingxiao Hang.  Deep attention-based classification network for robust depth prediction.  In Asian Conference on Computer Vision, pages 663–678. Springer, 2018.
  * [39] Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, and Yasutaka Furukawa.  Planenet: Piece-wise planar reconstruction from a single rgb image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2579–2588, 2018.
  * [40] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski.  An intriguing failing of convolutional neural networks and the coordconv solution.  In Advances in Neural Information Processing Systems, pages 9605–9616, 2018.
  * [41] Yue Luo, Jimmy Ren, Mude Lin, Jiahao Pang, Wenxiu Sun, Hongsheng Li, and Liang Lin.  Single view stereo matching.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 155–163, 2018.
  * [42] Faisal Mahmood and Nicholas J Durr.  Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy.  Medical image analysis, 48:230–243, 2018.
  * [43] Rafael Monroy, Sebastian Lutz, Tejo Chalasani, and Aljosa Smolic.  Salnet360: Saliency maps for omni-directional images with cnn.  Signal Processing: Image Communication, 69:26–34, 2018.
  * [44] Greire Payen de La Garanderie, Amir Atapour Abarghouei, and Toby P Breckon.  Eliminating the blind spot: Adapting 3d object detection and monocular depth estimation to 360 panoramic imagery.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 789–807, 2018.
  * [45] Marc Pollefeys, Reinhard Koch, and Luc Van Gool.  A simple and efficient rectification method for general motion.  In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 1, pages 496–501. IEEE, 1999.
  * [46] Gyula Pudics, Miklós Zsolt Szabó-Resch, and Zoltán Vámossy.  Safe robot navigation using an omnidirectional camera.  In 2015 16th IEEE International Symposium on Computational Intelligence and Informatics (CINTI), pages 227–231. IEEE, 2015.
  * [47] Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia.  Geonet: Geometric neural network for joint depth and surface normal estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 283–291, 2018.
  * [48] Ashutosh Saxena, Sung H Chung, and Andrew Y Ng.  Learning depth from single monocular images.  In Advances in neural information processing systems, pages 1161–1168, 2006.
  * [49] Daniel Scharstein and Richard Szeliski.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  International journal of computer vision, 47(1-3):7–42, 2002.
  * [50] Nikolai Smolyanskiy, Alexey Kamenev, and Stan Birchfield.  On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1007–1015, 2018.
  * [51] Yu-Chuan Su and Kristen Grauman.  Learning spherical convolution for fast features from 360 imagery.  In Advances in Neural Information Processing Systems, pages 529–539, 2017.
  * [52] Yu-Chuan Su and Kristen Grauman.  Kernel transformer networks for compact spherical convolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9442–9451, 2019.
  * [53] Keisuke Tateno, Nassir Navab, and Federico Tombari.  Distortion-aware convolutional filters for dense prediction in panoramic images.  In The European Conference on Computer Vision (ECCV), September 2018\.
  * [54] Fabio Tosi, Filippo Aleotti, Matteo Poggi, and Stefano Mattoccia.  Learning monocular depth estimation infusing traditional stereo knowledge.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9799–9809, 2019.
  * [55] Shubham Tulsiani, Richard Tucker, and Noah Snavely.  Layer-structured 3d scene inference via view synthesis.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 302–317, 2018.
  * [56] Julien Valentin, Adarsh Kowdle, Jonathan T Barron, Neal Wadhwa, Max Dzitsiuk, Michael Schoenberg, Vivek Verma, Ambrus Csaszar, Eric Turner, Ivan Dryanovski, et al.  Depth from motion for smartphone ar.  ACM Transactions on Graphics (ToG), 37(6):1–19, 2018.
  * [57] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai.  Bifuse: Monocular 360 depth estimation via bi-projection fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 462–471, 2020.
  * [58] Ning-Hsu Wang, Bolivar Solarte, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun.  360sd-net: 360 stereo depth estimation with learnable cost volume.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 582–588. IEEE, 2020.
  * [59] Jamie Watson, Oisin Mac Aodha, Daniyar Turmukhambetov, Gabriel J Brostow, and Michael Firman.  Learning stereo from single images.  arXiv preprint arXiv:2008.01484, 2020.
  * [60] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7467–7477, 2020.
  * [61] Changhee Won, Jongbin Ryu, and Jongwoo Lim.  Omnimvs: End-to-end learning for omnidirectional stereo matching.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8987–8996, 2019.
  * [62] Changhee Won, Jongbin Ryu, and Jongwoo Lim.  Sweepnet: Wide-baseline omnidirectional depth estimation.  In IEEE International Conference on Robotics and Automation (ICRA), pages 6073–6079, 2019.
  * [63] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 767–783, 2018.
  * [64] Wei Zeng, Sezer Karaoglu, and Theo Gevers.  Joint 3d layout and depth prediction from a single indoor panorama image.  In European Conference on Computer Vision, pages 666–682. Springer, 2020.
  * [65] Chaoqiang Zhao, Qiyu Sun, Chongzhen Zhang, Yang Tang, and Feng Qian.  Monocular depth estimation based on deep learning: An overview.  arXiv preprint arXiv:2003.06620, 2020.
  * [66] Qiang Zhao, Chen Zhu, Feng Dai, Yike Ma, Guoqing Jin, and Yongdong Zhang.  Distortion-aware cnns for spherical images.  In IJCAI, pages 1198–1204, 2018.
  * [67] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  arXiv preprint arXiv:1805.09817, 2018.
  * [68] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, Federico Alvarez, and Petros Daras.  Spherical view synthesis for self-supervised 360 depth estimation.  In 2019 International Conference on 3D Vision (3DV), pages 690–699. IEEE, 2019.
  * [69] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, and Petros Daras.  Omnidepth: Dense depth estimation for indoors spherical panoramas.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 448–465, 2018.

##  Appendix A Spherical Geometry Analysis

Figure 6: (a) Projection of a 3D point $P$ from spherical space to
equirectangular domain. (b) Vertical spherical disparity model with two
cameras placed at baseline $b$. (c) Horizontal spherical disparity model with
two cameras placed at baseline $b$.

The projection of a 3D point $P$ from spherical coordinates$(r,\phi,\theta)$
to the ERP image is shown in Figure 6(a), in which $\phi\in[0,\pi]$ represents
longitude, and $\theta\in[0,2\pi]$ is the complement of latitude. $r$ is the
distance from $P$ to the camera center, which is considered as depth in our
study. Therefore, given two cameras positioned at a fixed baseline $b$, we can
build the spherical stereo model. The displacement between the two cameras can
be either vertical (Figure 6(b)) or horizontal (Figure 6(c)).

The conversion from a 3D point $P=(x,y,z)$ in Cartesian coordinates to
spherical coordinates $(r,\phi,\theta)$ can be calculated as follows:

$\displaystyle r$ $\displaystyle=\sqrt{x^{2}+y^{2}+z^{2}}$ (8)
$\displaystyle\phi$ $\displaystyle=arctan(\frac{x}{z})$ $\displaystyle\theta$
$\displaystyle=arccos(\frac{y}{r})$

The conversion from spherical to Cartesian is defined as:

$\displaystyle x$ $\displaystyle=r\;sin\phi\;sin\theta$ (9) $\displaystyle y$
$\displaystyle=r\;cos\theta$ $\displaystyle z$
$\displaystyle=r\;cos\phi\;sin\theta$

If we derive the partial derivatives of the conversion process from Equ (9),
we would have:

$\displaystyle\partial\phi$
$\displaystyle=\frac{cos\phi}{r\;sin\theta}\partial
x-\frac{sin\phi}{r\;sin\theta}\partial z$ (10) $\displaystyle\partial\theta$
$\displaystyle=\frac{sin\phi\;cos\theta}{r}\partial
x-\frac{sin\theta}{r}\partial y+\frac{cos\phi\;cos\theta}{r}\partial z$

From Equ (10), we can calculate the relationship between baseline placement
$b=(dx,dy,dz)$ and spherical disparity $d=(d\phi,d\theta)$, for vertical model
where $b_{y}=(0,dy,0)$:

$d\phi=0,\;\;d\theta=-\frac{sin\theta}{r}b_{y}$ (11)

For horizontal model where $b_{x}=(dx,0,0)$:

$d\phi=\frac{cos\phi}{r\;sin\theta}b_{x},\;\;d\theta=\frac{sin\phi\;cos\theta}{r}b_{x}$
(12)

We can see from Equ (11) that there is no longitudinal displacements for
vertical stereo model, while there are both longitudinal and latitudinal
displacements for horizontal stereo model as described in Equ (12). Thus
estimating disparity for horizontal stereo model could be difficult with
displacements along both horizontal and vertical directions. Our Spherical
Warping Layer (SWL) solves this problem by directly working on depth and
calculating coordinate mapping based on depth. In Section C.2.5, we show the
quantitative comparison between horizontal and vertical stereo model. Based on
both the theoretical analysis and experimental results, we choose the vertical
stereo as our default setting for the second stage network.

Figure 7: Illustration of scene rendering procedure. Left is the illustration
of an example viewpoint (black dot) with three cameras placed along the same
vertical axis. On the right side, the three ERP images with purple, orange,
and yellow borders correspond to the three camera views (shown in
corresponding colors) in the left image.  Figure 8: Illustration of the
procedure for generating perspective images. For each ERP image rendered with
various baselines, we take multiple perspective views using defined field of
view (FoV), and map pixels within the FoV to perspective images. The
conversion process runs five times for the five FoV settings.  Figure 9: The
study of correlations between image FoV, baseline and synthesized view
quality. (a) Plots of MSE error over the baseline settings. (b) Colored error
map visualization of MSE for perspective images (row 1,2,3) and ERP images
(row 4).

##  Appendix B Network Architecture and Implementation Details

In this section, we discuss the implementation details of PanoDepth network
architecture. PanoDepth is model-agnostic, in other words, the sub-networks
are replaceable. We only show examples of network implementation with good
performance based on our experience.

###  B.1 Coarse Depth Estimation Network in View Synthesis Stage

For the coarse depth estimation in view synthesis stage, We customize CoordNet
[68] with ASPP module [10] and deeper convolution layers. From Table 3 in the
main paper, we can see that our customized CoordNet have less parameters than
RectNet [69], it outperforms RectNet in both one-stage and full two-stage
pipeline omnidirectional depth estimation. The detailed network implementation
is shown in Table 4, the architecture of the CoordNet as well as view
synthesis is shown in Figure 10. The output coarse depth is then fed into
depth image based rendering (DIBR) module together with original RGB 360 image
for novel view synthesis given baseline. The DIBR implementation is based on
[55]. The pixels from source image are splatted on an empty target image, then
occlusions are handled by soft z-buffering. The final prediction is the
weighted average of points which splat to the same pixel.

Although empirically the quality of view synthesis depends on the accuracy of
depth maps, the correlation between the two is not necessarily linear ([41,
57]). The key for our model-agnostic approach is to find the configuration
which achieves best performance with reasonable computation resources. We have
shown our results with different coarse depth estimation choices as guidelines
in main paper Table 3 to justify our choice of network implementation here.

Layer |  Add Coor. |  UpSample |  Conv. |  Batch Norm. |  ReLU |  Output Layer |  Output Dim.
---|---|---|---|---|---|---|---
conv1 |  Y |  - |  $7\times 7,s1,p3,d1$ |  Y |  Y |  conv2,3 |  $256\times 512\times 16$
conv2 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv4 |  $256\times 512\times 16$
conv3 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  N |  conv4 |  $256\times 512\times 16$
conv4 |  Y |  - |  $3\times 3,s2,p1,d1$ |  Y |  Y |  conv5,6 |  $128\times 256\times 32$
conv5 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv7,8 |  $128\times 256\times 32$
conv6 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  N |  conv7,8 |  $128\times 256\times 32$
conv7 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv9,10 |  $128\times 256\times 32$
conv8 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  N |  conv9,10 |  $128\times 256\times 32$
conv9-10 |  - |  - |  repeat conv7-8 |  - |  - |  - |  $128\times 256\times 32$
conv11 |  Y |  - |  $3\times 3,s2,p1,d1$ |  Y |  Y |  conv12,13 |  $64\times 128\times 64$
conv12 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv14,15 |  $64\times 128\times 64$
conv13 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  N |  conv14,15 |  $64\times 128\times 64$
conv14 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv16,17 |  $64\times 128\times 64$
conv15 |  Y |  - |  $3\times 3,s1,p1,d1$ |  Y |  N |  conv16,17 |  $64\times 128\times 64$
conv16-37 |  - |  - |  repeat conv14-15 |  - |  - |  - |  $64\times 128\times 64$
ASPP conv38 |  - |  - |  $1\times 1,s1,p1,d1$ |  Y |  - |  concat44 |  $64\times 128\times 256$
ASPP conv39 |  - |  - |  $3\times 3,s1,p6,d6$ |  Y |  - |  concat44 |  $64\times 128\times 256$
ASPP conv40 |  - |  - |  $3\times 3,s1,p12,d12$ |  Y |  - |  concat44 |  $64\times 128\times 256$
ASPP conv41 |  - |  - |  $3\times 3,s1,p18,d18$ |  Y |  - |  concat44 |  $64\times 128\times 256$
avgpool42 |  - |  - |  - |  - |  - |  conv43 |  -
ASPP conv43 |  - |  - |  $1\times 1,s1,p1,d1$ |  Y |  - |  concat44 |  $1\times 1\times 256$
concat44 |  - |  - |  - |  - |  - |  conv45 |  -
ASPP conv45 |  - |  - |  $1\times 1,s1,p1,d1$ |  Y |  - |  conv46 |  $64\times 128\times 64$
conv46 |  - |  2.0 |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv47 |  $128\times 256\times 32$
conv47 |  - |  - |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv48 |  $128\times 256\times 32$
conv48 |  - |  2.0 |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv49 |  $256\times 512\times 16$
conv49 |  - |  - |  $3\times 3,s1,p1,d1$ |  Y |  Y |  conv50 |  $256\times 512\times 16$
conv50 |  Y |  - |  $7\times 7,s1,p3,d1$ |  N |  N |  - |  $256\times 512\times 1$
Table 4: Our customized CoordNet has 50 convolution layers with ASPP module.
Each properties (s, p, d) means (stride, padding, dilation) in the
convolutional block. Figure 10: Illustration of view synthesis stage including
coarse depth estimation network and DBIR module.

###  B.2 Stereo Matching Network

Our stereo matching network adapts from PSMNet, with Spherical Warping Layer
(SWL) and other features enabled by SWL such as multi-view cost fusion and
cascade design. The architecture of our stereo matching network is illustrated
in Figure 11. Note that as PanoDepth is model-agnostic, we can replace PSMNet
with other deep stereo matching networks simply by integrating SWL in cost
volume construction.

During the construction of the cost volume, it is preferable that the inverse
depth of the depth hypothesis planes are uniformly distributed. Otherwise, the
estimated depth from the spherical stereo matching network could be biased. To
ensure the uniform distribution of depth hypothesis planes, we develop the
following process. The first step is to generate depth hypothesis planes. As
mentioned in our main paper, cascade mechanism is incorporated for more
accurate stereo matching. At the first cascade level $l=1$, we sample
uniformly on the inverse depth domain:

$\frac{1}{d_{j}}=\frac{1}{d_{max}}+(\frac{1}{d_{min}}-\frac{1}{d_{max}})\frac{v_{0}\times
j}{D-1},j=0,1,...,D-1$ (13)

where $D$ is the total number of hypothesis planes at level $l=1$, $d_{j}$ is
the $j^{th}$ depth plane, $d_{min}$ and $d_{max}$ are the minimum and maximum
of the depth image, $v_{0}$ is the initial plane interval. At a subsequent
level $l(l>1)$, hypothesis plane ranges are adjusted according to the
prediction from the previous level. With an initial interval $v_{l}$, we
update the minimum and maximum of depth and re-calculate the interval for each
individual pixel:

$\displaystyle d^{\prime}_{min}$
$\displaystyle=max(\frac{1}{\tilde{d}_{l-1}}-\frac{D_{l}\times
v_{l}}{2},\;\;\frac{1}{d_{max}})$ (14) $\displaystyle d^{\prime}_{max}$
$\displaystyle=min(\frac{1}{d_{min}},\;\frac{1}{\tilde{d}_{l-1}}+\frac{D_{l}\times
v_{l}}{2})$ $\displaystyle v^{\prime}_{l}$
$\displaystyle=(\frac{1}{d^{\prime}_{min}}-\frac{1}{d^{\prime}_{max}})/(D_{l}-1)$

where $D_{l}$ is the number of hypothesis planes for the current level,
$\tilde{d}_{l-1}$ is the depth prediction from level $l-1$, and
$v^{\prime}_{l},d^{\prime}_{min},d^{\prime}_{max}$ is the updated plane
interval, minimum and maximum depth respectively. Then the depth plane
hypothesis $d_{j,l}$ for level $l$ can be calculated based on Equ (13).

In the next step, we warp the feature map of each reference image to the
target image at each hypothesis plane. This warping is calculated using
coordinate mapping between the reference and target image, determined by
spherical disparity:

$C_{x}=0,\;C_{y}=\frac{cos(\theta)\times
baseline}{d_{j,l}}\times\frac{H}{\pi}$ (15)

where $H$ is the height of input feature map. Note that the range of $\theta$
is normalized into $[-\pi/2,\pi/2]$

Figure 11: Illustration of stereo matching network architecture with feature
extraction, spherical warping, cost volume construction, cost aggregation and
depth prediction.

##  Appendix C Experiments

###  C.1 View Synthesis Analysis

To analyze the correlation between image synthesis quality, image field of
view (FoV), and stereo baselines, a dataset consists of various combinations
of image FoV and baselines is required. Since there is no such benchmark
dataset available, we construct our own dataset by rendering 3D meshes from a
real-world dataset Stanford2D3D [3]. In our dataset, images are organized as
two parts, equirectangular projection (ERP) and perspective. For each 3D mesh,
we first generate the ERP dataset using Blender [12] to render panoramic image
sequence at virtual camera viewpoints sampled from a customized path. At each
viewpoint, we render RGB-D ERP images with cameras placed along the vertical
axis with baseline displacements of
$b\in\\{0m,0.01m,0.02m,0.04m,0.08m,0.16m,0.32m,0.64m\\}$. An illustration of
scene rendering is shown in Figure 7.

We collect results from over 1000 viewpoints, in total more than 8000 RGB-D
ERP images including original view point and 7 vertical baseline settings in
the ERP dataset. Using these rendered RGB-D ERP images, we then build the
perspective dataset given five FoV settings,
$\\{{64}^{\circ},{80}^{\circ},{96}^{\circ},{112}^{\circ},{128}^{\circ}\\}$.
The ERP dataset can also be considered as a special case of the FoV setting of
${360}^{\circ}$. Given a ERP RGB-D image and the FoV settings, we uniformly
sample $4\times 6$ pairs of $\theta$ and $\phi$ angles from the spherical
space. For each pair of $\theta$ and $\phi$ angles, we can define a homography
transformation $H$ from spherical space to perspective image space with the
FoV setting. By mapping the pixels from ERP image to perspective image using
transformation $H$, and filling the holes by image interpolation, the
perspective image is generated. In total, around 2 million perspective images
are collected. Figure 8 shows an illustration.

Using the constructed ERP and perspective datasets, we analyze the correlation
between image FoV, baseline displacement, and view synthesis quality. In
detail, we synthesize novel views for each viewpoint via Depth-Image-Based
Rendering (DIBR) [55] for both perspective images and ERP images with their
ground truth depth maps. To make a fair comparison, we stitch the synthesized
perspective images as ERP format and eliminate the image stitching error
during comparison. The error metric used in the comparison is the Mean Squared
Error(MSE).

Figure 9 shows both the qualitative and quantitative results of the
experiment. In Figure 9, we can see that synthesized ERP images in general
have the least errors and artifacts compared with perspective images at
different FoV settings. There are two major sources of errors in view
synthesis of perspective images: boundary effect and object occlusions. Unlike
perspective images, given ${360}^{\circ}$ horizontal and ${180}^{\circ}$
vertical FoV in one image, 360 images are free of boundary effect. From the
experiment, we can also observe that when the baseline increases, ERP and
large FoV images are less effected. We conclude that ERP images have
advantages in novel view synthesis, especially with large baselines. Such
advantage can furthermore benefit the stereo matching stage for better depth
prediction.

###  C.2 Ablation Studies

####  C.2.1 Convergence Analysis

Figure 12: Depth prediction error metrics of the coarse depth estimation(first
stage) and the final output(second stage) over epochs in terms of Abs Rel
(left) and log RMSE (right).

In Figure 12, we plot the depth error in the form of Abs Rel and RMSE(log) for
both the first stage and the final output. To demonstrate the advantage of
end-to-end training, we add the results from monocular depth estimation with
CoordNet [68] as backbone (stand-alone CoordNet) to the plot. The experiment
is conducted on Omnidirectional Stereo Dataset [68] where the left-down view
is used for training and testing. We can see that the coarse depth estimation
network converges fast at earlier epochs. This enables the second stage stereo
matching network to converge. Moreover, since our pipeline is en-to-end, the
gradient of the second stage flows back to the view synthesis stage and
facilitates the generation of better quality novel synthesized views. We
compare a stand-alone CoordNet with the CoordNet that is integrated in our
full pipeline. As shown in Figure 12, the stand-alone version has the largest
depth prediction error, which indicates that simultaneous training for
networks in two stage is beneficial for convergence.

####  C.2.2 Number of Synthesized Views

In Table 5, we show the impact of the number of synthesized views on
performance of stereo matching. The experiment is conducted on Omnidirectional
Stereo Dataset [68] where the left-down view is used for training and testing.
We use the ground truth depth to synthesize multiple novel views on different
baseline settings. Comparing with one novel view, more stereo constraints are
brought in when given two or more synthesized views. We observe that there is
a performance drop when using four views. As we explored in our analysis in
Section C.1, when the baseline becomes larger, it introduces more errors in
view synthesis. Thus, adding too many synthesized views could accumulate these
errors, which could lead to performance drop. Hence in this paper, we choose
three synthesized views as the default setting.

$\\#$ Synthesized Views(baseline(s)) |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  RMSElog$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---
1 View ($0.12m$) |  0.0849 |  0.0373 |  0.3180 |  0.1332 |  0.9267 |  0.9860 |  0.9961
1 View ($0.24m$) |  0.0845 |  0.0370 |  0.3164 |  0.1323 |  0.9274 |  0.9862 |  0.9963
2 Views ($-0.12m$, $0.12m$) |  0.0829 |  0.0364 |  0.3147 |  0.1311 |  0.9296 |  0.9864 |  0.9962
2 Views ($-0.24m$, $0.24m$) |  0.0842 |  0.0372 |  0.3187 |  0.1328 |  0.9274 |  0.9859 |  0.9961
3 Views ($-0.12m$,$0.12m$,$0.24m$) |  0.0830 |  0.0361 |  0.3129 |  0.1305 |  0.9302 |  0.9867 |  0.9963
3 Views ($-0.24m$,$0.24m$,$0.4m$) |  0.0839 |  0.0365 |  0.3141 |  0.1316 |  0.9288 |  0.9866 |  0.9962
4 Views ($-0.24m$,$-0.12m$,$0.12m$,$0.24m$) |  0.0834 |  0.0365 |  0.3152 |  0.1316 |  0.9287 |  0.9864 |  0.9962
4 Views ($-0.4m$,$-0.24m$,$0.24m$,$0.4m$) |  0.0840 |  0.0372 |  0.3175 |  0.1322 |  0.9281 |  0.9862 |  0.9961
Table 5: An ablation study of the number of synthesized views and the
baselines. All the views are generated along the vertical axis with predefined
baselines.

####  C.2.3 Number of Hypothesis Planes

Table 6 shows the ablation study of stereo matching network using different
cascade levels as well as different number of hypothesis planes. Adding
cascade level and planes improves the performance, but also increases the
computation cost. The experiment is trained and tested on Omnidirectional
Stereo Dataset [68] where bottom-up stereo pairs are used.

Cascade Level(s) |  Number of Hypothesis Planes |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  RMSElog$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---|---
1 |  $D=32$ |  0.0253 |  0.0222 |  0.2268 |  0.0686 |  0.9756 |  0.9874 |  0.9976
1 |  $D=48$ |  0.0236 |  0.0102 |  0.1860 |  0.0626 |  0.9854 |  0.9960 |  0.9984
1 |  $D=64$ |  0.0229 |  0.0087 |  0.1731 |  0.0606 |  0.9900 |  0.9969 |  0.9987
2 |  $D_{1}=32,D_{2}=16$ |  0.0186 |  0.0069 |  0.1491 |  0.0544 |  0.9920 |  0.9974 |  0.9989
2 |  $D_{1}=48,D_{2}=24$ |  0.0178 |  0.0064 |  0.1415 |  0.0519 |  0.9928 |  0.9976 |  0.9990
2 |  $D_{1}=64,D_{2}=32$ |  0.0166 |  0.0062 |  0.1419 |  0.0513 |  0.9933 |  0.9976 |  0.9989
Table 6: An ablation study of different number of cascade levels, and the
number of hypothesis planes. We see that two-level outperforms one-level
methods, and more hypothesis planes improve the stereo matching performance.

####  C.2.4 One-stage VS. Two-stage

In this section, we compare the performance between one-stage and two-stage
360 monocular depth estimation approach. Besides the vanilla one-stage
approach, we also add several one-stage approaches with techniques that
utilizing multi-tasking or additional refinement module to improve depth
quality. For multi-tasking, there are methods that combines normal and
boundary estimation [15], or semantic segmentation and layout prediction [64],
with depth estimation task. In this experiment, we train a multi-task depth
network with a shared weight encoder and two decoders for both depth and
normal maps prediction. For refinement, we append another CoordNet to the
first CoordNet as refinement network, which takes the original RGB and the
coarse depth estimation as input and outputs a refined depth map. The results
is shown in Table 7. Adding a refinement network boosts the performance by
7.8% (Abs Rel). Multi-tasking improves the performance by 15.9% (Abs Rel).
However, multi-tasking usually requires additional training labels which is
not suitable for many cases where only ground truth depth is available. Our
PanoDepth is able to outperform the vanilla CoordNet by 18.5%, which has the
best performance. Besides, since PanoDepth is model-agnostic, we can even add
multi-tasking to the coarse depth estimation network to further improve the
performance of the full pipeline.

Methods |  #params |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  RMSElog$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---|---
CoordNet |  6.1M |  0.1264 |  0.0888 |  0.4456 |  0.2084 |  0.8533 |  0.9558 |  0.9813
CoordNet with Refinement |  12.2M |  0.1165 |  0.0794 |  0.4186 |  0.1927 |  0.8668 |  0.9615 |  0.9852
CoordNet with multi-tasking |  7.2M |  0.1063 |  0.0695 |  0.4089 |  0.1877 |  0.8841 |  0.9677 |  0.9877
PanoDepth |  16.6M |  0.1030 |  0.0635 |  0.4001 |  0.1828 |  0.8821 |  0.9682 |  0.9877
Table 7: Performance comparisons between CoordNet, CoordNet with refinement, CoordNet with depth and normal prediction (multi-tasking), and our PanoDepth. PanoDepth is able to outperform the vanilla CoordNet by 18.5%, achieve the best performance comparing to other techniques such as adding refinement or multi-tasking. For PanoDepth stereo network setting, we use two-level cascade ($D_{1}=48,D_{2}=24$). Positions of stereo and baseline(s) |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  RMSElog$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---
V $0.24m$, V $-0.24m$ |  0.0553 |  0.0220 |  0.2561 |  0.0995 |  0.9625 |  0.9926 |  0.9977
H $0.24m$, H $-0.24m$ |  0.0937 |  0.0415 |  0.3496 |  0.1403 |  0.9057 |  0.9850 |  0.9966
H $0.24m$, V $0.24m$ |  0.0881 |  0.0365 |  0.3190 |  0.1339 |  0.9279 |  0.9863 |  0.9961
Table 8: Evaluations of multi-view stereo matching network on different horizontal and vertical spherical stereo settings. V represents vertical displacement, and H represents horizontal displacements. The vertical stereo setting (top row) achieves the best performance. Sampling method |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  RMSElog$\downarrow$ |  $\delta_{1}\uparrow$ |  $\delta_{2}\uparrow$ |  $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---
Random sampling |  0.0818 |  0.0410 |  0.3198 |  0.1204 |  0.9199 |  0.9893 |  0.9972
Uniform depth sampling |  0.0662 |  0.0334 |  0.2559 |  0.1065 |  0.9625 |  0.9922 |  0.9962
Uniform reverse depth sampling (proposed) |  0.0553 |  0.0220 |  0.2561 |  0.0995 |  0.9625 |  0.9926 |  0.9977
Table 9: Depth sampling methods comparison. We compare three sampling methods,
random sampling, uniform depth sampling, and uniform reverse depth sampling.
The uniform reverse depth sampling from our proposed Spherical Warping Layer
performs the best.

####  C.2.5 Spherical Stereo Model

To compare the horizontal and vertical stereo model performance, we use the
ground truth depth to generate synthesized views via DIBR with both horizontal
and vertical baselines as inputs. We then evaluate the multi-view spherical
stereo matching results using two synthesized views with different
combinations of spherical stereo model, including i) both horizontal, ii) both
vertical and iii) one horizontal + one vertical. As listed in Table 8, stereo
model with all vertical directions has the best performance. This is
consistent with the spherical geometry analysis in Section A. The experiment
is performed on Omnidirectional Stereo Dataset [68] where only left-down view
is used. The results are all collected at the 60th epoch as a fair comparison.

####  C.2.6 Depth Sampling Analysis

To validate the effectiveness of uniform sampling on reverse depth in our
proposed spherical warping layer, we compare it with other sampling strategy
in Table 9 including random sampling and uniform depth sampling. The numbers
of hypothesis planes $D_{1}$ and $D_{2}$ are 48 and 24 respectively. For
random sampling, we perturb a small Gaussian noise on the uniformly sampled
depth hypothesis. From Table 9, we see that our proposed uniform reverse depth
sampling has the best performance. The experiment is performed on
Omnidirectional Stereo Dataset [68] where only left-down view is used. The
results are all collected at the 60th epoch as a fair comparison.

Figure 13: Qualitative comparisons with the state-of-the-art approaches on
Stanford2D3D [3]. We show the results of RectNet proposed in Omnidepth [69]
(3rd column), BiFuse with Fusion [57] (4th column), and our PanoDepth (5th
column). We use red boxes to highlight the differences between predicted depth
maps.

###  C.3 Additional Qualitative Comparisons with State-of-the-art Algorithms

In Figure 13, we show additional qualitative comparisons between our method
and the state-of-art approaches [69, 57] on Stanford2D3d [3]. In Figure 14, we
show additional generalization capability comparisons between current state-
of-the-art BiFuse (with fusion) and our PanoDepth, with training on 360D [69]
and testing on a different real-world Laval HDR dataset [23] with no ground
truth depth. As can be seen from both figures, our method can more accurately
recover the scene structure, predict the depth with less artifacts, and
restore the object details, and has stronger generalization ability.

Figure 14: Generalization capability comparison between BiFuse (with fusion)
and our PanoDepth. We train the model on 360D [69] and test on a different
real-world dataset, Laval HDR [23], which does not have any ground truth
depth. Row 1 and 3 are the input images and the estimated depth maps, Row 2
and 4 are the reconstructed point clouds from the two methods. We can see that
PanoDepth is able to recover the structure of the room, capture the details
and sharp edges more accurately.