# PIV3CAMS: a multi-camera dataset for multiple computer vision problems and
its application to novel view-point synthesis

Sohyeong Kim Signal Processing Laboratory 5, EPFL, Switzerland Martin
Danelljan Computer Vision Laboratory, ETH Zurich, Switzerland  Radu Timofte
Computer Vision Laboratory, ETH Zurich, Switzerland  Luc Van Gool Computer
Vision Laboratory, ETH Zurich, Switzerland  Jean-Philippe Thiran Signal
Processing Laboratory 5, EPFL, Switzerland

###### Abstract

The modern approaches for computer vision tasks significantly rely on machine
learning, which requires a large number of quality images. While there is a
plethora of image datasets with a single type of images, there is a lack of
datasets collected from multiple cameras. In this thesis, we introduce Paired
Image and Video data from three CAMeraS, namely PIV3CAMS, aimed at multiple
computer vision tasks. The PIV3CAMS dataset consists of 8385 pairs of images
and 82 pairs of videos taken from three different cameras: Canon D5 Mark IV,
Huawei P20, and ZED stereo camera. The dataset includes various indoor and
outdoor scenes from different locations in Zurich (Switzerland) and Cheonan
(South Korea). Some of the computer vision applications that can benefit from
the PIV3CAMS dataset are image/video enhancement, view interpolation, image
matching, and much more. We provide a careful explanation of the data
collection process and detailed analysis of the data. The second part of this
thesis studies the usage of depth information in the view synthesizing task.
In addition to the regeneration of a current state-of-the-art algorithm, we
investigate several proposed alternative models that integrate depth
information geometrically. Through extensive experiments, we show that the
effect of depth is crucial in small view changes. Finally, we apply our model
to the introduced PIV3CAMS dataset to synthesize novel target views as an
example application of PIV3CAMS.

Keywords: Image processing, Big data, View interpolation, Deep learning, Novel
view synthesis

###### Contents

  1. 1 Introduction
    1. 1.1 Motivation
    2. 1.2 Contribution
  2. 2 Related Works
    1. 2.1 Datasets for Scene understanding
      1. 2.1.1 Single-image datasets
      2. 2.1.2 Paired datasets
      3. 2.1.3 Multi-modal datasets
      4. 2.1.4 Synthetic datasets
    2. 2.2 Novel view Synthesis
      1. 2.2.1 Geometry-based View Synthesis
      2. 2.2.2 Learning-based View Synthesis
  3. 3 Paired Image and Video dataset: PIV3CAMS dataset
    1. 3.1 Motivation
    2. 3.2 Overview of the dataset
    3. 3.3 Setup
      1. 3.3.1 Hardware: Selection of cameras
      2. 3.3.2 Hardware: Rig design
      3. 3.3.3 Synchronization: Triggering multiple cameras at the same time
      4. 3.3.4 Calibration: Stereo camera calibration
    4. 3.4 Data Acquisition
      1. 3.4.1 Locations and times
      2. 3.4.2 Image data collection and processing
      3. 3.4.3 Video data collection and processing
    5. 3.5 Data Analysis
      1. 3.5.1 Collection time
      2. 3.5.2 RAW image data
      3. 3.5.3 Object instance analysis
  4. 4 Novel View Synthesis using RGB-D images
    1. 4.1 Regenerated Baseline Network
      1. 4.1.1 Encoder and Decoder modules
      2. 4.1.2 Transforming latent vector
      3. 4.1.3 Depth branch
      4. 4.1.4 Pixel branch and Mask branch
      5. 4.1.5 Fusion
      6. 4.1.6 Training
    2. 4.2 Variation Networks
      1. 4.2.1 Variation 1: Without depth decoder
      2. 4.2.2 Variation 2: Without both depth decoder and mask decoder
    3. 4.3 Experiments
      1. 4.3.1 Datasets: ShapeNet
      2. 4.3.2 Datasets: KITTI
      3. 4.3.3 Datasets: PIV3CAMS
    4. 4.4 Metrics
      1. 4.4.1 Mean Absolute Error (L1)
      2. 4.4.2 Structural SIMilarity (SSIM) Index
    5. 4.5 Training Details
  5. 5 Results and Discussion
    1. 5.1 Novel view synthesis for Synthetic Objects
      1. 5.1.1 Influence of using ground-truth depth in view synthesizing
      2. 5.1.2 Influence of using visibility mask in view synthesizing
    2. 5.2 Novel view synthesis for Real Scenes
    3. 5.3 Novel view synthesis on the PIV3CAMS dataset
    4. 5.4 Limitations
      1. 5.4.1 The performance of the pixel branch
      2. 5.4.2 The sparsity of the depth map
  6. 6 Conclusion

## 1\. Introduction

###  1.1 Motivation

Computer vision is a field of artificial intelligence that aims to develop
systems that can interpret and understand the visual world as the human visual
system does. One of the earliest well-known projects in the computer vision
field is the summer computer vision project at MIT in 1966[1], intending to
make a computer explain what it saw. Early attempts at scene understanding
involved extracting edges[2], labeling lines[3, 4], representing objects as
interconnections of smaller surfaces, feature-based correspondence
algorithms[5, 6], optical flows[7], and motion estimation[8]. Since then, much
research has been done on the rigorous mathematical analysis of images and
quantitative aspects of computer vision. Most modern studies on computer
vision apply linear algebra, projective and differential geometry, as well as
statistics in conjunction with machine learning techniques and complex
optimization frameworks. For example, Convolutional Neural Networks
algorithms[9] have already shown promising results with superior accuracy,
even surpassing human-level accuracy on some tasks[10].

Several factors have converged to bring about a recent breakthrough in the
computer vision field, including the advancement of hardware and the amount of
data available. The growth of computing power has enabled the iterative
learning process with deep layers of neural network architectures. The built-
in cameras in smartphones and smaller digital cameras have made it easier to
take photos and videos, resulting in a tremendous amount of data becoming
accessible for research. The development of other types of image acquisition
hardware, such as structured-light 3D cameras, radar imaging, Light Detection
and Ranging (LIDAR) scanners, and magnetic resonance imaging, has given access
to different types of images beyond RGB images, enriching computer vision
research.

Over time, computer vision has partly merged with other closely related fields
such as image processing, photogrammetry, and computer graphics, broadening
its spectrum of potential applications. These applications of computer vision
include autonomous navigation, robotic control processes, 3D scene
reconstruction, novel view synthesis, virtual reality (VR), augmented reality
(AR), and vision-based human-computer interfaces. As these applications become
more popular in our lives, much research in computer vision has already
started and produced notable progress.

As there is a wide range of applications for computer vision, the organization
of its system is highly application-dependent. Not only the algorithm to
complete a task but also the means of image acquisition and pre-processing
those images are related to the application’s specific objective. For example,
for large-scale object detection, classification, and segmentation tasks, the
ImageNet[11] and MS COCO[12] datasets can be perfect to use. On the other
hand, for specific tasks such as house number detection, one can use SVHN
(Street View House Numbers)[13] for better performance. Some datasets, such as
KITTI[14], have pre-processed the collected data in various formats so that it
can be used for different tasks in applications for autonomous vehicles.

Much of the recent approaches for computer vision greatly rely on machine
learning, which drives the need for benchmark datasets for training and
evaluation. While there is a plethora of image datasets with a single type of
images, there is a lack of datasets collected from multiple imaging sensors.
To address this gap, we have collected a paired image dataset. We have also
chosen novel view synthesis as an example of a computer vision problem to
investigate. Synthesizing new views of a scene becomes more critical as it is
highly related to many compelling applications such as scene reconstruction,
AR/VR, free-viewpoint image-based rendering, and more.

###  1.2 Contribution

This thesis concerns two core parts of the modern computer vision pipeline,
namely data collection and system design. For the first task of the thesis, we
introduce Paired Image and Video data (Fig 1) from three CAMeras (PIV3CAMS)
collected for multiple computer vision tasks. Next, investigating novel view
synthesis with a learning-based approach is our second aim for the thesis.
Specifically, we wanted to experiment with how using depth information would
affect the performance of view synthesis.

Figure 1: Examples from PIV3CAMS dataset The upper row shows images from
different cameras with relative resolution sizes(Canon: 6720x4480, P20:
5120x3480, and ZED:2208x1242). The lower row shows the enlarged RGB-D image
and a confidence map from ZED stereo camera.

As the first main contribution of this thesis, the PIV3CAMS dataset is
proposed. It consists of images and videos from three different cameras
(Huawei P20, Canon 5D Mark IV, and ZED stereo camera). These cameras are
attached to a static rig, and we have collected data from two different
cities: Zurich, Switzerland, and Cheonan, South Korea. The purpose of this
dataset is to push forward the development of computer vision algorithms such
as view interpolation, image matching, and image/video super-resolution. It
includes RGB-D images and videos, depth confidence maps, and positions of the
stereo camera for videos. Compared to other existing paired datasets, such as
the KITTI[14] dataset, which is mainly comprised of road and traffic scenes,
our dataset includes various scenes for both outdoor and indoor settings
obtained using hand-held devices.

As the second main contribution, this thesis investigates the problem of novel
view synthesis (Fig 2). The goal of the novel view synthesis problem is to
generate a new image from a different point of view from the given source
image. While reproducing a state-of-the-art novel view synthesis
algorithm[15], we investigated the effect of utilizing depth information for
the view generation task. We have reorganized their networks by replacing the
depth branch with a ground truth depth map to warp the source view. To see how
sparse depth affects performance, we have also trained the networks with both
dense and sparse depth maps. We used three different datasets to validate our
approach: SYNTHIA[16], KITTI[14], and PIV3CAMS.

Figure 2: Novel view Synthesis task The goal of this task is to synthesize the
target view from the source view.

In summary, this work makes the following contributions: (1) Introduction of a
new image and video dataset aimed at multiple computer vision tasks. (2)
Reproduction of a state-of-the-art novel view synthesis algorithm. (3)
Investigation of the effect of depth information in the novel view synthesis
task by creating a variant network model from the baseline network model.

In Section 2, we will discuss existing datasets that aim to tackle similar
tasks as PIV3CAMS and related works on novel view synthesis. In Section 3, we
will provide the technical details of the collection process and raw data of
PIV3CAMS. We will also describe how the raw data is processed and rendered for
specific tasks such as novel view synthesis and image and video super-
resolution. In Section 4, we will explain the baseline network for the novel
view synthesis task and the variations we made to the networks to use depth
information. In Section 5, we will show the details of training and testing
the networks, and we will discuss the results. Finally, in Section 6, the
conclusion and future work for the thesis will be presented.

## 2\. Related Works

###  2.1 Datasets for Scene understanding

In the last decade, many image datasets have been released, and they have
played a significant role in computer vision research. This section discusses
some of these datasets according to their data organization. Most of the
datasets available for research are primarily focused on scene understanding,
which includes tasks such as object detection and recognition, image
classification, and segmentation. While some datasets, like DPED[17], aim to
enhance image quality, we have included this dataset in our discussion because
its data collection setup was similar to ours.

####  2.1.1 Single-image datasets

The ease of capturing and annotating single RGB images has led communities to
release large-scale RGB image-only datasets.

PASCAL VOC[18], CIFAR-10, and CIFAR-100[19] were among the earliest image
datasets used for image classification tasks. The images were labeled in
several classes: 20, 10, and 100 for PASCAL VOC, CIFAR-10, and CIFAR-100,
respectively. For training and validation, images in the PASCAL VOC dataset
were segmented, and objects within the images were annotated with bounding
boxes.

ImageNet[11] is one of the largest image datasets, providing more than 14
million images, about 1 million of which are annotated with bounding boxes. It
also provides descriptive words and SIFT features for the images.

More recent datasets such as COCO[12] and Open Images Dataset V5[20] provide
more complex annotations beyond just segmentation and bounding boxes to allow
for cross-task training and analysis. For example, COCO provides panoptic
segmentation by assigning a semantic label and instance ID for each pixel of
an image. It also provides key points for people’s poses if there are person
instances. This allows data to be used for person detection and localization
of key points for poses. The OpenImages Dataset V5 annotated all images
already containing bounding box annotations with visual relationships between
objects and other objects in the images. These visual relationship annotations
pushed scene understanding further into ’images to words.’

Some datasets consist of single sets of videos with segmentation annotations.
DAVIS[21] and YouTube-VOS[22] provide object segmentation in the video frames.
These datasets aim for video instance segmentation tasks.

The roles of single image and video datasets are very limited to object
detection, classification, and recognition since they only provide 2D-limited
information of the scenes. Table 1 summarizes common datasets of single images
or videos.

Dataset |  Year |  # Images |  Video length
(in minutes) |  # Categories |  # Objects
annotated
---|---|---|---|---|---
PASCAL VOC[18] |  2010 |  500,000 |  - |  80 |  27,450
CIFAR-10[19] |  2009 |  60,000 |  - |  10 |  -
CIFAR-100[19] |  2009 |  60,000 |  - |  100 |  -
ImageNet[11] |  2014 |  14,197,122 |  - |  200 |  534,309
COCO[12] |  2015 |  2,500,000 |  - |  80 |  886,284
OpenImages[20] |  2017 |  9,178,275 |  - |  600 |  15,440,132
DAVIS[21] |  2017 |  10,459 |  5.17 |  - |  376
Youtube-VOS[22] |  2018 |  - |  334.81 |  94 |  197,272
Table 1: Comparison of the single images and video datasets used for scene
understanding tasks

####  2.1.2 Paired datasets

With the advancement of image acquisition hardware and sensors, it has become
possible to capture images and videos of the same scene from different
sensors. The visual data acquired from various sensors is formed into paired
data. This additional paired data, on top of a single RGB image, provides
researchers with extra information about the scenes. Table 2 summarizes common
datasets of paired images or videos.

One of the most popular ways of collecting paired images is using depth-
sensing cameras such as Microsoft Kinect, Intel RealSense, and Asus Xtion. The
paired data obtained from the depth-sensing camera is commonly called an
’RGB-D’ image. The advantage of having depth information in an image is that
it enables the extraction and utilization of the geometrical structure of
scenes.

Some datasets are labeled or annotated with bounding boxes to provide a more
general understanding of the world. NYU Depth V1[23] and NYU Depth V2[24]
labeled images with dense multi-class labels. The Cornell-RGBD-Dataset[25]
labeled objects in a point cloud format. The Berkeley 3-D Object (B3DO)[26]
and RGB-D Scenes[27] datasets annotated objects with bounding boxes.
SUNRGB-D[28] merged the RGB-D images from NYU Depth V2, B3DO, and SUN3D[29]
and annotated the whole dataset with 2D polygons and 3D bounding boxes with
accurate object orientations, as well as a 3D room layout and scene
categories.

There are also some other datasets that go beyond the single, static-frame
modality. SUN3D[29] and ViDRILO[30] collected RGB-D data in video format.
Specifically, the SUN3D dataset includes eight annotated sequences consisting
of RGB-D images, camera poses, object segmentation, and point clouds
registered into global coordinate frames.

While the majority of RGB-D datasets capture data in indoor environments, the
DIML/CVL RGB-D dataset[31] comprises both indoor and outdoor RGB-D images.
Compared to our dataset, the DIML/CVL RGB-D dataset consists of a much larger
number of data, with 1M indoor images and 1M outdoor images. However, unlike
our dataset, which includes RGB-D video data, their dataset is limited to
image data.

Other than RGB-D datasets, another type of paired dataset using different
cameras exists. DPED[17] captured natural scenes using four different cameras
by triggering them simultaneously. They used an iPhone 3GS, BlackBerry
Passport, Sony Xperia Z, and Canon 70D DSLR to collect the data. This dataset
is useful for training image enhancement algorithms, for example, to improve
image quality from low to DSLR quality. Similarly, our dataset could be used
for image enhancement tasks and further extended to video enhancement tasks.

Dataset |  Year |  # Images |  Video |  Sensors |  Scene type
---|---|---|---|---|---
NYU V1[23] |  2011 |  2283 frames |  - |  Kinect |  indoor
NYU V2[24] |  2012 |  1449 frames
from 464 scenes |  - |  Kinect |  indoor
Cornell-RGBD dataset[25] |  2011 |  52 scenes |  - |  Kinect |  indoor
B3OD[26] |  2011 |  849 frames
from 79 scenes |  - |  Kinect |  indoor
RGB-D Scenes[27] |  2011 |  8 scenes |  - |  Kinect |  indoor
SUN RGB-D dataset[28] |  2015 |  10,335 frames |  - |  Kinect v1, Kinect v2,
Intel RealSense,
Axus Xiton |  indoor
SUN3D[29] |  2013 |  8 scenes |  o |  Kinect |  indoor
ViDRILO[30] |  2015 |  22,454 frames
from 5 scenes |  o |  Kinect |  indoor
DIML/CVL RGB-D dataset[31] |  2018 |  1M frames (indoor),
1M frames (outdoor) |  - |  Kinect v2, ZED stereo, Built-in Streo |  indoor, outdoor
DPED[17] |  2017 |  16,291 frames |  - |  iPhone 3GS, BlackBerry Passport, Sony Xperia, Canon 70D DSLR |  indoor
Table 2: Comparison of the paired images and video datasets used for scene
understanding tasks

####  2.1.3 Multi-modal datasets

The scene understanding problem in computer vision has become more important
with the rise of autonomous vehicles. Due to this specific application
setting, several image and video datasets with a view from a car have been
released, as summarized in Table 3. These datasets typically comprise images
from multiple cameras, range sensor (lidar, radar) data, and GPS/IMU data.
Multi-modal datasets are expensive to collect and annotate due to the
difficulties of integrating, synchronizing, and calibrating multiple sensors.

KITTI[14] was the pioneering multimodal dataset, providing dense point clouds
from lidar sensors as well as gray and color images and GPS/IMU data. It
offers their raw dataset with various annotations and settings so that it can
be used for many computer vision tasks.

The recently released H3D dataset[32] includes 160 crowded traffic scenes with
a total of 1 million labeled instances in 27k frames. It provides full
360-degree object annotations in point clouds, whereas the KITTI dataset
annotates objects in front of the car. However, all cameras in H3D are front-
facing, meaning that image data only covers 180 degrees. Additionally, both
KITTI and H3D datasets do not provide nighttime data.

The KAIST multispectral dataset[33] is a multimodal dataset for driving
scenarios that provides various illumination conditions, including nighttime.

The Cityscapes dataset[34] also collected urban street scenes in 50 different
cities. It provides high-quality pixel-level annotations for 5k images and
coarse annotations for 20k images.

BDD100k[35] and Apolloscape[36] released much larger datasets compared to
other multimodal datasets, containing 100k and 144k RGB images, respectively.
BDD100k also contains images captured in different weather conditions and
illumination settings.

It is true that most of the multimodal datasets target the specific
application of autonomous driving, so their visual data are limited to road
scenes. Therefore, diversity in captured scenes is needed for broader
applications.

Dataset |  Year |  # scenes |  # images |  sensors |  Locations
---|---|---|---|---|---
KITTI[14] |  2012 |  22 |  150,000 |  2 Grayscale camera, 2 RGB cameras, 4 Varifocal lenses, Lidar, GPS/IMU |  Karlsruhe
H3D[32] |  2019 |  160 |  83,000 |  3 RGB cameras, Lidar, GPS/IMU |  San Francisco
KAIST[33] |  2018 |  - |  89,000 |  RGB-thermal camera, RGB stereo camera, Lidar, GPS/IMU |  Daejeon
Cityscapes[34] |  2016 |  - |  25,000 |  Stereo camera, GPS/IMU, Thermometer |  Germany
BDD100k[35] |  2017 |  100,000 |  100M |  RGB-thermal camera, RGB stereo camera, Lidar, GPS/IMU |  NewYork,
San Francisco
ApolloScape[36] |  2018 |  - |  144,000 |  4 RGB cameras, 2 laser scanners, GNSS/IMU |  China
Table 3: Comparison of the multi-modal datasets used for scene understanding
tasks

####  2.1.4 Synthetic datasets

While real-world datasets are useful for enabling computer vision applications
to work well in real-world scenarios, collecting all the data is very
laborious. To avoid the difficulties in collecting real scenes, several
synthetic datasets were generated via simulation.

ShapeNet[37] and ObjectNet3D[38] are large-scale databases of 3D objects
classified into 3,135 and 100 categories, respectively. ShapeNet provides many
semantic annotations for each 3D model, such as consistent rigid alignments,
parts, and physical sizes.

SYNTHIA[16], CARLA[39], and Virtual KITTI[40] simulated virtual urban areas
using game engines and acquired photo-realistic frames rendered from multiple
viewpoints in the simulated world. CARLA allows flexible configuration of the
agent’s sensor suite so that users can extract numerous frames of the same
scene using different pseudo-sensors.

These synthetic datasets also provide a large amount of annotated images and
measurements without requiring too much human labor. However, a domain gap
still exists between synthesized images and real ones. In general, models
learned in real scenarios generalize better in real-world applications.

###  2.2 Novel view Synthesis

Novel view synthesis is the task of synthesizing new views from a given source
image. Its practical applications range from computer vision and computer
graphics to virtual/augmented reality. The novel view synthesis problem aims
to mimic humans’ ability to build a mental understanding of the 3D properties
of scenes from just 2D components. Humans can learn to combine and process all
the information, such as viewpoints and shape understandings, to predict a 3D
statistical understanding of the world. In computer vision, the integration of
data from 2D images is not trivial, and understanding each type of information
is tackled separately. In this section, we will introduce past research that
has tackled novel view synthesis tasks.

####  2.2.1 Geometry-based View Synthesis

One of the classical approaches to view synthesis is directly estimating the
underlying 3D structures of the object or scene and applying geometrical
transformations to the pixels in the input[41],[42],[43],[44],[45]. Given
abundant source data from multiple views, one can interpolate the views by
exploiting the 3D geometry of the scene[46]. [47] and [48] predicted depth
maps from the stereo images created by small motion and then reconstructed the
3D structure of scenes. [49] and [50] generated new views from arbitrary
camera positions without depth information but with light field arrays
rendered from images. This approach is successful as it can accurately
transfer original colors, textures, and local features from the source view to
the target view. However, it is still difficult to hallucinate new parts due
to dis-occlusion, and it is unable to recover the desired target views with
only a handful of images.

####  2.2.2 Learning-based View Synthesis

With the emergence of neural networks and the introduction of large-scale
RGB-D datasets[14] and synthetic 3D model datasets[16],[37], learning-based
approaches have been applied to tackle novel view synthesis problems.

[51], [52], and [53] built networks that learn how to generate pixels in the
target views. Directly regressing pixels can generate structurally consistent
results; however, it is prone to generating blurry results. To overcome this
blurriness, [54], [55], and [56] tried to predict the flows occurring from
view changes. While these approaches could generate realistic textures, they
were unable to generate regions not present in the source images. Recent
research has merged these two concepts, pixel generation and flow prediction,
to generate the target views [57], [58], and [59]. They also used generative
adversarial networks (GANs) to generate target views that are perceptually
better than previous works.

Furthermore, some novel view synthesis frameworks such as [52], [58], and [60]
aggregated multiple source views to derive the 3D statistical information of
the objects or scenes. Aggregating multiple source views is only possible when
there are images from numerous views of the same scene in the dataset.
Additionally, it is challenging to incorporate understandings from various
source images. On the other hand, [57] and [59] used a single source image to
generate a target view by modeling auto-encoder networks to extract
geometrically relevant features such as depth and masks. Then, by explicitly
transforming geometric information, they could derive images with target
views.

## 3\. Paired Image and Video dataset: PIV3CAMS dataset

In this chapter, we will provide a detailed description of the newly collected
dataset, Paired Image and Video from 3 Cameras (PIV3CAMS) dataset. The primary
motivation for creating this dataset is to promote applications related to
paired images based on deep learning. We will describe how the data was
collected and processed. We will then present the analysis results of the
collected images and videos and propose some computer vision tasks where our
dataset can be utilized.

###  3.1 Motivation

With the increasing number of smartphone users in recent years, more and more
people are taking pictures and videos with their smartphones. Even though
advancements in smartphone camera lenses and image processing technology have
minimized the gap in image and video quality between smartphones and DSLR
cameras, there is still a need for better quality smartphone images and
videos. At the same time, recent breakthroughs in AR/VR technology have
produced many appealing applications in fields such as healthcare, education,
e-commerce, and manufacturing. Many of these applications are facilitated by
the availability of depth information during training and/or testing. Inspired
by these needs, we have decided to create a dataset with various environments
that could support different applications: image and video enhancement, novel
view synthesis, and scene matching.

  * When designing and collecting a dataset, there are several important aspects to consider:

  * •

Selection of cameras \- What kinds of cameras should be used to cover the
different computer vision applications we are motivated by? Which models
should we choose?

  * •

Rig design \- How do we fix multiple cameras together?

  * •

Synchronization \- How are we going to trigger all cameras at the same time?

  * •

Calibration \- Which tools are we going to use to calibrate the cameras?

  * •

Locations \- Where do we collect the images and videos to ensure a variety of
scenes?

  * •

Capture mode \- Which modes should we use when taking images and videos?

  * •

Resolution \- At what resolution should we capture the images and videos?

These aspects are carefully addressed in this section. Please refer to the
following subsections for further details.

###  3.2 Overview of the dataset

The dataset consists of over 8,000 sets of paired images and 82 different
video scenes acquired by three different cameras: Huawei P20, Canon 5D Mark
IV, and ZED stereo camera. This dataset was collected over two months, from
June to July 2019, in Zurich (Switzerland) and Cheonan (South Korea), both
indoors and outdoors. Unlike existing outdoor data using stereo cameras
mounted on a moving vehicle, ours was taken using handheld stereo cameras.
Table 4 and Table 5 summarize the details of the image and video collection
settings, respectively. Figures 3 and 4 show subsets of the dataset as
examples.

|  Huawei P20 |  Canon Mark IV 5D |  ZED stereo
---|---|---|---
Color Resolution |  5120 x 3840 (4:3) |  6720 x 4480 (3:2) |  2208 x 1242 (16:9)
Depth Resolution |  |  |  2208 x 1242 (16:9)
Range of Depth |  |  |  0.5m to 20 m
Focus Mode |  Auto focus |  Auto focus |  Auto focus
Raw file Format |  DNG |  CR2 |  SVO
Data Acquisition |  RGB images,
RAW files |  RGB images,
RAW files |  Rectified left and right RGB images, Left and right depth images, Left confidence maps, Calibration parameters, RAW files
Collection Area | Zurich, Choenan, Indoor, Outdoor
Table 4: Summary of PIV3CAMS dataset settings for image

Figure 3: The samples from PIV3CAMS image dataset

|  Huawei P20 |  Canon Mark IV 5D |  ZED stereo
---|---|---|---
Color Resolution |  3840x2160 (16:9) |  1920 x 1080 (16:9) |  1920x1080 (16:9)
Depth Resolution |  |  |  1920x1080 (16:9)
Range of Depth |  |  |  0.5m to 20 m
FPS |  29.76 |  29.97 |  30 (10 fps for depth video)
Video Codec |  H.265(HEVC) |  H.264(AVC) |  Lossless(PNG)
Data Acquisition |  RGB Videos |  RGB Videos |  RGB videos(rectified left),
Depth videos(left), Calibration parameters, Poses of the camera
Video lengths | between 25 and 50 seconds
Collection Area | Zurich, Indoor, Outdoor
Table 5: Summary of PIV3CAMS dataset settings for videos

Figure 4: The frame samples from PIV3CAMS video dataset

###  3.3 Setup

####  3.3.1 Hardware: Selection of cameras

We aimed to select cameras that differ in their image and video quality to
cover applications such as image and video enhancement. At the same time, we
wanted our dataset to provide 3D information about the scenes. Therefore, we
selected three different cameras: Huawei P20 (P20), Canon 5D Mark IV (Canon),
and ZED stereo camera (ZED), each representing a smartphone camera, DSLR
camera, and 3D camera, respectively.

We wanted to use a smartphone well-known for its camera performance. There
were many models we could choose for smartphones: Samsung Galaxy series, Apple
iPhones, and more. Among them, we decided on the Huawei P20 since it was one
of the top-tier smartphones at the time of this thesis and is known for its
good camera.

Similarly, we wanted to use a high-performance DSLR camera, so we chose the
Canon 5D Mark IV. It is a professional-grade 30.1-megapixel full-frame DSLR
camera and was one of the best DSLR cameras at the time of this work.

The difference in sensor size, image processor, and lenses makes the DSLR
camera special. Notably, the sensor size of the Canon is 36.0x24.0mm, about 20
times larger than the sensor size of the P20, which is 1/1.7-inch (7.6x5.7mm),
and 35 times larger than the sensor size of the ZED, which is 1/3-inch
(4.8x3.6mm). The bigger the sensor size, the better the low-light sensitivity,
depth of field, and lower diffraction the camera has.

For the 3D camera, we chose the ZED due to its high resolution and high frame
rate for capturing 3D images and videos. Compared to other well-known 3D
cameras such as Microsoft Kinect v2 and Intel RealSense D415/D435, the ZED has
a much greater depth-sensing range, up to 20 meters indoors and outdoors.
Additionally, it has better light sensitivity compared to active Time-of-
Flight 3D cameras like the Kinect.

####  3.3.2 Hardware: Rig design

To fix the position of the cameras, we printed a customized rig with a 3D
printer and mounted the cameras on it, as shown in Figure 5. These were then
attached to a tripod for stabilization. A laptop (Dell XPS 15-9570) with an
Intel Core i7-8750H CPU (12 x 2.20GHz), 16GB DDR4 RAM, and a 4GB GPU (GeForce
GTX 1050 Ti) was used throughout the data collection.

Figure 5: Rig setup for cameras. (Left) Front view of the 3D printed rig with
cameras mounted on it. (Right) Side view of the rig with cameras mounted on
it.

Figure 6: Overall setup for data collection. (Left)All scenes were captured
steadily with a stationary tripod. The collector used a mouse to trigger
cameras while carrying the laptop. (Right) The Canon and P20 cameras were
connected to a USB hub, while the ZED camera was connected directly to the
laptop.

To achieve precise multi-sensor data alignment between cameras and to make
collection easier, we developed scripts that trigger multiple cameras
concurrently. The script runs on Linux OS (Ubuntu 18.04.3 LTS) and requires
ZED SDK version 2.8 with the CUDA 10 library. With a click of the mouse’s left
button, the script triggers the shooting of the three cameras connected to the
laptop.

Due to a lack of USB ports on the laptop, we connected the P20 and Canon to
the laptop using a USB hub (Fig 6). The ZED camera was connected directly to
the computer through a USB 3.0 port, as the USB hub could not source enough
power to run the ZED camera.

####  3.3.3 Synchronization: Triggering multiple cameras at the same time

We tested the cameras’ exposure time using a timer and achieved an average
time difference of less than 0.5 seconds, which can be regarded as
instantaneous exposure. Figure 7 shows the images testing the synchronization.

Figure 7: Synchronization test. Testing simultaneous camera triggering using
the script. Images are captured from Canon, P20, ZED right, and ZED left (left
to right, top to bottom). Over 20 tests were conducted to average
synchronization time differences for capturing images and videos.

####  3.3.4 Calibration: Stereo camera calibration

We performed the calibration for the ZED camera using the calibration toolbox
provided with the ZED SDK. This calibration app works by having the ZED camera
recognize checkerboard patterns shown on a monitor (Fig 8). Calibration is
completed after moving the ZED camera further back and tilting it in each
direction through several steps. The calibrated parameters are used to rectify
the raw images from the ZED camera and compute the depth map of the scenes.
These calibrated parameters are included within our dataset.

Figure 8: Calibration of the ZED stereo camera using ZED SDK. The goal is to
align the blue dot (ZED camera view) with the red dot.

###  3.4 Data Acquisition

####  3.4.1 Locations and times

The PIV3CAMS dataset was obtained using cameras mounted on a tripod, capturing
various outdoor and indoor scenes from different locations in Zurich
(Switzerland) and Cheonan (South Korea), as shown in Figure 9.

As mentioned in section 3.1, we aimed to ensure a variety of scenes in our
dataset. Therefore, we made a list of scene types before choosing the
locations. Some examples from the list include: business area, residential
area, playground, construction site, park, lake, cafeteria, office, and gym.
We then carefully selected areas within the cities for shooting to minimize
overlap between scene types. To ensure diversity in the dataset, we collected
images in the early morning, afternoon, evening, and at night.

Figure 9: Locations of data collection. The map shows data collection areas in
Zurich, Switzerland (top) and Cheonan, South Korea (bottom). Orange-shaded
areas indicate where images were taken, and red pins mark video recording
locations.

####  3.4.2 Image data collection and processing

Capture mode and resolution We captured images in the AUTO mode of each camera
since it is the most commonly used mode. We also set the image resolution to
the maximum achievable for each camera for the best quality pictures.

The image resolutions are 6720$\times$4480, 5120$\times$3840, and
2208$\times$1242 for Canon, P20, and ZED, respectively. For the Canon and P20,
we saved both the RAW files and JPEG-compressed RGB images. For the ZED, we
recorded images in Stereolab’s SVO format, which contains extra metadata such
as timestamps and sensor data. We chose the LOSSLESS (PNG) compression mode
for recording the SVO files.

Processing After the collection, we pre-processed the dataset to make it
suitable for research. First, we processed the ZED’s SVO files to extract
images. Using the ZED API, we loaded recorded SVO files and extracted left and
right RGB images, left and right depth images, depth confidence maps, and
other calibration parameters. When extracting depth images, we chose the ULTRA
mode, which offers the highest depth range and better preserves Z-accuracy
along with the sensing range (from 0.5 to 20 meters). We stored depth images
in millimeters using a 16-bit PNG format. Next, we manually sorted out pairs
of photos corrupted by blurriness, light reflection, and incorrect color
mapping.

Data format Here, we describe the structure and format of the image dataset.
There are a total of 8,385 pairs of images, and the total size of the dataset
is over 900GB (approximately 300GB for compressed images and 600GB for RAW
files). In addition to the image files, we provide the timestamps for when
each pair of images was taken and the calibration parameters for the ZED
camera.

.1 IMAGE_DATA. .2 Compressed. .3 CANON. .4 Canon_[6digits].JPG. .3 P20. .4
P20_[6digits].jpg. .3 ZED. .4 color. .5 left. .6 ZED_left_[6digits].png. .5
right. .6 ZED_right_[6digits].png. .4 depth. .5 left. .6
ZED_depth_left_[6digits].png. .5 right. .6 ZED_depth_right_[6digits].png. .4
confidence. .5 ZED_confidence_map_[6digits].png. .1 IMAGE_DATA. .2 RAW. .3
CANON. .4 Canon_[6digits].CR2. .3 P20. .4 P20_[6digits].dng. .3 ZED. .4
ZED_[6digits].svo. .3 calib.txt. .3 time.txt.

####  3.4.3 Video data collection and processing

Capture mode and resolution Similar to the image collection, we captured
videos using the DEFAULT video recording modes of each camera. We set the
frame rate to 30 frames per second (fps) and configured the resolution to the
maximum achievable at 30 fps. The video frame resolutions are
1920$\times$1080, 3840$\times$2160, and 1920$\times$1080 for Canon, P20, and
ZED, respectively. For the Canon and P20, the videos are compressed using
H.264 and H.265 coding methods, respectively. For the ZED camera, the video is
recorded in SVO format with LOSSLESS (PNG) mode.

We recorded the videos in two different settings: one with a static camera
position and the other with a moving camera position. In the first setting,
the tripod was fixed in one spot, and the cameras were moderately panned to
minimize fluctuations from walking. In the second setting, the videos were
recorded while walking to mimic a natural recording situation. Although we
synchronized the triggering time of all cameras to within 0.5 seconds, this
time difference can result in up to a 15-frame misalignment in the videos. To
reduce frame shifts between videos, we displayed a timer at the beginning of
each video.

Processing Similar to the image collection, we pre-processed the collected
videos. First, we extracted frames from each camera’s videos. We extracted
ZED’s SVO files into both RGB and depth frames from the left lens of the ZED
camera. For depth images, we used the ULTRA mode as we did for the image sets,
with a sensing range from 0.5 to 20 meters. Depth frames are stored in
millimeters using a 16-bit PNG format. Canon’s and P20’s videos were also
converted into frames. We then matched the starting frame for each pair of
scenes by checking the timer shown at the beginning of the videos. Depending
on the length of the original videos, we cropped the recordings to have
lengths between 25 and 50 seconds.

We also retrieved the ZED camera poses from the SVO file. The ZED uses a
three-dimensional Cartesian coordinate system (X, Y, Z) with a right-handed,
y-down configuration, and coordinates in millimeters. The motion of the ZED’s
left eye is described in the absolute real-world space, called the World
Frame, located where the ZED first started motion tracking; the very first
frame of the video. (Fig 10). The pose data of each frame, including rotation
and translation values, is stored in a text file.

Figure 10: Coordinate system of ZED

Data format We describe the structure and format of the video dataset below.
There are a total of 82 pairs of videos with different scenes. The total size
of the dataset is over 1.2TB (around 700GB for extracted frames and 500GB for
raw videos). In addition to the raw videos and their frames, we also provide
pose data of the ZED’s left lens for each scene.

.1 VIDEO_DATA. .2 Frames. .3 set[2digits]. .4 canon_frames. .6 [6digit].png.
.4 p20_frames. .6 [6digit].png. .4 zed_frames. .5 left. .6 [6digit].png. .5
depth. .6 [6digit].png. .5 calib.txt. .5 pose.txt. .1 VIDEO_DATA. .2 RAW. .3
Canon_[2digit].MOV. .3 P20_[2digit].mp4. .3 ZED_[2digit].svo.

###  3.5 Data Analysis

We analyze the statistics of the collection time, color distribution, density
of the depth map, and the number of object instances in PIV3CAMS. We expect
this analysis to give readers an understanding of how the dataset is composed
and provide ideas for future work with this dataset.

####  3.5.1 Collection time

Our dataset was collected at different times of the day, as seen in Figure 11.
While images were taken throughout the day from 6 a.m. to 11 p.m., videos were
mostly taken during the daytime. Considering that the collection took place
during the summer, there are limited numbers of night scenes. Due to
difficulties in capturing consistent quality RGB-D images from ZED cameras in
low light, many images and videos captured at night were deleted during the
pre-processing step.

Figure 11: Number of scenes taken at different times of the day. Histogram of
the scenes of images (left) and videos (right).

####  3.5.2 RAW image data

The RAW images that we saved, along with other compressed images, are
unprocessed outputs directly from the camera, showing more shades of colors
and a better representation of image parameters such as white balance,
contrast, exposure, gain, and more. Our RAW images, from the Canon and P20,
are three to four times larger in size than the compressed images.

The pixels in RAW images are overlaid with the Bayer color filtering
arrangement, also known as the Bayer mosaic pattern. It filters each pixel to
record a single primary color (red, green, or blue) and its pattern consists
of 50% green, 25% red, and 25% blue. Various demosaicing algorithms can be
used to interpolate RGB values in the Bayer mosaic pattern to obtain a full-
color image. Figures 13 and 13 show the RAW images and their Bayer mosaic
pattern for the Canon and P20 images. Note that the ZED camera outputs
processed images, so we do not save RAW images from the ZED.

As shown in Figures 13 and 13, the RAW images are relatively darker at the
edges. It is also possible to compare how much the image is processed by
comparing the RAW image and a JPEG compressed image. For the P20, the JPEG
compressed image has visually good quality compared to its RAW image.

Figure 12: Example of a RAW image from Canon. Images in the left column show
Bayer mosaic pattern of a RAW image. The middle column shows the RAW image and
the right column shows the JPEG compressed image.

Figure 13: Example of a RAW image from P20. The left column shows the Bayer
mosaic pattern of a RAW image, the middle column shows the RAW image, and the
right column shows the JPEG compressed image.

####  3.5.3 Object instance analysis

Although we have not annotated our scenes with objects, we analyzed the
PIV3CAMS dataset using an object detector to examine the diversity of the
scenes. We used the TensorFlow Object Detection API[61] provided by Google,
which includes popular object detection networks with pre-trained models. We
used the Faster R-CNN detector[62] with models trained on the Open Images
Dataset V4[20], which has 600 classes of object instances.

For the image dataset, we used all 8,385 images to test the object detector.
On average, 3.656 object instances were detected per scene. Figure 14 shows
the top 30 most frequently seen object instance in the image dataset. From
this figure, we notice that tree appears in almost every scene, likely because
many greens are planted throughout the cities. The window, building, house,
and car are the next most frequent objects. Since most of the image collection
locations were within urban areas, it was inevitable to capture houses and
cars.

Figure 14: Top 30 most frequent objects found in the image dataset. The
histogram shows 30 most frequent objects found throughout the image dataset.

For the video dataset, we sampled video frames every ten frames to avoid
excessive overlap between frames. On average, 5.019 object instances were
detected per scene. Figure 15 shows the top 30 most frequently seen object
instances in the video dataset. Similar to the image dataset, tree and
building are the two most frequently appearing objects. While we focused on
collecting static scenes for the image sets, we captured more dynamic scenes
for the video sets. Therefore, there are more active object instances, such as
person, car, and boat, appearing in the videos compared to the images.

Figure 15: Top 30 most frequent objects found in the video dataset. The
histogram shows 30 most frequent objects found throughout the video dataset.

## 4\. Novel View Synthesis using RGB-D images

Along with data collection, the second aim of this thesis is to explore a
computer vision task: novel view synthesis. As discussed in Chapter 2, there
are two main approaches to the novel view synthesis problem: geometry-based
and learning-based approaches. The method we chose to investigate is [59],
which is the state-of-the-art method at the time of writing this thesis. Their
method involves training an encoder-decoder network given a pair of RGB images
of source and target views and their poses. Our goal was first to reproduce
their method as our baseline. Then, we modified parts of their network to
apply depth images and investigate the effectiveness of using depth
information in the novel view synthesis task.

In this chapter, we will briefly explain the baseline method and how we re-
implemented their models. Then we will introduce several network variations
that we modified from the baseline network. The experimental results on each
network will be discussed in the next chapter.

###  4.1 Regenerated Baseline Network

The current state-of-the-art learning-based novel view synthesis method has
been proposed by [59]. The main idea of their work is to train a network with
different branches that learn the depth and the pixels of the target view. The
network has an encoder-decoder structure with three different branches that
learn the depth, mask, and pixels of the target view (Fig 16).

Inspired by [63], they transformed the latent vector, an output of the encoder
network, according to the given pose matrix between the source and target
views. This transformed latent vector is then given as an input to three
different decoders. The depth decoder predicts the depth map of the target
view and uses it to inverse warp the source view to the target view. The pixel
decoder directly predicts the RGB images of the target view. Finally, the mask
decoder learns to create a mask that merges the inverse warped image from the
depth decoder branch and the result of the pixel decoder.

Figure 16: Baseline Network Architecture The network consists of an encoder
and three decoders. The depth decoder predicts the depth map, the pixel
decoder predicts the RGB image, and the mask decoder merges their outputs.
Given a source view and a transformation matrix, the network outputs the
target view.

####  4.1.1 Encoder and Decoder modules

The authors of [59] did not release their source code or supplementary
materials at the time of this project. Therefore, we relied solely on the
network description from their paper and designed the encoder and decoder
modules as shown in Figure 17.

We constructed the encoder module with 7 convolution layers (stride of 2,
kernel size of 4, and padding size of 1) with batch normalization and the
leaky rectified linear unit (leaky ReLU) as the activation function. At the
end of the encoder module, a fully connected layer outputs the latent vector
with the chosen size.

The decoder module also consists of 7 deconvolution layers (stride of 2,
kernel size of 4, and padding size of 1). Similar to the encoder module, we
added batch normalization between each convolution layer and used the leaky
ReLU as the activation function. A fully connected layer is attached at the
beginning of each decoder to receive the rotated latent vector as input. Note
that the output of each decoder varies by its usage. For example, the pixel
branch decoder produces an RGB image of size $BatchSize\times W\times H\times
3$ while the depth and mask branch decoders produce a depth map and a mask of
size $BatchSize\times W\times H\times 1$.

Figure 17: Encoder and Decoder module diagram. The encoder (top) and decoder
(bottom) modules each consist of 7 convolution layers with batch normalization
and leaky ReLU activation functions between each layer.

####  4.1.2 Transforming latent vector

[59] deployed a Transforming Auto-Encoder (T-AE) network[63] by applying a 3D
transform to the latent vector $z^{T}=T_{src\rightarrow tgt}(z)$. The latent
vector z is represented as $z\in R^{n\times 3}$, where n is a hyper-parameter.
This representation is then multiplied with the ground-truth transformation
$T_{src\rightarrow tgt}$ which describes the camera pose between the source
view ($I_{src}$) and the target view ($I_{tgt}$). This 3D transformed latent
vector is given as input to the decoders in each branch.

####  4.1.3 Depth branch

The decoder in the depth branch decodes $z^{T}$ into a target depth mask
$D^{pred}_{tgt}$. We compute the warped target view $I^{warped}_{tgt}$
deterministically via perspective projection using $D^{pred}_{tgt}$,and
$I_{src}$.

Each pixel $(x_{t},y_{t})$ in the depth mask is mapped to 3D coordinates [X,
Y, Z] using the intrinsic parameter matrix K.

$[X,Y,Z]^{T}=K^{-1}D^{pred}_{tgt}(x_{t},y_{t})[x_{t},y_{t},1]^{T}$ (1)

We compute the new 3D coordinates of the pixel’s location in the source view
by multiplying transformation matrix $T_{tgt\rightarrow src}$ which describes
the camera rotation from the target view to the source view. These coordinates
are then projected onto the source image plane.

$[x^{tgt}_{s},y^{tgt}_{s},1]^{T}=KT_{tgt\rightarrow src}[X,Y,Z,1]^{T}$ (2)

Through the above equation, we obtain the correspondence between the pixels in
the target view and their positions in the source view. Note that the pixel
positions derived from equation 2 are non-integer. Therefore, we used a
differentiable bilinear sampling method proposed in [64] to compute the pixel
values in the target view from the source view.

$I^{warped}_{tgt}(x_{t},y_{t})=\sum_{x_{s}}\sum_{y_{s}}I_{src}max(0,1-|x_{s}-x^{tgt}_{s}|)max(0,1-|y_{s}-y^{tgt}_{s}|)$
(3)

Finally, we warp the source view to the target view using the pixel
correspondences. The operations, from equation 1 to equation 3, are indicated
as an inverse warping block in Figure 16. This operation is crucial for
generating target views as it helps preserve the texture and local details of
the source view.

####  4.1.4 Pixel branch and Mask branch

The decoder in the pixel branch decodes $z^{T}$ into a target RGB image
$I^{pixel}_{tgt}$. The decoder in the mask branch decodes $z^{T}$ into a
weighted mask $M^{pred}$ which is then used to fuse the results from the pixel
and depth branch.

####  4.1.5 Fusion

While the predicted target view from the depth branch contains textures and
local details close to the ground-truth target view, it is difficult to
reconstruct occluded parts in the source view. On the other hand, the
predicted target view from the pixel branch has a similar structure to the
ground-truth target view but lacks local details. Therefore, [59] employed a
fusion of the predicted target views from the depth and pixel branches using
the mask. The final predicted target view $I^{pred}_{tgt}$ is given by
equation 4 where $\cdot$ denotes element-wise multiplication.

$I^{pred}_{tgt}=(1-M^{pred})\cdot I^{warped}_{tgt}+M^{pred}\cdot
I^{pixel}_{tgt}$ (4)

####  4.1.6 Training

The baseline model requires only pairs of source and target views and their
transformation information to train the networks. We applied the L1 loss
between the predicted image outputs from each branch and the ground truth. All
three networks are trained by minimizing the reconstruction loss, as shown in
equation 5.

$loss_{recon}=||I^{pred}_{tgt}-I_{tgt}||_{1}+||I^{warped}_{tgt}-I_{tgt}||_{1}+||I^{pixel}_{tgt}-I_{tgt}||_{1}$
(5)

Additionally, we applied a least-square GAN loss[65] and a perceptual loss[66]
to the output of the pixel branch to enhance realism.

$loss_{LSGAN}=(1-Dis(I^{pixel}_{tgt}))^{2}$ (6)
$loss_{VGG}=||F_{vgg16}(I_{tgt})-F_{vgg16}(I^{pixel}_{tgt})||_{2}$ (7)

where $Dis$ is a discriminator and $F_{vgg16}$ is a feature-extracting
function base on a pre-trained VGG network[67].

The final loss is defined as:

$loss=\lambda_{L1}loss_{recon}+\lambda_{GAN}loss_{LSGAN}+\lambda_{VGG}loss_{VGG}$
(8)

where $\lambda_{L1},\lambda_{GAN}$, and $\lambda_{VGG}$ are the weight
parameters.

###  4.2 Variation Networks

Our motivation is to investigate the effectiveness of using ground-truth depth
maps instead of predicting depth masks from the network. To experiment with
our ideas, we made several variations to the baseline network by replacing the
depth decoder with ground-truth depth maps. Additionally, we examined the
importance of the mask, which plays a significant role in fusing the results
of the depth and pixel branches. Therefore, we altered the depth and mask
branches of the baseline network while keeping other parts unchanged.

In summary, we classified our proposed networks into two main categories.
Within each category, we made small changes, such as using different types of
input depth maps (source or target depth map) and masks for final prediction.
We explain the details of the modified networks in the following sub-sections.

  * (1)

Without depth decoder.

    * –

Using target depth maps.

    * –

Using source depth maps.

  * (2)

Without both depth decoder and mask decoder.

    * –

Using target depth maps and using no mask.

    * –

Using target depth maps and using visible source pixels in the target view as
a mask.

    * –

Using source depth maps and using visible source pixels in the target view as
a mask.

####  4.2.1 Variation 1: Without depth decoder

Using target depth maps
To investigate the effectiveness of using ground-truth depth maps for novel
view synthesis, our first approach is to replace the depth decoder with a
ground-truth target depth map. Since the role of the depth decoder in the
baseline model is to predict the target depth mask for inverse warping, we
have instead used the ground-truth target depth directly, as shown in Figure
18. We will refer to this variation model as No Depth Decoder - Target Depth
Map (ND-Tgt) for simplicity.

The encoder and decoder modules are the same as in the baseline network.
Additionally, the transformation of the latent vector $z$ and the inverse
warping of the target depth maps to produce warped target images are done as
explained earlier. Note that there are no weights to train in the depth branch
of the network. The final loss is defined as in equation 9, with
$loss_{LSGAN}$ and $loss_{VGG}$ as defined in equations 6 and 7, respectively.

$loss=\lambda_{L1}||I^{pred}_{tgt}-I_{tgt}||_{1}+\lambda_{L1}||I^{pixel}_{tgt}-I_{tgt}||_{1}+\lambda_{GAN}loss_{LSGAN}+\lambda_{VGG}loss_{VGG}$
(9)

Figure 18: ND-Tgt Network model. This variation replaces the depth decoder
with ground-truth target depth. The encoder and two decoders (pixel and mask)
predict the target RGB image and a fusion mask. Given a source view, ground-
truth target depth, and a pose matrix, the network outputs the predicted
target view.

Using source depth maps
Normally, RGB-D images come as a pair of source RGB image and source depth map
in real scenarios. Therefore, we modified the ND-Tgt network to use the source
depth map as an input, as shown in Figure Figure 19. We will refer to this
variation model as No Depth Decoder - Source Depth Map (ND-Src) for
simplicity.

Figure 19: ND-Src Network model. This variation replaces the depth decoder
with the ground-truth source depth. The encoder and two decoders (pixel and
mask) predict the target RGB image and a fusion mask. Given a source view,
ground-truth source depth, and a pose matrix, the network outputs the
predicted target view.

From the source depth map $D^{gt}_{src}$, we computed the warped target depth
map $D^{warped}_{tgt}$ using the forward projective projection. This warped
target depth $D^{warped}_{tgt}$ is used to compute the warped target image
$I^{warped}_{tgt}$. The forward warping process is similar to the inverse
warping process explained previously, except for the transformation matrix
$T_{src\rightarrow tgt}$,which describes the camera rotation from the source
view to the target view. Using forward warping, we can also compute a
visibility mask that shows pixels visible in the target view. Figure 20 shows
the depth branch using the source RGB-D image to compute the warped target
depth and visibility mask. This warped target depth is then used with the
source view to generate the target view.

Figure 20: Example of depth branch using source RGB-D image. Given a source
RGB-D image as an input, we compute the warped target depth map and visibility
mask via forward-warping. Then, the source image and warped target depth map
are used to compute the warped target image via inverse warping.

For the mask and pixel branches, we used the same encoder and decoder as the
baseline network. The latent vector $z$ is transformed and the inverse warping
of the target are done as well. Since there are no weights for thedepth
branch, the final loss is defined as equation 9

####  4.2.2 Variation 2: Without both depth decoder and mask decoder

Although the role of the depth branch is to provide more accurate texture and
local details to the predicted target view, this information is only available
for the parts that are not occluded from the source view. Therefore, we were
motivated to use the visibility map instead of learning the mask from the
decoder.

Using target depth maps and using no mask
First, we modified the baseline network by removing both the depth and mask
decoders, as shown in Figure 21. We will refer to this variation model as No
Depth Decoder, No Mask Decoder - Target Depth Map (NDNM-Tgt) for simplicity.

Figure 21: NDNM-Tgt Network model. This variation uses an encoder and a pixel
decoder, replacing the depth decoder with ground-truth target depth. The pixel
decoder predicts the target RGB image. Fusion is done by averaging pixel
values from the warped target view and the pixel branch output. Given a source
view, ground-truth target depth, and a pose matrix, the network outputs the
predicted target view.

The final predicted target image $I^{pred}_{tgt}$ is achieved by averaging the
result from the depth branch and the pixel branch.

$I^{pred}_{tgt}=\frac{1}{2}I^{warped}_{tgt}+\frac{1}{2}I^{pixel}_{tgt}$ (10)

Since the NDNM-Tgt network consists of only one encoder and one decoder, the
total loss is solely on the pixel branch, as shown in equeation 11

$loss=\lambda_{L1}||I^{pred}_{tgt}-I_{tgt}||_{1}+\lambda_{L1}||I^{pixel}_{tgt}-I_{tgt}||_{1}+\lambda_{GAN}loss_{LSGAN}+\lambda_{VGG}loss_{VGG}$
(11)

Using target depth maps and visible source pixels in the target view as a mask
Next, we computed the visibility mask $M^{vis}$ from the source RGB-D image
using forward-warping projection. This mask is then used to merge the results
from the depth and pixel branches (Eq. 12). The visibility mask is originally
sparse, as shown in Figure 20. We filled its gaps using the closing operation
of mathematical morphology to create a dense visibility mask. For our
experiment, we tried using both sparse and dense visibility masks.

$I^{pred}_{tgt}=M^{vis}\cdot I^{warped}_{tgt}+(1-M^{vis})\cdot
I^{pixel}_{tgt}$ (12)

We will refer to this variation model as No Depth Decoder, Visibility Mask -
Target depth map (NDVM-Tgt) for simplicity. Figure 22 shows the diagram of
this model. Note that this model uses the same method for transforming the
latent vector $z$. Additionally, the pixel branch is trained with GAN loss 6
and VGG loss 7 for realistic results. Therefore, the final loss is defined the
same as in equation 11.

Figure 22: NDVM-Tgt Network model.This variation uses an encoder and a pixel
decoder, replacing the depth decoder with ground-truth target depth. The pixel
decoder predicts the target RGB image. Fusion between depth and pixel branches
is done using the visibility mask from forward-warping projection.

Using source depth maps and visible source pixels in the target view as a mask
As mentioned above, in realistic scenarios, it is challenging to provide
ground-truth target depth maps with source RGB-D images. Therefore, our final
variation model uses only the source RGB-D image as input to predict the image
in a target view, as shown in Figure 23. This model will be referred to as No
Depth Decoder, Visibility Mask - Source Depth Map (NDVM-Src) for simplicity.

The encoder-decoder part for the pixel branch remains the same as in the
baseline model. The loss function applied to this model is the same as the
NDVM-Tgt model (Eq 11). Similar to the NDVM-Tgt model, we trained this model
with both sparse and dense visibility masks. The main difference of this model
compared to NDVM-Tgt is that it computes the warped target depth map from the
source depth map. The warped target depth map is a sparse depth map, which
contains less information when warping a source image into the target view.

Figure 23: NDVM-Src Network model. This variation uses an encoder and a pixel
decoder. The source RGB-D image is used to compute the warped target depth map
and visibility mask, which in turn predict the warped target image. The pixel
decoder predicts the target RGB image. Fusion between the depth and pixel
branches is done using the visibility mask from forward-warping projection.
Given a source RGB-D and a pose matrix, the network outputs the predicted
target view.

###  4.3 Experiments

We conducted experiments on two challenging datasets: the synthetic car
dataset from ShapeNet[37] and real driving scenes from KITTI[14]. First, we
trained and evaluated the baseline model and four different proposed variation
models with the synthetic car dataset. Then we trained the models with real
scenes from KITTI. Finally, we chose the most promising model to test on our
dataset, PIV3CAMS. The metrics, mean absolute error (L1) and structural
similarity (SSIM) index, widely used to assess reconstruction quality, were
used to evaluate the models quantitatively. In this section, we describe the
details about the organization of the training and testing sets and technical
implementations.

####  4.3.1 Datasets: ShapeNet

The synthetic car dataset from ShapeNet[37] consists of 8,195 car object files
that can be rendered using 3D computer graphics software such as Blender.

We used 6,555 car objects for training and 1,640 for testing, as suggested in
[59]. For training, we rendered 18 views for each car object by changing its
azimuth between 0°and 360°with a step size of 20°. We then randomly sampled
two views from each car object and used them as a pair of inputs to the
networks. For testing, we rendered 180 views for each car object by changing
its azimuth between 0°and 360°with a step size of 2°. For each car object, we
chose 15 pairs of different source and target views, resulting in a total of
24,600 instances used during evaluation.

####  4.3.2 Datasets: KITTI

The KITTI dataset[14] contains city road scenes captured from the frontal
views of cars. They provide sub-datasets with different settings for various
computer vision tasks.

We chose the sub-dataset for the odometry task following [59]. Since the
odometry dataset does not come with depth maps, we downloaded the
corresponding ground-truth sparse depth maps from the depth completion
dataset. We excluded sequences from the odometry dataset if there were no
corresponding depth maps. Additionally, we prepared dense depth maps using the
depth completion model proposed by [68].

In total, we used 21,432 frames to train the models and 1,071 frames for
testing. For both training and testing, we randomly sampled two images within
a three-frame difference. We also center-cropped each frame to form an image
with dimensions of 256$\times$256.

####  4.3.3 Datasets: PIV3CAMS

As an example of a computer vision application for PIV3CAMS, we tested novel
view synthesis on our proposed dataset. We selected three different ZED RGB-D
scenes from the video set since they include depth maps and transformation
data between frames. We did not perform any pre-processing on these sets
except for cropping the center of each frame to match the dimensions of the
other datasets, 256$\times$256\. We prepared 2,000 frames from three different
scenes, randomly sampled within a three-frame distance, as the source and
target views for the test.

###  4.4 Metrics

####  4.4.1 Mean Absolute Error (L1)

The mean absolute error, also called L1 loss, is defined as

$L_{1}=\frac{\sum_{i=1}^{h}\sum_{j=1}^{w}|x(i,j)-y(i,j)|}{hw}$ (13)

where $h$ and $w$ denotes the height and width of an image. This metric is
used to measure per-pixel color absolute differences between predicted target
images ($I^{pred}_{tgt}$) and the ground-truth target images ($I_{tgt}$).

####  4.4.2 Structural SIMilarity (SSIM) Index

Structural SIMilarity (SSIM) Index[69] measures the structural similarity
between two images by calculating on various windows of images. The measure
between two windows $x$ and $y$ of size $N\times N$ is defined as

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})}$
(14)

where $\mu_{x},\mu_{y}$ are the average of $x$ and $y$,
$\sigma_{x}^{2},\sigma_{y}^{2}$ are the variance of $x$ and $y$, $\sigma_{xy}$
is covariance of $x$ and $y$, and $c_{1},c_{2}$ are variables added to
stabilize the division. For our evaluation, we set window size $N=11$ and
$c_{1}=0.01^{2},c_{2}=0.03^{2}$ as proposed in [69]. This metric ranges from
-1 to 1, indicating the perceptual quality of two images. Note that when two
images are similar, the value is closer to 1.

###  4.5 Training Details

For a fair comparison between various models, we set the parameters the same
for all training sessions. The networks were trained with the Adam optimizer,
using a learning rate of 0.00006, $\beta_{1}$ as 0.5 and $\beta_{2}$ as 0.999.
The weights for the losses were chosen so they have similar absolute values
during training: $\lambda_{L1}=10,\lambda_{GAN}=2,\lambda_{VGG}=0.5$. The
batch size was set to 16. The maximum number of training epochs was 200 with
375 iterations for the car dataset and 100 with 1,375 iterations for the KITTI
dataset.

## 5\. Results and Discussion

###  5.1 Novel view synthesis for Synthetic Objects

First, we present the results of the novel view synthesis for synthetic car
data. The quantitative results are shown in Table 6. The NDVM-Tgt model, which
uses ground-truth target depth and source depth, performs slightly better than
other models on both L1 and SSIM metrics. Note that in this model, the source
depth is used only to generate the visibility mask. The results demonstrate
that using ground-truth target depth generally synthesizes the target view
better than using ground-truth source depth. The qualitative results are
presented in Figure 24

Model | $L_{1}$ | SSIM | Input Depth | Visibility Mask
---|---|---|---|---
Baseline | 0.0467 | 0.8763 | - | -
ND-Tgt | 0.0499 | 0.8737 | Target depth | -
ND-Src | 0.0495 | 0.8721 | Source depth | -
NDNM-Tgt | 0.4421 | 0.6861 | Target depth | -
NDVM-Tgt | 0.0455 | 0.8766 | Source and Target depth | sparse
NDVM-Tgt | 0.0454 | 0.8814 | Source and Target depth | dense
NDVM-Src | 0.0467 | 0.8726 | Source depth | sparse
NDVM-Src | 0.0468 | 0.8762 | Source depth | dense
Table 6: Quantitative results of novel view synthesis on synthetic car
objects[37]

Figure 24: Qualitative results of novel view synthesis on synthetic car
objects[37]. The rotation angles between the source and target view are 30°,
58°, 322°, and 150° in clock-wise, from top to bottom rows, respectively. The
density of the visibility mask used is indicated inside the parenthesis like
NDVM-Tgt (sparse).

####  5.1.1 Influence of using ground-truth depth in view synthesizing

The influence of ground-truth depth on novel view synthesis performance can be
observed by comparing the baseline model with the ND-Tgt and ND-Src models.
Note that in real scenarios, it is challenging to obtain ground-truth target
depth; however, we experimented with our models using ground-truth target
depth to establish an upper bound.

Quantitatively, the baseline model performs better than the other two models
on both L1 and SSIM metrics. The depth map predicted by the baseline model, as
shown in Figure 25 is a learned depth mask (refer to depth) that optimizes the
final predicted output in compliance with the fusion mask (refer to mask).
Consequently, naively replacing the baseline depth mask with the ground-truth
target map, as in the ND-Tgt model, did not improve the model’s performance.
Indeed, the depth mask from the baseline model differs significantly from the
ground-truth target depth map.

Qualitatively, the baseline, ND-Tgt, and ND-Src models all produce
perceptually acceptable results. We observe that using the ground-truth depth
map performs better in terms of transferring local details when the
transformation between the source view and target view is small. On the other
hand, the baseline model performs much better when there is a large view
transformation.

As observed in Figure 25, the results from the pixel decoder (refer to pixel)
show that the pixel decoders are capable of producing well-structured shapes
of the target view but lack detailed texture. The mask from the mask decoder
determines which result to use for the final prediction. We observe that it
learns the parts visible in the source and uses the warped image for those
parts. For parts not visible in the source, it uses the pixel image predicted
from the pixel branch.

Figure 25: Intermediate results of the baseline, ND-Tgt, and ND-Src models The
Depth column shows images used for inverse-warping projection. The results are
shown in the Warped column. Note that for the baseline model, Depth is
predicted by the depth decoder. The ND-Tgt model uses the ground-truth target
depth map for Depth. The ND-Src model uses Depth as the warped target depth
map computed from forward-warping. The Pixel and Mask columns show the output
of the pixel decoder and mask decoder of each model, respectively. The final
prediction is shown in the Predicted column.

####  5.1.2 Influence of using visibility mask in view synthesizing

Instead of predicting the mask from the mask decoder, we experimented with
using a visibility mask computed statistically via forward-warping. The NDVM-
Tgt and NDVM-Src models use the visibility mask to merge the results of the
pixel branch and the warped image. For each network, we synthesized the target
view using either a sparse or dense visibility mask, as shown in Figure 26.

Figure 26: Intermediate results of the NDVM-Tgt and NDVM-Src models The Depth
column shows images used for inverse-warping projection, and the results are
shown in the Warped column. The NDVM-Tgt model used the ground-truth target
depth map for Depth. The NDVM-Src model used Depth as a warped target depth
map computed from forward-warping. The Pixel column shows the outputs of the
pixel decoder, and the Mask column shows the sparse or dense visibility masks
computed via forward-warping.

Quantitatively, using a statistically computed visibility mask (NDVM models)
performs better than using a predicted mask (ND models). Specifically, NDVM-
Tgt with a dense visibility mask performs better than the baseline model on
both L1 and SSIM metrics. Comparing the intermediate results(Figure 25 and
Figure 26) of these models, we observe that the visibility mask provides more
precise information about the parts visible in the source view, whereas the
predicted mask provides rough information.

The density of the visibility mask affects the quantitative measures, with the
dense visibility mask resulting in higher SSIM values. Qualitatively, the
dense visibility mask produces smoother target results, especially in areas
visible in the source view. However, the final predicted target results still
suffer from speckles in the images for both types of visibility masks.

###  5.2 Novel view synthesis for Real Scenes

We also tested our proposed models on real scenes, where dense ground-truth
depth maps are usually unavailable and the images are more complex than those
in the synthetic car dataset. We trained and tested the baseline model and our
proposed models with the KITTI dataset[14]. Training with real scenes took
significantly longer than training with the synthetic car dataset; it took
eight days to train 100 epochs using a 12GB GPU (Tesla K40c). Due to limited
resources and time, we could only train the baseline, ND-Tgt, and ND-Src
models. Then, we used the trained encoder and pixel decoder of ND-Tgt and ND-
Src to test the performance of the other models, NDVM-Tgt and NDVM-Src,
respectively.

The quantitative results of the novel view synthesis on real scenes are shown
in Table 7. The ND-Tgt model performs better than the other models,
particularly outperforming the baseline model. Figure 27 shows the qualitative
results.

Model | $L_{1}$ | SSIM | Input Depth | Visibility Mask
---|---|---|---|---
Baseline | 0.3409 | 0.2503 | - | -
ND-Tgt | 0.2026 | 0.4216 | Target depth | -
ND-Src | 0.2061 | 0.4136 | Source depth | -
NDVM-Tgt | 0.2139 | 0.3490 | Source and Target depth | sparse
NDVM-Tgt | 0.2038 | 0.4017 | Source and Target depth | dense
NDVM-Src | 0.2141 | 0.3427 | Source depth | sparse
NDVM-Src | 0.2044 | 0.3928 | Source depth | dense
Table 7: Quantitative results of novel view synthesis on real scenes from
KITTI dataset[14]

Figure 27: Qualitative results of novel view synthesis on real scenes from
KITTI dataset[14]. The frame gaps between the source and target views are 1,
2, and 3, from top to bottom rows, respectively. The density of the visibility
mask used is indicated in parentheses, e.g., NDVM-Tgt (sparse).

The experiment on real scenes shows that the baseline model fails to predict
the target views, while the other methods using ground-truth depth maps can
generate at least some parts of the target views. A common issue noticed in
all models is that the output from the pixel branch is not satisfactory. One
possible explanation could be the small size of the latent vector used during
training. Since real scenes are more complex than synthetic car images, using
the same size latent vector as the synthetic car dataset may not be sufficient
to capture the features of real scenes.

The final output of the baseline model is almost identical to the result of
the pixel branch, indicating that the warped target image from the depth
branch is not adequate. The depth decoder did not learn the target depth
properly, resulting in poor warped target images. While [59] demonstrated good
performance with real scene images, it is likely that specific parameters need
to be applied to make the depth branch work effectively.

In contrast, the other models (ND-Tgt, ND-Src, NDVM-Tgt, and NDVM-Src)
produced reasonable results in some parts of the images, thanks to the ground-
truth depth. As shown in Figure 28, it is possible to warp the source RGB-D
image into a target view using 3D geometrical projections. These models use
the results from the pixel branch to inpaint the missing parts from the warped
target images, creating the final predicted target images. This result aligns
with the observations made with the synthetic car data.

Figure 28: 3D geometric projection of real scenes. Source and target depth
maps are completed using the [68] algorithm. Brighter pixels indicate further
distances. Yellow areas in the visibility mask show source pixels visible in
the target view.

Our experiment demonstrated that with a deficient result from the pixel
branch, the model’s performance relies heavily on the results from the depth
branch, specifically the warped target images. For example, using the dense
visibility mask produces a smoother final result by incorporating more pixels
from the warped target images. Among the NDVM models, those using dense
visibility masks scored lower L1 and higher SSIM. Additionally, when the
source and target images are closer in frames, more parts of the target view
are visible from the source view, resulting in better-predicted target images
(Fig 29).

Figure 29: Average L1 loss based on frame distance between the source and
target images from KITTI dataset. The horizontal axis shows the frame distance
from the source view. Negative values indicate the target view is before the
source view; positive values indicate it is after.

###  5.3 Novel view synthesis on the PIV3CAMS dataset

Finally, we tested one of our models trained on the real scenes on the
PIV3CAMS dataset. We used the ND-Tgt model for view synthesis as it showed the
best performance among the models (Fig 27). The calculated L1 and SSIM scores
are 0.2571 and 0.3942, respectively.

In Figure 30, we present some visual results. From these results, we observed
that the predicted masks prefer the result from the warped target images over
the pixel images. Therefore, except for the edge parts of the images, most of
the predicted views are the same as the warped target images. Since the
ground-truth target depth map used as an input was not a dense depth map, it
leaves missing pixel areas as gray in the warped images and thus in the
predicted views. Despite this, we have successfully demonstrated the potential
of using PIV3CAMS for novel view synthesis applications.

Figure 30: The novel view synthesizing results using PIV3CAMS dataset. The
ground-truth target depth map is a sparse depth map, where brighter pixels
indicate greater distances than darker pixels. Depth values set to zero are
represented in violet.

###  5.4 Limitations

####  5.4.1 The performance of the pixel branch

In the previous section, we have shown that the role of the mask image is to
fuse the two results: the warped target image and the pixel image. The mask
uses the warped target image for the areas visible in the source view, and the
pixel image for the rest. In other words, the occluded parts will be filled in
with the results from the pixel branch. Consequently, the performance of the
pixel branch plays an essential role in novel view synthesis.

Observing the results from the pixel branches of our proposed models, we
noticed that they do not predict the colors of the target view precisely,
hence degrading the performance. This issue is evident not only in real scene
cases but also in the synthetic car examples. The results of the NDNM-Tgt
model and the pepper-noised results from the NDVM models in Figure 24 show
that the predicted colors from the pixel branch do not match perfectly with
the ground-truth target view. Additionally, the pixel images in Figure 25 are
not sharp enough compared to the ground-truth target view. Therefore,
designing a network to produce better pixel images could enhance the
performance of novel view synthesis.

####  5.4.2 The sparsity of the depth map

From the real scene experiments in Sections 5.2 and 5.3, it becomes clear that
having a dense depth map is essential for novel view synthesis. The original
KITTI dataset provides an extremely sparse depth map, with less than 10% of
the pixels having depth values. Since the depth map plays a vital role in
computing the warped target image, we needed to complete the depth maps of
KITTI’s. Even if these completed depth maps are not perfect, they provide
reasonably good results for view synthesis. On the other hand, when we tested
the model with the PIV3CAMS dataset, which has sparse depth maps, holes
appeared in the predicted images. As our proposed models are not able to fill
those holes, the methods are limited to dense depth maps. Therefore, adopting
a network to fill those holes would expand the boundaries of novel view
synthesis using RGB-D images

## 6\. Conclusion

In this work, we introduced the PIV3CAMS dataset, which consists of paired
images and videos from three different cameras (Huawei P20, Canon 5D Mark IV,
and ZED stereo camera). The PIV3CAMS dataset comprises 8,385 pairs of images
and 82 pairs of videos. Each pair of images includes 2 RGB images, 2 RAW
images, 2 RGB-D images (left and right), and one confidence map, resulting in
about 75.4 million individual photos in the image set. The videos, captured at
30 frames per second with durations between 25 and 50 seconds, total more than
250 million frames. The data was collected from Zurich, Switzerland, and
Cheonan, South Korea, with carefully chosen locations to cover various indoor
and outdoor scenes. Additionally, the dataset includes scenes captured at
different times of the day, including night scenes. We anticipate that this
dataset will advance multiple computer vision algorithms using machine
learning approaches, such as view interpolation, image matching, and
image/video super-resolution.

We also investigated the novel view synthesis problem as an example of a
computer vision application. First, we reproduced one of the state-of-the-art
novel view synthesis methods as a baseline. This baseline model consists of
three branches that predict the target depth, target pixel image, and a mask
for fusion. We then modified the depth branch by using ground-truth target or
source depth maps as input. We experimentally showed with synthetic data that
having depth information is useful, especially when the transformation between
the source and target view is small. Furthermore, we trained our models with
real RGB-D images from the KITTI dataset, demonstrating that it is possible to
generalize the approaches to real-world imagery. Lastly, we tested the models
trained with the KITTI dataset on the PIV3CAMS dataset to demonstrate the
potential of using our proposed dataset for the novel view synthesis task.

In conclusion, our work covered the entire process of computer vision, from
data collection to application. For future work, we suggest annotating the
dataset with several object classes to expand its usage for more computer
vision tasks. Additionally, collecting more data from different locations
could increase the variety of the dataset. In terms of novel view synthesis
tasks, we recommend using the PIV3CAMS dataset to train networks, as it
contains more diverse scenes than the KITTI dataset. From our experiments, the
warped image using the RGB-D source view has perceptually acceptable results
except for some noise and holes. Therefore, an interesting direction for
future research is to tackle the novel view synthesis problem with inpainting
and denoising techniques.

## References

  * [1] S. A. Papert, “The summer vision project,” 1966.
  * [2] L. G. Roberts, “Machine perception of three-dimensional solids,” Ph.D. dissertation, Massachusetts Institute of Technology, 1963.
  * [3] D. A. Huffman, “Impossible objects as nonsense sentences,” 1971.
  * [4] M. B. Clowes, “On seeing things,” _Artif. Intell._ , vol. 2, no. 1, pp. 79–116, Jan. 1971. [Online]. Available: http://dx.doi.org/10.1016/0004-3702(71)90005-1
  * [5] D. Marr and T. Poggio, “A computational theory of human stereo vision,” _Proceedings of the Royal Society of London. Series B. Biological Sciences_ , vol. 204, no. 1156, pp. 301–328, 1979.
  * [6] W. E. L. Grimson, “Computational experiments with a feature based stereo algorithm,” 1984.
  * [7] B. K. Horn and B. G. Schunck, “Determining optical flow,” _Artificial intelligence_ , vol. 17, no. 1-3, pp. 185–203, 1981.
  * [8] S. Ullman, “The interpretation of structure from motion,” _Proceedings of the Royal Society of London. Series B. Biological Sciences_ , vol. 203, no. 1153, pp. 405–426, 1979.
  * [9] Y. LeCun, P. Haffner, L. Bottou, and Y. Bengio, “Object recognition with gradient-based learning,” in _Shape, contour and grouping in computer vision_. Springer, 1999, pp. 319–345.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [11] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” _International Journal of Computer Vision (IJCV)_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [12] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [13] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011.
  * [14] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [15] X. Chen, J. Song, and O. Hilliges, “Nvs machines: Learning novel view synthesis with fine-grained view control,” _arXiv preprint arXiv:1901.01880_ , 2019.
  * [16] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez, “The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3234–3243.
  * [17] A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Van Gool, “Dslr-quality photos on mobile devices with deep convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 3277–3285.
  * [18] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International Journal of Computer Vision_ , vol. 88, no. 2, pp. 303–338, Jun. 2010.
  * [19] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009.
  * [20] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, and V. Ferrari, “The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,” _arXiv:1811.00982_ , 2018.
  * [21] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, “A benchmark dataset and evaluation methodology for video object segmentation,” in _Computer Vision and Pattern Recognition_ , 2016.
  * [22] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, and T. Huang, “Youtube-vos: A large-scale video object segmentation benchmark,” _arXiv preprint arXiv:1809.03327_ , 2018.
  * [23] N. Silberman and R. Fergus, “Indoor scene segmentation using a structured light sensor,” in _Proceedings of the International Conference on Computer Vision - Workshop on 3D Representation and Recognition_ , 2011.
  * [24] P. K. Nathan Silberman, Derek Hoiem and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _ECCV_ , 2012.
  * [25] H. S. Koppula, A. Anand, T. Joachims, and A. Saxena, “Semantic labeling of 3d point clouds for indoor scenes,” in _Advances in neural information processing systems_ , 2011, pp. 244–252.
  * [26] A. Janoch, S. Karayev, Y. Jia, J. T. Barron, M. Fritz, K. Saenko, and T. Darrell, “A category-level 3d object dataset: Putting the kinect to work,” in _Consumer depth cameras for computer vision_. Springer, 2013, pp. 141–165.
  * [27] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” in _2011 IEEE international conference on robotics and automation_. IEEE, 2011, pp. 1817–1824.
  * [28] S. Song, S. P. Lichtenberg, and J. Xiao, “Sun rgb-d: A rgb-d scene understanding benchmark suite,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 567–576.
  * [29] J. Xiao, A. Owens, and A. Torralba, “Sun3d: A database of big spaces reconstructed using sfm and object labels,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2013, pp. 1625–1632.
  * [30] J. Martínez-Gómez, I. García-Varea, M. Cazorla, and V. Morell, “Vidrilo: The visual and depth robot indoor localization with objects information dataset,” _The International Journal of Robotics Research_ , vol. 34, no. 14, pp. 1681–1687, 2015.
  * [31] Diml/cvl rgb-d dataset. [Online]. Available: https://dimlrgbd.github.io/
  * [32] A. Patil, S. Malla, H. Gang, and Y.-T. Chen, “The h3d dataset for full-surround 3d multi-object detection and tracking in crowded urban scenes,” _arXiv preprint arXiv:1903.01568_ , 2019.
  * [33] Y. Choi, N. Kim, S. Hwang, K. Park, J. S. Yoon, K. An, and I. S. Kweon, “Kaist multi-spectral day/night data set for autonomous and assisted driving,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 3, pp. 934–948, 2018.
  * [34] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3213–3223.
  * [35] F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving video database with scalable annotation tooling,” _arXiv preprint arXiv:1805.04687_ , 2018.
  * [36] X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, and R. Yang, “The apolloscape dataset for autonomous driving,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 954–960.
  * [37] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su _et al._ , “Shapenet: An information-rich 3d model repository,” _arXiv preprint arXiv:1512.03012_ , 2015.
  * [38] Y. Xiang, W. Kim, W. Chen, J. Ji, C. Choy, H. Su, R. Mottaghi, L. Guibas, and S. Savarese, “Objectnet3d: A large scale database for 3d object recognition,” in _European Conference Computer Vision (ECCV)_ , 2016.
  * [39] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” _arXiv preprint arXiv:1711.03938_ , 2017.
  * [40] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, “Virtual worlds as proxy for multi-object tracking analysis,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4340–4349.
  * [41] P. Sturm and B. Triggs, “A factorization based algorithm for multi-image projective structure and motion,” in _European conference on computer vision_. Springer, 1996, pp. 709–720.
  * [42] M. Montemerlo, S. Thrun, D. Koller, B. Wegbreit _et al._ , “Fastslam: A factored solution to the simultaneous localization and mapping problem,” _Aaai/iaai_ , vol. 593598, 2002.
  * [43] P. E. Debevec, C. J. Taylor, and J. Malik, _Modeling and rendering architecture from photographs_. University of California, Berkeley, 1996.
  * [44] K. Rematas, C. H. Nguyen, T. Ritschel, M. Fritz, and T. Tuytelaars, “Novel views of objects from a single image,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 8, pp. 1576–1590, 2016.
  * [45] N. Kholgade, T. Simon, A. Efros, and Y. Sheikh, “3d object manipulation in a single photograph using stock 3d models,” _ACM Transactions on Graphics (TOG)_ , vol. 33, no. 4, p. 127, 2014.
  * [46] Y. Furukawa, C. Hernández _et al._ , “Multi-view stereo: A tutorial,” _Foundations and Trends® in Computer Graphics and Vision_ , vol. 9, no. 1-2, pp. 1–148, 2015.
  * [47] H. Ha, S. Im, J. Park, H.-G. Jeon, and I. So Kweon, “High-quality depth from uncalibrated small motion clip,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5413–5421.
  * [48] F. Yu and D. Gallup, “3d reconstruction from accidental motion,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 3986–3993.
  * [49] M. Levoy and P. Hanrahan, “Light field rendering,” in _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_. ACM, 1996, pp. 31–42.
  * [50] Z. Zhang, Y. Liu, and Q. Dai, “Light field from micro-baseline image pair,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3800–3809.
  * [51] J. Yang, S. E. Reed, M.-H. Yang, and H. Lee, “Weakly-supervised disentangling with recurrent transformations for 3d view synthesis,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 1099–1107.
  * [52] M. Tatarchenko, A. Dosovitskiy, and T. Brox, “Multi-view 3d models from single images with a convolutional network,” in _European Conference on Computer Vision_. Springer, 2016, pp. 322–337.
  * [53] D. Ji, J. Kwon, M. McFarland, and S. Savarese, “Deep view morphing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2155–2163.
  * [54] J. Flynn, I. Neulander, J. Philbin, and N. Snavely, “Deepstereo: Learning to predict new views from the world’s imagery,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5515–5524.
  * [55] T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, and A. A. Efros, “Learning dense correspondence via 3d-guided cycle consistency,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 117–126.
  * [56] T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros, “View synthesis by appearance flow,” in _European conference on computer vision_. Springer, 2016, pp. 286–301.
  * [57] E. Park, J. Yang, E. Yumer, D. Ceylan, and A. C. Berg, “Transformation-grounded image generation network for novel 3d view synthesis,” in _Proceedings of the ieee conference on computer vision and pattern recognition_ , 2017, pp. 3500–3509.
  * [58] S.-H. Sun, M. Huh, Y.-H. Liao, N. Zhang, and J. J. Lim, “Multi-view to novel view: Synthesizing novel views with self-learned confidence,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 155–171.
  * [59] X. Chen, J. Song, and O. Hilliges, “Nvs machines: Learning novel view synthesis with fine-grained view control,” _arXiv preprint arXiv:1901.01880_ , 2019.
  * [60] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, “Stereo magnification: Learning view synthesis using multiplane images,” _arXiv preprint arXiv:1805.09817_ , 2018.
  * [61] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama _et al._ , “Speed/accuracy trade-offs for modern convolutional object detectors,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7310–7311.
  * [62] R. Girshick, “Fast r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448.
  * [63] G. E. Hinton, A. Krizhevsky, and S. D. Wang, “Transforming auto-encoders,” in _International Conference on Artificial Neural Networks_. Springer, 2011, pp. 44–51.
  * [64] M. Jaderberg, K. Simonyan, A. Zisserman _et al._ , “Spatial transformer networks,” in _Advances in neural information processing systems_ , 2015, pp. 2017–2025.
  * [65] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2794–2802.
  * [66] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European conference on computer vision_. Springer, 2016, pp. 694–711.
  * [67] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [68] F. Ma, G. V. Cavalheiro, and S. Karaman, “Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3288–3295.
  * [69] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli _et al._ , “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.