The 3D-zoom operation is the positive translation of the camera in the Z-axis, perpendicular to the image plane. In contrast, the optical zoom changes the focal length and the digital zoom is used to enlarge a certain region of an image to the original image size. In this paper, we are the first to formulate an unsupervised 3D-zoom learning problem where images with an arbitrary zoom factor can be generated from a given single image. An unsupervised framework is convenient, as it is a challenging task to obtain a 3D-zoom dataset of natural scenes due to the need for special equipment to ensure camera movement is restricted to the Z-axis. Besides, the objects in the scenes should not move when being captured, which hinders the construction of a large dataset of outdoor scenes. We present a novel unsupervised framework to learn how to generate arbitrarily 3D-zoomed versions of a single image, not requiring a 3D-zoom ground truth, called the Deep 3D-Zoom Net. The Deep 3D-Zoom Net incorporates the following features: (i) transfer learning from a pre-trained disparity estimation network via a back re-projection reconstruction loss; (ii) a fully convolutional network architecture that models depth-image-based rendering (DIBR), taking into account high-frequency details without the need for estimating the intermediate disparity; and (iii) incorporating a discriminator network that acts as a no-reference penalty for unnaturally rendered areas. Even though there is no baseline to fairly compare our results, our method outperforms previous novel view synthesis research in terms of realistic appearance on large camera baselines. We performed extensive experiments to verify the effectiveness of our method on the KITTI and Cityscapes datasets. Novel view synthesis is the task of hallucinating an image seen from a different camera pose given a single image or a set of input images. In natural images, this is a challenging task due to occlusions, ambiguities, and complex 3D structures in the scene. In addition, the larger the baseline (relative distance between input camera pose and target camera pose) the more challenging the problem becomes, as occlusions and ambiguities become dominant. New view synthesis finds applications in robotics, image navigation, augmented reality, virtual reality, cinematography, and image stabilization. There is a large body of literature that has studied the novel view synthesis problem for the multiple input image scenario, in both classical and learning based approaches. On the other hand, few works have tackled the problem of single input image novel view synthesis, which is a more complex task, as the deep understanding of the underlying 3D structure of the scene is needed to synthesize a new view. Finally, 3D-zoom is a subset of the novel view synthesis problem that has not been studied separately as exemplified in Figure 1 . 3D-zoom is the positive translation of the camera in the Z-axis as depicted in Figure 2 . In contrast, digital and optical zoom are close to a change in focal length and don't require any knowledge about the scene 3D geometry. Generating a 3D-zoom dataset with natural scene imagery is a challenging task. Special devices would need to be used to ensure translation is restricted to the Z-axis. In addition, moving objects would need to be masked or avoided as they would represent ambiguities for the 3d-zoom model. Alternatively, some available driving datasets could be used by filtering the sequences that move in a straight line. However, it does not guarantee camera pose changes to be restricted to the Z-axis neither the absence of moving objects between captures in the scene. For these reasons, we propose to learn 3D-zoom in an unsupervised fashion by utilizing a pre- trained disparity estimation network with transfer learning. Our 3D-Zoom Net is based on a fully convolutional network architecture that learns the under-laying 3D structure of the scene without the need of intermediate disparity as it is trained based on a novel back re-projection reconstruction cost that enforces both 3D geometry and natural appearance. Additionally, we include an adversarial network that acts as a no-reference measure that penalizes unnaturally rendered areas. Our proposed model, Deep 3D-Zoom Net, can perform inference of naturally looking 3D-zoomed images very fast. We show the efficacy of our proposed model in generating 3D-Zoomed images at various zoom factors on the KITTI (Geiger et al., 2012; Menze & Geiger, 2015) and Cityscapes (Cordts et al., 2016) datasets. We formulated a new image synthesis problem, by constraining it to positive translations in the Z-axis, which we call 3D-zoom, and presented an unsupervised learning solution, called the Deep 3D-Zoom Net. We demonstrated that 3D-zoom can be learned in an unsupervised fashion, by (i) modeling the image synthesis as a blending operation between multiple up-scaled versions of the input image, (ii) by minimizing a novel back re-projection reconstruction loss that facilitates transfer learning from a pre-trained disparity estimation network and accounts for 3D structure and appearance, and (iii) incorporating an adversarial loss to reduce unnaturally synthesized areas. Our Deep 3D-Zoom Net produces naturally looking images for both the KITTI and Cityscapes dataset, establishing a state-of-the-art solution for this class of single image novel view synthesis problem. We believe our Deep 3D-Zoom Net can be used as a tool for cinematography and user 3D-visualization of 2D images. Our work could also be extended for virtual and augmented reality, and even in glasses-free 3D displays as having arbitrary 3D zoomed versions of the input image generates a 3D sensation.
A novel network architecture to perform Deep 3D Zoom or close-ups. A method for creating a "zoomed image" for a given input image,and a novel back re-projection reconstruction loss that allows the network to learn underlying 3D structure and maintain a natural appearance. An algorithm for synthesizing 3D-zoom behavior when the camera is moving forward, a network structure incorporating disparity estimation in a GANs framework to synthesize novel views, and a proposed new computer vision task.