We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours and sightseeing, the digital inspection of historical artifacts). A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object.
 As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data. In recent years, large progress has been made in 3D shape reconstruction of objects from photographs or depth streams. However, highly realistic re-rendering of such objects, e.g., in a virtual environment, is still very challenging. The reconstructed surface models and color information often exhibit inaccuracies or are comparably coarse (e.g., Izadi et al. (2011) ). Many objects also exhibit strong view-dependent appearance effects, such as specularities. These effects not only frequently cause errors already during image-based shape reconstruction, but are also hard to reproduce when re-rendering an object from novel viewpoints. Static diffuse textures are frequently reconstructed for novel viewpoint synthesis, but these textures lack view-dependent appearance effects. Imagebased rendering (IBR) introduced variants of view-dependent texturing that blend input images on the shape (Buehler et al., 2001; Heigl et al., 1999; Carranza et al., 2003; Zheng et al., 2009) . This enables at least coarse approximation of view-dependent effects. However, these approaches often produce ghosting artifacts due to view blending on inaccurate geometry, or artifacts at occlusion boundaries. Some algorithms reduce these artifacts by combining view blending and optical flow correction (Eisemann et al., 2008; Casas et al., 2015; Du et al., 2018) , or by combining viewdependent blending with view-specific geometry (Chaurasia et al., 2013; Hedman et al., 2016) or geometry with soft 3D visibility like Penner & Zhang (2017) . Hedman et al. (2018) reduces these artifacts using a deep neural network which is predicting per-pixel blending weights. In contrast, our approach explicitly handles view-dependent effects to output photo-realistic images and videos. It is a neural rendering approach that combines image-based rendering and the advances in deep learning. As input, we capture a short video of an object to reconstruct the geometry using multi-view stereo. Given this 3D reconstruction and the set of images of the video, we are able Figure 1 : Overview of our image-guided rendering approach: based on the nearest neighbor views, we predict the corresponding view-dependent effects using our EffectsNet architecture. The viewdependent effects are subtracted from the original images to get the diffuse images that can be reprojected into the target image space. In the target image space we estimate the new view-dependent effect and add them to the warped images. An encoder-decoder network is used to blend the warped images to obtain the final output image. During training, we enforce that the output image matches the corresponding ground truth image. to train our pipeline in a self-supervised manner. The core of our approach is a neural network called EffectsNet which is trained in a Siamese way to estimate view-dependent effects, for example, specular highlights or reflections. This allows us to remove view-dependent effects from the input images, resulting in images that contain view-independent appearance information of the object. This view-independent information can be projected into a novel view using the reconstructed geometry, where new view-dependent effects can be added. CompositionNet, a second network, composites the projected K nearest neighbor images to a final output. Since CompositionNet is trained to generate photo-realistic output images, it is resolving reprojection errors as well as filling regions where no image content is available. We demonstrate the effectiveness of our algorithm using synthetic and real data, and compare to classical computer graphics and learned approaches. To summarize, we propose a novel neural image-guided rendering method, a hybrid between classical image-based rendering and machine learning. The core contribution is the explicit handling of view-dependent effects in the source and the target views using EffectsNet that can be learned in a self-supervised fashion. The composition of the reprojected views to a final output image without the need of hand-crafted blending schemes is enabled using our network called CompositionNet. In this paper, we propose a novel image-guided rendering approach that outputs photo-realistic images of an object. We demonstrate the effectiveness of our method in a variety of experiments. The comparisons to competing methods show on-par or even better results, especially, in the presence of view-dependent effects that can be handled using our EffectsNet. We hope to inspire follow-up work in self-supervised re-rendering using deep neural networks.
We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis while considering view-dependent effects. This submission proposes a method to handle view-dependent effects in neural rendering, which improves the robustness of existing neural rendering methods.