Incorporating dense metric depth into neural 3D representations for view synthesis and relighting
Arkadeep Narayan Chaudhury Igor Vasiljevic Sergey Zakharov Vitor Guizilini Rares Ambrus Srinivasa Narasimhan Christopher G. Atkeson

[

Elephant trunk wrinkles:
 A mathematical model of function and form
    Yang Liu[Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, UK, Department of Mechanics, School of Mechanical Engineering, Tianjin University, Tianjin 300354, China]
  Alain Goriely[Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, UK]
  L. Angela Mihai[Corresponding author: L.A. Mihai, Email: , School of Mathematics, Cardiff University, Senghennydd Road, Cardiff, CF24 4AG, UK]

    
==========================================================================================================================================================================================================================================================================================================================================================================================================================================================


    < g r a p h i c s >

figureWe present an approach for the photo-realistic capture of small scenes by incorporating dense metric depth, multi-view, and multi-illumination images into neural 3D scene understanding pipelines. We use a robot mounted multi-flash stereo camera system, developed in-house, to capture the necessary supervision signals needed to optimize our representation with a few input views. The reconstruction of the LEGO plant and the face were generated with 11 and 2 stereo pairs respectively. We relight the textured meshes using <cit.>. Background design by <cit.>.
        


]
Elephant trunk wrinkles:
 A mathematical model of function and form
    Yang Liu[Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, UK, Department of Mechanics, School of Mechanical Engineering, Tianjin University, Tianjin 300354, China]
  Alain Goriely[Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, UK]
  L. Angela Mihai[Corresponding author: L.A. Mihai, Email: , School of Mathematics, Cardiff University, Senghennydd Road, Cardiff, CF24 4AG, UK]

    
==========================================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
Synthesizing accurate geometry and photo-realistic appearance of small scenes is an active area of research with compelling use cases in gaming, virtual reality, robotic-manipulation, autonomous driving, convenient product capture, and consumer-level photography. When applying scene geometry and appearance estimation techniques to robotics, we found that the narrow cone of possible viewpoints due to the limited range of robot motion and scene clutter caused current estimation techniques to produce poor quality estimates or even fail. On the other hand, in robotic applications, dense metric depth can often be measured directly using stereo and illumination can be controlled. Depth can provide a good initial estimate of the object geometry to improve reconstruction, while multi-illumination images can facilitate relighting. In this work we demonstrate a method to incorporate dense metric depth into the training of neural 3D representations and address an artifact observed while jointly refining geometry and appearance by disambiguating between texture and geometry edges. We also discuss a multi-flash stereo camera system developed to capture the necessary data for our pipeline and show results on relighting and view synthesis with a few training views. 


    


§ INTRODUCTION


Capturing photo realistic appearance and geometry of scenes is a fundamental problem in computer vision and graphics with a set of mature tools and solutions for content creation <cit.>, large scale scene mapping <cit.>, augmented reality and cinematography <cit.>. 
Enthusiast level 3D photogrammetry, especially for small or tabletop scenes, has been supercharged by more capable smartphone cameras and new toolboxes like RealityCapture and NeRFStudio. A subset of these solutions are geared towards view synthesis where the focus is on photo-realistic view interpolation rather than recovery of accurate scene geometry. These solutions take the “shape-radiance ambiguity”<cit.> into stride by decoupling the scene transmissivity (related to geometry) from the scene appearance prediction. But without of diverse training views, several neural scene representations (e.g. <cit.>) are prone to poor shape reconstructions while estimating accurate appearance. 
By only reasoning about appearance as cumulative radiance weighted with the scene's transmissivity, one can achieve convincing view interpolation results, with the quality of estimated scene geometry improving with the diversity and number of training views. However, capturing a diverse set of views, especially for small scenes, often becomes challenging due to the scenes' arrangement. 
Without of dense metric depth measurements, researchers have used sparse depth from structure from motion <cit.>, and dense monocular depth priors <cit.> to improve reconstruction, with a focus on appearance. Assimilating dense non-metric depth (e.g.  <cit.>) is often challenging due to the presence of an unknown affine degree of freedom which needs to be estimated across many views.     
However, without diversity of viewpoints, measuring the geometry directly is often useful. Several hardware solutions for digitizing objects exist, ranging from consumer level 3D scanners (e.g. <cit.>), and room scale metrology devices (<cit.>) to high precision hand held 3D scanners (e.g. <cit.>). Although these systems measure geometry very accurately, they interpret appearance as diffuse reflectance and often fall short in modelling view-dependent effects.
Despite the known effectiveness of incorporating depth and widespread availability of dense metric depth sensors in smartphone cameras (<cit.>) and as standalone devices (<cit.>, incorporation of dense metric depth into neural 3D scene understanding is underexplored. 
In this work:


  * we present a method to incorporate dense metric depth into the training of neural 3D fields, enabling state-of-the-art methods to use dense metric depth with minor changes. 

  * We investigate an artifact (<ref>) commonly observed while jointly refining shape and appearance. We identify its cause as existing methods' inability to differentiate between depth and texture discontinuities. We address it by using depth edges as an additional supervision signal. 





We demonstrate our ideas using a robot mounted multi-flash stereo camera rig developed in-house from off-the-shelf components. This device allows us to capture a diverse range of scenes with varying complexity in both appearance and geometry. Using the captured data, we demonstrate results in reconstruction, view interpolation, geometry capture, and relighting with a few views. We hope that our full-stack solution comprising of the camera system and algorithms will serve as a test bench for automatically capturing small scenes in the future. Additional results may be viewed at https://stereomfc.github.iohttps://stereomfc.github.io and in the supplementary document. 




§ RELATED WORK

View synthesis and reconstruction of shapes from multiple 3D measurements is an important problem in computer vision with highly efficient and general solutions like volumetric fusion (<cit.>), screened Poisson surface reconstruction (<cit.>), patch based dense stereopsis (<cit.>) and joint refinement of surface and appearance <cit.>. While these continue to serve as robust foundations, they fall short in capturing view-dependent appearance. Additionally, even with arbitrary levels of discretization, they often oversmooth texture and surfaces due to data association relying on weighted averages along the object surface. 
Recent neural 3D scene understanding approaches (e.g. <cit.>) have avoided this by adopting a continuous implicit volumetric representation to serve as the geometric and appearance back-end of the view synthesizer. Together with continuous models, reasoning about appearance as radiance, and high frequency preserving embeddings<cit.>, these approaches serve as highly capable view interpolators by reliably preserving view dependent appearance and minute geometric details. More recent work has included additional geometric priors in the form of monocular depth supervision (<cit.>), sparse depth supervision from structure-from-motion toolboxes (<cit.>), dense depth maps (<cit.>), patch based multi-view consistency <cit.>, and multi-view photometric consistency under assumed surface reflectance functions (<cit.>). Our work builds on the insights from using dense depth supervision to improve scene understanding with only a few training views available. 
Novel hardware is often used for collecting supervision signals in addition to color images to aid 3D scene understanding. <cit.> demonstrate a method to incorporate a time-of-flight sensor. <cit.> demonstrate a method to extract geometric and radiometric cues from scenes captured with a commercial RGBD sensor and improve view synthesis with a few views. Event based sensors have also been used to understand poorly lit scenes with fast moving cameras (<cit.>). Researchers have also combined illumination sources with cameras to capture photometric and geometric cues for dense 3D reconstruction of scenes with known reflectances (<cit.>). Similarly, <cit.> capture geometry and reflectance of objects by refining multi-view color, depth and multi-illumination images. Given the recent advances in stereo matching (<cit.>) we use a stereo camera to collect data for view synthesis to disambiguate between shape and appearance at capture. 
Pairing illumination sources with imaging can improve reasoning about the appearance in terms of surface reflectance parameters. <cit.> approaches the problem of material capture using a variety of neural and classical techniques. <cit.> leverage recent neural scene understanding techniques to jointly learn shape and appearance as reflectance of the scene. Our work also pairs illumination sources with stereo cameras to capture multi-illumination images from the scene and we build on modern neural techniques for view synthesis and relighting the scene. 



§ METHODS

We follow related works (<cit.>) and represent the scene with two neural networks – an intrinsic network 𝒩(θ) and an appearance network 𝒜(ϕ) which are jointly optimized to capture the shape and appearance of the object. 𝒩(θ) is a multi-layer perceptron (MLP) with parameters (θ) and uses multi-level hash grids to  encode the inputs (<cit.>). It is trained to approximate the intrinsic properties of the scene – the scene geometry as a neural signed distance field 𝒮(θ) and an embedding ℰ(θ). The appearance network 𝒜(ϕ) is another MLP which takes ℰ(θ) and a frequency encoded representation of the viewing direction and returns the scene radiance along a ray. 
Prior work has jointly learned 𝒮, 𝒩, 𝒜 with only multi-view images by optimizing a loss in the form of <ref> using stochastic gradient descent <cit.> along a batch of rays projected from known camera centers to the scene. 

    ℓ = ℓ_C + λ_gℓ_D + λ_c𝔼(|∇^2_𝐱𝒮(𝐱_s)|)

λs are hyperparameters and the third term in <ref> is the mean surface curvature minimized against the captured surface normals (see <cit.>). As the gradients of the loss functions ℓ_C (appearance loss)  and ℓ_D (geometry loss) propagate through 𝒜 and 𝒩 (and 𝒮 as it is part of  𝒩) the appearance and geometry are learned together.
We describe our method of incorporating dense metric depth in <ref> which enables a variety of neural 3D representations (<ref>) to use it. In <ref> we jointly optimize shape and appearance of a scene using information about scene depth edges. 



 §.§ Incorporating dense metric depth

Given a large number of orthogonal view pairs (viewpoint diversity), and the absence of very strong view dependent effects, <ref> is expected to guide 𝒮 to towards an unbiased estimate of the true scene depth (see e.g. <cit.>). We can accelerate the convergence by providing high quality biased estimate of the scene depth. Given the quality of modern deep stereo (<cit.>) and a well calibrated camera system, a handful of aligned RGBD sequences can serve as a good initial estimate of the true surface depth in absence of diverse viewpoints. 
In this section we describe our method to directly optimize 𝒮 with estimates of true surface depth to any surface point 𝐱_s. 
Although <cit.> use the depth estimates directly, they fall short of modelling view-dependent effects. To avoid that, we elect to learn a continuous and locally smooth function that approximates the signed distance function of the surface 𝐱_s which can then be transformed to scene density (<cit.>). To do this, we roughly follow <cit.> and consider a loss function of the form 

    ℓ_D(θ)    = ℓ_𝐱_s + λ𝔼(||∇_𝐱𝒮(𝐱^Δ, θ)|| - 1)^2
    where, ℓ_𝐱_s   = 1/NΣ_∀𝐱[ 𝒮(𝐱, θ) + 1-⟨∇_𝐱𝒮(𝐱, θ), 𝐧_x ⟩]

Through the two components of ℓ_𝐱_s, the loss encourages the function 𝒮(𝐱, θ) to vanish at the observed surface points and the gradients of the surface to align at the measured surface normals (𝐧_x). The second component in <ref> is the Eikonal term (<cit.>) which encourages the gradients of 𝒮 to have a unit L_2 norm everywhere. The individual terms of <ref> are averaged across all samples in a batch corresponding to N rays projected from a known camera. 
The Eikonal constraint applies to the neighborhood points 𝐱^Δ_s of each point in 𝐱_s. <cit.> identifies candidate 𝐱^Δ_s through a nearest neighbor search, where as <cit.> identifies 𝐱^Δ through random perturbations of the estimated surface point along the projected ray. As we have access to depth maps, we identify the variance of the neighborhood of 𝐱_s through a sliding window maximum filter on the depth images. This lets us avoid expensive nearest neighbor lookups for a batch of 𝐱_s to generate better estimates of 𝐱^Δ_s than <cit.> at train time. As a result, convergence is accelerated – (∼ 100× over <cit.>) with no loss of accuracy. As we used metric depth, noisy depth estimates for parts of the scene are implicitly averaged by 𝒮 optimized by minimizing <ref>, making us more robust to errors than <cit.>. We provide more details in the supplementary material.




 §.§ Incorporating depth edges in joint optimization of appearance and geometry


 

Prior works (<cit.>) show the benefits of jointly refining geometry and appearance as it affords some degree of geometric super-resolution and more stable training. However, some pathological cases may arise when the scene has a large variation in appearance corresponding to a minimal variation in geometry across two neighboring surface points – 𝐱_s and 𝐱_s^Δ. We investigate this effect by considering an extreme case – a checkerboard printed on matte paper with an inkjet printer, where there is no geometric variation (planar geometry) or view dependent artifacts (ink on matte paper is close to Lambertian) corresponding to a maximum variation in appearance (white on black). The qualitative results are presented in <ref>. 
Consider two rays r⃗_𝐱_s and r⃗_𝐱_s^Δ connecting the camera center and two neighboring points 𝐱_s and 𝐱_s^Δ on two sides of an checkerboard edge included in the same batch of the gradient descent. The total losses for those rays depend on the sum of the geometry and appearance losses (<ref>). By default, the current state of the art (<cit.> etc.) do not have a mechanism to disambiguate between texture and geometric edges (depth discontinuities).  

As seen in <ref>, given unsuitable hyperparameters, the approaches will continue to jointly update both geometry and appearance to minimize a combined loss (<ref>). 
This can often result in pathological reconstructions (left insets in <ref>) due to ℓ_C gradients dominating over ℓ_D. By gradually increasing the modelling capacity of 𝒩 we can somewhat avoid this artifact and force the gradient updates to focus on 𝒜 to minimize the cumulative loss. <cit.> recognize this and provide an excellent set of hyperparameters and training curricula to gradually increase the modelling capacity of 𝒩(ϕ). This results in remarkable geometric reconstructions for well known datasets (<cit.>). Alternatively, if we have per-pixel labels of geometric edges (𝐄, <ref>), we can preferentially sample image patches with low variation of geometric features when the model capacity is lower (𝒮(θ) tends to represent smoother surfaces), and focus on image patches with geometric edges when the model capacity has increased. The modelling capacity of 𝒜(ϕ) never changes. 
<Ref> describes our sampling procedure while learning a scene with a variety of geometric and texture edges. <Ref> is used to draw pixel samples – the probability of drawing pixel p_i is calculated as a linear blend of the likelihood that it belongs to the set of edge pixels 𝐄 and α is a scalar (α∈ [0, 1]) proportional to the progress of the training. 

    P(p_i|α) = (1-α)P(p_i ∈𝐄) + α P(p_i ∉𝐄)

To preserve the geometric nature of the edges while ruling out high frequency pixel labels, we use Euclidean distance transform (<cit.>) to dilate 𝐄 before applying <ref>. We provide  implementation details in the supplementary material for reproducibility. We discuss quantitative results in <ref>.



 §.§ Baselines augmented with depth

As baselines, we augment four state-of-the-art methods to incorporate metric depth: 
 is our implementation of AdaptiveShells (<cit.>) using metric depth. We retain the formulations for the scene geometry and appearance models, and adapt the formulation of the “shells” to use dense metric depth. Through we also demonstrate how dense metric depth can combine the advantages of volumetric and surface based representations in <ref>.  
is our augmented version of <cit.>, where we use the metric depths along the rays to optimize the geometry <ref>. All the other parts of the original approaches, including the methods for generating samples to minimize <ref> and scene density transforms are left unchanged. 
is our augmented version of <cit.>, where we also use <ref> to optimize the geometry. The rest of the algorithm including the background radiance field is left intact.  
is our deliberately hamstrung version of <cit.> where we force the the samples generated for the volumetric rendering step to have a very low variance around the current biased estimate of the surface. This makes the algorithm necessarily indifferent to the relative magnitudes of ℓ_D and ℓ_C in <ref>, and helps us exaggerate the pathological effects of not segregating texture and geometric edges. We choose to name the method after we (and <cit.>) observed that original method was vulnerable to this artifact under certain hyperparameter choices. Across all the methods, we implement and train 𝒩(θ) following <cit.>, and all of them use the same appearance network 𝒜(ϕ). and require a warm start – 𝒮 pre-optimized for 5K gradient steps. All methods except use the sampling strategy from <ref>. More details are in the supplementary material.



§ SETUP AND DATASET



 §.§ A multi-flash stereo camera


In addition to multi-view images, scene depth and depth edges are valuable signals to train neural 3D representations. To capture all the supervision signals, we designed and fabricated a multi-flash stereo camera based on insights from <cit.>, with off-the-shelf parts. We capture data by moving our camera rig in front of objects. For each camera pose we capture a stereo pair of high dynamic range (HDR) images, two depth maps from left and right stereo, two corresponding image aligned surface normals (as gradient of depth maps). We first tonemap the HDR images <cit.> and in-paint them with the depth edges before using <cit.> to preserve intricate surface details in the depth maps. Additionally we capture 12 pairs of multi-illumination images for 12 flash lights around the cameras, one light at a time. From the multi-flash images we recover a per pixel likelihood of depth edges in the scene and a label of pixels with a large appearance variation under changing illumination – relating to the specularity. We detail the design of our rig and the capture processes in the supplementary material. <Ref> shows a snapshot of the data captured, <ref> illustrates our camera rig prototype. 
We elected to calculate depth from stereo because it performs better than the following three alternatives we tested.  1) Recovering geometry from intrinsic-image-decomposition (<cit.>) and photometric stereo with a few lights (<cit.>) did not yield satisfactory results. PIE-Net(<cit.>) requires 256×256 images which were too low-resolution for reconstruction and, our captures were out-of-distribution for the pre-trained model. <cit.> assumes fixed lights – we’d need new light calibrations per-view. 2) Self-calibrating-photometric-stereo (<cit.>), demonstrated on <cit.>, needs  50-80 light views and accurate masks which we do not capture. Also, our lights are much closer to the camera than <cit.>. And, 3) modern camera-projector systems (<cit.>) yield better estimates of geometry than stereo, but is not fast enough to capture human subjects (<ref>). 



 §.§ Dataset

Although a data set is not the primary contribution of our research, we capture some salient aspects of the scene that are not present in several established datasets. We identify these aspects in <ref>. In the rows labeled “specularity” and “depth edges” we note if the dataset has explicit labels for the specular nature of the pixel or a presence of a depth edge at that pixel respectively. Under “illum. model” we note if an explicit illumination model is present per scene –  we do not capture an environment illumination model, and instead provide light poses. PaNDoRa does not have explicit specularity labels but polarization measurements at pixels may be used to derive high quality specularity labels, which are better than what our system natively captures. We differentiate between “OLAT” (one light at a time) and “flash” by the location of the source of illumination. Similar to <cit.>, our flashes are parallel to the imaging plane, located close (∼0.1f) to the camera, as opposed to ReNE and OpenIllumination.




§ EXPERIMENTS AND RESULTS



 §.§ Accuracy of incorporating metric depth

We reconstruct synthetic scenes with ground truth depth from <cit.> to measure the accuracy of our technique. We use 12-15 RGBD images to reconstruct the scenes and train for an average of 30k gradient steps (∼1500 epochs) in about 75 minutes. In contrast, <cit.> use  300+ RGBD tuples and 9+ hours of training on comparable hardware. Notably, <cit.> also optimizes for noise in camera poses and reports metrics with ground truth and optimized poses. We report the best metric among these two. <cit.> registers the images themselves. We register the RGBD images with a combination of rigid and photometric registration (<cit.>). We present the quantitative results in <ref>. We replicate or out-perform the baselines by using a fraction of the training data and gradient steps. Among all methods discussed in <ref>, and demonstrate similar performance, recovers a smoother surface at the expense of  ∼ 1.25× more gradient steps. Our errors on these synthetic datasets closely reflect the performance of <cit.> on approximating surfaces from low noise point clouds. These datasets do not have large view dependent appearance variations to affect the gradient updates. 


























 §.§ The effect of depth edges in training

We tested fused RGBD maps from stereo and four baselines from <ref> to investigate the effect of depth and texture edges. We use edge guided sampling (<ref>) for all except stereo and to prioritize learning geometric discontinuities over appearance.  We present the results in <ref>. All of the baselines except improve the reconstruction accuracy due to segregation of texture and depth edges. The smoothness enforced by the curvature loss in <ref> also improves the surface reconstruction over stereo.  





 §.§ View synthesis with dense depth

Incorporating dense metric depth and our sampling strategy from <ref> enables , , and to perform competitively across challenging scenes. Scene A (<ref>(a)) looks at a couple of reflective objects with large variation in view dependent appearance. Additionally, there are large local errors in the captured depth maps due to specularities in the scene. We capture six stereo pairs, train on 11 images and test on one image. Scene B (<ref>(b)) features a rough metallic object of relatively simple geometry  captured by a 16mm lens (450 mm focal length, shallow depth of field). We capture four stereo pairs, train on seven images and test on one image. Scene C (<ref>(c))  features a fairly complicated geometry and is captured with 12 stereo pairs. We train on 22 images and test on two. Quantitative results of our experiments are in <ref>. We observe that , which is roughly 15% faster per gradient step than , generally converges the fastest (wall clock time) to a target PSNR. When the geometry is very complicated (scene C), an equally complicated sampling volume negates the efficiency gains of our sampler. We could not find good parameters for for any of these sequences. 
View synthesis was unsuccessful without the inclusion of dense depth. We trained with no depth supervision (equivalent to <cit.>) until saturation (less than 0.1 PSNR increase for 1000 consecutive epochs). The reconstructions, none of which had a PSNR of 18 or higher, are shown in the last column of <ref>.




 §.§ Using noisy depth



To investigate the effects of noise in the depth maps, we obtain the depths of scenes using conventional stereo. We used semi-global matching stereo (<cit.>) with a dense census cost (<cit.>) and sub-pixel refinement on tone mapped HDR images to calculate the surface depth.  Surface normals were calculated using the spatial gradients of the depth maps. 

To focus on the performance of our approaches, we did not filter or smooth the depth obtained from conventional stereo. From the top of <ref>, we observe that strictly improves the quality of the surface reconstructed from just noisy stereo (row 1 and 2 versus row 3), especially when edge sampling is enabled. 

If the end goal is just view synthesis, , which blends the advantages of volumetric and surface based rendering, performs equally well with large noise in depth, whereas takes many more iterations to converge. This indicates that photorealistic view synthesis with a volumetric renderer is possible with noisy depth data. However conventional stereo often introduces large local errors which our approaches were unable to improve significantly. 
In the presence of noisy depth, the quality of the reconstructed surface was enhanced through edge-based sampling (<ref>). Our sampling strategy allocated samples away from depth edges, where the noise was more prevalent, leading to fewer gradient steps spent modelling areas with higher noise. <Ref> presents the quantitative details of the experiment.




 §.§ Relighting




We capture multi-illumination images with known light poses and recover geometry independently of appearance. This allows us to infer the illumination dependent appearance using a combination of physically based appearance parameters – e.g. the Disney Principled BRDF<cit.>. As a benchmark, we upgraded the closest related work, <cit.>, which uses the full gamut of the Disney BRDF parameters, with , to incorporate dense depth. For the data we collected, the optimization process  as implemented by <cit.>, was quite brittle and some parameters (e.g. `clearcoat-gloss') would often take precedence over other appearance parameters (e.g. `specular-tint') and drive the optimization to a poor local minima. We demonstrate this problem in detail in the supplementary material. We found the optimization of a subset of appearance parameters (`base-color', `specular-tint', and `roughness') to be the most stable. <cit.> conclude the same. 
For relighting, we explore two avenues – the inference step of our approach as a volumetric renderer and a mesh created with the appearance parameters as texture (<ref>). Quantitative and qualitative results of the volumetric renderer are shown in <ref>. We used <cit.> to unwrap the geometry and generate texture coordinates whose quality exceeded <cit.> and our implementation of <cit.>. None of our approaches worked on the ReNe dataset (<ref>, <cit.>) due to low view diversity, and the absence of metric depth. We used the labels in <ref> to allocate more gradient steps for learning the regions with higher appearance variation. We provide more details in the supplementary material.



§ LIMITATIONS
 
Although we achieve state of the art results in view-synthesis and relighting with a few views, our approach struggles to represent transparent objects and accurately capture the geometry of reflective surfaces. <cit.> address the problem of reflective objects by modelling background reflections and is based on the architecture proposed by <cit.>. As enables <cit.> to use possibly noisy metric depth, it can potentially be extended to model reflective objects. 
Our approaches require metric depth and depth edges for the best performance. Our approach relies on capture devices with reasonable quality depth measurements. Future work will address incorporation of monocular and sparse depth priors with depth edges. 

Incorporation of metric depth introduces a strong bias, often limiting super resolution of geometry sometimes achieved in neural 3D scene representation (see e.g. <cit.>). Decreasing the effect of <ref> during training may potentially encourage geometric superresolution and is future work. 

Finally, modern grid based representations (see e.g. <cit.>) produce very compelling view interpolation results at a fraction of the computational cost of a state of the art volumetric renderer (e.g. <cit.>). However, they need to be “distilled” from a pre-trained volumetric view interpolator. Future work can investigate the use of depth priors to train a grid based representation directly from color and depth images.  



§ CONCLUSIONS

We present a solution to incorporate dense metric depth into neural 3D reconstruction which enables state of  the art geometry reconstruction. We examine a corner case of jointly learning appearance and geometry and address it by incorporating additional supervision signals. Additionally, we describe a variant of the multi-flash camera to capture the salient supervision signals needed to improve photorealistic 3D reconstruction and demonstrate a pipeline for view synthesis and relighting of small scenes with a handful of training views.





    ieeenat_fullname
    





This supplementary document inherits the figure, equation, table, and reference numbers from the main document. Additional results may be viewed at https://stereomfc.github.iohttps://stereomfc.github.io. 



§ REPRESENTATIONS AND IMPLEMENTATION DETAILS

Our scene representation consists of two networks – an intrinsic network 𝒩(θ) and an appearance network 𝒜(ϕ). We follow <cit.> to build and train 𝒩(θ). We use 18 levels of hashgrid encodings <cit.> to encode the input and a two layer (128 neurons/layer) MLP to generate the intrinsic embedding. The first channel of the embedding, 𝒮(θ) is trained with <ref> to recover a signed distance field of the scene as described in <ref>. The rest of the 127 channels of the embedding ℰ(θ) are passed on to the appearance network 𝒜(ϕ) as an input. 
The appearance network takes ℰ(θ), the viewing direction (encoded with 6 levels of sinusoidal encodings following <cit.>), and optionally the illumination direction (if recovering BRDF) to generate colors. The neural network is built with 2 layers of fully connected MLPs (128 neurons/layer) with skip connections. 
The neural signed distance field 𝒮(θ) is optimized to return the signed distance of a point  from its nearest surface 𝒮(θ): ℝ^3 →ℝ. 

The surface of the object can be obtained from the zero-level set of 𝒮(θ) – i.e. for all surface points 𝐱_s ∈ℝ^3   |  𝒮(𝐱_s|θ) = 0. We train 𝒮(θ) by minimizing a geometric loss ℓ_D (<ref>). We follow <cit.> to transform the  distance of a point p⃗_i = r⃗|_t_i in a ray to its closest surface s_i = 𝒮(θ,p⃗_i) to the scene density (or transmissivity).

    Ψ_β(s) = 
                        0.5exp(s/β),     s≤0
    
                        1-0.5exp(-s/β),    otherwise.

To render the color 𝐂 of a single pixel of the scene at a target view with a camera centered at o⃗ and an outgoing ray direction d⃗, we calculate the ray corresponding to the pixel r⃗ = o⃗ + td⃗, and sample a set of points t_i along the ray. The networks 𝒩(θ) and 𝒜(ϕ) are then evaluated at all the 𝐱_i corresponding to t_i and the per point color 𝐜_i. The transmissivity τ_i is obtained and composited together using the quadrature approximation from <cit.> as:

    𝐂 = ∑_iexp( - ∑_j<iτ_j δ_j ) (1-exp(-τ_jδ_j))𝐜_i,      δ_i = t_i - t_i-1

The appearance can then be learned using a loss on the estimated and ground truth color 𝐂_gt

    ℓ_C = 𝔼[||𝐂 - 𝐂_gt||^2]


The appearance and geometry are jointly estimated by minimizing the losses in <ref> using stochastic gradient descent <cit.>. 

    ℓ = ℓ_C + λ_gℓ_D + λ_c𝔼(|∇^2_𝐱𝒮(𝐱_s)|)

λs are hyperparameters and the third term in <ref> is the mean surface curvature minimized against the captured surface normals. As the gradients of the loss functions ℓ_C and ℓ_D propagate through 𝒜 and 𝒩 (and 𝒮 as it is part of  𝒩) the appearance and geometry are learned together. 



 §.§ Details of our baselines


We implemented four baselines to investigate the effects of incorporating dense metric depth and depth edges into neural view synthesis pipelines. 
 is our method similar to VolSDF <cit.> and MonoSDF<cit.>. We represent the scene with 𝒩 and 𝒜 and train it with metric depth and color by minimizing <ref>. The samples for <ref> are drawn using the “error-bounded sampler” introduced by <cit.>. 
 represents a modified version of NeUS <cit.>, where we use the training schedule and structure of 𝒩 from <cit.>, the appearance network 𝒜 is adopted from NeUS and we optimize <ref> along with <ref>. In addition to 𝒜, also has a small 4 layer MLP (32 neurons per layer) to learn the radiance of the background as recommended in the original work by <cit.>. 
 is our method inspired by UniSurf<cit.>. We represent the scene's geometry using a pre-optimized implicit network 𝒩 as outlined in <ref>. We follow the recommendations of <cit.> to optimize 𝒜. UniSurf exposes a hyperparameter to bias sampling of <ref> towards the current estimate of the surface. As we pre-optimize the surface, we can find the surface point 𝐱_s = 𝐨+t_s𝐝 through sphere tracing 𝒮 along a ray. The intersection point t_s can then be used to generate N samples along the ray to optimize <ref>.  

    t_i = 𝒰[ t_s + ( 2i-2N - 1)Δ,  t_s + ( 2iN - 1)Δ]

<Ref> is the distribution used to draw samples and Δ is the hyperparameter that biases the samples to be close to the current surface estimate. We optimize 𝒮 independent of <ref> by just minimizing <ref> with registered depth maps (see <ref>). We use this method to study the effects of volumetric rendering versus surface rendering. We found this strategy to be very sensitive to the hyperparameter Δ and its decay schedule as the training progressed. While best parameters for some sequences resulted in very quick convergence, they were very hard to come across and generally, poorer choices led to undesirable artifacts (see e.g. <ref>). 
We describe  in the following section. 



 §.§ : Accelerating training with dense depth


The slowest step in training and inference for neural volumetric representations is generally the evaluation of <ref>. In this section we describe our method to accelerate training by incorporating metric depth. A method to make training more efficient involves drawing the smallest number of the most important samples of t_i for any ray. The sampling of t_i is based on the current estimate of the scene density and although these samples  can have a large variance, given a large number of orthogonal view pairs (viewpoint diversity), and the absence of very strong view dependent effects, the training procedure is expected to recover an unbiased estimate of the true scene depth (see e.g. <cit.>). We can accelerate the convergence by a) providing high quality biased estimate of the scene depth and b) decreasing the number of samples for t_i along the rays. 
Given the high quality of modern deep stereo (we use<cit.>) and a well calibrated camera system, stereo depth can serve as a good initial estimate of the true surface depth. We use stereo depth, aligned across multiple views of the scene to pre-optimize the geometry network 𝒮(θ). The other channels ℰ(θ) of 𝒩 remain un-optimized. A pre-optimized 𝒮 can then be used for high quality estimates of ray termination depths. 
<cit.> recommend using root finding techniques (e.g. bisection method) on scene transmissivity (<ref>) to estimate the ray termination depth. The samples for <ref> are then generated around the estimated surface point. Drawing high variance samples as 𝒩 and 𝒜 are jointly optimized reduces the effect of low quality local minima, especially in the initial stages of the optimization. As we have a pre-trained scene transmissivity field (𝒮 transformed with <ref>), we can draw a few high-quality samples to minimize the training effort. 
We found uniformly sampling  around the estimated ray-termination depth (baseline in <ref>) to be unsuitable. Instead, we pre-calculated a discrete sampling volume by immersing 𝒮 in an isotropic voxel grid and culling the voxels which report a lower than threshold scene density. We then used an unbiased sampler from <cit.> to generate the samples in this  volume. This let us greatly reduce the number of root-finding iterations and samples, while limiting the variance by the dimensions of the volume along a ray. As the training progresses, we decrease the culling threshold to converge to a thinner sampling volume around the surface while reducing the number of samples required.
We show the sampling volume (at convergence) and our reconstruction results in <ref> respectively. We retain the advantages of volumetric scene representation as demonstrated by the reconstruction of the thin structures in the scene, while reducing training effort. We dub our method to acknowledge  <cit.>, which demonstrates a related approach to accelerate inference. 



The “shells” shown in <ref> and the shells recovered by <cit.> for small scenes are physically similar quantities. <cit.> dilate and erode the original level-set of the scene (approximated by 𝒮) using a hyperparameter. Our “shells” are also jointly estimated with the geometry as the training progresses. <cit.> estimate the fall-off of the volume density values along a ray to determine the hyperparameters, which in turn determines the thickness of the “shell”. They subsequently use uniform sampling (similar to <ref>, where the Δ now denotes the local thickness of the shell) to generate samples for rendering. Our work takes a discrete approach by immersing the zero-level set (in form of pre-optimized 𝒮) in a dense isotropic voxel grid and culling the voxels which have a lower volume density, according to a preset hyperparameter that determines the thickness of the shell. Once the shell has been estimated, we use a unbiased density weighted sampler (instead of a uniform sampler) to generate samples along the ray inside the shell. We roughly follow <cit.> to generate samples along a segment of the ray guided by the voxels it intersects. The spatial density of samples is inversely proportional to their distance from the estimated surface. We implement this using the tools from NerfStudio<cit.>. Our sampling strategy is more robust to errors in estimated geometry (as shown in <ref>) than other approaches –notably and the original work (Adaptive Shells <cit.>) which is based on NeUS(<cit.>).  



 §.§ Training Details


We ran our experiments on a Linux workstation with an Intel Core i9 processor, 64GB RAM, and an Nvidia RTX3090Ti graphics card with 25GB of vRAM. Across all the experiments for learning scene radiance, we implemented a hard cut-off of 100K gradient steps amounting to less than 4.5 hours of training time across all the experiments. 
Across all our baselines (, , , and ) we used the intrinsic network proposed in <cit.>, with 2 layers of MLPs (128 neurons per fully connected layer) and 18 levels of input hash encodings activated gradually. Our input activation curriculum was based on the recommendations of <cit.>, and was used jointly with our edge-aware sampling strategy <ref> across all the scenes. 
The implementations of our baselines, design and bill-of-materials for the multi-flash camera system, dataset and the hyperparameters will be released soon. <Ref> denotes the training steps graphically.



 §.§ Difference between our and prior work on neural scene understanding with depth

IGR<cit.> was among the first to fit a neural surface to point samples of the surface. Our pipeline is largely inspired by that work. However, we have two main differences – we use a smaller network, and periodically activate multi-resolution hash encodings as recommended by <cit.> instead of using a fully connected set of layers with skip connections. Additionally, as we have access to depth maps, we identify the variance of the neighborhood of a point on the surface through a sliding window filter. We use this local estimate of variance in a normal distribution to draw samples for 𝐱_s^Δ along each ray. Our strategy assumes that image-space pixel neighbors are also world space neighbors, which is incorrect along the depth edges. However, as the Eikonal equation should be generally valid in ℝ^3 for 𝒮, the incorrect samples do not cause substantial errors and only contribute as minor inefficiencies in the pipeline. A more physically based alternative, following <cit.>, would be executing nearest neighbor queries at each surface point along the rays to estimate the variance for sampling. With about 80k rays per batch, ∼ 200K points in (𝐱_s), and about 40k gradient steps executed till convergence, and a smaller network, our approach was more than two orders of magnitude faster than <cit.>, with no measurable decrease in accuracy of approximating the zero-level set of the surface. 
NeuralRGBD<cit.> is the closest prior work based on data needed for the pipeline and its output. The scene is reconstructed using color and aligned dense metric depth maps. The authors aggregate the depth maps as signed distance fields and use the signed distance field to calculate weights for cumulative radiance along samples on a ray (<ref> in text). The weights are calculated with 

    w_i = σ(D_itr) ×σ(-D_itr)

where the D_i is the distance to the surface point along a ray, and the truncation tr denotes how fast the weights fall off away from the surface. <Ref> yields surface biased weights with a variance controlled by the parameter tr. Notably, the depth map aggregation does not yield a learned sign distance field (no Eikonal regularizer in the loss). The authors also include a `free-space' preserving loss to remove “floaters”. As implemented, the pipeline needs the truncation factor to be selected per-scene. As the depth maps are implicitly averaged by a neural network, it is implicitly smoothed and therefore the pipeline is robust to local noise in the depth map.
MonoSDF<cit.> is mathematically the closest prior method to our work and it uses dense scene depths and normals obtained by a monocular depth and normal prediction network (OmniData<cit.>). MonoSDF defines the ray length weighted with the scene density as the scene depth 𝐝_pred and minimizes 

    ℓ_D = ∑_r||𝐰𝐝_mono+𝐪 - 𝐝_pred||_2^2

where {𝐰,𝐪} are scale and shift parameters. Estimating an affine transformation on the monocular depth 𝐝_mono is important because in addition to gauge freedom (𝐰), monocular depths also have an affine degree of freedom (𝐪). The scale and shift can be solved using least squares to align 𝐝_mono and 𝐝_pred. The scene normals are calculated as gradients of 𝒮 weighted with scene density along a ray.  Through a scale and shift invariant loss, MonoSDF calculates one set of (𝐰,𝐪) for all the rays in the batch corresponding to a single training RGBD tuple.  In the earlier stages of the training, this loss helps the scene geometry converge. The underlying assumption is that there is an unique tuple {𝐰,𝐪} per training image that aligns 𝐝_mono to the actual scene depth captured by the intrinsic network 𝒩. 
Our experiments with MonoSDF indicate that the network probably memorizes the set of (𝐰,𝐪) tuples per training image. Explicitly passing an unique scalar tied to the training image (e.g. image index as proposed in <cit.>) speeds up convergence significantly. Success of MonoSDF in recovering both shape and appearance strongly depends on the quality of the monocular depth and normal predictions. Our experiments on using MonoSDF on the WildLight dataset(<cit.>) or the ReNe dataset (<cit.>) failed because the pre-trained Omnidata models performed poorly on these datasets. Unfortunately, as implemented, MonoSDF also failed to reconstruct scene geometry when the angles between the training views were small – ReNe dataset views are maximally 45^∘ apart. However, it demonstrates superior performance on the DTU and the BlendedMVS sequences while training with as low as three pre-selected views. Finally, our scenes were captured with a small depth of field and most of the background was out of focus, so the scene background depth was significantly more noisy than the foreground depth. We sidestepped this problem by assigning a fixed 1m depth to all the pixels that were in the background. Although this depth mask simplifies our camera pose estimation problem (by segregating the foreground from the background), it assigns multiple infeasible depths to a single background point. As we aggregate the depth maps into the intrinsic network (𝒩) by minimizing <ref>, the network learns the mean (with some local smoothing) of the multiple depths assigned to the single background point. However, the scale and shift invariant loss is not robust to this and with masked depth maps, we could not reliably optimize MonoSDF on our sequences. We suspect that this is because the scale and shift estimates for each instance of <ref> on the background points yielded very different results, de-stabilizing the optimization.  
<cit.> and <cit.>
use sparse scene depth in the form of SfM triangulated points. <cit.> use learnt spatial propagation <cit.> to generate dense depth maps from the sparse depth obtained by projecting the world points triangulated by SfM. <cit.> assign the closest surface depth at a pixel obtained by projecting the triangulated points to the image plane. Neither of these pipelines recover a 3D representation of the scene and focus on view synthesis using few views. 
<cit.> introduce a novel 3D representation – “Neural point clouds” which includes geometric and appearance feature descriptors (small MLPs) grounded to a point in 3D. The geometry is recovered as the anchors of the “neural points”. The appearance is calculated using a volumetric renderer which composits the outputs of the appearance descriptors of the neural points with the transmissivity of the neural points along the ray. The transmissivity of a neural point is calculated as a function of distances of a pre-set number of neighboring neural points. 



 §.§ Capturing approximate BRDF and generating textured meshes




Multi-illumination images captured by our camera system can be used to estimate surface reflectance properties. We  recover a truncated Disney BRDF model(<cit.>. Our model consists of a per pixel specular albedo, a diffuse RGB albedo, and a roughness value to interpret the observed appearance under varying illumination. To estimate the spatially varying reflectance, we first train a model (, or ) to convergence to learn the appearance as radiance. At convergence, the first channel 𝒮 of the intrinsic network 𝒩 encodes the geometry and the appearance network 𝒜 encodes the radiance. We use two of the embedding channels of ℰ to predict the roughness and specular albedo at every point on the scene. The diffuse albedo is obtained as the output of the converged appearance network 𝒜. To calculate the appearance, we apply the shading model (<cit.>) to calculate the color at every sample along a ray and volumetrically composite them using <ref> to infer appearance as reflectance. <Ref> describes our steps graphically.
Optimizing for the full set of the Disney BRDF parameters, following <cit.> did not work with our aproach as the optimization often got stuck at poor local minima. <Ref> shows one instance of optimizing the pipeline of <cit.>, where the strengths of the recovered `clearcoat' and `clearcoat-gloss' parameters dominated over the optimization of the other parameters, resulting in a waxy appearance. Choosing a more conservative set of parameters (only `base-color', `specular' and `roughness') in <ref> led to a  more realistic appearance. WildLight<cit.> is based on <cit.> – we substituted <cit.> with and the apperance model of <cit.> was not changed, minimizing the chances of introduction of a bug causing the artifact. 
Our process of generating texture and material properties roughly follows the methods described by <cit.> and <cit.>. We proceed through the following steps:


  * At convergence (see <ref>), we extracted the scene geometry using the method described in <cit.>. 

  * We calculate a depth mask by thresholding the depth images at every training view with an estimate of the scene depth to segregate the foreground from the background.

  * Next, we cull the resulting triangular mesh (step 1) by projecting rays from every unmasked (foreground) pixel corresponding to all the camera views. This lets us extract the main subject of our scene as a mesh. We use Embree<cit.> to implement this. 

  * We generate texture coordinates on the culled mesh using “Smart UV Unwrap” function from <cit.>. These results were qualitatively better than <cit.> and our  implementation of <cit.>. We then rasterize the culled mesh from step 3 to get points on the surface corresponding to the texture coordinates.

  * We project each of these surface points back on to each of the training views to get the image coordinates. Rays originating from a rasterized surface point and intersecting the surface before reaching the camera are removed to preserve self occlusion.

  * For all the valid projected points, we cast a ray onto the scene and use either , , or to generate the color at the pixel along the ray using <ref>. This is repeated for all the training views.

  * At the end of the previous step we have several measurements of colors at every texture coordinate of the scene. We apply a median filter (per color channel) to choose the color – taking averages or maxima of the samples introduces artifacts. If using the radiance as texture is sufficient (often the case for diffuse scenes) this textured mesh can be exported. <Ref> demonstrates using each of  , and, to calculate the diffuse color of the scene in <ref>. 

  * To generate material textures, we follow the same procedures with the corresponding material channels after , or has been trained on multi illumination images using the schedule outlined in <ref>. 

  * The material properties are also volumetrically composited using <ref> and median filtered like the base colors. This is different from just querying the value of the network at the estimated surface point in <cit.>. 

We use <cit.>, a web browser based tool that supports physically based rendering with the Disney BRDF parameters, and <cit.>  to generate the images in <ref> respectively.  



§ A MULTI-FLASH STEREO CAMERA







We capture the scene using a binocular stereo camera pair with a ring of lights that can be flashed at high intensity. For our prototype, we use a pair of machine vision cameras (<cit.>) with a 1”, 4MP CMOS imaging sensor of resolution of 2048 × 2048 pixels. As we focus mainly on small scenes, we use two sets of lenses that yield a narrow field of view – 12mm and 16mm fixed focal length lens (<cit.>). We use 80W 5600K white LEDs (<cit.>) flashed by a high-current DC power supply switched though MOSFETs controlled with an Arduino microcontroller. At each pose of our rig, we captured 12 images with each of the flash lights on (one light at a time) and one HDR image per camera. The cameras are configured to return a 12 bit Bayer image which is then de-Bayered to yield a 16 bit RGB image. 
For the HDR images, we performed a sweep of exposures from the sensor's maximum (22580 microseconds) in 8 stops and used <cit.> to fuse the exposures captured with ambient illumination (fluorescent light panels in a room). Following the recommendations of <cit.> we used an f-stop of 2.8 to ensure the whole scene is in the depth of field of the sensors. We found the recommendations from <cit.> to be incompatible with our pipeline, so we used Reinhard tone-mapping (<cit.>) to re-interpret the HDR images. Our image localization pipeline, and stereo matching also worked better with tonemapped images.
We set the left and right cameras to be triggered simultaneously by an external synchronization signal. We configured the camera frame acquisition and the illumination control programs to run in the same thread and synchronized the frame acquisition with the flashes through blocking function calls. <Ref> presents a schematic of our prototype device. 
Through experiments we observed that the vignetting at the edges of the frames were detrimental to the quality of reconstruction, so we only binned the central 1536 × 1536 pixels. A 16bit 1536×1536 frame saved as a PNG image was often larget than 10MB. To achieve a faster capture and training time without sacrificing the field of view, we down sampled the images to a resolution of 768×768 pixels for our experiments. Centered crops of our initial larger frames lead to failures of our pose-estimation pipelines due to the field of view being too narrow(<ref>), so we chose to down sample the images instead. For the images lit by a single LED, we used the camera's auto exposure function to calculate an admissible exposure for the scene and used 80% of the calculated exposure time for imaging – the built-in auto-exposure algorithm tended to over-expose the images a bit. Estimating the exposure takes about 2 seconds. Once the exposure value is calculated, it is used for all of the 12 flashes for each camera.
Several instances of these RGBD tuples are collected and the colored depth maps are registered in the 3D space in two stages – first coarsely using FGR <cit.> and then refined by optimizing a pose graph<cit.>. At the end of this global registration and odometry step, we retain a reprojection error of about 5 - 10 pixels. If the reprojection errors are not addressed, they will cause the final assets to have smudged color textures.  To address it, we independently align the color images using image-feature based alignment techniques common in multi-view stereo (<cit.>), so that a sub 1 pixel mean squared reprojection error is attained. The cameras aligned in the image-space are then robustly transformed to the world space poses using RANSAC<cit.> with Umeyama-Kabsch's algorithm<cit.>. Finally, we mask out the specular parts of the aligned images and use ColorICP <cit.> to refine the poses. The final refinement step helps remove any  small offset in the camera poses introduced by the robust alignment step. A subset of the data collected can be viewed on the project website.



 §.§ Identifying pixels along depth edges

To identify pixels along depth edges, we follow <cit.> and derive per-pixel likelihoods of depth edges. Assuming that the flashes are point light sources and the scene is Lambertian, we can model the observed image intensity for the k^th light illuminating a point 𝐱 with reflectance ρ(𝐱) on the object as 

    𝐈_k(𝐱) = μ_k ρ(𝐱) ⟨𝐥_k(𝐱), 𝐧(𝐱) ⟩

where μ_k is the intensity of the k^th source and 𝐥_k(𝐱) is the normalized light vector at the surface point. 𝐈_k(𝐱) is the image with the ambient component removed. With this, we can calculate a ratio image across all the illumination sources

    𝐑(𝐱) = 𝐈_k(𝐱)𝐈_max(𝐱) = μ_k  ⟨𝐥_k(𝐱), 𝐧(𝐱) ⟩/max_i (μ_i  ⟨𝐥_i(𝐱), 𝐧(𝐱) ⟩)

It is clear that the ratio image 𝐑(𝐱) of a surface point is exclusively a function of the local geometry. As the light source to camera baselines are much smaller than the camera to scene distance, except for a few detached shadows and inter-reflections, the ratio images (<ref>) are more sensitive to the variations in geometry than any other parameters. We exploit this effect to look for pixels with largest change in intensity along the direction of the epipolar line between the camera and the light source on the image. This yields a per-light confidence value of whether 𝐱 is located on a depth edge or not. Across all 12 illumination sources, we extract the maximum values of the confidences as the depth edge maps. Unlike <cit.>, we use 12 illumination sources 30^∘ apart, and we do not threshold the confidence values to extract a binary edge map. This lets us extract more edges especially for our narrow depth of field imaging system and gets rid of hyper parameters used for thresholding and connecting the edges.
Often parts of our scene violate the assumption of Lambertian reflectances resulting in spurious depth edges. When we use depth edges for sampling, these errors do not affect the accuracy of our pipeline. When using depth edges for enhancing stereo matching (<ref>) we ensure that the stereo pairs do not contain too many of these spurious edge labels to introduce noise in our depth maps. 



 §.§ Identifying patches with non-Lambertian reflectances


We modified the definition of differential images in the context of near-field photometric stereo introduced by <cit.> to identify non-Lambertian patches. Assuming uniform Lambertian reflectances, <ref> can be expanded as 

    𝐈_k(𝐱) = μ_k^* ρ(𝐱)  𝐧(𝐱)^T 𝐬_k-𝐱|𝐬_k-𝐱|^3

where 𝐬_k is the location and μ_k^* is the power of the k^th light source. We define the differential images as 𝐈_t = ∂𝐈/∂𝐬𝐬_t where, 𝐬_t = ∂𝐬/∂ t, which when applied to <ref> can be expanded as

    𝐈_t(𝐱) = 𝐈(𝐱)𝐧^T 𝐬_t𝐧^T(𝐬-𝐱) - 3𝐈(𝐱)(𝐬-𝐱)^T𝐬_t|𝐬-𝐱|^2

Observing that the light sources move in a circle around the center of projection on the imaging plane, 𝐬^T𝐬_t = 0. Also, the second term of <ref> is exceedingly small given that the plane spanned by 𝐬_t is parallel to the imaging plane and our choice of lenses limit the field of view of the cameras. The second term is further attenuated by the denominator |𝐬-𝐱|^2 because the camera-to-light baselines (𝐬) are at least an order of magnitude smaller than the camera to object distance (𝐱). As a result, under isotropic reflectances (Lambertian assumed for this analysis) the differential images 𝐈_t(𝐱) are invariant to circular light motions. Any observed variance therefore can be attributed to the violations of our isotropic BRDF assumptions. We identify specular patches by measuring the variance of this quantity across the 12 instances of the flashlit images.   
Although our pipelines for identifying depth edges and patches of varying appearances demonstrate satisfactory qualitative performance, sometimes they yield wrong labels because <ref> do not include additional terms for spatially varying BRDFs and interreflections respectively. These errors do not have any significant effect in our reconstruction pipeline as we use this information to generate samples during different phases of training to minimize photometric losses and we do not directly infer shape or reflectances from these steps. 



 §.§ Difference between <cit.> and our hardware

<cit.> was the first to propose pairing flashes with cameras and laid the groundwork for identifying depth edges from multi-flash images from a single viewpoint. However, <cit.> considered a monocular camera and only four flashes along the horizontal and vertical directions of the camera in the demonstrated device. Researchers (see e.g. <cit.>) have since extended it by placing multiple light sources far apart from a monocular camera and have demonstrated locating depth edges on objects with strictly Lambertian reflectances. In this work, we retain the original light and camera configuration from <cit.> and increase the number of lights from four to 12. 
<cit.> also investigated a stereo camera in a multi-flash configuration aimed at edge preserving stereo depth maps. They do not extend the application to synthesizing geometry or appearance by capturing and assimilating multiple views of the scene. For obtaining stereo depth maps, we use <cit.>, which performs much better than conventional stereo matching (<cit.>) largely deployed in off-the shelf systems (<cit.>). 
Both <cit.> discuss methods to detect specularities (termed “material edges”) through different transforms of the multi-light images. However, we achieve a more continuous circular motion of the lights around the cameras, so we choose to use the photometric invariants described by <cit.> instead. 











