Surface-Centric Modeling for High-Fidelity Generalizable Neural Surface Reconstruction
Rui Peng Shihe Shen Kaiqiang Xiong Huachen Gao Jianbo Jiao Xiaodong Gu Ronggang Wang




 



Surface-Centric Modeling for High-Fidelity Surface Reconstruction






R. Peng et al.




1School of Electronic and Computer Engineering, Peking University 
 2Peng Cheng Laboratory   3University of Birmingham   4Alibaba 

   



Surface-Centric Modeling for High-Fidelity Generalizable Neural Surface Reconstruction
    Rui Peng1,2 Shihe Shen1 Kaiqiang Xiong1 Huachen Gao1 
Jianbo Jiao3 Xiaodong Gu4 Ronggang Wang1,2

    
=====================================================================================================



§ ABSTRACT 
  Reconstructing the high-fidelity surface from multi-view images, especially sparse images, is a critical and practical task that has attracted widespread attention in recent years. However, existing methods are impeded by the memory constraint or the requirement of ground-truth depths and cannot recover satisfactory geometric details. To this end, we propose SuRF, a new Surface-centric framework that incorporates a new Region sparsification based on a matching Field, achieving good trade-offs between performance, efficiency and scalability. To our knowledge, this is the first unsupervised method achieving end-to-end sparsification powered by the introduced matching field, which leverages the weight distribution to efficiently locate the boundary regions containing surface. Instead of predicting an SDF value for each voxel, we present a new region sparsification approach to sparse the volume by judging whether the voxel is inside the surface region. In this way, our model can exploit higher frequency features around the surface with less memory and computational consumption. Extensive experiments on multiple benchmarks containing complex large-scale scenes show that our reconstructions exhibit high-quality details and achieve new state-of-the-art performance, i.e., 46% improvements with 80% less memory consumption. Code is available at <https://github.com/prstrive/SuRF>.
  







§ INTRODUCTION



Reconstructing surface from multi-view images is a fundamental and challenging task in computer vision with wide-ranging applications, including autonomous driving, robotics, virtual reality, and more. While many typical methods <cit.> have achieved satisfactory results through tedious multi-stage processes (, depth estimation, filtering and meshing), recent neural implicit methods <cit.> attract increasing attention due to their concise procedures and impressive reconstructions. They can directly extract the geometry through Marching Cube <cit.>, and avoid the accumulated errors. Despite their effectiveness, these methods are hampered by the cumbersome per-scene optimization and the requirement of a large number of input views, which makes them unsuitable for many applications. Even recent fast methods like <cit.> and 3D Gaussian Splatting methods <cit.> struggle to extract meshes in seconds and perform poorly under sparse input.
 
Recently, some generalizable neural surface methods <cit.> have been proposed to mitigate these problems by combining neural implicit representations with prior image information. However, as the pipeline comparisons shown in Fig. <ref>, they either rely on the non-end-to-end pipeline that leads to accumulated errors, or require constructing the dense volumes (or even separate volumes) for each view and consumes excessive memory and computation. We are interested in the question: why unsupervised end-to-end sparsification has not been achieved yet? To sparse the volume for the next fine model initialization, previous methods like SparseNeuS <cit.> require predicting SDF values for a large number of voxels and determining whether the SDF values are within a threshold. This is a time-consuming operation (about 10s), making it impossible to train the coarse and fine stages together.
We note that some concurrent methods <cit.> directly use a large reconstruction model to achieve sparse reconstruction, but these methods are computational expensive and can only generate the low-resolution 3D representation, thus limiting their reconstruction fidelity.

In this paper, we present SuRF, the first attempt, to our knowledge, towards simultaneously unsupervised, sparsified and end-to-end approach, which provides good trade-offs between performance, efficiency, and scalability. The main idea behind this is the surface-centric modeling we adopt, which focuses more attention on regions near the surface, called “surface regions”, a practice that improves both performance and efficiency. On the one hand, the projection feature in surface regions is more multi-view consistent and more useful for geometric reasoning. On the other hand, since the surface region only occupies a small proportion of the scene bounding box, this focusing strategy can obviously save memory and computational overhead, and enable the usage of high-resolution volumes. To achieve this, we design a module called Matching Field to locate surface regions, which poses two advantages: 1) it is the first to use the weight distribution along rays to represent the geometry, and enable the use of image warping loss to achieve unsupervised training; 2) it is highly efficient that only needs an additional single-channel volume and the very-fast trilinear interpolation. Concretely, at each scale, in addition to the n-channel feature volume used for final geometric inference, we construct another single-channel matching volume for predicting the matching field.

Based on the matching field, we propose a new strategy called Region Sparsification to generate sparse volumes for later high-resolution scales. Instead of predicting the SDF values for each voxel using MLPs like existing methods, we retain only voxels in surface regions visible from at least two views, which can circumvent the influence of occlusion. Thus, we can generate multi-scale and surface-centric feature volumes to remarkably improve the geometric details of the reconstruction with less memory and computational consumption, as shown in Fig. <ref>. Extensive experiments on DTU <cit.> BlendedMVS <cit.>, Tanks and Temples <cit.> and ETH3D <cit.> datasets validate the efficiency of the proposed model, surpassing the baseline model <cit.> by more than 46% and saving more than 80% memory consumption compared with previous state-of-the-art methods <cit.>. In summary, our main contributions are highlighted below:


  * We make the first attempt to achieve unsupervised end-to-end sparsification in neural surface model for high-fidelity sparse reconstruction. 

  * We present a novel matching field to locate surface regions, which apply the weight distribution to represent the geometry and use image warping loss to achieve unsupervised training.

  * We introduce a new region sparsification strategy based on the extracted surface region that is robust to occlusions.

  * Extensive experiments on standard benchmarks validate the effectiveness of our approach from the perspectives of accuracy, efficiency and scalability.






§ RELATED WORKS



Multi-view stereo. Multi-view stereo (MVS) is a type of methods that take the stereo correspondence as the main cue to reconstruct geometry from multi-view images. Taking the scene representation as an axis of taxonomy, it can be broadly categorized into three types: voxel grids-based <cit.>, point clouds-based <cit.>, and depth map-based <cit.>.  Among them, depth map-based methods decompose complex 3D reconstructions into explicit 2D depth map estimates, becoming the most common one due to convenience. In particular, many learning-based methods <cit.> have been proposed to improve the matching accuracy through a more robust cost volume. However, the surface reconstruction of these methods is based on a multi-stage pipeline, which is cumbersome and inevitably introduces accumulated errors.

Neural surface reconstruction. Although previous volumetric methods <cit.> have achieved high-quality reconstructions, neural implicit functions have recently revealed significant potential in 3D reconstruction <cit.> and appearance modeling <cit.>. Some work <cit.> apply surface rendering to reconstruct plausible geometry without 3D supervision, but they often require extra priors like object masks <cit.> or sparse points <cit.>. Inspired by the success of NeRF <cit.> in novel view synthesis, more and more methods integrate volume rendering into shape modeling. They treat the density of volume rendering as the function of different implicit representations, , <cit.> adopts the occupancy network to represent the geometry and <cit.> apply the signed distance function to replace the local transparency function. Nevertheless, such methods suffer from lengthy per-scene optimization, cannot generalize to new scenes and perform poorly with sparse inputs.

Generalizable neural surface reconstruction. Similar to the generalizable novel view synthesis methods <cit.>, several methods <cit.> are proposed to solve the generalization of neural surface reconstruction. By replacing the input from spatial coordinates with image features, these methods can achieve impressive cross-scene generalization. Method <cit.> is the first attempt to achieve this through a multi-stage pipeline, but still struggles to recover geometric details. Even recent methods have tried to improve this through the view-dependent representation <cit.>, transformer architecture <cit.> and even build a separate cost volume for each view <cit.>, still failing to balance performance, efficiency, and scalability. To be specific, these methods are restricted by the requirement of the ground-truth depth <cit.>, cannot use high-resolution feature volumes <cit.> and cannot scale to cases with more input views <cit.> due to memory constraints. In this paper, we propose SuRF, which can reconstruct more geometric details with limited memory consumption.



§ METHODOLOGY



In this paper, our goal is to reconstruct the finely detailed and globally smooth surface 𝒮 from an arbitrary number of inputs with limited memory and computational consumption, which is achieved through our surface-centric modeling. The overall framework of our model is illustrated in Fig. <ref>. We first introduce our overall pipeline in Sec. <ref>, including how to aggregate multi-view features and reason about geometry and appearance. Then we depict our matching field in Sec. <ref>, including the unsupervised training and surface regions localization, and detail how to construct the multi-scale surface-centric feature volumes based on our new sparsification strategy in Sec. <ref>. The combination of the final loss function is described in Sec. <ref>.







 §.§ Overall Pipeline

 

Given a set of calibrated images {I_i ∈ℝ^3 × H × W}_i=1^N captured from N different viewpoints, we first extract the multi-scale features {F_i^j ∈ℝ^C × H × W}_i,j=1,1^N,L through a weight-shared FPN <cit.> network ℱ_img. To aggregate these multi-view features, we adopt an adaptive cross-scale fusion strategy, which can grasp both global and local features and is more robust to occlusion.

Cross-scale fusion. For a volume V with U number of voxels, we project each voxel 𝐯=(x,y,z) to the pixel position of corresponding viewpoint with camera intrinsics {K_i}_i=1^N and extrinsic {[R,𝐭]_i}_i=1^N:

    𝐪_i=π(K_iR_i^T(𝐯-𝐭_i)),

where π([x,y,z]^T)=[x/z,y/z]^T. The corresponding multi-scale features {𝐟_i^j ∈ℝ^C}_i,j=1,1^N,L are then sampled from all image planes via bilinear interpolation. We treat the high-scale features as detail residuals of low-scale features, and sum them together as multi-view features {𝐟_i ∈ℝ^C}_i=1^N, which are then input to a fusion network ℱ_fus to generate view's fusion weights {w_i}_i=1^N. The final fused feature for each voxel is the concatenation of weighted mean and variance features [Mean(𝐯), Var(𝐯)]:

    Mean(𝐯)=∑_i=1^Nw_i𝐟_i,  Var(𝐯)=∑_i=1^Nw_i(𝐟_i-Mean(𝐯))^2.


Further regularizing above fused features through a 3D network ℱ_3d, we can get the final single-channel matching volume V_m ∈ℝ^1× U and n-channel feature volume V_f ∈ℝ^C'× U.  Note that in our surface-centric modeling, we will generate the multi-scale feature volumes {V_f^j}_j=1^L and only voxels in the surface regions will be retained at high-resolution scales. The detailed procedure of surface region localization using our matching field will be stated in Sec. <ref>, and how to construct the multi-scale surface-centric feature volumes using our new region sparsification strategy will be elaborated in Sec. <ref>.

In this way, the surface can be reconstructed by the zero-level set of SDF values, which is estimated through a surface prediction network ℱ_sdf, which concatenate interpolations of multi-scale feature volumes as input:

    𝒮={𝐩∈ℝ^3 | ℱ_sdf(𝐩,<{V_f^j(𝐩)}_j=1^L>)=0},

where <·> is a concatenation operator. Meanwhile, since the traditional sampling operator cannot interpolate from the sparse volume, we implement a sparse trilinear sampling algorithm to achieve interpolation efficiently. Inherited from <cit.>, most methods employ a similar blending strategy to predict the color of each point on a ray:

    𝐜=∑_i=1^Nη_i𝐜̂_̂î,

where {𝐜̂_̂î}_i=1^N is the projected colors from source views, and {η}_i=1^N is the softmax-activated blending weights estimated through a color prediction network ℱ_color,
which takes projected image features and viewing direction differences as input.

Finally, alpha-composition of samples {𝐩(t_k)=𝐨+t_k𝐝|k=1,...,M} is performed to produce the color of a ray emitting from camera center 𝐨 in view direction 𝐝:

    Ĉ=∑_k=1^M T_kα_k𝐜_k,  T_k=∏_l=1^k-1(1-α_l),

where α is formulated in an unbiased and occlusion-aware conversion of SDF values:

    α_k=max(Φ_s(ℱ_sdf(𝐩(t_k)))-Φ_s(ℱ_sdf(𝐩(t_k+1)))/Φ_s(ℱ_sdf(𝐩(t_k))),0),

where Φ is the sigmoid function and s is an anneal factor, please refer to <cit.> for more details.






 §.§ Matching Field



Fig. <ref> illustrates the overall pipeline of our matching field, which is the cornerstone of surface-centric modeling. In this section, we will elaborate on it in twofold: how it achieves the surface region localization, and how it achieves unsupervised training. Note that these procedures are the same for all scales, and we omit the subscript of scales for convenience.

Surface region localization. For efficiency, we need this procedure to hold three important properties: 


  * It requires encoding the entire scene geometry with limited memory consumption

. Existing methods like <cit.> or <cit.> borrow the main idea of MVS methods <cit.> to construct a separate cost volume for each view, which is sometimes impractical for surface reconstruction, especially when there are many input views.

  * It needs to rapidly locate the surface region with a small computational cost, which makes multi-stage training or the use of extra networks unworkable.

  * It needs to be occlusion-aware and view-dependent, , those surfaces that are behind or not visible from the input views are unuseful and unsolvable.


Motivated by these properties, we implement the matching field as a weight distribution along the ray obtained from matching volume interpolation.

As shown in Fig. <ref>, instead of representing the geometry as the occupancy, density or SDF value, we employ the view-dependent weight distribution, where larger values represent closer proximity to the surface. Concretely, to extract the surface of a ray 𝐫=(𝐨,𝐝), we first uniformly sample M_s points {𝐩(t_k)=𝐨+t_k𝐝}_k=1^M_s within the current surface region (Note that M_s decreases as the scale increases). Next, we directly interpolate the corresponding value for each point from the matching volume V_m, and then go through a softmax operator to generate the weight distribution {γ_k}_k=1^M_s along this ray. In this way, we can infer the rough position of the surface point 𝐩_s=𝐨+t_s𝐝, where:

    t_s=∑_k=1^M_sγ_kt_k.


Finally, the surface region that we need is defined as: sr=[t_s-ϵ, t_s+ϵ], and ϵ is a hyperparameter that gradually decreases as the scale increases. And the surface region is set to the length of scene bounds for the first scale.

Unsupervised training. With the surface points, we can conveniently leverage the image warping loss <cit.> to constrain the matching field. Supposing the reference image I_0 has a resolution of H× W, through the matching field, we can efficiently retrieve the surface point of all rays emitting through the pixels of I_0 to form the “surface map” E_0∈ℝ^3× H× W. Then we project these points to the pixel positions of source images {I_i}_i=1^N through Eq. (<ref>), and interpolate the colors to generate the warped images {I_i^0}_i=1^N. Theoretically, the projected colors of these surface points should remain consistent across multiple viewpoints. Therefore, we can generate the constraints through the difference between the ground-truth reference image and the warped images from source images. Furthermore, we combine the pixel-wise color loss with the patch-based SSIM <cit.>:

    WL_i=0.8 ×1 - SSIM(I_0, I_i^0)/2+0.2 × |I_0-I_i^0|.


To avoid the influence of occlusions, we take the average of the K smallest warping losses as the final constraint to optimize our matching field:

    L_wl=1/K∑_i=1^KWL_i.


In this way, we can optimize our matching field unsupervised to locate the surface region efficiently.






 §.§ Feature Volume Construction based on Region Sparsification



As aforementioned and the pipeline comparisons shown in Fig. <ref>, previous methods <cit.> rely on the dense volume or multi-stage training <cit.>, and they either are limited by the memory constraints or introduce cumulative errors. To this end, based on the surface region located through our matching field, we propose a region sparsification strategy to construct the multi-scale and surface-centric feature volumes to mitigate these drawbacks.

Region sparsification. Taking a certain scale j as an example, assume that we have generated the matching volume V_m^j∈ℝ^1× U^j and feature volume V_f^j∈ℝ^C'× U^j according to the process in Sec. <ref>, and obtained the surface maps {E_i^j∈ℝ^3× H× W}_i=1^N of all views following the pipeline in Sec. <ref>. To prune those voxels away from the surface, we project all voxels Vox^j={𝐯_h}_h=1^U^j to the pixel position of all surface maps through Eq. (<ref>) and interpolate the corresponding surface points {Ê_i^j ∈ℝ^3× U^j}_i=1^N visible from each view through bilinear sampling. We then can determine whether the voxel is inside the surface region based on the distance between the voxel and the interpolated surface point:

    H_i^j(𝐯)=𝚏𝚕𝚘𝚊𝚝(∥Ê_i^j(𝐯)-𝐯∥_2 < ϵ^j),

where 𝚏𝚕𝚘𝚊𝚝 is the operator that converts bool values to float values. Furthermore, to maintain the view consistency, we only retain voxels that simultaneously fall into the surface region of at least two views:

    Vox^j+1={𝐯 | 𝚜𝚞𝚖(H^j(𝐯))≥ 2},

where 𝚜𝚞𝚖 is the summation operator. This is an important step to mitigate the impact of occlusion, as regions visible only from a small number of views are meaningless, and we depict some examples in Fig. <ref>. Then we can halve these surviving voxels to aggregate higher-frequency information for the next scale.

Repeating the region sparsification for each scale, we can generate the final multi-scale feature volumes {V_f^j}_j=1^L. While this multi-scale strategy is beneficial for the model to reconstruct surfaces with high-frequency detail and global smoothness like <cit.>, our volumes are surface-centric and can achieve higher resolution with less memory consumption. Meanwhile, since the surface region in the coarse stage is wide, using multi-scale features to predict the geometry makes the model more robust when the surface region location in the fine stage is wrong. Before employing the volume rendering to produce the color of a ray, we propose surface sampling to efficiently sample more points for surface regions. 

Surface sampling. With the off-the-shelf surface regions {sr^j}_j=1^L provided by the matching field, it's natural to sample more points within these regions because voxels inside these regions contain the most valuable information about the surface. We uniformly sample a decreasing number of points within the surface region from low-resolution to high-resolution scales, which results in more sampling points near the surface as shown in Fig. <ref>. When interpolating from multi-scale feature volumes, we fill the feature of those sampling points that are outside the surface region of certain scales with zero. Therefore, we do not need other networks to resample more fine points like <cit.>.




 §.§ Loss Function



Our overall loss function consists of three components:

    L=L_surf + L_mf,

where L_surf is used to optimize surface network and L_mf is used to optimize multi-stage matching fields. 

Following existing methods <cit.>, our surface loss L_surf is defined as:

    L_surf=L_color + L_mfc +α L_ek + β L_pe,

where L_color is computed as the average color loss of all sampled pixels Q:

    L_color=1/|Q|∑_q∈ Q |C(q)-Ĉ(q)|,

L_mfc is the feature consistency following <cit.>, and eikonal loss <cit.> is:

    L_ek=1/|P|∑_p∈ P(||∇ℱ_sdf(p)||_2-1)^2,

where P is a set of sampled 3D points. Similar to <cit.>, we leverage the pseudo label generated from the unsupervised multi-view stereo method <cit.> to enhance and accelerate model convergence. We apply a very strict filtering strategy to obtain relatively accurate pseudo point clouds P̂. The pseudo loss is:

    L_pe=1/|P̂|∑_p∈P̂|ℱ_sdf(p)|.


The loss of the matching field is defined as the weighted sum of all scales' warping loss:

    L_mf=∑_j=1^Lμ^j L_wl^j,

where μ_j is the weight of stage j, and it increases from coarse to fine scale.





§ EXPERIMENTS



In this section, we first introduce our implementation details and compared datasets, then we enumerate extensive experiments and ablation studies. Note that the reported results here are based on our model without any finetuning. Please refer to Supp. Mat. for the finetuning results.

Implementation details.
We implement our model in PyTorch <cit.>, and build our surface-centric feature volume in L=4 scales. The range of surface regions for each scale is defined as ϵ^1:ϵ^2:ϵ^3:ϵ^4=1:0.3:0.1:0.01. In our matching field, we set the number of sampling points for each scale as 128, 64, 32, 16 to extract surface regions. To apply the final volume rendering, the total number of sampling points of each ray is set to M=120, which consists of 64, 32, 16 and 8 for our surface sampling. During training, we adopt Adam optimizer <cit.> to train our model for 10 epochs. The base learning rate is set to 1e-3 for feature networks and 5e-4 for MLPs. We set the number of source images as N=4 and resize the resolution to 640× 480. The volume resolution of the first low-resolution scale is set to R^1=64×64×64. The weight of each loss term is set to α=0.1, β=1.0, and the warping loss weights of each scale are set to 0.25, 0.5, 0.75 and 1.0.  During testing, we take N=2 source images with a resolution of 800 × 576 as input, and set R^1=80×80×80. Our meshes are extracted using Marching Cubes <cit.>.

Datasets.
Following existing practices <cit.>, we train our model on the DTU dataset <cit.>, and we employ the same splitting strategies as <cit.>, , 75 scenes for training and two sets of images from 15 non-overlapping scenes for testing. To validate our generalization ability, we further conduct some qualitative comparisons on BlendedMVS <cit.>, Tanks and Temples <cit.> and ETH3D <cit.> datasets.



 §.§ Results on DTU



For a fair comparison, we adopt the same evaluation strategy as previous methods <cit.>, , reconstruct the surface using only three input views and report the average chamfer distance of two image sets.
The quantitative results on the DTU dataset are summarized in Tab. <ref>, which indicate that our SuRF can bring a satisfactory improvement to the baseline, , more than 46% improvement compared with SparseNeuS <cit.>. Meanwhile, we can surpass those methods <cit.> that employ ground-truth depth for supervision. Even compared with the method that construct separate cost volumes for each input view <cit.>, our model still offers plausible advantages in efficiency as well as scalability, that is, we only need to construct a global volume and is computation and memory insensitive to the number of input views. For those classical MVS methods <cit.>, our method can win out in most metrics. Compared with the recent fast method <cit.> that converges in minutes, our model can still reconstruct finer details in seconds. Qualitative results in Fig. <ref> further show that our SuRF can reconstruct finer surfaces only through fast network inference. The results in Fig. <ref> indicate that the 3DGS-based method SuGaR <cit.> failed with sparse inputs and our approach shows much better performance even with sparse inputs.





Number of input views. 
Fig. <ref> indicates that the reconstruction quality of our model gradually improves as the number of views increases, but it tends to stabilize when the input views are enough. Meanwhile, a larger number of inputs does not lead to a significant increase in consumption, which is an obvious advantage compared to C2F2NeuS <cit.>, as the comparison shown in Tab. <ref>.
 





 §.§ Generalization





To verify the generalization capabilities of our method, we further test on BlendedMVS <cit.>, Tanks and Temples <cit.> and ETH3D <cit.> datasets using the model pre-trained on the DTU dataset <cit.>. Some qualitative comparisons with existing methods are shown in Fig. <ref> and Fig. <ref>. The results prove that our method exhibits great generalization ability even under these difficult scenes. The volume-based method struggles on large scenes since most voxels are empty, but ours can still reconstruct meshes with fine details even without any finetuning.







 §.§ Ablation Studies


Our ablation experiments are performed on the first image set of the DTU dataset, which is the same as <cit.>. 
To investigate the effectiveness of our end-to-end sparsification, we compare it with existing solutions, , the multi-stage training strategy in SparseNeuS <cit.> and the multi-scale dense structure in GenS <cit.>. The results in (a) of Tab. <ref> show that all these solutions bring improvements to the baseline model, and our solution performs the best mainly because our features are surface-centric and can be continuously optimized. We perform another study to understand how the resolution of volumes and images affects the model. The results in (b) of Tab. <ref> show that the higher resolution favors the model but stabilizes when a certain resolution is reached. 

Efficiency and scalability. Compared with existing methods, our SuRF exhibits advantages in terms of efficiency and scalability as shown in Fig. <ref>. Our model can leverage volumes of higher resolution with less memory and computational consumption, , only 3G memory for the model with the highest 256^3 resolution volumes while at least 20G for previous methods <cit.>. Compared with methods that require predicting depth maps <cit.> or constructing cost volumes <cit.> for each view, the running time and memory consumption of our model is relatively insensitive to the number of input views as shown in Tab. <ref>, making it scalable to handle different input numbers. 
Meanwhile, the capability of using high-resolution volumes gives our method the potential to reconstruct very large-scale scenes as the results shown in Fig. <ref>.



§ CONCLUSION





In this paper, we proposed a new generalizable neural surface model, SuRF, to accomplish high-fidelity reconstruction even from sparse inputs with satisfactory trade-offs between performance, efficiency and scalability. To the best of our knowledge, it is the first unsupervised method to achieve end-to-end sparsification based on our surface-centric modeling, which consists of a novel matching field module and a new region sparsification strategy. The proposed matching field adopts the weight distribution to represent geometry and introduces the image warping loss to achieve unsupervised training, which can efficiently locate the surface region. Then we adopted the region sparsification strategy to prune voxels outside the surface regions and generated the multi-scale surface-centric feature volumes. Extensive experiments on multiple public benchmarks demonstrate that our model exhibits great generalization ability in diverse scenes and can reconstruct higher-frequency details with less memory and computational consumption. 






§ ACKNOWLEDGEMENTS


This work is financially supported by Outstanding Talents Training Fund in Shenzhen, Shenzhen Science and Technology Program-Shenzhen Cultivation of Excellent Scientific and Technological Innovation Talents project(Grant No. RCJ-C20200714114435057), Shenzhen Science and Technology Program-Shenzhen Ho-ng Kong joint funding project (Grant No. SGDX20211123144400001), National Natural Science Foundation of China U21B2012, R24115SG MIGU-PKU META VISION TECHNOLOGY INNOVATION LAB, Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology. 
Jianbo Jiao is supported by the Royal Society Short Industry Fellowship (SIF∖R1∖231009).
In addition, we sincerely thank all assigned anonymous reviewers in ECCV 2024, whose comments were constructive and very helpful to our writing and experiments.







splncs04












Surface-Centric Modeling for High-Fidelity Surface Reconstruction



R. Peng et al.



Supplementary Materials
    
    
=======================










§ RESULTS OF FINE-TUNING



As the qualitative and quantitative comparisons shown in our main paper, the reconstructions of our model without fine-tuning exhabit finest geometric details even compared with some fine-tuned models like SparseNeuS-ft <cit.>. Here, we illustrate some results of our model after fast fine-tuning. Different with methods <cit.> that require reconstructing separate cost volume for each view, our model only builds the global volume, which makes our model easily fine-tuned (only 2.5k iterations, about 10 minutes). The quantitative results in Tab. <ref> show that our model still ranks the first in most scenes and has the best mean chamfer distance. Meanwhile, it is worth noting that our volume is sparse and more memory and computationally efficient. And the qualitative results of some scenes are visulized in Fig. <ref>. Note that there are only three input views during fine-tuning.










§ MORE COMPARISONS WITH RC-MVSNET


We show some visual and metric comparisons with the TSDF fusion result of RC-MVSNet <cit.> in Fig. <ref>. The results show that the reconstruction of our model is smoother and more complete, especially in low-texture regions, leading to better results in the chamfer distance metric. To further verify the effectiveness of our surface-centric modeling, we compare with two baselines which directly use the surface point of RC-MVSNet to prune voxels: Baseline1 directly replaces the surface region of our trained model with that of RC-MVSNet; Baseline2 uses the surface region of RC-MVSNet to train a new model. Results in Tab. <ref> show that even simply using the surface region of RC-MVSNet can achieve superior results. And our full model, trained together with the surface location module (Our matching field), achieves the best performance. This is reasonable because the surface region of these two baselines was not optimized or corrected with the model when directly using the results of RC-MVSNet. 







§ DETAILED RESULTS OF VOLRECON AND RETR



As mentioned in our main paper, we report the reproduced results of VolRecon <cit.> and ReTR <cit.> on two image sets using their official repositories and released model checkpoints. The detailed reproduction results of all scenes at two image sets are illustrated in Tab. <ref>, which are slightly different from the results reported in their papers. We speculate that there are something inconsistent in the experimental configurations, but this inconsistency doesn't affect the valuable of their contributions.



§ MORE ABLATION RESULTS

Here, we report more ablation results of our model, and we set the training time to a quarter of the overall process (different from our main paper to save time) and only test on the first image set for convenience.

r0.5

  

  Ablation results on DTU dataset. 
  

  
  1.0!
  
  Number of scales     Surface sampling     Cross-scale fusion     Mean 

  1 scales     51     51     1.38 

  2 scales     51     51     1.22 

  3 scales     51     51     1.15 

  4 scales     51     51     1.11 

  5 scales     51     51     1.13 

  4 scales     51     55     1.15 

  4 scales     55     51     1.13 

  
  
  



Number of scales. We conduct some ablations to evaluate the effect of the number of scales. We set the resolution of the finest stage of each model to be similar. The results in Tab. <ref> show that the overall quality first remarkably increases and then slightly decreses, reaching the optimum in 4 scales. We illustrate some visual results of the model with differnet scales in Fig. <ref>. The single-scale model performs the worst, with reconstructions that are noisy and lack geometric detail, while the four-scales model can reconstruct smooth geometry and restore more geometric details.



r0.5
  

  Ablation results of loss weight on DTU dataset. 
  

  
  1.0!
  
  Method     β     μ^1 : μ^2 : μ^3 : μ^4     Mean 

  A     0.0     0.25:0.50:0.75:1.00     1.17 

  B     1.0     0.25:0.50:0.75:1.00     1.11 

  C     1.0     1.00:1.00:1.00:1.00     1.16 

  D     1.0     1.00:0.75:0.50:0.25     1.25 

  
  
  



Ablations on loss weight. We further conduct some experiments to verify the effect of the weight of each loss term on model performance. Concretely, we change the weight of the pseudo loss β and the weight combination of different stages of matching field loss μ^j. The ablation model is based on the 4-scales model and the results are shown in Tab. <ref>. Through the comparison between model A and model B, we can see that the pseudo point clouds generated from the unsupervised multi-view stereo method <cit.> can guide the model towards better convergence. To avoid the influence of erroneous pseudo points, we apply a very strict filtering strategy, , Only point clouds whose projection distance from at least 3 viewpoints does not exceed 0.2 pixels and whose relative depth error does not exceed 0.001 can be left. From the results of the model (B, C, D) adopt different weight combinations of the matching field loss, we can see that model B which has μ^1 : μ^2 : μ^3 : μ^4=0.25:0.50:0.75:1.00 performs the best and model D performs the worst. This indicates that applying greater weight to the high-resolution scale is beneficial to model convergence. Because there is no need to obtain very accurate predictions in the low-resolution scale, and the gradient of the high-resolution scale will be transmitted back to the low-resolution scale, it is reasonable to have a lower weight in the low-resolution scale. And we show some visual comparisons of these models in Fig. <ref>.



r0.5
  

  Ablation results of the range of surface regions.
  

  
  1.0!
  
  Method     ϵ^1 : ϵ^2 : ϵ^3 : ϵ^4     Mean 

  B     1.00:0.40:0.10:0.01     1.11 

  E     1.00:0.30:0.10:0.01     1.10 

  F     1.00:0.30:0.05:0.01     1.12 

  
  
  



Ablations on the range of surface region. Here, we employ an additional ablation experiment to study the sensitivity of the range of surface regions. ϵ^0 is the range of the first scale, and its value is fixed at 1, which represents covering the entire near and far area. And the value of later scales means the percentage of coverage. As the results shown in Tab. <ref>, the differences of these three groups of experiments are not large, as long as the surface region is gradually tightened, and model F which has a range combination of ϵ^1 : ϵ^2 : ϵ^3 : ϵ^4=1.00:0.30:0.10:0.01 performs the best. 



§ MORE RESULTS



Because C2F2NeuS <cit.> doesn't release the code, the memory of C2F2NeuS in Tab. <ref> is refer to the implementation of CasMVSNet <cit.>. Fig. <ref> shows additional comparisons with COLMAP <cit.>, NeuS <cit.>, SparseNeuS <cit.>, SparseNeuS-ft <cit.>, VolRecon <cit.> and ReTR <cit.> on DTU dataset. We can see that our method can stably achieve superior results and exhibit finer geometry details. We further show some visual comparisons with the fast per-scene overfitting method Voxurf <cit.> in Fig. <ref>. While Voxurf still requires more than 30 minutes of training time per scene, it struggles to reconstruct smooth and accurate surface from sparse inputs. 







§ VISUALIZATION OF THE SURFACE REGION



To understand how the surface region changes as scale increases, we show some visualization results of the surface region at different scales in Fig. <ref>. For convenience, we show the depth of the middle position of the surface region. We can see that the located surface region at the higher resolution scale is indeed sharper, which proves the effectiveness of our design. 





§ LIMITATIONS AND FUTURE WORK



Despite exhibiting efficiency over existing methods, our model still struggled to extract the surface in real time due to the inherent drawback of MLP-based implicit methods. In the future, we will be focusing on addressing this deficiency issue, and we have constructed a lite-version model, which will be released latter. Furthermore, we plane to train our model on more large-scale dataset like Objaverse <cit.> and expand the scale of the model like <cit.>.

