Optimizing 3D Gaussian Splatting for Sparse Viewpoint Scene Reconstruction
Shen Chen Jiale Zhou Lei Li




Optimal Regularity for Fully Nonlinear Nonlocal Equations with Unbounded Source Terms
    Disson S. dos Prazeres and Makson S. Santos
    Received 16 July 2024; accepted 04 September 2024
=====================================================================================

empty
empty



§ ABSTRACT 
3D Gaussian Splatting (3DGS) has emerged as a promising approach for 3D scene representation, offering a reduction in computational overhead compared to Neural Radiance Fields (NeRF). However, 3DGS is susceptible to high-frequency artifacts and demonstrates suboptimal performance under sparse viewpoint conditions, thereby limiting its applicability in robotics and computer vision. 
To address these limitations, we introduce SVS-GS, a novel framework for Sparse Viewpoint Scene reconstruction that integrates a 3D Gaussian smoothing filter to suppress artifacts.
Furthermore, our approach incorporates a Depth Gradient Profile Prior (DGPP) loss with a dynamic depth mask to sharpen edges and 2D diffusion with Score Distillation Sampling (SDS) loss to enhance geometric consistency in novel view synthesis. Experimental evaluations on the MipNeRF-360 and SeaThru-NeRF datasets demonstrate that SVS-GS markedly improves 3D reconstruction from sparse viewpoints, offering a robust and efficient solution for scene understanding in robotics and computer vision applications.









§ INTRODUCTION


The use of RGB cameras in robotic vision systems for 3D scene reconstruction is essential for acquiring multiple viewpoints, a fundamental requirement for high-quality novel view synthesis (NVS). However, in practical scenarios, obtaining dense multi-view data is often impractical, especially in resource-constrained or complex environments. This limitation necessitates developing methods that can achieve effective scene reconstruction from sparse viewpoints. Traditional Neural Radiance Fields (NeRF) <cit.> have shown strong performance in NVS, but their pixel-level ray rendering is computationally intensive and not well-suited for scenarios with sparse input data, requiring substantial resources and processing time.

In contrast, 3D Gaussian Splatting (3DGS) <cit.> employs an explicit representation that significantly reduces both training and rendering times while maintaining high-quality outputs. This method initializes a set of 3D Gaussians from point clouds generated by Structure from Motion (SfM) <cit.> or via random initialization. It uses adaptive density control to clone and prune these Gaussians, enhancing scene detail representation. Leveraging the smooth, differentiable properties of Gaussian distributions, 3DGS enables rapid rasterization by projecting 3D Gaussians onto 2D image planes, supporting efficient rendering and interpolation <cit.>.



3D Gaussian distributions effectively capture details across multiple scales, and their projection onto a 2D plane simplifies the rasterization process. While this method is capable of efficiently representing complex, large-scale scenes or objects, the absence of size constraints for each 3D Gaussian primitive leads to a loss of detail when reconstructing fine objects, especially upon zooming in. This limitation is particularly evident when dealing with extremely thin lines, where it can result in inaccuracies that hinder the precise capture and reproduction of slender structures and small features, thereby compromising the overall visual realism and detail fidelity of the scene <cit.>. Moreover, in practical applications, 3D Gaussian Splatting (3DGS) requires densely sampled multi-view scenes to achieve optimal results <cit.>. However, obtaining such extensive viewpoint data is often impractical in resource-constrained or complex environments. The unconstrained size of primitives in 3DGS and the reliance on dense multi-view image data present significant challenges for practical applications, such as autonomous vehicle navigation.



3DGS methods are heavily dependent on the density and quality of initial point clouds derived from dense multi-view inputs, which limits their effectiveness in sparse-viewpoint scenarios.  To address the inherent limitations of 3DGS, we propose a sparse-view 3DGS framework, termed SVS-GS. To impose size constraints on the 3D Gaussian primitives, we introduce a 3D smoothing filter <cit.>. This filter regulates the diffusion range of Gaussian primitives in both 3D space and their 2D projections, ensuring the preservation of more details during reconstruction, particularly for small and thin structures. In standard 3DGS, the initial 3D Gaussian primitives are derived from point cloud data generated by COLMAP <cit.>. However, sparse views yield a limited number of initial points, resulting in low point cloud density, which adversely affects the distribution and quality of Gaussian primitives. To enhance the density of these initial 3D Gaussian primitives, we introduce a local adaptive density scaling module. This module dynamically increases the density of Gaussian primitives based on the sparse point clouds, producing a denser set of 3D Gaussian primitives.


For the optimization of the 3D Gaussian primitives, we employ score distillation sampling (SDS) loss <cit.> to integrate 3DGS with 2D diffusion, incorporating depth prior information to constrain the positions and sizes of the 3D Gaussian primitives. Additionally, we introduce a dynamic depth mask and Gradient Profile Prior (GPP) loss <cit.> to enhance the sharpness of edges in the depth maps. SVS-GS effectively addresses gaps in the sparse point cloud data while simultaneously improving the uniformity and spatial coverage of the initial Gaussian primitives, thereby enhancing precision and detail fidelity in 3D scene reconstruction.












Our main contributions are as follows:

    
  * Novel Sparse-View Framework: SVS-GS reduces dependency on dense multi-view data by optimizing Gaussian primitive distributions, improving practicality and efficiency.
    
  * Adaptive Density Scaling: A local adaptive density scaling module generates denser initial 3D Gaussian primitives, addressing the problem of sparse point clouds.
    
  * Enhanced Optimization Techniques: Integration of SDS loss with 2D diffusion, dynamic depth masks, and depth priors ensures precise control over Gaussian primitives, improving detail reconstruction.











 §.§ Novel View Synthesis


Implicit representations for novel view synthesis (NVS), particularly Neural Radiance Field (NeRF)-based methods, have gained substantial attention in recent years <cit.>. NeRF <cit.> utilizes a multi-layer perceptron (MLP) <cit.> to predict radiance and density at 3D locations and viewing directions, leveraging classical volume rendering techniques <cit.> to generate high-quality novel views. Despite their strengths, these methods can produce artifacts when handling high-frequency details. To address this, Mip-NeRF <cit.> introduces multi-scale features and anti-aliased conical frustums to minimize blurring. While NeRF-based approaches are effective for objects and small-scale scenes, inaccuracies in camera parameters can accumulate errors in large-scale, unbounded environments, affecting reconstruction quality. Mip-NeRF 360 <cit.> alleviates these issues with non-linear scene parameterization and online distillation techniques to reduce artifacts in large-scale scenes.

In scenarios with sparse input views, NeRF models are prone to overfitting, which limits their ability to generalize to novel perspectives <cit.>. Several methods have been proposed to enhance reconstruction accuracy in such settings. Depth-Supervised NeRF (DSNeRF) <cit.> combines color and depth supervision to produce more detailed scenes, while SPARF <cit.> uses pixel matching and depth consistency loss to achieve high-precision 3D scene generation from sparse inputs. 
















 §.§ Primitive-Based Rendering


Primitive-based rendering techniques, which rasterize geometric primitives onto a 2D plane, have gained widespread adoption due to their high efficiency <cit.>. Differentiable point-based rendering methods <cit.> are particularly effective for novel view synthesis (NVS) because they offer optimization-friendly representations of complex scene structures. Recently, the introduction of 3D Gaussian Splatting (3DGS) <cit.> has renewed interest in explicit representation methods. Unlike implicit representations, explicit representations directly encode the geometry and lighting information of a scene, reducing computational complexity. However, 3DGS adapts Gaussian primitives to each training image independently, often neglecting the global structural coherence of the scene <cit.>. Additionally, the lack of size constraints during training can lead to artifacts in rendered novel views. To address these issues, Structured 3D Gaussians (Scaffold-GS) <cit.> introduces anchor points to guide the distribution of 3D Gaussian primitives, enhancing the structural integrity of the scene. Mip-Splatting <cit.> further improves 3DGS by incorporating a 3D smoothing filter and a 2D mipmap filter to constrain the size of Gaussian primitives, thereby capturing finer scene details.

Most 3DGS-based methods initialize using point clouds generated from Structure-from-Motion (SfM) techniques, such as COLMAP. These methods rely on dense input images to maintain sufficient point cloud density, which is crucial for high-quality scene reconstruction. When the input images are sparse, the resulting point clouds also become sparse, limiting the capacity of 3D Gaussian primitives to capture intricate geometric details during generation and optimization <cit.>. This sparsity can cause the models to overfit to the limited training views, thereby hindering generalization to novel viewpoints and reducing the effectiveness of scene reconstruction. SparseGS <cit.> attempts to mitigate the dependency on dense input by incorporating 2D diffusion and depth information.







§ PRELIMINARIES


3DGS employs anisotropic Gaussians to effectively capture the varying scales and orientations present within a scene. Each 3D Gaussian primitive, denoted as {𝒢_n | n = 1, …, N }, is characterized by several parameters: a center position μ_n ∈ℝ^3 × 1, a covariance Σ_n ∈ℝ^7, a color c_n ∈ℝ^3, and an opacity α_n ∈ℝ^1. The Gaussian function is defined as:

    𝒢_n(x) = e^-1/2(x-μ_n)^T Σ^-1_n (x-μ_n),

where x denotes points queried around the center position μ_n. The size and orientation of each 3D Gaussian primitive are determined by the semi-definite parameters Σ_n = R_n S_n (R_n S_n)^T, where R_n ∈ℝ^4 represents a rotation matrix and S_n ∈ℝ^3 is a scaling matrix.

To render images from different viewpoints, differential splatting is applied to project the 3D Gaussians onto camera planes. This process involves the viewing transformation W_n and the Jacobian matrix J_n, resulting in a transformed covariance:

    Σ'_n = J_n W_n Σ_n (J_n W_n)^T.

For color construction, 3DGS utilizes spherical harmonics to model the color c_n of each Gaussian, incorporating its opacity α_n. When rendering from a novel viewpoint, the 3D Gaussians are projected onto 2D planes, and the resulting color C_r(x) for a given ray r is computed as:

    C_r(x) = ∑_i ∈ M c_i σ_i ∏_j=1^i-1 (1 - σ_j),   σ_i = α_i 𝒢^2D_i(x),

where c_i and α_i represent the color and opacity of the i-th Gaussian, respectively. Here, the ray r originates from the camera center corresponding to the observation viewpoint. Finally, an adaptive density control mechanism is implemented to dynamically clone and prune the 3D Gaussians, maintaining a balance between computational efficiency and scene detail.





























§ METHODS




 §.§ Problem Formulation


In the context of scene reconstruction, optimizing the initialized 3D Gaussian primitives necessitates a set of multi-view images I = {I_1, I_2, …, I_k} and the corresponding point clouds P = { p_1, p_2, …, p_M }. The multi-view images I are first utilized to generate an initial point cloud P through Structure-from-Motion (SfM) techniques. Subsequently, these images guide the optimization of 3D Gaussian Splatting (3DGS) by comparing them with the rendered images, thereby refining the 3D Gaussian primitives to improve scene representation. 

The quality of novel view synthesis (NVS) in 3DGS is heavily influenced by the density and distribution of point clouds P and the quality of the input multi-view images I. When robotic vision systems rely exclusively on RGB cameras with limited data, the resulting sparse point clouds and input images can significantly impair the completeness and level of detail in the geometric representation, limiting the capacity of 3DGS to accurately capture scene complexity. This limitation becomes particularly critical in complex or unbounded environments, where inadequate data hampers the ability to represent intricate geometric structures and variations in lighting, thereby reducing the effectiveness of scene reconstruction.








 §.§ Initialize Adaptive Dense

In 3D scene reconstruction, a combined strategy of global and local processing is employed to balance the accuracy of the overall structure with the refinement of local details. 
Global processing is responsible for capturing the broad geometric structure of the entire scene, while local processing focuses on enhancing the detail representation within specific regions.


  §.§.§ Global Processing

The primary objective of global processing is to ensure the geometric consistency of the entire scene. Using the point clouds P_init = {p_i | i = 1, …, k} generated by SfM, we first address the overall structure to obtain a comprehensive spatial framework and point cloud density distribution.
The global processing optimizes P_init to derive a global density function ρ(p):

    ρ_global(p) = ∫_Pexp( - p - q^2/2σ_p^2) f(q)   dq,

where each point p_i ∈ P has coordinates (x_i, y_i, z_i), 
q represents the potential nearest neighbors of the point p.
f(q) is the density function, representing the weight or density at point q. 
This density function is utilized to assess the distribution of points across the point clouds, ensuring that the essential geometric structures are retained at the global level.



  §.§.§ Local Processing

Following global processing, the point clouds are partitioned into several local regions N, where each region undergoes more detailed optimization. The main goal of local processing is to enhance the representation of fine details. 
For a local region R_i, the bounding box is defined as:

    p_min_i = min(p_R_i),    p_max_i = max(p_R_i),

where p_R_i denotes the points within the region R_i. The position of the newly generated points p_r∈[ p_min_i, p_max_i] is determined by uniform sampling within this bounding box.
The local point cloud density function ρ_local(p) is further refined to capture intricate geometric details:

    ρ_local(p_r) = ∫_R_iexp( - p_r - q_r^2/2σ_p_r^2) f(q_r)   dq_r,

where R_i represents the integration domain, which encompasses the entire range of possible values for the local region around p_r; 
q_r represents the potential nearest neighbors of the point p_r.



  §.§.§ Density-weighted selection

Upon completing the local and global density estimations, the point selection process strategically integrates these results, optimizing the balance between local precision and global coherence to enhance the overall quality of the reconstruction.

Initially, within each local region, a KD-tree <cit.> is constructed to identify the k nearest neighbors p_i for each point p. 
The distances between p and these neighbors are calculated and then converted into local density values ρ_local(p) using a Gaussian function. Based on these density values, the probability of retaining each point ℙ_local is determined:

    ℙ_local(p_r_j∈ p_r) ∝ρ_local(p_r).

Simultaneously, a similar process is conducted at the global level. The global density ρ_global(p) is estimated by calculating the distances to the global nearest neighbors p_i, and the corresponding global retention probability ℙ_global is computed:

    ℙ_global(p_i∈ p) ∝ρ_global(p).


The selected points from both the local P_local and global P_global density estimations are combined with the initial point cloud P_init using a union operation, resulting in the final point cloud P_final:

    P_final = P_init⊕ P_local⊕ P_global.

This approach ensures that both the global structural integrity and local detail accuracy are maintained, thereby improving the overall quality and precision.




 §.§ 3D Smoothing

The intrinsic and extrinsic parameters of the camera are not fixed, leading to varying degrees of artifacts when rendering novel views, especially upon magnification.
In the optimization process, the coordinates o_i = (x_o_i,y_o_i,z_o_i) of any arbitrary 3D Gaussian need to be transformed from the world coordinate system to each coordinate system of camera:

    e_i = o_i R_i + T_i = (x_e_i,y_e_i,z_e_i),

where R_i and T_i represent the rotation matrix and  translation matrix for the i-th camera. The transformed point is then projected onto the image plane using the intrinsic matrix of the camera:

    x^s_i = x_e_i/z_e_i· f_i,x + W_i/2,    y^s_i = y_e_i/z_e_i· f_i,y + H_i/2,

where f_i represents the focal length of the i-th camera; H_i and W_i represent the height and width of the image, respectively. 
The maximum Gaussian point frequency β_k is obtained using the observed positions of the 3D Gaussians on the screen:

    ζ_k = sup( f_i/z_e_i),

where x_i^s ∈ [-α W_i, (1+α) W_i] and y_i^s ∈ [-α H_i, (1+α) H_i]. The hyperparameter α is used to extend the boundary of the image plane, ensuring that points near the image edges are considered.

After 3D smoothing filtering, the 3D Gaussian is represented as follows:

    𝒢_k(x) = √(Σ_k/Σ_k_s)· 
        e^-1/2(𝐱 - μ_k)^T Σ_k_s^-1 (𝐱 - μ_k),

where Σ_k_s = Σ_k + s/ζ_k^2·𝐈 represents the covariance matrix after filtering. 



 §.§ Depth SDS as Optimization Guidance

Using the diffusion model to generate spatially aligned RGB images and depth maps, we can guide the 3DGS optimization process in both structure and texture.
The depth map for each view is computed by accumulating the depth values of 𝒩 ordered Gaussian primitives along the ray, using point-based α blending:

    D_r(x) = ∑_i ∈𝒩 d_μ_iσ_i ∏_j=1^i-1 (1 - σ_j),

where d_μ_i is the depth of the i-th Gaussian primitive center μ_i in the camera view. All depth maps from the training views are normalized for subsequent depth-based loss calculation.

We employ SDS <cit.> to guide the optimization of 3DGS through 2D diffusion <cit.>.
The rendered image Ĩ and depth map D̃ from unseen viewpoints v are jointly used to optimize 3DGS through SDS:

    ∇_θℒ_SDS   = λ_1 ·𝔼_ϵ_I, t[ w_t ( ϵ_ϕ (I_t; Ĩ^v,t) - ϵ_I ) ∂ I_t/∂θ] 
       + λ_2 ·𝔼_ϵ_D, t[ w_t ( ϵ_ϕ (D_t; D̃^v, t) - ϵ_D ) ∂ D_t/∂θ],

where λ_1 and λ_2 are coefficients that balance the influence of image and depth; 
ϵ_ϕ(.) is the denoising function of 2D diffusion; 
ϵ_I, ϵ_D ∼ N(0, I) are independent Gaussian noises.
By integrating the 2D diffusion model, 3DGS can be optimized more effectively, enabling the generated images and depth maps from new viewpoints to more accurately reflect the geometric structure and textural details of the actual scene.























 §.§ Depth mask and Gradient Profile Prior


Since noise and irrelevant details in the distant background can negatively impact the gradient calculation process, leading to blurred edges and loss of detail in the reconstruction, we introduce a dynamic depth mask to effectively suppress high-frequency noise and artifacts from distant objects, thereby improving the geometric accuracy and visual quality of the reconstruction. 
To accommodate scenes with varying depth distributions, q_f for the far-distance threshold is calculated as follows:

    q_f = q_b + (β_D/β_D + α_D) ×Δ q,

where α_D and β_D represent the mean and standard deviation of the depth map D, respectively. p_b is the base quantile, and Δ p is the dynamic adjustment range. The generated mask M is defined as:

    M = 1_D ≤ T_f = 1_D ≤Quantile(D, q_f),

where 1_(·) is an indicator function that assesses the visibility of depth map D. The mask is determined by calculating the value T_f at the quantile p_f of the depth map D.
The final masked depth map (D_m = D ⊙ M) is used for gradient operations.


The Depth Gradient Profile Prior (DGPP) is introduced to enhance the sharpness and accuracy of edges in the depth map, particularly focusing on refining the texture and geometric details.
The GPP loss is formulated to enforce the alignment of gradient profiles between the rendered depth map D̂_m and the target depth map D_m. 
When the pixel positions b of D̂_m and D_m correspond one-to-one, the DGPP loss function is defined as:

    ℒ_DGPP = 1/b_1 - b_0∫_b_0^b_1∇D̂_m(b) - ∇D_m(b)_1   db,

where ∇D̂_m and ∇ D_m represent the gradient fields of the rendered and target depth maps, respectively. 
The depth alignment ensures that the sharpness of edges is preserved and that the 3D reconstruction accurately reflects the underlying geometry.



 §.§ Loss Function


To optimize the 3D Gaussian representation ({θ_k=(μ_k, Σ_k,α_k, c_k) }^K_k), we designed a final optimization function that integrates various loss terms.
Our final loss function for optimizing 3D Gaussians is defined as: 

    ℒ_final   =
        ℒ_RGB(Î(θ),I) + λ_depthℒ_depth(D̂(θ),D,M)_loss of know view
        + λ_SDSℒ_SDS(Ĩ^v(θ), D̃^v(θ))_loss of novel view,

where Î, Ĩ^v  represent the RGB images rendered by the 3D Gaussian primitives; I represents the reference RGB image; D̂, Ĩ^v represent the depth maps rendered by the 3D Gaussian primitives; D represents the reference depth map.



§ EXPERIMENTS









 §.§ Datasets and Evaluation Metrics


For unbounded scenes, we select six 360° coverage scenes from Mip-NeRF 360 <cit.> to evaluate our model. For underwater scenes, we used the SeaThru-NeRF dataset <cit.> to evaluate the applicability of our framework to other complex scenes.
We employ tree metrics (PSNR, SSIM <cit.>, and LPIPS <cit.>), to evaluate and compare our method against existing approaches.



 §.§ Implementation Details

Our method is implemented using the PyTorch <cit.> framework and the open-source 3DGS <cit.> codebase. AdamW <cit.> is employed as the optimizer. For all scenes, the models were trained for 30K iterations using the same loss function, Gaussian density control strategy, and hyperparameters to optimize the 3D Gaussian primitives. Both Gaussian training and rendering tests were performed on a NVIDIA^TM RTX 4090 GPU.



 §.§ Qualitative and Quantitative Evaluation



 







In the qualitative analysis on MipNeRF360, as shown in the Fig.<ref>, we compared the performance of different methods in reconstructing complex scenes. 

When reconstructing unbounded scenes, SVS-GS, with its integration of 3D Gaussian smoothing and depth priors, clearly outperforms traditional 3DGS and SparseGS methods by successfully capturing more intricate structures and lighting variations. These results further confirm the advantages and practical effectiveness of SVS-GS in sparse view scene reconstruction. Similarly, on the SeaThru-NeRF underwater dataset, SVS-GS again outperformed other methods, as shown in the Fig.<ref>. Particularly in handling the challenges of complex underwater lighting conditions and sparse viewpoints, SVS-GS demonstrated greater robustness and accuracy, successfully reducing visual distortions and preserving more scene details. These quantitative results underscore the broad applicability of SVS-GS across different scenarios and viewpoint conditions.




In the quantitative analysis, we systematically evaluated the performance of SVS-GS against other methods on the MipNeRF360 and SeaThru-NeRF datasets, as shown in the Table.<ref>. Comparison of the PSNR, SSIM, and LPIPS metrics clearly demonstrates the significant advantages of SVS-GS in terms of reconstruction accuracy and image quality. On the MipNeRF360 dataset, SVS-GS achieved the highest scores in both PSNR and SSIM, indicating its superior ability to reconstruct geometric and textural details in sparse views, while also exhibiting the lowest perceptual error in the LPIPS, further validating its visual fidelity.




 §.§ Ablations and Analysis

As shown in Table.<ref>, we conducted ablation studies to evaluate the impact of key components in our method. 
The dynamic depth mask plays a crucial role in effectively reducing noise and artifacts in distant areas, confirming its importance in filtering out irrelevant depth information.
DGPP sharpens edge contours, highlighting its importance in preserving details. 
Additionally, omitting the 3D Gaussian smoothing filter results in a noticeable increase in surface noise and artifacts, demonstrating its essential role in maintaining the smoothness and consistency of the reconstructed surfaces. The lack of SDS leads to geometric inconsistencies in the synthesized novel views, emphasizing the necessity of this component in ensuring geometric coherence and minimizing visual discrepancies.
Each component contributes to the effectiveness of achieving high-quality 3D scene reconstruction.






§ CONCLUSION

In this paper, we introduce SVS-GS, a novel framework for 3D scene reconstruction from sparse viewpoints, optimized for both robotic vision systems and broader computer vision tasks using only RGB cameras. Our method utilizes a dynamic depth mask to enhance geometric accuracy by selectively retaining critical depth information. Additionally, by incorporating depth priors, a 3D Gaussian smoothing filter, and Depth Gradient Profile Prior (DGPP) loss, our approach sharpens edges and preserves fine details in complex scenes. To ensure high-quality and consistent novel view synthesis, we integrate Score Distillation Sampling (SDS) loss, which reduces noise and maintains geometric coherence across different viewpoints. Experimental results demonstrate that SVS-GS outperforms existing methods in sparse viewpoint scenarios, achieving superior visual fidelity and geometric consistency. Furthermore, our framework shows robust performance across various challenging environments, making it an efficient and effective solution for 3D scene reconstruction in both robotics and computer vision applications.

















IEEEbib


