LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors
Hanyang Yu Xiaoxiao Long Ping Tan






  



Journal of  Class Files, Vol. 14, No. 8, Sep 2024
Shell et al.: A Sample Article Using IEEEtran.cls for IEEE Journals

LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors
    Hanyang Yu^1,
        Xiaoxiao Long^1,
        and Ping Tan^1
        ^ Xiaoxiao Long is the corresponding author (xxlong@connect.hku.hk).
^1 The Hong Kong University of Science and Technology 
Submitted for review on Sep. 4th, 2024.
    September 9, 2024
=============================================================================================================================================================================================================================================




§ ABSTRACT 
We aim to address sparse-view reconstruction of a 3D scene by leveraging priors from large-scale vision models. While recent advancements such as 3D Gaussian Splatting (3DGS) have demonstrated remarkable successes in 3D reconstruction, these methods typically necessitate hundreds of input images that densely capture the underlying scene, making them time-consuming and impractical for real-world applications. However, sparse-view reconstruction is inherently ill-posed and under-constrained, often resulting in inferior and incomplete outcomes. This is due to issues such as failed initialization, overfitting on input images, and a lack of details.
To mitigate these challenges, we introduce LM-Gaussian, a method capable of generating high-quality reconstructions from a limited number of images. Specifically, we propose a robust initialization module that leverages stereo priors to aid in the recovery of camera poses and the reliable point clouds. Additionally, a diffusion-based refinement is iteratively applied to incorporate image diffusion priors into the Gaussian optimization process to preserve intricate scene details. Finally, we utilize video diffusion priors to further enhance the rendered images for realistic visual effects.



Overall, our approach significantly reduces the data acquisition requirements compared to previous 3DGS methods. We validate the effectiveness of our framework through experiments on various public datasets, demonstrating its potential for high-quality 360-degree scene reconstruction. Visual results are on our websitehttps://runningneverstop.github.io/lm-gaussian.github.io/(lm-gaussian.github.io).





sparse-view, reconstruction, gaussian splatting, large models.




§ INTRODUCTION


3D scene reconstruction and novel view synthesis from sparse-view images present significant challenges in the field of computer vision. Recent advancements in neural radiance fields (NeRF) <cit.> and 3D Gaussian splatting (3DGS) <cit.> have made notable progresses in synthesizing novel views, but they typically require hundreds of images to reconstruct a scene. Capturing such a dense set of images is often impractical, raising the inconvenience for utilizing these technologies. Although efforts have been made to address sparse-view settings, existing works are still limited to straightforward facing scenarios, such as the LLFF dataset <cit.>, which involve small-angle rotations and simple orientations. For large-scale 360-degree scenes, the problems of being ill-posed and under-constrained hinder the employment of these methods. In this work, we present a new method that is capable of producing high-quality reconstruction from sparse input images, demonstrating promising results even in challenging 360-degree scenes.


There are three main obstacles that prevent 3D Gaussian splatting from achieving high-quality 3D reconstruction with sparse-view images. 1) Failed initialization: 3DGS heavily relies on pre-calculated camera poses and point clouds for initializing Gaussian spheres. However, traditional Structure-from-Motion (SfM) techniques <cit.> cannot successfully handle the sparse-view setting due to insufficient overlap among the input images, therefore yielding inaccurate camera poses and unreliable point clouds for 3DGS initialization. 2) Overfitting on input images: Lacking sufficient images to provide constraints, 3DGS tends to be overfitted on the sparse input images and therefore produces novel synthesized views with severe artifacts. 3) Lack of details: Given limited multi-view constraints and geometric cues, 3DGS always fails to recover the details of the captured 3D scene and the unobserved regions, which significantly degrades the final reconstruction quality.

To tackle these challenges, we introduce LM-Gaussian, a novel method capable of producing high-quality reconstructions from sparse input images by incorporating large model priors. The key idea is leveraging the power of various large model priors to boost the reconstruction of 3D gaussian splatting with three primary objectives: 1) Robust initialization; 2) Overfitting prevention; 3) Detail preservation.


For robust initialization, instead of relying on traditional SfM methods <cit.>, we propose a novel initialization module utilizing stereo priors from DUSt3R <cit.>. DUSt3R is a comprehensive stereo model that takes pairs of images as input and directly generates corresponding 3D point clouds. Through a global optimization process, it derives camera poses from the input images and establishes a globally registered point cloud.
However, the global point cloud often exhibits artifacts and floaters in background regions due to the inherent bias of DUSt3R towards foreground regions.
To mitigate this issue, we introduce a Background-Aware Depth-guided Initialization module. Initially, we use depth priors to refine the point clouds produced by DUSt3R, particularly in the background areas of the scene. Additionally, we employ iterative filtering operations to eliminate unreliable 3D points by conducting geometric consistency checks and confidence-based evaluations. This approach ensures the generation of a clean and reliable 3D point cloud for initializing 3D Gaussian splatting.

Once a robust initialization is obtained, photo-metric loss is commonly used to optimize 3D Gaussian spheres. 
However, in the sparse-view setting, solely using photo-metric loss will make 3DGS overfit on input images. To address this issue, we introduce multiple geometric constraints to regularize the optimization of 3DGS effectively.
Firstly, a multi-scale depth regularization term is incorporated to encourage 3DGS to capture both local and global geometric structures of depth priors. Secondly, a cosine-constrained normal regularization term is introduced to ensure that the geometric variations of 3DGS to be aligned with normal priors. Lastly, a weighted virtual-view regularization term is applied to enhance the resilience of 3DGS to unseen view directions.

To preserve intricate scene details, we introduce Iterative Gaussian Refinement Module, which leverages diffusion priors to recover high-frequency details. 
We leverage a diffusion-based Gaussian repair model to restore the images rendered from 3DGS, aiming to enhance image details with good visual effects. The enhanced images are used as additional pseudo ground-truth to optimize 3DGS. Such a refinement operation is iteratively employed in 3DGS optimization, which gradually inject the image diffusion priors into 3DGS for detail enhancement.
Specifically, the Gaussian repair model is built on ControlNet with injected Lora layers, where the sparse input images are used to finetune the Lora layers so that repair model could work well on specific scenes.



By combining the strengths of different large model priors, LM-Gaussian can synthesize new views with competitive quality and superior details compared to state-of-the-art methods in sparse-view settings, particularly in 360-degree scenes. The contributions of our method can be summarized as follows:


  * We propose a new method capable of generating high-quality novel views in a sparse-view setting with large model priors. Our method surpasses recent works in sparse-view settings, especially in large-scale 360-degree scenes.


  * We introduce a Background-Aware Depth-guided Initialization Module, capable of simultaneously reconstructing high-quality dense point clouds and camera poses for initialization.


  * We introduce a Multi-modal Regularized Gaussian Reconstruction Module that leverages regularization techniques from various domains to avoid overfitting issues.


  * We present an Iterative Gaussian Refinement Module which uses diffusion priors to recover scene details and achieve high-quality novel view synthesis results.




§ RELATED WORK

3D Representations for Novel-view synthesis 
Novel view synthesis (NVS) involves rendering unseen viewpoints of a scene from a given set of images. One popular approach is Neural Radiance Fields (NeRF), which uses a Multilayer Perceptron (MLP) to represent 3D scenes and renders via volume rendering. Several works have aimed to enhance NeRF's performance by addressing aspects such as speed <cit.>, quality <cit.>, and adapting it to novel tasks <cit.>. While NeRF relies on a neural network to represent the radiance field, 3D Gaussian Splatting (3DGS)<cit.> stands out by using an ensemble of anisotropic 3D Gaussians to represent the scene and employs differentiable splatting for rendering. This approach has shown remarkable success in efficiently and accurately reconstructing complex real-world scenes with superior quality. Recent works have further extended the capabilities of 3DGS to perform various downstream tasks, including text-to-3D generation<cit.>, dynamic scene representation <cit.>, editing <cit.>, compression <cit.>, SLAM <cit.>, animating humans <cit.>, and other novel tasks <cit.>.




Sparse View Scene Reconstruction and Synthesis
Sparse view reconstruction aims to reconstruct a scene using a limited number of input views. Existing methods can be classified into regularization techniques and generalizable reconstruction priors. Several works <cit.> address this challenge by employing strategies like depth regularization and frequency regularization. Some approaches <cit.> utilize pre-trained models as a regularization mechanism to guide training based on established knowledge. Another research direction <cit.> focuses on training priors to synthesize novel views across diverse scenes. Building on the achievements of 3D Gaussian Splatting, recent methods such as SparseGS <cit.>, pixelSplat <cit.>, and MVSplat <cit.> leverage stereo view interpolation to facilitate training. GeNVS <cit.> and latentSplat <cit.> utilize rendering view-conditioned feature fields followed by 2D generative decoding to generate novel views. While these methods have demonstrated improved results in new view synthesis, they still encounter challenges in producing clear views under high uncertainty and may encounter difficulties in handling 360-degree scenes.







Unposed Scene Reconstruction
The methods mentioned above all rely on known camera poses, and Structure from Motion (SfM) algorithms often struggle to predict camera poses and point clouds with sparse inputs, mainly due to a lack of image correspondences. Therefore, removing camera parameter preprocessing is another active line of research. For instance, iNeRF <cit.> demonstrates that poses for new view images can be estimated using a reconstructed NeRF model. NeRFmm <cit.> concurrently optimizes camera intrinsics, extrinsics, and NeRF training. BARF <cit.> introduces a coarse-to-fine positional encoding strategy for joint optimization of camera poses and NeRF. GARF <cit.> illustrates that utilizing Gaussian-MLPs simplifies and enhances the accuracy of joint pose and scene optimization. Recent works like Nope-NeRF <cit.>, LocalRF <cit.>, and CF-3DGS <cit.> leverage depth information to constrain NeRF or 3DGS optimization. While demonstrating promising outcomes on forward-facing datasets such as LLFF <cit.>, these methods encounter challenges when dealing with complex camera trajectories involving significant camera motion, such as 360-degree large-scale scenes.


§ PRELIMINARY




 §.§ 3D Gaussian Splatting

3D Gaussian Splatting (3D-GS) represents a 3D scene with a set of 3D Gaussians. Specifically, a Gaussian primitive can be defined by a center μ∈ℝ^3, a scaling factor s ∈ℝ^3, and a rotation quaternion q ∈ℝ^4. Each 3D Gaussian is characterized by:

    G(x)=1/(2π)^3/2|Σ|^1/2e^-1/2(x - μ)^TΣ^-1(x - μ)

where the covariance matrix Σ can be derived from the scale s and rotation q.

To render an image from a specified viewpoint, the color of each pixel p is computed by blending K ordered Gaussians {G_i | i=1, ⋯ ,K} that overlap with p using the following blending equation:


    c(p)=∑_i=1^K c_i α_i ∏_j=1^i-1(1-α_j),


where α_i is determined by evaluating a projected 2D Gaussian from G_i at p multiplied by a learned opacity of G_i, and c_i represents the learnable color of G_i. The Gaussians covering p are sorted based on their depths under the current viewpoint. Leveraging differentiable rendering techniques, all attributes of the Gaussians can be optimized end-to-end through training for view reconstruction.


Rasterizing Depth for Gaussians 
Following the depth calculation approach introduced in RaDe-GS <cit.>, the center μ_i of a Gaussian G_i is initially projected into the camera coordinate system as μ_i^'. Upon obtaining the center value (x_i', y_i', z_i') for each Gaussian, the depth (x,y,z) of each pixel is computed as:

    d =  z_i' + 𝐩[ Δ x; Δ y ], 
        μ_i^' = [[ x_i'; y_i'; z_i' ]]=𝐖μ_i+𝐭,

where z_i' represents the depth of the Gaussian center, Δ x = x_i' - x and Δ y = y_i' - y denote the relative pixel positions. The vector 𝐩 is determined by the Gaussian parameters [𝐖, 𝐭] ∈ℝ^3 × 4. 
 

Rasterizing Normal for Gaussians
In accordance with RaDe-GS, the normal direction of the projected Gaussian is aligned with the plane's normal. To compute the normal map, we transform the normal vector from the 'rayspace' to the 'camera space' as follows:

    𝐧 = -𝐉^⊤(
    𝐳_𝐢'/𝐳_𝐢𝐩   1 )^⊤,

where 𝐉 represents the local affine matrix, and the vector 𝐩 has been defined earlier. 



 §.§ Diffusion model

In recent years, diffusion models have emerged as the state-of-the-art approach for image synthesis. These models are characterized by a predefined forward noising process {𝐳_𝐭}_t=1^T that progressively corrupts the data by introducing random noise ϵ.

    z_t = √(α_t)z_0 + √(1-α_t)ϵ, ϵ∈𝐍(0, 𝐈),

where t ∈ [1, T] denotes the time step and α_t=α_1·α_2...α_t represents a decreasing sequence. These models can generate samples from the underlying data distribution given pure noise by training a neural network to learn a reversed denoising process. Having learned from hundreds of millions of images from the internet, diffusion priors exhibit a remarkable capacity to recover real-world details.




§ METHOD






 §.§ Overview

In this paper, we introduce a new method called LM-Gaussian, which aims to generate high-quality novel views of 360-degree scenes using a limited number of input images. Our approach integrates multiple large model priors and is composed of four key modules:
1) Background-Aware Depth-guided Initialization:  This module extends DUSt3R for camera pose estimation and detailed 3D point cloud creation. By integrating depth priors and point cleaning, we achieve a high-quality point cloud for Gaussian initialization (see Section <ref>).
2) Multi-Modal Regularized Gaussian Reconstruction: In addition to the photometric loss used in 3DGS, we incorporate depth, normal, and virtual-view constraints to regularize the optimization process (see Section <ref>).
3) Iterative Gaussian Refinement: We use image diffusion priors to enhance rendered images from 3DGS. These improved images further refine 3DGS optimization iteratively, incorporating diffusion model priors to boost detail and quality in novel view synthesis (see Section <ref>).
4) Scene Enhancement: In addition to image diffusion priors, we apply video diffusion priors to further enhance the rendered images from 3DGS, enhancing the realism of visual effects (refer to Section <ref>).






 §.§ Background-Aware Depth-guided Initialization 
 

Traditionally, 3DGS relies on point clouds and camera poses calculated through Structure from Motion (SfM) methods for initialization. However, SfM methods often encounter challenges in sparse view settings. To address this issue, we propose leveraging stereo priors <cit.> as a solution. DUSt3R, an end-to-end dense stereo model, can take sparse views as input and produce dense point clouds along with camera poses. Nevertheless, the point clouds generated by DUSt3R are prone to issues such as floating objects, artifacts, and distortion, particularly in the background of the 3D scene.

To overcome these challenges, we introduce the Background-Aware Depth-guided Initialization module to generate dense and precise point clouds. This module incorporates four key techniques:
1) Camera Pose Recovery: Initially, sparse images are used to generate point clouds for each image using DUSt3R. Subsequently, the camera poses and point clouds are aligned into a globally consistent coordinate system.
2) Depth-guided Optimization: Depth-guided optimization is then employed to refine the aligned point cloud. In this step, a monocular estimation model is used as guidance for the optimization process.
3) Point Cloud Cleaning: Two strategies are implemented for point cloud cleaning: geometry-based cleaning and confidence-based cleaning. During optimization, after every ξ iterations, a geometry-based cleaning step is executed to remove unreliable floaters. Following the optimization process, confidence-based cleaning is applied to distinguish between foreground and background, utilizing specific filtering techniques to preserve the final output point cloud. 
Next, we will provide detailed insights into the implementation of each component within this module.


Camera pose recovery 

We first use minimum spanning tree algorithm <cit.> to align all camera poses and point clouds into a unified coordinate system. An optimization scheme is then utilized to enhance the quality of the aligned point clouds. Initially, following the approach of DUSt3R, a point cloud projection loss ℒ_pc is minimized. Consider the image pair  {I_k, I_l} where P_k and P_l denote the point map in the  k_th and l_th camera's coordinate system. The objective is to evaluate the consistency of 3D points in the k_th coordinate system with those in the l_th coordinate system. The projection loss is computed by projecting the point map P_l to the k_th coordinate system using a transformation matrix T_k,l that converts from the l_th coordinate system to the k_th coordinate system. The loss parameters include the transformation matrix T_k,l, a scaling factor σ_k,l, and P_k. This process is repeated for the remaining image pairs.

    ℒ_pc = ∑_k∈ K∑_l∈ K∖{k}η_k·η_l ‖P_k - σ_k,lT_k,lP_l‖


The purpose of this loss function is to systematically pair each input image like I_k with all other images such as I_l. For the image pair {I_k, I_l}, the loss function measures the disparity between the point map P_k in the k_th coordinate system and the transformed point map σ_k,lT_k,lP_l. These comparisons are weighted by their respective confidence maps η_k and η_l.

Depth-guided Optimization
The optimization based solely on the projection loss may not be sufficient for reconstructing large-scale scenes, as it could lead to issues like floaters and scene distortion that can impact subsequent reconstructions. To tackle scene distortion, we integrate a robust model prior to guide the optimization network. Marigold, a diffusion-based monocular depth estimation model known for its top-tier performance in this domain, is employed to provide insights into the scene's depth information. 


















The monocular depth estimation model significantly enhances depth perception across different scales. Its guidance is pivotal in mitigating distortion issues and improving overall scene depth perception.
Within the optimization network, we merge DUSt3R outputs with depth guidance by incorporating a point cloud projection loss, a multi-scale depth loss and a depth smoothness loss.


    ℒ_opt = ℒ_pc + α_dℒ_D + α_sℒ_smooth

where ℒ_opt refers to the total optimization loss. α_d and α_s are loss weights of multi-scale depth loss ℒ_D and depth smoothness loss ℒ_smooth. The smoothness loss encourages depth map smoothness by penalizing depth gradient changes, weighted by the image gradients and details of the multi-scale depth loss term ℒ_D would be discussed later (see Sec <ref>).




Point cloud Cleaning

In order to eliminate floaters and artifacts, we implement two strategies for cleaning the point cloud: geometry-based cleaning and confidence-based cleaning.

In geometry-based cleaning, we adopt an iterative approach to remove unreliable points during the depth optimization process. For a set of K input images, as illustrated in Figure <ref>, the method involves systematically pairing image I_k with all other K-1 images within a single iteration. For the image pair I_k, I_l, a pixel q in I_l corresponds to a 3D point Q in the scene, represented as (X_q, Y_q, Z_q) in the l_th coordinate system. This point can be translated into the k_th coordinate system using the transformation matrix T_k,l. The projected point intersects with I_k at pixel r, and its depth in the k_th coordinate system is denoted as Z_q'. Conversely, pixel r also corresponds to another 3D point R, denoted as (X_r, Y_r, Z_r) in the k_th coordinate system.












To tackle the floating issue, if we detect that the difference between the projected depth Z_q' and the depth Z_r of point R exceeds a threshold τ_1, and the confidence η_q of point Q exceeds the confidence η_r of point R by more than τ_2, we label point R as unreliable. As a result, we remove this point from the set of 3D points.

    |Z_r - Z_q'| > τ_1  and  η_q - η_r > τ_2 ⇒Exclude point  R

The cleaning operation of the point clouds is executed once every ξ iterations, with τ_1 and τ_2 serving as hyperparameters.




In addition to the geometry-based cleaning process, we also implement a confidence-based cleaning step post-optimization. Each point within the point clouds is assigned a confidence value. The original DUSt3R method applies a basic confidence threshold to filter out points with confidence below a certain level. However, due to the distance bias, this approach may inadvertently exclude many background elements. To tackle this challenge, as depicted in Figure <ref>, we differentiate between foreground and background regions by arranging the depths of all points and selecting the median depth as the separation boundary. Points in the foreground, typically observed in multiple-view images, tend to have higher confidence levels. Consequently, we establish a high-confidence threshold for foreground objects. On the contrary, the background area, often captured from a distance and present in only a few images, tends to exhibit lower confidence levels. Hence, we adopt a more lenient strategy for this region, employing a lower confidence threshold for point cleaning.




 §.§ Multi-modal Regularized Gaussian Reconstruction


Dense point clouds and camera poses are acquired through Background-Aware Depth-guided Initialization. These variables serve as the initialization of Gaussian kernels. Vanilla 3DGS methods utilize photo-metric loss functions such as ℒ_1 and ℒ_SSIM to optimize 3DGS kernels and enable them to capture the underlying scene geometry.
However, challenges arise in scenarios with extremely sparse input images. Due to the inherent biases of the Gaussian representation, the Gaussian kernels are prone to overfitting on the training views and cause degradation on unseen perspectives. To mitigate this issue, we enhance the Gaussian optimization process by integrating photo-metric loss, multi-scale depth loss, cosine-constrained normal loss, and norm-weighted virtual-view loss.

Photo-metric Loss In line with vanilla 3DGS, we initially compute the photo-metric loss between the input RGB images and Gaussian-rendered images. The photo-metric loss function combines ℒ_1 with an SSIM term ℒ_SSIM. 

    ℒ_pho = (1-λ) ℒ_1+λℒ_SSIM

where λ represents a hyperparameter, and ℒ_pho denotes the photo-metric loss.

Multi-scale Depth Regularization 

To address above challenges, depth information can be incorporated into the Gaussian scene to provide regularization during training. Similar to the Initialization Module, we initially employ the monocular estimation model Marigold to predict depth images {D̂_k}_k=0^K-1 from sparse input images. Since monocular depth estimation models typically offer relative depth predictions without realistic scene scale information, we utilize the Pearson Correlation Coefficient (PCC) <cit.> as a metric to gauge the similarity between depth maps.
The Pearson Correlation Coefficient is a fundamental statistical correlation coefficient that quantifies the linear correlation between two data sets. Essentially, it assesses the resemblance between two distinct distributions X and Y.

    PCC(X,Y) = E[XY] - E[X]E[Y]/√(E[Y^2] - E[Y]^2)√(E[X^2] - E[X]^2)

where E represents the mathematical expectation.

Inspired by previous works <cit.> <cit.>, to enhance the capture of local structures, we go beyond assessing the correlation between depth maps at the source scales. We divide depth images into small patches and compare the correlation among these depth patches. The Pearson correlation coefficient has a strong connection with normalized cross-correlation, implying that the utilization of this loss function promotes high cross-correlation values for patches at corresponding locations in both depth maps, regardless of depth value variations.
During each iteration, we randomly select F non-overlapping patches to evaluate the depth correlation loss, defined as:

    ℒ_depth = 1/F∑^F-1_f=0 1 -  PCC(Γ̅_f, Γ̂_f)
 where Γ̅_f denotes the f_th patch of Gaussian-rendered depth maps and Γ̂_f denotes the f_th patch of depth maps predicted by monocular estimation model. Intuitively, this loss works to align depth maps of the Gaussian representation with the depth map of monocular prediction, mitigating issues related to inconsistent scale and shift.

Cosine-constrained Normal Regularization While depth provides distance information within the scene, normals are also essential for shaping surfaces and ensuring smoothness. Therefore, we introduce a normal-prior regularization to constrain the training process.



We utilize cosine similarity to quantify the variance between the predicted normal maps got from normal prior <cit.> and the normal maps rendered using Gaussian representations.

    ℒ_normal = 1/K∑^K-1_k=0 1 - COS(N̅_k, N̂_k)

where N̅_k ∈ℝ^H× W× 3 represents the Gaussian-rendered normal maps, and N̂_k signifies the normal maps predicted by the monocular estimation model. The function COS() denotes the cosine similarity function.




Weighted Virtual-view Regularization 
In cases where the training views are sparse, the Gaussian scene may deteriorate when presented with new views due to the lack of supervision from these training views. Hence, we introduce a virtual-view regularization strategy to preserve the original point cloud information throughout the optimization process.

As illustrated in Figure <ref>, we randomly sample K_v virtual views in 3D space. For each virtual camera, we project the point clouds onto the 2D plane of the view. A weighted blending algorithm is employed to render the 3D points into RGB point-rendered images. These point-rendered images serve as guidance for the Gaussian optimization process.



When creating a point-rendered RGB image from a virtual view, the color of each pixel i is determined by the U nearest projected 3D points. As shown in Figure <ref>, these points are selected based on their proximity to pixel i within a radius π, where π is defined as one-third of the pixel width. Subsequently, these selected points are arranged in order of their distances {d_u}_u=0^U-1 from the viewpoint. Weights are then allocated to these ordered points according to their distances, with closer 3D points to the virtual viewpoint receiving higher weights.

    c(i) =
    ∑_u=0^U-1 c_u w_u,  
    w_u = e^-d_u/∑_u=0^U-1 e^-d_u   if valid points
    
    c_bg   otherwise

Here, c(i) represents the color of pixel i after point rasterization. c_u denotes the color of the u_th 3D point, and w_u is its corresponding weight. In cases where a pixel has no valid point projection (i.e., U=0), we assign the pixel the predefined background color c_bg, which, in this instance, is white.

By employing the norm-weighted blending algorithm, we obtain K_v point-rendered RGB images denoted as {I_k^pr}_k=0^K_v-1. These images are subsequently utilized to regulate Gaussian kernels, thereby imposing constraints on optimization and preventing overfitting. The virtual-view loss function at this stage is presented below.

    ℒ_vir =(1-λ) ℒ_1(I̅_k,I_k^pr) + 
    λℒ_SSIM(I̅_k,I_k^pr),k ∈ K_v

where λ, ℒ_1, ℒ_SSIM are the same as original 3d Gaussian splatting and I̅_k is the Gaussian-rendered RGB image from k_th view.

Multi-modal Joint Optimization Throughout the Multi-modal Regularized Gaussian Reconstruction phase, in addition to the photo-metric loss, Multi-scale Depth Regularization, Cosine-constrained Normal Regularization, and Norm-weighted Virtual-view Regularization are incorporated to steer the training process. These methodologies are pivotal in alleviating overfitting and upholding the output quality.

    ℒ_multi = ℒ_pho+β_virℒ_vir + β_depℒ_depth + β_norℒ_normal

where ℒ_multi represents the loss function utilized in the Multi-modal Regularized Gaussian Reconstruction. The weights β_vir, β_dep, β_nor serve as hyperparameters to regulate their impact, with further elaboration provided in the Experiment Section.






 §.§ Iterative Gaussian Refinement



During this phase, we implement an iterative optimization approach to progressively enhance scene details. Initially, we uniformly enhance the Gaussian-rendered images from virtual viewpoints using a Gaussian repair model. This model refines blurry Gaussian-rendered images into sharp, realistic representations. Following this enhancement, these refined images act as supplementary guidance, facilitating the optimization of Gaussian kernels in conjunction with depth and normal regularization terms. After ζ optimization steps, we re-render the Gaussian images and subject them to the repair model once more, replacing the previously refined images for another iteration of supervision.





  §.§.§ Iterative Gaussian Optimization



Initially, we generate K_v images using Gaussian kernels from virtual viewpoints. Subsequently, the Gaussian Repair Model is utilized to uniformly enhance these images, resulting in a set of repaired images denoted as {I_k^repair}_k=0^K_v-1.
To maintain scene coherence and reduce potential conflicts, we set the denoise strength to a low value and gradually reintroduce limited details to the Gaussian-rendered images during each repair process. These repaired virtual-view images, along with monocular depth and normal maps from training views as outlined in Section <ref>, are then employed to regulate the Gaussian optimization. The overall optimization loss in the Gaussian refinement stage, denoted as ℒ_refine, is determined by:

    ℒ_refine = ℒ_pho+β_repℒ_rep + β_depℒ_depth + β_norℒ_normal

Here, ℒ_depth and ℒ_normal correspond to the Multi-modal Regularized Gaussian Reconstruction. β_rep signifies the weight of the repair loss. The repair loss ℒ_rep is defined as follows:

    ℒ_rep =(1-λ) ℒ_1(I̅_k,I_k^repair) + 
    λℒ_SSIM(I̅_k, I_k^repair),k ∈ K_v

In this formulation, we leverage the photo-metric loss between the repaired images {I_k^repair}_k=0^K_v-1and the Gaussian-rendered images {I̅_k}_k=0^K_v-1 in one loop, utilizing the repaired images as a guiding reference. The parameters λ, ℒ_1, and ℒ_SSIM remain consistent with the original 3D Gaussian splatting methodology. The above operations will be repeated.


Through this iterative optimization strategy, the newly generated images gradually enhance in sharpness without being affected by blurring caused by view disparities. The optimization process persists until it is ascertained that the diffusion process no longer produces satisfactory outcomes, as evidenced by deviations from the initial scene or inconsistencies across various viewpoints. 




  §.§.§ Gaussian Repair Model
 In this section, we present the Gaussian repair model utilized earlier. Its primary objective is to enhance blurry Gaussian-rendered images into sharp, realistic images while preserving the style and content of the original image.
 
Model Architecture The architecture of the Gaussian Repair Model is illustrated in Figure <ref>. It takes Gaussian-rendered images and real-world input images as inputs. In Figure <ref>(b), the Gaussian-rendered image I̅ undergoes image encoding to extract latent image features. The real-world input images are processed through a GPT API for a description prompt σ, then encoded to obtain text latent features. These image and text latent features act as conditions for a ControlNet <cit.> to predict noise ϵ_θ and progressively remove noise from the Gaussian-rendered image. The model is a controlnet finetuned by injecting lora into its layer and it can produce the repaired Gaussian-rendered image. Figure <ref>(c) provides insight into the Lora-ControlNet, where Lora <cit.> weights are integrated into each transformer layer of the ControlNet's UNet. We maintain the original parameters of the transformer blocks constant and solely train the low-rank compositions A, B, where A ∈ℝ^d × r, B ∈ℝ^r × k, with a rank r ≪min(d, k). Concerning the text encoder, as depicted in Figure <ref>(d), Lora weights are integrated into each self-attention layer of the encoder. The input of the Lora-Text Encoder is the scene prompt, and the output is the text embedding.



Training process In this section, we will explore the training process of the Gaussian repair model. Initially, for data preparation, we collect image pairs from Section <ref>, where input images within each scene act as reference images. For each training perspective, we randomly select ω Gaussian-rendered images and pair them with input images to create training pairs.
Subsequently, these image pairs are utilized to train our Gaussian Repair Model. As shown in Figure <ref>(a), the input image I undergoes a forward diffusion process. Specifically, the image is input into an image encoder to extract the latent representation z_1. This latent representation then undergoes a diffusion process where noise ϵ is gradually introduced over T steps. After obtaining latent z_T, a reverse diffusion process is initiated, as illustrated in Figure <ref>(b), where a Lora-UNet and a Lora-ControlNet are employed to predict noise ϵ_θ at each step. This predicted noise, combined with the noise introduced during the diffusion process, contributes to calculating the loss function, aiding in the training process. The loss function for any input image can be defined as follows.

    ℒ_Control = E_I,t,I̅,ϵ∈𝒩(0, 1)[‖(ϵ_θ(I, t, I̅, σ)-ϵ)‖_2^2]

Here, σ refers to the text prompt of the reconstructed scene. I represents the actual input image, and I̅ is the Gaussian-rendered image obtained from Section <ref>. E_I,t,I̅,ϵ∈𝒩(0, 1) denotes the expectation over the input image I, the time step t, the text prompt σ, the condition image I̅, and the noise ϵ drawn from a normal distribution with mean 0 and standard deviation 1. ϵ_θ indicates the predicted noise.

 






 §.§ Scene Enhancement



Given the sparse input images and the restricted training perspectives, it is expected that rendered images from adjacent new viewpoints may display discrepancies. In order to ensure high-quality and consistent rendering along a specified camera path, we propose a View Enhancement module, which utilizes video diffusion priors to improve the coherence of rendered images.


This module concentrates on enhancing the visual consistency of rendered images without delving into Gaussian kernel refinement. Initially, multiple images are rendered along a predetermined camera trajectory and grouped for processing. Subsequently, a video diffusion UNet is employed to denoise these images to generate enhanced images.
In the video diffusion model, DDIM inversion <cit.> is utilized to map Gaussian-rendered images back to the latent space, the formulation can be expressed as:

    z_t+1=α_t+1/α_tz_t+(1/α_t+1-1-1/α_t-1)ϵ_θ(z_t,t,σ),

where t∈[1,T] is the time step and α_t denotes a decreasing sequence that guides the diffusion process. σ serves as an intermediate representation that encapsulates the textual condition.

The rationale behind mapping Gaussian-rendered images to a latent space is to leverage the continuous nature of the latent space, preserving relationships between different views. By denoising images collectively in the latent space, the aim is to enhance visual quality without sacrificing spatial consistency.



In the scene enhancement model, we utilized Zeroscope-XL as the video-diffusion prior and set the denoising strength to 0.1.



§ EXPERIMENTS









 §.§ Experimental Setup

Dataset Our experiments were conducted using three datasets: the Tanks and Temples Dataset <cit.>, the MipNeRF360 Dataset <cit.>, and the LLFF Dataset <cit.>. The Tanks and Temples and MipNeRF360 datasets feature 360-degree real-world scenes, while the LLFF dataset comprises feed-forward scenes. From the Tanks and Temples Dataset, we uniformly selected 200 images covering scenes like Family, Horse, Ignatius, and Trunk to represent the entire 360-degree environments. In the MipNeRF360 dataset, we chose the initial 48 frames capturing various elevations across a full 360-degree rotation, including scenes such as Garden, Bicycle, Bonsai, and Stump. Additionally, scenes like flowers, orchids, and ferns from the LLFF Dataset have been incorporated as well.

Train/Test Datasets Split For 360-degree scenes, we varied the view number K from 4 to 16 to assess all the algorithms under consideration. In a training set comprising K images, the remaining images were allocated to the test set. Concerning the feed-forward dataset, we adhered to the setup outlined in previous research <cit.>, employing 3 views for training and the remainder for testing.

Metrics  In the assessment of novel view synthesis, we present Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM)<cit.>, and Learned Perceptual Image Patch Similarity (LPIPS)<cit.> scores as quantitative measures to evaluate the reconstruction performance.


Baselines We compare our method against 7 baseline approaches. Our evaluation included sparse-view reconstruction methods such as DNGaussian <cit.>, FreeNeRF <cit.>, SparseNeRF <cit.>, PixelNeRF <cit.>, MVSNeRF <cit.>, DietNeRF <cit.>, and RegNerf <cit.>. Additionally, we compared our method with the vanilla 3DGS approach. Notably, all these baseline methods employed Colmap for the pre-computation of camera parameters.







 §.§ Implementation Details


We implemented our entire framework in PyTorch 2.0.1 and conducted all experiments on an A6000 GPU. In the Background-Aware Depth-guided Initialization stage, 

loss weights α_g, α_s and α_l were set to 0.01, 0.01 and 0.1. Geometry-based cleaning was applied every 50 iterations. For Confidence-based cleaning, τ_1 and τ_2 were set to 0.1 and 0, respectively. Moving on to the Multi-modal Regularized Gaussian Reconstruction stage, we trained the Gaussian model for 6,000 iterations with specified loss function weights: β_vir = 0.5, β_dep = 0.3, and β_nor = 0.1 across all experiments. We use ControlNet <cit.> as our foundational model. 

The low-rank r was set to 64, and the model was trained for 2000 steps. In the Iterative Gaussian Refinement Module, the denoising strength was set at 0.3 for each repair iteration, and the repair process was repeated every 4,000 iterations. The value of β_rep was adjusted based on the distance between the virtual view and its nearest training view, within the range of (0, 1). This iterative cycle was set to 3 in our experiments. 



 §.§ Quantative and Qualitative Results
 









Tanks and Temples & MipNerf360 The quantitative results, presented in Table <ref> and Table <ref>, demonstrate that our method outperforms others in terms of PSNR, SSIM, and LPIPS metrics, showcasing superior performance and finer details. Visual results are also showcased in Figure <ref> and Figure <ref>. With 16 input images, LM-Gaussian produces high-quality reconstruction results, preserving most structures and details, whereas DNGaussian and SparseNerf yield cluttered outcomes. Despite DNGaussian and SparseNerf utilizing frequency and depth regularization to prevent overfitting, their performance falls short for two primary reasons. Firstly, their dependence on Colmap for initializing point clouds and camera poses proves challenging in sparse-view scenarios where traditional SfM methods struggle to provide reliable outputs. Secondly, these methods neglect to introduce significant model priors to restore intricate details. Given the sparse nature of the input data, critical details may be lost without additional information.





LLFF Besides challenging 360-degree large-scale scenes, we also conducted experiments on feed-forward scenes like the LLFF Dataset to ensure the thoroughness of our study and validate the robustness of our method. Following previous sparse-view reconstruction works, we take 3 images as input, with quantitative results presented in Table <ref> and qualitative results shown in Figure <ref>. It is observed that sparse-view methods like DNGaussian also demonstrate commendable visual outcomes and relatively high PSNR and SSIM values. This can be attributed to the nature of the LLFF Dataset, which does not encompass a 360-degree scene but rather involves movement within a confined area. This results in higher image overlap and fewer unobserved areas, making it easier to reconstruct the scene. However, despite the impressive performance of these methods, our approach still exhibits certain advantages. In addition to the numerical enhancements demonstrated in TABLE <ref>, taking the flower scene as an example, our method excels in visual results by restoring finer details such as flower textures. Moreover, our method maintains superior performance in regions with less overlap, as exemplified by the leaves in the surroundings.






 §.§ Ablation Study
 



Colmap Initialization Initially, we investigated the conventional Colmap method within our sparse-view settings. It fails to reconstruct point clouds with 8 input images in 360-degree scenes. We progressively increased the number of input images until Colmap could eventually generate sparse point clouds. With 16 input images, as illustrated in Figure <ref>, Colmap's resulting point clouds were significantly sparse, comprising only 1342 points throughout the scene. In contrast, our Background-Aware Depth-guided Initialization method excels in generating high-quality dense point clouds.





Effect of Background-Aware Depth-guided Initialization


Through visual demonstrations, we highlight DUSt3R's limitations in reconstructing high-quality background scenes, plagued by artifacts and distortions. As depicted in Figure <ref>(a)(b), DUSt3R either lacks background details or presents poor background quality, resulting in subpar reconstructions.


In contrast, as illustrated in Figure <ref> (c), our module adeptly reconstructs background scenes with minimal distortion while preserving high-quality foreground scenes, exhibiting fewer artifacts and floaters compared to Figure (b)'s point clouds. Additionally, we present visualizations of depth maps before and after depth optimization. In Figure <ref>, the original DUSt3R's depth maps reveal background blurriness, blending elements like the sky and street lamps. With guidance from Marigold, our Background-Aware Depth-guided Initialization notably enhances the reconstruction of background scenes compared to the original DUSt3R output.

Moreover, Table <ref> provides quantitative comparisons among our method, Colmap, and DUSt3R. While Colmap yields less favorable results, DUSt3R shows significant improvement. Despite DUSt3R's performance, our initialization module achieves state-of-the-art outcomes, leading to improvements in PSNR and SSIM metrics.


Effect of Multi-modal regularized Gaussian Reconstruction
We conducted ablation studies on the multi-modal regularization, incorporating depth, normal, and virtual-view regularization. The quantitative outcomes are detailed in Table <ref>. Following the integration of these regularization techniques, the novel view synthesis demonstrates improved results, indicated by higher PSNR and SSIM values, as well as finer details with lower LPIPS scores. With the implementation of multi-modal regularization, as depicted in Figure <ref>, the Gaussian-rendered images showcase smoother surfaces and reduced artifacts within the scene. Conversely, images lacking regularization exhibit black holes and sharp angles, diminishing the overall quality.







Effect of Iterative Gaussian Refinement We further explore the usefulness of the iterative Gaussian refinement module. As illustrated in Figure <ref>, we present a comparison between the images before and after Gaussian Repair. The noticeable outcomes highlight that the repaired images exhibit enhanced details and a reduction in artifacts, emphasizing the effectiveness of the Gaussian Repair Model. Quantitative results before and after the Iterative Gaussian Refinement are also detailed in Table <ref>. While a marginal improvement in PSNR is noted, more intricate metrics such as LPIPS and SSIM demonstrate substantial enhancements. These findings align seamlessly with our primary objective of restoring intricate details within the images.







The number of input images In Figure <ref>, we assess our method using different number of sparse input images N. We compare LM-Gaussian with the original 3DGS across view splits of growing sizes K ∈{4, 8, 16, 24, 32}. Notably, in sparse view scenarios, our method consistently enhances the performance of 3DGS. Furthermore, as the number of views increases, we observe that our proposed techniques, leveraging extensive model priors for regularization and repair, diminish in significance. While LM-Gaussian's rate of quality enhancement slows with increasing views, 3DGS showcases a steady linear progression.


§ CONCLUSIONS

We introduce LM-Gaussian, a sparse-view 3D reconstruction method that harnesses priors from large vision models. Our method includes a robust initialization module that utilizes stereo priors to aid in recovering camera poses and reliable Gaussian spheres. Multi-modal regularizations leverage monocular estimation priors to prevent network overfitting. Additionally, we employ iterative diffusion refinement to incorporate extra image diffusion priors into Gaussian optimization, enhancing scene details. Furthermore, we utilize video diffusion priors to further improve the rendered images for realistic visual effects. Our approach significantly reduces the data acquisition requirements typically associated with traditional 3DGS methods and can achieve high-quality results even in 360-degree scenes. LM-Gaussian currently is built on standard 3DGS that only works well on static scenes, and we would like incorporate dynamic 3DGS techniques to enable dynamic modeling in the future.




plain

    












