GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Driving
Huasong Han Kaixuan Zhou Xiaoxiao Long Yusen Wang Chunxia Xiao


Diffusion-limited settling of highly porous particles in density-stratified fluids
    Daniel M. Harris
    September 3, 2024
==================================================================================



§ ABSTRACT 

























We propose GGS, a Generalizable Gaussian Splatting method for Autonomous Driving which can achieve realistic rendering under large viewpoint changes. Previous generalizable 3D gaussian splatting methods are limited to rendering novel views that are very close to the original pair of images, which cannot handle large differences in viewpoint. Especially in autonomous driving scenarios, images are typically collected from a single lane. The limited training perspective makes rendering images of a different lane very challenging. To further improve the rendering capability of GGS under large viewpoint changes, 
we introduces a novel virtual lane generation module into GSS method
to enables high-quality lane switching even without a multi-lane dataset. Besides, we design a diffusion loss to supervise the generation of virtual lane image to further address the problem of lack of data in the virtual lanes. 
Finally, we also propose a depth refinement module to optimize depth estimation in the GSS model. Extensive validation of our method, compared to existing approaches, demonstrates state-of-the-art performance.










§ INTRODUCTION


















Novel view synthesis is an essential task in the field of computer vision, with significant potential applications in autonomous driving <cit.>, object detection, and digital human representations. To enhance the robustness of autonomous driving systems, it is imperative to establish a simulation environment for testing these systems effectively. However, the majority of existing datasets are limited to single-lane scenarios. This limitation presents significant challenges in inferring adjacent lane scenarios from the current viewpoint. If lane switching is not supported, the test samples provided to the autonomous driving simulation system will be incomplete, making it impossible to conduct better simulation testing and requiring a significant amount of data collection costs.
















The methods based on NeRF <cit.> often rely on lidar to better generate novel views in autonomous driving scenarios. READ <cit.> introduces a new rendering method that adopts a neural rendering approach different from NeRF. It learns neural descriptors of the original point cloud with explicit geometry to render images, instead of learning implicit geometry in NeRF methods. However, the efficiency of training and rendering using these methods is very low.


















The efficiency in training and rendering speed, coupled with the high reconstruction quality of 3D Gaussian Splatting <cit.>, contributes to its widespread application of novel view synthesis in autonomous driving. GaussianPro <cit.> introduces multi-view stereo to improve the geometry of generated gaussian splats. DC-Gaussian<cit.> introduces an adaptive image decomposition module to address the impact of glass reflections on the quality of novel view synthesis. However, these methods still cannot have effective novel view synthesis in lane switching, as they do not address the main problem that only single lanes of data are collected.






To address the problem of the sparse view synthesis, many methods have sought to optimize this process using generative models <cit.>. generative models are trained across large amount of scenes to enhance performance in sparse view scenarios. However, the generative model still lacks of data from multi-lanes to learn how to synthesize novel views for other lanes from single lanes data. 

Therefore, we propose a virtual lane module into generative Gaussian splatting to address the synthesis of new views involving lane changes, despite the lack of multi-lane training datasets for supervision. In the module, we first use 3D gaussians generated from images in the single lanes using a generative model to predict images from virtual lanes, then use 3D gaussians generated from the virtual images to predict back the image collected in the single lanes. In this way, we can let generative model to learn how to generate the best images in the other lanes even with only single lanes of data. In addition, we propose a diffusion loss from a latent diffusion model <cit.> to virtual generated images to further improve the lane switching of our GGS. Finally, as improving geometry of generated 3D gaussians also improves novel view synthesis in sparse view collections, we employs points from traditional multiview stereo reconstruction to refine the depth estimated in GGS. 


































The main contributions of this paper can be summarized as follows:





  * We propose a novel virtual lanes module into the generative gaussian splatting to improve the quality of lane switching novel view with only single lanes of data.



  * We introduce a diffusion loss to directly supervise the image from virtual lanes predicted by GGS to further improve the novel view synthesis from limited collected views. 


  * We  propose to fusing MVS geometry into the generative 3D gaussian splating to improve geometry estimation.


  * We conduct extensive experiments on a wide range of scenarios to validate the effectiveness of our algorithm, and achieve state-of-the-art street novel view synthesis even without LiDAR.

















§ RELATED WORK




















 §.§ 3D Gaussian Splatting


3D Gaussian Splatting <cit.> employs a point-cloud-based 3D reconstruction method, which combines the position information of each point with Gaussian distribution to convert point cloud data into a 3D surface. However, the quality of street novel synthesis is still problematic due to limited view collections in the street.





GaussianPro <cit.> has improved 3D Gaussian Splatting by introducing a novel progressive propagation strategy to guide Gaussian densification based on the scene's surface structure. Although improving geometries helps to mitigate the novel synthesis in sparse views, the quality of novel synthesis in another lanes is still low. Deformable 3D Gaussians <cit.> employs a framework for extending 3D Gaussian Splatting in dynamic scenes using a deformation field, enabling the learning of 3D Gaussians in a normalized space. There are also other methods based on 3D Gaussian Splatting such as <cit.>. Although street view synthesis has only been improved on the collected lanes, they have not solved the problem of sparse views, leaving lower levels of novel view synthesis when changing lanes.










 §.§ Generalized model

To solve the problem of novel view synthesis in sparse views, some methods propose a generalized model-based approach. PixelNeRF <cit.> employs a generalized model for novel view synthesis based on volume rendering method, which can be trained directly from images without explicit 3D supervision. However, the generation quality is not high and the training efficiency is low. 
mvsplat <cit.> introduces an efficient feedforward 3D Gaussian splash model learned from sparse multi view images, and constructs a cost volume to represent the cross view feature similarity of different candidate depths, providing valuable clues for depth estimation. MVSGaussian <cit.> employs a mixture Gaussian rendering method that integrates efficient volume rendering design for novel view synthesis. Compared with the original 3D Gaussian Splatting, MVSGaussian achieves better view synthesis results while reducing training computational costs. However, it cannot be well completed for scenes with obstacles.






 §.§ Diffusion Model

For occluded scenes, using generalized models cannot generate better results, so some algorithms introduce diffusion model <cit.> to imagine unknown regions. ReconFusion <cit.> further utilizes the generative capacity of large models to infer unknown areas, and integrates diffusion prior into NeRF's 3D reconstruction process. DrivingDiffusion <cit.> introduces a spatiotemporally consistent diffusion framework, incorporating multi-view attention to generate realistic multi-view videos controlled by 3D layouts. These diffusion model methods only consider a single lane and do not utilize multi-lane features for better completion.



























































§ METHODOLOGY













Although generalized models can assist in synthesizing novel views in sparse views, insufficient view information leads to inaccurate depth estimation. Our method further optimizes the generalization model. The overall framework diagram of our GGS method is as shown in Figure <ref>. We input four different frame images and introduce neighborhood feature in the Mutli-View Depth Refinement Module to better address scenes with occlusions. And we introduce more global information to optimize the predicted depth map By using MVS. In the Virtual Lane Generation Module, we introduce the concept of virtual lanes and solve the problem of not having a multi-lane dataset by switching back after switching, allowing the model to flexibly switch lanes. In addition, we introduce the Multi-Lane Diffusion Loss to supervise the novel view synthesis.






 §.§ Background

















MVSplat <cit.> is a generalizable 3D Gaussian Splatting method, which can synthesize novel views from sparse inputs. MVSplat takes transformer based structure and adopts cross view attention strategy to build a cost volume for each input view, then following a U-Net to predict the depth and the parameters of Gaussian primitives for each pixel.







The 3D Gaussian parameters consist of the Gaussian center position x, scale s, rotation angle q, opacity α, and color c. Given the predicted depth map D and the projection matrix P with camera parameters K, pixels located at x are back-projected from the image plane to 3D space as follows:

    x_p_x = Π_P^-1(p_x, D),

where Π represents the back-projection operation, and p_x and D represents pixel coordinates and estimated depth, respectively. The opacity α is represented by the matching confidence directly.


The remaining Gaussian parameters scale s, rotation angle q, and color c are decoded from the encoded features as follows:

    s_p_x = Softplus(h_s(Γ(p_x))),


    q_p_x = Norm(h_q(Γ(p_x))),


    c_p_x = Sigmoid(h_c(Γ(p_x))),

where Γ represents the high-dimensional feature vector, p_x represents pixel coordinates, and h_s, h_q, and h_c represent the scaling head, rotation head, and color head, respectively.

































 §.§ Mutli-View Depth Refinement Module


We enhanced MVSplat by our Mutli-View Depth Refinement Module, i.e. Modified MVSplat. It can produce more accurate 3D gaussian primitives and improve the quality of novel view synthesis. To better infer unknown regions, we incorporated the color feature information of the neighborhood of this view. Our model takes this into consideration. We use the back-projected point cloud map reconstructed through Agisoft Metashape as an additional input color feature for U-Net. The feature representation of the neighborhood is:

    F_neighbor_i = {F_m | m ∈ [i-k, i+k]},

where i represents the i-th frame in the video, and F_i represents the color feature of the i-th frame. k represents neighborhood distance.

Neighborhood color features are merged into depth features through concatenation, high-dimensional Gaussian parameter features are output through UNet, decoded using a Gaussian parameter decoder, and finally generate Gaussian parameter representations.






    dep_ref = 𝒰(F_neighbor_i, dep_i),

where 𝒰 represents the U-Net. By introducing color information from multiple neighborhood perspectives in this way, the synthesis ability of the generalized model under obstacle occlusion is enhanced.


In addition, to refine the depth, we introduce a confidence based method. The lower the transparency of the predicted 3D Gaussian, the lower the confidence level of the predicted depth. When the confidence level is high, the predicted depth remains unchanged. When the confidence level is low, we correct the predicted depth map by reconstructing the back-projected depth map through Agisoft Metashape <cit.>. The optimized depth value is:

    dep_i = {βd̂êp̂_̂î+(1-β)D_i,  if α_i<α
    d̂êp̂_̂î,  if α_i≥α.,

where D_i represents the depth of projected depth map. d̂êp̂_̂î represents the predict depth. α and β represent the transparency threshold and depth threshold, respectively.




























































































































 §.§ Virtual Lane Generation Module




Previous generalizable 3D gaussian splatting methods are limited to rendering novel views that are very close to the original pair of images, which cannot handle large differences in viewpoint. Especially in autonomous driving scenarios, images are typically collected from a single lane. The limited training perspective makes rendering images of a different lane very challenging. We have obtained a 3D Gaussian using our modified depth refinement module. To further improve the rendering capability of GGS under large viewpoint changes, We introduce the virtual lane approach that enables high-quality lane switching even without a multi-lane dataset,  inspired by <cit.>. 



The virtual lane converter is used to select the appropriate virtual lane, so that after lane switching, no information can be seen from the virtual perspective due to excessive switching amplitude. Then generate a pose for the virtual lane by performing a vertical translation along the lane. Finally, generate a virtual perspective based on the pose of the virtual lane. After introducing the virtual lane module, our GGS module process mainly includes two stages.

In the first stage, we input a set of N images:

    ISet_1 = {I_1, I_2, ...I_N},

then we output the target image through the model:

    Î^̂1̂= 𝒢(ISet_1),

where 𝒢 represents GGS module, and ISet_1 is a rendered image without shifting the view, and the rendered view is consistent with the ground truth. The current lane generates a collection of virtual lane rendering images through lane converters. The rendered image of the virtual lane is represented as:

    ISet_2 = {𝒱(Î^̂1̂_̂k̂,γ sinθ)|k_f≤ k≤ k_l, θ=ω k},

where 𝒱 represents the virtual lane converter, γ represents the translation coefficient, k_f and k_l represent the index of the first and last frames of the input, respectively. ω represents the switching period angle, and the switching angle of each frame changes periodically in order.









In the second stage, use the virtual lane generated in the first stage as input. Using our model, switch back from the virtual lane to the real lane and output a rendered image of the real lane:

    Î^̂2̂= 𝒢(ISet_2),

where 𝒢 represents GGS module. This forms a closed-loop process of switching to a new lane and then switching back. The advantage of doing so is that even without the ground truth of the left and right lanes, we can still enhance the quality of the model's rendering of the left and right lanes by establishing virtual lanes, allowing the model to improve the quality of lane switching, as shown in Figure <ref>.












 §.§ Multi-Lane Diffusion Loss













































There is no ground truth available for training when switching lanes. When the lane switching amplitude is large, obstacles can obstruct the view during lane changes, making it impossible to collect information about the new lane from the current lane, as shown in Figure <ref>. Therefore, in order to better address this issue, we use diffusion prior knowledge to imagine color information from a novel lane view. 

The traditional diffusion model denoising method directly completes the generated image, but due to the diversity of the generated models, it can lead to inconsistent results between frames. Our method calculates the loss of the denoised image and the image before denoising, and generates a new perspective supervised by diffusion. Additionally we construct multi-lane novel view images, instead of utilizing image of the current lane as input for U-Net denoising. This approach helps ensure that the autonomous driving lane remains visible in the image following a change in viewpoint.

Specifically, we adapt the Stable Diffusion framework <cit.>, and use the Variational AutoEncoder <cit.> to encode the multi-lane images into latent code, including the left lane, middle lane and right lane. Then, we perform several denoising steps on the latent code as an initialization parameter for Denoising U-Net, fixing the input text as the autonomous driving label. Generated through the CLIP <cit.>, denoised through several steps, and then decoded into images using the Variational AutoEncoder. These images serve as supervision to guide the synthesis of novel views.




































 §.§ Loss Function





































Our model is trained on a single lane dataset and introduces a method of constructing virtual lanes to generate unknown domains through diffusion models. Therefore, our method mainly includes reconstruction loss, depth loss, virtual lane switching loss, and diffusion loss. The overall loss function is represented as follows:

    ℒ=ℒ_recon +ℒ_depth +ℒ_switch +ℒ_diffusion .


























 

 























































































 

 
















































































Reconstruction loss. Our GGS model is a generative model for novel views on autonomous driving. During the training process, we construct a reconstruction loss function by comparing the rendered image with the ground truth using mean square error loss.

    ℒ_recon =1/n∑_i=1^n(y_i-ŷ_i)^2,

where y_i represents the color value in Ground Truth corresponding to a certain pixel, and ŷ_i) represents the color value in the rendered image corresponding to the same pixel.













Depth loss. In most autonomous driving scenarios, lanes are regular and even, so the depth of adjacent pixels should be smooth to avoid abrupt changes. Therefore, we construct the depth loss function as follows:

    ℒ_depth =1/n∑_i=1^n (dD_i/dx +dD_i/dy + λ(d^2D_i/dx^2 +d^2D_i/dy^2)),

where dD_i/dx, dD_i/dy, d^2D_i/dx^2 and d^2D_i/dy^2 represents the first and second derivatives of the depth in the x and y-axis directions of the image, respectively. And λ is the depth smoothing adjustment factor.

Lane switching loss. Due to the lack of lane switching data, we train the model by constructing virtual lanes and switching back, and construct a lane switching loss.

    ℒ_switch =1/n∑_i=1^n(y_i-Ψ(Φ (ŷ_i)))^2,

where Φ represents constructing virtual lanes and Ψ represents switching from the virtual lane to the current lane.

Multi-lane diffusion loss. When we switch lanes in autonomous driving, changes in view can cause artifacts, so we use denoising methods to eliminate noise. 



    ℒ_diffusion =𝔼_π, t[β(t) (y-ŷ_π_1+ℒ_lpips(y, ŷ_π))],

where π represents the camera pose of the selected views, y represents the multi-lane images, ŷ_π represents the output images from the denoising model, β(t) is a weight function related to the noise level, and ℒ_lpips represents perceptual loss, which aims to emulate human perception of image similarity to better capture visual differences between images.




§ EXPERIMENTS








We compare GGS with ADOP <cit.>, READ <cit.>, 3DGaussian <cit.>, GaussianPro <cit.>, UC-NeRF <cit.> and DC-Gaussian <cit.>. We use Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), perceptual loss (VGG loss), perceptual metrics, and Learned Perceptual Image Patch Similarity (LPIPS) as evaluation metrics. 
















 §.§ Evaluation on KITTI and BrnoUrban



























From Table <ref>, methods based on 3D Gaussian Splatting, such as GaussianPro, and DC-Gaussian, generate slightly better quality than other methods based on neural radiation fields. However, in some scenes, the rendering quality is inferior, and our model performs better.




As illustrated in Figure <ref>, GaussainPro and DC-Gaussian fail to capture details such as tree leaves and utility poles. The rendering quality of the READ is inadequate, and UC-NeRF does not render the white lines in the middle of the road. The comparison methods of different models for lane switching are shown in Figure <ref>. Compared to other models, our method demonstrates excellent overall rendering quality and lane switching quality. 





 §.§ Assessing Cross-dataset Generalization

Our method GGS has the advantage of generalization in extending to new scenarios outside the distribution. To evaluate the generalization of our model, we conduct two cross-dataset evaluations. Specifically, we train the model on KITTI dataset and test it on Brno Urban dataset <cit.>. Conversely, we train the model on Brno Urban and test it on KITTI, as shown in Figure <ref>.








 §.§ Ablation Study











Effect of Virtual Lane Generation module. To demonstrate the effectiveness of the virtual lane generation module, we use FID <cit.> to conduct lane-switching experiments on different models, as shown in Table <ref>. FID@LEFT and FID@RIGHT represents the distance between the rendered images of the left and right lanes and the GT. The qualitative experimental results are illustrated in Figure <ref>. Our model achieves high rendering quality while ensuring that quality remains unaffected during lane switching, with quantitative results shown in Table <ref> and qualitative results shown in Figure <ref>.




















Effect of Multi-Lane Diffusion Loss. Due to limited input view information, some unknown areas cannot be synthesized after lane switching. Therefore, a diffusion model is used to imagine the unknown areas and optimize the generation quality, as shown in Table <ref>.




Effect of Depth Refinement Module. The depth refinement module introduces neighborhood feature information to optimize depth estimation in the presence of occluded objects, as shown in Table <ref>. After removing the deep refinement module, each metrics affected slightly.



§ CONCLUSIONS




In this paper, we have proposed a generative framework based on MVS and 3D Gaussian Splatting fusion, which can repair unknown regions to optimize generation quality. By simulating the virtual lanes, our method effectively switches driving lanes in autonomous driving scenarios, suitable for simulation testing of autonomous driving systems. Our method has some limitations, the quality of lane switching generation needs to be improved when encountering dynamic scenes with complex road conditions, multiple people, and mixed vehicles.












