Object Gaussian for Monocular 6D Pose Estimation from Sparse Views
Luqing Luo Shichu Sun Jiangang Yang Linfang Zheng Jinwei Du Jian Liu


BMI Prediction from  Handwritten English Characters Using a Convolutional Neural Network





    N. T. Diba1, N. Akter2, S. A. H. Chowdhury3
Dept. of Electronics & Telecommunication Engineering 

Rajshahi University of Engineering & Technology

Rajshahi, Bangladesh

nishattasnimediba09@gmail.com1,
 nasrinmanha123@gmail.com2, arif.1968.ruet@gmail.com3
J. E. Giti4
Dept. of Electrical & Electronic Engineering 

Rajshahi University of Engineering & Technology

Rajshahi, Bangladesh

jishan.e.giti@gmail.com4

    September 9, 2024
===============================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Monocular object pose estimation, as a pivotal task  in computer vision and robotics, heavily depends on accurate 2D-3D correspondences, which often demand costly CAD models that may not be readily available. Object 3D reconstruction methods offer an alternative, among which recent advancements in 3D Gaussian Splatting (3DGS) afford a compelling potential. Yet its performance still suffers and tends to overfit with fewer input views. Embracing this challenge, we introduce SGPose, a novel framework for sparse view object pose estimation using Gaussian-based methods. Given as few as ten views, SGPose generates a geometric-aware representation by starting with a random cuboid initialization, eschewing reliance on Structure-from-Motion (SfM) pipeline-derived geometry as required by traditional 3DGS methods. SGPose removes the dependence on CAD models by regressing dense 2D-3D correspondences between images and the reconstructed model from sparse input and random initialization, while the geometric-consistent depth supervision and online synthetic view warping are key to the success. Experiments on typical benchmarks, especially on the Occlusion LM-O dataset, demonstrate that SGPose outperforms existing methods even under sparse view constraints, under-scoring its potential in real-world applications. 













§ INTRODUCTION


Monocular pose estimation in 3D space, while inherently ill-posed, is a necessary step for many tasks involving human-object interactions, such as robotic grasping and planning <cit.>, augmented reality <cit.>, and autonomous driving <cit.>. 
Influenced by deep learning approaches, its evolution has enabled impressive performance even in cluttered environments.  
The most studied task in this field assumes that the CAD model of the object is known a priori <cit.>, but the accessibility of such a predefined geometry information prevents its applicability in real-world settings. 
To reduce reliance on specific object CAD models, recent research has shifted toward category-level pose estimation <cit.>, aiming to generalize across objects within the same category. However, these methods typically ask for extra depth information, and can falter with instances of varying appearances.



Emerging real-world demands call for an object pose estimator generalizable, flexible, and computationally efficient. 
Ideally, a new object can be reconstructed from casually taken reference images, sans the need for fine-grained well-textured 3D structures.
Reconstruction-based methods have shown the feasibility of this proposal <cit.>, which basically reconstruct the 3D object from the multi-view RGB images to substitute missing CAD model.
however, reconstruction-based methods have long relied on a fixed budget of high-quality given images and the prerequisite use of Structure-from-Motion (SfM) techniques, resulting in a notably tedious and costly training process.
Our method deviates from these requirements by pioneering an efficient object reconstruction method that thrives on limited reference images and the convenience of random initialization.
Capitalizing on the high-quality scene representation and real-time rendering of 3DGS <cit.>, we unveil SGPose, an novel framework of Sparse View Object Gaussian for monocular 6D Pose estimation. 
The proposed SGPose develops geometric-aware depth to guide object-centric 3D representations from RGB only input. Requiring a mere ten views, SGPose achieves competitive performance for object pose estimation, heralding its readiness for real-world deployment.

In our work, we extend a variant of 3DGS <cit.> by formulating Gaussian primitives as elliptic disks instead of ellipsoids to derive depth rendering.
As illustrated in Fig. <ref>, the conceived geometric-consistent constraints guide depth acquisition.
The resulting depth enables reliable online synthetic view warping and eases the challenges of sparse views for traditional 3DGS methods.
Additionally, an online pruning is incorporated in terms of the geometric-consistent depth supervision, toning down common sparse view reconstruction issues like floaters and background collapses. 
In contrast to the conventional reliance on SfM pipelines for point cloud initialization in 3DGS, the proposed SGPose opts for a random initialization from a cuboid of 4,096 points. 
The proposed object Gaussian generates dense image pixels and object coordinates correspondence (2D-3D correspondence) maps efficiently using geometric-aware depth rendering, serving as a keystone advantage for monocular 6D pose estimation. An adapted GDRNet++ framework <cit.> is utilized to assess 6D pose estimation on the LM <cit.> and Occlusion LM-O <cit.> datasets. 
Our SGPose takes sparse view images and pose annotations to create synthetic views, object masks, and dense correspondence maps.
Noteworthy, for the Occlusion LM-O dataset, we render data similar to PBR (Physically Based Rendering) data <cit.>, which further enhance the performance of the proposed method.
By matching state-of-the-art performance across CAD-based and CAD-free approaches, we highlight the efficiency and flexibility of our method. 
To sum up, Our main contributions are:


  * By intaking only RGB images, the proposed geometric-aware object Gaussian derives accurate depth rendering from random point initialization;

  * The rendered depth ensures a reliable synthetic view warping and an effective online pruning, addressing the issue of overfitting under sparse views at an impressively low time cost;

  * By generating dense 2D-3D correspondences and images that simulate real occlusions using the proposed object Gaussian, our SGPose framework achieves CAD-free monocular pose estimation that is both efficient and robust. 




§ RELATED WORK




  §.§.§ CAD-Based Object Pose Estimation


Many previous works on pose estimation rely on known CAD models. Regression-based methods <cit.> estimate pose parameters directly from features in regions of interest (RoIs), while keypoint-based methods establish correspondences between 2D image pixels and 3D object coordinates either by regression <cit.> or by voting <cit.>, often solve poses by using a variant of Perspective-n-Points (PnP) algorithms <cit.>.NOCS <cit.> establishes correspondences between image pixels and Normalized Object Coordinates (NOCS) shared across a category, reducing dependency on CAD models at test time. Later works <cit.> build upon this idea by leveraging category-level priors to recover more accurate shapes.
A limitation of these methods is that objects within the same category can have significant variations in shape and appearance, which challenges the generalization of trained networks. Additionally, accurate CAD models are required for generating ground-truth NOCS maps during training. In contrast, our framework reconstructs 3D object models from pose-annotated images, enabling CAD-free object pose estimation during both training and testing phases.



  §.§.§ CAD-Free Object Pose Estimation


Some endeavors have been made to relax the constraints of CAD models of the objects. 
RLLG <cit.> uses multi-view consistency to supervise coordinate prediction by minimizing reprojection error.  
NeRF-Pose <cit.> trains a NeRF-based <cit.> implicit neural representation of object and regresses object coordinate for pose estimation.
Gen6D <cit.> initializes poses using detection and retrieval but requires accurate 2D bounding boxes and struggles with occlusions. 
GS-Pose <cit.> improves on Gen6D <cit.> by employing a joint segmentation method and 3DGS-based refinement. 
OnePose <cit.> reconstructs sparse point clouds of objects and extracts 2D-3D correspondences, though its performance is limited on symmetric or textureless objects due to its reliance on repeatable keypoint detection.
While OnePose++ <cit.> removes the dependency on keypoints resulting in a performance enhancement.
Unlike these methods, which require numerous input images for training, we directly leverage the power of 3DGS <cit.> for geometric-aware object reconstruction from sparse, pose-annotated images to achieve pose estimation.



§ METHODS




An overview of the proposed method is presented in Fig. <ref>.
Given sparse views as input, the dense 2D-3D correspondence maps are encoded in the conceived object Gaussian naturally, by supervising geometric-aware depth. Consequently, the created synthetic views and correspondence maps are availed to a downstream pose estimator, to achieve CAD-free monocular Pose Estimation.



 §.§ Depth Rendering of Geometric-aware Object Gaussian


The object geometry is described by Gaussian primitives of probability density function as <cit.>, 

    𝒢(𝐱)=e^-1/2(𝐱-μ)^⊤Σ^-1(𝐱-μ),

where 𝐱 is a point in world space to describe the target object and μ is the mean of each Gaussian primitive (which also is the geometric center). Thus, the difference vector 𝐱-μ indicts the probability density of 𝐱, which peaks at the center μ and decreases as departing from it. 

By treating Gaussian primitives as the elliptical disks <cit.>, the covariance matrix Σ is parameterized on a local tangent plane centered at μ with a rotation matrix and a scaling matrix. 
Concretely, the rotation matrix R is comprised of three vectors 𝐭_u, 𝐭_v and 𝐭_w, where two orthogonal tangential vectors 𝐭_u and 𝐭_v indicate the orientations within the local tangent plane, and 𝐭_w=𝐭_u ×𝐭_v represents the normal perpendicular to the plane. 
The scaling matrix S depicts the variances of Gaussian primitives on corresponding directions, noted that there is no distribution in the direction of 𝐭_w since the Gaussian primitive is defined on a flat elliptical disk. 
Thereby, the 3 × 3 rotation matrix R=[𝐭_u, 𝐭_v, 𝐭_w] and the scaling matrix S=[s_u, s_v, 0] form up the covariance matrix as Σ = RSS^⊤R^⊤.

By leveraging the world-to-camera transformation matrix W and the Jacobian of the affine approximation of the projective transformation matrix J, the projected 2D covariance matrix Σ^' in camera coordinates is given as, Σ^'=J W Σ W^⊤ J^⊤.
By virtue of the same structure and properties are maintained by skipping the third row and column of Σ^'  <cit.>, a 2 × 2 variance matrix Σ^2 D (corresponding to 𝒢^2 D) is obtained,

    𝒢^2 D(𝐱^') =  e^-1/2(𝐱^'-μ^')^⊤(Σ^2 D)^-1(𝐱^'-μ^'),

where 𝐱^' and μ^' stands for the projected points of  𝐱 and μ in the screen space, respectively.

Furthermore, the local tangent plane is defined as,

    X(u, v)=μ+s_u 𝐭_u u+s_v 𝐭_v v=𝐇(u, v, 1,1)^⊤,

where 𝐇=[[ s_u 𝐭_u s_v 𝐭_v       0       μ;       0       0       0       1 ]]=[[ RS  μ;  0  1 ]] is a homogeneous transformation matrix mapping point (u,v) on local tangent plane into the world space. 

Suppose there is a ray emitting from the camera optical center onto the screen space. 
The geometric-aware depth d^geo is hence defined as the distance between the camera and the Gaussian primitive along the ray. Accordingly, the homogeneous coordinate of the point (u,v) projected onto the screen is <cit.>, 

    (u^', v^', d, w)^⊤=W X(u, v)=W 𝐇(u, v, 1,1)^⊤.

where w is usually set to 1 (the homogeneous representation describes a point when w ≠ 0, while it depicts a ray when w = 0). 
This point can be further represented as the intersection of two orthogonal planes corresponding to u^' and v^' <cit.>.
Specifically , u^'-plane is defined by a normal vector (-1, 0, 0) and an offset u^', the 4D homogeneous plane thus is 𝐡_u^'=(-1, 0, 0, u^'). Similarly, v^'-plane is 𝐡_v^'=(0, -1, 0, v^'). Conversely, both planes can be transformed back to the local tangent plane coordinates as,

    𝐡_u=(W𝐇)^⊤𝐡_u^',   𝐡_v=(W𝐇)^⊤𝐡_v^',

in which (W𝐇)^⊤ is equivalent to (W𝐇)^-1 as show in <cit.>.
According to <cit.>, since the screen point (u^', v^') must lie on both u^'-plane and v^'-plane, for any point (u, v, 1, 1) on the elliptical disk, the dot product of the transformed plane 𝐡_u and 𝐡_v with the point (u, v, 1, 1)  should be zero,

    𝐡_u ·(u, v, 1,1)^⊤=𝐡_v ·(u, v, 1,1)^⊤=0,

by solving the equation above, the coordinates of the screen point (u^', v^') on the local tangent plane are yielded,

    u=𝐡_u^2 𝐡_v^4-𝐡_u^4 𝐡_v^2/𝐡_u^1 𝐡_v^2-𝐡_u^2 𝐡_v^1,    v=𝐡_u^4 𝐡_v^1-𝐡_u^1 𝐡_v^4/𝐡_u^1 𝐡_v^2-𝐡_u^2 𝐡_v^1,

where 𝐡_u^i, 𝐡_v^i are the elements of the 4D homogeneous plane parameters.

Thus far, the Gaussian primitives can be expressed with respect to (u,v). Suppose Σ^2 D M=I, by transforming the 2D covariance matrix Σ^2 D into the identity matrix I, the probability density function can be rewritten as standardized Gaussian (with mean of zero and deviation of one),

    𝒢(𝐱^')=e^-1/2(M(𝐱^'-μ^'))^⊤(M(𝐱^'-μ^')),

where 𝐱^' can be further replaced by (u, v) via some linear transformations as,

    𝒢(u, v)=e^ -1/2(u^2+v^2).

To further take account of numerical instability introduced by inverse homogeneous transformations of Eq. <ref>,  a lower bounded Gaussian is imposed <cit.>, 

    𝒢̂(u,v)=max{𝒢(u,v), 𝒢((u^',v^')-μ^'/r)}.

When the elliptic disk is projected as the segment line in some cases, a low-pass filter (centered at μ^' with radius r) is wielded to guarantee sufficient points passed toward the screen space (the radius is set as √(2) / 2 empirically by following <cit.>).

Suppose the opacity of i-th Gaussian primitive is α_i, by considering the alpha-weighted contribution along the ray, the accumulated transmittance is,

    T_i=∏_j=1^i-1(1-α_j 𝒢̂_̂ĵ(u, v)).

To this end, the proposed object Gaussian renders both image and depth map of the object. The final color is c^α=∑_i ∈𝒩T_i α_i 𝒢̂_̂î(u, v) c_i, with c_i as the view-dependent appearance represented by spherical harmonics  (SH) <cit.>.
The alpha-blended depth map is formulated via the summation of geometric-aware depth d^geo of each Gaussian primitive as,

    d^α=∑_i ∈𝒩T_i α_i 𝒢̂_̂î(u, v) max{d^geo_i | T_i>σ},

where σ=0.5 is a threshold deciding whether the rendered depth valid. Noted that the maximum depth along the ray is selected if T_i does not reach the threshold as in <cit.>. 



 §.§ Geometric-Consistency under Sparse Views


In the circumstance of extremely sparse view reconstruction, the object Gaussian struggles with over-fitting <cit.>, where the background collapse and floaters are commonly witnessed even the rendered view deviates marginally from the given one <cit.>. 


In principle, the effective solutions involve online synthetic view augmentation and geometric-consistent depth supervision.
Notably, effective online synthetic view augmentation remarkably reduces the need of a high budget of real images, and the multi-view geometric consistency prevents significant fluctuations on rendered depth.



  
Synthetic View Warping
Given what is at stake, it is intuitive to warp synthetic views online to enrich training samples, which encourages model to adapt from a diverse set of perspectives and brings better generalization capabilities on unseen views. 

Owing to lack of ground truth of synthetic views, the proposed alpha-blended depth map plays an essential role in warping process. 
Specifically, the rendered depth d^α is used to transform the given view into 3D points, which are re-projected as pixels of synthetic views. 
Formally, the pixel (u^'_g,v^'_g) of a given view is warped as (u^'_w,v^'_w) of an unseen view, which is

    (u_w^', v_w^')=K T[d^α K^-1(u_g^', v_g^', 1)^⊤],

where K is the camera intrinsic, and the rendered depth d^α serves as the pixel-wise scaling factor to confine the re-projection within a meaningful range.

Moreover, the transformation T from give view to warped view is obtained via perturbations (including rotations and translations) sampled from normal distribution randomly, by making use of tool provided in <cit.>.
The warped pixels are assembled as the ground truth image I_w,

    ℒ_warp=ℒ_1 (Î,    I_w).

While the ground truth image I_w and the rendered image Î of a specific synthetic view establish supervision of the image warping loss via ℒ_1.



  
Depth Supervision under Geometric-Consistent
Considering each Gaussian is in tandem with the depth distribution of a certain region in the scene, to concentrate the geometric-aware depth of Gaussian primitives along the ray is beneficial to refine each Gaussian's contribution to overall distribution. 
Accordingly, the geometric-consistent loss is employed as

    ℒ_geo=∑_i, jω_i ω_j|d^geo_i-d^geo_j|,

where ω_i= T_i α_i 𝒢̂_̂î(u, v) is the blending weight of i-th Gaussian primitive <cit.>.



  
Geometric-consistency Guided Online Pruning
Lastly, inspired by <cit.>, an online floaters pruning strategy is implemented by introducing a peak depth,

    d^peak= d^geo_max _i(ω_i).

The peak depth <cit.> is acquired by selecting the Gaussian of highest blending weight, which also is the Gaussian of the highest opacity.

To implement the multi-view geometric-aware depth comparison, the alpha-blending depth d^α and peak depth d^peak are compared under each given view.
Generally, the alpha-blending depth locates slightly behind the peak depth, the differences result in a candidate region for pruning. 
While the corresponding opacity α_i of peak depth within the region guides the online floater pruning. 



 §.§ 2D-3D Correspondence Generation


The proposed SGPose starts from RGB data alone, without taking advantage of external depth information, yet it effectively renders reliable geometric-aware depth maps.
Unlike traditional CAD-free pipelines that heavily rely on geometric initialization from SfM methods such as COLMAP <cit.>, our method handles the random initialization of a cuboid that approximates the bounding box of object. 
The differentiable optimization of the proposed object Gaussian is expressed as

    ℒ=ℒ_image+ℒ_warp+λ_1 ℒ_geo+λ_2 ℒ_normal.

Concretely, ℒ_image is the image rendering loss combining ℒ_1 with the D-SSIM term from <cit.>, which is implemented in the given views only.
ℒ_image for given views and ℒ_warp for synthetic views follow the identical optimization pipeline, which update the object Gaussian model alternatively. 
The reasons of implementing such a training strategy are two-folded. 
Firstly, data from respective views exhibit disparate geometric details, tackling with them independently accommodates the model to the diversified data distributions; Secondly, the stand-alone Gaussian densification and pruning mitigate fluctuations brought by view alternating. 
λ_1 is set as 10^4 to align up the scale of depth term with the other ones, and λ_2=0.005 for normal loss ℒ_normal=∑_i ω_i(1-n_i^⊤ϕ(u,v)) to facilitate the gradients of depth maps ϕ(u,v) in line with normal maps n_i <cit.>.

Overall, the geometric-consistent supervision under sparse views reconstructs the desirable object Gaussian.
The dense 2D-3D correspondences, generated to fully replace the CAD models, along with synthetic view color images and object masks, serve as the ground truth for a modified GDRNet++ <cit.> to perform monocular pose regression. 
Among them, the generation of 2D-3D correspondences and the simulation of realistic occlusions in images are essential for the task.

For dense 2D-3D correspondences, object points are obtained by transforming the rendered depth map into 3D points of camera coordinates via the known camera's intrinsic, and in turn mapping the points to world space via the specific view parameters (rotations and translations). Pixel coordinates are calculated from the rendered object mask.
Thus, the 3D points in world space and corresponding pixel coordinates are stack orderly as dense 2D-3D correspondences of any specific view, which is

    𝐌_2D-3D=
    [[ 𝐑_obj^⊤(d^α K^-1(u^', v^', 1)^⊤-𝐭_obj);                        (u^', v^')_mask ]],

where 𝐑_obj and 𝐭_obj are the specific view parameters. 












§ EXPERIMENTS


In this section, extensive experiments are conducted to demonstrate the competitive performance of the proposed SGPose.  



  §.§.§ Datasets


The proposed SGPose is evaluated on two commonly-used datasets, which are LM <cit.> and LM-O <cit.>.
LM is a standard benchmark for 6D object pose estimation of textureless objects, which consist of individual sequences of 13 objects 

of various sizes in the scenes with strong clutters and slight occlusion. Each contains about 1.2k real images with pose annotations. The dataset is split as 15% for training and 85% for testing.
LM-O is an extension of LM, from which to annotate one sequence of 8 objects with more severe occlusions of various degrees.

For the LM dataset, approximately 1k  images with 2D-3D correspondence maps are rendered for each object.
For LM-O, which is designed for pose estimation in scenarios with occlusions, 50k images with significant occlusions on transparent backgrounds are rendered and involved in training with a ratio of 10:1, by following the convention of CAD-based methods for a fair comparison <cit.>.



  §.§.§ Implementation Details








Ten real images and corresponding pose annotations are taken as input of the proposed geometric-aware object Gaussian, which are selected from real data in LM.
Different selection strategies are conducted, e.g., selecting samples uniformly, randomly, in term of maximum rotation differences and maximum Intersection over Union (IoU). 
The simple uniform selection is adopted in our method taking account of real-world practice.

The proposed object Gaussian tailors respective optimization strategies to the supervision signals.
The number of points of random initialization is set as 4,096.
The geometric-consistent loss involves in training at iteration 3,000 and the normal loss is enabled at iteration 7,000 as in <cit.>.
Ten sparse views are given, and two synthetic views are created online around each give view at iteration 4,999, that is, 30 images involve in training for each object.
The synthetic image warping spans 40% of training phase once it is activated, which updates the model alternatively with the image rendering loss.
Both image rendering loss and image warping loss are endowed with equal weights, dominating the training process.

Noted that it’s a little tricky to conduct pruning effectively in our problem settings.
Regions of Interests (RoIs) are remained for training and backgrounds are masked out, it is possible that pruning techniques working well for distant floaters remove part of the foreground objects mistakenly, which is more pronounced when the objects are thin and tall.
Thus, the objects phone and driller do not apply pruning empirically.

For the occluded scene generation for LM-O dataset, we utilize pose annotations from PBR (Physically Based Rendering) <cit.> for image rendering.
Each object is rendered onto the image using its respective geometric-aware object Gaussian, with all objects rendered sequentially in a single image. 
Realizing that individual object Gaussian do not inherently represent occlusions, we overlay the rendered images with the visible masks from PBR data to simulate occlusion scenarios effectively. 

The rendered masks, crafted from the proposed geometric-aware object Gaussian, is generated by mapping color images to a binary format. 
That is, assigning “1” to pixels within the object and “0” to those outside, thus producing a boolean array congruent with the original image's dimensions. 
It is possible that incorporating an extra mask loss for supervision could improve the performance of mask rendering in future work.

The 2D bounding boxes for pose estiamtion are obtained by borrow an off-the-shelf object detector yolov3 <cit.>.




  §.§.§ Evaluation Metrics

We evaluate our method with the most commonly used metrics including ADD(S)-0.1d and Proj@5pix.

ADD(S)-0.1d measures the mean distance between the model points transformed from the estimated pose and the ground truth. If the percentage of mean distance lies below 10% of the object’s diameter (0.1d), the estimated pose is regarded as correct. For symmetric objects with pose ambiguity, ADD(-S) measures the deviation to the closet model point <cit.>. 
Proj@5pix computes the mean distance between the projection of 3D model points with given predicted and ground truth object poses. The estimated pose is considered correct if the mean projection distance is less than 5 pixels.






 §.§ Comparison with State-of-the-Arts




  §.§.§ Results on LM




The proposed method is compared with the CAD-based methods DPOD <cit.>, PVNet <cit.>, CDPN <cit.>, GDR-Net <cit.>, SO-Pose <cit.> and CAD-free methods RLLG <cit.> , Gen6D <cit.>, OnePose <cit.>, OnePose++ <cit.>, GS-Pose <cit.>, and Nerf-Pose <cit.> on metric of ADD(S)-0.1d and Proj@5pix.
As shown in Tab. <ref>, even under the setting of sparse view training, our method achieves comparable performance compared to most CAD-free baselines that are trained with more than 100 views, and is on par with the CAD-based methods. 
Notably, our proposed object Gaussian is trained on only 10 views, whereas Nerf-Pose uses 156 views for OBJ-NeRF training.  
In brief, the objects where our method outperforms the best CAD-free method (i.e., NeRF-Pose^†) are highlighted in bold, and where it surpasses the best CAD-based method (i.e., SO-Pose) are in italic bold. 
Noteworthy, Gen6D^† is refined on a subset of the LM dataset, and NeRF-Pose^† is trained on relative camera pose annotations.
As show in Tab. <ref>, SGPose demonstrates an impressive 98.51% average performance using only 10 given views, outperforming all baselines according to the metric of Proj@5pix. Noted that our method uses YOLOv3 <cit.> as the object detector, while all the others use the more recent YOLOv5 <cit.>.






  §.§.§ Results on LM-O


As demonstrated in Tab. <ref>, our method is compared with state-of-the-arts w.r.t. the metric of average recall (%) of ADD(-S). 
Among CAD-based methods, “real+pbr” outperforms “real+syn” because “pbr” data <cit.> incorporate occlusions in object placement, with random textures, materials, and lighting, simulating a more natural environment compared to individually rendered synthetic data. Given the heavy occlusions typical of the LM-O dataset, training with “pbr” data significantly enhances performance. 
In our setting, we do not have access to CAD models nor do we leverage “pbr” data. 
Instead, we render the synthetic images that replicates the occlusion scenarios found in the LM-O dataset, using our proposed object Gaussian.
Exemplary, we exceed Nerf-Pose <cit.> by 5.83% with 55.03% compared to 49.2%, also rival NeRF-Pose^†, which is trained on relative camera poses instead of ground truth pose annotations, by 3.63%. Impressively, we even slightly outperform the best CAD-based method SO-Pose <cit.>.



 §.§ Ablations




  §.§.§ Qualitative comparison of 2D-3D Correspondence


The qualitative results of selected objects are presented in Fig. <ref>, where the transformed 3D bounding boxes are overlaid with the corresponding images. 
As observed, the predicted poses (green boxes) mostly align with the ground truth (blue boxes). 
The images are cropped and zoomed into the area of interest for better visualization.
The rendered images are exhibited in column (c) and (h), compared to the reference CAD models in column (a), our object Gaussian successfully retains both the silhouette and the details of the objects. 
The accurate geometric shapes rendered from the object Gaussian ensure the performance of the subsequent pose estimation.  
Nonetheless, given the inherently challenging nature of sparse view reconstruction, some imperfections in the predicted shapes are also evident. 



  §.§.§ Training with Occlusions


Since LM-O is a more challenging dataset presenting complex occlusions of objects, the integration of synthetic data that captures diverse poses and realistic occlusions is beneficial for enhancing performance. 
We thus avail the object Gaussian to render such images to enrich training.
Quantitatively, as shown in Tab. <ref>,  two different synthetic data are rendered for training, "Occluded object" indicates images containing multiple objects with occlusions, whereas "Individual object" signifies images with a single unoccluded object. 
We observe that the use of "Occluded object" rendering improves the performance of the proposed SGPose by large margins under all the objects.


For the LM-O dataset, the proposed SGPose generates images and 2D-3D correspondences that demonstrate a diverse range of poses and realistic occlusions, as shown in Fig. <ref>. 
In the training process, the synthetic images are integrated at a 10:1 ratio with real images, meaning that for every ten real images, one synthetic image is included.
The 2D-3D correspondence maps are projected onto the target object in the images for visualization. 
Compared to training with individual object rendering, the inclusion of occluded object rendering remarkably enhances the model's performance in complex scenarios where objects have partial visibility.



  §.§.§ Effectiveness of Synthetic View Warping


As shown in Tab. <ref>, successful reconstruction of the object ape in LM requires a minimum of 33 images without synthetic view warping. 
By synthesising 20 novel view alone with 10 given images, the proposed SGPose maintains the performance. 
This demonstrates the effectiveness of synthetic view warping in reducing the reliance on real images.



  §.§.§ Effectiveness of Online Pruning


Point cloud accuracy is quantified as the proportion of reconstructed point clouds that fall within a specified distance threshold (e.g., 3mm) relative to the ground truth point clouds, where the vertices of the object meshes serve as the ground truth reference <cit.>.
The point cloud accuracy is evaluated without and with online pruning for our proposed object Gaussian, following the established protocols in <cit.> and <cit.>.  
The results presented in Tab. <ref> demonstrate that the online pruning removes outliers from sparse view object Gaussian reconstruction, resulting in a more accurate and compact representation.
Additionally, pose estimation comparison of selected objects w.r.t. ADD(S)-0.1d and Proj@5pix without and with online pruning is presented in Tab. <ref>.
Besides, sparse view reconstruction poses a challenge for object with thin and long geometry, such as lamp and glue. The timely application of online pruning, initiated as divergence threatens, ensures the model to be reconstructed successfully.



 §.§ Qualitative Results of Synthetic View Warping


The synthetic view for each real image is generated by introducing a controlled amount of noise to the given view, ensuring that the synthetic images retain a realistic and plausible appearance. The perturbation parameters are carefully chosen to keep the object within the camera's field of view.
Specifically, the Euler angles for rotation perturbation are sampled from a normal distribution with a standard deviation of 15°, capped at an upper limit of 45°. The translation perturbation along each axis is independently sampled from a normal distribution with standard deviations of 0.01 m for the x and y axes, and 0.05 m for the z-axis, respectively.
Fig. <ref> displays the qualitative results. Column (a) and (e) presents the ground truth of given views, while column (b) and (f) shows the corresponding rendered images from SGPose. Columns (c) and (g) illustrate the ground truth of synthetic views, and columns (d) and (h) exhibit the rendered synthetic images, respectively. 
The rendered results are obtained at the iteration 30k upon completion of the training.








 §.§ Implementation and Runtime Analysis


The experiments are conduct on an platform with an Intel(R) Xeon(R) Gold 5220R 2.20GHz CPU and Nvidia RTX3090 GPUs of 24GB Memory.
Given ten 640×480 images as input, the object Gaussian costs about 10 minutes to reconstruct one object and real-time renders image, mask, and 2D-3D correspondence.
An ImageNet <cit.> pre-trained ConvNeXt <cit.> network is leveraged as the backbone of our pose regression network, for a 640×480 image, the proposed SGPose takes about 24ms for inference.



§ LIMITATIONS


In future work, we plan to reduce the training time to enable portable online reconstruction and pose estimation, thereby facilitating real-time, end-to-end pose estimation suitable for real-world applications.



§ CONCLUSION


The proposed SGPose presents a monocular object pose estimation framework, effectively addressing the limitations of traditional methods that rely on CAD models. 
By introducing a novel approach that requires as few as ten reference views, the derived geometric-aware depth guides the object-centric Gaussian model to perform synthetic view warping and online pruning effectively, showcasing its robustness and applicability in real-world scenarios under sparse view constrains. 
The occlusion data rendered from the proposed object Gaussian substantially enhances the performance of post estimation, set our SGPose a state-of-the-art on the Occlusion LM-O dataset.







