# Confronting Ambiguity in 6D Object Pose Estimation
via Score-Based Diffusion on SE(3)

Tsu-Ching Hsiao, Hao-Wei Chen, Hsuan-Kung Yang, and Chun-Yi Lee
Elsa Lab, National Tsing Hua University
{joehsiao, jaroslaw1007<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Addressing pose ambiguity in 6D object pose estimation from single RGB images
presents a significant challenge, particularly due to object symmetries or
occlusions. In response, we introduce a novel score-based diffusion method
applied to the $SE(3)$ group, marking the first application of diffusion
models to $SE(3)$ within the image domain, specifically tailored for pose
estimation tasks. Extensive evaluations demonstrate the method’s efficacy in
handling pose ambiguity, mitigating perspective-induced ambiguity, and
showcasing the robustness of our surrogate Stein score formulation on $SE(3)$.
This formulation not only improves the convergence of denoising process but
also enhances computational efficiency. Thus, we pioneer a promising strategy
for 6D object pose estimation.

##  1 Introduction

Estimating the six degrees of freedom (DoF) pose of objects from a single RGB
image remains a formidable task, primarily due to the presence of ambiguity
induced by symmetric objects and occlusions. Symmetric objects exhibit
identical visual appearance from multiple viewpoints, whereas occlusions arise
when key aspects of an object are concealed either by another object or its
own structure. This can complicate the determination of its shape and
orientation. Pose ambiguity presents a unique challenge as it transforms the
direct one-to-one correspondence between an image and its associated object
pose into a complex one-to-many scenario, which can potentially leads to
significant performance degradation for methods reliant on one-to-one
correspondence. Despite extensive exploration in the prior object pose
estimation literature [39, 21, 10, 41, 19], pose ambiguity still remains a
persisting and unresolved issue.

Recent advancements in pose regression have introduced the use of symmetry-
aware annotations to improve pose estimation accuracy [39, 44, 64, 60]. These
methods typically employ symmetry-aware losses that can tackle the pose
ambiguity problem. The efficacy of these losses, nevertheless, depend on the
provision of symmetry annotations, which can be particularly challenging to
obtain for objects with intricate shapes or under occlusion. An example is a
texture-less cup, where the true orientation becomes ambiguous if the handle
is not visible.The manual labor and time required to annotate the equivalent
views of each object under such circumstances is impractical.

Figure 1:  Visualization of the denoising process of our score-based diffusion
method on $SE(3)$ for 6DoF pose estimation.

Several contemporary studies have sought to eliminate the reliance on symmetry
annotations by treating ‘equivalent poses’ as a multi-modal distribution,
reframing the original pose estimation problem as a density estimation
problem. Methods such as Implicit-PDF [41] and HyperPose-PDF [23] leverage
neural networks to implicitly characterize the non-parametric density on the
rotation manifold $SO(3)$. While these advances are noteworthy, they also
introduce new complexities. For instance, the computation during training
requires exhaustive sampling across the whole $SO(3)$ space. Moreover, the
accuracy of inference is dependent on the resolution of the grid search, which
necessitates a significant amount of grid sampling. These computational
limitations are magnified when extending to larger spaces such as $SE(3)$ due
to the substantial memory requirements.

Recognizing these challenges, the research community is pivoting towards
diffusion models (DMs) [57, 16, 56, 58], which are effective in handling
multi-modal distributions. Their effectiveness lies in the iterative sampling
process, which incorporates noises and enables a more focus exploration of the
pose space while reducing computational demands. As diffusion models refrain
from explicit density estimation, this property enables them to handle large
spaces and high-dimensional distributions. In prior endeavors, the authors in
[33, 28] applied the denoising diffusion probabilistic model (DDPM) [16] and
score-based generative model (SGM) [58] to the $SO(3)$ rotation manifold,
effectively recovering unknown densities on the $SO(3)$ space. On the other
hand, other research efforts [61, 71] have extended the application of
diffusion models to the more complex $SE(3)$ space, which enlightens the
potential applicability of diffusion models in object pose estimation tasks.

In light of the above motivations, we introduce a novel approach that applies
diffusion models to the $SE(3)$ group for object pose estimation tasks,
specifically aimed at addressing the pose ambiguity problem. This method draws
its inspiration from the correlation observed between rotation and translation
distributions, a phenomenon often resultant from the perspective effect
inherent in image projection. We propose that by jointly estimating the
distribution of rotation and translation on $SE(3)$, we may secure more
accurate and reliable results as shown in Fig. 1. To the best of our
knowledge, this is the first work to apply diffusion models to $SE(3)$ within
the context of image space. To substantiate our approach, we have developed a
new synthetic dataset, called SYMSOL-T, based on the original SYMSOL dataset
[41]. It enhances the original dataset with randomly sampled translations,
offering a more rigorous testbed to evaluate our method’s effectiveness in
capturing the joint density of object rotations and translations.

Following the motivations discussed above, we have extensively evaluated our
$SE(3)$ diffusion model using the synthetic SYMSOL-T dataset and a real-world
T-LESS [20] dataset. The experimental results affirm the model’s competence in
handling $SE(3)$, which successfully addresses the pose ambiguity problem in
6D object pose estimation. Moreover, the $SE(3)$ diffusion model has proven
effective in enhancing rotation estimation accuracy and robustness.
Importantly, the surrogate Stein score formulation we propose on $SE(3)$
exhibits improved convergence in the denoising process compared to the score
calculated via automatic differentiation. This not only highlights the
robustness of our method, but also demonstrates its potential to handle
complex dynamics in object pose estimation tasks.

##  2 Background

###  2.1 Lie Groups and Their Applications

A Lie group, denoted by $\mathcal{G}$, serves as a mathematical structure with
broad applicability due to its dual nature as both a group and a smooth (or
differentiable) manifold. The latter is a topological space that can be
locally approximated as a linear space. In accordance with the axioms
governing groups, a composition operation is formally defined as a mapping
$\circ:\mathcal{G}\times\mathcal{G}\to\mathcal{G}$. The composition operation,
along with the associated inversion map, exhibits smoothness properties
consistent with the group structure. For notational convenience in subsequent
analyses, the composition of two group elements $X,Y\in\mathcal{G}$ is
succinctly denoted as $X\circ Y=XY$. Every Lie group $\mathcal{G}$ has an
associated Lie algebra, denoted as $\mathfrak{g}$. A Lie group and its
associated Lie algebra are related through the following mappings:
$\text{Exp}:\mathfrak{g}\rightarrow\mathcal{G},~{}\text{Log}:\mathcal{G}\rightarrow\mathfrak{g}$.
In the context of pose estimation, two Lie groups are commonly employed:
$SO(3)$ and $SE(3)$. The Lie group $SO(3)$ and its associated Lie algebra
$\mathfrak{so}(3)$ can represent rotations in three-dimensional Euclidean
space. On the other hand, the Lie group $SE(3)$, along with its corresponding
Lie algebra $\mathfrak{se}(3)$, can be employed to describe rigid-body
transformations, which incorporate both rotational and translational elements
in Euclidean space. Such group structures form the mathematical basis for
analyzing and solving complex problems, especially for six Degrees of Freedom
(6DoF) pose estimation.

###  2.2 Lie Group Representation of Transformations

A variety of parametrizations for these transformation groups are discussed in
[55]. This work considers two types of transformation groups, each
characterized by a distinct manifold structure and the accompanying
parametrizations: $R^{3}SO(3)$ and $SE(3)$. The former parametrization, which
segregates rotations $R\in SO(3)$ and translations $T\in\mathbb{R}^{3}$ into a
composite manifold $\langle\mathbb{R}^{3},SO(3)\rangle$, denotes its Lie
algebra as $\langle\mathbb{R}^{3},\mathfrak{so}(3)\rangle$. $R^{3}SO(3)$
employs a composition rule defined by
$(R_{2},T_{2})(R_{1},T_{1})=(R_{2}R_{1},T_{2}+T_{1})$. This parametrization,
which is prevalent in several prior diffusion models on $R^{3}SO(3)$ due to
its simplicity as discussed in [71, 61], induces a separate diffusion process
for both $R$ and $T$. Another parametrization, $SE(3)$, formulates elements
within the Lie algebra as $\tau=(\rho,\phi)\in\mathfrak{se}(3)$, wherein
$\rho$ and $\phi$ correspond to infinitesimal translations and rotations at
the identity element’s tangent space, respectively. The corresponding group
elements within $SE(3)$ are represented as
$(R,T)=(\text{Exp}(\phi),\mathbf{J}_{l}(\phi)\rho)$, where $\mathbf{J}_{l}$
denotes the left-Jacobian of $SO(3)$. The composition rule for the $SE(3)$
parametrization is expressed as
$(R_{2},T_{2})(R_{1},T_{1})=(R_{2}R_{1},T_{2}+R_{2}T_{1})$. The integration of
both rotations and translations within $SE(3)$ gives rise to a diffusion
process that emulates the elaborate dynamics of rigid-body motion.

###  2.3 Score-Based Generative Modeling

Consider independent and identically distributed (i.i.d.) samples
$\\{\mathbf{x}_{i}\in\mathbb{R}^{D}\\}^{N}_{i=1}$ drawn from a data
distribution $p_{\text{data}}(\mathbf{x})$. The (Stein) score of a probability
density $p(\mathbf{x})$ is the gradient of its logarithm, denoted as
$\nabla_{\mathbf{x}}\log p(\mathbf{x})$ [27]. In the framework of score-based
generative models (SGMs), an important formulation within the spectrum of
diffusion models, data undergo a gradual transformation toward a known prior
distribution. Such a distribution is often selected for computational
tractability [63], and this process is termed the forward process. The forward
process is characterized by a series of increasing noise levels
$\\{\sigma_{i}\\}^{L}_{i=1}$, which are ordered such that
$\sigma_{\text{min}}=\sigma_{1}<\sigma_{2}<\ldots<\sigma_{L}=\sigma_{\text{max}}$.
The selection of $\sigma_{\text{min}}$ and $\sigma_{\text{max}}$ as
sufficiently small and large values respectively facilitates the approximation
of $p_{\sigma_{\text{min}}}(\mathbf{x})$ to $p_{\text{data}}(\mathbf{x})$ and
of $p_{\sigma_{\text{max}}}(\mathbf{x})$ to the Gaussian distribution
$\mathcal{N}(\mathbf{x};\mathbf{0},\sigma_{\text{max}}^{2}\mathbf{I})$. This
process utilizes a perturbation kernel
$p_{\sigma}(\tilde{\mathbf{x}}|\mathbf{x})=\mathcal{N}(\tilde{\mathbf{x}};\mathbf{x},\sigma^{2}\mathbf{I})$,
and the perturbed distribution is given by
$p_{\sigma}(\tilde{\mathbf{x}})=\int
p_{\text{data}}(\mathbf{x})p_{\sigma}(\tilde{\mathbf{x}}|\mathbf{x})d\mathbf{x}$.
In the Noise Conditional Score Network (NCSN) [57], a network
$s_{\boldsymbol{\theta}}(\mathbf{x},\sigma)$ parameterized by $\theta$ is
trained to estimate the score via a Denoising Score Matching (DSM) objective
[63] as follows:

${\scriptsize\begin{split}\boldsymbol{\theta}^{\ast}&=\mathop{\arg\min}_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{{\theta}};\sigma)\\\
&\triangleq\frac{1}{2}\mathbb{E}_{p_{\text{data}}(\mathbf{x})}\mathbb{E}_{\tilde{\mathbf{x}}\sim\mathcal{N}(\mathbf{x},\sigma^{2}I)}\left[\left\|s_{\boldsymbol{\theta}}(\tilde{\mathbf{x}},\sigma)-\nabla_{\tilde{\mathbf{x}}}\log
p_{\sigma}(\tilde{\mathbf{x}}|\mathbf{x})\right\|^{2}_{2}\right].\end{split}}$
(1)

The optimal score-based model
$s_{\boldsymbol{\theta}^{\ast}}(\mathbf{x},\sigma)$ aims to match
$\nabla_{\mathbf{x}}\log p(\mathbf{x})$ as closely as possible across the
entire range of $\sigma$ values in the set $\\{\sigma_{i}\\}^{L}_{i=1}$.
During the sample generation phase, score-based generative models employ an
iterative reverse process. Specifically, in the context of the Noise
Conditional Score Network (NCSN), the Langevin Markov Chain Monte Carlo (MCMC)
method is utilized to execute $M$ steps. This process is designed to produce
samples in a sequential manner from each $p_{\sigma_{i}}(\mathbf{x})$,
expressed as follows:

$\footnotesize\tilde{\mathbf{x}}^{m}_{i}=\tilde{\mathbf{x}}^{m-1}_{i}+\epsilon_{i}s_{\boldsymbol{\theta}^{\ast}}(\tilde{\mathbf{x}}^{m-1}_{i},\sigma_{i})+\sqrt{2\epsilon_{i}}\mathbf{z}^{m}_{i},\quad
m=1,2,...,M,$ (2)

where $\epsilon_{i}>0$ denotes the step size, and $\mathbf{z}^{m}_{i}$
represents a standard normal variable. Overall, diffusion based models,
especially SGMs, provide a solid framework for handling complex data
distributions. They serve as the foundation for the denoising procedure
employed by our methodology.

##  3 Related Work

Table 1: Comparison of different methods. $\triangle$ means closed form but
with approximation. $\mathcal{N}_{SE(3)}$ please refer to Eq. (3).

Baselines | Group | Distribution | Closed Form | Diffusion Method | Diffusion Space | App. Domain
---|---|---|---|---|---|---
Leach _et al_. [33] | $SO(3)$ | $IG_{SO(3)}$ | ✗ | DDPM | $SO(3)$ | Vector
Jagvaral _et al_. [28] | $SO(3)$ | $IG_{SO(3)}$ | ✗ | Score / Autograd | $SO(3)$ | Vector
Urain _et al_. [61] | $R^{3}SO(3)$ | $\mathcal{N}_{\mathbb{R}^{3}}\times\mathcal{N}_{SO(3)}$ | ✓ | Score / Autograd | $R^{3}SO(3)$ | Vector
Yim _et al_. [71] | $R^{3}SO(3)$ | $\mathcal{N}_{\mathbb{R}^{3}}\times IG_{SO(3)}$ | ✗ | Score / Autograd | $\langle\mathbb{R}^{3},\mathfrak{so}(3)\rangle$ | Vector
Ours | $SE(3)$ | $\mathcal{N}_{SE(3)}$ | $\triangle$ | Score / Closed Form | $SE(3)$ | Image

###  3.1 Methodologies for Dealing with Pose Ambiguity

#### Non-probabilistic modeling.

In the realm of object pose estimation, pose ambiguity remains a significant
challenge, often stemming from an object that exhibits identical visual
appearances from different perspectives [39]. A variety of strategies have
been explored in the literature to directly address this issue, including the
application of symmetry supervisions and point matching algorithms [66, 1].
Regression-based approaches, such as those presented in [32, 64, 11, 60], aim
to minimize pose discrepancy by selecting the closest candidate within a set
of ambiguous poses. Some researchers [46, 48], on the other hand, introduce
constraints to the regression targets (especially regarding rotation angles)
to mitigate ambiguity. Moreover, certain approaches [44, 65, 25] suggest
regressing to a predetermined set of geometric features derived from symmetry
annotations. These prior arts often necessitate manual annotations of
equivalent poses and are limited in dealing with other sources of pose
ambiguities, such as those caused by occlusion and self-occlusion [39].

#### Probabilistic modeling.

On the other hand, several studies have investigated methods to model the
inherent uncertainty in pose ambiguity. This involves the quantification and
representation of uncertainty associated with the estimated poses. Some works
have employed parametric distributions such as Bingham distributions [43, 12,
10] and von-Mises distributions [47, 72] to model orientation uncertainty.
Other approaches, such as in [38], utilize normalizing flows [50] to model
distributions within rotational space. A number of studies [41, 23, 31] employ
non-parametric distributions to implicitly represent rotation uncertainty on
$SO(3)$. These methods primarily focus on modeling distributions on $SO(3)$,
leaving the joint distribution modeling of rotation and translation
unexplored.

###  3.2 Diffusion Probabilistic Models and Their Application Domains

#### Diffusion models on Euclidean space.

Diffusion probabilistic models [68, 16, 56, 58, 57] represent a class of
generative models designed to learn the underlying probability distribution of
data. They have been applied to various generative tasks, and have shown
impressive results in several application domains, including image [49, 52,
51, 53, 2, 3, 7], video [69, 18, 17], audio [26, 67], and natural language
processing [13, 35]. In the realm of human pose estimation, diffusion models
have also been found useful in addressing joint location ambiguity, which
arises from the projection of 2D keypoints into 3D space [9, 24].

#### Diffusion models on non-Euclidean space.

To accommodate data residing on a manifold, the authors in [5] extended
diffusion models to Riemannian manifolds, and leveraged Geodesic Random Walk
[29] for sampling. Other studies [28, 33] applied the Denoising Diffusion
Probabilistic Models (DDPM) [16] and score-based generative models [58, 57] to
the $SO(3)$ manifold to recover the density of data on $SO(3)$. Further
extensions of diffusion models have been attempted for tasks such as unfolding
protein structures [71] and arm manipulations [61]. These approaches typically
used $R^{3}SO(3)$ parametrization, which treated rotation and translation as
separate entities for diffusion.

###  3.3 Diffusion Models on Lie Groups

Diffusion models on Lie groups have been explored in a range of applications
[33, 28, 61, 71]. Nevertheless, these implementations vary in their choices of
distributions and computational methods, which lead to diverse outcomes and
different levels of computational efficiency. Table 1 presents a comparison of
several previous diffusion model approaches along with our own. It highlights
the distinct groups, distributions, methods, as well as diffusion spaces each
method utilizes. Several earlier studies [33, 28] have introduced techniques
that operate within the $SO(3)$ space, and adopted normal distributions
defined on $SO(3)$ [42] (denoted as $IG_{SO(3)}$). Unfortunately, a primary
drawback of $IG_{SO(3)}$ is its absence of a closed form, which poses
challenges in its computational efficiency. In a similar vein, the authors in
[71] developed a method that operates in the tangent space of $R^{3}SO(3)$.
This method’s distribution also does not possess a closed form, which
complicates the computational procedure. On the other hand, the authors in
[61] employed a joint Gaussian distribution within the $\mathbb{R}^{3}$ and
$SO(3)$ spaces. This distribution benefits from the presence of a closed form
and thus offers the potential for increased computational efficiency. However,
this approach is confined to the $\mathbb{R}^{3}\times SO(3)$ space and treats
rotation and translation as separate entities for diffusion. As a result, it
may not be able to offer the advantages that $SE(3)$ can provide.

##  4 Methodology

Given an RGB image $I$ that displays the object of interest, our goal is to
estimate the 6D object poses $X=(R,T)\in SE(3)$, which represent the
transformation from the camera frame to the object. This estimation involves
sampling poses from a conditional distribution $X\sim p(X|I)$, which captures
the inherent pose uncertainty of the object depict in $I$. To facilitate this
process, our method employs a score-based generative model on $SE(3)$ to
recover this underlying distribution. Poses are then sampled via a reverse
process that gradually refines noisy pose hypotheses $\tilde{X}\sim
p(\tilde{X})$ drawn from a known prior distribution $p(\tilde{X})$,
specifically a Gaussian distribution on $SE(3)$. Both the forward and reverse
processes are performed on Lie groups and leverage the associated group
operations. It is important to note that our approach does not utilize 3D
models of the objects or symmetry annotations during either the training or
inference phases, instead relying exclusively on RGB images and the associated
ground truth (GT) poses for training.

###  4.1 Score-Based Pose Diffusion on a Lie Group

To apply score-based generative modeling to a Lie group $\mathcal{G}$, we
first establish a perturbation kernel on $\mathcal{G}$ that conforms to the
Gaussian distribution [54, 8]. The kernel is given by:

${\scriptsize\begin{split}p_{\Sigma}(Y|X)&:=\mathcal{N}_{\mathcal{G}}(Y;X,\Sigma)\\\
&\triangleq\frac{1}{\zeta(\Sigma)}\exp\left(-\frac{1}{2}\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\text{Log}(X^{-1}Y)\right),\end{split}}$
(3)

where $\Sigma$ is the covariance matrix with diagonal entries populated by
$\sigma$ for representing the scale of the perturbation, $\zeta(\Sigma)$ is
the normalizing constant, and $X,Y\in\mathcal{G}$ denote the group elements.
The score on $\mathcal{G}$ then corresponds to the gradient of the log-density
of the data distribution with respect to the group element $Y$. It can be
formulated as follows:

$\footnotesize\nabla_{Y}\log
p_{\Sigma}(Y|X)=-\mathbf{J}_{r}^{-\top}(\text{Log}(X^{-1}Y))\Sigma^{-1}\text{Log}(X^{-1}Y).$
(4)

This term can be expressed in closed form if the inverse of the right-Jacobian
$\mathbf{J}_{r}^{-1}$ on $\mathcal{G}$ exists in a closed form. Nevertheless,
an alternative approach suggested by the authors in [61] would be to compute
this term using automatic differentiation [45]. By substituting $Y$ with
$\tilde{X}$, assuming
$\tilde{X}=X\text{Exp}(z),~{}z\sim\mathcal{N}(0,\sigma_{i}^{2}I)$, and
integrating the above definition, the score on $\mathcal{G}$ can be
reformulated as follows:

$\nabla_{\tilde{X}}\log
p_{\sigma}(\tilde{X}|X)=-\frac{1}{\sigma^{2}}\mathbf{J}_{r}^{-\top}(z)z.$ (5)

A score model $s_{\boldsymbol{\theta}}(\tilde{X},\sigma)$ can then be trained
using the DSM objective shown in Eq. (1), which takes the following form:

${\scriptsize\begin{split}\boldsymbol{\theta}^{\ast}&=\mathop{\arg\min}_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{{\theta}};\sigma)\\\
\triangleq&\frac{1}{2}\mathbb{E}_{p_{\text{data}}(X)}\mathbb{E}_{\tilde{X}\sim\mathcal{N}_{\mathcal{G}}(X,\Sigma)}\left[\left\|s_{\boldsymbol{\theta}}(\tilde{X},\sigma)-\nabla_{\tilde{X}}\log
p_{\sigma}(\tilde{X}|X)\right\|^{2}_{2}\right].\end{split}}$ (6)

For the denoising process, we employ a variant of the Geodesic Random Walk
[5], tailored to the Lie group context, as a means to generate a sample from a
noise distribution. The procedure is expressed as follows:

${\small\begin{split}\tilde{X}_{i+1}=\tilde{X}_{i}\text{Exp}(\epsilon_{i}s_{\theta}(\tilde{X}_{i},\sigma_{i})+\sqrt{2\epsilon_{i}}z_{i}),\quad
z_{i}\sim\mathcal{N}(0,I).\end{split}}$ (7)

###  4.2 Efficient Computation of the Stein Score

Even with the above derivation, obtaining the closed-form score remains
challenging due to its dependency on the selected distribution. For instance,
deriving the closed-form score for the $IG_{SO(3)}$ distribution [42] poses
difficulties. Furthermore, computing the score depends on the existence of a
closed-form expression for the Jacobian matrix on $\mathcal{G}$. Even if such
an expression exists, it may not guarantee computational efficiency compared
to automatic differentiation. Therefore, we next discuss a simplification
method of the Stein score under certain conditions for reducing computational
costs on $\mathcal{G}$. This can be expressed in a closed-form if the Jacobian
matrix on $\mathcal{G}$ is invertible and if the left and right Jacobian
matrices conform to the following relation:

$\mathbf{J}_{l}(z)=\mathbf{J}^{\top}_{r}(z),\quad\mathbf{J}_{l}^{-1}(z)=\mathbf{J}_{r}^{-\top}(z),$
(8)

where $z\in\mathfrak{g}$. As pointed out by [55], $SO(3)$ exhibits this
property. Its closed-form score can then be simplified by utilizing the
following property, which holds on any $\mathcal{G}$ as
$\mathbf{J}_{l}(z)z=z$. The derivation is in the supplementary material. The
score on $SO(3)$ can then be expressed as follows:

$\nabla_{\tilde{X}}\log
p_{\sigma}(\tilde{X}|X)=-\frac{1}{\sigma^{2}}\mathbf{J}_{l}^{-1}(z)z=-\frac{1}{\sigma^{2}}z.$
(9)

This shows that the score on $SO(3)$ can be simplified to the sampled Gaussian
noise $z$ scaled by $-1/{\sigma^{2}}$, thus eliminating the need for both
automatic differentiation and Jacobian calculations. Similarly, the score on
$R^{3}SO(3)$ also has a closed-form as its Jacobians satisfy the relations in
Eq. (8):

$\mathbf{J}_{l}(z)=(I,\mathbf{J}_{l}(\phi))=(I,\mathbf{J}_{r}^{\top}(\phi))=\mathbf{J}_{r}^{\top}(z),$
(10)

where in the case of $R^{3}SO(3)$,
$z=(T,\phi)\in\langle\mathbb{R}^{3},\mathfrak{so}(3)\rangle$. This implies
that the score on $R^{3}SO(3)$ can also be simplified according to the
formulation represented by Eq. (9).

Figure 2: Left: Framework overview. Right: Visualization of a denoising step
from a noisy sample $\tilde{X}$ to its cleaned counterpart $X$ on $SE(2)$. The
contours are the distances to $X$ in 2D Euclidean space. Each line represents
a denoising path with varying sub-sampling steps.

###  4.3 Surrogate Stein Score Calculation on $SE(3)$

While the score on $SO(3)$ and $R^{3}SO(3)$ can be simplified as described in
the preceding sections, it can be shown that $SE(3)$ does not possess the
property in Eq. (8). Consider the inverse of the left-Jacobian on $SE(3)$ at
$z=(\rho,\phi)\in\mathfrak{se}(3)$, expressed as
$\mathbf{J}_{l}^{-1}(z)=\left[\begin{smallmatrix}\mathbf{J}_{l}^{-1}(\phi)&\mathbf{Z}(\rho,\phi)\\\
0&\mathbf{J}_{l}^{-1}(\phi)\\\ \end{smallmatrix}\right]$, where
$\mathbf{Z}(\rho,\phi)=-\mathbf{J}^{-1}_{l}(\phi)\mathbf{Q}(\rho,\phi)\mathbf{J}^{-1}_{l}(\phi)$.
The complete form of $\mathbf{Q}(\rho,\phi)$ can be found in [55, 4] and our
supplementary material. The property
$\mathbf{Q}^{\top}(-\rho,-\phi)=\mathbf{Q}(\rho,\phi)$, as derived in the
references, leads to the following inequality:

${\footnotesize\begin{split}\mathbf{J}_{r}^{-\top}(z)=(\mathbf{J}_{l}^{-1}(-z))^{\top}=\left[\begin{smallmatrix}\mathbf{J}^{-1}_{l}(\phi)&0\\\
\mathbf{Z}(\rho,\phi)&\mathbf{J}^{-1}_{l}(\phi)\end{smallmatrix}\right]\neq\mathbf{J}^{-1}_{l}(z).\end{split}}$
(11)

This inequality indicates the potential discrepancy between the score vector
and the denoising direction due to the curvature of the manifold, which may
impede the convergence of the reverse process and necessitate additional
denoising steps. To address this problem, we turn to higher-order
approximation methods by breaking one step of reverse process into multiple
smaller sub-steps. Fig. 2 (right) illustrates this one-step denoising process
on $SE(2)$ from a noisy sample $\tilde{X}=X\text{Exp}(z)$ to its cleaned
counterpart $X$, with contour lines representing the distance to $X$ in 2D
Euclidean space. We observe that increasing the number of sub-steps eventually
leads the integral of those small transformations approaches the inverse of
$z$. As a result, we propose substituting the true score in Eq. (5) with a
surrogate score in our training objective of Eq. (6) on $SE(3)$, defined as
follows:

${\small\begin{split}\tilde{s}_{X}(\tilde{X},\sigma)\triangleq-\frac{1}{\sigma^{2}}z.\end{split}}$
(12)

Note that the detailed training and sampling procedures are described and
elaborated in our supplementary material.

###  4.4 The Proposed Framework

Fig. 2 (left) presents an overview of our framework, which consists of a
conditioning part and a denoising part. The conditioning part is responsible
for generating the condition variable $c$, which is crucial for guiding the
denoising process. This variable $c$ can be derived either from an image
encoder which extracts features from an image, or from a positional embedding
module [62] that encodes a time index $i$. In our experiments, we employ
ResNet [14] as the image encoder. The separation of the two parts in our
framework eliminates the need of image feature extraction in every denoising
step, which offers efficiency in the inference phase. For the denoising part,
our score model is composed of multiple multi-layer perceptron (MLP) blocks.
This structure is inspired by the recent conditional generative models [16,
57], while we have modified their approaches by substituting linear layers for
the convolutional ones. The score model processes a noisy pose
$\tilde{x}_{i}\in\mathfrak{g}$ embedded using a positional encoding. It then
computes an estimated score $s_{\theta}(\tilde{x}_{i},\sigma_{i})$. This
estimated score is subsequently utilized in the denoising process (i.e., Eq.
(7)). Please note that the input and output of the denoising part are
represented in vector forms within the corresponding Lie algebra space.

Regarding the design of the conditioning mechanism in MLPs, a few prior
studies [16, 57] employ scale-bias condition, which is formulated as
$f(x,c)=\mathbf{A}(c)x+\mathbf{B}(c)$. Nevertheless, our empirical
observations suggest that this conditioning mechanism does not perform
satisfactorily when learning distributions on $SO(3)$. This may be
attributable to the limited expressivity of the underlying neural networks.
Inspired by [73, 34], we introduce a modified Fourier-based conditioning
mechanism, which is formulated as follows:

${\footnotesize\begin{split}f_{i}(x,c)=\sum^{d-1}_{j=0}\mathbf{W}_{ij}\left(\mathbf{A}_{j}(c)\cos(\pi
x_{j})+\mathbf{B}_{j}(c)\sin(\pi x_{j})\right),\end{split}}$ (13)

where $d$ represents the dimension of our linear layer. This form bears
similarity to the Fourier series
$f(t)=\sum^{\infty}_{k=0}\mathbf{A}_{k}\cos\left(\frac{2\pi
kt}{P}\right)+\mathbf{B}_{k}\sin\left(\frac{2\pi kt}{P}\right)$. Our
motivation stems from the fact that the pose distribution on SO(3) is
circular, and can therefore be represented as periodic functions. By the
definition of periodic functions, their derivatives are also periodic. It is
worth noting that this conditioning mechanism does not introduce additional
parameters in our neural network design, as $\mathbf{W}_{ij}$ is provided by
the subsequent linear layer. Our experimental findings suggest that this
conditioning scheme enhances the ability of neural network to capture periodic
features of score fields on $SO(3)$.

##  5 Experimental Results

In this section, we demonstrate that our score-based diffusion model can
produce precise pose estimation on both $SO(3)$ and $SE(3)$ compared with
previous probabilistic approaches. In addition, we present our method’s
superior performance on the real-world T-LESS [20] dataset without relying on
reconstructed 3D models or symmetric annotations. Note that, to the best of
our knowledge, our approach is the first probabilistic model that conduct the
experiments on the complete T-LESS dataset and reports the accuracy, in
contrast to previous methods confined to a limited subset of objects. The
extensive evaluation substantiate the robustness and scalability of our score-
based diffusion model.

###  5.1 Experimental Setups

#### SYMSOL.

SYMSOL is a dataset specifically designed for evaluating density estimators in
the $SO(3)$ space. This dataset, first introduced by [41], comprises 250k
images of five texture-less and symmetric objects, with each subject to random
rotations. The objects include tetrahedron (tet.), cube, icosahedron (icosa.),
cone, and cylinder (cyl.), with each exhibiting unique symmetries that
introduce various degrees of pose ambiguity. For this dataset, our score model
is compared in the $SO(3)$ space with several recent works [10, 41, 23, 37].
The baseline models compared with utilize a pre-trained ResNet50 [15] as their
backbones. Note that we report the average angular distances in degrees.

#### SYMSOL-T.

To extend our evaluation into the $SE(3)$ space, we developed the SYMSOL-T
dataset by incorporating random translations based on SYMSOL, which introduces
an additional layer of complexity due to perspective-induced ambiguity.
Similar to SYMSOL, it features the same five symmetric shapes and the same
number of random samples. For SYMSOL-T, we benchmark our proposed methods
against two pose regression methods. These two methods are trained using a
symmetry-aware loss, but with different strategies: one directly estimates the
pose from an image, while the other employs iterative refinement. We report
the average angular distances in degrees for rotation and the average
distances for translation.

#### T-LESS.

T-LESS [20] has been recognized as a challenging benchmark in the BOP
challenge [22], which consists of thirty texture-less industrial objects. The
objects in this dataset are characterized by a range of discrete and
continuous symmetries. In this dataset, the pose ambiguities arise not only
from the intrinsic object symmetries but also the environmental factors such
as occlusion and self-occlusion due to its cluttered settings. The T-LESS
dataset features a training set with 50k physically based rendering (PBR) [22]
images from synthetic images, and an additional 37k images from real-world
scanning. The testing set encompasses 10k real-world scanned images. The
evaluation methods employed in our study include three standard metrics from
the BOP challenge: Maximum Symmetry-Aware Projection Distance (MSPD), Maximum
Symmetry-Aware Surface Distance (MSSD), and Visible Surface Discrepancy (VSD).
To reflect the emphasis of our work on symmetry, we further introduced
symmetry-aware metrics: R@2, R@5, and R@10, which represent predictions with
rotational errors within 2, 5, and 10 degrees, respectively. Similarly, T@2,
T@5, and T@10 are estimations with translational errors within 2, 5, and 10
centimeters, respectively.

#### Visualization

To visualize the density predictions, we adopt the strategy employed in [41]
to represent the rotation densities generated by our model in the $SO(3)$
space. Specifically, we use the Mollweide projection for visualizing the
$SO(3)$ space, with longitude and latitude values representing the yaw and
pitch of the object’s rotation, respectively. The color in the $SO(3)$ space
indicates the roll of the object’s rotation. The circles denote sets of
equivalent poses, with each dot representing a single sample. For each plot,
we generate a total of $1,000$ random samples from our model. For the
translation part, we illustrate the rendered results of the estimated poses
below their original images.

Table 2: Evaluation results on SYMSOL.

Methods | SYMSOL (Spread in degrees $\downarrow$)
---|---
Avg. | tet. | cube | icosa. | cone | cyl.
DBN [10] | 22.44 | 16.70 | 40.70 | 29.50 | 10.10 | 15.20
Implicit-PDF [41] | 3.96 | 4.60 | 4.00 | 8.40 | 1.40 | 1.40
HyperPosePDF [23] | 1.94 | 3.27 | 2.18 | 3.24 | 0.55 | 0.48
Normalizing Flows [37] | 0.70 | 0.60 | 0.60 | 1.10 | 0.50 | 0.50
Ours (ResNet34) | 0.42 | 0.43 | 0.44 | 0.52 | 0.35 | 0.35
Ours (ResNet50) | 0.37 | 0.28 | 0.32 | 0.40 | 0.53 | 0.31

Table 3: Evaluation results on SYMSOL-T.

Methods |  SYMSOL-T (Spread in degrees $\downarrow$)
---|---
| tet. | cube | icosa. | cone | cyl.
| $R$ | $t$ | $R$ | $t$ | $R$ | $t$ | $R$ | $t$ | $R$ | $t$
Regression | 2.92 | 0.064 | 2.86 | 0.05 | 2.46 | 0.037 | 1.84 | 0.058 | 2.24 | 0.049
Iterative regression | 4.25 | 0.048 | 4.2 | 0.037 | 29.33 | 0.026 | 1.63 | 0.037 | 2.34 | 0.032
Ours ($R^{3}SO(3)$) | 1.38 | 0.017 | 1.93 | 0.010 | 29.35 | 0.009 | 1.33 | 0.016 | 0.86 | 0.010
Ours ($SE(3)$) | 0.59 | 0.016 | 0.58 | 0.011 | 0.64 | 0.012 | 0.54 | 0.016 | 0.41 | 0.011

Table 4: Evaluation results on T-LESS (Average of 30 objects).

Methods | T-LESS (Accuracy % $\uparrow$)
---|---
MSPD | MSSD | VSD | R@2 | R@5 | R@10 | T@2 | T@5 | T@10
GDRNPP [64] | 90.17 | 75.06 | 67.60 | 21.60 | 71.18 | 90.56 | 90.31 | 96.09 | 98.10
Ours ($R^{3}SO(3)$) | 85.73 | 52.03 | 48.41 | 27.98 | 72.42 | 89.26 | 60.37 | 79.75 | 89.62
Ours ($SE(3)$) | 93.16 | 60.17 | 56.88 | 47.21 | 86.94 | 94.78 | 71.72 | 92.03 | 97.15

###  5.2 Quantitative Results on SYMSOL

In this section, we present the quantitative results evaluated on SYMSOL, and
compare our diffusion-based methods with non-parametric ones. We assess the
performance of our score model on $SO(3)$ across various shapes using both
ResNet34 and ResNet50 as the backbones, with the results reported in Table 2.
Our model demonstrates promising performance, consistently surpassing the
contemporary non-parametric baseline models. It is observed that our model,
even when based on the less complex ResNet34 backbone, is still able to
achieve results that exceed those of the other baselines using the more
complex ResNet50 backbone. The average angular errors are consistently below
$1$ degree across all shape categories. The performance further improves when
employing ResNet50, which emphasizes the potential robustness and scalability
of using diffusion models for addressing the pose ambiguity problem. However,
it is important to observe that our model with ResNet50 exhibits a slightly
reduced performance for the cone shape compared to the ResNet34 variant. This
discrepancy can be attributed to our practice of training a single model
across all shapes, a strategy that parallels those adopted by Implicit-PDF
[41] and HyperPosePDF [23]. Such an approach may lead to mutual influences
among shapes with diverse pose distributions, and potentially compromise
optimal performance for certain shapes. This observation highlights
opportunities for future improvements to our model, specifically in enhancing
its ability to effectively learn from data spanning various domains. Such
endeavors would potentially shed light on the diverse complexities associated
with distinct shapes and characteristics.

Figure 3: Visualization of our $SE(3)$ diffusion results on SYMSOL-T. Each
plot contains $1,000$ sampled poses generated by our model. The first row
depicts the densities of discrete symmetrical shapes: (a) tetrahedron, (b)
cube, (c) icosahedron, each possessing 12, 24 and 60 discrete symmetries,
respectively. The second row presents the densities of continuous symmetrical
objects: (d) cone and (e) cylinder, with each shape exhibiting 1 and 2
continuous symmetries, respectively. Figure 4: Visualization of our $SE(3)$
diffusion results on T-LESS. In the first row, we present our estimation
results of three objects in cluttered scenes: (a) Object 9, characterized by 2
discrete symmetries; (b) Object 27, featuring 4 discrete symmetries; and (c)
object 14, possessing 1 continuous symmetries. The second row illustrates pose
ambiguities arising from occlusion and self-occlusion, particularly related to
Object 4. Notably, this object is annotated with 1 continuous symmetry by
human annotator, which does not accurately capture the true ambiguities in
certain cases. We explore the scenarios where (d) the object has no symmetry
if the top feature is visible; (e) 2 discrete symmetries when the feature is
self-occluded, but revealing the two screw holes at the bottom; and (f) 1
continuous symmetry if the screw holes are also occluded by the scene. Each
plot contains $1,000$ pose samples from our model. The samples are
concentrated on each mode of the distribution, indicating that our models can
generate precise rotation estimations across different objects.

###  5.3 Quantitative Results on SYMSOL-T

We report the quantitative results obtained from the SYMSOL-T dataset
evaluation, as shown in Table 3. The results reveal that our $SE(3)$ and
$R^{3}SO(3)$ score models outperform the pose regression and iterative
regression baselines in terms of estimation accuracy. However, the
$R^{3}SO(3)$ score model encounters difficulty when learning the distribution
of the icosahedron shape. In contrast, our $SE(3)$ score model excels in
estimating rotation across all shapes and achieves competitive results in
translation compared to the $R^{3}SO(3)$ score model, thus demonstrating its
ability to model the joint distribution of rotation and translation. Please
note that the $SE(3)$ and $R^{3}SO(3)$ score models do not rely on symmetry
annotations, which distinguish them from the pose regression and iterative
regression baselines that leverage symmetry supervision. This supports our
initial hypothesis that score models are capable of addressing the pose
ambiguity problem in the image domain. In the comparison between the
$R^{3}SO(3)$ score model and iterative regression, both models employ
iterative refinement. However, our $R^{3}SO(3)$ score model consistently
outperforms iterative regression on tetrahedron, cube, cone, and cylinder
shapes. The key difference is that iterative regression focuses on minimizing
pose errors without explicitly learning the underlying true distributions. In
contrast, our $R^{3}SO(3)$ score model captures different scales of noise,
enabling it to learn the true distribution of pose uncertainty and achieve
more accurate results. Regarding translation performance, the $R^{3}SO(3)$
score model takes the lead over the $SE(3)$ score model. The former’s
performance can be credited to its assumption of independence between rotation
and translation, which effectively eliminates mutual interference. On the
other hand, the $SE(3)$ score model learns the joint distribution of rotation
and translation, which leads to more robust rotation estimations. The
observations therefore support our hypothesis that $SE(3)$ can provide a more
comprehensive pose estimation than $R^{3}SO(3)$. Fig. 3 show the visualization
derived by our model on the $SE(3)$ group.

Table 5: Inference time (second per sample) across different denoising steps
on the T-LESS dataset.

Methods | Steps | Inference time | FPS | MSPD | MSSD | VSD
---|---|---|---|---|---|---
Ours ($R^{3}SO(3)$) | 100 | 0.041 | 24 | 85.73 | 52.03 | 48.41
50 | 0.021 | 47 | 85.46 | 52.18 | 48.41
10 | 0.005 | 188 | 85.57 | 52.25 | 48.77
5 | 0.003 | 307 | 85.67 | 53.11 | 49.59
Ours ($SE(3)$) | 100 | 0.050 | 20 | 93.16 | 60.17 | 56.88
50 | 0.026 | 38 | 93.00 | 59.96 | 56.64
10 | 0.006 | 161 | 92.79 | 60.35 | 57.08
5 | 0.004 | 250 | 92.40 | 59.30 | 56.15

###  5.4 Quantitative Results on T-LESS

We evaluate our $SE(3)$ diffusion model on T-LESS, and demonstrate the
effectiveness of our approaches in real-world cluttered scenarios. In this
experiment, a single model with a ResNet34 backbone is trained across 30
T-LESS objects. We crop the Region of Interest (RoI) confined within bounding
boxes from RGB images and employ segmentation masks to isolate the visible
parts of objects. To introduce randomness during training while preserving the
RoI aspect ratios, we leverage the Dynamic Zoom-In [36] method. In addition,
we apply hard image augmentations [64] to the RoIs, including random colors,
Gaussian blur, and noise. It is crucial to note that our method assumes the
availability of ground truth bounding boxes and segmentation masks for the
visible parts of objects. Table 4 presents the quantitative results. For
comparison, we include GDRNPP [64], a regression-based method that stands as
the state-of-the-art approach from the BOP challenge in 2022 [59]. The results
indicate that our $SE(3)$ diffusion model outperforms its $R^{3}SO(3)$
counterpart across all metrics. Furthermore, our $SE(3)$ diffusion model
demonstrates superior rotation estimation compared to GDRNPP, albeit with a
slightly inferior performance in translation. This discrepancy is attributed
to GDRNPP’s use of geometry guidance derived from 3D models to enhance depth
estimation. Fig. 4 presents the visualization results. Please note that more
details are presented in the supplementary material.

###  5.5 Inference Time Analysis

To assess the inference time performance of our models, they are evaluated
using the T-LESS dataset and employing JAX [6] as the deep learning package.
Our experiments are conducted on an AMD Ryzen Threadripper 2990WX CPU and an
RTX 2080 Ti GPU. The models, based on the ResNet34 backbone and an input size
of 224 x 224 pixels, demonstrate noticeable efficiency across various
denoising steps when parametrized on the $SE(3)$ and $R^{3}SO(3)$ spaces, as
detailed in Table 5. For $SE(3)$, we achieve up to 250 FPS at minimal
denoising steps, while for $R^{3}SO(3)$, the performance reaches 307 FPS.
These results suggest the practical applicability of our models in real-time
scenarios.

##  6 Conclusion

In this paper, we presented a novel approach that applies diffusion models to
the $SE(3)$ group for object pose estimation, effectively addressing the pose
ambiguity issue. Inspired by the correlation between rotation and translation
distributions caused by image projection effects, we jointly estimated their
distributions on $SE(3)$ for improved accuracy. This is the first work to
apply diffusion models to $SE(3)$ in the image domain. To validate it, we
developed the SYMSOL-T dataset, which enriches the original SYMSOL dataset
with randomly sampled translations. Our experiments confirmed the
applicability of our $SE(3)$ diffusion model in the image domain and the
advantage of $SE(3)$ parametrization over $R^{3}SO(3)$. Moreover, our
experiments on T-LESS exhibits the efficacy of our $SE(3)$ diffusion model in
real-world applications.

##  7 Acknowledgement

The authors gratefully acknowledge the support from the National Science and
Technology Council (NSTC) in Taiwan under grant numbers MOST
111-2223-E-007-004-MY3, Taiwan. The authors would also like to express their
appreciation for the donation of the GPUs from NVIDIA Corporation and NVIDIA
AI Technology Center (NVAITC) used in this work. Furthermore, the authors
extend their gratitude to the National Center for High-Performance Computing
(NCHC) for providing the necessary computational and storage resources.

## References

  * Amini et al. [2022] Arash Amini, Arul Selvam Periyasamy, and Sven Behnke.  Yolopose: Transformer-based multi-object 6d pose estimation using keypoint regression.  In _Intelligent Autonomous Systems (IAS)_ , pages 392–406, 2022.
  * Amit et al. [2021] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf.  Segdiff: Image segmentation with diffusion probabilistic models.  _arXiv preprint arXiv:2112.00390_ , 2021.
  * Baranchuk et al. [2021] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  _arXiv preprint arXiv:2112.03126_ , 2021.
  * Barfoot and Furgale [2014] Timothy D. Barfoot and Paul Timothy Furgale.  Associating uncertainty with three-dimensional poses for use in estimation problems.  _IEEE Trans. Robotics_ , 30:679–693, 2014.
  * Bortoli et al. [2022] Valentin De Bortoli, Emile Mathieu, Michael John Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet.  Riemannian score-based generative modelling.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Bradbury et al. [2018] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018.
  * Chen et al. [2022] Shoufa Chen, Peize Sun, Yibingimp Song, and Ping Luo.  Diffusiondet: Diffusion model for object detection.  _CoRR_ , abs/2211.09788, 2022.
  * Chirikjian and Kobilarov [2014] Gregory Chirikjian and Marin Kobilarov.  Gaussian approximation of non-linear measurement models on lie groups.  In _53rd IEEE Conference on Decision and Control_ , pages 6401–6406. IEEE, 2014.
  * Choi et al. [2022] Jeongjun Choi, Dongseok Shim, and H. Jin Kim.  Diffupose: Monocular 3d human pose estimation via denoising diffusion probabilistic model.  _CoRR_ , abs/2212.02796, 2022.
  * Deng et al. [2020] Haowen Deng, Mai Bui, Nassir Navab, Leonidas Guibas, Slobodan Ilic, and Tolga Birdal.  Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation, 2020.
  * Di et al. [2021] Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, and Federico Tombari.  So-pose: Exploiting self-occlusion for direct 6d pose estimation.  In _Proc. IEEE Int. Conf. on Computer Vision (ICCV)_ , pages 12376–12385, 2021.
  * Gilitschenski et al. [2020] Igor Gilitschenski, Roshni Sahoo, Wilko Schwarting, Alexander Amini, Sertac Karaman, and Daniela Rus.  Deep orientation uncertainty learning based on a bingham loss.  In _International conference on learning representations_ , 2020.
  * Gong et al. [2022] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong.  Diffuseq: Sequence to sequence text generation with diffusion models.  _arXiv preprint arXiv:2210.08933_ , 2022.
  * He et al. [2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016a.
  * He et al. [2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016b.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _Proc. Conf. on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Ho et al. [2022a] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022a.
  * Ho et al. [2022b] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  _arXiv preprint arXiv:2204.03458_ , 2022b.
  * Hodaň et al. [2017] Tomáš Hodaň, Pavel Haluza, Štěpán Obdržálek, Jiří Matas, Manolis Lourakis, and Xenophon Zabulis.  T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects.  _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2017.
  * Hodan et al. [2017] Tomáš Hodan, Pavel Haluza, Štepán Obdržálek, Jiri Matas, Manolis Lourakis, and Xenophon Zabulis.  T-less: An rgb-d dataset for 6d pose estimation of texture-less objects.  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 880–888. IEEE, 2017.
  * Hodan et al. [2020] Tomás Hodan, Dániel Baráth, and Jiri Matas.  EPOS: estimating 6d pose of objects with symmetries.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 11700–11709, 2020.
  * Hodaň et al. [2020] Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas.  Bop challenge 2020 on 6d object localization.  In _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pages 577–594. Springer, 2020.
  * Höfer et al. [2023] Timon Höfer, Benjamin Kiefer, Martin Messmer, and Andreas Zell.  Hyperposepdf-hypernetworks predicting the probability distribution on so (3).  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2369–2379, 2023.
  * Holmquist and Wandt [2022] Karl Holmquist and Bastian Wandt.  Diffpose: Multi-hypothesis human pose estimation using diffusion models.  _arXiv preprint arXiv:2211.16487_ , 2022.
  * Huang et al. [2022a] Lin Huang, Tomas Hodan, Lingni Ma, Linguang Zhang, Luan Tran, Christopher D. Twigg, Po-Chen Wu, Junsong Yuan, Cem Keskin, and Robert Wang.  Neural correspondence field for object pose estimation.  In _Proc. European Conf. on Computer Vision (ECCV)_ , pages 585–603, 2022a.
  * Huang et al. [2022b] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 2595–2605, 2022b.
  * Hyvärinen and Dayan [2005] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Jagvaral et al. [2023] Yesukhei Jagvaral, Francois Lanusse, and Rachel Mandelbaum.  Diffusion generative models on so(3).  https://openreview.net/pdf?id=jHA-yCyBGb, 2023.
  * Jørgensen [1975] Erik Jørgensen.  The central limit problem for geodesic random walks.  _Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete_ , 32(1-2):1–64, 1975.
  * Kingma and Ba [2015] Diederick P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _Proc. Int. Conf. on Learning Representations (ICLR)_ , 2015.
  * Klee et al. [2023] David M Klee, Ondrej Biza, Robert Platt, and Robin Walters.  Image to sphere: Learning equivariant features for efficient pose prediction.  _arXiv preprint arXiv:2302.13926_ , 2023.
  * Labbé et al. [2020] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic.  Cosypose: Consistent multi-view multi-object 6d pose estimation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16_ , pages 574–591. Springer, 2020.
  * Leach et al. [2022] Adam Leach, Sebastian M Schmon, Matteo T. Degiacomi, and Chris G. Willcocks.  Denoising diffusion probabilistic models on so(3) for rotational alignment.  In _Proc. Int. Conf. on Learning Representations Workshop (ICLRW)_ , 2022.
  * Lee et al. [2021] Jiyoung Lee, Wonjae Kim, Daehoon Gwak, and Edward Choi.  Conditional generation of periodic signals with fourier-based decoder.  _arXiv preprint arXiv:2110.12365_ , 2021.
  * Li et al. [2022] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto.  Diffusion-lm improves controllable text generation.  _Advances in Neural Information Processing Systems_ , 35:4328–4343, 2022.
  * Li et al. [2019] Zhigang Li, Gu Wang, and Xiangyang Ji.  CDPN: coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation.  In _Proc. IEEE Int. Conf. on Computer Vision (ICCV)_ , pages 7677–7686, 2019.
  * Liu et al. [2023a] Yulin Liu, Haoran Liu, Yingda Yin, Yang Wang, Baoquan Chen, and He Wang.  Delving into discrete normalizing flows on so(3) manifold for probabilistic rotation modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 21264–21273, 2023a.
  * Liu et al. [2023b] Yulin Liu, Haoran Liu, Yingda Yin, Yang Wang, Baoquan Chen, and He Wang.  Delving into discrete normalizing flows on so (3) manifold for probabilistic rotation modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21264–21273, 2023b.
  * Manhardt et al. [2019] Fabian Manhardt, Diego Martín Arroyo, Christian Rupprecht, Benjamin Busam, Tolga Birdal, Nassir Navab, and Federico Tombari.  Explaining the ambiguity of object detection and 6d pose from visual data.  In _Proc. IEEE Int. Conf. on Computer Vision (ICCV)_ , pages 6840–6849, 2019.
  * Matthies et al. [1988] Siegfried Matthies, J Muller, GW Vinel, et al.  On the normal distribution in the orientation space.  _Texture, Stress, and Microstructure_ , 10:77–96, 1988.
  * Murphy et al. [2021] Kieran A. Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia.  Implicit-pdf: Non-parametric representation of probability distributionson the rotation manifold.  In _Proc. Int. Conf. on Machine Learning (ICML)_ , pages 7882–7893, 2021.
  * Nikolayev and Savyolov [1970] Dmitry I Nikolayev and Tatjana I Savyolov.  Normal distribution on the rotation group so (3).  _Textures and Microstructures_ , 29, 1970.
  * Okorn et al. [2020] Brian Okorn, Mengyun Xu, Martial Hebert, and David Held.  Learning orientation distributions for object pose estimation.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 10580–10587. IEEE, 2020.
  * Park et al. [2019] Kiru Park, Timothy Patten, and Markus Vincze.  Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation.  In _Proc. IEEE Int. Conf. on Computer Vision (ICCV)_ , pages 7667–7676, 2019.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Peng et al. [2019] Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao.  Pvnet: Pixel-wise voting network for 6dof pose estimation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 4561–4570, 2019.
  * Prokudin et al. [2018] Sergey Prokudin, Peter Gehler, and Sebastian Nowozin.  Deep directional statistics: Pose estimation with uncertainty quantification.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 534–551, 2018.
  * Rad and Lepetit [2017] Mahdi Rad and Vincent Lepetit.  BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth.  In _Proc. IEEE Int. Conf. on Computer Vision (ICCV)_ , pages 3848–3856, 2017.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rezende et al. [2020] Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer.  Normalizing flows on tori and spheres.  In _International Conference on Machine Learning_ , pages 8083–8092. PMLR, 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2022] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  _arXiv preprint arXiv:2208.12242_ , 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Said et al. [2017] Salem Said, Lionel Bombrun, Yannick Berthoumieu, and Jonathan H. Manton.  Riemannian gaussian distributions on the space of symmetric positive definite matrices.  _IEEE Trans. Inf. Theory_ , 63(4):2153–2170, 2017.
  * Solà et al. [2018] Joan Solà, Jérémie Deray, and Dinesh Atchuthan.  A micro lie theory for state estimation in robotics.  _CoRR_ , abs/1812.01537, 2018.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _Proc. Int. Conf. on Learning Representations (ICLR)_ , 2021a.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  In _Proc. Conf. on Neural Information Processing Systems (NeurIPS)_ , pages 11895–11907, 2019.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _Proc. Int. Conf. on Learning Representations (ICLR)_ , 2021b.
  * Sundermeyer et al. [2023] Martin Sundermeyer, Tomáš Hodaň, Yann Labbe, Gu Wang, Eric Brachmann, Bertram Drost, Carsten Rother, and Jiří Matas.  Bop challenge 2022 on detection, segmentation and pose estimation of specific rigid objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2784–2793, 2023.
  * Thalhammer et al. [2023] Stefan Thalhammer, Timothy Patten, and Markus Vincze.  COPE: end-to-end trainable constant runtime object pose estimation.  In _Proc. IEEE Winter Conf. on Applications of Computer Vision (WACV)_ , pages 2859–2869, 2023.
  * Urain et al. [2022] Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki.  Se(3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion.  _CoRR_ , abs/2209.03855, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vincent [2011] Pascal Vincent.  A connection between score matching and denoising autoencoders.  _Neural Comput._ , 23(7):1661–1674, 2011.
  * Wang et al. [2021] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji.  Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 16611–16621, 2021.
  * Wang et al. [2019] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2642–2651, 2019.
  * Xiang et al. [2018] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  In _Robotics: Science and Systems XIV_ , 2018.
  * Yang et al. [2023] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu.  Diffsound: Discrete diffusion model for text-to-sound generation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * Yang et al. [2022a] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  _arXiv preprint arXiv:2209.00796_ , 2022a.
  * Yang et al. [2022b] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion probabilistic modeling for video generation.  _arXiv preprint arXiv:2203.09481_ , 2022b.
  * Yi et al. [2021] Brent Yi, Michelle Lee, Alina Kloss, Roberto Martín-Martín, and Jeannette Bohg.  Differentiable factor graph optimization for learning smoothers.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021.
  * Yim et al. [2023] Jason Yim, Brian L. Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi S. Jaakkola.  SE(3) diffusion model with application to protein backbone generation.  _CoRR_ , abs/2302.02277, 2023.
  * Yin et al. [2023] Yingda Yin, Yang Wang, He Wang, and Baoquan Chen.  A laplace-inspired distribution on SO(3) for probabilistic rotation estimation.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Ziyin et al. [2020] Liu Ziyin, Tilman Hartwig, and Masahito Ueda.  Neural networks fail to learn periodic functions and how to fix it.  _Advances in Neural Information Processing Systems_ , 33:1583–1594, 2020.

Supplementary Material

##  8 Ablation Studies

Figure 5: Visualizing pose ambiguity caused by image perspective. The
rotations between the four cubes differ by an angle of 15 degrees. Figure 6:
The distribution of angular errors of the $SE(3)$ and $R^{3}SO(3)$ score
models with three configurations and four shapes, in which the width
represents the density of data points at a particular range. Please note that
the results of $R^{3}SO(3)$ on icosa. are not reported as this model fails to
adequately handle this particular shape.

###  8.1 Analysis of $SE(3)$ and $R^{3}SO(3)$ in the Presence of Image
Perspective Ambiguity

In the realm of pose estimation, the effect of image perspective present a
notable challenge. It intertwines rotation and translation in the image space,
leading to the phenomenon of pose ambiguity. Fig. 5 exemplifies this through
four cubes, each of which appears similarly oriented but actually differs in
rotation degrees, complicating model predictions for accurate rotation angles.
The parametrizations of $R^{3}SO(3)$ and $SE(3)$ offer different approaches to
dealing with this problem. Specifically, $R^{3}SO(3)$ does not factor in the
relationship between rotation and translation, whereas $SE(3)$ actively
incorporates it into its structure. As a result, it is reasonable to
hypothesize that $SE(3)$ might be more capable of mitigating performance
degradation stemming from the image perspective effect. This potential
advantage of $SE(3)$, further elaborated in Section 2.2.

To delve deeper into the effects of image perspective on our pose estimation
methods, we additionally synthesized three variants of the SYMSOL-T dataset:
Uniform, Edge, and Centered. The Uniform variant consists of uniformly sampled
translations, the Edge variant includes translations at the maximum distance
from the center, and the Centered variant comprises zero translations. Fig. 6
showcases a comparison of the evaluation results for these three variants. We
present the distributions of angular errors made by the $SE(3)$ and
$R^{3}SO(3)$ diffusion models on these dataset variants and four shapes:
tetrahedron, cube, cone, and cylinder. These distributions of angular errors
depict the uncertainty of the pose estimations. In line with our hypothesis,
the Edge variant, which is most influenced by image perspective, exhibits
greater uncertainty compared to the Centered variant. The Uniform variant
situates itself between these two. It is evident that both the $R^{3}SO3$ and
$SE(3)$ score models demonstrate higher uncertainty on the Edge dataset across
all shapes, with reduced uncertainty on the Centered dataset. The $SE(3)$
score model demonstrates an impressive ability to counter the pose ambiguity
introduced by image perspective, a capability that becomes evident when
compared with the $R^{3}SO(3)$ score model. The observation therefore confirms
our hypothesis that $SE(3)$ does exhibit greater robustness to the ambiguity
caused by the image perspective issue.

Table 6: Evaluation results for various denoising steps applied to score
models on $SE(3)$, trained using automatic differentiation and surrogate
scores.

Methods | Steps | SYMSOL-T (Spread in degrees $\downarrow$)
---|---|---
tet. | cube | icosa. | cone | cyl.
$R$ | $t$ | $R$ | $t$ | $R$ | $t$ | $R$ | $t$ | $R$ | $t$
$SE(3)$-autograd | 100 | 0.60 | 0.019 | 0.59 | 0.012 | 0.67 | 0.012 | 0.58 | 0.018 | 0.41 | 0.012
50 | 0.61 | 0.019 | 0.61 | 0.013 | 0.66 | 0.013 | 0.58 | 0.019 | 0.41 | 0.013
10 | 2.89 | 0.102 | 3.21 | 0.113 | 3.24 | 0.113 | 3.12 | 0.104 | 3.16 | 0.108
5 | 12.93 | 0.418 | 13.07 | 0.407 | 10.33 | 0.302 | 10.83 | 0.377 | 10.09 | 0.345
$SE(3)$-surrogate (Ours) | 100 | 0.59 | 0.016 | 0.58 | 0.011 | 0.64 | 0.012 | 0.55 | 0.016 | 0.41 | 0.011
50 | 0.56 | 0.017 | 0.58 | 0.011 | 0.65 | 0.012 | 0.54 | 0.017 | 0.41 | 0.011
10 | 0.63 | 0.017 | 0.70 | 0.012 | 1.71 | 0.015 | 0.56 | 0.019 | 0.43 | 0.014
5 | 1.22 | 0.024 | 2.00 | 0.028 | 5.31 | 0.048 | 0.72 | 0.035 | 0.62 | 0.031

###  8.2 Performance Analysis: Surrogate Score versus Automatically
Differentiated True Score

To evaluate our hypothesis concerning convergence speed, we compare two
versions of our score model. The first version, termed $SE(3)$-surrogate, is
trained with the surrogate score described in Eq. (12). The second version,
termed as $SE(3)$-autograd, is trained with the true score described in Eq.
(5) and calculated by automatic differentiation as described in Section 9.2.
We trained both estimators and evaluated their performance using different
steps of denoising process. The results are reported in Table 6. Our findings
show that when a larger number of denoising steps (e.g., $100$ steps) are
used, both score models produce comparable results. However, the performance
of $SE(3)$-autograd significantly declines in comparison to $SE(3)$-surrogate
when the number of sampling steps decreases from $50$ to $10$ and then to $5$.
This performance drop is due to the curved manifold represented by the $SE(3)$
parametrization, which can result in the score vector not consistently
pointing towards the noise-free data. These results substantiate our
hypothesis, and suggest that the application of the surrogate score can lead
to faster convergence than the use of the true score calculated through
automatic differentiation.

Table 7: Comparison with other diffusion-based approaches.

Methods | Distribution | Loss | SYMSOL (Spread in degrees $\downarrow$)
---|---|---|---
| Avg. | tet. | cube | icosa. | cone | cyl.
Leach _et al_. [33] | $\mathcal{IG}_{SO(3)}$ | DDPM | 0.63 | 0.59 | 0.65 | 0.75 | 0.73 | 0.41
Jagvaral _et al_. [28] | $\mathcal{IG}_{SO(3)}$ | MLE | 30.45 | 12.21 | 15.18 | 28.76 | 86.77 | 9.35
Ours w/o fourier | $\mathcal{IG}_{SO(3)}$ | DSM | 1.18 | 0.52 | 0.77 | 3.97 | 0.32 | 0.32
Ours w/o fourier | $\mathcal{N}_{SO(3)}$ | DSM | 0.51 | 0.50 | 0.46 | 0.91 | 0.33 | 0.34
Ours | $\mathcal{N}_{SO(3)}$ | DSM | 0.42 | 0.43 | 0.44 | 0.52 | 0.35 | 0.35

###  8.3 Comparison of Diffusion Models on $SO(3)$

In this experiment, we further compare our $SO(3)$ score model with the
diffusion models proposed by [33] and [28] using the SYMSOL dataset. While
these studies do not specifically address object pose estimation, we have
adapted their methods to fit within our framework. The authors of [33] extend
the DDPM [16] to $SO(3)$ using an analogy approach. They employ an $SO(3)$
variant of DDPM loss during the training process. On the another hand, the
authors of [28] reformulate the SGM [57] to apply it to the $SO(3)$ space and
proposed to train with maximum log-likelihood loss (MLE). The results of these
comparisons are presented in Table 7. Our analysis shows that the models
employing DDPM or Denoising Score Matching (DSM) losses can learn the
distributions on $SO(3)$ effectively, while the model employing MLE loss
fails. When comparing our score models with different distributions, we can
observe that the one with $\mathcal{N}_{SO(3)}$ performs better than it
$\mathcal{IG}_{SO(3)}$ counterpart. Furthermore, when incorporating the
Fourier-based conditioning descreibed in Section 4.4, our score model can
achieve the best performance on SYMSOL. This suggests that Fourier-based
conditioning enhances our models ability to learn pose distributions.

Table 8: Evaluation results on T-LESS (30 objects).

Objects | T-LESS (Accuracy % $\uparrow$)
---|---
MSPD | MSSD | VSD | R@2 | R@5 | R@10 | T@2 | T@5 | T@10
1 | 90.05 | 32.29 | 29.60 | 38.22 | 78.20 | 89.10 | 40.78 | 72.14 | 89.50
2 | 92.22 | 35.56 | 31.73 | 48.07 | 85.49 | 92.97 | 42.63 | 73.92 | 91.61
3 | 97.55 | 47.29 | 43.88 | 52.86 | 92.45 | 98.70 | 60.42 | 90.10 | 96.88
4 | 92.27 | 48.84 | 46.07 | 44.28 | 86.36 | 93.43 | 52.86 | 85.52 | 95.12
5 | 96.32 | 76.47 | 74.18 | 49.47 | 91.58 | 96.84 | 81.05 | 95.79 | 98.95
6 | 98.57 | 78.06 | 75.71 | 60.20 | 92.86 | 97.96 | 84.69 | 95.92 | 97.96
7 | 93.96 | 85.44 | 80.50 | 54.80 | 94.00 | 98.00 | 80.80 | 95.60 | 99.60
8 | 90.40 | 86.53 | 79.49 | 44.67 | 93.33 | 98.00 | 70.00 | 96.00 | 98.00
9 | 96.54 | 84.15 | 79.46 | 47.15 | 93.09 | 97.97 | 82.93 | 97.56 | 99.59
10 | 98.39 | 68.88 | 63.35 | 50.35 | 90.91 | 99.30 | 72.03 | 95.10 | 99.30
11 | 95.20 | 57.26 | 51.52 | 25.14 | 77.71 | 91.43 | 68.00 | 93.14 | 98.86
12 | 96.76 | 62.23 | 56.47 | 38.85 | 87.05 | 95.68 | 64.75 | 93.53 | 97.12
13 | 99.36 | 47.79 | 44.89 | 70.00 | 96.43 | 100.00 | 62.86 | 91.43 | 99.29
14 | 97.60 | 63.36 | 60.05 | 71.92 | 95.21 | 98.63 | 71.92 | 94.52 | 97.26
15 | 97.95 | 59.93 | 57.72 | 73.97 | 97.95 | 98.63 | 69.86 | 93.15 | 98.63
16 | 97.34 | 61.81 | 59.40 | 67.02 | 96.28 | 97.87 | 76.06 | 92.55 | 97.87
17 | 98.56 | 82.19 | 78.47 | 78.08 | 98.63 | 100.00 | 85.62 | 96.58 | 97.26
18 | 83.42 | 72.33 | 75.22 | 16.44 | 59.59 | 78.77 | 82.19 | 93.84 | 95.21
19 | 94.03 | 64.71 | 60.83 | 28.80 | 79.58 | 94.76 | 70.16 | 92.67 | 97.91
20 | 88.71 | 61.62 | 54.42 | 22.92 | 70.83 | 92.08 | 65.00 | 90.00 | 97.92
21 | 80.06 | 58.00 | 56.74 | 37.71 | 72.57 | 77.71 | 68.57 | 84.57 | 90.86
22 | 83.94 | 59.20 | 58.82 | 29.26 | 72.34 | 84.57 | 70.21 | 90.96 | 96.28
23 | 92.58 | 78.06 | 73.75 | 25.00 | 78.63 | 94.76 | 72.98 | 96.77 | 98.39
24 | 96.98 | 62.29 | 59.27 | 56.77 | 95.31 | 97.40 | 65.10 | 92.71 | 98.96
25 | 94.84 | 74.84 | 71.48 | 48.42 | 91.58 | 97.89 | 78.95 | 95.79 | 97.89
26 | 97.17 | 81.41 | 78.73 | 49.49 | 97.98 | 98.99 | 90.91 | 96.97 | 98.99
27 | 89.69 | 79.90 | 75.27 | 33.33 | 81.25 | 94.79 | 82.29 | 93.75 | 97.92
28 | 88.12 | 73.12 | 72.58 | 39.58 | 78.65 | 90.62 | 75.52 | 91.15 | 95.31
29 | 95.82 | 84.90 | 83.78 | 53.06 | 90.82 | 97.96 | 84.69 | 96.94 | 98.98
30 | 97.85 | 69.86 | 67.50 | 60.42 | 91.67 | 98.61 | 77.78 | 92.36 | 97.22
Avg(30) | 93.16 | 60.17 | 56.88 | 47.21 | 86.94 | 94.78 | 71.72 | 92.03 | 97.15

###  8.4 Full Evaluation Results on T-LESS

Table 8 presents the evaluation results of our $SE(3)$ diffusion model on each
T-LESS object. Please note that a single model with ResNet34 backbone is
trained across thirty T-LESS objects. More visualization results are presented
in Fig. 8.

Table 9: Evaluation results on T-LESS (Average of 30 objects).

Methods | T-LESS (Accuracy % $\uparrow$)
---|---
MSPD | MSSD | VSD | R@2 | R@5 | R@10 | T@2 | T@5 | T@10
GDRNPP [64] | 90.17 | 75.06 | 67.60 | 21.60 | 71.18 | 90.56 | 90.31 | 96.09 | 98.10
Ours ($R^{3}SO(3)$) | 85.73 | 52.03 | 48.41 | 27.98 | 72.42 | 89.26 | 60.37 | 79.75 | 89.62
Ours ($SE(3)$) | 93.16 | 60.17 | 56.88 | 47.21 | 86.94 | 94.78 | 71.72 | 92.03 | 97.15
| x@2 | x@5 | x@10 | y@2 | y@5 | y@10 | z@2 | z@5 | z@10
GDRNPP [64] | 98.12 | 98.84 | 99.47 | 98.56 | 99.35 | 99.59 | 91.21 | 96.67 | 98.56
Ours ($R^{3}SO(3)$) | 98.00 | 99.66 | 99.92 | 96.46 | 99.82 | 99.99 | 61.68 | 80.23 | 89.94
Ours ($SE(3)$) | 99.20 | 99.63 | 99.88 | 99.19 | 99.81 | 99.99 | 73.33 | 92.51 | 97.33

###  8.5 Translation Analysis on T-LESS

In this section, we further analyze the error sources of our $SE(3)$ diffusion
model and GDRNPP [64]. The translation accuracies on $x$, $y$ and $z$ axes are
reported in Table 9. It can be observed that the $SE(3)$ diffusion model is
able to predict the $x$ and $y$ translations as accurate as GDRNPP. However,
the $SE(3)$ diffusion model exhibits a slightly less effective performance in
predicting the depth value $z$ compared to GDRNPP. This is because GDRNPP
employs geometry guidance [64] by the reconstructed 3D models of the objects
to enhance depth estimation, while our $SE(3)$ diffusion model exclusively
depends on RGB inputs and ground truth poses for supervision. Nevertheless,
these results still highlight the significant potential of our diffusion
models to compete with contemporary state-of-the-art methods on the real-world
datasets.

###  8.6 Failure Analysis on T-LESS

The failure cases are provided in Fig. 8. In Fig. 8 (a), our approach predicts
the pose as exhibiting one continuous symmetry. However, in reality, there
should be only six discrete symmetries. This presents a failure case arising
from the objective of probabilistic modeling, which aims to approximate the
distribution across the entire space. Our assumption regarding the possible
reasons is twofold: (a) we fit one model to multiple objects, which may have
difficulty representing and learning all the distributions accurately, as they
may interfere with each other; (b) another limitation of our diffusion-based
approach is its reliance on a sufficient volume of data samples. Without
these, it could fail to accurately model the correct distribution of poses.

##  9 Additional implementation Details

###  9.1 Isotropic Gaussian on $SO(3)$

Isotropic Gaussian on $SO(3)$ [42], denoted as $\mathcal{IG}_{SO(3)}$, is a
heat kernel that can be used to model the distribution on $SO(3)$ rotation
space, which has the following form:

${\footnotesize\begin{split}f_{\epsilon}(\phi)=\lim_{N\to\infty}\sum^{N}_{\ell=0}(2\ell+1)e^{-\epsilon\ell(\ell+1)}\frac{\sin((2\ell+1)\phi/2)}{\sin(\phi/2)},\end{split}}$
(14)

where $\phi\in[0,\pi]$ is the rotation angle and $\epsilon>0$ is the
concentration parameter. Note that a normalizing factor
$Z(\phi)=(1-\cos(\phi))/\pi$ is applied to this distribution. For an
$\epsilon\ll 1$, this infinite series converge slowly and could lead to
inefficient computation. In the previous literature, the authors in [71]
proposed to truncate the series by letting $N=2000$, while the authors in [28]
attempted to use another closed-form approximation, expressed as follows:

${\footnotesize\begin{split}f_{\epsilon}(\phi)&\approx\sqrt{\pi}\epsilon^{-\frac{3}{2}}e^{\frac{\epsilon}{4}-\frac{(\phi/2)^{2}}{\epsilon}}\\\
&\cdot\left(\frac{\phi-e^{-\frac{\pi^{2}}{\epsilon}}\left((\phi-2\pi)e^{\frac{\pi\phi}{\epsilon}}+(\phi+2\pi)e^{-\frac{\pi\phi}{\epsilon}}\right)}{2\sin(\phi/2)}\right).\end{split}}$
(15)

As shown in [40], this approximation closely aligns with Eq. (14) when
$\epsilon<1$. To draw samples from this distribution, a common approach is to
utilize the inverse transform sampling. The steps are described as follows.
First, a sample is drawn from a uniform distribution within $[0,\pi]$.
Subsequently, the cumulative distribution function (CDF) of
$\mathcal{IG}_{SO(3)}$ is calculated for inverse sampling. The sampling
procedure is described in Listing LABEL:lst:igso3.

Unfortunately, $\mathcal{IG}_{SO(3)}$ still exists several drawbacks. The main
concern is the intractability of the inverse CDF for $\mathcal{IG}_{SO(3)}$,
which necessitates interpolation in the calculation of inverse sampling.
Moreover, numerical instability could arise during the inverse sampling when
$\epsilon$ is close to zero. As a result, this distribution is not suitable
for applications that require precise computations. Therefore, the proposed
method opt to utilize an alternative distribution to enhance performance and
reliability.

⬇

from math import pi

from jaxlie import SO3

import jax

import jax.numpy as jnp

def normalize(v):

return v / jnp.linalg.norm(v)

def rsub(y:SO3, x:SO3):

return (x.inverse() @ y).log()

# geodesic distance

def geodesic(y:SO3, x:SO3):

return jnp.linalg.norm( rsub(y, x) )

# Ep. (15)

def f_igso3(phi, scale):

eps = scale ** 2

return 0.5 * jnp.sqrt(jnp.pi) * (eps**-1.5) \

* jnp.exp((eps-(phi**2/eps))/4) / jnp.sin(phi/2) \

* (phi-((phi-2*pi)*(jnp.exp(pi*(phi-pi)/eps))) \

+ (phi+2*pi)*(jnp.exp(-pi*(phi+pi)/eps))))

def cdf(steps=1024):

x = jnp.linspace(0.0, 1.0, steps) * pi

y = (1-jnp.cos(x)) / pi * f_igso3(x)

y = jnp.cumsum(y) * pi / steps

return y / y.max(), x

# Inverse transform sampling

def sample(seed):

y, x = cdf_igso3()

key1, key2 = jax.random.split(seed, 2)

rnd = jax.random.uniform((), key=key1)

ang = jnp.interp(rnd, y, x)

axis = jnp.random.normal((3,), key=key2)

axis = normalize(axis)

tan = ang[…, jnp.newaxis] * axis

return SO3.exp(tan)

# log-likelihood IG_SO3

def log_prob(x:SO3, mu:SO3, scale):

phi = geodesic(mu, x)

prob = f_igso3(phi, scale)

return jnp.log(prob)

Listing 1: Isotropic Gaussian $SO(3)$ in JAX.

###  9.2 Concentrated Gaussian on $SO(3)$

Concentrated Gaussian distribution [8, 4] is a distribution that used for
modeling the density on Lie groups. We denote such distribution as
$\mathcal{N}_{\mathcal{G}}$, where $\mathcal{G}$ implies specifically applying
it on Lie group $\mathcal{G}$. This distribution usually assumes that the
noises $z\sim\mathcal{N}(\mathbf{0},\Sigma)$ are relatively small compared to
the domain of the distribution and concentrated around zero in the
corresponding vector space. By the definition of multivariate Gaussian
distribution, the probability density of $z\in\mathbb{R}^{\kappa}$ is
described as follows:

${\footnotesize\begin{split}p_{\Sigma}(z)&:=\mathcal{N}(\mathbf{0},\Sigma)\triangleq\frac{1}{\sqrt{(2\pi)^{\kappa}|\Sigma|}}\exp\left(-\frac{1}{2}z^{\top}\Sigma^{-1}z\right),\end{split}}$
(16)

where $\Sigma\in\mathbb{R}^{\kappa\times\kappa}$ is the covariance matrix.
Assuming that $X,Y\in\mathcal{G}$ and $z\in\mathfrak{g}$, and given the
relation $Y=X\text{Exp}(z)$, the inverse relation can be expressed as
$z=\text{Log}(X^{-1}Y)$. Substituting this into Eq. (16) results in a
concentrated Gaussian on $\mathcal{G}$ centered at $X$. This result
corresponds to Eq. (3) in our main paper and can be expressed as follows:

${\scriptsize\begin{split}p_{\Sigma}(Y|X)&:=\mathcal{N}_{\mathcal{G}}(Y;X,\Sigma)\\\
&\triangleq\frac{1}{\zeta(\Sigma)}\exp\left(-\frac{1}{2}\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\text{Log}(X^{-1}Y)\right),\end{split}}$
(17)

where $\zeta(\Sigma)$ is the normalizing factor. To draw samples from this
distribution, it is accomplished by first drawing a random variable from the
normal distribution $z\sim\mathcal{N}(\mathbf{0},\Sigma)$. Subsequently, $z$
is applied to the center parameter $X$ to yield $Y=X\text{Exp}(z)$. The
sampling procedure is detailed in Listing LABEL:lst:gauss. The primary
advantage of this distribution is its elimination of the need for
approximation and inverse sampling. Due to its simplicity, this method has
been extensively utilized in prior literature for modeling the distribution on
$SO(3)$ [8], $SE(3)$ [4, 61] and manifolds [54].

⬇

from math import pi

from jaxlie import SO3

import jax

import jax.numpy as jnp

def sample(seed, scale):

tan = jax.random.normal(shape=n+(3,), key=seed)

tan = tan * scale

return SO3.exp(tan)

# log-likelihood concentrated Gaussian

def log_prob(x:SO3, mu:SO3, scale):

var = (scale ** 2)

log_sc = jnp.log(scale)

nm = jnp.log(jnp.sqrt(2 * pi))

z = rsub(mu, x)

return -((z ** 2) / (2 * var) - log_sc - nm).sum()

Listing 2: Concentrated Gaussian $SO(3)$ in JAX.

⬇

from jaxlie import SO3, SE3

import jax

import jax.numpy as jnp

Lie = SO3 # Specify Lie groups

# Eq. (25)

def calc_score(y, x, sigma=1.0):

return jax.grad(

lambda tau: log_prob(

Lie.exp(y) @ Lie.exp(tau),

Lie.exp(x),

sigma

)

)(jnp.zeros(Lie.tangent_dim))

# tangent_dim=3 for SO3, 6 for SE3

Listing 3: Calculation of Stein scores using automatic differentiation.

###  9.3 Calculation of Stein Scores Using Automatic Differentiation in JAX

As stated by [28], the Stein scores can be computed as follows:

$\nabla_{Y}\log p_{\Sigma}(Y|X)=\left.\frac{\partial}{\partial k}\log
p_{\Sigma}(Y\text{Exp}(k\tau)|X)\right|_{k=0},$ (18)

where $k\in\mathbb{R}$, $\tau\in\mathfrak{g}$, and $k\tau$ indicates a small
perturbation on $\mathcal{G}$. In practice, this can be computed by automatic
differentiation. Listing LABEL:lst:true_score demonstrates our implementation
based on JAX [6] and jaxlie [70].

Require: $s_{\boldsymbol{\theta}}$, $\\{\sigma_{i}\\}^{L}_{i=0}$,
$p_{\text{data}}$

for  _$j\in\\{0,\dots,N_{\text{iter}}-1\\}$_ do

$i\sim\mathcal{U}(0,L-1)$

$X\sim p_{\text{data}}(X)$

$\tilde{X}=X\text{Exp}(z),~{}z\sim\mathcal{N}(0,\sigma^{2}_{i}I)$

$\ell_{\theta}=\|s_{\mathbf{\theta}}(\tilde{X},\sigma_{i})-\tilde{s}_{X}(\tilde{X},\sigma_{i})\|^{2}_{2}$

$\theta\leftarrow\text{optimize}(\theta,\ell_{\theta})$

end for

Algorithm 1 Training a Score Model using Denoising Score Matching on
$\mathcal{G}$

Require: $s_{\boldsymbol{\theta}}$, $\\{\sigma_{i}\\}^{L}_{i=0}$,
$\\{\epsilon_{i}\\}^{L}_{i=0}$, $\tilde{X}_{0}$

for  _$i\in\\{0,\dots,L-1\\}$_ do

$z_{i}\sim\mathcal{N}(0,I)$

$\tilde{X}_{i+1}=\tilde{X}_{i}\text{Exp}(\epsilon_{i}s_{\theta}(\tilde{X}_{i},\sigma_{i})+\sqrt{2\epsilon_{i}}z^{m}_{i})$

end for

return _$\tilde{X}_{L}$_

Algorithm 2 Sampling Through Geodesic Random Walk on $\mathcal{G}$

###  9.4 Algorithms

The algorithms used for our training and sampling procedures are presented in
Algorithms 1 and 2, respectively. The notations employed conform to those
detailed in the main manuscript.

###  9.5 Datasets

The SYMSOL-T dataset contains 250k images of five symmetric, texture-less
three-dimensional objects. Following the structure of SYMSOL [41], each shape
has 45k training images and 5k testing images. The dataset ensures that
translations over the $x$, $y$, and $z$ axes are uniformly sampled within the
range of $[-1,1]$. In the experiments examining image perspective ambiguity in
Section 8.1, each of the dataset variants (i.e., Uniform, Edge, and Centered)
comprises 200 images per shape. Our analysis is performed based on 1k randomly
generated poses from our score models for each image.

###  9.6 Hyperparameters

In our experiments, we utilize a pre-trained ResNet34 model [15] as the
standard backbone across all methods, unless explicitly stated otherwise.
During training, we sample a batch containing 16 images and the corresponding
ground truth poses in each iteration. Each of these samples is perturbed to
generate 256 random poses, resulting in 4,096 noisy samples. The proposed
score-based model is then trained for 400k steps to denoise these samples. In
the SYMSOL-T experiments, the pose regression approach is trained for 400k
steps. Meanwhile, the iterative regression and both our $R^{3}SO(3)$ and
$SE(3)$ score models are subjected to an extended training duration of 800k
steps. In the T-LESS experiments, the size of the batch is increased to 32.
The score-based model is trained for 400k steps. We employ the Adam optimizer
[30] with an initial learning rate set at $10^{-4}$. During the latter half of
the training schedule, we apply an exponential decay, which lowers the
learning rate to $10^{-5}$. For the diffusion process, we use a linear noise
scheduling approach that ranges from $10^{-4}$ to $1.0$, divided into 100
discrete steps.

Table 10: Hyperparameters.

Hyperparameters | SYMSOL | SYMSOL-T | T-LESS
---|---|---|---
Learning rate | $\left[10^{-4},10^{-5}\right]$ | $\left[10^{-4},10^{-5}\right]$ | $\left[10^{-4},10^{-5}\right]$
Batch size | 16 | 16 | 32
Number of noisy samples | 256 | 256 | 256
Training steps | 400k | 800k | 400k
Optimizer | Adam | Adam | Adam
Noise scale | $\left[10^{-4},1.0\right]$ | $\left[10^{-4},1.0\right]$ | $\left[10^{-4},1.0\right]$
Denoising steps | 100 | 100 | 100
Number of MLP blocks | 1 | 1 | 1

###  9.7 Evaluation Metrics

In the SYMSOL experiments, we adopt the minimum angular distance, measured in
degrees, between a set of ground truth equivalent rotations and the estimated
rotations as the evaluation metric. For the SYMSOL-T experiments, we
incorporate the Euclidean distance between the ground truth and the estimated
translations as our metric to evaluate the accuracy of translation. Each of
these distance metrics is computed per sample, and we report their averages
over all samples in our results. In the T-LESS experiments, we adopt three
standard metrics used in the BOP challenge [22]: Maximum Symmetry-Aware
Projection Distance (MSPD), Maximum Symmetry-Aware Surface Distance (MSSD),
and Visible Surface Discrepancy (VSD).

###  9.8 Visualization of SYMSOL-T Results

Figure 7: Visualization of our SYMSOL-T results. Please refer to Section 9.8
for the detailed descriptions.

In Fig. 7, we present the SYMSOL-T results obtained from our $SE(3)$ diffusion
model for each shape. The model predictions are displayed in green and
correlate to the corresponding original input images that are illustrated in
gray. Our visualization strategy is described in Section 5.1. For each plot,
we generate a total of $1,000$ random samples from our model. Please note that
both the cone and the cylinder exhibit continuous symmetries. This causes the
circles on $SO(3)$ to overlap densely and connect, which gives rise to tilde
shapes on the sphere. In the case of $\mathbb{R}^{3}$, a single circle is
present due to the unique solution for the translation. The samples generated
from our score model are tightly concentrated in the center of each circle.
This evidence highlights the capability of our model to accurately capture
equivalent object poses originating from either discrete or continuous
symmetries.

Figure 8: Visualization of our $SE(3)$ diffusion results on T-LESS.

##  10 Proofs

###  10.1 Closed-Form of Stein Scores

In this section, we present the derivation of the closed-form solution for the
Stein scores. We begin with a revisitation of the Gaussian distribution on the
Lie group $\mathcal{G}$, which is formulated as follows:

$\displaystyle p_{\Sigma}(Y|X)$
$\displaystyle:=\mathcal{N}_{\mathcal{G}}(Y;X,\Sigma)$ (19)
$\displaystyle\triangleq\frac{1}{\zeta(\Sigma)}\text{exp}\left(-\frac{1}{2}\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\text{Log}(X^{-1}Y)\right).$

To derive Eq. (4), we utilize the definition of Stein scores, which is defined
as the derivative of log-density of the data distribution with respect to the
group element $Y\in\mathcal{G}$, expressed as follows:

$\displaystyle\nabla_{Y}$ $\displaystyle\log p_{\Sigma}(Y|X)^{\top}$ (20)
$\displaystyle=\frac{\partial}{\partial
Y}\left(-\frac{1}{2}\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\text{Log}(X^{-1}Y)\right)$
$\displaystyle=\frac{\partial}{\partial\text{Log}(X^{-1}Y)}\left(-\frac{1}{2}\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\text{Log}(X^{-1}Y)\right)\frac{\partial\text{Log}(X^{-1}Y)}{\partial
Y}$
$\displaystyle=-\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\left(\frac{\partial\text{Log}(X^{-1}Y)}{\partial(X^{-1}Y)}\cdot\frac{\partial(X^{-1}Y)}{\partial
Y}\right)$
$\displaystyle=-\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\left(\mathbf{J}_{r}^{-1}(\text{Log}(X^{-1}Y))\cdot
I\right)$
$\displaystyle=-\text{Log}(X^{-1}Y)^{\top}\Sigma^{-1}\mathbf{J}_{r}^{-1}(\text{Log}(X^{-1}Y)).$

Based on the above derivation, the closed-form solution for the Stein scores
can be obtained as follows:

$\displaystyle\nabla_{Y}\log
p_{\Sigma}(Y|X)=-\mathbf{J}_{r}^{-\top}(\text{Log}(X^{-1}Y))\Sigma^{-1}\text{Log}(X^{-1}Y).$
(21)

###  10.2 Left and Right Jacobians on $SO(3)$

In this section, we present the derivation of Eq. (8). Let
$z=\left[z_{x},z_{y},z_{z}\right]\in\mathfrak{so}(3)$ and
$\phi=\|z\|^{2}_{2}$. The skew-symmetric matrix induced by $z$ can therefore
be represented as follows:

$z_{\times}=\left[\begin{matrix}0&-z_{z}&z_{y}\\\ z_{z}&0&-z_{x}\\\
-z_{y}&z_{x}&0\\\ \end{matrix}\right]$ (22)

As demonstrated in [55], the left and the right Jacobian on $SO(3)$ can be
expressed as the following closed-form expressions:

$\displaystyle\begin{aligned}
\mathbf{J}_{r}(z)&=I-\frac{1-\cos\phi}{\phi^{2}}z_{\times}+\frac{\phi-\sin\phi}{\phi^{3}}z_{\times}^{2}\\\
\mathbf{J}^{-1}_{r}(z)&=I+\frac{1}{2}z_{\times}+\left(\frac{1}{\phi}-\frac{1+\cos\phi}{2\phi\sin\phi}\right)z^{2}_{\times}\\\
\mathbf{J}_{l}(z)&=I+\frac{1-\cos\phi}{\phi^{2}}z_{\times}+\frac{\phi-\sin\phi}{\phi^{3}}z_{\times}^{2}\\\
\mathbf{J}^{-1}_{l}(z)&=I-\frac{1}{2}z_{\times}+\left(\frac{1}{\phi}-\frac{1+\cos\phi}{2\phi\sin\phi}\right)z^{2}_{\times}.\\\
\end{aligned}$ (23)

As a result, Eq. (8) of the main manuscript can be derived as follow:

$\mathbf{J}_{l}(z)=\mathbf{J}_{r}^{\top}(z),\qquad\mathbf{J}_{l}^{-1}(z)=\mathbf{J}_{r}^{-\top}(z).$
(24)

###  10.3 Eigenvector of The Jacobians

For the purpose of proving $\mathbf{J}_{l}(z)z=z$, we consider the derivative
of exponential mapping on $\mathcal{G}$, where $k\in\mathbb{R}$ and
$z\in\mathfrak{g}$. More specifically, by applying the chain rule on the
derivative of the small perturbation $\text{Exp}(kz)$ on $\mathcal{G}$ with
respect to $k$, we can obtain the resultant equation as follows:

$\frac{\partial\text{Exp}(kz)}{\partial
k}=\frac{\partial\text{Exp}(kz)}{\partial(kz)}\frac{\partial(kz)}{\partial
k}=\mathbf{J}_{l}(kz)z.$ (25)

On the other hand, by applying the differential rule, the following equations
can be derived:

$\displaystyle\frac{\partial\text{Exp}(kz)}{\partial k}$
$\displaystyle=\lim_{h\to
0}\frac{\text{Log}(\text{Exp}((k+h)z)\text{Exp}(kz)^{-1})}{k}$ (26)
$\displaystyle=\lim_{h\to
0}\frac{\text{Log}(\text{Exp}(hz)\text{Exp}(kz)\text{Exp}(kz)^{-1})}{h}=z.$

By further combining Eqs. (25) and (26) and setting $k=1$, the following
equation can be derived:

$\mathbf{J}_{l}(z)z=z.$ (27)

The resultant Eq. (27) suggests that $z$ is an eigenvector of
$\mathbf{J}_{l}(z)$. Please note that the same rule can also be employed to
provide a proof for the right-Jacobian as follows:

$\mathbf{J}_{r}(z)z=z.$ (28)

###  10.4 Closed-Form of Stein Scores on $SE(3)$

In this section, we delve into the closed-form solution of Stein scores on
$SE(3)$, which is referenced in Section 4.3. Let
$z=(\rho,\phi)\in\mathfrak{se}(3)$, where $\rho$ represents the translational
part and $\phi$ denotes the rotational part. We define
$\hat{\phi}=\|\phi\|^{2}_{2}$ and recall the inverse of the left-Jacobian on
$SE(3)$ as follows:

$\mathbf{J}^{-1}_{l}(z)=\left[\begin{matrix}\mathbf{J}^{-1}_{l}(\phi)&\mathbf{Z}(\rho,\phi)\\\
0&\mathbf{J}^{-1}_{l}(\phi)\end{matrix}\right],$ (29)

where
$\mathbf{Z}(\rho,\phi)=-\mathbf{J}^{-1}_{l}(\phi)\mathbf{Q}(\rho,\phi)\mathbf{J}^{-1}_{l}(\phi)$.
The complete form of $\mathbf{Q}(\rho,\phi)$ is defined in [55, 4] as follows:

$\displaystyle\mathbf{Q}($
$\displaystyle\rho,\phi)=\frac{1}{2}\rho_{\times}+\frac{\hat{\phi}-\sin\hat{\phi}}{\hat{\phi}^{3}}(\phi_{\times}\rho_{\times}+\rho_{\times}\phi_{\times}+\phi_{\times}\rho_{\times}\phi_{\times})$
(30)
$\displaystyle-\frac{1-\frac{\hat{\phi}^{2}}{2}-\cos\hat{\phi}}{\hat{\phi}^{4}}(\phi^{2}_{\times}\rho_{\times}+\rho_{\times}\phi^{2}_{\times}-3\phi_{\times}\rho_{\times}\phi_{\times})$
$\displaystyle-\frac{1}{2}\left(\frac{1-\frac{\hat{\phi}^{2}}{2}-\cos\hat{\phi}}{\hat{\phi}^{4}}-3\frac{\hat{\phi}-\sin\hat{\phi}-\frac{\hat{\phi}^{3}}{6}}{\hat{\phi}^{5}}(\phi_{\times}\rho_{\times}\phi^{2}_{\times}+\phi^{2}_{\times}\rho_{\times}\phi_{\times})\right).$

From the Eq. (30), an essential property can be observed and expressed as
follows:

$\mathbf{Q}^{\top}(-\rho,-\phi)=\mathbf{Q}(\rho,\phi).$ (31)

Based on the above derivation, the closed-form expression of the inverse
transposed right-Jacobian on $SE(3)$ combined with the property outlined in
Eq. (31) can be derived as follows:

$\displaystyle\mathbf{J}_{r}^{-\top}(z)$
$\displaystyle=\left(\mathbf{J}_{l}^{-1}(-z)\right)^{\top}$ (32)
$\displaystyle=\left[\begin{matrix}\mathbf{J}^{-1}_{l}(-\phi)&\mathbf{Z}(-\rho,-\phi)\\\
0&\mathbf{J}^{-1}_{l}(-\phi)\end{matrix}\right]^{\top}$
$\displaystyle=\left[\begin{matrix}\mathbf{J}^{-1}_{r}(\phi)&-\mathbf{J}_{r}^{-1}(\phi)\mathbf{Q}(-\rho,-\phi)\mathbf{J}^{-1}_{r}(\phi)\\\
0&\mathbf{J}^{-1}_{r}(\phi)\end{matrix}\right]^{\top}$
$\displaystyle=\left[\begin{matrix}\mathbf{J}^{-\top}_{r}(\phi)&0\\\
-\mathbf{J}_{r}^{-\top}(\phi)\mathbf{Q}^{\top}(-\rho,-\phi)\mathbf{J}^{-\top}_{r}(\phi)&\mathbf{J}^{-\top}_{r}(\phi)\end{matrix}\right]$
$\displaystyle=\left[\begin{matrix}\mathbf{J}^{-1}_{l}(\phi)&0\\\
-\mathbf{J}_{l}^{-1}(\phi)\mathbf{Q}(\rho,\phi)\mathbf{J}^{-1}_{l}(\phi)&\mathbf{J}^{-1}_{l}(\phi)\end{matrix}\right]$
$\displaystyle=\left[\begin{matrix}\mathbf{J}^{-1}_{l}(\phi)&0\\\
\mathbf{Z}(\rho,\phi)&\mathbf{J}^{-1}_{l}(\phi).\end{matrix}\right]$

The closed-form solution of Stein score on $SE(3)$ can then be computed by the
definition of Stein score as follows:

$\nabla_{Y}\log
p_{\sigma}(\tilde{X}|X)=-\frac{1}{\sigma^{2}}\left[\begin{matrix}\mathbf{J}^{-1}_{l}(\phi)&0\\\
\mathbf{Z}(\rho,\phi)&\mathbf{J}^{-1}_{l}(\phi)\end{matrix}\right]z.$ (33)

After examining the derivation process, it is clear that this computation
involves the costly calculation of Jacobians, and does not confer any
computational benefits when using automatic differentiation. However, by
adopting the surrogate score presented in Eq. (12), it is possible to reduce
the computation of the Jacobian $\mathbf{J}^{-\top}_{r}(z)$, while
simultaneously improving performance, as explained in Section 8.2.