# 6-DoF Stability Field via Diffusion Models

Takuma Yoneda∗ Tianchong Jiang∗ Gregory Shakhnarovich Matthew R. Walter Takuma
Yoneda, Gregory Shakhnarovich, and Matthew R. Walter are with the Toyota
Technological Institute at Chicago (TTIC), Chicago, IL USA,
<EMAIL_ADDRESS>Jiang is with the University of
Chicago, Chicago, IL USA<EMAIL_ADDRESS>
###### Abstract

fancyline,color=green!60]MW: I updated the abstract A core capability for
robot manipulation is reasoning over where and how to stably place objects in
cluttered environments. Traditionally, robots have relied on object-specific,
hand-crafted heuristics in order to perform such reasoning, with limited
generalizability beyond a small number of object instances and object
interaction patternsfancyline,color=yellow!40]TJ: originally:and classes, but
we don’t have many classes in the experiments. Recent approaches instead learn
notions of physical interaction, namely motion prediction, but require
supervision in the form of labeled object information or come at the cost of
high sample complexity, and do not directly reason over stability or object
placement. We present 6-DoFusion, a generative model capable of generating 3D
poses of an object that produces a stable configuration of a given scene.
Underlying 6-DoFusion is a diffusion model that incrementally refines a
randomly initialized $\textrm{SE}(3)$ pose to generate a sample from a
learned, context-dependent distribution over stable poses. We evaluate our
model on different object placement and stacking tasks, demonstrating its
ability to construct stable scenes that involve novel object classes as well
as to improve the accuracy of state-of-the-art 3D pose estimation
methods.fancyline,color=green!60]MW: Added reference to pose estimation
results

**footnotetext: Equal contribution.

##  I Introduction

In order for robots to operate effectively in our homes and workplaces, they
must be able to reason over where and how to place objects in our cluttered
environments—whether it is to prepare a dining table for a meal or to put
dishes away in a cupboard. Indeed, arguably the most common use of robot
manipulators is for pick-and-place tasks. A common approach to endowing robots
with this level of reasoning is to manually define a set of heuristics that
attempt to identify valid object placements, for example by detecting empty
locations and hard-coding stable orientations for every object [1, 2, 3].
These heuristics are often specified in an individual, object- and
environment-specific manner, and thus tend to be restricted to relatively few
objectsfancyline,color=blue!10]TY: Are we referring to any specific
literature? fancyline,color=green!60]MW: Added refs, particularly when the
robot needs to reason over placements that involve object interactions (e.g.,
stacking one mug on top of another). The limited ability to generalize to new
objects and scenes precludes the use of such heuristics in human environments
that are often complex, cluttered, and contain a diverse array of objects [4].

Data-driven methods provide a promising alternative to the reliance on
heuristics to reason over stable object placement [5, 6, 7]. Indeed, such
strategies have the potential to better generalize to novel object and
environment instances. However, earlier data-driven methods trade-off hand-
engineered heuristics and instead rely on hand-engineered features to
represent key properties of inter-object placement [8]. More recent methods
instead learn these representations in the context of a model trained to
classify the stability of a candidate pose for the objects in the scene, where
these candidates may be randomly generated (e.g., by sampling from a uniform
distribution) or sampled from a guided distribution [9]. These stability
prediction models typically require access to a large amount of training data
in order to be representative of a diverse set of object types and poses,
which can be particularly costly when it requires the robot to interact with
the environment to collect data. Indeed, approaches that employ rejection
sampling can be inefficient, particularly when the support of the target event
is small (e.g., objects requiring careful placement to be stable).

Reverse diffusion process Figure 1: Given (top-left) an initial 3D scene that
consists of two objects (a green rectangle and blue wedge), 6-DoFusion
initializes a given object (red cube) in a random pose and (top, left-to-
right) employs a reverse diffusion process to generate a (top-right) valid
$\textrm{SE}(3)$ pose that results in each object in the scene being stable.
As depicted in the bottom row (bottom), 6-DoFusion is able to produce a
diverse array of candidate stable poses.

Instead, we propose 6-DoFusion (Fig. 1), a framework that generates poses for
a _query_ object that stabilizes a given scene (i.e., the _context_) by
sampling from a learned context-dependent distribution over stable poses. As a
generative model, an advantage of our approach is that it requires access to
only positive examples of stable scene configurations (object poses), and does
not rely upon a separate stable pose classifier nor does it require rejection
sampling to produce stable poses. Instead, inspired by the work of [10], our
6-DoFusion framework utilizes a diffusion model [11], a type of generative
model that has proven highly effective at complex generation tasks, for
generating samples from a distribution over 6-DoF object poses. Diffusion
models are comprised of two primary processes. The first process, referred to
as forward process, iteratively adds noise of increasing scale to the input
(e.g., the 6-DoF pose of the query object). The second process, known as
reverse process, is then trained to iteratively denoise from this noisy input,
generating a sample from the desired distribution over stable poses.

Our model reasons over the shape and 6-DoF pose of each object in the
workspace, and generates the 6-DoF pose of an additional (query) object, such
that it and the existing blocks remain stable. 6-DoFusion employs a diffusion
model to generate this pose as a sample from a learned, context-dependent
distribution over stable object poses. We train our model on examples of
stable object-object interactions and evaluate it in scenarios that involve
placing and stacking a variety of different object shapes in 3D scenes. We
evaluate the quality and diversity of the generated poses and show that
6-DoFusion is able to successfully place objects to achieve complex object
interactions that render an entire scene containing both seen and unseen
(novel) objects stable.

##  II Related Work

Particularly relevant to our work are methods that consider the related
problems of object rearrangement [4] and object stacking [12, 13, 14, 15].
This includes recent neuro-symbolic approaches [5, 16, 6] to task and motion
planning in the context of long-horizon pick and place tasks. These methods
are restricted to a set of known objects and do not explicitly reason over the
stability of 6-DoF placement pose. Perhaps more relevant is the work of [7],
who propose a framework that identifies where and how (in terms of pose) an
object can be placed in a stable environment given an RGBD image, such that
the resulting pose is both physically stable and consistent with learned
semantic relationships. Our method also reasons over how to place an object so
as to preserve what is already a stable scene, albeit with regards to the
object’s full six-DoF pose rather than only its planar rotation as in their
work. More significant is that unlike their framework, 6-DoFusion determines
how to place an object so as to make an otherwise unstable environment
stable.fancyline,color=yellow!40]TJ: and they cannot make a unstable scene
stable? fancyline,color=green!60]MW: Good catch. Updated.

Increasing attention has been paid to the problem of learning physical
intuition [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. This includes methods
that reason over the stability of a given scene as well as those that are
concerned with modeling forward dynamics. Our model differs from this body of
work in that it directly generates stable 3D object poses by sampling from a
learned distribution, rather than relying on rejection sampling on top of a
learned forward prediction model.

In addition to using a notion of stability to guide object manipulation,
another line of work explores the benefits of using scene stability as an
inductive bias to improve scene understanding. This includes the idea of using
learned stability prediction to score candidate scene predictions (e.g., the
estimated shape and pose of the objects) [28, 29, 30, 31, 32], using the
intuition that the true scene is stable.

Diffusion models [11] have recently been applied to many problems including
image generation, image editing, and text-conditioned image and video
generation.fancyline,color=green!60]MW: I added this StructDiffusion reference
here. It was in a pargraph by itself below In the context of robotics, [33]
propose a diffusion-based planning model that is able to generate a diverse
set of feasible trajectories that reach a desired goal. The ability for
diffusion models to generate samples from a potentially multi-modal
distribution, a capability that we exploit here, has also been used in the
context of other robotics tasks including imitation learning [34], policy
learning via offline reinforcement learning (RL) [35], and shared autonomy
[36]. [10] take advantage of this capability to propose an object-conditioned
grasp generation model built on top of a diffusion model, which involves
careful considerations of the representation of the $\textrm{SE}(3)$ gripper
pose. Very recently, several works [37, 38] propose to use diffusion models
for rearrangement task, yet these approaches weigh on generating locally
plausible placement poses, rather than considering stability of the entire
scene including object interactions as we do.

##  III Method

Figure 2: Overview of the presented architecture, where
$\bm{x}\in\mathbb{R}^{6}$ denotes a stable pose sampled from the training
dataset, $\delta x_{t}$ represents the noise sampled from the corresponding
noise scale, and $\bar{\bm{p}}$ is a pointcloud with two extra dimensions, a
binary flag and SDF value, appended to each point.

In this section, after formally define our problem, we describe how to apply
diffusion models in $SE(3)$ space, as well as the approach to encode object
interactions. Before introducing our 6-DoFusion framework (Figure 2, Algorithm
1), we first provide a mathematical introduction to a general framework of
diffusion models [39]. fancyline,color=yellow!40]TJ: need to change

###  III-A Background on Diffusion Models

Diffusion models [11] are a type of generative model that have proven highly
effective for complex generation tasks (e.g., image synthesis), often
outperforming their generative adversarial network (GAN) [40] and variational
autoencoder (VAE) [41] counterparts. Diffusion models involve two core
processes. Modeled as a first-order Markov chain, the forward diffusion
process involves iteratively adding zero-mean isotropic Gaussian noise of
increasing scale $\sigma_{t}\in\mathbb{R}$ to an initial sample $\bm{x}\sim
q(\bm{x})$ drawn from an unknown data distribution. The model is then trained
to iteratively denoise a noise-corrupted input through the reverse diffusion
process that is conditioned on the noise scale $\sigma_{t}$. Each step of
training a denoising model $D_{\theta}$ minimizes the following objective

$\mathcal{J}_{\theta}=\mathbb{E}_{\begin{subarray}{c}\bm{x}\sim
P_{\textrm{data}}\\\ t\sim\textrm{Uniform}[1,T]\\\
\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})\end{subarray}}\bigg{[}\lVert
D_{\theta}(\bm{x}+\sigma_{t}\bm{\epsilon},\sigma_{t})-\bm{x}\rVert^{2}\bigg{]},$
(1)

where $\bm{x}$ is a data sample. In our proposed method, we adopt the
denoising diffusion probabilistic model (DDPM) [42] formulation, where we have
a discrete set of noise scales of increasing magnitude
$\\{\sigma_{1},\ldots,\sigma_{T}\\}$, and we sample a noise scale uniformly
during training. Once trained, the model generates a sample by iteratively
denoising zero-mean isotropic Gaussian noise via the reverse diffusion
process. Formally, this procedure starts with an initial sample
$\bm{x}_{T}\sim\mathcal{N}(\bm{0},\bm{I})$ and involves recursively generating
a sample in a previous diffusion timestep, until it reaches $\bm{x}_{0}$

$\displaystyle\hat{\bm{x}}_{0}$
$\displaystyle=D_{\theta}(\bm{x}_{t},\sigma_{t})$ (2a)
$\displaystyle\bm{x}_{t-1}$
$\displaystyle=\hat{\bm{x}}_{0}+\sigma_{t-1}\bm{\epsilon},~{}\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}).$
(2b)

###  III-B Problem Formulation

We consider a setting in which there are $N_{\textrm{obj}}-1$ objects in the
environment with poses
$\bm{H}_{2},\ldots,\bm{H}_{N_{\textrm{obj}}}\in\textrm{SE}(3)$ that constitute
a _context_. In this context, we assume that we can place a new _query_ object
in a pose $\bm{H}_{1}$ that makes the entire scene stable. As an example, the
context may contain $N_{\textrm{obj}}-1=2$ blocks, one resting on a table and
the other elevated above it (e.g., Fig. 3, second
row)fancyline,color=blue!10]TY: This is very hard to
understand.fancyline,color=green!60]MW: I reworded While this configuration is
unstable, it is possible to place a new block between the two to support the
floating block. In this way, the task is to identify a 6-DoF pose of a query
object $\hat{\bm{H}}_{1}$ such that the full set of objects becomes stable. We
note that there may be many valid poses $\hat{\bm{H}}_{1}$. We assume access
to the object pose $\bm{H}_{i}\in\textrm{SE}(3)$, pointcloud
$\bm{p}_{i}\in\mathbb{R}^{N_{pts}\times 3}$, and the associated signed
distance field (SDF) [43] $f_{i}:\mathbb{R}^{3}\mapsto\mathbb{R}$ for each
object in the workspace ($i\in[2,N_{\textrm{objs}}]$). We also assume access
to the point cloud for the object to be placed and that we can query its SDF
for a given point.

###  III-C Diffusion Models in $\textrm{SE}(3)$ Space

Learning and reasoning over a stable configuration necessitates that we work
in $\textrm{SE}(3)$ space. The $\textrm{SE}(3)$ pose of an object can be
represented by a homogeneous matrix
$\bm{H}=\left[\begin{smallmatrix}\bm{R}&\bm{t}\\\
\bm{0}&1\end{smallmatrix}\right]\in\mathbb{R}^{4\times 4}$, where
$\bm{R}\in\textrm{SO}(3)$ is a $3\times 3$ rotation matrix and
$\bm{t}\in\mathbb{R}^{3}$ is the translation vector.

In diffusion models, both the training and generation processes involve
iteratively adding Gaussian noise of different scales to the input. Applying
diffusion model formulation in $\textrm{SE}(3)$ space requires careful
consideration of object pose in order to ensure that the transformation matrix
remains valid, e.g., to guarantee that $\bm{R}\in\textrm{SO}(3)$. As with
Urain et al. [10], we represent pose as a six-dimensional vector that consists
of the position of the object center $\bm{t}\in\mathbb{R}^{3}$ in a Cartesian
reference frame and an exponential coordinate representation
$\bm{e}\in\mathbb{R}^{3}$ of its orientation. We can add Gaussian noise
$\bm{\epsilon}_{r}\in\mathbb{R}^{3}$ to an $\textrm{SO}(3)$ orientation as
follows,

$\tilde{\bm{R}}=\bm{R}\text{Expmap}(\bm{\epsilon}_{r}),~{}\bm{\epsilon}_{r}\sim\mathcal{N}(\bm{0},\sigma^{2}\bm{I}),~{}\bm{\epsilon}_{r}\in\mathbb{R}^{3}.$
(3)

In summary, the procedure for adding noise to a sample $\bm{H}$ from
$\textrm{SE}(3)$ space involves (1) sampling noise from an isotropic zero-mean
Gaussian $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\sigma^{2}\bm{I})$, where
$\bm{\epsilon}\in\mathbb{R}^{6}$; (2) treating the first three dimensions
$\bm{\epsilon}_{t}=\bm{\epsilon}_{0:3}$ as additive noise on position, and the
last three entries $\bm{\epsilon}_{r}=\bm{\epsilon}_{3:6}$ as noise added to
the exponential coordinate; and (3) constructing $\hat{\bm{H}}$ from
$\hat{\bm{t}}=\bm{\epsilon}_{t}$ and
$\hat{\bm{R}}=\textrm{Expmap}(\bm{\epsilon}_{r})$. Adopting this formulation,
we use DDPM [42] to learn and generate a 6-DoF object pose.

###  III-D Encoding Object Interactions

Now that we have established a means of applying diffusion models in
$\textrm{SE}(3)$ space, we now describe the process by which we model object
interactions. These interactions, which include contacts and object-on-object
support relationships are integral to reasoning over the stability of a
candidate configuration of objects in the scene. In principle, the model can
extract such information from the individual pointclouds. However, as a form
of supplementary information, we include SDF values evaluated on the points as
an auxiliary input to the network.

We define $\bm{p}_{i,n}\in\mathbb{R}^{3}$ as the coordinates $(x,y,z)$ of
point $n$ within the pointcloud of object $o_{i}$, and
$\bm{H}^{\textrm{wo}}_{i}\in\mathbb{R}^{4\times 4}$ as the transformation
matrix that transforms 3D points from the reference frame associated with
object $o_{i}$ to the world frame. Denoting
$f_{j}:\mathbb{R}^{3}\mapsto\mathbb{R}$ as a function that returns the SDF for
all other objects $j\in[1,N_{\text{objs}}]\setminus\\{i\\}$, we then compute

$s_{i,n}=\min_{j}f_{j}(\bm{H}^{\textrm{ow}}_{j}\bm{H}^{\textrm{wo}}_{i}\bm{p}_{i,n}),$
(4)

where $\bm{H}^{\textrm{ow}}_{j}=(\bm{H}^{\textrm{wo}}_{j})^{-1}$. Taking the
minimum over SDF values in this fashion can be interpreted as virtually
merging all objects with the exception of $o_{i}$, and computing the SDF value
of point $\bm{p}_{i,n}\in\mathbb{R}^{3}$ with respect to the merged object. We
perform this operation for all objects, including the query object. After
obtaining the SDF value $s_{i,n}\in\mathbb{R}$ for each point in the
pointcloud, we append it to the world-frame coordinate $\bm{p}_{i,n}$ along
with a binary flag that distinguishes the set of points that belong to the
object being diffused from those that belong to one of the other objects,
resulting in a five-dimensional vector. In the end, we end up with augmented
pointcloud $\bar{\bm{p}}\in\mathbb{R}^{N_{\text{objs}}\cdot
N_{\text{pts}}\times 5}$.

As depicted in Figure 2, we provide the resulting augmented pointcloud as
input to the pointcloud encoder $F_{\varphi}:\mathbb{R}^{N_{\text{objs}}\cdot
N_{\text{pts}}\times 5}\mapsto Z$ that outputs a vector-valued latent
embedding $\bm{z}$ of the augmented pointcloud. This embedding along with a
noisy 6-DoF pose of the diffusing object
$\bm{x}+\sigma_{t}\bm{\epsilon}\in\mathbb{R}^{6}$ and noise scale factor $t$
are then fed to the denoising network $D_{\theta}$. The denoising network
predicts the denoised pose $\hat{x}$ and is trained in the same way as a
standard DDPM [42].

1: Input: pointclouds of objects: $\bm{p}_{1},\dots,\bm{p}_{n}$,

object poses in the workspace: $\bm{H}_{2},\dots,\bm{H}_{n}$

2: while True do

3: Sample $t\sim\textrm{Uniform}[0,T]$

4: $\sigma_{t}\leftarrow\textrm{NoiseSchedule}(t)$

5:
$\bm{\epsilon}\sim\mathcal{N}(\bm{0},\sigma^{2}_{t}\bm{I}),~{}\bm{\epsilon}\in\mathbb{R}^{6}$

6: $\tilde{\bm{H}}_{i}\leftarrow\textrm{AddNoise}(\bm{H}_{i},\bm{\epsilon})$

7: for $i,j$ in $1$ to $N_{\textrm{objs}}$ do

8: $\bm{p}_{i,j}\leftarrow\bm{H}_{j}^{-1}\tilde{\bm{H}}_{i}\bm{p}_{i}$

9: $\textrm{SDF}_{i,j}\leftarrow\textrm{ComputeSDF}(\bm{p}_{i,j}$, $z_{i})$

10: end for

11: for $i$ in $1$ to $N_{\textrm{objs}}$ do

12: $\textrm{cond}_{i}\leftarrow 1~{}\textbf{if}~{}i=1~{}\textbf{else}~{}0$

13:
$\textrm{SDF}_{i}\leftarrow\min_{j}(\textrm{SDF}_{i,j})~{}~{}j\in[1,N_{\textrm{objs}}]\setminus\\{i\\}$

14:
$\bar{\bm{p}}_{i}\leftarrow\textrm{Concatenate}(\bm{p}_{i},\textrm{SDF}_{i},\textrm{cond}_{i})$

15: $\Psi\leftarrow\textrm{DGCNN}_{\varphi}(\tilde{\bm{p}}_{i})$

16:
$\hat{\bm{\epsilon}}\leftarrow\textrm{Decoder}_{\theta}(\Psi,\bm{x}_{i}+\bm{\epsilon},t)$

17:
$\textrm{Optimizer.step}\big{(}\nabla_{\theta,\varphi}\mathcal{L}(\hat{\bm{\epsilon}},\bm{\epsilon})\big{)}$

18: end for

19: end while

Algorithm 1 6-DoFusion Training Procedure

##  IV Experiments

We evaluate the effectiveness with which our model is able to generate stable
poses for seen and unseen (i.e., novel) objects in the context of three task
settings: single-block placement, multi-block stacking, and unstructured
multi-block placement. For each domain, 6-DoFusion generates the pose of a
single (query) block given the context (i.e., the poses and augmented
pointclouds of the other blocks). We then evaluate the stability of the
generated poses by simulating the effects of gravity and object interactions
until the objects settle and then measure their displacements. We note that
our task setting, where we aim to generate a $\textrm{SE}(3)$ stable object
pose, is quite new that we are not aware of any other work that is directly
comparable to ours. For example, Paxton et al. [7] address a similar issue,
however, they only reason over a planar rotation of the object. Liu et al.
[37] adopt an object-centric diffusion model to reason over placement, but
their framework expects language conditioning. As such, we compare against
random sampling, followed by forward simulation in some cases as the baseline.

###  IV-A Setup

####  IV-A1 Dataset generation

We consider 3D blocks with seven different shapes. These shapes are based on
those used by Janner et al. [24]. We generate an instance of stable block
poses by first randomly sampling block shapes with replacement, dropping them
one-by-one from a random $\textrm{SE}(3)$ pose above the ground biased to
encourage object interaction, and then use the MuJoCo physics simulator [44]
to simulate the effects of gravity and object interactions. We follow this
procedure to generate two datasets, one for unstructured block placement and
the other for two- and three-object block stacking. We generate $300$K stable
block configurations with three blocks, $200$K configurations with two blocks,
and $100$K with one block.

####  IV-A2 Architecture

We employ DGCNN [45] to encode the augmented pointcloud
$\bm{p}\in\mathbf{R}^{N_{\text{objs}}\cdot N_{\text{pts}}\times 5}$
($F_{\varphi}$ in Fig. 2). DGCNN interleaves the construction of a
$k$-nearest-neighbor ($k$-NN) graph on the points and graph convolution. For
the first layer, we apply $k$-NN graph construction on the four dimensions,
i.e., the position and the binary flag for the query object. We adopt the
pointcloud classification architecture of [45], by removing its classification
head. This network gives us a single latent vector
$\mathbf{z}\in\mathbb{R}^{512}$. We follow this pointcloud encoder by a three-
layer MLP with ReLU activation. The MLP takes in the concatenation of the
noisy pose of the query object $\bm{x}$, the diffusion noise scale $t$, and
the latent vector $\bm{z}$. The MLP outputs the predicted noise that is then
added to the pose of the query object.

####  IV-A3 Evaluating a generated pose

To evaluate the stability of a generated pose, we would ideally want to
quantify the difference between the pose and the “nearest” stable pose.
However, there is no clear way to identify the “nearest” pose in a non-brute-
force manner. Thus, we evaluate the stability of a generated pose by
initializing the query object in the MuJoCo simulator at the generated pose,
and running forward simulation until every object in the scene settles. We
then measure the _translational displacement_ as the Euclidean distance
between the initial and settled object positions, and the _rotational
displacement_ as the magnitude of the axis-angle representation of the
relative rotation.

We compare our model to a baseline that randomly samples poses above the
existing blocks. The baseline samples the orientation of the placing block
randomly, and the $(x,y)$ coordinates from a Gaussian centered at the average
$(x,y)$ coordinate of the existing blocks. In the block stacking experiment in
Section IV-B, we set the $(x,y)$ coordinates of the baseline to match the
block at the top.

Aside from one model trained on all of the shapes, in order to test the
ability of our model to generalize to new shapes, we train seven separate
models, each with a different held-out shape. In evaluation, we split the test
dataset into seven subsets, each corresponding to a unique shape. We ensure
that the corresponding shape always shows up in every scene of the subset. For
each test subset, we run a model trained on all shapes (in-distribution, ID)
and a model that has not seen the corresponding shape (out-of-distribution,
OOD).

Shapes that have longer principle dimensions, such as the long triangle, are
more likely to have large translational displacements when settling. As such,
we normalize the translational displacements of each block by dividing the raw
translational displacement by the diameter of the block. For translational and
rotational displacements, we score each instance using the maximum
displacement over all objects (both the context and query objects) and report
the median across instances.

###  IV-B Single-Block Placement

As a means of validating that 6-DoFusion is able to learn to generate
reasonable poses, we first train the model on the single-block placement
dataset that includes a single instance of different shapes at stable
configurations in the scene.

TABLE I: Performance on single-block placement | ID | OOD | Random
---|---|---|---
Translational Displacement (%) | $1.2$ | $2.4$ | $90.7$
Rotational Displacement (deg) | $1.2$ | $1.4$ | $37.2$

Table I summarizes the performance of our model for single-block placement
(Fig. 3) in terms of the median translational and rotational displacements
over the test set. As we see, 6-DoFusion is able to place the object in stable
3D poses and is also able to generalize to unseen shapes. In contrast, the
baseline produces poses that tend to be far less stable, as indicated by the
large displacement values.

###  IV-C Block Stacking

Next, we evaluate the ability of our model to place a given query block in a
stable pose on top of existing blocks (Fig. 3, second and the third rows). We
train a single model on a union of the single-block placement and block
stacking datasets, where the total number of blocks in a scene ranges from one
to three. The evaluation tasks 6-DoFusion with generating a pose that stacks
the given object in a scene that consists of one or two objects.

TABLE II: Success rate (%) on block stacking (on one block) |  Block | ID | OOD | Rand. Ori.
---|---|---|---|---
|  Tall Triangle | $81.6$ | $75.6$ | $67.6$
|  Middle Triangle | $87.2$ | $83.8$ | $36.2$
|  Half Rectangle | $87.0$ | $75.6$ | $46.8$
|  Rectangle | $94.6$ | $87.6$ | $51.0$
|  Cube | $95.6$ | $89.6$ | $71.6$
|  Tetrahedron | $80.0$ | $52.2$ | $23.4$
|  Hat | $75.6$ | $68.0$ | $69.8$

We first consider the rate at which 6-DoFusion successfully stacks a given
object in a single-object scene without falling. Table II summarizes the in-
distribution and out-of-distribution success rates of 6-DoFusion compared to
the baseline, which sets the placing block at the same $(x,y)$ coordinate as
the block below, and randomly samples its orientation. We see that 6-DoFusion
performs well on seen and unseen objects.

TABLE III: Success rate (%) on block stacking (on two blocks) |  Block | ID | OOD | Rand. Ori.
---|---|---|---|---
|  Tall Triangle | $46.6$ | $31.8$ | $18.4$
|  Middle Triangle | $53.2$ | $43.8$ | $16.8$
|  Half Rectangle | $65.8$ | $54.2$ | $15.4$
|  Rectangle | $80.2$ | $65.0$ | $13.0$
|  Cube | $90.8$ | $79.6$ | $14.4$
|  Tetrahedron | $65.2$ | $26.6$ | $22.4$
|  Hat | $46.6$ | $18.8$ | $28.2$

Table III summarizes the statistics for the scenario in which the initial
scene consists of two stacked blocks. This task is significantly more
difficult. As the stack is taller, it requires more precise placement (pose)
of the block to avoid collapsing the existing stack. It is also indicated by
the noticeable decrease in the baseline performance. 6-DoFusion is able to
stack blocks successfully in more cases than the baseline even for OOD
objects.

###  IV-D Conditional Pose Generation

TABLE IV: Performance of unstructured block placement with two other blocks |  | Trans. Disp. (%) $\downarrow$ | Rot. Disp. (deg) $\downarrow$
---|---|---|---
| Block |  6-DoFusion |  Random |  6-DoFusion |  Random
In-Distribution (ID)
| Tall Triangle |  $25.2$ |  $\hphantom{0}76.4$ |  $26.8$ |  $54.6$
| Middle Triangle |  $\hphantom{0}6.8$ |  $125.0$ |  $\hphantom{0}5.7$ |  $50.3$
| Half Rectangle |  $26.2$ |  $\hphantom{0}87.5$ |  $22.2$ |  $53.5$
| Rectangle |  $12.6$ |  $\hphantom{0}80.5$ |  $10.0$ |  $48.9$
| Cube |  $\hphantom{0}2.3$ |  $103.1$ |  $\hphantom{0}1.8$ |  $43.9$
| Tetrahedron |  $\hphantom{0}5.5$ |  $125.8$ |  $\hphantom{0}3.5$ |  $45.7$
| Hat |  $13.4$ |  $115.5$ |  $12.1$ |  $48.1$
Out-of-Distribution (OOD)
| Tall Triangle |  $17.9$ |  $\hphantom{0}78.7$ |  $21.0$ |  $51.0$
| Middle Triangle |  $\hphantom{0}9.5$ |  $125.9$ |  $\hphantom{0}7.5$ |  $47.3$
| Half Rectangle |  $32.8$ |  $\hphantom{0}86.3$ |  $25.7$ |  $52.9$
| Rectangle |  $40.0$ |  $\hphantom{0}76.7$ |  $22.0$ |  $53.9$
| Cube |  $\hphantom{0}2.7$ |  $105.1$ |  $\hphantom{0}2.3$ |  $43.9$
| Tetrahedron |  $\hphantom{0}8.5$ |  $132.4$ |  $\hphantom{0}5.9$ |  $46.6$
| Hat |  $18.5$ |  $110.6$ |  $15.2$ |  $49.5$

We consider the more general setting in which the model needs to reason over a
placement pose in a cluttered environment (Fig. 3, bottom two rows). This
setting is particularly challenging since the object interactions are far more
complex than with block stacking. If a block is leaning on another block, the
set of poses is no longer stable if we remove the supporting. In such cases,
the model needs to generate poses that are not only stable themselves, but
that also support existing blocks in the scene such that the entire
configuration is stable. We train a single model on the union of the single-
block and all unstructured block datasets, where the number of blocks in a
scene ranges from one to three. Table IV summarizes the ID and OOD results for
scenes with one and two blocks, respectively. Again, we see that 6-DoFusion
produces far more stable poses than the baseline.

Placement |  |  |  |  |
---|---|---|---|---|---
Placement |  |  |  |  |
Placement |  |  |  |  |
Stacking |  |  |  |  |
Stacking |  |  |  |  |
| Initial | Generated | Settled | Random | Settled
Figure 3: Samples of conditionally generated poses with environments with zero, one, or two blocks and for tasks that involve general placement and object stacking, where each row corresponds to a different scenario. The first column on the left shows the initial scene, while the second column from the left provides a visualization of the pose that 6-DoFusion generates for the new object in red and the third column shows the pose of the objects after settling. We compare these results to the poses generated by the random baseline (fourth column) and their resulting settled poses (fifth column). Initial |  | $\rightarrow$ |  |  |  |  Generated
---|---|---|---|---|---|---
|  |  |  |  |  |  Settled
Initial |  | $\rightarrow$ |  |  |  |  Generated
|  |  |  |  |  |  Settled
Initial |  | $\rightarrow$ |  |  |  |  Generated
|  |  |  |  |  |  Settled
Figure 4: Examples demonstrating the diversity with which our algorithm
generates poses of a new object (shown in red) for three different initial
scene contexts. Each of the three columns on the right visualizes (top row)
different poses that 6-DoFusion generates for the new object along with the
(bottom row) settled poses. Note that in the top scenario, 6-DoFusion
stabilizes the blue triangle by placing the given object underneath it.

###  IV-E Ablation Studies

Next, we ablate the different components of our framework in order to better
understand their contribution. In particular, we consider the effect of
removing SDF information as an explicit indication of inter-object geometry,
as well as the effect of removing explicit pose of the placing object from the
input to the model.

We consider the task of placing a block in an unstructured scene that contains
two blocks, and compute the translational and rotational displacements, as
well as the success rates.

TABLE V: Ablation study | In-distribution | Out-of-distribution
---|---|---
Block |  Full |  w/o SDF |  w/o pos |  Full |  w/o SDF |  w/o pos
Trans. Displ. (%) |  $13.1$ |  $18.5$ |  $70.0$ |  $18.6$ |  $27.7$ |  $82.7$
Rot. Displ. (deg) |  $11.7$ |  $15.6$ |  $74.0$ |  $14.3$ |  $25.5$ |  $80.5$

Table V summarizes the result of these ablations. This suggests that the model
gains meaningful information from both the SDF and the explicit pose of the
placed object.

##  V Conclusion

We presented 6-DoFusion, a diffusion-based model that generates 6-DoF object
poses that result in stable multi-object scenes. We evaluated our model on
multi-object placement and stacking tasks, and demonstrated that it is able to
reason over the full 6-DoF pose of novel objects placed in complex,
unstructured scenes. A limitation of 6-DoFusion is that it assumes knowledge
of the pose and SDF of relevant objects. Future work includes updating the
model to reason over noisy estimates of object poses and pointclouds, such as
those that can be inferred from RGBD images of the scene.

## References

  * Schmitt et al. [2017] P. S. Schmitt, W. Neubauer, W. Feiten, K. M. Wurm, G. V. Wichert, and W. Burgard, “Optimal, sampling-based manipulation planning,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 3426–3432.
  * Xian et al. [2017] Z. Xian, P. Lertkultanon, and Q.-C. Pham, “Closed-chain manipulation of large objects by multi-arm robotic systems,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 4, pp. 1832–1839, 2017.
  * Garrett et al. [2018] C. R. Garrett, T. Lozano-Perez, and L. P. Kaelbling, “FFRob: Leveraging symbolic planning for efficient task and motion planning,” _International Journal of Robotics Research_ , vol. 37, no. 1, pp. 104–136, 2018.
  * Batra et al. [2020] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi _et al._ , “Rearrangement: A challenge for embodied ai,” _arXiv preprint arXiv:2011.01975_ , 2020.
  * Huang et al. [2019] D.-A. Huang, D. Xu, Y. Zhu, A. Garg, S. Savarese, L. Fei-Fei, and J. C. Niebles, “Continuous relaxation of symbolic planner for one-shot imitation learning,” in _Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019.
  * Zhu et al. [2021] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu, “Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * Paxton et al. [2022] C. Paxton, C. Xie, T. Hermans, and D. Fox, “Predicting stable configurations for semantic placement of novel objects,” in _Proceedings of the Conference on Robot Learning (CoRL)_ , 2022.
  * Jiang et al. [2012] Y. Jiang, M. Lim, C. Zheng, and A. Saxena, “Learning to place new objects in a scene,” _International Journal of Robotics Research_ , 2012.
  * Cheng et al. [2021] S. Cheng, K. Mo, and L. Shao, “Learning to regrasp by learning to place,” in _Proceedings of the Conference on Robot Learning (CoRL)_ , 2021.
  * Urain et al. [2023] J. Urain, N. Funk, J. Peters, and G. Chalvatzaki, “SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion,” _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Sohl-Dickstein et al. [2015] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2015.
  * Deisenroth et al. [2011] M. P. Deisenroth, C. E. Rasmussen, and D. Fox, “Learning to control a low-cost manipulator using data-efficient reinforcement learning,” in _Proceedings of Robotics: Science and Systems (RSS)_ , 2011.
  * Rusu et al. [2016] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” _arXiv preprint arXiv:1606.04671_ , 2016.
  * Nair et al. [2018] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Overcoming exploration in reinforcement learning with demonstrations,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2018.
  * Li et al. [2020] R. Li, A. Jabri, T. Darrell, and P. Agrawal, “Towards practical multi-object manipulation using relational reinforcement learning,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2020.
  * Xu et al. [2019] D. Xu, R. Martín-Martín, D.-A. Huang, Y. Zhu, S. Savarese, and L. F. Fei-Fei, “Regression planning networks,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Wu et al. [2017] J. Wu, E. Lu, P. Kohli, W. T. Freeman, and J. B. Tenenbaum, “Learning to see physics via visual de-animation,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Wang et al. [2018] Z. Wang, S. Rosa, B. Yang, S. Wang, A. Trigoni, and A. Markham, “3D-PhysNet: Learning the intuitive physics of non-rigid object deformations,” in _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2018.
  * Li et al. [2016] W. Li, S. Azimi, A. Leonardis, and M. Fritz, “To fall or not to fall: A visual approach to physical stability prediction,” _arXiv preprint arXiv:1604.00066_ , 2016.
  * Lerer et al. [2016] A. Lerer, S. Gross, and R. Fergus, “Learning physical intuition of block towers by example,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2016.
  * Finn et al. [2016] C. Finn, I. Goodfellow, and S. Levine, “Unsupervised learning for physical interaction through video prediction,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016.
  * Chang et al. [2016] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum, “A compositional object-based approach to learning physical dynamics,” _arXiv preprint arXiv:1612.00341_ , 2016.
  * Yi et al. [2019] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum, “CLEVRER: Collision events for video representation and reasoning,” _arXiv preprint arXiv:1910.01442_ , 2019.
  * Janner et al. [2019] M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu, “Reasoning about physical interactions with object-oriented prediction and planning,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019.
  * Veerapaneni et al. [2020] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. Tenenbaum, and S. Levine, “Entity abstraction in visual model-based reinforcement learning,” in _Proceedings of the Conference on Robot Learning (CoRL)_ , 2020.
  * Piloto et al. [2022] L. S. Piloto, A. Weinstein, P. Battaglia, and M. Botvinick, “Intuitive physics learning in a deep-learning model inspired by developmental psychology,” _Nature Human Behaviour_ , 2022.
  * Driess et al. [2022] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint, “Learning multi-object dynamics with compositional Neural Radiance Fields,” _arXiv preprint arXiv:2202.11855_ , 2022.
  * Zheng et al. [2015a] B. Zheng, Y. Zhao, J. Yu, K. Ikeuchi, and S.-C. Zhu, “Scene understanding by reasoning stability and safety,” _International Journal on Computer Vision_ , 2015.
  * Silberman et al. [2012] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2012.
  * Jia et al. [2013] Z. Jia, A. Gallagher, A. Saxena, and T. Chen, “3D-based reasoning with blocks, support, and stability,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2013.
  * Zheng et al. [2015b] B. Zheng, Y. Zhao, J. Yu, K. Ikeuchi, and S.-C. Zhu, “Scene understanding by reasoning stability and safety,” _International Journal on Computer Vision_ , 2015.
  * Du et al. [2018] Y. Du, Z. Liu, H. Basevi, A. Leonardis, B. Freeman, J. Tenenbaum, and J. Wu, “Learning to exploit stability for 3D scene parsing,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * Janner et al. [2022] M. Janner, Y. Du, J. Tenenbaum, and S. Levine, “Planning with diffusion for flexible behavior synthesis,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2022.
  * Pearce et al. [2023] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann _et al._ , “Imitating human behaviour with diffusion models,” _arXiv preprint arXiv:2301.10677_ , 2023.
  * Wang et al. [2022] Z. Wang, J. J. Hunt, and M. Zhou, “Diffusion policies as an expressive policy class for offline reinforcement learning,” _arXiv preprint arXiv:2208.06193_ , 2022.
  * Yoneda et al. [2023] T. Yoneda, L. Sun, G. Yang, B. Stadie, and M. Walter, “To the noise and back: Diffusion for shared autonomy,” _arXiv preprint arXiv:2302.12244_ , 2023.
  * Liu et al. [2022] W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton, “StructDiffusion: Language-guided creation of physically-valid structures using unseen objects,” _arXiv preprint arXiv:2211.04604_ , 2022.
  * Simeonov et al. [2023] A. Simeonov, A. Goyal, L. Manuelli, L. Yen-Chen, A. Sarmiento, A. Rodriguez, P. Agrawal, and D. Fox, “Shelving, stacking, hanging: Relational pose diffusion for multi-modal rearrangement,” _arXiv preprint arXiv:2307.04751_ , 2023.
  * Karras et al. [2022] T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Saharia et al. [2022] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in _Proceeding of the International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH)_ , 2022.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 10 684–10 695.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _arXiv preprint arxiv:2006.11239_ , 2020.
  * Park et al. [2019] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “DeepSDF: Learning continuous signed distance functions for shape representation,” 2019.
  * Todorov et al. [2012] E. Todorov, T. Erez, and Y. Tassa, “MuJoCo: A physics engine for model-based control,” in _Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2012.
  * Wang et al. [2019] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph CNN for learning on point clouds,” _ACM Transactions on Graphics (TOG)_ , 2019.