capbtabboxtable[][]

# Fourier Transporter:
Bi-Equivariant Robotic Manipulation in 3D

Haojie Huang1, Owen Howell2,$\star$, Dian Wang1,$\star$, Xupeng Zhu1,$\star$,
Robert Platt1,$\dagger$, Robin Walters1,$\dagger$
Khoury College of Computer Sciences 1 ,
Department of Electrical and Computer Engineering 2
Northeastern University,
Boston MA, 02115, USA
{huang.latehaoj; howell.o; wang.dian; zhu.xup; r.walters;
<EMAIL_ADDRESS>$\star$ denotes equal contribution, $\dagger$
denotes equal advising

###### Abstract

Many complex robotic manipulation tasks can be decomposed as a sequence of
pick and place actions. Training a robotic agent to learn this sequence over
many different starting conditions typically requires many iterations or
demonstrations, especially in 3D environments. In this work, we propose
Fourier Transporter (FourTran) which leverages the two-fold
$\mathrm{SE}(d)\times\mathrm{SE}(d)$ symmetry in the pick-place problem to
achieve much higher sample efficiency. FourTran is an open-loop behavior
cloning method trained using expert demonstrations to predict pick-place
actions on new environments. FourTran is constrained to incorporate symmetries
of the pick and place actions independently. Our method utilizes a fiber space
Fourier transformation that allows for memory-efficient construction. We test
our proposed network on the RLbench benchmark and achieve state-of-the-art
results across various tasks.

##  1 Introduction

Imitation learning for manipulation tasks in $\mathrm{SE}(3)$ has emerged as a
key topic in robotic learning. Imitation learning is attractive because of its
practicality. In contrast to reinforcement learning wherein training happens
via a period of autonomous interaction with the environment (which can
potentially damage the robot and the environment), imitation learning requires
only human demonstrations of the task to be performed which is often safer and
easier to provide. Sample efficiency is critical here; the robot needs to
learn to perform a task without requiring the human to provide an undue number
of demonstrations. Unfortunately, many current state-of-the-art methods are
not sample efficient. For example, even after training with one hundred
demonstrations, methods like PerAct and RVT still struggle to solve standard
RLBench tasks like Stack Wine or Stack Cups (Shridhar et al., 2023; Goyal et
al., 2023).

(a) 2D Bi-Equivariance

(b) 3D Bi-Equivariance

Figure 1:  Illustration of bi-equivariance in 2D (left) and 3D (right). The
place action, $a^{\prime}=g_{2}ag_{1}^{-1}$, is symmetric with respect to both
the orientation of the object to be picked, $g_{1}$, and the orientation of
the place target, $g_{2}$.

Why is sample efficiency such a challenge in three dimensions? A big reason is
that these tasks are defined over $\mathrm{SE}(3)$ action spaces where the
learned policy must output both a 3D position and $\mathrm{SO}(3)$
orientation. The orientation component here is a particular challenge for
robot learning because $\mathrm{SO}(3)$ is not Euclidean and standard
convolutional layers are not a good fit. Discretization is also not ideal. For
example, a grid sampling 1000 different orientations in $\mathrm{SO}(3)$ still
only realizes an angular resolution of $36^{\degree}$, which is insufficient
for many manipulation tasks. Instead, existing methods, e.g. Shridhar et al.
(2023); Goyal et al. (2023), often fall back on generic self-attention layers
that do not take advantage of the geometry of $\mathrm{SO}(3)$.

One approach to this problem is to leverage neural network policy models that
are symmetric in $\mathrm{SO}(2)$ or $\mathrm{SO}(3)$. This has been explored
primarily in $\mathrm{SO}(2)$ and $\mathrm{SE}(2)$ settings, e.g. Wang et al.
(2021) and Jia et al. (2023), where significant gains in sample efficiency
have been made using policy networks that incorporate steerable convolution
kernels (Cohen & Welling, 2017). In $\mathrm{SO}(3)$, symmetric models have
been mainly applied to pose or descriptor inference, e.g. Klee et al. (2023)
and Ryu et al. (2022), but not directly to policy learning. Moreover, these
works do not address the dual symmetry present in many pick and place
problems, sometimes referred to as _bi-equivariance_ (Ryu et al., 2022), where
the pick-place action distribution transforms symmetrically (equivariantly)
when a transformation is applied independently to either the pick or the place
pose. This is illustrated in Figure 1. Independent rotations of the gear
($g_{1}$) and the slot ($g_{2}$) result in a change
($a^{\prime}=g_{2}ag_{1}^{-1}$) in the requisite action needed to perform the
insertion. While bi-equivariance in policy learning has been studied in
$\mathrm{SO}(2)$ (Huang et al., 2023a), models for encoding $\mathrm{SO}(3)$
bi-equivariance in a general policy learning setting have not yet been
developed.

This paper proposes _Fourier Transporter_ (FourTran), an approach to modeling
$\mathrm{SE}(3)$ bi-equivariance using 3D convolutions and a Fourier
representation of rotations. Unlike existing methods, e.g. Equivariant
Descriptor Fields (Ryu et al., 2022) and TAX-Pose (Pan et al., 2023), our
method encodes $\mathrm{SO}(3)$ bi-equivariance inside a general purpose
policy learning model rather than relying on point descriptors. Our key
innovation is to parameterize action distributions over $\mathrm{SO}(3)$ in
the Fourier domain as coefficients of Wigner $D$-matrix entries where (by the
Convolution Theorem) convolution becomes a simple dot product. We embed this
representation within a 3D translational convolution, thereby enabling us to
do convolutions directly in $\mathrm{SE}(3)$ without excessive computational
cost and with minimal memory requirements. The end result is a policy learning
model for imitation learning with high sample efficiency and high angular
resolution that can outperform existing $\mathrm{SE}(3)$ methods by
significant margins (Table 1). Our contributions are:

  * •

We analyze problems with bi-equivariant symmetry and provide a general
theoretical solution to leverage the coupled symmetries.

  * •

We propose Fourier Transporter (FourTran) for leveraging bi-equivariant
structure in manipulation pick-place problems in 2D and 3D.

  * •

We achieve state-of-the-art performance on several RLbench tasks (James et al.
(2020)). Specifically, FourTran outperforms baselines by a margin between six
percent (Stack-Wine task) and _two-hundred_ percent (Stack-Cups task).

##  2 Related Work

Action-centric manipulation. Action-centric manipulation associates each
pixel, voxel, or point with a target position of the end-effector, providing
an efficient framework that evaluates a large amount of action with a dense
output map. Early works utilizing this method (Zhu et al., 2014; Zeng et al.,
2017; Deng et al., 2020; Wen et al., 2022) often require a pretrained vision
model, and may struggle with deformable or granular objects. Transporter
Networks (Zeng et al., 2021; Seita et al., 2021) combine action-centric
representation with end-to-end learning, showing fast convergence speed and
strong generalization ability. However, end-to-end learning methods are often
limited to 2D top-down settings and cannot efficiently or accurately solve 3D
pick and place tasks.

Recent works have made great progress in extending action-centric manipulation
to the full $\mathrm{SE}(3)$ action space. Lin et al. (2023) synthesize
different views and apply Transporter Networks on those views to realize 3D
pick-place. C2FARM (James et al., 2022) represents the workspace with multi-
resolution voxel grids and infers the translation in a coarse-to-fine fashion.
Transformer based methods like PerAct (Shridhar et al., 2023) first tokenize
the voxel grids together with a language description of the task and learns a
language-conditioned policy with Perceiver Transformer (Jaegle et al., 2021).
RVT (Goyal et al., 2023) projects the 3D observation onto orthographic images
and uses the generated dense feature map of each image to generate 3D actions.
However, they all require a large number of expert demonstrations to learn
even simple 3D pick and place skills. Compared with the previous works, our
proposed architecture is an end-to-end method that demonstrates significant
increase in sample efficiency in both 2D and 3D pick-place tasks.

Symmetries and Robot Learning. Robot learning methods can benefit
significantly through leveraging the underlying symmetry in the task. Zeng et
al. (2018); Morrison et al. (2018) show the translational equivariance of FCN
can improve learning efficiency for manipulation tasks. Recent works explore
the use of Equivariant Networks (Cohen & Welling, 2016; 2017; Weiler et al.,
2018; Weiler & Cesa, 2019; Cesa et al., 2021) to embed the 2D rotational
symmetries, resulting in a dramatic improvement in sample efficiency. Wang et
al. (2021); Zhu et al. (2022) encode the $\mathrm{SO}(2)$ rotation symmetry of
the manipulation policy and realize on-robot grasp learning. Wang et al.
(2022b; a); Jia et al. (2023); Nguyen et al. (2023) exploit the
$\mathrm{SO}(2)$ and $\mathrm{O}(2)$ symmetries and solve multi-step
manipulation tasks with a closed-loop policy. Huang et al. (2022; 2023a)
analyze the bi-equivariant symmetry of pick and place on the 2D rotation
groups. However, they all limit in a 2D action space.

Several works have attempted to utilize the $\mathrm{SO}(3)$ symmetry in robot
learning. Neural Descriptor Fields (NDF) (Simeonov et al., 2022) uses Vector
Neurons (Deng et al., 2021) to generate $\mathrm{SO}(3)$-equivariant key point
descriptors to define the object’s pose for pick and place. However, NDF and
its variations (Chun et al., 2023; Huang et al., 2023b) require well-segmented
point cloud and pre-trained descriptor network in advance. TAX-Pose (Pan et
al., 2023) generates $\mathrm{SO}(3)$-invariant dense correspondences for
pick-place tasks, but is limited to reasoning two objects. Equivariant
Descriptor Field (Ryu et al., 2022) achieves Bi-Equivariance by encoding the
$\mathrm{SO}(3)$-equivariant point feature with Tensor Field Network (Thomas
et al., 2018) and $\mathrm{SE}(3)$ Transformer (Fuchs et al., 2020). However,
it requires lots of samples in $\mathrm{SE}(3)$ to train and test the energy-
based model. In comparison, our proposed method generalizes the Bi-Equvariance
to both 2D and 3D manipulation pick-place problems, infers the pick-and-place
distribution over the entire action space with a single pass, and utilizes the
convolution in Fourier space to improve computation efficiency.

##  3 Background

We provide some preliminary knowledge of symmetry and groups that will be used
in our method. Please see Appendix A.2 for a more thorough mathematical
introduction.

Groups and Representations. In this work, we are primarily interested in the
Euclidean group $\mathrm{SE}(d)=\mathrm{SO}(d)\ltimes\mathbb{R}^{d}$ which
includes all rotations and translations of $\mathbb{R}^{d}$. The discrete
$\mathrm{SO}(2)$ subgroup $C_{n}=\\{\mathrm{Rot}_{\theta}:\theta=2\pi
i/n,0\leq i<n\\}$ contains rotations by angles which are multiples of
$2\pi/n$. The icosahedral rotation group group $I_{60}$ and octohedral
rotation group $O_{24}$ are finite subgroups of the group $\mathrm{SO}(3)$
which give the symmetries of the icosohedron and octohedron and contain $60$
rotations and $24$ rotations, respectively.

An $n$-dimensional _representation_ $\rho\colon G\to\mathrm{GL}_{n}$ of a
group $G$ assigns to each element $g\in G$ an invertible
$n\\!\times\\!n$-matrix $\rho(g)$ where for all $g,g^{\prime}\in G$ the
matrices satisfy $\rho(gg^{\prime})=\rho(g)\rho(g^{\prime})$. Different
representations of $\mathrm{SO}(d)$ or its subgroups describe how different
signals are transformed under rotations. We consider several examples. The
trivial representation $\rho_{0}\colon\mathrm{SO}(d)\to\mathrm{GL}_{1}$
assigns $\rho_{0}(g)=1$ for all $g\in G$, i.e. there is no transformation
under rotation. The standard representation $\rho_{1}$ of $\mathrm{SO}(d)$
assigns each group element its standard $d\times d$ rotation matrix. For
finite groups $G$, the regular representation $\rho_{\mathrm{reg}}$ acts on
$\mathbb{R}^{|G|}$. Label a basis of $\mathbb{R}^{|G|}$ by $\\{e_{g}:g\in
G\\}$, then $\rho_{\mathrm{reg}}(g)(e_{h})=e_{gh}$. Both $\mathrm{SO}(2)$ and
$\mathrm{SO}(3)$ have irreducible representations, i.e. representations with
no non-trivial fixed subspaces, indexed by non-negative integers
$k\in\mathbb{Z}_{\geq 0}$. The irreducible representations of $\mathrm{SO}(3)$
are known as Wigner D-matrices, have dimension $(2k+1)\times(2k+1)$ and are
denoted $D^{k}$. The irreducible representation $\rho_{k}$ of $\mathrm{SO}(2)$
is given by $2\times 2$ rotation matrices
$\rho_{k}(\theta)=\mathrm{Rot}_{k\theta}$.

Steerable Feature Maps. We consider all features to be steerable feature
vector fields $f\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{c}$, which assign a
feature vector $f(\mathbf{x})\in\mathbb{R}^{c}$ to each position
$\mathbf{x}\in\mathbb{R}^{d}$. An element $g\in\mathrm{SO}(d)$ acts on a
steerable $\rho$-field $f$ by acting on both the base space $\mathbb{R}^{d}$
by rotating the pixel or voxel positions and on the fiber space
$\mathbb{R}^{c}$ by some representation $\rho$. We define the induced
representation from $\rho$ on $f$ by
$\mathrm{Ind}_{\rho}(g)(f)(\mathbf{x})=\rho(g)\cdot
f(\rho_{1}(g)^{-1}\mathbf{x})$. For a more thorough discussion of induced
representations please see A.4. We denote the base space action alone by
$(\beta(g)f)(\mathbf{x})=f(\rho_{1}(g)^{-1}\mathbf{x}).$ Note that
$\beta=\mathrm{Ind}_{\rho_{0}}$.

$G$-Equivariant Mappings. A mapping $F\colon X\to Y$ is _equivariant_ to a
group $G$ acting on $X$ by $\rho_{X}$ and $Y$ by $\rho_{Y}$ if it intertwines
the two group actions $\rho_{Y}(g)F[f]=F[\rho_{X}(g)f]\quad\forall g\in G,f\in
X$ Equivariant linear mappings, i.e. intertwiners, between spaces of steerable
feature fields are given by convolution with _G-steerable kernels_ (Weiler et
al., 2018; Jenner & Weiler, 2021). Assume the input field type transforms as
$\mathrm{Ind}_{\rho_{\mathrm{in}}}$ and the output field type as
$\mathrm{Ind}_{\rho_{\mathrm{out}}}$. Then by Cohen et al. (2019), Theorem
3.3, convolution with the kernel
$K\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d_{\mathrm{out}}\times
d_{\mathrm{in}}}$ is $G$-equivariant if and only if its satisfies the
_steerability constraint_ , $K(g\cdot
x)=\rho_{\mathrm{out}}(g)K(x)\rho_{\mathrm{in}}(g)^{-1}$ A complete
characterization and explicit parametrization of steerable kernels is given in
Lang & Weiler (2021).

$\mathrm{SO}(d)$ Fourier Transformation. Signals defined over the group
$\mathrm{SO}(d)$ can be decomposed as limits of linear combinations of complex
exponential functions (for $\mathrm{SO}(2)$) or Wigner D-matrices (for
$\mathrm{SO}(3)$). We refer to the Fourier transform that maps
$\mathrm{SO}(d)$-signals to the coefficients of the basis functions as
$\mathcal{F}^{+}$ and the inverse Fourier Transform as $\mathcal{F}^{-1}$. For
a more in depth discussion of the Fourier Transform, we refer the reader to C.

##  4 Method

###  4.1 Problem Statement

This paper focuses on behavior cloning for robotic pick-place problems. Given
a set of expert demonstrations that contain a sequence of one or more
observation-action pairs $(o_{t},a_{t})$, the objective is to learn a policy
$p(a_{t}|o_{t})$ where the action
$a_{t}=(a_{\mathrm{pick}},a_{\mathrm{place}})$ describes both the pick and
place components of the action and the observation $o_{t}$ describes the
current state of the workspace. In 2D manipulation tasks, $o_{t}$ is in the
format of a top-down image, and in 3D manipulation tasks, $o_{t}$ is a voxel
grid. Our model will factor the policy $p(a_{t},o_{t})$ as

$\displaystyle
p(a_{t}|o_{t})=p(a_{\mathrm{place}}|o_{t},a_{\mathrm{pick}})p(a_{\mathrm{pick}}|o_{t})$

where $p(a_{\mathrm{pick}}|o_{t})$ and
$p(a_{\mathrm{place}}|o_{t},a_{\mathrm{pick}})$ are parameterized as the
output of two separate neural networks. The pick action
$a_{\mathrm{pick}}\in\mathrm{SE}(d)$ and place action
$a_{\mathrm{place}}\in\mathrm{SE}(d)$ are decomposed in terms of translation
and orientation $(T,R)\in\mathrm{SE}(d)$, where $T$ is pixel coordinates
$(u,v)$ in 2D or the voxel coordinates $(i,j,k)$ in 3D111Each pixel or voxel
corresponds to a unique $(x,y,z)$ spatial coordinate in the workspace. The
rotational part of the action $R\in\mathrm{SO}(d)$ denotes the gripper
orientation, which is constrained as planar rotation $\theta$ in 2D tasks and
a three-dimensional rotation $R\in\mathrm{SO}(3)$ in 3D tasks. The pick
rotation $R_{\mathrm{pick}}\in\mathrm{SO}(d)$ is defined with respect to the
world frame and $R_{\mathrm{place}}\in\mathrm{SO}(d)$ is the relative rotation
between the pick pose and place pose.

###  4.2 $\mathrm{SE}(d)$-Equivariant Pick

We first analyze the symmetry of the robotic pick task in $\mathrm{SE}(d)$,
where $d=\\{2,3\\}$ indicates the 2D pick and 3D pick. Then, we present our
pick network to realize the symmetry.

Symmetry of the Pick Action $f_{\mathrm{pick}}$. The pick network takes an
input observation $o_{t}$ and outputs the pick pose probability distribution
over $\mathrm{SE}(d)$, i.e., $f_{\mathrm{pick}}\colon o_{t}\mapsto
p(a_{\mathrm{pick}}|o_{t})$ . We represent $p(a_{\mathrm{pick}}|o_{t})$ as a
steerable field $\mathbb{R}^{d}\to\\{SO(3)\to\mathbb{R}\\}$ in which the base
space determines the pick location and the fiber space encodes the
distribution over pick orientations. The distribution
$p:\mathrm{SO}(3)\to\mathbb{R}$ transforms by $(\rho_{L}(g)p)(h)=p(g^{-1}h)$
where the subscript $L$ denotes the left-action of $g^{-1}$ corresponding to
post-composing with the rotation. A consistent pick network should satisfy the
pick symmetry relation,

$\forall g\in\mathrm{SE}(d),\quad f_{\mathrm{pick}}(g\cdot
o_{t})=\mathrm{Ind}_{\rho_{L}}(g)f_{\mathrm{pick}}(o_{t})$ (1)

Equation 1 illustrates the underlying symmetry of robotic picking problem,
i.e., if there is a transformation $g\in\mathrm{SE}(d)$ on $o_{t}$ (RHS of
Equation 1), the pick pose distribution $p_{\mathrm{pick}}$ should transform
accordingly by $\mathrm{Ind}_{\rho_{L}}$. Here the action on the base space
rotates the pick location and the fiber action transforms the pick
orientation.

Pick Symmetry Constraints. To construct $f_{\mathrm{pick}}$ satisfying
Equation 1, we parameterize it with equivariant convolutional layers. The pick
network $f_{\mathrm{pick}}$ takes the input observation $o_{t}$ as a
$\rho_{0}$ field and outputs a $\rho_{\mathrm{irrep}}$ field where
$\rho_{\mathrm{irrep}}$ represents the direct sum of irreducible
representations giving the truncated Fourier space representation of functions
over $\mathrm{SO}(d)$ . To be specific, $f_{\mathrm{pick}}$ can be encoded as
an equivariant U-Net that takes the observation and generates the coefficients
of $\mathrm{SO}(d)$ basis functions. As a result, $f_{\mathrm{pick}}$
generates an un-normalized distribution over $\mathrm{SO}(2)$ above each pixel
coordinate or over $\mathrm{SO}(3)$ above each voxel coordinate. The pick pose
distribution over $\mathrm{SE}(d)$ is calculated by jointly normalizing the
signal on translation and orientation. The best pick pose
$\bar{a}_{\mathrm{pick}}$ can be evaluate by
$\bar{a}_{\mathrm{pick}}=\operatorname*{arg\,max}p(a_{\mathrm{pick}}|o_{t})$.

Figure 2:  Architecture of FourTran. $f_{\mathrm{pick}}$ first detects a task-
appropriate pick pose. The crop $c$ centered at the pick location is fed to
$\psi$ network. The lift operation generates a stack of rotated features and
Fourier Transformation $\mathcal{F}^{+}$ is applied to the channel space of
the feature to output the dynamic kernel $\kappa(c)$. The cross correlation is
conducted in Fourier space.

###  4.3 $\mathrm{SE}(d)\times\mathrm{SE}(d)$-Equivariant Place

We first clarify the coupled symmetries inside the place tasks and propose a
general solution to achieve the symmetries. Then, we present how FourTran
realizes the solution.

Symmetry of the Place Action $f_{\mathrm{place}}$. After the pick action
$\bar{a}_{\mathrm{pick}}$ is determined, the place network
$f_{\mathrm{place}}$ infers the place action $a_{\mathrm{place}}$ to transport
the object to be grasped to the target placement. We assume the object does
not move or deform during grasping so that $a_{\mathrm{pick}}$ may be
geometrically represented by an image or voxel patch222The pick pose can also
be geometrically represented by occupied voxel. centered on the pick location,
as shown in Figure 2. Our place network is described

$f_{\mathrm{place}}:(c,o_{t})\mapsto
p(a_{\mathrm{place}}|o_{t},a_{\mathrm{pick}})$ (2)

where $p(a_{\mathrm{place}}|o_{t},a_{\mathrm{pick}})$ denotes the probability
that the object grasped at $a_{\mathrm{pick}}$ in scene $o_{t}$ should be
placed at $a_{\mathrm{place}}$.

Since the place action is conditioned on the pick action, a consistent place
$f_{\mathrm{place}}$ network should satisfy the following bi-equivariance
constraint. For $f\colon\mathrm{SO}(3)\to\mathbb{R}$, denote
$(\rho_{R}(g)f)(h)=f(hg^{-1})$ the right action on orientation distributions
corresponding to pre-rotation by $g$. Then $f_{\mathrm{place}}$ is
$\mathrm{SE}(d)\times\mathrm{SE}(d)$-equivariant:

$\forall g_{1},g_{2}\in\mathrm{SE}(d),\quad f_{\mathrm{place}}(g_{1}\cdot
c,g_{2}\cdot
o_{t})=\mathrm{Ind}_{\rho_{L}}(g_{2})\rho_{R}(g_{1}^{-1})f_{\mathrm{place}}(c,o_{t})$
(3)

Equation 3 states that if $g_{1}\in\mathrm{SE}(d)$ acts on the picked object
and $g_{2}\in\mathrm{SE}(d)$ acts on the observation $o_{t}$ 333Strictly,
$g_{2}$ acts on $o_{t}\backslash c$, the observation excluding the patch,
since the picked object is transformed by $g_{1}$. that contains the placement
of interest, the desired place action will transform accordingly. Recalling
$\mathrm{Ind}_{\rho_{L}}$ acts on the location and orientation, the place
orientation $R_{\mathrm{place}}$ will be transformed to
$\rho_{1}(g_{2})R_{\mathrm{place}}\rho_{1}(g_{1}^{-1})$, and the place
location $T_{\mathrm{place}}$ will be transformed to
$\rho_{1}(g_{2})T_{\mathrm{place}}$.

Realizing place symmetry: Theory. In order to design a network architecture
$f_{\mathrm{place}}$ which satisfies Equation 2, we follow the previous works
(Zeng et al., 2021; Huang et al., 2022) which treat the placing as a template
matching problem and encodes $f_{\mathrm{place}}$ with two separate functions
$\phi$ and $\kappa$ to process $o_{t}$ and $c$ respectively. Then
$p(a_{\mathrm{place}}|o_{t},a_{\mathrm{pick}})$ is computed as the cross-
correlation between $\kappa(c)$ and $\phi(o_{t})$,

$f_{\mathrm{place}}(c,o_{t})=\kappa(c)\star\phi(o_{t})$ (4)

where $\star$ denotes the cross correlation. Specifically,
$\kappa(c)\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d_{\mathrm{out}}\times
d_{\mathrm{in}}}$ is a dynamic kernel and
$\phi(o_{t})\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d_{\mathrm{in}}}$ is a
feature map generated from the workspace observation $o_{t}$.

###### Proposition 1

Equation 4 satisfies the Bi-Equivariant symmetry stated in Equation 3 if the
following constraints hold:

  1. 1.

$\kappa(c)\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d_{\mathrm{out}}\times
d_{\mathrm{trivial}}}$ is a _steerable_ convolutional kernel with _trivial_
type input

  2. 2.

$\kappa(c)$ satisfies the equivariant property that $\kappa(g\cdot
c)=\beta(g)\kappa(c)$

  3. 3.

$\phi(o_{t})$ satisfies the equivariant property that $\phi(g\cdot
o_{t})=\beta(g)\phi(o_{t})$

The detailed proofs of Proposition 1 are derived in Appendix A.10.
Intuitively, rotations on the crop $c$ and the placement $o_{t}$ result in
consistent transformations of the kernel $\kappa(c)$ and feature map
$\phi(o_{t})$. The steerability of the kernel bridges the two transformations
during the cross-correlation and is necessary for the correlation to be
equivariant to transformations of $o_{t}$.

Realizing Place Symmetry: Solution. Our proposed FourTran satisfies the three
constraints listed in Proposition 1. A schematic of our network architecture
is shown in Figure 2. In the top branch, we encode $\phi$ with equivariant
convolutional layers that take input observation $o_{t}$ as $\rho_{0}$ signal
and output $\rho_{0}$ dense feature map. It satisfies the third constraint of
Proposition 1. In the bottom branch, the crop is processed by $\psi$ that has
the same architecture as $\phi$ to generate a dense $\psi$ feature map for the
crop $c$. Then, we lift $\psi(c)$ with a finite number of rotations $\\{g_{i}\
|g_{i}\in\mathrm{SO}(d)\\}_{i=1}^{m}$ to generate a stack of rotated feature
maps. Specifically, we define $\mathcal{L}^{\uparrow\tilde{G}}$ as

$\displaystyle\forall
x\in\mathbb{R}^{n},\quad\mathcal{L}^{\uparrow}[f](x)=\\{{f(g^{-1}_{1}x)},{f(g^{-1}_{2}x)}\cdots,{f(g^{-1}_{m}x)}\\}$

that generates a stack of orbit-traversing signals above each pixel/voxel. We
then apply a Fourier Transform to the fiber-space feature to get the
$\mathrm{SO}(d)$ representation. Our dynamic kernel generator $\kappa$ is
summarized as

$\kappa(c)=\mathcal{F}^{+}[\mathcal{L}^{\uparrow}(\psi(c))]$ (5)

where $\mathcal{F}^{+}$ denotes the Fourier Transform in the fiber space.
Appendix.A.1 shows the pseudocode of the inference step of FourTran.

###### Proposition 2

Let $f:\mathbb{R}^{3}\rightarrow\mathbb{R}^{k}$ be an
$\mathrm{SO}(3)$-steerable feature field. Then, the lifting operator
$\mathcal{L}^{\uparrow I_{60}}(f)$ generates a $I_{60}$-steerable kernel with
regular-type output and trivial input. The fiber space Fourier transformation
$\mathcal{F}^{+}[\mathcal{L}^{\uparrow}(f)]$ is an approximately an
$SO(3)$-steerable kernel with trivial input and output type
$\bigoplus_{\ell=0}^{\ell_{max}}(2\ell+1)D^{\ell}$.

Proposition 2 states that $\kappa(c)$ in Equation 5 is a dynamic steerable
kernel generator, which satisfies the first constraint in Proposition 1. The
proof of Proposition 2 is derived in Appendix A.11. Finally, the cross-
correlation between $\kappa(c)$ and $\phi(o_{t})$ is performed in the Fourier
space.

Representing $\mathrm{SO}(d)$ rotations in Fourier space can achieve high
angular resolution and enormously save the computation load. In contrast,
directly cross-correlating between a number of rotated $\psi(c)$ and
$\phi(o_{t})$ independently is impossible when the action space contains a
large number of rotations, especially in 3D.

###  4.4 Sampling rotations in coarse-to-fine fashion

Coarse to fine sampling methods (James et al., 2022) are numerically efficient
importance sampling schemes that only sample in regions of dense signal. The
pick $f_{\mathrm{pick}}$ and place $f_{\mathrm{place}}$ network output the
$\mathrm{SO}(d)$ distribution for each pixel/voxel in the format of
coefficients of $\mathrm{SO}(d)$ basis functions. We can sample a finite
number of rotations for each element with inverse Fourier Transformation.
However, the memory requirements for the voxel grid are cubic in the
resolution of the grid and therefore limit the the number of sampled rotations
for each voxel. To solve it, we first sample a small number of rotations and
evaluate the best coarse pick action $(T_{\mathrm{pick}},R_{\mathrm{pick}})$
and the place action $(T_{\mathrm{place}},R_{\mathrm{place}})$. Then, we
locate the best pick and place location $T_{\mathrm{pick}}$ and
$T_{\mathrm{place}}$ and generate higher-order irrep signals with equivariant
layers from the corresponding hidden features. Finally, we conduct a second
sampling to generate a large number of fine $\mathrm{SO}(3)$ rotations for
$T_{\mathrm{pick}}$and $T_{\mathrm{place}}$ and calculate the best fine
orientation $R^{\star}_{\mathrm{pick}}$ and $R^{\star}_{\mathrm{pick}}$. The
robot will receive commands of
$(T_{\mathrm{pick}},R^{\star}_{\mathrm{pick}}),(T_{\mathrm{place}},R^{\star}_{\mathrm{place}})$
in the same time step and execute the actions.

##  5 Experiments

###  5.1 Model Architecture Details

In FourTran, $f_{\mathrm{pick}}$ is a single convolution network and
$f_{\mathrm{place}}$ is composed of two equivariant convolution networks,
$\phi$ and $\psi$. We implement them as an 18-layer residual network with a
U-Net (Ronneberger et al., 2015b) as the backbone. The U-Net has 8 residual
blocks and each block contains two equivariant convolution layers and one skip
connection. The first layer maps the trivial-representation $o_{t}$ to regular
representation, and the last equivariant layer transforms the regular-
representation feature to trivial representation for $\psi$ and $\phi$ and to
irreducible representation for $f_{\mathrm{pick}}$. We use pointwise ReLU
activations (Nair & Hinton, 2010) inside the network.

For 2D FourTran, we select the $C_{4}$ regular representation in the
intermediate layers of the three 2D convolution networks and use the $C_{90}$
group to lift $\psi(c)$. After the Fourier transform, the frequency is
truncated with the max order of 37. In our 3D convolution networks, we use
$O_{24}$ regular representation in the hidden layers and lift the 3D crop
features with a set of 384 approximately uniformly sampled rotations of
$\mathrm{SO}(3)$. We show that this random lifting approximately generates a
steerable kernel in the appendix B. To decode the _irrep_ feature, we first
sample 384 rotations for every voxel and conduct a fine sampling of 26244
rotations. The max order $\ell_{max}$ of the Winger-D matrices in the coarse-
to-fine sampling is $\ell_{max}=2$ and $\ell_{max}=4$, respectively.

###  5.2 3D Pick-Place

3D pick-place tasks are difficult due to the large observation and action
spaces. We conduct our primary experiments on five tasks shown in Figure 3
from RLbench (James et al., 2020) and compare with three strong baselines
(James et al., 2022; Shridhar et al., 2023; Goyal et al., 2023).

3D Task Description. We choose five the most difficult tasks from James et al.
(2020) to test our proposed method. Stack-blocks: The stack blocks task
consists of stacking two blocks of the red color on the green platform. Stack-
cups: In stack-cups, the agent must stack two blue cups on top of the red
color cup. Stack-wine: The agent must grab the wine bottle and put it on the
wooden rack at one of three specified locations. Place-cups: The agent must
place one mug on the mug holder. This is a very high precision task where the
handle of the mug has to be exactly aligned with the spoke of the holder for
the placement to succeed. Existing methods have trouble with this task. Put-
plate: The agent is asked to pick up the plate and insert it between the red
spokes on the dish rack. This is also a high-precision task. The different 3D
tasks are shown graphically in Figure 3. Note that the object poses are
randomly sampled at the beginning of each episode and the agent needs to learn
to generalize to novel object poses.

3D Baselines. Our method is compared against three state-of-the-art baselines:
C2FARM-BC (James et al., 2022) represents the scene with multi-resolution
voxels and infers the next key-frame action using a coarse-to-fine scheme.
PerAct (Shridhar et al., 2023) is the state-of-the-art multi-task behavior
cloning agent that tokenizes the voxel grids together with a language
description of the task and learns a language-conditioned policy with
Perceiver Transformer (Jaegle et al., 2021). RVT (Goyal et al., 2023) projects
the 3D observation onto five orthographic images and uses the dense feature
map of each image to generate 3D actions.

Training and Metrics. We train our method with $\\{1,5,10\\}$ demonstrations
and train the baselines with 10 demonstrations on each task individually. The
single-task versions of the baselines are denoted as ‘-Single’. All methods
are trained for 15K SGD steps, and we evaluate them on 25 unseen
configurations every 5K steps. Each evaluation is averaged over 3 evaluation
seeds, and we report the best evaluation across the training process. For the
favor of the baselines, we also include the results of multi-task versions of
the baselines trained on 16 different tasks of RLbench with 100 demonstrations
per task from Goyal et al. (2023), denotes as ‘-Multi’. To measure the effects
of discretization error and path planning, we also report the expert
performance in the discrete action space used by our method.

3D Results. We report the results of our method and the baselines in Table 1.
Several conclusions can be drawn from Table 1: 1). FourTran significantly
outperforms all baselines trained with 10 demos on all the tasks 2). For tasks
with a high-precision requirement, e.g., stack-cups, FourTran keeps a high
success rate while all the baselines fail to learn a good policy. 3). FourTran
achieves better sample efficiency and with $\\{1,5\\}$ demonstrations, it can
outperform baselines trained with hundreds of demonstrations.

Figure 3: 3D pick and place task descriptions. Leftmost: Stack-blocks task. Left-center: Stack-Cups task. Center: Stack-Wine task. Right-center: Place-Cups task. Rightmost: Put-Plate task  Model | # demos | stack-blocks | stack-cups | stack-wine | place-cups | put-plate 444This task is not covered in RVT (Goyal et al., 2023)
---|---|---|---|---|---|---
FourTran (ours) | 1 | 4 | 2.6 | 89.3 | 8 | 16
FourTran (ours) | 5 | 76 | 88 | 100 | 10.6 | 32
FourTran (ours) | 10 | 80 | 92 | 100 | 26.6 | 66.6
RVT-Single | 10 | 9.3 | 13.3 | 33.3 | 1.3 | 54.6
PerAct-Single | 10 | 52 | 1.3 | 12 | 1.3 | 8
C2FARM-BC-Single | 10 | 36 | 0 | 1.3 | 0 | 1.3
RVT-Multi | 100 | 28.8 | 26.4 | 91.0 | 4.0 | -
PerAct-Multi | 100 | 36 | 0 | 12 | 0 | -
C2FARM-BC-Multi | 100 | 0 | 0 | 8 | 0 | -
Discrete Expert | - | 100 | 92 | 100 | 90.6 | 65.3
Table 1: Performance comparisons on RL benchmark. Success rate (%) on 25 tests
v.s. the number of demonstration episodes (1, 5, 10) used in training. Even
with only 5 demos, our method is able to outperform existing baselines by a
significant margin.

###  5.3 2D Pick-Place

We further evaluate the ability of FourTran for solving precise pick-place
tasks in 2D where the action space is $(u,v,\theta)$, i.e., $x,y$ translation
and top-down rotation. We adopt three tasks shown in Figure 4 from Ravens
Benchmark (Zeng et al., 2021).

Figure 4: 2D pick and place task descriptions. Left: Block-insertion task.
Center: Assembling kits task. Right: Sweeping-piles task.

2D Task Description. block-insertion: The agent must pick up an L-shape block
and place it into an L-shaped fixture; assembling-kits: The agent needs to
pick 5 shaped objects (randomly sampled with replacement from a set of 20) and
fit them to corresponding silhouettes on a board. sweeping-piles: The agent
must push piles of small objects (randomly initialized) into a desired target
goal zone on the tabletop marked with green boundaries. Detailed task settings
and descriptions can be found in Appendix A.9.

2D Baselines. We compare our method against two strong baselines. Transporter
Net (Zeng et al., 2021) lifts the image crop $c$ with $C_{n}$ before feeding
them to $\psi$ network and encode $\phi$ and $\psi$ with ResNet-43.
Equivariant Transporter (Huang et al., 2022) is the variation of our proposed
method with the use of the _regular-type_ steerable kernel and achieves the
discretized $C_{n}\times C_{n}$ equivariance. Since tasks of Ravens Benchmark
uses a symmetric suction gripper, all outputs of picking angle $\theta$ are
equivalent. The picking angle $\theta$ is thus a nuisance variable and our
pick network outputs trivial-type features (instead of regular or higher-order
irreducible features).

Training and Metrics. We train each model with 10 expert demonstrations and
measure the performance with the two reward functions, the low-resolution
reward function credits agent for translation and rotation error relative to
target pose within $\tau=1\mathrm{cm}$ and $\omega=15^{\degree}$ and the high-
resolution reward function elevates the threshold to $\tau=0.5\mathrm{cm}$ and
$\omega=7.5^{\degree}$.

Results. Table 2 shows the performance of all models trained with 10
demonstrations for 10K steps. All tests are evaluated on 100 unseen
configurations. First, FourTran and Equivariant Transporter realize the Bi-
equivariance of and achieve a higher success rate than Transporter Net on
single-step task and multi-step task. Second, as the criteria increase from
$15^{\degree}$ rotation requirement to $7.5^{\degree}$ rotation requirement,
FourTran maintain the performance better than others. It indicates that the
$\mathrm{SO}(2)\times\mathrm{SO}(2)$ equivariance of FourTran is more precise.

| block-insertion-10 | assembling-kits-10 | sweeping-piles-10
---|---|---|---
Model | $15^{\degree}$ | $7.5^{\degree}$ | $15^{\degree}$ | $7.5^{\degree}$ |
FourTran (ours) | 100 | 100 | 86.2 | 78.0 | 99.8
Equivariant Transporter | 100 | 98.0 | 85.0 | 76.0 | 100
Transporter Net | 100 | 88.0 | 80.0 | 64.0 | 90.4
Table 2: Performance comparisons on 2D tasks. Success rate (%) on 100 tests.
Results for both low-resolution ($15^{\degree}$) reward and high-resolution
reward ($7.5^{\degree}$). Note that FourTran outperforms existing methods by a
larger margin on the higher resolution reward.

##  6 Conclusion

In this work, we propose the FourTran architecture for pick and place
problems. Similar to previous pick and place methods (Ryu et al., 2022; Huang
et al., 2023a), FourTran leverages the coupled
$\mathrm{SE}(d)\times\mathrm{SE}(d)$-symmetries inherent in pick-place tasks.
FourTran uses a novel fiber space Fourier transform method to construct a bi-
equivarient architecture in a memory efficient manner. The use of the Fourier
space convolution allows our architecture to process high resolution features
without need for extensive sampling. We evaluate our proposed architecture on
various tasks and empirically demonstrate that our method significantly
improves sample efficiency and success rate while lowering memory requirements
and needed pixel/voxel discritization. Specifically, in the two-dimensional
case, FourTran achieves slightly better performance than existing SOTA methods
(Huang et al., 2023a; Zeng et al., 2021) on the Ravens benchmark. For three-
dimensional pick and place, our method achieves SOTA results on a subset of
tasks in the James et al. (2019) benchmark by a significant (in some cases up
to two-hundred percent) margin. This empirical success establishes FourTran as
a powerful architecture for pick and place manipulation tasks.

One limitation of the formulation of manipulation tasks in this paper is that
it relies on open-loop control and does not take path planning and collision
awareness into account. Moreover, the current model is limited in a single-
task setting, and extending it to a multi-task, language-conditioned
equivariant agent is an important future direction. Lastly, this paper
considers only robotic manipulation problems, while the Bi-equivariant
architecture proposed here may have uses outside of robotic manipulation.
Specifically, various binding in biochemistry, like rigid protein-ligand
interaction (Ganea et al., 2022), have the same bi-equivariant symmetry as
pick-place problems.

### Disclosure of Funding

Haojie Huang, Xupeng Zhu, Dian Wang, Robin Walters and Rob Platt were
supported in part by NSF 1724257, NSF 1724191, NSF1763878, NSF 1750649, NSF
2107256, NSF 2134178 and NASA 80NSSC19K1474. Owen Howell was supported by the
NSF-GRFP. Dian Wang was funded by the JPMorgan Chase PhD fellowship.

## References

  * Ceccherini-Silberstein et al. (2008) T. Ceccherini-Silberstein, F. Scarabotti, and F. Tolli.  _Harmonic Analysis on Finite Groups: Representation Theory, Gelfand Pairs and Markov Chains_.  Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2008.  ISBN 9780521883368.  URL https://books.google.com/books?id=6qZUnwEACAAJ.
  * Cesa et al. (2021) Gabriele Cesa, Leon Lang, and Maurice Weiler.  A program to build e (n)-equivariant steerable cnns.  In _International Conference on Learning Representations_ , 2021.
  * Chun et al. (2023) Ethan Chun, Yilun Du, Anthony Simeonov, Tomas Lozano-Perez, and Leslie Kaelbling.  Local neural descriptor fields: Locally conditioned object representations for manipulation.  _arXiv preprint arXiv:2302.03573_ , 2023.
  * Cohen & Welling (2016) Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In _International conference on machine learning_ , pp. 2990–2999. PMLR, 2016.
  * Cohen & Welling (2017) Taco S Cohen and Max Welling.  Steerable cnns.  In _International Conference on Learning Representations_ , 2017.
  * Cohen et al. (2019) Taco S Cohen, Mario Geiger, and Maurice Weiler.  A general theory of equivariant cnns on homogeneous spaces.  _Advances in neural information processing systems_ , 32, 2019.
  * Deng et al. (2021) Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas.  Vector neurons: A general framework for so (3)-equivariant networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 12200–12209, 2021.
  * Deng et al. (2020) Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox.  Self-supervised 6d object pose estimation for robot manipulation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 3665–3671. IEEE, 2020.
  * Fuchs et al. (2020) Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling.  Se (3)-transformers: 3d roto-translation equivariant attention networks.  _Advances in Neural Information Processing Systems_ , 33:1970–1981, 2020.
  * Ganea et al. (2022) Octavian-Eugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause.  Independent se(3)-equivariant models for end-to-end rigid protein docking, 2022.
  * Goyal et al. (2023) Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox.  RVT: Robotic view transformer for 3d object manipulation.  In _7th Annual Conference on Robot Learning_ , 2023.  URL https://openreview.net/forum?id=0hPkttoGAf.
  * Howell et al. (2023) Owen Howell, David Klee, Ondrej Biza, Linfeng Zhao, and Robin Walters.  Equivariant single view pose prediction via induced and restricted representations, 2023.
  * Huang et al. (2022) Haojie Huang, Dian Wang, Robin Walters, and Robert Platt.  Equivariant Transporter Network.  In _Proceedings of Robotics: Science and Systems_ , New York City, NY, USA, June 2022.  doi: 10.15607/RSS.2022.XVIII.007.
  * Huang et al. (2023a) Haojie Huang, Dian Wang, Arsh Tangri, Robin Walters, and Robert Platt.  Leveraging symmetries in pick and place.  _arXiv preprint arXiv:2308.07948_ , 2023a.
  * Huang et al. (2023b) Zeyu Huang, Juzhan Xu, Sisi Dai, Kai Xu, Hao Zhang, Hui Huang, and Ruizhen Hu.  Nift: Neural interaction field and template for object manipulation.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1875–1881. IEEE, 2023b.
  * Jaegle et al. (2021) Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al.  Perceiver io: A general architecture for structured inputs & outputs.  _arXiv preprint arXiv:2107.14795_ , 2021.
  * James et al. (2019) Stephen James, Marc Freese, and Andrew J Davison.  Pyrep: Bringing v-rep to deep robot learning.  _arXiv preprint arXiv:1906.11176_ , 2019.
  * James et al. (2020) Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison.  Rlbench: The robot learning benchmark & learning environment.  _IEEE Robotics and Automation Letters_ , 2020.
  * James et al. (2022) Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J Davison.  Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13739–13748, 2022.
  * Jenner & Weiler (2021) Erik Jenner and Maurice Weiler.  Steerable partial differential operators for equivariant neural networks.  _arXiv preprint arXiv:2106.10163_ , 2021.
  * Jia et al. (2023) Mingxi Jia, Dian Wang, Guanang Su, David Klee, Xupeng Zhu, Robin Walters, and Robert Platt.  Seil: Simulation-augmented equivariant imitation learning.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1845–1851. IEEE, 2023.
  * Klee et al. (2023) David Klee, Ondrej Biza, Robert Platt, and Robin Walters.  Image to sphere: Learning equivariant features for efficient pose prediction.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=_2bDpAtr7PI.
  * Lang & Weiler (2021) Leon Lang and Maurice Weiler.  A wigner-eckart theorem for group equivariant convolution kernels, 2021\.
  * Lin et al. (2023) Yen-Chen Lin, Pete Florence, Andy Zeng, Jonathan T Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, and Phillip Isola.  Mira: Mental imagery for robotic affordances.  In _Conference on Robot Learning_ , pp. 1916–1927. PMLR, 2023.
  * Morrison et al. (2018) Douglas Morrison, Juxi Leitner, and Peter Corke.  Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach.  In _Proceedings of Robotics: Science and Systems_ , Pittsburgh, Pennsylvania, June 2018.  doi: 10.15607/RSS.2018.XIV.021.
  * Murphy et al. (2022) Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia.  Implicit-pdf: Non-parametric representation of probability distributions on the rotation manifold, 2022.
  * Nair & Hinton (2010) Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Icml_ , 2010.
  * Nguyen et al. (2023) Hai Huu Nguyen, Andrea Baisero, David Klee, Dian Wang, Robert Platt, and Christopher Amato.  Equivariant reinforcement learning under partial observability.  In _7th Annual Conference on Robot Learning_ , 2023.  URL https://openreview.net/forum?id=AnDDMQgM7-.
  * Pan et al. (2023) Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held.  Tax-pose: Task-specific cross-pose estimation for robot manipulation.  In _Conference on Robot Learning_ , pp. 1783–1792. PMLR, 2023.
  * Rohmer et al. (2013) Eric Rohmer, Surya PN Singh, and Marc Freese.  V-rep: A versatile and scalable robot simulation framework.  In _2013 IEEE/RSJ international conference on intelligent robots and systems_ , pp. 1321–1326. IEEE, 2013.
  * Ronneberger et al. (2015a) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015a.
  * Ronneberger et al. (2015b) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pp. 234–241. Springer, 2015b.
  * Ryu et al. (2022) Hyunwoo Ryu, Hong-in Lee, Jeong-Hoon Lee, and Jongeun Choi.  Equivariant descriptor fields: Se (3)-equivariant energy-based models for end-to-end visual robotic manipulation learning.  _arXiv preprint arXiv:2206.08321_ , 2022.
  * Seita et al. (2021) Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, and Andy Zeng.  Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 4568–4575. IEEE, 2021.
  * Shridhar et al. (2023) Mohit Shridhar, Lucas Manuelli, and Dieter Fox.  Perceiver-actor: A multi-task transformer for robotic manipulation.  In _Conference on Robot Learning_ , pp. 785–799. PMLR, 2023.
  * Simeonov et al. (2022) Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann.  Neural descriptor fields: Se (3)-equivariant object representations for manipulation.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pp. 6394–6400. IEEE, 2022.
  * Thomas et al. (2018) Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.  Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.  _arXiv preprint arXiv:1802.08219_ , 2018.
  * Wang et al. (2021) Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt.  Equivariant $Q$ learning in spatial action spaces.  In _5th Annual Conference on Robot Learning_ , 2021.
  * Wang et al. (2022a) Dian Wang, Mingxi Jia, Xupeng Zhu, Robin Walters, and Robert Platt.  On-robot learning with equivariant models.  In _6th Annual Conference on Robot Learning_ , 2022a.
  * Wang et al. (2022b) Dian Wang, Robin Walters, and Robert Platt.  $\mathrm{SO}(2)$-equivariant reinforcement learning.  In _The Tenth International Conference on Learning Representations_ , 2022b.
  * Weiler & Cesa (2019) Maurice Weiler and Gabriele Cesa.  General e (2)-equivariant steerable cnns.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Weiler et al. (2018) Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen.  3d steerable cnns: Learning rotationally equivariant features in volumetric data.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Wen et al. (2022) Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal.  You only demonstrate once: Category-level manipulation from single visual demonstration.  _arXiv preprint arXiv:2201.12716_ , 2022.
  * Zee (2016) A. Zee.  _Group Theory in a Nutshell for Physicists_.  In a Nutshell. Princeton University Press, 2016.  ISBN 9780691162690.  URL https://books.google.com/books?id=FWkujgEACAAJ.
  * Zeng et al. (2017) Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao.  Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge.  In _2017 IEEE international conference on robotics and automation (ICRA)_ , pp. 1386–1383. IEEE, 2017.
  * Zeng et al. (2018) Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al.  Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pp. 3750–3757. IEEE, 2018.
  * Zeng et al. (2021) Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al.  Transporter networks: Rearranging the visual world for robotic manipulation.  In _Conference on Robot Learning_ , pp. 726–747. PMLR, 2021.
  * Zhu et al. (2014) Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Daniilidis.  Single image 3d object detection and pose estimation for grasping.  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 3936–3943. IEEE, 2014.
  * Zhu et al. (2022) Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt.  Sample efficient grasp learning using equivariant models.  _Proceedings of Robotics: Science and Systems (RSS)_ , 2022.

##  Appendix A Appendix

###  A.1 Pseudocode

Algorithm 1 FourTran inference

1:Given observation $o_{t}$, pick neural network $f_{\text{pick}}$,
observation and crop encoder $\phi,\psi$, discretization of rotation
$\hat{\mathrm{SO}}(d)$:

2:if $\mathrm{SE}(d)$-equivariant pick then

3: Calculate pick logits in Fourier:
$\tilde{f}_{\mathrm{pick}}=f_{\mathrm{pick}}(o_{t})$

4: Calculate pick probability in Fourier:
$\tilde{p}(a_{\mathrm{pick}}|o_{t})=\mathrm{softmax}(\tilde{f}_{\mathrm{pick}})$

5: Calculate pick probability in spacial:
$p(a_{\mathrm{pick}}|o_{t})=\mathcal{F}^{-}[\tilde{p}(a_{\mathrm{pick}}|o_{t})]$

6: The best pick pose $\bar{a}_{\mathrm{pick}}$ can be evaluate by
$\bar{a}_{\mathrm{pick}}=\operatorname*{arg\,max}p(a_{\mathrm{pick}}|o_{t})$

7:end if

8:if $\mathrm{SE}(d)\times\mathrm{SE}(d)$-equivariant place then

9: Crop the observation at pick place:
$c=\text{crop}(o_{t},\bar{a}_{\mathrm{pick}})$

10: Encode observation and crop: $\phi(o_{t}),\psi(c)$

11: $\mathcal{L}^{\uparrow}(\phi(c))=[]$

12: for All $g\in\hat{\mathrm{SO}}(d)$ do

13: Lift the crop encoding: $\mathcal{L}^{\uparrow}(\phi(c))$ append
$f(g^{-1}x)$

14: end for

15: Calculate lifted crop enoding in Fourier domain:
$\kappa(c)=\mathcal{F}^{+}[\mathcal{L}^{\uparrow}(\psi(c))]$

16: Calculate place logits in fourier:
$\tilde{f}_{\mathrm{place}}=\kappa(c)\star\phi(o_{t})$

17: Calculate place probability in Fourier:
$\tilde{p}(a_{\mathrm{place}}|o_{t})=\mathrm{softmax}(\tilde{f}_{\mathrm{place}})$

18: Calculate place probability in spacial:
$p(a_{\mathrm{place}}|o_{t})=\mathcal{F}^{-}[\tilde{p}(a_{\mathrm{place}}|o_{t})]$

19: The best place pose $\bar{a}_{\mathrm{place}}$ can be evaluate by
$\bar{a}_{\mathrm{place}}=\operatorname*{arg\,max}p(a_{\mathrm{place}}|o_{t})$

20:end if

###  A.2 Mathematical Background

We establish some notation and review some elements of group theory and
representation theory. For a comprehensive review of representation theory,
please see Zee (2016); Ceccherini-Silberstein et al. (2008).

####  A.2.1 Group Theory

At a high level, a group is the mathematical description of a symmetry.
Formally, a group $G$ is a non-empty set combined with a associative binary
operation $\cdot:G\times G\rightarrow G$ that satisfies the following
properties

$\displaystyle\text{existence of identity: }e\in G,\text{ s.t. }\forall g\in
G,\enspace e\cdot g=g\cdot e=g$ $\displaystyle\text{existence of inverse:
}\forall g\in G,\exists g^{-1}\in G,\enspace g\cdot g^{-1}=g^{-1}\cdot g=e$

The identity element of any group $G$ will be denoted as $e$. Note that the
set consisting of just the identity element $e$ is a group.

####  A.2.2 Representation Theory

Let $V$ be a vector space over the field $\mathbb{C}$. A representation
$(\rho,V)$ of a group $G$ consists of $V$ and a group homomorphism
$\rho:G\rightarrow\text{Hom}[V,V]$. By definition, the $\rho$ map satisfies

$\displaystyle\forall g,g^{\prime}\in G,\enspace\forall v\in
V,\enspace\rho(g)\rho(g^{\prime})v=\rho(gg^{\prime})v$

Heuristically, a group can be thought of as the embedding of an group (which
is an abstract mathematical object) into a set of matrices. Two representation
$(\rho,V)$ and $(\sigma,W)$ are said to be equivalent representations if there
exists a invertable matrix $U$

$\displaystyle\forall g\in G,\quad U\rho(g)=\sigma(g)U$

The linear map $U$ is said to be a $G$-intertwiner of the $(\rho,V)$ and
$(\sigma,W)$ representations. A representation is said to be reducible if it
breaks into a direct sum of smaller representations. Specifically, a unitary
representation $\rho$ is reducible if there exists an unitary matrix $U$ such
that

$\displaystyle\forall g\in
G,\quad\rho(g)=U[\bigoplus_{i=1}^{k}\sigma_{i}(g)]U^{\dagger}$

where $k\geq 2$ and $\sigma_{i}$ are smaller irreducible representations of
$G$. The set of all non-equivalent representations of a group $G$ will be
denoted as $\hat{G}$. All representations of compact groups $G$ can be
decomposed into direct sums of irreducible representations. Specifically, if
$(\sigma,V)$ is a $G$-representation,

$\displaystyle(\sigma,V)=U[\bigoplus_{\rho\in\hat{G}}m^{\rho}_{\sigma}(\rho,V_{\rho})]U^{\dagger}$

where $U$ is a unitary matrix and the integers $m^{\rho}_{\sigma}$ denote the
number of copies of the irreducible $(\rho,V_{\rho})$ in the representation
$(\sigma,V)$.

####  A.2.3 Wigner $D$-Matrices

The fundamental representation of the rotation group in three-dimensions is
given by

$\displaystyle SO(3)=\\{\enspace R\enspace|\enspace R\in\mathbb{R}^{d\times
d},\enspace R^{T}R=\mathbb{I}_{3},\enspace\text{det}(R)=1\enspace\\}.$

It should be noted that a group is an abstract mathematical object and that
the standard parameterization is a _choice_. There are multiple non-equivalent
representations of the group $SO(3)$. Irreducible representations of $SO(3)$
are known as _Wigner_ $D$-matrices. The irreducible representations of
$\mathrm{SO}(3)$ are known as Wigner $D$-matrices $(D^{\ell},V^{\ell})$. The
Wigner $D$-matrix of order $\ell$ is a real representation of dimension
$(2\ell+1)\times(2\ell+1)$. Although Wigner $D$-matrices are difficult to
visualize, the $\ell=1$ representation is just the standard $3\times 3$ matrix
representation of $SO(3)$.

###  A.3 Peter-Weyl Theorem and Fiber Space Fourier Transforms

The Peter-Weyl theorem Ceccherini-Silberstein et al. (2008) states that all
representations of compact groups can be decomposed into a countably infinite
sets of irreducible representations. Consider the set of functions

$\displaystyle\mathcal{F}=\\{\enspace f\enspace|\enspace
f:G\rightarrow\mathbb{C}\enspace\\}$

of all complex valued function defined on $G$. The set $\mathcal{F}$ forms a
vector space over the field $\mathbb{C}$. The group $G$ acts on vector space
$\mathcal{F}$. Specifically, define the group action
$\lambda:G\times\mathcal{F}\rightarrow\mathcal{F}$ as

$\displaystyle\forall f\in\mathcal{F},\enspace\forall g,g^{\prime}\in
G,\quad(\lambda_{g}\cdot f)(g^{\prime})=f(g^{-1}g)\in\mathcal{F}$

The action satisfies $\lambda_{g}\lambda_{g^{\prime}}=\lambda_{gg^{\prime}}$
and is a group homeomorphism. The left-regular representation of a group is
defined as $(\lambda,\mathcal{F})$. The Peter-Weyl theorem Ceccherini-
Silberstein et al. (2008) states that

$\displaystyle(\lambda,\mathcal{F})=U[\bigoplus_{\rho\in\hat{G}}d_{\rho}(\rho,V_{\rho})]U^{\dagger}$

where $U$ is the unitary matrix. Thus, the left-regular representation
decomposes into $d_{\rho}$ copies of each $(\rho,V_{\rho})$ irreducible. In
other words, the Peter-Weyl theorem states that matrix elements of irreducible
$G$-representations form an orthonormal base of the space of square integrable
functions on $G$.

####  A.3.1 Irreducible Representation Orthogonality Relations

Matrix elements of irriducibles representations satisfy a set of orthogonality
relations Zee (2016). Specifically, let $\rho$ and $\sigma$ be irreducible
representations of the group $G$. Then,

$\displaystyle\sum_{g\in
G}\rho_{kk^{\prime}}(g)\sigma(g)_{nn^{\prime}}^{\dagger}=\frac{|G|}{d_{\rho}}\delta_{\rho,\sigma}\delta_{kn}\delta_{k^{\prime}n^{\prime}}$

where $|G|$ is the cardinality of the group. The orthogonality relations of
different irreducible representations is an analogue of harmonics of different
frequencies in Fourier analysis. As an example, consider the case with
$G=SO(2)$. Irreducible representations of $SO(2)$ are all one dimensional and
labeled by an integer. They are all of the form

$\displaystyle\rho_{k}(\phi)=\exp(ik\phi)$

The orthogonality relations applied to $G=SO(2)$ then state that

$\displaystyle\int_{0}^{2\pi}d\phi\rho_{k}(\phi)\rho(\phi)_{k^{\prime}}^{\dagger}=\int_{0}^{2\pi}d\phi\exp(i(k-k^{\prime})\phi)=2\pi\delta_{kk^{\prime}}$

which is the standard orthogonality relation for different Fourier harmonics.

###  A.4 Induced Representations

The induced representation is a way to construct representations of a larger
group $G$ out of representations of a subgroup $H\subseteq G$. Let $(\rho,V)$
be a representation of $H$. The induced representation of $(\rho,V)$ from $H$
to $G$ is denoted as $\mathrm{Ind}_{H}^{G}[(\rho,V)]$. Define the space of
functions

$\displaystyle\mathcal{F}=\\{\enspace f\enspace|\enspace f:G\rightarrow
V,\enspace\forall h\in H,\enspace f(gh)=\rho(h^{-1})f(g)\enspace\\}$

Then the induced representation is defined as
$(\pi,\mathcal{F})=\mathrm{Ind}_{H}^{G}[(\rho,V)]$ where the induced action
$\pi$ acts on the function space $\mathcal{F}$ via

$\displaystyle\forall g,g^{\prime}\in G,\enspace\forall
f\in\mathcal{F},\quad(\pi(g)\cdot f)(g^{\prime})=f(g^{-1}g^{\prime})$

The induced representation was originally used in Cohen & Welling (2017) to
design networks that are equivariant with respect to both rotations and
translations. Induced representations be used to change the underling group
equivariance Cesa et al. (2021); Weiler & Cesa (2019); Howell et al. (2023).
Specifically, induced representations can be used to design
$SE(3)$-equivariant maps from $SO(3)$-equivariant maps.

###  A.5 Training Details

We evaluate our method on both 2D and 3D manipulation pick-place tasks.
Specifically, we train a single-task policy for each task with a dataset of
$n$ experiment demonstrations. Each demonstration contains one or more
observation-action pairs
$(o_{t},\tilde{a}_{\mathrm{pick}}\tilde{a}_{\mathrm{place}})$, where
$\tilde{a}_{\mathrm{pick}}$ denotes the expert pick action and
$\tilde{a}_{\mathrm{place}}$ is the expert place action. We use expert actions
to generate one-hot maps as the ground-truth labels for our picking model and
placing model. Due to the computation load of equivariant convolutional layers
on 3D voxel grids, we slightly lift the third constraint of Proposition 1 by
encoding our $\phi$ with a traditional U-net (Ronneberger et al., 2015a).
U-net with the long skip connection also maintains a certain amount of the
$\rho_{0}\mapsto\rho_{0}$ equivariance. Both models are trained end to end
using a cross-entropy loss. The model is trained using the Adam optimizer with
fixed learning rate=$1e^{-4}$. We report the training time and GPU memory
requirement of 3D FourTran in Table 4.

###  A.6 Ablation Studies

We perform two ablation studies to explore the functionality of our proposed
architecture. We first replace the equivariant U-net of $\psi$ with a
traditional U-net. This modification reduces the architecture to satisfy the
second constraint in Proposition 1. The second ablated version of our model is
that we remove the lifting and Fourier transform and directly generate the
irreducible features for each element, i.e., the model is forced to learn the
third constraint of Proposition 1 without prior knowledge. Table 3 shows the
performances of all ablations. Comparing the first row with the second row, we
find that the results are consistent. We hypothesize that the resons are: 1)
traditional U-net with the skip connections also captures the
$\mathrm{trivial}\mapsto\mathrm{trival}$ equivariance and the equivariant
constraints of the equivariant layers with $O_{24}$ group in FourTran is not
strong. 2) Data augmentation is applied to both models to learn the
equivariance. Comparing the first row to the third row, the ablated model
attempting to learn the coefficients of the basis function is not as well as
our proposal to generate a dynamic steerable kernel. On the other hand, given
the fact that lifting with a fixed number of $\mathrm{SO}(3)$ rotations
approximates the steerability of a 3D kernel, this ablated version avoids
lifting to reduce the computation load.

Model | stack-blocks | stack-wine | phone-on-base | put-plate
---|---|---|---|---
FourTran (ours) | $\mathbf{76}$ | $\mathbf{100}$ | $\mathbf{96}$ | $\mathbf{32}$
Fourier Transporter w.o equiv. of $\psi$ | $\mathbf{76}$ | 96 | $\mathbf{96}$ | 24
Fourier Transporter w.o lifting | 72 | 88 | $\mathbf{96}$ | 28
Table 3: Ablation Study: Performance of three variants of FourTran on
different RLbench tasks James et al. (2019) Each model is trained with 5
demonstrations and evaluated on 25 tests. Best performances are highlighted in
bold.

###  A.7 Model Comparisons

We train a single-task agent for each 3D baseline on the five tasks shown in
Figure 3 with the same settings presented in their paper. Note that we have
$\\{$pre-pick, pick, post-pick$\\}$ and $\\{$pre-place, place, post-place$\\}$
where the pre-action and post-action are defined as relative to the pick and
place action while the baseline predicts the action for the next keyframe
without a clear line between pick and place.

Model | Parameters (M) | Memory(Gb) | Training Time (secs/sgd step)
---|---|---|---
FourTran-pick |  3.1 M |  8.5 | 1.5
FourTran-place |  1.6 M |  13.6 | 1.6
RVT |  36 M | 12.5 | 0.46
PerAct |  33 M | 12 | 1.5
C2FARM |  3.6 M | 2 | 0.07
Table 4: Comparison of FourTran (ours) architecture and existing pick-place
methods. Tests were performed on NVIDIA 3090 GPU.

Table 4 compares FourTran with the baseline architectures. Note that our model
has about the same number of parameters as C2FARM James et al. (2020)
significantly less parameters than the transformer based methods PerActJaegle
et al. (2021) (33 M) and RVTGoyal et al. (2023) (36 M).

###  A.8 Detailed 3D task settings and Metrics

The RLbench James et al. (2020) is implemented in CoppelaSim (Rohmer et al.,
2013) and interfaced through PyRep (James et al., 2019). All experiments use a
Franka Panda robot with a parallel gripper. The input observations are
captured from four RGB-D cameras. The captured point could is parameterized as
$72\times 96\times 56$ voxel grid and each voxel represents a $0.94$cm3 cube
in our settings.

Figure 5 shows the expert $\mathrm{SO}(3)$ rotation action distributions in
stack-wine and put-plate tasks. The agent needs to reason about the
$\mathrm{SO}(3)$ rotation space to finish the tasks. For more information of
plotting elements of $SO(3)$, please see Murphy et al. (2022).

Figure 5: Visualization of expert $\mathrm{SO}(3)$ actions from 10
demonstrations. First column: expert pick action. Second column: expert place
action. First row: stack-wine. Second row: put-plate. The orientation
visualization follows “YXY” convention. For more detail on plot formatting,
please see Murphy et al. (2022)

###  A.9 Detailed 2D task settings and metrics

At the beginning of each episode, the poses of objects and placements in each
task are randomly sampled in the workspace without collision. The visual
observation $o_{t}$ is a top-down projection of the workspace with 3 simulated
RGB-D cameras pointing towards the workspace. For two-dimensional simulations,
the pixel discretization is $320\times 160$ for the $1$m $\times$ $0.5$m
workspace in our model. We measure performance the same way as it was measured
in Transporter Net Zeng et al. (2021) – using a metric in the range of 0
(failure) to 100 (success). Partial scores are assigned to multiple-action
tasks. For example, in the assembling kit task where the agent needs to
assemble 5 objects, each successful rearrangement is credited with a score of
0.2. We report the highest validation performance during training, averaging
over 100 unseen tests for each task.

###  A.10 Proof of Proposition 1

To prove proposition 1, we begin with 2 lemma.

###### Lemma 1

A steerable kernel
$K\colon\mathbb{R}^{n}\rightarrow\mathbb{R}^{d_{\mathrm{out}}\times
d_{\mathrm{trivial}}}$ satisfies

$\beta(g)K(x)=\rho_{\mathrm{out}}(g^{-1})K(x)$ (6)

###### proof 1

Recall that $\rho_{0}(g)$ is an identity mapping. Substituting
$\rho_{\mathrm{in}}$ with $\rho_{0}(g)$ and $g^{-1}$ with $g$ in the
steerability constraint $K(g\cdot
x)=\rho_{\mathrm{out}}(g)K(x)\rho_{\mathrm{in}}(g)^{-1}$ completes the proof.

$\displaystyle\beta(g)$ $\displaystyle=K(g^{-1}x)$
$\displaystyle=\rho_{\mathrm{out}}(g^{-1})K(x)\rho_{\mathrm{in}}(g)$
$\displaystyle=\rho_{\mathrm{out}}(g^{-1})K(x)$

###### Lemma 2

cross correlation satisfies that

$\displaystyle(\beta(g)(K\star
f))(\vec{v})=((\beta(g)K)\star(\beta(g)f))(\vec{v})$ (7)

###### proof 2

We evaluate the left-hand side of Equation 7:

$\displaystyle\beta(g)(K\star f)(\vec{v})$
$\displaystyle=\sum_{\vec{w}\in\mathbb{Z}^{2}}f(g^{-1}\vec{v}+\vec{w})K(\vec{w}).$

Re-indexing the sum with $\vec{y}=g\vec{w}$,

$\displaystyle=\sum_{\vec{y}\in\mathbb{Z}^{2}}f(g^{-1}\vec{v}+g^{-1}\vec{y})K(g^{-1}\vec{y})$

is by definition

$\displaystyle=\sum_{\vec{y}\in\mathbb{Z}^{2}}(\beta(g)f)(\vec{v}+\vec{y})(T^{0}_{g}K)(\vec{y})$
$\displaystyle=((\beta(g)K)\star(\beta(g)f))(\vec{v})$

as desired.

We first prove that if there is a rotation $g$ acting on the observation
$o_{t}$, $\kappa(c)\ast\phi(g\cdot
o_{t})=\mathrm{Ind}_{\rho_{L}}(g)\kappa(c)\ast\phi(o_{t})$.

###### proof 3

$\displaystyle\kappa(c)\ast\phi(g\cdot o_{t})$
$\displaystyle=k(c)\ast(\beta(g)o_{t})\>\>\text{Equiv. of $\phi$}$
$\displaystyle=\beta(g)\beta(g^{-1})k(c)\ast\beta(g)\phi(o_{t})$
$\displaystyle=\beta(g)(\beta(g^{-1})k(c)\ast\phi(o_{t}))\>\>\text{lemma 2}$
$\displaystyle=\beta(g)(\rho(g)k(c)\ast\phi(o_{t})\>\>\text{lemma 1}$

Then, we prove that this a rotation $g$ acting on the crop $c$, $\kappa(g\cdot
c)\ast\phi(o_{t})={\rho}(g^{-1})\kappa(c)\ast\phi(o_{t})$

###### proof 4

$\displaystyle\kappa(\beta(g)c)\ast\phi(o_{t})$
$\displaystyle=\beta(g)\kappa(c)\ast\phi(o_{t})\>\>\text{equiv. of}\>\>\kappa$
$\displaystyle=\rho(g^{-1})\kappa(c)\ast\phi(o_{t})\>\>\text{lemma 1}$

Combining proof 4 and proof 3 finishes the proofs of proposition A.10.

###  A.11 Proof of Proposition 2

####  A.11.1 The $I_{60}$-Lifting Map is A Steerable Kernel with Trivial Input
Type and Regular Output Type

The lifting map $\mathcal{L}^{\uparrow}(f)$ generates a $I_{60}$-steerable
kernel with regular-type output. To see this, note that, by the definition of
$\mathcal{L}^{\uparrow}$,

$\displaystyle\forall
x\in\mathbb{R}^{d},\quad\mathcal{L}^{\uparrow}(f(x))=\\{g_{1}\cdot
f(x),g_{2}\cdot f(x),...,g_{m}\cdot f(x)\\}$

where each $g_{i}$ are elements of $I_{60}$. Thus, $\forall g\in I_{60}$, we
have that

$\displaystyle\mathcal{L}^{\uparrow}(f(g\cdot x))=\\{g_{1}\cdot
f(gx),g_{2}\cdot f(gx),...,g_{m}\cdot
f(gx)\\}=\\{f(gg^{-1}_{1}x),f(g^{-1}_{2}gx),...,f(g^{-1}_{m}gx)\\}$

Now, let $g$ be a fixed element of the icosahedral group. Left multiplication
by an element of $g$ is a group homeomorphism of $I_{60}$. Let us define
$g_{g(i)}$ to be the element of the sampled set $g_{i}$ so that

$\displaystyle g_{g(i)}=g\cdot g_{i}$

Let $1$ be the identity element on $SO(d)$. Then, note that

$\displaystyle g_{1(i)}=g_{i}$

The map $g_{g(i)}$ satisfies an additional composition property. Let
$g,g^{\prime}$ be elements of $SO(d)$, then,

$\displaystyle g_{(gg^{\prime})(i)}=g_{g(g^{\prime}(i))}$

holds exactly. Thus, the map $g(\cdot)\in$ The expression for
$\mathcal{L}^{\uparrow}(f)$ can be rewritten as

$\displaystyle\mathcal{L}^{\uparrow}(f(g\cdot
x))=\rho(g)\\{f(g^{-1}_{1}x),f(g^{-1}_{2}x)...f(g^{-1}_{m}x)\\}=\rho(g)\mathcal{L}^{\uparrow}(f(x))$

where the matrix $\rho$ has elements given by

$\displaystyle\rho(g)_{ij}=\delta_{ig(j)}$

Using $g_{1(i)}=g_{i}$, we have that the matrix $\rho$ satisfies,

$\displaystyle\rho(1)=\mathbb{I}_{60}$

where $\mathbb{I}_{60}$ is the identity matrix in $60$ dimensions. Using the
identity $g_{(gg^{\prime})(i)}=g_{g(g^{\prime}(i))}$ the matrix $\rho$
satisfies the relation

$\displaystyle\rho(g)\rho(g^{\prime})=\rho(gg^{\prime})$

which is the definition of a group representation. Thus, the matrix $\rho$ is
exactly the permutation representation of $I_{60}$. Thus, the lifting operator
$\mathcal{L}^{\uparrow}(f)$ is an $I_{60}$-steerable kernel with trivial input
type and regular output type.

####  A.11.2 Fiber Space Fourier Transform

Let us suppose that the map $\mathcal{L}^{\uparrow}(f)$ is exactly a
$I_{60}$-steerable kernel with trivial input type and regular output type. We
can then compute the fiber space $SO(3)$-Fourier transform of
$\mathcal{L}^{\uparrow}(f)$

$\displaystyle\rho_{out}(g)=\bigoplus_{\ell=0}^{\ell_{\max}}m_{\ell}D^{\ell}(g)$

where $D^{\ell}$ is the $\ell$-th irreducible of $SO(3)$.

Let $F^{+}$ be the Fourier transform in the fiber space given by

$\displaystyle
F^{+}(\mathcal{L}^{\uparrow}(f))=\sum_{\ell=0}^{\ell_{max}}C^{\ell}(x)D^{\ell}$

where the irreducible coefficients are given by the fiber space Fourier
transform

$\displaystyle C^{\ell}(x)=\int_{R\in SO(3)}dR\text{
}D^{\ell}(R^{-1})\rho(R)\mathcal{L}^{\uparrow}(f)(R^{-1}x)$

##  Appendix B The Stochastically Sampled Lifting Map is Approximately A
$SO(3)$-Steerable Kernel

The stochastically sampled lifting map $\mathcal{L}^{\uparrow}(f)$
approximately generates a steerable kernel with regular-type output. To see
this, note that, by the definition of $\mathcal{L}^{\uparrow}$,

$\displaystyle\forall
x\in\mathbb{R}^{n},\quad\mathcal{L}^{\uparrow}(f(x))=\\{g_{1}\cdot
f(x),g_{2}\cdot f(x),...,g_{m}\cdot f(x)\\}$

where each $g_{i}$ is sampled iid from $SO(d)$. Thus, $\forall g\in SO(d)$, we
have that

$\displaystyle\mathcal{L}^{\uparrow}(f(g\cdot x))=\\{g_{1}\cdot
f(gx),g_{2}\cdot f(gx),...,g_{m}\cdot
f(gx)\\}=\\{f(gg^{-1}_{1}x),f(g^{-1}_{2}gx),...,f(g^{-1}_{m}gx)\\}$

Now, let $g$ be a fixed element of $SO(d)$. Left multiplication by an element
of $g$ is a group homeomorphism of $SO(d)$. Let us define $g_{g(i)}$ to be the
closest element of the sampled set $g_{i}$ so that

$\displaystyle g_{g(i)}=\text{argmax}_{j=1,2,...m}||g\cdot g_{i}-g_{j}||$

where the norm is the geodesic distance on $SO(d)$. Let us assume that we work
in the regime where the number of samples $m$ is large and

$\displaystyle\text{max}_{j=1,2,...m}||g\cdot
g_{i}-g_{j}||\leq\frac{\epsilon}{m}$

For iid uniform $g_{i}$, this property holds generically for large values of
$m$. This can be proved rigorously using concentration of measure phenomena.

Let $1$ be the identity element on $SO(d)$. Then, note that

$\displaystyle g_{1(i)}=\text{argmax}_{j=1,2,...m}||g_{i}-g_{j}||=g_{i}$

The map $g_{g(i)}$ satisfies an additional composition property. Let
$g,g^{\prime}$ be elements of $SO(d)$, then,

$\displaystyle
g_{(gg^{\prime})(i)}=\text{argmax}_{j=1,2,...m}||(gg^{\prime})\cdot
g_{i}-g_{j}||=\text{argmax}_{j=1,2,...m}||g(g^{\prime}\cdot g_{i})-g_{j}||$

Thus,

$\displaystyle g_{(gg^{\prime})(i)}=g_{g(g^{\prime}(i))}$

holds approximately.

We may decompose the expression for $\mathcal{L}^{\uparrow}(f)(gx)$ as

$\displaystyle\mathcal{L}^{\uparrow}(f(g\cdot
x))=\\{f(g^{-1}_{g(1)}x),f(g^{-1}_{g(2)}x)...f(g^{-1}_{g(m)}x)\\}+\textbf{error}$

where the error term can be written as

$\displaystyle\textbf{error}=\\{f(g^{-1}_{1}gx)-f(g^{-1}_{g(1)}x),f(g^{-1}_{2}gx)-f(g^{-1}_{g(2)}x),...,f(g^{-1}_{m}gx)-f(g^{-1}_{g(m)}x)\\}$

if we assume that the function $f$ is $L$-Lipschitz continuous, then,

$\displaystyle||f(g^{-1}_{k}gx)-f(g^{-1}_{g(k)}x)||\leq
L||g^{-1}_{k}g-g^{-1}_{g(k)}||\leq\frac{L\epsilon}{m}$

and so as long as $m$ is large, the error term is small (roughly
$\mathcal{O}(\frac{L}{m}))$. Ignoring the error term, the expression for
$\mathcal{L}^{\uparrow}(f)$ can be rewritten as

$\displaystyle\mathcal{L}^{\uparrow}(f(g\cdot
x))=\rho(g)\\{f(g^{-1}_{1}x),f(g^{-1}_{2}x)...f(g^{-1}_{m}x)\\}=\rho(g)\mathcal{L}^{\uparrow}(f(x))$

where the matrix $\rho$ (which implicitly depends on the sampled $g_{i}$) has
elements given by

$\displaystyle\rho(g)_{ij}=\delta_{ig(j)}$

Using $g_{1(i)}=g_{i}$, we have that the matrix $\rho$ satisfies,

$\displaystyle\rho(1)=\mathbb{I}_{m}$

where $\mathbb{I}_{m}$ is the $m$-dimensional identify matrix. Using the
identity $g_{(gg^{\prime})(i)}=g_{g(g^{\prime}(i))}$ the matrix $\rho$
approximately satisfies the relation

$\displaystyle\rho(g)\rho(g^{\prime})=\rho(gg^{\prime})$

which is the definition of a group representation. Thus, up to approximation,
the matrix $\rho$ is the permutation representation of $SO(d)$. The $g_{i}$
are sampled at some numerical resolution with corresponding bandwidth
$\ell_{max}$. Ergo, for large $m$, the matrix $\rho$ is a good approximation
to the $SO(d)$ permutation representation at bandwidth $\ell_{max}$.

##  Appendix C  $SO(3)$-Fourier Transform

Let $f:SO(3)\rightarrow\mathbb{R}$ be a real valued function defined on
$SO(3)$. Then, by the Peter-Weyl theorem $f$ can be decomposed as

$\displaystyle\forall g\in SO(3),\quad
f(g)=\sum_{\ell=0}^{\infty}\sum_{k,k^{\prime}=-\ell}^{\ell}\hat{f}^{\ell}_{kk^{\prime}}D^{\ell}_{kk^{\prime}}(g)$

where $D^{\ell}$ are the Wigner $D$-matrices. Wigner $D$-matrices of order
$\ell$ are irreducible representations of dimension $2\ell+1$. The Fourier
transform over $\mathrm{SO}(3)$ is defined
$\mathcal{F}(f)=(\hat{f}^{\ell})_{\ell=0}^{\infty}$, where each of the
$\hat{f}^{\ell}$ is a matrix of size $(2\ell+1)\times(2\ell+1)$. The inverse
Fourier transform can be computed using the orthogonality of the Wigner
$D$-matrices,

$\displaystyle\hat{f}^{\ell}_{kk^{\prime}}=\int_{g\in SO(3)}dg\text{
}f(g)D^{\ell}_{kk^{\prime}}(g^{-1})$

where $dg$ denotes the $SO(3)$ Haar measure. In practice, we will truncate the
$\ell$ index at some maximum value $\ell_{\max}$.