# Numerical study of GCR proton transport

Yurovsky V.O  Faculty of Physics, MSU, Moscow, Russia  Peryatinskaya A. I
Faculty of Physics, MSU, Moscow, Russia  Kudryashov I.A  SINP MSU, Moscow,
Russia

###### Abstract

The paper numerically investigates the propagation of galactic cosmic rays in
a model magnetic field, which is a composition of an isotropic turbulent field
with the Kolmogorov turbulence spectrum and a regular constant field. The
dependence of the diffusion tensor components on the particle energy is
studied. It is shown that the transport anisotropy increases with decreasing
energy.

##  1 Introduction

It is well known that there are various inhomogeneities in the spectra of
Galactic Cosmic Rays (hereinafter GCR), such as a change in the amplitude and
phase rotation of the GCR dipole anisotropy in the energy range 10 TeV - 100
TeV [3], discovered by the NUCLEON[1] experiment. Also, this inhomogenety was
confirmed by other experiments, for example, such as CALET[2] and DAMPE[6],
the spectral inhomogeneity in the GCR energy spectrum in the close range of
100 TeV - 1 EeV. To explain these phenomena, a detailed understanding of the
mechanisms of GCR transport in this energy range is required. Existing
packages such as CRPROPA[4] were unable to solve this problem, which forced us
to create our own solution based on dynamic simulation of the magnetic field,
as in the papers, and particle propagation simulation using the Cash-Karp
method, which is used in CRPROPA [4].

##  2 Our model

###  2.1 Magnetic field

In the present work, we study the transport of particles in two types of
magnetic fields.

The first type of field is a random isotropic turbulent field, which is
defined as follows: the random field $\boldsymbol{b}$ is represented by the
sum of a finite number of several randomly generated modes.

$\boldsymbol{b}=\sum_{n}{A_{n}\boldsymbol{P}_{n}cos(\boldsymbol{r}\boldsymbol{k}_{n}+\phi_{n})}$
(1)

In the present work $N=500$. The same magnetic field model was studied, for
example, in [7]. The modes are generated with a logarithmic step in the
spatial size from 100 a.u. up to 100 pc, the correlation length is 20 parsecs.
The rms value of the turbulent field is 6 micro Gauss.

The second type of field is the sum of a field of the first type
$\boldsymbol{b}$ with a constant field $\boldsymbol{B}_{0}$, the value of
constant field is proportional to the rms value of the turbulent field. In
this work we study cases with $B_{0}=0;0.25;0.5;1\cdot b$.

$\boldsymbol{B}=\boldsymbol{B}_{0}+\boldsymbol{b},$ (2)

where $\boldsymbol{B}_{0}$ is the average value of the field over some area,
and $\boldsymbol{b}$ are the deviations of the field value from the average in
this area. The spectrum of the field is shown in Fig.1

Figure 1: Spectrum of the turbulent component of the field

Unlike CRPROPA[4], the field is not calculated in advance at the grid nodes,
and then interpolated, but dynamically calculated at each point of the
trajectory, which makes it possible to take into account small-scale
turbulence.

###  2.2 Propagation

In this work, the following equations are solved by the numerical method,
which describe the propagation of a particle in a magnetic field:

$\begin{cases}\frac{d\boldsymbol{r}}{dt}=\boldsymbol{v}\\\ \gamma
m\frac{d\boldsymbol{v}}{dt}=q[\boldsymbol{v}\times\boldsymbol{B}].\end{cases}$
(3)

We can equations 3 as:

$\begin{cases}\frac{d\boldsymbol{r}}{dt}=c\tilde{\boldsymbol{v}}\\\
\frac{d\tilde{\boldsymbol{v}}}{dt}=\frac{qc^{2}}{E}[\tilde{\boldsymbol{v}}\times\boldsymbol{B}],\end{cases}$
(4)

where E is the energy of the particle, and $\tilde{\boldsymbol{v}}$ is a unit
length vector directed along the particle’s velocity.

These equations are solved by the Cash-Karp method, one of the methods of the
Runge-Kutta family of the 4th order of accuracy. Methods from the Runge-Kutta
family are non-conservative. The modification of Newton’s equations shown
above is needed in order for the energy conservation law to be fulfilled in an
explicit form. The time step was chosen empirically to ensure correct modeling
of particle motion in a constant magnetic field. To check whether the step is
small enough in the case of the model field, we ran the simulation with a step
10 times larger than the chosen one and the result was statistically
indistinguishable from the initially chosen one. In particular, for a particle
energy of 100 TeV, steps of 100 AU were chosen. ( 1/30 gyroradius) and 1000
a.u. ( 1/3 gyroradius).

$dt=\frac{E\frac{au}{TeV}}{c}$ (5)

##  3 Calculation of diffusion coefficients

To calculate diffusion coefficients we averaged trajectories of particles in
three realizations of the magnetic field for each energy value. Each run was a
simulation of the propagation of 32 particles for a mileage of 800 kps. Each
attempt used a new random implementation of the turbulent field. The running
diffusion coefficient is defined as

$D_{ij}(l)=\frac{x_{i}(l)x_{j}(l)}{l/c}.$ (6)

From the results of the experiment, it can be seen that for sufficiently large
l it becomes constant, so that in the end we obtain simply the diffusion
coefficient as

$D_{ij}=\lim_{l\to\infty}D_{ij}(l).$ (7)

Figure 2: An example of obtaining a diffusion coefficient

##  4 Results

###  4.1 Magnetic Field Generation Check

The method of generation of the turbulent field described above was tested as
follows: histograms of $Bi$, $B_{i}B_{j}$ values were constructed for randomly
measured $10^{7}$ values. You can see the result in the figure 9.

This figure shows, that the distribution of the components is normal, and
there are also no correlations between the components, which can be seen from
the distributions of $B_{i}B_{j}$.

###  4.2 Field line diffusion

For the case of isotropic turbulence (without a regular field), the diffusion
coefficient of field lines was calculated as

$D^{f}_{ij}=\lim_{l\to\infty}\frac{x_{i}(l)x_{j}(l)}{l}.$ (8)

This parameter is important because, in the low-energy limit, the particles
are completely captured by the field (the leading center approximation is
applicable) and the entanglement of the field lines has the greatest effect on
particle transport. The simulation results are shown in fig. 3.

Figure 3: Values of diffusion coefficients of field lines for three
realizations of the field

The dimension of this coefficient is equal to the dimension of length. If we
want to compare this coefficient with the diffusion coefficient of particles,
then we can multiply by $c/2$, where $c$ is the speed of light. This
corresponds to the fact that the particles fly at the speed of light, and the
angle between the speed and the direction of the field line is uniformly
distributed. We get that $D_{ii}\approx 1e26m^{2}/s$.

###  4.3 Isotropic turbulence

The energy dependence of the diffusion coefficient for the case of isotropic
turbulence is shown in Fig. 4.

Figure 4: Averaged values of $Drr$ in the isotropic case. Green line shows
diffusion coefficient of field lines

It can be seen that $D_{rr}$ is a constant for low energies, and then begins
to increase as the energy increases. This can be explained as follows: when
the gyroradius of a particle in the mean field is small compared to the
characteristic radius of a magnetic tube (i.e., we can speak of field lines
and the guiding center approximation is applicable), the particles
predominantly propagate ballistically along the field lines of force, and
their spatial propagation is determined mainly by the diffusion of field
lines, which will be separately studied in subsequent papers. When the
gyroradius of the particle in the mean field approaches the characteristic
radius of the tube, the particle diffusion regime begins.

If we compare the value of the coefficient of the field line with the value
obtained from the dependence of the diffusion coefficient for particles in the
case of an isotropic turbulent field, we will see that the coefficient for
particles is an order of magnitude lower than for the field. This may be due
to the influence of magnetic traps.

###  4.4 Anisotropic turbulence

Recall that we studied cases with $B_{reg}/b_{turb}=0;0.25;0.5;1$.

On fig. 5 shows the energy dependence of the diffusion coefficient for the
cases $B_{reg}=0$ and $B_{reg}=b_{turb}$. There is also a second horizontal
axis on the chart. The values on it are $x=R(E)/L_{corr}$, where
$R(E)=\frac{p_{\perp}}{qB}=\frac{E}{qcb_{turb}}$ is the gyroradius of the
particle in the rms turbulent field, and $L_{corr}$ is the correlation length
of the magnetic field.

Figure 5: Average values of $Dzz$ for $B_{reg}=0;b_{turb}$

On fig. 6 the dependencies of $D_{zz}$ on energy are presented for all the
studied cases of the ratio $B_{reg}/b_{turb}$.

Figure 6: Average values of $Dzz$ for $B_{reg}=0;0.25\cdot b_{turb};0.5\cdot
b_{turb};b_{turb}$

This figure shows a general trend: $D_{zz}$ first falls, and the larger the
ratio $B_{reg}b_{turb}$, the steeper the slope. Then, when the graph
$D_{zz}(E)$ intersects the graph $D_{zz}(E)$ for the case $B_{reg}=0$, the
graph starts growing, and the growth of $D_{zz}$ does not depend on relations
$B_{reg}/b_{turb}$. Similar behavior was also observed for other magnetic
field models, for example, in [5].

###  4.5 Perpendicular transport

On fig. 7 the dependences of the diffusion coefficient in the plane
perpendicular to the direction of the constant magnetic field are shown, which
are fixed to 2: $D_{rr}=\frac{Dxx+Dyy}{2}$. The normalization by 2 is needed
in order for $D_{rr}$ to reflect the displacement projection onto some vector
(in the XY plane), which, in turn, is necessary for the correct comparison of
$Dzz$ and $Drr$ values.

Figure 7: Average $Drr$

It can be seen that the behavior of $D_{rr}$ is even more complicated than
that of $Dzz$. For low and high energies, this is a constant (except for the
case of $B_{reg}=0$), and only in a rather narrow energy range is there some
increase, the larger, the smaller the ratio $B_{reg}/b_{turb}$. The
explanation of this phenomenon requires further research.

###  4.6 Transport anisotropy

On fig. 8 the energy dependence of the anisotropy of the diffusion tensor
$a=\frac{D_{zz}}{D_{rr}}$ for the case $B_{reg}=b_{turb}$ is plotted.

Figure 8: Energy dependence of diffusion tensor anisotropy at
$B_{reg}=b_{turb}$

This figure shows, that the anisotropy is the greater, the larger the ratio
$B_{reg}/B_{turb}$. It is also seen that the anisotropy is always > 1, which
means that in the presence of a constant field, diffusion is anisotropic.

##  5 Conclusion

The lower the energy of the particle, the more the particles are entrained by
the magnetic field. For low energies, the propagation of particles is caused
by the superposition of one-dimensional motion of particles along the magnetic
field lines and the diffusion of the magnetic lines themselves. The study of
magnetic line diffusion and pitch-angle diffusion will be covered in future
articles.

The diffusion tensor is anisotropic at any energy and for all
$B_{reg}/b_{turb}$ ratios considered, but the observed anisotropy at a point
is determined by the cosmic ray concentration gradient only in those cases
where the particle does not move along the field line. In the same case, if
the particle is captured by the field, then the observed anisotropy at the
point shows the direction of the field line of force passing through the point
of observation. This issue will be considered in more detail in future works.

Figure 9: Histograms of magnetic field values. A. $B_{x}$ B. $B_{y}$ C.
$B_{z}$ D. $B_{x}^{2}$ E. $B_{y}^{2}$ F. $B_{z}^{2}$ G. $B_{x}B_{y}$ H.
$B_{x}B_{z}$ K. $B_{y}B_{z}$

## References

  * [1] V. L. Bulatov et al A. D. Panov, E. V. Atkin.  Review of the results from the nucleon space experiment.  Bulletin of the Russian Academy of Sciences: Physics, 83(8):980–982, 2019.
  * [2] Asano K. et al Adriani O., Akaike Y.  Direct measurement of the cosmic-ray proton spectrum from 50 gev to 10 tev with the calorimetric electron telescope on the international space station.  Phys. Rev. Lett., 122:181102, May 2019.
  * [3] Markus Ahlers.  The dipole anisotropy of galactic cosmic rays.  Journal of Physics: Conference Series, 1181:012004, feb 2019.
  * [4] Dundovic et al Alves Batista, Rafael.  CRPropa 3 - a Public Astrophysical Simulation Framework for Propagating Extraterrestrial Ultra-High Energy Particles.  JCAP, 05:038, 2016.
  * [5] Julián Candia and Esteban Roulet.  Diffusion and drift of cosmic rays in highly turbulent magnetic fields.  Journal of Cosmology and Astroparticle Physics, 2004(10):007–007, oct 2004.
  * [6] DAMPE Collaboration et al.  Measurement of the cosmic ray proton spectrum from 40 gev to 100 tev with the dampe satellite.  Science Advances, 5(9):eaax3793, 2019.
  * [7] R. C. Tautz and A. Dosch.  On numerical turbulence generation for test-particle simulations.  Physics of Plasmas, 20(2):022302, feb 2013.


#  Superfluid vacuum theory and deformed dispersion relations

Konstantin G. Zloshchastiev  Institute of Systems Science, Durban University
of Technology, Durban 4000, South Africa

(received: 19 Sep 2019 [WSPC]; published: 31 Jan 2019 [IJMPA])

###### Abstract

Using the logarithmic superfluid model of physical vacuum, one can formulate a
quantum theory, which successfully recovers Einstein’s theory of relativity in
low-momenta limit, but otherwise has different foundations and predictions. We
present an analytical example of the dispersion relation and argue that it
should have a Landau “roton” form which ensures the suppression of dissipative
fluctuations. We show that at small momenta, a dispersion relation becomes
relativistic with small deformations, such that a photon acquires effective
mass, but a much more complex picture arises at large momenta.

###### pacs:

03.75.Kk, 47.37.+q, 47.55.nb, 64.70.Tg

Keywords: superfluid vacuum, logarithmic superfluid, photon dispersion
relation

††preprint: Int. J. Mod. Phys. A 35, 2040032 (2020) [DOI:
10.1142/S0217751X20400321]

It is a general consensus that physical vacuum is a nontrivial dynamical
matter underlying all phenomena. Its dynamics and structure are a subject of
intensive studies and debates based on different views and approaches, which
generally agree on a main paradigm of physical vacuum being a quantum
object,dir51  but differ in details.volbook ; huabook  One of such views is
superfluid vacuum theory (SVT) – a post-relativistic approach in high-energy
physics and gravity, which advocates that physical vacuum is a superfluid and
elementary particles are excitations thereof. The term ‘post-relativistic’
implies that SVT can generally be a non-relativistic theory, but contains
relativity as a subset, special case or limit with respect to some dynamical
value, thus fulfilling the correspondence principle. The superfluid itself is
usually understood as a non-relativistic quantum liquid with suppressed
dissipative fluctuations, thus resulting in zero macroscopic viscosity.z12eb
Together with wavefunction’s isotropy, this would make such a vacuum
insensitive to the Michelson-Morley-type experiments, thus different from the
classical aether which theory was abandoned long ago.

One can describe quantum liquid by a condensate wavefunction $\Psi$ obeying a
nonlinear wave equation, which can be chosen to have a minimal
$U(1)$-symmetric form:
$i\partial_{t}\Psi=\left[-\frac{\hbar}{2m}\nabla^{2}+V_{\text{ext}}(\textbf{x},t)-F(|\Psi|^{2})\right]\Psi$,
where $m$ is the mass of a constituent particle, $F(\rho)$ is a differentiable
function on a positive semi-axis $\rho$, and $V_{\text{ext}}(\textbf{x},t)$ is
an external potential representing a trapping potential or container (we shall
neglect it in what follows). The condensate wavefunction must be normalized:
$\int_{\cal V}|\Psi|^{2}d{\cal V}=\int_{\cal V}\rho\,d{\cal V}={\cal M}>0$,
where ${\cal M}$ and ${\cal V}$ are the total mass and volume of the liquid.

SVT assumes that physical vacuum is described by a similar equation, while
photon-like excitations are analogous to acoustic waves in superfluid which
propagate with the velocity $c_{s}\propto\sqrt{p^{\prime}(\rho)}$, where prime
denotes a derivative. The correspondence principle requires that in low-
momenta limit, SVT must recover Einstein’s theory of relativity. One of
postulates of the latter implies that the speed of photon-like excitations of
vacuum should not depend on density, at least in a leading order with respect
to $\hbar$. At low momenta, $c_{s}$ tends to $c_{0}\approx c$, where
$c=2.9979\times 10^{10}$ $\text{cm}\,\text{s}^{-1}$ is historically called the
speed of light in vacuum. As shown in Ref. z11appb, , this results in the
following equation
$\rho|F^{\prime}(\rho)|=mc_{s}^{2}/\hbar\approx\text{const}(\rho)$, where
$\text{const}(\rho)$ is a function which does not dependent on density. The
solution of this differential equation is a logarithmic function:
$F(\rho)=b\ln{(\rho/\bar{\rho})}$, where $b$ and $\bar{\rho}$ are real
parameters. The wave equation thus becomes

$i\partial_{t}\Psi=\left[-\frac{\hbar}{2m}\nabla^{2}+V_{\text{ext}}(\textbf{x},t)-b\ln{(|\Psi|^{2}/\bar{\rho})}\right]\Psi.$
(1)

This introduces a model with logarithmic nonlinearity as a further development
and generalization of the non-perturbative theory of quantum gravity with non-
exact Lorentz symmetry. It tremendously facilitates analytical studies that
the ground state solution for positive values of $b$ was known, though for
purposes other than ours, since the works of Rosen and Bialynicki-Birula and
Mycielski.ros68 ; bb76  In Cartesian coordinates, it is a Gaussian wave packet
modulated by the de Broglie plane wave; in a rotationally symmetric 3D case,
it is a Gaussian function of a radius-vector.

A theory of superfluid vacuum can be formulated in different ways. The first
way is to treat vacuum effects as a small perturbation of a Lorentz-covariant
theory.z10gc ; z11pla  The second, more recent, approach is to consider
particle-like excitations within the framework of a post-relativistic
superfluid vacuum theory where spacetime is an induced phenomenon, as
discussed above. Correspondingly, dispersion relations turn out to be
different in each approach. For the former approach, the relations were
derived in Sec. 4 of Ref. z10gc, , therefore, we do not consider them here; we
just mention that their energy has a non-polynomial dependence on momentum.
This alerts us that a perturbative series cannot provide a full description of
vacuum, therefore the latter requires non-perturbative treatment.

(a) $E_{p}/E_{a}$ (b) $c^{2}_{p}/c^{2}_{0}$

Figure 1: Profiles of the energy (2) and corresponding velocity squared,
versus $p/p_{a}$, for the following values of $\eta$: $1$ (dashed curve),
$11.43$ (dash-dotted), $20.66$ (solid), $25.98$ (dash-double-dotted).

The dispersion relations for the second approach can be introduced by analogy
with conventional superfluids. In superfluid 4He, energy as a function of
momentum has a distinct shape, usually dubbed the Landau “roton” form. As the
excitation’s momentum increases, this energy grows from zero, until it reaches
a local maximum (called the maxon peak), which is crucial for suppressing
dissipative fluctuations.z12eb  Then it dives down to a local minimum (called
the “roton” minimum, for historical reasons), and eventually it starts growing
again, all the way to a border of a theory’s applicability domain, akin to a
solid curve in Fig. 1a. In the regime of small momenta, the dispersion
relation is approximately linear with respect to momentum, which is a
behaviour typical for relativistic particles.

We thus assume that the physical vacuum has a similar energy spectrum of
excitations - with its own values of parameters of course. It also means that
one can use the relativistic approach (with perhaps some tweaking by adding
additional fields) for a large range of energies - until momentum reaches a
value corresponding to the maxon peak, which can be rather large, up to a
Planck scale. These intermediate models can still provide valuable
understanding about various fundamental phenomena, such as the mass generation
mechanism and non-zero extent of particles.z11appb ; az11 ; dz11 ; dmz15
However, conceptually new physics steps in near and above the maxon threshold;
vacuum Cherenkov radiation and luminal boom being examples thereof.z11pla

An analytical example of a dispersion relation, which has a Landau form and
correct proportions for local extrema, is given by the formula:

$E_{p}=\frac{p^{2}}{2M_{a}}\sqrt{1+\eta\,f\\!\left(\frac{p}{p_{a}}\right)},\ \
f(k)\equiv\frac{1}{2k^{2}}\left[1-\aleph\,\text{e}^{-k^{2}}+\left(\frac{1}{k}-2k\right)D_{+}(k)\right]\\!,$
(2)

where $D_{+}(k)$ is a Dawson function,
$\aleph=\sqrt{\pi}/2+1/(\text{e}\,\text{erf}(1))\approx 1.32$,
$p_{a}=2\hbar/a$ is de Broglie momentum corresponding to the length scale
$a=\hbar/\sqrt{2m|b|}$, and $M_{a}=p_{a}^{2}/2E_{a}\propto m$. Here, $p_{a}$,
$E_{a}$ and $M_{a}$ set the momentum, energy and mass scales of the theory,
whereas the parameter $\eta$ controls local extrema of the dispersion curve:
one can check that $E_{p}$ acquires the Landau form, cf. a solid curve in Fig.
1a, only if $11.43<\eta<25.98$.

Furthermore, Fig. 1b displays the excitations’ velocity
$c_{p}\equiv\sqrt{dE_{p}/dp}$ divided by its low-momentum value
$c_{0}=\lim\limits_{p\to 0}c_{p}$. As in Fig. 1a, it is the solid curve which
is of interest to us. One can see that at small momenta, its behaviour follows
the relativistic pattern, but becomes a rather nontrivial as $p$ grows.
Initially it decreases with increasing $p$, which corresponds to a photon’s
slow-down and thus it can be interpreted as a photon acquiring the effective
mass $\mu$ – when expanded in Taylor series at small $p$, Eq. (2) expectantly
yields a deformed-relativistic dispersion:

$E_{p}^{2}=c_{0}^{2}p^{2}+\mu^{2}(p)\,c_{0}^{4},\ \
\mu(p)\equiv\left\\{A_{4}(p/p_{a})^{4}+A_{6}(p/p_{a})^{6}+{\cal
O}\\!\left[(p/p_{a})^{8}\right]\right\\}^{1/2},$ (3)

where $c_{0}=(p_{a}/2M_{a})\sqrt{\eta\,(1-\aleph/2)}$,
$A_{4}=(p_{a}/c_{0})^{2}(\aleph-8/3+2/\eta)/(2-\aleph)$,
$A_{6}=(p_{a}/c_{0})^{2}(8/5-\aleph/2)/(2-\aleph)$, and ${\cal O}[k^{n}]$
denotes terms of order $k^{n}$ and above.

Note that this effective mass generation mechanism is different from the gap
mechanism in conventional quantum condensed matter such as superconductors: it
is a post-relativistic effect and it happens before one reaches any of the
local extrema. Since $c_{0}\approx c$, the following experimental constraint
applies to our parameters:
$aM_{a}/\sqrt{\eta}=(\hbar/c_{0})\sqrt{1-\aleph/2}\approx 2.04\times
10^{-38}\,\text{g}\,\text{cm}$, whereas $A_{6}$ does not explicitly depend on
$\eta$.

A series (3) converges only at $p<p_{a}$: in the vicinity of local extrema
$p\sim p_{a}$, not to mention at $p\to\infty$, a formula changes drastically,
which reaffirms necessity of non-perturbative treatment. As momentum grows,
energy reaches first the maxon peak which marks the beginning of a region in
which $c_{p}^{2}$ becomes negative thus indicating absence of any classical
propagation. Starting from that region, classical 4D spacetime is no longer a
robust description, and the definition of a particle can no longer be based on
irreducible representations of a Poincarè group. This region extends until
object’s momentum approaches a value corresponding to the “roton” minimum,
after which the maximum attainable velocity can eventually go above $c_{0}$.
This is where the luminal boom occurs, which is the vacuum analogue of sonic
boom in air.z11pla  Beyond that value, breakdown of the quantum liquid occurs:
a moving object or observer no longer experiences vacuum condensate nor
induced spacetime.

###### Acknowledgements.

This work is supported by Department of Higher Education and Training of South
Africa and in part by National Research Foundation of South Africa.
Proofreading of the manuscript by P. Stannard is greatly appreciated.

## References

  * (1) P. A. M. Dirac, Nature 168, 906 (1951).
  * (2) G. E. Volovik, The Universe in a Helium Droplet (Oxford University Press, Oxford, 2009).
  * (3) K. Huang, A Superfluid Universe (World Scientific, Singapore, 2016).
  * (4) K. G. Zloshchastiev, Eur. Phys. J. B 85, 273 (2012).
  * (5) K. G. Zloshchastiev, Acta Phys. Polon. 42, 261 (2011).
  * (6) G. Rosen, J. Math. Phys. 9, 996 (1968).
  * (7) I. Bialynicki-Birula and J. Mycielski, Ann. Phys. (N. Y.) 100, 62 (1976).
  * (8) K. G. Zloshchastiev, Grav. Cosmol. 16, 288 (2010).
  * (9) K. G. Zloshchastiev, Phys. Lett. A 375, 2305 (2011).
  * (10) A. V. Avdeenkov and K. G. Zloshchastiev, J. Phys. B: At. Mol. Opt. Phys. 44, 195303 (2011).
  * (11) V. Dzhunushaliev and K. G. Zloshchastiev, Central Eur. J. Phys. 11, 325 (2013).
  * (12) V. Dzhunushaliev, A. Makhmudov and K. G. Zloshchastiev, Phys. Rev. D 94, 096012 (2016).


# Amplifying the kinematics of origami mechanisms with spring joints

###### Abstract

Due to its rigid foldability and predictable kinematics, the reverse fold is
the fundamental mechanism behind some of the most well known origami kinematic
structures, including the Miura Ori, Yoshimura, and waterbomb patterns.
However, the reverse fold only has one parameter to control its behavior: the
starting fold angle. In this paper I provide an alternative to the traditional
reverse fold—based on the spring-into-action pattern—called the spring joint.
This novel rigidly foldable mechanism is able to couple multiple reverse folds
into a compact space to amplify the kinematic output of a traditional reverse
fold by up to ten times, and to add one parameter for each reverse fold,
giving more programmatic control of origami structures. Methods of
parameterizing both the starting angle, the path of travel, and the axis of
motion are also introduced such that the spring joint can be engineered to any
application within compliant mechanisms, deployable structures and robotics.
Unfortunately, this versatility comes at the cost of a large buildup of
layers, making the spring joint impractical for thick origami mechanisms. To
solve this problem, I also introduce a modular alternative to the spring joint
that has no additional layers, with the same kinematic properties. Both of
these mechanisms are tested as replacements for the reverse fold in both
traditional and custom origami structures.

##  1 Introduction

Figure 1: An illustration of the spring joint mechanism when folded from a
square of paper

Origami is a vast field of study, from hyper realistic figurative models to
fractal tessellations [Ikegami 23]. Where origami most closely overlaps with
fields of engineering and robotics is in the area of pleat based tessellations
known as corrugations. Examples of pleat based origami corrugations that have
been used in engineering are the Miura Ori, Yoshimura, Chicken-wire, and
Waterbomb patterns. Origami corrugations have proved incredibly useful because
they offer a nonrestrictive method of deploying surfaces from compact states
to large surfaces. This was the original motivation for the Miura Ori as
described in [Koryo 85]. Pleats are also an effective way to transmit
mechanical information via the pleat dihedral angle. This is used to
facilitate the simultaneous motion of large deployable origami structures.
These properties of pleat based origami make it ripe for new innovations and
customized designs. However, there has been very little in terms of new
applied designs within origami corrugations. Furthermore, the patterns
mentioned earlier in this introduction are virtually the only corrugation
patterns with applications in engineering, are the only templates loaded into
the design tool Crane [Suto et al. 23], and are–the Miura Ori and Waterbomb
fold especially–the only patterns applied to research in aforementioned
neighboring fields [Lee et al. 21][Song et al. 14].

The greatest innovations, then, within pleat based origami have related to the
parameterization of existing designs. Tachi [Tachi 10] created an algorithm
for adapting corrugations to any three dimensional mesh when deployed.
Gardiner et al., [Gardiner et al. 18] developed a design process for adapting
in context applications of periodic tessellations. What all of these
approaches have in common is that they take existing origami folds and greatly
expand their potential for application by adding parameters and program-
ability. What all of the ubiquitous designs mentioned have in common is that
they are all composed of reverse folds, the most basic unit of corrugations.
Reverse folds are simple, but have limited parameters for their rigid motion.
The spring joint mechanism (figure 1) described in this paper offers a
parameterized substitution to the reverse fold. The spring joint can amplify
the motion of origami mechanisms, and has the parameters such that custom
paths of motion can be programmed into the model. Since it is pleat based, it
can be integrated to share the dihedral angle motion of any corrugation.
Variations and generalizations are introduced such that the spring joint can
be applied to real world thick origami applications.

##  2 Construction

In this section, I present methods of design and construction based on the
intuition behind the kinematics of reverse folds. The term spring joint
originates from the Spring-Into-Action model by Jeff Beynon, a model that has
been used in origami robotics applications [Hu et al. 20]. The units described
in this paper are a corrugation based generalization of the Spring-Into-
Action. The closest to my implementation of this fold, and an inspiration for
this paper, is the Squishy Spring-Into-Action Tessellation Tower by Jeremy
Shafer[Shafer 13]. The unit that is used in this model can be generalized as a
Spring Joint, and the model in this video is similar to the Miura-Ori Pattern
with spring joints replacing the reverse folds.

###  2.1 Reverse Fold Kinematics

For the purposes of this paper, the kinematics of reverse folds will be
described with an isotropic generalization, as described in [Tachi 09]. It is
isotropic because the degree 4 vertex is parallel along the X axis. Under this
framework, the angle of the reverse fold $\phi$ can be modeled as a function
of the dihedral fold angle $\xi$, where $\phi_{0}$ is the initial fold angle
as specified by the crease pattern:

$\phi(\xi,\phi_{0})=2arctan\left(cos\frac{\xi}{2}tan\frac{\phi_{0}}{2}\right)$
(1)

Reverse folds can be constructed in horizontal sequence, simply by connecting
two parallel reverse folds with an opposite dihedral angle in between. Any
combination of congruent reverse folds will have one degree of freedom, and
the kinematic relation continues. The fold angle of the series of reverse
folds will have the same kinematic relation between the reverse fold angle and
the collective dihedral angles.

In this model, graphed in figure 2, there is a trade off between the travel
distance—with higher values of $\phi_{0}$ traveling further to reach the
unfolded state—and the linearity of the relationship between the two
variables—with smaller values of $\phi_{0}$ having a more uniform relationship
between $\xi$ and $\phi$. Two things should be noted from this visualization:
1. If the reverse fold is reversed, so that all mountain folds become valleys
and vice versa, the kinematics will be reversed, and 2. If you take the limit
as $\phi_{0}$ approaches $\pi$, or the maximum fold angle, the fold angle
$\phi$ no longer has any relation to the dihedral angle $\xi$. These folds
will be called $\pi$ folds.

(a)

(b)

$\frac{\pi}{4}$$\frac{\pi}{2}$$\frac{3\pi}{4}$$\pi$$\frac{\pi}{4}$$\frac{\pi}{2}$$\frac{3\pi}{4}$$\frac{9\pi}{10}$$\pi$Dihedral
Angle $\xi$Fold Angle $\phi$ (c)

Figure 2: (a) Reverse fold crease pattern (b) Reverse fold folded state (c)
Graph of the fold angle $\phi$ as a function of the dihedral angle $\xi$ for
different values of the starting fold angle.

###  2.2 Compound Reverse Folds

The goal of a spring joint is to combine the kinematics of multiple reverse
folds. This will amplify the change in the fold angle and increase the design
flexibility by increasing the number of variables that can be adjusted.
However, attempts to create a crease pattern that include multiple reverse
folds run into the issue that the fold angle of the set of reverse folds will
compound, so that the maximum combined fold angle must be less than $\pi$.

(a)

(b)

(c)

$\frac{\pi}{4}$$\frac{\pi}{2}$$\frac{3\pi}{4}$$\pi$$-\frac{\pi}{4}$$-\frac{\pi}{2}$$-\frac{3\pi}{4}$$-\pi$Dihedral
Angle $\xi$Fold Angle $\phi$ (d)

Figure 3: (a) Compound reverse fold crease pattern with four reverse folds and
a starting fold angle of zero. The starting fold angles of the constituent
reverse folds $\phi_{0}$ are shown $\phi_{0}$. (b) Compressed crease pattern
of the compound reverse fold, known as a spring joint. (c) Folded spring joint
(d) Graph of the fold angle $\phi$ of this spring joint. A dotted line is
drawn to indicate the hard stop of the paper hitting itself

A reasonable attempt to solve this issue would be to use reverse folds of the
opposite sign to reduce the fold angle. However, if constructed, then the
reverse folds of opposite sign will move counter to each other and the
actuation is reduced. Instead, we use the reverse folds of the opposite sign
with a fold angle of $\pi$ due to their property that the fold angle is
unchanging in $\xi$. Using this construction method, each successive reverse
fold is paired with a $\pi$ fold of the opposite sign. This process is
illustrated in figure 2. In the case of two reverse folds and one $\pi$ fold,
the combined fold angle $\phi^{dual}_{0}$ can be described by:

$\displaystyle\phi^{dual}_{0}=\phi_{0}^{0}-\pi+\phi_{0}^{1}$ (2)
$\displaystyle\phi^{dual}=\phi(\xi,\phi_{0}^{0})-\pi+\phi(\xi,\phi_{0}^{1})$

The starting angle of the two individual reverse folds are two parameters that
can be used to control the motion of this mechanism. More generally, for any
number of reverse folds constructed in this manner:

$\displaystyle\phi^{compound}_{0}=-\pi(n-1)+\sum^{n}_{k=0}\phi_{0}^{k}$ (3)
$\displaystyle\phi^{compound}=-\pi(n-1)+\sum^{n}_{k=0}\phi(\xi,\phi_{0}^{k})$

###  2.3 Parameters

Figure 3, shows the extent to which motion can be amplified by this process.
Much more relative motion can be achieved by combining the kinematics of
multiple reverse folds. This method is also very compact, taking up no more
space in the final folded model than a reverse fold. For this reason, it can
be used as a direct substitution for a reverse fold in applications that
require a much higher velocity ratio. Since there is no spacial cost to
including more complex spring joints, a reverse fold can be split into
multiple separate reverse folds as many times as required for the application.
As the limit of the number of reverse folds approaches infinity, and the
constituent starting fold angles $\phi_{0}$ approach $\pi$, the maximum
kinematics of a spring joint, given the starting fold angle $\phi_{0}^{max}$,
can be represented as:

$\displaystyle\phi^{max}=(\phi_{0}^{max}-\pi)sec(\frac{\xi}{2})+\phi_{0}^{max}$
(4)

The spring joint fold is also extremely programmable. The angle $\phi_{0}$ of
each reverse fold changes the path of the resultant model. For each parameter,
then, we are able to add an additional constraint to the path of the motion.
For example, we could require that the motion includes two value pairs
$(\xi,\phi)$, so long as these values exist in the space below the maximum
motion modeled in equation 4 and figure 7. We could find a spring joint for an
arbitrary starting fold angle $\phi_{0}$ and a desired fold angle when
unfolded $\phi$. The reverse fold function is not sufficiently nonlinear to
produce arbitrary functions of $\xi$, but it can still be applied to many
applications where a slight ”program” embedded into the model itself is
beneficial. A larger space of functions is possible when you also control the
fold angle of the $\pi$ folds, making the model double back or decouple, but
these mechanisms rely heavily on the perfect mathematical model, and do not
adequately transfer to the real world. Paper prototypes of decoupling spring
joints have only weakly demonstrated this effect, due to the flexibility and
memory of the paper, and any compliance in a working model would be compounded
in harming this effect. For this reason, the $\pi$ fold angle is not treated
as a parameter in the spring joint.

##  3 Axis Variations

###  3.1 Tilted Axis Reverse Folds

Up until this point, I have only discussed reverse folds that follow an axis
perpendicular to the pleats, wherein the reverse folds in a sequence are
aligned horizontally. Although this mechanism is useful and adaptable, it does
not represent the full extent of pleat based rigidly foldable mechanisms. In
this section I show that, by using reverse folds at a different axis,
functional spring joints can be designed. Increasing the potential designs for
spring joints.

Figure 4: A series of reverse folds, when folded with an axis tilt (and flat
folded based on the Kawasaki-Justin theorem), will spread out the pleats on
the opposite side of the reverse fold.

There is one glaring issue with this model–shown in figure 4: In order to
construct the constituent reverse folds of this model, I use the Kawasaki-
Justin Theorem [Kawasaki 91] to generate the fold angles of the resulting
pleats. These pleats, however, must spread out under this system and are no
longer symmetrical. Any non zero fold angle reverse fold will have this issue.
In order for a compound system of reverse folds to preserve the symmetries of
the pleats, the initial fold angle of the final system must be zero.

$\displaystyle\phi_{0}^{tilt}=0$ (5)

###  3.2 Tilted Axis Spring Joints

By designing crease patterns to hold with this condition, I generated
functional spring joints that operate at different angles of rotation. Given
that each constituent reverse fold is rigidly foldable, the resultant
mechanism is also rigidly foldable. A crease pattern for this mechanism is
shown and folded in figure 5. The mechanism, instead of pivoting about an axis
perpendicular to the direction of expansion, instead pivots about the tilted
axis. When folded flat, the spring joint section of the mechanism is spread
out along the paper, rather than be stacked in one area. Because of the
complex references and the repeated use of the Kawasaki-Justin Theorem to
guarantee flat foldability, there exists no practical method of finding
references to fold this model. In my prototypes, I have printed the crease
pattern on paper before folding.

(a)  (b)

Figure 5: (a) An example tilted spring joint crease pattern and folded state
with starting fold angle $\phi^{tilt}_{0}=0$ (b) Physical prototype where the
mechanism actuates along a different axis.

###  3.3 Crease Pattern Design

The method of generating these crease patterns is detailed in figure 6. The
parameters for construction–instead of being the fold angles of the individual
reverse folds–are the distances between $\pi$ folds and the number of reverse
folds. It is much more difficult to control the fold angle of any one reverse
fold, since, the angle of the pleat changes throughout the spring joint area.
This can be seen in the curves of pleat lines shown in figures 5 and 6.
Therefore the fold angle will change depending on the distance between $\pi$
folds and the angle of the pleat, which depends on the fold angle of the
reverse folds determined before it, and so on. This iterative process makes it
impractical to define individual fold angles by hand. Instead, I define a set
distance $l$ between $\pi$ folds and a length $l$ of pleat segments.
Generally, the ratio of $\frac{d}{l}$ should be smaller for more reverse
folds. Steps 2-4 generate the spring joint pattern based on the new axis and
the flat foldability requirement. Steps 5-7 then guarantee that
$\phi_{0}^{tilt}=0$ by forcing the resultant pleats parallel to the initial
pleats (5) and bisecting those pleats to avoid spread (6). The crease pattern
can be generated based on these restrictions. Computational methods of
designing tilted axis spring joints could circumvent this step by solving
directly for the final fold angle.

Figure 6: The step by step process for creating a tilted spring joint pattern.

###  3.4 Kinematics

The isotropic assumption used to derive the kinematic model of the basic
spring joint requires symmetric around a pleat and so can not longer apply to
tilted axis reverse folds. Furthermore, the desired value–”fold angle”–is
loosely defined when the mechanism actuates around a different axis, an axis
that itself is changing as the pleats open. For these reasons, finding a
specific kinematic model to describe the desirable relationships of tilted
axis spring joints is an open problem. Nevertheless, the ability to program
the axis of rotation into the mechanism serves as another tool to expand the
potential applications of this mechanism.

##  4 Alternate Modular Design

###  4.1 Layer Thickness Accommodation

Although there is no theoretical limit to the complexity of a spring joint,
the multitude of layers makes this model impractical for many real world
applications with non zero layer thickness. Of the existing methods for layer
thickness accommodation, none of them work with the existing of the spring
joint described in figure 3. The tapered panels method described by Tachi in
[Tachi et al. 11] cannot accommodate full $\pi$ fold angles, and so does not
work with the $\pi$ folds of the spring joint. The offset panel technique
method by [10\. 14] does not extend to work with so many layers on top of each
other. And the axis offset method used by [Hoberman 88] and [Trautz and
Kunstler 10] fails to preserve the kinematics of the original mechanism.

(a)  (b)  (c)

(d)
$\frac{\pi}{4}$$\frac{\pi}{2}$$\frac{3\pi}{4}$$\pi$$-\frac{\pi}{4}$$-\frac{\pi}{2}$$-\frac{3\pi}{4}$$-\pi$Dihedral
Angle $\xi$Fold Angle $\phi$ (e)

Figure 7:  Steps for the construction of a minimal layer spring joint: (a) The
basic unit for the mechanism (b) Place the two units together as shown, and
connect them with hinges along the dashed lines. (c) Connect two mirrored
compound units along the dashed line. (d) Finished minimal layer spring joint.
(e) The graph of the fold angle (solid) of the minimal layer spring joint
compared to the 4 reverse fold spring joint (dashed) from figure 3.  Figure 8:
A variation of the modular units and assembly to replicate a tilted axis
spring joint. Half of the units have a open cut in replacement of the slit
such that, when placed on itself, a greater horizontal displacement is
asymmetrically created in one pair of units, causing an axis shift.

###  4.2 Minimal Layer Spring Joints

In this section I introduce an adaptation of the spring joint fold to reduce
the layers of the mechanism. The trade off is that this adaptation is not
foldable from a planar sheet of paper. Instead, it is constructed by a series
of rectangular units with semicircular ends.

A method of construction for this variation is outlined in figure 7. In order
for this mechanism to operate well, all connections and the semicircular end
of the unit must be made with a flexible material. For paper prototypes,
scotch tape is sufficient.

This mechanism is also rigidly foldable, and, by replacing the semicircular
end with a polygon, it can be made to replicate the motion of any spring
joint. Each vertex of the polygon is equivalent a reverse fold-$\pi$ fold
pair, where: $\theta^{vertex}=\phi_{0}$. Therefore–in it’s theoretically
circular form with infinite vertices–it approximates the limit of spring joint
kinematics as described in equation 4.

This high velocity ratio is especially beneficial in applications where there
is a lack of rigidity in the material–such as origami action models and
compliant mechanisms–because it becomes much more responsive and motion is not
dampened in the mechanism. The downside to this high velocity ratio, is, of
course, a much lower mechanical advantage. For this reason it is not ideal for
high force transfer applications, and works well with light materials. The
pleats or corrugations also serve to reinforce the material, giving it more
structural integrity.

###  4.3 Tilted Axis Modular

Modifications to the original unit, shown in figure 8, allow for a modular
replication of the tilted axis. Unlike the basic minimal layer spring joint,
this is not a one to one replica of its origami alternative, but it replicates
a similar motion.

##  5 Reverse Fold Substitution

(a)  (b)

Figure 9: (a) The crease pattern and folded illustration of a Miura-Ori
pattern where the reverse folds are substituted by spring joints with
$\phi_{0}^{compound}=\frac{5\pi}{6}$. (b) The physical prototype of this
model, shown at varying dihedral angles $\xi$.

In origami corrugation design, you can interpret a reverse fold as a segment
of the sheet that reverses the pleat sign. Any fold that accomplishes this can
be used as a direct substitution for a reverse fold in the crease pattern,
with no interference. Since a spring joint is a composition of reverse folds,
it has this property and can be used as a substitution for reverse folds to
either amplify the motion of a corrugation mechanism, or to display additional
properties.

To illustrate this, I created a paper prototype for a Miura Ori where the
reverse folds are substituted for spring joints with
$\phi_{0}^{compound}=\frac{5\pi}{6}$ and four reverse folds each, shown in
figure 9. The Miura Ori, first shown in [Koryo 85], is by far the most popular
and widely used origami mechanism. It has the property of a negative Poisson’s
Ratio, where the model can expand in two orthogonal directions at the same
time, making it ideal for the deployment of large surfaces–such as that of
solar panels in space.

My design of the spring joint Miura Ori mechanism had a number of unique
properties. Where the fold angle $\phi>0$, the Miura-Ori is deploying as
normal, with a negative Poisson’s Ratio, in this case when the dihedral angle
$\xi<2.495$. At $\xi=2.495$, $\phi=0$, therefore the mechanism is in it’s
maximum deployed state. In the rest of the domain where $2.495<\xi>2.878$, the
model starts to collapse again as $\phi<0$ with a positive Poisson’s Ratio.
When $\xi=2.878$, $\phi=-\pi$, so the pleats collide with each other. These
states are shown in figure 9. In this prototype, it is much easier to deploy
to the maximum state at $\xi=2.495$, unlike with paper renditions of the
original Miura Ori, where the paper’s memory holds the folds in place. This
property implies a use in applications of a Miura Ori from a compliant
material, like the paper tested here.

##  6 Discussion

###  6.1 Further Development

Although this paper introduces many variants and generalizations of the spring
joint pattern, more generalizations will naturally lead to more applications.
The generalizations chosen for this paper were picked for their rigid
foldability, and therefore predictable kinematics. Further progress in this
field would require exploring spring joint generalizations for non parallel
pleats, and a shift in axis in the paper. These generalizations would allow
spring joints to adapt to the complex instances of reverse folds seen in
freeform tessellation variants [Tachi 10]. These generalizations are not
rigidly foldable, and cannot be designed by hand. In order to achieve these,
computer assisted design tools such as Crane [Suto et al. 23], Tree Maker
[Lang 96], Tess [Bateman et al. 02] and Origamizer [Demaine and Tachi 17] are
required. In my view, software assisted design and generalization is the
surest direction of progress for the spring joint mechanism.

###  6.2 In Origami

(a)

(b)

Figure 10: Origami cranes augmented with spring joints to engineer complex
motion patterns (a) Simple. (b) Complex.

Origami as an art form, unlike Engineering, is intentionally very restrictive.
The requirement of only being able to fold from a single unbroken sheet of
paper means that all of the traditional mechanisms found in engineering cannot
apply to action origami models. The spring joint, as described in this paper,
can be used as a mechanism to engineer motions within pure origami, and so
serves as a replacement for traditional mechanisms like gears ratios when
trying to create mechanical advantage in origami. For example, by grafting
pleats into a traditional crane, spring joints can be added to create motion
along multiple axis for a more realistic flapping pattern, shown in figure 10.

##  7 Conclusion

The spring joint mechanism provides a programmable, amplified alternative to
the reverse fold in origami mechanisms. This high velocity ratio is important
in compliant mechanisms and origami models in order to make them more
responsive. And the program-ability by the mathematical model allows for
arbitrary angles to be formed by deploying this mechanism. For these reasons I
have shown that it has applications in the engineering of new origami
structures.

## References

  * [10. 14] _An Offset Panel Technique for Thick Rigidily Foldable Origami_ , International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Volume 5B: 38th Mechanisms and Robotics Conference, 2014.  doi:10.1115/DETC2014-35606.  Available online (https://doi.org/10.1115/DETC2014-35606).
  * [Bateman et al. 02] Alex Bateman et al.  “Computer tools and algorithms for origami tessellation design.”  In _Origami3: Proceedings of the 3rd International Meeting of Origami Science, Mathematics, and Education_ , pp. 121–127, 2002.
  * [Demaine and Tachi 17] Erik Demaine and Tomohiro Tachi.  “Origamizer: A practical algorithm for folding any polyhedron.”
  * [Gardiner et al. 18] Matthew Gardiner, Roland Aigner, Hideaki Ogawa, and Rachel Hanlon.  “Fold mapping: parametric design of origami surfaces with periodic tessellations.”  _Origami_ 7 (2018), 105–118.
  * [Hoberman 88] Charles S Hoberman.  “Reversibly expandable three-dimensional structure.”, 1988.  US Patent 4,780,344.
  * [Hu et al. 20] Fuwen Hu, Wei Wang, Jingli Cheng, and Yunchang Bao.  “Origami spring–inspired metamaterials and robots: An attempt at fully programmable robotics.”  _Science Progress_ 103:3 (2020), 0036850420946162.
  * [Ikegami 23] Ushio Ikegami.  “Fractal Origami for Beginners.”  _The Fold_ 79\.
  * [Kawasaki 91] Toshikazu Kawasaki.  “On the relation between mountain-creases and valley-creases of a flat origami.”  In _Proceedings of the First International Meeting of Origami Science and Technology, 1991_ , 1991.
  * [Koryo 85] Miura Koryo.  “Method of packaging and deployment of large membranes in space.”  _The Institute of Space and Astronautical Science report_ :618 (1985), 1–9.
  * [Lang 96] Robert J Lang.  “A computational algorithm for origami design.”  In _Proceedings of the twelfth annual symposium on Computational geometry_ , pp. 98–105, 1996.
  * [Lee et al. 21] Dae-Young Lee, Jae-Kyeong Kim, Chang-Young Sohn, Jeong-Mu Heo, and Kyu-Jin Cho.  “High–load capacity origami transformable wheel.”  _Science Robotics_ 6:53 (2021), eabe0201.
  * [Shafer 13] Jeremy Shafer.  “Check out the new model I just designed!… Squishy Spring into Action Tesselation Tower Demo.”, 2013.  Available online (https://www.youtube.com/watch?v=qMgOPznUnK8).
  * [Song et al. 14] Zeming Song, Teng Ma, Rui Tang, Qian Cheng, Xu Wang, Deepakshyam Krishnaraju, Rahul Panat, Candace K Chan, Hongyu Yu, and Hanqing Jiang.  “Origami lithium-ion batteries.”  _Nature communications_ 5:1 (2014), 3140.
  * [Suto et al. 23] Kai Suto, Yuta Noma, Kotaro Tanimichi, Koya Narumi, and Tomohiro Tachi.  “Crane: An Integrated Computational Design Platform for Functional, Foldable, and Fabricable Origami Products.”  _ACM Transactions on Computer-Human Interaction_ 30:4 (2023), 1–29.
  * [Tachi et al. 11] Tomohiro Tachi et al.  “Rigid-foldable thick origami.”  _Origami_ 5:5 (2011), 253–264.
  * [Tachi 09] Tomohiro Tachi.  “One-Dof cylindrical deployable structures with rigid quadrilateral panels.”, 2009.  Available online (https://api.semanticscholar.org/CorpusID:8229142).
  * [Tachi 10] Tomohiro Tachi.  “Freeform variations of origami.”  _J. Geom. Graph_ 14:2 (2010), 203–215.
  * [Trautz and Kunstler 10] Martin Trautz and Arne Kunstler.  “Deployable folded plate structures-folding patterns based on 4-fold-mechanism using stiff plates.”  In _Symposium of the International Association for Shell and Spatial Structures (50th. 2009. Valencia). Evolution and Trends in Design, Analysis and Construction of Shell and Spatial Structures: Proceedings_. Editorial Universitat Politècnica de València, 2010.

Malcolm Smith e-mail<EMAIL_ADDRESS>


# Visual Affordance Prediction for Guiding Robot Exploration

Homanga Bharadhwaj, Abhinav Gupta, Shubham Tulsiani The authors are with the
Robotics Institute, Carnegie Mellon University. Link to project website
https://sites.google.com/andrew.cmu.edu/affordance-robotics/home

###### Abstract

Motivated by the intuitive understanding humans have about the space of
possible interactions, and the ease with which they can generalize this
understanding to previously unseen scenes, we develop an approach for learning
visual affordances for guiding robot exploration. Given an input image of a
scene, we infer a distribution over plausible future states that can be
achieved via interactions with it. We use a Transformer-based model to learn a
conditional distribution in the latent embedding space of a VQ-VAE and show
that these models can be trained using large-scale and diverse passive data,
and that the learned models exhibit compositional generalization to diverse
objects beyond the training distribution. We show how the trained affordance
model can be used for guiding exploration by acting as a goal-sampling
distribution, during visual goal-conditioned policy learning in robotic
manipulation.

##  I Introduction

Consider the images $o_{g}$ in Figure 1. We humans can effortlessly understand
the depicted scenes e.g. a bottle lying on the table in the top image, or a
teddy lying next to a pot. More importantly, we can infer not only what _is_ ,
but also what _can be_ e.g. one can imagine the bottle being placed at a
different location on the table, or turned to lie horizontally, or perhaps
taking the teddy to place it inside the pan. We instinctively make such
judgements across a myriad of everyday scenarios, understanding that an open
cabinet can be closed, or that an egg can be broken, or spilled liquid wiped
away. In this work, our goal is to build a computational system with similar
capabilities, which can be used directly as a goal-sampling distribution for
robot exploration. Given just a single image depicting generic (possibly
unseen) objects, we wish to predict possible configurations that may occur as
a result of a human (or a robot) interacting in the scene.

We are inspired by Gibson’s seminal work on _affordances_ which argues for
developing intelligent agents with an intuitive understanding of possible
interactions they can have with their environment [13]. Initial attempts in
the vision community formalized this as a pixelwise labeling task (e.g.
‘sittable’ surface), but these do not explicitly model the actions or their
effects [6, 42]. An alternative approach has been to use geometric models [15,
11] to predict human-centric affordances. Towards a richer parametrization,
recent approaches have pursued ‘visuo-motor’ affordances where the space of
possible interactions is modeled via predicting possible low-level actions
that an agent can execute to affect its environment [32, 36, 28, 43]. While
such visuo-motor affordances can be directly translated to agent behavior, we
argue that the requirement of inferring precise actions associated can be
restrictive e.g. babies may understand that a fruit can be cut even if they
can’t do it themselves, as perhaps an adult may know that a bike’s tire can be
removed even if not sure how to precisely do so. In addition, training models
for such visuo-motor affordances requires annotated data, which can be
restrictive to scale for diverse settings. In our work, we therefore pursue
‘visual affordances’, where instead of modeling the action trajectories, we
aim to model what their _effects_ can be. More concretely, given a single
input image, we pursue the task of conditionally generating diverse and
plausible images of the same scene that can be achieved as a result of an
agent’s interaction.

This task of inferring visual affordances is an increasingly common one in the
robot learning community, where an understanding of interesting plausible
actions can help guide exploration. However, the typical methods only learn
domain-specific affordances — relying on active interaction in a specific
(lab) setting, they learn models for interactions with the particular objects
in the particular environment [32, 36, 22, 31]. In addition to their limited
generalizability, these active interaction-based methods also do not learn
more complex behaviors (e.g. stacking blocks) as random interactions from an
agent are unlikely to lead to such interesting transitions. Our key insight in
this work is that instead of only relying on active interactions with limited
variation, such visual affordances can be learned from large-scale diverse
passive data. Just as we humans can learn from watching others (e.g.
instructional videos), our system learns visual affordances using interaction
videos depicting generic human and robot interactions across varied settings.
Using this trained affordance model, when a robot is placed in a certain
scene, we can generate goals corresponding to plausible manipulations of the
objects in the scene, and learn a goal-conditioned policy through self-
supervised exploration.

Specifically, we consider the setting of exploration in goal-conditioned
policy learning and use the affordance model for sampling goals that the
policy tries to reach during training. The affordance model consists of two
stages: 1) we learn discrete latent embeddings from images by training a VQ-
VAE [41] on the entire dataset 2) we learn conditional prediction of latent
codes through an auto-regressive generation procedure using a Transformer
architecture [5]. Because of the diversity seen in training, the affordance
model is able to generalize to new environments and unseen objects, and
predict interesting plausible configurations in settings never seen in
training. Due to the diversity of sampled goals, the policy is incentivized to
explore broadly and learn interesting behaviors like stacking, compared to
curiosity-based exploration strategies.

Figure 1: We develop an approach for goal-directed robot exploration by
learning a goal-sampling distribution followed by self behavior cloning with
exploration trajectories. The goal-sampling distribution is a visual
affordance model trained from large passive datasets of image pairs to predict
a distribution over goal images given an initial image of the scene. We show
that this affordance model enables goal-directed robot exploration, for
learning diverse behaviors like pushing, grasping, and stacking in a table-top
manipulation setting.

##  II Related Works

Actionable Information with Visual Observations. Prior work has explored the
problem of learning how to interact with objects in the scene from visual
observations, in different settings like dextrous manipulation [27] and mobile
navigation [30]. These approaches typically use either passive observations
(for example human videos) [29, 2, 11] or active interactions with a robotic
agent [35]. Some prior works also leverage robot simulators to randomize the
generation of different types of objects in order to learn affordances for
tasks like grasping [21] and pushing [45] objects. The unifying idea in all
these approaches is learning how to interact with objects in the scene. They
tackle a problem slightly orthogonal to ours, where by learning affordances,
we refer to visualizing the result of interactions, and not specifically how
to obtain those interactions.

Generative Modeling. There have been significant recent advances in deep
generative models, with high-quality realistic images being produced by
StyleGAN [19, 20] and BigGAN [3, 7] among others that use transformers [41,
40, 33, 5, 9]. While most of these models tackle the problem of unconditional
generation, a closely related problem to our setting is that of conditional
image generation. This is typically defined as the task of generating new
images from a dataset conditional on certain class attributes [39, 10]. Video
generation methods [24] using transformers perform per-timestep prediction,
whereas our affordance model samples (long-horizon) goal images that we show
are useful for robot exploration. Some style transfer approaches [18, 46] also
have a similar conditional generation formulation, where the conditioning is
done on a source image as opposed to a class attribute. Our affordance model
is an approach for better conditioning on images, where we want the
generations to capture configurational changes in the attributes of objects in
the source image.

Robot Exploration. Generative models have been used for goal-sampling in
visual robot learning [32, 31, 36, 26]. Learning a good goal-sampling strategy
for diverse goals and training goal-conditioned policies to try and reach
sampled goals is a paradigm for exploration without the use of explicit
heuristics like curiosity [34, 38] or reduction of uncertainty [4, 8]. Most
prior work in this vein has learned goal-sampling models only in the
experiment domain either by collecting large robot interaction datasets, or
training the goal-sampler online jointly with policy learning [32, 26, 36,
43]. Collecting large datasets in the lab is expensive, and the goal-sampling
models trained solely on a specific-lab setting is unlikely to generalize to
other settings without requiring re-training. In contrast to these approaches
we learn a model for generating visual affordances from various internet data
- not data collected in the experimenter’s lab. We show how the learned
affordance model can be used to guide exploration for goal-conditioned policy
learning in environments with unseen objects.

Figure 2: Affordance-driven exploration and policy learning through hindsight
goal re-labeling. Given an initial configuration, we sample a goal with the
affordance model, execute rollouts with the current policy, and store the
transitions in the replay buffer. For updating the policy, we sample
transitions from the replay buffer and re-label goals to be the final frames
in the corresponding trajectory. The process is described in detail in section
III-B On the right we show two different views of the robot workspace, with a
Franka Panda arm and an overhead Realsense camera.

##  III Approach

We develop an approach for learning visual affordances from passive data that
can be used as goals for guiding robot exploration. We use the term affordance
to mean the set of possible interactions in a particular scene. Given an image
of a scene, we want the affordance model to generate a new image with
different configurations of the same objects in the scene. This model can be
used for goal-directed exploration, such that in a new scene, the agent can
sample a goal conditioned on the current image, and execute actions to try and
reach the sampled goal.

Since collecting data in-lab is expensive due to a lot of manual effort, we
make use of the rich diversity of passive data existing in the internet, to
learn the affordance model, and use it directly for goal-sampling. The
affordance model is two-staged: we first learn a VQ-VAE [41] to discretize the
space of continuous images, and then learn a transformer based auto-regressive
model that (in latent space) predicts a possible goal given the input. We then
project the generated latent goal, back to the image-space with the decoder of
the VQ-VAE. This framework lets us both: a) generate diverse goals, and b)
produce realistic and high-resolution images.

For goal-conditioned policy learning, self-supervised approaches typically
adopt a learning pipeline consisting of two (often inter-leaved) components:
1) exploration to collect data of interactions with the environment, and 2)
policy learning using the interaction trajectories. We show that it is
possible to drive better exploration by using the trained affordance model as
a conditional goal-sampling distribution that samples diverse realistic goals
in an unseen scene.

Concretely, we learn a goal-conditioned policy through exploration, where the
goals are sampled from a learned affordance model in the beginning of the
trajectory. In order to learn such a policy through self-supervised
exploration, the sampled goals must be plausible and diverse, corresponding to
different arrangement of the objects in the scene. Depending on the objects in
the scene, the different arrangements could be putting a lid on a pot, pushing
a cup across the table, grasping an object and rotating it etc. We show that
such behaviors emerge from the trained affordance model, and describe how we
can learn a goal-conditioned policy through hindsight re-labeling of
exploration trajectories.

In the next sub-sections, we first describe the architecture and training
details of the affordance model, and then discuss how we use the model as a
goal-sampler for guiding robot exploration in manipulation tasks.

###  III-A Learning to Imagine Goals

Instead of modeling in the image-space directly, in order to reduce spatial
redundancies and implicitly abstract out interesting objects in the scene, we
first learn a down-sampled encoding of high resolution images. To learn these
discrete latent embeddings, we employ a VQ-VAE to learn lossy encoder and
decoder models that can transform latents to generate realistic image samples.
Having a discrete latent space allows flexibility in modeling features (such
as objects in the scene) that span multiple pixels in the image, and do not
encode local imperceptible differences that are not significant.

Training details of the VQ-VAE. Let the images be $o\in\mathbb{R}^{H\times
W\times 3}$. The VQ-VAE model consists of an encoder
$E(o)\in\mathbb{R}^{h\times w\times L}$, a codebook $C=\\{e_{k}\\}_{k=1}^{K}$
with elements $e_{i}$ of size $L$ and a decoder $D(e)\in\mathbb{R}^{H\times
W\times 3}$. The quantization happens in the channel space, where all the
$h\times w$ vectors of size $L$ are replaced by their nearest codebook vectors
$e_{i}$, and the resulting quantized latent code of dimension $h\times w\times
L$ is fed to the decoder. The training objective for this model, following
prior work [41] is as follows:

$\displaystyle{\mathcal{L}}_{\text{vqvae}}=\mathbb{E}_{o\sim{\mathcal{D}}}\left[||o-D(e)||_{2}+||E(o)-\text{sg}[e]||_{2}\right]$

Here, sg refers to the stopgradient operator, that is defined as identity
during forward-computation.

After training the VQ-VAE, corresponding to each image $o_{c}$, we obtain a
downsampled encoding $z_{c}$. We then learn an auto-regressive prior using a
Transformer architecture to model $p(z_{g}|z_{c})$. Finally, with the decoder
of the VQ-VAE we obtain a corresponding image $o_{g}$ from the latent $z_{g}$.
Learning this conditional generation model in the latent space with a powerful
Transformer architecture enables us to achieve compositional generalization,
by generation of plausible goals in scenes where there are multiple objects.
The stochasticity of the model is helpful in ensuring that the generated
samples are allowed to be diverse.

Auto-regressive goal sampling. In Fig. 3, we show a forward pass through our
affordance model that generates a goal image $o_{g}$, given a conditioning
image $o_{c}$. The initial image $o_{c}$ is first encoded by the VQ-VAE to a
discrete latent code $z_{c}$. The Transformer then generates the latent code
corresponding to the goal image, $z_{g}\sim p(z_{g}|z_{c})$. The generation
process happens pixel by pixel in raster-scan order and channel by channel for
one pixel. Hence we can denote the auto-regressive generation process as:

$\displaystyle p(z_{g})=\prod_{i}p\left(z_{(g,i)}|z_{(g,<i)},z_{c}\right)$

This equation shows that the prediction of the $i^{\text{th}}$ pixel in the
latent code $z_{(g,i)}$ is conditioned on the already predicted pixels
$z_{(g,<i)}$ and the initial latent code $z_{c}$.

Figure 3: Illustration of a forward pass through the affordance model. The
initial scene $o_{c}$ is first encoded by a VQ-VAE encoder, and converted to a
discretized latent code by swapping nearest neighbor encodings from a codebook
$C$. The resulting discrete latent code $z_{c}$ is passed through the
Transformer. The autoregressive generation yields a latent code $z_{g}$ which
is decoded by the VQ-VAE decoder to a plausible goal image $o_{g}$.

Learning objective. We instantiate the affordance model as
$p_{\psi}(o_{g}|o_{c})$ and aim to maximize
$\mathbb{E}_{(o_{g},o_{c})\sim{\mathcal{D}}}[\log p_{\psi}(o_{g}|o_{c})]$.
Given image pairs $(o_{c},o_{g})\sim{\mathcal{D}}$, we transform each image to
their respective quantized latent code $z_{c}$ and $z_{g}$. The autoregressive
transformer model is used to generate $\hat{z}_{g}$ conditioned on $z_{c}$,
i.e. $\hat{z}_{g}\sim p(z_{g}|z_{c})$. We train the model to maximize
$\mathbb{E}_{(z_{g},z_{c})\sim{\mathcal{D}}}[\log p(z_{g}|z_{c})]$.

Training Details of the Transformer. For the above generation, we use an
architecture similar to Image-GPT [5]. It consists of an encoder-decoder model
with masked convolutions and self-attention layers in the decoder. The auto-
regressive model is preserved by padding the input, and replacing the paddded
pixel values with the generated values and repeating the process recursively.
Finally, all the values in $z_{g}$ correspond to valid generations that become
more and more accurate as training progresses, and so we eventually have a
model of the form $p(z_{g}|z_{c})$. We then feed in $z_{g}$ to the VQ-VAE
decoder, obtaining the decoded image frame $o_{g}$. This completes the
structure of the affordance model $p_{\psi}(o_{g}|o_{c})$. After training the
model, given a new initial image $\hat{o}_{c}$, we can sample a plausible goal
$o_{g}\sim p_{\psi}(o_{g}|\hat{o}_{c})$. We empirically analyze this model in
the next section, and show results of goal generations in unseen scenes.

###  III-B Affordance-driven Robot Exploration

We use the affordance model trained solely on passive data for goal-
conditioning and do not fine-tune on any lab-specific data. Because we trained
the model on diverse passive data, we observe good generalization for
affordance prediction in the robot learning setting. We obtain exploration
trajectories with interesting behaviors, and are eventually able to learn
better goal-conditioned policies compared to alternate exploration strategies
like curiosity [34] that learn from scratch in the environment, or those that
sample goals from other sampling distributions like a VAE-prior [32].

Given the affordance model, we train a policy that chooses actions for
executing a particular task specified by a goal image. We refer to the
observation the robot sees at time-step $t$ of an episode as $o_{t}$, and
denote the goal image as $o_{g}$. We denote the goal-conditioned policy as
$\pi(a_{t}|o_{t},o_{g})$. We learn this policy by simple behavior cloning with
goal-relabeling. After sampling a goal $o_{g}\sim p_{\psi}(o_{g}|o_{1})$ and
executing a trajectory $(o_{1},a_{1},o_{2},a_{2},...,o_{T};o_{g})$ to reach
the goal, even if the final configuration $o_{T}$ is not similar to the goal
configuration, the executed trajectory provides useful information for
reaching the configuration the system ended up in. We describe the overall
framework in Fig. 2. During deployment, test goal images are sampled from some
distribution $o_{g}\sim p({\mathcal{G}})$, and the robot must interact with
the objects in the scene to reach the goal configuration.

Training the goal-conditioned policy. Given observation $o_{1}$ corresponding
to the initial scene, we sample a goal from the affordance model $o_{g}\sim
p_{\psi}(o_{g}|o_{i})$ and execute a trajectory
$(o_{1},a_{1},o_{2},a_{2},...,o_{T};o_{g})$, which we store in the replay
buffer ${\mathcal{R}}$. In Fig. 2 we see examples of goals sampled in a scene.
The affordance model enables sampling diverse goals and encourages the policy
to try and reach them, thereby ensuring interesting interactions. Given the
data of interactions in the replay buffer ${\mathcal{R}}$, we perform goal re-
labeling and use tuples $(o_{t},a_{t},o_{t+1},o_{g}=o_{T})$ to update the
policy $\pi(a_{t}|o_{t},o_{g})$. This corresponds to a variation of hindsight-
experience replay, that has been shown to be useful in prior works [1, 12,
25]. The rationale for this is that during exploration, although the
incompletely trained policy might not reach the sampled goal, the final state
it reaches is still a potential goal, and the executed sequence of actions
provides guidance on reaching this goal. The policy update method is to do
simple behavior cloning that maximizes the the likelihood
$\mathbb{E}_{(o_{t},a_{t},o_{t+1},o_{g}=o_{T})\sim{\mathcal{R}}}[\log\pi(a_{t}|o_{t},o_{g})]$.
Note that we do not need samples to be on-policy for this update and so we can
interleave exploration of a few trajectories with a policy update phase where
we randomly sample tuples from different trajectories in the replay buffer.

##  IV Experiments

Through experiments, we aim to understand the following research questions:

  * •

How effective is the affordance model in generating diverse scenes with
plausible object manipulations?

  * •

How effective are the generated plausible and diverse affordances in guiding
robot exploration?

###  IV-A Setting

Data. For training the affordance model, we use data from three different
internet sources: somethingsomethingv2 [14], Berkeley [22], and JHU CoStar
[17]. We did not collect any additional data ourselves, and use only pairwise
frames extracted from these datasets for training. We use the trained visual
affordance model as-is for robot experiments and do not do any fine-tuning in
the lab.

Baselines. We compare our model against two relevant baselines, a Conditional
VAE (CVAE) [39], and a Conditional GAN (Pix2Pix) [18] for conditional
generation and robot exploration, and in addition with a curiosity baseline
[34] for robot exploration. We train these models on the same data as our
model, and perform qualitative evaluations as well as quantitative comparisons
through several metrics described in the next sub-section. After analyzing the
visual generations of the baselines in section IV-B, we evaluate policy
learning behaviors in section 5

Figure 4: Qualitative analysis of the proposed affordance model with the
baselines CVAE and Pix2Pix. Given initial conditioning frame $o_{c}$ that
corresponds to an unseen configuration during training, we see that the
sampled affordances $o_{g}$ from our model are diverse and correspond to
plausible interactions in the scene. For the baselines, we see that the
generated frames are not diverse and sometimes omit certain objects from the
scene or introduce different artifacts.

###  IV-B Evaluating Predicted Affordances

As highlighted by prior work, evaluating the quality of synthesized images is
challenging [18, 46]. Typical metrics used in assessing image reconstruction
(like the pixel mean-squared error) do not translate well to assessing the
quality of novel image generations. Further, these metrics will not be useful
for understanding how diverse and plausible are the generated affordances for
novel scenes. Hence we consider a metric based on human perceptual evaluation.
Motivated by evaluation protocols in prior works [16, 18, 46, 44], we conduct
an perceptual study with Amazon Mechanical Turk (MTurk). We run the MTurk
perceptual study, following the protocol from prior work [18, 46] where images
are shown on the screen for 3 seconds and workers are asked to guess certain
proprieties of the images. In our setting, we show three images per screen and
ask the workers to choose one among the two rightmost images. We mention in
the instructions that the task is to guess which of the two images on the
right correspond to plausible manipulations of objects on the left image. We
randomize the ordering of images such that one image is from our model and the
other is from a baseline and compare the average number of times workers
choose ours compared to the baselines.

Qualitative results. In Fig. 4 we perform a qualitative analysis of the
proposed affordance model with the baselines CVAE and Pix2Pix. For each of the
three initial scenes $o_{c}$ in the first column, we generate three
affordances with our model, three with the CVAE model, and one with the
Pix2Pix model (since the model is not stochastic). The initial scenes are
unseen during training. We can see that the affordances generated by our model
are diverse and involve plausible manipulations of the different objects in
the scene. We observe the emergence of interesting affordances like stacking
of blocks and re-orientation of the bottle. Whereas for the baselines,
sometimes certain objects are omitted from the scene, are non-realistically
generated, and do not involve diverse manipulations of the objects. Given that
the objects have never appeared in the training data before, this provides
evidence that our method can generalize to new scenes and compose affordances
for different objects.

Quantitative results. In Table I, we report results from the MTurk perceptual
study. 40 workers participated in our study. In each trial, workers saw an
image on the left, and were asked to choose which of the two images on the
right corresponded to plausible manipulations of objects on the left image.
The numbers show the $\%$ of trials where workers chose a sample from our
model as opposed to that from the baselines, averaged over all the workers.
Since the numbers are $>50\%$, we see that workers preferred samples from our
model compared to the baselines. This confirms the plausibility of our
generated affordances.

Table I: Pairwise comparison of results for the MTurk perceptual study. We tabulate the $\%$ of trials where workers chose a sample from our model as opposed to that from the baselines, averaged over 40 workers who participated in our study. Higher is better. A number $>50\%$ shows that workers chose our model’s sample more number of times compared to the baselines. | Pix2Pix | CVAE
---|---|---
Our | $69.8\pm 11.9$ | $75.5\pm 10.8$

###  IV-C Benchmarking Affordance-Guided Exploration

Figure 5: We show visualizations of exploration during training, corresponding to different episodes. On the right, we show the respective goal sampled from the affordance model. Towards the end, we see the evolution of interesting behaviors like grasping and the policy leading to behaviors that reach the goal image.  Table II: We show results of tests for various robot manipulation tasks. For each task, we have two test goal configurations, and report success rate over 10 trials each.  | Pushing | Pick and Place | Stacking
---|---|---|---
Curiosity | 50% | 40% | 30%
CVAE | 30% | 20% | 10%
Pix2Pix | 30% | 10% | 10%
Our | 70% | 60% | 60%
Figure 6: We show examples of two evaluation runs of our method and the
curiosity baseline corresponding to two test goals in stacking and pick and
place respectively. On the right are the goal images that the policy is
conditioned on for evaluation, and the sequence of observations
$o_{1},...,o_{T}$ corresponds to the executed trajectory.

In this section, we empirically analyze the framework in terms of generating
diverse affordances that serve as goals for aiding exploration in robot policy
learning. We consider a goal-directed policy learning setting, where the robot
needs to set its own goals, and explore the environment to try and reach those
goals. There is no notion of tasks during the exploration phase, but for
evaluation, the experimenters provide goal images corresponding to tasks like
pushing, and stacking that can be performed in the scene. The overall aim is
to evaluate goal-reaching behaviors that can be learned by training a goal-
conditioned policy described in section III-B.

Setup. Our setup is shown in Fig. 2, where we use a Franka Emika Panda robot
arm, with an overhead Intel Realsense camera for observations. The robot is
controlled through end-effector (EE) control, and the action-space is four
dimensional - (x,y,z) position of the EE and opening/closing of the gripper.
We place certain objects in the scene, and let the robot interact with them.
We choose diverse everyday objects like teddy bears, cloth, ketchup bottle,
blocks etc. such that different behaviors like pushing, pick and place,
stacking are plausible to emerge through interaction. We reset the scene after
$T$ steps of interaction while introducing new objects and/or changing the
position of existing objects. As is standard in evaluating goal-conditioned
policies, after training, we measure % success in reaching a set of test goal-
images. For comparison, in addition to goal-sampling from CVAE and Pix2Pix
baselines, we consider a curiosity-based exploration baseline [34] for data
collection, and follow the same training protocol as our method to train a
goal-conditioned policy with the exploration data.

Results. We visualize the progression of exploration during training in Fig.
5, for different episodes corresponding to different sampled goals. Towards
the end, we see the evolution of interesting behaviors like grasping and the
policy leading to behaviors that reach the goal image. This illustrates that
the proposed affordance-guided exploration described in section III-B is
helpful in training a policy with emergent manipulation capabilities.

After training the policy through affordance-driven exploration, for
evaluation we consider a set of goal images, and evaluate the fraction success
of the policy in reaching goal configurations. We emphasize that only a single
goal-conditioned policy is trained and evaluated, but for ease of analysis we
show results in Table. II split across 3 different type of tasks, with the
same trained policy. In Table. II, we show comparisons of the success rates of
the two approaches, across three types of tasks - pushing, pick and place, and
stacking. For each task, we consider two goal-images, and tabulate % success
across 10 trials.

In Fig. 6 we visualize some evaluation runs for our method and the curiosity
baseline. Please refer to the supplementary video for additional
visualizations. We can see an average of 25% higher success rates compared to
the baselines, demonstrating the efficacy of the affordance model for goal-
directed exploration. We observe significantly lower success rates for the
CVAE and Pix2Pix baselines, which we believe is because the images generated
either do not correspond to realistic reachable goals for the robot or do not
change the configuration at all (see examples in Fig. 4), and thus are unable
to help in guiding exploration for policy learning. This confirms the
generalization of the proposed affordance model trained on diverse passive
data, for aiding in goal-directed exploration.

##  V Discussion

In this paper we developed a framework for learning visual affordances from
passive data such that the learned model can be directly used for goal-
directed exploration in unseen scenes. Given an image of an initial scene, our
affordance model could generate diverse images corresponding to plausible
interactions. Further, its ability to generalize allowed us to directly
leverage this affordance model to drive exploration in robot learning. We
believe our approach is indicative of the broader potential of large-scale and
diverse visual data in developing intelligent agents that can act in generic
environments. However, ours is but a first step towards this and we believe
several exciting questions remain to be addressed. First, our notion of
‘visual affordances’, while capturing high-level changes, did not model the
actions required. In extending this approach to capture ‘visuomotor
affordances’, one must overcome the challenges of varying morphology variation
across humans and hands, while also developing action representations beyond
low-level trajectories. It would also be interesting to generate intermediate
‘checkpoints’ to continuously depict the transition, but these may not be
directly useful for robots due to the presence of human hands. Broadly, we are
hopeful that extending on our work, there will be future approaches that can
leverage increasingly diverse and multi-modal passive datasets for
generalization in robot learning, alleviating the necessity of in-domain data
collection that has greatly bottle-necked the field.

## References

  * [1] Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., Zaremba, W.: Hindsight experience replay. Advances in neural information processing systems 30 (2017)
  * [2] Brahmbhatt, S., Ham, C., Kemp, C.C., Hays, J.: Contactdb: Analyzing and predicting grasp contact via thermal imaging. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8709–8719 (2019)
  * [3] Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)
  * [4] Burda, Y., Edwards, H., Storkey, A., Klimov, O.: Exploration by random network distillation. arXiv preprint arXiv:1810.12894 (2018)
  * [5] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., Sutskever, I.: Generative pretraining from pixels. In: International Conference on Machine Learning. pp. 1691–1703. PMLR (2020)
  * [6] Delaitre, V., Fouhey, D.F., Laptev, I., Sivic, J., Gupta, A., Efros, A.A.: Scene semantics from long-term observation of people. In: European Conference on Computer Vision. pp. 732–745. Springer (2012)
  * [7] Donahue, J., Simonyan, K.: Large scale adversarial representation learning. Advances in Neural Information Processing Systems 32 (2019)
  * [8] Dwaracherla, V., Lu, X., Ibrahimi, M., Osband, I., Wen, Z., Van Roy, B.: Hypermodels for exploration. arXiv preprint arXiv:2006.07464 (2020)
  * [9] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12873–12883 (2021)
  * [10] Fetaya, E., Jacobsen, J.H., Grathwohl, W., Zemel, R.: Understanding the limitations of conditional generative models. arXiv preprint arXiv:1906.01171 (2019)
  * [11] Fouhey, D.F., Delaitre, V., Gupta, A., Efros, A.A., Laptev, I., Sivic, J.: People watching: Human actions as a cue for single view geometry. In: European Conference on Computer Vision. pp. 732–745. Springer (2012)
  * [12] Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., Levine, S.: Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088 (2019)
  * [13] Gibson, J.J.: The ecological approach to visual perception. (1979)
  * [14] Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al.: The" something something" video database for learning and evaluating visual common sense. In: Proceedings of the IEEE international conference on computer vision. pp. 5842–5850 (2017)
  * [15] Gupta, A., Satkin, S., Efros, A.A., Hebert, M.: From 3d scene geometry to human workspace. In: CVPR (2011)
  * [16] Gupta, A., Johnson, J., Alahi, A., Fei-Fei, L.: Characterizing and improving stability in neural style transfer. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 4067–4076 (2017)
  * [17] Hundt, A., Jain, V., Lin, C.H., Paxton, C., Hager, G.D.: The costar block stacking dataset: Learning with workspace constraints. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1797–1804. IEEE (2019)
  * [18] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
  * [19] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4401–4410 (2019)
  * [20] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8110–8119 (2020)
  * [21] Khansari, M., Kappler, D., Luo, J., Bingham, J., Kalakrishnan, M.: Action image representation: Learning scalable deep grasping policies with zero real world data. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 3597–3603. IEEE (2020)
  * [22] Khazatsky, A., Nair, A., Jing, D., Levine, S.: What can i do here? learning new skills by imagining visual affordances. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 14291–14297. IEEE (2021)
  * [23] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [24] Le Moing, G., Ponce, J., Schmid, C.: Ccvs: Context-aware controllable video synthesis. Advances in Neural Information Processing Systems 34, 14042–14055 (2021)
  * [25] Li, A., Pinto, L., Abbeel, P.: Generalized hindsight for reinforcement learning. Advances in neural information processing systems 33, 7754–7767 (2020)
  * [26] Lin, X., Baweja, H.S., Held, D.: Reinforcement learning without ground-truth state. arXiv preprint arXiv:1905.07866 (2019)
  * [27] Mandikal, P., Grauman, K.: Learning dexterous grasping with object-centric visual affordances. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 6169–6176. IEEE (2021)
  * [28] Mo, K., Guibas, L.J., Mukadam, M., Gupta, A., Tulsiani, S.: Where2act: From pixels to actions for articulated 3d objects. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6813–6823 (2021)
  * [29] Nagarajan, T., Feichtenhofer, C., Grauman, K.: Grounded human-object interaction hotspots from video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8688–8697 (2019)
  * [30] Nagarajan, T., Grauman, K.: Learning affordance landscapes for interaction exploration in 3d environments. Advances in Neural Information Processing Systems 33, 2005–2015 (2020)
  * [31] Nair, A., Bahl, S., Khazatsky, A., Pong, V., Berseth, G., Levine, S.: Contextual imagined goals for self-supervised robotic learning. In: Conference on Robot Learning. pp. 530–539. PMLR (2020)
  * [32] Nair, A.V., Pong, V., Dalal, M., Bahl, S., Lin, S., Levine, S.: Visual reinforcement learning with imagined goals. Advances in neural information processing systems 31 (2018)
  * [33] Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., Tran, D.: Image transformer. In: International Conference on Machine Learning. pp. 4055–4064. PMLR (2018)
  * [34] Pathak, D., Agrawal, P., Efros, A.A., Darrell, T.: Curiosity-driven exploration by self-supervised prediction. In: International conference on machine learning. pp. 2778–2787. PMLR (2017)
  * [35] Pinto, L., Gupta, A.: Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In: 2016 IEEE international conference on robotics and automation (ICRA). pp. 3406–3413. IEEE (2016)
  * [36] Pong, V.H., Dalal, M., Lin, S., Nair, A., Bahl, S., Levine, S.: Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698 (2019)
  * [37] Rakhimov, R., Volkhonskiy, D., Artemov, A., Zorin, D., Burnaev, E.: Latent video transformer. arXiv preprint arXiv:2006.10704 (2020)
  * [38] Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., Gelly, S.: Episodic curiosity through reachability. In: International Conference on Learning Representations (ICLR) (2019)
  * [39] Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. Advances in neural information processing systems 28 (2015)
  * [40] Tulsiani, S., Gupta, A.: Pixeltransformer: Sample conditioned signal generation. arXiv preprint arXiv:2103.15813 (2021)
  * [41] Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems 30 (2017)
  * [42] Wang, X., Girdhar, R., Gupta, A.: Binge watching: Scaling affordance learning from sitcoms. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2596–2605 (2017)
  * [43] Wang, Y., Gautham, N., Lin, X., Okorn, B., Held, D.: Roll: Visual self-supervised reinforcement learning with object reasoning. In: Conference on Robot Learning (2020)
  * [44] Woods, A.T., Velasco, C., Levitan, C.A., Wan, X., Spence, C.: Conducting perception research over the internet: a tutorial review. PeerJ 3, e1058 (2015)
  * [45] Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., Funkhouser, T.: Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4238–4245. IEEE (2018)
  * [46] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)

### Implementation details for the Affordance model and baselines

VQ-VAE Encoder and Decoder. We have input image sizes $64\times 64\times 3$
and latent code dimensions $32\times 32\times 1$. The codebook size is 1024,
and the codebook dimension is 256. The encoder and decoder architectures are
same as [41]. We use a learning rate of $5e-4$ with ADAM optimizer [23], a
batch size of 32, and train for 300K training steps. We use exponential moving
average (EMA) to update dictionary items, instead of the commitment loss, with
$\gamma=0.99$ as described in Appendix A of [41].

Transformer. We follow the ImageGPT [5, 37] architecture for the auto-
regressive Transformer model that predicts goal latent code, given the initial
latent code. The latent code dimensions input are $32\times 32\times 1$
obtained from the VQ-VAE encoder. There are 4 attention heads and 16 attention
layers, the vocabulary size is 1024, the embedding size is 512, and the
feedforward hidden size is 2048. We use a learning rate of $2e-4$ with ADAM
optimizer [23], a batch size of 32, and train for 300K training steps.

### Experiment details for robot learning

The robot workspace dimensions shown in Fig. 6 of the main paper are
approximately $1\text{m}\times 1\text{m}\times 0.4\text{m}$. The observations
are obtained from a overhead IntelRealsense camera, and we re-size the images
to be of dimensions $64\times 64\times 3$. The policy network takes as input
the image observation at time-step $t$, $o_{t}$ and encodes it with the
trained VQ-VAE encoder to a latent $z_{t}$ of dimension $32\times 32\times 1$.
Similarly, the goal image $o_{g}$ is encoded to $z_{g}$. The two embeddings
$z_{t}$ and $z_{g}$ are concatenated, and then there are fully connected
layers of dimensions $2048\times 256$, $256\times 64$, $64\times 4$, with ReLU
non-linearities. The action-space is 4 dimensional, and includes the different
$x,y,z$ position of the end-effector (i.e. how much to move from the current
position) and the final coordinate is for open/close of the gripper.

###  More Qualitative results

In this section, we show more generated affordances for our model for random
initial images, in Fig. 7. Visualization of robot exploration as training
progresses is in the project website
https://sites.google.com/andrew.cmu.edu/affordance-robotics/home along with an
explainer video of motivation, method, and results.

Figure 7: Qualitative analysis of the affordance model, for random initial
images. We choose $o_{1}$ to be a random unseen image, and show randomly
sampled affordances $o_{g}$. We see that the sampled affordances $o_{g}$ from
our model are diverse and correspond to plausible interactions in the scene.